Novel approaches for super resolution of intensity/range image using sparse representation (PhD)

Abstract

Super resolution (SR) aims to restore an approximated version of the original high resolution (HR) scene from the given low resolution (LR) image. As there could be different possibilities of HR scenes that can produce the same LR image, the process of SR is illposed in nature. In order to achieve an HR image, the inverse problem has to be regularized with prior knowledge about the HR image. Sparsity nducing norm can be used to address the regularization issue, but it can not take care about preserving edges, which are perceptually important in any image. To mitigate with this concern, we propose an edge preserving constraint that preserves the edges of the input image in the SR result in the framework of sparse representation. This constraint is useful to improve the quality of the result of SR for intensity images, and is further investigated for range image (or depth map). It is found that the edge preserving SR is well suited for this modality. This is because the resolution enhancement of range images is primarily gauged in terms of retention of object shape and inter-object discontinuities. Further, we address the issues of higher up-sampling as well as non-uniform up-sampling requirement for depth map. The non-uniform up-sampling requirement is caused by the sparse point cloud that is generated from structure from motion part in the pipeline of depth estimation. The sparse point cloud can be interpreted as a nonuniformly sampled LR depth map. To up-sample the non-uniformly sampled LR depth map, we generalize the SR framework using a mask operator. Here, the missing depths at HR grid is filled using the dictionary of exemplars in sparse domain. Dictionary plays an important role in the sparsity based SR. Often, the dictionary is learned using either structural information (dominant edge orientation) or statistical information (mean of intensity values) of image patches. The complimentary nature of both kind of information has not been explored, and an approach is proposed to address the same using example patches. The example patches are first clustered based on their dominant edge orientation to generate structurally similar clusters, which may vary statistically. Hence, the structurally similar clusters are further divided using K-means clustering to generate clusters that consist of structurally as well as statistically similar patches. This kind of clustering strategy can produce dictionaries that can represent the target patch appropriately. Availability of good HR example image patches are very important to learn the dictionaries. If the example patches are unavailable, one has to explore the information available in the given LR image. Thus, we construct image pyramid by up/down-sampling the given LR image, and patches from the pyramids can be used to learn dictionary. Further, we choose the image patch details for SR, as it contains perceptually significant information. Here, image patch details is computed by subtracting the patch from the non-local mean of similar patches. If the given LR image is contaminated by noise, considering patch detail for SR will emphasize the noise also. To mitigate with this issue, we derive few parameters from the given LR image that reflects the strength of noise present in the image. These parameters are used: i) to derive a threshold that is employed in the sparse coding stage using iterative shrinkage/thresholding algorithm, and ii) to choose between the noise suppressing non-local mean component and the detail component. By enhancing suitable component using iterative thresholding algorithm, we are able to suppress noise while super-resolving a single image. Hence, we do not require the strength and type of additive noise in super-resolving a noisy LR image. Further, the problem of distant face recognition is addressed by an edge based SR strategy, where edge information is employed either explicitly by super-resolving edge related information or implicitly by preserving edges using a constraint in SR of gray scale face image