Super-resolution: a comprehensive survey <s> 3D SR <s> The author derives an estimate-maximise (EM) formulation of a Bayesian super-resolution algorithm for reconstructing scattering cross sections from coherent images. He generalises this result to obtain an autofocus/super-resolution method, which simultaneously autofocusses an imaging system and super-resolves its image data. Autofocus/super-resolution might be applied to the interpretation of airborne synthetic aperture radar images that are subject to defocussing effects. > <s> BIB001 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> Using a set of low resolution images it is possible to reconstruct high resolution information by merging low resolution data on a finer grid. A 3D super-resolution algorithm is proposed, based on a probabilistic interpretation of the n-dimensional version of Papoulis' (1977) generalized sampling theorem. The algorithm is devised for recovering the albedo and the height map of a Lambertian surface in a Bayesian framework, using Markov random fields for modeling the a priori knowledge. <s> BIB002 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> Bayesian inference has been used successfully for many problems where the aim is to infer the parameters of a model of interest. In this paper we formulate the three dimensional reconstruction problem as the problem of inferring the parameters of a surface model from image data, and show how Bayesian methods can be used to estimate the parameters of this model given the image data. Thus we recover the three dimensional description of the scene. This approach also gives great flexibility. We can specify the geometrical properties of the model to suit our purpose, and can also use different models for how the surface reflects the light incident upon it. In common with other Bayesian inference problems, the estimation methodology requires that we can simulate the data that would have been recorded for any values of the model parameters. In this application this means that if we have image data we must be able to render the surface model. However it also means that we can infer the parameters of a model whose resolution can be chosen irrespective of the resolution of the images, and may be super-resolved. We present results of the inference of surface models from simulated aerial photographs for the case of super-resolution, where many surface elements project into a single pixel in the low-resolution images. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> Super-resolution is a technique to restore the detailed information from the degenerated data. Lots of previous work is for 2D images while super-resolution of 3D models was little addressed. This paper focuses on the super-resolution of 3D human faces. We firstly extend the 2D image pyramid model to the progressive resolution chain (PRC) model in 3D domain, to describe the detail variation during resolution decreasing. Then a consistent planar representation of 3D faces is presented, which enables the analysis and comparison among the features of the same facial part for the subsequent restoration process. Finally, formulated as solving an iterative quadratic system by maximizing a posteriori, a 3D restoration algorithm using PRC features is given. The experimental results on USF HnmanID 3D face database demonstrate tlie effectiveness of the proposed approach. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> This paper examines a simultaneous estimation of both high resolution (high accuracy) 3D information and high-resolution image from multiple low-resolution stereo-pair images. The proposed method is based on the MAP (maximum a posteriori) framework and these two relevant problems (estimation of both high-resolution 3 D and intensity images) are incorporated into a unified cost function. Then the solution is obtained iteratively by a fixed point algorithm, which yeilds 3 D information while the high-resolution image is fixed, and vice versa. Each solution can be obtained by solving a linear equation. It can be expected that the proposed method improves accuracy of both 3 D information and high-resolution image simultaneously. Some experimental results are shown to validate the proposed method. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> We present a new post-processing step to enhance the resolution of range images. Using one or two registered and potentially high-resolution color images as reference, we iteratively refine the input low-resolution range image, in terms of both its spatial resolution and depth precision. Evaluation using the Middlebury benchmark shows across-the-board improvement for sub-pixel accuracy. We also demonstrated its effectiveness for spatial resolution enhancement up to 100 times with a single reference image. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> Image analysis and enhancement tasks such as tone mapping, colorization, stereo depth, and photomontage, often require computing a solution (e.g., for exposure, chromaticity, disparity, labels) over the pixel grid. Computational and memory costs often require that a smaller solution be run over a downsampled image. Although general purpose upsampling methods can be used to interpolate the low resolution solution to the full resolution, these methods generally assume a smoothness prior for the interpolation. We demonstrate that in cases, such as those above, the available high resolution input image may be leveraged as a prior in the context of a joint bilateral upsampling procedure to produce a better high resolution solution. We show results for each of the applications above and compare them to traditional upsampling methods. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> We present a hybrid camera that combines the advantages of a high resolution camera and a high speed camera. Our hybrid camera consists of a pair of low-resolution high-speed (LRHS) cameras and a single high-resolution low-speed (HRLS) camera. The LRHS cameras are able to capture fast-motion with little motion blur. They also form a stereo pair and provide a low-resolution depth map. The HRLS camera provides a high spatial resolution but also introduces severe motion blur when capturing fast moving objects. We develop efficient algorithms to simultaneously motion-deblur the HRLS image and reconstruct a high resolution depth map. Our method estimates the motion flow in the LRHS pair and then warps the flow field to the HRLS camera to estimate the point spread function (PSF). We then deblur the HRLS image and use the resulting image to enhance the low-resolution depth map using joint bilateral filters. We demonstrate the hybrid camera in depth map super-resolution and motion deblurring with spatially varying kernels. Experiments show that our framework is robust and highly effective. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> This paper focuses on hallucinating a facial shape from a low-resolution 3D facial shape. Firstly, we give a constrained conformal embedding of 3D shape in R2, which establishes an isomorphic mapping between curved facial surface and 2D planar domain. With such conformal embedding, two planar representations of 3D shapes are proposed: Gaussian curvature image (GCI) for a facial surface, and surface displacement image (SDI) for a pair of facial surfaces. The conformal planar representation reduces the data complexity from 3D irregular curved surface to 2D regular grid while preserving the necessary information for hallucination. Then, hallucinating a low resolution facial shape is formalized as inference of SDI from GCIs by modeling the relationship between GCI and SDI by RBF regression. The experiments on USF HumanID 3D face database demonstrate the effectiveness of the approach. Our method can be easily extended to hallucinate those category-specific 3D surfaces sharing with similar geometric structures. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> This paper proposes a face hallucination method for the reconstruction of high-resolution facial images from single-frame, low-resolution facial images. The proposed method has been derived from example-based hallucination methods and morphable face models. First, we propose a recursive error back-projection method to compensate for residual errors, and a region-based reconstruction method to preserve characteristics of local facial regions. Then, we define an extended morphable face model, in which an extended face is composed of the interpolated high-resolution face from a given low-resolution face, and its original high-resolution equivalent. Then, the extended face is separated into an extended shape and an extended texture. We performed various hallucination experiments using the MPI, XM2VTS, and KF databases, compared the reconstruction errors, structural similarity index, and recognition rates, and showed the effects of face detection errors and shape estimation errors. The encouraging results demonstrate that the proposed methods can improve the performance of face recognition systems. Especially the proposed method can enhance the resolution of single-frame, low-resolution facial images. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> In this paper we propose a novel way of using time-of-flight camera depth and intensity images to produce a higher resolution depth image with prior knowledge of spatial noise distribution, which is correlated with the incident light falling on each pixel. The proposed method is compared to well-established methods, and results with real image data are presented. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> Depth maps captured with time-of-flight cameras have very low data quality: the image resolution is rather limited and the level of random noise contained in the depth maps is very high. Therefore, such flash lidars cannot be used out of the box for high-quality 3D object scanning. To solve this problem, we present LidarBoost, a 3D depth superresolution method that combines several low resolution noisy depth images of a static scene from slightly displaced viewpoints, and merges them into a high-resolution depth image. We have developed an optimization framework that uses a data fidelity term and a geometry prior term that is tailored to the specific characteristics of flash lidars. We demonstrate both visually and quantitatively that LidarBoost produces better results than previous methods from the literature. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> This paper presents an artifact-free super resolution texture mapping from multiple-view images. The multiple-view images are upscaled with a learning-based super resolution technique and are mapped onto a 3D mesh model. However, mapping multiple-view images onto a 3D model is not an easy task, because artifacts may appear when different upscaled images are mapped onto neighboring meshes. We define a cost function that becomes large when artifacts appear on neighboring meshes, and our method seeks the image-and mesh assignment that minimizes the cost function. Experimental results with real images demonstrate the effectiveness of our method. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> We present an integrated method for post-processing of range data which removes outliers, smoothes the depth values and enhances the lateral resolution in order to achieve visually pleasing 3D models from low-cost depth sensors with additional (registered) color images. The algorithm is based on the non-local principle and adapts the original NL-Means formulation to the characteristics of typical depth data. Explicitly handling outliers in the sensor data, our denoising approach achieves unbiased reconstructions from error-prone input data. Taking intra-patch similarity into account, we reconstruct strong discontinuities without disturbing artifacts and preserve fine detail structures, obtaining piece-wise smooth depth maps. Furthermore, we exploit the dependencies of the depth data with additionally available color information and increase the lateral resolution of the depth maps. We finally discuss how to parallelize the algorithm in order to achieve fast processing times that are adequate for post-processing of data from fast depth sensors such as time-of-flight cameras. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> This paper presents a Markov Random Field (MRF)-based approach for depth map super-resolution and enhancement. Given a low-resolution or moderate quality depth map, we study the problem of enhancing its resolution or quality with a registered high-resolution color image. Different from the previous methods, this MRF-based approach is based on a novel data term formulation that fits well to the unique characteristics of depth maps. We also discuss a few important design choices that boost the performance of general MRF-based methods. Experimental results show that our proposed approach achieves high-resolution depth maps at more desirable quality, both qualitatively and quantitatively. It can also be applied to enhance the depth maps derived with state-of-the-art stereo methods, resulting in the raised ranking based on the Middlebury benchmark. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> The fast development of time-of-flight (ToF) cameras in recent years enables capture of high frame-rate 3D depth maps of moving objects. However, the resolution of depth map captured by ToF is rather limited, and thus it cannot be directly used to build a high quality 3D model. In order to handle this problem, we propose a novel joint example-based depth map super-resolution method, which converts a low resolution depth map to a high resolution depth map, using a registered high resolution color image as a reference. Different from previous depth map SR methods without training stage, we learn a mapping function from a set of training samples and enhance the resolution of the depth map via sparse coding algorithm. We further use a reconstruction constraint to make object edges sharper. Experimental results show that our method outperforms state-of-the-art methods for depth map super-resolution. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> 3D SR <s> A new method to up-sample low-resolution depth maps to high quality and high resolution is proposed. Range sensors such as time-of-flight cameras yield low-resolution depth maps and the output includes heavy noise at the edges of objects. Recently, many techniques have been proposed to up-sample low-resolution depth maps. However, there is no effective countermeasure to the edge noise problem. The proposed up-sampling method is based on Markov random fields and addresses this edge noise using newly designed confidence weights. The performance of our method is evaluated using error rate and mean absolute error through comparison with existing methods, and results show that the proposed method outperforms conventional methods. <s> BIB017
HR 3D sensors are usually expensive, and the available 3D depth sensors like time of flight (TOF) cameras usually produce depth images of LR; therefore, SR is of great importance for working with depth images. Much research on 3D SR has been reported BIB001 BIB002 33, BIB003   BIB008 BIB009 BIB013 BIB014 BIB016 . Most of these methods have used 2D observations to extract the depth information. However, in BIB004  BIB005 BIB006  BIB009 BIB011 BIB012 BIB016 3D-3D SR algorithms have been developed to generate HR 3D images from LR: -3D inputs using a hallucination algorithm BIB009 BIB010 . -Stereo pair images BIB005 . -Images captured by a TOF camera using a MAP SR algorithm BIB011 BIB012 . To do so, e.g., in BIB009 two planer representations have been used: Gaussian curvature image (GCI) and surface displacement image (SDI). GCI and SDI model the intrinsic geometric information of the LR input meshes and the relationship between these LR meshes and their HR counterparts, respectively. Then, the hallucination process obtains SDI from GCI, which has been done by RBF networks. In BIB010 , the result of a hallucination algorithm has been applied to a morphable face model to extract a frontal HR face image from the LR inputs. Another group of 3D SR algorithms, a.k.a joint image upsampling, works with a LR depth map and a HR image to super-resolve a HR depth image, like in BIB007 BIB015 BIB017 . This technique has been well applied to aerial imaging, in which for example a HR panchromatic image and a LR multi-spectral image are used to produce a HR multi-spectral image. This method in aerial applications is known as pansharpening, like in [54,114,159,604].