Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> Abstract Accurate computation of image motion enables the enhancement of image sequences. In scenes having multiple moving objects the motion computation is performed together with object segmentation by using a unique temporal integration approach. After the motion for the different image regions is computed, these regions can be enhanced by fusing several successive frames covering the same region. Enhancements treated here include improvement of image resolution, filling-in occluded regions, and reconstruction of transparent objects. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> We propose a Bayesian approach to image hallucination. Given a generic low resolution image, we hallucinate a high resolution image using a set of training images. Our work is inspired by recent progress on natural image statistics that the priors of image primitives can be well represented by examples. Specifically, primal sketch priors (e.g., edges, ridges and corners) are constructed and used to enhance the quality of the hallucinated high resolution image. Moreover, a contour smoothness constraint enforces consistency of primitives in the hallucinated image by a Markov-chain based inference algorithm. A reconstruction constraint is also applied to further improve the quality of the hallucinated image. Experiments demonstrate that our approach can hallucinate high quality super-resolution images. <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> We propose a single-frame, learning-based super-resolution restoration technique by using the wavelet domain to define a constraint on the solution. Wavelet coefficients at finer scales of the unknown high-resolution image are learned from a set of high-resolution training images and the learned image in the wavelet domain is used for further regularization while super-resolving the picture. We use an appropriate smoothness prior with discontinuity preservation in addition to the wavelet-based constraint to estimate the super-resolved image. The smoothness term ensures the spatial correlation among the pixels, whereas the learning term chooses the best edges from the training set. Because this amounts to extrapolating the high-frequency components, the proposed method does not suffer from oversmoothing effects. The results demonstrate the effectiveness of the proposed approach. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol 14, 105–112, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.20013 <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> We develop a framework for learning generic, expressive image priors that capture the statistics of natural scenes and can be used for a variety of machine vision tasks. The approach extends traditional Markov random field (MRF) models by learning potential functions over extended pixel neighborhoods. Field potentials are modeled using a Products-of-Experts framework that exploits nonlinear functions of many linear filter responses. In contrast to previous MRF approaches all parameters, including the linear filters themselves, are learned from training data. We demonstrate the capabilities of this Field of Experts model with two example applications, image denoising and image inpainting, which are implemented using a simple, approximate inference scheme. While the model is trained on a generic image database and is not tuned toward a specific application, we obtain results that compete with and even outperform specialized techniques. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> We propose a learning-based, single-image super-resolution reconstruction technique using the contourlet transform, which is capable of capturing the smoothness along contoursmaking use of directional decompositions. The contourlet coefficients at finer scales of the unknown high-resolution image are learned locally from a set of high-resolution training images, the inverse contourlet transform of which recovers the super-resolved image. In effect, we learn the high-resolution representation of an oriented edge primitive from the training data. Our experiments show that the proposed approach outperforms standard interpolation techniques as well as a standard (Cartesian) wavelet-based learning both visually and in terms of the PSNR values, especially for images with arbitrarily oriented edges. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> Effective image prior is necessary for image super resolution, due to its severely under-determined nature. Although the edge smoothness prior can be effective, it is generally difficult to have analytical forms to evaluate the edge smoothness, especially for soft edges that exhibit gradual intensity transitions. This paper finds the connection between the soft edge smoothness and a soft cut metric on an image grid by generalizing the Geocuts method (Y. Boykov and V. Kolmogorov, 2003), and proves that the soft edge smoothness measure approximates the average length of all level lines in an intensity image. This new finding not only leads to an analytical characterization of the soft edge smoothness prior, but also gives an intuitive geometric explanation. Regularizing the super resolution problem by this new form of prior can simultaneously minimize the length of all level lines, and thus resulting in visually appealing results. In addition, this paper presents a novel combination of this soft edge smoothness prior and the alpha matting technique for color image super resolution, by normalizing edge segments with their alpha channel description, to achieve a unified treatment of edges with different contrast and scale. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> In this paper, we propose a novel learning-based method for image hallucination, with image super-resolution being a specific application that we focus on here. Given a low-resolution image, its underlying higher-resolution details are synthesized based on a set of training images. In order to build a compact yet descriptive training set, we investigate the characteristic local structures contained in large volumes of small image patches. Inspired by progress in manifold learning research, we take the assumption that small image patches in the low-resolution and high-resolution images form manifolds with similar local geometry in the corresponding image feature spaces. This assumption leads to a super-resolution approach which reconstructs the feature vector corresponding to an image patch by its neighbors in the feature space. In addition, the residual errors associated with the reconstructed image patches are also estimated to compensate for the information loss in the local averaging process. Experimental results show that our hallucination method can synthesize higher-quality images compared with other methods. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> In this paper we propose a new method for upsampling images which is capable of generating sharp edges with reduced input-resolution grid-related artifacts. The method is based on a statistical edge dependency relating certain edge features of two different resolutions, which is generically exhibited by real-world images. While other solutions assume some form of smoothness, we rely on this distinctive edge dependency as our prior knowledge in order to increase image resolution. In addition to this relation we require that intensities are conserved; the output image must be identical to the input image when downsampled to the original resolution. Altogether the method consists of solving a constrained optimization problem, attempting to impose the correct edge relation and conserve local intensities with respect to the low-resolution input image. Results demonstrate the visual importance of having such edge features properly matched, and the method's capability to produce images in which sharp edges are successfully reconstructed. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> A learning-based super-resolution system consisting of training and synthesis processes is presented. In the proposed system, a multiresolution wavelet approach is applied to carry out the robust synthesis of both the global geometric structure and the local high-frequency detailed features of a facial image. In the training process, the input image is transformed into a series of images of increasingly lower resolution using the Haar discrete wavelet transform (DWT). The images at each resolution level are divided into patches, which are then projected onto an eigenspace to derive the corresponding projection weight vectors. In the synthesis process, a low-resolution input image is divided into patches, which are then projected onto the same eigenspace as that used in the training process. Modeling the resulting projection weight vectors as a Markov network, the maximum a posteriori (MAP) estimation approach is then applied to identity the best-matching patches with which to reconstruct the image at a higher level of resolution. The experimental results demonstrate that the proposed reconstruction system yields better results than the bi-cubic spline interpolation method. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> High resolution image reconstruction is an image process to reconstruct a high resolution image from a set of blurred, degraded and shifted low resolution images. In this paper, the reconstruction problem is treated as a function approximation. We use linear interpolation to build up an algorithm to obtain the relationship between the detail coefficients in wavelet subbands and the set of low resolution images. We use Haar wavelet as an example and establish the connection between the Haar wavelet subband and the low resolution images. Experiments show that we can use just 3 low resolution images to obtain a high resolution image which has better quality than Tikhonov least-squares approach and Chan et al. Algorithm 3 in low noise cases. We also propose an error correction extension for our method which can lead to very good results even in noisy cases. Moreover, our approach is very simple to implement and very efficient. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> In this paper, a novel algorithm for single image super resolution is proposed. Back-projection [1] can minimize the reconstruction error with an efficient iterative procedure. Although it can produce visually appealing result, this method suffers from the chessboard effect and ringing effect, especially along strong edges. The underlining reason is that there is no edge guidance in the error correction process. Bilateral filtering can achieve edge-preserving image smoothing by adding the extra information from the feature domain. The basic idea is to do the smoothing on the pixels which are nearby both in space domain and in feature domain. The proposed bilateral back-projection algorithm strives to integrate the bilateral filtering into the back-projection method. In our approach, the back-projection process can be guided by the edge information to avoid across-edge smoothing, thus the chessboard effect and ringing effect along image edges are removed. Promising results can be obtained by the proposed bilateral back-projection method efficiently. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> We propose a simple but effective upsampling method for automatically enhancing the image/video resolution, while preserving the essential structural information. The main advantage of our method lies in a feedback-control framework which faithfully recovers the high-resolution image information from the input data, without imposing additional local structure constraints learned from other examples. This makes our method independent of the quality and number of the selected examples, which are issues typical of learning-based algorithms, while producing high-quality results without observable unsightly artifacts. Another advantage is that our method naturally extends to video upsampling, where the temporal coherence is maintained automatically. Finally, our method runs very fast. We demonstrate the effectiveness of our algorithm by experimenting with different image/video data. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> In this paper, we propose an image super-resolution approach using a novel generic image prior - gradient profile prior, which is a parametric prior describing the shape and the sharpness of the image gradients. Using the gradient profile prior learned from a large number of natural images, we can provide a constraint on image gradients when we estimate a hi-resolution image from a low-resolution image. With this simple but very effective prior, we are able to produce state-of-the-art results. The reconstructed hi-resolution image is sharp while has rare ringing or jaggy artifacts. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> In learning based single image super-resolution(SR) approach, the super-resolved image are usually found or combined from training database through patch matching. But because the representation ability of small patch is limited, it is difficult to guarantee that the super-resolved image is best under global view. To tackle this problem, we propose a statistical learning method for SR with both global and local constraints. Firstly, we use Maximum a Posteriori (MAP) estimation with learned image priors by Fields of Experts (FoE) model, and regularize SR globally guided by the image priors. Secondly, for each overlapped patch, the higher-order Markov random fields (MRFs) is used to model its local relationship with corresponding high-resolution candidates, then belief propagation is used to find high-resolution image. Compared with traditional patch based learning method without global constraint, our method could not only preserve the global image structure, but also restore the local details well. Experiments verify the idea of our global and local constraint SR method. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> Designing effective image priors is of great interest to image super-resolution (SR), which is a severely under-determined problem. An edge smoothness prior is favored since it is able to suppress the jagged edge artifact effectively. However, for soft image edges with gradual intensity transitions, it is generally difficult to obtain analytical forms for evaluating their smoothness. This paper characterizes soft edge smoothness based on a novel SoftCuts metric by generalizing the Geocuts method . The proposed soft edge smoothness measure can approximate the average length of all level lines in an intensity image. Thus, the total length of all level lines can be minimized effectively by integrating this new form of prior. In addition, this paper presents a novel combination of this soft edge smoothness prior and the alpha matting technique for color image SR, by adaptively normalizing image edges according to their alpha-channel description. This leads to the adaptive SoftCuts algorithm, which represents a unified treatment of edges with different contrasts and scales. Experimental results are presented which demonstrate the effectiveness of the proposed method. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> This paper addresses the super-resolution problem for low quality cartoon videos widely distributed on the web, which are generated by downsampling and compression from the sources. To effectively eliminate the compression artifacts and meanwhile preserve the visually salient primitive components (e.g., edges, ridges and corners), we propose an adaptive regularization method depending on the degradation grade of each frame, followed by learning-based pair matching to further enhance the primitives in the upsampled frames. In addition, temporal consistency is considered a directive constraint in both the regularization and enhancement processes. Experimental results demonstrate our solution achieves a good balance between artifacts removal and primitive enhancement, providing perceptually high quality super-resolution results for various web cartoon videos. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> Image upsampling is a common yet challenging task, since it is severely underconstrained. While considerable progress was made in preserving the sharpness of salient edges, current methods fail to reproduce the fine detail typically present in the textured regions bounded by these edges, resulting in unrealistic appearance. In this paper we address this fundamental shortcoming by integrating higher-level image analysis and custom low-level image synthesis. Our approach extends and refines the patch-based image model of Freeman et al. [10] and interprets the image as a tiling of distinct textures, each of which is matched to an example in a database of relevant textures. The matching is not done at the patch level, but rather collectively, over entire segments. Following this model fitting stage, which requires some user guidance, a higher-resolution image is synthesized using a hybrid approach that incorporates principles from example-based texture synthesis. We show that for images that comply with our model, our method is able to reintroduce consistent fine-scale detail, resulting in enhanced appearance textured regions. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> We present a novel approach for online shrinkage functions learning in single image super-resolution. The proposed approach leverages the classical Wavelet Shrinkage denoising technique where a set of scalar shrinkage functions is applied to the wavelet coefficients of a noisy image. In the proposed approach, a unique set of learned shrinkage functions is applied to the overcomplete representation coefficients of the interpolated input image. The super-resolution image is reconstructed from the post-shrinkage coefficients. During the learning stage, the lowresolution input image is treated as a reference high-resolution image and a super-resolution reconstruction process is applied to a scaled-down version of it. The shapes of all shrinkage functions are jointly learned by solving a Least Squares optimization problem that minimizes the sum of squared errors between the reference image and its super-resolution approximation. Computer simulations demonstrate superior performance compared to state-of-the-art results. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> In this paper, we propose a new learning-based approach for super-resolving an image captured at low spatial resolution. Given the low spatial resolution test image and a database consisting of low and high spatial resolution images, we obtain super-resolution for the test image. We first obtain an initial high-resolution (HR) estimate by learning the high-frequency details from the available database. A new discrete wavelet transform (DWT) based approach is proposed for learning that uses a set of low-resolution (LR) images and their corresponding HR versions. Since the super-resolution is an ill-posed problem, we obtain the final solution using a regularization framework. The LR image is modeled as the aliased and noisy version of the corresponding HR image, and the aliasing matrix entries are estimated using the test image and the initial HR estimate. The prior model for the super-resolved image is chosen as an Inhomogeneous Gaussian Markov random field (IGMRF) and the model parameters are estimated using the same initial HR estimate. A maximum a posteriori (MAP) estimation is used to arrive at the cost function which is minimized using a simple gradient descent approach. We demonstrate the effectiveness of the proposed approach by conducting the experiments on gray scale as well as on color images. The method is compared with the standard interpolation technique and also with existing learning-based approaches. The proposed approach can be used in applications such as wildlife sensor networks, remote surveillance where the memory, the transmission bandwidth, and the camera cost are the main constraints. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> Pansharpening is a technique that fuses a low resolution multispectral image and a high resolution panchromatic image, to obtain a multispectral image with the spatial resolution and quality of the panchromatic image while preserving spectral information of the multispectral image. In this paper, we present a new pansharpening method based on super-resolution and contourlet transform. The experimental results show that the proposed method not only enhances the spatial resolution of the pansharpened image, but also preserves the spectral information of the original multispectral image. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> The problem of creating artifact-free upscaled images appearing sharp and natural to the human observer is probably more interesting and less trivial than it may appear. The solution to the problem, often referred to also as “single-image super-resolution,” is related both to the statistical relationship between low-resolution and high-resolution image sampling and to the human perception of image quality. In many practical applications, simple linear or cubic interpolation algorithms are applied for this task, but the results obtained are not really satisfactory, being affected by relevant artifacts like blurring and jaggies. Several methods have been proposed to obtain better results, involving simple heuristics, edge modeling, or statistical learning. The most powerful ones, however, present a high computational complexity and are not suitable for real-time applications, while fast methods, even if edge adaptive, are not able to provide artifacts-free images. In this paper, we describe a new upscaling method (iterative curvature-based interpolation) based on a two-step grid filling and an iterative correction of the interpolated pixels obtained by minimizing an objective function depending on the second-order directional derivatives of the image intensity. We show that the constraints used to derive the function are related with those applied in another well-known interpolation method, providing good results but computationally heavy (i.e., new edge-directed interpolation (NEDI). The high quality of the images enlarged with the new method is demonstrated with objective and subjective tests, while the computation time is reduced of one to two orders of magnitude with respect to NEDI so that we were able, using a graphics processing unit implementation based on the nVidia Compute Unified Device Architecture technology, to obtain real-time performances. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> In this paper, we propose a novel generic image prior-gradient profile prior, which implies the prior knowledge of natural image gradients. In this prior, the image gradients are represented by gradient profiles, which are 1-D profiles of gradient magnitudes perpendicular to image structures. We model the gradient profiles by a parametric gradient profile model. Using this model, the prior knowledge of the gradient profiles are learned from a large collection of natural images, which are called gradient profile prior. Based on this prior, we propose a gradient field transformation to constrain the gradient fields of the high resolution image and the enhanced image when performing single image super-resolution and sharpness enhancement. With this simple but very effective approach, we are able to produce state-of-the-art results. The reconstructed high resolution images or the enhanced images are sharp while have rare ringing or jaggy artifacts. <s> BIB022 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> We present a self-learning approach for single image super-resolution (SR), with the ability to preserve high frequency components such as edges in resulting high resolution (HR) images. Given a low-resolution (LR) input image, we construct its image pyramid and produce a super pixel dataset. By extracting context information from the super-pixels, we propose to deploy context-specific contour let transform on them in order to model the relationship (via support vector regression) between the input patches and their associated directional high-frequency responses. These learned models are applied to predict the SR output with satisfactory quality. Unlike prior learning-based SR methods, our approach advances a self-learning technique and does not require the self similarity of image patches within or across image scales. More importantly, we do not need to collect training LR/HR image data in advance and only require a single LR input image. Empirical results verify the effectiveness of our approach, which quantitatively and qualitatively outperforms existing interpolation or learning-based SR methods. <s> BIB023 </s> Super-resolution: a comprehensive survey <s> Reconstruction-based single-image SR algorithms <s> In recent years, super-resolution techniques in the field of computer vision have been studied actively owing to the potential applicability in various fields. In this paper, we propose a single-image, super-resolution approach using GMM (Gaussian Mixture Model)-based conversion. The conversion function is constructed by GMM using the input image and its self-reduction image. The high-resolution image is obtained by applying the conversion function to the enlarged input image without any outside database. We confirmed the effectiveness of this proposed method through the experiments. <s> BIB024
These algorithms BIB002 BIB004 BIB006 BIB007 BIB008 BIB012 BIB013 BIB014 BIB015 BIB016 BIB017  BIB021 BIB022 similar to their peer multiple imagebased SR algorithms try to address the aliasing artifacts that are present in the LR input image. These algorithms can be classified into the following three groups. Primal sketches The idea of a class-based a priori, which is used in most of the hallucination-based algorithms, mostly for the class of face images, was extended to generic a prioris in BIB002 BIB007 BIB016  , where primal sketches were used as the a priori. The key point in primal sketches is that the hallucination algorithm is applied only to the primitives (like edges, ridges, corners, T-junctions, and terminations) but not to the non-primitive parts of the image. This is because the a priori term related to the primitives can be learned but not those for the non-primitives. Having an LR input image they BIB002 first interpolate (using bicubic interpolation) it to the target resolution, then for every primitive point (basically every point on the contours inside the image) a 9 × 9 patch is considered. Then, based on the primal sketch prior, and using a Markov chain inference, the corresponding HR patch for every LR patch is found and replaced. This step actually hallucinates the high-frequency counterparts of the primitives. This hallucinated image is then used as the starting point for the IBP algorithm of BIB001 to produce an HR image. Gradient profile Using the fact that the shape statistics of the gradient profiles in natural images is robust against changes in image resolution, a gradient profile prior has been introduced in BIB013 BIB022 . Similar to the previous approaches that learn the relationship between the LR and HR images, gradient profile-based methods learn the similarity between the shape statistics of the LR and HR images in a training step. This learned information will be used to apply a gradientbased constraint to the reconstruction process. The distribution of this gradient profile prior is defined by a general exponential family distribution [generalized Gaussian distribution (GGD)] BIB013 BIB022 : where Γ is the Gamma function and α = Γ ( is the improvement factor. An interesting property of α is that it makes the second moments of the above GGD equal to σ 2 . This means that the second order of this prior can be used for estimating σ . Finally, λ controls the shape of the distribution BIB013 BIB022 . Fields of experts Fields of experts is an a priori for learning the heavy non-Gaussian statistics of natural images BIB004 BIB014 . Here usually contrastive divergence is used to learn a set of filters, J , from a training database. Having learned these filters or bases, the a priori term is represented as: where is the number of cliques, N is the number of experts, and the αs are positive parameters which make the experts have proper distributions. The experts, φ i , are defined by: Other reported reconstruction-based a prioris in this group include: -Wavelet based BIB003 BIB009 BIB010 BIB018 BIB019 . -Contourlet transform based BIB005 BIB020 BIB023 . -Bilateral filters used for edge preserving in an IBP method in BIB011 for dealing with the chessboard and ringing effects of IBP. -Gaussian mixture model-based methods BIB024 .