Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In several applications it is required to reconstruct a high-resolution noise-free image from multipath frames of undersampled low-resolution noisy images. Using the aliasing relationship between the undersamples frames and the reference image, an algorithm based on weighted recursive least-squares theory is developed in the wavenumber domain. This algorithm is efficient because interpolation and noise removal are performed recursively, and is highly suitable for implementation via the massively parallel computational architectures currently available. Success in the use of the algorithm is demonstrated through various simulated examples. > <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> The authors address the problem of reconstruction of a high-resolution image from a number of lower-resolution (possibly noisy) frames of the same scene where the successive frames are uniformly based versions of each other at subpixel displacements. In particular, two previously proposed methods, a frequency-domain method and a method based on projections onto convex sets (POCSs), are extended to take into account the presence of both sensor blurring and observation noise. A new two-step procedure is proposed, and it is shown that the POCS formulation presented for the high-resolution image reconstruction problem can also be used as a new method for the restoration of spatially invariant blurred images. Some simulation results are provided. > <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> A theoretical analysis is proposed to evaluate the robustness of the total least square (TLS) algorithm for image reconstruction from a sequence of undersampled noisy and blurred frames in the wavenumber domain. Since the data samples in the wavenumber domain are complex-valued, the results from the TLS theory developed for the case of real-valued data are adapted to the situation at hand. It is shown that the image quality can be improved as more frames become available. In the case of blurred frames, higher resolution images may be reconstructed using the TLS algorithm with post-deblurring. Finally, computer simulation results are provided to demonstrate the robustness of the TLS algorithm for image reconstruction. > <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution is a technique that produces higher resolution images from low resolution images (LRIs). In practice, the improvement in resolution is limited. The aim of this paper is to address the problem of whether fundamental limits exist for super-resolution? Specifically, this paper provides explicit limits for a major class of super-resolution algorithms, called reconstruction-based algorithms, under both real and synthetic conditions. Our analysis is based on perturbation theory of linear systems. We also show that a sufficient number of LRIs can be determined to reach the limit. Both real and synthetic experiments are carried out to verify our analysis. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> We consider the estimation of the unknown hyperparameters for the problem of reconstructing a high-resolution image from multiple undersampled, shifted, degraded frames with subpixel displacement errors. We derive mathematical expressions for the iterative calculation of the maximum likelihood estimate (MLE) of the unknown hyperparameters given the low resolution observed images. Experimental results are presented for evaluating the accuracy of the proposed method. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution aims to produce a high-resolution image from a set of one or more low-resolution images by recovering or inventing plausible high-frequency image content. Typical approaches try to reconstruct a high-resolution image using the sub-pixel displacements of several low-resolution images, usually regularized by a generic smoothness prior over the high-resolution image space. Other methods use training data to learn low-to-high-resolution matches, and have been highly successful even in the single-input-image case. Here we present a domain-specific image prior in the form of a p.d.f. based upon sampled images, and show that for certain types of super-resolution problems, this sample-based prior gives a significant improvement over other common multiple-image super-resolution techniques. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Superresolution is a technique that can produce images of a higher resolution than that of the originally captured ones. Nevertheless, improvement in resolution using such a technique is very limited in practice. This makes it significant to study the problem: "Do fundamental limits exist for Superresolution?" In this paper, we focus on a major class of superresolution algorithms, called the reconstruction-based algorithms, which compute high-resolution images by simulating the image formation process. Assuming local translation among low-resolution images, this paper is the first attempt to determine the explicit limits of reconstruction-based algorithms, under both real and synthetic conditions. Based on the perturbation theory of linear systems, we obtain the superresolution limits from the conditioning analysis of the coefficient matrix. Moreover, we determine the number of low-resolution images that are sufficient to achieve the limit. Both real and synthetic experiments are carried out to verify our analysis. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Recently, there has been much work developing super-resolution algorithms for combining a set of low quality images to produce a set of higher quality images. In most cases, such algorithms must first register the collection of images to a common sampling grid and then reconstruct the high resolution image. While many such algorithms have been proposed to address each one of these subproblems, no work has addressed the overall performance limits for this joint estimation problem. In this paper, we analyze the performance limits from statistical first principles using the Cramer-Rao bound. We offer insight into the fundamental bottlenecks limiting the performance of multiframe image reconstruction algorithms and hence super-resolution. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a novel method for solving single-image super-resolution problems. Given a low-resolution image as input, we recover its high-resolution counterpart using a set of training examples. While this formulation resembles other learning-based methods for super-resolution, our method has been inspired by recent manifold teaming methods, particularly locally linear embedding (LLE). Specifically, small image patches in the lowand high-resolution images form manifolds with similar local geometry in two distinct feature spaces. As in LLE, local geometry is characterized by how a feature vector corresponding to a patch can be reconstructed by its neighbors in the feature space. Besides using the training image pairs to estimate the high-resolution embedding, we also enforce local compatibility and smoothness constraints between patches in the target high-resolution image through overlapping. Experiments show that our method is very flexible and gives good empirical results. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution has been applied mainly to grayscale images, but producing a high-resolution color image using a single CCD sensor has not been investigated in detail until recently. This work aims at producing a high-resolution color image directly from raw "color mosaic" images obtained by a single CCD equipped with a color filter array. This method is based on a generalized formulation of super-resolution that simultaneously performs both resolution enhancement and demosaicing. The other key factor of our topic is a precise sub-pixel registration of multiple raw images. We examined direct registration of raw images based on an imaging model, which shows precise estimation of motion among severely aliased raw images. The proposed method is verified through experiments using synthetic and real images. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> We propose a single-frame, learning-based super-resolution restoration technique by using the wavelet domain to define a constraint on the solution. Wavelet coefficients at finer scales of the unknown high-resolution image are learned from a set of high-resolution training images and the learned image in the wavelet domain is used for further regularization while super-resolving the picture. We use an appropriate smoothness prior with discontinuity preservation in addition to the wavelet-based constraint to estimate the super-resolved image. The smoothness term ensures the spatial correlation among the pixels, whereas the learning term chooses the best edges from the training set. Because this amounts to extrapolating the high-frequency components, the proposed method does not suffer from oversmoothing effects. The results demonstrate the effectiveness of the proposed approach. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol 14, 105–112, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.20013 <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Superresolution is the process of producing a high resolution image from a collection of low resolution images. This process has potential application in a wide spectrum of fields in which navigation, surveillance, and observation are important, yet in which target images have limited resolution. There have been numerous methods proposed and developed to implement superresolution, each with its own advantages and limitations. However, there is no standard method or software for superresolution. In this paper a genetic algorithm solution for determining the registration and point spread function (PSF) parameters for superresolution is proposed and implemented, and a superresolved image is generated using genetic algorithm optimization of an existing superresolution method. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Wavelet coefficient thresholding is effective in reducing spatial domain noise in wavelet-based super-resolution algorithms. Here, the effect of the threshold level on reconstructed image quality in second-generation wavelet super-resolution is investigated. The choice of optimal threshold involves a tradeoff between noise filtering and blurring introduced by thresholding. A measure based on the singular values of the image matrix is employed as a reliable gauge of generated high-resolution image quality. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we study the problem of reconstructing a high-resolution image from several blurred low-resolution image frames. The image frames consist of decimated, blurred and noisy versions of the high-resolution image. The high-resolution image is modeled as a Markov random field (MRF), and a maximum a posteriori (MAP) estimation technique is used for the restoration. We show that with the periodic boundary condition, the high-resolution image can be restored efficiently by using fast Fourier transforms. We also apply the preconditioned conjugate gradient method to restore the high-resolution image. Computer simulations are given to illustrate the effectiveness of the proposed method. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution is the problem of generating one or a set of high-resolution images from one or a sequence of low-resolution frames. Most methods have been proposed for super-resolution based on multiple low resolution images of the same scene, which is called multiple-frame super-resolution. Only a few approaches produce a high-resolution image from a single low-resolution image, with the help of one or a set of training images from scenes of the same or different types. It is referred to as single-frame super-resolution. This article reviews a variety of single-frame super-resolution methods proposed in the recent years. In the paper, a new manifold learning method: locally linear embedding (LLE) and its relation with single-frame super-resolution is introduced. Detailed study of a critical issue: "neighborhood issue" is presented with related experimental results and analysis and possible future research is given. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we consider the super-resolution image reconstruction problem. We propose a Markov chain Monte Carlo (MCMC) approach to find the maximum a posterior probability (MAP) estimation of the unknown high-resolution image. Firstly, Gaussian Markov random field (GMRF) is exploited for modeling the prior probability distribution of the unknown high-resolution image. Then, a MCMC technique (in particular, the Gibbs sampler) is introduced to generate samples from the posterior probability distribution to compute the MAP estimation of the unknown high-resolution image, which is obtained as the mean of the samples. Moreover, we derive a bound on the convergence time of the proposed MCMC approach. Finally, the experimental results are presented to verify the superior performance of the proposed approach and the validity of the proposed bound. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution imaging is to overcome the inherent limitations of image acquisition to create high-resolution images from their low-resolution counterparts. In this paper, a novel state-space approach is proposed to incorporate the temporal correlations among the low-resolution observations into the framework of the Kalman filtering. The proposed approach exploits both the temporal correlations information among the high-resolution images and the temporal correlations information among the low-resolution images to improve the quality of the reconstructed high-resolution sequence. Experimental results show that the proposed framework is superior to bi-linear interpolation, bi-cubic spline interpolation and the conventional Kalman filter approach, due to the consideration of the temporal correlations among the low-resolution images. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> We propose a new solution to the problem of obtaining a single high-resolution image from multiple blurred, noisy, and undersampled images. Our estimator, derived using the Bayesian stochastic framework, is novel in that it employs a new hierarchical non-stationary image prior. This prior adapts the restoration of the super-resolved image to the local spatial statistics of the image. Numerical experiments demonstrate the effectiveness of the proposed approach. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a robust approach to super-resolution static sprite generation from multiple low-resolution images. Considering both short-term and long-term motion influences, a hybrid global motion estimation technique is first presented for sprite generation. An iterative super-resolution reconstruction algorithm is then proposed for the super-resolution sprite construction. This algorithm is robust to outliers and results in a sprite image with high visual quality. In addition, the generated super-resolution sprite can provide better reconstructed images than a conventional low-resolution sprite. Experimental results demonstrate the effectiveness of the proposed methods. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution image reconstruction produces a high-resolution image from a set of shifted, blurred, and decimated versions thereof. Previously published papers have not addressed the computational complexity of this ill-conditioned large scale problem adequately. In this paper, the computational complexity of MAP-based multiframe super-resolution algorithms is studied, and a new fast algorithm, as well as methods for parallel image reconstruction is also presented. The proposed fast algorithm splits the multiple input low-resolution images into several subsets according to their translation relations, and then applies normal MAP algorithm to each subset, the reconstructed images are processed subsequently at a successive level until the desired resolution is achieved. Experiment results are also provided to demonstrate the efficiency of the proposed techniques. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Novel theoretical results for the super-resolution reconstruction (SRR) are presented under the case of arbitrary image warping. The SRR model is reasonably separated into two parts of anti-aliasing and deblurring. The anti-aliasing part is proved to be well-posed. The ill-posedness of the entire SRR process is shown to be mainly caused by the deblurring part. The motion estimation error results in a multiplicative perturbation to the warping matrix, and the perturbation bound is derived. The common regularization algorithms used in SRR are analyzed through the discrete Picard condition, which provides a theoretical measure for limits on SRR. Experiments and examples are supplied to validate the presented theories. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> A typical image formation model for super-resolution (SR) introduces blurring, aliasing, and added noise. The enhancement itself may also introduce ringing. In this paper, we use subjective tests to assess the visual quality of SR-enhanced images. We then examine how well some existing objective quality metrics can characterize the observed subjective quality. Even full-reference metrics like MSE and SSIM do not always capture visual quality of SR images with and without residual aliasing. <s> BIB022 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Recently, there has been a great deal of work developing super-resolution algorithms for combining a set of low-quality images to produce a set of higher quality images. Either explicitly or implicitly, such algorithms must perform the joint task of registering and fusing the low-quality image data. While many such algorithms have been proposed, very little work has addressed the performance bounds for such problems. In this paper, we analyze the performance limits from statistical first principles using Crame/spl acute/r-Rao inequalities. Such analysis offers insight into the fundamental super-resolution performance bottlenecks as they relate to the subproblems of image registration, reconstruction, and image restoration. <s> BIB023 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution is the process of combining multiple low-resolution images to produce a higher resolution image. Because the super-resolution problem is an ill-posed one, many regularized algorithms have been proposed. Those algorithms usually use the same regularization term for a whole image. However, since an image generally contains various regions of different characteristics, simple regularization is not good enough for the prospective result. In this paper, we propose a region-based super-resolution algorithm to apply a suitable regularization term for each region. In the algorithm, the image is divided into homogeneous and inhomogeneous regions. According to the type of region, we apply different filters for regularization. The regularization parameters are also adaptively determined during the iteration. Simulation results show that the proposed algorithm is superior to the conventional algorithm in terms of objective quality as well as subjective one <s> BIB024 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper proposes a new algorithm to address blind image super-resolution by fusing multiple blurred low-resolution (LR) images to render a high-resolution (HR) image. Conventional super-resolution (SR) image reconstruction algorithms assume either the blurring during the image formation process is negligible or the blurring function is known a priori. This assumption, however, is impractical as it is difficult to eliminate blurring completely in some applications or characterize the blurring function fully. In view of this, we present a new maximum a posteriori (MAP) estimation framework that performs joint blur identification and HR image reconstruction. An iterative scheme based on alternating minimization is developed to estimate the blur and HR image progressively. A blur prior that incorporates the soft parametric blur information and smoothness constraint is introduced in the proposed method. Experimental results show that the new method is effective in performing blind SR image reconstruction where there is limited information about the blurring function. <s> BIB025 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> We propose a learning-based, single-image super-resolution reconstruction technique using the contourlet transform, which is capable of capturing the smoothness along contoursmaking use of directional decompositions. The contourlet coefficients at finer scales of the unknown high-resolution image are learned locally from a set of high-resolution training images, the inverse contourlet transform of which recovers the super-resolved image. In effect, we learn the high-resolution representation of an oriented edge primitive from the training data. Our experiments show that the proposed approach outperforms standard interpolation techniques as well as a standard (Cartesian) wavelet-based learning both visually and in terms of the PSNR values, especially for images with arbitrarily oriented edges. <s> BIB026 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a learning-based method for video super-resolution. There are two main contributions of the proposed method. First, information from cameras with different spatial-temporal resolutions is combined in our framework. This is achieved by constructing training dictionary using the high resolution images captured by still camera and the low resolution video is enhanced via searching in this customized database. Second, we enforce the spatio-temporal constraints using the conditional random field (CRF) and the problem of video super-resolution is posed as finding the high resolution video that maximizes the conditional probability. We apply the algorithm to video sequences taken from different scenes using cameras with different qualities and promising results are presented. <s> BIB027 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper we present a new super resolution Bayesian method for pansharpening of multispectral images which: a) incorporates prior knowledge on the expected characteristics of the multispectral images, b) uses the sensor characteristics to model the observation process of both panchromatic and multispectral images, and c) performs the estimation of all the unknown parameters in the model. Using real data, the pan-sharpened multispectral images are compared with the images obtained by other parsharpening methods and their quality is assessed both qualitatively and quantitatively. <s> BIB028 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution reconstruction produces one or a set of high-resolution (HR) images from a sequence of low-resolution (LR) images. Due to translational registration, super-resolution reconstruction can apply only on the sequences that have simple translation motion. This paper proposed a novel super-resolution reconstruction that that can apply on real sequences or complex motion sequences. The proposed super-resolution reconstruction uses a high accuracy registration algorithm, the fast affine block-based registration [16], in the maximum likelihood framework. Moreover, the regularization is used to compensate the missing measurement information. The experimental results show that the proposed reconstruction can apply on real sequence such as Foreman and Suzie. <s> BIB029 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-Resolution Reconstruction (SRR) aims to produces one or a set of high-resolution (HR) images from a sequence of low-resolution (LR) images. Due to translational registration, super-resolution reconstruction can apply only on the sequences that have simple translation motion. This paper proposed a novel super-resolution reconstruction that that can apply on real sequences or complex motion sequences. The proposed SRR uses a high accuracy registration algorithm, the fast affine block-based registration [16], in the maximum likelihood framework. Moreover, the BTV (Bi-Total Variance) regularization [12] is used to compensate the missing measurement information. The experimental results show that the proposed reconstruction can apply on real sequence such as Suzie. <s> BIB030 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution algorithms reconstruct a high-resolution image from a set of low-resolution images of a scene. Precise alignment of the input images is an essential part of such algorithms. If the low-resolution images are undersampled and have aliasing artifacts, the performance of standard registration algorithms decreases. We propose a frequency domain technique to precisely register a set of aliased images, based on their low-frequency, aliasing-free part. A high-resolution image is then reconstructed using cubic interpolation. Our algorithm is compared to other algorithms in simulations and practical experiments using real aliased images. Both show very good visual results and prove the attractivity of our approach in the case of aliased input images. A possible application is to digital cameras where a set of rapidly acquired images can be used to recover a higher-resolution final image. <s> BIB031 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Regularization plays a vital role in inverse problems, and especially in ill-posed ones. Along with classical regularization techniques based on smoothness, entropy, and sparsity, an emerging powerful regularization is one that leans on image examples. In this paper, we propose an efficient scheme for using image examples as driving a powerful regularization, applied to the image scale-up (super-resolution) problem. In this work, we target specifically scanned documents containing written text, graphics, and equations. Our algorithm starts by assigning per each location in the degraded image several candidate high-quality patches. Those are found as the nearest-neighbors (NN) in an image-database that contains pairs of corresponding low- and high-quality image patches. The found examples are used for the definition of an image prior expression, merged into a global MAP penalty function. We use this penalty function both for rejecting some of the irrelevant outlier examples, and then for reconstructing the desired image. We demonstrate our algorithm on several scanned documents with promising results. <s> BIB032 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Reconstruction-based super-resolution algorithms use either sub-pixel shifts or relative blur among low-resolution observations as a cue to obtain a high-resolution image. In this paper, we propose a super-resolution algorithm that exploits the information available in the low-resolution observations due to both sub-pixel shifts and relative blur to yield a better quality image. Performance analysis is carried out based on the Cramer-Rao lower bound. Several experimental results on synthetic and real images are given for validation. <s> BIB033 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Motion blur can degrade the quality of images and is considered a nuisance for computer vision problems. In this paper, we show that motion blur can in-fact be used for increasing the resolution of a moving object. Our approach utilizes the information in a single motion-blurred image without any image priors or training images. As the blur size increases, the resolution of the moving object can be enhanced by a larger factor, albeit with a corresponding increase in reconstruction noise. Traditionally, motion deblurring and super-resolution have been ill-posed problems. Using a coded-exposure camera that preserves high spatial frequencies in the blurred image, we present a linear algorithm for the combined problem of deblurring and resolution enhancement and analyze the invertibility of the resulting linear system. We also show a method to selectively enhance the resolution of a narrow region of high-frequency features, when the resolution of the entire moving object cannot be increased due to small motion blur. Results on real images showing up to four times resolution enhancement are presented. <s> BIB034 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a maximum a posteriori ramework for the super-resolution problem, i.e., reconstructing high-resolution images from shifted, rotated, low-resolution degraded observations. The main contributions of this work are two; first, the use of a new locally adaptive edge preserving prior for the super-resolution problem. Second an efficient two-step reconstruction methodology that includes first an initial registration using only the low-resolution degraded observations. This is followed by a fast iterative algorithm implemented in the discrete Fourier transform domain in which the restoration, interpolation and the registration subtasks of this problem are preformed simultaneously. We present examples with both synthetic and real data that demonstrate the advantages of the proposed framework. <s> BIB035 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Although the topic of Super-Resolution Reconstruction (SRR) has recently received considerable attention within the traditional research community, the SRR estimations are based on LI or L2 statistical norm estimation. Therefore, these SRR methods are very sensitive to their assumed data and noise models, which limit their applications. The real noise models that corrupt the measure sequence are unknown; consequently, SRR algorithm using LI or L2 norm may degrade the image sequence rather than enhance it. The robust norm applicable to several noise and data models is desired in SRR algorithms. This paper proposes an alternate SRR approach based on the stochastic regularization technique of Bayesian MAP estimation by minimizing a cost function. The Lorentzian norm is used for measuring the difference between the projected estimate of the high-resolution image and each low resolution image, removing outliers in the data. Tikhonov regularization is used to remove artifacts from the final result and improve the rate of convergence. In order to cope with real sequences and complex motion sequences, the fast affine block-based registration is used in the registration step of SRR. The experimental results show that the proposed reconstruction can be applied on real sequences such as Suzie sequence and confirm the effectiveness of our method and demonstrate its superiority to other super-resolution methods based on LI and L2 norm for several noise models such as AWGN, Poisson and Salt & Pepper noise. <s> BIB036 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Recently, there has been a great deal of work developing super-resolution reconstruction (SRR) algorithms. While many such algorithms have been proposed, the almost SRR estimations are based on L1 or L2 statistical norm estimation, therefore these SRR algorithms are usually very sensitive to their assumed noise model that limits their utility. The real noise models that corrupt the measure sequence are unknown; consequently, SRR algorithm using L1 or L2 norm may degrade the image sequence rather than enhance it. Therefore, the robust norm applicable to several noise and data models is desired in SRR algorithms. This paper first comprehensively reviews the SRR algorithms in this last decade and addresses their shortcomings, and latter proposes a novel robust SRR algorithm that can be applied on several noise models. The proposed SRR algorithm is based on the stochastic regularization technique of Bayesian MAP estimation by minimizing a cost function. For removing outliers in the data, the Lorentzian error norm is used for measuring the difference between the projected estimate of the high-resolution image and each low-resolution image. Moreover, Tikhonov regularization and Lorentzian-Tikhonov regularization are used to remove artifacts from the final answer and improve the rate of convergence. The experimental results confirm the effectiveness of our method and demonstrate its superiority to other super-resolution methods based on L1 and L2 norms for several noise models such as noiseless, additive white Gaussian noise (AWGN), poisson noise, salt and pepper noise, and speckle noise. <s> BIB037 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In many applications, the sampling frequency is limited by the physical characteristics of the components: the pixel pitch, the rate of the analog-to-digital (AID) converter, etc. A low- pass filter is usually applied before the sampling operation to avoid aliasing. However, when multiple copies are available, it is possible to use the information that is inherently present in the aliasing to reconstruct a higher resolution signal. If the different copies have unknown relative offsets, this is a nonlinear problem in the offsets and the signal coefficients. They are not easily separable in the set of equations describing the super-resolution problem. Thus, we perform joint registration and reconstruction from multiple unregistered sets of samples. We give a mathematical formulation for the problem when there are M sets of N samples of a signal that is described by L expansion coefficients. We prove that the solution of the registration and reconstruction problem is generically unique if MN ges L + M - 1. We describe two subspace-based methods to compute this solution. Their complexity is analyzed, and some heuristic methods are proposed. Finally, some numerical simulation results on one- and two-dimensional signals are given to show the performance of these methods. <s> BIB038 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a new super-resolution face hallucination and recognition method based on Independent Component Analysis (ICA). Firstly, ICA is used to build a linear mixing relationship between high-resolution (HR) face image and independent HR source faces images. The linear mixing coefficients are retained, thus the corresponding low-resolution (LR) face image is represented by linear mixture of down-sampled source faces images. So, when the source faces images are obtained by training a set of HR face images, unconstrained least square is utilized to obtain mixing coefficients to a LR image for hallucination and recognition. Experiments show that the accuracy of face recognition is insensitive to image size and the number of HR source faces images when image size is larger than 8×8, and the resolution and quality of the hallucinated face image are greatly enhanced over the LR ones, which is very helpful for human recognition. <s> BIB039 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper presents a neural network-based method for image super-resolution. In this technique, the super-resolution is considered as an ill-posed inverse problem which is solved by minimizing an evaluation function established based on an observation model that closely follows the physical image acquisition process. A Hopfield neural network is created to obtain an optimal solution to the problem. Not like some other single-frame super-resolution techniques, this technique takes into consideration PSF (point spread function) blurring as well as additive noise and generates high-resolution images with more preserved or restored image details. Experimental results demonstrate that the high-resolution images obtained by this technique have a very high quality in terms of PSNR (peak signal-to-noise ratio) and visually look more pleasant <s> BIB040 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Learning-based superresolution (SR) are popular SR techniques that use application dependent priors to infer the missing details in low resolution images (LRIs). However, their performance still deteriorates quickly when the magnification factor is moderately large. This leads us to an important problem: "Do limits of learning-based SR algorithms exist?" In this paper, we attempt to shed some light on this problem when the SR algorithms are designed for general natural images (GNIs). We first define an expected risk for the SR algorithms that is based on the root mean squared error between the superresolved images and the ground truth images. Then utilizing the statistics of GNIs, we derive a closed form estimate of the lower bound of the expected risk. The lower bound can be computed by sampling real images. By computing the curve of the lower bound w.r.t. the magnification factor, we can estimate the limits of learning-based SR algorithms, at which the lower bound of expected risk exceeds a relatively large threshold. We also investigate the sufficient number of samples to guarantee an accurate estimation of the lower bound. <s> BIB041 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper we propose a novel algorithm for super resolution based on total variation prior and variational distribution approximations. We formulate the problem using a hierarchical Bayesian model where the reconstructed high resolution image and the model parameters are estimated simultaneously from the low resolution observations. The algorithm resulting from this formulation utilizes variational inference and provides approximations to the posterior distributions of the latent variables. Due to the simultaneous parameter estimation, the algorithm is fully automated so parameter tuning is not required. Experimental results show that the proposed approach outperforms some of the state-of-the-art super resolution algorithms. <s> BIB042 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In facial image analysis, image resolution is an important factor which has great influence on the performance of face recognition systems. As for low-resolution face recognition problem, traditional methods usually carry out super-resolution firstly before passing the super-resolved image to a face recognition system. In this paper, we propose a new method which predicts high-resolution images and the corresponding features simultaneously. More specifically, we propose ldquofeature hallucinationrdquo to project facial images with low-resolution into an expected feature space. As a result, the proposed method does not require super-resolution as an explicit preprocessing step. In addition, we explore a constrained hallucination that considers the local consistency in the image grid. In our method, we use the index of local visual primitives as features and a block-based histogram distance to measure the similarity for the face recognition. Experimental results on FERET face database verify that the proposed method can improve both visual quality and recognition rate for low-resolution facial images. <s> BIB043 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a new super resolution method Maximum a Posteriori based on a universal hidden Markov tree model (MAP-uHMT) for remote sensing images. The hidden Markov tree theory in the wavelet domain is used to set up a prior model for reconstructing super resolution images from a sequence of warped, blurred, sub-sampled and contaminated low resolution images. Both the simulation results with a Landsat7 panchromatic image and actual results with four Landsat7 panchromatic images which were captured on different dates show that our method achieves better super resolution images both visually and quantitatively than other methods, based on PSNR in the simulation and derived PSF with actual data. <s> BIB044 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper a novel independent component analysis (ICA) based face super-resolution algorithm is proposed. In this scheme, the independent components (ICs) are obtained by offline training high-resolution face images. Given a low-resolution image, the high-resolution image is reconstructed by the linear combination of the ICs where the weight coefficients are obtained by the method of maximum a posteriori (MAP). The prior of ICA coefficients are estimated by performing PCA on training images. Meanwhile, a structure-tensor based filter is proposed to refine the intermediate SR result which is estimated by inverse ICA. Experimental results demonstrate that the proposed algorithm is robust to various pose, expressions and lighting conditions. The hallucination results preserve both the global structure and the high spatial-frequency information better such as sharp edges and high contrast. <s> BIB045 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper we formulate a new time dependent convolutional model for super-resolution based on a constrained variational model that uses the total variation of the signal as a regularizing functional. We propose an iterative refinement procedure based on Bregman iteration to improve spatial resolution. The model uses a dataset of low resolution images and incorporates a downsampling operator to relate the high resolution scale to the low resolution one. We present an algorithm for the model and we perform a series of numerical experiments to show evidence of the good behavior of the numerical scheme and quality of the results. <s> BIB046 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a new super-resolution face hallucination method based on Bilateral-projection-based Two-Dimensional Principal Component Analysis (B2DPCA). Firstly, the high-resolution (HR) face image and its corresponding low-resolution (LR) face image are projected to the HR and LR B2DPCA feature spaces, respectively. In these spaces, the linear mixing relationship between HR and LR feature is estimated from a training set. For reconstructing the HR image from the observed LR image, the LR image is firstly projected to LR feature space and then mapped to HR feature. Finally, the HR feature is reconstructed to the HR face image. Experiments on the well-known face databases show that the performance of our proposed method. The resolution and quality of the hallucinated face images are greatly enhanced over the LR ones, which is very helpful for human recognition. <s> BIB047 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper we propose a novel super-resolution algorithm based on motion compensation and edge-directed spatial interpolation succeeded by fusion via pixel classification. Two high-resolution images are constructed, the first by means of motion compensation and the second by means of edge-directed interpolation. The AdaBoost classifier is then used to fuse these images into an high-resolution frame. Experimental results show that the proposed method surpasses well-known resolution enhancement methods while maintaining moderate computational complexity. <s> BIB048 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose an image super-resolution approach using a novel generic image prior - gradient profile prior, which is a parametric prior describing the shape and the sharpness of the image gradients. Using the gradient profile prior learned from a large number of natural images, we can provide a constraint on image gradients when we estimate a hi-resolution image from a low-resolution image. With this simple but very effective prior, we are able to produce state-of-the-art results. The reconstructed hi-resolution image is sharp while has rare ringing or jaggy artifacts. <s> BIB049 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> A robust whole-image super-resolution is highly demanded. Whole-image super-resolution requires target-region extraction and motion estimation because an image sequence usually includes multiple targets and a background. A single- motion model is insufficient to represent whole-image motion. In addition, high-accuracy motion estimation is necessary for super-resolution. Simultaneous target-region extraction and high-accuracy motion estimation are challenging problems. We propose a robust and accurate algorithm that can provide extracted single-motion regions and their motion parameters. The proposed algorithm consists of three phases: feature-based motion estimation, single-motion region extraction, and region-based motion estimation. Once we obtain motion parameters of the whole image, we can reconstruct a high-resolution whole image by applying a robust super- resolution. Results of experiments demonstrate that the proposed approach can robustly reconstruct complex multiple- target scenes. <s> BIB050 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> A novel image registration algorithm for super-resolution is proposed. The quality of reconstructed images becomes poor when the arrangement of the registered positions is irregular. However, there are no known ways to acquire the positions arranged regularly without using active cameras. By using a conventional algorithm, the positions registered with subpixel accuracy are often distributed near integer values in the low-resolution grid when the aperture problem occurs, for example, in the case of processing straight edges. Since this phenomenon causes an irregular arrangement, the high- resolution images are reconstructed inefficiently. The proposed method estimates corresponding positions on the shifted image whose image grid is shifted by a desired subpixel displacement. By using the proposed method, the distribution of the corresponding positions is shifted artificially so that the quality of the reconstructed image is improved. The effectiveness of the proposed method is demonstrated in experiments. <s> BIB051 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution algorithms combine multiple low resolution images into a single high resolution image. They have received a lot of attention recently in various application domains such as HDTV, satellite imaging, and video surveillance. These techniques take advantage of the aliasing present in the input images to reconstruct high frequency information of the resulting image. One of the major challenges in such algorithms is a good alignment of the input images: subpixel precision is required to enable accurate reconstruction. In this paper, we give an overview of some subspace techniques that address this problem. We first formulate super-resolution in a multichannel sampling framework with unknown offsets. Then, we present three registration methods: one approach using ideas from variable projections, one using a Fourier description of the aliased signals, and one using a spline description of the sampling kernel. The performance of the algorithms is evaluated in numerical simulations. <s> BIB052 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper introduces a new color image superresolution algorithm in an adaptive, robust M-estimation framework. Using a robust error norm in the objective function, and adapting the estimation process to each of the low-resolution frames, the proposed method effectively suppresses the outliers due to violations of the assumed observation model, and results in color superresolution estimates with crisp details and no color artifacts, without the use of regularization. Experiments on both synthetic and real sequences demonstrate the superior performance over using the L2 and L1 error norms in the objective function. <s> BIB053 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In video surveillance, face images that are captured by common cameras usually have a low resolution, which is a great obstacle to face recognition. In this paper, we show that classification accuracy drops very quickly and algebraic features of face image change greatly as resolution decreases, and we further indicate that two orthogonal matrices (TOMs) of SVD contain the leading information for recognition. Based on the phenomena, we propose a feature-based face super-resolution method using eigentransformation by 2DPCA. TOMs of SVD on face image matrices are reconstructed instead of super-resolution being performed in pixel domain. Reconstructed algebraic features are formed into high-resolution face images and also be used for face recognition directly. Experiments demonstrate that our method obtains hallucinated face images with good vision, and improves performance of low-resolution face recognition. <s> BIB054 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Faces captured by surveillance cameras are often of very low resolution. This significantly deteriorates face recognition performance. Super-resolution techniques have been proposed in the past to mitigate this. This paper proposes the novel use of a Locality Preserving Projections (LPP) algorithm called Direct Locality Preserving Projections (DLPP) for super resolution of facial images, or ldquoface hallucinationrdquo in other words. Because DLPP doesnpsilat require any dimensionality reduction preprocessing via Principle Component Analysis (PCA), it retains more discriminating power in its feature space than LPP. Combined with non-parametric regression using a generalized regression neural network (GRNN), the proposed work can render high-resolution face image from an image of resolution as low as 8x7 with a large zoom factor of 24. The resulting technique is powerful and efficient in synthesizing faces similar to ground-truth faces. Simulation results show superior results compared to other well-known schemes. <s> BIB055 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper proposes a new algorithm to generate a super-resolution image from a single, low-resolution input without the use of a training data set. We do this by exploiting the fact that the image is highly compressible in the wavelet domain and leverage recent results of compressed sensing (CS) theory to make an accurate estimate of the original high-resolution image. Unfortunately, traditional CS approaches do not allow direct use of a wavelet compression basis because of the coherency between the point-samples from the downsampling process and the wavelet basis. To overcome this problem, we incorporate the downsampling low-pass filter into our measurement matrix, which decreases coherency between the bases. To invert the downsampling process, we use the appropriate inverse filter and solve for the high-resolution image using a greedy, matching-pursuit algorithm. The result is a simple and efficient algorithm that can generate high quality, high-resolution images without the use of training data. We present results that show the improved performance of our method over existing super-resolution approaches. <s> BIB056 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In the present paper, we propose a new magnetic resonance fast scan technique using the phase-scrambling Fourier transform imaging (PSFT), in which the strategy of data acquisition is similar to that of the standard Fourier imaging technique. The Fourier transformed image in the PSFT has a phase modulation in which the local spatial frequency is increased as the position moves away from the center of the image. Therefore, the signal length of the PSFT signal can be expanded using the restraint that the reconstructed image is a real-value object in the super-resolution algorithm. Simulation and experimental studies using a 0.0187T MR scanner were performed for the new improving image resolution method. It was shown that the spatial resolution could be improved in proportion to the distance from the center of the image by the quadratic phase modulation function, irrespective of the amount of noise, and the maximum resolution improvement is 2.0 times, as compared to the standard imaging method. <s> BIB057 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Video scaling is a crucial video processing component that finds a variety of applications in video display, enhancement and analysis. Thus, video scaling has been an important research topic for several years now. With the emergence of hybrid-cameras that simultaneously capture a low-resolution (LR) video and periodic high-resolution (HR) stills, methods that use information in HR stills for enhancing the LR video are being studied. We present a multi-layer super-resolution (SR) method to rescale the input LR video with the help of a few HR still image guides. Resolution enhancement is achieved by blending the high frequency details into the input video frames that are scaled-up by a conventional, spatial scaling method. The missing high frequency details for each video frame are derived by spatio-temporal warping of the high frequency details extracted from HR still image guides. The Non-Local-Means (NLM) concept is used for computing the warping weights. The proposed SR method handles both global and local motions in a video, without requiring any explicit image registration. Experimental results comparing the proposed method with some of the existing video scaling techniques clearly points its strong potential. <s> BIB058 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> The joint usage of low- and full-resolution images in multiview systems provides an attractive opportunity for data size reduction while maintaining good quality in 3D applications. In this paper we present a novel application of a super-resolution method for usage within a mixed resolution multiview setup. The technique borrows high-frequency content from neighboring full resolution images to enhance particular low-resolution views. Occlusions are handled through matching of low-resolution images. Both the stereo and the more general multiview cases are considered using the multiview video-plus-depth format. Results demonstrate significant gains in PSNR and in visual quality for test sequences. <s> BIB059 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper we present a new image up scaling (single image super resolution) algorithm. It is based on the refinement of a simple pixel decimation followed by an optimization step maximizing the smoothness of the second order derivatives of the image intensity while keeping the sum of the brightness values of each subdivided pixel (i.e. the estimated irradiance on the area) constant. The method is physically grounded and creates images that appear very sharp and with reduced artifacts. Subjective and objective tests demonstrate the high quality of the results obtained. <s> BIB060 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> We formulate the multiframe blind deconvolution problem in an incremental expectation maximization (EM) framework. Beyond deconvolution, we show how to use the same framework to address: (i) super-resolution despite noise and unknown blurring; (ii) saturation-correction of overexposed pixels that confound image restoration. The abundance of data allows us to address both of these without using explicit image or blur priors. The end result is a simple but effective algorithm with no hyperparameters. We apply this algorithm to real-world images from astronomy and to super resolution tasks: for both, our algorithm yields increased resolution and deconvolved images simultaneously. <s> BIB061 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> A novel two-step super-resolution (SR) method for face images is proposed in this paper. The critical issue of global face reconstruction in the two-step SR framework is to construct the relationship between high resolution (HR) and low resolution (LR) features. We choose the Principal Component Analysis (PCA) coefficients of LR/HR face images as the features for global faces. These features are considered as inputs and outputs of an unknown linear system. The mapping between the inputs and outputs is estimated from training sets as the system response. The HR features corresponding to a test LR image can be obtained by applying the learnt mapping to the LR features, and hence we can reconstruct the global face. Ultimately, an HR face image is generated by using the patch-based neighbor reconstruction that imposes facial details into the global face. Experiments indicate that our method produces HR faces of higher quality and is easier to implement than traditional methods based on two-step framework. <s> BIB062 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution reconstruction of face image is the problem of reconstructing a high resolution face image from one or more low resolution face images. Assuming that high and low resolution images share similar intrinsic geometries, various recent super-resolution methods reconstruct high resolution images based on a weights determined from nearest neighbors in the local embedding of low resolution images. These methods suffer disadvantages from the finite number of samples and the nature of manifold learning techniques, and hence yield unrealistic reconstructed images. To address the problem, we apply canonical correlation analysis (CCA), which maximizes the correlation between the local neighbor relationships of high and low resolution images. We use it separately for reconstruction of global face appearance, and facial details. Experiments using a collection of frontal human faces show that the proposed algorithm improves reconstruction quality over existing state-of-the-art super-resolution algorithms, both visually, and using a quantitative peak signal-to-noise ratio assessment. <s> BIB063 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper proposes a framework for single-image super-resolution. The underlying idea is to learn a map from input low-resolution images to target high-resolution images based on example pairs of input and output images. Kernel ridge regression (KRR) is adopted for this purpose. To reduce the time complexity of training and testing for KRR, a sparse solution is found by combining the ideas of kernel matching pursuit and gradient descent. As a regularized solution, KRR leads to a better generalization than simply storing the examples as has been done in existing example-based algorithms and results in much less noisy images. However, this may introduce blurring and ringing artifacts around major edges as sharp changes are penalized severely. A prior model of a generic image class which takes into account the discontinuity property of images is adopted to resolve this problem. Comparison with existing algorithms shows the effectiveness of the proposed method. <s> BIB064 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper proposes a context-constrained hallucination approach for image super-resolution. Through building a training set of high-resolution/low-resolution image segment pairs, the high-resolution pixel is hallucinated from its texturally similar segments which are retrieved from the training set by texture similarity. Given the discrete hallucinated examples, a continuous energy function is designed to enforce the fidelity of high-resolution image to low-resolution input and the constraints imposed by the hallucinated examples and the edge smoothness prior. The reconstructed high-resolution image is sharp with minimal artifacts both along the edges and in the textural regions. <s> BIB065 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> The objective of super-resolution (SR) imaging is to reconstruct a single higher-resolution image based on a set of lower-resolution images that were acquired from the same scene to overcome the limitations of image acquisition process for facilitating better visualization and content recognition. In this paper, a stochastic Markov chain Monte Carlo (MCMC) SR image reconstruction approach is proposed. First, a Bayesian inference formulation, which is based on the observed low-resolution images and the prior high-resolution image model, is mathematically derived. Second, to exploit the MCMC sample-generation technique for the stochastic SR image reconstruction, three fundamental issues are observed as follows. First, since the hyperparameter value of the prior image model controls the degree of regularization and intimately affects the quality of the reconstructed high-resolution image, how to determine an optimal hyperparameter value for different low-resolution input images becomes a very challenging task. Rather than exploiting the exhaustive search, an iterative updating approach is developed in this paper by allowing the value of hyperparameter being simultaneously updated in each sample-generation iteration. Second, the samples generated during the so-called burn-in period (measured in terms of the number of samples initially generated) of the MCMC-based sample-generation process are considered unreliable and should be discarded. To determine the length of the burn-in period for each set of low-resolution input images, a time-period bound in closed form is mathematically derived. Third, image artifacts could be incurred in the reconstructed high-resolution image, if the number of samples (counting after the burn-in period) generated by the MCMC-based sample-generation process is insufficient. For that, a variation-sensitive bilateral filter is proposed as a 'complementary' post-processing scheme, to improve the reconstructed high-resolution image quality, when the number of samples is insufficient. Extensive simulation results have clearly shown that the proposed stochastic SR image reconstruction method consistently yields superior performance. <s> BIB066 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> We present a complete super-resolution system using a camera, that is assumed to be on a vibrating platform and continually capturing frames of a static scene, that have to be super-resolved in particular regions of interest. In a practical system the shutter of the camera is not synchronised with the vibrations it is subjected to. So, we propose a novel method for frame selection according to their degree of blurring and we combine a tracker with the sequence of selected frames to identify the subimages containing the region of interest. The extracted subimages are subsequently co-registered using a state of the art sub-pixel registration algorithm. Further selection of the co-registered subimages takes place, according to the confidence in the registration result. Finally, the subimage of interest is super-resolved using a state of the art super-resolution algorithm. The proposed frame selection method is of generic applicability and it is validated with the help of manual frame quality assessment. <s> BIB067 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> We present an integrated method for post-processing of range data which removes outliers, smoothes the depth values and enhances the lateral resolution in order to achieve visually pleasing 3D models from low-cost depth sensors with additional (registered) color images. The algorithm is based on the non-local principle and adapts the original NL-Means formulation to the characteristics of typical depth data. Explicitly handling outliers in the sensor data, our denoising approach achieves unbiased reconstructions from error-prone input data. Taking intra-patch similarity into account, we reconstruct strong discontinuities without disturbing artifacts and preserve fine detail structures, obtaining piece-wise smooth depth maps. Furthermore, we exploit the dependencies of the depth data with additionally available color information and increase the lateral resolution of the depth maps. We finally discuss how to parallelize the algorithm in order to achieve fast processing times that are adequate for post-processing of data from fast depth sensors such as time-of-flight cameras. <s> BIB068 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> From the perspective of linear algebra, the performance of super-resolution reconstruction (SR) depends on the conditioning of the linear system characterizing the degradation model. This is analyzed in the Fourier domain using the perturbation theory. By proposing a new SR error bound in terms of the point spread function (PSF), we reveal that the blur function dominates the condition number (CN) of degradation matrix, and the advantage of non-integer magnification factors (MFs) over the integer ones comes from sampling zero crossings of the DFT of the PSF. We also explore the effect of regularization by integrating it into the SR model, and investigate the influence of the optimal regularization parameter. A tighter error bound is derived given the optimal regularization parameter. Two curves of error bounds vs. MFs are presented, and verified by processing real images. It explains that with proper regularization, SR at the integer MFs is still valid. <s> BIB069 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we address the super resolution (SR) problem from a set of degraded low resolution (LR) images to obtain a high resolution (HR) image. Accurate estimation of the sub-pixel motion between the LR images significantly affects the performance of the reconstructed HR image. In this paper, we propose novel super resolution methods where the HR image and the motion parameters are estimated simultaneously. Utilizing a Bayesian formulation, we model the unknown HR image, the acquisition process, the motion parameters and the unknown model parameters in a stochastic sense. Employing a variational Bayesian analysis, we develop two novel algorithms which jointly estimate the distributions of all unknowns. The proposed framework has the following advantages: 1) Through the incorporation of uncertainty of the estimates, the algorithms prevent the propagation of errors between the estimates of the various unknowns; 2) the algorithms are robust to errors in the estimation of the motion parameters; and 3) using a fully Bayesian formulation, the developed algorithms simultaneously estimate all algorithmic parameters along with the HR image and motion parameters, and therefore they are fully-automated and do not require parameter tuning. We also show that the proposed motion estimation method is a stochastic generalization of the classical Lucas-Kanade registration algorithm. Experimental results demonstrate that the proposed approaches are very effective and compare favorably to state-of-the-art SR algorithms. <s> BIB070 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Most popular learning-based super-resolution (SR) approaches suffer from complicated learning structures and highly intensive computation, especially in resource-limited applications. We propose a novel frontal facial image SR approach by using multiple local linear transformations to approximate the nonlinear mapping between low-resolution (LR) and high-resolution (HR) images in the pixel domain. We adopt Procrustes analysis to obtain orthogonal matrices representing the learned linear transformations, which cannot only well capture appearance variations in facial patches but also greatly simplify the transformation computation to matrices manipulation. An HR image can be directly reconstructed from a single LR image without need of the large training data, thus avoiding the use of a large redundant LR and HR patch database. Experimental results show that our approach is computationally fast as well the SR quality compares favorably with the state-of-the-art approaches from both subjective and objective evaluations. Besides, our approach is insensitive to the size of training data and robust to a wide range of facial variations like occlusions. More importantly, the proposed method is also much more effective than other comparative methods to reconstruct real-world images captured from the Internet and webcams. <s> BIB071 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Spatial Super Resolution (SR) aims to recover fine image details, smaller than a pixel size. Temporal SR aims to recover rapid dynamic events that occur faster than the video frame-rate, and are therefore invisible or seen incorrectly in the video sequence. Previous methods for Space-Time SR combined information from multiple video recordings of the same dynamic scene. In this paper we show how this can be done from a single video recording. Our approach is based on the observation that small space-time patches (‘ST-patches’, e.g., 5×5×3) of a single ‘natural video’, recur many times inside the same video sequence at multiple spatio-temporal scales. We statistically explore the degree of these ST-patch recurrences inside ‘natural videos’, and show that this is a very strong statistical phenomenon. Space-time SR is obtained by combining information from multiple ST-patches at sub-frame accuracy. We show how finding similar ST-patches can be done both efficiently (with a randomized-based search in space-time), and at sub-frame accuracy (despite severe motion aliasing). Our approach is particularly useful for temporal SR, resolving both severe motion aliasing and severe motion blur in complex ‘natural videos’. <s> BIB072 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a novel generic image prior-gradient profile prior, which implies the prior knowledge of natural image gradients. In this prior, the image gradients are represented by gradient profiles, which are 1-D profiles of gradient magnitudes perpendicular to image structures. We model the gradient profiles by a parametric gradient profile model. Using this model, the prior knowledge of the gradient profiles are learned from a large collection of natural images, which are called gradient profile prior. Based on this prior, we propose a gradient field transformation to constrain the gradient fields of the high resolution image and the enhanced image when performing single image super-resolution and sharpness enhancement. With this simple but very effective approach, we are able to produce state-of-the-art results. The reconstructed high resolution images or the enhanced images are sharp while have rare ringing or jaggy artifacts. <s> BIB073 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper proposes a new algorithm to perform single-frame image super-resolution (SR) of vehicle license plate (VLP) using soft learning prior. Conventional single-frame SR/interpolation methods such as bi-cubic interpolation often experience over-smoothing near the edges and textured regions. Therefore, learning-based methods have been proposed to handle these shortcomings by incorporating a learning term so that the reconstructed high-resolution images can be guided towards these models. However, existing learning-based methods employ a binary hard-decision approach to determine whether the prior models are fully relevant or totally irrelevant. This approach, however, is inconsistent with many practical applications as the degree of relevance for the prior models may vary. In view of this, this paper proposes a new framework that adopts a soft learning approach in license plate super-resolution. The method integrates image SR with optical character recognition (OCR) to perform VLP SR. The importance of the prior models is estimated through relevance scores obtained from the OCR. These are then incorporated as a soft learning term into a new regularized cost function. Experimental results show that the proposed method is effective in handling license plate SR in both simulated and real experiments. <s> BIB074 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Sparse representation provides a new method of generating a super-resolution image from a single low resolution input image. An over-complete base for sparse representation is an essential part of such methods. However, discovering the over-complete base with efficient representation from a large amount of image patches is a difficult problem. In this paper, we propose a super-resolution construction based on multi-space sparse representation to efficiently solve the problem. In the representation, image patches are decomposed into a structure component and a texture component represented by the over-complete bases of their own spaces so that their high-level features can be captured by the bases. In the implementation, a prior knowledge about low resolution images generation is combined to the typical base construction for high construction quality. Experiment results demonstrate that the proposed method significantly improves the PSNR, SSIM and visual quality of reconstructed high-resolution image. <s> BIB075 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper we address the problem of producing a high-resolution image from a single low-resolution image without any external training set. We propose a framework for both magnification and deblurring using only the original low-resolution image and its blurred version. In our method, each pixel is predicted by its neighbors through the Gaussian process regression. We show that when using a proper covariance function, the Gaussian process regression can perform soft clustering of pixels based on their local structures. We further demonstrate that our algorithm can extract adequate information contained in a single low-resolution image to generate a high-resolution image with sharp edges, which is comparable to or even superior in quality to the performance of other edge-directed and example-based super-resolution algorithms. Experimental results also show that our approach maintains high-quality performance at large magnifications. <s> BIB076 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this letter, we present the first application of the Generalized Gaussian Markov Random Field (GGMRF) to the problem of video super-resolution. The GGMRF prior is employed to perform a maximum a posteriori (MAP) estimation of the desired high-resolution image. Compared with traditional prior models, the GGMRF can describe the distribution of the high-resolution image much better and can also preserve better the discontinuities (edges) of the original image. Previous work that used GGMRF for image restoration in which the temporal dependencies among video frames has not considered. Since the corresponding energy function is convex, gradient descent optimization techniques are used to solve the MAP estimation. Results show the super-resolved images using the GGMRF prior not only offers a good enhancement of visual quality, but also contain a significantly smaller amount of noise. <s> BIB077 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> While researchers strive to improve automatic face recognition performance, the relationship between image resolution and face recognition performance has not received much attention. This relationship is examined systematically and a framework is developed such that results from super-resolution techniques can be compared. Three super-resolution techniques are compared with the Eigenface and Elastic Bunch Graph Matching face recognition engines. Parameter ranges over which these techniques provide better recognition performance than interpolated images is determined. <s> BIB078 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, a new two-step method is proposed to infer a high-quality and high-resolution (HR) face image from a low-quality and low-resolution (LR) observation based on training samples in the database. First, a global face image is reconstructed based on the non-linear relationship between LR and HR face images, which is established according to radial basis function and partial least squares (RBF-PLS) regression. Based on the reconstructed global face patches manifold (formed by the image patches at the same position of all global face images), whose local geometry is more consistent with that of original HR face patches manifold than noisy LR one is, the Neighbor Embedding is applied to induce the target HR face image by preserving the similar local geometry between global face patches manifold and the original HR face patches manifold. A comparison of some state-of-the-art methods shows the superiority of our method, and experiments also demonstrate the effectiveness both under simulation and real conditions. <s> BIB079 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In this paper, we propose a new two-step face hallucination method to induce a high-resolution (HR) face image from a low-resolution (LR) observation. Especially for low-quality surveillance face image, an RBF-PLS based variable selection method is presented for the reconstruction of global face image. Further more, in order to compensate for the reconstruction errors, which are lost high frequency detailed face features, the Neighbor Embedding (NE) based residue face hallucination algorithm is used. Compared with current methods, the proposed RBF-PLS based method can generate a global face more similar to the original face and less sensitive to noise, moreover, the NE algorithm can reduce the reconstruction errors caused by misalignment on the basis of a carefully designed search strategy. Experiments show the superiority of the proposed method compared with some state-of-the-art approaches and the efficacy both in simulation and real surveillance condition. <s> BIB080 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Recently, compressive sensing (CS) has emerged as a powerful tool for solving a class of inverse/underdetermined problems in computer vision and image processing. In this paper, we investigate the application of CS paradigms on single image super-resolution (SR) problems that are considered to be the most challenging in this class. In light of recent promising results, we propose novel tools for analyzing sparse representation-based inverse problems using redundant dictionary basis. Further, we provide novel results establishing tighter correspondence between SR and CS. As such, we gain insights into questions concerning regularizing the solution to the underdetermined problem, such as follows. 1) Is sparsity prior alone sufficient? 2) What is a good dictionary? 3) What is the practical implication of noncompliance with theoretical CS hypothesis? Unlike in other underdetermined problems that assume random down-projections, the low-resolution image formation model employed in CS-based SR is a deterministic down-projection that may not necessarily satisfy some critical assumptions of CS. We further investigate the impact of such projections in concern to the above questions. <s> BIB081 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Ultrasound image resolution enhancement is an ongoing challenge to date. Though many works have been performed using device-based approach, there exists few works dealing with post-processing methods. This paper investigates a technique based on the Alternating Direction Method of Multipliers for the resolution enhancement in ultrasound imaging, which includes the deblurring and denoising tasks. We here point out the characteristics of the proposed technique and thereby underline the features that must be dealt with for successful ultrasound image restoration. Synthetic and in vivo ultrasound images are processed in order to asses its performances. <s> BIB082 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution is a widely applied technique that improves the resolution of input images by software methods. Most conventional reconstruction-based super-resolution algorithms assume accurate dense optical flow fields between the input frames, and their performance degrades rapidly when the motion estimation result is not accurate enough. However, optical flow estimation is usually difficult, particularly when complicated motion is presented in real-world videos. In this paper, we explore a new way to solve this problem by using sparse feature point correspondences between the input images. The feature point correspondences, which are obtained by matching a set of feature points, are usually precise and much more robust than dense optical flow fields. This is because the feature points represent well-selected significant locations in the image, and performing matching on the feature point set is usually very accurate. In order to utilize the sparse correspondences in conventional super-resolution, we extract an adaptive support region with a reliable local flow field from each corresponding feature point pair. The normalized prior is also proposed to increase the visual consistency of the reconstructed result. Extensive experiments on real data were carried out, and results show that the proposed algorithm produces high-resolution images with better quality, particularly in the presence of large-scale or complicated motion fields. <s> BIB083 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Super-resolution technology provides an effective way to increase image resolution by incorporating additional information from successive input images or training samples. Various super-resolution algorithms have been proposed based on different assumptions, and their relative performances can differ in regions of different characteristics within a single image. Based on this observation, an adaptive algorithm is proposed in this paper to integrate a higher level image classification task and a lower level super-resolution process, in which we incorporate reconstruction-based super-resolution algorithms, single-image enhancement, and image/video classification into a single comprehensive framework. The target high-resolution image plane is divided into adaptive-sized blocks, and different suitable super-resolution algorithms are automatically selected for the blocks. Then, a deblocking process is applied to reduce block edge artifacts. A new benchmark is also utilized to measure the performance of super-resolution algorithms. Experimental results with real-life videos indicate encouraging improvements with our method. <s> BIB084 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper proposes a new multi-frame super-resolution (SR) approach to reconstruct a high-resolution (HR) image from low-resolution (LR) images with zooming motion. Existing SR methods often assume that the relative motion between acquired LR images consists of only translation and possibly rotation. This restricts the application of these methods in cases when there is zooming motion among the LR images. There are currently only a few methods focusing on zooming SR. Their formulation usually ignores the registration errors or assumes them to be negligible during the HR reconstruction. In view of this, this paper presents a new SR reconstruction approach to handle a more flexible motion model including translation, rotation and zooming. An iterative framework is developed to estimate the motion parameters and HR image progressively. Both simulated and real-life experiments show that the proposed method is effective in performing image SR. <s> BIB085 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> The paper presents the resolution-invariant image representation (@?IIR) framework. It applies sparse-coding with multi-resolution codebook to learn resolution-invariant sparse representations of local patches. An input image can be reconstructed to higher resolution at not only discrete integer scales, as that in many existing super-resolution works, but also continuous scales, which functions similar to 2-D image interpolation. The @?IIR framework includes the methods of building a multi-resolution bases set from training images, learning the optimal sparse resolution-invariant representation of an image, and reconstructing the missing high-frequency information at continuous resolution level. Both theoretical and experimental validations of the resolution invariance property are presented in the paper. Objective comparison and subjective evaluation show that the @?IIR framework based image resolution enhancement method outperforms existing methods in various aspects. <s> BIB086 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Low resolution severely degrades the detail of face images, while bad illumination condition further increases the difficulty of face recognition. In this paper, we present a new differential images based face recognition approach to deal with low resolution and variable illumination problems. Firstly, we resort to down-sampling and interpolation to build differential image database. Then feature extraction is preformed to project the differential images into differential feature subspace. For recognizing input low-resolution (LR) face images that suffer from illumination variation, normalization are employed to compensate the illumination of LR images and hallucination method is to infer the corresponding project coefficients in the differential feature subspace. Finally, similarity measure is used for face discrimination. Different from Eigenspace-based face recognition techniques, the utilization of differential images instead of original images ensures the distinction between face images of different subjects more effectively. Experiments demonstrate that the proposed approach can improve recognition rate of LR face under variable illumination significantly. <s> BIB087 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Recently, the demand for face recognition to identify persons from surveillance video cameras has rapidly increased. Since surveillance cameras are usually placed at positions far from a person's face, the quality of face images captured by the cameras tends to be low. This degrades the recognition accuracy. Therefore, aiming to improve the accuracy of the low-resolution-face recognition, we propose a video-based super-resolution method. The proposed method can generate a high-resolution face image from low-resolution video frames including non-rigid deformations caused by changes of face poses and expressions without using any positional information of facial feature points. Most existing techniques use the facial feature points for image alignment between the video frames. However, it is difficult to obtain the accurate positions of the feature points from low-resolution face images. To achieve the alignment, the proposed method uses a free-form deformation method that flexibly aligns each local region between the images. This enables super-resolution of face images from low-resolution videos. Experimental results demonstrated that the proposed method improved the performance of super-resolution for actual videos in terms of both image quality and face recognition accuracy. <s> BIB088 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> Total variation (TV) has been used as a popular and effective image prior model in regularization-based image processing fields, such as denoising, deblurring, super-resolution (SR), and others, because of its ability to preserve edges. However, as the TV model favors a piecewise constant solution, the processing results in the flat regions of the image being poor, and it cannot automatically balance the processing strength between different spatial property regions in the image. In this paper, we propose a spatially weighted TV image SR algorithm, in which the spatial information distributed in different image regions is added to constrain the SR process. A newly proposed and effective spatial information indicator called difference curvature is used to identify the spatial property of each pixel, and a weighted parameter determined by the difference curvature information is added to constrain the regularization strength of the TV regularization at each pixel. Meanwhile, a majorization-minimization algorithm is used to optimize the proposed spatially weighted TV SR model. Finally, a significant amount of simulated and real data experimental results show that the proposed spatially weighted TV SR algorithm not only efficiently reduces the “artifacts” produced with a TV model in fat regions of the image, but also preserves the edge information, and the reconstruction results are less sensitive to the regularization parameters than the TV model, because of the consideration of the spatial information constraint. <s> BIB089 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> This paper proposes a new approach to the image blind super-resolution (BSR) problem in the case of affine interframe motion. Although the tasks of image registration, blur identification, and high-resolution (HR) image reconstruction are coupled in the imaging process, when dealing with nonisometric interframe motion or without the exact knowledge of the blurring process, classic SR techniques generally have to tackle them (maybe in some combinations) separately. The main difficulty is that state-of-the-art deconvolution methods cannot be straightforwardly generalized to cope with the space-variant motion. We prove that the operators of affine warping and blur commute with some additional transforms and derive an equivalent form of the BSR observation model. Using this equivalent form, we develop an iterative algorithm to jointly estimate the triple-coupled variables, i.e., the motion parameters, blur kernels, and HR image. Experiments on synthetic and real-life images illustrate the performance of the proposed technique in modeling the space-variant degradation process and restoring local textures. <s> BIB090 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> We propose a simple preconditioning method for accelerating the solution of edge-preserving image super-resolution (SR) problems in which a linear shift-invariant point spread function is employed. Our technique involves reordering the high-resolution (HR) pixels in a similar manner to what is done in preconditioning methods for quadratic SR formulations. However, due to the edge preserving requirements, the Hessian matrix of the cost function varies during the minimization process. We develop an efficient update scheme for the preconditioner in order to cope with this situation. Unlike some other acceleration strategies that round the displacement values between the low-resolution (LR) images on the HR grid, the proposed method does not sacrifice the optimality of the observation model. In addition, we describe a technique for preconditioning SR problems involving rational magnification factors. The use of such factors is motivated in part by the fact that, under certain circumstances, optimal SR zooms are nonintegers. We show that, by reordering the pixels of the LR images, the structure of the problem to solve is modified in such a way that preconditioners based on circulant operators can be used. <s> BIB091 </s> Super-resolution: a comprehensive survey <s> Improvement factor and lower bounds <s> In applications such as super-resolution imaging and mosaicking, multiple video sequences are registered to reconstruct video with enhanced resolution. However, not all computed registration is reliable. In addition, not all sequences contribute useful information towards reconstruction from multiple non-uniformly distributed sample sets. In this paper we present two algorithms that can help determine which low resolution sample sets should be combined in order to maximize reconstruction accuracy while minimizing the number of sample sets. The first algorithm computes a confidence measure which is derived as a combination of two objective functions. The second algorithm is an iterative ranked-based method for reconstruction which uses confidence measures to assign priority to sample sets that maximize information gain while minimizing reconstruction error. Experimental results with real and synthetic sequences validate the effectiveness of the proposed algorithms. Application of our work in medical visualization and super-resolution reconstruction of MRI data are also presented. <s> BIB092
The bounds derived so far for the improvement factor of a reconstruction-based SR algorithm are of two types. The first type is deterministic, based on linear models of the problem and looking at, e.g., the condition numbers of the matrices and the available number of LR observations. These deterministic bounds tend to be very conservative and yield relatively low numbers for the possible improvement factors (around 1.6). It is shown in BIB004 BIB007 BIB022 that for these deterministic bounds, if the number of available LR images is K and the improvement factors along the x and y directions are s x and s y , respectively, under local translation condition between the LR images, the sufficient number of LR images for a reconstruction-based SR algorithm is 4(s x s y ) 2 . Having more LR images may only marginally improve the quality of the reconstructed image, e.g., in BIB004 BIB007 BIB022 it is discussed that such an improvement can be seen in only some disjoint intervals. Another class of bounds on the improvement factors of reconstruction-based SR algorithms is stochastic in nature and gives statistical performance bounds which are much more complex to interpret. These are more accurate and have proven to be more reliable, but they are not used very often because there are many "it depends" scenarios, which are to some extent clarified in BIB008 BIB023 BIB033 BIB056 where using BIB001 BIB002 BIB003 46, BIB005 BIB006   BIB009 BIB010 BIB011 BIB012 BIB013 BIB014 BIB015 BIB016 BIB017 BIB018 BIB019 BIB020 BIB024 BIB025 BIB026 BIB027 BIB028 BIB029 BIB030 BIB031 BIB034 BIB035 BIB032  BIB036 BIB037 BIB033 BIB038 BIB039 BIB040 BIB042 BIB043 BIB044 BIB045 BIB046 BIB047 BIB048 BIB049 BIB050 BIB051 BIB052 BIB053 BIB057 BIB058 BIB059 BIB060 BIB061 BIB062 BIB063 BIB064  BIB065 BIB066 BIB070  BIB071 BIB067 BIB072 BIB073 BIB074 BIB077 BIB078 BIB068 BIB079 BIB080 BIB081  BIB082 BIB083 BIB084 BIB085 BIB086 BIB087 BIB088 BIB089 BIB090 614 ,617] 3 [ 84,85,87,113,124,126,138,151,160,174,226-228,241,245,251,252,279,281,283, 285,306,350,368,381,392,404,405,407,440,443,444,494,515,518,521,546,552, 558,559,564,577,580,587,594,596,598,600,602,606,609,611,613,618] 4 [ 30,36,37,61,84,85,90,91,94,105,112,113,127,128,136,140,144,151,157,160,164, 165,167,174,180,187-189,191-194,200,201,207,208,211,217,226-228,235,248,253,257,259,267,270,273,279,287,302,303,310,311,313,314,321, 323,337,344,346,348,350,354,368,370,374,378,380,387,392,393,395,404,405, 407,412-414,424,426,436,440,443,444,446,450,452,455,457-459,472,475,476,481-483,494-496,503,506,512,515,521,526,528,532,536,538, 541,543,546,548,549,551,556,557, It is discussed in BIB021 that one may check the practical limits of reconstruction-based SR algorithms which are regularized by Tikhonov-like terms by discrete picard condition (DPC) . In BIB041  , a closed-form lower bound for the expected risk of the root mean square error (RMSE) function between the ground truth images and the super-resolved images obtained by learning-based SR algorithms has been estimated. Here, it is discussed that having obtained the curve of the lower bound against the improvement factor, a threshold at which the lower bound of the expected risk of learningbased SR algorithms might exceed some acceptable values can be estimated. In BIB069 BIB091 , it is discussed that preconditioning the system can allow for rational improvement factors BIB086 . In BIB092 , it is discussed that all the images in the available LR sequence do not necessarily provide useful information for the actual reconstructed image. To measure the usefulness of the images, a priority measure is assigned to each of them based on a confidence measure, to maximize the reconstruction accuracy, and at the same time to minimize the number of the sample sets. Despite the above-mentioned theoretical thresholds, the reported improvement factors listed in Table 12 are mostly higher than these bounds. Besides the improvement factors reported in the table the following ones have been reported by at least one paper: 9 BIB075 , 10 BIB054 BIB076 , 12 BIB087 , and 24 BIB055 .