Super-resolution: a comprehensive survey <s> Blur estimation <s> Abstract Given a low resolution camera, we would like to get an image with improved resolution using camera motion. From several low resolution images, with subpixel relative displacements, we get an improved resolution image. We find a high resolution image such that when simulating the imaging process we get low resolution images closest to the observed images. The method is tested on computer simulations, and simulated annealing is also used for the optimization process. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> An iterative algorithm to increase image resolution is described. Examples are shown for low-resolution gray-level pictures, with an increase of resolution clearly observed after only a few iterations. The same method can also be used for deblurring a single blurred image. The approach is based on the resemblance of the presented problem to the reconstruction of a 2-D object from its 1-D projections in computer-aided tomography. The algorithm performed well for both computer-simulated and real images and is shown, theoretically and practically, to converge quickly. The algorithm can be executed in parallel for faster hardware implementation.<<ETX>> <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> In several applications it is required to reconstruct a high-resolution noise-free image from multipath frames of undersampled low-resolution noisy images. Using the aliasing relationship between the undersamples frames and the reference image, an algorithm based on weighted recursive least-squares theory is developed in the wavenumber domain. This algorithm is efficient because interpolation and noise removal are performed recursively, and is highly suitable for implementation via the massively parallel computational architectures currently available. Success in the use of the algorithm is demonstrated through various simulated examples. > <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> Abstract Image resolution can be improved when the relative displacements in image sequences are known accurately, and some knowledge of the imaging process is available. The proposed approach is similar to back-projection used in tomography. Examples of improved image resolution are given for gray-level and color images, when the unknown image displacements are computed from the image sequence. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> A method for detecting and tracking multiple moving objects, using both a large spatial region and a large temporal region, without assuming temporal motion constancy is described. When the large spatial region of analysis has multiple moving objects, the motion parameters and the locations of the objects are computed for one object after another. A method for segmenting the image plane into differently moving objects and computing their motions using two frames is presented. The tracking of detected objects using temporal integration and the algorithms for enhancement of tracked objects by filling-in occluded regions and by improving the spatial resolution of the imaged objects are described. > <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> The authors address the problem of reconstruction of a high-resolution image from a number of lower-resolution (possibly noisy) frames of the same scene where the successive frames are uniformly based versions of each other at subpixel displacements. In particular, two previously proposed methods, a frequency-domain method and a method based on projections onto convex sets (POCSs), are extended to take into account the presence of both sensor blurring and observation noise. A new two-step procedure is proposed, and it is shown that the POCS formulation presented for the high-resolution image reconstruction problem can also be used as a new method for the restoration of spatially invariant blurred images. Some simulation results are provided. > <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> This paper introduces a new algorithm for enhancing image resolution from an image sequence. The approach we propose herein uses the integrating resampler for warping. The method is a direct computation, which is fundamentally different from the iterative back-projection approaches proposed in previous work. This paper shows that image-warping techniques may have a strong impact on the quality of image resolution enhancement. By coupling the degradation model of the imaging system directly into the integrating resampler, we can better approximate the warping characteristics of real sensors, which also significantly improve the quality of super-resolution images. Examples of super-resolutions are given for gray-scale images. Evaluations are made visually by comparing the resulting images and those using bi-linear resampling and back-projection and quantitatively using OCR as a fundamental measure. The paper shows that even when the images are qualitatively similar, quantitative differences appear in machine processing. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> Super-resolution reconstruction deals with the recovery of a single high-resolution image from a set of warped, blurred and decimated versions of it. Several algorithms were already proposed for the solution of this general problem. We concentrate on a special case where the warps are pure translations, the blur is space invariant and the same for all the images, and the noise is white. We exploit previous results to develop a new highly efficient super-resolution reconstruction algorithm for this case, which separates the treatment to de-blurring and measurements fusion. The fusion part is shown to be a non-iterative and very simple algorithm, preserving the optimality of the entire reconstruction process, in the maximum-likelihood sense. Simulations demonstrate the capabilities of the proposed algorithm. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> In many image restoration/resolution enhancement applications, the blurring process, i.e., point spread function (PSF) of the imaging system, is not known or is known only to within a set of parameters. We estimate these PSF parameters for this ill-posed class of inverse problem from raw data, along with the regularization parameters required to stabilize the solution, using the generalized cross-validation method (GCV). We propose efficient approximation techniques based on the Lanczos algorithm and Gauss quadrature theory, reducing the computational complexity of the GCV. Data-driven PSF and regularization parameter estimation experiments with synthetic and real image sequences are presented to demonstrate the effectiveness and robustness of our method. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> This paper introduces a practical approach for superresolution, the process of reconstructing a high-resolution image from the low-resolution input ones. The emphasis of our work is to super-resolve frames from dynamic video sequences, which may contain significant object occlusion or scene changes. As the quality of super-resolved images highly relies on the correctness of image alignment between consecutive frames, we employ the robust optical flow method to accurately estimate motion between the image pair. An efficient and reliable scheme is designed to detect and discard incorrect matchings, which may degrade the output quality. We also introduce the usage of elliptical weighted average (EWA) filter to model the spatially variant point spread function (PSF) of acquisition system in order to improve accuracy of the model. A number of complex and dynamic video sequences are tested to demonstrate the applicability and reliability of our algorithm. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> Super-Resolution reconstruction produces one or a set of high-resolution images from a sequence of low-resolution frames. This article reviews a variety of Super-Resolution methods proposed in the last 20 years, and provides some insight into, and a summary of, our recent contributions to the general Super-Resolution problem. In the process, a detailed study of several very important aspects of Super-Resolution, often ignored in the literature, is presented. Spe- cifically, we discuss robustness, treatment of color, and dynamic operation modes. Novel methods for addressing these issues are accompanied by experimental results on simulated and real data. Finally, some future challenges in Super-Resolution are outlined and discussed. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 14, 47-57, 2004; Published online in Wiley InterScience (www.interscience.wiley. com). DOI 10.1002/ima.20007 <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods neglect camera response function, exposure time, white balancing, and external illumination changes. In this paper, we show how to extend traditional super-resolution reconstruction methods to handle these factors. We provide formulations for several different super-resolution reconstruction approaches. Experimental results are also included. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> In traditional digital image restoration, the blurring process of the optic is assumed known. Many previous research efforts have been trying to reconstruct the degraded image or video sequence with either partially known or totally unknown point spread function (PSF) of the optical lens, which is commonly called the blind deconvolution problem. Many methods have been proposed in the application to image restoration. However, little work has been done in the super-resolution scenario. In this paper, we propose a generalized framework of regularized image/video iterative blind deconvolution / super-resolution (IBD-SR) algorithm, using some information from the more matured blind deconvolution techniques from image restoration. The initial estimates for the image restoration help the iterative image/video super-resolution algorithm converge faster and be stable. Experimental results are presented and conclusions are drawn. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods assume that exposure time is fixed for all observations, which is not necessarily true. In reality, cameras have limited dynamic range and nonlinear response to the quantity of light received, and exposure time might be adjusted automatically or manually to capture the desired portion of the scene's dynamic range. In this letter, we propose a Bayesian super-resolution algorithm based on an imaging model that includes camera response function, exposure time, sensor noise, and quantization error in addition to spatial blurring and sampling. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> In this paper, we present a super-resolution (SR) method based on iterative backprojections. Motion information is used for the synthesis of the restoration filter in the SR method. Both, the blur estimation and the choice of the degradation model, are based on estimated global motion. The method is applied to highly under-sampled images such as DC images of MPEG compressed video and medical magnetic resonance (MR) images. Results of the comparison of our blur model with some common blur models are encouraging. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> This paper addresses the problem of super-resolving a single image and recovering the characteristics of the sensor using a learning-based approach. In particular, the point spread function (PSF) of the camera is sought by minimizing the mean Euclidean distance function between patches from the input frame and from degraded versions of high-resolution training images. Once an estimate of the PSF is obtained, a supervised learning algorithm can then be used as is. Results are compared with another method for blind super-resolution by using a series of quality measures. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> Motion blur can degrade the quality of images and is considered a nuisance for computer vision problems. In this paper, we show that motion blur can in-fact be used for increasing the resolution of a moving object. Our approach utilizes the information in a single motion-blurred image without any image priors or training images. As the blur size increases, the resolution of the moving object can be enhanced by a larger factor, albeit with a corresponding increase in reconstruction noise. Traditionally, motion deblurring and super-resolution have been ill-posed problems. Using a coded-exposure camera that preserves high spatial frequencies in the blurred image, we present a linear algorithm for the combined problem of deblurring and resolution enhancement and analyze the invertibility of the resulting linear system. We also show a method to selectively enhance the resolution of a narrow region of high-frequency features, when the resolution of the entire moving object cannot be increased due to small motion blur. Results on real images showing up to four times resolution enhancement are presented. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> In image reconstruction, dynamic super resolutionimage reconstruction algorithms have been investigated toenhance video frames sequentially, where explicit motionestimation is considered as a major factor in theperformance. This paper proposes a novel measurementvalidation method to attain robust image reconstructionresults under inaccurate motion estimation. In addition, wepresent an effective scene change detection methoddedicated to the proposed super resolution technique forminimizing erroneous results when abrupt scene changesoccur in the video frames. Representative experimentalresults show excellent performance of the proposedalgorithm in terms of the reconstruction quality andprocessing speed. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> Spatial Super Resolution (SR) aims to recover fine image details, smaller than a pixel size. Temporal SR aims to recover rapid dynamic events that occur faster than the video frame-rate, and are therefore invisible or seen incorrectly in the video sequence. Previous methods for Space-Time SR combined information from multiple video recordings of the same dynamic scene. In this paper we show how this can be done from a single video recording. Our approach is based on the observation that small space-time patches (‘ST-patches’, e.g., 5×5×3) of a single ‘natural video’, recur many times inside the same video sequence at multiple spatio-temporal scales. We statistically explore the degree of these ST-patch recurrences inside ‘natural videos’, and show that this is a very strong statistical phenomenon. Space-time SR is obtained by combining information from multiple ST-patches at sub-frame accuracy. We show how finding similar ST-patches can be done both efficiently (with a randomized-based search in space-time), and at sub-frame accuracy (despite severe motion aliasing). Our approach is particularly useful for temporal SR, resolving both severe motion aliasing and severe motion blur in complex ‘natural videos’. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Blur estimation <s> Although multi-frame super resolution has been extensively studied in past decades, super resolving real-world video sequences still remains challenging. In existing systems, either the motion models are oversimplified, or important factors such as blur kernel and noise level are assumed to be known. Such models cannot deal with the scene and imaging conditions that vary from one sequence to another. In this paper, we propose a Bayesian approach to adaptive video super resolution via simultaneously estimating underlying motion, blur kernel and noise level while reconstructing the original high-res frames. As a result, our system not only produces very promising super resolution results that outperform the state of the art, but also adapts to a variety of noise levels and blur kernels. Theoretical analysis of the relationship between blur kernel, noise level and frequency-wise reconstruction rate is also provided, consistent with our experimental results. <s> BIB020
This step is responsible for compensating for the blur differences between the LR images, with the ultimate goal of deblurring the super-resolved HR image. In most of the SR works, blur is explicitly involved in the imaging model. The blur effects in this group of algorithms are caused by the imaging device, atmospheric effects, and/or by motion. The blurring effect of the imaging device is usually modeled by a so-called point spread function, which is usually a squared Gaussian kernel with a suitable standard deviation, e.g., 3×3 with standard deviation of 0.4  , 5×5 with standard deviation of 1 BIB011 BIB012 BIB014 , 15 × 15 with standard deviation of 1.7 BIB013 , and so on. If the point spread function of the lens is not available from its manufacturer, it is usually estimated by scanning a small dot on a black background BIB002 BIB004 BIB005 . If the imaging device is not available, but only a set of LR images is, it can be estimated by techniques known as Blind Deconvolution BIB016 in which the blur can be estimated by degradation of features like small points or sharp edges or by techniques like generalized cross validation (GCV) [68, BIB009 . In this group, the blur can be estimated globally (space-invariant blurring) BIB001  BIB002 BIB003 BIB004 BIB005 or locally (space-variant blurring). Local blurring effects for SR were first proposed by Chiang et al. [30, 36, 37] by modeling the edges of the image as a step function v + δu where v is the unknown intensity value and δ is the unknown amplitude of the edge. The local blur of the edge is then modeled by a Gaussian blur kernel with an unknown standard deviation. The unknowns are found by imposing some constraints on the reconstruction model that they use [30, 36, 37] . Shekarfroroush and Chellappa  used a generalization of Papoulis's sampling theorem and shifting property between consecutive frames to estimate local blur for every frame. The blurring caused by motion depends on the direction of the motion, the velocity, and the exposure time BIB015 BIB017 BIB019 . It is shown in BIB019 that temporal blur induces temporal aliasing and can be exploited to improve the SR of moving objects in video sequences. Instead of estimating the point spread function, in a second group of SR algorithms known as direct methods (Sect. 5.1.3), a deblurring filter is used after the actual reconstruction of the HR image [30, 36, 37, 58, BIB007 . Using a high-pass filter for deblurring in the context of SR was first proposed by Keren et al.  . In Tekalp et al. BIB006 and then in [58, BIB008 BIB015 BIB018 BIB020 a Wiener filter and in BIB010 an elliptical weighted area filter has been used for this purpose.