Super-resolution: a comprehensive survey <s> Error and noise <s> This paper describes a Bayesian method for constructing a super-resolved surface model by combining information from a set of images of the given surface. We develop the theory and algorithms in detail for the 2-D reconstruction problem, appropriate for the case where all images are taken from roughly the same direction and under similar lighting conditions. We show the results of this 2-D reconstruction on Viking Martian data. These results show dramatic improvements in both spatial and gray-scale resolution. The Bayesian approach uses a neighbor correlation model as well as pixel data from the image set. Some extensions of this method are discussed, including 3-D surface reconstruction and the resolution of diffraction blurred images. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> An iterative algorithm to increase image resolution is described. Examples are shown for low-resolution gray-level pictures, with an increase of resolution clearly observed after only a few iterations. The same method can also be used for deblurring a single blurred image. The approach is based on the resemblance of the presented problem to the reconstruction of a 2-D object from its 1-D projections in computer-aided tomography. The algorithm performed well for both computer-simulated and real images and is shown, theoretically and practically, to converge quickly. The algorithm can be executed in parallel for faster hardware implementation.<<ETX>> <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> Abstract Image resolution can be improved when the relative displacements in image sequences are known accurately, and some knowledge of the imaging process is available. The proposed approach is similar to back-projection used in tomography. Examples of improved image resolution are given for gray-level and color images, when the unknown image displacements are computed from the image sequence. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> A method for detecting and tracking multiple moving objects, using both a large spatial region and a large temporal region, without assuming temporal motion constancy is described. When the large spatial region of analysis has multiple moving objects, the motion parameters and the locations of the objects are computed for one object after another. A method for segmenting the image plane into differently moving objects and computing their motions using two frames is presented. The tracking of detected objects using temporal integration and the algorithms for enhancement of tracked objects by filling-in occluded regions and by improving the spatial resolution of the imaged objects are described. > <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> Superresolution reconstruction produces a high-resolution image from a set of low-resolution images. Previous iterative methods for superresolution had not adequately addressed the computational and numerical issues for this ill-conditioned and typically underdetermined large scale problem. We propose efficient block circulant preconditioners for solving the Tikhonov-regularized superresolution problem by the conjugate gradient method. We also extend to underdetermined systems the derivation of the generalized cross-validation method for automatic calculation of regularization parameters. The effectiveness of our preconditioners and regularization techniques is demonstrated with superresolution results for a simulated sequence and a forward looking infrared (FLIR) camera image sequence. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> In the last two decades, many papers have been published, proposing a variety methods of multiframe resolution enhancement. These methods are usually very sensitive to their assumed model of data and noise, which limits their utility. This paper reviews some of these methods and addresses their shortcomings. We propose a different implementation using L/sub 1/ norm minimization and robust regularization to deal with different data and noise models. This computationally inexpensive method is robust to errors in motion and blur estimation, and results in sharp edges. Simulation results confirm the effectiveness of our method and demonstrate its superiority to other robust super-resolution methods. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> Super-Resolution reconstruction produces one or a set of high-resolution images from a sequence of low-resolution frames. This article reviews a variety of Super-Resolution methods proposed in the last 20 years, and provides some insight into, and a summary of, our recent contributions to the general Super-Resolution problem. In the process, a detailed study of several very important aspects of Super-Resolution, often ignored in the literature, is presented. Spe- cifically, we discuss robustness, treatment of color, and dynamic operation modes. Novel methods for addressing these issues are accompanied by experimental results on simulated and real data. Finally, some future challenges in Super-Resolution are outlined and discussed. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 14, 47-57, 2004; Published online in Wiley InterScience (www.interscience.wiley. com). DOI 10.1002/ima.20007 <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> In this paper, we propose a robust approach to super-resolution static sprite generation from multiple low-resolution images. Considering both short-term and long-term motion influences, a hybrid global motion estimation technique is first presented for sprite generation. An iterative super-resolution reconstruction algorithm is then proposed for the super-resolution sprite construction. This algorithm is robust to outliers and results in a sprite image with high visual quality. In addition, the generated super-resolution sprite can provide better reconstructed images than a conventional low-resolution sprite. Experimental results demonstrate the effectiveness of the proposed methods. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods neglect camera response function, exposure time, white balancing, and external illumination changes. In this paper, we show how to extend traditional super-resolution reconstruction methods to handle these factors. We provide formulations for several different super-resolution reconstruction approaches. Experimental results are also included. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> Theoretical and practical limitations usually constrain the achievable resolution of any imaging device. Super-Resolution (SR) methods are developed through the years to go beyond this limit by acquiring and fusing several low-resolution (LR) images of the same scene, producing a high-resolution (HR) image. The early works on SR, ::: although occasionally mathematically optimal for particular models of data and noise, produced poor results when applied to real images. In this paper, we discuss two of the main issues related to designing a practical SR system, namely reconstruction accuracy and computational efficiency. Reconstruction accuracy refers to the problem of designing a robust SR method applicable to images from different imaging systems. We study a general framework for optimal reconstruction of images from grayscale, color, or color filtered (CFA) cameras. The performance of our proposed method is boosted by using powerful priors and is robust to both measurement (e.g. CCD read out noise) and system noise (e.g. motion estimation error). Noting that the motion estimation is often considered a bottleneck in terms of SR performance, we introduce the concept of "constrained motions" for enhancing the quality of super-resolved images. We show that using such constraints will enhance the quality of the motion estimation and therefore results in more accurate reconstruction of the HR images. We also justify some practical assumptions that greatly reduce the computational complexity and memory requirements of the proposed methods. We use efficient approximation of the Kalman Filter (KF) and adopt a dynamic point of view to the SR problem. Novel methods for addressing these issues are accompanied by experimental results on real data. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods assume that exposure time is fixed for all observations, which is not necessarily true. In reality, cameras have limited dynamic range and nonlinear response to the quantity of light received, and exposure time might be adjusted automatically or manually to capture the desired portion of the scene's dynamic range. In this letter, we propose a Bayesian super-resolution algorithm based on an imaging model that includes camera response function, exposure time, sensor noise, and quantization error in addition to spatial blurring and sampling. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> In the last two decades, two related categories of problems have been studied independently in image restoration literature: super-resolution and demosaicing. A closer look at these problems reveals the relation between them, and, as conventional color digital cameras suffer from both low-spatial resolution and color-filtering, it is reasonable to address them in a unified context. In this paper, we propose a fast and robust hybrid method of super-resolution and demosaicing, based on a maximum a posteron estimation technique by minimizing a multiterm cost function. The L1 norm is used for measuring the difference between the projected estimate of the high-resolution image and each low-resolution image, removing outliers in the data and errors due to possibly inaccurate motion estimation. Bilateral regularization is used for spatially regularizing the luminance component, resulting in sharp edges and forcing interpolation along the edges and not across them. Simultaneously, Tikhonov regularization is used to smooth the chrominance components. Finally, an additional regularization term is used to force similar edge location and orientation in different color channels. We show that the minimization of the total cost function is relatively easy and fast. Experimental results on synthetic and real data sets confirm the effectiveness of our method. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> Although the topic of Super-Resolution Reconstruction (SRR) has recently received considerable attention within the traditional research community, the SRR estimations are based on LI or L2 statistical norm estimation. Therefore, these SRR methods are very sensitive to their assumed data and noise models, which limit their applications. The real noise models that corrupt the measure sequence are unknown; consequently, SRR algorithm using LI or L2 norm may degrade the image sequence rather than enhance it. The robust norm applicable to several noise and data models is desired in SRR algorithms. This paper proposes an alternate SRR approach based on the stochastic regularization technique of Bayesian MAP estimation by minimizing a cost function. The Lorentzian norm is used for measuring the difference between the projected estimate of the high-resolution image and each low resolution image, removing outliers in the data. Tikhonov regularization is used to remove artifacts from the final result and improve the rate of convergence. In order to cope with real sequences and complex motion sequences, the fast affine block-based registration is used in the registration step of SRR. The experimental results show that the proposed reconstruction can be applied on real sequences such as Suzie sequence and confirm the effectiveness of our method and demonstrate its superiority to other super-resolution methods based on LI and L2 norm for several noise models such as AWGN, Poisson and Salt & Pepper noise. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> Recently, there has been a great deal of work developing super-resolution reconstruction (SRR) algorithms. While many such algorithms have been proposed, the almost SRR estimations are based on L1 or L2 statistical norm estimation, therefore these SRR algorithms are usually very sensitive to their assumed noise model that limits their utility. The real noise models that corrupt the measure sequence are unknown; consequently, SRR algorithm using L1 or L2 norm may degrade the image sequence rather than enhance it. Therefore, the robust norm applicable to several noise and data models is desired in SRR algorithms. This paper first comprehensively reviews the SRR algorithms in this last decade and addresses their shortcomings, and latter proposes a novel robust SRR algorithm that can be applied on several noise models. The proposed SRR algorithm is based on the stochastic regularization technique of Bayesian MAP estimation by minimizing a cost function. For removing outliers in the data, the Lorentzian error norm is used for measuring the difference between the projected estimate of the high-resolution image and each low-resolution image. Moreover, Tikhonov regularization and Lorentzian-Tikhonov regularization are used to remove artifacts from the final answer and improve the rate of convergence. The experimental results confirm the effectiveness of our method and demonstrate its superiority to other super-resolution methods based on L1 and L2 norms for several noise models such as noiseless, additive white Gaussian noise (AWGN), poisson noise, salt and pepper noise, and speckle noise. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> A hybrid error model with L 1 and L 2 norm minimization criteria is proposed in this paper for image/video super-resolution. A membership function is defined to adaptively control the tradeoff between the L 1 and L 2 norm terms. Therefore, the proposed hybrid model can have the advantages of both L 1 norm minimization (i.e. edge preservation) and L 2 norm minimization (i.e. smoothing noise). In addition, an effective convergence criterion is proposed, which is able to terminate the iterative L 1 and L 2 norm minimization process efficiently. Experimental results on images corrupted with various types of noises demonstrate the robustness of the proposed algorithm and its superiority to representative algorithms. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Error and noise <s> As digital video cameras spread rapidly, critical locations such as airports, train stations, military compounds and airbases and public ''hot-spots'' are placed in the spotlight of video surveillance. Video surveillance provides visual information, in order to maintain security at the monitored areas. Indeed, super-resolution (SR) technique can be useful for extracting additional information from the captured video sequences. Generally, video surveillance cameras capture moving objects while cameras are moving in some cases, which means that many of the existing SR algorithms, that cannot cope with moving objects, are not applicable in this case. In this paper, we propose a SR algorithm that takes into account inaccurate registration at the moving regions and therefore copes with moving objects. We propose to adaptively weight each region according to its reliability where regions that have local motion and/or occlusion have different registration error level. Also, the regularization parameter is simultaneously estimated for each region. The regions are generated by segmenting the reference frame using watershed segmentation. Our approach is tested on simulated and real data coming from videos with different difficulties taken by a hand-held camera. The experimental results show the effectiveness of the proposed algorithm compared to four state-of-the-art SR algorithms. <s> BIB016
In real-world applications, the discussed registration steps are error prone. This gets aggregated when inconsistent pixels are present in some of the LR input images. Such pixels may emerge when there are, e.g., -moving objects that are present in only some LR images, like a bouncing ball or a flying bird   BIB016 . -Outliers in the input. Outliers are defined as data points with different distributional characteristics than the assumed model BIB006  BIB007   . A system is said to be robust if it is not sensitive to these errors. To study the robustness of an algorithm against outlier, the concept of breakdown point is used. A breaksown point is, the smallest percentage of outlier contamination leading the results of the estimation to be outside of some acceptable range  . For example, a single outlier is enough to move the results of a mean estimator outside of any predicted range, i.e., the breakdown point of a mean estimator is zero. This value for a median estimator is 0.5, i.e., this estimator is robust to outliers when their contamination is less than 50 percent of all the data point  BIB008 . Besides errors in estimating the parameters of the system, an SR system may suffer from noise. Several sources of noise can be imagined in such a system, including telemetry noise (e.g., in satellite imaging) BIB001 , measurement noise (e.g., shot noise in a CCD, analog to digital conversion noise BIB010 ), and thermal noise BIB009 BIB011 . The performance of the HR estimator has a sensitive dependence on the model assumed for the noise. If this model does not fully describe the measured data, the results of the estimator will be erroneous. Several types of noise models are used with SR algorithms, for example: -linear noise (i.e., additive noise) addressed by: -averaging the LR pixels  BIB002 BIB003 BIB004 BIB005 -modeling the noise as a: • Gaussian (using an l 2 norm estimator). • Laplacian (using an l 1 norm estimator) BIB006  BIB007   BIB010 BIB012  BIB015  which has been shown to be more accurate than a Gaussian distribution. -non-linear noise (i.e., multiplicative noise) addressed by: -eliminating extreme LR pixels  BIB002 BIB003 BIB004 . -Lorentzian modeling. In BIB013 BIB014 it has been discussed that employing l 1 and l 2 norms for modeling the noise is valid only if the noise involved in the imaging model of Eq. (4) is additive white Gaussian noise, but the actual model of the noise is not known. Therefore, it has been discussed to use Lorentzian norm for modeling the noise, which is more robust than l 1 and l 2 from a statistical point of view. This norm is defined by: where r is the reconstruction error and T is the Lorentzian constant. Having discussed the imaging model and the parameters involved in the typical SR algorithms, the actual reconstructions of these algorithms are discussed in the following sections, according to the order given in Fig. 1 .