Super-resolution: a comprehensive survey <s> Iterative back projection <s> An iterative algorithm to increase image resolution is described. Examples are shown for low-resolution gray-level pictures, with an increase of resolution clearly observed after only a few iterations. The same method can also be used for deblurring a single blurred image. The approach is based on the resemblance of the presented problem to the reconstruction of a 2-D object from its 1-D projections in computer-aided tomography. The algorithm performed well for both computer-simulated and real images and is shown, theoretically and practically, to converge quickly. The algorithm can be executed in parallel for faster hardware implementation.<<ETX>> <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Abstract Image resolution can be improved when the relative displacements in image sequences are known accurately, and some knowledge of the imaging process is available. The proposed approach is similar to back-projection used in tomography. Examples of improved image resolution are given for gray-level and color images, when the unknown image displacements are computed from the image sequence. <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> A method for detecting and tracking multiple moving objects, using both a large spatial region and a large temporal region, without assuming temporal motion constancy is described. When the large spatial region of analysis has multiple moving objects, the motion parameters and the locations of the objects are computed for one object after another. A method for segmenting the image plane into differently moving objects and computing their motions using two frames is presented. The tracking of detected objects using temporal integration and the algorithms for enhancement of tracked objects by filling-in occluded regions and by improving the spatial resolution of the imaged objects are described. > <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Abstract Accurate computation of image motion enables the enhancement of image sequences. In scenes having multiple moving objects the motion computation is performed together with object segmentation by using a unique temporal integration approach. After the motion for the different image regions is computed, these regions can be enhanced by fusing several successive frames covering the same region. Enhancements treated here include improvement of image resolution, filling-in occluded regions, and reconstruction of transparent objects. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> The method for reconstruction and restoration of super-resolution images from sets of low-resolution images presented is an extension of the algorithm proposed by Irani and Peleg (1991). After estimating the projective transformation parameters between the image sequence frames, the observed data are transformed into a sequence with only quantised sub-pixel translations. The super-resolution reconstruction is an iterative process, in which a high-resolution image is initialised and iteratively improved. The improvement is achieved by back-projecting the errors between the translated low-resolution images and the respective images obtained by simulating the imaging system. The imaging system's point-spread function (PSF) and the back-projection function are first estimated with a resolution higher than that of the super-resolution image. The two functions are then decimated so that two banks of polyphase filters are obtained. The use of the polyphase filters allows exploitation of the input data without any smoothing and/or interpolation operations. The presented experimental results show that the resolution improvement is better than the results obtained with Irani and Peleg's algorithm. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Mosaicing and super resolution are two ways to combine information from multiple frames in video sequences. Mosaicing displays the information of multiple frames in a single panoramic image. Super-resolution uses regions which appear in multiple frames to improve resolution and reduce noise. The aim of this work is constructing a high resolution mosaic from a video sequence in an efficient way. Simple combination of the two methods is problematic, since the alignment used in mosaicing may not be accurate enough for super resolution. Another issue is the efficiency of the super resolution algorithm, which requires heavy computations, especially when applied to large images such as panoramic mosaics. This paper introduces two novelties. First, a framework for super resolution algorithms is presented, which enables the development of very efficient algorithms. Second, a method for applying super resolution to panoramic mosaics is presented. This method preserves the geometry of the original mosaic image, while improving its resolution. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> A robust approach for super-resolution is, presented, which is especially valuable in the presence of outliers. Such outliers may be due to motion errors, inaccurate blur models, noise, moving objects, motion blur etc. This robustness is needed since super-resolution methods are very sensitive to such errors. A robust median estimator is combined in an iterative process to achieve a super resolution algorithm. This process can increase resolution even in regions with outliers, where other super resolution methods actually degrade the image. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Subsequent to the work of Kim, Bose, and Valenzuela in 1990 on the simultaneous filtering and interpolation of a registered sequence of undersampled noisy and shift-invariant blur degraded images, Bose and Boo tackled in 1998 the problem of reconstructing a high-resolution image from multiple undersampled, shifted, degraded frames with subpixel displacement errors. This led to a formulation involving a periodically shift-variant system model. Lertrattanapanich and Bose advanced in 1999 a procedure for construction of a high-resolution video mosaic following the estimation of motion parameters between successive frames in a video sequence generated from a video camera. Current research is focused on simultaneous blur identification and robust superresolution. The blur is not restricted to be linear shift-invariant and could not only be of the linear shift-variant type but also some nonlinear blurs could be accommodated. The optimal tuning parameter may, if desired, be calculated analytically and not by trial-and-error. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> In the last two decades, many papers have been published, proposing a variety methods of multiframe resolution enhancement. These methods are usually very sensitive to their assumed model of data and noise, which limits their utility. This paper reviews some of these methods and addresses their shortcomings. We propose a different implementation using L/sub 1/ norm minimization and robust regularization to deal with different data and noise models. This computationally inexpensive method is robust to errors in motion and blur estimation, and results in sharp edges. Simulation results confirm the effectiveness of our method and demonstrate its superiority to other robust super-resolution methods. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Super-resolution imaging is to overcome the inherent limitations of image acquisition to create high-resolution images from their low-resolution counterparts. In this paper, a novel state-space approach is proposed to incorporate the temporal correlations among the low-resolution observations into the framework of the Kalman filtering. The proposed approach exploits both the temporal correlations information among the high-resolution images and the temporal correlations information among the low-resolution images to improve the quality of the reconstructed high-resolution sequence. Experimental results show that the proposed framework is superior to bi-linear interpolation, bi-cubic spline interpolation and the conventional Kalman filter approach, due to the consideration of the temporal correlations among the low-resolution images. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Resolution improvement from several images is typically restricted to simple planar rotations and translations. In this paper, a super-resolution algorithm that allows 3D ego-motion of the camera system is proposed. By exploiting the trifocal tensor constraint of a stereo camera system, the intermediate step of scene structure recovery is effectively skipped. 3D motion is estimated recursively through an extended Kalman filter and used by a novel image warping procedure to perform resolution enhancement. Real image experiment with comparison to well known Irani-Peleg approach confirms the validity of the algorithm. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Theoretical and practical limitations usually constrain the achievable resolution of any imaging device. Super-Resolution (SR) methods are developed through the years to go beyond this limit by acquiring and fusing several low-resolution (LR) images of the same scene, producing a high-resolution (HR) image. The early works on SR, ::: although occasionally mathematically optimal for particular models of data and noise, produced poor results when applied to real images. In this paper, we discuss two of the main issues related to designing a practical SR system, namely reconstruction accuracy and computational efficiency. Reconstruction accuracy refers to the problem of designing a robust SR method applicable to images from different imaging systems. We study a general framework for optimal reconstruction of images from grayscale, color, or color filtered (CFA) cameras. The performance of our proposed method is boosted by using powerful priors and is robust to both measurement (e.g. CCD read out noise) and system noise (e.g. motion estimation error). Noting that the motion estimation is often considered a bottleneck in terms of SR performance, we introduce the concept of "constrained motions" for enhancing the quality of super-resolved images. We show that using such constraints will enhance the quality of the motion estimation and therefore results in more accurate reconstruction of the HR images. We also justify some practical assumptions that greatly reduce the computational complexity and memory requirements of the proposed methods. We use efficient approximation of the Kalman Filter (KF) and adopt a dynamic point of view to the SR problem. Novel methods for addressing these issues are accompanied by experimental results on real data. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> In the last two decades, two related categories of problems have been studied independently in image restoration literature: super-resolution and demosaicing. A closer look at these problems reveals the relation between them, and, as conventional color digital cameras suffer from both low-spatial resolution and color-filtering, it is reasonable to address them in a unified context. In this paper, we propose a fast and robust hybrid method of super-resolution and demosaicing, based on a maximum a posteron estimation technique by minimizing a multiterm cost function. The L1 norm is used for measuring the difference between the projected estimate of the high-resolution image and each low-resolution image, removing outliers in the data and errors due to possibly inaccurate motion estimation. Bilateral regularization is used for spatially regularizing the luminance component, resulting in sharp edges and forcing interpolation along the edges and not across them. Simultaneously, Tikhonov regularization is used to smooth the chrominance components. Finally, an additional regularization term is used to force similar edge location and orientation in different color channels. We show that the minimization of the total cost function is relatively easy and fast. Experimental results on synthetic and real data sets confirm the effectiveness of our method. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Effective image prior is necessary for image super resolution, due to its severely under-determined nature. Although the edge smoothness prior can be effective, it is generally difficult to have analytical forms to evaluate the edge smoothness, especially for soft edges that exhibit gradual intensity transitions. This paper finds the connection between the soft edge smoothness and a soft cut metric on an image grid by generalizing the Geocuts method (Y. Boykov and V. Kolmogorov, 2003), and proves that the soft edge smoothness measure approximates the average length of all level lines in an intensity image. This new finding not only leads to an analytical characterization of the soft edge smoothness prior, but also gives an intuitive geometric explanation. Regularizing the super resolution problem by this new form of prior can simultaneously minimize the length of all level lines, and thus resulting in visually appealing results. In addition, this paper presents a novel combination of this soft edge smoothness prior and the alpha matting technique for color image super resolution, by normalizing edge segments with their alpha channel description, to achieve a unified treatment of edges with different contrast and scale. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> In this paper, a novel algorithm for single image super resolution is proposed. Back-projection [1] can minimize the reconstruction error with an efficient iterative procedure. Although it can produce visually appealing result, this method suffers from the chessboard effect and ringing effect, especially along strong edges. The underlining reason is that there is no edge guidance in the error correction process. Bilateral filtering can achieve edge-preserving image smoothing by adding the extra information from the feature domain. The basic idea is to do the smoothing on the pixels which are nearby both in space domain and in feature domain. The proposed bilateral back-projection algorithm strives to integrate the bilateral filtering into the back-projection method. In our approach, the back-projection process can be guided by the edge information to avoid across-edge smoothing, thus the chessboard effect and ringing effect along image edges are removed. Promising results can be obtained by the proposed bilateral back-projection method efficiently. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Super-resolution reconstruction of image sequences is highly dependent on the quality of the motion estimation between successive frames. This work presents a statistical analysis of the least mean square (LMS) algorithm applied to super-resolution reconstruction of an image sequence. Deterministic recursions are derived for the mean and mean square behaviors of the reconstruction error as functions of the registration errors. The new model describes the behavior of the algorithm in realistic situations, and significantly improves the accuracy of a simple model available in the literature. Monte Carlo simulations show good agreement between actual and predicted behaviors <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> This paper addresses the problem of generating a super-resolution (SR) image from a single low-resolution input image. We approach this problem from the perspective of compressed sensing. The low-resolution image is viewed as downsampled version of a high-resolution image, whose patches are assumed to have a sparse representation with respect to an over-complete dictionary of prototype signal-atoms. The principle of compressed sensing ensures that under mild conditions, the sparse representation can be correctly recovered from the downsampled signal. We will demonstrate the effectiveness of sparsity as a prior for regularizing the otherwise ill-posed super-resolution problem. We further show that a small set of randomly chosen raw patches from training images of similar statistical nature to the input image generally serve as a good dictionary, in the sense that the computed representation is sparse and the recovered high-resolution image is competitive or even superior in quality to images produced by other SR methods. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> Designing effective image priors is of great interest to image super-resolution (SR), which is a severely under-determined problem. An edge smoothness prior is favored since it is able to suppress the jagged edge artifact effectively. However, for soft image edges with gradual intensity transitions, it is generally difficult to obtain analytical forms for evaluating their smoothness. This paper characterizes soft edge smoothness based on a novel SoftCuts metric by generalizing the Geocuts method . The proposed soft edge smoothness measure can approximate the average length of all level lines in an intensity image. Thus, the total length of all level lines can be minimized effectively by integrating this new form of prior. In addition, this paper presents a novel combination of this soft edge smoothness prior and the alpha matting technique for color image SR, by adaptively normalizing image edges according to their alpha-channel description. This leads to the adaptive SoftCuts algorithm, which represents a unified treatment of edges with different contrasts and scales. Experimental results are presented which demonstrate the effectiveness of the proposed method. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> The super-resolution reconstruction (SRR) of images is an ill posed problem. Traditionally, it is treated as a regularized minimization problem. Moreover, one of the major problems concerning SRR is its dependence on an accurate registration. In this paper, we show that a certain amount of registration error may, in fact, be beneficial for the performance of the least mean square SRR (LMS-SRR) adaptive algorithm. In these cases, the regularization term may be avoided, leading to reduction in computational cost that can be important in real-time SRR applications. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> We present experimental results of digital super resolution (DSR) techniques on low resolution data collected using PANOPTES, a multi-aperture miniature folded imaging architecture. The flat form factor of PANOPTES architecture results in an optical system that is heavily blurred with space variant PSF which makes super resolution challenging. We also introduce a new DSR method called SRUM (Super-Resolution with Unsharpenning Mask) which can efficiently highlight edges by embedding an unsharpenning mask to the cost function. This has much better effect than just applying the mask after all iterations as a post-processing step. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> With the VASI, we propose a low-power video acquisition, which captures a video at low-resolution (LR) interlaced with some at full-resolution. The power consumed during acquisition is reduced by turning off some of the sensors when acquiring LR frames. Afterwards, an efficient super-resolution (SR) technique is employed to reconstruct the LR frames to high-resolution (HR). Another application of the proposed framework is to overcome the pixel transfer rate limit to obtain high-frame-rate and HR video. Experiments verify the efficacy of the proposed framework in both saving power and enhancing video quality. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Iterative back projection <s> This paper discusses resolution enhancement of a set of images with varying exposure durations, having a high combined dynamic range. So far, little has been said in relation to the Human Visual System when it comes to Super-Resolution and High Dynamic Range fusion, unlike the case for traditional Super-Resolution where errors are measured with respect to human perception in the pixel domain. We propose a Super-Resolution method in the L*a*b* domain to bridge that gap and present some image reconstruction results. <s> BIB022
Iterative back projection (IBP) methods (Table 5) were among the first methods developed for spatial-based SR. Having defined the imaging model like, e.g., the one given in Eq. (5), these methods then try to minimize ||A f − g|| 2 2 . To do so, usually an initial guess for the HR targeted image is generated and then it is refined. Such a guess can be obtained by registering the LR images over an HR grid and then averaging them BIB001 BIB002 BIB003 BIB004 . To refine this initial guess f (0) , the imaging model given in Eq. (4) is used to simulate the set of the available LR observations, g Then the error between the simulated LR images and the observed ones, which is computed by k || 2 2 (t is the number of iterations), is obtained and back-projected to the coordinates of the HR image to improve the initial guess BIB004 . This process is either repeated for a specific number of iterations or until no further improvement can be achieved. To do so, usually the following Richardson iteration is used in this group of algorithms: in which t is an iteration parameter, w k is the inverse of the warping kernel of Eq. (4),ḋ is an upsampling operator, * represents a convolution operation, andḣ is a debluring kernel which has the following relationship with the blurring kernel of the imaging model of Eq. (4) BIB004 : wherein δ is the unity pulse function centered at (0, 0) BIB004 . If the value of a pixel does not change for a specific number of iterations, its value will be considered as found and the pixel will not accompany the other pixels in the rest of the iterations. This increases the speed of the algorithm. As can be seen from Eq. (22), the back-projected error is the mean of the errors that each LR image causes. In BIB007 BIB009  it has been suggested to replace this mean by the median to get a faster algorithm. In BIB011 this method has been extended to the case where the LR images are captured by a stereo setup. The main problem with the above-mentioned IBP methods is that the response of the iteration can either converge to one of the possible solutions or it may oscillate between them BIB001 BIB002 BIB003 BIB004 BIB005 BIB008 . However, this can be dealt with by incorporating a priori knowledge about the solution, as has been done in BIB006 BIB008 BIB009 BIB014 BIB015 BIB017 BIB018 BIB020 BIB021 BIB022 . In this case, these algorithms will try to minimize ||A f − g|| 2 + λ||ρ( f )|| 2 , wherein λ is a regularization coefficient and ρ is a constraint on the solution. In BIB009  BIB012 BIB013 it has been suggested to replace the l 2 norm by l 1 in both the residual term and the regularization term. Besides increasing the speed of the algorithm it has been shown that this increases the robustness of the algorithm against the outliers which can be generated by different sources of errors, such as errors in the motion estimation BIB009 . (24) in which B k models the relationship between the current and the previous HR image and ζ represents the error of estimating B k . Besides these two equations, which were considered in the original works on using the Kalman filter for SR [62] [63] [64] BIB016 BIB019 , it is shown in BIB010 that modeling the relationship between the LR images can also be incorporated into the estimation of the HR images. To do so, a third equation is employed: