Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> Super-resolution reconstruction deals with the recovery of a single high-resolution image from a set of warped, blurred and decimated versions of it. Several algorithms were already proposed for the solution of this general problem. We concentrate on a special case where the warps are pure translations, the blur is space invariant and the same for all the images, and the noise is white. We exploit previous results to develop a new highly efficient super-resolution reconstruction algorithm for this case, which separates the treatment to de-blurring and measurements fusion. The fusion part is shown to be a non-iterative and very simple algorithm, preserving the optimality of the entire reconstruction process, in the maximum-likelihood sense. Simulations demonstrate the capabilities of the proposed algorithm. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> In the last two decades, many papers have been published, proposing a variety methods of multiframe resolution enhancement. These methods are usually very sensitive to their assumed model of data and noise, which limits their utility. This paper reviews some of these methods and addresses their shortcomings. We propose a different implementation using L/sub 1/ norm minimization and robust regularization to deal with different data and noise models. This computationally inexpensive method is robust to errors in motion and blur estimation, and results in sharp edges. Simulation results confirm the effectiveness of our method and demonstrate its superiority to other robust super-resolution methods. <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> The attainment of super resolution (SR) from a sequence of degraded undersampled images could be viewed as reconstruction of the high-resolution (HR) image from a finite set of its projections on a sampling lattice. This can then be formulated as an optimization problem whose solution is obtained by minimizing a cost function. The approaches adopted and their analysis to solve the formulated optimization problem are crucial, The image acquisition scheme is important in the modeling of the degradation process. The need for model accuracy is undeniable in the attainment of SR along with the design of the algorithm whose robust implementation will produce the desired quality in the presence of model parameter uncertainty. To keep the presentation focused and of reasonable size, data acquisition with multisensors instead of, say a video camera is considered. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> Super-resolution image reconstruction produces a high-resolution image from a set of shifted, blurred, and decimated versions thereof. Previously published papers have not addressed the computational complexity of this ill-conditioned large scale problem adequately. In this paper, the computational complexity of MAP-based multiframe super-resolution algorithms is studied, and a new fast algorithm, as well as methods for parallel image reconstruction is also presented. The proposed fast algorithm splits the multiple input low-resolution images into several subsets according to their translation relations, and then applies normal MAP algorithm to each subset, the reconstructed images are processed subsequently at a successive level until the desired resolution is achieved. Experiment results are also provided to demonstrate the efficiency of the proposed techniques. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> Image superresolution methods process an input image sequence of a scene to obtain a still image with increased resolution. Classical approaches to this problem involve complex iterative minimization procedures, typically with high computational costs. In this paper is proposed a novel algorithm for super-resolution that enables a substantial decrease in computer load. First, a probabilistic neural network architecture is used to perform a scattered-point interpolation of the image sequence data. The network kernel function is optimally determined for this problem by a multi-layer perceptron trained on synthetic data. Network parameters dependence on sequence noise level is quantitatively analyzed. This super-sampled image is spatially filtered to correct finite pixel size effects, to yield the final high-resolution estimate. Results on a real outdoor sequence are presented, showing the quality of the proposed method. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> We study the problem of reconstruction of a high-resolution image from several blurred low-resolution image frames. The image frames consist of blurred, decimated, and noisy versions of a high-resolution image. The high-resolution image is modeled as a Markov random field (MRF), and a maximum a posteriori (MAP) estimation technique is used for the restoration. We show that with the periodic boundary condition, a high-resolution image can be restored efficiently by using fast Fourier transforms. We also apply the preconditioned conjugate gradient method to restore high-resolution images in the aperiodic boundary condition. Computer simulations are given to illustrate the effectiveness of the proposed approach. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> Super-Resolution Reconstruction (SRR) aims to produces one or a set of high-resolution (HR) images from a sequence of low-resolution (LR) images. Due to translational registration, super-resolution reconstruction can apply only on the sequences that have simple translation motion. This paper proposed a novel super-resolution reconstruction that that can apply on real sequences or complex motion sequences. The proposed SRR uses a high accuracy registration algorithm, the fast affine block-based registration [16], in the maximum likelihood framework. Moreover, the BTV (Bi-Total Variance) regularization [12] is used to compensate the missing measurement information. The experimental results show that the proposed reconstruction can apply on real sequence such as Suzie. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> A computationally simple super-resolution algorithm using a type of adaptive Wiener filter is proposed. The algorithm produces an improved resolution image from a sequence of low-resolution (LR) video frames with overlapping field of view. The algorithm uses subpixel registration to position each LR pixel value on a common spatial grid that is referenced to the average position of the input frames. The positions of the LR pixels are not quantized to a finite grid as with some previous techniques. The output high-resolution (HR) pixels are obtained using a weighted sum of LR pixels in a local moving window. Using a statistical model, the weights for each HR pixel are designed to minimize the mean squared error and they depend on the relative positions of the surrounding LR pixels. Thus, these weights adapt spatially and temporally to changing distributions of LR pixels due to varying motion. Both a global and spatially varying statistical model are considered here. Since the weights adapt with distribution of LR pixels, it is quite robust and will not become unstable when an unfavorable distribution of LR pixels is observed. For translational motion, the algorithm has a low computational complexity and may be readily suitable for real-time and/or near real-time processing applications. With other motion models, the computational complexity goes up significantly. However, regardless of the motion model, the algorithm lends itself to parallel implementation. The efficacy of the proposed algorithm is demonstrated here in a number of experimental results using simulated and real video sequences. A computational analysis is also presented. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> In this paper, we propose a maximum a posteriori ramework for the super-resolution problem, i.e., reconstructing high-resolution images from shifted, rotated, low-resolution degraded observations. The main contributions of this work are two; first, the use of a new locally adaptive edge preserving prior for the super-resolution problem. Second an efficient two-step reconstruction methodology that includes first an initial registration using only the low-resolution degraded observations. This is followed by a fast iterative algorithm implemented in the discrete Fourier transform domain in which the restoration, interpolation and the registration subtasks of this problem are preformed simultaneously. We present examples with both synthetic and real data that demonstrate the advantages of the proposed framework. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> Although the topic of Super-Resolution Reconstruction (SRR) has recently received considerable attention within the traditional research community, the SRR estimations are based on LI or L2 statistical norm estimation. Therefore, these SRR methods are very sensitive to their assumed data and noise models, which limit their applications. The real noise models that corrupt the measure sequence are unknown; consequently, SRR algorithm using LI or L2 norm may degrade the image sequence rather than enhance it. The robust norm applicable to several noise and data models is desired in SRR algorithms. This paper proposes an alternate SRR approach based on the stochastic regularization technique of Bayesian MAP estimation by minimizing a cost function. The Lorentzian norm is used for measuring the difference between the projected estimate of the high-resolution image and each low resolution image, removing outliers in the data. Tikhonov regularization is used to remove artifacts from the final result and improve the rate of convergence. In order to cope with real sequences and complex motion sequences, the fast affine block-based registration is used in the registration step of SRR. The experimental results show that the proposed reconstruction can be applied on real sequences such as Suzie sequence and confirm the effectiveness of our method and demonstrate its superiority to other super-resolution methods based on LI and L2 norm for several noise models such as AWGN, Poisson and Salt & Pepper noise. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> We propose a novel, learning-based algorithm for image super-resolution. First, an optimal distance-based weighted interpolation of the image sequence is performed using a new neural architecture, hybrid of a multi-layer perceptron and a probabilistic neural network, trained on synthetic image data. Secondly, a linear filter is applied with coefficients learned to restore residual interpolation artifacts in addition to low-resolution blurring, providing noticeable improvements over lens-detector Wiener restorations. Our method has been evaluated on real visible and IR sequences with widely different contents, providing significantly better results that a two-step method with high computational requirements. Results were similar or better than those of a maximum-a-posteriori estimator, with a reduction in processing time by a factor of almost 300. This paves the way to high-quality, quasi-real time applications of super-resolution techniques. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> In this paper we propose a novel super-resolution algorithm based on motion compensation and edge-directed spatial interpolation succeeded by fusion via pixel classification. Two high-resolution images are constructed, the first by means of motion compensation and the second by means of edge-directed interpolation. The AdaBoost classifier is then used to fuse these images into an high-resolution frame. Experimental results show that the proposed method surpasses well-known resolution enhancement methods while maintaining moderate computational complexity. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> In general, all the video super-resolution (SR) algorithms present the important drawback of a very high computational load, mainly due to the huge amount of operations executed by the motion estimation (ME) stage. Commonly, there is a trade-off between the accuracy of the estimated motion, given as a motion vector (MV), and the computational cost associated. In this sense, the ME algorithms that explore more exhaustively the search area among images use to deliver better MVs, at the cost of a higher computational load and resources use. Due to this reason, the proper choice of a ME algorithm is a key factor not only to reach real-time applications, but also to obtain high quality video sequences independently of their characteristics. Under the hardware point of view, the preferred ME algorithms are based on matching fixed-size blocks in different frames. In this paper, a comparison of nine of the most representative Fast Block Matching Algorithms (FBMAs) is made in order to select the one which presents the best tradeoff between video quality and computational cost, thus allowing reliable real-time hardware implementations of video super-resolution systems. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> A typical dynamic reconstruction-based super-resolution video involves three independent processes: registration, fusion and restoration. Fast video super-resolution systems apply translational motion compensation model for registration with low computational cost. Traditional motion compensation model assumes that the whole spectrum of pixels is consistent between frames. In reality, the low frequency component of pixels often varies significantly. We propose a translational motion compensation model via frequency classification for video super-resolution systems. A novel idea to implement motion compensation by combining the up-sampled current frame and the high frequency part of the previous frame through the SAD framework is presented. Experimental results show that the new motion compensation model via frequency classification has an advantage of 2dB gain on average over that of the traditional motion compensation model. The SR quality has 0.25dB gain on average after the fusion process which is to minimize error by making use of the new motion compensated frame. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> A non-iterative method of image super-resolution based on weighted median filtering with Gaussian weights is proposed. Visual tests and basic edges metrics were used to examine the method. It was shown that the weighted median filtering reduces the errors caused by inaccurate motion vectors. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Speed of SR algorithms <s> Image Super-Resolution: Historical Overview and Future Challenges, J. Yang and T. Huang Introduction to Super-Resolution Notations Techniques for Super-Resolution Challenge issues for Super-Resolution Super-Resolution Using Adaptive Wiener Filters, R.C. Hardie Introduction Observation Model AWF SR Algorithms Experimental Results Conclusions Acknowledgments Locally Adaptive Kernel Regression for Space-Time Super-Resolution, H. Takeda and P. Milanfar Introduction Adaptive Kernel Regression Examples Conclusion AppendiX Super-Resolution With Probabilistic Motion Estimation, M. Protter and M. Elad Introduction Classic Super-Resolution: Background The Proposed Algorithm Experimental Validation Summary Spatially Adaptive Filtering as Regularization in Inverse Imaging, A. Danielyan, A. Foi, V. Katkovnik, and K. Egiazarian Introduction Iterative filtering as regularization Compressed sensing Super-resolution Conclusions Registration for Super-Resolution, P. Vandewalle, L. Sbaiz, and M. Vetterli Camera Model What Is Resolution? Super-Resolution as a Multichannel Sampling Problem Registration of Totally Aliased Signals Registration of Partially Aliased Signals Conclusions Towards Super-Resolution in the Presence of Spatially Varying Blur, M. Sorel, F. Sroubek and J. Flusser Introduction Defocus and Optical Aberrations Camera Motion Blur Scene Motion Algorithms Conclusion Acknowledgments Toward Robust Reconstruction-Based Super-Resolution, M. Tanaka and M. Okutomi Introduction Overviews Robust SR Reconstruction with Pixel Selection Robust Super-Resolution Using MPEG Motion Vectors Robust Registration for Super-Resolution Conclusions Multi-Frame Super-Resolution from a Bayesian Perspective, L. Pickup, S. Roberts, A. Zisserman and D. Capel The Generative Model Where Super-Resolution Algorithms Go Wrong Simultaneous Super-Resolution Bayesian Marginalization Concluding Remarks Variational Bayesian Super Resolution Reconstruction, S. Derin Babacan, R. Molina, and A.K. Katsaggelos Introduction Problem Formulation Bayesian Framework for Super Resolution Bayesian Inference Variational Bayesian Inference Using TV Image Priors Experiments Estimation of Motion and Blur Conclusions Acknowledgements Pattern Recognition Techniques for Image Super-Resolution, K. Ni and T.Q. Nguyen Introduction Nearest Neighbor Super-Resolution Markov Random Fields and Approximations Kernel Machines for Image Super-Resolution Multiple Learners and Multiple Regressions Design Considerations and Examples Remarks Glossary Super-Resolution Reconstruction of Multi-Channel Images, O.G. Sezer and Y. Altunbasak Introduction Notation Image Acquisition Model Subspace Representation Reconstruction Algorithm Experiments & Discussions Conclusion New Applications of Super-Resolution in Medical Imaging, M.D.Robinson, S.J. Chiu, C.A. Toth, J.A. Izatt, J.Y. Lo, and S. Farsiu Introduction The Super-Resolution Framework New Medical Imaging Applications Conclusion Acknowledgment Practicing Super-Resolution: What Have We Learned? N. Bozinovic Abstract Introduction MotionDSP: History and Concepts Markets and Applications Technology Results Lessons Learned Conclusions <s> BIB016
SR algorithms usually suffer from high computational requirements, which obviously reduces their speed performances. For example, POCS algorithms may oscillate between the responses and get too slow to converge. Some MAP methods, if they use non-convex prior terms for their regularization, may require computationally heavy optimization methods, like simulated annealing. Some researchers have tried to develop fast SR systems by imposing some conditions on the input images of the system. For example, in - BIB001 , a ML-based method was developed which was fast but only when the motion between its LR inputs is pure translational. Similar approach of considering translational motion has been followed in BIB014 . - BIB002 BIB012 BIB015 , a ML method was developed for the case that SR problem is over-determined. In this case the there is no need for a regularization term and the huge matrix calculations can be replaced by shift and add operations. This is mostly the case for the direct SR algorithm in which different types of filters are utilized, for example in BIB008 BIB016 adaptive Wiener filter has been used (Sect. 5.1.3). - BIB003 BIB006 , preconditioning of the system has been used so that one can use a faster optimization method, like CG. In some SR algorithms it has been tried to improve the speed of a specific step of the system. For example: -in BIB007 BIB009 BIB010 BIB013 faster registration algorithms are used. -In BIB004 parallel processing has been used. -Some of the learning-based approaches perform fast in their testing phase, but they need a time-consuming training step and are mostly application dependent, like in BIB005  BIB011 .