Super-resolution: a comprehensive survey <s> Imaging models <s> An iterative algorithm to increase image resolution is described. Examples are shown for low-resolution gray-level pictures, with an increase of resolution clearly observed after only a few iterations. The same method can also be used for deblurring a single blurred image. The approach is based on the resemblance of the presented problem to the reconstruction of a 2-D object from its 1-D projections in computer-aided tomography. The algorithm performed well for both computer-simulated and real images and is shown, theoretically and practically, to converge quickly. The algorithm can be executed in parallel for faster hardware implementation.<<ETX>> <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Accurate image expansion is important in many areas of image analysis. Common methods of expansion, such as linear and spline techniques, tend to smooth the image data at edge regions. This paper introduces a method for nonlinear image expansion which preserves the discontinuities of the original image, producing an expanded image with improved definition. The maximum a posteriori (MAP) estimation techniques that are proposed for noise-free and noisy images result in the optimization of convex functionals. The expanded images produced from these methods will be shown to be aesthetically and quantitatively superior to images expanded by the standard methods of replication, linear interpolation, and cubic B-spline expansion. > <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> In many applications, like surveillance, image sequences are of poor quality. Motion blur in particular introduces significant image degradation. An interesting challenge is to merge these many images into one high-quality, estimated still. We propose a method to achieve this. Firstly, an object of interest is tracked through the sequence using region based matching. Secondly, degradation of images is modelled in terms of pixel sampling, defocus blur and motion blur. Motion blur direction and magnitude are estimated from tracked displacements. Finally, a high-resolution deblurred image is reconstructed. The approach is illustrated with video sequences of moving people and blurred script. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> We analyze the super-resolution reconstruction constraints. In particular we derive a sequence of results which all show that the constraints provide far less useful information as the magnification factor increases. It is well established that the use of a smoothness prior may help somewhat, however for large enough magnification factors any smoothness prior leads to overly smooth results. We therefore propose an algorithm that learns recognition-based priors for specific classes of scenes, the use of which gives far better super-resolution results for both faces and text. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> In this chapter, we present a parametric method for generating a super-resolution image from a sequence consisting of blurred and noisy observations. The high resolution image is modeled as a Markov random field (MRF) and a maximum a posteriori (MAP) estimation technique is used for super-resolution restoration. Unlike other super-resolution imaging methods, the proposed technique does not require sub-pixel registration of given observations. A simple gradient descent method is used to optimize the cost. The discontinuities in the intensity process can be preserved by introducing suitable line processes. Superiority of this technique to standard methods of image interpolation is illustrated. The motivation for using blur as a cue is also explained. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Multiframe resolution enhancement ("superresolution") methods are becoming widely studied, but only a few procedures have been developed to work with compressed video, despite the fact that compression is a standard component of most image- and video-processing applications. One of these methods uses quantization-bound information to define convex sets and then employs a technique called "projections onto convex sets" (POCS) to estimate the original image. Another uses a discrete cosine transformation (DCT)-domain Bayesian estimator to enhance resolution in the presence of both quantization and additive noise. The latter approach is also capable of incorporating known source statistics and other reconstruction constraints to impose blocking artifact reduction and edge enhancement as part of the solution. We propose a spatial-domain Bayesian estimator that has advantages over both of these approaches. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Face images that are captured by surveillance cameras usually have a very low resolution, which significantly limits the performance of face recognition systems. In the past, super-resolution techniques have been proposed that attempt to increase the resolution by combining information from multiple images. These techniques use super-resolution as a preprocessing system to obtain a high resolution image that can later be passed to a face recognition system. Considering that most state-of-the-art face recognition systems use an initial dimensionality reduction method, we propose embedding the super-resolution algorithm into the face recognition system so that super-resolution is not performed in the pixel domain, but is instead performed in a reduced dimensional domain. The advantage of such an approach is a significant decrease in the computational complexity of the super-resolution algorithm because the algorithm no longer tries to construct a visually improved high quality image, but instead constructs the information required by the recognition algorithm directly in the lower dimensional domain without any unnecessary overhead. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Surveys the field of super resolution (SR) processing for compressed video. The introduction of motion vectors, compression noise, and additional redundancies within the image sequence makes this problem fertile ground for novel processing methods. In conducting this survey, though, we develop and present all techniques within the Bayesian framework. This adds consistency to the presentation and facilitates comparison between the different methods. The article is organized as follows. We define the acquisition system utilized by the surveyed procedures. Then we formulate the HR problem within the Bayesian framework and survey models for the acquisition and compression systems. This requires consideration of both the motion vectors and transform coefficients within the compressed bit stream. We survey models for the original HR image intensities and displacement values. We discuss solutions for the SR problem and provide examples of several approaches. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Some models for super-resolution restoration assume that low-resolution (LR) images are formed from high-resolution (HR) ones by a warping first, blurring second (followed by decimating) imaging process; whereas others adopt a blurring first, warping second imaging constraint. These models are abused to some extent and respective conditions for usability of them are not identified. Such ambiguities are analyzed and cleared up in this letter. Conclusions are: The warping-blurring model coincides with the general imaging physics, but it is usable only if the motion among HR images is known a priori. When the motion is estimated from LR images, this model may cause systematic error, but the use of the blurring-warping model is more appropriate, and leads to better performance. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Face hallucination is used to synthesize a high-resolution facial image from a low-resolution input. In this paper, we present a framework for face hallucination with pose variation. We derive a texture model consisting of a set of linear mappings between the Gabor wavelet features of the facial images of every two possible poses. Given a low-resolution facial image, its pose is first estimated using a SVM classifier. Then, the Gabor wavelet feature corresponding to the frontal face is computed by our texture model and the low-resolution frontal facial image is reconstructed from its Gabor wavelet feature by a novel algorithm we propose. Finally, the high-resolution face can be hallucinated by one of the hallucination approaches for frontal faces. Our framework is demonstrated by extensive experiments with high-quality hallucinated results. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> In this paper, we propose a novel technique for super-resolution imaging of a scene from observations at different zoom levels. Given a sequence of images with different zoom factors of a static scene, the problem is to obtain a picture of the entire scene at a resolution corresponding to the most zoomed image in the scene. We not only obtain the super-resolved image for known integer zoom factors, but also for unknown arbitrary zoom factors. We model the super-resolution image as a Markov random field (MRF) and a maximum a posteriori (MAP) estimation method is used to derive a cost function which is then optimized to recover the high-resolution field. The entire observation conforms to the same MRF, but is viewed at the different resolution pyramid. Since there is no relative motion between the scene and the camera, as is the case with most of the super-resolution techniques, we do away with the correspondence problem. Results of the experimentation on real data are presented. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Super-Resolution reconstruction produces one or a set of high-resolution images from a sequence of low-resolution frames. This article reviews a variety of Super-Resolution methods proposed in the last 20 years, and provides some insight into, and a summary of, our recent contributions to the general Super-Resolution problem. In the process, a detailed study of several very important aspects of Super-Resolution, often ignored in the literature, is presented. Spe- cifically, we discuss robustness, treatment of color, and dynamic operation modes. Novel methods for addressing these issues are accompanied by experimental results on simulated and real data. Finally, some future challenges in Super-Resolution are outlined and discussed. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 14, 47-57, 2004; Published online in Wiley InterScience (www.interscience.wiley. com). DOI 10.1002/ima.20007 <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Novel theoretical results for the super-resolution reconstruction (SRR) are presented under the case of arbitrary image warping. The SRR model is reasonably separated into two parts of anti-aliasing and deblurring. The anti-aliasing part is proved to be well-posed. The ill-posedness of the entire SRR process is shown to be mainly caused by the deblurring part. The motion estimation error results in a multiplicative perturbation to the warping matrix, and the perturbation bound is derived. The common regularization algorithms used in SRR are analyzed through the discrete Picard condition, which provides a theoretical measure for limits on SRR. Experiments and examples are supplied to validate the presented theories. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> In traditional digital image restoration, the blurring process of the optic is assumed known. Many previous research efforts have been trying to reconstruct the degraded image or video sequence with either partially known or totally unknown point spread function (PSF) of the optical lens, which is commonly called the blind deconvolution problem. Many methods have been proposed in the application to image restoration. However, little work has been done in the super-resolution scenario. In this paper, we propose a generalized framework of regularized image/video iterative blind deconvolution / super-resolution (IBD-SR) algorithm, using some information from the more matured blind deconvolution techniques from image restoration. The initial estimates for the image restoration help the iterative image/video super-resolution algorithm converge faster and be stable. Experimental results are presented and conclusions are drawn. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> We propose a technique for super-resolution imaging of a scene from observations at different camera zooms. Given a sequence of images with different zoom factors of a static scene, we obtain a picture of the entire scene at a resolution corresponding to the most zoomed observation. The high-resolution image is modeled through appropriate parameterization, and the parameters are learned from the most zoomed observation. Assuming a homogeneity of the high-resolution field, the learned model is used as a prior while super-resolving the scene. We suggest the use of either a Markov random field (MRF) or an simultaneous autoregressive (SAR) model to parameterize the field based on the computation one can afford. We substantiate the suitability of the proposed method through a large number of experimentations on both simulated and real data. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Hyperspectral images are used for aerial and space imagery applications, including target detection, tracking, agricultural, and natural resource exploration. Unfortunately, atmospheric scattering, secondary illumination, changing viewing angles, and sensor noise degrade the quality of these images. Improving their resolution has a high payoff, but applying super-resolution techniques separately to every spectral band is problematic for two main reasons. First, the number of spectral bands can be in the hundreds, which increases the computational load excessively. Second, considering the bands separately does not make use of the information that is present across them. Furthermore, separate band super resolution does not make use of the inherent low dimensionality of the spectral data, which can effectively be used to improve the robustness against noise. In this paper, we introduce a novel super-resolution method for hyperspectral images. An integral part of our work is to model the hyperspectral image acquisition process. We propose a model that enables us to represent the hyperspectral observations from different wavelengths as weighted linear combinations of a small number of basis image planes. Then, a method for applying super resolution to hyperspectral images using this model is presented. The method fuses information from multiple observations and spectral bands to improve spatial resolution and reconstruct the spectrum of the observed scene as a combination of a small number of spectral basis functions. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods neglect camera response function, exposure time, white balancing, and external illumination changes. In this paper, we show how to extend traditional super-resolution reconstruction methods to handle these factors. We provide formulations for several different super-resolution reconstruction approaches. Experimental results are also included. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> We propose a method for constructing a video sequence of high space-time resolution by combining information from multiple low-resolution video sequences of the same dynamic scene. Super-resolution is performed simultaneously in time and in space. By "temporal super-resolution," we mean recovering rapid dynamic events that occur faster than regular frame-rate. Such dynamic events are not visible (or else are observed incorrectly) in any of the input sequences, even if these are played in "slow-motion." The spatial and temporal dimensions are very different in nature, yet are interrelated. This leads to interesting visual trade-offs in time and space and to new video applications. These include: 1) treatment of spatial artifacts (e.g., motion-blur) by increasing the temporal resolution and 2) combination of input sequences of different space-time resolutions (e.g., NTSC, PAL, and even high quality still images) to generate a high quality video sequence. We further analyze and compare characteristics of temporal super-resolution to those of spatial super-resolution. These include: the video cameras needed to obtain increased resolution; the upper bound on resolution improvement via super-resolution; and, the temporal analogue to the spatial "ringing" effect. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> An efficient algorithm using Maximum a posteriori- Markov random field (MAP-MRF) based approach for recovering a high-resolution image from multiple sub-pixel shifted low-resolution images is proposed. The algorithm can be used for super-resolution of both space-invariant and space-variant blurred images. We prove an important theorem that the posterior is also Markov and derive the exact posterior neighborhood structure in the presence of warping, blurring and down-sampling operations. The posterior being Markov enables us to perform all matrix operations as local image domain operations thereby resulting in a considerable speedup. Experimental results are given to demonstrate the effectiveness of our method. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> This paper presents a new method for super-resolution (SR) reconstruction of a high-resolution (HR) image from several low-resolution (LR) images. The HR image is assumed to be composed of homogeneous regions. Thus, the a priori distribution of the pixels is modeled by a finite mixture model (FMM) and a Potts Markov model (PMM) for the labels. The whole a priori model is then a hierarchical Markov model. The LR images are assumed to be obtained from the HR image by lowpass filtering, arbitrarily translation, decimation, and finally corruption by a randomnoise. The problem is then put in a Bayesian detection and estimation framework, and appropriate algorithms are developed based on Markov chain Monte Carlo (MCMC) Gibbs sampling. At the end, we have not only an estimate of the HR image but also an estimate of the classification labels which leads to a segmentation result. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Recently, new sampling schemes were presented for signals with finite rate of innovation (FRI) using sampling kernels reproducing polynomials or exponentials [1] [2]. In this paper, we extend those sampling schemes to a distributed acquisition architecture in which numerous and randomly located sensors are pointing to the same area of interest. We emphasize the importance played by moments and show how to acquire efficiently FRI signals with a set of sensors. More importantly, we also show that those sampling schemes can be used for accurate registration of affine transformed and low-resolution images. Based on this, a new super-resolution algorithm was developed and showed good preliminary results. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> This paper develops a multi-frame image super-resolution approach from a Bayesian view-point by marginalizing over the unknown registration parameters relating the set of input low-resolution views. In Tipping and Bishop's Bayesian image super-resolution approach [16], the marginalization was over the super-resolution image, necessitating the use of an unfavorable image prior. By integrating over the registration parameters rather than the high-resolution image, our method allows for more realistic prior distributions, and also reduces the dimension of the integral considerably, removing the main computational bottleneck of the other algorithm. In addition to the motion model used by Tipping and Bishop, illumination components are introduced into the generative model, allowing us to handle changes in lighting as well as motion. We show results on real and synthetic datasets to illustrate the efficacy of this approach. <s> BIB022 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Super-resolution reconstruction of image sequences is highly dependent on the data outliers and on the quality of the motion estimation. This work addresses the design of the least mean square (LMS) algorithm applied to super-resolution reconstruction. Based on a statistical model of the algorithm behavior, we propose a design strategy to reduce the effects of outliers on the reconstructed image sequence. We show that the proposed strategy can improve the algorithm performance in both transient and steady-state phases of adaptation in practical situations, when registration errors are considered. <s> BIB023 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Optical zooming is an important feature of imaging systems. In this paper, we investigate a low-cost signal processing alternative to optical zooming-synthetic zooming by super-resolution (SR) techniques. Synthetic zooming is achieved by registering a sequence of low-resolution (LR) images acquired at varying focal lengths and reconstructing the SR image at a larger focal length or increased spatial resolution. Under the assumptions of constant scene depth and zooming speed, we argue that the motion trajectories of all physical points are related to each other by a unique vanishing point and present a robust technique for estimating its 3D coordinate. Such a line-geometry-based registration is the foundation of SR for synthetic zooming. We address the issue of data inconsistency arising from the varying focal length of optical lens during the zooming process. To overcome the difficulty of data inconsistency, we propose a two-stage Delaunay-triangulation-based interpolation for fusing the LR image data. We also present a PDE-based nonlinear deblurring to accommodate the blindness and variation of sensor point spread functions. Simulation results with real-world images have verified the effectiveness of the proposed SR techniques for synthetic zooming. <s> BIB024 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> In this paper we present a new super resolution Bayesian method for pansharpening of multispectral images which: a) incorporates prior knowledge on the expected characteristics of the multispectral images, b) uses the sensor characteristics to model the observation process of both panchromatic and multispectral images, and c) performs the estimation of all the unknown parameters in the model. Using real data, the pan-sharpened multispectral images are compared with the images obtained by other parsharpening methods and their quality is assessed both qualitatively and quantitatively. <s> BIB025 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods assume that exposure time is fixed for all observations, which is not necessarily true. In reality, cameras have limited dynamic range and nonlinear response to the quantity of light received, and exposure time might be adjusted automatically or manually to capture the desired portion of the scene's dynamic range. In this letter, we propose a Bayesian super-resolution algorithm based on an imaging model that includes camera response function, exposure time, sensor noise, and quantization error in addition to spatial blurring and sampling. <s> BIB026 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> We present a super-resolution technique where the 3-D shape preservation is used as a constraint while super-resolving a scene. Given the observations under different illuminant positions, we combine these observations to obtain the super-resolved image and the spatially enhanced scene structure simultaneously. The use of shape cue in the form of photometric measurements, instead of the motion cue, eliminates the need for image registration with a sub-pixel accuracy. We model the high-resolution image, the structure, and the albedo of the surface as separate Markov random fields and super-resolve them using a suitable regularization scheme. Results of the experimentation on real data are presented. <s> BIB027 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Motion blur can degrade the quality of images and is considered a nuisance for computer vision problems. In this paper, we show that motion blur can in-fact be used for increasing the resolution of a moving object. Our approach utilizes the information in a single motion-blurred image without any image priors or training images. As the blur size increases, the resolution of the moving object can be enhanced by a larger factor, albeit with a corresponding increase in reconstruction noise. Traditionally, motion deblurring and super-resolution have been ill-posed problems. Using a coded-exposure camera that preserves high spatial frequencies in the blurred image, we present a linear algorithm for the combined problem of deblurring and resolution enhancement and analyze the invertibility of the resulting linear system. We also show a method to selectively enhance the resolution of a narrow region of high-frequency features, when the resolution of the entire moving object cannot be increased due to small motion blur. Results on real images showing up to four times resolution enhancement are presented. <s> BIB028 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> We present a novel method of Bayesian image super-resolution in which marginalization is carried out over latent parameters such as geometric and photometric registration and the image point-spread function. Related Bayesian super-resolution approaches marginalize over the high-resolution image, necessitating the use of an unfavourable image prior, whereas our method allows for more realistic image prior distributions, and reduces the dimension of the integral considerably, removing the main computational bottleneck of algorithms such as Tipping and Bishop's Bayesian image super-resolution. We show results on real and synthetic datasets to illustrate the efficacy of our method. <s> BIB029 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Super-resolution reconstruction of image sequences is highly dependent on the quality of the motion estimation between successive frames. This work presents a statistical analysis of the least mean square (LMS) algorithm applied to super-resolution reconstruction of an image sequence. Deterministic recursions are derived for the mean and mean square behaviors of the reconstruction error as functions of the registration errors. The new model describes the behavior of the algorithm in realistic situations, and significantly improves the accuracy of a simple model available in the literature. Monte Carlo simulations show good agreement between actual and predicted behaviors <s> BIB030 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> We present a hybrid camera that combines the advantages of a high resolution camera and a high speed camera. Our hybrid camera consists of a pair of low-resolution high-speed (LRHS) cameras and a single high-resolution low-speed (HRLS) camera. The LRHS cameras are able to capture fast-motion with little motion blur. They also form a stereo pair and provide a low-resolution depth map. The HRLS camera provides a high spatial resolution but also introduces severe motion blur when capturing fast moving objects. We develop efficient algorithms to simultaneously motion-deblur the HRLS image and reconstruct a high resolution depth map. Our method estimates the motion flow in the LRHS pair and then warps the flow field to the HRLS camera to estimate the point spread function (PSF). We then deblur the HRLS image and use the resulting image to enhance the low-resolution depth map using joint bilateral filters. We demonstrate the hybrid camera in depth map super-resolution and motion deblurring with spatially varying kernels. Experiments show that our framework is robust and highly effective. <s> BIB031 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> In this paper, we propose a new super-resolution face hallucination method based on Bilateral-projection-based Two-Dimensional Principal Component Analysis (B2DPCA). Firstly, the high-resolution (HR) face image and its corresponding low-resolution (LR) face image are projected to the HR and LR B2DPCA feature spaces, respectively. In these spaces, the linear mixing relationship between HR and LR feature is estimated from a training set. For reconstructing the HR image from the observed LR image, the LR image is firstly projected to LR feature space and then mapped to HR feature. Finally, the HR feature is reconstructed to the HR face image. Experiments on the well-known face databases show that the performance of our proposed method. The resolution and quality of the hallucinated face images are greatly enhanced over the LR ones, which is very helpful for human recognition. <s> BIB032 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> A typical dynamic reconstruction-based super-resolution video involves three independent processes: registration, fusion and restoration. Fast video super-resolution systems apply translational motion compensation model for registration with low computational cost. Traditional motion compensation model assumes that the whole spectrum of pixels is consistent between frames. In reality, the low frequency component of pixels often varies significantly. We propose a translational motion compensation model via frequency classification for video super-resolution systems. A novel idea to implement motion compensation by combining the up-sampled current frame and the high frequency part of the previous frame through the SAD framework is presented. Experimental results show that the new motion compensation model via frequency classification has an advantage of 2dB gain on average over that of the traditional motion compensation model. The SR quality has 0.25dB gain on average after the fusion process which is to minimize error by making use of the new motion compensated frame. <s> BIB033 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> We wish to recover an original image u from several blurry-noisy versions f k , called frames. We assume a more severe degradation model, in which the image u has been blurred by a noisy (stochastic) point spread function. We consider the problem of restoring the degraded image in a variational framework. Since the recovery of u from one single frame f is a highly ill-posed problem, we formulate two minimization problems based on the multiframe approach proposed for image super-resolution by Marquina-Osher [13]. Several experimental results for image restoration are shown, illustrating that the proposed models give visually satisfactory results. <s> BIB034 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Super-resolution reconstruction (SRR) has long been relying on very accurate motion estimation between the frames for a successful process. However, recent works propose SRR that bypasses the need for an explicit motion estimation. In this correspondence, we present a new framework that ultimately leads to the same algorithm as in our prior work. The contribution of this paper is two-fold. First, the suggested approach is much simpler and more intuitive, relying on the classic SRR formulation, and using a probabilistic and crude motion estimation. Second, the new approach offers various extensions not covered in our previous work, such as more general re-sampling tasks (e.g., de-interlacing). <s> BIB035 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Super-resolution reconstruction proposes a fusion of several low-quality images into one higher quality result with better optical resolution. Classic super-resolution techniques strongly rely on the availability of accurate motion estimation for this fusion task. When the motion is estimated inaccurately, as often happens for nonglobal motion fields, annoying artifacts appear in the super-resolved outcome. Encouraged by recent developments on the video denoising problem, where state-of-the-art algorithms are formed with no explicit motion estimation, we seek a super-resolution algorithm of similar nature that will allow processing sequences with general motion patterns. In this paper, we base our solution on the Nonlocal-Means (NLM) algorithm. We show how this denoising method is generalized to become a relatively simple super-resolution algorithm with no explicit motion estimation. Results on several test movies show that the proposed method is very successful in providing super-resolution on general sequences. <s> BIB036 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> The super-resolution reconstruction (SRR) of images is an ill posed problem. Traditionally, it is treated as a regularized minimization problem. Moreover, one of the major problems concerning SRR is its dependence on an accurate registration. In this paper, we show that a certain amount of registration error may, in fact, be beneficial for the performance of the least mean square SRR (LMS-SRR) adaptive algorithm. In these cases, the regularization term may be avoided, leading to reduction in computational cost that can be important in real-time SRR applications. <s> BIB037 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Given a set of images captured using different integer zoom settings of a camera, we propose a fast approach to obtain super-resolution (SR) for the least zoomed image at a resolution of the most zoomed image. We first obtain SR approximation to the superresolved image using a learning based approach that uses training database consisting of low-resolution (LR) and their high-resolution (HR) images. We model the LR observations as the aliased and noisy versions of their HR parts and estimate the decimation using the learned SR approximation and the available least zoomed observation. A discontinuity preserving Markov random field (MRF) is used as a prior and its parameters are estimated using the SR approximation. Finally Maximum a posteriori (MAP)-MRF formulation is used and the final cost function is optimized using particle swarm optimization (PSO) technique which is computationally efficient compared to simulated annealing. The proposed method can be used in multiresolution fusion for remotely sensed images where the available HR panchromatic image can be used to obtain HR multispectral images. Another interesting application is in immersive navigation for walk through application. Here one can change zoom setting without compromising on the spatial resolution. <s> BIB038 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Video scaling is a crucial video processing component that finds a variety of applications in video display, enhancement and analysis. Thus, video scaling has been an important research topic for several years now. With the emergence of hybrid-cameras that simultaneously capture a low-resolution (LR) video and periodic high-resolution (HR) stills, methods that use information in HR stills for enhancing the LR video are being studied. We present a multi-layer super-resolution (SR) method to rescale the input LR video with the help of a few HR still image guides. Resolution enhancement is achieved by blending the high frequency details into the input video frames that are scaled-up by a conventional, spatial scaling method. The missing high frequency details for each video frame are derived by spatio-temporal warping of the high frequency details extracted from HR still image guides. The Non-Local-Means (NLM) concept is used for computing the warping weights. The proposed SR method handles both global and local motions in a video, without requiring any explicit image registration. Experimental results comparing the proposed method with some of the existing video scaling techniques clearly points its strong potential. <s> BIB039 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> We propose a simple preconditioning method for accelerating the solution of edge-preserving image super-resolution (SR) problems in which a linear shift-invariant point spread function is employed. Our technique involves reordering the high-resolution (HR) pixels in a similar manner to what is done in preconditioning methods for quadratic SR formulations. However, due to the edge preserving requirements, the Hessian matrix of the cost function varies during the minimization process. We develop an efficient update scheme for the preconditioner in order to cope with this situation. Unlike some other acceleration strategies that round the displacement values between the low-resolution (LR) images on the HR grid, the proposed method does not sacrifice the optimality of the observation model. In addition, we describe a technique for preconditioning SR problems involving rational magnification factors. The use of such factors is motivated in part by the fact that, under certain circumstances, optimal SR zooms are nonintegers. We show that, by reordering the pixels of the LR images, the structure of the problem to solve is modified in such a way that preconditioners based on circulant operators can be used. <s> BIB040 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> This paper proposes a new approach to the image blind super-resolution (BSR) problem in the case of affine interframe motion. Although the tasks of image registration, blur identification, and high-resolution (HR) image reconstruction are coupled in the imaging process, when dealing with nonisometric interframe motion or without the exact knowledge of the blurring process, classic SR techniques generally have to tackle them (maybe in some combinations) separately. The main difficulty is that state-of-the-art deconvolution methods cannot be straightforwardly generalized to cope with the space-variant motion. We prove that the operators of affine warping and blur commute with some additional transforms and derive an equivalent form of the BSR observation model. Using this equivalent form, we develop an iterative algorithm to jointly estimate the triple-coupled variables, i.e., the motion parameters, blur kernels, and HR image. Experiments on synthetic and real-life images illustrate the performance of the proposed technique in modeling the space-variant degradation process and restoring local textures. <s> BIB041 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> The low resolution of images has been one of the major limitations in recognising humans from a distance using their biometric traits, such as face and iris. Superresolution has been employed to improve the resolution and the recognition performance simultaneously, however the majority of techniques employed operate in the pixel domain, such that the biometric feature vectors are extracted from a super-resolved input image. Feature-domain superresolution has been proposed for face and iris, and is shown to further improve recognition performance by capitalising on direct super-resolving the features which are used for recognition. However, current feature-domain superresolution approaches are limited to simple linear features such as Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), which are not the most discriminant features for biometrics. Gabor-based features have been shown to be one of the most discriminant features for biometrics including face and iris. This paper proposes a framework to conduct super-resolution in the non-linear Gabor feature domain to further improve the recognition performance of biometric systems. Experiments have confirmed the validity of the proposed approach, demonstrating superior performance to existing linear approaches for both face and iris biometrics. <s> BIB042 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> The spatial resolution of a hyperspectral image is often coarse because of the limitations of the imaging hardware. Super-resolution reconstruction (SRR) is a promising signal post-processing technique for hyperspectral image resolution enhancement. This paper proposes a maximum a posteriori (MAP) based multi-frame super-resolution algorithm for hyperspectral images. Principal component analysis (PCA) is utilized in both parts of the proposed algorithm: motion estimation and image reconstruction. A simultaneous motion estimation method with the first few principal components, which contain most of the information of a hyperspectral image, is proposed to reduce computational load and improve motion field accuracy. In the image reconstruction part, different image resolution enhancement techniques are applied to different groups of components, to reduce computational load and simultaneously remove noise. The proposed algorithm is tested on both synthetic images and real image sequences. The experimental results and comparative analyses verify the effectiveness of this algorithm. <s> BIB043 </s> Super-resolution: a comprehensive survey <s> Imaging models <s> Ultrasound image resolution enhancement is an ongoing challenge to date. Though many works have been performed using device-based approach, there exists few works dealing with post-processing methods. This paper investigates a technique based on the Alternating Direction Method of Multipliers for the resolution enhancement in ultrasound imaging, which includes the deblurring and denoising tasks. We here point out the characteristics of the proposed technique and thereby underline the features that must be dealt with for successful ultrasound image restoration. Synthetic and in vivo ultrasound images are processed in order to asses its performances. <s> BIB044
The imaging model of reconstruction-based SR algorithms describes the process by which the observed images have been obtained. In the simplest case, this process can be modeled linearly as BIB002 : where g is an observed LR image, f is the original HR scene, q is a decimation factor or sub-sampling parameter which is assumed to be equal for both x and y directions, x and y are the coordinates of the HR image, and m and n of the LR images. The LR image is assumed to be of size M 1 × M 2 , and the HR image is of size N 1 × N 2 where N 1 = q M 1 and N 2 = q M 2 . The imaging model in Eq. (1) states that an LR observed image has been obtained by averaging the HR intensities over a neighborhood of q 2 pixels BIB002 BIB005 . This model becomes more realistic when the other parameters involved in the imaging process are taken into account. As shown in Fig. 3 , these parameters, aside from decimation, are the blurring, warping and noise. The inclusion of these factors in the model of Eq. (1) results in BIB001 : where w is a warping function, h is a blurring function, d is a down-sampling operator, and η is an additive noise. The down-sampling operator defines the way by which the HR scene is sub-sampled. For example, in Eq. (1) every window of size q 2 pixels in the HR scene is replaced by only one pixel at the LR observed image by averaging the q 2 pixel values of the HR window. The warping function stands for any transformations between the LR observed image and the HR scene. For example, in Eq. (1) the warping function is uniform. But, if the LR image of g(m, n) is displaced from the HR scene of f (x, y) by a translational vector as (a, b) and a rotational angle of θ , the warping function (in homogeneous coordinates) will be as: The above function will of course change depending on the type of motion between the HR scene and the LR observations. The blurring function [which, e.g., in Eq. (1) is uniform] models any blurring effect that is imposed on the LR observed image, for example, by the optical system (lens and/or the sensor) BIB004 BIB040 or by atmospheric effects   BIB019 BIB033 BIB034 BIB035 BIB036  . The registration and blur estimation are discussed more in Sects. 3.1 and 3.2, respectively. If the number of LR images is more than one, the imaging model of Eq. (2) becomes: where k changes from 1 to the number of the available LR images, K . In matrix form, this can be written as: in which A stands for the above-mentioned degradation factors. This imaging model has been used in many SR works (Table 4 ). Figure 4 shows graphically how three different LR images are generated from a HR scene using different parameters of the imaging model of Eq. (4.) Instead of the more typical sequence of applying warping and then blurring [as in Eq. (4)], some researchers have considered reversing the order by first applying the blurring and then the warping BIB009 BIB013 BIB020 . It is discussed in BIB009 BIB013 that the former coincides more with the general imaging physics (where the camera blur is dominant), but it may result in systematic errors if motion is being estimated from the LR images BIB009  . However, some other researchers have mentioned that these two operations can commute and be assumed as block-circulate matrices, if the point spread function is space invariant, normalized, has nonnegative elements, and the motion between the LR images is translational    BIB014 BIB040 BIB041 . The imaging model of Eq. (4) has been modified by many other researchers (Table 4) , for example: -in BIB003 45, BIB028 , in addition to the blurring effect of the optical system, motion blur has been taken into account. In this case, the blur's point spread function, h, of Eq. (4) is replaced by three point spread functions such as h sensor * h lens * h motion . -  BIB021 BIB022  BIB029  BIB031 assume that a global affine photometric correction resulting from multiplication and addition across all pixels by scalars λ and γ respectively is also involved in the imaging process: The above affine photometric model can only handle small photometric changes, therefore it has been extended to a non-linear model in  , which also discusses the fact that feature extraction for finding similarities would be easier between similarly exposed images than it would with images having large differences in exposure. Therefore, it can be more efficient to carry out a photometric registration to find similarly exposed images and then do the geometric registration. - [62] [63] [64] BIB023 BIB030 BIB037  assume that a sequence of LR observations are blurred and down-sampled versions of a respective sequence of HR images, i.e., they do not consider warping effect between LR images and their corresponding HR ones, instead they involve warping between the super-resolved HR images. This provides the possibility of using temporal information between consecutive frames of a video sequence. - BIB006  BIB008  change the above imaging model to consider the quantization error which is introduced by the compression process. - BIB007  change the imaging model of Eq. (4) in such a way that the SR is applied to the feature vectors of some LR face image observations instead of their pixel values. The result in this case will not be a higher resolution face image but a higher-dimensional feature vector, which helps in increasing the recognition rate of a system which uses these feature vectors. The same modification is followed in BIB010 , wherein Gabor wavelet filters are used as the features. This method is also followed in BIB032 BIB042 , but here the extracted features are directly used to reconstruct the super-resolved image. - BIB015 BIB011 BIB024  BIB038 597 ] change the imaging model of Eq. (4) to include the effect of different zooming in the LR images. - BIB012    160 ] change the imaging model of Eq. (4) to reflect the effect of color filtering for color images which comes into play when the color images are taken by cameras with only one charge-coupled device (CCD). Here some color filters are used to make every pixel sensitive to only one color. Then, the other two color elements of the pixels are obtained by demosaicing techniques. - BIB016 BIB025 BIB043 adapt the imaging model of Eq. (4) to hyper-spectral imaging in such a way that the information of different spectra of different LR images is involved in the model. - BIB017 BIB026 reflect the effects of a non-linear camera response function, exposure time, white balancing and external illumination changes which cause vignetting effects. In this case, the imaging model of Eq. (5) is changed to: where κ is the non-linear camera response function, α k is a gain factor modeling the exposure time, β k is an offset factor modeling the white balancing, η k is the sensor noise, and k is the quantization error. - BIB018 extends the imaging model of Eq. (4) to the case where multiple video cameras capture the same scene. It is discussed here that, just as spatial misalignment can be used to improve the resolution in SR algorithms, temporal misalignment between the videos captured by different cameras can also be exploited to produce a video with higher frame rates per second than any of the individual cameras. - BIB027 uses an imaging model in which it is assumed that the LR images are obtained from the HR scene by a process which is a function of (1) sub-sampling the HR scene, (2) the HR structural information representing the surface gradients, (3) the HR reflectance field such as albedo, and (4) Gaussian noise of the process. Using this structure preserving imaging model, there is no need for sub-pixel misalignment between the LR images, and consequently, no registration algorithm is needed. - BIB035 BIB036 BIB039 remove the explicit motion parameter of the imaging model of Eq. (4). Instead, the idea of a probabilistic motion is introduced (this will be explained in more detail in Sect. 5.1.3). - BIB044 changes the imaging model of Eq. (4) for the purpose of ultrasound imaging: where g is the acquired radio frequency signal and f is the tissue scattering function.