Super-resolution: a comprehensive survey <s> Geometric registration <s> We propose a super-resolution technique specifically aimed at enhancing low-resolution text images from handheld devices. The Teager filter, a quadratic unsharp masking filter, is used to highlight high frequencies which are then combined with the warped and interpolated image sequence following motion estimation using Taylor series decomposition. Comparative performance evaluation is presented in the form of OCR results of the super-resolution <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> An iterative algorithm to increase image resolution is described. Examples are shown for low-resolution gray-level pictures, with an increase of resolution clearly observed after only a few iterations. The same method can also be used for deblurring a single blurred image. The approach is based on the resemblance of the presented problem to the reconstruction of a 2-D object from its 1-D projections in computer-aided tomography. The algorithm performed well for both computer-simulated and real images and is shown, theoretically and practically, to converge quickly. The algorithm can be executed in parallel for faster hardware implementation.<<ETX>> <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Abstract Image resolution can be improved when the relative displacements in image sequences are known accurately, and some knowledge of the imaging process is available. The proposed approach is similar to back-projection used in tomography. Examples of improved image resolution are given for gray-level and color images, when the unknown image displacements are computed from the image sequence. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> A method for detecting and tracking multiple moving objects, using both a large spatial region and a large temporal region, without assuming temporal motion constancy is described. When the large spatial region of analysis has multiple moving objects, the motion parameters and the locations of the objects are computed for one object after another. A method for segmenting the image plane into differently moving objects and computing their motions using two frames is presented. The tracking of detected objects using temporal integration and the algorithms for enhancement of tracked objects by filling-in occluded regions and by improving the spatial resolution of the imaged objects are described. > <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Abstract Accurate computation of image motion enables the enhancement of image sequences. In scenes having multiple moving objects the motion computation is performed together with object segmentation by using a unique temporal integration approach. After the motion for the different image regions is computed, these regions can be enhanced by fusing several successive frames covering the same region. Enhancements treated here include improvement of image resolution, filling-in occluded regions, and reconstruction of transparent objects. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Cameras with bellows give photographers flexibility for controlling perspective, but once the picture is taken, its perspective is set. We introduce 'virtual bellows' to provide control over perspective after a picture has been taken. Virtual bellows can be used to align images taken from different viewpoints, an important initial step in applications such as creating a high-resolution still image from video. We show how the virtual bellows, which implements the projective group, is an exact model fit to both pan and tilt. Specifically, we identify two important classes of image sequences accommodated by the virtual bellows. Examples of constructing high-quality stills are shown for the two cases: multiple frames taken of a flat object, and multiple frames taken from a fixed point. > <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Faces often appear very small in surveillance imagery because of the wide fields of view that are typically used and the relatively large distance between the cameras and the scene. For tasks such as face recognition, resolution enhancement techniques are therefore generally needed. Although numerous resolution enhancement algorithms have been proposed in the literature, most of them are limited by the fact that they make weak, if any, assumptions about the scene. We propose an algorithm to learn a prior on the spatial distribution of the image gradient for frontal images of faces. We proceed to show how such a prior can be incorporated into a resolution enhancement algorithm to yield 4- to 8-fold improvements in resolution (i.e., 16 to 64 times as many pixels). The additional pixels are, in effect, hallucinated. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We analyze the super-resolution reconstruction constraints. In particular we derive a sequence of results which all show that the constraints provide far less useful information as the magnification factor increases. It is well established that the use of a smoothness prior may help somewhat, however for large enough magnification factors any smoothness prior leads to overly smooth results. We therefore propose an algorithm that learns recognition-based priors for specific classes of scenes, the use of which gives far better super-resolution results for both faces and text. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> The objective of the work presented is the super-resolution restoration of a set of images, and we investigate the use of learnt image models within a generative Bayesian framework. It is demonstrated that restoration of far higher quality than that determined by classical maximum likelihood estimation can be achieved by either constraining the solution to lie on a restricted sub-space, or by using the sub-space to define a spatially varying prior. This sub-space can be learnt from image examples. The methods are applied to both real and synthetic images of text and faces, and results are compared to R.R. Schultz and R.L. Stevenson's (1996) MAP estimator. We consider in particular images of scenes for which the point-to-point mapping is a plane projective transformation which has 8 degrees of freedom. In the real image examples, registration is obtained from the images using automatic methods. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Superresolution reconstruction produces a high-resolution image from a set of low-resolution images. Previous iterative methods for superresolution had not adequately addressed the computational and numerical issues for this ill-conditioned and typically underdetermined large scale problem. We propose efficient block circulant preconditioners for solving the Tikhonov-regularized superresolution problem by the conjugate gradient method. We also extend to underdetermined systems the derivation of the generalized cross-validation method for automatic calculation of regularization parameters. The effectiveness of our preconditioners and regularization techniques is demonstrated with superresolution results for a simulated sequence and a forward looking infrared (FLIR) camera image sequence. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution is usually posed as a reconstruction problem. The low resolution input images are assumed to be noisy, downsampled versions of an unknown super-resolution image that is to be estimated. A common way of inverting the down-sampling process is to write down the reconstruction constraints and then solve them, often adding a smoothness prior to regularize the solution. In this paper, we present two results which both show that there is more to super-resolution than image reconstruction. We first analyze the reconstruction constraints and show that they provide less and less useful information as the magnification factor increases. Afterwards, we describe a “hallucination” algorithm, incorporating the recognition of local features in the low resolution images, which outperforms existing reconstruction-based algorithms. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> A variety of super-resolution algorithms have been described in this book. Most of them are based on the same source of information however; that the super-resolution image should generate the lower resolution input images when appropriately warped and down-sampled to model image formation. (This information is usually incorporated into super-resolution algorithms in the form of reconstruction constraints which are frequently combined with a smoothness prior to regularize their solution.) In this final chapter, we first investigate how much extra information is actually added by having more than one image for super-resolution. In particular, we derive a sequence of analytical results which show that the reconstruction constraints provide far less useful information as the decimation ratio increases. We validate these results empirically and show that for large enough decimation ratios any smoothness prior leads to overly smooth results with very little high-frequency content however many (noiseless) low resolution input images are used. In the second half of this chapter, we propose a super-resolution algorithm which uses a completely different source of information, in addition to the reconstruction constraints. The algorithm recognizes local “features” in the low resolution images and then enhances their resolution in an appropriate manner, based on a collection of high and low-resolution training samples. We call such an algorithm a hallucination algorithm. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This chapter develops very high definition image acquisition system which is based on the signal-processing approach with multiple cameras. The approach produces an improved resolution image with sufficiently high signal-to-noise ratio by processing and integrating multiple images taken simultaneously with multiple cameras. Originally, in this approach, we used multiple cameras with the same pixel aperture, but in this case there needs to be severe limitations both in the arrangement of multiple cameras and in the configuration of the scene, in order to guarantee the spatial uniformity of the resultant resolution. To overcome this difficulty completely, this chapter presents the utilization of multiple cameras with different pixel apertures, and develops a new, alternately iterative signal-processing algorithm available in the different aperture case. Experimental simulations and results are also presented. These results show that the utilization of multiple different-aperture cameras prospects well and that the alternately iterative algorithm behaves satisfactorily. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Image sensing is usually done with multiple sensors, like the RGB sensors in color imaging, the IR and EO sensors in surveillance and satellite imaging, etc. The resolution of each sensor can be increased by considering the images of the other sensors, and using the statistical redundancy among the sensors. Particularly, we use the fact that most discontinuities in the image of one sensor correspond to discontinuities in the other sensors. Two applications are presented: Increasing the resolution of a single color image by using the correlation among the three color channels, and enhancing noisy IR images. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Reconstruction-based super-resolution algorithms require very accurate alignment and good choice of filters to be effective. Often these requirements are hard to satisfy, for example, when we adopt optical flow as the motion model. In addition, the condition of having enough sub-samples may vary from pixel to pixel. We propose an alternative super-resolution method based on image fusion (called super-fusion hereafter). Image fusion has been proven to be effective in many applications. Extending image fusion to super-resolve images, we show that super-fusion is a faster alternative that imposes less requirements and is more stable than traditional super-resolution methods. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Abstract It has been known for some time that temporal dependence (motion) plays a key role in the super-resolution (SR) reconstruction of a single frame (or sequence of frames). While the impact of global time-invariant translations is relatively well known, the general motion case has not been studied in detail. In this paper, we discuss SR reconstruction for both motion models from a frequency-domain point of view. A noniterative algorithm for SR reconstruction is presented using spatio-temporal filtering. The concepts of motion-compensated windows and sinc interpolation kernels are utilized, resulting in a finite impulse response (FIR) filter realization. In the simulations, we assume a priori knowledge of the motion (optical flow), which is commonly done throughout much of the SR reconstruction literature. The proposed process is localized in nature, and this enables the selective reconstruction of desired parts of a particular frame or sequence of frames. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper introduces a practical approach for superresolution, the process of reconstructing a high-resolution image from the low-resolution input ones. The emphasis of our work is to super-resolve frames from dynamic video sequences, which may contain significant object occlusion or scene changes. As the quality of super-resolved images highly relies on the correctness of image alignment between consecutive frames, we employ the robust optical flow method to accurately estimate motion between the image pair. An efficient and reliable scheme is designed to detect and discard incorrect matchings, which may degrade the output quality. We also introduce the usage of elliptical weighted average (EWA) filter to model the spatially variant point spread function (PSF) of acquisition system in order to improve accuracy of the model. A number of complex and dynamic video sequences are tested to demonstrate the applicability and reliability of our algorithm. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In this paper we consider the problem of reconstructing a high resolution image from a set of undersampled and degraded frames, all of them obtained from high resolution images with unknown shifting displacements between them. We derive an iterative method to estimate the unknown shifts and the high resolution image given the low resolution observations. Finally, the proposed method is tested on real images. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> 1 Introduction.- 1.1 Background.- 1.2 Modelling assumptions.- 1.3 Applications.- 1.4 Principal contributions.- 2 Literature Survey.- 2.1 Image registration.- 2.1.1 Registration by a geometric transformation.- 2.1.2 Ensuring global consistency.- 2.1.3 Other parametric surfaces.- 2.2 Image mosaicing.- 2.3 Super-resolution.- 2.3.1 Simple super-resolution schemes.- 2.3.2 Methods using a generative model.- 2.3.3 Super-resolution using statistical prior image models.- 3 Registration: Geometric and Photometric.- 3.1 Introduction.- 3.2 Imaging geometry.- 3.3 Estimating homographies.- 3.3.1 Linear estimators.- 3.3.2 Non-linear refinement.- 3.3.3 The maximum likelihood estimator of H.- 3.4 A practical two-view method.- 3.5 Assessing the accuracy of registration.- 3.5.1 Assessment criteria.- 3.5.2 Obtaining a ground-truth homography.- 3.6 Feature-based vs. direct methods.- 3.7 Photometric registration.- 3.7.1 Sources of photometric difference.- 3.7.2 The photometric model.- 3.7.3 Estimating the parameters.- 3.7.4 Results.- 3.8 Application: Recovering latent marks in forensic images.- 3.8.1 Motivation.- 3.8.2 Method.- 3.8.3 Further examples.- 3.9 Summary.- 4 Image Mosaicing.- 4.1 Introduction.- 4.2 Basic method.- 4.2.1 Outline.- 4.2.2 Practical considerations.- 4.3 Rendering from the mosaic.- 4.3.1 The reprojection manifold.- 4.3.2 The blending function.- 4.3.3 Eliminating seams by photometric registration.- 4.3.4 Eliminating seams due to vignetting.- 4.3.5 A fast alternative to median filtering.- 4.4 Simultaneous registration of multiple views.- 4.4.1 Motivation.- 4.4.2 Extending the two-view framework to N-views.- 4.4.3 A novel algorithm for feature-matching over N-views.- 4.4.4 Results.- 4.5 Automating the choice of reprojection frame.- 4.5.1 Motivation.- 4.5.2 Synthetic camera rotations.- 4.6 Applications of image mosaicing.- 4.7 Mosaicing non-planar surfaces.- 4.8 Mosaicing "user's guide".- 4.9 Summary.- 4.9.1 Further examples.- 5 Super-resolution: Maximum Likelihood and Related Approaches.- 5.1 Introduction.- 5.2 What do we mean by "resolution"?.- 5.3 Single-image methods.- 5.4 The multi-view imaging model.- 5.4.1 A note on the assumptions made in the model.- 5.4.2 Discretization of the imaging model.- 5.4.3 Related approaches.- 5.4.4 Computing the elements in Mn.- 5.4.5 Boundary conditions.- 5.5 Justification for the Gaussian PSF.- 5.6 Synthetic test images.- 5.7 The average image.- 5.7.1 Noise robustness.- 5.8 Rudin's forward-projection method.- 5.9 The maximum-likelihood estimator.- 5.10 Predicting the behaviour of the ML estimator.- 5.11 Sensitivity of the ML estimator to noise sources.- 5.11.1 Observation noise.- 5.11.2 Poorly estimated PSF.- 5.11.3 Inaccurate registration parameters.- 5.12 Irani and Peleg's method.- 5.12.1 Least-squares minimization by steepest descent.- 5.12.2 Irani and Peleg's algorithm.- 5.12.3 Relationship to the ML estimator.- 5.12.4 Convergence properties.- 5.13 Gallery of results.- 5.14 Summary.- 6 Super-resolution Using Bayesian Priors.- 6.1 Introduction.- 6.2 The Bayesian framework.- 6.2.1 Markov random fields.- 6.2.2 Gibbs priors.- 6.2.3 Some common cases.- 6.3 The optimal Wiener filter as a MAP estimator.- 6.4 Generic image priors.- 6.5 Practical optimization.- 6.6 Sensitivity of the MAP estimators to noise sources.- 6.6.1 Exercising the prior models.- 6.6.2 Robustness to image noise.- 6.7 Hyper-parameter estimation by cross-validation.- 6.8 Gallery of results.- 6.9 Super-resolution "user's guide".- 6.10 Summary.- 7 Super-resolution Using Sub-space Models.- 7.1 Introduction.- 7.2 Bound constraints.- 7.3 Learning a face model using PCA.- 7.4 Super-resolution using the PCA model.- 7.4.1 An ML estimator (FS-ML).- 7.4.2 MAP estimators.- 7.5 The behaviour of the face model estimators.- 7.6 Examples using real images.- 7.7 Summary.- 8 Conclusions and Extensions.- 8.1 Summary.- 8.2 Extensions.- 8.2.1 Application to digital video.- 8.2.2 Model-based super-resolution.- 8.3 Final observations.- A Large-scale Linear and Non-linear Optimization.- References. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution reconstruction algorithm produces a high-resolution image from a low-resolution image sequence. The accuracy and the stability of the motion estimation (ME) are essential for the whole restoration. In this paper, a new super-resolution reconstruction algorithm is developed using a robust ME method, which fuses multiple estimated motion vectors within the sequence. The new algorithm has two major improvements compared with the previous research. First, instead of only two frames, the whole sequence is used to obtain a more accurate and stable estimation of the motion vector of each frame; second, the reliability of the ME is quantitatively measured and introduced into the cost function of the reconstruction algorithm. The algorithm is applied to both synthetic and real sequences, and the results are presented in the paper. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We propose a method for constructing a video sequence of high space-time resolution by combining information from multiple low-resolution video sequences of the same dynamic scene. Super-resolution is performed simultaneously in time and in space. By "temporal super-resolution," we mean recovering rapid dynamic events that occur faster than regular frame-rate. Such dynamic events are not visible (or else are observed incorrectly) in any of the input sequences, even if these are played in "slow-motion." The spatial and temporal dimensions are very different in nature, yet are interrelated. This leads to interesting visual trade-offs in time and space and to new video applications. These include: 1) treatment of spatial artifacts (e.g., motion-blur) by increasing the temporal resolution and 2) combination of input sequences of different space-time resolutions (e.g., NTSC, PAL, and even high quality still images) to generate a high quality video sequence. We further analyze and compare characteristics of temporal super-resolution to those of spatial super-resolution. These include: the video cameras needed to obtain increased resolution; the upper bound on resolution improvement via super-resolution; and, the temporal analogue to the spatial "ringing" effect. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Hyperspectral images are used for aerial and space imagery applications, including target detection, tracking, agricultural, and natural resource exploration. Unfortunately, atmospheric scattering, secondary illumination, changing viewing angles, and sensor noise degrade the quality of these images. Improving their resolution has a high payoff, but applying super-resolution techniques separately to every spectral band is problematic for two main reasons. First, the number of spectral bands can be in the hundreds, which increases the computational load excessively. Second, considering the bands separately does not make use of the information that is present across them. Furthermore, separate band super resolution does not make use of the inherent low dimensionality of the spectral data, which can effectively be used to improve the robustness against noise. In this paper, we introduce a novel super-resolution method for hyperspectral images. An integral part of our work is to model the hyperspectral image acquisition process. We propose a model that enables us to represent the hyperspectral observations from different wavelengths as weighted linear combinations of a small number of basis image planes. Then, a method for applying super resolution to hyperspectral images using this model is presented. The method fuses information from multiple observations and spectral bands to improve spatial resolution and reconstruct the spectrum of the observed scene as a combination of a small number of spectral basis functions. <s> BIB022 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> To increase the range of sizes of video scene text recognizable by optical character recognition (OCR), we developed a Bayesian super-resolution algorithm that uses a text-specific bimodal prior. We evaluated the effectiveness of the bimodal prior, compared with and in conjunction with a piecewise smoothness prior, visually and by measuring the accuracy of the OCR results on the variously super-resolved images. The bimodal prior improved the readability of 4- to 7-pixel-high scene text significantly better than bicubic interpolation, and increased the accuracy of OCR results better than the piecewise smoothness prior. <s> BIB023 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We address the problem of estimating the relative motion between the frames of a video sequence. In comparison with the commonly applied pairwise image registration methods, we consider global consistency conditions for the overall multi-frame motion estimation problem, which is more accurate. We review the recent work on this subject and propose an optimal framework, which can apply the consistency conditions as both hard constraints in the estimation problem, or as soft constraints in the form of stochastic (Bayesian) priors. The framework is applicable to virtually any motion model and enables us to develop a robust approach, which is resilient against the effects of outliers and noise. The effectiveness of the proposed approach is confirmed by a super-resolution application on synthetic and real data sets <s> BIB024 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution image reconstruction produces a high-resolution image from a set of shifted, blurred, and decimated versions thereof. Previously published papers have not addressed the computational complexity of this ill-conditioned large scale problem adequately. In this paper, the computational complexity of MAP-based multiframe super-resolution algorithms is studied, and a new fast algorithm, as well as methods for parallel image reconstruction is also presented. The proposed fast algorithm splits the multiple input low-resolution images into several subsets according to their translation relations, and then applies normal MAP algorithm to each subset, the reconstructed images are processed subsequently at a successive level until the desired resolution is achieved. Experiment results are also provided to demonstrate the efficiency of the proposed techniques. <s> BIB025 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In this paper, we propose a robust approach to super-resolution static sprite generation from multiple low-resolution images. Considering both short-term and long-term motion influences, a hybrid global motion estimation technique is first presented for sprite generation. An iterative super-resolution reconstruction algorithm is then proposed for the super-resolution sprite construction. This algorithm is robust to outliers and results in a sprite image with high visual quality. In addition, the generated super-resolution sprite can provide better reconstructed images than a conventional low-resolution sprite. Experimental results demonstrate the effectiveness of the proposed methods. <s> BIB026 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Image superresolution involves the processing of an image sequence to generate a still image with higher resolution. Classical approaches, such as bayesian MAP methods, require iterative minimization procedures, with high computational costs. Recently, the authors proposed a method to tackle this problem, based on the use of a hybrid MLP-PNN architecture. In this paper, we present a novel superresolution method, based on an evolution of this concept, to incorporate the use of local image models. A neural processing stage receives as input the value of model coefficients on local windows. The data dimensionality is firstly reduced by application of PCA. An MLP, trained on synthetic sequences with various amounts of noise, estimates the high-resolution image data. The effect of varying the dimension of the network input space is examined, showing a complex, structured behavior. Quantitative results are presented showing the accuracy and robustness of the proposed method. <s> BIB027 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Video surveillance systems are becoming an indispensable tool in today's environment, particularly for security related applications. Surveillance footage is often routinely used to identify faces of criminals "caught in the act" or for tracking individuals in a crowded environment. Most face images captured with these systems however, are small and coarse, making it extremely difficult to identify an individual through human observation or via automatic face recognition systems. ::: Super-resolution (SR) is a technique that can overcome this limitation by combining complimentary information from several frames of a video sequence to produce high resolution images of a subject. A problem that plagues many existing SR systems is that they can only deal with simple, rigid inter-frame transformations, thus performing poorly with face images as faces are non-planar, non-rigid, non-lambertian and can self-occlude. ::: This paper presents a SR system to overcome these limitations by using a robust optical flow technique. An investigation into the quality of the super-resolved images and their dependency on the number of video sequence frames used in the reconstruction is undertaken. Different fusion techniques are also investigated and experiments are conducted over two image sequences. Results show significant improvement of the image quality and resolution over the original low resolution sequences. <s> BIB028 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods neglect camera response function, exposure time, white balancing, and external illumination changes. In this paper, we show how to extend traditional super-resolution reconstruction methods to handle these factors. We provide formulations for several different super-resolution reconstruction approaches. Experimental results are also included. <s> BIB029 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Video cameras must produce images at a reasonable frame-rate and with a reasonable depth of field. These requirements impose fundamental physical limits on the spatial resolution of the image detector. As a result, current cameras produce videos with a very low resolution. The resolution of videos can be computationally enhanced by moving the camera and applying super-resolution reconstruction algorithms. However, a moving camera introduces motion blur, which limits super-resolution quality. We analyze this effect and derive a theoretical result showing that motion blur has a substantial degrading effect on the performance of super-resolution. The conclusion is that, in order to achieve the highest resolution motion blur should be avoided. Motion blur can be minimized by sampling the space-time volume of the video in a specific manner. We have developed a novel camera, called the "jitter camera," that achieves this sampling. By applying an adaptive super-resolution algorithm to the video produced by the jitter camera, we show that resolution can be notably enhanced for stationary or slowly moving objects, while it is improved slightly or left unchanged for objects with fast and complex motions. The end result is a video that has a significantly higher resolution than the captured one. <s> BIB030 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In minimally invasive surgery, a small field-of-view is often required for achieving a large magnification factor during micro-scale tasks such as coronary anastomosis. Constant change of the orientation and focal length of the laparoscope camera, however, is cumbersome and can impose extra visual and cognitive load to the operating surgeon in realigning the visual pathways and anatomical landmarks. The purpose of this paper is to investigate the use of fixational movements for robotic assisted minimal invasive surgery such that the perceived resolution of the foveal field-of-view is greater than the intrinsic resolution of the laparoscope camera. The proposed technique is based on super resolution imaging using projection onto convex sets. Validation with both phantom and in vivo data from totally endoscopic coronary artery bypass surgery is provided. <s> BIB031 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper proposes a new algorithm to address blind image super-resolution by fusing multiple blurred low-resolution (LR) images to render a high-resolution (HR) image. Conventional super-resolution (SR) image reconstruction algorithms assume either the blurring during the image formation process is negligible or the blurring function is known a priori. This assumption, however, is impractical as it is difficult to eliminate blurring completely in some applications or characterize the blurring function fully. In view of this, we present a new maximum a posteriori (MAP) estimation framework that performs joint blur identification and HR image reconstruction. An iterative scheme based on alternating minimization is developed to estimate the blur and HR image progressively. A blur prior that incorporates the soft parametric blur information and smoothness constraint is introduced in the proposed method. Experimental results show that the new method is effective in performing blind SR image reconstruction where there is limited information about the blurring function. <s> BIB032 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Unlike most super-resolution (SR) methods described in literature, which perform only SR reconstruction on the background of an image scene, we propose a framework that performs SR reconstruction simultaneously on the background and on moving objects. After registration of the background, moving objects are detected and to each moving object registration is applied. The fusion and deblurring tasks of the framework are performed by the well-known method of Hardie. To evaluate the performance of the framework the triangle orientation discrimination (TOD) method is used. The TOD method quantitatively measures the SR performance on the image background and on moving objects. From experiments it can be concluded that under proper conditions, the SR performance on moving objects is similar as the SR performance on the background. <s> BIB033 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In a cable-and-sleeve connector, variations in the real or effective length of the cable 3 will tend to vary the curvature of the sleeve 7, and this can be accommodated by variations in the effective length of the sleeve 7. For this purpose, one end of the sleeve 7 is adjustably fixed to an anchor 6 by a clutch 30. The clutch consists of four collet members 31 that, when effective as a clutch, are wedged within the tapering aperture 29 so as to be held with the serrations 32 on the internal surfaces of the collet members 31 mating with external serrations on the rigid end of the sleeve 7. When the clutch 30 is disengaged, the collet members 31 move out of the aperture 29, in the direction of the sleeve 7, and tilt loosely about their wider ends so that the wider ends are only just free of the serrations 21. Disengagement is effected, when the cable 3 is relaxed, by a member 40 being moved by the sleeve part 9 to push the collet members 31 out of the aperture 29 and lifting them out of engagement with the serrations 21. When the cable is brought under tension, the effect of the member 40 is removed and the spring 35, through the member 36, pushes the collet members 31 into their effective condition. Since the serrations of the wider ends of the collet members were only just clear of the serrations on the rigid end of the sleeve 7, the chance of mis-match on re-engagement is small so the connection operates reliably and quietly. Projections 36a co-operate with recesses 31a in the collet members 31 to limit the movement of the latter while allowing them sufficient freedom to move clear of the serrations 21. <s> BIB034 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Reconstruction-based super-resolution has been widely treated in computer vision. However, super-resolution of facial images has received very little attention. Since different parts of a face may have different motions in normal videos, this paper proposes a new method for enhancing the resolution of low-resolution facial image by handling the facial image non-uniformly. We divide low-resolution face image into different regions based on facial features and estimate motions of each of these regions using different motion models. Our experimental results show we can achieve better results than applying super-resolution on the whole face image uniformly. <s> BIB035 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods assume that exposure time is fixed for all observations, which is not necessarily true. In reality, cameras have limited dynamic range and nonlinear response to the quantity of light received, and exposure time might be adjusted automatically or manually to capture the desired portion of the scene's dynamic range. In this letter, we propose a Bayesian super-resolution algorithm based on an imaging model that includes camera response function, exposure time, sensor noise, and quantization error in addition to spatial blurring and sampling. <s> BIB036 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Recently, new sampling schemes were presented for signals with finite rate of innovation (FRI) using sampling kernels reproducing polynomials or exponentials [1] [2]. In this paper, we extend those sampling schemes to a distributed acquisition architecture in which numerous and randomly located sensors are pointing to the same area of interest. We emphasize the importance played by moments and show how to acquire efficiently FRI signals with a set of sensors. More importantly, we also show that those sampling schemes can be used for accurate registration of affine transformed and low-resolution images. Based on this, a new super-resolution algorithm was developed and showed good preliminary results. <s> BIB037 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper proposes a new algorithm to integrate image registration into image super-resolution (SR). Image SR is a process to reconstruct a high-resolution (HR) image by fusing multiple low-resolution (LR) images. A critical step in image SR is accurate registration of the LR images or, in other words, effective estimation of motion parameters. Conventional SR algorithms assume either the estimated motion parameters by existing registration methods to be error-free or the motion parameters are known a priori. This assumption, however, is impractical in many applications, as most existing registration algorithms still experience various degrees of errors, and the motion parameters among the LR images are generally unknown a priori. In view of this, this paper presents a new framework that performs simultaneous image registration and HR image reconstruction. As opposed to other current methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated simultaneously and improved progressively. Further, unlike most algorithms that focus on the translational motion model, the proposed method adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images. <s> BIB038 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We investigate new ways of improving the performance of Bayesian-based super-resolution image reconstruction by using a discontinuity adaptive image prior distribution based on robust statistics and a fast and efficient way of initializing the optimization process. The latter is an adapted normalized convolution (NC) technique that incorporates the uncertainty induced by registration errors. We present both qualitative and quantitative results on real video sequences and demonstrate the advantages of the proposed method compared to conventional methodologies. <s> BIB039 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super resolution image reconstruction allows the recovery of a high-resolution (HR) image from several low-resolution images that are noisy, blurred, and down sampled. In this paper, we present a joint formulation for a complex super-resolution problem in which the scenes contain multiple independently moving objects. This formulation is built upon the maximum a posteriori (MAP) framework, which judiciously combines motion estimation, segmentation, and super resolution together. A cyclic coordinate descent optimization procedure is used to solve the MAP formulation, in which the motion fields, segmentation fields, and HR images are found in an alternate manner given the two others, respectively. Specifically, the gradient-based methods are employed to solve the HR image and motion fields, and an iterated conditional mode optimization method to obtain the segmentation fields. The proposed algorithm has been tested using a synthetic image sequence, the "Mobile and Calendar" sequence, and the original "Motorcycle and Car" sequence. The experiment results and error analyses verify the efficacy of this algorithm <s> BIB040 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution image reconstruction is a powerful methodology for resolution enhancement from a set of blurred and noisy low-resolution images. Following a Bayesian framework, we propose a procedure for super-resolution image reconstruction based on Markov random fields (MRF), where a Potts-Strauss model is assumed for the a priori probability density function of the actual image. The first step is given by aligning all the low-resolution observations over a high-resolution grid and then improving the resolution through the iterated conditional modes (ICM) algorithm. The method was analyzed considering a number of simulated low-resolution and globally translated observations and the results demonstrate the effectiveness of the algorithm in reconstructing the desirable high-resolution image. <s> BIB041 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We propose a novel, learning-based algorithm for image super-resolution. First, an optimal distance-based weighted interpolation of the image sequence is performed using a new neural architecture, hybrid of a multi-layer perceptron and a probabilistic neural network, trained on synthetic image data. Secondly, a linear filter is applied with coefficients learned to restore residual interpolation artifacts in addition to low-resolution blurring, providing noticeable improvements over lens-detector Wiener restorations. Our method has been evaluated on real visible and IR sequences with widely different contents, providing significantly better results that a two-step method with high computational requirements. Results were similar or better than those of a maximum-a-posteriori estimator, with a reduction in processing time by a factor of almost 300. This paves the way to high-quality, quasi-real time applications of super-resolution techniques. <s> BIB042 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We propose a computationally efficient super-resolution (SR) algorithm to produce high-resolution video from low-resolution (LR) video using partition-based weighted sum (PWS) filters. First, subpixel motion parameters are estimated from the LR video frames. These are used to position the observed LR pixels into a high-resolution (HR) grid. Finally, PWS filters are employed to simultaneously perform nonuniform interpolation (to fully populate the HR grid) and perform deconvolution of the system point spread function. The PWS filters operate with a moving window. At each window location, the output is formed using a weighted sum of the present pixels within the window. The weights are selected from a filter bank based on the configuration of missing pixels in the window and the intensity structure of the present pixels. We present an algorithm for applying the PWS SR filters to video that is computationally efficient and suitable for parallel implementation. A number of experimental results are presented to demonstrate the efficacy of the proposed algorithm in comparison to several previously published methods. A detailed computational analysis of the partition-based SR filters is also presented <s> BIB043 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper examines a simultaneous estimation of both high resolution (high accuracy) 3D information and high-resolution image from multiple low-resolution stereo-pair images. The proposed method is based on the MAP (maximum a posteriori) framework and these two relevant problems (estimation of both high-resolution 3 D and intensity images) are incorporated into a unified cost function. Then the solution is obtained iteratively by a fixed point algorithm, which yeilds 3 D information while the high-resolution image is fixed, and vice versa. Each solution can be obtained by solving a linear equation. It can be expected that the proposed method improves accuracy of both 3 D information and high-resolution image simultaneously. Some experimental results are shown to validate the proposed method. <s> BIB044 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Characteristics of surveillance video generally include low resolution and poor quality due to environmental, storage and processing limitations. It is extremely difficult for computers and human operators to identify individuals from these videos. To overcome this problem, super-resolution can be used in conjunction with an automated face recognition system to enhance the spatial resolution of video frames containing the subject and narrow down the number of manual verifications performed by the human operator by presenting a list of most likely candidates from the database. As the super-resolution reconstruction process is ill-posed, visual artifacts are often generated as a result. These artifacts can be visually distracting to humans and/or affect machine recognition algorithms. While it is intuitive that higher resolution should lead to improved recognition accuracy, the effects of superresolution and such artifacts on face recognition performance have not been systematically studied. This paper aims to address this gap while illustrating that super-resolution allows more accurate identification of individuals from low-resolution surveillance footage. The proposed optical flow-based super-resolution method is benchmarked against Baker et al.'s hallucination and Schultz et al.'s super-resolution techniques on images from the Terrascope and XM2VTS databases. Ground truth and interpolated images were also tested to provide a baseline for comparison. Results show that a suitable super-resolution system can improve the discriminability of surveillance video and enhance face recognition accuracy. The experiments also show that Schultz et al.'s method fails when dealing surveillance footage due to its assumption of rigid objects in the scene. The hallucination and optical flow-based methods performed comparably, with the optical flow-based method producing less visually distracting artifacts that interfered with human recognition. <s> BIB045 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In many situations, interdependency between motion estimation and other estimation tasks is observable. This is for instance true in the area of super-resolution (SR). In order to successfully reconstruct a SR image, accurate motion vector fields are needed. On the other hand, one can only get accurate (subpixel) motion vectors, if there exist highly accurate higher-resolution reference images. Neglecting this interdependency may lead to poor estimation results for motion estimation as well as for the SR image. To address this problem, a new motion estimation scheme is presented that jointly optimises the motion vector field and a SR reference image. For this purpose and in order to attenuate aliasing and noise, which deteriorate the motion estimation, an observation model for the image acquisition process is applied and a maximum a posteriori (MAP) optimisation is performed, using Markov random field image models for regularisation. Results show that the new motion estimator provides more accurate motion vector fields than classical block motion estimation techniques. The joint optimisation scheme yields an estimate of the motion vector field as well as a SR image. <s> BIB046 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> The problem of image super-resolution from a set of low resolution multiview images has recently received much attention and can be decomposed, at least conceptually, into two consecutive steps as: registration and restoration. The ability to accurately register the input images is key to the success and the quality of image super-resolution algorithms. Using recent results from the sampling theory for signals with finite rate of innovation (FRI), we propose in this paper a new technique for subpixel extraction from low resolution images of local features like step edges and corners for image registration. By exploiting the knowledge of the sampling kernel, we are able to locate exactly the step edges on synthetic images. We also present results of full frame super-resolution of real low resolution images using our registration technique. We obtain super-resolved images with a much improved visual quality compared to using a standard local feature detection approach like a subpixel Harris corner detector. <s> BIB047 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Face recognition at a distance is a challenging and important law-enforcement surveillance problem, with low image resolution and blur contributing to the difficulties. We present a method for combining a sequence of video frames of a subject in order to create a super-resolved image of the face with increased resolution and reduced blur. An Active Appearance Model (AAM) of face shape and appearance is fit to the face in each video frame. The AAM fit provides the registration used by a robust image super-resolution algorithm that iteratively solves for a higher resolution face image from a set of video frames. This process is tested with real-world outdoor video using a PTZ camera and a commercial face recognition engine. Both improved visual perception and automatic face recognition performance are observed in these experiments. <s> BIB048 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> A computationally simple super-resolution algorithm using a type of adaptive Wiener filter is proposed. The algorithm produces an improved resolution image from a sequence of low-resolution (LR) video frames with overlapping field of view. The algorithm uses subpixel registration to position each LR pixel value on a common spatial grid that is referenced to the average position of the input frames. The positions of the LR pixels are not quantized to a finite grid as with some previous techniques. The output high-resolution (HR) pixels are obtained using a weighted sum of LR pixels in a local moving window. Using a statistical model, the weights for each HR pixel are designed to minimize the mean squared error and they depend on the relative positions of the surrounding LR pixels. Thus, these weights adapt spatially and temporally to changing distributions of LR pixels due to varying motion. Both a global and spatially varying statistical model are considered here. Since the weights adapt with distribution of LR pixels, it is quite robust and will not become unstable when an unfavorable distribution of LR pixels is observed. For translational motion, the algorithm has a low computational complexity and may be readily suitable for real-time and/or near real-time processing applications. With other motion models, the computational complexity goes up significantly. However, regardless of the motion model, the algorithm lends itself to parallel implementation. The efficacy of the proposed algorithm is demonstrated here in a number of experimental results using simulated and real video sequences. A computational analysis is also presented. <s> BIB049 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> The appearance of a distant object, when viewed through a telephoto-lens, is often deformed nonuniformly by the influence of hot-air optical turbulence. The deformation is unsteady: an image sequence can include nonuniform movement of the object even if a stationary camera is used for a static object. This study proposes a multi-frame super-resolution reconstruction from such an image sequence. The process consists of the following three stages. In the first stage, an image frame without deformation is estimated from the sequence. However, there is little detailed information about the object. In the second stage, each frame in the sequence is aligned non-rigidly to the estimated image using a non-rigid deformation model. A stable non-rigid registration technique with a B-spline function is also proposed in this study for dealing with a textureless region. In the third stage, a multi-frame super-resolution reconstruction using the non-rigid deformation recovers the detailed information in the frame obtained in the first stage. Experiments using synthetic images demonstrate the accuracy and stability of the proposed non-rigid registration technique. Furthermore, experiments using real sequences underscore the effectiveness of the proposed process. <s> BIB050 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> The MRI image reconstruction supported by Toeplitz-based iterative super-resolution algorithm is presented in the paper. It is shown that the approach improves MRI spatial resolution in cases when multi-blade sequences are used. The PROPELLER MRI method collects data from rectangular regions (blades) rotated around the origin of the k-space. Inter-blade patient motion is the premise for the use of super-resolution technique. Images obtained from sets of irregularly located frequency domain samples are combined into the high resolution MRI image. The super-resolution reconstruction replaces usually applied direct averaging of low-resolution images. Furthermore high order affine motion model has been applied. <s> BIB051 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> A novel image registration algorithm for super-resolution is proposed. The quality of reconstructed images becomes poor when the arrangement of the registered positions is irregular. However, there are no known ways to acquire the positions arranged regularly without using active cameras. By using a conventional algorithm, the positions registered with subpixel accuracy are often distributed near integer values in the low-resolution grid when the aperture problem occurs, for example, in the case of processing straight edges. Since this phenomenon causes an irregular arrangement, the high- resolution images are reconstructed inefficiently. The proposed method estimates corresponding positions on the shifted image whose image grid is shifted by a desired subpixel displacement. By using the proposed method, the distribution of the corresponding positions is shifted artificially so that the quality of the reconstructed image is improved. The effectiveness of the proposed method is demonstrated in experiments. <s> BIB052 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In this paper we propose a novel super-resolution algorithm based on motion compensation and edge-directed spatial interpolation succeeded by fusion via pixel classification. Two high-resolution images are constructed, the first by means of motion compensation and the second by means of edge-directed interpolation. The AdaBoost classifier is then used to fuse these images into an high-resolution frame. Experimental results show that the proposed method surpasses well-known resolution enhancement methods while maintaining moderate computational complexity. <s> BIB053 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In general, all the video super-resolution (SR) algorithms present the important drawback of a very high computational load, mainly due to the huge amount of operations executed by the motion estimation (ME) stage. Commonly, there is a trade-off between the accuracy of the estimated motion, given as a motion vector (MV), and the computational cost associated. In this sense, the ME algorithms that explore more exhaustively the search area among images use to deliver better MVs, at the cost of a higher computational load and resources use. Due to this reason, the proper choice of a ME algorithm is a key factor not only to reach real-time applications, but also to obtain high quality video sequences independently of their characteristics. Under the hardware point of view, the preferred ME algorithms are based on matching fixed-size blocks in different frames. In this paper, a comparison of nine of the most representative Fast Block Matching Algorithms (FBMAs) is made in order to select the one which presents the best tradeoff between video quality and computational cost, thus allowing reliable real-time hardware implementations of video super-resolution systems. <s> BIB054 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> A robust whole-image super-resolution is highly demanded. Whole-image super-resolution requires target-region extraction and motion estimation because an image sequence usually includes multiple targets and a background. A single- motion model is insufficient to represent whole-image motion. In addition, high-accuracy motion estimation is necessary for super-resolution. Simultaneous target-region extraction and high-accuracy motion estimation are challenging problems. We propose a robust and accurate algorithm that can provide extracted single-motion regions and their motion parameters. The proposed algorithm consists of three phases: feature-based motion estimation, single-motion region extraction, and region-based motion estimation. Once we obtain motion parameters of the whole image, we can reconstruct a high-resolution whole image by applying a robust super- resolution. Results of experiments demonstrate that the proposed approach can robustly reconstruct complex multiple- target scenes. <s> BIB055 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution algorithms combine multiple low resolution images into a single high resolution image. They have received a lot of attention recently in various application domains such as HDTV, satellite imaging, and video surveillance. These techniques take advantage of the aliasing present in the input images to reconstruct high frequency information of the resulting image. One of the major challenges in such algorithms is a good alignment of the input images: subpixel precision is required to enable accurate reconstruction. In this paper, we give an overview of some subspace techniques that address this problem. We first formulate super-resolution in a multichannel sampling framework with unknown offsets. Then, we present three registration methods: one approach using ideas from variable projections, one using a Fourier description of the aliased signals, and one using a spline description of the sampling kernel. The performance of the algorithms is evaluated in numerical simulations. <s> BIB056 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper explains and implements our efficient multi-frame super-resolution mosaicking algorithm. In this algorithm, feature points between images are matched using SIFT, and then random M-least squares is used to estimate the homography between frames. Next, separate frames are registered and the overlapping region is extracted. A generative model is then adopted and combined with maximum a posteriori estimation to construct the underdetermined sparse linear system. To solve the ill-posed large-scale inverse system, we derive and implement a new hybrid regularization (bilateral total variance Hubert) method. Cross validation is utilized to estimate the derivative of the blur kernel as well as the regularization parameter. Super-resolution is then applied to the individual sub-frames from the overlapping region. Finally, multi-band blending is used to stitch these resolution-enhanced frames to form the final image. The whole process is semi-real time (roughly 30 seconds for 35 frames) and the effectiveness of our algorithm is validated by applying it to real and synthetic UAV video frames. <s> BIB057 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution (SR) of facial images from video suffers from facial expression changes. Most of the existing SR algorithms for facial images make an unrealistic assumption that the "perfect" registration has been done prior to the SR process. However, the registration is a challenging task for SR with expression changes. This paper proposes a new method for enhancing the resolution of low-resolution (LR) facial image by handling the facial image in a non-rigid manner. It consists of global tracking, local alignment for precise registration and SR algorithms. A B-spline based Resolution Aware Incremental Free Form Deformation (RAIFFD) model is used to recover a dense local non-rigid flow field. In this scheme, low-resolution image model is explicitly embedded in the optimization function formulation to simulate the formation of low resolution image. The results achieved by the proposed approach are significantly better as compared to the SR approaches applied on the whole face image without considering local deformations. <s> BIB058 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> A typical dynamic reconstruction-based super-resolution video involves three independent processes: registration, fusion and restoration. Fast video super-resolution systems apply translational motion compensation model for registration with low computational cost. Traditional motion compensation model assumes that the whole spectrum of pixels is consistent between frames. In reality, the low frequency component of pixels often varies significantly. We propose a translational motion compensation model via frequency classification for video super-resolution systems. A novel idea to implement motion compensation by combining the up-sampled current frame and the high frequency part of the previous frame through the SAD framework is presented. Experimental results show that the new motion compensation model via frequency classification has an advantage of 2dB gain on average over that of the traditional motion compensation model. The SR quality has 0.25dB gain on average after the fusion process which is to minimize error by making use of the new motion compensated frame. <s> BIB059 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In this paper, we propose a novel face hallucination method which mainly focuses on recovering the lost high-frequency information of low resolution face images. Both the global information and local detailed features are taken into account in the proposed method. Firstly, the resolution of the low-resolution face image is enhanced by an edge-protecting interpolation algorithm. Then, the structural information of the interpolated face is extracted by principal component analysis. Utilizing the training database, we synthesize the local high-frequency details according to its corresponding structural information. At last, we can get a final high-resolution result by integrating the interpolated face with its corresponding local high-frequency details. Experiments demonstrate that this method can achieve face hallucination with good visual quality. <s> BIB060 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In this paper, we present new results in performance analysis of super-resolution (SR) image reconstruction. We investigate bounds on the improvement in resolution that can be achieved and its relation to the image sequence. We derive lower bounds on the resolution enhancement factor based on a frequency-domain SR algorithm. We subsequently show that the bounds remain valid for other SR algorithms. Moreover, we consider an image sequence model in the presence of affine motion. We demonstrate theoretically and experimentally that incorporation of affine motion into the image model can be used to increase the enhancement factor in comparison to purely translational motion. Finally, we discuss the extension of the performance bounds to temporal super-resolution methods and its implications on the image sequence. <s> BIB061 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> The accurate registration of multiview images is of central importance in many advanced image processing applications. Image super-resolution, for example, is a typical application where the quality of the super-resolved image is degrading as registration errors increase. Popular registration methods are often based on features extracted from the acquired images. The accuracy of the registration is in this case directly related to the number of extracted features and to the precision at which the features are located: images are best registered when many features are found with a good precision. However, in low-resolution images, only a few features can be extracted and often with a poor precision. By taking a sampling perspective, we propose in this paper new methods for extracting features in low-resolution images in order to develop efficient registration techniques. We consider, in particular, the sampling theory of signals with finite rate of innovation and show that some features of interest for registration can be retrieved perfectly in this framework, thus allowing an exact registration. We also demonstrate through simulations that the sampling model which enables the use of finite rate of innovation principles is well suited for modeling the acquisition of images by a camera. Simulations of image registration and image super-resolution of artificially sampled images are first presented, analyzed and compared to traditional techniques. We finally present favorable experimental results of super-resolution of real images acquired by a digital camera available on the market. <s> BIB062 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper proposes a cooperative example-based face hallucination method using multiple references. The proposed method first uses clustering and residual prototype faces construction to improve the performance of hallucinating a single low-resolution (LR) face to obtain a high-resolution (HR) counterpart. In the case that multiple LR face images for a person are available, a unique feature of the proposed method is to cooperatively enhance the qualities of hallucinated HR images by taking into account the multiple input face images jointly as prior models. Experimental results demonstrate that the proposed cooperative method achieve significant subjective and objective improvement over single-prior schemes. <s> BIB063 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper presents a high-resolution image reconstruction method from low-resolution image sequence. It is difficult to recognize details from a low-resolution image because of severe aliasing and poor image quality, hence recognition from the low-resolution image may result in false recognition decision. In order to improve the recognition performance, the proposed method performs a reconstruction-based super-resolution technique as a preprocessing. Then, we adopt a learning-based super-resolution technique to make high-resolution images. The proposed method also considers the illumination change between an input image and training images. To verify the accuracy and reliability of the proposed method, experiments and numerical analyses were performed with several video sequences of a moving car that simulate real surveillance systems. <s> BIB064 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We present experimental results of digital super resolution (DSR) techniques on low resolution data collected using PANOPTES, a multi-aperture miniature folded imaging architecture. The flat form factor of PANOPTES architecture results in an optical system that is heavily blurred with space variant PSF which makes super resolution challenging. We also introduce a new DSR method called SRUM (Super-Resolution with Unsharpenning Mask) which can efficiently highlight edges by embedding an unsharpenning mask to the cost function. This has much better effect than just applying the mask after all iterations as a post-processing step. <s> BIB065 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper addresses the localization and super resolution of license plate in a unifying approach. Higher quality license plate can be obtained using super resolution on successive lower resolution plate images. All existing methods assume that plate zones are correctly extracted from every frame. However, the accurate localization needs a sufficient quality of the image, which is not always true in real video. Super-resolution on all pixels is a possible but much time consuming alternative. We propose a framework which interlaces successfully these two modules. First, coarse candidates are found by an weak but fast license plate detection based on edge map sub-sampling. Then, an improved fast MAP-based super-resolution, using local phase accurate registration and edge preserving prior, applied on these regions of interest. Finally, our robust ICHT-based localizer rejects false-alarms and localizes the high resolution license plate more accurately. Experiments which were conducted on synthetic and real data, proved the robustness of our approach with real-time possibility. <s> BIB066 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We present a complete super-resolution system using a camera, that is assumed to be on a vibrating platform and continually capturing frames of a static scene, that have to be super-resolved in particular regions of interest. In a practical system the shutter of the camera is not synchronised with the vibrations it is subjected to. So, we propose a novel method for frame selection according to their degree of blurring and we combine a tracker with the sequence of selected frames to identify the subimages containing the region of interest. The extracted subimages are subsequently co-registered using a state of the art sub-pixel registration algorithm. Further selection of the co-registered subimages takes place, according to the confidence in the registration result. Finally, the subimage of interest is super-resolved using a state of the art super-resolution algorithm. The proposed frame selection method is of generic applicability and it is validated with the help of manual frame quality assessment. <s> BIB067 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> A hybrid error model with L 1 and L 2 norm minimization criteria is proposed in this paper for image/video super-resolution. A membership function is defined to adaptively control the tradeoff between the L 1 and L 2 norm terms. Therefore, the proposed hybrid model can have the advantages of both L 1 norm minimization (i.e. edge preservation) and L 2 norm minimization (i.e. smoothing noise). In addition, an effective convergence criterion is proposed, which is able to terminate the iterative L 1 and L 2 norm minimization process efficiently. Experimental results on images corrupted with various types of noises demonstrate the robustness of the proposed algorithm and its superiority to representative algorithms. <s> BIB068 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Face recognition systems are very sensitive to the quality and resolution of their input face images. This makes such systems unreliable when working with long surveillance video sequences without employing some selection and enhancement algorithms. On the other hand, processing all the frames of such video sequences by any enhancement or even face recognition algorithm is demanding. Thus, there is a need for a mechanism to summarize the input video sequence to a set of key-frames and then applying an enhancement algorithm to this subset. This paper presents a system doing exactly this. The system uses face quality assessment to select the key-frames and a hybrid super-resolution to enhance the face image quality. The suggested system that employs a linear associator face recognizer to evaluate the enhanced results has been tested on real surveillance video sequences and the experimental results show promising results. <s> BIB069 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Reconstruction-based super-resolution algorithms are widely employed for enhancing the quality of low-resolution face images. However, these algorithms are very sensitive to the registration errors of their input images. The registration errors aggravate when working with face images coming from video sequences. The longer the video the bigger is the registration error (due to the motion of the subject). Furthermore, the improvement factor of these algorithms is limited by factors smaller than two. The proposed system in this paper deals with these two problems. In order to restrict the registration errors of the system a fuzzy-based face quality assessment is employed. To cope with the second problem, a hierarchy of different types of super-resolution algorithms is used to reach an improvement factor of four. The proposed system has been tested using real video sequences of different longs and the experimental results are promising. <s> BIB070 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Regularization based super-resolution (SR) methods have been widely used to improve video resolution in recent years. These methods, however, only minimize the sum of difference between acquired low resolution (LR) images and observation model without considering video local structure. In this paper, we proposed an idea, which employs adaptive kernel regression on regularization based SR methods, to improve super-resolution performance. Arbitrary motions in input video are also considered and well modeled in our work. It is shown that the proposed idea can provide better visual quality as well as higher Peak Signal-to-Noise Ratio (PSNR) than approaches using regularized scheme or adaptive kernel regression alone. <s> BIB071 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super resolution is a technique to recover a high resolution image from a low resolution image. We develop a variational super resolution method for the subpixel accurate optical flow computation using variational optimisation. We combine variational super resolution and the variational optical flow computation for the super resolution optical flow computation. <s> BIB072 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> With the VASI, we propose a low-power video acquisition, which captures a video at low-resolution (LR) interlaced with some at full-resolution. The power consumed during acquisition is reduced by turning off some of the sensors when acquiring LR frames. Afterwards, an efficient super-resolution (SR) technique is employed to reconstruct the LR frames to high-resolution (HR). Another application of the proposed framework is to overcome the pixel transfer rate limit to obtain high-frame-rate and HR video. Experiments verify the efficacy of the proposed framework in both saving power and enhancing video quality. <s> BIB073 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Video camera is now commonly used and demand of capturing a single frame from video sequence is increasing. Since resolution of video camera is usually lower than digital camera and video data usually contains a many motion blur in the sequence, simple frame capture can produce only low quality image; image restoration technique is inevitably required. In this paper, we propose a method to restore a sharp and highresolution image from a video sequence by motion deblur for each frame followed by super-resolution technique. Since the frame-rate of the video camera is high and variance of feature appearance in successive frames and motion of feature points are usually small, we can still estimate scene geometries from video data with blur. Therefore, by using such geometric information, we first apply motion deblur for each frame, and then, super-resolve the images from the deblurred image set. For better result, we also propose an adaptive super-resolution technique considering different defocus blur effects dependent on depth. Experimental results are shown to prove the strength of our method. <s> BIB074 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper proposes a two-step prototype-face-based scheme of hallucinating the high-resolution detail of a low-resolution input face image. The proposed scheme is mainly composed of two steps: the global estimation step and the local facial-parts refinement step. In the global estimation step, the initial high-resolution face image is hallucinated via a linear combination of the global prototype faces with a coefficient vector. Instead of estimating coefficient vector in the high-dimensional raw image domain, we propose a maximum a posteriori (MAP) estimator to estimate the optimum set of coefficients in the low-dimensional coefficient domain. In the local refinement step, the facial parts (i.e., eyes, nose and mouth) are further refined using a basis selection method based on overcomplete nonnegative matrix factorization (ONMF). Experimental results demonstrate that the proposed method can achieve significant subjective and objective improvement over state-of-the-art face hallucination methods, especially when an input face does not belong to a person in the training data set. <s> BIB075 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Feeding low-resolution and low-quality images, from inexpensive surveillance cameras, to systems like, e.g., face recognition, produces erroneous and unstable results. Therefore, there is a need for a mechanism to bridge the gap between on one hand low-resolution and low-quality images and on the other hand facial analysis systems. The proposed system in this paper deals with exactly this problem. Our approach is to apply a reconstruction-based super-resolution algorithm. Such an algorithm, however, has two main problems: first, it requires relatively similar images with not too much noise and second is that its improvement factor is limited by a factor close to two. To deal with the first problem we introduce a three-step approach, which produces a face-log containing images of similar frontal faces of the highest possible quality. To deal with the second problem, limited improvement factor, we use a learning-based super-resolution algorithm applied to the result of the reconstruction-based part to improve the quality by another factor of two. This results in an improvement factor of four for the entire system. The proposed system has been tested on 122 low-resolution sequences from two different databases. The experimental results show that the proposed system can indeed produce a high-resolution and good quality frontal face image from low-resolution video sequences. <s> BIB076 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> We propose a Bayesian image super-resolution (SR) method with a causal Gaussian Markov random field (MRF) prior. SR is a technique to estimate a spatially high-resolution image from given multiple low-resolution images. An MRF model with the line process supplies a preferable prior for natural images with edges. We improve the existing image transformation model, the compound MRF model, and its hyperparameter prior model. We also derive the optimal estimator -- not the joint maximum a posteriori (MAP) or marginalized maximum likelihood (ML), but the posterior mean (PM) -- from the objective function of the L2-norm (mean square error) -based peak signal-to-noise ratio (PSNR). Point estimates such as MAP and ML are generally not stable in ill-posed high-dimensional problems because of overfitting, while PM is a stable estimator because all the parameters in the model are evaluated as distributions. The estimator is numerically determined by using variational Bayes. Variational Bayes is a widely used method that approximately determines a complicated posterior distribution, but it is generally hard to use because it needs the conjugate prior. We solve this problem with simple Taylor approximations. Experimental results have shown that the proposed method is more accurate or comparable to existing methods. <s> BIB077 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In this paper, we propose an energy-based algorithm for motion-compensated video super-resolution (VSR) targeted on upscaling of standard definition (SD) video to high-definition (HD) video. Since the motion (flow field) of the image sequence is generally unknown, we introduce a formulation for the joint estimation of a super-resolution (SR) sequence and its flow field. Via the calculus of variations, this leads to a coupled system of partial differential equations for image sequence and motion estimation. We solve a simplified form of this system and, as a by-product, we indeed provide a motion field for super-resolved sequences. To the best of our knowledge, computing super-resolved flows has not been done before. Most advanced SR methods found in literature cannot be applied to general video with arbitrary scene content and/or arbitrary optical flows, as it is possible with our simultaneous VSR method. A series of experiments shows that our method outperforms other VSR methods when dealing with general video input and that it continues to provide good results even for large scaling factors up to 8 × 8. <s> BIB078 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Spatial Super Resolution (SR) aims to recover fine image details, smaller than a pixel size. Temporal SR aims to recover rapid dynamic events that occur faster than the video frame-rate, and are therefore invisible or seen incorrectly in the video sequence. Previous methods for Space-Time SR combined information from multiple video recordings of the same dynamic scene. In this paper we show how this can be done from a single video recording. Our approach is based on the observation that small space-time patches (‘ST-patches’, e.g., 5×5×3) of a single ‘natural video’, recur many times inside the same video sequence at multiple spatio-temporal scales. We statistically explore the degree of these ST-patch recurrences inside ‘natural videos’, and show that this is a very strong statistical phenomenon. Space-time SR is obtained by combining information from multiple ST-patches at sub-frame accuracy. We show how finding similar ST-patches can be done both efficiently (with a randomized-based search in space-time), and at sub-frame accuracy (despite severe motion aliasing). Our approach is particularly useful for temporal SR, resolving both severe motion aliasing and severe motion blur in complex ‘natural videos’. <s> BIB079 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper presents a video super-resolution algorithm to interpolate an arbitrary frame in a low resolution video sequence from sparsely existing high resolution key-frames. First, a hierarchical block-based motion estimation is performed between an input and low resolution key-frames. If the motion-compensated error is small, then an input low resolution patch is temporally super-resolved via bi-directional overlapped block motion compensation. Otherwise, the input patch is spatially super-resolved using the dictionary that has been already learned from the low resolution and its corresponding high resolution key-frame pair. Finally, possible blocking artifacts between temporally super-resolved patches and spatially super-resolved patches are concealed using a specific de-blocking filter. The experimental results show that the proposed algorithm provides significantly better subjective visual quality as well as higher peak-to-peak signal-to-noise ratio than those by previous interpolation algorithms. <s> BIB080 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This paper proposes a new multi-frame super-resolution (SR) approach to reconstruct a high-resolution (HR) image from low-resolution (LR) images with zooming motion. Existing SR methods often assume that the relative motion between acquired LR images consists of only translation and possibly rotation. This restricts the application of these methods in cases when there is zooming motion among the LR images. There are currently only a few methods focusing on zooming SR. Their formulation usually ignores the registration errors or assumes them to be negligible during the HR reconstruction. In view of this, this paper presents a new SR reconstruction approach to handle a more flexible motion model including translation, rotation and zooming. An iterative framework is developed to estimate the motion parameters and HR image progressively. Both simulated and real-life experiments show that the proposed method is effective in performing image SR. <s> BIB081 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> This manuscript proposes a posterior mean (PM) super-resolution (SR) method with a compound Gaussian Markov random field (MRF) prior. SR is a technique to estimate a spatially high-resolution image from observed multiple low-resolution images. A compound Gaussian MRF model provides a preferable prior for natural images that preserves edges. PM is the optimal estimator for the objective function of peak signal-to-noise ratio (PSNR). This estimator is numerically determined by using variational Bayes (VB). We then solve the conjugate prior problem on VB and the exponential-order calculation cost problem of a compound Gaussian MRF prior with simple Taylor approximations. In experiments, the proposed method roughly overcomes existing methods. <s> BIB082 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In this letter, we present the first application of the Generalized Gaussian Markov Random Field (GGMRF) to the problem of video super-resolution. The GGMRF prior is employed to perform a maximum a posteriori (MAP) estimation of the desired high-resolution image. Compared with traditional prior models, the GGMRF can describe the distribution of the high-resolution image much better and can also preserve better the discontinuities (edges) of the original image. Previous work that used GGMRF for image restoration in which the temporal dependencies among video frames has not considered. Since the corresponding energy function is convex, gradient descent optimization techniques are used to solve the MAP estimation. Results show the super-resolved images using the GGMRF prior not only offers a good enhancement of visual quality, but also contain a significantly smaller amount of noise. <s> BIB083 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Super-resolution is a widely applied technique that improves the resolution of input images by software methods. Most conventional reconstruction-based super-resolution algorithms assume accurate dense optical flow fields between the input frames, and their performance degrades rapidly when the motion estimation result is not accurate enough. However, optical flow estimation is usually difficult, particularly when complicated motion is presented in real-world videos. In this paper, we explore a new way to solve this problem by using sparse feature point correspondences between the input images. The feature point correspondences, which are obtained by matching a set of feature points, are usually precise and much more robust than dense optical flow fields. This is because the feature points represent well-selected significant locations in the image, and performing matching on the feature point set is usually very accurate. In order to utilize the sparse correspondences in conventional super-resolution, we extract an adaptive support region with a reliable local flow field from each corresponding feature point pair. The normalized prior is also proposed to increase the visual consistency of the reconstructed result. Extensive experiments on real data were carried out, and results show that the proposed algorithm produces high-resolution images with better quality, particularly in the presence of large-scale or complicated motion fields. <s> BIB084 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> In this paper, a coarse-to-fine framework is proposed to register accurately the local regions of interest (ROIs) of images with independent perspective motions by estimating their deformation parameters. A coarse registration approach based on control points (CPs) is presented to obtain the initial perspective parameters. This approach exploits two constraints to solve the problem with a very limited number of CPs. One is named the point-point-line topology constraint, and the other is named the color and intensity distribution of segment constraint. Both of the constraints describe the consistency between the reference and sensed images. To obtain a finer registration, we have converted the perspective deformation into affine deformations in local image patches so that affine refinements can be used readily. Then, the local affine parameters that have been refined are utilized to recover precise perspective parameters of a ROI. Moreover, the location and dimension selections of local image patches are discussed by mathematical demonstrations to avoid the aperture effect. Experiments on simulated data and real-world sequences demonstrate the accuracy and the robustness of the proposed method. The experimental results of image super-resolution are also provided, which show a possible practical application of our method. <s> BIB085 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> The spatial resolution of a hyperspectral image is often coarse because of the limitations of the imaging hardware. Super-resolution reconstruction (SRR) is a promising signal post-processing technique for hyperspectral image resolution enhancement. This paper proposes a maximum a posteriori (MAP) based multi-frame super-resolution algorithm for hyperspectral images. Principal component analysis (PCA) is utilized in both parts of the proposed algorithm: motion estimation and image reconstruction. A simultaneous motion estimation method with the first few principal components, which contain most of the information of a hyperspectral image, is proposed to reduce computational load and improve motion field accuracy. In the image reconstruction part, different image resolution enhancement techniques are applied to different groups of components, to reduce computational load and simultaneously remove noise. The proposed algorithm is tested on both synthetic images and real image sequences. The experimental results and comparative analyses verify the effectiveness of this algorithm. <s> BIB086 </s> Super-resolution: a comprehensive survey <s> Geometric registration <s> Recently, the demand for face recognition to identify persons from surveillance video cameras has rapidly increased. Since surveillance cameras are usually placed at positions far from a person's face, the quality of face images captured by the cameras tends to be low. This degrades the recognition accuracy. Therefore, aiming to improve the accuracy of the low-resolution-face recognition, we propose a video-based super-resolution method. The proposed method can generate a high-resolution face image from low-resolution video frames including non-rigid deformations caused by changes of face poses and expressions without using any positional information of facial feature points. Most existing techniques use the facial feature points for image alignment between the video frames. However, it is difficult to obtain the accurate positions of the feature points from low-resolution face images. To achieve the alignment, the proposed method uses a free-form deformation method that flexibly aligns each local region between the images. This enables super-resolution of face images from low-resolution videos. Experimental results demonstrated that the proposed method improved the performance of super-resolution for actual videos in terms of both image quality and face recognition accuracy. <s> BIB087
For multiple-image SR to produce missing HR frequencies, some level of aliasing is required to be present in the LR acquired frames. In other words, multiple-image based SR is possible if at least one of the parameters involved in the imaging model employed changes from one LR image to another. These parameters include motion, blur (optical, atmospheric, and/or motion blur), zoom, multiple aperture BIB013 BIB065 , multiple images from different sensors BIB014 BIB021 , and different channels of a color image BIB014 . Therefore, in multiple-image SR prior to the actual reconstruction, a registration step is required to compensate for such changes, though, some of the methods (discussed in Sect. 6.1.2) do the reconstruction and the compensation of the changes simultaneously. The two most common types of compensation for the changes between LR images are geometric registration and blur estimation. The geometric registration is discussed in this section and the blur estimation in Sect. 3.2. Geometric registration compensates for the geometric misalignment (motion) between the LR images, with the ultimate goal of their registration to an HR framework. Such misalignments are usually the result of global and/or local motions  BIB002 BIB005 35, 36, 55,  BIB031 BIB050 . Global motion is a result of motion of the object and/or camera, while local motion is due to the non-rigid nature of the object, e.g., the human face, or due to imaging condition, e.g., the effect of hot air BIB050 . Global motion can be modeled by: -a translational model (which is common in satellite imaging) BIB018 BIB022 BIB023 BIB024 BIB025 BIB032 BIB059 , -an affine model BIB005 55, BIB001 BIB038 BIB039 BIB051 BIB060 BIB061 BIB066 BIB067 , or -a projective model BIB019 BIB009  BIB026 BIB040 BIB068 , while local motion is modeled by a non-rigid motion model BIB050 . In a typical non-rigid motion model, a set of control points on a given image is usually combined using a weighting system to represent the positional information both in the reference image and in the new image to be registered with the reference image. The first image registration algorithm used for SR was proposed in  , in which translational and rotational motions between the LR observations and the targeted HR image were assumed. Therefore, according to the imaging model given in Eq. (4), the coordinates of the LR and HR images will be related to each other according to: where (x t k , y t k ) is the translation of the kth frame, θ k is its rotation, and q x and q y are the sampling rates along the x and y direction, respectively. To find these parameters,  BIB002 BIB005 BIB001 BIB027 BIB031 BIB081 used the Taylor series expansions of the LR images. To do so, two LR images g 1 and g 2 taken from the same scene which are displaced from each other by a horizontal shift a, vertical shift b, and rotation θ are first described by: Then, sin θ and cos θ in Eq. (10) are expanded in their Taylor series expansions (up to two terms): Then, g 1 is expanded into its own Taylor series expansion (up to two terms): From this, the error of mapping one of these images on the other one can be obtained as: where the summation is over the overlapping area of the two images. The minimum of this error can be found by taking its derivatives with respect to a, b and θ and solving the equations obtained. It was shown in  that this method is valid only for small translational and rotational displacements between the images. This algorithm was later on used (or slightly changed) in many other works BIB003 BIB004 BIB006 51, BIB010 BIB020 BIB027 BIB033  BIB041 BIB042 BIB043   BIB069 BIB070  BIB076 BIB082 BIB077 . It is discussed in BIB002 that the above-mentioned method of  can be used for modeling other types of motion, such as perspective transformations, if the images can be divided into blocks such that each block undergoes some uniform motion BIB002 BIB052 . To speed up this registration algorithm, it was suggested to use a Gaussian resolution pyramid BIB002 . The idea is that even large motions in the original images will be converted to small motions in the higher levels (lower resolution images) of the pyramid. Therefore, these small motions are first found in the smaller images and then are interpolated in the lower level (higher resolution) images of the pyramid until the original image is met. This method, known as optical flow, works quite well when motion is to be computed between objects, which are non-rigid, non-planar, nonLambertian, and are subject to self-occlusion, like human faces, [55, 57, 58, BIB007 BIB011 BIB008 BIB012  BIB015 BIB016 BIB017 BIB034 BIB028 BIB035   BIB044 BIB045   BIB071 BIB072 BIB073 BIB074 BIB078  [ BIB083 BIB084 . It is discussed in BIB084 BIB053 BIB052 BIB059 BIB079 BIB080 , SSD BIB052 , sum of absolute transform differences, sum of squared transform differences BIB046 . A comparison of these search techniques can be found in BIB054 . Having applied one of these search techniques, the block with the smallest distance is considered to be the corresponding block of the current block. This process is repeated for every block until the motion vectors between every two corresponding blocks are found. This technique works fine but fails to estimate vectors properly over flat image intensity regions [55, BIB034 . To deal with this problem, it is suggested in BIB034 that motion vectors should only be calculated from textured regions and not from smooth regions. - BIB029 BIB026 BIB036  , feature points are extracted by Harris corner detection and then are matched using normalized cross correlation. After removing the outliers by RANSAC, the homographies between the LR images are found by again applying RANSAC but this time to the inliers. - BIB037 BIB047 BIB062 , the sampling theory of signals with finite rate of innovation (FRI) is used to detect step edges and corners and then use them for registration in an SR algorithm. It is shown in BIB047 that this method works better than registration algorithms based on Harris corners. - BIB044 , normalized cross correlation has been used to obtain the disparity for registration in a stereo setup for 3D SR. - BIB048 BIB063 BIB075 , active appearance model has been used for registration of facial images in a video sequence. - BIB055 , a feature-based motion estimation is performed using SIFT features (and PROSAC algorithm for matching) to obtain an initial estimate for the motion vectors between an input image and a reference image. These estimated vectors are then used to extract individual regions in the input image which have similar motions. Then, a region-based motion estimation method using local similarity and local motion error between the reference image and the input image is used to refine the initial estimate of the motion vectors. This method is shown to be able to handle multiple motions in the input images BIB055 . - BIB056 , Fourier description-based registration has been used. - BIB057 BIB064 BIB084 BIB085 , SIFT and RANSAC have been used. - [401], a mesh-based warping is used. - [450], depth information is used for finding the registration parameters. - BIB086 , principal component analysis (PCA) of hyperspectral images is used for motion estimation and registration. Each motion model has its own pros and cons. The proper motion estimation method depends on the characteristics of the image, the motion's velocity, and the type of motion (local or global). The methods mentioned above are mostly global methods, i.e., they treat all the pixels the same. This might be problematic if there are several objects in the scene having different motions (multiple motions) or if different parts of an object have different motions, like different parts of a face image BIB035 BIB058 . To deal with the former cases, in BIB005 and later on in [42, 43,  BIB017  BIB030 BIB033  it was suggested to find the motion vectors locally for each object and use the temporal information if there is any. To do so, in  BIB030 Tukey M-estimator error functions of the grayscale differences of the inlier regions are used. These are the regions which are correctly aligned. Since these regions are dominated by aliasing, the standard deviation of the aliasing can be used for estimating the standard deviation of the grayscale differences. The standard deviation of the aliasing can be estimated using the results on the statistics of the natural images  BIB030 . To give further examples of finding local motions, the following can be mentioned: -in BIB035 , a system is proposed for face images, in which the face images are first divided into sub-regions and then the motions between different regions are calculated independently. -in BIB058 BIB087 , a free from deformation (FFD) model is proposed for modeling the local deformation of facial components. -in [35] , the motion vectors of three channels of a color image are found independently and then combined to improve the accuracy of the estimation. Global motion estimation between the images of a sequence can be carried out in two ways: differential (progressive) and cumulative (anchored). In the differential method, the motion parameters are found between every two successive frames BIB049 . In the cumulative method, one of the frames of the sequence is chosen as the reference frame and the motions of the other frames are computed relative to this reference frame. If the reference frame is not chosen suitably, e.g., if it is noisy or if it is partially occluded, the motion estimation and therefore the entire SR algorithm will be erroneous. To deal with that, -Wang and Wang BIB020 use subsequences of the input sequence to compute an indirect motion vector for each frame instead of computing the motion vector between only two images (a new image and the reference image). These motion vectors are then fused to make the estimation more accurate. Furthermore, they have included a reliability measure to compensate for the inaccuracy in the estimation of the motion vectors. -Ye et al. BIB026 propose using the frame before the current frame as the new reference frame if the overlap between the reference frame and the current frame is less than a threshold (e.g., 60 %). -Nasrollahi and Moeslund BIB069 BIB070  BIB076 and then BIB067 propose using some quality measures to pick the best frame of the sequence as the reference frame.