Super-resolution: a comprehensive survey <s> Direct methods <s> We propose a super-resolution technique specifically aimed at enhancing low-resolution text images from handheld devices. The Teager filter, a quadratic unsharp masking filter, is used to highlight high frequencies which are then combined with the warped and interpolated image sequence following motion estimation using Taylor series decomposition. Comparative performance evaluation is presented in the form of OCR results of the super-resolution <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> This paper introduces a new algorithm for enhancing image resolution from an image sequence. The approach we propose herein uses the integrating resampler for warping. The method is a direct computation, which is fundamentally different from the iterative back-projection approaches proposed in previous work. This paper shows that image-warping techniques may have a strong impact on the quality of image resolution enhancement. By coupling the degradation model of the imaging system directly into the integrating resampler, we can better approximate the warping characteristics of real sensors, which also significantly improve the quality of super-resolution images. Examples of super-resolutions are given for gray-scale images. Evaluations are made visually by comparing the resulting images and those using bi-linear resampling and back-projection and quantitatively using OCR as a fundamental measure. The paper shows that even when the images are qualitatively similar, quantitative differences appear in machine processing. <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Reconstruction-based super-resolution algorithms require very accurate alignment and good choice of filters to be effective. Often these requirements are hard to satisfy, for example, when we adopt optical flow as the motion model. In addition, the condition of having enough sub-samples may vary from pixel to pixel. We propose an alternative super-resolution method based on image fusion (called super-fusion hereafter). Image fusion has been proven to be effective in many applications. Extending image fusion to super-resolve images, we show that super-fusion is a faster alternative that imposes less requirements and is more stable than traditional super-resolution methods. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Video surveillance systems are becoming an indispensable tool in today's environment, particularly for security related applications. Surveillance footage is often routinely used to identify faces of criminals "caught in the act" or for tracking individuals in a crowded environment. Most face images captured with these systems however, are small and coarse, making it extremely difficult to identify an individual through human observation or via automatic face recognition systems. ::: Super-resolution (SR) is a technique that can overcome this limitation by combining complimentary information from several frames of a video sequence to produce high resolution images of a subject. A problem that plagues many existing SR systems is that they can only deal with simple, rigid inter-frame transformations, thus performing poorly with face images as faces are non-planar, non-rigid, non-lambertian and can self-occlude. ::: This paper presents a SR system to overcome these limitations by using a robust optical flow technique. An investigation into the quality of the super-resolved images and their dependency on the number of video sequence frames used in the reconstruction is undertaken. Different fusion techniques are also investigated and experiments are conducted over two image sequences. Results show significant improvement of the image quality and resolution over the original low resolution sequences. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> In this paper, we propose a robust approach to super-resolution static sprite generation from multiple low-resolution images. Considering both short-term and long-term motion influences, a hybrid global motion estimation technique is first presented for sprite generation. An iterative super-resolution reconstruction algorithm is then proposed for the super-resolution sprite construction. This algorithm is robust to outliers and results in a sprite image with high visual quality. In addition, the generated super-resolution sprite can provide better reconstructed images than a conventional low-resolution sprite. Experimental results demonstrate the effectiveness of the proposed methods. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Characteristics of surveillance video generally include low resolution and poor quality due to environmental, storage and processing limitations. It is extremely difficult for computers and human operators to identify individuals from these videos. To overcome this problem, super-resolution can be used in conjunction with an automated face recognition system to enhance the spatial resolution of video frames containing the subject and narrow down the number of manual verifications performed by the human operator by presenting a list of most likely candidates from the database. As the super-resolution reconstruction process is ill-posed, visual artifacts are often generated as a result. These artifacts can be visually distracting to humans and/or affect machine recognition algorithms. While it is intuitive that higher resolution should lead to improved recognition accuracy, the effects of superresolution and such artifacts on face recognition performance have not been systematically studied. This paper aims to address this gap while illustrating that super-resolution allows more accurate identification of individuals from low-resolution surveillance footage. The proposed optical flow-based super-resolution method is benchmarked against Baker et al.'s hallucination and Schultz et al.'s super-resolution techniques on images from the Terrascope and XM2VTS databases. Ground truth and interpolated images were also tested to provide a baseline for comparison. Results show that a suitable super-resolution system can improve the discriminability of surveillance video and enhance face recognition accuracy. The experiments also show that Schultz et al.'s method fails when dealing surveillance footage due to its assumption of rigid objects in the scene. The hallucination and optical flow-based methods performed comparably, with the optical flow-based method producing less visually distracting artifacts that interfered with human recognition. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> A computationally simple super-resolution algorithm using a type of adaptive Wiener filter is proposed. The algorithm produces an improved resolution image from a sequence of low-resolution (LR) video frames with overlapping field of view. The algorithm uses subpixel registration to position each LR pixel value on a common spatial grid that is referenced to the average position of the input frames. The positions of the LR pixels are not quantized to a finite grid as with some previous techniques. The output high-resolution (HR) pixels are obtained using a weighted sum of LR pixels in a local moving window. Using a statistical model, the weights for each HR pixel are designed to minimize the mean squared error and they depend on the relative positions of the surrounding LR pixels. Thus, these weights adapt spatially and temporally to changing distributions of LR pixels due to varying motion. Both a global and spatially varying statistical model are considered here. Since the weights adapt with distribution of LR pixels, it is quite robust and will not become unstable when an unfavorable distribution of LR pixels is observed. For translational motion, the algorithm has a low computational complexity and may be readily suitable for real-time and/or near real-time processing applications. With other motion models, the computational complexity goes up significantly. However, regardless of the motion model, the algorithm lends itself to parallel implementation. The efficacy of the proposed algorithm is demonstrated here in a number of experimental results using simulated and real video sequences. A computational analysis is also presented. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> We propose a computationally efficient super-resolution (SR) algorithm to produce high-resolution video from low-resolution (LR) video using partition-based weighted sum (PWS) filters. First, subpixel motion parameters are estimated from the LR video frames. These are used to position the observed LR pixels into a high-resolution (HR) grid. Finally, PWS filters are employed to simultaneously perform nonuniform interpolation (to fully populate the HR grid) and perform deconvolution of the system point spread function. The PWS filters operate with a moving window. At each window location, the output is formed using a weighted sum of the present pixels within the window. The weights are selected from a filter bank based on the configuration of missing pixels in the window and the intensity structure of the present pixels. We present an algorithm for applying the PWS SR filters to video that is computationally efficient and suitable for parallel implementation. A number of experimental results are presented to demonstrate the efficacy of the proposed algorithm in comparison to several previously published methods. A detailed computational analysis of the partition-based SR filters is also presented <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> In this paper, we make contact with the field of nonparametric statistics and present a development and generalization of tools and results for use in image processing and reconstruction. In particular, we adapt and expand kernel regression ideas for use in image denoising, upscaling, interpolation, fusion, and more. Furthermore, we establish key relationships with some popular existing methods and show how several of these algorithms, including the recently popularized bilateral filter, are special cases of the proposed framework. The resulting algorithms and analyses are amply illustrated with practical examples. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> In this paper we propose a novel super-resolution algorithm based on motion compensation and edge-directed spatial interpolation succeeded by fusion via pixel classification. Two high-resolution images are constructed, the first by means of motion compensation and the second by means of edge-directed interpolation. The AdaBoost classifier is then used to fuse these images into an high-resolution frame. Experimental results show that the proposed method surpasses well-known resolution enhancement methods while maintaining moderate computational complexity. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Super-resolution reconstruction (SRR) has long been relying on very accurate motion estimation between the frames for a successful process. However, recent works propose SRR that bypasses the need for an explicit motion estimation. In this correspondence, we present a new framework that ultimately leads to the same algorithm as in our prior work. The contribution of this paper is two-fold. First, the suggested approach is much simpler and more intuitive, relying on the classic SRR formulation, and using a probabilistic and crude motion estimation. Second, the new approach offers various extensions not covered in our previous work, such as more general re-sampling tasks (e.g., de-interlacing). <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Super-resolution reconstruction proposes a fusion of several low-quality images into one higher quality result with better optical resolution. Classic super-resolution techniques strongly rely on the availability of accurate motion estimation for this fusion task. When the motion is estimated inaccurately, as often happens for nonglobal motion fields, annoying artifacts appear in the super-resolved outcome. Encouraged by recent developments on the video denoising problem, where state-of-the-art algorithms are formed with no explicit motion estimation, we seek a super-resolution algorithm of similar nature that will allow processing sequences with general motion patterns. In this paper, we base our solution on the Nonlocal-Means (NLM) algorithm. We show how this denoising method is generalized to become a relatively simple super-resolution algorithm with no explicit motion estimation. Results on several test movies show that the proposed method is very successful in providing super-resolution on general sequences. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> The need for precise (subpixel accuracy) motion estimates in conventional super-resolution has limited its applicability to only video sequences with relatively simple motions such as global translational or affine displacements. In this paper, we introduce a novel framework for adaptive enhancement and spatiotemporal upscaling of videos containing complex activities without explicit need for accurate motion estimation. Our approach is based on multidimensional kernel regression, where each pixel in the video sequence is approximated with a 3-D local (Taylor) series, capturing the essential local behavior of its spatiotemporal neighborhood. The coefficients of this series are estimated by solving a local weighted least-squares problem, where the weights are a function of the 3-D space-time orientation in the neighborhood. As this framework is fundamentally based upon the comparison of neighboring pixels in both space and time, it implicitly contains information about the local motion of the pixels across time, therefore rendering unnecessary an explicit computation of motions of modest size. The proposed approach not only significantly widens the applicability of super-resolution methods to a broad variety of video sequences containing complex motions, but also yields improved overall performance. Using several examples, we illustrate that the developed algorithm has super-resolution capabilities that provide improved optical resolution in the output, while being able to work on general input video with essentially arbitrary motion. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Methods for super-resolution can be broadly classified into two families of methods: (i) The classical multi-image super-resolution (combining images obtained at subpixel misalignments), and (ii) Example-Based super-resolution (learning correspondence between low and high resolution image patches from a database). In this paper we propose a unified framework for combining these two families of methods. We further show how this combined approach can be applied to obtain super resolution from as little as a single image (with no database or prior examples). Our approach is based on the observation that patches in a natural image tend to redundantly recur many times inside the image, both within the same scale, as well as across different scales. Recurrence of patches within the same image scale (at subpixel misalignments) gives rise to the classical super-resolution, whereas recurrence of patches across different scales of the same image gives rise to example-based super-resolution. Our approach attempts to recover at each pixel its best possible resolution increase based on its patch redundancy within and across scales. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Regularization based super-resolution (SR) methods have been widely used to improve video resolution in recent years. These methods, however, only minimize the sum of difference between acquired low resolution (LR) images and observation model without considering video local structure. In this paper, we proposed an idea, which employs adaptive kernel regression on regularization based SR methods, to improve super-resolution performance. Arbitrary motions in input video are also considered and well modeled in our work. It is shown that the proposed idea can provide better visual quality as well as higher Peak Signal-to-Noise Ratio (PSNR) than approaches using regularized scheme or adaptive kernel regression alone. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> A non-iterative method of image super-resolution based on weighted median filtering with Gaussian weights is proposed. Visual tests and basic edges metrics were used to examine the method. It was shown that the weighted median filtering reduces the errors caused by inaccurate motion vectors. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> We present an integrated method for post-processing of range data which removes outliers, smoothes the depth values and enhances the lateral resolution in order to achieve visually pleasing 3D models from low-cost depth sensors with additional (registered) color images. The algorithm is based on the non-local principle and adapts the original NL-Means formulation to the characteristics of typical depth data. Explicitly handling outliers in the sensor data, our denoising approach achieves unbiased reconstructions from error-prone input data. Taking intra-patch similarity into account, we reconstruct strong discontinuities without disturbing artifacts and preserve fine detail structures, obtaining piece-wise smooth depth maps. Furthermore, we exploit the dependencies of the depth data with additionally available color information and increase the lateral resolution of the depth maps. We finally discuss how to parallelize the algorithm in order to achieve fast processing times that are adequate for post-processing of data from fast depth sensors such as time-of-flight cameras. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Super-resolution enhancement is the technique of increasing spatial resolution of an image from the image itself or from several noisy images of the same scene but with additional information such as motion. This paper presents a kernel regression approach to reconstruct a high resolution image from several low resolution video frames. In many applications including cell phone and webcam the imaging sensors capture low resolution images due to low cost sensors or physical limitation of the hardware. So, the proposed resolution enhancement technique could be used as an inexpensive software alternative. The performance of the proposed algorithm is evaluated with several grayscale and color video streams and found successful when compared to other state of the art techniques. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> This paper proposes a framework for single-image super-resolution. The underlying idea is to learn a map from input low-resolution images to target high-resolution images based on example pairs of input and output images. Kernel ridge regression (KRR) is adopted for this purpose. To reduce the time complexity of training and testing for KRR, a sparse solution is found by combining the ideas of kernel matching pursuit and gradient descent. As a regularized solution, KRR leads to a better generalization than simply storing the examples as has been done in existing example-based algorithms and results in much less noisy images. However, this may introduce blurring and ringing artifacts around major edges as sharp changes are penalized severely. A prior model of a generic image class which takes into account the discontinuity property of images is adopted to resolve this problem. Comparison with existing algorithms shows the effectiveness of the proposed method. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> Video sequence resolution enhancement became a popular research area during the last two decades. Although traditional super-resolution techniques have been successful in dealing with image sequences, many constraints such as global translation between frames, have to be imposed to obtain good performance. In this paper, we present a new wavelet-based nonlocal-means (WNLM) framework to bypass the motion estimation stage. It can handle complex motion changes between frames. Compared with the nonlocal-means (NLM) super-resolution framework, the proposed method provides better result in terms of PSNR and faster processing. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> In this paper, we propose an iterative Wiener filter which can simultaneously perform interpolation and restoration by using non-local means to directly model the correlation between the desired high-resolution image and observed low-resolution image. A novel mechanism is proposed to control the decay speed of the correlation function while iteratively updating both estimated correlation and high-resolution image. During the iterations, the image is decomposed into patches with similar intensities at initial iterations and the patches are connected naturally with good convergence. Experimental results show that the proposed algorithm is able to produce natural image structures, and provides better PSNR and visual quality than the state-of-the-art algorithms using the sparse representation and natural image priors. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Direct methods <s> In this paper, we present a novel method for Super Resolution (SR) reconstruction with rotation invariance and search window relocation. To combine complementary information in observed images to generate a higher resolution image, we first relocate search window to involve potential similar patches and then use rotation invariance similarity measure to find accurate similar patches. Comparing with Nonlocal Means SR, our algorithm can find more similar patches for weighted average. Experimental results demonstrate superior performance of the proposed method in terms of both objective measurements and subjective evaluation. <s> BIB022
Given a set of LR observations, in the first SR algorithms of this group (Table 6 ) the following simple steps were involved: first, one of the LR images was chosen as a reference image and the others were registered against it (e.g., by optical flow [58,  BIB003 BIB004 BIB006 BIB015 ), then the reference image is scaled up by a specific scaling factor and the other LR images were warped into that using the registration information. Then, the HR image is generated by fusing all the images together and finally an optional deblurring kernel may be applied to the result. For fusing the scaled LR images, different filters can be used, such as mean and median filters    BIB004 BIB005 BIB006 BIB016 , adaptive normalized averaging  , Adaboost classifier BIB010 , and SVD-based filters  . These algorithms have been shown to be much faster than the IBP algorithms [30, 36, 37, 42, 43, BIB002 . In    BIB005 , it was shown that the median fusion of the LR images when they are registered is equivalent to the ML estimation of the residual of the imaging model of Eq. (4), and results in a robust SR algorithm if the motion between the LR images is translational and the blur is spatially locally invariant. The order of the above-mentioned steps has sometimes been changed by some authors. For example, in BIB001  , after finding the motion information from the LR images, they are mapped to an HR grid to make an initial estimate of the super-resolved image. Then, a quadratic Teager filter, which is an unsharping filter, is applied to the LR images and they are mapped to the HR grid using the previously found motion information to generate a second superresolved image. Finally, these two super-resolved images are fused using a median filter to generate the end result. It has been shown in BIB001  that this method can increase the readability of text images. As opposed to the above algorithms, in some other algorithms of this group, such as, e.g., in BIB007 , after finding the registration parameters, the LR pixels of the different LR images are not quantized to a finite HR grid, but they are weighted and then combined based on their positions in a local moving window. The weights are adaptively found in each position of the moving window. To combine the LR pixels after registration, in BIB008 , partition-based weighted sum filters are used. Using a moving window which meets all the locations of the HR image, the HR pixel in the center of the moving window is obtained based on the weighted sum of the LR pixels in the window. In each window location, the weights of the available pixels are obtained from a filter bank using the configuration of the missing pixels in the window and the intensity structure of the available pixels BIB008 . In a recently developed set of algorithms, known as nonparametric SR algorithms BIB011 (Table 6 ), which can also be classified in the group of Direct methods, the two steps of motion estimation and fusion are combined. In this group of algorithms, which is shown to work very well with video sequences, the common requirement of SR algorithms for explicit motion estimation between LR input images has been replaced by the newly introduced concept of fuzzy motion. These methods can handle occlusion, local and complex motions like those for facial components when facial expressions change. Here, the LR images are first divided into patches. Then, every patch of a given image is compared to a set of patches, including the corresponding patch and some patches in its neighborhood, in the next LR image. Then based on the similarity of these patches and their distances from the current patch, a weight is associated to every patch indicating the importance of this patch for producing an output patch. Different methods have been developed for defining these weights. For example, in - BIB011 BIB012  BIB017 BIB021 BIB022 , they are defined based on the non-local-means algorithm which has been used for video denoising: whereẐ is the reconstructed image which later on should be deblurred,w[k, l] which defined as: is the center of the current patch, N (k, l) is the neighborhood of the pixel (k, l), Ω is the support of the entire image, D p is the down-sampling operator, R H k,l is a patch extraction operator at the location of (k, l) of the HR image, t is a time operator which changes to cover all the available LR images, T , R L i, j is a patch extraction operator at the location of (i, j) of the LR image, y is the LR input image, and w[k, l, i, j, t] is the weight associated to patch (i, j) for reconstruction of patch (k, l) and is defined by: where the first term considers the radiometric similarity of the two patches and the second one, f , considers the geometric similarities and may take any form, such as a Gaussian, a box, or a constant, and the parameter σ r controls the effect of the gray level difference between the two pixels BIB011 BIB012 . - BIB009 BIB013 BIB018 BIB019 BIB015 , they are defined based on steering kernel regression which takes into account the correlation between the pixel positions and their values. -  , they are found using Zernike moments. - [618], they are found based on the similarity of some clusters in the images. These measures are defined by Gaussian functions based on the structure and intensity distance between the clusters. - BIB020 , they are found using wavelet decomposition. It worth mentioning that sometimes, as in BIB014 , the aboveexplained method of BIB011 BIB012 has been applied to only one single image. In such a case, usually a resolution pyramid of the single-input image is built and then the method of BIB011 BIB012 is applied to the images of this pyramid BIB014 .