Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Superresolution produces high quality, high resolution images from a set of degraded, low resolution frames. We present a new and efficient wavelet-based algorithm for image superresolution. The algorithm is a combination of interpolation and restoration processes. Unlike previous work, our method exploits the interlaced sampling structure in the low resolution data. Numerical experiments and analysis demonstrate the effectiveness of our approach and illustrate why the computational complexity only doubles for 2-D superresolution versus 1-D case. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Super-Resolution reconstruction produces one or a set of high-resolution images from a sequence of low-resolution frames. This article reviews a variety of Super-Resolution methods proposed in the last 20 years, and provides some insight into, and a summary of, our recent contributions to the general Super-Resolution problem. In the process, a detailed study of several very important aspects of Super-Resolution, often ignored in the literature, is presented. Spe- cifically, we discuss robustness, treatment of color, and dynamic operation modes. Novel methods for addressing these issues are accompanied by experimental results on simulated and real data. Finally, some future challenges in Super-Resolution are outlined and discussed. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 14, 47-57, 2004; Published online in Wiley InterScience (www.interscience.wiley. com). DOI 10.1002/ima.20007 <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> To increase the range of sizes of video scene text recognizable by optical character recognition (OCR), we developed a Bayesian super-resolution algorithm that uses a text-specific bimodal prior. We evaluated the effectiveness of the bimodal prior, compared with and in conjunction with a piecewise smoothness prior, visually and by measuring the accuracy of the OCR results on the variously super-resolved images. The bimodal prior improved the readability of 4- to 7-pixel-high scene text significantly better than bicubic interpolation, and increased the accuracy of OCR results better than the piecewise smoothness prior. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Face images of non-frontal views under poor illumination resolution reduce dramatically face recognition accuracy. This is evident most compellingly by the very low recognition rate of all existing face recognition systems when applied to live CCTV camera input. In this paper, we present a Bayesian framework to perform multimodal (such as variations in viewpoint and illumination) face image super-resolution for recognition in tensor space. Given a single modal low-resolution face image, we benefit from the multiple factor interactions of training sensor and super-resolve its high-resolution reconstructions across different modalities for face recognition. Instead of performing pixel-domain super-resolution and recognition independently as two separate sequential processes, we integrate the tasks of super-resolution and recognition by directly computing a maximum likelihood identity parameter vector in high-resolution tensor space for recognition. We show results from multi-modal super-resolution and face recognition experiments across different imaging modalities, using low-resolution images as testing inputs and demonstrate improved recognition rates over standard tensorface and eigenface representations <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Video surveillance systems are becoming an indispensable tool in today's environment, particularly for security related applications. Surveillance footage is often routinely used to identify faces of criminals "caught in the act" or for tracking individuals in a crowded environment. Most face images captured with these systems however, are small and coarse, making it extremely difficult to identify an individual through human observation or via automatic face recognition systems. ::: Super-resolution (SR) is a technique that can overcome this limitation by combining complimentary information from several frames of a video sequence to produce high resolution images of a subject. A problem that plagues many existing SR systems is that they can only deal with simple, rigid inter-frame transformations, thus performing poorly with face images as faces are non-planar, non-rigid, non-lambertian and can self-occlude. ::: This paper presents a SR system to overcome these limitations by using a robust optical flow technique. An investigation into the quality of the super-resolved images and their dependency on the number of video sequence frames used in the reconstruction is undertaken. Different fusion techniques are also investigated and experiments are conducted over two image sequences. Results show significant improvement of the image quality and resolution over the original low resolution sequences. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In video surveillance, the faces of interest are often of small size. Image resolution is an important factor affecting face recognition by human and computer. In this paper, we propose a new face hallucination method using eigentransformation. Different from most of the proposed methods based on probabilistic models, this method views hallucination as a transformation between different image styles. We use Principal Component Analysis (PCA) to fit the input face image as a linear combination of the low-resolution face images in the training set. The high-resolution image is rendered by replacing the low-resolution training images with high-resolution ones, while retaining the same combination coefficients. Experiments show that the hallucinated face images are not only very helpful for recognition by humans, but also make the automatic recognition procedure easier, since they emphasize the face difference by adding more high-frequency details. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In this paper we propose a robust learning-based face hallucination algorithm, which predicts a high-resolution face image from an input low-resolution image. It can be utilized for many computer vision tasks, such as face recognition and face tracking. With the help of a database of other high-resolution face images, we use a steerable pyramid to extract multi-orientation and multi-scale information of local low-level facial features both from the input low-resolution face image and other high-resolution ones, and use a pyramid-like parent structure and local best match approach to estimate the best prior; then, this prior is incorporated into a Bayesian maximum a posterior (MAP) framework, and finally the high-resolution version is optimized by a steepest decent algorithm. The experimental results show that we can enhance a 24x32 face image into a 96x128 one while the visual effect is relatively good. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Image superresolution involves the processing of an image sequence to generate a still image with higher resolution. Classical approaches, such as bayesian MAP methods, require iterative minimization procedures, with high computational costs. Recently, the authors proposed a method to tackle this problem, based on the use of a hybrid MLP-PNN architecture. In this paper, we present a novel superresolution method, based on an evolution of this concept, to incorporate the use of local image models. A neural processing stage receives as input the value of model coefficients on local windows. The data dimensionality is firstly reduced by application of PCA. An MLP, trained on synthetic sequences with various amounts of noise, estimates the high-resolution image data. The effect of varying the dimension of the network input space is examined, showing a complex, structured behavior. Quantitative results are presented showing the accuracy and robustness of the proposed method. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Hyperspectral images are used for aerial and space imagery applications, including target detection, tracking, agricultural, and natural resource exploration. Unfortunately, atmospheric scattering, secondary illumination, changing viewing angles, and sensor noise degrade the quality of these images. Improving their resolution has a high payoff, but applying super-resolution techniques separately to every spectral band is problematic for two main reasons. First, the number of spectral bands can be in the hundreds, which increases the computational load excessively. Second, considering the bands separately does not make use of the information that is present across them. Furthermore, separate band super resolution does not make use of the inherent low dimensionality of the spectral data, which can effectively be used to improve the robustness against noise. In this paper, we introduce a novel super-resolution method for hyperspectral images. An integral part of our work is to model the hyperspectral image acquisition process. We propose a model that enables us to represent the hyperspectral observations from different wavelengths as weighted linear combinations of a small number of basis image planes. Then, a method for applying super resolution to hyperspectral images using this model is presented. The method fuses information from multiple observations and spectral bands to improve spatial resolution and reconstruct the spectrum of the observed scene as a combination of a small number of spectral basis functions. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper presents comparisons of two learning-based super-resolution algorithms as well as standard interpolation methods. To allow quality assessment of results, a comparison of a variety of image quality measures is also performed. Results show that a MRF-based super-resolution algorithm improves a previously interpolated image. The estimated degree of improvement varies both according to the quality measure chosen for the comparison as well as the image class. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> A typical image formation model for super-resolution (SR) introduces blurring, aliasing, and added noise. The enhancement itself may also introduce ringing. In this paper, we use subjective tests to assess the visual quality of SR-enhanced images. We then examine how well some existing objective quality metrics can characterize the observed subjective quality. Even full-reference metrics like MSE and SSIM do not always capture visual quality of SR images with and without residual aliasing. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Unlike most super-resolution (SR) methods described in literature, which perform only SR reconstruction on the background of an image scene, we propose a framework that performs SR reconstruction simultaneously on the background and on moving objects. After registration of the background, moving objects are detected and to each moving object registration is applied. The fusion and deblurring tasks of the framework are performed by the well-known method of Hardie. To evaluate the performance of the framework the triangle orientation discrimination (TOD) method is used. The TOD method quantitatively measures the SR performance on the image background and on moving objects. From experiments it can be concluded that under proper conditions, the SR performance on moving objects is similar as the SR performance on the background. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper demonstrates a super-resolution method for improving the resolution in clinical positron emission tomography (PET) scanners. Super-resolution images were obtained by combining four data sets with spatial shifts between consecutive acquisitions and applying an iterative algorithm. Super-resolution attenuation corrected PET scans of a phantom were obtained using the two-dimensional and three-dimensional (3-D) acquisition modes of a clinical PET/computed tomography (CT) scanner (Discovery LS, GEMS). In a patient study, following a standard 18F-FDG PET/CT scan, a super-resolution scan around one small lesion was performed using axial shifts without increasing the patient radiation exposure. In the phantom study, smaller features (3 mm) could be resolved axially with the super-resolution method than without (6 mm). The super-resolution images had better resolution than the original images and provided higher contrast ratios in coronal images and in 3-D acquisition transaxial images. The coronal super-resolution images had superior resolution and contrast ratios compared to images reconstructed by merely interleaving the data to the proper axial location. In the patient study, super-resolution reconstructions displayed a more localized 18F-FDG uptake. A new approach for improving the resolution of PET images using a super-resolution method has been developed and experimentally confirmed, employing a clinical scanner. The improvement in axial resolution requires no changes in hardware. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In the last two decades, two related categories of problems have been studied independently in image restoration literature: super-resolution and demosaicing. A closer look at these problems reveals the relation between them, and, as conventional color digital cameras suffer from both low-spatial resolution and color-filtering, it is reasonable to address them in a unified context. In this paper, we propose a fast and robust hybrid method of super-resolution and demosaicing, based on a maximum a posteron estimation technique by minimizing a multiterm cost function. The L1 norm is used for measuring the difference between the projected estimate of the high-resolution image and each low-resolution image, removing outliers in the data and errors due to possibly inaccurate motion estimation. Bilateral regularization is used for spatially regularizing the luminance component, resulting in sharp edges and forcing interpolation along the edges and not across them. Simultaneously, Tikhonov regularization is used to smooth the chrominance components. Finally, an additional regularization term is used to force similar edge location and orientation in different color channels. We show that the minimization of the total cost function is relatively easy and fast. Experimental results on synthetic and real data sets confirm the effectiveness of our method. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> We address the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color superresolved images from low-quality monochromatic, color, or mosaiced frames. Our approach includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed method is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the presented algorithms, and their strength. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> A computationally simple super-resolution algorithm using a type of adaptive Wiener filter is proposed. The algorithm produces an improved resolution image from a sequence of low-resolution (LR) video frames with overlapping field of view. The algorithm uses subpixel registration to position each LR pixel value on a common spatial grid that is referenced to the average position of the input frames. The positions of the LR pixels are not quantized to a finite grid as with some previous techniques. The output high-resolution (HR) pixels are obtained using a weighted sum of LR pixels in a local moving window. Using a statistical model, the weights for each HR pixel are designed to minimize the mean squared error and they depend on the relative positions of the surrounding LR pixels. Thus, these weights adapt spatially and temporally to changing distributions of LR pixels due to varying motion. Both a global and spatially varying statistical model are considered here. Since the weights adapt with distribution of LR pixels, it is quite robust and will not become unstable when an unfavorable distribution of LR pixels is observed. For translational motion, the algorithm has a low computational complexity and may be readily suitable for real-time and/or near real-time processing applications. With other motion models, the computational complexity goes up significantly. However, regardless of the motion model, the algorithm lends itself to parallel implementation. The efficacy of the proposed algorithm is demonstrated here in a number of experimental results using simulated and real video sequences. A computational analysis is also presented. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper examines a simultaneous estimation of both high resolution (high accuracy) 3D information and high-resolution image from multiple low-resolution stereo-pair images. The proposed method is based on the MAP (maximum a posteriori) framework and these two relevant problems (estimation of both high-resolution 3 D and intensity images) are incorporated into a unified cost function. Then the solution is obtained iteratively by a fixed point algorithm, which yeilds 3 D information while the high-resolution image is fixed, and vice versa. Each solution can be obtained by solving a linear equation. It can be expected that the proposed method improves accuracy of both 3 D information and high-resolution image simultaneously. Some experimental results are shown to validate the proposed method. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> We propose a computationally efficient super-resolution (SR) algorithm to produce high-resolution video from low-resolution (LR) video using partition-based weighted sum (PWS) filters. First, subpixel motion parameters are estimated from the LR video frames. These are used to position the observed LR pixels into a high-resolution (HR) grid. Finally, PWS filters are employed to simultaneously perform nonuniform interpolation (to fully populate the HR grid) and perform deconvolution of the system point spread function. The PWS filters operate with a moving window. At each window location, the output is formed using a weighted sum of the present pixels within the window. The weights are selected from a filter bank based on the configuration of missing pixels in the window and the intensity structure of the present pixels. We present an algorithm for applying the PWS SR filters to video that is computationally efficient and suitable for parallel implementation. A number of experimental results are presented to demonstrate the efficacy of the proposed algorithm in comparison to several previously published methods. A detailed computational analysis of the partition-based SR filters is also presented <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Identifying an individual from surveillance video is a difficult, time consuming and labour intensive process. The proposed system aims to streamline this process by filtering out unwanted scenes and enhancing an individual’s face through super-resolution. An automatic face recognition system is then used to identify the subject or present the human operator with likely matches from a database. A person tracker is used to speed up the subject detection and super-resolution process by tracking moving subjects and cropping a region of interest around the subject’s face to reduce the number and size of the image frames to be super-resolved respectively. In this paper, experiments have been conducted to demonstrate how the optical flow super-resolution method used improves surveillance imagery for visual inspection as well as automatic face recognition on an Eigenface and Elastic Bunch Graph Matching system. The optical flow based method has also been benchmarked against the “hallucination” algorithm, interpolation methods and the original low-resolution images. Results show that both super-resolution algorithms improved recognition rates significantly. Although the hallucination method resulted in slightly higher recognition rates, the optical flow method produced less artifacts and more visually correct images suitable for human consumption. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Characteristics of surveillance video generally include low resolution and poor quality due to environmental, storage and processing limitations. It is extremely difficult for computers and human operators to identify individuals from these videos. To overcome this problem, super-resolution can be used in conjunction with an automated face recognition system to enhance the spatial resolution of video frames containing the subject and narrow down the number of manual verifications performed by the human operator by presenting a list of most likely candidates from the database. As the super-resolution reconstruction process is ill-posed, visual artifacts are often generated as a result. These artifacts can be visually distracting to humans and/or affect machine recognition algorithms. While it is intuitive that higher resolution should lead to improved recognition accuracy, the effects of superresolution and such artifacts on face recognition performance have not been systematically studied. This paper aims to address this gap while illustrating that super-resolution allows more accurate identification of individuals from low-resolution surveillance footage. The proposed optical flow-based super-resolution method is benchmarked against Baker et al.'s hallucination and Schultz et al.'s super-resolution techniques on images from the Terrascope and XM2VTS databases. Ground truth and interpolated images were also tested to provide a baseline for comparison. Results show that a suitable super-resolution system can improve the discriminability of surveillance video and enhance face recognition accuracy. The experiments also show that Schultz et al.'s method fails when dealing surveillance footage due to its assumption of rigid objects in the scene. The hallucination and optical flow-based methods performed comparably, with the optical flow-based method producing less visually distracting artifacts that interfered with human recognition. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Face recognition at a distance is a challenging and important law-enforcement surveillance problem, with low image resolution and blur contributing to the difficulties. We present a method for combining a sequence of video frames of a subject in order to create a super-resolved image of the face with increased resolution and reduced blur. An Active Appearance Model (AAM) of face shape and appearance is fit to the face in each video frame. The AAM fit provides the registration used by a robust image super-resolution algorithm that iteratively solves for a higher resolution face image from a set of video frames. This process is tested with real-world outdoor video using a PTZ camera and a commercial face recognition engine. Both improved visual perception and automatic face recognition performance are observed in these experiments. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In this paper, we propose a new super-resolution face hallucination and recognition method based on Independent Component Analysis (ICA). Firstly, ICA is used to build a linear mixing relationship between high-resolution (HR) face image and independent HR source faces images. The linear mixing coefficients are retained, thus the corresponding low-resolution (LR) face image is represented by linear mixture of down-sampled source faces images. So, when the source faces images are obtained by training a set of HR face images, unconstrained least square is utilized to obtain mixing coefficients to a LR image for hallucination and recognition. Experiments show that the accuracy of face recognition is insensitive to image size and the number of HR source faces images when image size is larger than 8×8, and the resolution and quality of the hallucinated face image are greatly enhanced over the LR ones, which is very helpful for human recognition. <s> BIB022 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> A two-phase face hallucination approach is proposed in this paper to infer high-resolution face image from the low-resolution observation based on a set of training image pairs. The proposed locality preserving hallucination (LPH) algorithm combines locality preserving projection (LPP) and radial basis function (RBF) regression together to hallucinate the global high-resolution face. Furthermore, in order to compensate the inferred global face with detailed inartificial facial features, the neighbor reconstruction based face residue hallucination is used. Compared with existing approaches, the proposed LPH algorithm can generate global face more similar to the ground truth face efficiently, moreover, the patch structure and search strategy carefully designed for the neighbor reconstruction algorithm greatly reduce the computational complexity without diminishing the quality of high-resolution face detail. The details of synthetic high-resolution face are further improved by a global linear smoother. Experiments indicate that our approach can synthesize distinct high-resolution faces with various facial appearances such as facial expressions, eyeglasses efficiently. <s> BIB023 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper proposes a face hallucination method for the reconstruction of high-resolution facial images from single-frame, low-resolution facial images. The proposed method has been derived from example-based hallucination methods and morphable face models. First, we propose a recursive error back-projection method to compensate for residual errors, and a region-based reconstruction method to preserve characteristics of local facial regions. Then, we define an extended morphable face model, in which an extended face is composed of the interpolated high-resolution face from a given low-resolution face, and its original high-resolution equivalent. Then, the extended face is separated into an extended shape and an extended texture. We performed various hallucination experiments using the MPI, XM2VTS, and KF databases, compared the reconstruction errors, structural similarity index, and recognition rates, and showed the effects of face detection errors and shape estimation errors. The encouraging results demonstrate that the proposed methods can improve the performance of face recognition systems. Especially the proposed method can enhance the resolution of single-frame, low-resolution facial images. <s> BIB024 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Moving objects are often the most interesting parts in image sequences. When images from a camera are undersampled and the moving object is depicted small on the image plane, processing afterwards may help to improve the visibility as well as automatic recognition of the object. This paper presents an approach which performs Super-Resolution (SR) specifically on small moving objects. The presented approach estimates simultaneously a sub-pixel accurate boundary and an intensity description of the small moving object. This approach has the advantage that it permits explicit modeling of the partial volume effect for pixels which are influenced by both the moving object and the background of the scene. This paper describes the setup of the proposed SR algorithm for small moving objects and shows its superior performance in comparison with a standard SR approach. <s> BIB025 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> We propose a new algorithm for recognition of low-resolution faces for cases when multiple images of the same face are available at matching. Specifically, this is the case of multiple-frame, or video, face recognition, and recognition with multiple cameras. Face recognition degrades when probe faces are of lower resolution than those available for training. There are two paradigms to alleviate this problem, but both have clear disadvantages. One is to use super-resolution algorithms to enhance the image, but as resolution decreases, super-resolution becomes more vulnerable to environmental variations, and it introduces distortions that affect recognition. On the other hand, it is possible to match in the low-resolution domain by downsampling the training set, but this is undesirable because features important for recognition depend on high frequency details that are erased by downsampling. We recently proposed a new framework for recognition that is different from these two paradigms, and we have shown that recognition is considerably improved for still-image face recognition. In this work, we show that the proposed method can be generalized to use a stream of frames, and produce even better recognition performance. The new algorithm incorporates the underlying assumptions of super-resolution methods with subspace distance metrics used for classification, simultaneously. We extend our previous formulation to use multiple frames, and we show that it can also be generalized to use multiple image formation processes, modeling different cameras. In this manuscript we consider two scenarios: face recognition with multiple frames, and the case of having a camera sensor network with dual cameras in a master-slave setting. Using the Multi-PIE database, our results show an increase of rank-1 recognition accuracy of about 6% for both cases compared to the single frame settings. <s> BIB026 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Face recognition degrades when faces are of very low resolution since many details about the difference between one person and another can only be captured in images of sufficient resolution. In this work, we propose a new procedure for recognition of low-resolution faces, when there is a high-resolution training set available. Most previous super-resolution approaches are aimed at reconstruction, with recognition only as an after-thought. In contrast, in the proposed method, face features, as they would be extracted for a face recognition algorithm (e.g., eigenfaces, Fisher-faces, etc.), are included in a super-resolution method as prior information. This approach simultaneously provides measures of fit of the super-resolution result, from both reconstruction and recognition perspectives. This is different from the conventional paradigms of matching in a low-resolution domain, or, alternatively, applying a super-resolution algorithm to a low-resolution face and then classifying the super-resolution result. We show, for example, that recognition of faces of as low as 6 times 6 pixel size is considerably improved compared to matching using a super-resolution reconstruction followed by classification, and to matching with a low-resolution training set. <s> BIB027 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In this paper a novel independent component analysis (ICA) based face super-resolution algorithm is proposed. In this scheme, the independent components (ICs) are obtained by offline training high-resolution face images. Given a low-resolution image, the high-resolution image is reconstructed by the linear combination of the ICs where the weight coefficients are obtained by the method of maximum a posteriori (MAP). The prior of ICA coefficients are estimated by performing PCA on training images. Meanwhile, a structure-tensor based filter is proposed to refine the intermediate SR result which is estimated by inverse ICA. Experimental results demonstrate that the proposed algorithm is robust to various pose, expressions and lighting conditions. The hallucination results preserve both the global structure and the high spatial-frequency information better such as sharp edges and high contrast. <s> BIB028 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> The facial change caused by aging progression might significantly degrade the performance of a face recognition system. One major way to deal with this problem is to predict the aging process. The work presented in this paper proposed a framework to simulate the face aging process by means of super-resolution. Considered the nature of multi- modalities in face image set, multi-linear algebra is introduced into the super-resolution method to represent and process the whole image set in tensor space. The simulating results represented in the paper are compared with the ground truth face image of the same people. <s> BIB029 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> The size of training set as well as the usage thereof is an important issue of learning-based super-resolution. In this paper, we presented an adaptive learning method for face hallucination using locality preserving projections (LPP). By virtue of the ability to reveal the non-linear structure hidden in the high-dimensional image space, LPP is an efficient manifold learning method to analyze the local intrinsic features on the manifold of local facial areas. By searching out patches online in the LPP sub-space, which makes the resultant training set tailored to the testing patch, our algorithm performed the adaptive sample selection and then effectively restored the lost high-frequency components of the low-resolution face image by patch-based eigen transformation using the dynamic training set. Finally, experiments fully demonstrated that the proposed method can achieve good performance of super-resolution reconstruction by utilizing a relative small sample. <s> BIB030 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In this paper, we propose a new super resolution method Maximum a Posteriori based on a universal hidden Markov tree model (MAP-uHMT) for remote sensing images. The hidden Markov tree theory in the wavelet domain is used to set up a prior model for reconstructing super resolution images from a sequence of warped, blurred, sub-sampled and contaminated low resolution images. Both the simulation results with a Landsat7 panchromatic image and actual results with four Landsat7 panchromatic images which were captured on different dates show that our method achieves better super resolution images both visually and quantitatively than other methods, based on PSNR in the simulation and derived PSF with actual data. <s> BIB031 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> The MRI image reconstruction supported by Toeplitz-based iterative super-resolution algorithm is presented in the paper. It is shown that the approach improves MRI spatial resolution in cases when multi-blade sequences are used. The PROPELLER MRI method collects data from rectangular regions (blades) rotated around the origin of the k-space. Inter-blade patient motion is the premise for the use of super-resolution technique. Images obtained from sets of irregularly located frequency domain samples are combined into the high resolution MRI image. The super-resolution reconstruction replaces usually applied direct averaging of low-resolution images. Furthermore high order affine motion model has been applied. <s> BIB032 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper presents a high-resolution image reconstruction method from low-resolution image sequence. It is difficult to recognize details from a low-resolution image because of severe aliasing and poor image quality, hence recognition from the low-resolution image may result in false recognition decision. In order to improve the recognition performance, the proposed method performs a reconstruction-based super-resolution technique as a preprocessing. Then, we adopt a learning-based super-resolution technique to make high-resolution images. The proposed method also considers the illumination change between an input image and training images. To verify the accuracy and reliability of the proposed method, experiments and numerical analyses were performed with several video sequences of a moving car that simulate real surveillance systems. <s> BIB033 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Human faces and license plates are the most important targets for many digital surveillance systems. The image quality is too poor to recognize the targets due to the uncontrollable effects such as poor light conditions or far distances from the concerned objects. The cameras are zoomed in to make the targets to be discernible. Image interpolation and super-resolution are two popular techniques for removing the blurry effects of the zoomed images. In this paper, the super-resolution approach is applied on the reconstructions of facial and license plate images. Training samples are collected and their pyramidal edge images are built. The intuition idea for enhancing image quality is to preserve the edge data of the high resolutional images. Face and license plate hallucination are constructed. Some experimental results are conducted to show the effectiveness of the proposed approach. Some conclusions and future works are given. <s> BIB034 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper proposes a cooperative example-based face hallucination method using multiple references. The proposed method first uses clustering and residual prototype faces construction to improve the performance of hallucinating a single low-resolution (LR) face to obtain a high-resolution (HR) counterpart. In the case that multiple LR face images for a person are available, a unique feature of the proposed method is to cooperatively enhance the qualities of hallucinated HR images by taking into account the multiple input face images jointly as prior models. Experimental results demonstrate that the proposed cooperative method achieve significant subjective and objective improvement over single-prior schemes. <s> BIB035 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In this paper, we propose a region-based eigentransformation method for human face hallucination. The algorithm first uses the snake method to locate the targeted regions with high-frequency features in an edge-smoothed face mask, and then employs the eigentransformation to reconstruct the corresponding high-resolution counterpart. The resulting high-resolution face can be obtained by combining all of the reconstructed regions. As the boundaries of the respective high-frequency regions are also smooth, so the merging process at the boundaries will not produce visual artifacts. In addition, by segmenting the facial features into different regions, the eigentransformation can achieve a better performance level in reconstructing the high-frequency information. Experiments show that our proposed algorithm outperforms the original eigentransformation for face hallucination. <s> BIB036 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In learning based single image super-resolution(SR) approach, the super-resolved image are usually found or combined from training database through patch matching. But because the representation ability of small patch is limited, it is difficult to guarantee that the super-resolved image is best under global view. To tackle this problem, we propose a statistical learning method for SR with both global and local constraints. Firstly, we use Maximum a Posteriori (MAP) estimation with learned image priors by Fields of Experts (FoE) model, and regularize SR globally guided by the image priors. Secondly, for each overlapped patch, the higher-order Markov random fields (MRFs) is used to model its local relationship with corresponding high-resolution candidates, then belief propagation is used to find high-resolution image. Compared with traditional patch based learning method without global constraint, our method could not only preserve the global image structure, but also restore the local details well. Experiments verify the idea of our global and local constraint SR method. <s> BIB037 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Low resolution faces are the main barrier to efficient face recognition and identification in several problems primarily surveillance systems. To mitigate this problem we proposes a novel learning based two-step approach by the use of Direct Locality Preserving Projections (DLPP), Maximum a posterior estimation (MAP) and Kernel Ridge Regression (KRR) for super-resolution of face images or in other words Face Hallucination. First using DLPP for manifold learning and MAP estimation, a smooth Global high resolution image is obtained. In second step to introduce high frequency components KRR is used to model the Residue high resolution image, which is then added to Global image to get final high quality detail featured Hallucinated face image. As shown in experimental results the proposed system is robust and efficient in synthesizing low resolution faces similar to the original high resolution faces. <s> BIB038 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Today's camera sensors usually have a high gray-scale resolution, e.g. 256, however, due to the dramatic lighting variations, the gray-scales distributed to the face region might be far less than 256. Therefore, besides low spatial resolution, a practical face recognition system must also handle degraded face images of low gray-scale resolution (LGR). In the last decade, low spatial resolution problem has been studied prevalently, but LGR problem was rarely studied. Aiming at robust face recognition, this paper makes a first primary attempt to investigate explicitly the LGR problem and empirically reveals that LGR indeed degrades face recognition method significantly. Possible solutions to the problem are discussed and grouped into three categories: gray-scale resolution invariant features, gray-scale degradation modeling and Gray-scale Super-Resolution (GSR). Then, we propose a Coupled Subspace Analysis (CSA) based GSR method to recover the high gray-scale resolution image from a single input LGR image. Extensive experiments on FERET and CMU-PIE face databases show that the proposed method can not only dramatically increase the gray-scale resolution and visualization quality, but also impressively improve the accuracy of face recognition. <s> BIB039 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Practical face recognition systems are sometimes confronted with low-resolution face images. Traditional two-step methods solve this problem through employing super-resolution (SR). However, these methods usually have limited performance because the target of SR is not absolutely consistent with that of face recognition. Moreover, time-consuming sophisticated SR algorithms are not suitable for real-time applications. To avoid these limitations, we propose a novel approach for LR face recognition without any SR preprocessing. Our method based on coupled mappings (CMs), projects the face images with different resolutions into a unified feature space which favors the task of classification. These CMs are learned through optimizing the objective function to minimize the difference between the correspondences (i.e., low-resolution image and its high-resolution counterpart). Inspired by locality preserving methods for dimensionality reduction, we introduce a penalty weighting matrix into our objective function. Our method significantly improves the recognition performance. Finally, we conduct experiments on publicly available databases to verify the efficacy of our algorithm. <s> BIB040 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Face recognition systems are very sensitive to the quality and resolution of their input face images. This makes such systems unreliable when working with long surveillance video sequences without employing some selection and enhancement algorithms. On the other hand, processing all the frames of such video sequences by any enhancement or even face recognition algorithm is demanding. Thus, there is a need for a mechanism to summarize the input video sequence to a set of key-frames and then applying an enhancement algorithm to this subset. This paper presents a system doing exactly this. The system uses face quality assessment to select the key-frames and a hybrid super-resolution to enhance the face image quality. The suggested system that employs a linear associator face recognizer to evaluate the enhanced results has been tested on real surveillance video sequences and the experimental results show promising results. <s> BIB041 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper proposes a new nonlinear face super resolution algorithm to address an important issue in face recognition from surveillance video namely, recognition of low resolution face image with nonlinear variations. The proposed method learns the nonlinear relationship between low resolution face image and high resolution face image in (nonlinear) kernel feature space. Moreover, the discriminative term can be easily included in the proposed framework. Experimental results on CMU-PIE and FRGC v2.0 databases show that proposed method outperforms existing methods as well as the recognition based on high resolution images. <s> BIB042 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper proposes a two-step prototype-face-based scheme of hallucinating the high-resolution detail of a low-resolution input face image. The proposed scheme is mainly composed of two steps: the global estimation step and the local facial-parts refinement step. In the global estimation step, the initial high-resolution face image is hallucinated via a linear combination of the global prototype faces with a coefficient vector. Instead of estimating coefficient vector in the high-dimensional raw image domain, we propose a maximum a posteriori (MAP) estimator to estimate the optimum set of coefficients in the low-dimensional coefficient domain. In the local refinement step, the facial parts (i.e., eyes, nose and mouth) are further refined using a basis selection method based on overcomplete nonnegative matrix factorization (ONMF). Experimental results demonstrate that the proposed method can achieve significant subjective and objective improvement over state-of-the-art face hallucination methods, especially when an input face does not belong to a person in the training data set. <s> BIB043 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In this paper, a Wavelet-based Adaptive Wiener Filter (WAWF) super-resolution (SR) algorithm is presented. A redundant discrete dyadic wavelet transform (DDWT) is applied to the input sequence to classify the LR frames into subbands of similar contextual information. The similar subbands from each LR frame are registered using subpixel motion and merged on the same HR grid to form HR subbands of similar contextual and statistical information. Then an adaptive Wiener filter approach is applied locally to each subband in order to interpolate the missing information on the HR gird. The weights of the filter are designed using a parametric circularly symmetric statistical model that adapts to the statistics and the spatial proximity of the neighboring wavelet coefficients. Simulation results show that the proposed WAWF SR algorithm results in a superior performance as compared to existing recent SR schemes. <s> BIB044 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Learning-based approaches for super-resolution (SR) have been studied in the past few years. In this paper, a novel single-image SR framework based on the learning of sparse image representation with support vector regression (SVR) is presented. SVR is known to offer excellent generalization ability in predicting output class labels for input data. Given a low resolution image, we approach the SR problem as the estimation of pixel labels in its high resolution version. The feature considered in this work is the sparse representation of different types of image patches. Prior studies have shown that this feature is robust to noise and occlusions present in image data. Experimental results show that our method is quantitatively more effective than prior work using bicubic interpolation or SVR methods, and our computation time is significantly less than that of existing SVR-based methods due to the use of sparse image representations. <s> BIB045 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> We formulate the multiframe blind deconvolution problem in an incremental expectation maximization (EM) framework. Beyond deconvolution, we show how to use the same framework to address: (i) super-resolution despite noise and unknown blurring; (ii) saturation-correction of overexposed pixels that confound image restoration. The abundance of data allows us to address both of these without using explicit image or blur priors. The end result is a simple but effective algorithm with no hyperparameters. We apply this algorithm to real-world images from astronomy and to super resolution tasks: for both, our algorithm yields increased resolution and deconvolved images simultaneously. <s> BIB046 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Image enhancement is of great importance in medical imaging where image resolution remains a crucial point in many image analysis algorithms. In this paper, we investigate brain hallucination (Rousseau, 2008), or generating a high-resolution brain image from an input low-resolution image, with the help of another high-resolution brain image. We propose an approach for image super-resolution by using anatomical intermodality priors from a reference image. Contrary to interpolation techniques, in order to be able to recover fine details in images, the reconstruction process is based on a physical model of image acquisition. Another contribution to this inverse problem is a new regularization approach that uses an example-based framework integrating non-local similarity constraints to handle in a better way repetitive structures and texture. The effectiveness of our approach is demonstrated by experiments on realistic Brainweb Magnetic Resonance images and on clinical images from ADNI, generating automatically high-quality brain images from low-resolution input. <s> BIB047 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> A novel face hallucination method is proposed in this paper for the reconstruction of a high-resolution face image from a low-resolution observation based on a set of high- and low-resolution training image pairs. Different from most of the established methods based on probabilistic or manifold learning models, the proposed method hallucinates the high-resolution image patch using the same position image patches of each training image. The optimal weights of the training image position-patches are estimated and the hallucinated patches are reconstructed using the same weights. The final high-resolution facial image is formed by integrating the hallucinated patches. The necessity of two-step framework or residue compensation and the differences between hallucination based on patch and global image are discussed. Experiments show that the proposed method without residue compensation generates higher-quality images and costs less computational time than some recent face image super-resolution (hallucination) techniques. <s> BIB048 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper addresses the reconstruction of high-resolution omnidirectional images from multiple low-resolution images with inexact registration. When omnidirectional images from low-resolution vision sensors can be uniquely mapped on the 2-sphere, such a reconstruction can be described as a transform-domain super-resolution problem in a spherical imaging framework. We describe how several spherical images with arbitrary rotations in the SO(3) rotation group contribute to the reconstruction of a high-resolution image with help of the spherical Fourier transform (SFT). As low-resolution images might not be perfectly registered in practice, the impact of inaccurate alignment on the transform coefficients is analyzed. We then cast the joint registration and super-resolution problem as a total least-squares norm minimization problem in the SFT domain. A l1-regularized total least-squares problem is considered and solved efficiently by interior point methods. Experiments with synthetic and natural images show that the proposed methods lead to effective reconstruction of high-resolution images even when large registration errors exist in the low-resolution images. The quality of the reconstructed images also increases rapidly with the number of low-resolution images, which demonstrates the benefits of the proposed solution in super-resolution schemes. Finally, we highlight the benefit of the additional regularization constraint that clearly leads to reduced noise and improved reconstruction quality. <s> BIB049 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper proposes a new algorithm to perform single-frame image super-resolution (SR) of vehicle license plate (VLP) using soft learning prior. Conventional single-frame SR/interpolation methods such as bi-cubic interpolation often experience over-smoothing near the edges and textured regions. Therefore, learning-based methods have been proposed to handle these shortcomings by incorporating a learning term so that the reconstructed high-resolution images can be guided towards these models. However, existing learning-based methods employ a binary hard-decision approach to determine whether the prior models are fully relevant or totally irrelevant. This approach, however, is inconsistent with many practical applications as the degree of relevance for the prior models may vary. In view of this, this paper proposes a new framework that adopts a soft learning approach in license plate super-resolution. The method integrates image SR with optical character recognition (OCR) to perform VLP SR. The importance of the prior models is estimated through relevance scores obtained from the OCR. These are then incorporated as a soft learning term into a new regularized cost function. Experimental results show that the proposed method is effective in handling license plate SR in both simulated and real experiments. <s> BIB050 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Feeding low-resolution and low-quality images, from inexpensive surveillance cameras, to systems like, e.g., face recognition, produces erroneous and unstable results. Therefore, there is a need for a mechanism to bridge the gap between on one hand low-resolution and low-quality images and on the other hand facial analysis systems. The proposed system in this paper deals with exactly this problem. Our approach is to apply a reconstruction-based super-resolution algorithm. Such an algorithm, however, has two main problems: first, it requires relatively similar images with not too much noise and second is that its improvement factor is limited by a factor close to two. To deal with the first problem we introduce a three-step approach, which produces a face-log containing images of similar frontal faces of the highest possible quality. To deal with the second problem, limited improvement factor, we use a learning-based super-resolution algorithm applied to the result of the reconstruction-based part to improve the quality by another factor of two. This results in an improvement factor of four for the entire system. The proposed system has been tested on 122 low-resolution sequences from two different databases. The experimental results show that the proposed system can indeed produce a high-resolution and good quality frontal face image from low-resolution video sequences. <s> BIB051 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In this paper, we present a kernel-based eigentransformation framework to hallucinate the high-resolution (HR) facial image of a low-resolution (LR) input. The eigentransformation method is a linear subspace approach, which represents an image as a linear combination of training samples. Consequently, those novel facial appearances not included in the training samples cannot be super-resolved properly. To solve this problem, we devise a kernel-based extension of the eigentransformation method, which takes higher-order statistics of the image data into account. To generate HR face images with higher fidelity, the HR face image reconstructed using this kernel-based eigentransformation method is treated as an initial estimation of the target HR face. The corresponding high-frequency components of this estimation are extracted to form a prior in the maximum a posteriori (MAP) formulation of the SR problem so as to derive the final reconstruction result. We have evaluated our proposed method using different kernels and configurations, and have compared these performances with some current SR algorithms. Experimental results show that our kernel-based framework, along with a proper kernel, can produce good HR facial images in terms of both visual quality and reconstruction errors. <s> BIB052 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> We consider the problem of super-resolution from unregistered aliased images with unknown spatial scaling factors and shifts. Due to the limitation of pixel size in the image sensor, the sampling rate for each image is lower than the Nyquist rate of the scene. Thus, we have aliasing in captured images, which makes it hard to register the low-resolution images and then generate a high-resolution image. To work out this problem, we formulate it as a multichannel sampling and reconstruction problem with unknown parameters, spatial scaling factors and shifts. We can estimate the unknown parameters and then reconstruct the high-resolution image by solving a nonlinear least square problem using the variable projection method. Experiments with synthesized 1-D signals and 2-D images show the effectiveness of the proposed algorithm. <s> BIB053 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In this paper, we present a highly data-driven approach to the task of single image super-resolution. Super-resolution is a challenging problem due to its massively under-constrained nature - for any low-resolution input there are numerous high-resolution possibilities. Our key observation is that, even with extremely low-res input images, we can use global scene descriptors and Internet-scale image databases to find similar scenes which provide ideal example textures to constrain the image upsampling problem. We quantitatively show that the statistics of scene matches are more predictive than internal image statistics for the super-resolution task. Finally, we build on recent patch-based texture transfer techniques to hallucinate texture detail and compare our super-resolution with other recent methods. <s> BIB054 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Until now, neighbor-embedding-based (NE) algorithms for super-resolution (SR) have carried out two independent processes to synthesize high-resolution (HR) image patches. In the first process, neighbor search is performed using the Euclidean distance metric, and in the second process, the optimal weights are determined by solving a constrained least squares problem. However, the separate processes are not optimal. In this paper, we propose a sparse neighbor selection scheme for SR reconstruction. We first predetermine a larger number of neighbors as potential candidates and develop an extended Robust-SL0 algorithm to simultaneously find the neighbors and to solve the reconstruction weights. Recognizing that the k-nearest neighbor (k-NN) for reconstruction should have similar local geometric structures based on clustering, we employ a local statistical feature, namely histograms of oriented gradients (HoG) of low-resolution (LR) image patches, to perform such clustering. By conveying local structural information of HoG in the synthesis stage, the k-NN of each LR input patch is adaptively chosen from their associated subset, which significantly improves the speed of synthesizing the HR image while preserving the quality of reconstruction. Experimental results suggest that the proposed method can achieve competitive SR quality compared with other state-of-the-art baselines. <s> BIB055 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> The spatial resolution of a hyperspectral image is often coarse because of the limitations of the imaging hardware. Super-resolution reconstruction (SRR) is a promising signal post-processing technique for hyperspectral image resolution enhancement. This paper proposes a maximum a posteriori (MAP) based multi-frame super-resolution algorithm for hyperspectral images. Principal component analysis (PCA) is utilized in both parts of the proposed algorithm: motion estimation and image reconstruction. A simultaneous motion estimation method with the first few principal components, which contain most of the information of a hyperspectral image, is proposed to reduce computational load and improve motion field accuracy. In the image reconstruction part, different image resolution enhancement techniques are applied to different groups of components, to reduce computational load and simultaneously remove noise. The proposed algorithm is tested on both synthetic images and real image sequences. The experimental results and comparative analyses verify the effectiveness of this algorithm. <s> BIB056 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> While researchers strive to improve automatic face recognition performance, the relationship between image resolution and face recognition performance has not received much attention. This relationship is examined systematically and a framework is developed such that results from super-resolution techniques can be compared. Three super-resolution techniques are compared with the Eigenface and Elastic Bunch Graph Matching face recognition engines. Parameter ranges over which these techniques provide better recognition performance than interpolated images is determined. <s> BIB057 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> The low resolution of images has been one of the major limitations in recognising humans from a distance using their biometric traits, such as face and iris. Superresolution has been employed to improve the resolution and the recognition performance simultaneously, however the majority of techniques employed operate in the pixel domain, such that the biometric feature vectors are extracted from a super-resolved input image. Feature-domain superresolution has been proposed for face and iris, and is shown to further improve recognition performance by capitalising on direct super-resolving the features which are used for recognition. However, current feature-domain superresolution approaches are limited to simple linear features such as Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), which are not the most discriminant features for biometrics. Gabor-based features have been shown to be one of the most discriminant features for biometrics including face and iris. This paper proposes a framework to conduct super-resolution in the non-linear Gabor feature domain to further improve the recognition performance of biometric systems. Experiments have confirmed the validity of the proposed approach, demonstrating superior performance to existing linear approaches for both face and iris biometrics. <s> BIB058 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Low resolution severely degrades the detail of face images, while bad illumination condition further increases the difficulty of face recognition. In this paper, we present a new differential images based face recognition approach to deal with low resolution and variable illumination problems. Firstly, we resort to down-sampling and interpolation to build differential image database. Then feature extraction is preformed to project the differential images into differential feature subspace. For recognizing input low-resolution (LR) face images that suffer from illumination variation, normalization are employed to compensate the illumination of LR images and hallucination method is to infer the corresponding project coefficients in the differential feature subspace. Finally, similarity measure is used for face discrimination. Different from Eigenspace-based face recognition techniques, the utilization of differential images instead of original images ensures the distinction between face images of different subjects more effectively. Experiments demonstrate that the proposed approach can improve recognition rate of LR face under variable illumination significantly. <s> BIB059 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Recently, the demand for face recognition to identify persons from surveillance video cameras has rapidly increased. Since surveillance cameras are usually placed at positions far from a person's face, the quality of face images captured by the cameras tends to be low. This degrades the recognition accuracy. Therefore, aiming to improve the accuracy of the low-resolution-face recognition, we propose a video-based super-resolution method. The proposed method can generate a high-resolution face image from low-resolution video frames including non-rigid deformations caused by changes of face poses and expressions without using any positional information of facial feature points. Most existing techniques use the facial feature points for image alignment between the video frames. However, it is difficult to obtain the accurate positions of the feature points from low-resolution face images. To achieve the alignment, the proposed method uses a free-form deformation method that flexibly aligns each local region between the images. This enables super-resolution of face images from low-resolution videos. Experimental results demonstrated that the proposed method improved the performance of super-resolution for actual videos in terms of both image quality and face recognition accuracy. <s> BIB060 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In video surveillance, the face recognition usually aims at recognizing a nonfrontal low resolution face image from the gallery in which each person has only one high resolution frontal face image. Traditional face recognition approaches have several challenges, such as the difference of image resolution, pose variation and only one gallery image per person. This letter presents a regression based method that can successfully recognize the identity given all these difficulties. The nonlinear regression models from the specific nonfrontal low resolution image to frontal high resolution features are learnt by radial basis function in subspace built by canonical correlation analysis. Extensive experiments on benchmark database show the superiority of our method. <s> BIB061 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> This paper addresses the very low resolution (VLR) problem in face recognition in which the resolution of the face image to be recognized is lower than 16 × 16. With the increasing demand of surveillance camera-based applications, the VLR problem happens in many face application systems. Existing face recognition algorithms are not able to give satisfactory performance on the VLR face image. While face super-resolution (SR) methods can be employed to enhance the resolution of the images, the existing learning-based face SR methods do not perform well on such a VLR face image. To overcome this problem, this paper proposes a novel approach to learn the relationship between the high-resolution image space and the VLR image space for face SR. Based on this new approach, two constraints, namely, new data and discriminative constraints, are designed for good visuality and face recognition applications under the VLR problem, respectively. Experimental results show that the proposed SR algorithm based on relationship learning outperforms the existing algorithms in public face databases. <s> BIB062 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> Instead of using probabilistic graph based or manifold learning based models, some approaches based on position-patch have been proposed for face hallucination recently. In order to obtain the optimal weights for face hallucination, they represent image patches through those patches at the same position of training face images by employing least square estimation or convex optimization. However, they can hope neither to provide unbiased solutions nor to satisfy locality conditions, thus the obtained patch representation is not the best. In this paper, a simpler but more effective representation scheme- Locality-constrained Representation (LcR) has been developed, compared with the Least Square Representation (LSR) and Sparse Representation (SR). It imposes a locality constraint onto the least square inversion problem to reach sparsity and locality simultaneously. Experimental results demonstrate the superiority of the proposed method over some state-of-the-art face hallucination approaches. <s> BIB063 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In satellite remote sensing, spatial resolutions of multispectral images over a particular region can be enhanced using better spatial resolution panchromatic images for the same region by a process called image fusion, or more generally data fusion. A fusion method is considered successful, if the spatial detail of thepanchromatic image is transferred into the multispectral image and the spectral content of the original multispectral image is preserved in the fused product. This research proposes a novel image fusion algorithm which takes aim at producing both spatially enhanced and spectrally appealing fused multispectral images.In the proposed method, first an intermediary image is created using original panchromatic and multispectral images. This intermediary image contains the high frequency content of the panchromatic source image such that it is the one closest to the given multispectral source image (upsampled) by a natural semi innerproduct defined. The final fused image is obtained by applying a function which performs convex linear combination of the intermediary image and the upsampled multispectral image. The function used depends on the local standard deviations of the source images. To test the performance of the method, the imagesfrom IKONOS sensor are fused using the Brovey, IHS, PCA, wavelet transform based methods, and the proposed method. Both visual and quantitative evaluation results indicate that the proposed method yields to both spectrally and spatially appealing results as the wavelet transform based method, and it gives a betterperformance when both spatial detail enhancement and spectral content preservation in the fused products are considered. It is also obvious that the method has a potential to get better results if a better fitting, more complex function is found. <s> BIB064 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> 4D-CT plays an important role in lung cancer treatment. However, due to the inherent high-dose exposure associated with CT, dense sampling along superior-inferior direction is often not practical. As a result, artifacts such as lung vessel discontinuity and partial volume are typical in 4D-CT images and might mislead dose administration in radiation therapy. In this paper, we present a novel patch-based technique for super-resolution enhancement of the 4D-CT images along the superior-inferior direction. Our working premise is that the anatomical information that is missing at one particular phase can be recovered from other phases. Based on this assumption, we employ a patch-based mechanism for guided reconstruction of super-resolution axial slices. Specifically, to reconstruct each targeted super-resolution slice for a CT image at a particular phase, we agglomerate a dictionary of patches from images of all other phases in the 4D-CT sequence. Then we perform a sparse combination of the patches in this dictionary to reconstruct details of a super-resolution patch, under constraint of similarity to the corresponding patches in the neighboring slices. By iterating this procedure over all possible patch locations, a superresolution 4D-CT image sequence with enhanced anatomical details can be eventually reconstructed. Our method was extensively evaluated using a public dataset. In all experiments, our method outperforms the conventional linear and cubic-spline interpolation methods in terms of preserving image details and suppressing misleading artifacts. <s> BIB065 </s> Super-resolution: a comprehensive survey <s> Assessment of SR algorithms <s> In multi-slice magnetic resonance imaging (MRI) the resolution in the slice direction is usually reduced to allow faster acquisition times and to reduce the amount of noise in each 2-D slice. In this paper, a novel image super resolution (SR) algorithm is presented that is used to improve the resolution of the 3D MRI volumes in the slice direction. The proposed SR algorithm uses a complex wavelet-based de-blurring approach with a Gaussian scale mixture model sparseness constraint. The algorithm takes several multi-slice volumes of the same anatomical region captured at different angles and combines these low-resolution images together to form a single 3D volume with much higher resolution in the slice direction. Our results show that the 3D volumes reconstructed using this approach have higher quality than volumes produced by the best previously proposed approaches. <s> BIB066
Both subjective and objective methods have been used for assessing the results of SR algorithms. In the subjective method, usually human observers assess the quality of the produced image. Here the results of the SR algorithms are presented in the form of mean opinion scores and variance opinion score  . With objective methods, the results of SR algorithms are usually compared against the ground truth using measures like MSE and (peak) signal to noise ratio (PSNR). MSE is defined as: The smaller the MSE, the closer the result is to the ground truth. PSNR is defined as: where RMSE is the square root of the MSE of Eq. (63). Though these two measures have been very often used by SR researchers, they do not represent the human visual system very well BIB010 . Therefore, other measures, such as the correlation coefficient (CC) and structural similarity measure (SSIM), have also been involved in SR algorithms BIB010 BIB011 . CC is defined by: BIB016 BIB017 BIB018 BIB024  BIB053 where and μf are the mean of f andf , respectively. The maximum value of CC is one which means a perfect match between the reconstructed image and the ground truth BIB010 . The other measure, SSIM, is defined by: where C 1 and C 2 are constants, σ f and σf are the standard deviations of the associated images, and σ ff is defined by: To use SSIM, the image is usually first divided into subimages, and then Eq. (67) is applied to every sub-image and the mean of all the values is used as the actual measure between the super-resolved image and the ground truth. A mean value of SSIM close to unity means a perfect match between the two images. It is discussed in BIB054 that SSIM favors blur over texture misalignment, and therefore may favor algorithms which do not provide enough texture details. Table 13 shows an overview over the objective measurements used in SR algorithms. Besides those reported in the table, the following objective measurements have been used in at least one paper: -Correlation coefficient in BIB010 BIB038 , -Feature similarity in BIB055 , -Triangle orientation discrimination BIB012  BIB025  -Point reproduction ratio (PRR) and mean segment reproduction ratio (MSRR)  : PRR measures the closeness of two sets of interest points (extracted by Harris, or SIFT): one generated from the super-resolved image and the other from the ground truth. MSRR works with similar sets but the interest points are replaced by the results of some segmentation algorithms. -Feature similarity in BIB055 using histogram of oriented gradients, -Blind image quality index and sharpness index  , -Expressiveness and predictive power of image patches for single-image based SR BIB054 , -Spectral angle mapper BIB056 for hyper-spectral imaging, and -Fourier ring correlation BIB049 . Besides these assessment measures, some authors have shown that feeding some real-world applications by images that are produced by some SR algorithms improves the performance of those applications to some degree, for example, in - BIB003  BIB050 , the readability of the employed optical character recognition system has been improved. - BIB004 BIB005 BIB006 BIB019 BIB020 BIB021 BIB022 BIB026 BIB027 BIB028  BIB039 , BIB040 BIB041 BIB042 BIB051 BIB057 BIB058 BIB059 BIB060 BIB061  BIB062 , the recognition rates of the employed face recognition systems are increased. - BIB013 , better contrast ratios in the super-resolved PET images are obtained. - BIB033 , the recognition rate of a number plate reader has been improved. Table 14 shows the list of the most common 2D facial databases that have been used in SR algorithms. Table 15 shows more details about these databases. One feature which is interesting for SR databases is providing ground truth data for different resolutions (i.e., different distances from the camera). Among the most common databases reviewed (shown in Table 15 ), only CAS-PEAL facial database provides such a data. Besides the list shown in Table 14 , the following facial databases have been used in at least one system: CohnKanade face database BIB007 , NUST BIB022 BIB028 , BioID BIB034  , Terrascope BIB020 , Asian Face Database PF01 BIB023 , FG-NET Database BIB029 , Korean Face Database BIB024 , Max Planck Institute Face Database BIB024 , IMM Face Database BIB030 , PAL BIB035 BIB043 , USC-SIPI BIB044 BIB045  , Georgia Tech BIB036 BIB052 , ViDTIMIT  , FRI CVL BIB041 , Face96 BIB041 , FEI face database BIB063 , MBGC face and iris database BIB058 , SOFTPIA Japan Face Database BIB060 . The following list shows some of the reported databases that SR algorithms have used in other applications than facial imaging: -Aerial imaging: forward looking infrared BIB001 , IKONOS satellite BIB008 BIB064 , Moffett Field and BearFruitGray for hyper-spectral imaging BIB009 , Landsat7 BIB031 . -MDSP dataset (a multi-purpose dataset): BIB002  BIB014 BIB015 BIB046 599 ]. -Medical imaging: PROPELLER MRI database BIB032 , Alzheimer's disease neuroimaging initiative (ADNI) database. BIB047 , DIR-lab 4D-CT dataset BIB065 , and 3D MRI of the brain web simulated database BIB047 BIB066 566 ]. -Natural images: Berkeley segmentation database BIB037 . -Stereo pair database BIB048 . Moreover, there are many SR works that are tested on common images, like Lena (for single-image SR) and resolution chart (for single-and multiple-image SR).