Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Many interesting and globally ordered patterns of behavior, such as solidification, arise in statistical physics and are generally referred to as collective phenomena. The obvious analogies to parallel computation can be extended quite far, so that simple computations may be endowed with the most desirable properties of collective phenomena: robustness against circuit defects, extreme parallelism, asynchronous operation and efficient implementation in silicon. To obtain these advantages for more complicated and useful computations, the relatively simple pattern recognition task of fingerprint identification has been selected. Simulations show that an intuitively understandable neural network can generate fingerprint-like patterns within a framework which should allow control of wire length and scale invariance. The purpose of generating such patterns is to create a network whose stable states are noiseless fingerprint patterns, so that noisy fingerprint patterns used as input to the network will evoke the corresponding noiseless patterns as output. There is a developing theory for predicting the behavior of such networks and thereby reducing the amount of simulation that must be done to design them. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Accurate image expansion is important in many areas of image analysis. Common methods of expansion, such as linear and spline techniques, tend to smooth the image data at edge regions. This paper introduces a method for nonlinear image expansion which preserves the discontinuities of the original image, producing an expanded image with improved definition. The maximum a posteriori (MAP) estimation techniques that are proposed for noise-free and noisy images result in the optimization of convex functionals. The expanded images produced from these methods will be shown to be aesthetically and quantitatively superior to images expanded by the standard methods of replication, linear interpolation, and cubic B-spline expansion. > <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> A hidden Markov model, which describes the evolution of a (binary) edge-image along the resolution axis, is presented. The model integrates two layers: A hidden layer consists of sources having the ability of "breeding" along the resolution axis according to a Markovian rule. A second layer consists of a Gibbs random field which is defined by all the sources. The available image is a realization of this field. After fitting such a model to a given pyramid, it is possible to estimate the super-resolution images by synthesizing additional levels of the process which created the pyramid. The hidden Markov model is found to be a useful tool, allowing us to incorporate selected properties in the process of evolution along the resolution axis, while simultaneously providing an interpretation of this process. The properties incorporated into the model significantly influence the super-resolution image. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Faces often appear very small in surveillance imagery because of the wide fields of view that are typically used and the relatively large distance between the cameras and the scene. For tasks such as face recognition, resolution enhancement techniques are therefore generally needed. Although numerous resolution enhancement algorithms have been proposed in the literature, most of them are limited by the fact that they make weak, if any, assumptions about the scene. We propose an algorithm to learn a prior on the spatial distribution of the image gradient for frontal images of faces. We proceed to show how such a prior can be incorporated into a resolution enhancement algorithm to yield 4- to 8-fold improvements in resolution (i.e., 16 to 64 times as many pixels). The additional pixels are, in effect, hallucinated. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> We analyze the super-resolution reconstruction constraints. In particular we derive a sequence of results which all show that the constraints provide far less useful information as the magnification factor increases. It is well established that the use of a smoothness prior may help somewhat, however for large enough magnification factors any smoothness prior leads to overly smooth results. We therefore propose an algorithm that learns recognition-based priors for specific classes of scenes, the use of which gives far better super-resolution results for both faces and text. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Super-resolution is usually posed as a reconstruction problem. The low resolution input images are assumed to be noisy, downsampled versions of an unknown super-resolution image that is to be estimated. A common way of inverting the down-sampling process is to write down the reconstruction constraints and then solve them, often adding a smoothness prior to regularize the solution. In this paper, we present two results which both show that there is more to super-resolution than image reconstruction. We first analyze the reconstruction constraints and show that they provide less and less useful information as the magnification factor increases. Afterwards, we describe a “hallucination” algorithm, incorporating the recognition of local features in the low resolution images, which outperforms existing reconstruction-based algorithms. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> A variety of super-resolution algorithms have been described in this book. Most of them are based on the same source of information however; that the super-resolution image should generate the lower resolution input images when appropriately warped and down-sampled to model image formation. (This information is usually incorporated into super-resolution algorithms in the form of reconstruction constraints which are frequently combined with a smoothness prior to regularize their solution.) In this final chapter, we first investigate how much extra information is actually added by having more than one image for super-resolution. In particular, we derive a sequence of analytical results which show that the reconstruction constraints provide far less useful information as the decimation ratio increases. We validate these results empirically and show that for large enough decimation ratios any smoothness prior leads to overly smooth results with very little high-frequency content however many (noiseless) low resolution input images are used. In the second half of this chapter, we propose a super-resolution algorithm which uses a completely different source of information, in addition to the reconstruction constraints. The algorithm recognizes local “features” in the low resolution images and then enhances their resolution in an appropriate manner, based on a collection of high and low-resolution training samples. We call such an algorithm a hallucination algorithm. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> The objective of the work presented is the super-resolution restoration of a set of images, and we investigate the use of learnt image models within a generative Bayesian framework. It is demonstrated that restoration of far higher quality than that determined by classical maximum likelihood estimation can be achieved by either constraining the solution to lie on a restricted sub-space, or by using the sub-space to define a spatially varying prior. This sub-space can be learnt from image examples. The methods are applied to both real and synthetic images of text and faces, and results are compared to R.R. Schultz and R.L. Stevenson's (1996) MAP estimator. We consider in particular images of scenes for which the point-to-point mapping is a plane projective transformation which has 8 degrees of freedom. In the real image examples, registration is obtained from the images using automatic methods. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we study face hallucination, or synthesizing a high-resolution face image from low-resolution input, with the help of a large collection of high-resolution face images. We develop a two-step statistical modeling approach that integrates both a global parametric model and a local nonparametric model. First, we derive a global linear model to learn the relationship between the high-resolution face images and their smoothed and down-sampled lower resolution ones. Second, the residual between an original high-resolution image and the reconstructed high-resolution image by a learned linear model is modeled by a patch-based nonparametric Markov network, to capture the high-frequency content of faces. By integrating both global and local models, we can generate photorealistic face images. Our approach is demonstrated by extensive experiments with high-quality hallucinated faces. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> The problem of super-resolution involves generating feasible higher resolution images, which are pleasing to the eye and realistic, from a given low resolution image. This might be attempted by using simple filters for smoothing out the high resolution blocks or through applications where substantial prior information is used to imply the textures and shapes which will occur in the images. In this paper we describe an approach which lies between the two extremes. It is a generic unsupervised method which is usable in all domains, but goes beyond simple smoothing methods in what it achieves. We use a dynamic tree-like architecture to model the high resolution data. Approximate conditioning on the low resolution image is achieved through a mean field approach. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In video surveillance, the faces of interest are often of small size. Image resolution is an important factor affecting face recognition by human and computer. In this paper, we study the face recognition performance using different image resolutions. For automatic face recognition, a low resolution bound is found through experiments. We use an eigentransformation based hallucination method to improve the image resolution. The hallucinated face images are not only much helpful for recognition by human, but also make the automatic recognition procedure easier, since they emphasize the face difference by adding some high frequency details. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> The super-resolution of a single image of unknown point spread-function (PSF) is addressed by extending a learning framework using blind deconvolution with an uncertainty around the resulting PSF. Results indicate success in refining the estimate of the PSF as well as to restoring the image. A novel disparity measure is also proposed to quantify the results. <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Face hallucination is used to synthesize a high-resolution facial image from a low-resolution input. In this paper, we present a framework for face hallucination with pose variation. We derive a texture model consisting of a set of linear mappings between the Gabor wavelet features of the facial images of every two possible poses. Given a low-resolution facial image, its pose is first estimated using a SVM classifier. Then, the Gabor wavelet feature corresponding to the frontal face is computed by our texture model and the low-resolution frontal facial image is reconstructed from its Gabor wavelet feature by a novel algorithm we propose. Finally, the high-resolution face can be hallucinated by one of the hallucination approaches for frontal faces. Our framework is demonstrated by extensive experiments with high-quality hallucinated results. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we propose a novel method for solving single-image super-resolution problems. Given a low-resolution image as input, we recover its high-resolution counterpart using a set of training examples. While this formulation resembles other learning-based methods for super-resolution, our method has been inspired by recent manifold teaming methods, particularly locally linear embedding (LLE). Specifically, small image patches in the lowand high-resolution images form manifolds with similar local geometry in two distinct feature spaces. As in LLE, local geometry is characterized by how a feature vector corresponding to a patch can be reconstructed by its neighbors in the feature space. Besides using the training image pairs to estimate the high-resolution embedding, we also enforce local compatibility and smoothness constraints between patches in the target high-resolution image through overlapping. Experiments show that our method is very flexible and gives good empirical results. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper we propose a robust learning-based face hallucination algorithm, which predicts a high-resolution face image from an input low-resolution image. It can be utilized for many computer vision tasks, such as face recognition and face tracking. With the help of a database of other high-resolution face images, we use a steerable pyramid to extract multi-orientation and multi-scale information of local low-level facial features both from the input low-resolution face image and other high-resolution ones, and use a pyramid-like parent structure and local best match approach to estimate the best prior; then, this prior is incorporated into a Bayesian maximum a posterior (MAP) framework, and finally the high-resolution version is optimized by a steepest decent algorithm. The experimental results show that we can enhance a 24x32 face image into a 96x128 one while the visual effect is relatively good. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Super-resolution is a technique to restore the detailed information from the degenerated data. Lots of previous work is for 2D images while super-resolution of 3D models was little addressed. This paper focuses on the super-resolution of 3D human faces. We firstly extend the 2D image pyramid model to the progressive resolution chain (PRC) model in 3D domain, to describe the detail variation during resolution decreasing. Then a consistent planar representation of 3D faces is presented, which enables the analysis and comparison among the features of the same facial part for the subsequent restoration process. Finally, formulated as solving an iterative quadratic system by maximizing a posteriori, a 3D restoration algorithm using PRC features is given. The experimental results on USF HnmanID 3D face database demonstrate tlie effectiveness of the proposed approach. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we present a novel learning based framework for performing super-resolution using multiple images. We model the image as an undirected graphical model over image patches in which the compatibility functions are represented as non-parametric kernel densities which are learnt from training data. The observed images are translation rectified and stitched together onto a high resolution grid and the inference problem reduces to estimating unknown pixels in the grid. We solve the inference problem by using an extended version of the non-parametric belief propagation algorithm. We show experimental results on synthetic digit images and real face images from the ORL face dataset. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Given a person's neutral face image, we can predict his/her expressive face images by machine learning techniques. Different from the prior expression cloning or image analogy approaches, we try to hallucinate the person's plausible facial expression with the help of a face expression database. In the first step, nonlinear manifold learning technique is used to obtain a smooth estimation for the expressive face image. In the second step, Markov network is adopted to learn the low-level local facial feature's relationship between the residual neutral and the expressional face image's patches in the training set, then belief propagation is employed to infer the expressional residual face image for that person. By integrating the two approaches, we obtain the final result. The experimental results show that the hallucinated facial expression is not only expressive but also close to the ground truth. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Super-resolution is the problem of generating one or a set of high-resolution images from one or a sequence of low-resolution frames. Most methods have been proposed for super-resolution based on multiple low resolution images of the same scene, which is called multiple-frame super-resolution. Only a few approaches produce a high-resolution image from a single low-resolution image, with the help of one or a set of training images from scenes of the same or different types. It is referred to as single-frame super-resolution. This article reviews a variety of single-frame super-resolution methods proposed in the recent years. In the paper, a new manifold learning method: locally linear embedding (LLE) and its relation with single-frame super-resolution is introduced. Detailed study of a critical issue: "neighborhood issue" is presented with related experimental results and analysis and possible future research is given. <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we propose a learning-based method for video super-resolution. There are two main contributions of the proposed method. First, information from cameras with different spatial-temporal resolutions is combined in our framework. This is achieved by constructing training dictionary using the high resolution images captured by still camera and the low resolution video is enhanced via searching in this customized database. Second, we enforce the spatio-temporal constraints using the conditional random field (CRF) and the problem of video super-resolution is posed as finding the high resolution video that maximizes the conditional probability. We apply the algorithm to video sequences taken from different scenes using cameras with different qualities and promising results are presented. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Regularization plays a vital role in inverse problems, and especially in ill-posed ones. Along with classical regularization techniques based on smoothness, entropy, and sparsity, an emerging powerful regularization is one that leans on image examples. In this paper, we propose an efficient scheme for using image examples as driving a powerful regularization, applied to the image scale-up (super-resolution) problem. In this work, we target specifically scanned documents containing written text, graphics, and equations. Our algorithm starts by assigning per each location in the degraded image several candidate high-quality patches. Those are found as the nearest-neighbors (NN) in an image-database that contains pairs of corresponding low- and high-quality image patches. The found examples are used for the definition of an image prior expression, merged into a global MAP penalty function. We use this penalty function both for rejecting some of the irrelevant outlier examples, and then for reconstructing the desired image. We demonstrate our algorithm on several scanned documents with promising results. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Image enhancement of low-resolution images can be done through methods such as interpolation, super-resolution using multiple video frames, and example-based super-resolution. Example-based super-resolution, in particular, is suited to images that have a strong prior (for those frameworks that work on only a single image, it is more like image restoration than traditional, multiframe super-resolution). For example, hallucination and Markov random field (MRF) methods use examples drawn from the same domain as the image being enhanced to determine what the missing high-frequency information is likely to be. We propose to use even stronger prior information by extending MRF-based super-resolution to use adaptive observation and transition functions, that is, to make these functions region-dependent. We show with face images how we can adapt the modeling for each image patch so as to improve the resolution. <s> BIB022 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we study face hallucination, or synthesizing a high-resolution face image from an input low-resolution image, with the help of a large collection of other high-resolution face images. Our theoretical contribution is a two-step statistical modeling approach that integrates both a global parametric model and a local nonparametric model. At the first step, we derive a global linear model to learn the relationship between the high-resolution face images and their smoothed and down-sampled lower resolution ones. At the second step, we model the residue between an original high-resolution image and the reconstructed high-resolution image after applying the learned linear model by a patch-based non-parametric Markov network to capture the high-frequency content. By integrating both global and local models, we can generate photorealistic face images. A practical contribution is a robust warping algorithm to align the low-resolution face images to obtain good hallucination results. The effectiveness of our approach is demonstrated by extensive experiments generating high-quality hallucinated face images from low-resolution input with no manual alignment. <s> BIB023 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> This paper addresses the problem of super-resolving a single image and recovering the characteristics of the sensor using a learning-based approach. In particular, the point spread function (PSF) of the camera is sought by minimizing the mean Euclidean distance function between patches from the input frame and from degraded versions of high-resolution training images. Once an estimate of the PSF is obtained, a supervised learning algorithm can then be used as is. Results are compared with another method for blind super-resolution by using a series of quality measures. <s> BIB024 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we propose a new super-resolution face hallucination and recognition method based on Independent Component Analysis (ICA). Firstly, ICA is used to build a linear mixing relationship between high-resolution (HR) face image and independent HR source faces images. The linear mixing coefficients are retained, thus the corresponding low-resolution (LR) face image is represented by linear mixture of down-sampled source faces images. So, when the source faces images are obtained by training a set of HR face images, unconstrained least square is utilized to obtain mixing coefficients to a LR image for hallucination and recognition. Experiments show that the accuracy of face recognition is insensitive to image size and the number of HR source faces images when image size is larger than 8×8, and the resolution and quality of the hallucinated face image are greatly enhanced over the LR ones, which is very helpful for human recognition. <s> BIB025 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Face super-resolution is to synthesize a high-resolution facial image from a low-resolution input, which can significantly improve the recognition for computer and human. In this paper, we propose a new method of super-resolution based on hybrid model including a linear model of eigenface super-resolution and a Bayesian formulation model. Principal Component Analysis (PCA) is used to approximately represent the input face image by linear combination of limited eigenface images. Then preliminary estimation of super-resolution result can be given by hallucinating the low-resolution eigenface images in the linear combination representation respectively. Finally, we use a Bayesian estimation algorithm to consider of the effect brought by subspace representation error and observation noise. Our method is demonstrated by extensive experiments with promising results of high-quality hallucinated results. <s> BIB026 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper a novel independent component analysis (ICA) based face super-resolution algorithm is proposed. In this scheme, the independent components (ICs) are obtained by offline training high-resolution face images. Given a low-resolution image, the high-resolution image is reconstructed by the linear combination of the ICs where the weight coefficients are obtained by the method of maximum a posteriori (MAP). The prior of ICA coefficients are estimated by performing PCA on training images. Meanwhile, a structure-tensor based filter is proposed to refine the intermediate SR result which is estimated by inverse ICA. Experimental results demonstrate that the proposed algorithm is robust to various pose, expressions and lighting conditions. The hallucination results preserve both the global structure and the high spatial-frequency information better such as sharp edges and high contrast. <s> BIB027 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In facial image analysis, image resolution is an important factor which has great influence on the performance of face recognition systems. As for low-resolution face recognition problem, traditional methods usually carry out super-resolution firstly before passing the super-resolved image to a face recognition system. In this paper, we propose a new method which predicts high-resolution images and the corresponding features simultaneously. More specifically, we propose ldquofeature hallucinationrdquo to project facial images with low-resolution into an expected feature space. As a result, the proposed method does not require super-resolution as an explicit preprocessing step. In addition, we explore a constrained hallucination that considers the local consistency in the image grid. In our method, we use the index of local visual primitives as features and a block-based histogram distance to measure the similarity for the face recognition. Experimental results on FERET face database verify that the proposed method can improve both visual quality and recognition rate for low-resolution facial images. <s> BIB028 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In video surveillance, face images that are captured by common cameras usually have a low resolution, which is a great obstacle to face recognition. In this paper, we show that classification accuracy drops very quickly and algebraic features of face image change greatly as resolution decreases, and we further indicate that two orthogonal matrices (TOMs) of SVD contain the leading information for recognition. Based on the phenomena, we propose a feature-based face super-resolution method using eigentransformation by 2DPCA. TOMs of SVD on face image matrices are reconstructed instead of super-resolution being performed in pixel domain. Reconstructed algebraic features are formed into high-resolution face images and also be used for face recognition directly. Experiments demonstrate that our method obtains hallucinated face images with good vision, and improves performance of low-resolution face recognition. <s> BIB029 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we propose a new super-resolution face hallucination method based on Bilateral-projection-based Two-Dimensional Principal Component Analysis (B2DPCA). Firstly, the high-resolution (HR) face image and its corresponding low-resolution (LR) face image are projected to the HR and LR B2DPCA feature spaces, respectively. In these spaces, the linear mixing relationship between HR and LR feature is estimated from a training set. For reconstructing the HR image from the observed LR image, the LR image is firstly projected to LR feature space and then mapped to HR feature. Finally, the HR feature is reconstructed to the HR face image. Experiments on the well-known face databases show that the performance of our proposed method. The resolution and quality of the hallucinated face images are greatly enhanced over the LR ones, which is very helpful for human recognition. <s> BIB030 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we address the problem of hallucinating a high resolution face given a low resolution input face. The problem is approached through sparse coding. To exploit the facial structure, non-negative matrix factorization (NMF) is first employed to learn a localized part-based subspace. This subspace is effective for super-resolving the incoming low resolution face under reconstruction constraints. To further enhance the detailed facial information, we propose a local patch method based on sparse representation with respect to coupled overcomplete patch dictionaries, which can be fast solved through linear programming. Experiments demonstrate that our approach can hallucinate high quality super-resolution faces. <s> BIB031 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> We address a learning-based method for super resolution. Training sample set provide a candidate high resolution interpretation for the low-resolution images.Modeling image patches as Markov network node, and we learn the parameters of the network from training set,compute probability distribution by K-means algorithm. Given a new low-resolution image to enhance, we select from the training data a set of 10 candidate high-resolution patches for each patch of low-resolution image. In Bayesian belief propagation, we use compatibility relationship between neighboring candidate patches to select the most probable high-resolution candidate. The experimental results show that this method can obtain better result. <s> BIB032 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we propose a novel face hallucination method which mainly focuses on recovering the lost high-frequency information of low resolution face images. Both the global information and local detailed features are taken into account in the proposed method. Firstly, the resolution of the low-resolution face image is enhanced by an edge-protecting interpolation algorithm. Then, the structural information of the interpolated face is extracted by principal component analysis. Utilizing the training database, we synthesize the local high-frequency details according to its corresponding structural information. At last, we can get a final high-resolution result by integrating the interpolated face with its corresponding local high-frequency details. Experiments demonstrate that this method can achieve face hallucination with good visual quality. <s> BIB033 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> This paper demonstrates how Kernel Principal Component Analysis (KPCA) can be used for face hallucination. Different with other KPCA-based methods, KPCA in this paper handles samples from two subspaces, namely the high- and low- resolution image spaces. As KPCA learns not only linear features but also non-linear features, it is anticipated that more detailed facial features could be synthesized. We propose a new model and give theoretical analysis on when it is applicable. Algorithm is then developed for implementation. Experiments are conducted to compare the proposed method with the existing well-known face hallucination methods in terms of visual quality and mean square error. Our results are better and encouraging. Motivated by Wang and Tang's (2) work and the successful applications of KPCA for other face image preprocessing (10)(12), we develop a new method to learn both linear and non-linear intrinsic properties of low-resolution image and high-resolution image using KPCA. Unlike PCA, KPCA is nonlinear and can model structures in given data set better, so that more detailed features could be synthesized. However, we would analyze in next section that the KPCA-based method can not be developed in a straightforward way just by simple combination of kernel and the PCA-based method. As will be analyzed later, the main difficulty is that it is generally difficult to find the relationship between the two kernel feature spaces corresponding to low- and high- resolution data respectively, while the assumption used in PCA-based face hallucination cannot be applied for kernel approach. In order to solve such problem, we propose a new model by assuming that when suitable kernels are used, different resolution images can approximately have the same intrinsic features in their corresponding kernel principal subspaces. Theoretical analysis why this assumption can be applicable will be given. Following this model, the KPCA-based face hallucination method is then developed. In addition, we also propose a new two-step face hallucination approach by integrating the residue compensation to enhance the equality of reconstructed high- resolution images. <s> BIB034 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Given a set of images captured using different integer zoom settings of a camera, we propose a fast approach to obtain super-resolution (SR) for the least zoomed image at a resolution of the most zoomed image. We first obtain SR approximation to the superresolved image using a learning based approach that uses training database consisting of low-resolution (LR) and their high-resolution (HR) images. We model the LR observations as the aliased and noisy versions of their HR parts and estimate the decimation using the learned SR approximation and the available least zoomed observation. A discontinuity preserving Markov random field (MRF) is used as a prior and its parameters are estimated using the SR approximation. Finally Maximum a posteriori (MAP)-MRF formulation is used and the final cost function is optimized using particle swarm optimization (PSO) technique which is computationally efficient compared to simulated annealing. The proposed method can be used in multiresolution fusion for remotely sensed images where the available HR panchromatic image can be used to obtain HR multispectral images. Another interesting application is in immersive navigation for walk through application. Here one can change zoom setting without compromising on the spatial resolution. <s> BIB035 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, we propose a new face hallucination using Eigen transformation with error regression model. Normally in the traditional methods, a high-resolution (HR) face image is reconstructed only from low-resolution (LR) face image. Nevertheless, none of these works interested to take advantage of reconstruction error. Therefore, the error information is included in our framework to correct the final result. In this way, the error estimation can be used from the existing LR feature in eigen space to be found by regression analysis, in order to improve the performance of facial image reconstruction. Our framework consists of two-phase series. In the first phase, learning process is from the mistakes in reconstruct face images of the training dataset by Eigen transformation, then finding the relationship between input and error by regression analysis. In the second phase, hallucinating process uses normal method by Eigen transformation, after that the result is corrected with the error estimation. Experimental results on the well-known face databases show that the resolution and quality of the hallucinated face images are greatly enhanced over the traditional Eigen transformation method, which is very helpful for face recognition. <s> BIB036 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> This paper proposes a new face hallucination technique for color face image reconstruction using Eigentransformation with error regression model. Generally, a high-resolution (HR) face image is reconstructed only from low-resolution (LR) face image. Nevertheless, previous researches neglect to gain benefit from error of face reconstruction. Therefore, in order to improve the performance of facial image reconstruction, the error information is included in our framework to correct the final result. In this way, the regression analysis was used to find the error estimation, which can be obtained from the existing LR in eigen space for each color channel. To handle with color image, each color channel in RGB model is separately processed by our framework. Our framework consists of two-phase. First phase learn error of face image reconstruction from training dataset by Eigentransfomation, then regression analysis is performed to find relationship between input and error. Second phase hallucinates using normal method by Eigentransformation, after that the result is corrected with the error estimation. Experimental results on the well-known database show that the resolution and quality of the hallucinated color face images are greatly enhanced over the traditional Eigentransformation method, which is very helpful for face recognition. <s> BIB037 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Image super-resolution reconstructions (SR) require image degradation model (DM) as the prior, however, the actual DM is often unknown in practical applications. In this work, a novel framework is proposed for single image SR, where the explicit DM is unknown. Based on Bayesian MAP, an iteration scheme is adopted to update the reconstructed SR image and the DM estimate. During reconstruction, MRF-Gibbs image prior is incorporated for regularization and example-based machine learning technique is employed to draw the DM estimations back to the potential DM space. The SR resulted images by the proposed method are superior to the ones produced by bicubic interpolation and conventional SR algorithm with incorrect DM, in both aesthetical and quantitative aspects. <s> BIB038 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Today's camera sensors usually have a high gray-scale resolution, e.g. 256, however, due to the dramatic lighting variations, the gray-scales distributed to the face region might be far less than 256. Therefore, besides low spatial resolution, a practical face recognition system must also handle degraded face images of low gray-scale resolution (LGR). In the last decade, low spatial resolution problem has been studied prevalently, but LGR problem was rarely studied. Aiming at robust face recognition, this paper makes a first primary attempt to investigate explicitly the LGR problem and empirically reveals that LGR indeed degrades face recognition method significantly. Possible solutions to the problem are discussed and grouped into three categories: gray-scale resolution invariant features, gray-scale degradation modeling and Gray-scale Super-Resolution (GSR). Then, we propose a Coupled Subspace Analysis (CSA) based GSR method to recover the high gray-scale resolution image from a single input LGR image. Extensive experiments on FERET and CMU-PIE face databases show that the proposed method can not only dramatically increase the gray-scale resolution and visualization quality, but also impressively improve the accuracy of face recognition. <s> BIB039 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> A novel two-step super-resolution (SR) method for face images is proposed in this paper. The critical issue of global face reconstruction in the two-step SR framework is to construct the relationship between high resolution (HR) and low resolution (LR) features. We choose the Principal Component Analysis (PCA) coefficients of LR/HR face images as the features for global faces. These features are considered as inputs and outputs of an unknown linear system. The mapping between the inputs and outputs is estimated from training sets as the system response. The HR features corresponding to a test LR image can be obtained by applying the learnt mapping to the LR features, and hence we can reconstruct the global face. Ultimately, an HR face image is generated by using the patch-based neighbor reconstruction that imposes facial details into the global face. Experiments indicate that our method produces HR faces of higher quality and is easier to implement than traditional methods based on two-step framework. <s> BIB040 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Super-resolution reconstruction of face image is the problem of reconstructing a high resolution face image from one or more low resolution face images. Assuming that high and low resolution images share similar intrinsic geometries, various recent super-resolution methods reconstruct high resolution images based on a weights determined from nearest neighbors in the local embedding of low resolution images. These methods suffer disadvantages from the finite number of samples and the nature of manifold learning techniques, and hence yield unrealistic reconstructed images. To address the problem, we apply canonical correlation analysis (CCA), which maximizes the correlation between the local neighbor relationships of high and low resolution images. We use it separately for reconstruction of global face appearance, and facial details. Experiments using a collection of frontal human faces show that the proposed algorithm improves reconstruction quality over existing state-of-the-art super-resolution algorithms, both visually, and using a quantitative peak signal-to-noise ratio assessment. <s> BIB041 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Super-resolution algorithms typically transform images into 1D vectors and operate on these vectors to obtain a high-resolution image. In this study, the authors first propose a 2D method for super-resolution using a 2D model that treats images as matrices. We then apply this 2D model to the super-resolution of face images. Two-directional two-dimensional principal component analysis (PCA) [(2D) 2 -PCA] is an efficient face representation technique where the images are treated as matrices instead of vectors. We use (2D) 2 -PCA to learn the face subspace and use it as a prior to super-resolve face images. Experimental results show that our approach can reconstruct high quality face images with low computational cost. <s> BIB042 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In actual imaging environment, a variety of factors have an impact on the quality of images, which leads to pixel distortion and aliasing. The traditional face super-resolution algorithm only uses the difference of image pixel values as similarity criterion, which degrades similarity and identification of reconstructed facial images. Image semantic information with human understanding, especially structural information, is robust to the degraded pixel values. In this paper, we propose a face super-resolution approach using shape semantic model. This method describes the facial shape as a series of fiducial points on facial image. And shape semantic information of input image is obtained manually. Then a shape semantic regularization is added to the original objective function. The steepest descent method is used to obtain the unified coefficient. Experimental results demonstrate that the proposed method outperforms the traditional schemes significantly both in subjective and objective quality. <s> BIB043 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper, a car plate image restoration system is presented, in which the PCA (Principle Component Analysis) based super-resolution method is applied to restore very low resolution car plate images (height < 10 pixels) from mainland China and Hong Kong. Several algorithms are proposed to improve the robustness and performance of this system. The core part of this system is a dynamic PCA algorithm, in which the training set is adjusted adaptively for achieving better restoration results. Extensive experiments on both simulated test images and practical test images show that our system performances satisfactorily in terms of the restoration error rate and the flexibility in practical applications. <s> BIB044 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> A face hallucination scheme based on independent residual features (IRFs) is proposed in this paper. In the proposed scheme, a high-resolution (HR) face image is assumed as composition of two parts: an approximate and a residual image, where the approximate image is obtained by interpolating the corresponding low-resolution (LR) face image. According to this assumption, a residual image training database is established based on a set of training HR face images and their corresponding LR ones. And independent component analysis is used to extract IRFs from this established database. With constructing a cost function of least square, the IRFs are used to derive the corresponding residual image from any given LR face image and an expected HR one can be rendered. Experiments show the proposed scheme can effectively improve the resolution of a face image, on matter whether the face image belongs to the training face image set or not, under different light conditions. <s> BIB045 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> In this paper we propose to convert the task of face hallucination into an image decomposition problem, and then use the morphological component analysis (MCA) for hallucinating a single face image, based on a novel three-step framework. Firstly, a low-resolution input image is up-sampled by interpolation. Then, the MCA is employed to decompose the interpolated image into a high-resolution image and an unsharp masking, as MCA can properly decompose a signal into special parts according to typical dictionaries. Finally, a residue compensation, which is based on the neighbor reconstruction of patches, is performed to enhance the facial details. The proposed method can effectively exploit the facial properties for face hallucination under the image decomposition perspective. Experimental results demonstrate the effectiveness of our method, in terms of the visual quality of the hallucinated face images. <s> BIB046 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> This paper addresses the localization and super resolution of license plate in a unifying approach. Higher quality license plate can be obtained using super resolution on successive lower resolution plate images. All existing methods assume that plate zones are correctly extracted from every frame. However, the accurate localization needs a sufficient quality of the image, which is not always true in real video. Super-resolution on all pixels is a possible but much time consuming alternative. We propose a framework which interlaces successfully these two modules. First, coarse candidates are found by an weak but fast license plate detection based on edge map sub-sampling. Then, an improved fast MAP-based super-resolution, using local phase accurate registration and edge preserving prior, applied on these regions of interest. Finally, our robust ICHT-based localizer rejects false-alarms and localizes the high resolution license plate more accurately. Experiments which were conducted on synthetic and real data, proved the robustness of our approach with real-time possibility. <s> BIB047 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> This paper proposes a two-step prototype-face-based scheme of hallucinating the high-resolution detail of a low-resolution input face image. The proposed scheme is mainly composed of two steps: the global estimation step and the local facial-parts refinement step. In the global estimation step, the initial high-resolution face image is hallucinated via a linear combination of the global prototype faces with a coefficient vector. Instead of estimating coefficient vector in the high-dimensional raw image domain, we propose a maximum a posteriori (MAP) estimator to estimate the optimum set of coefficients in the low-dimensional coefficient domain. In the local refinement step, the facial parts (i.e., eyes, nose and mouth) are further refined using a basis selection method based on overcomplete nonnegative matrix factorization (ONMF). Experimental results demonstrate that the proposed method can achieve significant subjective and objective improvement over state-of-the-art face hallucination methods, especially when an input face does not belong to a person in the training data set. <s> BIB048 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Most popular learning-based super-resolution (SR) approaches suffer from complicated learning structures and highly intensive computation, especially in resource-limited applications. We propose a novel frontal facial image SR approach by using multiple local linear transformations to approximate the nonlinear mapping between low-resolution (LR) and high-resolution (HR) images in the pixel domain. We adopt Procrustes analysis to obtain orthogonal matrices representing the learned linear transformations, which cannot only well capture appearance variations in facial patches but also greatly simplify the transformation computation to matrices manipulation. An HR image can be directly reconstructed from a single LR image without need of the large training data, thus avoiding the use of a large redundant LR and HR patch database. Experimental results show that our approach is computationally fast as well the SR quality compares favorably with the state-of-the-art approaches from both subjective and objective evaluations. Besides, our approach is insensitive to the size of training data and robust to a wide range of facial variations like occlusions. More importantly, the proposed method is also much more effective than other comparative methods to reconstruct real-world images captured from the Internet and webcams. <s> BIB049 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> Human faces in surveillance video images usually have low resolution and poor quality. They need to be reconstructed in super-resolution for identification. The traditional subspace-based face super-resolution algorithms are sensitive to light. For solving the problem, this paper proposes a face super-resolution method based on illumination invariant feature. The method firstly extracts the illumination invariant features of an input low resolution image by using adaptive L1–L2 total variation model and self-quotient image in logarithmic domain. Then it projects the feature onto non-negative basis obtained by Nonnegative Matrix Factorization(NMF) in face image database. Finally it reconstructs the high resolution face images under the framework of Maximum A Posteriori (MAP) probability. Experimental results demonstrate that the proposed method outperforms the compared methods both in subjective and objective quality under poor light conditions. <s> BIB050 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> The low resolution of images has been one of the major limitations in recognising humans from a distance using their biometric traits, such as face and iris. Superresolution has been employed to improve the resolution and the recognition performance simultaneously, however the majority of techniques employed operate in the pixel domain, such that the biometric feature vectors are extracted from a super-resolved input image. Feature-domain superresolution has been proposed for face and iris, and is shown to further improve recognition performance by capitalising on direct super-resolving the features which are used for recognition. However, current feature-domain superresolution approaches are limited to simple linear features such as Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), which are not the most discriminant features for biometrics. Gabor-based features have been shown to be one of the most discriminant features for biometrics including face and iris. This paper proposes a framework to conduct super-resolution in the non-linear Gabor feature domain to further improve the recognition performance of biometric systems. Experiments have confirmed the validity of the proposed approach, demonstrating superior performance to existing linear approaches for both face and iris biometrics. <s> BIB051 </s> Super-resolution: a comprehensive survey <s> Learning-based single-image SR algorithms <s> This paper describes a novel method for single-image super-resolution (SR) based on a neighbor embedding technique which uses Semi-Nonnegative Matrix Factorization (SNMF). Each low-resolution (LR) input patch is approximated by a linear combination of nearest neighbors taken from a dictionary. This dictionary stores low-resolution and corresponding high-resolution (HR) patches taken from natural images and is thus used to infer the HR details of the super-resolved image. The entire neighbor embedding procedure is carried out in a feature space. Features which are either the gradient values of the pixels or the mean-subtracted luminance values are extracted from the LR input patches, and from the LR and HR patches stored in the dictionary. The algorithm thus searches for the K nearest neighbors of the feature vector of the LR input patch and then computes the weights for approximating the input feature vector. The use of SNMF for computing the weights of the linear approximation is shown to have a more stable behavior than the use of LLE and lead to significantly higher PSNR values for the super-resolved images. <s> BIB052
These algorithms, a.k.a, as learning-based or Hallucination algorithms (Table 9 ) were first introduced in BIB001 in which a neural network was used to improve the resolution of fingerprint images. These algorithms contain a training step in which the relationship between some HR examples (from a specific class like face images, fingerprints, etc.) and their LR counterparts is learned. This learned knowledge is then incorporated into the a priori term of the reconstruction. The training database of learning-based SR algorithms needs to have a proper generalization capability BIB020 . To measure this, the two factors of sufficiency and predictability have been introduced in BIB020 . Using a larger database does not necessarily generate better results, on the contrary, a larger number of irrelevant examples not only increase the computational time of searching for the best matches, but also disturb this search  . To deal with this, in  it is suggested to use a content-based classification of image patches (like codebook) during the training. Different types of learning-based SR algorithms are discussed in the following subsections. Feature pyramids The notable work in this group was developed by Baker and Kanade [57, BIB004 BIB006 BIB005 BIB007 for face hallucination. In the training step of this algorithm, each HR face image is first down-sampled and blurred several times to produce a Gaussian resolution pyramid. Then, from these Gaussian pyramids, Laplacian pyramids and then Feature pyramids are generated. Features can be simply the derivatives of the Gaussian pyramids which contain the low-frequency information of the face images, Laplacian pyramids which contain the band-pass frequency of the face images, steerable pyramids which contain multiorientational information of the face images BIB015 , etc. Having trained the system, for an LR test image the most similar LR image among the available LR images in all the pyramids is found. In the original work of [57, 58, BIB004 BIB006 , the nearest neighbor technique was used for finding the most similar images/patches. But in BIB021  , the LR images/patches were arranged in a tree structure, which allows fast search techniques to find the most similar patches. Having found the most similar image/patch, the relationships between the found LR image and its higher counterparts (which have been coded as child-parent structures) are used to predict/hallucinate the high-resolution details of the LR input image as an a priori term in a MAP algorithm similar to the one proposed in BIB002 which uses an HMRF a priori term. However, the employed a priori term similar to most of the MAP methods considers local constraints in the reconstructed image. Exactly this technique has been extended in BIB016 to hallucinate 3D HR face models from LR 3D inputs. Belief network The notable work of this group was developed by Freeman and Pasztor [65, 66] . These algorithms use a belief network such as a Markov network or a tree structure. In the case of the Markov network BIB003 65, 66,   BIB012   BIB022 BIB024 BIB032 BIB035 both the LR image and its HR counterparts are divided into patches. Then the corresponding patches in the two images are associated to each other by a so-called observation function. The observation function defines how well a candidate HR patch matches a known LR patch BIB022 . The neighbor patches in the superresolved HR image are assumed to be related to each other by a so-called transition function BIB022 . Having trained the model (parameters), it infers the missing HR details of LR input images using a belief propagation algorithm to obtain a MAP super-resolved image. For learning and inferring, the Markov assumptions are used to factorize the posterior probability. For inferring the HR patches, the following equation is used: where f i is the ith patch of the HR image f , g i is its associated LR patch, and j changes to include all the neighbor nodes of the ith patch and L i j is defined by: whereL jk is the L jk from the previous iteration with initial values of 1 BIB003 65, 66,   BIB012 . In BIB017 , in addition to the association potential function of [65, 66,   BIB012 , which models the relationship between the LR and HR patches, a so-called interaction potential function is also used to model the relationship between the HR patches. It is shown in BIB017 that these two potential functions can be used effectively to model the temporal information which can be available in an input LR video sequence. In BIB022 a system has been developed for hallucination of face images in which the transition and the observation functions of BIB003 65, 66,   BIB012 are no longer static for the entire image, but are adapted to different regions of the face image. Similar to the above system, BIB010 has used a mixture of tree structures in the belief network for learning the relationship between the LR images and their corresponding HR ones in the training step. Projection The a priori term learned in [57, BIB004 BIB006 BIB005 BIB007 imposes local constraints on the reconstructed super-resolved image. For imposing global constraints, some SR algorithms have used projection-based methods for learning the a priori term of the employed MAP algorithm, e.g., in BIB008 BIB011 BIB026  BIB033 BIB036 BIB037 BIB038 BIB039 BIB040 BIB041 BIB042 BIB043 BIB044 PCA in BIB025 BIB027 BIB045 BIB049 independent component analysis (ICA), and in BIB046 morphological component analysis (MCA) have been used. In PCA every face is represented by its PC basis, which is computed from training images at the desired resolution f = V y − μ, where V is the set of PC basis vectors and μ is the average of the training images BIB008 . To consider both local and global constraints, Liu et al. BIB009 BIB023 combined non-parametric Markov networks (for considering the local variances) and a PCA-based a priori (for considering the global variances) for face hallucination. In  BIB042 , a Kernel-PCA based prior that is a non-linear extension of the common PCA was embedded in a MAP method to take into account more complex correlations of human face images. In BIB034 again Kernel PCA but this time with radius basis function (RBF) was used for face hallucination. In  PCA was again used for hallucination, but not directly for hallucination of face images, but for their features, i.e., the hallucination technique was applied to the feature vector of the LR face image to hallucinate the feature vector of the corresponding HR image, without actual reconstruction of the image. This hallucinated feature vector was then used for recognition. The same technique of  but with different types of features has been used in some other works, e.g., in - [164, BIB013 BIB047 BIB051 , Gabor filter responses are used as features to hallucinate a texture model of the targeted HR image. These systems can hallucinate a frontal face image from rotated LR input images. - BIB014 BIB018 BIB019 BIB028 , local visual primitives have been used as features and then have been hallucinated using a locally linear embedding (LLE) technique. - BIB029 two orthogonal matrices of SVD of facial images which contain the most important features for recognition have been used in the SR algorithm. In PCA-based methods, usually the matrices representing each training image are first vectorized (by arranging, e.g., all the columns of each matrix in only one column) and then they are combined into a large matrix to obtain the covariance matrix of the training data for modeling the eigenspace. It is discussed in BIB030 that such vectorization of training images may not fully retain their spatial structure information. Instead, it is suggested to apply such a vectorization to the features extracted from training data and use them for SR. It is shown in BIB030 that bilateral projections can produce such features. It has been discussed in BIB031 BIB048 BIB050 that PCA-based hallucination methods are very sensitive to occlusion since the PCA bases are holistic and they mostly render the hallucination results towards the mean face. Therefore, it has been proposed to use non-negative matrix factorization to divide the face images into relatively independent parts, such as eyes, eyebrows, noses, mouthes, checks, and chins. Then, a MAP sparse-representation based algorithm has been applied to these parts. A similar technique has been used in BIB052 .