Super-resolution: a comprehensive survey <s> Neural networks <s> Fuzzy classification techniques have been developed recently to estimate the class composition of image pixels, but their output provides no indication of how these classes are distributed spatially within the instantaneous field of view represented by the pixel. As such, while the accuracy of land cover target identification has been improved using fuzzy classification, it remains for robust techniques that provide better spatial representation of land cover to be developed. Such techniques could provide more accurate land cover metrics for determining social or environmental policy, for example. The use of a Hopfield neural network to map the spatial distribution of classes more reliably using prior information of pixel composition determined from fuzzy classification was investigated. An approach was adopted that used the output from a fuzzy classification to constrain a Hopfield neural network formulated as an energy minimization tool. The network converges to a minimum of an energy function, defined as a goal and several constraints. Extracting the spatial distribution of target class components within each pixel was, therefore, formulated as a constraint satisfaction problem with an optimal solution determined by the minimum of the energy function. This energy minimum represents a "best guess" map of the spatial distribution of class components in each pixel. The technique was applied to both synthetic and simulated Landsat TM imagery, and the resultant maps provided an accurate and improved representation of the land covers studied, with root mean square errors (RMSEs) for Landsat imagery of the order of 0.09 pixels in the new fine resolution image recorded. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper presents an edge-enhancing super-resolution algorithm using anisotropic diffusion technique. Because we solve the super-resolution problem by incorporating anisotropic diffusion, our technique does more than merely reconstruct a high-resolution image from several overlapping noisy low-resolution images and preserve them. In addition to reducing image noise during the restoration process, our method also enhances edges. We apply this technique to a video stream which can be aligned by 3/spl times/3 projective transformations. <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> The paper presents a new method for image resolution enhancement from multiple image frames using an integrated recurrent neural network (IRNN). The IRNN is a set of feedforward neural networks working collectively with the ability of having feedback of information from its output to its input. As such, it is capable of both learning and searching the optimal solution in the solution space for optimisation problems. In other words, it combines the advantages of both the Hopfield network and the multilayered feedforward network in solving the enhanced image reconstruction problem. Simulation results demonstrate that the proposed IRNN can successfully be used to enhance image resolution. The proposed neural network based method is promising for real-time applications, especially when the inherent parallelism of computation of the neural network is explored. Further, it can adapt itself to the various conditions of the reconstruction problem by learning. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a novel method for solving single-image super-resolution problems. Given a low-resolution image as input, we recover its high-resolution counterpart using a set of training examples. While this formulation resembles other learning-based methods for super-resolution, our method has been inspired by recent manifold teaming methods, particularly locally linear embedding (LLE). Specifically, small image patches in the lowand high-resolution images form manifolds with similar local geometry in two distinct feature spaces. As in LLE, local geometry is characterized by how a feature vector corresponding to a patch can be reconstructed by its neighbors in the feature space. Besides using the training image pairs to estimate the high-resolution embedding, we also enforce local compatibility and smoothness constraints between patches in the target high-resolution image through overlapping. Experiments show that our method is very flexible and gives good empirical results. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> The super-resolution of a single image of unknown point spread-function (PSF) is addressed by extending a learning framework using blind deconvolution with an uncertainty around the resulting PSF. Results indicate success in refining the estimate of the PSF as well as to restoring the image. A novel disparity measure is also proposed to quantify the results. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a novel patch-based face hallucination framework, which employs a dual model to hallucinate different components associated with one facial image. Our model is based on a statistical learning approach: associative learning. It suffices to learn the dependencies between low-resolution image patches and their high-resolution ones with a new concept hidden parameter space as a bridge to connect those patches with different resolutions. To compensate higher frequency information of images, we present a dual associative learning algorithm for orderly inferring main components and high frequency components of faces. The patches can be finally integrated to form a whole high-resolution image. Experiments demonstrate that our approach does render high quality superresolution faces. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Image superresolution methods process an input image sequence of a scene to obtain a still image with increased resolution. Classical approaches to this problem involve complex iterative minimization procedures, typically with high computational costs. In this paper is proposed a novel algorithm for super-resolution that enables a substantial decrease in computer load. First, a probabilistic neural network architecture is used to perform a scattered-point interpolation of the image sequence data. The network kernel function is optimally determined for this problem by a multi-layer perceptron trained on synthetic data. Network parameters dependence on sequence noise level is quantitatively analyzed. This super-sampled image is spatially filtered to correct finite pixel size effects, to yield the final high-resolution estimate. Results on a real outdoor sequence are presented, showing the quality of the proposed method. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Image superresolution involves the processing of an image sequence to generate a still image with higher resolution. Classical approaches, such as bayesian MAP methods, require iterative minimization procedures, with high computational costs. Recently, the authors proposed a method to tackle this problem, based on the use of a hybrid MLP-PNN architecture. In this paper, we present a novel superresolution method, based on an evolution of this concept, to incorporate the use of local image models. A neural processing stage receives as input the value of model coefficients on local windows. The data dimensionality is firstly reduced by application of PCA. An MLP, trained on synthetic sequences with various amounts of noise, estimates the high-resolution image data. The effect of varying the dimension of the network input space is examined, showing a complex, structured behavior. Quantitative results are presented showing the accuracy and robustness of the proposed method. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Given a person's neutral face image, we can predict his/her expressive face images by machine learning techniques. Different from the prior expression cloning or image analogy approaches, we try to hallucinate the person's plausible facial expression with the help of a face expression database. In the first step, nonlinear manifold learning technique is used to obtain a smooth estimation for the expressive face image. In the second step, Markov network is adopted to learn the low-level local facial feature's relationship between the residual neutral and the expressional face image's patches in the training set, then belief propagation is employed to infer the expressional residual face image for that person. By integrating the two approaches, we obtain the final result. The experimental results show that the hallucinated facial expression is not only expressive but also close to the ground truth. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Super-resolution is the problem of generating one or a set of high-resolution images from one or a sequence of low-resolution frames. Most methods have been proposed for super-resolution based on multiple low resolution images of the same scene, which is called multiple-frame super-resolution. Only a few approaches produce a high-resolution image from a single low-resolution image, with the help of one or a set of training images from scenes of the same or different types. It is referred to as single-frame super-resolution. This article reviews a variety of single-frame super-resolution methods proposed in the recent years. In the paper, a new manifold learning method: locally linear embedding (LLE) and its relation with single-frame super-resolution is introduced. Detailed study of a critical issue: "neighborhood issue" is presented with related experimental results and analysis and possible future research is given. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Face images of non-frontal views under poor illumination with low resolution reduce dramatically face recognition accuracy. To overcome these problems, super-resolution techniques can be exploited. In this paper, we present a Bayesian framework to perform multi-modal (such as variations in viewpoint and illumination) face image super-resolutions in tensor space. Given a single modal low-resolution face image, we benefit from the multiple factor interactions of training tensor, and super-resolve its high-resolution reconstructions across different modalities. Instead of performing pixel-domain super-resolutions, we reconstruct the high-resolution face images by computing a maximum likelihood identity parameter vector in high-resolution tensor space. Experiments show promising results of multi-view and multi-illumination face image super-resolutions respectively. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Face images of non-frontal views under poor illumination resolution reduce dramatically face recognition accuracy. This is evident most compellingly by the very low recognition rate of all existing face recognition systems when applied to live CCTV camera input. In this paper, we present a Bayesian framework to perform multimodal (such as variations in viewpoint and illumination) face image super-resolution for recognition in tensor space. Given a single modal low-resolution face image, we benefit from the multiple factor interactions of training sensor and super-resolve its high-resolution reconstructions across different modalities for face recognition. Instead of performing pixel-domain super-resolution and recognition independently as two separate sequential processes, we integrate the tasks of super-resolution and recognition by directly computing a maximum likelihood identity parameter vector in high-resolution tensor space for recognition. We show results from multi-modal super-resolution and face recognition experiments across different imaging modalities, using low-resolution images as testing inputs and demonstrate improved recognition rates over standard tensorface and eigenface representations <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a new face hallucination framework based on image patches, which integrates two novel statistical super-resolution models. Considering that image patches reflect the combined effect of personal characteristics and patch-location, we first formulate a TensorPatch model based on multilinear analysis to explicitly model the interaction between multiple constituent factors. Motivated by locally linear embedding, we develop an enhanced multilinear patch hallucination algorithm, which efficiently exploits the local distribution structure in the sample space. To better preserve face subtle details, we derive the coupled PCA algorithm to learn the relation between high-resolution residue and low-resolution residue, which is utilized for compensate the error residue in hallucinated images. Experiments demonstrate that our framework on one hand well maintains the global facial structures, on the other hand recovers the detailed facial traits in high quality. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a novel face hallucination framework based on image patches, which exploits local geometry structures of overlapping patches to hallucinate different components associated with one facial image. To achieve local fidelity while preserving smoothness in the target high-resolution image, we develop a neighbor combination super-resolution model for high-resolution patch synthesis. For further enhancing the detailed information, we propose another model, which effectively learns neighbor transformations between low- and high-resolution image patch residuals to compensate modeling errors caused by the first model. Experiments demonstrate that our approach can hallucinate high quality super-resolution faces. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> An optimal recovery based neural-network super resolution algorithm is developed. The proposed method is computationally less expensive and outputs images with high subjective quality, compared with previous neural-network or optimal recovery algorithms. It is evaluated on classical SR test images with both generic and specialized training sets, and compared with other state-of-the-art methods. Results show that our algorithm is among the state-of-the-art, both in quality and efficiency. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a sequential approach to hallucinate/ synthesize high-resolution images of multiple facial expressions. We propose an idea of multi-resolution tensor for super-resolution, and decompose facial expression images into small local patches. We build a multi-resolution patch tensor across different facial expressions. By unifying the identity parameters and learning the subspace mappings across different resolutions and expressions, we simplify the facial expression hallucination as a problem of parameter recovery in a patch tensor space. We further add a high-frequency component residue using nonparametric patch learning from high-resolution training data. We integrate the sequential statistical modelling into a Bayesian framework, so that given any low-resolution facial image of a single expression, we are able to synthesize multiple facial expression images in high-resolution. We show promising experimental results from both facial expression database and live video sequences. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Inspired by multi-scale tensor voting, a computational framework for perceptual grouping and segmentation, we propose an edge-directed technique for color image superresolution given a single low-resolution color image. Our multi-scale technique combines the advantages of edgedirected, reconstruction-based and learning-based methods, and is unique in two ways. First, we consider simultaneously all the three color channels in our multi-scale tensor voting framework to produce a multi-scale edge representation to guide the process of high-resolution color image reconstruction, which is subject to the back projection constraint. Fine details are inferred without noticeable blurry or ringing artifacts. Second, the inference of highresolution curves is achieved by multi-scale tensor voting, using the dense voting field as an edge-preserving smoothness prior which is derived geometrically without any timeconsuming learning procedure. Qualitative and quantitative results indicate that our method produces convincing results in complex test cases typically used by state-of-theart image super-resolution techniques. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> A regression model in the tensorPCA subspace is proposed in this paper for face super-resolution reconstruction. An approximate conditional probability model is used for the tensor subspace coefficients and maximum-likelihood estimator gives a linear regression model. The approximation is corrected by adding non-linear component from a RBF-type regressor. Experiments on face images from FERET database validate the algorithm. Although each projection coefficient is estimated by a local estimator, tensorPCA subspace analysis is still a global descriptor, which makes the algorithm have certain ability to deal with partially occluded images. <s> BIB018 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper presents a neural network-based method for image super-resolution. In this technique, the super-resolution is considered as an ill-posed inverse problem which is solved by minimizing an evaluation function established based on an observation model that closely follows the physical image acquisition process. A Hopfield neural network is created to obtain an optimal solution to the problem. Not like some other single-frame super-resolution techniques, this technique takes into consideration PSF (point spread function) blurring as well as additive noise and generates high-resolution images with more preserved or restored image details. Experimental results demonstrate that the high-resolution images obtained by this technique have a very high quality in terms of PSNR (peak signal-to-noise ratio) and visually look more pleasant <s> BIB019 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> We propose a novel, learning-based algorithm for image super-resolution. First, an optimal distance-based weighted interpolation of the image sequence is performed using a new neural architecture, hybrid of a multi-layer perceptron and a probabilistic neural network, trained on synthetic image data. Secondly, a linear filter is applied with coefficients learned to restore residual interpolation artifacts in addition to low-resolution blurring, providing noticeable improvements over lens-detector Wiener restorations. Our method has been evaluated on real visible and IR sequences with widely different contents, providing significantly better results that a two-step method with high computational requirements. Results were similar or better than those of a maximum-a-posteriori estimator, with a reduction in processing time by a factor of almost 300. This paves the way to high-quality, quasi-real time applications of super-resolution techniques. <s> BIB020 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper presents the original and versatile architecture of a modular neural network and its application to super-resolution. Each module is a small multilayer perceptron, trained with the Levenberg-Marquardt method, and is used as a generic building block. By connecting the modules together to establish a composition of their individual mappings, we elaborate a lattice of modules that implements full connectivity between the pixels of the low-resolution input image and those of the higher-resolution output image. After the network is trained with patterns made up of low and high-resolution images of objects or scenes of the same kind, it will be able to enhance dramatically the resolution of a similar object's representation. The modular nature of the architecture allows the training phase to be readily parallelized on a network of PCs. Finally, it is shown that the network performs global-scale reconstruction of human faces from very low resolution input images. <s> BIB021 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> A two-phase face hallucination approach is proposed in this paper to infer high-resolution face image from the low-resolution observation based on a set of training image pairs. The proposed locality preserving hallucination (LPH) algorithm combines locality preserving projection (LPP) and radial basis function (RBF) regression together to hallucinate the global high-resolution face. Furthermore, in order to compensate the inferred global face with detailed inartificial facial features, the neighbor reconstruction based face residue hallucination is used. Compared with existing approaches, the proposed LPH algorithm can generate global face more similar to the ground truth face efficiently, moreover, the patch structure and search strategy carefully designed for the neighbor reconstruction algorithm greatly reduce the computational complexity without diminishing the quality of high-resolution face detail. The details of synthetic high-resolution face are further improved by a global linear smoother. Experiments indicate that our approach can synthesize distinct high-resolution faces with various facial appearances such as facial expressions, eyeglasses efficiently. <s> BIB022 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> A novel method for robust super-resolution of face images is proposed in this paper. Face super-resolution is a particular interest in video surveillance where face images have typically very low-resolution quality and there is a need to apply face enhancement or super-resolution algorithms. In this paper, we apply a manifold learning method which has hardly been used for super-resolution. A manifold is a natural generalization of a Euclidean space to a locally Euclidean space. Manifold learning algorithms are more powerful than other pattern recognition methods which analyze a Euclidean space because they can reveal the underlying nonlinear distribution of the face space; however, there are some practical problems which prevent these algorithms from being applied to super-resolution. Almost all of the manifold learning methods cannot generate mapping functions for new test images which are absent from a training set. Another factor is that super-resolution seeks to recover a high-dimensional image from a lower-dimensional one while manifold learning methods perform the exact opposite as they are applied to dimensionality reduction. In this paper, we break the limitation of applying manifold learning methods for face super-resolution by proposing a novel method using locality preserving projections (LPP). <s> BIB023 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a novel method for performing robust super-resolution of face images. Face super-resolution is to recover a high-resolution face image from a given low-resolution face image by modeling a face image space in view of multiple resolutions. The proposed method is based on the assumption that a low-resolution image space and a high-resolution image space have similar local geometries but also have partial distortions of neighborhood relationships between facial images. In this paper, local geometry is analyzed by an idea inspired by locally linear embedding (LLE), the state-of-the art manifold learning method. Using the analyzed neighborhood relationships, two sets of neighborhoods in the low-and high-resolution image spaces become more similar in an iterative way. In this paper, we show that changing resolution causes the partial distortions of neighborhood embeddings obtained by a manifold learning method. Experimental results show that the proposed method produces more reliable results of face super-resolution than the traditional way using neighbor embedding. <s> BIB024 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper focuses on hallucinating a facial shape from a low-resolution 3D facial shape. Firstly, we give a constrained conformal embedding of 3D shape in R2, which establishes an isomorphic mapping between curved facial surface and 2D planar domain. With such conformal embedding, two planar representations of 3D shapes are proposed: Gaussian curvature image (GCI) for a facial surface, and surface displacement image (SDI) for a pair of facial surfaces. The conformal planar representation reduces the data complexity from 3D irregular curved surface to 2D regular grid while preserving the necessary information for hallucination. Then, hallucinating a low resolution facial shape is formalized as inference of SDI from GCIs by modeling the relationship between GCI and SDI by RBF regression. The experiments on USF HumanID 3D face database demonstrate the effectiveness of the approach. Our method can be easily extended to hallucinate those category-specific 3D surfaces sharing with similar geometric structures. <s> BIB025 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Faces captured by surveillance cameras are often of very low resolution. This significantly deteriorates face recognition performance. Super-resolution techniques have been proposed in the past to mitigate this. This paper proposes the novel use of a Locality Preserving Projections (LPP) algorithm called Direct Locality Preserving Projections (DLPP) for super resolution of facial images, or ldquoface hallucinationrdquo in other words. Because DLPP doesnpsilat require any dimensionality reduction preprocessing via Principle Component Analysis (PCA), it retains more discriminating power in its feature space than LPP. Combined with non-parametric regression using a generalized regression neural network (GRNN), the proposed work can render high-resolution face image from an image of resolution as low as 8x7 with a large zoom factor of 24. The resulting technique is powerful and efficient in synthesizing faces similar to ground-truth faces. Simulation results show superior results compared to other well-known schemes. <s> BIB026 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Generally face images may be visualized as points drawn on a low-dimensional manifold embedded in high-dimensional ambient space. Many dimensionality reduction techniques have been used to learn this manifold. Orthogonal locality preserving projection (OLPP) is one among them which aims to discover the local structure of the manifold and produces orthogonal basis functions. In this paper, we present a two-step patch based algorithm for face superresolution. In first step a MAP based framework is used to obtain high resolution patch from its low resolution counterpart where the face subspace is learnt using OLPP. To enhance the quality of the image further, we propose a method which uses kernel ridge regression to learn the relation between low and high resolution residual patches. Experimental results show that our approach can reconstruct high quality face images. <s> BIB027 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> The size of training set as well as the usage thereof is an important issue of learning-based super-resolution. In this paper, we presented an adaptive learning method for face hallucination using locality preserving projections (LPP). By virtue of the ability to reveal the non-linear structure hidden in the high-dimensional image space, LPP is an efficient manifold learning method to analyze the local intrinsic features on the manifold of local facial areas. By searching out patches online in the LPP sub-space, which makes the resultant training set tailored to the testing patch, our algorithm performed the adaptive sample selection and then effectively restored the lost high-frequency components of the low-resolution face image by patch-based eigen transformation using the dynamic training set. Finally, experiments fully demonstrated that the proposed method can achieve good performance of super-resolution reconstruction by utilizing a relative small sample. <s> BIB028 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In facial image analysis, image resolution is an important factor which has great influence on the performance of face recognition systems. As for low-resolution face recognition problem, traditional methods usually carry out super-resolution firstly before passing the super-resolved image to a face recognition system. In this paper, we propose a new method which predicts high-resolution images and the corresponding features simultaneously. More specifically, we propose ldquofeature hallucinationrdquo to project facial images with low-resolution into an expected feature space. As a result, the proposed method does not require super-resolution as an explicit preprocessing step. In addition, we explore a constrained hallucination that considers the local consistency in the image grid. In our method, we use the index of local visual primitives as features and a block-based histogram distance to measure the similarity for the face recognition. Experimental results on FERET face database verify that the proposed method can improve both visual quality and recognition rate for low-resolution facial images. <s> BIB029 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> The facial change caused by aging progression might significantly degrade the performance of a face recognition system. One major way to deal with this problem is to predict the aging process. The work presented in this paper proposed a framework to simulate the face aging process by means of super-resolution. Considered the nature of multi- modalities in face image set, multi-linear algebra is introduced into the super-resolution method to represent and process the whole image set in tensor space. The simulating results represented in the paper are compared with the ground truth face image of the same people. <s> BIB030 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Existing learning-based face super-resolution (hallucination) techniques generate high-resolution images of a single facial modality (i.e., at a fixed expression, pose and illumination) given one or set of low-resolution face images as probe. Here, we present a generalized approach based on a hierarchical tensor (multilinear) space representation for hallucinating high-resolution face images across multiple modalities, achieving generalization to variations in expression and pose. In particular, we formulate a unified tensor which can be reduced to two parts: a global image-based tensor for modeling the mappings among different facial modalities, and a local patch-based multiresolution tensor for incorporating high-resolution image details. For realistic hallucination of unregistered low-resolution faces contained in raw images, we develop an automatic face alignment algorithm capable of pixel-wise alignment by iteratively warping the probing face to its projection in the space of training face images. Our experiments show not only performance superiority over existing benchmark face super-resolution techniques on single modal face hallucination, but also novelty of our approach in coping with multimodal hallucination and its robustness in automatic alignment under practical imaging conditions. <s> BIB031 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we address the problem of hallucinating a high resolution face given a low resolution input face. The problem is approached through sparse coding. To exploit the facial structure, non-negative matrix factorization (NMF) is first employed to learn a localized part-based subspace. This subspace is effective for super-resolving the incoming low resolution face under reconstruction constraints. To further enhance the detailed facial information, we propose a local patch method based on sparse representation with respect to coupled overcomplete patch dictionaries, which can be fast solved through linear programming. Experiments demonstrate that our approach can hallucinate high quality super-resolution faces. <s> BIB032 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper addresses the problem of generating a super-resolution (SR) image from a single low-resolution input image. We approach this problem from the perspective of compressed sensing. The low-resolution image is viewed as downsampled version of a high-resolution image, whose patches are assumed to have a sparse representation with respect to an over-complete dictionary of prototype signal-atoms. The principle of compressed sensing ensures that under mild conditions, the sparse representation can be correctly recovered from the downsampled signal. We will demonstrate the effectiveness of sparsity as a prior for regularizing the otherwise ill-posed super-resolution problem. We further show that a small set of randomly chosen raw patches from training images of similar statistical nature to the input image generally serve as a good dictionary, in the sense that the computed representation is sparse and the recovered high-resolution image is competitive or even superior in quality to images produced by other SR methods. <s> BIB033 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Assuming that the local geometry of low-resolution image patches is similar to that of the high-resolution counterparts, neighbor embedding based super-resolution methods learn a high-resolution image from one or more low-resolution input images by embedding its patches optimally with training ones. However, their performance suffers from inappropriate choices of features, neighborhood sizes and training patches. To address the issues, we propose an extended Neighbor embedding based super-resolution through edge detection and Feature Selection (henceforth NeedFS). Three major contributions of NeedFS are: (1) A new combination of features are proposed, which preserve edges and smoothen color regions better; (2) the training patches are learned discriminately with different neighborhood sizes based on edge detection; (3) only those edge training patches are bootstrapped to provide extra useful information with least redundancy. Experiments show that NeedFS performs better in both quantitative and qualitative evaluation. NeedFS is also robust even with a very limited training set and thus is promising for real applications. <s> BIB034 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Many learning-based super-resolution methods are based on the manifold assumption, which claims that point-pairs from the low-resolution representation manifold (LRM) and the corresponding high-resolution representation manifold (HRM) possess similar local geometry. However, the manifold assumption does not hold well on the original coupled manifolds (i.e., LRM and HRM) due to the nonisometric one-to-multiple mappings from low-resolution (LR) image patches to high-resolution (HR) ones. To overcome this limitation, we propose a solution from the perspective of manifold alignment. In this context, we perform alignment by learning two explicit mappings which project the point-pairs from the original coupled manifolds into the embeddings of the common manifold (CM). For the task of SR reconstruction, we treat HRM as target manifold and employ the manifold regularization to guarantee that the local geometry of CM is more consistent with that of HRM than LRM is. After alignment, we carry out the SR reconstruction based on neighbor embedding between the new couple of the CM and the target HRM. Besides, we extend our method by aligning the multiple coupled subsets instead of the whole coupled manifolds to address the issue of the global nonlinearity. Experimental results on face image super-resolution verify the effectiveness of our method. <s> BIB035 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we revisit the manifold assumption which has been widely adopted in the learning-based image super-resolution. The assumption states that point-pairs from the high-resolution manifold share the local geometry with the corresponding low-resolution manifold. However, the assumption does not hold always, since the one-to-multiple mapping from LR to HR makes neighbor reconstruction ambiguous and results in blurring and artifacts. To minimize the ambiguous, we utilize Locality Preserving Constraints (LPC) to avoid confusions through emphasizing the consistency of localities on both manifolds explicitly. The LPC are combined with a MAP framework, and realized by building a set of cell-pairs on the coupled manifolds. Finally, we propose an energy minimization algorithm for the MAP with LPC which can reconstruct high quality images compared with previous methods. Experimental results show the effectiveness of our method. <s> BIB036 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> We proposed a compressed sensing Super Resolution algorithm based on wavelet. The proposed algorithm performs well with a smaller quantity of training image patches and outputs images with satisfactory subjective quality. It is tested on classical images commonly adopted by Super Resolution researchers with both generic and specialized training sets for comparison with other popular commercial software and state-of-the-art methods. Experiments demonstrate that, the proposed algorithm is competitive among contemporary Super Resolution methods. <s> BIB037 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper proposes a new algorithm to generate a super-resolution image from a single, low-resolution input without the use of a training data set. We do this by exploiting the fact that the image is highly compressible in the wavelet domain and leverage recent results of compressed sensing (CS) theory to make an accurate estimate of the original high-resolution image. Unfortunately, traditional CS approaches do not allow direct use of a wavelet compression basis because of the coherency between the point-samples from the downsampling process and the wavelet basis. To overcome this problem, we incorporate the downsampling low-pass filter into our measurement matrix, which decreases coherency between the bases. To invert the downsampling process, we use the appropriate inverse filter and solve for the high-resolution image using a greedy, matching-pursuit algorithm. The result is a simple and efficient algorithm that can generate high quality, high-resolution images without the use of training data. We present results that show the improved performance of our method over existing super-resolution approaches. <s> BIB038 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Practical face recognition systems are sometimes confronted with low-resolution face images. Traditional two-step methods solve this problem through employing super-resolution (SR). However, these methods usually have limited performance because the target of SR is not absolutely consistent with that of face recognition. Moreover, time-consuming sophisticated SR algorithms are not suitable for real-time applications. To avoid these limitations, we propose a novel approach for LR face recognition without any SR preprocessing. Our method based on coupled mappings (CMs), projects the face images with different resolutions into a unified feature space which favors the task of classification. These CMs are learned through optimizing the objective function to minimize the difference between the correspondences (i.e., low-resolution image and its high-resolution counterpart). Inspired by locality preserving methods for dimensionality reduction, we introduce a penalty weighting matrix into our objective function. Our method significantly improves the recognition performance. Finally, we conduct experiments on publicly available databases to verify the efficacy of our algorithm. <s> BIB039 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Low resolution faces are the main barrier to efficient face recognition and identification in several problems primarily surveillance systems. To mitigate this problem we proposes a novel learning based two-step approach by the use of Direct Locality Preserving Projections (DLPP), Maximum a posterior estimation (MAP) and Kernel Ridge Regression (KRR) for super-resolution of face images or in other words Face Hallucination. First using DLPP for manifold learning and MAP estimation, a smooth Global high resolution image is obtained. In second step to introduce high frequency components KRR is used to model the Residue high resolution image, which is then added to Global image to get final high quality detail featured Hallucinated face image. As shown in experimental results the proposed system is robust and efficient in synthesizing low resolution faces similar to the original high resolution faces. <s> BIB040 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In video surveillance, the resolution of face image is very important for Police Criminal Investigation. A novel method for super-resolution of face images is proposed in this paper. The method use LPP(Locality Preserving Projection) to discover the hidden nonlinear structure in high-dimensional data space, and construct the feature subspace of face image by analyze its internal manifold. We divided the input low resolution face image into patches and select the nearest patches for each patch in the subspace. Then use the selected samples restore the high-resolution face images in the absence of high frequency components. Experimental results demonstrate that the proposed method outperforms the traditional schemes significantly both in subjective and objective quality. <s> BIB041 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Example-based image super-resolution techniques model the co-occurrence patterns between the middle and high frequency layers of example images to estimate the missing high frequency component for low resolution input. However, many existing approaches seek to estimate the optimal solution within a small set of candidates by using empirical criteria. Hence their representational performance is limited by the quality of the candidate set, and the generated super-resolution image is unstable, with noticeable artifacts. In this paper, we propose a novel image super-resolution method based on learning the sparse association between input image patches and the example image patches. We improve an existing sparse-coding algorithm to find sparse association between image patches. We also propose an iterative training strategy to learn a redundancy reduced basis set to speed up the super-resolution process. Comparing to existing example-based approaches, the proposed method significantly improves image quality, and the produced super-resolution images are sharp and natural, with no obvious artifact. <s> BIB042 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Learning-based approaches for super-resolution (SR) have been studied in the past few years. In this paper, a novel single-image SR framework based on the learning of sparse image representation with support vector regression (SVR) is presented. SVR is known to offer excellent generalization ability in predicting output class labels for input data. Given a low resolution image, we approach the SR problem as the estimation of pixel labels in its high resolution version. The feature considered in this work is the sparse representation of different types of image patches. Prior studies have shown that this feature is robust to noise and occlusions present in image data. Experimental results show that our method is quantitatively more effective than prior work using bicubic interpolation or SVR methods, and our computation time is significantly less than that of existing SVR-based methods due to the use of sparse image representations. <s> BIB043 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper presents a new approach to single-image superresolution, based upon sparse signal representation. Research on image statistics suggests that image patches can be well-represented as a sparse linear combination of elements from an appropriately chosen over-complete dictionary. Inspired by this observation, we seek a sparse representation for each patch of the low-resolution input, and then use the coefficients of this representation to generate the high-resolution output. Theoretical results from compressed sensing suggest that under mild conditions, the sparse representation can be correctly recovered from the downsampled signals. By jointly training two dictionaries for the low- and high-resolution image patches, we can enforce the similarity of sparse representations between the low-resolution and high-resolution image patch pair with respect to their own dictionaries. Therefore, the sparse representation of a low-resolution image patch can be applied with the high-resolution image patch dictionary to generate a high-resolution image patch. The learned dictionary pair is a more compact representation of the patch pairs, compared to previous approaches, which simply sample a large amount of image patch pairs , reducing the computational cost substantially. The effectiveness of such a sparsity prior is demonstrated for both general image super-resolution (SR) and the special case of face hallucination. In both cases, our algorithm generates high-resolution images that are competitive or even superior in quality to images produced by other similar SR methods. In addition, the local sparse modeling of our approach is naturally robust to noise, and therefore the proposed algorithm can handle SR with noisy inputs in a more unified framework. <s> BIB044 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> As a powerful statistical image modeling technique, sparse representation has been successfully used in various image restoration applications. The success of sparse representation owes to the development of the l1-norm optimization techniques and the fact that natural images are intrinsically sparse in some domains. The image restoration quality largely depends on whether the employed sparse domain can represent well the underlying image. Considering that the contents can vary significantly across different images or different patches in a single image, we propose to learn various sets of bases from a precollected dataset of example image patches, and then, for a given patch to be processed, one set of bases are adaptively selected to characterize the local sparse domain. We further introduce two adaptive regularization terms into the sparse representation framework. First, a set of autoregressive (AR) models are learned from the dataset of example image patches. The best fitted AR models to a given patch are adaptively selected to regularize the image local structures. Second, the image nonlocal self-similarity is introduced as another regularization term. In addition, the sparsity regularization parameter is adaptively estimated for better image restoration performance. Extensive experiments on image deblurring and super-resolution validate that by using adaptive sparse domain selection and adaptive regularization, the proposed method achieves much better results than many state-of-the-art algorithms in terms of both PSNR and visual perception. <s> BIB045 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper proposes a two-step prototype-face-based scheme of hallucinating the high-resolution detail of a low-resolution input face image. The proposed scheme is mainly composed of two steps: the global estimation step and the local facial-parts refinement step. In the global estimation step, the initial high-resolution face image is hallucinated via a linear combination of the global prototype faces with a coefficient vector. Instead of estimating coefficient vector in the high-dimensional raw image domain, we propose a maximum a posteriori (MAP) estimator to estimate the optimum set of coefficients in the low-dimensional coefficient domain. In the local refinement step, the facial parts (i.e., eyes, nose and mouth) are further refined using a basis selection method based on overcomplete nonnegative matrix factorization (ONMF). Experimental results demonstrate that the proposed method can achieve significant subjective and objective improvement over state-of-the-art face hallucination methods, especially when an input face does not belong to a person in the training data set. <s> BIB046 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper proposes a new algorithm to perform single-frame image super-resolution (SR) of vehicle license plate (VLP) using soft learning prior. Conventional single-frame SR/interpolation methods such as bi-cubic interpolation often experience over-smoothing near the edges and textured regions. Therefore, learning-based methods have been proposed to handle these shortcomings by incorporating a learning term so that the reconstructed high-resolution images can be guided towards these models. However, existing learning-based methods employ a binary hard-decision approach to determine whether the prior models are fully relevant or totally irrelevant. This approach, however, is inconsistent with many practical applications as the degree of relevance for the prior models may vary. In view of this, this paper proposes a new framework that adopts a soft learning approach in license plate super-resolution. The method integrates image SR with optical character recognition (OCR) to perform VLP SR. The importance of the prior models is estimated through relevance scores obtained from the OCR. These are then incorporated as a soft learning term into a new regularized cost function. Experimental results show that the proposed method is effective in handling license plate SR in both simulated and real experiments. <s> BIB047 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper deals with the super-resolution (SR) problem based on a single low-resolution (LR) image. Inspired by the local tangent space alignment algorithm in [16] for nonlinear dimensionality reduction of manifolds, we propose a novel patch-learning method using locally affine patch mapping (LAPM) to solve the SR problem. This approach maps the patch manifold of low-resolution image to the patch manifold of the corresponding high-resolution (HR) image. This patch mapping is learned by a training set of pairs of LR/HR images, utilizing the affine equivalence between the local low-dimensional coordinates of the two manifolds. The latent HR image of the input (an LR image) is estimated by the HR patches which are generated by the proposed patch mapping on the LR patches of the input. We also give a simple analysis of the reconstruction errors of the algorithm LAPM. Furthermore we propose a global refinement technique to improve the estimated HR image. Numerical results are given to show the efficiency of our proposed methods by comparing these methods with other existing algorithms. <s> BIB048 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Sparse representation provides a new method of generating a super-resolution image from a single low resolution input image. An over-complete base for sparse representation is an essential part of such methods. However, discovering the over-complete base with efficient representation from a large amount of image patches is a difficult problem. In this paper, we propose a super-resolution construction based on multi-space sparse representation to efficiently solve the problem. In the representation, image patches are decomposed into a structure component and a texture component represented by the over-complete bases of their own spaces so that their high-level features can be captured by the bases. In the implementation, a prior knowledge about low resolution images generation is combined to the typical base construction for high construction quality. Experiment results demonstrate that the proposed method significantly improves the PSNR, SSIM and visual quality of reconstructed high-resolution image. <s> BIB049 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In video surveillance, the face recognition usually aims at recognizing a nonfrontal low resolution face image from the gallery in which each person has only one high resolution frontal face image. Traditional face recognition approaches have several challenges, such as the difference of image resolution, pose variation and only one gallery image per person. This letter presents a regression based method that can successfully recognize the identity given all these difficulties. The nonlinear regression models from the specific nonfrontal low resolution image to frontal high resolution features are learnt by radial basis function in subspace built by canonical correlation analysis. Extensive experiments on benchmark database show the superiority of our method. <s> BIB050 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> This paper describes a novel method for single-image super-resolution (SR) based on a neighbor embedding technique which uses Semi-Nonnegative Matrix Factorization (SNMF). Each low-resolution (LR) input patch is approximated by a linear combination of nearest neighbors taken from a dictionary. This dictionary stores low-resolution and corresponding high-resolution (HR) patches taken from natural images and is thus used to infer the HR details of the super-resolved image. The entire neighbor embedding procedure is carried out in a feature space. Features which are either the gradient values of the pixels or the mean-subtracted luminance values are extracted from the LR input patches, and from the LR and HR patches stored in the dictionary. The algorithm thus searches for the K nearest neighbors of the feature vector of the LR input patch and then computes the weights for approximating the input feature vector. The use of SNMF for computing the weights of the linear approximation is shown to have a more stable behavior than the use of LLE and lead to significantly higher PSNR values for the super-resolved images. <s> BIB051 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a novel patch-based face-hallucination algorithm, which is based on the local structure kernels established via the relation between interpolated low-resolution (LR) images and their corresponding high-resolution counterparts. In our algorithm, the local linear embedding (LLE) algorithm is used to extract local structures, and the kernels are then constructed based on non-overlapped patches in the interpolated LR images. The information about local structures as described by the kernels is propagated to the corresponding regions of the HR images. Sub-pixel distortions are refined by solving a constrained problem at pixel level via iterative procedures. Experimental results show that our proposed method can provide a good performance in terms of reconstruction errors and visual quality. <s> BIB052 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> We explore in this paper efficient algorithmic solutions to single image super-resolution (SR). We propose the GESR, namely Graph Embedding Super-Resolution, to super-resolve a high-resolution (HR) image from a single low-resolution (LR) observation. The basic idea of GESR is to learn a projection matrix mapping the LR image patch to the HR image patch space while preserving the intrinsic geometrical structure of original HR image patch manifold. While GESR resembles other manifold learning-based SR methods in persevering the local geometric structure of HR and LR image patch manifold, the innovation of GESR lies in that it preserves the intrinsic geometrical structure of original HR image patch manifold rather than LR image patch manifold, which may be contaminated because of image degeneration (e.g., blurring, down-sampling and noise). Experiments on benchmark test images show that GESR can achieve very competitive performance as Neighbor Embedding based SR (NESR) and Sparse representation based SR (SSR). Beyond subjective and objective evaluation, all experiments show that GESR is much faster than both NESR and SSR. <s> BIB053 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, a new two-step method is proposed to infer a high-quality and high-resolution (HR) face image from a low-quality and low-resolution (LR) observation based on training samples in the database. First, a global face image is reconstructed based on the non-linear relationship between LR and HR face images, which is established according to radial basis function and partial least squares (RBF-PLS) regression. Based on the reconstructed global face patches manifold (formed by the image patches at the same position of all global face images), whose local geometry is more consistent with that of original HR face patches manifold than noisy LR one is, the Neighbor Embedding is applied to induce the target HR face image by preserving the similar local geometry between global face patches manifold and the original HR face patches manifold. A comparison of some state-of-the-art methods shows the superiority of our method, and experiments also demonstrate the effectiveness both under simulation and real conditions. <s> BIB054 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a new two-step face hallucination method to induce a high-resolution (HR) face image from a low-resolution (LR) observation. Especially for low-quality surveillance face image, an RBF-PLS based variable selection method is presented for the reconstruction of global face image. Further more, in order to compensate for the reconstruction errors, which are lost high frequency detailed face features, the Neighbor Embedding (NE) based residue face hallucination algorithm is used. Compared with current methods, the proposed RBF-PLS based method can generate a global face more similar to the original face and less sensitive to noise, moreover, the NE algorithm can reduce the reconstruction errors caused by misalignment on the basis of a carefully designed search strategy. Experiments show the superiority of the proposed method compared with some state-of-the-art approaches and the efficacy both in simulation and real surveillance condition. <s> BIB055 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Until now, neighbor-embedding-based (NE) algorithms for super-resolution (SR) have carried out two independent processes to synthesize high-resolution (HR) image patches. In the first process, neighbor search is performed using the Euclidean distance metric, and in the second process, the optimal weights are determined by solving a constrained least squares problem. However, the separate processes are not optimal. In this paper, we propose a sparse neighbor selection scheme for SR reconstruction. We first predetermine a larger number of neighbors as potential candidates and develop an extended Robust-SL0 algorithm to simultaneously find the neighbors and to solve the reconstruction weights. Recognizing that the k-nearest neighbor (k-NN) for reconstruction should have similar local geometric structures based on clustering, we employ a local statistical feature, namely histograms of oriented gradients (HoG) of low-resolution (LR) image patches, to perform such clustering. By conveying local structural information of HoG in the synthesis stage, the k-NN of each LR input patch is adaptively chosen from their associated subset, which significantly improves the speed of synthesizing the HR image while preserving the quality of reconstruction. Experimental results suggest that the proposed method can achieve competitive SR quality compared with other state-of-the-art baselines. <s> BIB056 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Instead of using probabilistic graph based or manifold learning based models, some approaches based on position-patch have been proposed for face hallucination recently. In order to obtain the optimal weights for face hallucination, they represent image patches through those patches at the same position of training face images by employing least square estimation or convex optimization. However, they can hope neither to provide unbiased solutions nor to satisfy locality conditions, thus the obtained patch representation is not the best. In this paper, a simpler but more effective representation scheme- Locality-constrained Representation (LcR) has been developed, compared with the Least Square Representation (LSR) and Sparse Representation (SR). It imposes a locality constraint onto the least square inversion problem to reach sparsity and locality simultaneously. Experimental results demonstrate the superiority of the proposed method over some state-of-the-art face hallucination approaches. <s> BIB057 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Recently, compressive sensing (CS) has emerged as a powerful tool for solving a class of inverse/underdetermined problems in computer vision and image processing. In this paper, we investigate the application of CS paradigms on single image super-resolution (SR) problems that are considered to be the most challenging in this class. In light of recent promising results, we propose novel tools for analyzing sparse representation-based inverse problems using redundant dictionary basis. Further, we provide novel results establishing tighter correspondence between SR and CS. As such, we gain insights into questions concerning regularizing the solution to the underdetermined problem, such as follows. 1) Is sparsity prior alone sufficient? 2) What is a good dictionary? 3) What is the practical implication of noncompliance with theoretical CS hypothesis? Unlike in other underdetermined problems that assume random down-projections, the low-resolution image formation model employed in CS-based SR is a deterministic down-projection that may not necessarily satisfy some critical assumptions of CS. We further investigate the impact of such projections in concern to the above questions. <s> BIB058 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> The fast development of time-of-flight (ToF) cameras in recent years enables capture of high frame-rate 3D depth maps of moving objects. However, the resolution of depth map captured by ToF is rather limited, and thus it cannot be directly used to build a high quality 3D model. In order to handle this problem, we propose a novel joint example-based depth map super-resolution method, which converts a low resolution depth map to a high resolution depth map, using a registered high resolution color image as a reference. Different from previous depth map SR methods without training stage, we learn a mapping function from a set of training samples and enhance the resolution of the depth map via sparse coding algorithm. We further use a reconstruction constraint to make object edges sharper. Experimental results show that our method outperforms state-of-the-art methods for depth map super-resolution. <s> BIB059 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> The choice of the over-complete dictionary that sparsely represents data is of prime importance for sparse coding-based image super-resolution. Sparse coding is a typical unsupervised learning method to generate an over-complete dictionary. However, most of the sparse coding methods for image super-resolution fail to simultaneously consider the geometrical structure of the dictionary and corresponding coefficients, which may result in noticeable super-resolution reconstruction artifacts. In this paper, a novel sparse coding method is proposed to preserve the geometrical structure of the dictionary and the sparse coefficients of the data. Moreover, the proposed method can preserve the incoherence of dictionary entries, which is critical for sparse representation. Inspired by the development on non-local self-similarity and manifold learning, the proposed sparse coding method can provide the sparse coefficients and learned dictionary from a new perspective, which have both reconstruction and discrimination properties to enhance the learning performance. Extensive experimental results on image super-resolution have demonstrated the effectiveness of the proposed method. <s> BIB060 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> We introduce a two-step face hallucination frame work as one of classifying among sparse residual compensation model. In the first step, the optimal coefficients of the interpolated training images are used to construct a global face image. In the second step, a class of priors is computed based on mixing a set of linear priors related to dissimilar priors. The blocks of coefficients are considered to find the sparse mixing weights. In order to find the best improved information of the face image in the residual compensation of step-two, a sparse signal representation is considered over coefficients in a frame. Finally, we obtain a hallucinated face image by integrating these two steps. The extensive experiments on publicly available database show the effectiveness of the framework. <s> BIB061 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> The paper presents the resolution-invariant image representation (@?IIR) framework. It applies sparse-coding with multi-resolution codebook to learn resolution-invariant sparse representations of local patches. An input image can be reconstructed to higher resolution at not only discrete integer scales, as that in many existing super-resolution works, but also continuous scales, which functions similar to 2-D image interpolation. The @?IIR framework includes the methods of building a multi-resolution bases set from training images, learning the optimal sparse resolution-invariant representation of an image, and reconstructing the missing high-frequency information at continuous resolution level. Both theoretical and experimental validations of the resolution invariance property are presented in the paper. Objective comparison and subjective evaluation show that the @?IIR framework based image resolution enhancement method outperforms existing methods in various aspects. <s> BIB062 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Recently, single image super-resolution reconstruction (SISR) via sparse coding has attracted increasing interest. In this paper, we proposed a multiple-geometric-dictionaries-based clustered sparse coding scheme for SISR. Firstly, a large number of high-resolution (HR) image patches are randomly extracted from a set of example training images and clustered into several groups of “geometric patches,” from which the corresponding “geometric dictionaries” are learned to further sparsely code each local patch in a low-resolution image. A clustering aggregation is performed on the HR patches recovered by different dictionaries, followed by a subsequent patch aggregation to estimate the HR image. Considering that there are often many repetitive image structures in an image, we add a self-similarity constraint on the recovered image in patch aggregation to reveal new features and details. Finally, the HR residual image is estimated by the proposed recovery method and compensated to better preserve the subtle details of the images. Some experiments test the proposed method on natural images, and the results show that the proposed method outperforms its counterparts in both visual fidelity and numerical measures. <s> BIB063 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> In this paper, we propose a novel coupled dictionary training method for single-image super-resolution (SR) based on patchwise sparse recovery, where the learned couple dictionaries relate the low- and high-resolution (HR) image patch spaces via sparse representation. The learning process enforces that the sparse representation of a low-resolution (LR) image patch in terms of the LR dictionary can well reconstruct its underlying HR image patch with the dictionary in the high-resolution image patch space. We model the learning problem as a bilevel optimization problem, where the optimization includes an l1-norm minimization problem in its constraints. Implicit differentiation is employed to calculate the desired gradient for stochastic gradient descent. We demonstrate that our coupled dictionary learning method can outperform the existing joint dictionary training method both quantitatively and qualitatively. Furthermore, for real applications, we speed up the algorithm approximately 10 times by learning a neural network model for fast sparse inference and selectively processing only those visually salient regions. Extensive experimental comparisons with state-of-the-art SR algorithms validate the effectiveness of our proposed approach. <s> BIB064 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Reconstruction- and example-based super-resolution (SR) methods are promising for restoring a high-resolution (HR) image from low-resolution (LR) image(s). Under large magnification, reconstruction-based methods usually fail to hallucinate visual details while example-based methods sometimes introduce unexpected details. Given a generic LR image, to reconstruct a photo-realistic SR image and to suppress artifacts in the reconstructed SR image, we introduce a multi-scale dictionary to a novel SR method that simultaneously integrates local and non-local priors. The local prior suppresses artifacts by using steering kernel regression to predict the target pixel from a small local area. The non-local prior enriches visual details by taking a weighted average of a large neighborhood as an estimate of the target pixel. Essentially, these two priors are complementary to each other. Experimental results demonstrate that the proposed method can produce high quality SR recovery both quantitatively and perceptually. <s> BIB065 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> 4D-CT plays an important role in lung cancer treatment. However, due to the inherent high-dose exposure associated with CT, dense sampling along superior-inferior direction is often not practical. As a result, artifacts such as lung vessel discontinuity and partial volume are typical in 4D-CT images and might mislead dose administration in radiation therapy. In this paper, we present a novel patch-based technique for super-resolution enhancement of the 4D-CT images along the superior-inferior direction. Our working premise is that the anatomical information that is missing at one particular phase can be recovered from other phases. Based on this assumption, we employ a patch-based mechanism for guided reconstruction of super-resolution axial slices. Specifically, to reconstruct each targeted super-resolution slice for a CT image at a particular phase, we agglomerate a dictionary of patches from images of all other phases in the 4D-CT sequence. Then we perform a sparse combination of the patches in this dictionary to reconstruct details of a super-resolution patch, under constraint of similarity to the corresponding patches in the neighboring slices. By iterating this procedure over all possible patch locations, a superresolution 4D-CT image sequence with enhanced anatomical details can be eventually reconstructed. Our method was extensively evaluated using a public dataset. In all experiments, our method outperforms the conventional linear and cubic-spline interpolation methods in terms of preserving image details and suppressing misleading artifacts. <s> BIB066 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Based on the assumption about the sparse representation of natural images and the theory of compressed sensing, very promising results about single-image super-resolution were obtained by an excellent algorithm introduced by Yang et al. [45]. However, their success could not be well explained theoretically. The lack of theoretical insight has hindered the further improvement of the algorithm. In this paper, Yang's algorithm is revisited in the view of learning theory. According to this point, Yang's algorithm can be considered as a linear regression method in a special feature space which is named as sparse coding space by us. In fact, it has been shown that Yang's algorithm is a result of optimal linear estimation in sparse coding space. More importantly, our theoretical analysis suggests that Yang's algorithm can be improved by using more flexible regression methods than the linear regression method. Following the idea, a novel single-image super-resolution algorithm which is designed based on the framework of L"2-Boosting is proposed in the paper. The experimental results show the effectiveness of the proposed algorithm by comparing with other methods, which verify our theoretical analysis about Yang's algorithm. <s> BIB067 </s> Super-resolution: a comprehensive survey <s> Neural networks <s> Given multiple source images of the same scene, image fusion integrates the inherent complementary information into one single image, and thus provides a more complete and accurate description. However, when the source images are of low-resolution, the resultant fused image can still be of low-quality, hindering further image analysis. To improve the resolution, a separate image super-resolution step can be performed. In this paper, we propose a novel framework for simultaneous image fusion and super-resolution. It is based on the use of sparse representations, and consists of three steps. First, the low-resolution source images are interpolated and decomposed into high- and low-frequency components. Sparse coefficients from these components are then computed and fused by using image fusion rules. Finally, the fused sparse coefficients are used to reconstruct a high-resolution fused image. Experiments on various types of source images (including magnetic resonance images, X-ray computed tomography images, visible images, infrared images, and remote sensing images) demonstrate the superiority of the proposed method both quantitatively and qualitatively. <s> BIB068
The same concept of belief-network based methods, but using different types of neural networks (NN), has been employed in many different SR algorithms. Examples of such networks are linear associative memories with single [61] and dual associative learning BIB006 , Hopfield NN BIB001 BIB019 , probabilistic NN BIB007 BIB020 , integrated recurrent NN BIB003 , multi-layer perceptron BIB008 BIB025 BIB021 BIB047 , feed forward NN, BIB015 BIB015 , and RBF BIB022 BIB050 . Manifold Manifold-based methods BIB004 BIB009 BIB010 BIB023 BIB024 BIB022 BIB026 BIB027 BIB028 BIB034 BIB035 BIB036 BIB039 BIB040 BIB048 BIB051 BIB052 BIB053 BIB054 BIB055 assume that the HR and LR images form manifolds with similar local geometries in two distinct feature spaces BIB027 . Similar to PCA, these methods are also usually used for dimensionality reduction. These methods generally consist of the following steps: -Generate HR and LR manifolds in the training step using some HR images and their corresponding LR counterparts. -In the testing step, first divide the input LR test image into a set of LR patches. For each LR patch of this image, l x : -Find its k-nearest neighbor patches, l j , on the LR manifold. -Use these k-nearest neighbors to calculate the weights, w L x , for reconstructing the patch l x : arg min where the summation of the weights is equal to one. -Use the involved weights and neighbors to find the corresponding objects in the HR manifold to reconstruct the HR patch of l x : where N L (x) includes the neighbors of l x , and the h j s are the HR patches associated to the LR neighbor patches of l x . Though most of the manifold-based methods use this manifold assumption that the reconstruction HR weights are similar to the LR weights, i.e.: w L x j = w H x j , it is shown in BIB035 BIB036 470 ] that this might not be completely true and therefore it is suggested to align the manifolds before the actual reconstruction. To do so, in BIB035 BIB036 BIB039 , a so-called common manifold has been used, which helps the manifold alignment by learning two explicit mappings which map the paired manifolds into the embeddings of the common manifold. Unlike traditional dimensionality reduction techniques, such as PCA and LDA, manifold-based methods can handle non-Euclidean structures. Reported manifold-based methods for SR have mostly been applied to face hallucination and can be classified into three groups: -Locally linear embedding (LLE) BIB004 BIB009 BIB010 BIB029 BIB051 BIB052 598 ]. -Locality preserving projections (LPP) BIB023 BIB024 BIB022 BIB026 BIB028 BIB041 BIB040 . -Orthogonal locality preserving projections (OLPP) BIB027 . The main difference between LPP and LLE is that LPP can handle cases which have not been seen during training better than LLE BIB026 . OLPP can produce orthogonal basis functions and therefore can provide better locality preserving than LPP. As opposed to the majority of the previously mentioned learning-based SR systems, in which for generating each HR image patch only one nearest LR patch and its corresponding HR patch in the training database are used, in manifoldbased methods BIB004 BIB009 BIB010 , multiple nearest neighbor LR patches are used simultaneously. Manifold-based methods are usually applied in two steps. In the first step, they are combined with a MAP method BIB023 BIB024 BIB040 BIB053 BIB054 BIB055 or a Markov-based learning method BIB009 like those in [65, 66,   BIB005  to apply a global constraint over the super-resolved image. In the second step, they use a different technique like kernel ridge regression BIB027 BIB040 , graph embedding BIB053 , radial basis function and partial least squares (RBF-PLS) regression BIB054 BIB055 to apply local constraints to the super-resolved image by finding the transformation between low and HR residual patches. In these algorithms, the sizes of the patches, their amount of overlap, the number of the patches involved (the nearest LR patches), and the features employed for representing the patches (e.g., first-and second-order gradients BIB004 , edges BIB034 ) are of great importance. If the number of patches are too large, the neighborhood embedding analysis gets difficult. If this number is too small, the global structure of the given data space cannot be captured BIB023 BIB024 BIB034 . Tensor Tensors are a higher-order generalization of vectors (first order) and matrices (second order) BIB030 . Tensor analysis can be considered as a generalized extension of traditional linear methods such as PCA for which the mappings between multiple factor spaces are studied BIB031 . It provides a means of decomposing the entire available data into multimodal spaces and then studies the mappings between these spaces BIB002 BIB011 BIB012 BIB006 BIB013 BIB016 BIB017 BIB018 BIB031 BIB030  . In BIB011 the super-resolved reconstructed image is computed by an ML identity parameter vector in an HR tensor space, which is a space that has been obtained by applying tensor decomposition to the HR training images. In BIB012 it is shown that the method of BIB011 can improve the recognition rate of a face recognition system. In BIB013 BIB014 , a patchbased tensor analysis is applied to a given LR input image to hallucinate a face image. To do so, the K 1 nearest images of the input image are first found in the training database. Then, the input image is divided into overlapping patches and for each patch the K 2 nearest patches among the previously found K 1 training images are found. The patches are then weighted in a way that their fusion generates an HR image which in case of down-sampling possesses the minimum distance from the down-sampled versions of the HR images involved. This patch-based approach enforces local consistency between the patches in the hallucinated image. In BIB031 , a tensor-based hallucination has been generalized to different facial expressions and poses. Compressive sensing Though the sparsity of images had been implicitly utilized in many SR works, recent advances in sparse signal processing have introduced the possibility of recovering linear relationships between HR signals and their LR projections, and have motivated many researchers to explicitly use the sparsity of the images for SR algorithms. In sparse coding it is assumed that there is an overcomplete dictionary like D ∈ R n×K which contains K prototype image patches. It is assumed that an HR image f ∈ R n can be represented as a liner combination of these patches: f = Dα where α is a very sparse coefficient vector, i.e., ||α|| 0 << K . These algorithms BIB032 BIB033 BIB037  BIB042 BIB043 BIB044 BIB045 BIB056 BIB057 BIB049 BIB058 BIB059 BIB060 BIB061 BIB067 BIB062 BIB063 BIB064 BIB068 BIB065 , BIB066 usually assume that there are two overcomplete dictionaries: one for the HR patches, D h , and one for their corresponding LR counterparts, D l . The latter has been produced from the former by a degradation process like that in Eq. (4) . Usually the mean of the pixel values of each patch is subtracted from its pixels to make the dictionaries more texture representative than intensity. Given an input LR image, g, the above-mentioned discussion for sparse coding is used to find a set of coefficients,α, which minimizes: extraction operator (usually a high-pass filter BIB032 BIB033 BIB044 BIB063 ), P extracts the overlapping area between the current target patch and the previously reconstructed image, and β is a control parameter for finding a tradeoff between matching the LR input and finding an HR patch which is compatible with its neighbors. Having found such an α minimizing Eq. (57), the following equation is used to obtain an estimate for the reconstructed HR image: The reconstructedf from Eq. (58) has been mostly used as an a priori term in combination with other SR methods. For example, in the following works, it has been combined with: -a MAP method BIB032 BIB037 BIB046 BIB045 BIB056 BIB061 BIB067 BIB062 BIB063 BIB064 BIB068 , -an IBP method BIB033 BIB044 , -a support vector regression BIB043 , and -a Wavelet-based method BIB038 . CS algorithms have been successfully used for SR along with wavelet-based methods. However, due to coherency between the LR images resulting from the down-sampling process and the wavelet basis, CS algorithms have problems with being directly employed in the wavelet domain. To reduce this coherency, BIB038 applies a low-pass filter to the LR inputs before using the CS technique in the wavelet domain. It is discussed in BIB060 that preserving the local topological structure in the data space results in better reconstruction results. While trying to do so, the incoherency of the dictionary entries should be taken into account. To do this: - [424] applies a low-pass filter to the LR inputs before using the CS technique in the wavelet domain, and - BIB060 uses the non-local self-similarity concept of manifold-based learning methods.