Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> This paper describes a Bayesian method for constructing a super-resolved surface model by combining information from a set of images of the given surface. We develop the theory and algorithms in detail for the 2-D reconstruction problem, appropriate for the case where all images are taken from roughly the same direction and under similar lighting conditions. We show the results of this 2-D reconstruction on Viking Martian data. These results show dramatic improvements in both spatial and gray-scale resolution. The Bayesian approach uses a neighbor correlation model as well as pixel data from the image set. Some extensions of this method are discussed, including 3-D surface reconstruction and the resolution of diffraction blurred images. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> Using a set of low resolution images it is possible to reconstruct high resolution information by merging low resolution data on a finer grid. A 3D super-resolution algorithm is proposed, based on a probabilistic interpretation of the n-dimensional version of Papoulis' (1977) generalized sampling theorem. The algorithm is devised for recovering the albedo and the height map of a Lambertian surface in a Bayesian framework, using Markov random fields for modeling the a priori knowledge. <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> While Bayesian methods can significantly improve the quality of tomographic reconstructions, they require the solution of large iterative optimization problems. Recent results indicate that the convergence of these optimization problems can be improved by using sequential pixel updates, or Gauss-Seidel iterations. However, Gauss-Seidel iterations may be perceived as less useful when parallel computing architectures are use. We show that for degrees of parallelism of typical practical interest, the Gauss-Seidel iterations updates may be computed in parallel with little loss in convergence speed. In this case, the theoretical speed up of parallel implementations is nearly linear with the number of processors. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> We present a maximum a posteriori solution to the problem of obtaining a high-resolution image from a set of degraded low-resolution images of the same scene. The proposed algorithm has the advantage that no prior knowledge of blurring functions is required and it can handle unknown misregistrations between the input images. An efficient implementation scheme of alternating minimizations is presented together with experiments that demonstrate the performance of the algorithm. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> This paper presents a new technique for generating a high resolution image from a blurred image sequence; this is also referred to as super-resolution restoration of images. The image sequence consists of decimated, blurred and noisy versions of the high resolution image. The high resolution image is modeled as a Markov random field (MRF) and a maximum a posteriori (MAP) estimation technique is used. A simple gradient descent method is used to optimize the functional. Further, line fields are introduced in the cost function and optimization using Graduated Non-Convexity (GNC) is shown to yield improved results. Lastly, we present results of optimization using Simulated Annealing (SA). <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> In this chapter, we present a parametric method for generating a super-resolution image from a sequence consisting of blurred and noisy observations. The high resolution image is modeled as a Markov random field (MRF) and a maximum a posteriori (MAP) estimation technique is used for super-resolution restoration. Unlike other super-resolution imaging methods, the proposed technique does not require sub-pixel registration of given observations. A simple gradient descent method is used to optimize the cost. The discontinuities in the intensity process can be preserved by introducing suitable line processes. Superiority of this technique to standard methods of image interpolation is illustrated. The motivation for using blur as a cue is also explained. <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> 1 Introduction.- 1.1 Background.- 1.2 Modelling assumptions.- 1.3 Applications.- 1.4 Principal contributions.- 2 Literature Survey.- 2.1 Image registration.- 2.1.1 Registration by a geometric transformation.- 2.1.2 Ensuring global consistency.- 2.1.3 Other parametric surfaces.- 2.2 Image mosaicing.- 2.3 Super-resolution.- 2.3.1 Simple super-resolution schemes.- 2.3.2 Methods using a generative model.- 2.3.3 Super-resolution using statistical prior image models.- 3 Registration: Geometric and Photometric.- 3.1 Introduction.- 3.2 Imaging geometry.- 3.3 Estimating homographies.- 3.3.1 Linear estimators.- 3.3.2 Non-linear refinement.- 3.3.3 The maximum likelihood estimator of H.- 3.4 A practical two-view method.- 3.5 Assessing the accuracy of registration.- 3.5.1 Assessment criteria.- 3.5.2 Obtaining a ground-truth homography.- 3.6 Feature-based vs. direct methods.- 3.7 Photometric registration.- 3.7.1 Sources of photometric difference.- 3.7.2 The photometric model.- 3.7.3 Estimating the parameters.- 3.7.4 Results.- 3.8 Application: Recovering latent marks in forensic images.- 3.8.1 Motivation.- 3.8.2 Method.- 3.8.3 Further examples.- 3.9 Summary.- 4 Image Mosaicing.- 4.1 Introduction.- 4.2 Basic method.- 4.2.1 Outline.- 4.2.2 Practical considerations.- 4.3 Rendering from the mosaic.- 4.3.1 The reprojection manifold.- 4.3.2 The blending function.- 4.3.3 Eliminating seams by photometric registration.- 4.3.4 Eliminating seams due to vignetting.- 4.3.5 A fast alternative to median filtering.- 4.4 Simultaneous registration of multiple views.- 4.4.1 Motivation.- 4.4.2 Extending the two-view framework to N-views.- 4.4.3 A novel algorithm for feature-matching over N-views.- 4.4.4 Results.- 4.5 Automating the choice of reprojection frame.- 4.5.1 Motivation.- 4.5.2 Synthetic camera rotations.- 4.6 Applications of image mosaicing.- 4.7 Mosaicing non-planar surfaces.- 4.8 Mosaicing "user's guide".- 4.9 Summary.- 4.9.1 Further examples.- 5 Super-resolution: Maximum Likelihood and Related Approaches.- 5.1 Introduction.- 5.2 What do we mean by "resolution"?.- 5.3 Single-image methods.- 5.4 The multi-view imaging model.- 5.4.1 A note on the assumptions made in the model.- 5.4.2 Discretization of the imaging model.- 5.4.3 Related approaches.- 5.4.4 Computing the elements in Mn.- 5.4.5 Boundary conditions.- 5.5 Justification for the Gaussian PSF.- 5.6 Synthetic test images.- 5.7 The average image.- 5.7.1 Noise robustness.- 5.8 Rudin's forward-projection method.- 5.9 The maximum-likelihood estimator.- 5.10 Predicting the behaviour of the ML estimator.- 5.11 Sensitivity of the ML estimator to noise sources.- 5.11.1 Observation noise.- 5.11.2 Poorly estimated PSF.- 5.11.3 Inaccurate registration parameters.- 5.12 Irani and Peleg's method.- 5.12.1 Least-squares minimization by steepest descent.- 5.12.2 Irani and Peleg's algorithm.- 5.12.3 Relationship to the ML estimator.- 5.12.4 Convergence properties.- 5.13 Gallery of results.- 5.14 Summary.- 6 Super-resolution Using Bayesian Priors.- 6.1 Introduction.- 6.2 The Bayesian framework.- 6.2.1 Markov random fields.- 6.2.2 Gibbs priors.- 6.2.3 Some common cases.- 6.3 The optimal Wiener filter as a MAP estimator.- 6.4 Generic image priors.- 6.5 Practical optimization.- 6.6 Sensitivity of the MAP estimators to noise sources.- 6.6.1 Exercising the prior models.- 6.6.2 Robustness to image noise.- 6.7 Hyper-parameter estimation by cross-validation.- 6.8 Gallery of results.- 6.9 Super-resolution "user's guide".- 6.10 Summary.- 7 Super-resolution Using Sub-space Models.- 7.1 Introduction.- 7.2 Bound constraints.- 7.3 Learning a face model using PCA.- 7.4 Super-resolution using the PCA model.- 7.4.1 An ML estimator (FS-ML).- 7.4.2 MAP estimators.- 7.5 The behaviour of the face model estimators.- 7.6 Examples using real images.- 7.7 Summary.- 8 Conclusions and Extensions.- 8.1 Summary.- 8.2 Extensions.- 8.2.1 Application to digital video.- 8.2.2 Model-based super-resolution.- 8.3 Final observations.- A Large-scale Linear and Non-linear Optimization.- References. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> This paper presents a prior model for text image super-resolution in the Bayesian framework. In contrast to generic image super-resolution task, super-resolution of text images can be benefited from strong prior knowledge of the image class: firstly, low-resolution images are assumed to be generated from a high-resolution image by a sort of degradation which can be grasped through example pairs of the original and the corresponding degradation; secondly, text images are composed of two homogeneous regions, text and background regions. These properties were represented in a Markov random field (MRF) framework. Experiments showed that our model is more appropriate to text image super-resolution than the other prior models. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> In this paper we present a new super resolution Bayesian method for pansharpening of multispectral images which: a) incorporates prior knowledge on the expected characteristics of the multispectral images, b) uses the sensor characteristics to model the observation process of both panchromatic and multispectral images, and c) performs the estimation of all the unknown parameters in the model. Using real data, the pan-sharpened multispectral images are compared with the images obtained by other parsharpening methods and their quality is assessed both qualitatively and quantitatively. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> Super-resolution reconstruction produces one or a set of high-resolution (HR) images from a sequence of low-resolution (LR) images. Due to translational registration, super-resolution reconstruction can apply only on the sequences that have simple translation motion. This paper proposed a novel super-resolution reconstruction that that can apply on real sequences or complex motion sequences. The proposed super-resolution reconstruction uses a high accuracy registration algorithm, the fast affine block-based registration [16], in the maximum likelihood framework. Moreover, the regularization is used to compensate the missing measurement information. The experimental results show that the proposed reconstruction can apply on real sequence such as Foreman and Suzie. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> An efficient algorithm using Maximum a posteriori- Markov random field (MAP-MRF) based approach for recovering a high-resolution image from multiple sub-pixel shifted low-resolution images is proposed. The algorithm can be used for super-resolution of both space-invariant and space-variant blurred images. We prove an important theorem that the posterior is also Markov and derive the exact posterior neighborhood structure in the presence of warping, blurring and down-sampling operations. The posterior being Markov enables us to perform all matrix operations as local image domain operations thereby resulting in a considerable speedup. Experimental results are given to demonstrate the effectiveness of our method. <s> BIB011 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> Super resolution image reconstruction allows the recovery of a high-resolution (HR) image from several low-resolution images that are noisy, blurred, and down sampled. In this paper, we present a joint formulation for a complex super-resolution problem in which the scenes contain multiple independently moving objects. This formulation is built upon the maximum a posteriori (MAP) framework, which judiciously combines motion estimation, segmentation, and super resolution together. A cyclic coordinate descent optimization procedure is used to solve the MAP formulation, in which the motion fields, segmentation fields, and HR images are found in an alternate manner given the two others, respectively. Specifically, the gradient-based methods are employed to solve the HR image and motion fields, and an iterated conditional mode optimization method to obtain the segmentation fields. The proposed algorithm has been tested using a synthetic image sequence, the "Mobile and Calendar" sequence, and the original "Motorcycle and Car" sequence. The experiment results and error analyses verify the efficacy of this algorithm <s> BIB012 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> We present a novel method of Bayesian image super-resolution in which marginalization is carried out over latent parameters such as geometric and photometric registration and the image point-spread function. Related Bayesian super-resolution approaches marginalize over the high-resolution image, necessitating the use of an unfavourable image prior, whereas our method allows for more realistic image prior distributions, and reduces the dimension of the integral considerably, removing the main computational bottleneck of algorithms such as Tipping and Bishop's Bayesian image super-resolution. We show results on real and synthetic datasets to illustrate the efficacy of our method. <s> BIB013 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> Super-resolution reconstruction (SRR) produces a high-resolution image from multiple low-resolution images. Many image SRR algorithms assume that the blurring process, i.e., point spread function (PSF) of the imaging system is known in advance. However, the blurring process is not known or is known only to within a set of parameters in many practical applications. In this letter, we propose an approach for solving the joint blur identification and image SRR based on the principle similar to the variable projection method. The approach can avoid some shortcomings of cyclic coordinate descent optimization procedure. We also propose an efficient implementation based on Lanczos algorithm and Gauss quadrature theory. Experimental results are presented to demonstrate the effectiveness of our method. <s> BIB014 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> In this paper we formulate a new time dependent convolutional model for super-resolution based on a constrained variational model that uses the total variation of the signal as a regularizing functional. We propose an iterative refinement procedure based on Bregman iteration to improve spatial resolution. The model uses a dataset of low resolution images and incorporates a downsampling operator to relate the high resolution scale to the low resolution one. We present an algorithm for the model and we perform a series of numerical experiments to show evidence of the good behavior of the numerical scheme and quality of the results. <s> BIB015 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> In many surveillance video applications, it is of interest to recognize a region of interest (ROI), which often occupies a small portion of a low-resolution, noisy video. This paper proposes an edge-preserving maximum a posteriori (MAP) estimation based super-resolution algorithm using a weighted directional Markov image prior model for a ROI from more than one low-resolution surveillance image. Conjugate gradient (CG) optimization based on standard operations on images is then developed to improve the computational efficiency of the algorithm. The proposed algorithm is tested on different series of surveillance images. The experimental results indicate that the proposed algorithm has considerable effectiveness in terms of both objective measurements and visual evaluation. <s> BIB016 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> In this letter, we present the first application of the Generalized Gaussian Markov Random Field (GGMRF) to the problem of video super-resolution. The GGMRF prior is employed to perform a maximum a posteriori (MAP) estimation of the desired high-resolution image. Compared with traditional prior models, the GGMRF can describe the distribution of the high-resolution image much better and can also preserve better the discontinuities (edges) of the original image. Previous work that used GGMRF for image restoration in which the temporal dependencies among video frames has not considered. Since the corresponding energy function is convex, gradient descent optimization techniques are used to solve the MAP estimation. Results show the super-resolved images using the GGMRF prior not only offers a good enhancement of visual quality, but also contain a significantly smaller amount of noise. <s> BIB017 </s> Super-resolution: a comprehensive survey <s> Markov random fields (MRF) <s> Total variation (TV) has been used as a popular and effective image prior model in regularization-based image processing fields, such as denoising, deblurring, super-resolution (SR), and others, because of its ability to preserve edges. However, as the TV model favors a piecewise constant solution, the processing results in the flat regions of the image being poor, and it cannot automatically balance the processing strength between different spatial property regions in the image. In this paper, we propose a spatially weighted TV image SR algorithm, in which the spatial information distributed in different image regions is added to constrain the SR process. A newly proposed and effective spatial information indicator called difference curvature is used to identify the spatial property of each pixel, and a weighted parameter determined by the difference curvature information is added to constrain the regularization strength of the TV regularization at each pixel. Meanwhile, a majorization-minimization algorithm is used to optimize the proposed spatially weighted TV SR model. Finally, a significant amount of simulated and real data experimental results show that the proposed spatially weighted TV SR algorithm not only efficiently reduces the “artifacts” produced with a TV model in fat regions of the image, but also preserves the edge information, and the reconstruction results are less sensitive to the regularization parameters than the TV model, because of the consideration of the spatial information constraint. <s> BIB018
The first a priori term used in the literature for SR along with a MAP approach was introduced in BIB001 (Table 8) , by which the values of the obtained pixels in the super-resolved image were updated according to the values of the neighboring pixels (4 × 4 or 8 × 8). In other words, it imposes a similarity on the structures of the neighbors. This similarity can be best modeled by MRF. Therefore, MRFs have been widely used in the SR literature for modeling a priori knowledge. However, for reasons of feasibility, MRFs are usually expressed by the equivalent Gibbs distribution as in Eq. (41). In this case the energy function of Eq. (41) can be written as: where V r is a function of a set of local points r , which are called cliques, R is the set of all of these cliques, and ρ r are potential functions defined over the pixels in each clique r . The first two sums are for meeting all the pixels and the last  where Q is a symmetric, positive definite matrix. In the simplest case Q = I (a minimal energy regularization which limits the total energy of the image) results in the a priori term || f || 2 assuming a zero mean and an i.i.d. Gaussian distribution for the pixel values. This results in the following Gibbs distribution for the a priori term BIB007 BIB005   BIB008 BIB009  BIB010 BIB004 BIB012 BIB014 : The Tikhonov regularization term does not allow discontinuities in the solution and therefore may not recover the edges properly BIB015 . To preserve sharp edges, which carry important information, a distribution should be used which penalizes them less severely BIB013  . To do so, Huber MRFs (HMRF) [34] are designed. The Gibbs distribution of a HMRF is defined as: where x here can be the first or the second derivative of the image and α is a parameter separating the quadratic and linear regions BIB011 . A HMRF a priori is an example of a convex but non-quadratic prior, which results in non-linear cost functions (Table 8) . If Q in Eq. (44) is non-diagonal, its off-diagonal elements model the spatial correlations between neighboring pixels, which results in a multi-variate Gaussian distribution over f . This a priori term is known as a Gaussian MRF (GMRF) BIB002 . In this case, Γ [of Eq. (43)] can be defined as a linear operator applied to f , like ||Γ f || 2 , which estimates the first and the second derivatives of f and imposes spatial smoothness (Table 8) . Therefore, it removes the pixels with highfrequency energies. This helps to remove noise, but at the same time smooths any sharp edges. Using the energy function of Eq. (43), it can be expressed as: in which ρ r is a quadratic function of the directional smooth measurements of d m i, j which for every pixel located at (i, j) are defined by BIB016 : It is discussed in BIB016 that the proper weighting of this directional smoothness measure can improve the performance of this regularization term. If the image gradients are modeled by generalized Gaussian distributions, a so-called generalized Gaussian MRF (GGMRF) BIB003 a priori term can be used, which has the following form: where 1 < p < 2. Similar to HMRF, GGMRF is also a convex non-quadratic a priori BIB003 BIB007  BIB017 . In BIB006  , this function has been defined as a quadratic cost function on pairwise cliques on a first-order neighborhood: Total variation (TV) If α in the Huber formulation of HMRF tends to zero, the Huber a priori term converts to a so-called TV norm that applies similar penalties for a smooth and a step edge and tends to preserve edges and avoid ringing effects (Table 8) . It is defined by: where ∇ is the gradient operator. In BIB018 the TV terms are weighted with an adaptive spatial algorithm based on differences in the curvature.  (Table 8) , which is used to approximate TV, is defined by: