Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> We consider the problem of reconstructing remotely obtained images from image-plane detector arrays. Although the individual detectors may be larger than the blur spot of the imaging optics, high-resolution reconstructions can be obtained by scanning or rotating the image with respect to the detector. As an alternative to matrix inversion or least-squares estimation [Appl. Opt. 26, 3615 (1987)], the method of convex projections is proposed. We show that readily obtained prior knowledge can be used to obtain good-quality imagery with reduced data. The effect of noise on the reconstruction process is considered. <s> BIB001 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> We propose a novel image restoration and super-resolution algorithm based on a convex set theoretic approach. This procedure differs from the currently popular super-resolution algorithms, which typically are derived by employing statistical approaches, in offering a greater flexibility for incorporating scene-related prior information into the restoration processing. Novel methods for modeling and for extrapolation of scene information for efficient use in the restoration process are described. These include a border extraction method that permits a background-foreground separation of the image, a template-based information modeling and a method of extrapolating the image frequency content by a Taylor series expansion. A class of iterative projection algorithms for processing diffraction-limited images by using convex sets modeling various types of a priori information about the object or scene imaged is developed. The restoration and super-resolution performance of these algorithms are established through a variety of simulation experiments. <s> BIB002 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> In this paper we present a technique that may be applied to surveillance video data to obtain a higher-quality image from a sequence of lower-quality images. The increase in quality is derived through a deconvolution of optical blur and/or an increase in spatial sampling. To process sequences of real forensic video data, three main steps are required: frame and region selection, displacement estimation, and original image estimation. A user-identified region-of-interest (ROI) is compared to other frames in the sequence. The areas that are suitable matches are identified and used for displacement estimation. The calculated displacement vector images describe the transformation of the desired high-quality image to the observed low quality images. The final stage is based on the Projection Onto Convex Sets (POCS) super-resolution approach of Patti, Sezan, and Tekalp. This stage performs a deconvolution using the observed image sequence, displacement vectors, and an a priori known blur model. A description of the algorithmic steps are provided, and an example input sequence with corresponding output image is given. <s> BIB003 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> Signal and image enhancement is considered in the context of a new type of diffusion process that simultaneously enhances, sharpens, and denoises images. The nonlinear diffusion coefficient is locally adjusted according to image features such as edges, textures, and moments. As such, it can switch the diffusion process from a forward to a backward (inverse) mode according to a given set of criteria. This results in a forward-and-backward (FAB) adaptive diffusion process that enhances features while locally denoising smoother segments of the signal or image. The proposed method, using the FAB process, is applied in a super-resolution scheme. The FAB method is further generalized for color processing via the Beltrami flow, by adaptively modifying the structure tensor that controls the nonlinear diffusion process. The proposed structure tensor is neither positive definite nor negative, and switches between these states according to image features. This results in a forward-and-backward diffusion flow where different regions of the image are either forward or backward diffused according to the local geometry within a neighborhood. <s> BIB004 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> In many surveillance video applications, it is of interest to recognize an object or a person, which occupies a small portion of a low-resolution, noisy video. This paper addresses the problem of super-resolution recovery of a region of interest from more than one low-resolution view of a scene recorded by multiple cameras. The multiple camera scenario alleviates the difficulty in registration of multiple frames of video that contain non-rigid or multiple object motion in the single camera case. With proper temporal registration of multiple videos, arbitrary scene motion can be handled. The success of super-resolution recovery from multiple views in real applications vitally depends on two factors: i) the accuracy of multiple view registration results, and ii) the accuracy of the camera and data acquisition model. We propose a system, which consists of a method for sub-pixel accurate spatio-temporal alignment of multiple video sequences for view registration and the projections onto convex sets method for super-resolution recovery. Experiments were implemented using two commercial analog video cameras, which do not perform on-board compression. Experimental results show that the super resolution recovery of dynamic scenes can be achieved as long as the multiple views of the scene can be registered with sub-pixel accuracy. <s> BIB005 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> Super-Resolution reconstruction produces one or a set of high-resolution images from a sequence of low-resolution frames. This article reviews a variety of Super-Resolution methods proposed in the last 20 years, and provides some insight into, and a summary of, our recent contributions to the general Super-Resolution problem. In the process, a detailed study of several very important aspects of Super-Resolution, often ignored in the literature, is presented. Spe- cifically, we discuss robustness, treatment of color, and dynamic operation modes. Novel methods for addressing these issues are accompanied by experimental results on simulated and real data. Finally, some future challenges in Super-Resolution are outlined and discussed. © 2004 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 14, 47-57, 2004; Published online in Wiley InterScience (www.interscience.wiley. com). DOI 10.1002/ima.20007 <s> BIB006 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> Hyperspectral images are used for aerial and space imagery applications, including target detection, tracking, agricultural, and natural resource exploration. Unfortunately, atmospheric scattering, secondary illumination, changing viewing angles, and sensor noise degrade the quality of these images. Improving their resolution has a high payoff, but applying super-resolution techniques separately to every spectral band is problematic for two main reasons. First, the number of spectral bands can be in the hundreds, which increases the computational load excessively. Second, considering the bands separately does not make use of the information that is present across them. Furthermore, separate band super resolution does not make use of the inherent low dimensionality of the spectral data, which can effectively be used to improve the robustness against noise. In this paper, we introduce a novel super-resolution method for hyperspectral images. An integral part of our work is to model the hyperspectral image acquisition process. We propose a model that enables us to represent the hyperspectral observations from different wavelengths as weighted linear combinations of a small number of basis image planes. Then, a method for applying super resolution to hyperspectral images using this model is presented. The method fuses information from multiple observations and spectral bands to improve spatial resolution and reconstruct the spectrum of the observed scene as a combination of a small number of spectral basis functions. <s> BIB007 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> Super-resolution reconstruction is the process of reconstructing a high-resolution image from multiple low-resolution images. Most super-resolution reconstruction methods neglect camera response function, exposure time, white balancing, and external illumination changes. In this paper, we show how to extend traditional super-resolution reconstruction methods to handle these factors. We provide formulations for several different super-resolution reconstruction approaches. Experimental results are also included. <s> BIB008 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> An efficient scheme is presented to the color super-resolution problem for recovery of a color high-resolution image with knowledge of multiple Bayer filtered low-resolution images. To recover a visually natural high-resolution image, we restrict fairly smooth initial candidates to all images satisfying all bounds imposed on the several nonsmooth convex color total variations as well as a non-smooth convex inter cross correlation measure among color channels. In the proposed scheme, the data-fidelity is optimized in a systematic way, over all initial candidates, with the hybrid steepest descent method for quasi-nonexpansive mappings [Yamada & Ogura 2004], by minimizing successively an weighted average of pure mean square errors between the low-resolution transforms of the high-resolution estimate and the multiple low-resolution images. Numerical examples show that the proposed scheme recovers visually natural high resolution images by resolving the tradeoff between noise suppression and edge preservation of the recovered image while keeping fair inter channel cross correlation among color channels. <s> BIB009 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> In minimally invasive surgery, a small field-of-view is often required for achieving a large magnification factor during micro-scale tasks such as coronary anastomosis. Constant change of the orientation and focal length of the laparoscope camera, however, is cumbersome and can impose extra visual and cognitive load to the operating surgeon in realigning the visual pathways and anatomical landmarks. The purpose of this paper is to investigate the use of fixational movements for robotic assisted minimal invasive surgery such that the perceived resolution of the foveal field-of-view is greater than the intrinsic resolution of the laparoscope camera. The proposed technique is based on super resolution imaging using projection onto convex sets. Validation with both phantom and in vivo data from totally endoscopic coronary artery bypass surgery is provided. <s> BIB010 </s> Super-resolution: a comprehensive survey <s> Projection onto convex sets <s> The MRI image reconstruction supported by Toeplitz-based iterative super-resolution algorithm is presented in the paper. It is shown that the approach improves MRI spatial resolution in cases when multi-blade sequences are used. The PROPELLER MRI method collects data from rectangular regions (blades) rotated around the origin of the k-space. Inter-blade patient motion is the premise for the use of super-resolution technique. Images obtained from sets of irregularly located frequency domain samples are combined into the high resolution MRI image. The super-resolution reconstruction replaces usually applied direct averaging of low-resolution images. Furthermore high order affine motion model has been applied. <s> BIB011
Another group of iterative methods is those based on the concept of POCS BIB001 38, 42, 43, 49, 53, BIB002 BIB003  BIB004 BIB005 BIB006 BIB007 BIB008 BIB009 BIB010 BIB011 . These algorithms define an implicit cost function for solving the SR problem BIB006 . Considering the imaging model given in Eq. (4), in the POCS method it is assumed that each LR image imposes an a priori knowledge on the final solution. It is assumed that this a priori knowledge is a closed convex set, S k , which is defined as [38] : where g k is the kth LR image, f is the solution, and δ l and -Channel by channel total variation, luminance total variation, inter-channel cross correlation sets, boundedness, and non-negativity constraints for color images BIB009 . It has been discussed in  that since projections onto sets defined at edge locations cause a ringing effect, it is better to reduce the amount of deblurring at the edges using blur functions at the direction of the edge gradient more like an impulse. Furthermore, in BIB008 it is discussed that defining the constraints on individual pixel values rather than on the whole image leads to a simpler implementation.