Variational Bayesian Blind Image Deconvolution: A review <s> E. Deconvolution in mobile devices <s> Abstract In this paper image restoration applications, where multiple distorted versions of the same original image are available, are considered. A general adaptive restoration algorithm is derived on the basis of a set theoretic regularization technique. The adaptivity of the algorithm is introduced in two ways: (a) by a constraint operator which incorporates properties of the response of the human visual system into the restoration process and (b) by a weight matrix which assigns greater importance for the deconvolution process to areas of high spatial activity than to areas of low spatial activity. Different degrees of trust are assigned to the various distorted images depending on the amounts of noise. The proposed algorithm is general and can be used for any type of linear distortion and constraint operators. It can also be used to restore signals other than images. Experimental results obtained by an iterative implementation of the proposed algorithms are presented. <s> BIB001 </s> Variational Bayesian Blind Image Deconvolution: A review <s> E. Deconvolution in mobile devices <s> Long-exposure handheld photography is degraded with blur, which is difficult to remove without prior information about the camera motion. In this work, we utilize inertial sensors (accelerometers and gyroscopes) in modern smartphones to detect exact motion trajectory of the smartphone camera during exposure and remove blur from the resulting photography based on the recorded motion data. The whole system is implemented on the Android platform and embedded in the smartphone device, resulting in a close-to-real-time deblurring algorithm. The performance of the proposed system is demonstrated in real-life scenarios. <s> BIB002 </s> Variational Bayesian Blind Image Deconvolution: A review <s> E. Deconvolution in mobile devices <s> This paper presents a new Metropolis-adjusted Langevin algorithm (MALA) that uses convex analysis to simulate efficiently from high-dimensional densities that are log-concave, a class of probability distributions that is widely used in modern high-dimensional statistics and data analysis. The method is based on a new first-order approximation for Langevin diffusions that exploits log-concavity to construct Markov chains with favourable convergence properties. This approximation is closely related to Moreau-Yoshida regularisations for convex functions and uses proximity mappings instead of gradient mappings to approximate the continuous-time process. The proposed method complements existing MALA methods in two ways. First, the method is shown to have very robust stability properties and to converge geometrically for many target densities for which other MALA are not geometric, or only if the step size is sufficiently small. Second, the method can be applied to high-dimensional target densities that are not continuously differentiable, a class of distributions that is increasingly used in image processing and machine learning and that is beyond the scope of existing MALA and HMC algorithms. To use this method it is necessary to compute or to approximate efficiently the proximity mappings of the logarithm of the target density. For several popular models, including many Bayesian models used in modern signal and image processing and machine learning, this can be achieved with convex optimisation algorithms and with approximations based on proximal splitting techniques, which can be implemented in parallel. The proposed method is demonstrated on two challenging high-dimensional and non-differentiable models related to image resolution enhancement and low-rank matrix estimation that are not well addressed by existing MCMC methodology. <s> BIB003 </s> Variational Bayesian Blind Image Deconvolution: A review <s> E. Deconvolution in mobile devices <s> Smartphones are now widely used as photographic devices. Equipped with cheap cameras they are prone to many degradations, most notably handshake in combination with rolling shutter causes severe space-variant blur. Removing blur without any information about the camera motion is a computationally demanding and unstable process. We use built-in gyroscopes to record the motion trajectory of the camera during exposure and then remove blur from the acquired photograph based on the reconstructed trajectory. The proposed deblurring application is implemented on Android smartphones with close-to-real-time performance. <s> BIB004 </s> Variational Bayesian Blind Image Deconvolution: A review <s> E. Deconvolution in mobile devices <s> Hand-held mobile phone photography usually suffers from motion blur. Without information about the camera motion, it is difficult for traditional methods to remove such blur. We study sensor-assisted single image deblurring on modern smartphones. Information about the camera motion is obtained from the smartphones' built-in sensors such as ac-celerometers, gyroscopes, and magnetometers. Addressing the characteristics of mobile photo capture, we propose a camera model to estimate the point spread function (PSF) based on the 3D camera orientation computed from the fused sensor data. A convenient framework is developed for automatically calibrating the camera and sensors using a series of consecutively captured photos, without requiring any extra device. In addition, to accommodate users' sensitivity in artifacts over human faces, a face-adaptive deblurring framework is proposed. Features measuring image quality are computed over the automatically detected face regions, based on which the appropriate deblurring algorithm is selected. We evaluate our approach over 500 consumer photos featuring various levels of blur and illumination, where our algorithm shows clear advantages compared with state-of-the-art traditional blind and non-blind deconvolution methods in both de-blurring quality and speed. <s> BIB005 </s> Variational Bayesian Blind Image Deconvolution: A review <s> E. Deconvolution in mobile devices <s> Image deblurring to remove blur caused by camera shake has been intensively studied. Nevertheless, most methods are brittle and computationally expensive. In this paper we analyze multi-image approaches, which capture and combine multiple frames in order to make deblurring more robust and tractable. In particular, we compare the performance of two approaches: align-and-average and multi-image deconvolution. Our deconvolution is non-blind, using a blur model obtained from real camera motion as measured by a gyroscope. We show that in most situations such deconvolution outperforms align-and-average. We also show, perhaps surprisingly, that deconvolution does not benefit from increasing exposure time beyond a certain threshold. To demonstrate the effectiveness and efficiency of our method, we apply it to still-resolution imagery of natural scenes captured using a mobile camera with flexible camera control and an attached gyroscope. <s> BIB006
The ubiquity of mobile devices, such as smartphones and tablets, and the not-so-high quality of their cameras make the restoration of images taken with those devices a succulent market. Running the deconvolution process on mobile devices is, nevertheless, difficult given their limited computational power. Some commercial applications that claim to remove blur from images are available for the different platforms (see DeblurIt Pro or Photo Fix de Blur for Android or Photo Doctor for iOS). However they seem to deal only with out-of-focus blur with a manually selected radius and implement simple deconvolution algorithms. Smartphones and tablets are more than simple cameras. They usually have other built-in sensors to capture the device position and trajectory and the capability of processing images. Hence, some methods are being proposed to perform deconvolution on those devices. In an effort to take advantage of the sensors present on the mobile phones,Šindelář andŠroubek BIB002 used the information provided by the gyroscope to keep track of the device motion while taking the picture and, hence, obtain an estimation of the blur by rendering the camera trajectory on the image plane. This blur estimate is used to deconvolve the image by a simple Wiener filter. An extension considering spatially variant blur and rolling shutter compensation is presented in BIB004 . A similar approach was used in BIB005 where the blur is obtained from a combination of the kernel estimated from the fusion of gyroscope, magnetometer and accelerometer measurements and a Gaussian kernel with small variance to take into account the out-of-focus blur due to the motion of the camera from the finger movement on pressing on the screen. Additionally, to minimize artifacts on faces, a face detection algorithm is applied to the image and an SVM classifier is trained and used to select between deblurring followed by denoising or sharpening the image, depending on the face characteristics. Using a developer tablet modified by attaching a USB connected external gyroscope, a multi-image deconvolution which captures and combines multiple frames in order to make deblurring more robust and tractable is proposed in BIB006 . Blur is obtained from the gyroscope data and multi-image deconvolution is performed by minimizing where λ is a regularization parameter and ∆ is the gradient operator. The authors conclude that this deconvolution procedure outperforms, in most situations, the align-and-average strategy, that is, averaging multiple noisy images captured using a short exposure time, and hence blur-free, aligned using the gyroscope data. The optimization problem in Eq. BIB003 was first utilized in the work by Katsaggelos BIB001 .