Previous CNN-based video super-resolution approaches need to align multiple frames to the reference. In this paper, we show that proper frame alignment and motion compensation is crucial for achieving high quality results. We accordingly propose a sub-pixel motion compensation' (SPMC) layer in a CNN framework. Analysis and experiments show the suitability of this layer in video SR. The final end-to-end, scalable CNN framework effectively incorporates the SPMC layer and fuses multiple frames to reveal image details. Our implementation can generate visually and quantitatively high-quality results, superior to current state-of-the-arts, without the need of parameter tuning.
For deep video SR, Liao al @cite_3 adopted a separate step to construct high-resolution SR-drafts, which are obtained under different flow parameters. Kappeler al @cite_10 estimated optical flow and selected corresponding patches across frames to train a CNN. In both methods, motion estimation is separated from training. Recently, Caballero al @cite_8 proposed the first end-to-end video SR framework, which incorporates motion compensation as a submodule.
@cite_10 @cite_3 @cite_8
2589026418 2210480155 2951845494 2750704532
Video superresolution (SR) techniques are of essential usages for high-resolution display devices due to the current lack of high-resolution videos. Although many algorithms have been proposed, video SR still remains a very challenging inverse problem under different conditions. In this paper, we propose a new method for video SR named motion compensation and residual net (MCResNet). We use optical flow algorithm for motion estimation and motion compensation as a preprocessing step. Then, we employ a novel deep residual convolutional neural network (CNN) to predict a high-resolution image using multiple motion compensated observations. The new residual CNN model preserves the low-frequency contents and facilitates the restoration of high-frequency details. Our method is able to handle large and complex motions adaptively. Extensive experimental results validate that our proposed method outperforms state-of-the-art single-image-based and multi-frame-based algorithms for video SR quantitatively and qualitatively. We propose a new direction for fast video super-resolution (VideoSR) via a SR draft ensemble, which is defined as the set of high-resolution patch candidates before final image deconvolution. Our method contains two main components -- i.e., SR draft ensemble generation and its optimal reconstruction. The first component is to renovate traditional feedforward reconstruction pipeline and greatly enhance its ability to compute different super resolution results considering large motion variation and possible errors arising in this process. Then we combine SR drafts through the nonlinear process in a deep convolutional neural network (CNN). We analyze why this framework is proposed and explain its unique advantages compared to previous iterative methods to update different modules in passes. Promising experimental results are shown on natural video sequences. This paper shows how to extract dense optical flow from videos with a convolutional neural network (CNN). The proposed model constitutes a potential building block for deeper architectures to allow using motion without resorting to an external algorithm, for recognition in videos. We derive our network architecture from signal processing principles to provide desired invariances to image contrast, phase and texture. We constrain weights within the network to enforce strict rotation invariance and substantially reduce the number of parameters to learn. We demonstrate end-to-end training on only 8 sequences of the Middlebury dataset, orders of magnitude less than competing CNN-based motion estimation methods, and obtain comparable performance to classical methods on the Middlebury benchmark. Importantly, our method outputs a distributed representation of motion that allows representing multiple, transparent motions, and dynamic textures. Our contributions on network design and rotation invariance offer insights nonspecific to motion estimation. Convolutional neural networks (CNNs) have recently been applied to the optical flow estimation problem. As training the CNNs requires sufficiently large ground truth training data, existing approaches resort to synthetic, unrealistic datasets. On the other hand, unsupervised methods are capable of leveraging real-world videos for training where the ground truth flow fields are not available. These methods, however, rely on the fundamental assumptions of brightness constancy and spatial smoothness priors which do not hold near motion boundaries. In this paper, we propose to exploit unlabeled videos for semi-supervised learning of optical flow with a Generative Adversarial Network. Our key insight is that the adversarial loss can capture the structural patterns of flow warp errors without making explicit assumptions. Extensive experiments on benchmark datasets demonstrate that the proposed semi-supervised algorithm performs favorably against purely supervised and semi-supervised learning schemes.