Deep image prior (DIP), which utilizes a deep convolutional network (ConvNet) structure itself as an image prior, has attracted huge attentions in computer vision community.   It empirically shows the effectiveness of ConvNet structure for various image restoration applications.   However, why the DIP works so well is still unknown, and why convolution operation is essential for image reconstruction or enhancement is not very clear. In this study, we tackle these questions. The proposed approach is dividing the convolution into ``delay-embedding'' and ``transformation (\ie encoder-decoder)'', and proposing a simple, but essential, image/tensor modeling method which is closely related to dynamical systems and self-similarity. The proposed method named as manifold modeling in embedded space (MMES) is implemented by using a novel denoising-auto-encoder in combination with multi-way delay-embedding transform. In spite of its simplicity, the image/tensor completion and super-resolution results of MMES are quite similar even competitive to DIP in our extensive experiments, and these results would help us for reinterpreting/characterizing the DIP from a perspective of ``low-dimensional patch-manifold prior''. The most important piece of information for image/tensor restoration would be the "prior" which usually converts the optimization problems from ill-posed to well-posed, and/or gives some robustness for specific noises and outliers. Many priors were studied in computer science problems such as low-rank representation (Pearson, 1901; Hotelling, 1933; Hitchcock, 1927; Tucker, 1966) , smoothness (Grimson, 1981; Poggio et al., 1985; Li, 1994) , sparseness (Tibshirani, 1996) , non-negativity (Lee & Seung, 1999; Cichocki et al., 2009) , statistical independence (Hyvarinen et al., 2004) , and so on. Particularly in today's computer vision problems, total variation (TV) (Guichard & Malgouyres, 1998; Vogel & Oman, 1998) , low-rank representation (Liu et al., 2013; Ji et al., 2010; Zhao et al., 2015; Wang et al., 2017) , and non-local similarity (Buades et al., 2005; Dabov et al., 2007) priors are often used for image modeling. These priors can be obtained by analyzing basic properties of natural images, and categorized as "unsupervised image modeling". By contrast, the deep image prior (DIP) (Ulyanov et al., 2018) has been come from a part of "supervised" or "data-driven" image modeling framework (i.e., deep learning) although the DIP itself is one of the state-of-the-art unsupervised image restoration methods. The method of DIP can be simply explained to only optimize an untrained (i.e., randomly initialized) fully convolutional generator network (ConvNet) for minimizing squares loss between its generated image and an observed image (e.g., noisy image), and stop the optimization before the overfitting. Ulyanov et al. (2018) explained the reason why a high-capacity ConvNet can be used as a prior by the following statement: Network resists "bad" solutions and descends much more quickly towards naturally-looking images, and its phenomenon of "impedance of ConvNet" was confirmed by toy experiments. However, most researchers could not be fully convinced from only above explanation because it is just a part of whole. One of the essential questions is why is it ConvNet? or in more practical perspective, to explain what is "priors in DIP" with simple and clear words (like smoothness, sparseness, low-rank etc) is very important. In this study, we tackle the question why ConvNet is essential as an image prior, and try to translate the "deep image prior" with words. For this purpose, we divide the convolution operation into "embedding" and "transformation" (see Fig. 9 in Appendix). Here, the "embedding" stands for delay/shift-embedding (i.e., Hankelization) which is a copy/duplication operation of image-patches by sliding window of patch size (τ, τ ). The embedding/Hankelization is a preprocessing to capture the delay/shift-invariant feature (e.g., non-local similarity) of signals/images. This "transformation" is basically linear transformation in a simple convolution operation, and it also indicates some nonlinear transformation from the ConvNet perspective. To simplify the complicated "encoder-decoder" structure of ConvNet used in DIP, we consider the following network structure: Embedding H (linear), encoding φ r (non-linear), decoding ψ r (non-linear), and backward embedding H † (linear) (see Fig. 1 ). Note that its encoder-decoder part (φ r , ψ r ) is just a simple multi-layer perceptron along the filter domain (i.e., manifold learning), and it is sandwitched between forward and backward embedding (H, H † ). Hence, the proposed network can be characterized by Manifold Modeling in Embedded Space (MMES). The proposed MMES is designed as simple as possible while keeping a essential ConvNet structure. Some parameters τ and r in MMES are corresponded with a kernel size and a filter size in ConvNet. When we set the horizontal dimension of hidden tensor L with r, each τ 2 -dimensional fiber in H, which is a vectorization of each (τ, τ )-patch of an input image, is encoded into r-dimensional space. Note that the volume of hidden tensor L looks to be larger than that of input/output image, but representation ability of L is much lower than input/output image space since the first/last tensor (H,H ) must have Hankel structure (i.e., its representation ability is equivalent to image) and the hidden tensor L is reduced to lower dimensions from H. Here, we assume r < τ 2 , and its lowdimensionality indicates the existence of similar (τ, τ )-patches (i.e., self-similarity) in the image, and it would provide some "impedance" which passes self-similar patches and resist/ignore others. Each fiber of Hidden tensor L represents a coordinate on the patch-manifold of image. It should be noted that the MMES network is a special case of deep neural networks. In fact, the proposed MMES can be considered as a new kind of auto-encoder (AE) in which convolution operations have been replaced by Hankelization in pre-processing and post-processing. Compared with ConvNet, the forward and backward embedding operations can be implemented by convolution and transposed convolution with one-hot-filters (see Fig. 12 in Appendix for details). Note that the encoder-decoder part can be implemented by multiple convolution layers with kernel size (1,1) and non-linear activations. In our model, we do not use convolution explicitly but just do linear transform and non-linear activation for "filter-domain" (i.e., horizontal axis of tensors in Fig. 1 ). The contributions in this study can be summarized as follow: (1) A new and simple approach of image/tensor modeling is proposed which translates the ConvNet, (2) effectiveness of the proposed method and similarity to the DIP are demonstrated in experiments, and (3) most importantly, there is a prospect for interpreting/characterizing the DIP as "low-dimensional patch-manifold prior". A beautiful manifold representation of complicated signals in embedded space has been originally discovered in a study of dynamical system analysis (i.e., chaos analysis) for time-series signals (Packard et al., 1980) . After this, many signal processing and computer vision applications have been studied but most methods have considered only linear approximation because of the difficulty of non-linear modeling (Van Overschee & De Moor, 1991; Szummer & Picard, 1996; Li et al., 1997; Ding et al., 2007; Markovsky, 2008) . However nowadays, the study of non-linear/manifold modeling has been well progressed with deep learning, and it was successfully applied in this study. Interestingly, we could apply this non-linear system analysis not only for time-series signals but also natural color images and tensors (this is an extension from delay-embedding to multi-way shiftembedding). The best of our knowledge, this is the first study to apply Hankelization with AE into general tensor data reconstruction. MMES is a novel and simple image reconstruction model based on the low-dimensional patchmanifold prior which has many connections to ConvNet. We believe it helps us to understand how work ConvNet/DIP through MMES, and support to use DIP for various applications like tensor/image reconstruction or enhancement (Gong et al., 2018; Yokota et al., 2019; Van Veen et al., 2018; Gandelsman et al., 2019) . Finally, we established bridges between quite different research areas such as the dynamical system analysis, the deep learning, and the tensor modeling. The proposed method is just a prototype and can be further improved by incorporating other methods such as regularizations, multi-scale extensions, and adversarial training. We can see the anti-diagonal elements of above matrix are equivalent. Such matrix is called as "Hankel matrix". For a two-dimensional array we consider unfold of it and inverse folding by unfold , and The point here is that we scan matrix elements column-wise manner. Hankelization of this twodimensional array (matrix) with τ = [2, 2] is given by scanning a matrix with local (2,2)-window column-wise manner, and unfold and stack each local patch left-to-right. Thus, it is given as We can see that it is not a Hankel matrix. However, it is a "block Hankel matrix" in perspective of block matrix, a matrix that its elements are also matrices. We can see the block matrix itself is a Hankel matrix and all elements are Hankel matrices, too. Thus, Hankel matrix is a special case of block Hankel matrix in case of that all elements are scalar. In this paper, we say simply "Hankel structure" for block Hankel structure. Figure 9 shows an illustrative explanation of valid convolution which is decomposed into delayembedding/Hankelization and linear transformation. 1D valid convolution of f with kernel h = [h 1 , h 2 , h 3 ] can be provided by matrix-vector product of the Hankel matrix and h. In similar way, 2D valid convolution can be provided by matrix-vector product of the block Hankel matrix and unfolded kernel.
We propose a new auto-encoder incorporated with multiway delay-embedding transform toward interpreting deep image prior.
{"src": "scitldr-AIC_train", "paper_id": "SJgBra4YDS"}