To understand the inner work of deep neural networks and provide possible theoretical explanations, we study the deep representations through the untrained, random weight CNN-DCN architecture. As a convolutional AutoEncoder, CNN indicates the portion of a convolutional neural network from the input to an intermediate convolutional layer, and DCN indicates the corresponding deconvolutional portion. As compared with DCN training for pre-trained CNN, training the DCN for random-weight CNN converges more quickly and yields higher quality image reconstruction. Then, what happens for the overall random CNN-DCN? We gain intriguing results that the image can be reconstructed with good quality. To gain more insight on the intermediate random representation, we investigate the impact of network width versus depth, number of random channels, and size of random kernels on the reconstruction quality, and provide theoretical justifications on empirical observations. We further provide a fast style transfer application using the random weight CNN-DCN architecture to show the potential of our observation. Deep neural networks have achieved impressive performance on various machine learning tasks. However, our understanding of how these deep learning models operate remains limited. Providing a theoretical explanation or empirical interpretation for their success is an important research area. Existing works Arora et al. (2015; 2014) ; Paul & Venkatasubramanian (2014) propose mathematical models for learning architectures, however, the theoretical analysis of which fails to capture the state-of-the-art architectures. Gilbert et al. (2017) ; Chang et al. (2018) leverage either compressive sensing or ordinary differential equations to facilitate the understanding of CNNs. Ma et al. (2018) ; Hand & Voroninski (2017) deliver rigorous proofs about the invertibility of convolutional generative models. Despite these promising progress, there is no solid theoretical foundation on why the overall random CNN-DCN architecture is capable for image reconstruction. In this paper, we bridge the gap between the empirical observation and theoretical explanation of CNNs, especially the invertibility of the overall random CNN-DCN architecture. To understand the deep representations of intermediate layers, a variety of visualization techniques have been developed in order to unveil the feature representation and hence the inner mechanism of convolutional neural networks (CNNs) Zeiler & Fergus (2014) ; Mahendran & Vedaldi (2015) ; Yosinski et al. (2015) ; Xu et al. (2015) . In this work we propose applying randomization on deconvolutional networks (DCNs) for a systematic investigation of deep representations, and provide insights on the intrinsic properties of deep convolutional networks. We first observe that training the DCN for reconstruction, the random CNN preserves richer information in the feature space. The training on DCN converges faster for the random CNN contrasted to pre-trained CNN and yields higher quality image reconstruction. It indicates there is rich information encoded in the random features; the pre-trained CNN discards some information irrelevant for classification and encodes relevant features in a way favorable for classification but harder for reconstruction. This leads us to be curious about what happens if we feed the images to a CNN-DCN architecture where both the CNN and the DCN have random weights. Our motivation for studying the overall random CNN-DCN architecture is threefold. First, a series of works empirically showed that a certain feature learning architecture with random weights allowed satisfactory discriminative validity on object recognition tasks Jarrett et al. (2009) , and certain convolutional pooling architectures even with random weights can be inherently frequency selective and translation invariant, leading to the potential application of fast search of network architectures Saxe et al. (2011) . Second, studying a complex system with random weights rather than learned determin-istic ones may lead to a better understanding of the system even in the learned case. For example, in the field of compressed sensing, random sampling leads to breakthroughs in the understanding of the number of required measurements for a stable reconstruction of the signal Giryes et al. (2016) ; Gilbert et al. (2017) . For highly complicated systems with nonlinear operations along the hidden layers, there are already some investigations on random deep neural networks Saxe et al. (2011); Arora et al. (2014) ; Ulyanov et al. (2017a) . Third, as a reversible encoder-decoder architecture, deconvolution is a valuable visualization technique for studying the feature representation of deep convolutional nets. To our knowledge there is no existing work on the random deconvolutional networks in the literature. Our work on using deconvolution to study the random intermediate features of CNN provides new insights and inspires possible applications with untrained deep neural models. Our main results and contributions are as follows. We study the overall random CNN-DCN architecture to investigate the randomness in deconvolutional networks, i.e. there is no training at all for inverting the inputs that passes their information through a random weight convolutional network. Surprisingly, the image is inverted with satisfactory quality. The geometric and photometric features of the inputs are well preserved given a sufficient number of channels. We provide empirical evidence as well as theoretical analysis on the reconstruction quality, and bound the error in terms of the number of random nonlinearities, the network architecture, the distribution of the random weights, and local similarity of the input which is high for natual images. Extensive empirical study by varying the network width, depth, or kernel size has been performed to show the effectiveness on the inversion. The CNN-DCN architecture with random weights can be very useful on texture synthesis, style transfer, image segmentation, image inpainting, etc. As an example, we illustrate how fast style transfer can be applied using random weight CNN-DCN architecture. Note that our approach can save a big amount of time and energy as we do not need to do the pre-training on deep models, and it is very flexible as we can easily try whatever nerual network architecture as we wish. In this work, we introduce a novel investigation on deep random representations through the convolution-deconvolution architecture, which to our knowledge is the first study on the randomness of deconvolutional networks in the literature. We extensively explore the potential of randomness for image reconstruction on deep neural networks, and found that images can be reconstructed with satisfactory quality when there are a sufficient number of channels. Extensive investigations have been performed to show the effectiveness of the reconstruction. We also provide theoretical analysis that a slight variant of the random CNN architecture has the ability to reconstruct the input image, and the output converges to the input image when the width of the network, i.e. number of channels, goes to infinity. We also bound the reconstruction error between the input and the convergence value as a function of the network width and depth. (2015) and AlexNet Krizhevsky et al. (2012) . A convolutional layer is usually followed by a pooling layer, except for the last convolutional layer, Conv5. For consistency, we will explore the output after the convolutional layer but before the pooling layer. In what follows, "feature representation" or "image representation" denotes the feature vectors after the linear convolutional operator and the nonlinear activation operator but before the pooling operator for dimension reduction. We build a CNN-DCN architecture on the layer of feature representation to be studied. The convolution operator of a deconvolutional layer in DCN is the same as the convolution operator in CNN, and an upsampling operator is applied in DCN to invert the corresponding pooling operator in CNN, as designed in Dosovitskiy & Brox (2016). We will focus on the representations of the convolutional layers, since Dosovitskiy et al. Dosovitskiy & Brox (2016) build DCNs for each layer of the pre-trained AlexNet and find that the predicted image from the fully connected layers becomes very vague. For the activation operator, we apply the leaky ReLU nonlinearity with slope 0.2, that is, r(x) = x if x ≥ 0 and otherwise r(x) = 0.2x. At the end of the DCN, a final Crop layer is added to cut the output of DeConv1 to the same shape as the original images. We build deconvolutional networks on both VGG16 and AlexNet, and most importantly, we focus on the random features of the CNN structure when training the corresponding DCN. Then we do no training for deconvolution and explore the properties of the purely random CNN-DCN architecture on VGG16.
We investigate the deep representation of untrained, random weight CNN-DCN architectures, and show their image reconstruction quality and possible applications.
{"src": "scitldr-AIC_train", "paper_id": "S1xTMyHYwB"}