Based on our observation that there exists a dramatic drop for the singular values of the fully connected layers or a single feature map of the convolutional layer, and that the dimension of the concatenated feature vector almost equals the summation of the dimension on each feature map, we propose a singular value decomposition (SVD) based approach to estimate the dimension of the deep manifolds for a typical convolutional neural network VGG19. We choose three categories from the ImageNet, namely Persian Cat, Container Ship and Volcano, and determine the local dimension of the deep manifolds of the deep layers through the tangent space of a target image. Through several augmentation methods, we found that the Gaussian noise method is closer to the intrinsic dimension, as by adding random noise to an image we are moving in an arbitrary dimension, and when the rank of the feature matrix of the augmented images does not increase we are very close
 to the local dimension of the manifold. We also estimate the dimension of the deep manifold based on the tangent space for each of the maxpooling layers. Our results show that the dimensions of different categories are close to each other and decline quickly along the convolutional layers and fully connected layers. Furthermore, we show that the dimensions decline quickly inside the Conv5 layer. Our work provides new insights for the intrinsic structure of deep neural networks and helps unveiling the inner organization of the black box of deep neural networks. To have a better understanding of deep neural networks, a recent important trend is to analyze the structure of the high-dimensional feature space. Capitalizing on the manifold hypothesis BID1 BID12 , the distribution of the generated data is assumed to concentrate in regions of low dimensionality. In other words, it is assumed that activation vectors of deep neural networks lie on different low dimensional manifolds embedded in high dimensional feature space.Note that the rationality of many manifold learning algorithms based on deep learning and autoencoders is that one learns an explicit or implicit coordinate system for leading factors of variation. These factors can be thought of as concepts or abstractions that help us understand the rich variability in the data, which can explain most of the structure in the unknown data distribution. See BID3 for more information.The dimension estimation is crucial in determining the number of variables in a linear system, or in determining the number of degrees of freedom of a dynamic system, which may be embedded in the hidden layers of neural networks. Moreover, many algorithms in manifold learning require the intrinsic dimensionality of the data as a crucial parameter. Therefore, the problem of estimating the intrinsic dimensionality of a manifold is of great importance, and it is also a crucial start for manifold learning.Unfortunately, the manifold of interest in AI (especially for deep neural networks), is such a rugged manifold with a great number of twists, ups and downs with strong curvature. Thus, there is a fundamental difficulty for the manifold learning, as raised in BID0 , that is, if the manifolds are not very smooth, one may need a considerable number of training examples to cover each one of these variations, and there is no chance for us to generalize to unseen variations.Our work is based on an important characterization of the manifold, namely, the set of its tangent hyperplanes. For a point p on a d-dimensional manifold, the tangent hyperplane is given by a local basis of d vectors that span the local directions of variations allowed on the manifold. As illustrated in Figure 1 , these local directions specify how one can change p infinitesmally while staying on the manifold. Figure 1 : A two-dimensional manifold with a small region where data points concentrate, along with a tangent plane and associated tangent directions, forming a basis that specifies the directions of small moves one can make to stay on the manifold.Based on above analysis, our work focuses on a thorough exploration of the local hyperplane dimension of the activation manifold in deep neural networks. Creating an artificial data cluster concentrated in regions of the local tangent hyperplane, we apply SVD to the data cluster in different layers or feature maps in neural networks. Through thorough analysis, we reach the following fascinating results.• There exists a dramatic drop for the singular values of the fully connected layers or a single feature map of the convolutional layer.• For convolutional layers, the dimension of the concatenated feature vector almost equals the summation of the dimension on each feature map.• The dimensions of different image categories are close and the dimension declines quickly along the layers.To our knowledge this is the first thorough exploration of manifold dimension on very deep neural networks. We wish our work sheds light on new understandings and inspires further investigations on the structure of manifolds in deep neural networks. Through extensive experiments, we found that there exists a dramatic drop for the singular values of the fully connected layers or a single feature map of the convolutional layer, and the dimension of the concatenated feature vector almost equals the summation of the dimension of each feature map for several feature maps randomly picked. Based on the interesting observations we obtained, we developed an efficient and effective SVD based method to estimate the local dimension of deep manifolds in the VGG19 neural network. We found that the dimensions are close for different images of the same category and even images of different categories, and the dimension declines quickly along the convolutional layers and fully connected layers. Our results supports the lowdimensional manifold hypothesis for deep networks, and our exploration helps unveiling the inner organization of deep networks. Our work will also inspire further possibility of observing every feature map separately for the dimension of convolutional layers, rather than directly working on the whole activation feature maps, which is costly or even impossible for the current normal computing power.
We propose a SVD based method to explore the local dimension of activation manifold in deep neural networks.
{"src": "scitldr-AIC_train", "paper_id": "Sy3nGCYXz"}