The ResNet and the batch-normalization (BN) achieved high performance even when only a few labeled data are available. However, the reasons for its high performance are unclear. To clear the reasons, we analyzed the effect of the skip-connection in ResNet and the BN on the data separation ability, which is an important ability for the classification problem. Our results show that, in the multilayer perceptron with randomly initialized weights, the angle between two input vectors converges to zero in an exponential order of its depth, that the skip-connection makes this exponential decrease into a sub-exponential decrease, and that the BN relaxes this sub-exponential decrease into a reciprocal decrease. Moreover, our analysis shows that the preservation of the angle at initialization encourages trained neural networks to separate points from different classes. These imply that the skip-connection and the BN improve the data separation ability and achieve high performance even when only a few labeled data are available. The architecture of a neural network heavily affects its performance especially when only a few labeled data are available. The most famous example of one such architecture is the convolutional neural network (CNN) BID6 . Even when convolutional layers of CNN were randomly initialized and kept fixed and only the last fully-connected layer was trained, it achieved a competitive performance compared with the traditional CNN BID5 BID14 . Recent other examples are the ResNet BID3 and the batch-normalization (BN) BID4 . The ResNet and the BN are widely used in few-shot learning problems and achieved high performance BID8 BID9 .One reason for the success of neural networks is that their architectures enable its feature vector to capture prior knowledge about the problem. The convolutional layer of CNN enable its feature vector to capture statistical properties of data such as the shift invariance and the compositionality through local features, which present in images BID13 . However , effects of the skip-connection in ResNet and the BN on its feature vector are still unclear.To clear the effects of the skip-connection and the BN, we analyzed the transformations of input vectors by the multilayer perceptron, the ResNet, and the ResNet with BN. Our results show that the skip-connection and the BN preserve the angle between input vectors. This preservation of the angle is a desirable ability for the classification problem because the last output layer should separate points from different classes and input vectors in different classes have a large angle BID11 BID10 . Moreover, our analysis shows that the preservation of the angle at initialization encourages trained neural networks to separate points from different classes. These imply that the skip-connection and the BN improve the data separation ability and achieve high performance even when only a few labeled data are available. The ResNet and the BN achieved high performance even when only a few labeled data are available. To clear the reasons for its high performance, we analyzed effects of the skip-connection in ResNet and the BN on the transformation of input vectors through layers. Our results show that the skip-connection and the BN preserve the angle between input vectors, which is a desirable ability for the classification problem. Moreover, our analysis shows that the preservation of the angle at initialization encourages trained neural networks to separate points from different classes. These results imply that the skip-connection and the BN improve the data separation ability and achieve high performance even when only a few labeled data are available.
The Skip-connection in ResNet and the batch-normalization improve the data separation ability and help to train a deep neural network.
{"src": "scitldr-AIC_train", "paper_id": "S1xU74med4"}