OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer

A deep neural network of multiple nonlinear layers forms a large function space, which can easily lead to overfitting when it encounters small-sample data. To mitigate overfitting in small-sample classification, learning more discriminative features from small-sample data is becoming a new trend. To this end, this paper aims to find a subspace of neural networks that can facilitate a large decision margin. Specifically, we propose the Orthogonal Softmax Layer (OSL), which makes the weight vectors in the classification layer remain orthogonal during both the training and test processes. The Rademacher complexity of a network using the OSL is only $\frac{1}{K}$, where $K$ is the number of classes, of that of a network using the fully connected classification layer, leading to a tighter generalization error bound. Experimental results demonstrate that the proposed OSL has better performance than the methods used for comparison on four small-sample benchmark datasets, as well as its applicability to large-sample datasets. Codes are available at: https://github.com/dongliangchang/OSLNet.


I. INTRODUCTION
In recent years, due to the advances of deep learning, largescale image classification has achieved massive success [1], [2]. However, in many real-world problems, big data are difficult to obtain [3]- [5]. In addition, when human beings learn a concept, millions or billions of data samples are unnecessary [6], [7]. Therefore, small-sample classification [8]- [10] has received much attention recently in the deep learning community [7], [11]- [15]. However, a deep neural network usually models a large function space, which results in overfitting and instability problems that are difficult to avoid in small-sample classification [16]- [18].
Small-sample classification can be roughly classified into two families depending on whether there are unseen categories to be predicted [19]- [21]. In this work, we mainly focus on the one with no unseen category to be predicted. A key challenge in this direction is how to avoid overfitting. Up to now, many methods have been proposed to mitigate overfitting in smallsample classification, such as data augmentation [22]- [24], domain adaptation [3], [18], regularization [25], ensemble methods [26] and learning discriminative features [27].
Recently, learning discriminative features has become a new trend to improve the classification performance in deep learning. Methods such as the large-margin loss and the virtual softmax method [28] work well on both large-sample and small-sample data. However, these methods either add some constraints on the loss function or make some assumptions on data, which increases the difficulty of optimization and limits the applicable types of data. On the contrary, our goal in this work is to find a subspace of neural networks that can readily obtain a large decision margin and learn highly discriminative features. Specifically, we aim at obtaining a large decision margin through achieving large angles between the weight vectors of different classes in the classification layer (i.e., output layer), in part motivated by the observation that the larger the angles between the weight vectors in the classification layer are, the better the generalization performance is, as shown in Fig. 1.
Therefore, we propose the Orthogonal Softmax Layer (OSL) for neural networks as a replacement of the fully connected classification layer. In the proposed OSL, some connections are removed and the weight vectors of different classes are pairwise orthogonal. Due to fewer connections and larger between-class angles, the OSL can mitigate the coadaptation [29] of a network while enhancing the discrimination ability of features, as we will show in this work. Compared with traditional networks with a fully connected classification layer, a neural network with the OSL has significantly lower model complexity and is ideally suitable for small-sample classification. Experimental results demonstrate that the proposed OSL performs better on four small-sample benchmark datasets than the methods used for comparison as well as its applicability to large-sample datasets.
A number of methods have been proposed to maintain the orthogonality of the weight vectors during the training process of networks [30]- [36]. These methods maintain the orthogonality of weight vectors either for reducing gradients vanishing and obtaining a stable feature distribution [30], [31], [33] or for generating decorrelated feature representation [32], [37]. Unlike these methods, our work does not constrain the optimization process to obtain an orthogonal weight matrix, but constructs a network structure with a fixed orthogonal classification layer by removing some connections so that it can mitigate the co-adaptation between the parameters. The main contributions of this paper are threefold: 1) A novel layer, namely the Orthogonal Softmax Layer (OSL), is proposed. The OSL is an alternative to the fully connected classification layer and can be used as the classification layer (i.e., the output layer) of any neural network for classification. 2) The proposed OSL can reduce the difficulty of network optimization due to militating the co-adaptation between the parameters in the classification layer. 3) A network with the proposed OSL can have a lower generalization error bound than a network with a fully connected classification layer.
II. RELATED WORK Data augmentation. Data augmentation, which artificially inflates the training set with label preserving transformations, is well-suited to limited training data [38], [39], such as the methods of deformation [40], [41], generating more training samples [42], and pseudo-label [43]. However, data augmentation is computationally costly to implement [23].
Domain adaptation. The goal of domain adaptation [44], [45] is to use a model, which is trained in the source domain with a sufficient amount of annotated training data while trained in the target domain with little or no training data [46]- [49]. The simplest approach to domain adaptation is to use the available annotated data in the target domain to finetune a convolutional neural network (CNN) pre-trained on the source data, such as ImageNet [48], [50]- [52], which is a commonly used method for small-sample classification [53]. However, as both the initial learning rate and the optimization strategy of neural networks affect the final performance, this kind of methods is difficult to avoid the overfitting problem in small-sample classification. In addition, the knowledge distillation [54], a kind of method for knowledge transfer, compresses the knowledge in an ensemble into a single model. Overall, this type of methods have some limitations: the original domain and the target domain cannot be far away from each other, and overfitting of neural networks on smallsample data remains difficult to avoid.
Learning discriminative features. There are several recent studies that explicitly encourage the learning of discriminative features and enlarge the decision margin, such as the virtual softmax [28], the L-softmax loss [27], the A-softmax loss [55], the GM loss [56], and the center loss [57]. The virtual softmax enhances the discrimination ability of learned features by injecting a dynamic virtual negative class into the original softmax, and it indeed encourages the features to be more compact and separable across the classes [28]. The L-softmax loss and the A-softmax loss are built on the cross-entropy loss. They introduce new classification scores to enlarge the decision margin. However, the two losses increase the difficulty of network optimization. The GM loss assumes that the deep features of sample points follow a Gaussian mixture distribution. It still has some limitations since many real data are not well suitable to be modelled by Gaussian mixture distributions. The center loss [57] developed a regularization term for the softmax loss function, that is, features of the samples from the same class must be close in the Euclidean distance. In addition to the loss functions above, other loss functions either consider the imbalance of data [58] or the noisy labels in the training data [59]. The focal loss [58] places different weights on different training samples: the samples that are difficult to be identified will be assigned a large weight. The truncated L q loss [59], a noise-robust loss function, can overcome noisy labels in the training data. Unlike these studies for improving the loss function, our method obtains discriminative features by constructing a fixed orthogonal classification layer by removing some connections.
Ensemble methods. The ensemble methods [60]- [64] have been shown to be effective to address the overfitting problem. The SnapShot ensembling [26] only trains one time and obtains multiple base classifiers for free. This method leverages the nonconvex nature of neural networks and the ability of the stochastic gradient descent (SGD) to converge to or escape from local minima on demand. It finds multiple local minima of loss, saves the weights of the corresponding base networks, and combines the predictions of the corresponding base networks. The temporal ensembling, a parallel work to the SnapShot ensembling, can be trained on a single network. The predictions made on different epochs correspond to an ensemble prediction of multiple subnetworks due to dropout regularization [63]. Both the SnapShot ensembling and the temporal ensembling work well on small-sample data. However, generally speaking, using an entire ensemble of models to make prediction is cumbersome and computationally costly.
Regularization. The L 2 regularization method [65] is often applied to mitigate overfitting in neural networks. Dropout [66] mitigates overfitting mainly by randomly dropping some units (with their connections) from the neural network during training, which prevents the units from coadapting too much. Drop-Connect [67] is a variant of Dropout, where each connection can be dropped with probability p. Both approaches introduce dynamic sparsity within the model. The fully connected layer with DropConnect becomes a sparsely connected layer, where the connections are randomly selected during the training stage. In addition, the orthogonality regularization is also a kind of regularization techniques that are able to stabilize the distribution of activation over layers within CNNs and improve the performance of CNNs [37], [68].
In addition to these methods, there exist implicit regularization techniques, such as batch normalization and its variants [69], [70]. Batch normalization (BN) [71] aims to normalize the distribution of the layer input of neural networks, hence it can reduce the internal covariate shift in the training process. Built on BN, the decorrelated batch normalization (DBN) [69] decorrelates the layer input and improved BN on CNNs. Iterative normalization (IterNorm) [70] further improves DBN towards efficient whitening and iteratively normalizes the data along the eigenvectors in the training process. Both DBN and IterNorm adjust the distribution of samples so that features of samples are pairwise orthogonal. Unlike them, the proposed OSL forces the weights in the classification layer to be pairwise orthogonal (Please refer to Fig. 2 for details.) The orthogonality in OSL is to assign the input neurons to different output neurons so that it can mitigate the co-adaptation between the parameters.

III. THE ORTHOGONAL SOFTMAX LAYER
Dropping some connections in the neural networks are adopted by Dropout and DropConnect, as well as the proposed OSL. To make the proposed OSL easy to understand, we first review Dropout and DropConnect.

A. Preliminaries
We denote the input vector and the output vector of a layer in the neural network as v = [v 1 , v 2 , . . . , v D ] T and r = [r 1 , r 2 , . . . , r K ] T , respectively, and denote the activation function as a. Following the formulation in [67], the fully connected layer is defined as r = a(Wv) with weight matrix W.
Based on the above denotations, the fully connected layer with Dropout is defined as r = m a (Wv), where represents element wise product, m is a K-dimension binary vector. The jth element of m, m j ∼ Bernoulli (1 − p), j ∈ {1, 2, . . . , K} and p is probability for dropping a neuron. Moreover, the fully connected layer with DropConnect is defined as r = a ((M W) v), where represents element wise product and each entry of binary matrix M is M ij ∼ Bernoulli (1 − q), i ∈ {1, 2, . . . , D} and j ∈ {1, 2, . . . , K}, and q is the probability for dropping a connection.

B. Mathematical Interpretation of the OSL
To maintain large angles among the weights in the classification layer, we drop some connections in the fully connected classification layer, make the weights in the classification layer be pairwise orthogonal and propose the Orthogonal Softmax Layer (OSL, see Fig. 2). The OSL is defined as where represents element wise product, and M, the mask matrix of W, is a fixed and predesigned block diagonal matrix as where K is the number of classes, M ij is a column vector whose elements are 1, and 0 ij is a zero column vector where every element is 0. The matrix is fixed during the training and test phases. If we consider M W as one matrix, the column vectors are pairwise orthogonal, which is equivalent to introducing a strong prior that the angles between the weights of different categories are all 90 • .

C. Remarks for OSL
Here are some remarks on the OSL. First, the OSL is an alternative to the classification layer (i.e., the last fully connected layer) of a neural network. In contrast, the Dropconnect cannot be effectively used in the classification layer, because if all the connections of a output neuron are dropped, the neural network or part of it cannot be trained.
Second, a neural network with the OSL is a single model throughout the training and test processes, since it first drops some connections and subsequently fixes the structure in both the training and test phases. This is in contrast with the Drop-Connect method. DropConnect randomly drops connections with a given probability during the training phase, but no connection is dropped in the test phase. DropConnect can be considered an implicit ensemble method.
Third, in the OSL, different neurons of the last hidden layer connect to different output neurons. This setup assigns each hidden neuron to one and only one specific class that they are responsible prior to the start of training, so the difficulty level of training the neural network is reduced. Moreover, the solution space of a network with the OSL is a subset of the solution space of its corresponding network with a fully connected classification layer, which implies low model complexity of the former.

D. Implementation of the OSLNet
The OSL can be used in any type of networks for classification, such as the fully connected network or CNN. For convenience, we call a neural network with the OSL as OSLNet.
The OSLNet uses the OSL as the classification layer instead of a fully connected classification layer, and the structure of the other layers are kept the same as a standard network, e.g., a neural network with the fully connected classification layer. Different from the implementation of the fully connected classification layer, the forward computation of the OSL needs to conduct the dot product (see (1)) between the predesigned matrix M (see (2)) and weights matrix W.

E. The Generalization Error Bound of the OSLNet
In this section, we discuss the generalization error bound of the OSLNet in terms of the Rademacher complexity [72], [73]. We define the entire network into two parts: the classification layer and the layers of extracting features. The classification layer refers to the last fully connected layer in a neural network, and the layers of extracting features refer to all layers except the classification layer. Based on the definitions, a standard network with the fully connected classification layer can be represented as f (x; W s , W g ) and the OSLNet can be represented as f (x; M W s , W g ), where W s and M W s are the parameters of the classification layer for the two networks, respectively, and W g are the parameters of the layers for extracting features.
, and H denote a linear transform function H : Theorem 2. (Complexity of OSLNet). Following the notation in Lemma 1, we further letR l (O) denote the empirical Rademacher complexity of an OSLNet. If the weights Proof. First, we denote the empirical Rademacher complexity of a standard network with the fully connected classification layer asR l (O 0 ) =R l (H 0 • G). Since no connection has been removed in a standard network with the fully connected classification layer, H 0 is a linear transform function H 0 : Because in the OSLNet some connections are removed and the weight vectors of different classes are pairwise orthogonal, H becomes a linear transform function H: The analysis above shows that the bound of empirical Rademacher complexity for the OSLNet is only 1 K of that for a standard network. In addition, the empirical error of the OSLNet is close to the standard model in terms of accuracy and cross-entropy loss on the training data, as shown in Fig. 6. Therefore, according to the relationship between a model generalization error bound and an empirical Rademacher complexity (Theorem 3.1 in [72]), the OSLNet has a lower model generalization error bound than the standard network.

IV. EXPERIMENTAL EVALUATION
To evaluate the performance of the proposed OSL, we compare OSLNet with other methods on four small-sample datasets and three large-sample datasets. These evaluation serve four purposes:

A. Small-Sample Datasets
For experiments on small-sample image classification, we selected the following four small-sample datasets: • UIUC-Sports dataset (UIUC) 1    For the 15Scenes and 80-AI datasets, in both the training and test datasets, each class contains 100 samples. For the UIUC and Caltech101 datasets, unlike the 15Scenes and 80-AI datasets, different classes have different number of samples, and we randomly split the data of each class into the training and test sets evenly.
We adopted a CNN feature extractor, VGG16 [77], which was pre-trained on ImageNet. First, we resized the images into identical sizes of 256 × 256 and directly extracted the

B. The Compared Methods and Their Implementation
To evaluate the classification performance of the proposed OSL, we compare the following methods: 1) fully connected network (FC); 2) Focal loss (Focal) [58]; 3) Center loss (Center) [57] For FC, we used a fully connected network with two layers, where the activation functions of the first and second layers are rectified linear unit function (Relu) and Softmax, respectively. FC is optimized by minimizing the softmax cross-entropy loss based on the minibatch gradient descent. The optimization algorithm is the RMSprop with the initial learning rate 0.001, the batch size is 32, and the number of epochs is 100.
For DBN and IterNorm, we placed them before each linear layer of FC. For DBN, we set group-number = 2 for the UIUC and 15 scenes datasets and set group-number = 16 for the 80-AI and Caltech101 datasets. For IterNorm, we set T = 10 and group-number = 8 for all the four datasets.
For Focal, Center, T-L q , and Lsoftmax, we adopted the focal loss [27], center loss, truncated L q loss, and large-margin softmax loss [27] to replace the softmax cross-entropy loss used in FC, respectively. For these four loss functions, we tried multiple sets of parameter values and selected the setting with best performance. Specifically, for Focal, we selected γ = 0.5 for the 80-AI dataset and γ = 0.3 for other three datasets. For Center, we selected a small loss weight, λ = 1e − 10. For T-L q , we set q = 0.5 and k = 0.1 for the UIUC dataset, and q = 0.1 and k = 0.1 for the 15Scenes dataset. However, on the 80-AI and Caltech101 datasets, we did not find a set of q and k that makes T-L q fit the training data. For Lsoftmax, we set m = 2. Other settings were kept the same as those for FC.
For Dropout, we added a Dropout layer after the hidden layer of FC. The probability that a neuron unit should be dropped is 0.5. Other settings of Dropout are identical to FC.
The SnapShot here is a SnapShot ensembling built on the FC network, where the learning rate is learned by using a cyclic cosine annealing method [78], and the number of SnapShots is 2. For OS, we replaced the fully connected classification layer with the proposed OSL in FC and other settings were identical to that of FC. For OS-SnapShot, except for the structure of base network, other settings are identical to that of SnapShot.
All the methods have been implemented with PyTorch [79].

C. Classification Accuracy
We ran FC, Focal, Center, T-L q , IterNorm, DBN, Dropout, Lsoftmax, SnapShot, OS and OS-SnapShot on the UIUC, 15Scenes, 80-AI and Caltech101 datasets for 60 rounds each. The mean and the standard deviation of the classification accuracy are listed in Table I, and the boxplot of classification accuracy is shown in Fig. 3. Larger mean and smaller standard deviation indicate better performance. Table I shows that, on the four datasets, FC is easy to overfit and has unstable performance. Moreover, Focal, Center, and T-L q underperform FC. IterNorm outperforms FC on 80-AI and DNB performs better than FC on Caltech101, and in other cases IterNorm and DBN underperform FC. Dropout performs slightly better than FC on UIUC, competitively with FC on 80-AI, and worse than FC on 15Scenes and Caltech101. Lsoftmax performs better than FC on UIUC, 15Scenes and Caltech101 but slightly worse than FC on 80-AI. SnapShot always has better performance than FC on the four datasets. OS performs better than FC, Dropout and Lsoftmax on UIUC, 15Scenes and 80-AI, showing larger mean and smaller variance. OS performs better than FC and Dropout, but worse than Lsoftmax and SnapShot, on Caltech101. We also evaluated the SnapShot ensembling version of OS, denoted as OS-SnapShot, and it obtains better performance on all the four datasets. Fig. 3 demonstrates that on the UIUC-Sports dataset, the boxplot of OS is more compact than those of FC, Focal, Center, T-L q , IterNorm, DBN, Dropout, Lsoftmax and SnapShot. Moreover, it has no bad-performing outlier. On the 15Scenes dataset, the boxplot of OS is more compact than those of FC, Dropout and SnapShot, but is close to that of Lsoftmax. On the 80-AI dataset, the boxplot of OS is also more compact than those of other methods, and both the central mark and the edges of the box are higher than those of other methods used for comparison. On the Caltech101 dataset, the boxplot  of OS is also more compact and higher than those of FC and Dropout, but is lower than those of Lsoftmax and SnapShot. Finally, the boxplots of OS-SnapShot is always more compact than those of all other compared methods, with both the central mark and the edges of the boxes higher. Table II and Fig. 4 show that, on the UIUC datasets with decreased training set sizes, Dropout shows small improvement over FC. Lsoftmax achieves larger mean and lower variance than FC and Dropout. SnapShot, an ensemble method, obtains larger mean, but the variance is similar to the others. It is encouraging to see that OS has larger mean and lower standard deviation than all the four compared methods on all the different sizes of training sets. OS-SnapShot outperforms OS in terms of the mean accuracy.

D. Classification Accuracy for Different Training Set Sizes
On the 15Scenes dataset, Dropout performs worse than FC, in terms of variance and mean. Lsoftamx consistently performs better than FC. SnapShot shows slight improvement but is worse than Lsoftmax. OS still performs well, as expected. OS-SnapShot improves the mean compared with OS.
On the 80-AI dataset, Lsoftmax and Dropout perform worse than FC. Lsoftmax notably is difficult to converge with the four training sizes. SnapShot performs better than FC, Dropout and Lsoftmax. OS has smaller variances and larger means and performs much better than all other methods.
On the Caltech101 dataset, SnapShot performs better than FC, Dropout and Lsoftmax. OS has smaller variances and larger means than FC and Dropout and has similar performance to Lsoftmax. OS-SnapShot performs the best.
In summary, the proposed OSL performs better than other compared methods on these datasets.

E. Paired Student's t-test
The experimental results presented in the previous sections show that the proposed OSL obtains better performance. To confirm that the improvement is statistically significant, we performed a paired Student's t-test [80] for OS vs. FC, OS vs. Dropout, OS vs. Lsoftmax, and OS vs. SnapShot, and the p-values are listed in Table III. Following [81], we set the significance level as 0.05.
For the tests of OS vs. FC, all the p-values are much smaller than the significance level. Thus, the null hypothesis that FC and OS have the identical mean is always rejected. Likewise, the null hypothesis that Dropout has the identical mean to OS is also rejected.
In terms of Lsoftmax, on the UIUC, 80-AI and Caltech101 datasets, the null hypothesis that Dropout and OS have the identical mean is always rejected. However, on the 15Scenes dataset, the p-values are larger than the significance level in most cases. Thus, on 15Scenes, the null hypothesis that Lsoftmax and OS have the identical mean is generally not rejected.
For SnapShot, on the 80-AI and Caltech101 datasets, the null hypothesis that SnapShot and OS have the identical mean is always rejected. However, on the UIUC and 15Scenes datasets, the null hypothesis that SnapShot has the identical mean with OS is not always rejected.

F. Effect of Changing the Width of the Hidden Layers on OSLNet
In the previous experiments, our method and all the baseline methods used the network structure with 32 neurons in the last hidden layer. To explore the effect of the width of the hidden layer on the performance of OSLNet, we changed the number of hidden neurons in both FC and OS from 16 to 512 on the UIUC dataset, with other settings unchanged. At the same time, we also changed the number of the hidden neurons from 32 to 512 on the 15Scenes dataset, with other settings unchanged. FC and OS were evaluated on the two datasets for 60 rounds, respectively.
From Table IV, we can observe that, with the increase in the width of network, the mean of FC has generally slight improvement on UIUC, and the standard deviation decreases. That is, FC generally performs better when the number of the hidden neurons is increased. In contrast, OS performs better when the number of the hidden neurons is small. Particularly, when the number of the hidden neurons is 16, OS is much  V  COMPARISON OF THE CLASSIFICATION ACCURACIES ON THE UIUC AND 15SCENES DATASETS WHEN WE VARY THE DEPTH OF THE NEURAL NETWORKS.  THE SIZES OF THE HIDDEN LAYERS ARE LISTED. EACH STRUCTURE OF FC AND OS IS TESTED FOR 60 ROUNDS; THE MEAN AND STANDARD DEVIATION  (STD.) ARE LISTED IN THE CELLS OF THE TABLE.   UIUC  Measure  32-8  64-32-8 128-64-32-8 256-128-64-32-8

G. Effect of Changing the Depth of the Hidden Layers on OSLNet
We also evaluate the effect of the depth of the network on OSLNet. In particular, we varied the depth of the network from 2 to 5 layers and from 2 to 4 layers in both FC and OS on the UIUC and 15Scenes datasets, respectively. Each structure of FC and OS is evaluated on UIUC and 15Scenes for 60 rounds. The size of each layer, the depth of each layer, the corresponding mean and standard deviation of accuracy are listed in Table V.
From Table V, we can observe that, on UIUC, the mean values of FC and OS become smaller, when the depth is increased, that is, the performance of both FC and OS decreases, and OS decreases faster than FC. A similar pattern is also found on 15Scenes. Therefore, shallow structures of both FC and OS perform better than deep ones of FC and OS on the two datasets.

H. Effect of Changing Feature Extractor on OSLNet
In all the experiments presented above, we used a pretrained VGG16 as the feature extractor. In this section, we changed the feature extractor to a pre-trained DenseNet121 and ran all the compared methods on the UIUC and 15Scenes datasets for 60 rounds each. The classification results are shown in Table VI. Here, we do not list the performance of T-L q and Lsoftmax, as it cannot fit the training data if the feature extractor is DenseNet121.
From Table VI, we can observe that: 1) On the UIUC and 15scenes datasets, each method performs better with the VGG16 feature extractor than with the DenseNer121 feature extractor. 2) When the feature extractor is DenseNet121, on the UIUC dataset, the proposed OS outperforms all the compared methods, and on the 15scenes dataset, OS underperforms SnapShot and outperform other compared methods. Then we can conclude that OS-SnapShot performs best on both datasets.
In summary, the performance comparision on DenseNet121 is similar to those on VGG16, which further shows the applicability of the OSL.

I. Feature Visualization
To gain insights on the proposed OSL, we visualize the input features of OSL. Take the experiments on the UIUC dataset as an example: the input dimension of OSL is 32, and the output dimension of OSL is 8, as determined by the number of classes. Therefore, according to the design of OSL, each output neuron (i.e., each class) is assigned 4 input neurons. For example, the 1st-4th dimensional input features are for the class 0, and the 5th-8th dimensional input features are for the class 1 (The class labels are from 0 to 7 on the UIUC dataset). We use t-SNE [82] to visualize the input features of OSL in Fig. 5. In Fig. 5, the first column is for the 32-dimensional input features without truncation, and the following three columns are for the 4th-8th dimensional features, the 13th-16th dimensional features, the 21th-24th dimensional features, which correspond to the classes 1, 3 and 5 in the OSL, respectively.
From Fig. 5, it can be observed that, in the first column, the samples from different classes are separate. In the following three columns, the samples of the corresponding class are put to one end of a feature curve or a place far from the features of other classes. A similar pattern appears for other classes in the UIUC dataset and in other datasets. This demonstrates that, in the proposed OSL, the input neurons assigned to each class are indeed responsible for learning discriminative features for differentiating that class from other classes.

J. OSLNet on Large-sample Datasets
To evaluate the applicability of our OSL on large-sample classification, we selected the following three datasets.
• CIFAR-10: This dataset [83] Table VII and demonstrated in Fig. 6. On the CIFAR-10 and CIFAR-100 datasets, for CNN, we used the VGG16 style network, where the convolutional layer has batch normalization, and the fully connected parts have 2 hidden layers of 16 units each. The epoch number is 400, and the learning rate decreases from 0.01 to 0 by using the cosine annealed method. The optimization method iss SGD. For OS-CNN, we only replaced the last fully connected layer with the OS layer in CNN, while other settings unchanged. For CNN-SnapShot, the learning rate decreases from 0.01 to 0 by using the annealed cosine method within 200 epochs, which restarted to the identical process one more time. The total number of epochs is 400, and the number of SnapShot networks is 2. Other settings are identical to those for CNN. For OS-CNN-SnapShot, except for the classification layer, other settings are identical to that for CNN-SnapShot.
On the MNIST dataset, we followed the CNN structure published by PyTorch for MNIST. In the structure, there are two modules including convolution (Maxpooling and Relu activation) and two fully connected layers with 50 hidden neurons. The optimization method remains to be SGD, and the learning rate decreases from 0.01 to 0 as the cosine annealed method is used. The epoch number is 400. For OS-CNN, CNN-SnapShot and OS-CNN-SnapShot, except for the structure of CNN, other settings were the same as that for the CIFAR-10 and CIFAR-100 datasets.
From the classification accuracy shown in Table VII, the curves of the cross-entropy loss, and the accuracy shown in  The left-hand column is for the training data, and the right-hand column is for the test data.  Fig. 6, it can be observed that CNN and OS-CNN have similar performance on the three datasets. That is, the proposed OSL is also applicable for large-sample classification.

K. Discussion
The focal loss places large weights on the samples that are difficult to identify; the center loss constructs a constraints that the features of samples from the same class must be close in the Euclidean distance; and the truncated L q loss is a noiserobust loss. In the experiments above, the three loss functions are not effective on the four small-sample datasets. It can also be observed that IterNorm and DBN perform unstably, which may be because, while accelerating optimization of neural networks, they do not have any particular mechanism to improve the classification performance of networks.
Among all the compared methods in the experiments on small-samples datasets, Dropout is an implicit ensemble method: in the training process it trains many subnetworks of the original network, and in the test phase no neuron is dropped. On the four small-sample datasets, Dropout do not show remarkable advantages over the network without dropouts.
Lsoftmax introduces a large margin into the cross-entropy softmax loss function to learn more discriminative features.
The key problem of this method is that it is not easy to converge. Thus, it does not perform well in some experiments with a reduced training size. In contrast, the proposed OSL method converges easily.
The SnapShot is a strong baseline among the compared methods and obtains relatively larger mean values and relatively smaller variances. However, although the method is trained only once in the training process and combines the predictions of all SnapShot networks in the test phase, due to the correlation of the base networks, the SnapShot increases the performance of FC slightly.
The proposed method on small-samples datasets shows higher accuracy and better stability, which are mainly attributed to the OSL, where some connections are removed and the angles among the weights of different classes are maintained as 90 • from the beginning to the end during training. Having large angles among the weights of different classes in the classification layer is an important precondition to obtain a decision rule with a large margin, when the L 2 norms of the weights are not considered. In addition, which class each neuron of the last hidden layer should serve exclusively is determined prior to the start of training, so the difficulty of training can be significantly reduced. The experimental results of changing the width and depth of the network suggest that when a thin and shallow structure is selected, the OSL can obtain a better performance. In addition, experimental results of our OSL on the three large-sample image datasets show that, although developed for small-sample classification, OSL can also be well applied to large-sample classification.
Even though OSL has demonstrated superior performance, it has its own limitations. For example, due to the design of OSL, the number of neurons of the last hidden layer should be larger than the class number. Therefore, when the class number is large, e.g., 1000, a large number of neurons will be required, which will increase the flexibility of network. We leave it as an open problem for the future.

V. CONCLUSIONS
In this paper, we proposed a new classification layer called Orthogonal Softmax Layer (OSL). A network with OSL (OSLNet) has two advantages, i.e., easy optimization and low Rademacher complexity. The Rademacher complexity of OSLNet is 1 K , where K is the number of classes, of that of a network with the fully connected classification layer. Experimental results on four small-sample datasets provide the following observations for an OSLNet in small-sample classification: 1) It is able to obtain higher accuracy with larger mean and smaller variance than those of the baselines used for comparison. 2) It is statistically significantly better than the baselines. 3) It is more suitable for thin and shallow networks than a fully connected network. Further experiments on three large-sample datasets show that, compared with a CNN with the fully connected classification layer, a CNN with the OSL has a competitive performance for both training and test data.