One of the main challenges of deep learning methods is the choice of an appropriate training strategy. In particular, additional steps, such as unsupervised pre-training, have been shown to greatly improve the performances of deep structures. In this article, we propose an extra training step, called post-training, which only optimizes the last layer of the network. We show that this procedure can be analyzed in the context of kernel theory, with the first layers computing an embedding of the data and the last layer a statistical model to solve the task based on this embedding. This step makes sure that the embedding, or representation, of the data is used in the best possible way for the considered task. This idea is then tested on multiple architectures with various data sets, showing that it consistently provides a boost in performance. The experiments presented in Section 4 show that post-training improves the performances of all the networks considered -including recurrent, convolutional and fully connected networks. The gain is significant, regardless of the time at which the regular training is stopped and the posttraining is done. In both the CIFAR10 and the PTB experiment, the gap between the losses with and without post-training is more pronounced if the training is stopped early, and tends to be smaller as the network converges to a better solution (see Figure 4 and Figure 3 ). The reduction of the gap between the test performances with and without post-training is made clear in TAB0 . For the MNIST data set, with a small-size convolutional neural network, while the error rate drops by 1.5% when post-training is applied after 5000 iterations, this same error rate only drops by 0.2% when it is applied after 20000 iterations. This same observation can be done for the other results reported in TAB0 . However, while the improvement is larger when the network did not fully converge prior to the post-training, it is still significant when the network has reached its minimum: for example in PTB the final test perplexity is 81.7 with post-training and 82.4 without; in CIFAR10 the errors are respectively 0.147 and 0.154.If the networks are allowed to moderately overfit, for instance by training them with regular algorithm for a very large number of iterations, the advantage provided by post-training vanishes: for example in PTB the test perplexity after 2000 iterations (instead of 400) is 83.2 regardless of posttraining. This is coherent with the intuition behind the post-training: after overfitting, the features learned by the network become less appropriate to the general problem, therefore their optimal usage obtained by post-training no longer provide an advantage.It is important to note that the post-training computational cost is very low compared to the full training computations. For instance, in the CIFAR10 experiment, each iteration for post-training is 4× faster on the same GPU than an iteration using the full gradient. Also, in the different experiments, post-training produces a performance gap after using as little as 100 batches. There are multiple reasons behind this efficiency: first, the system reaches a local minimum relatively rapidly for post-training as the problem FORMULA3 has a small number of parameters compared to the dimensionality of the original optimization problem. Second, the iterations used for the resolution of (4) are computationally cheaper, as there is no need to chain high dimensional linear operations, contrarily to regular backpropagation used during the training phase. Finally, since the post-training optimization problem is generally convex, the optimization is guaranteed to converge rapidly to the optimal weights for the last layer.Another interesting point is that there is no evidence that the post-training step leads to overfitting. In CIFAR10, the test error is improved by the use of post-training, although the training loss is similar. The other experiments do not show signs of overfitting either as the test error is mostly improved by our additional step. This stems from the fact that the post-training optimization is much simpler than the original problem as it lies in a small-dimensional space -which, combined with the added 2 -regularization, efficiently prevents overfitting. The regularization parameter λ plays an important role in post-training. Setting λ to be very large reduces the explanatory capacity of the networks whereas if λ is too small, the capacity can become too large and lead to overfitting. Overall, our experiments highlighted that the post-training produces significant results for any choice of λ reasonably small (i.e 10 −5 ≤ λ ≤ 10 −2 ). This parameter is linked to the regularization parameter of the kernel methods, as stated in Section 3.Overall, these results show that the post-training step can be applied to most trained networks, without prerequisites about how optimized they are since post-training does not degrade their performances, providing a consistent gain in performances for a very low additional computational cost.In Subsection 4.3, numerical experiments highlight the link between post-training and kernel methods. As illustrated in TAB1 , using the optimal weights derived from kernel theory immediately a performance boost for the considered network. The post-training step estimate numerically this optimal layer with the gradient descent optimizer. However, computing the optimal weights for the last layer is only achievable for small data set due to the required matrix inversion. Moreover, the closed form solution is known only for specific problems, such as kernelized least square regression. But post-training approaches the same performance in these cases solving (4) with gradient-based methods.The post-training can be linked very naturally to the idea of pre-training, developed notably by BID12 , and BID5 . The unsupervised pre-training of a layer is designed to find a representation that captures enough information from the data to be able to reconstruct it from its embedding. The goal is thus to find suitable parametrization of the general layers to extract good features, summarizing the data. Conversely, the goal of the posttraining is, given a representation, to find the best parametrization of the last layer to discriminate the data. These two steps, in contrast with the usual training, focus on respectively the general or specific layers. In this work, we studied the concept of post-training, an additional step performed after the regular training, where only the last layer is trained. This step is intended to take fully advantage of the data representation learned by the network. We empirically shown that post-training is computationally inexpensive and provide a non negligible increase of performance on most neural network structures. While we chose to focus on post-training solely the last layer -as it is the most specific layer in the network and the resulting problem is strongly convex under reasonable prerequisites -the relationship between the number of layers frozen in the post-training and the resulting improvements might be an interesting direction for future works.Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.arXiv preprint, arXiv:1409(2329), 2014.
We propose an additional training step, called post-training, which computes optimal weights for the last layer of the network.
{"src": "scitldr-AIC_train", "paper_id": "H1O0KGC6b"}