Vision-based deep reinforcement learning (RL) typically obtains performance benefit by using high capacity and relatively large convolutional neural networks (CNN). However, a large network leads to higher inference costs (power, latency, silicon area, MAC count). Many inference optimizations have been developed for CNNs. Some optimization techniques offer theoretical efficiency, such as sparsity, but designing actual hardware to support them is difficult. On the other hand, distillation is a simple general-purpose optimization technique which is broadly applicable for transferring knowledge from a trained, high capacity teacher network to an untrained, low capacity student network. DQN distillation extended the original distillation idea to transfer information stored in a high performance, high capacity teacher Q-function trained via the Deep Q-Learning (DQN) algorithm. Our work adapts the DQN distillation work to the actor-critic Proximal Policy Optimization algorithm. PPO is simple to implement and has much higher performance than the seminal DQN algorithm. We show that a distilled PPO student can attain far higher performance compared to a DQN teacher. We also show that a low capacity distilled student is generally able to outperform a low capacity agent that directly trains in the environment. Finally, we show that distillation, followed by "fine-tuning" in the environment, enables the distilled PPO student to achieve parity with teacher performance. In general, the lessons learned in this work should transfer to other modern actor-critic RL algorithms.
Distillation was proposed in @cite_2 as a method to transfer knowledge from a trained teacher classifier neural network into an untrained student network. There are various techniques to implement distillation, and here we review the version most relevant to RL. Initially, assume a high capacity teacher classifier network has been trained to high performance, and a smaller network is to be trained with distillation. Additionally, assume access to the training inputs @math used for teacher training, but no access to class @math training labels @math . In this case, we may derive a loss function for the student network by providing training inputs @math to the teacher network and using its class probability distribution @math as a soft target for the student network's output probability distribution @math , where the student is parameterized by @math . The student's loss is defined as the distance between distributions @math and @math and may be measured using a standard metric, such as Kullback-Leibler divergence: where @math and @math represent the probability for class @math , given input @math . The gradient of @math may then be taken with respect to the student's parameters, which may then be updated using gradient descent.
@cite_2
1821462560
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.