The State of Sparse Training in Deep Reinforcement Learning
The use of sparse neural networks has seen rapid growth in recent years,
particularly in computer vision. Their appeal stems largely from the reduced
number of parameters required to train and store, as well as in an increase in
learning efficiency. Somewhat surprisingly, there have been very few efforts
exploring their use in Deep Reinforcement Learning (DRL). In this work we
perform a systematic investigation into applying a number of existing sparse
training techniques on a variety of DRL agents and environments. Our results
corroborate the findings from sparse training in the computer vision domain -
sparse networks perform better than dense networks for the same parameter count
- in the DRL domain. We provide detailed analyses on how the various components
in DRL are affected by the use of sparse networks and conclude by suggesting
promising avenues for improving the effectiveness of sparse training methods,
as well as for advancing their use in DRL.