Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey
State-of-the-art deep learning models have a parameter count that reaches
into the billions. Training, storing and transferring such models is energy and
time consuming, thus costly. A big part of these costs is caused by training
the network. Model compression lowers storage and transfer costs, and can
further make training more efficient by decreasing the number of computations
in the forward and/or backward pass. Thus, compressing networks also at
training time while maintaining a high performance is an important research
topic. This work is a survey on methods which reduce the number of trained
weights in deep learning models throughout the training. Most of the introduced
methods set network parameters to zero which is called pruning. The presented
pruning approaches are categorized into pruning at initialization, lottery
tickets and dynamic sparse training. Moreover, we discuss methods that freeze
parts of a network at its random initialization. By freezing weights, the
number of trainable parameters is shrunken which reduces gradient computations
and the dimensionality of the model's optimization space. In this survey we
first propose dimensionality reduced training as an underlying mathematical
model that covers pruning and freezing during training. Afterwards, we present
and discuss different dimensionality reduced training methods.