Along with research directions into network uncertainty and robustness, neural network sparsity has become a subject of interest for many machine learning researchers  [1]}, [2]}, [3]}. Sparse neural networks are desirable because they require less computation at test time, demand less memory [1]}, and are less likely to overfit to training data [5]}. In the context of our investigation, the tendency for sparse neural networks to overfit more slowly to training data can allow them to avoid memorizing noisy training labels. We can evaluate network sparsity on a per-neuron level: which neurons never or rarely activate, for any and all test samples, and how common are these neurons throughout the entire model? Network sparsity can be defined as the subset of neurons output a value that is always zero  [1]}, or very close: these neurons do not affect the final predictions in any significant manner. The larger the subset of neurons with this property, the more sparse the network's learned representation is. Because neural networks can easily overfit to noise in training labels [7]}, we are interested in the observed property of sparse models to overfit more slowly. With fewer tunable parameters available, sparse models have fewer degrees of freedom to overfit to noise.
