Boundary between noise and information applied to filtering neural network weight matrices
Deep neural networks have been successfully applied to a broad range of
problems where overparametrization yields weight matrices which are partially
random. A comparison of weight matrix singular vectors to the Porter-Thomas
distribution suggests that there is a boundary between randomness and learned
information in the singular value spectrum. Inspired by this finding, we
introduce an algorithm for noise filtering, which both removes small singular
values and reduces the magnitude of large singular values to counteract the
effect of level repulsion between the noise and the information part of the
spectrum. For networks trained in the presence of label noise, we indeed find
that the generalization performance improves significantly due to noise
filtering.