Over-parameterized deep neural networks (DNNs), which have more number of trainable parameters than that of data instances, usually achieve excellent performance on both the training dataset and the test dataset.
However, it has also been found that a DNN has the capability to perfectly fit pure random labels, and this behaviour of a DNN when trained on random labels or random inputs has been implicitly referred to as the phenomenon of memorization [1]}, [2]}, [3]}, [4]}.
Understanding the memorization of deep neural networks is a key step towards characterizing its optimization dynamics and interpreting the model's generalization ability.
