The phenomenon of memorization can also be observed on a real dataset when there are no random labels.
[1]} studied memorization through the lens of self-influence estimation, and provided compelling evidences for the long tail theory [2]}. They believe that memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed.
Modern datasets usually follows a long tail distribution, where the subpopulations at the head of the distribution include a large amount of frequent and easy examples while the subpopulations at the tail include rare and difficult examples.
