However, the insights of these prior work normally are limited through some computationally intensive estimation.
For example, [1]}, [2]} only study the memorization phenomenon on random data, and it is not clear if such observations can be generalized to real data.
Similarly, the study on influence estimation [3]}, [4]} focuses on the cross-influence of a training instance over a test instance, instead of self-influence nor its implication on optimization.
To this end, we introduce the memorization score (i.e. self-influence)—as a unified and computationally efficient metric—on both real data and random data to estimate the difficulty of training instances; such efficient estimation allows us to study the interpolation between memorization, optimization, and its implication on generalization.
