Recently, [1]}, [2]}, [3]} and other authors suggest that for real-world problems, the learning objective often has a power-law \(\sim m^{-\alpha }\)  dependence of training set size \(m\) , where the exponent \(\alpha \)  is a constant that usually independent from \(m\) . A surprising finding is that, for certain triplets \(( \mathcal {M}, I)\)  , \(\alpha \)  can grow as \(m\)  becomes larger, i.e., data improves data efficiency (DIDE); see Fig. REF . In what follows, we first examine the learning curve of \(\mathsf {VEC}_n\)  to better understand the huge performance gap between \(\mathsf {VEC}_n\)  and \(\mathsf {VEC}_\infty \) . We then move to SoTA models, in which we observe a “cusp” in the learning curve. Finally, we provide possible explanations for the observed phenomena.
