Many authors have matched the \(O(d/n)\)  rate (REF ) for ERM-based algorithms under significantly less restrictive assumptions than that of a well-specified model with Gaussian covariates (e.g., assuming a favorable covariance structure and sub-Gaussian noise [1]}, assuming \(L_{q}\) –\(L_2\)  (for some \(q > 2\) ) moment equivalence of the marginals \(\langle \omega , X \rangle \)  and the noise random variable \(\xi \)  [2]}, [3]}, [4]}, [5]}, or the weaker small-ball assumption [6]}, [7]}). Moment equivalence type assumptions allow for modelling heavy-tailed distributions, and, in particular, they have played a crucial role in recent developments in the robust statistics literature (e.g., [4]}, [9]}, [10]}, [11]}); however, in some cases, such assumptions only hold with constants that can deteriorate arbitrarily with respect to the parameters of the unknown distribution \(P\) , even for light-tailed or bounded distributions. For instance, the smallest constant with respect to which \(\operatorname{Bernoulli}(p)\)  distribution satisfies the \(L_{4}\) –\(L_{2}\)  moment equivalence can get arbitrarily large for small \(p\) . In the context of linear regression, the work [4]} highlights that some of the prior results on the performance of least squares relying on such assumptions can have constants that may unintentionally depend on the dimension of the covariates \(d\) . Recent literature has further accentuated this problem and witnessed an emerging interest in refining moment equivalence and small-ball assumptions [13]}, [9]}, [15]}.
