The analysis in [1]} presented below is akin to the one used in our ridge regression proof (cf. Appendix ), albeit with one simplifying modification. Instead of decomposing the excess risk into the stability and fitting terms, we use the following leave-one-out decomposition:
\(&\operatorname{\mathbb {E}}_{S_{n+1}} \left(\widehat{f}^{(-(n+1))}(X_{n+1}) - Y_{n+1}\right)^{2}- \inf _{\omega \in {\mathbb {R}}^{d}} R(\omega ) \\&= \operatorname{\mathbb {E}}_{S_{n+1}} \left( \frac{1}{n+1}\sum _{i=1}^{n+1} \left(\widehat{f}^{(-i)}(X_{i}) - Y_{i}\right)^{2} \right)- \inf _{\omega \in {\mathbb {R}}^{d}} \operatorname{\mathbb {E}}_{S_{n+1}} \left(\frac{1}{n+1} \sum _{i=1}^{n} \left(\langle \omega , X_{i} \rangle - Y_{i}\right)^{2}\right) \\&\le \operatorname{\mathbb {E}}_{S_{n+1}} \left( \frac{1}{n+1}\sum _{i=1}^{n+1} \left(\widehat{f}^{(-i)}(X_{i}) - Y_{i}\right)^{2} \right)- \operatorname{\mathbb {E}}_{S_{n+1}} \inf _{\omega \in {\mathbb {R}}^{d}} \left(\frac{1}{n+1} \sum _{i=1}^{n} \left(\langle \omega , X_{i} \rangle - Y_{i}\right)^{2}\right) \\&= \operatorname{\mathbb {E}}_{S_{n+1}} \left( \frac{1}{n+1}\sum _{i=1}^{n+1} \left(\widehat{f}^{(-i)}(X_{i}) - Y_{i}\right)^{2} \right)- \operatorname{\mathbb {E}}_{S_{n+1}} \left(\frac{1}{n+1} \sum _{i=1}^{n} \left(\langle \widehat{\omega }_{\infty }, X_{i} \rangle - Y_{i}\right)^{2}\right) \\&= \operatorname{\mathbb {E}}_{S_{n+1}} \left( \frac{1}{n+1}\sum _{i=1}^{n+1} \left(\widehat{f}^{(-i)}(X_{i}) - Y_{i}\right)^{2}- \left(\langle \widehat{\omega }_{\infty }, X_{i} \rangle - Y_{i}\right)^{2} \right) .\) 
