For the experiments involving neural networks, we use synthetic datasets very similar to the linear network experiment in section L.3 of [1]}.
For a neural network as described above with dimension \(d\) , we consider the problem of mapping \(n\)  inputs \({\mathbf {x}}_1, \dots , {\mathbf {x}}_n\in \mathbb {R}^d\)  to \(n\)  outputs \({\mathbf {y}}_1, \dots , {\mathbf {y}}_n\in \mathbb {R}^d\) . Let \({\mathbf {X}}\in \mathbb {R}^{d\times d}\)  and \({\mathbf {Y}}\in \mathbb {R}^{d\times d}\)  denote the vertically stack inputs and outputs respectively. We generate \({\mathbf {X}}\)  as a whitened matrix such that \({\mathbf {X}}{\mathbf {X}}^T = d{\mathbf {I}}_d\)  and generate \(Y\)  by \({\mathbf {Y}}={\mathbf {X}}{\mathbf {A}}\)  where \({\mathbf {A}}= \text{diag}(1, \frac{d-1}{d},\dots , \frac{2}{d}, \frac{1}{d})\) .
