Dimensionality reduction for large models. One potential problem for LTI is that the gradients \(\nabla _{\mathbf {w}}\ell (f_{\mathbf {w}}(\mathbf {x}_{\rm aux}), y_{\rm aux})\)  can be extremely high-dimensional. For example, both ResNet18 [1]} for vision tasks and a three-layer transformer [2]} for language tasks have approximately \(1.1\)  million trainable parameters. Such high-dimensional input to the model \(g_\theta \)  can lead to memory issues, as the first layer of the MLP would have \(11M\times h\)  parameters, where \(h\)  denotes the size of the first hidden layer.
