Expanding \(J_{{\theta }_l}f_{\theta }(\mathbf {x})\)  with the chain rule, one realises that the Jacobian can be computed as a function of the Jacobian of the next layer. fig:backprophessian illustrate that an intermediate quantity \(M\) , which is initialised as \(\nabla ^2_{\mathbf {x}_\text{rec}}\mathcal {L}(\mathbf {x}_\text{rec})\) , can be efficiently backpropagated through multiplication with the Jacobian w.r.t. input of each layer.
This process leads to a block diagonal approximation of the Hessian as illustrated in fig:ggnmethods(a). However, diagonal blocks are generally too large to store and invert. To combat this, each block can be further approximated by its exact diagonal [1]} as depicted in fig:ggnmethods(b). This scales linearly w.r.t. parameters, but still scales quadratically w.r.t. the output resolution (tab:computationalcost).
<FIGURE><FIGURE>