Using Eq. (REF ) and (REF ) we can now compute the image-space gradient \(\frac{\delta I}{\partial p}\) .
Assuming that the camera model is differentiable, the desired partial derivatives (REF ) are computed with the chain rule.
However, after implementing a direct bundle adjustment with perceptual loss, we found that the gradients in Eq. (REF ) are not accurate.
In some extreme cases, we even observed divergence of the optimization.
We believe that this problem comes from high variations inside perceptual loss paired with the large step size \(h=1\)  in gradient computation.
Other work may have found similar results as they mention that gradient computation is not viable for one-pixel points [1]} and has to be skipped [2]}.
