Recursively Updating the Hessian vs. Using an Uninformed Prior: In our formulation the first term of the RHS is the previous precision, which we “discount” with a forgetting term \(\alpha \)  that comes from the error inherited by moving away from the previous linearization. In the formulation of [1]} this is a prior that is assumed to be in the form \(\gamma ^2\mathbb {I}\) , where the scalar \(\gamma \)  is optimized at every step through maximization of the evidence (Eq. 6 in their paper). This evidence maximization comes with some drawbacks: (1) It tends to make the Laplace approximation overconfident to outliers. They partially address this issue by adding to the evidence an auxiliary term that depends on an ood dataset, penalizing it (Eq. 12 in their Appendix). (2) Besides inducing the avoidable need for an ood dataset, this technique have the same pitfall as vaes, namely uncertainty should be a derived quantity, not a learned one.
