As demonstrated by [1]} la is a scalable and well-behaved alternative to Bayesian nns if used post-hoc to approximate the intractable posterior over the weights after maximum-a-posteriori (map) training in classification and regression. The general utility of la has also motivated its use as an approximation to the marginal likelihood over nn weights. Recent methods, including [1]}, have explored this path to find model hyperparameters [3]} or learning invariances [4]}. However, its computational burden, for instance in Hessian matrices, has prescribed diagonal or Kronecker factored approximations [5]}, [6]}, [7]}, which are now widely used for second-order optimization. We provide more details on the connections to existing hessian based methods [1]}, [9]}, [10]}, Bayes by Backpropagation [11]} and Adam [12]} in sec:extendedrelatedworks.
