@inproceedings{
dinh2024hamiltonian,
title={Hamiltonian Monte Carlo on Re{LU} Neural Networks is Inefficient},
author={Vu C. Dinh and Lam Si Tung Ho and Cuong V. Nguyen},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=aBmiyi7iA7}
}
We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $\Omega(\epsilon)$ rather than the classical error rate of $\mathcal{O}(\epsilon^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks.
Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient
Vu C. Dinh Lam Si Tung Ho Cuong V. Nguyen