@inproceedings{
soen2024tradeoffs,
title={Trade-Offs of Diagonal Fisher Information Matrix Estimators},
author={Alexander Soen and Ke Sun},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=TVbCKAqoD8}
}
The Fisher information matrix can be used to characterize the local geometry of
the parameter space of neural networks. It elucidates insightful theories and
useful tools to understand and optimize neural networks. Given its high
computational cost, practitioners often use random estimators and evaluate only
the diagonal entries. We examine two popular estimators whose accuracy and sample
complexity depend on their associated variances. We derive bounds of the
variances and instantiate them in neural networks for regression and
classification. We navigate trade-offs for both estimators based on analytical
and numerical studies. We find that the variance quantities depend on the
non-linearity w.r.t. different parameter groups and should not be neglected when
estimating the Fisher information.
Trade-Offs of Diagonal Fisher Information Matrix Estimators
Alexander Soen Ke Sun