Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees
We consider transfer learning approaches that fine-tune a pretrained deep
neural network on a target task. We investigate generalization properties of
fine-tuning to understand the problem of overfitting, which often happens in
practice. Previous works have shown that constraining the distance from the
initialization of fine-tuning improves generalization. Using a PAC-Bayesian
analysis, we observe that besides distance from initialization, Hessians affect
generalization through the noise stability of deep neural networks against
noise injections. Motivated by the observation, we develop Hessian
distance-based generalization bounds for a wide range of fine-tuning methods.
Next, we investigate the robustness of fine-tuning with noisy labels. We design
an algorithm that incorporates consistent losses and distance-based
regularization for fine-tuning. Additionally, we prove a generalization error
bound of our algorithm under class conditional independent noise in the
training dataset labels. We perform a detailed empirical study of our algorithm
on various noisy environments and architectures. For example, on six image
classification tasks whose training labels are generated with programmatic
labeling, we show a 3.26% accuracy improvement over prior methods. Meanwhile,
the Hessian distance measure of the fine-tuned network using our algorithm
decreases by six times more than existing approaches.