The difference between a model's performance on the training data versus its performance on test data is known as the generalization gap. It is practically useful and theoretically impactful to predict a neural network's generalization gap. Most recent work does so by finding a set of features that is predictive of the generalization, e.g., by estimating data margins [1]}, [2]}, [3]}. Jiang et al. [4]} and Yak et al. [5]} demonstrate how the margin signatures of a neural network can predict the generalization gap with small errors. Besides, the network complexity and noise stability are also useful cues [6]}, [7]}, [1]}, [9]}. Our problem substantially differs from predicting the neural networks' generalization gap, which is concerned with the training and test data sets that share the same underlying distribution. We instead care about the results after fine-tuning a network's checkpoint.
