Recall@\(k\) :
A practitioner may have resources to test up to \(k\)  checkpoints for their task of interest. We consider it a success if a measure ranks the highest-performance checkpoint into the top \(k\) . The measure's Recall@\(k\)  is the ratio between the number of downstream tasks on which it succeeds and the total number of tasks. We employ \(k=1\)  and \(k=3\)  in the experiments.
Top-\(k\)  relative accuracy (Rel@\(k\) ):
Given a task, a ranking measure returns an ordered list of the checkpoints. If the measure selects a high-performing checkpoint to the top \(k\)  despite that it misses the highest-performance one, we do not want to overly penalize it.
This Rel@\(k\)  is the ratio between the best fine-tuning accuracy on the downstream task with the top \(k\)  checkpoints and the the best fine-tuning accuracy with all the checkpoints.
Pearson correlation:
We incorporate Pearson's \(r\)  [1]} to compute the linear correlation between a measure' ranking scores \(\mathbf {R}_t\)  and the evaluation procedure's final accuracies \(\mathbf {G}_t\) .
Kendall ranking correlation:
We also include Kendall's \(\tau \)  [2]} to measure the ordinal association between a ranking measure \(\mathbf {R}\)  and the evaluation procedure \(\mathbf {G}\)  for each task. After all, what matter is the order of the checkpoints rather than the precise ranking scores.

