Standardized benchmarks drive progress in machine learning.
However, with repeated testing, the risk of overfitting grows as algorithms over-exploit benchmark idiosyncrasies. In our work, we seek to mitigate this challenge by compilingever-expandinglarge-scale benchmarks calledLifelong Benchmarks. As exemplars of our approach, we createLifelong-CIFAR10andLifelong-ImageNet, containing (for now)  1.69M and 1.98M test samples, respectively. While reducing overfitting, lifelong benchmarks introduce a key challenge: the high cost of evaluating a growing number of models across an ever-expanding sample set. To address this challenge, we also introduce an efficient evaluation framework:Sort & Search (S&S) , which reuses previously evaluated models by leveraging dynamic programming algorithms to selectively rank and sub-select test samples, enabling cost-effective lifelong benchmarking. Extensive empirical evaluations across31,000 models demonstrate thatS&Sachieves highly-efficient approximate accuracy measurement, reducing compute cost from 180 GPU days to 5 GPU hours (1000x reduction)  on a single A100 GPU, with low approximation error. As such, lifelong benchmarks offer a robust, practical solution to the “benchmark exhaustion” problem.