However, benchmarks have limitations, such as being misleading and motivating narrow research goals [162]. In particular, they risk becoming saturated after a period of time if models can overfit to them [155]. Some benchmarks have also been criticized for low ecological validity, as their component tests do not closely approximate real-world data [163, 164]. Therefore, constructing more ecologically valid benchmarks that generalize to real-world scenarios is an active area of research [19]. Notably, several projects have sought to rethink benchmarking in order to make it more challenging and valid, such as Dynabench [165], which uses human-and-model-in-the-loop evaluation. We aim to take these limitations and concerns into account as we develop our benchmark.