Many research teams introduce their own unique agent benchmarks alongside their agent implementation which makes comparing multiple agent implementations on the same benchmark challenging. Additionally, many of these new agent-specific benchmarks include a hand-crafted, highly complex, evaluation set where the results are manually scored [2]. This can provide a high-quality assessment of a method's capabilities, but it also lacks the robustness of a larger dataset and risks introducing bias into the evaluation, since the ones developing the method are also the ones writing and scoring the results. Agents can also have problems generating a consistent answer over multiple iterations, due to variability in the models, environment, or problem state. This added randomness poses a much larger problem to smaller, complex evaluation sets.