Many language model benchmarks are designed to be solved in a single iteration, with no tool calls, such as MMLU or GSM8K [3; 10]. While these are important for measuring the abilities of base language models, they are not good proxies for agent capabilities because they do not account for agent systems' ability to reason over multiple steps or access outside information. StrategyQA improves upon this by assessing models' reasoning abilities over multiple steps, but the answers are limited to Yes/No responses [7]. As the industry continues to pivot towards agent focused use-cases additional measures will be needed to better assess the performance and generalizability of agents to tasks involving tools that extend beyond their training data.