Some agent specific benchmarks like AgentBench evaluate language model-based agents in a variety of different environments such as web browsing, command-line interfaces, and video games [17]. This provides a better indication for how well agents can generalize to new environments, by reasoning, planning, and calling tools to achieve a given task. Benchmarks like AgentBench and SmartPlay introduce objective evaluation metrics designed to evaluate the implementation's success rate, output similarity to human responses, and overall efficiency [17; 30]. While these objective metrics are important to understanding the overall reliability and accuracy of the implementation, it is also important to consider more nuanced or subjective measures of performance. Metrics such as efficiency of tool use, reliability, and robustness of planning are nearly as important as success rate but are much more difficult to measure. Many of these metrics require evaluation by a human expert, which can be costly and time consuming compared to LLM-as-judge evaluations.