Assisting human evaluation.Recent studies have shown that in many cases, synthetic judgements from large-scale LMs (LLMs) can serve as qualified, fast, and low-cost alternatives to actual human evaluation (Gilardi et al., 2023a). Using GPT-4 as the judge, Alpaca Eval (Li et al., 2023b) and MT Bench (Zheng et al., 2023) are two popular benchmarks that measure the comprehensive abilities of LM-based ChatBot. In coding tasks, synthetic environment is a common choice to aid human evaluation, as humans can make the assessment more efficiently via actual executions and analysis on running logs. Gu et al. (2024) propose CRUXEval, a code execution reasoning benchmark consisting of 800 Python functions generated by CodeLAMA-34B. Similarly, Liu et al. (2024a) introduce CodeMind, a framework to gauge the code reasoning abilities of LLMs on Independent Execution Reasoning (IER), Dependent Execution Reasoning (DER), and Specification Reasoning (SR). All these evaluations based on synthetic data show strong correlation with real human judgements.