To support the v0.5 benchmark, MLCommons has developed an open-source evaluation tool, which consists of the ModelBench benchmark runner (which can be used to implement the benchmark) and the ModelGauge test execution engine (which contains the actual test items). This tool enables standardized, reproducible benchmark runs using versioned tests and SUTs. The tool is designed with a modular plug-in architecture, allowing model providers to easily implement and add new SUTs to the platform for evaluation. As the AI Safety Benchmark evolves, new versions of tests will be added to the platform. Details on how to access and use the platform can be found in the ModelBench Git repository on GitHub.6 ModelBench and ModelGauge were developed in collaboration with the Holistic Evaluation of Language Models [12] team at the Stanford Center for Research on Foundation Models (CRFM), and build upon the HELM team's experience of creating a widely-adopted open-source model evaluation framework for living leaderboards.