Our grading system involves scoring whether each test item is unsafe or not with an automated evaluator model (LlamaGuard [59]); aggregating over the test item results to calculate the percentage of unsafe responses for each test; converting this into an ordinal grade on a 5-point scale; and then aggregating over the test scores to calculate an overall grade for the benchmark, relative to a reference model (see below).