SUTs' performance on each **test item** is scored by automatically assessing whether their response is unsafe or not. We use LlamaGuard [59] for this purpose, which is an input-output safety filter built on top of the Llama 2 model [23]. It flags violations and provides additional flags for the type of safety risk (e.g. "Criminal Planning"). Although the training data is not available, we chose LlamaGuard as its weights are open and we can interrogate its design. In the future, we can fine-tune it to improve performance. We also considered the Perspective API20, which is widely used for content moderation and safety evaluation [43, 60, 61]. Perspective is popular because it is well-documented, reasonably robust, and scalable. However, it is limited by poor performance on some types of content, biases across social groups, and only a singular attribute that is widely used for assessments, namely "toxicity" [62, 63, 64].