Across all test items in v0.5, we collected 560,170 responses from SUTs in total (13 SUTs x 43,090 test items). LlamaGuard flagged 27,045 responses (4.8%) as unsafe. To verify LlamaGuard's accuracy, we selected a subset of the responses for human labelling (the "human eval set"). We used stratified sampling, rather than random sampling, to maximize coverage of different SUTs and hazard categories. First, from the flagged unsafe responses, we sampled a maximum of 10 responses from each combination of hazard category and SUT (e.g., from the responses of SUT 1 to the Violent Crimes test items). In total, we sampled 660 responses that were flagged as unsafe, ranging from 71 responses for Indiscriminate Weapons (CBRNE) test items to 130 responses for Hate test items; and from 16 responses for the SUT with the least unsafe responses to 70 for the SUT with the most unsafe response.21 We then sampled an equal number of responses not flagged as unsafe, with the same stratification for hazard category and SUT. The human eval set contains 1,320 responses (and prompts) in total.