Of the 660 items in the human eval set categorised as unsafe (based on the classifications provided by LlamaGuard), 358 are unsafe (based on the annotators' gold standard labels) and 302 are not unsafe. Accuracy is 54.2%. Of the 660 items in the human eval set categorised as not unsafe (based on the classifications provided by LlamaGuard), 88 are unsafe (based on the annotators' gold standard labels) and 572 are not unsafe. Accuracy is 86.7%. Overall, the accuracy of LlamaGuard is 70.4% on the human eval set. Due to the relatively small size of the human eval set, we do not analyse more finegrained categories, such as the performance of LlamaGuard on specific hazard categories or SUTs. The heavily imbalanced nature of the full dataset of 560,170 responses (of which only 533,125 or 95.2% are labelled Not Unsafe), and the better performance of the evaluator model on responses categorised as not unsafe, means that accuracy of LlamaGuard on the full dataset is higher than the eval set. If we weight the full dataset by the distribution of responses, accuracy is 85.6% (based on 86.7% x 95.2% plus 54.2% x 4.8%). Overall, although performance could be improved, LlamaGuard is sufficiently accurate at identifying unsafe and not unsafe responses to be used for the v0.5 Benchmark. We aim to improve performance of the evaluator model in future releases.