Directed evaluationDirected evaluation involves principled and clearly defined evaluation of models for known risks. Typically, models are tested against a set of clearly defined prompts that have been assigned to a clear set of categories and subcategories. Benchmarks and evaluation suites are typically directed evaluation, such as [30, 31, 152, 153]. Another form of directed evaluation is testing models' Natural Language Understanding for toxic content, which involves using LMs as zero-shot or few-shot classifiers to assess whether user-generated content is a violation of safety policies. If models are good at this task, it indicates that they have a strong natural language understanding of hazardous content [154], and therefore have the potential to be safe. The primary benefit of directed evaluation is that the results are highly interpretable and standardized, which enables us to make comparisons across time and across models. However, one limitation is that since the tests are not tailored to the characteristics or capabilities of the individual models, they may not fully challenge or evaluate the unique aspects of each model. Further, it takes time to develop, release and update directed evaluation test sets, which risks them going out of date given the rapid pace of AI development [155].