With this white paper, we introduce v0.5 of the **AI Safety Benchmark**. The benchmark is designed to assess the safety risks of AI systems that use chat-tuned Language Models (LMs).5 We focus on LMs as a tractable starting point because they have been extensively researched and are widely deployed in production, and several LM benchmarks already exist (e.g., HELM [19] and BIG-bench [20]). In the future, we will benchmark the safety risks of models for other modalities (such as image-to-text models, text-to-image models, and speech-to-speech models [21, 22]), and expand to LMs in languages other than English.