Over the past two years, AI safety has been an active and fast-growing area of research and practice [13], with a spate of new initiatives and projects that have sought to advance fundamental AI Safety research, policymaking, and development of practical tools, including the MLCommons AI Safety WG. Unsafe AI can lead to serious harm, ranging from the proliferation of highly persuasive scams and election disinformation to existential threats like biosurfare and rogue AI agents [14]. Further, because generative AI models are stochastic and their inner workings are not fully understood, AI systems cannot be simplistically 'ironclad' to protect against such risks.