@inproceedings{
gruetzemacher2023an,
title={An International Consortium for {AI} Risk Evaluations},
author={Ross Gruetzemacher and Alan Chan and {\v{S}}t{\v{e}}p{\'a}n Los and Kevin Frazier and Sim{\'e}on Campos and Matija Franklin and James Fox and Jose Hernandez-Orallo and Christin Manning and Philip Tomei and Kyle Kilian},
booktitle={Socially Responsible Language Modelling Research},
year={2023},
url={https://openreview.net/forum?id=HoIEKQhiRs}
}
Given rapid progress in AI and potential risks from next-generation frontier AI systems, the urgency to create and implement AI governance and regulatory schemes is apparent. A regulatory gap has permitted labs to conduct research, development, and deployment with minimal oversight or guidance. In response, frontier AI evaluations have been proposed as a way of assessing risks from the development and deployment of frontier AI systems. Yet, the budding AI risk evaluation ecosystem faces significant present and future coordination challenges, such as a limited diversity of evaluators, suboptimal allocation of effort, and races to the bottom. As a solution, this paper proposes an international consortium for AI risk evaluations, comprising both AI developers and third-party AI risk evaluators. Such a consortium could play a critical role in international efforts to mitigate societal-scale risks from advanced AI. In this paper, we discuss the current evaluation ecosystem and its problems, introduce the proposed consortium, review existing organizations performing similar functions in other domains, and, finally, we recommend concrete steps toward establishing the proposed consortium.
An International Consortium for AI Risk Evaluations
Ross Gruetzemacher Alan Chan Štěpán Los Kevin Frazier Siméon Campos Matija Franklin James Fox Jose Hernandez-Orallo Christin Manning Philip Tomei Kyle Kilian