In the long term, publishing test items can compromise a benchmark's integrity and usefulness. One well-established concern is that the dataset could appear in web-scraped corpora used to train models [27, 28, 29]. This means that models could just regurgitate the correct answers and score highly on the AI Safety Benchmark, even if they still have critical safety weaknesses. Alternatively, model providers could choose to intentionally optimize their models to perform well against the benchmark. For instance, the UK AISI states that details of its methodology are "kept confidential to prevent the risk of manipulation if revealed."7. By keeping the exact evaluation dataset hidden but providing clear definitions of the targeted hazards, model developers can be incentivized to focus on holistically improving the safety of their models, rather than overfitting to a known static test set. However, the benefits of hidden evaluation need to be considered against the lack of trust that might be created, as well as possible missed opportunities to improve understanding and knowledge of AI safety within the community.