@inproceedings{
gil2023holistic,
title={Holistic chemical evaluation reveals pitfalls in reaction prediction models},
author={Victor Sabanza Gil and Andres M Bran and Malte Franke and Jeremy S. Luterbacher and Philippe Schwaller},
booktitle={NeurIPS 2023 AI for Science Workshop},
year={2023},
url={https://openreview.net/forum?id=yLydB04RxR}
}
The prediction of chemical reactions has gained significant interest within the machine learning community in recent years, owing to its complexity and crucial applications in chemistry. However, model evaluation for this task has been mostly limited to simple metrics like top-k accuracy, which obfuscates fine details of a model's limitations. Inspired by progress in other fields, we propose a new assessment scheme that builds on top of current approaches, steering towards a more holistic evaluation. We introduce the following key components for this goal: ChORISO, a curated dataset along with multiple tailored splits to recreate chemically relevant scenarios, and a collection of metrics that provide a holistic view of a model's advantages and limitations. Application of this method to state-of-the-art models reveals important differences on sensitive fronts, especially stereoselectivity and chemical out-of-distribution generalization. Our work paves the way towards robust prediction models that can ultimately
accelerate chemical discovery.
Holistic chemical evaluation reveals pitfalls in reaction prediction models
Victor Sabanza Gil Andres M Bran Malte Franke Rémi Schlama Jeremy S. Luterbacher Philippe Schwaller