@inproceedings{
torrisi2023do,
title={Do chemical language models provide a better compound representation?},
author={Mirko Torrisi and Saeid Asadollahi and Antonio De la Vega de Leon and Kai Wang and Wilbert Copeland},
booktitle={NeurIPS 2023 Workshop on New Frontiers of AI for Drug Discovery and Development},
year={2023},
url={https://openreview.net/forum?id=yOmTE8XCyF}
}
In recent years, several chemical language models have been developed, inspired by the success of protein language models and advancements in natural language processing. In this study, we explore whether pre-training a chemical language model on billion-scale compound datasets, such as Enamine and ZINC20, can lead to improved compound representation in the drug space. We compare the learned representations of these models with de the facto standard compound representation, and evaluate their potential application in drug discovery and development by benchmarking them on biophysics, physiology, and physical chemistry datasets. Our findings suggest that the conventional masked language modeling approach on these extensive pre-training datasets is insufficient in enhancing compound representations. This highlights the need for additional physicochemical inductive bias in the modeling beyond scaling the dataset size.
Do chemical language models provide a better compound representation?
Mirko Torrisi Saeid Asadollahi Antonio De la Vega de Leon Kai Wang Wilbert Copeland