Chemical transformer compression for accelerating both training and inference of molecular modeling
Transformer models have been developed in molecular science with excellent
performance in applications including quantitative structure-activity
relationship (QSAR) and virtual screening (VS). Compared with other types of
models, however, they are large, which results in a high hardware requirement
to abridge time for both training and inference processes. In this work,
cross-layer parameter sharing (CLPS), and knowledge distillation (KD) are used
to reduce the sizes of transformers in molecular science. Both methods not only
have competitive QSAR predictive performance as compared to the original BERT
model, but also are more parameter efficient. Furthermore, by integrating CLPS
and KD into a two-state chemical network, we introduce a new deep lite chemical
transformer model, DeLiCaTe. DeLiCaTe captures general-domains as well as
task-specific knowledge, which lead to a 4x faster rate of both training and
inference due to a 10- and 3-times reduction of the number of parameters and
layers, respectively. Meanwhile, it achieves comparable performance in QSAR and
VS modeling. Moreover, we anticipate that the model compression strategy
provides a pathway to the creation of effective generative transformer models
for organic drug and material design.