TITLE: Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces 

ABSTRACT: In this work, we benchmark a variety of single- and multi-task graph neural
network (GNN) models against lower-bar and higher-bar traditional machine
learning approaches employing human engineered molecular features. We consider
four GNN variants -- Graph Convolutional Network (GCN), Graph Attention Network
(GAT), Message Passing Neural Network (MPNN), and Attentive Fingerprint
(AttentiveFP). So far deep learning models have been primarily benchmarked
using lower-bar traditional models solely based on fingerprints, while more
realistic benchmarks employing fingerprints, whole-molecule descriptors and
predictions from other related endpoints (e.g., LogD7.4) appear to be scarce
for industrial ADME datasets. In addition to time-split test sets based on
Genentech data, this study benefits from the availability of measurements from
an external chemical space (Roche data). We identify GAT as a promising
approach to implementing deep learning models. While all GNN models
significantly outperform lower-bar benchmark traditional models solely based on
fingerprints, only GATs seem to offer a small but consistent improvement over
higher-bar benchmark traditional models. Finally, the accuracy of in vitro
assays from different laboratories predicting the same experimental endpoints
appears to be comparable with the accuracy of GAT single-task models,
suggesting that most of the observed error from the models is a function of the
experimental error propagation.
In this work, we benchmark a variety of single- and multi-task graph neural
network (GNN) models against lower-bar and higher-bar traditional machine
learning approaches employing human engineered molecular features. We consider
four GNN variants -- Graph Convolutional Network (GCN), Graph Attention Network
(GAT), Message Passing Neural Network (MPNN), and Attentive Fingerprint
(AttentiveFP). So far deep learning models have been primarily benchmarked
using lower-bar traditional models solely based on fingerprints, while more
realistic benchmarks employing fingerprints, whole-molecule descriptors and
predictions from other related endpoints (e.g., LogD7.4) appear to be scarce
for industrial ADME datasets. In addition to time-split test sets based on
Genentech data, this study benefits from the availability of measurements from
an external chemical space (Roche data). We identify GAT as a promising
approach to implementing deep learning models. While all GNN models
significantly outperform lower-bar benchmark traditional models solely based on
fingerprints, only GATs seem to offer a small but consistent improvement over
higher-bar benchmark traditional models. Finally, the accuracy of in vitro
assays from different laboratories predicting the same experimental endpoints
appears to be comparable with the accuracy of GAT single-task models,
suggesting that most of the observed error from the models is a function of the
experimental error propagation.
Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces
http://arxiv.org/abs/2111.13964v1