On the Prediction Instability of Graph Neural Networks
Instability of trained models, i.e., the dependence of individual node
predictions on random factors, can affect reproducibility, reliability, and
trust in machine learning systems. In this paper, we systematically assess the
prediction instability of node classification with state-of-the-art Graph
Neural Networks (GNNs). With our experiments, we establish that multiple
instantiations of popular GNN models trained on the same data with the same
model hyperparameters result in almost identical aggregated performance but
display substantial disagreement in the predictions for individual nodes. We
find that up to one third of the incorrectly classified nodes differ across
algorithm runs. We identify correlations between hyperparameters, node
properties, and the size of the training set with the stability of predictions.
In general, maximizing model performance implicitly also reduces model
instability.