GOOD: A Graph Out-of-Distribution Benchmark
Out-of-distribution (OOD) learning deals with scenarios in which training and
test data follow different distributions. Although general OOD problems have
been intensively studied in machine learning, graph OOD is only an emerging
area of research. Currently, there lacks a systematic benchmark tailored to
graph OOD method evaluation. In this work, we aim at developing an OOD
benchmark, known as GOOD, for graphs specifically. We explicitly make
distinctions between covariate and concept shifts and design data splits that
accurately reflect different shifts. We consider both graph and node prediction
tasks as there are key differences when designing shifts. Overall, GOOD
contains 8 datasets with 14 domain selections. When combined with covariate,
concept, and no shifts, we obtain 42 different splits. We provide performance
results on 7 commonly used baseline methods with 10 random runs. This results
in 294 dataset-model combinations in total. Our results show significant
performance gaps between in-distribution and OOD settings. Our results also
shed light on different performance trends between covariate and concept shifts
by different methods. Our GOOD benchmark is a growing project and expects to
expand in both quantity and variety of resources as the area develops. The GOOD
benchmark can be accessed via
$\href{https://github.com/divelab/GOOD/}{\text{https://github.com/divelab/GOOD/}}$.