NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search
Graph neural architecture search (GraphNAS) has recently aroused considerable
attention in both academia and industry. However, two key challenges seriously
hinder the further research of GraphNAS. First, since there is no consensus for
the experimental setting, the empirical results in different research papers
are often not comparable and even not reproducible, leading to unfair
comparisons. Secondly, GraphNAS often needs extensive computations, which makes
it highly inefficient and inaccessible to researchers without access to
large-scale computation. To solve these challenges, we propose NAS-Bench-Graph,
a tailored benchmark that supports unified, reproducible, and efficient
evaluations for GraphNAS. Specifically, we construct a unified, expressive yet
compact search space, covering 26,206 unique graph neural network (GNN)
architectures and propose a principled evaluation protocol. To avoid
unnecessary repetitive training, we have trained and evaluated all of these
architectures on nine representative graph datasets, recording detailed metrics
including train, validation, and test performance in each epoch, the latency,
the number of parameters, etc. Based on our proposed benchmark, the performance
of GNN architectures can be directly obtained by a look-up table without any
further computation, which enables fair, fully reproducible, and efficient
comparisons. To demonstrate its usage, we make in-depth analyses of our
proposed NAS-Bench-Graph, revealing several interesting findings for GraphNAS.
We also showcase how the benchmark can be easily compatible with GraphNAS open
libraries such as AutoGL and NNI. To the best of our knowledge, our work is the
first benchmark for graph neural architecture search.