GraphMAE: Self-Supervised Masked Graph Autoencoders
Self-supervised learning (SSL) has been extensively explored in recent years.
Particularly, generative SSL has seen emerging success in natural language
processing and other fields, such as the wide adoption of BERT and GPT. Despite
this, contrastive learning--which heavily relies on structural data
augmentation and complicated training strategies--has been the dominant
approach in graph SSL, while the progress of generative SSL on graphs,
especially graph autoencoders (GAEs), has thus far not reached the potential as
promised in other fields. In this paper, we identify and examine the issues
that negatively impact the development of GAEs, including their reconstruction
objective, training robustness, and error metric. We present a masked graph
autoencoder GraphMAE that mitigates these issues for generative self-supervised
graph learning. Instead of reconstructing structures, we propose to focus on
feature reconstruction with both a masking strategy and scaled cosine error
that benefit the robust training of GraphMAE. We conduct extensive experiments
on 21 public datasets for three different graph learning tasks. The results
manifest that GraphMAE--a simple graph autoencoder with our careful
designs--can consistently generate outperformance over both contrastive and
generative state-of-the-art baselines. This study provides an understanding of
graph autoencoders and demonstrates the potential of generative self-supervised
learning on graphs.