@inproceedings{
lin2024scalable,
title={Scalable Multi-Source Pre-training for Graph Neural Networks},
author={Mingkai Lin and Wenzhong Li and Xiaobin Hong and Sanglu Lu},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=rFRerAPdwI}
}
Graph Neural Networks (GNNs) have been shown as powerful tools in various scenarios, such as multimodal and multimedia. A fundamental approach, pre-training on available graphs and subsequently transferring the acquired knowledge to optimize downstream tasks with limited labels, was widely exploited to mitigate the demand for extensive labeled training data. However, previous works commonly assumed that pre-training and fine-tuning occur in the same or closely related domains that share similar feature/label spaces and graph distributions. A limitation is that for each individual graph without accessible pre-training data, a GNN must be trained from scratch, imposing high training overhead and hindering the ability of generalization. In this paper, we address the \emph{GNN multi-domain pre-training problem}, which intends to pre-train a transferable GNN model from heterogeneous multi-source graph domains and then apply it in an unseen one with minor fine-tuning costs. To this end, we propose a sca\underline{LA}ble \underline{M}ulti-source \underline{P}re-training (LAMP) method. For pre-training, LAMP presents a graph dual-distillation approach to distill massive knowledge from various graph domains to form synthetic homogeneous graphs. Simultaneously, high-level meta-knowledge from the synthetic graphs is extracted to train the GNN model, whose capability can be adjusted according to target graph contexts through a co-training modulation architecture. For fine-tuning, LAMP respectively aligns the target graph distribution, graph context, and graph task with the pretext so that the downstream task in the unseen domain can be reshaped to leverage the transferable knowledge efficiently. Extensive experiments on four real-world graph domain datasets demonstrate the superiority of LAMP, showcasing notable improvements in various downstream graph learning tasks. Our codes are publicly available on GitHub.
Scalable Multi-Source Pre-training for Graph Neural Networks
Mingkai Lin Wenzhong Li Xiaobin Hong Sanglu Lu