Graph Neural Networks as Gradient Flows
Dynamical systems minimizing an energy are ubiquitous in geometry and
physics. We propose a gradient flow framework for GNNs where the equations
follow the direction of steepest descent of a learnable energy. This approach
allows to explain the GNN evolution from a multi-particle perspective as
learning attractive and repulsive forces in feature space via the positive and
negative eigenvalues of a symmetric "channel-mixing" matrix. We perform
spectral analysis of the solutions and conclude that gradient flow graph
convolutional models can induce a dynamics dominated by the graph high
frequencies which is desirable for heterophilic datasets. We also describe
structural constraints on common GNN architectures allowing to interpret them
as gradient flows. We perform thorough ablation studies corroborating our
theoretical analysis and show competitive performance of simple and lightweight
models on real-world homophilic and heterophilic datasets.