In Figure REF , we compare identical models with anisotropic Sum, Mean and Max aggregation functions.
As baselines, we consider the Transformer encoder on full graphs [1]}, [2]} as well as a structure-agnostic MLP on each node, which can be instantiated by not using any aggregation function in Eq.(REF ), i.e. \(h_i^{\ell +1} = h_i^{\ell } + \text{ReLU} \left( \textsc {Norm} \left( U^\ell h_i^{\ell } \right) \right)\) .
We find that the choice of GNN aggregation function does not have an impact when evaluating models within the training size range TSP20-50.
As we tackle larger graphs, GNNs with aggregation functions that are agnostic to node degree (Mean and Max) are able to outperform Transformers and MLPs.
Importantly, the theoretically more expressive Sum aggregator [3]} generalizes worse than structure-agnostic MLPs, as it cannot handle the distribution shift in node degree and neighborhood statistics across graph sizes, leading to unstable or exploding node embeddings [4]}.
We use the Max aggregator in further experiments, as it generalizes well for both AR and NAR decoders (not shown).
