To ensure all of the baseline models as well as the different AgentNet versions have a fair shot at solving the given tasks we perform a generous grid search for all of them. We consider batch size \(\in \lbrace 50, 300\rbrace \)  and hidden units \(\in \lbrace 64, 128\rbrace \)  and learning rate \(\in \lbrace 0.001, 0.0005, 0.0001\rbrace \) . For AgentNet we set number of agents \(k = n\)  (16 for 4-Cycles and 3-WL, 41 for Circular Skip Links) and consider \(\ell \in \lbrace 16, 64\rbrace \) . For other GNN architectures, we also include \(\ell \in \lbrace 4, 8\rbrace \)  as they tend to perform worse with high depth. On top of this for AgentNet we consider the number of agents \(k \in \lbrace 2, n\rbrace \)  as these problems should be solvable even with few agents. In fact, we did observe that having \(k=n\)  agents can make the model convergence slower and for 3-WL dataset results in mean accuracy of \(98\pm 2 \%\) .
We train each configuration using 10 random seeds and report the mean and the standard deviation of the best configuration after 10 thousand training steps. For the baseline models, we use the same training setup as for AgentNet. The baselines themselves were chosen as the most expressive models among their class of expressive GNNs. PPGN [1]} represents the higher-order GNNs and matches 3-WL in expressive power, GIN with random features [2]}, [3]} represents non-equivariant node identification, SMP [4]} represents the equivariant node identification scheme, while DropGNN [5]} represents the models that use many different versions of a graph to make a prediction [5]}, [7]}, [8]}.
