The co-activation graph approach follows the idea of layouting a graph structure in which nodes and edges represent neurons and their activation similarity, respectively.
We first compute the pairwise Cosine similarity of neurons according to their NAP values.
We then create a graph with one node corresponding to each neuron.
For each of the \(7.5\%\)  most similar pairs of neurons, we draw an edge between the corresponding nodes in the graph.
We empirically choose the distance threshold based on the connectedness of the graphs for several MLP models and layers.
The resulting graph can have separated subsets of nodes, which leads to large gaps in the layout.
To avoid these gaps, we further ensure that the entire graph is connected.
To this end, we first identify all maximal subsets of nodes where a path exists between all nodes, called connected components.
Then, we link each smaller connected component to the largest one by drawing an edge between the most similar pair of neurons out of the two components.
Finally, we layout the connected graph with the force-directed Fruchterman Reingold algorithm [1]} from the NetworkX package [2]}.
From the obtained graph, we use the node coordinates as the topographic layout of the neurons.
For brevity, we refer to the co-activation graph technique as the “graph” method.
