Finally, following the categorization of approaches to understanding community (group or cluster) structure in AI models presented by Watanabe et al. [1]}, our overarching approach to finding class encoding patterns falls into the category
“Analysis of trained layered neural networks”
and combines two subcategories: analysis of unit outputs and their mutual relationships and analysis of the influence on neural network inference by data. Overall, our approach can be related to modular partitioning [2]}, [3]}, and unsupervised disentanglement of a learned representation [4]}, [5]}.
In [2]}, [3]}, the authors search for local specializations of AI model subgraphs by using spectral clustering of AI model computation graphs and introducing two metrics, such as accuracy changes during neuron pruning ablation (neuron importance) and class-specific accuracy drop in a subcluster of neurons (input-feature coherence).
In contrast to [2]}, [3]}, our clustering of computation units does not use "strong" and "weak" structural undirected connectivity of neurons as in spectral clustering, but, rather, repetitive co-occurrence patterns of utilization values in connected computation units.
From the perspective of representation disentanglement, two studies by Locatello et al. [4]}, [5]} perform extensive experiments on \(14\,000\)  models trained on eight datasets to conclude that well-disentangled models cannot be identified without supervision and the evaluation metrics do not always agree on what should be considered as “disentangled”. While we tacitly assume that high-dimensional data can be explained by lower dimensional semantically meaningful latent variables as by Locatello et al. [4]}, [5]}, we do not attempt to fully automate finding subgraphs (i.e., a human is always in the loop) to follow the conclusions by Locatello et al. [4]}, [5]}. In addition, we do not aim at fully partitioning all AI model computation graphs into poly-semantic and mono-semantic subgraphs based on the poly-semantic and mono-semantic classification of neurons in subgraphs according to Olah et al. [16]} and Räuker et al. [17]}.
