To investigate how the DNN processes the input data in general, the model-internal representations of the data can be analyzed.
To this end, the hidden layer activations for the complete data set or a subset of it are computed and further evaluated in-depth.
For example, training linear models to classify the representations in a hidden layer indicates how well particular properties are encoded in this layer.
The classification targets of the linear models can be the targets of the DNN itself [1]} or any user-defined property by providing groups of inputs [2]}.
A more general approach than investigating linear separability is to analyze the representational similarity between groups of related examples.
Previous work used PCA [3]}, CCA [4]} or clustering techniques [5]}, [6]} to find co-activated neurons or to compare representations of different groups of examples.
Our introduced technique aims to analyze activations, as well, but in contrast to existing approaches, it allows to compare high-dimensional representations by visual inspection.
Explaining DNN by investigating their activations only gives insight into model behavior for groups that are contained in the used test data.
