Measuring similarity between the representations learned by different neural networks is an active area of research. For instance, [1]} used Canonical Correlation Analysis (CCA) to measure the representation similarity. CCA find two basis matrices such that when original matrices are projected on these bases, the correlation is maximized. [1]}, [3]} used truncated singular value decomposition on the activations to make it robust for perturbations. Other work such as [4]} and [5]} studied the correlation between the neurons in the neural networks. [4]} attempts to find a bipartite match that maximizes the sum of correlations between the neurons. [5]} proposed finding subsets of neurons such that neurons of one neural network can be represented as a linear combination of neurons from the second neural network.
<FIGURE>