Similarity Scores –
Canonical Correlation Analysis (CCA) methods are proposed to estimate the representational similarity across different models ([1]}, [2]}).
Centered Kernel Alignment (CKA) is an improved extension of CCA ([3]}).
While these methods effectively quantify the degree of similarity, they commonly require expensive computations.
For example, SVCCA performs singular vector decomposition of the model and CKA computes Hilbert-Schmidt Independence Criterion multiple times ([4]}).
In addition, the representational similarity does not deliver any information regarding the gradient difference that is strongly related to the convergence property.
We will propose a practical metric for estimating the layer-wise model discrepancy, which is cheap enough to be used at run-time.
