@inproceedings{
wald2024decoupling,
title={Decoupling Semantic Similarity from Spatial Alignment for Neural Networks.},
author={Tassilo Wald and Constantin Ulrich and Priyank Jaini and Gregor Koehler and David Zimmerer and Stefan Denner and Fabian Isensee and Michael Baumgartner and Klaus Maier-Hein},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=ypFgcT147Z}
}
What representation do deep neural networks learn? How similar are images to each other for neural networks? Despite the overwhelming success of deep learning methods key questions about their internal workings still remain largely unanswered, due to their internal high dimensionality and complexity. To address this, one approach is to measure the similarity of activation responses to various inputs.
Representational Similarity Matrices (RSMs) distill this similarity into scalar values for each input pair.
These matrices encapsulate the entire similarity structure of a system, indicating which input lead to similar responses.
While the similarity between images is ambiguous, we argue that the spatial location of semantic objects does neither influence human perception nor deep learning classifiers. Thus this should be reflected in the definition of similarity between image responses for computer vision systems. Revisiting the established similarity calculations for RSMs we expose their sensitivity to spatial alignment. In this paper we propose to solve this through _semantic RSMs_, which are invariant to spatial permutation. We measure semantic similarity between input responses by formulating it as a set-matching problem. Further, we quantify the superiority of  _semantic_ RSMs over _spatio-semantic_ RSMs through image retrieval and by comparing the similarity between representations to the similarity between predicted class probabilities.
Decoupling Semantic Similarity from Spatial Alignment for Neural Networks.
Tassilo Wald Constantin Ulrich Priyank Jaini Gregor Koehler David Zimmerer Stefan Denner Fabian Isensee Michael Baumgartner Klaus Maier-Hein