We applied RSA to study the ability of different neural network models – multi- or uni-modal – to explain the fMRI activity patterns in various brain regions. Based on recent findings [1]}, our hypothesis was that CLIP (and similar multimodal networks) would be specifically adept at explaining brain activity in the hippocampus–where `concept cells' are found. This hypothesis was supported by the data: the multimodal nature of a model was as a key component in explaining the activity in the human hippocampus – a trend that proved robust to different methods of voxel selection and distance metrics.
