The discovery of concept cells in artificial networks raises a natural question — Can a multimodal model like CLIP explain the activity of brain regions known to possess concept cells better than a purely visual model? In this work we investigate this question by using publicly available fMRI data [1]}, and asking if CLIP can explain the activity of the hippocampus region better than a comparable feedforward visual model, i.e., ResNet. Because fMRI data does not provide us with the spatial resolution to identify individual concept cells, we address this question at the level of multi-dimensional representation spaces rather than at the level of individual neurons. We also extend our analysis to a variety of models from the literature, trained with unimodal or multimodal objectives. Using Representational Similarity Analysis (RSA) [2]}, we report that multimodal networks consistently rank higher than their unimodal counterparts in their ability to explain fMRI activity in the human hippocampus. We provide all the code for reproducing our results, and easily extending the analysis to other models in the future, on GithubCode available at: https://github.com/bhavinc/mutlimodal-concepts.
