@inproceedings{
rajendran2024from,
title={From Causal to Concept-Based Representation Learning},
author={Goutham Rajendran and Simon Buchholz and Bryon Aragam and Bernhard Sch{\"o}lkopf and Pradeep Kumar Ravikumar},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=r5nev2SHtJ}
}
To build intelligent machine learning systems, modern representation learning attempts to recover latent generative factors from data, such as in causal representation learning. A key question in this growing field is to provide rigorous conditions under which latent factors can be identified and thus, potentially learned. Motivated by extensive empirical literature on linear representations and concept learning, we propose to relax causal notions with a geometric notion of concepts. We formally define a notion of concepts and show rigorously that they can be provably recovered from diverse data. Instead of imposing assumptions on the "true" generative latent space, we assume that concepts can be represented linearly in this latent space. The tradeoff is that instead of identifying the "true" generative factors, we identify a subset of desired human-interpretable concepts that are relevant for a given application. Experiments on synthetic data, multimodal CLIP models and large language models supplement our results and show the utility of our approach. In this way, we provide a foundation for moving from causal representations to interpretable, concept-based representations by bringing together ideas from these two neighboring disciplines.
From Causal to Concept-Based Representation Learning
Goutham Rajendran Simon Buchholz Bryon Aragam Bernhard Schölkopf Pradeep Kumar Ravikumar