@inproceedings{
huang2024lgcav,
title={{LG}-{CAV}: Train Any Concept Activation Vector with Language Guidance},
author={Qihan Huang and Jie Song and Mengqi Xue and Haofei Zhang and Bingde Hu and Huiqiong Wang and Hao Jiang and Xingen Wang and Mingli Song},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=MjD9Y05Q6i}
}
Concept activation vector (CAV) has attracted broad research interest in explainable AI, by elegantly attributing model predictions to specific concepts. However, the training of CAV often necessitates a large number of high-quality images, which are expensive to curate and thus limited to a predefined set of concepts. To address this issue, we propose Language-Guided CAV (LG-CAV) to harness the abundant concept knowledge within the certain pre-trained vision-language models (e.g., CLIP). This method allows training any CAV without labeled data, by utilizing the corresponding concept descriptions as guidance. To bridge the gap between vision-language model and the target model, we calculate the activation values of concept descriptions on a common pool of images (probe images) with vision-language model and utilize them as language guidance to train the LG-CAV. Furthermore, after training high-quality LG-CAVs related to all the predicted classes in the target model, we propose the activation sample reweighting (ASR), serving as a model correction technique, to improve the performance of the target model in return. Experiments on four datasets across nine architectures demonstrate that LG-CAV achieves significantly superior quality to previous CAV methods given any concept, and our model correction method achieves state-of-the-art performance compared to existing concept-based methods. Our code is available at https://github.com/hqhQAQ/LG-CAV.
LG-CAV: Train Any Concept Activation Vector with Language Guidance
Qihan Huang Jie Song Mengqi Xue Haofei Zhang Bingde Hu Huiqiong Wang Hao Jiang Xingen Wang Mingli Song