@inproceedings{
lv2024rethinking,
title={Rethinking the Effect of Uninformative Class Name in Prompt Learning},
author={Fengmao Lv and Changru Nie and Jianyang Zhang and Guowu Yang and Guosheng Lin and Xiao Wu and Tianrui Li},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=ENMl53MpKm}
}
Large pre-trained vision-language models like CLIP have shown amazing zero-shot recognition performance. To adapt pre-trained vision-language models to downstream tasks, recent studies have focused on the "learnable context + class name" paradigm, which learns continuous prompt contexts on downstream datasets. In practice, the learned prompt context tends to overfit the base categories and cannot generalize well to novel categories out of the training data. Recent works have also noticed this problem and have proposed several improvements. In this work, we draw a new insight based on empirical analysis, that is, uninformative class names lead to degraded base-to-novel generalization performance in prompt learning, which is usually overlooked by existing works. Under this motivation, we advocate to improve the base-to-novel generalization performance of prompt learning by enhancing the semantic richness of class names. We coin our approach as the Information Disengagement based Associative Prompt Learning (IDAPL) mechanism which considers the associative, meanwhile, decoupled learning of prompt context and class name embedding. IDAPL can effectively alleviate the phenomenon of learnable context overfitting to base classes, meanwhile, learning more informative semantic representation of base classes by fine-tuning the class name embedding, leading to improved performance on both base and novel classes. Experimental results on eleven widely used few-shot learning benchmarks clearly validate the effectiveness of our proposed approach. Code is available at https://github.com/tiggers23/IDAPL
Rethinking the Effect of Uninformative Class Name in Prompt Learning
Fengmao Lv Changru Nie Jianyang Zhang Guowu Yang Guosheng Lin Xiao Wu Tianrui Li