@inproceedings{
chen2024ssatadapter,
title={{SSAT}-Adapter: Enhancing Vision-Language Model Few-shot Learning with Auxiliary Tasks},
author={Bowen Chen and Yun Sing Koh and Gillian Dobbie},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=uaKuvzR74K}
}
Traditional deep learning models often struggle in few-shot learning scenarios, where limited labeled data is available.
While the Contrastive Language-Image Pre-training (CLIP) model demonstrates impressive zero-shot capabilities, its performance in few-shot scenarios remains limited. Existing methods primarily aim to leverage the limited labeled dataset, but this offers limited potential for improvement. To overcome the limitations of small datasets in few-shot learning, we introduce a novel framework, SSAT-Adapter, that leverages CLIP's language understanding to generate informative auxiliary tasks and improve CLIP's performance and adaptability in few-shot settings. We utilize CLIP's language understanding to create decision-boundary-focused image latents. These latents form auxiliary tasks, including inter-class instances to bridge CLIP's pre-trained knowledge with the provided examples, and intra-class instances to subtly expand the representation of target classes. A self-paced training regime, progressing from easier to more complex tasks, further promotes robust learning. Experiments show our framework outperforms the state-of-the-art online few-shot learning method by an average of 2.2\% on eleven image classification datasets. Further ablation studies on various tasks demonstrate the effectiveness of our approach to enhance CLIP's adaptability in few-shot image classification.
SSAT-Adapter: Enhancing Vision-Language Model Few-shot Learning with Auxiliary Tasks
Bowen Chen Yun Sing Koh Gillian Dobbie