@inproceedings{
zhang2024vecaf,
title={Ve{CAF}: Vision-language Collaborative Active Finetuning with Training Objective Awareness},
author={Rongyu Zhang and Zefan Cai and Huanrui Yang and Zidong Liu and Denis A Gudovskiy and Tomoyuki Okuno and Yohei Nakata and Kurt Keutzer and Baobao Chang and Yuan Du and Li Du and Shanghang Zhang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=NTpIYCW90q}
}
Finetuning a pretrained vision model (PVM) is a common technique for learning downstream vision tasks.
However, the conventional finetuning process with randomly sampled data points results in diminished training efficiency.
To address this drawback, we propose a novel approach, Vision-language Collaborative Active Finetuning (VeCAF). 
With the emerging availability of labels and natural language annotations of images through web-scale crawling or controlled generation, VeCAF makes use of these information to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence to meet the performance goal. This process is assisted by the inherent semantic richness of the text embedding space which we use to augment image features. 
Furthermore, the flexibility of text-domain augmentation allows VeCAF to handle out-of-distribution scenarios without external data.
Extensive experiments show the leading performance and high computational efficiency of VeCAF that is superior to baselines in both in-distribution and out-of-distribution image classification tasks. 
On ImageNet, VeCAF uses up to 3.3$\times$ less training batches to reach the target performance compared to full finetuning, and achieves an accuracy improvement of 2.8\% over the state-of-the-art active finetuning method with the same number of batches.
VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness
Rongyu Zhang Zefan Cai Huanrui Yang Zidong Liu Denis A Gudovskiy Tomoyuki Okuno Yohei Nakata Kurt Keutzer Baobao Chang Yuan Du Li Du Shanghang Zhang