@inproceedings{
zhao2024coplparameterefficient,
title={Co{PL}:Parameter-Efficient Collaborative Prompt Learning for Audio-Visual Tasks},
author={Yihan Zhao and Wei Xi and Cui Yuhang and Gairui Bai and Xinhui Liu and Jizhong Zhao},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=adqDa13uXh}
}
Parameter-Efficient Fine Tuning (PEFT) has been demonstrated to be effective and efficient for transferring foundation models to downstream tasks. Transferring pretrained uni-modal models to multi-modal downstream tasks helps alleviate substantial computational costs for retraining multi-modal models. However, existing approaches primarily focus on multi-modal fusion, while neglecting the modal-specific fine-tuning, which is also crucial for multi-modal tasks. To this end, we propose parameter-efficient $Co$llaborative $P$rompt $L$earning ($CoPL$) to fine-tune both uni-modal and multi-modal features. Specifically, the collaborative prompts consist of modal-specific prompts and modal-interaction prompts. The modal-specific prompts are tailored for fine-tuning each modality, while the modal-interaction prompts are customized to explore inter-modality association. Furthermore, prompt bank-based mutual coupling is introduced to extract instance-level features, further enhancing the model's generalization ability. Extensive experimental results demonstrate that our approach achieves comparable or higher performance on various audio-visual downstream tasks while utilizing approximately 1% extra trainable parameters.
CoPL:Parameter-Efficient Collaborative Prompt Learning for Audio-Visual Tasks