@inproceedings{
zhou2023multihead,
title={Multi-head {CLIP}: Improving {CLIP} with Diverse Representations and Flat Minima},
author={Mo Zhou and Xiong Zhou and Li Erran Li and Stefano Ermon and Rong Ge},
booktitle={OPT 2023: Optimization for Machine Learning},
year={2023},
url={https://openreview.net/forum?id=4jEOdGqVC0}
}
Contrastive Language-Image Pre-training (CLIP) has shown remarkable success in the field of multimodal learning by enabling joint understanding of text and images. In this paper, we introduce a novel method called Multi-head CLIP, inspired by Stein Variational Gradient Descent (SVGD) and Sharpness-aware Minimization (SAM). Our approach aims to enhance CLIP's learning capability by encouraging the model to acquire diverse features while also promoting convergence towards a flat loss region, resulting in improved generalization performance. We conduct extensive experiments on two benchmark datasets, YFCC15M and CC3M, to evaluate the effectiveness of our proposed method. The experimental results consistently demonstrate that multi-head CLIP outperforms both the original CLIP architecture and CLIP with the SAM optimizer.
Multi-head CLIP: Improving CLIP with Diverse Representations and Flat Minima
Mo Zhou Xiong Zhou Li Erran Li Stefano Ermon Rong Ge