@inproceedings{
zhang2023zeroshot,
title={Zero-shot Improvement of Object Counting with {CLIP}},
author={Ruisu Zhang and Yicong Chen and Kangwook Lee},
booktitle={R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models},
year={2023},
url={https://openreview.net/forum?id=AJiBZ1BPH5}
}
We focus on the object counting limitations of vision-language models, with a particular emphasis on Contrastive Language-Image Pre-Training (CLIP) models. We assess the counting performance of CLIP using a custom dataset, which uncovers significant variations across diverse objects. To address this, we introduce a zero-shot, training-free method aimed at improving counting accuracy by manipulating the text embedding space of CLIP. Through comprehensive experiments, we demonstrate that our method not only enhances the counting capabilities of CLIP but also boosts the performance of text-to-image generative models like Stable Diffusion, particularly in generating images with precise object counts.
Zero-shot Improvement of Object Counting with CLIP
Ruisu Zhang Yicong Chen Kangwook Lee