@inproceedings{
chen2023understanding,
title={Understanding the Vulnerability of {CLIP} to Image Compression},
author={Cangxiong Chen and Vinay P. Namboodiri and Julian Padget},
booktitle={R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models},
year={2023},
url={https://openreview.net/forum?id=eLmjmG39KP}
}
CLIP is a widely used foundational vision-language model that is used for zero-shot image recognition and other image-text alignment tasks. We demonstrate that CLIP is vulnerable to change in image quality under compression. This surprising result is further analysed using an attribution method-Integrated Gradients. Using this attribution method, we are able to better understand both quantitatively and qualitatively exactly the nature in which the compression affects the zero-shot recognition accuracy of this model. We evaluate this extensively on CIFAR-10 and STL-10. Our work provides the basis to understand this vulnerability of CLIP and can help us develop more effective methods to improve the robustness of CLIP and other vision-language models.
Understanding the Vulnerability of CLIP to Image Compression
Cangxiong Chen Vinay P. Namboodiri Julian Padget