Rethinking the Openness of CLIP
Contrastive Language-Image Pre-training (CLIP) has demonstrated great
potential in realizing open-vocabulary image classification in a matching
style, because of its holistic use of natural language supervision that covers
unconstrained real-world visual concepts. However, it is, in turn, also
difficult to evaluate and analyze the openness of CLIP-like models, since they
are in theory open to any vocabulary but the actual accuracy varies. To address
the insufficiency of conventional studies on openness, we resort to an
incremental view and define the extensibility, which essentially approximates
the model's ability to deal with new visual concepts, by evaluating openness
through vocabulary expansions. Our evaluation based on extensibility shows that
CLIP-like models are hardly truly open and their performances degrade as the
vocabulary expands to different degrees. Further analysis reveals that the
over-estimation of openness is not because CLIP-like models fail to capture the
general similarity of image and text features of novel visual concepts, but
because of the confusion among competing text features, that is, they are not
stable with respect to the vocabulary. In light of this, we propose to improve
the openness of CLIP from the perspective of feature space by enforcing the
distinguishability of text features. Our method retrieves relevant texts from
the pre-training corpus to enhance prompts for inference, which boosts the
extensibility and stability of CLIP even without fine-tuning.