In this work, we propose to leverage a recent advance of image-level vision-language learning model, i.e., CLIP [1]}, to address this issue, which has learnt a strong vision-category alignment model using rich image-caption data. A main difficulty in leveraging this model lies in that the CLIP model is learnt at image-level, which is different in granularity than that of the pixel-level semantic segmentation task. One way to bridge this granularity difference is to reformulate the image-level CLIP model to a pixel-level one such that an alignment between pixel-level vision features and the text feature can be converted from the image-level alignment. This pixel-level vision-category alignment model is then leveraged for the FCN based zero-shot semantic segmentation. However, the pixel-level alignment performs unsatisfactorily due to that the model is learnt in image-level.
