
Directly using the feature map generated by the CLIP vision encoder to perform pixel-wise classification. Note that in the original CLIP model, the feature of an image are represented by the feature of [cls] token, not the feature map, and this difference may lead to performance degradation. In addition, the original CLIP model uses the image size of \(224\times 224\)  during pre-training, while semantic segmentation usually requires a higher image resolution (e.g., shorter size is 640). Therefore, the direct use of high-resolution image during inference may lead to inferior performance due to inconsistency in image size. To alleviate this problem, we try to use the sliding window technique, which is widely used in previous works [1]} for performing multi-scale inference. We empirically found that it can improve performance(see tab:fcn).

The training data of the seen classes cannot be utilized in the first approach. Instead, we can retrain an FCN-based vision encoder on seen classes via the similar method introduced in Sec. REF . Specifically, we use the CLIP text encoder to generate a fixed classifier weight. Therefore, the retrained model can obtain a certain generalization ability to the unseen classes.

