To better leverage the strong vision-category correspondence capability involved in the image-level CLIP model, we pursue mask proposal based semantic segmentation approaches such as MaskFormer [1]}, which first extracts a set of class-agnostic mask proposals and then classifies each mask proposal into a specific category. This two-stage approach decouples the semantic segmentation task into two sub-tasks of class-agnostic mask generation and mask category classification. Both sub-tasks prove well adaptable to handle novel categories: firstly, the class-agnostic mask proposal generation trained using seen categories is observed well generalizable to novel categories; secondly, the second mask proposal classification stage is at a same recognition granularity than that used in a CLIP model. To further bridge the gap with a CLIP model, the masked image crop of each proposal is used as input to the CLIP model for zero-shot categorization. In addition, we employ a prompt-learning approach [2]} to further improve the zero-shot classification accuracy given a pre-trained CLIP model.
