We consider the VLM image encoder in two parts: 1) the feature extractor \(\mathcal {F}(\cdot )\) , e.g. ResNet-50 [1]}, and 2) the last feature pooling layer \(\mathcal {P}(\cdot )\) , e.g. an attention pooling layer [1]}. We adopt the same backbone architecture as the image feature extractor \(\mathcal {F}(\cdot )\) , which allows us to directly use the frozen weights and inherit the rich semantic knowledge (see Fig. REF ). Along with the backbone initialization, we also adopt the same image pre-processing scheme as the VLM pretraining to maintain its open-vocabulary recognition ability. We use the last VLM pooling layer \(\mathcal {P}(\cdot )\)  for open-vocabulary recognition at test time only (see Sec. REF ). Building upon the frozen backbone features, we adopt Mask R-CNN [3]} head and feature pyramid network [4]} as the detector head following previous works [5]}, [6]}, [7]}. The detector head is randomly initialized and is the only trainable component of F-VLM. Despite the image-level pretraining, we found empirically that the frozen VLM backbone contains adequate locality sensitive features to enable accurate downstream detection (see Appendix ).
<FIGURE>