@inproceedings{
ren2024crossclass,
title={Cross-Class Domain Adaptive Semantic Segmentation with Visual Language Models},
author={Wenqi Ren and Ruihao Xia and Meng Zheng and Ziyan Wu and Yang Tang and Nicu Sebe},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=TjFn6xktTm}
}
This paper addresses the issue of cross-class domain adaptation (CCDA) in semantic segmentation, where the target domain contains both shared and novel classes that are either unlabeled or unseen in the source domain. This problem is challenging, as the absence of labels for novel classes hampers the accurate segmentation of both shared and novel classes. Since Visual Language Models (VLMs) are capable of generating zero-shot predictions without requiring task-specific training examples, we propose a label alignment method by leveraging VLMs to relabel pseudo labels for novel classes. Considering that VLMs typically provide only image-level predictions, we embed a two-stage method to enable fine-grained semantic segmentation and design a threshold based on the uncertainty of pseudo labels to exclude noisy VLM predictions. To further augment the supervision of novel classes, we devise memory banks with an adaptive update scheme to effectively manage accurate VLM predictions, which are then resampled to increase the sampling probability of novel classes. Through comprehensive experiments, we demonstrate the effectiveness and versatility of our proposed method across various CCDA scenarios.
Cross-Class Domain Adaptive Semantic Segmentation with Visual Language Models
Wenqi Ren Ruihao Xia Meng Zheng Ziyan Wu Yang Tang Nicu Sebe