@inproceedings{
yi2024learning,
title={Learning Spectral-decomposited Tokens for Domain Generalized Semantic Segmentation},
author={Jingjun Yi and Qi Bi and Hao Zheng and Haolan Zhan and Wei Ji and Yawen Huang and Yuexiang Li and Yefeng Zheng},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=HbIxwfMJ4W}
}
The rapid development of Vision Foundation Model (VFM) brings superior out-domain generalization for a variety of down-stream tasks.
Among them, domain generalized semantic segmentation (DGSS) holds unique challenges as the cross-domain images share common pixel-wise content information (i.e., semantics) but vary greatly in terms of the style variation (e.g., urban landscape, environment dependencies).  
How to effectively fine-tune VLM for DGSS has recently become an open research topic for the vision community.
In this paper, we present a novel Spectral-decomposited Tokens (SET) learning framework to push the frontier.
Delving into further than existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on how to learn style-invariant features from these learnable tokens. 
Specifically, the frozen VLM features are first decomposited into the phase and amplitude component respectively in the frequency space, where the phase / amplitude component reflects more on the content / style, respectively. 
Then, learnable tokens are adapted to learn the content and style, respectively.
As the cross-domain differences mainly rest in the style from the amplitude component, such information is decoupled from the tokens.
Consequently, the refined feature maps are more stable to represent the pixel-wise content despite the style variation. 
Extensive cross-domain experiments under a variety of backbones and VFMs show the state-of-the-art performance.
We will make the source code publicly available.
Learning Spectral-decomposited Tokens for Domain Generalized Semantic Segmentation
Jingjun Yi Qi Bi Hao Zheng Haolan Zhan Wei Ji Yawen Huang Yuexiang Li Yefeng Zheng