@inproceedings{
wang2024semflow,
title={SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow},
author={Chaoyang Wang and Xiangtai Li and Lu Qi and Henghui Ding and Yunhai Tong and Ming-Hsuan Yang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=E3P1X94Y51}
}
Semantic segmentation and semantic image synthesis are two representative tasks in visual perception and generation. While existing methods consider them as two distinct tasks, we propose a unified framework (SemFlow) and model them as a pair of reverse problems. Specifically, motivated by rectified flow theory, we train an ordinary differential equation (ODE) model to transport between the distributions of real images and semantic masks. As the training object is symmetric, samples belonging to the two distributions, images and semantic masks, can be effortlessly transferred reversibly. For semantic segmentation, our approach solves the contradiction between the randomness of diffusion outputs and the uniqueness of segmentation results. For image synthesis, we propose a finite perturbation approach to enhance the diversity of generated results without changing the semantic categories. Experiments show that our SemFlow achieves competitive results on semantic segmentation and semantic image synthesis tasks. We hope this simple framework will motivate people to rethink the unification of low-level and high-level vision.
SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow
Chaoyang Wang Xiangtai Li Lu Qi Henghui Ding Yunhai Tong Ming-Hsuan Yang