@inproceedings{
chen2024learning,
title={Learning A Low-Level Vision Generalist via Visual Task Prompt},
author={Xiangyu Chen and Yihao Liu and Yuandong Pu and Wenlong Zhang and Jiantao Zhou and Yu Qiao and Chao Dong},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=tgSl2DkJCQ}
}
Building a unified model for general low-level vision tasks has important research and practical value. However, existing methods still face challenges when dealing with diverse low-level vision problems. Multi-task restoration approaches can simultaneously address various degradation-to-clean restoration tasks, while their applicability to tasks with different target domains (e.g., image stylization) remains limited. Existing methods like PromptGIP that can handle tasks with multiple input-target domains mainly rely on the Masked Autoencoder (MAE) training paradigm. Unfortunately, these approaches are restricted by coupling to the ViT architecture, resulting in suboptimal image reconstruction quality. In addition, they tend to be sensitive to prompt content and often fail when handling more tasks that involve low-frequency information processing, such as color and style. In this paper, we present a Visual task Prompt-based Image Processing (VPIP) framework to address the above challenges. This framework employs the visual task prompt to process tasks with different input-target domains. Besides, it provides the flexibility to select a backbone network suitable for various low-level vision tasks. A prompt cross-attention mechanism is introduced to deal with the information interaction between the input and prompt information. Based on the VPIP framework, we train a low-level vision generalist model, namely GenLV, on 30 diverse tasks. Experimental results show that GenLV can successfully address a variety of low-level tasks, and it significantly outperforms existing methods both quantitatively and qualitatively.
Learning A Low-Level Vision Generalist via Visual Task Prompt
Xiangyu Chen Yihao Liu Yuandong Pu Wenlong Zhang Jiantao Zhou Yu Qiao Chao Dong