@inproceedings{
hanziwang2024qmoe,
title={Q-MoE: Connector for {MLLM}s with Text-Driven Routing},
author={Hanziwang and Jiamin Ren and Yifeng Ding and Lei Ren and Huixing Jiang and Chen Wei and Fangxiang Feng and Xiaojie Wang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=wNaXMRWPmq}
}
Multimodal Large Language Models (MLLMs) have showcased remarkable advances in handling various vision-language tasks. These models typically consist of a Large Language Model (LLM), a vision encoder and a connector structure, which is used to bridge the modality gap between vision and language. It is challenging for the connector to filter the right visual information for LLM according to
the task in hand. Most of the previous connectors, such as light-weight projection and Q-former, treat visual information for diverse tasks uniformly, therefore lacking task-specific visual information extraction capabilities. To address the issue, this paper proposes Q-MoE, a query-based connector with Mixture-of-Experts  (MoE) to extract task-specific information with text-driven routing. Furthermore,
an optimal path based training strategy is also proposed to find an optimal expert combination. Extensive experiments on two popular open-source LLMs and several different visual-language tasks demonstrate the effectiveness of the Q-MoE connecter. We will open our codes upon publication.
Q-MoE: Connector for MLLMs with Text-Driven Routing
Hanziwang Jiamin Ren Yifeng Ding Lei Ren Huixing Jiang Chen Wei Fangxiang Feng Xiaojie Wang