@inproceedings{
guo2024llavaultra,
title={{LL}a{VA}-Ultra: Large Chinese Language and Vision Assistant for Ultrasound},
author={Xuechen Guo and Wenhao Chai and Shi-Yan Li and Gaoang Wang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=7ZYEoB71Vd}
}
Multimodal Large Language Model (MLLM) has recently garnered significant attention as a prominent research focus. By harnessing the capability of powerful Large Language Model (LLM), it facilitates the transition of conversational generative AI from unimodal text to performing multimodal tasks. This blooming development has begun to significantly impact the medical field. However, visual language models in the general domain lack sophisticated comprehension required for medical visual conversations. Even some models specifically tailored for the medical domain often produce answers that tend to be vague and weakly related to the visual contents. In this paper, we propose a fine-grained and adaptive visual language model architecture for Chinese medical visual conversations through parameter-efficient tuning. Specifically, we devise a fusion module with fine-grained vision encoders to achieve enhancement for subtle medical visual semantics. Then we note data redundancy that is common in medical scenes but ignored in most prior works. In cases of a single text paired with multiple figures, we utilize weighted scoring with knowledge distillation to adaptively screen valid images mirroring text descriptions. For execution, we leverage a large-scale Chinese ultrasound multimodal dataset obtained first-hand from the hospital database. We create instruction-following data based on text derived from doctors, which ensures professionality and thus contributes to effective tuning. With enhanced architecture and quality data, our Large Chinese Language and Vision Assistant for Ultrasound (LLaVA-Ultra) shows strong capability and robustness to medical scenarios. On three medical visual question answering datasets, LLaVA-Ultra surpasses previous state-of-the-art models on various metrics.
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
Xuechen Guo Wenhao Chai Shi-Yan Li Gaoang Wang