@inproceedings{
chen2024efficiency,
title={Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Models},
author={Jiawei Chen and Dingkang Yang and Yue Jiang and Mingcheng Li and Jinjie Wei and Xiaolu Hou and Lihua Zhang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=gYxocD2XGO}
}
In the realm of Medical Visual Language Models (VLMs), the quest for universal efficient fine-tuning mechanisms remains paramount, especially given researchers in interdisciplinary fields are often extremely short of training resources, yet largely unexplored.
Most of the current Parameter-Efficient Fine-Tuning(PEFT) methods, not only have not been comprehensively evaluated on Med-VLMs but also mostly focus on adding some components to the model's structure or input. However, fine-tuning intrinsic model components often yields better generality and consistency, and its impact on the ultimate performance of Med-VLMs has been widely overlooked and remains understudied. In this paper, we endeavour to explore an alternative to traditional PEFT methods, especially the impact of fine-tuning LayerNorm and Attention layers on Med-VLM. Our comprehensive study spans both small-scale and large-scale Med-VLMs, evaluating their performance under various fine-tuning paradigms across tasks such as Medical Visual Question Answering and Medical Imaging Report Generation. The findings reveal that fine-tuning solely the LayerNorm layers not only surpasses the efficiency of traditional PEFT methods but also retains the model's accuracy and generalization capabilities across a spectrum of medical downstream tasks. The experiments demonstrate LayerNorm fine-tuning's superior adaptability and scalability, particularly in the context of large-scale medical VLMs. We hope this work will contribute to the ongoing discourse on optimizing efficient fine-tuning strategies for medical VLMs.
Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Models
Jiawei Chen Dingkang Yang Yue Jiang Mingcheng Li Jinjie Wei Xiaolu Hou Lihua Zhang