@inproceedings{
pan2024towards,
title={Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration},
author={Kaihang Pan and Zhaoyu Fan and Juncheng Li and Qifan Yu and Hao Fei and Siliang Tang and Richang Hong and Hanwang Zhang and Qianru Sun},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=kf80ZS3fVy}
}
The swift advancement in Multimodal LLMs (MLLMs) also presents significant challenges for effective knowledge editing. Current methods, including intrinsic knowledge editing and external knowledge resorting, each possess strengths and weaknesses, struggling to balance the desired properties of reliability, generality, and locality when applied to MLLMs. In this paper, we propose \textbf{UniKE}, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting. Both types of knowledge are conceptualized as vectorized key-value memories, with the corresponding editing processes resembling the assimilation and accommodation phases of human cognition, conducted at the same semantic levels.  Within such a unified framework, we further promote knowledge collaboration by disentangling the knowledge representations into the semantic and truthfulness spaces. Extensive experiments validate the effectiveness of our method, which ensures that the post-edit MLLM simultaneously maintains excellent reliability, generality, and locality. The code for UniKE is available at https://github.com/beepkh/UniKE.
Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration
Kaihang Pan Zhaoyu Fan Juncheng Li Qifan Yu Hao Fei Siliang Tang Richang Hong Hanwang Zhang Qianru Sun