@inproceedings{
he2024everyday,
title={Everyday Object Meets Vision-and-Language Navigation Agent via Backdoor},
author={Keji He and Kehan Chen and Jiawang Bai and Yan Huang and Qi Wu and Shu-Tao Xia and Liang Wang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=rXGxbDJadh}
}
Vision-and-Language Navigation (VLN) requires an agent to dynamically explore environments following natural language.
The VLN agent, closely integrated into daily lives, poses a substantial threat to the security of privacy and property upon the occurrence of malicious behavior.
However, this serious issue has long been overlooked.
In this paper, we pioneer the exploration of an object-aware backdoored VLN, achieved by implanting object-aware backdoors during the training phase. 
Tailored to the unique VLN nature of cross-modality and continuous decision-making, we propose a novel backdoored VLN paradigm: IPR Backdoor. 
This enables the agent to act in abnormal behavior once encountering the object triggers during language-guided navigation in unseen environments, thereby executing an attack on the target scene.
Experiments demonstrate the effectiveness of our method in both physical and digital spaces across different VLN agents, as well as its robustness to various visual and textual variations. Additionally, our method also well ensures navigation performance in normal scenarios with remarkable stealthiness.
Everyday Object Meets Vision-and-Language Navigation Agent via Backdoor
Keji He Kehan Chen Jiawang Bai Yan Huang Qi Wu Shu-Tao Xia Liang Wang