CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations
Vision-and-Language Navigation (VLN) tasks require an agent to navigate
through the environment based on language instructions. In this paper, we aim
to solve two key challenges in this task: utilizing multilingual instructions
for improved instruction-path grounding and navigating through new environments
that are unseen during training. To address these challenges, we propose CLEAR:
Cross-Lingual and Environment-Agnostic Representations. First, our agent learns
a shared and visually-aligned cross-lingual language representation for the
three languages (English, Hindi and Telugu) in the Room-Across-Room dataset.
Our language representation learning is guided by text pairs that are aligned
by visual information. Second, our agent learns an environment-agnostic visual
representation by maximizing the similarity between semantically-aligned image
pairs (with constraints on object-matching) from different environments. Our
environment agnostic visual representation can mitigate the environment bias
induced by low-level visual information. Empirically, on the Room-Across-Room
dataset, we show that our multilingual agent gets large improvements in all
metrics over the strong baseline model when generalizing to unseen environments
with the cross-lingual language representation and the environment-agnostic
visual representation. Furthermore, we show that our learned language and
visual representations can be successfully transferred to the Room-to-Room and
Cooperative Vision-and-Dialogue Navigation task, and present detailed
qualitative and quantitative generalization and grounding analysis. Our code is
available at https://github.com/jialuli-luka/CLEAR