What do navigation agents learn about their environment?
Today's state of the art visual navigation agents typically consist of large
deep learning models trained end to end. Such models offer little to no
interpretability about the learned skills or the actions of the agent taken in
response to its environment. While past works have explored interpreting deep
learning models, little attention has been devoted to interpreting embodied AI
systems, which often involve reasoning about the structure of the environment,
target characteristics and the outcome of one's actions. In this paper, we
introduce the Interpretability System for Embodied agEnts (iSEE) for Point Goal
and Object Goal navigation agents. We use iSEE to probe the dynamic
representations produced by these agents for the presence of information about
the agent as well as the environment. We demonstrate interesting insights about
navigation agents using iSEE, including the ability to encode reachable
locations (to avoid obstacles), visibility of the target, progress from the
initial spawn location as well as the dramatic effect on the behaviors of
agents when we mask out critical individual neurons. The code is available at:
https://github.com/allenai/iSEE