Approach.
Given procedural houses populated with Objaverse-LVIS, the task is to navigate to the proximity of a chosen target object and invoke a task-completion action when the target object is in sight, given an open-vocabulary description formed with the template “a {name} {category}". The name is the object name given by its creator, which is often descriptive. We filter each by whether it is detected as being written in English by a language detector [1]}, [2]}, and fall back to a class-only description for non-English name. Examples of the possible expressions include “a victorian-monobike motorcycle", “a unicorn pony", or “a dino ghost lizard". The agent, similar to the ones in [3]}, observes an RGB egocentric view of the environment, pre-processed by the visual branch of a frozen ResNet-50 CLIP model [4]} – the target description is pre-processed by the corresponding text branch. We train the agent with DD-PPO [5]} and evaluate on houses with floor plans, objects, and descriptions unseen in training. We use the AllenAct [6]} framework to train our agent. Our trained agent achieves a success rate of 19.9%, for a random policy success of 5.1%. For more details about the experiment refer to the appendix.
