We presented PONI, a modular approach for ObjectGoal navigation. Our key idea is to treat `where to look for an unseen object?' purely as a perception problem and address it without any interactions. To this end, we proposed the potential function network, an encoder-decoder model that predicts two complementary potential functions to decide `where to look?' for a goal object. We proposed a novel strategy to train this model in a supervised fashion using a dataset of semantic maps obtained from 3D semantic annotations, unlike existing ObjectNav methods which design reward functions for RL-based policy learning. Through experiments on Gibson and Matterport3D, we demonstrated that our method achieves the state-of-the-art for ObjectNav while incurring significantly lower training cost. We discuss the broader impact of our work in the supplementary, and will release code and pre-trained models upon acceptance. We hope that our work will spur future research into compute-efficient training for embodied navigation.


Supplementary Materials
This document provides additional information about our experimental settings and supporting qualitative visualizations. Below is a summary of the sections in the supplementary file:

(§) Limitations

(§) Additional experimental details

(§) Examples of semantic maps

(§) Examples of potential functions

(§) Visualizing ObjectNav episodes


Limitations
In Sec. 5 from the main paper, we discussed the benefits of our proposed PONI method both in terms of achieving state-of-the-art results on ObjectNav, as well as computational benefits during training. However, we would like to acknowledge some limitations of our approach.
One of our main limitations is our reliance on the semantic map as the only source for deciding when an object is found (i.e., to execute STOP). As discussed in the ablation study from Sec. 5, our performance is sensitive to the image segmentation quality. The success rate goes down by \(14.9\%\)  in Gibson and \(49.0\%\)  on MP3D relative to the performance with ground-truth segmentation (see tab:limitationsegm). Note that our ratio of SPL to success remains relatively stable (only \(6\%\)  reduction), indicating that our search efficiency is not affected significantly by segmentation errors. Unlike end-to-end RL methods which may learn to be robust to the sensory noise, we do not have an inbuilt mechanism to handle failures in segmentation. This limitation of interaction-free learning can potentially be addressed by using the latest advances in segmentation. Additionally, segmentation errors in simulation can be caused by reconstruction artifacts in the 3D scenes. Experimenting on higher quality scenes, or testing in the real world may address this limitation.
We also rely on access to human-annotated semantic information in 3D scenes. While this is standard practice for most approaches in ObjectNav [1]}, [2]}, [3]}, [4]}, alternative strategies exist for learning ObjectNav without access to any ground-truth semantic annotations in 3D scenes [5]}. Such self-supervised approaches have the potential to be more scalable than our supervised approach. However, to the best of our knowledge, there are no purely self-supervised methods that achieve state-of-the-art results for ObjectNav.
<TABLE>
Additional experimental details
We provide additional information about the experiments to supplement the main paper. The Gibson ObjectNav dataset from [1]} consists of 6 object categories: `chair', `couch', `potted plant', `bed', `toilet', and `tv'. The train split episodes are generated on-the-fly during training from 25 train scenes in Gibson tiny. The val split consists of \(1@000\)  episodes from 5 val scenes in Gibson tiny and is publicly accessible.Gibson ObjectNav dataset is accessible at https://github.com/devendrachaplot/Object-Goal-Navigation. The MP3D ObjectNav dataset from the Habitat challenge consists of 21 object categories: `chair', `table', `picture', `cabinet', `cushion', `sofa', `bed', `chest of drawers', `plant', `sink', `toilet', `stool', `towel', `tv monitor', `shower', `bathtub', `counter', `fireplace', `gym equipment', `seating', and `clothes'. The train / val splits consist of \(263@2422\)  / \(2@195\)  episodes from 61 / 11 MP3D scenes and are publicly released.MP3D ObjectNav dataset is accessible at https://github.com/facebookresearch/habitat-lab.

Examples of semantic maps
We show examples from the semantic map datasets we used for training the potential function network in fig:gibsonsemanticmaps,fig:mp3dsemanticmaps. The Gibson semantic maps contain up to 15 object categories of which 6 categories are goal categories (same as [1]}). The MP3D semantic maps contain up to 21 object categories of which all are goal categories (same as the Habitat challenge [8]}). These maps are computed by performing an orthographic projection of the 3D point-cloud annotations (following [9]}). In addition to the pipeline from [9]}, we perform additional pre-processing to obtain per-floor maps. Specifically, we segment the 3D semantic point-cloud from Gibson [11]} and MP3D [12]} annotations into different floors. We do this by loading each scene into Habitat [13]}, identifying the navigable points and clustering them along the Y-coordinate to automatically discover the number of floors and their extents using DBScan [14]}. We then perform orthographic projection independently for each floor of the scene.
<FIGURE><FIGURE>
Examples of potential functions
In Fig. 3 from the main paper, we showed an example of potential functions from MP3D. We now show more examples of such potential functions for Gibson and MP3D in fig:gibsonpotentialfns,fig:mp3dpotentialfns.
<FIGURE><FIGURE>
Visualizing ObjectNav episodes
In Figure 5 from the main paper, we qualitatively visualized an episode showing how the potential functions are used to perform ObjectNav. We provide two additional examples in fig:pfqualsupp.
<FIGURE>