EgoEnv: Human-centric environment representations from egocentric video

First-person video highlights a camera-wearer's activities in the context of their persistent environment. However, current video understanding approaches reason over visual features from short video clips that are detached from the underlying physical space and capture only what is immediately visible. We present an approach that links egocentric video and the environment by learning representations that are predictive of the camera-wearer's (potentially unseen) local surroundings to facilitate human-centric environment understanding. We train such models using videos from agents in simulated 3D environments where the environment is fully observable, and test them on human-captured real-world videos from unseen environments. On two human-centric video tasks, we show that state-of-the-art video models equipped with our environment-aware features consistently outperform their counterparts with traditional clip features. Moreover, despite being trained exclusively on simulated videos, our approach successfully handles real-world videos from HouseTours and Ego4D. Project page: https://vision.cs.utexas.edu/projects/ego-env/


Introduction
Egocentric video offers a unique view into human activities through the eyes of a camera-wearer. Understanding this type of video is core to building augmented reality (AR) applications that can assist humans by answering their questions, recalling events, or anticipating their needs based on their activity. Ego-video is thus the subject of several recent datasets and benchmarks that are driving new research [12,26,45,70].
A key feature of the egocentric setting is the tight coupling of a camera-wearer and their environment as they traverse a persistent physical space, potentially over multiple hours (or days). This raises an important need for humancentric environment understanding -to learn representations from video that capture the camera-wearer's activities in the context of their environment. Such representations are important to answer questions like "what room is the person in right now?", or "when was the person last near the sink?". The former requires reasoning over video frames spread in time to build a coherent picture of the physical space. The latter requires reasoning about the camera-wearer's location relative to other objects, regardless of their visibility (e.g., a person at a sink, but looking at a mirror above). Despite its importance, there has been only limited work on learning human-centric environment representations. Current video models segment a video into short clips (1-2s long) and then aggregate clip features over time (e.g., with recurrent, graph or transformer-based networks) for tasks like action forecasting [23,25,26,54], temporal action localization [48,49,89,90], episodic memory [14,26], and movie understanding [85]. Critically, the clip features encode what is immediately visible in a short time window, and their aggregation over time does not equate to linking them in physical space (as needed to answer, e.g., where is the camera-wearer? what objects are within reach?). Other approaches use explicit camera pose information (e.g., from SLAM), but model the agent and the surrounding environment independently. For example, forecasting models [27, 55,64] use pose only to localize the camera-wearer, ignoring its relation to objects in the surrounding space. Affordance learning approaches [54,63] use pose to group activities by location, but do not learn representations for agent video. Moreover, these approaches require camerapose paired with each video frame, which precludes their use for egocentric video where characteristic blur and quick head motions prevent reliable pose estimates.
To address these shortcomings, we propose to learn environment-aware video representations that encode the surrounding physical space. Specifically, we define the local environment state at each time-step of an egocentric video as the set of objects (and their rough distance) in front, to the left, right, and behind the camera-wearer. We use this state as supervision to train a transformer-based video encoder model that aggregates visual information across a video to build an environment memory which can be queried to predict the local state at any point in the video.
The local state captures the rough layout of objects relative to the camera-wearer and is important for understanding physical space -it provides a semantic signal to localize people (e.g., in the living room, from the relative arrangement of couches, lamps and tables), as well as human behavior -people move towards layouts that support activities (e.g., stove-top areas, dressers). Predicting the local state thus involves capturing the natural statistics of object layouts across different homes and then translating contexts across environments to reason about new ones.See Fig. 1. Once trained, given an observation from a new video, our model produces a "drop-in" EgoEnv feature which encodes environment information to complement existing video clip features. Importantly, our EgoEnv feature encodes general information about the local surroundings and can benefit multiple downstream tasks.
An important practical question is how to supervise such a representation. Sourcing local state labels requires agent and object positions and omnidirectional visibility at each time step. This is challenging as egocentric videos only offer sparse coverage of the environment. Furthermore, they are prone to object detection, tracking, and SLAM failures. Therefore, for training we turn to videos generated by agents in simulation. This allows us to sample diverse, large-scale trajectories to cover the environment, while also providing ground-truth local state. Once trained with simulated video, we apply our models to real-world videos from new, unseen environments.
We demonstrate our approach on human-centric environment understanding tasks in video, where joint reasoning of both human motion and action in video and the underlying physical space is required. These include inferring the room category that the camera wearer is physically in as they move through their environment, and localizing the answer to a natural language query in an egocentric video. These tasks support many potential applications, including AR systems that can offer context-relevant assistance. For example, an AR assistant that offers location-specific sug-gestions for chores to complete based on the rooms visited by the user; or one that recalls the last time the user was near the sink, to check if the tap was left turned-on.
We are the first to demonstrate the value of 3D simulation data for real-world ego-video understanding. Our experiments show that by capitalizing on both geometric and semantic cues in our proposed "local environment state" task, we can leverage video walkthroughs from simulated agents in HM3D scenes [60] to ultimately enable downstream human-centric environment models on real-world videos from HouseTours [7] and Ego4D [26]. Furthermore, models equipped with our EgoEnv features outperform both popular scene classification [91] and state-of-the-art natural language video localization models [89].

Related work
Video understanding in 3D environments. Prior work encodes short video clips [4,33,57,80] or temporally aggregates them to add additional context [23,48,49,84,85,89,90]. However, these methods treat the video as a temporal sequence and fail to capture the spatial context from the underlying persistent environment. For egocentric video, prior work has used structure from motion (SfM) to map people and objects for trajectory [55] and activity forecasting [27] and action grounding in 3D [13,63]. These approaches localize the camera-wearer but do not learn representations for the camera-wearer's surroundings. The model of [50] associates features to voxel maps to localize actions; however, they require a pre-computed 3D scan of the environment at both training and inference. Prior work groups clips by rough spatial location as topological graphs [54] or activity threads [58], but they stop short of learning representations using these groups. In contrast, we go further and explicitly learn features that relate clips based on their spatial layout, for each step of an egocentric video.
Video representation learning. Traditional video understanding methods learn representations by training models on large, manually curated video action recognition datasets [4,26,52]. Recent self-supervised learning (SSL) approaches eliminate the supervision requirement by leveraging implicit temporal signals [22,31,38,53,62,76,79,[81][82][83]. In contrast, we learn features that encode the local spatial-state of the environment (as opposed to temporal signals). Further, we show how to leverage state information that is readily accessible in simulation, but not in video datasets (i.e., locations and semantic classes of objects surrounding the agent) for training.
Environment features for embodied AI. In embodied AI, pose-estimates are used to build maps [8,10,59], as edge features in graphs [7,9], as spatial embeddings for episodic memories [19], to project features to a grid map [5,28,36] or to learn environment features for visual navigation [61].
However, these approaches are explored solely in simulation and typically require accurate pose-estimates or smooth action spaces, and thus are not directly applicable to egocentric videos. Research on world-models [17,29,30] and unseen panorama reconstruction [39,42,71] hallucinate the effect of agent actions to aid decision-making in simulation. Our approach is similarly motivated, however we aim to learn environment features for an egocentric camera-wearer to aid real-world video understanding.
Learning from simulated data. Prior work has proposed cost-effective ways to generate large-scale synthetic image datasets for various vision tasks [15,18,37,66,72,74]. In robotics, simulation environments have been developed to quickly and safely train policies, with the eventual goal of transferring them to real world applications [6,41,43,60,68,73,75,86]. The resulting sim-to-real problem, where models must adapt to changes in simulator and real-world domains, is an active area of research for robot navigation [1,3,40,67,74]. Simulated data for video understanding is much less explored. Prior work has synthesized data for human body pose estimation [11,16,77,87], trajectory forecasting [46] and action recognition [65]. Rather than model human behavior, our approach is the first to directly capture the semantics of the environment surrounding the camerawearer for real-world video understanding tasks.

Approach
Our goal is to learn EgoEnv representations that encode the local surroundings of the camera-wearer -where they are in relation to nearby, potentially obscured objects. Such a representation implicitly maintains a semantic memory of surrounding objects beyond what is immediately visible, and once coupled with a standard video feature, it allows a model to jointly reason about activities in video and the underlying physical space. Directly appending camerawearer pose with each video frame may capture the local state, however noise in pose estimated from ego-video with quick head motions and characteristic blur limits its utility (see Supp. for experiments). Moreover, training a model to effectively utilize pose is itself non-trivial as camera-wearer location alone does not reveal the relative poses of objects.
Instead, we introduce an approach that leverages simulated environments where perfect state information is available to train models that can link visual information to the physical surroundings. To this end, we first define the local state prediction task in simulation (Sec. 3.1). Next, we introduce our transformer-based architecture that predicts the local state in videos (Sec. 3 Figure 2. Local environment state illustration. Views in each direction are shown for clarity (left panel), but only the forward view is accessible at each step (top left). Not all parts of the environment are seen during a walkthrough (white vs. grey regions)models must link seen observations based on their shared space, as well as anticipate unseen surroundings. Best viewed in color.

Local environment state
We require a model that is aware of not just what is immediately visible in a single frame, but also of the camerawearer's surroundings. We therefore define the local environment state of the camera-wearer as the set of objects in each relative direction -i.e., what objects are to the front, left, right or behind the camera-wearer, together with their rough distance from the camera-wearer -and train a model to predict this state. Our definition of local state takes inspiration from cognitive science [32,44], and offers supervision signals that are both geometric (relative object locations) and semantic (semantic object labels), which we observe leads to strong representations.
More formally, let O be a set of object classes. For a frame f from a video trajectory in an environment, the local state is a tuple (y o , y r ). y o is a 4 × |O| dimensional matrix which represents instances of each object class in the four cardinal directions relative to the camera-wearer. y r is a matrix of the same size containing the distance of the objects in y o from the camera-wearer, discretized into 5 distance ranges between 0.25 − 5.0m 1 . For direction i and object class j, the labels are calculated as follows where p a , p j are the poses of the camera-wearer and object o j respectively, d(p a , p j ) is the euclidean distance between them, δ is a distance threshold for nearby objects (we set δ = 5.0m, beyond which visible objects are small), θ(p a , p j ) ∈ [0..  camera-wearer's possibly hidden surroundings to produce environment-aware features. However, obtaining supervision for camera-wearer pose and every object's location is non-trivial for egocentric videos where trajectories have limited environment coverage, and where camera localization and tracking is error prone. We address these issues in the next section by leveraging videos in simulated environments for training.

Environment-aware pretraining in simulation
To source local state labels, we generate a dataset of video walkthroughs of agents in simulated 3D environments where agent and object poses are accessible at all times (see Sec. 4.1). We train our model in two stages described below.

Pose embedding learning
While ground truth camera pose is available from the simulator at training time, a model trained to rely on it will fail on real-world egocentric video at test time, where pose estimates are noisy and in many cases unavailable. With the goal of handling arbitrary indoor egocentric video, we instead explore representations that implicitly encode coarse pose information. See Supp. for pose-related experiments.
Specifically, for a sequence of RGB frames {f t } T t=1 and camera poses {θ t } T t=1 , we generate pose embeddings {p t } T t=1 = P(f 1 , ..., f T ) using a transformer encoder network. These embeddings are used to predict the relative pose between each observation pair using a bilinear layer whereθ i,j is the predicted relative pose and V p , W p , b p are trainable parameters. In practice, we discretize the relative pose into 12 angles and 4 distance ranges as it is difficult to regress to fine-grained pose values.
The network is trained to minimize cross-entropy between the predicted and the target relative pose labels for all observation pairs i,j L ce (θ i,j , θ i,j ). The trained pose embeddings p t encode information to help relate video observations based on their location and orientation in the environment. Note that once trained, pose embeddings are inferred directly from video frame sequences -ground truth pose is only required for training.

Local state pretraining
Next, we train a model to embed visual information from a video walkthrough into an environment memory, which can then be queried to infer the local state corresponding to a given video frame from the same video. We implement this model as a transformer encoder-decoder model.
Specifically, for a video walkthrough V with RGB frames {f t } T t=1 , and a query frame f q , we predict the local state y q = (y o , y r ) as follows. First, pose embeddings {p t } T t=1 are generated for the video and query frames following Sec. 3.2.1. Then, each frame is encoded jointly with the pose embedding using a linear transform M p .
Next, we uniformly sample K video frames to construct an environment memory using a transformer encoder E, which updates frame representations using self-attention: The resulting memory represents features for each timestep that contain propagated information from all other time-steps. Compared to prior work [23,48,49,89,90], our encoder has the ability to relate observations based on not just visual characteristics and their temporal ordering, but  also their relative spatial layout in the environment. A transformer decoder D then attends over the memory using query x q to produce the output EgoEnv representation h q : which is finally used to predict the local state using two linear classifiers M o and M r for object class and distance predictions respectively. The network is trained to minimize the combination of losses over the predicted and the target state labels for each direction: where L bce and L ce refer to binary cross-entropy and crossentropy losses respectively. We set λ = 0.1 which balances the contributions of each loss function based on validation experiments. The distance loss is computed only for objects that are in the local state (y o = 1). See Fig. 3 (left) and Supp. for more architecture details. Learning to predict the local state involves aggregating information about observed objects across time, as well as anticipating unseen objects based on learned priors from the layout of objects in other scenes (e.g., TVs are usually in front of couches; kitchens have particular arrangements of sinks, refrigerators, and stove-tops).
Once trained, given a video in a new environment and a target time-point of interest, our model constructs an environment memory, predicting the local state based on information aggregated through the entire test video. Importantly, h q -the EgoEnv feature -contains valuable information about the camera-wearer's surroundings offering environment-centric features to complement traditional video features (e.g., for a person watching TV, also encode features for the couch they are sitting on, the lamp nearby).

Environment-memory for video understanding
Next, we leverage our environment-memory model for real-world video tasks. A video understanding task defines a mapping from a sequence of video clips {c 1 , ..., c N } from longer video V to a task label. We consider two tasks: (1) ROOMPRED: where the model must classify which room r t the camera-wearer is in (e.g., living room, kitchen) at time t in the video, and (2) NLQ: natural language queries, an episodic memory task popularized recently in Ego4D [26] where the model must identify the temporal window (t s , t e ) in the video that answers an environmentcentric query q specified in natural language. See Fig. 4.
Current models produce clip features that encode only what is immediately visible. This is sufficient for shorthorizon tasks (e.g., action recognition), but as we will show, falls short on the tasks above that require extra reasoning about the agent's surroundings. Our environment-memory model addresses this by enhancing standard clip features with context from the camera-wearer's surroundings.
To do this, for each input clip in c i ∈ V, we select the center frame f i of the clip as the query frame. Following Sec. 3.2, we uniformly sample K frames from the video around the query frame, encode them along with their pose embeddings (Eqn. 4), and build an environment memory using our environment encoder E (Eqn. 5). Finally, we use our decoder D to produce the output feature. This results in set of output EgoEnv features, one per input clip.
Each environment feature then enhances the original clip feature as follows where g i is the original clip feature for clip c i (e.g., ResNet, SlowFast, EgoVLP.) and W E , b E are linear transform parameters. See Fig. 3 (right).
The new clip features {g 1 , ..., g N } consolidate features from what is directly visible in a short video clip and features of the (potentially unseen) space surrounding the camera-wearer. Put simply, our approach implicitly widens the field of view for tasks that reason about short video clips by providing a way to access features of their surroundings in a persistent, geometrically consistent manner.
Critically, our core approach is task-agnostic-we use the exact same EgoEnv representations to tackle both video understanding tasks. This is a departure from traditional sim-to-real approaches where a task-specific dataset needs to be carefully designed for every downstream task, which may be impractical for a large number of tasks, or complex tasks beyond the simulator's capability (e.g., simulating human motion, hand-object interaction). Note that our approach is best suited to videos in indoor environments to stay consistent with the scenes used for pretraining. In addition, indoor layouts have a higher density of nearby objects, providing richer local state labels to train our models with.

Experiments
We evaluate how our EgoEnv features learned in simulation benefit real-world video understanding models.

Simulator environments and video datasets
For training, we use the Habitat simulator [68] with photo-realistic HM3D [60] scenes to generate simulated video walkthroughs. We generate 15k walkthroughs from 900 HM3D scenes, each 512 steps long, taken by a shortestpath agent that randomly samples goal locations in the environment and navigates towards them (move forward, turn right/left 30 • ). For each time-step, we obtain the groundtruth local state from the simulator required in Sec. 3.1 (i.e., object labels and relative pose). For object labels, we map instance predictions across |O| = 23 categories to the 3D scenes using a pretrained instance segmentation model [20] trained on the subset of scenes with semantic labels.
Though the walkthroughs involve discrete actions, they share characteristics with real-world video (cameras at head-level; views covering the environment) making them suitable for transfer. See Supp. for further details.
We evaluate our models on three egocentric video sources. (1) Matterport3D (MP3D) [6] contains simulated video walkthroughs from 90 photo-realistic 3D scenes. We use 146 long video walkthroughs and standard data splits [5]. (2) HouseTours [7] contains 119 hours of realworld video footage of house tours from YouTube. We use 32 hours of video from 886 houses where the camera can be localized and create data splits based on houses. (3) Ego4D [26] contains 3k hours of real-world video of people performing daily activities. We use all 227 hours annotated for the NLQ benchmark and apply the provided data splits.
These datasets provide an ideal test-bed with increasing realism and associated challenges. MP3D offers novel scenes with distinct visual characteristics and object distributions compared to HM3D, allowing us to test our model's robustness to domain shift in a controlled simulated video setting. HouseTours captures high quality videos captured by people, allowing us to test generalization to real-world visuals as well as new, unseen houses. Ego4D captures nat-  ural human activity in diverse cluttered environments allowing us to further test our model's ability to relate human action and the underlying environment.
We collect labels for each task, which we will publicly share. For ROOMPRED these are room category labels (e.g., living room, kitchen) for each time-step. On HouseTours and Ego4D, we crowd-source room labels across 21 categories. For NLQ these are natural language queries and corresponding response tracks in the video. On House-Tours, we crowd-source queries across 8 templates (e.g., "where did I last see my phone in the kitchen", "when did I first visit the bathtub"). We group the templates into "see" queries, where the object of interest is directly visible in frames, and "visit" queries, where the camera-wearer is near, but may not actually see the object, which requires extra environment-level reasoning to localize. On Ego4D, we use the existing NLQ benchmark annotations which contain queries across 13 templates that require reasoning over actions, objects, and locations (e.g., "what tool did I pick up from the table", "where did I hang the pink cloth"). On MP3D, we source all labels directly from the simulator -9 room categories, and automatically generated NLQ queries from simulator object labels and locations. See Supp. for data collection details and Fig. 4 for examples.

Experiment setup
For pre-training, we use 2048-D ImageNet-pretrained ResNet50 [35] features for each video frame. Our encoderdecoder models P, E, D are 2-layer transformers [78] with hidden dimension 128. K = 64 frames are sampled from each video to populate the memory. We train models for 2.5k epochs and select the model with the lowest validation loss. For downstream tasks, we sample K = 64 frames for our memory. For ROOMPRED, we generate a single EgoEnv feature aligned with the query clip. For NLQ we generate one feature per input clip. Full architecture and training details are in Supp.

Baselines
We compare the following approaches to enhance the clip representations in a baseline model. For ROOM-PRED the baseline is a popular scene recognition model PLACESCNN [91]. For NLQ we use a state-of-the-art moment localization model VSLNET [47,51,88].
• FRAMEFEAT uses a pretrained ResNet50 [35] model to generate a frame feature corresponding to each clip.
• OBJFEAT trains an object detector on all available simulated HM3D data and generates backbone features for each clip. We use the QueryInst model [20].
• MAE [34] trains a state-of-the-art self-supervised learning approach to reconstruct the pixels of masked patches in frames from our simulated video trajectories.
• EGOTOPO [54] trains a graph convolutional network (GCN) over the video graph built following [54]. Graph construction details are in Supp.
• EPC [61] trains an environment memory model to predict masked zone features conditioned on pose.
• TRF (SCRATCH) trains a scene-memory transformer model [19] that shares our model architecture but does not benefit from pretraining.
These baselines represent various strategies to incorporate environment information into clip representations ranging from frame features (MAE, FRAMEFEAT, OB-JFEAT), to topological graph-based features (EGOTOPO), to pose-based features (EPC). Note that OBJFEAT, MAE and EPC all benefit from pre-training on the same walkthrough videos as our approach. OBJFEAT further benefits from ground-truth object labels from the simulator. Features from these approaches augment the input clip representations following Eqn. 8 -baseline architectures remain unchanged. Note that EPC requires privileged informationground-truth camera poses at inference time-whereas our model does not. Supp. contains pose estimation details and further experiments.

Local state pretraining
We begin by evaluating how well our model can infer the local state, reporting average precision (AP) in each direction. Given a forward view, objects can be reliably recognized (37.8 AP) compared to naively outputting the distribution of objects seen at training (5.4 AP). Moreover, our approach can link views in the video trajectory to also infer and anticipate objects in other directions, i.e., to the right, left and behind (21.5, 24.9 and 20.2 AP respectively) given the forward view. We visualize the attention weights learned by our model to link relevant observations to the query in Fig. 5. Our model learns to select informative views for the task beyond just temporally adjacent frames or views with high visual overlap. For example, the view with highest attention score (0.82) looks at the dining table directly behind (red box), allowing our model to benefit from information beyond its field of view.

EgoEnv features for room prediction
Next, we evaluate our method on predicting what room the camera-wearer visits in the video. All models have access to the full video, but inference is at at each time-step. Baseline. The PLACESCNN model is a 2-layer MLP classifier trained on features from a Places365 [91] scene classification model. Features are max-pooled across a clip of N = 8 frames around the time-step of interest for additional context before prediction, as a single frame may be uninformative (e.g., facing a wall). We generate an environment feature aligned with the center of the clip. Evaluation. We report top-1 accuracy as a function of dataset difficulty.Difficulty is measured as the prediction entropy of the Places365 model trained on canonical scene images. Instances are "hard" (high entropy) where frame-level information is insufficient for predicting the room type.
Our results are in Fig. 6. The accuracy on the full dataset is at the far-right end. All models perform better on House-Tours compared to MP3D and Ego4D since the house tours were captured explicitly to provide informative views of

EgoEnv features for episodic memory
Next we evaluate our method on localizing the responses to natural language queries in egocentric video. Baseline. We use VSLNET [88] and provide it with N = 128 clips sampled uniformly from the full video to generate predictions. We use SlowFast [21] clip features and generate environment features aligned with each input clip. Evaluation. We report Rank n@m metrics commonly used in moment localization [24,88,89], which measures the percentage of correctly localized moments in the top n predictions. A predicted moment is correct if it overlaps with the ground truth moment (IoU > m). We report Rank 1@[0.3, 0.5] and their mean following prior work [26].
Our results are in Table 1. Similar to ROOMPRED, instances are harder in MP3D than in HouseTours as the agents in the videos follow shortest paths across the environment, and do not focus on particular objects while navigating. As a result, moments are quick transitions between objects and locations, and contain only short glimpses of them. The global, video-level information from EGOTOPO improves performance slightly on both datasets. Strong image-level supervision (object labels in OBJFEAT and FRAMEFEAT) results in the largest improvements; MAE, which has access to the same data but trains self-supervised representations, does not show very large improvements. EPC which has access to ground-truth pose at training and inference performs well on MP3D where accurate pose is available from the simulator, but underperforms on House-Tours where only noisy pose estimates are available (see Supp.). Our EgoEnv approach performs the best overall, outperforming even EPC which has access to ground-truth pose at training and inference.
Finally, we evaluate our method on the Ego4D NLQ benchmark challenge. We augment current state-of-the-art approaches with our EgoEnv features and report our results on the validation set in Table 1 (right). Our approach enhances features from state-of-the-art action (SLOWFAST) and vision-language pretrained (EGOVLP) video features to achieve the best results. Note that Ego4D videos are fairly different from our simulated walkthrough videos in pretraining. They are in-the-wild, capture natural human actions and object-interactions, and take place in diverse scenes, both indoor and outdoor (e.g., workshops, gardens). This sim-to-real gap translates to smaller absolute improvements on Ego4D compared to MP3D and HouseTours. Despite this, our approach is still able to consistently improve all models, demonstrating the value of our environmentcentric feature learning approach.
Ablations show that pose information adds to our approach only when reliable and that both the geometric and semantic components of our local state definition are important for downstream performance (Supp.).

Conclusion
We proposed a framework to learn environment-aware representations in simulation and transfer them to video understanding tasks on challenging real-world datasets. Our approach outperforms state-of-the-art representations for predicting visited rooms and retrieving important moments from natural language queries, despite a significant sim-toreal gap. Future work can explore downstream AR applications like action forecasting or helping robots learn better from human demonstrations.

Supplementary Material
This section contains supplementary material to support the main paper text. The contents include: A. Videos illustrating our approach.
B. Data collection and annotation details.

A. Videos illustrating our approach
We present videos illustrating the local state prediction task (Sec. 3.2), downstream video tasks (Sec. 3.3) and our main results on our project page.
In the first part of the video, we demonstrate the local state prediction task from Sec. 3.2 of the main paper. The video shows the first-person view of the camera-wearer (left panel). The right panel shows the top-down view of the environment with the agent trajectory (blue gradient) and nearby objects (colored squares). Note that models only see the egocentric view -the top-down map is for illustration only. Given a simulated video walkthrough and a query time-stamp, the model must predict the direction and rough distance of each object near it. Correct, missing and false positive predictions are shown for each direction (left panel). Correct predictions on the top-down map are highlighted in cyan.
In the second part of the video, we show examples of the two downstream video tasks from Sec. 3.3 of the main paper, on MP3D, HouseTours and Ego4D. In the ROOM-PRED examples, the model must predict which room the camera-wearer is in from a short clip. As mentioned in Sec. 4.5, the clips show quick motions and often contain ambiguous views making it hard to predict room labels directly using traditional scene recognition methods. For example, the "staircase" is not visible as the person descends it (Ego4D, bottom right video). In the NLQ examples, the model must predict the moment in time that answers a particular environment-centric query. The video examples show how this requires reasoning about the camerawearer's surroundings. For example, in the HouseTours clip ("when did I visit the sink in the bathroom?") the sink is only seen briefly in the video but the response requires the window of time that the camera-wearer was physically near it (within arms reach) regardless of visibility. In the Ego4D clip ("Did I leave the drawer open"), the model must know where the drawer is relative to the camera-wearer and link their actions to this physical location in order to respond.

B. Data collection and annotation details.
We present data collection and annotation details for simulated walkthroughs, and our two downstream tasks (ROOMPRED and NLQ) for all three datasets (MP3D, HouseTours, Ego4D).

B.1. Walkthrough generation details
As mentioned in Sec. 4.1 of the main paper, we generate simulated walkthroughs in HM3D [60] scenes to train our models. Given an environment, we first cluster all navigable points using KMeans, selecting between 4-64 clusters depending on the environment size. With each cluster centroid as a starting location, we sample 8-16 nearest goal locations, shuffle them (to allow re-visitation), and make an agent visit the goals in sequence. We use a shortest-path planning agent that uses the underlying navigation graph to reach goals in the fewest number of steps. We collect a dataset of 15k episodes, each of 512 steps, of our agents visiting such goal sequences for experiments in Sec. 3.2. Fig. 1 shows a random sample of walkthroughs.
Note that the walkthroughs are generated in environments where objects are not moved, however a large part of real-world environments are in fact static. This includes static scene elements like doors, windows, counter tops, staircases, and most objects that are typically not moved like refrigerators, beds, couches, TV sets. Encoding these SCENES ROOMPRED NLQ HM3D [60] 800 / 100 / ---Matterport3D [ objects and scene elements can thus still provide value for human-centric environment understanding, even when some objects may have moved around.

B.2. Data collection and annotation details
As mentioned in Sec. 4.1 of the main paper, we collect labels for each video dataset. For Matterport3D, we directly use the ground truth information available through the simulator to extract labels. For HouseTours and Ego4D, we crowd-source annotations where required. We describe the data collection process and present data statistics for each dataset and task. The resulting dataset splits can be seen in Table 1.

B.2.1 Annotation requirements
ROOMPRED For this task, room labels are required at each time-step of the video. The 9 room categories used in Mat-terport3D are in Table 2 (left). These categories are predefined in the simulator. The 21 room categories used in HouseTours and Ego4D are in Table 2 (right). These categories were generated manually from a combination of Matterport3D room categories and a relevant subset of Places365 categories corresponding to indoor scenes. NLQ For this task, natural language queries and corresponding moment boundaries (start and end times) are required. For HouseTours, we define 7 query templates where o refers to objects and r refers to rooms: "see o", "see o in r", "see o 1 then o 2 ", "visit r 1 then r 2 ", "visit o/r", "visit o 1 then o 2 ", "visit o in r". Each template captures a type of question that requires a different mechanism of reasoning. "see" queries require reasoning about what is immediately visible; "visit" queries require an understanding of where the camera-wearer is in the environment and what objects are nearby (within arms reach); "see/visit o 1 then o 2 " and "see/visit o in r" require both spatial and temporal reasoning. Natural language queries follow from these templates. For example, for the "visit r 1 then r 2 " template, a natural language query may be "When did I first walk from the kitchen to the bathroom?". The list of query templates, examples and descriptions can be seen in Table 3. The task definition follows prior work [26] but is adapted for the datasets used, and contains more environment-centric queries. For Ego4D, we use the existing NLQ benchmark task annotations. Fig. 4 of the main paper. The video highlights the stark contrast between prediction in static images (third-person photos) which contains well-framed images that are easy to recognize, and egocentric video which is much more challenging. In this setting, video is tied to quick ego-motion as the camerawearer moves around the environment and objects are seen only briefly (or not at all) in non-canonical viewpoints.

B.2.2 Matterport3D annotations
For ROOMPRED, navigable location are mapped to room categories using information from the simulator. We map agent positions for each video frame to these categories. For NLQ we use room labels and extracted object positions to generate queries from the 7 templates above. We define objects as "seen" if they occupy at least 5% of the pixels in a given frame. We define objects as "visited" if the agent is < 1.0m from the object, regardless of its visibility, following embodied navigation protocol (ObjectNav [2]). Rooms are "visited" using the position → room category mapping above. We generate queries for each template by tracking objects and rooms that are seen and visited over time. If there are multiple visitations to an object or room, we ensure unique responses to queries by adapting them to consider only the first (or last) visit.

B.2.3 HouseTours annotations
We crowd-source annotations for real-world videos from HouseTours. For ROOMPRED, we ask annotators to watch a video and mark the start and end time of each "visit" to a room. They must then label each visit with one of the 21 room categories (from Table 2, right). An illustration of the annotation interface is shown in Fig. 5.
For NLQ, we ask annotators to identify an interesting moment (e.g., where a person sees a salient object, moves from one room to another, visits an important object) which serves as the answer to a query. The moment is specified by a start and end time, while the query is specified as natural language text generated following one of the 7 template classes from Table 3. An illustration of the annotation interface is shown in Fig. 6. Fig. 3 shows annotation statistics for both tasks on HouseTours. In general, trajectories are relatively shorter than Matterport3D, though they involve a similar number

B.2.4 Ego4D annotations
Following the same procedure as HouseTours, we crowdsource room visit labels on Ego4D videos (see Fig. 4). Ego4D videos are much longer (30 mins on average). As a consequence, visits are naturally a much smaller fraction of the overall video and the number of room transitions are much higher (16 on average). The distribution of room categories is similar to HouseTours. Annotation statistics for NLQ can be found in Section F.4 of the supplementary material of the Ego4D paper [26].

Template
Example Description see o Where did I first see the remote control? Objects must be visible for the moment duration see o in r When did I see the mirror in the bathroom? Object must be physically inside the room see o 1 then o 2 Where did I see a table then a chair? Objects can either be seen together or in quick succession visit r 1 then r 2 When did I walk from the living room to the kitchen? Start = when the person begins to leave; End = they are fully inside the kitchen. visit o/r When did I last visit the couch?; When did I last visit the bedroom? Visit = physically near an object (within arms reach) or physically inside a room visit o 1 then o 2 When did I visit the lamp then the couch? Same as see o 1 then o 2 , but using the "visit" criteria above visit o in r When did I visit the mirror in the bathroom?
Same as see o 1 in o 2 , but using the "visit" criteria above       . Annotation interface for collecting NLQ labels. Annotators must identify an interesting moment in time (start and end time) and associate a query template, a natural language query and object/room labels that fill the query template slots.

C. Implementation and training details
We present architecture and training details for all approaches.

C.1. Architecture and training details (pretraining)
We present additional details for our model in Sec. 3 of the main paper, as well as the baselines in Sec. 4.3. The full list of hyperparameters are in Table 5 (panels 1-3).
Pose embedder P, environment encoder E and decoder D. We build on the transformer [78] architecture for our model. The inputs f ∈ R 2048 are features from an ImageNet-pretrained ResNet-50 . These are encoded into 128-dimension vectors using a visual encoder (2-layer MLP).
First, 128-D pose embeddings are generated following Eqn. 3. These pose embeddings are concatenated with the visual embedding, and then transformed back to a 128dimension vector using M p . Sin-cosine position embeddings are added to this input following [78] resulting in the transformer input {x 1 , ..., x N } in Eqn. 4. Note that a separate visual encoder generates input features for the encoder E and the pose embedder P.
The encoder E performs multi-headed self-attention using these inputs, with 2 layers, 8 heads and hidden dimension 128. The decoder D is a 2-layer transformer with hidden dimension 128 which attends to the outputs of E to generate the output representation. We predict relative directions for 23 object classes in HM3D (see Table 4). We use the transformer implementation from PyTorch [56].
Our architecture is similar to prior embodied navigation approaches [19,61] but does not require pose (and instead uses pose embeddings), includes the environment decoder that queries the memory based on pose embeddings and visual content, and is trained using our proposed learning objective in Sec. 3.2. See Table 5 (panel 1).
EPC architecture details. EPC [61] is a transformer encoder-decoder model that masks out frames from physical locations and predicts the features of these masked zones given a query pose. We generate graphs to train this baseline following their approach.
Specifically, for every video frame {f 1 , ..., f T }, we compute the geometric viewpoint overlap with every other frame. The viewpoint overlap ψ(f i , f j ) is calculated by projecting pixels from the frames to 3D point-clouds using camera intrinsics, agent pose and depth measurements, and measuring the percentage of shared points across frames. We use ψ(f i , f j ) as our distance metric to cluster all frames in video into zones using hierarchical agglomerative clustering. We set the distance threshold to 0.8 as larger values result in too few zones.
We sample 4 unseen zones per instance in a batch of which one is "positive" and three are "negatives" for con-chair  table  picture  cabinet  cushion  sofa  bed  chest of drawers  plant  sink  toilet  stool  towel  tv monitor shower  bathtub  counter  fireplace  shelving seating  furniture appliances clothes   Table 4. HM3D object taxonomy trastive learning. We collect negatives from all instances in a batch during training. The network is trained using noisecontrastive estimation following [61]. See Table 5 (panel  2).
EgoTopo architecture details. EgoTopo [54] is an approach to translate egocentric video frames into a topological graph, where each node contains a list of clips that correspond to a physical location, and edges correspond to rough spatial layout.
For MP3D and HouseTours, ee directly use available pose information to determine whether two frames belong to the same zone or not (i.e., we do not train a retrieval network to approximate this). This amounts to a clustering of the trajectory based on pose (position and heading) and represents an enhanced version of EgoTopo that benefits from pose data. We use affinity propagation to cluster frames into nodes. To compute edges, we calculate the distance between the centroid of each node and assign an edge if this distance is < 3.0m to be consistent with Eqn. 2. For Ego4D, pose information is not available. We fall back to clustering visual frames based on ImageNet pre-trained ResNet-50 frame features.
Node features are calculated as the average of features assigned to that node. We use a 2-layer graph convolutional neural network (GCN) to aggregate features across nodes, and then average them to form a single video encoding following [54]. See Table 5 (panel 2).
MAE architecture details. We use all frames from our generated walkthroughs to train an MAE VIT-large model with 16 × 16 patch size. We use the authors existing code and train a model for 200 epochs.
ObjFeat architecture details. We train a QueryInst instance segmentation model [20] with a R-50-FPN backbone, using a dataset sampled from the 100 HM3D scenes with ground-truth semantic object annotations. We use the implementation in the MMDetection package.
Training details. As mentioned in Sec. 4.2 of the main paper, we train our models for 2500 epochs using the Adam optimizer with learning rate 1e − 4. We sample K = 64 frames to construct our memory. At training, we sample frames from the video but randomly offset frame indices to train robust models. During inference, we uniformly sam-

C.2. Architecture and training details (downstream)
We present additional details for the approaches in Sec. 3.3 of the main paper. The full list of hyperparameters are in Table 5 (panels 4-5).
Room prediction models. As mentioned in Sec. 4.5 of the main paper, for our PLACESCNN baseline model, we build a classifier on top of features from the wide ResNet-18 model from [91]. We use the authors existing code and their provided pretrained models to initialize the model. The classifier head is a 2-layer MLP with hidden dimension 512. The backbone is frozen and only the classifier is fine-tuned. N = 8 frames around the target frame are used to provide additional context. The features are max-pooled before classification.
For our models, we use the target frame as the query to produce a single environment-feature, which is then concatenated with all N = 8 frames and aggregated following Eqn. 8. This new, enhanced input is fed into the baseline model as described above.
We train all models for 80 epochs with learning rate 1e − 4 using the Adam optimizer. The full list of hyperparameters are in Table 5 (panel 4) Episodic memory retrieval models. As mentioned in Sec. 4.6 of the main paper, we build on the VSLNet model from [88]. We use an existing implementation based on the authors original code. Visual inputs are encoded as N = 128 clip features, created by adaptive average pooling of SlowFast-R50 clip features. Note that for MP3D we use ResNet-50 features as the walkthroughs contain discrete agent steps, rather than smooth video frames.
For our models, we select the center frame of each of the N = 128 inputs and use them as query frames to produce N = 128 environment features. Each pair of input feature and environment feature is aggregated following Eqn. 8, and then input to the VSLNet model described above. Note that we perform this aggregation after the video affine and feature encoding layers in VSLNet.
We train all models for 200 epochs with a learning rate of 1e − 4 using the Adam optimizer. The full list of hyperparameters are in Table 5 (panel 5).
For Ego4D experiments, we use the hyperparameters described in the respective benchmark whitepapers [26,47,51] and only aggregate precomputed environment features as described above.

D. Ablations and additional experiments
We present ablation experiments for various model design choices and additional experiments.

D.1. Additional experiments with pose embeddings and ground-truth pose
As noted in Sec. 3.2.1 ground truth pose may be utilized directly by our model, however it is not easily available in real-world egocentric video, which makes our use of inferred pose embeddings a strength.

D.1.1 Importance of pose embeddings
As noted in Sec. 3.2.1 of the main paper, we train pose embeddings to help relate observations by their visual content as well as relative orientation of capture. We measure the effect of pose embeddings on our local state prediction task. Our models see improvements in predicting objects in each of the cardinal directions (26.1 vs. 24.9 mAP) as well as improved object distance prediction (59% vs 58%). This improvement is small for predicting objects in the forward view (+0.7 mAP) where scene information is directly visible, but large for other views that need to be inferred: mAP improvements of +2.2 (right), +0.9 (behind) and +1.0 (left).

D.1.2 Importance of ground truth pose
In this section, we explore using pose estimates obtained directly from the simulator or using off-the-shelf structure for motion methods. Note that the existing method EPC in Fig. 6 and Table 1 already uses this ground-truth pose information, which other models (including ours) do not have access to.
Extracting pose information. For HouseTours videos, we run COLMAP [69], a structure from motion framework to compute camera-pose information associated with each frame of the video. For this, we first extract frames from each video at 2fps. Then we run COLMAP using a precomputed vocabulary tree file (flickr100K words32K). The resulting trajectories are inherently noisy due to the approximate nature of the SfM pipeline and the absence of true camera parameters for such in-the-wild video. We postprocess them by removing erroneous pose values (ones that cause jumps in pose atypical to smooth motion).
Overall, COLMAP successfully localizes 32 hours of video from 886 houses out of the original 119 hours available in [7]. Some examples of video trajectories are shown in Fig. 8. Comparing these to the simulated trajectories in Fig. 1, we see smoother trajectories overall, but with unrealistic jumps in localizations and loop-closure failures. Note that we visualize only the trajectory, not obstacles in the environment, as we do not have access to occupancy maps for the real-world videos.   Note that these videos are tours of indoor spaces with the intention of visually covering a large area. As a result, the camera-wearer moves slowly and smoothly to show parts of the house. Despite this, only a fraction of trajectories can be localized highlighting the difficulty of estimating pose from monocular video. In contrast, the same procedure fails to localize the camera-wearer in Ego4D videos due to rapid head motions and motion blur. On MP3D, pose is available directly from the simulator.
Performance with ground-truth pose. In Fig. 7, we investigate the role of pose information for ROOMPRED (top) and NLQ (bottom), by directly embedding groundtruth pose as part of the input to the baseline and our model. We find that our model can benefit from pose on MP3D, but falls short on HouseTours, due to noise in extracted pose (compared to simulator-provided pose in MP3D). However, our approach (with and without pose) outperforms EPC, which explicitly leverages pose both at train and test time.

D.2. Ego4D NLQ baseline comparisons.
We evaluate Ego4D NLQ performance against the remaining baselines from Sec. 4.3 for results analogous to comparisons in Table 1 (left) in the main paper. Our approach outperforms baselines that have access to all extra simulated training data.

D.3. Alternate pretraining task formulations.
In Sec. 3.2.2 in the main paper, we introduced our local state prediction task that is used to pretrain our video encoders on simulated walkthrough videos. We investigate alternate pretraining objectives to validate our task choice. We compare against the following:  Each of these objectives explicitly encodes a combination of semantic and geometric information. For example, PANOFEAT and PANOCONTRAST encodes primarily semantic information as they require reconstruction of image features. POSEEMBED encodes primarily geometric information to predict the relative pose between observation pairs. CARDINALOBJ encodes semantics from object categories and weak geometric information from their relative orientations. Table 7 highlights the performance of these variants on both downstream tasks. Our approach that requires predicting both object labels, orientations as well as rough distances offers a balance of both cues during pretraining, translating to strong downstream performance.

D.4. Ablation studies.
We perform ablation experiments for several model design choices listed in Sec. 4.2 of the main paper. All abla- tion experiments are performed on on validation data splits of MP3D and HouseTours.
Window size W . The window size controls the density of sampled frames for building our environment memory. Larger windows imply temporally separated frame inputs. Our results are in Fig. 10 (left column). We find that W = 64 is sufficient for localizing the room category for ROOMPRED, while a W = 256 is best for NLQ which requires reasoning over longer horizons.
Memory size K. The memory size controls the number of frames sampled from a window of size W for building our environment memory. Our results in Fig. 10 Table 7. Alternate pretraining tasks. Our local state prediction task offers a good balance of semantic and geometric cues that lead to features with strong downstream performance. For ROOMPRED (left) we report accuracy (%). For NLQ (right) we report mean Rank1@(0.3, 0.5). column) show the sensitivity of our model to this parameter. On both tasks and datasets, we see only marginal improvements with higher memory sizes, though K = 64 results in the best performance.
Loss weight term λ. λ controls the weighting between the object and distance prediction term in the local state prediction loss in Sec. 3.2.2. Our results in Fig. 10 (right column) show that λ = 0.1 results in the best balance between the two semantic and geometric terms.

D.5. Additional attention visualizations
We present additional examples visualizing the learned attention values in our transformer decoder model in Fig. 11 to supplement Fig. 5 of the main paper. Our model learns to attend to diverse views that are not simply based on temporal adjacency or visual overlap -they capture the surroundings of the camera-wearer. D.6. Easy vs. Hard instances in the ROOMPRED task As mentioned in Sec. 4.5 of the main paper, we sort instances for evaluation by difficulty based on the prediction entropy of a pre-trained scene classifier model. In Fig. 9 (top), we show the distribution of this entropy score across all three datasets. In general, Ego4D contains the hardest instances due to characteristic egocentric motion patterns, while HouseTours contains easier examples where the camera-wearer tends to dwell in one particular location to showcase it. We show examples of easy vs. hard instances in Fig. 9 (bottom). Note that the figure only shows the center frame of the clip that is used to predict the room label to highlight the difference between easy and hard frames. Our results in Fig. 6 highlight the advantage of our approach on these hard instances where environment-level reasoning is essential. See our video in Sec. A for more context.

D.7. Memory vs. anticipation during pretraining
As mentioned in Sec. 3.2 of the main paper, our pretraining task involves elements of both aggregating information about relevant views spread across the walkthrough, as well as anticipating objects that are rarely (or never seen). We quantify this statement in Fig. 12 where we show the percentage of training instances where objects are rarely seen, for different definitions of rarity. For example, 23% of training instances involve predicting objects that appear in only k < 4 frames. 5% of training instances involve anticipating completely unseen object instances (k < 1).

D.8. Task-specific pre-training in simulation
As mentioned in Sec. 3.3 of the main paper, our goal is to train task-agnostic representations using videos from simulated agents. The end result is a single model that can generate features for multiple tasks (in our experiments, ROOM-PRED and NLQ). This is different from traditional sim-toreal approaches where a new dataset needs to be collected for every downstream task, and a separate model has to be trained on it. Such a dataset needs to be well balanced and carefully designed to match the downstream task. Moreover, as tasks are added, new datasets per task need to be created which may be impractical, especially when they require data beyond the simulator's capability (e.g., simulating human motion, hand-object interaction).
To investigate further, for the ROOMPRED task, we generate a dataset in simulation that maps trajectory frames to room labels 2 . The room categories are estimated directly from the simulator by matching each frame to the nearest navigable point in an annotated room region. We train a ResNet18 model to predict the room category (a six-way classification) and then use features from this model following baselines in Sec. 4.3. The new baseline benefits from representations learned for the same task -room prediction -and on the same volume of simulated training data.
On MP3D this performs better than the PlacesCNN baseline (43.3 vs. 42.4%) but is weaker than our model (50.4%). On HouseTours, it performs worse than PlacesCNN (57.9 vs. 58.2%) and our approach (62.7%). The low performance may be attributed to the small label space (only six categories), resulting in features that are not discriminative for the large-scale, diverse data downstream. More generally, the task-specific approach is more susceptible to failures due to the sim-to-real gap, which manifests as a lower performance on real-world video frames from HouseTours.   Percentage of training instances that involve "rare" objects. The x-axis sets a threshold for what is considered rare. For example, 5% of training instances involve anticipating completely unseen object instances (k < 1).