b) Pixel-aligned feature extractor \(f_\text{enc}\) .
Pixel-aligned features help REDO make 3D predictions that are aligned with the visual 2D input.
To achieve this, we develop \(f_\text{enc}({\mathbf {q}}, I_i)\)  where the first argument is a point \({\mathbf {q}}\)  in 3D world space and the second argument is a frame \(I_i\) . We first project the point \({\mathbf {q}}\)  in world space to the frame \(I_i\) , and then use a pre-trained convolutional neural net [1]} to extract the 2D feature map of a video frame \(I_i\) . For points that fall within the instance mask of a frame, we extract its visual representation using bi-linear interpolation at the projected location. We then append a positional encoding [2]} of the frame time-step \(t_i\)  to this feature which helps to retain temporal information. We illustrate this process in Fig. REF . The resulting feature is combined with visual cues from other frames to serve as input to the temporal aggregator \(f_\text{agg}\) .
