a) Temporal aggregator \(f_\text{agg}\) .
To deal with partial visibility caused by occlusions or camera settings, we develop a transformer-based temporal aggregator \(f_\text{agg}\)  as shown in Fig. REF . \(f_\text{agg}\)  is a set function which computes \(K\) -dimensional point features \({_{\mathbf {p}}}\in \mathbb {R}^K\) . Assume the flow-field \(\Phi \)  is given and object \(j\)  is of interest.
We first propagate a query point within the canonical space, , the point \({\mathbf {p}}\in \) , to
every time-step \(t_i\) , obtaining locations \(\psi ( \Phi ({\mathbf {p}}, t_c, t_i, v_\theta )) \in {\mathbb {R}}^3\)  \(\forall i\in \lbrace 1, \dots , N\rbrace \) .
We then project each 3D point \(\psi ( \Phi ({\mathbf {p}}, t_c, t_i, v_\theta ) )\)  back to the corresponding 2D image frame \(I_i\)  using the associated camera matrix.
For all points that are projected into the mask area \(m_{ij}\) , we extract a visual representation using the pixel-aligned feature extractor \(f_\text{enc}\) , which we detail below. Note that due to partial visibility, we may not be able to extract features from every frame in the clip. To cope with this, the aggregator \(f_\text{agg}\)  is designed as a transformer-based [1]} set function. For more information, please see the implementation details in § REF  and Appendix § .
