We estimate 3D scene flow by combining the known cameras, dense 2D optical flow, and per-frame depth maps.
We estimate the 2D optical flow using RAFT [1]}.
When metric depth is not available, e.g., on previous datasets [2]}, [3]}, [4]}, we use DPT [5]} for monocular depth estimation.
Additionally, we need a foreground mask for the object, which we obtain through a video segmentation network [6]}.
For each pixel location \(\mathbf {u}_t\)  at time \(t\)  in the foreground mask, we can compute its 3D position \(\mathbf {x}_t\)  by back-projection with the depth \(z_t\) .
We then get the 2D pixel correspondence at time \(t+1\)  by simply following the 2D optical flow \(\mathbf {u}_{t+1} = \mathbf {u}_t + \mathbf {f}_{t \rightarrow t+1}(\mathbf {u}_t)\) , where \(\mathbf {f}_{t\rightarrow t+1}\)  is a bilinearly interpolated forward flow map.
After back-projection, we obtain the corresponding 3D point position \(\mathbf {x}_{t+1}\)  at frame \(t+1\) .
In practice, extra care is needed for handling the unknown depth scale from model prediction and occlusion, discussed next.
