Understanding object motion is one of the core problems in computer vision. It requires segmenting and tracking objects over time. Significant progress has been made in instance segmentation, but such models cannot track objects, and more crucially, they are unable to reason in both 3D space and time.
 We propose a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation. Our model includes a temporal network that learns to model temporal context and motion, which is essential to produce smooth embeddings over time. Further, our model also estimates monocular depth, with a self-supervised loss, as the relative distance to an object effectively constrains where it can be next, ensuring a time-consistent embedding. Finally, we show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset. Explicitly predicting the motion of actors in a dynamic scene is a critical component of intelligent systems. Humans can seamlessly track moving objects in their environment by using cues such as appearance, relative distance, and temporal consistency. The world is rarely experienced in a static way: motion (or its absence) provides essential information to understand a scene. Similarly, incorporating past context through a temporal model is essential to segment and track objects consistently over time and through occlusions. From a computer vision perspective, understanding object motion involves segmenting instances, estimating depth, and tracking instances over time. Instance segmentation, which requires segmenting individual objects at the pixel level, has gained traction with challenging datasets such as COCO (Lin et al., 2014) , Cityscapes (Cordts et al., 2016) and Mapillary Vistas (Neuhold et al., 2017) . Such datasets, which only contain single-frame annotations, do not allow the training of video models with temporally consistent instance segmentation, nor does it allow self-supervised monocular depth estimation, that necessitates consecutive frames. Yet, navigating in the real-world involves a three-dimensional understanding of the other agents with consistent instance segmentation and depth over time. More recently, a new dataset containing video instance segmentation annotations was released, the KITTI Multi-Object and Tracking Dataset (Voigtlaender et al., 2019) . This dataset contains pixel-level instance segmentation on more than 8,000 video frames which effectively enables the training of video instance segmentation models. In this work, we propose a new spatio-temporal embedding loss that learns to map video-pixels to a high-dimensional space 1 . This space encourages video-pixels of the same instance to be close together and distinct from other instances. We show that this spatio-temporal embedding loss, jointly with a deep temporal convolutional neural network and self-supervised depth loss, produces consistent instance segmentations over time. The embedding accumulates temporal context thanks to the temporal model, as otherwise, the loss would only be based on appearance. The temporal model is a causal 3D convolutional network, which is only conditioned on past frames to predict the current embedding and is capable of real-time operation. Finally, we show that predicting depth improves the quality of the embedding as knowing the distance to an instance constrains its future location given that objects move smoothly in space. To summarise our novel contributions, we: • introduce a new spatio-temporal embedding loss for video instance segmentation, • show that having a temporal model improves embedding consistency over time, • improve how the embedding disambiguates objects with a self-supervised monocular depth loss, • handle occlusions, contrary to previous IoU based instance correspondence. We demonstrate the efficacy of our method by advancing the state-of-the-art on the KITTI MultiObject and Tracking Dataset (Voigtlaender et al., 2019 ). An example of our model's output is given by Section 1. We proposed a new spatio-temporal embedding loss that generates consistent instance segmentation over time. The temporal network models the past temporal context and the depth network constrains the embedding to aid disambiguation between objects. We demonstrated that our model could effectively track occluded instances or instances with missed detections, by leveraging the temporal context. Our method advanced the state-of-the-art at video instance segmentation on the KITTI Multi-Object and Tracking Dataset. Encoder. The encoder is a ResNet-18 convolutional layer (He et al., 2016) , with 128 output channels. Temporal model. The temporal model contains 12 residual 3D convolutional blocks, with only the first and last block convolving over time. Each residual block is the succession of: projection layer of kernel size 1×1×1 to halve the number of channels, 3D causal convolutional layer t×3×3, projection layer 1 × 1 × 1 to double the number of channels. We set the temporal kernel size to t = 2, and the number of output channels to 128. Decoders. The decoders for instance embedding and depth estimation are identical and consist of 7 convolutional layers with channels [64, 64, 32, 32, 16, 16 ] and 3 upsampling layers. The final convolutional layer contains p channels for instance embedding and 1 channel for depth. Depth Masking. During training, we remove from the photometric reprojection loss the pixels that violate the rigid scene assumption, i.e. the pixels whose appearance do not change between adjacents frames. We set the mask M to only include pixels where the reprojection error is lower with the warped imageÎ s→t than the unwarped source image I s : M = min s e(I t ,Î s→t ) < min s e(I t , I s ) Pose Network. The pose network is the succession of a ResNet-18 model followed by 4 convolutions with [256, 256, 256, 6 ] channels. The last feature map is averaged to output a single 6-DoF transformation matrix. Mask Network. The mask network is trained separately to mask the background and is the succession of the Encoder and Decoder described above.
We introduce a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation, even with occlusions and missed detections, using appearance, geometry, and temporal context.
{"src": "scitldr-AIC_train", "paper_id": "HyxTJxrtvr"}