For semi-supervised video object segmentation, the annotated mask of the first frame is given, and the algorithm is designed to predict the masks of the rest frames in the video. There are three categories in this spectrum. The first one, which include MSK [1]}, MPNVOS [2]} etc, is to use optical flow to track the mask from the previous frame to the current frame. Similarly, the second category formulates the optical flow and segmentation in two parallel branches, and utilizes the predicted mask from the previous frame as a guidance, some representatives are
Segflow [3]},VSOF [4]}, RGMP [5]}, OSNM [6]} etc. The final class which keeps the state-of-the-art performance on Davis benchmark [7]} is to try to over-fit the appearance of the target object(s), and expect the method can generalize in the subsequent frames. Specifically, OSVOS [8]} uses one-shot learning mechanism to conduct fine-tuning on the first frame of test video to capture the appearance of the target object(s), and conduct inference on the rest frames. The drawbacks of OSVOS are: (1) it can not adapt to the unseen parts (2) when dramatic changes of appearance happen in subsequent frames, the method's performance significantly degrade. Inspired by the overall design of OSVOS, there are some following methods which employ various additional ingredients to improve the segmentation accuracy.
In particular, OSVOS-S [9]} combines the semantic instance information to remove the noisy objects coming from the same category. OnVOS [10]} utilizes on-line adaption mechanism to overcome the limits of OSVOS when drastic appearance changes occur. CINM [11]} utilizes a CNN-based markov random field (MRF) to estimate the probabilities of the pixels belonging to the target object(s) in spatial domain, and employs optical flow to track segmented pixels in temporal domain. Albeit those methods improve the segmentation performance of OSVOS to some extent, they are still time-consuming during inference since the on-line fine-tuning is necessary. And usually, at lease a dense CRF [12]} and more techniques are applied as the post-processing step to acquire the better segmentation results.
