Inspired by the meta-learning method of [1]}, we propose an intuitive yet powerful algorithm for video object segmentation, in which the reference frame is available with its annotated mask. Our objective is to train a system that can “adapt” this annotation information to subsequent frames in a fast yet flexible way at inference time.
Specifically, at inference time the reference frame (i.e. one with ground-truth annotation) is mapped to vector in a high dimensional embedding space \(X = \phi (I)\)  using a CNN \(\phi \) . We then determine using ridge regression [2]}, the coefficients of a matrix \(W\)  that best maps \(X\)  to the ground truth, \(Y=WX\) . \(W\)  is then the video-specific “adaptor”, and it maps the feature vectors for every query image (every other image in the video sequence) to their predicted segmentation masks. Training comprises the process of learning the mapping \(\phi (.)\)  by presenting the network with pairs of images (from a variety of videos but with each pair coming from the same video), each with ground-truth annotation, and back-propagating the loss through \(\phi \) . This is illustrated in Figure REF  and described in more detail later in the paper.
