BACKGROUND

Video object segmentation aims to separate foreground objects from background regions in a video sequence. Video object segmentation is an important task in video analysis and editing and often requires significant manual interaction to achieve satisfactory results. Semi-supervised video object segmentation (Semi-VOS) segments object instances of interest from video sequences, given a fine-grained mask in the first frame is provided. While Semi-VOS has broad applications in robotics, autonomous driving, and video editing tasks, most of Semi-VOS implementations require real-time responses and therefore, speed and quality become limiting factors. Early Semi-VOS methods relied on a provided mask to fine-tune a network to fit target objects during inference, leading to slow execution speeds. Accordingly, some Semi-VOS methods remove the fine-tuning step by extracting information from provided masks and corresponding images in a feed-forward manner to predict masks for specific objects. However, such methods are still time consuming and different object masks require separate processing. As an example, the Space-Time Memory (STM) method proposes a three-step process and includes feature encoding, correlation-based feature transformations, and feature decoding. Such a process requires object masks and the related images to be encoded by a heavy encoder network to generate object specific memory features; in addition, feature transforms are recomputed for each object. The fast end-to-end embedding learning for video object segmentation method (FEELVOS) uses a semantic pixel-wise embedding together with a global and a local matching mechanism to transfer information from the first frame and from the previous frame of the video to the current frame. However, FEELVOS requires the computation of high resolution feature maps to preserve feature details; accordingly, FEELVOS may require high computational resources and may execute at a speed that is slower than other encoding-decoding approaches.

It is with respect to these and other general considerations that embodiments have been described. Although relatively specific problems have been discussed, it should be understood that the examples described herein should not be limited to solving the specific problems identified in the background above.

SUMMARY

In accordance with examples of the present disclosure, systems and methods directed to a disentangled feature transform architecture that disentangles the feature computation for individual objects. Specifically, pairwise feature transforms propagate information from previous frames to a current frame that is agnostic to any object in a video sequence. Object-specific features are then computed using an efficient lightweight network. A feature decoding module is also included and shares decoded image features across multiple target objects. In this way, the disentangled feature transform architecture requires a single pass of feature encoding and decoding of each image, and minimally adds computational costs for each object.

In accordance with at least one example of the present disclosure, a method for performing video object segmentation is described. The method may include receiving video data representing a sequence of image frames, receiving an object mask, generating image features based on a first image frame of the sequence of image frames, generating object features based on the object mask, generating image features based on a second image frame of the sequence of image frames, computing a transform matrix based on the image features of the first image frame and the image features of the second image frame, applying the transform matrix to the object features resulting in transformed object features, and obtaining a predicted object mask associated with the second image frame by decoding the transformed object features.

In accordance with at least one example of the present disclosure, a system configured to perform video object segmentation is described. The system may include a processor, and memory including instructions, which when executed by the processor, causes the processor to: receive video data representing a sequence of image frames, receive an object mask, generate image features based on a first image frame of the sequence of image frames, generate object features based on the object mask, generate image features based on a second image frame of the sequence of image frames, compute a transform matrix based on the image features of the first image frame and the image features of the second image frame, apply the transform matrix to the object features resulting in transformed object features, and obtain a predicted object mask associated with the second image frame by decoding the transformed object features.

In accordance with at least one example of the present disclosure, a computer-readable storage medium including instructions for performing video object segmentation is described. The computer-readable storage medium may include instructions, which when executed by a processor, cause the processor to: receive video data representing a sequence of image frames, receive an object mask, generate image features based on a first image frame of the sequence of image frames, generate object features based on the object mask, generate image features based on a second image frame of the sequence of image frames, compute a transform matrix based on the image features of the first image frame and the image features of the second image frame, apply the transform matrix to the object features resulting in transformed object features, and obtain a predicted object mask associated with the second image frame by decoding the transformed object features.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

