Transformer-based VIS:
Wang et al. [1]} introduce a transformer-based encoder-decoder architecture, named VisTR, that formulates VIS as a direct end-to-end parallel sequence prediction task.
In the encoder, VisTR utilizes a single-scale attention that computes similarities between all pairs of features from multiple spatial locations of a low-resolution feature map, across frames. The work of [2]} introduces inter-frame communication transformers, where memory tokens are used to communicate between frames. The recent SeqFormer [3]}, built on Deformable DETR framework [4]}, utilizes per-frame multi-scale features during attention computations. While demonstrating promising results, SeqFormer struggles in case of target deformations likely due to not explicitly capturing the spatio-temporal feature relationships during attention computation. To address these issues, we proposed a framework comprising an encoder that captures multi-scale spatio-temporal feature relationships. We also introduced an attention block in decoder to enhance temporal consistency of detected instance in different frames and an adversarial loss during training that ensures better fg-bg separability within multi-scale spatio-temporal feature space.
