Self Supervised Relationship Probing
Structured representations of images that model visual relationships are beneficial for many vision and vision language applications. However, current humanannotated visual relationship datasets suffer from the long tailed predicate distribution problem which limits the potential of visual relationship models. In this work, we introduce a self supervised method that implicitly learns the visual relationships without relying on any ground truth visual relationship annotations. Our method relies on 1) intraand inter modality encodings to respectively model relationships within each modality separately and jointly, and 2) relationship probing, which seeks to discover the graph structure within each modality. By leveraging masked language modeling, contrastive learning, and dependency tree distances for self supervision, our method learns better object features as well as implicit visual relationships. We verify the effectiveness of our proposed method on various vision language tasks that benefit from improved visual relationship understanding.
NeurIPS