ReCo: Retrieve and Co-segment for Zero-shot Transfer
Semantic segmentation has a broad range of applications, but its real-world
impact has been significantly limited by the prohibitive annotation costs
necessary to enable deployment. Segmentation methods that forgo supervision can
side-step these costs, but exhibit the inconvenient requirement to provide
labelled examples from the target distribution to assign concept names to
predictions. An alternative line of work in language-image pre-training has
recently demonstrated the potential to produce models that can both assign
names across large vocabularies of concepts and enable zero-shot transfer for
classification, but do not demonstrate commensurate segmentation abilities. In
this work, we strive to achieve a synthesis of these two approaches that
combines their strengths. We leverage the retrieval abilities of one such
language-image pre-trained model, CLIP, to dynamically curate training sets
from unlabelled images for arbitrary collections of concept names, and leverage
the robust correspondences offered by modern image representations to
co-segment entities among the resulting collections. The synthetic segment
collections are then employed to construct a segmentation model (without
requiring pixel labels) whose knowledge of concepts is inherited from the
scalable pre-training process of CLIP. We demonstrate that our approach, termed
Retrieve and Co-segment (ReCo) performs favourably to unsupervised segmentation
approaches while inheriting the convenience of nameable predictions and
zero-shot transfer. We also demonstrate ReCo's ability to generate specialist
segmenters for extremely rare objects.