We have investigated two other alternatives to represent an instance, but neither works well. If we feed the instance token as the input to the transformer as in ViT [1]}, optimization becomes unstable, potentially due to the large masking rate for the input. With average pooling over the token representations as the instance representation, the model would find a shortcut on learning averaged patch features without learning attention across patches.
