Additionally, we also add dense-only loss that omits the sparse logits (i.e. original loss in<cite> Seo et al. (2019)</cite> ) to the final loss, in which case we find that we obtain higher-quality dense phrase representations.