@inproceedings{liu-etal-2024-unifying,
    title = "Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval",
    author = "Liu, Haowei  and
      Shi, Yaya  and
      Xu, Haiyang  and
      Yuan, Chunfeng  and
      Ye, Qinghao  and
      Li, Chenliang  and
      Yan, Ming  and
      Zhang, Ji  and
      Huang, Fei  and
      Li, Bing  and
      Hu, Weiming",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1480",
    pages = "17031--17041",
    abstract = "In video-text retrieval, most existing methods adopt the dual-encoder architecture for fast retrieval, which employs two individual encoders to extract global latent representations for videos and texts. However, they face challenges in capturing fine-grained semantic concepts. In this work, we propose the UNIFY framework, which learns lexicon representations to capture fine-grained semantics and combines the strengths of latent and lexicon representations for video-text retrieval. Specifically, we map videos and texts into a pre-defined lexicon space, where each dimension corresponds to a semantic concept. A two-stage semantics grounding approach is proposed to activate semantically relevant dimensions and suppress irrelevant dimensions. The learned lexicon representations can thus reflect fine-grained semantics of videos and texts. Furthermore, to leverage the complementarity between latent and lexicon representations, we propose a unified learning scheme to facilitate mutual learning via structure sharing and self-distillation. Experimental results show our UNIFY framework largely outperforms previous video-text retrieval methods, with 4.8{\%} and 8.2{\%} Recall@1 improvement on MSR-VTT and DiDeMo respectively.",
}

In video-text retrieval, most existing methods adopt the dual-encoder architecture for fast retrieval, which employs two individual encoders to extract global latent representations for videos and texts. However, they face challenges in capturing fine-grained semantic concepts. In this work, we propose the UNIFY framework, which learns lexicon representations to capture fine-grained semantics and combines the strengths of latent and lexicon representations for video-text retrieval. Specifically, we map videos and texts into a pre-defined lexicon space, where each dimension corresponds to a semantic concept. A two-stage semantics grounding approach is proposed to activate semantically relevant dimensions and suppress irrelevant dimensions. The learned lexicon representations can thus reflect fine-grained semantics of videos and texts. Furthermore, to leverage the complementarity between latent and lexicon representations, we propose a unified learning scheme to facilitate mutual learning via structure sharing and self-distillation. Experimental results show our UNIFY framework largely outperforms previous video-text retrieval methods, with 4.8{\%} and 8.2{\%} Recall@1 improvement on MSR-VTT and DiDeMo respectively.
Liu, Haowei Shi, Yaya Xu, Haiyang Yuan, Chunfeng Ye, Qinghao Li, Chenliang Yan, Ming Zhang, Ji Huang, Fei Li, Bing Hu, Weiming
Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval