@inproceedings{osebe-etal-2024-towards,
    title = "Towards Multi-Modal Co-Reference Resolution in Conversational Shopping Agents",
    author = "Osebe, Samuel  and
      Wanigasekara, Prashan  and
      Gueudre, Thomas  and
      Tran, Thanh  and
      Sharma, Rahul  and
      Yang, Fan  and
      Hu, Qian  and
      Ruan, Weitong  and
      Barut, Emre  and
      Su, Chengwei",
    editor = "Malmasi, Shervin  and
      Fetahu, Besnik  and
      Ueffing, Nicola  and
      Rokhlenko, Oleg  and
      Agichtein, Eugene  and
      Guy, Ido",
    booktitle = "Proceedings of the Seventh Workshop on e-Commerce and NLP @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.ecnlp-1.2",
    pages = "8--18",
    abstract = "The context of modern smart voice assistants is often multi-modal, where images, audio and video content are consumed by users simultaneously. In such a setup, co-reference resolution is especially challenging, and runs across modalities and dialogue turns. We explore the problem of multi-modal co-reference resolution in multi-turn dialogues and quantify the performance of multi-modal LLMs on a specially curated dataset of long, image-interleaved conversations between a voice assistant and human in a shopping use case. We propose a custom architecture for multi-modal embedding alignment using a novel parameter augmentation technique. Our proposed Parameter Augmented LLM approach shows a 4.9{\%} absolute F1 improvement above a cross-attention baseline while reducing the number of parameters being trained by 4x.",
}

The context of modern smart voice assistants is often multi-modal, where images, audio and video content are consumed by users simultaneously. In such a setup, co-reference resolution is especially challenging, and runs across modalities and dialogue turns. We explore the problem of multi-modal co-reference resolution in multi-turn dialogues and quantify the performance of multi-modal LLMs on a specially curated dataset of long, image-interleaved conversations between a voice assistant and human in a shopping use case. We propose a custom architecture for multi-modal embedding alignment using a novel parameter augmentation technique. Our proposed Parameter Augmented LLM approach shows a 4.9{\%} absolute F1 improvement above a cross-attention baseline while reducing the number of parameters being trained by 4x.
Osebe, Samuel Wanigasekara, Prashan Gueudre, Thomas Tran, Thanh Sharma, Rahul Yang, Fan Hu, Qian Ruan, Weitong Barut, Emre Su, Chengwei
Towards Multi-Modal Co-Reference Resolution in Conversational Shopping Agents