@inproceedings{
qianxinhuang2024similarity,
title={Similarity Preserving Transformer Cross-Modal Hashing for Video-Text Retrieval},
author={qianxinhuang and Siyao Peng and Xiaobo Shen and Yun-Hao Yuan and Shirui Pan},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=fuV2VIFNbe}
}
As social networks grow exponentially, there is an increasing demand for video  retrieval using natural language.  Cross-modal hashing that encodes multi-modal data using compact hash code has be widely used in large-scale image-text retrieval, primarily due to its computation and storage efficiency.  When applied to video-text retrieval, existing unsupervised cross-modal hashing extracts the frame- or word-level features individually, and thus ignores long-term dependencies.  In addition, effective exploit of multi-modal structure poses a significant challenge due to intricate nature of video and text. To address the above issues, we propose Similarity Preserving Transformer Cross-Modal Hashing (SPTCH), a new unsupervised deep cross-modal hashing method for video-text retrieval.  SPTCH encodes video and text by bidirectional Transformer encoder that exploits their long-term dependencies.  SPTCH constructs a multi-modal collaborative graph to model correlations among multi-modal data, and applies semantic aggregation by employing Graph Convolutional Network (GCN) on such graph.   SPTCH designs unsupervised multi-modal contrastive loss and neighborhood reconstruction loss to effectively exploit inter- and intra-modal similarity structure among videos and texts.  The empirical results on three video benchmark datasets demonstrate that the proposed SPTCH generally outperforms state-of-the-arts in video-text retrieval.
Similarity Preserving Transformer Cross-Modal Hashing for Video-Text Retrieval
qianxinhuang Siyao Peng Xiaobo Shen Yun-Hao Yuan Shirui Pan