@inproceedings{
huxiaowan2024spatiotemporal,
title={Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval},
author={HuXiaowan and Yiyi Chen and Yan Li and Minquan Wang and Haoqian Wang and Quan Chen and Han Li and Peng Jiang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=BGQkeDHSio}
}
With the rapid expansion of e-commerce, more consumers have become accustomed to making purchases via livestreaming. Accurately identifying the products being sold by salespeople, i.e., livestreaming product retrieval (LPR), poses a fundamental and daunting challenge. The LPR task encompasses three primary dilemmas in real-world scenarios: 1) the recognition of intended products from distractor products present in the background; 2) the video-image heterogeneity that the appearance of products showcased in live streams often deviates substantially from standardized product images in stores; 3) there are numerous confusing products with subtle visual nuances in the shop. To tackle these challenges, we propose the Spatiotemporal Graphing Multi-modal Network (SGMN). First, we employ a text-guided attention mechanism that leverages the spoken content of salespeople to guide the model to focus toward intended products, emphasizing their salience over cluttered background products. Second, a long-range spatiotemporal graph network is further designed to achieve both instance-level interaction and frame-level matching, solving the misalignment caused by video-image heterogeneity. Third, we propose a multi-modal hard example mining, assisting the model in distinguishing highly similar products with fine-grained features across the video-image-text domain. Through extensive quantitative and qualitative experiments, we demonstrate the superior performance of our proposed SGMN model, surpassing the state-of-the-art methods by a substantial margin. The code and models will be public soon.
Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval
HuXiaowan Yiyi Chen Yan Li Minquan Wang Haoqian Wang Quan Chen Han Li Peng Jiang