@inproceedings{
liu2024label,
title={Label Text-aided Hierarchical Semantics Mining for Panoramic Activity Recognition},
author={Tianshan Liu and Kin-man Lam and Bingkun BAO},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=I9y11jzmNz}
}
Panoramic activity recognition is a comprehensive yet challenging task in crowd scene understanding, which aims to concurrently identify multi-grained human behaviors, including individual actions, social group activities, and global activities. Previous studies tend to capture cross-granularity activity-semantics relations from solely the video input, thus ignoring the intrinsic semantic hierarchy in label-text space. To this end, we propose a label text-aided hierarchical semantics mining (THSM) framework, which explores multi-level cross-modal associations by learning hierarchical semantic alignment between visual content and label texts. Specifically, a hierarchical encoder is first constructed to encode the visual and text inputs into semantics-aligned representations at different granularities. To fully exploit the cross-modal semantic correspondence learned by the encoder, a hierarchical decoder is further developed, which progressively integrates the lower-level representations with the higher-level contextual knowledge for coarse-to-fine action/activity recognition. Extensive experimental results on the public JRDB-PAR benchmark validate the superiority of the proposed THSM framework over state-of-the-art methods.
Label Text-aided Hierarchical Semantics Mining for Panoramic Activity Recognition
Tianshan Liu Kin-man Lam Bingkun BAO