@inproceedings{
jiang2024sparseinteraction,
title={SparseInteraction: Sparse Semantic Guidance for Radar and Camera 3D Object Detection},
author={Shengyin Jiang and Shaoqing Xu and lifang and Li Liu and Ziying Song and Yang Bo and Zhi-Xin Yang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=3XaxpSG9F8}
}
Multi-modal fusion techniques, such as radar and images, enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. However, existing fusion methods for surround-view images and radar are challenged by the inherent noise and positional ambiguity of radar, which leads to significant performance losses. To address this limitation effectively, our paper presents a robust, end-to-end fusion framework dubbed SparseInteraction. First, we introduce the Noisy Radar Filter (NRF) module to extract foreground features by creatively using queried semantic features from the image to filter out noisy radar features. Furthermore, we implement the Sparse Cross-Attention Encoder (SCAE) to effectively blend foreground radar eatures and image features to address positional ambiguity issues at a sparse level. Ultimately, to facilitate model convergence and performance, the foreground prior queries containing position information of the foreground radar are concatenated with predefined queries and fed into the subsequent transformer-based decoder. The experimental results demonstrate that the proposed fusion strategies markedly enhance detection performance and achieve new state-of-the-art results on the nuScenes benchmark. Source code is available at https://github.com/GG-Bonds/SparseInteraction.
SparseInteraction: Sparse Semantic Guidance for Radar and Camera 3D Object Detection
Shengyin Jiang Shaoqing Xu lifang Li Liu Ziying Song Yang Bo Zhi-Xin Yang