@inproceedings{
he2024sniffing,
title={Sniffing Threatening Open-World Objects in Autonomous Driving by Open-Vocabulary Models},
author={Yulin He and Siqi Wang and Wei Chen and Tianci Xun and Yusong Tan},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=5RPP4QoFKw}
}
Autonomous driving (AD) is a typical application that requires effectively exploiting multimedia information. For AD, it is critical to ensure safety by detecting unknown objects in an open world, driving the demand for open world object detection (OWOD). However, existing OWOD methods treat generic objects beyond known classes in the training set as unknown objects and prioritize recall in evaluation. This encourages excessive false positives and endangers safety of AD. To address this issue, we restrict the definition of unknown objects to threatening objects in AD, and introduce a new evaluation protocol, which is built upon a new metric named U-ARecall, to alleviate biased evaluation caused by neglecting false positives. Under the new evaluation protocol, we re-evaluate existing OWOD methods and discover that they typically perform poorly in AD. Then, we propose a novel OWOD paradigm for AD based on fine-tuning foundational open-vocabulary models (OVMs), as they can exploit rich linguistic and visual prior knowledge for OWOD. Following this new paradigm, we propose a brand-new OWOD solution, which effectively addresses two core challenges of fine-tuning OVMs via two novel techniques: 1) the maintenance of open-world generic knowledge by a dual-branch architecture; 2) the acquisition of scenario-specific knowledge by the visual-oriented contrastive learning scheme. Besides, a dual-branch prediction fusion module is proposed to avoid post-processing and hand-crafted heuristics. Extensive experiments show that our proposed method not only surpasses classic OWOD methods in unknown object detection by a large margin ($\sim$3$\times$ U-ARecall), but also notably outperforms OVMs without fine-tuning in known object detection ($\sim$ 20\% K-mAP). Our codes are available at https://github.com/harrylin-hyl/AD-OWOD.
Sniffing Threatening Open-World Objects in Autonomous Driving by Open-Vocabulary Models
Yulin He Siqi Wang Wei Chen Tianci Xun Yusong Tan