@inproceedings{
wang2024efficient,
title={Efficient Temporal Action Segmentation via Boundary-aware Query Voting},
author={Peiyao Wang and Yuewei Lin and Erik Blasch and Jie Wei and Haibin Ling},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=jij4vOVU7i}
}
Although the performance of Temporal Action Segmentation (TAS) has been improved in recent years, achieving promising results often comes with a high computational cost due to dense inputs, complex model structures, and resource-intensive post-processing requirements. To improve the efficiency while keeping the high performance, we present a novel perspective centered on per-segment classification. By harnessing the capabilities of Transformers, we tokenize each video segment as an instance token, endowed with intrinsic instance segmentation. To realize efficient action segmentation, we introduce BaFormer, a boundary-aware Transformer network. It employs instance queries for instance segmentation and a global query for class-agnostic boundary prediction, yielding continuous segment proposals. During inference, BaFormer employs a simple yet effective voting strategy to classify boundary-wise segments based on instance segmentation. Remarkably, as a single-stage approach, BaFormer significantly reduces the computational costs, utilizing only 6% of the running time compared to the state-of-the-art method DiffAct, while producing better or comparable accuracy over several popular benchmarks. The code for this project is publicly available at  https://github.com/peiyao-w/BaFormer.
Efficient Temporal Action Segmentation via Boundary-aware Query Voting
Peiyao Wang Yuewei Lin Erik Blasch Jie Wei Haibin Ling