@inproceedings{
liling2024translating,
title={Translating Motion to Notation: Hand Labanotation for Intuitive and Comprehensive Hand Movement Documentation},
author={LiLing and Wenrui Yang and Xinchun Yu and Junliang Xing and Xiao-Ping Zhang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=7B93ulPZ9K}
}
Symbols play a pivotal role in the documentation and dissemination of art. For instance, we use musical scores and dance notation to document musical compositions and choreographic movements. Existing hand representations do not fit well with hand movement documentation since (1) data-oriented representations, e.g., coordinates of hand keypoints, are not intuitive and vulnerable to noise, and (2) the sign language, another widely adopted representation for hand movements, focuses solely on semantic interaction rather than action encoding. To balance intuitiveness and precision, we propose a novel notation system, named Hand Labanotation (HL), for hand movement documentation. We first introduce a new HL dataset comprising $4$M annotated images. Thereon, we propose a novel multi-view transformer architecture for automatically translating hand movements to HL. Extensive experiments demonstrate the promising capacity of our method for representing hand movements. This makes our method a general tool for hand movement documentation, driving various downstream applications like using HL to control robotic hands.
Translating Motion to Notation: Hand Labanotation for Intuitive and Comprehensive Hand Movement Documentation
LiLing Wenrui Yang Xinchun Yu Junliang Xing Xiao-Ping Zhang