Snapture -- A Novel Neural Architecture for Combined Static and Dynamic Hand Gesture Recognition
As robots are expected to get more involved in people's everyday lives,
frameworks that enable intuitive user interfaces are in demand. Hand gesture
recognition systems provide a natural way of communication and, thus, are an
integral part of seamless Human-Robot Interaction (HRI). Recent years have
witnessed an immense evolution of computational models powered by deep
learning. However, state-of-the-art models fall short in expanding across
different gesture domains, such as emblems and co-speech. In this paper, we
propose a novel hybrid hand gesture recognition system. Our architecture
enables learning both static and dynamic gestures: by capturing a so-called
"snapshot" of the gesture performance at its peak, we integrate the hand pose
along with the dynamic movement. Moreover, we present a method for analyzing
the motion profile of a gesture to uncover its dynamic characteristics and
which allows regulating a static channel based on the amount of motion. Our
evaluation demonstrates the superiority of our approach on two gesture
benchmarks compared to a CNNLSTM baseline. We also provide an analysis on a
gesture class basis that unveils the potential of our Snapture architecture for
performance improvements. Thanks to its modular implementation, our framework
allows the integration of other multimodal data like facial expressions and
head tracking, which are important cues in HRI scenarios, into one
architecture. Thus, our work contributes both to gesture recognition research
and machine learning applications for non-verbal communication with robots.