We address the problem of automatic American Sign Language fingerspelling recognition from video. Prior work has largely relied on frame-level labels, hand-crafted features, or other constraints, and has been hampered by the scarcity of data for this task. We introduce a model for fingerspelling recognition that addresses these issues. The model consists of an auto-encoder-based feature extractor and an attention-based neural encoder-decoder, which are trained jointly. The model receives a sequence of image frames and outputs the fingerspelled word, without relying on any frame-level training labels or hand-crafted features. In addition, the auto-encoder subcomponent makes it possible to leverage unlabeled data to improve the feature learning. The model achieves 11.6 and 4.4 absolute letter accuracy improvement respectively in signer-independent and signer- adapted fingerspelling recognition over previous approaches that required frame-level training labels.
Abstract We study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL. Recognizing fingerspelling is challenging for a number of reasons: it involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work we collect and annotate a new data set of continuous fingerspelling videos, compare several types of recognizers, and explore the problem of signer variation. Our best-performing models are segmental (semi-Markov) conditional random fields using deep neural network-based features. In the signer-dependent setting, our recognizers achieve up to about 92 letter accuracy. The multi-signer setting is much more challenging, but with neural network adaptation we achieve up to 83 letter accuracies in this setting.  We propose a new principle for recognizing fingerspelling sequences from American Sign Language (ASL) Instead of training a system to recognize the static posture for each letter from an isolated frame, we recognize the dynamic gestures corresponding to transitions between letters This eliminates the need for an explicit temporal segmentation step, which we show is error-prone at speeds used by native signers We present results from our system recognizing 82 different words signed by a single signer, using more than an hour of training and test video We demonstrate that recognizing letter-to-letter transitions without temporal segmentation is feasible and results in improved performance. This paper presents the Australian sign language (Auslan) Fingerspelling Recognizer (AFR): a system capable of recognizing signs consisting of Auslan manual alphabet letters from video sequences. The AFR system uses a combination of geometric features and motion features based on optical flow which are extracted from video sequences. The sequence of features are then classified using Hidden Markov Models (HMMs). Tests using a vocabulary of twenty signed words showed the system could achieve 97 accuracy at the letter level and 88 at the word level by using a finite state grammar network and embedded training. We investigate the problem of recognizing words from video, fingerspelled using the British Sign Language (BSL) fingerspelling alphabet. This is a challenging task since the BSL alphabet involves both hands occluding each other, and contains signs which are ambiguous from the observer's viewpoint. The main contributions of our work include: (i) recognition based on hand shape alone, not requiring motion cues; (ii) robust visual features for hand shape recognition; (iii) scalability to large lexicon recognition with no re-training. We report results on a dataset of 1,000 low quality webcam videos of 100 words. The proposed method achieves a word recognition accuracy of 98.9 .