We address the problem of automatic American Sign Language fingerspelling recognition from video. Prior work has largely relied on frame-level labels, hand-crafted features, or other constraints, and has been hampered by the scarcity of data for this task. We introduce a model for fingerspelling recognition that addresses these issues. The model consists of an auto-encoder-based feature extractor and an attention-based neural encoder-decoder, which are trained jointly. The model receives a sequence of image frames and outputs the fingerspelled word, without relying on any frame-level training labels or hand-crafted features. In addition, the auto-encoder subcomponent makes it possible to leverage unlabeled data to improve the feature learning. The model achieves 11.6 and 4.4 absolute letter accuracy improvement respectively in signer-independent and signer- adapted fingerspelling recognition over previous approaches that required frame-level training labels.
Despite the importance of fingerspelling in spontaneous sign language, there has been relatively little work explicitly addressing fingerspelling recognition. Most prior work on fingerspelling recognition is focused on restricted settings. One typical restriction is the size of the lexicon. When the lexicon is fixed to a small size (20-100 words), excellent recognition accuracy has been achieved @cite_23 @cite_11 @cite_27 , but this restriction is impractical. For ASL fingerspelling, the largest available open-vocabulary dataset to our knowledge is the TTIC UChicago Fingerspelling Video Dataset (Chicago-FSVid), containing 2400 open-domain word instances produced by 4 signers @cite_8 , which we use here. Another important restriction is the signer identity. In the signer-dependent setting, letter error rates below 10 The best-performing prior approaches for open-vocabulary fingerspelling recognition have been based on HMMs or segmental conditional random fields (SCRFs) using deep neural network (DNN) frame classifiers to define features @cite_8 . This prior work has largely relied on frame-level labels for training data, but these are hard to obtain. In addition, because of the scarcity of data, prior work has largely relied on human-engineered image features, such as histograms of oriented gradients (HOG) @cite_0 , as the initial image representation.
@cite_8 @cite_0 @cite_27 @cite_23 @cite_11
2963408148 2964081342 1580385328 2122585011
Abstract We study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL. Recognizing fingerspelling is challenging for a number of reasons: it involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work we collect and annotate a new data set of continuous fingerspelling videos, compare several types of recognizers, and explore the problem of signer variation. Our best-performing models are segmental (semi-Markov) conditional random fields using deep neural network-based features. In the signer-dependent setting, our recognizers achieve up to about 92 letter accuracy. The multi-signer setting is much more challenging, but with neural network adaptation we achieve up to 83 letter accuracies in this setting. Sign language recognition is important for natural and convenient communication between deaf community and hearing majority. We take the highly efficient initial step of automatic fingerspelling recognition system using convolutional neural networks (CNNs) from depth maps. In this work, we consider relatively larger number of classes compared with the previous literature. We train CNNs for the classification of 31 alphabets and numbers using a subset of collected depth data from multiple subjects. While using different learning configurations, such as hyper-parameter selection with and without validation, we achieve 99.99 accuracy for observed signers and 83.58 to 85.49 accuracy for new signers. The result shows that accuracy improves as we include more data from different subjects during training. The processing time is 3 ms for the prediction of a single image. To the best of our knowledge, the system achieves the highest accuracy and speed. The trained model and dataset is available on our repository1. We propose a new principle for recognizing fingerspelling sequences from American Sign Language (ASL) Instead of training a system to recognize the static posture for each letter from an isolated frame, we recognize the dynamic gestures corresponding to transitions between letters This eliminates the need for an explicit temporal segmentation step, which we show is error-prone at speeds used by native signers We present results from our system recognizing 82 different words signed by a single signer, using more than an hour of training and test video We demonstrate that recognizing letter-to-letter transitions without temporal segmentation is feasible and results in improved performance. Recognizing lines of unconstrained handwritten text is a challenging task. The difficulty of segmenting cursive or overlapping characters, combined with the need to exploit surrounding context, has led to low recognition rates for even the best current recognizers. Most recent progress in the field has been made either through improved preprocessing or through advances in language modeling. Relatively little work has been done on the basic recognition algorithms. Indeed, most systems rely on the same hidden Markov models that have been used for decades in speech and handwriting recognition, despite their well-known shortcomings. This paper proposes an alternative approach based on a novel type of recurrent neural network, specifically designed for sequence labeling tasks where the data is hard to segment and contains long-range bidirectional interdependencies. In experiments on two large unconstrained handwriting databases, our approach achieves word recognition accuracies of 79.7 percent on online data and 74.1 percent on offline data, significantly outperforming a state-of-the-art HMM-based system. In addition, we demonstrate the network's robustness to lexicon size, measure the individual influence of its hidden layers, and analyze its use of context. Last, we provide an in-depth discussion of the differences between the network and HMMs, suggesting reasons for the network's superior performance.