Sign language recognition is important for natural and convenient communication between deaf community and hearing majority. We take the highly efficient initial step of automatic fingerspelling recognition system using convolutional neural networks (CNNs) from depth maps. In this work, we consider relatively larger number of classes compared with the previous literature. We train CNNs for the classification of 31 alphabets and numbers using a subset of collected depth data from multiple subjects. While using different learning configurations, such as hyper-parameter selection with and without validation, we achieve 99.99 accuracy for observed signers and 83.58 to 85.49 accuracy for new signers. The result shows that accuracy improves as we include more data from different subjects during training. The processing time is 3 ms for the prediction of a single image. To the best of our knowledge, the system achieves the highest accuracy and speed. The trained model and dataset is available on our repository.
ASL sentence recognition and verification has also been explored. Zafrulla al @cite_3 proposed a system which recognizes a sentence of three to five words. The word should be one of 19 signs in their dictionary. They also used Hidden Markov Models on extracted features.
@cite_3
2146221819 2157548127 2342090483 2015551990
We present two real-time hidden Markov model-based systems for recognizing sentence-level continuous American sign language (ASL) using a single camera to track the user's unadorned hands. The first system observes the user from a desk mounted camera and achieves 92 percent word accuracy. The second system mounts the camera in a cap worn by the user and achieves 98 percent accuracy (97 percent with an unrestricted grammar). Both experiments use a 40-word lexicon. Hidden Markov models (HMMs) have been used prominently and successfully in speech recognition and, more recently, in handwriting recognition. Consequently, they seem ideal for visual recognition of complex, structured hand gestures such as are found in sign language. We describe a real-time HMM-based system for recognizing sentence level American Sign Language (ASL) which attains a word accuracy of 99.2 without explicitly modeling the fingers. We propose a new approach to modeling transition information between signs in continuous Sign Language Recognition (SLR) and address some scalability issues in designing SLR systems. In contrast to Automatic Speech Recognition (ASR) in which the transition between speech sounds is often brief and mainly addressed by the coarticulation effect, the sign transition in continuous SLR is far from being clear and usually not easily and exactly characterized. Leveraging upon hidden Markov modeling techniques from ASR, we proposed a modeling framework for continuous SLR having the following major advantages, namely: (i) the system is easy to scale up to large-vocabulary SLR; (ii) modeling of signs as well as the transitions between signs is robust even for noisy data collected in real-world SLR; and (iii) extensions to training, decoding, and adaptation are directly applicable even with new deep learning algorithms. A pair of low-cost digital gloves affordable for the deaf and hard of hearing community is used to collect a collection of training and testing data for real-world SLR interaction applications. Evaluated on 1,024 testing sentences from five signers, a word accuracy rate of 87.4p is achieved using a vocabulary of 510 words. The SLR speed is in real time, requiring an average of 0.69s per sentence. The encouraging results indicate that it is feasible to develop real-world SLR applications based on the proposed SLR framework. Recently, great progress has been made in sign language recognition. Most approaches are based on the Hidden Markov Model (HMM) with various features, such as motion trajectory. Recognition for sign sentences is obtained from optimal path by Viterbi algorithm, however, some wrong jumps are usually caused by transitional movements between signs. To address the problem, in this paper, we propose an approach consisting of two stages: offline training and online recognition. In the offline training stage, we propose a threshold matrix and rate thresholds. Each element of the threshold matrix describes the minimal probability when a segment belongs to a sign, and rate thresholds are defined as the average probability for signs. So, if certain segment's evaluation is smaller than all the thresholds, it is regarded as a transitional movement and then it should be removed. In the online recognition stage, coarse segmentation, based on the threshold matrix, records the time interval for fine segmentation, and fine segmentation, based on Dynamic Time Warping(DTW) and Length-Root method, determines the endpoint for each candidate sign and selects the most possible one. The final recognition is obtained by concatenating the most possible signs. We evaluate our approach with Kinect-based dataset and the experiments demonstrate the effectiveness of our approach.