Researches on signed languages still strongly dissociate linguistic issues related on phonological and phonetic aspects, and gesture studies for recognition and synthesis purposes. This paper focuses on the imbrication of motion and meaning for the analysis, synthesis and evaluation of sign language gestures. We discuss the relevance and interest of a motor theory of perception in sign language communication. According to this theory, we consider that linguistic knowledge is mapped on sensory-motor processes, and propose a methodology based on the principle of a synthesis-by-analysis approach, guided by an evaluation process that aims to validate some hypothesis and concepts of this theory. Examples from existing studies illustrate the different concepts and provide avenues for future work.
Alternatively, data-driven animation methods can be substituted for these pure synthesis methods. In this case the motions of a real signer are captured with different combinations of motion capture techniques. Though these methods significantly improve the quality and credibility of animations, there are nonetheless several challenges to the reuse of motion capture data in the production of sign languages. Some of them are related to the spatialization of the content, but also to the rapidity and precision required in motion performances, and to the dynamic aspects of movements. All these factors are responsible for phonological inflection processes. Incorrectly manipulated, they may lead to imperfections in the performed signs (problems in timing variations or synchronization between channels) that can alter the semantic content of the sentence. A detailed discussion on the important factors for the design of virtual signers in regard to the animation problems is proposed in @cite_15 .
@cite_15
2015551990 2077367280 2001234210 2766375149
Recently, great progress has been made in sign language recognition. Most approaches are based on the Hidden Markov Model (HMM) with various features, such as motion trajectory. Recognition for sign sentences is obtained from optimal path by Viterbi algorithm, however, some wrong jumps are usually caused by transitional movements between signs. To address the problem, in this paper, we propose an approach consisting of two stages: offline training and online recognition. In the offline training stage, we propose a threshold matrix and rate thresholds. Each element of the threshold matrix describes the minimal probability when a segment belongs to a sign, and rate thresholds are defined as the average probability for signs. So, if certain segment's evaluation is smaller than all the thresholds, it is regarded as a transitional movement and then it should be removed. In the online recognition stage, coarse segmentation, based on the threshold matrix, records the time interval for fine segmentation, and fine segmentation, based on Dynamic Time Warping(DTW) and Length-Root method, determines the endpoint for each candidate sign and selects the most possible one. The final recognition is obtained by concatenating the most possible signs. We evaluate our approach with Kinect-based dataset and the experiments demonstrate the effectiveness of our approach. In this article we present a multichannel animation system for producing utterances signed in French Sign Language (LSF) by a virtual character. The main challenges of such a system are simultaneously capturing data for the entire body, including the movements of the torso, hands, and face, and developing a data-driven animation engine that takes into account the expressive characteristics of signed languages. Our approach consists of decomposing motion along different channels, representing the body parts that correspond to the linguistic components of signed languages. We show the ability of this animation system to create novel utterances in LSF, and present an evaluation by target users which highlights the importance of the respective body parts in the production of signs. We validate our framework by testing the believability and intelligibility of our virtual signer. This paper presents an animation system that generates an animation from natural language texts such as movie scripts or stories. It also proposes a framework for a motion database that stores numerous motion clips for various characters. We have developed semantic analysis methods to extract information for motion search and scheduling from script-like input texts. Given an input text, the system searches for an appropriate motion clip in the database for each verb in the input text. Temporal constraints between verbs are also extracted from the input text and are used to schedule the motion clips found. In addition, when necessary, certain automatic motions such as locomotion, taking an instrument, changing posture, and cooperative motions are searched for in the database. An animation is then generated using an external motion synthesis system. With our system, users can make use of existing motion clips. Moreover, because it takes natural language text as input, even novice users can use our system. Automatic generation of video caption is a challenging task as video is an information-intensive media with complex variations. Most existing methods, either based on language templates or sequence learning, have treated video as a flat data sequence while ignoring intrinsic multimodality nature. Observing that different modalities (e.g., frame, motion, and audio streams), as well as the elements within each modality, contribute differently to the sentence generation, we present a novel deep framework to boost video captioning by learning Multimodal Attention Long-Short Term Memory networks (MA-LSTM). Our proposed MA-LSTM fully exploits both multimodal streams and temporal attention to selectively focus on specific elements during the sentence generation. Moreover, we design a novel child-sum fusion unit in the MA-LSTM to effectively combine different encoded modalities to the initial decoding states. Different from existing approaches that employ the same LSTM structure for different modalities, we train modality-specific LSTM to capture the intrinsic representations of individual modalities. The experiments on two benchmark datasets (MSVD and MSR-VTT) show that our MA-LSTM significantly outperforms the state-of-the-art methods with 52.3 BLEU@4 and 70.4 CIDER-D metrics on MSVD dataset, respectively.