Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures. The project website with video, code and data can be found at this http URL .
McNeill @cite_6 divides gestures into several classes @cite_6 : have specific conventional meanings ( thumbs up!''); convey physical shapes or direction of movements; describe abstract content using concrete motion; are pointing gestures, and are repetitive, fast hand motions that provide a temporal framing to speech.
@cite_6
2118163921 1708911126 2008496867 2124120479
What is the relation between gestures and speech? In terms of symbolic forms, of course, the spontaneous and unwitting gestures we make while talking differ sharply from spoken language itself. Whereas spoken language is linear, segmented, standardized, and arbitrary, gestures are global, synthetic, idiosyncratic, and imagistic. In Hand and Mind, David McNeill presents a bold theory of the essential unity of speech and the gestures that accompany it. This long-awaited, provocative study argues that the unity of gestures and language far exceeds the surface level of speech noted by previous researchers and in fact also includes the semantic and pragmatic levels of language. In effect, the whole concept of language must be altered to take into account the nonsegmented, instantaneous, and holistic images conveyed by gestures. McNeill and his colleagues carefully devised a standard methodology for examining the speech and gesture behavior of individuals engaged in narrative discourse. A research subject is shown a cartoon like the 1950 Canary Row--a classic Sylvester and Tweedy Bird caper that features Sylvester climbing up a downspout, swallowing a bowling ball and slamming into a brick wall. After watching the cartoon, the subject is videotaped recounting the story from memory to a listener who has not seen the cartoon. Painstaking analysis of the videotapes revealed that although the research subjects--children as well as adults, some neurologically impaired--represented a wide variety of linguistic groupings, the gestures of people speaking English and a half dozen other languages manifest the same principles. Relying on data from more than ten years of research, McNeill shows thatgestures do not simply form a part of what is said and meant but have an impact on thought itself. He persuasively argues that because gestures directly transfer mental images to visible forms, conveying ideas that language cannot always express, we must examine language and gesture Gesturing is such an integral yet unconscious part of communication that we are mostly oblivious to it. But if you observe anyone in conversation, you are likely to see his or her fingers, hands, and arms in some form of spontaneous motion. Why? David McNeill, a pioneer in the ongoing study of the relationship between gesture and language, set about answering this question in "Gesture and Thought" with an unlikely accomplice - Tweety Bird. McNeill argues that gestures are active participants in both speaking and thinking. He posits that gestures are key ingredients in an "imagery-language dialectic" that fuels speech and thought; gestures are the "imagery" and also the components of "language," rather than mere consequences. The smallest unit of this dialectic is the "growth point," a snapshot of an utterance at its beginning psychological stage. Enter Tweety Bird. In "Gesture and Thought", the central growth point comes from a cartoon. In his quest to eat Tweety Bird, Sylvester the cat first scales the outside of a rain gutter to reach his prey. Unsuccessful, he makes a second attempt by climbing up the inside of the gutter. Tweety, however, drops a bowling ball down the gutter; Sylvester swallows the ball. Over the course of twenty-five years, McNeill showed this cartoon to numerous subjects who spoke a variety of languages. A fascinating pattern emerged. Those who remembered the exact sequence of the cartoon while retelling it all used the same gesture to describe Sylvester's position inside the gutter. Those who forgot, in the retelling, that Sylvester had first climbed the outside of the gutter did not use this gesture at all. Thus that gesture becomes part of the "growth point" - the building block of language and thought. An ambitious project in the ongoing study of the relationship of how we communicate and its connection to thought, "Gesture and Thought" is a work of such consequence that it will influence all subsequent linguistic and evolutionary theory on the subject. McNeill (1985) claimed that speech and its accompanying gestures "share a common computational stage" that is located early in the speech production process. This position depends on two crucial arguments that we believe are unsound: first, that gestures "are synchronized with linguistic units in speech" and, second, that gestures "have semantic and pragmatic functions that parallel those of speech." Careful analysis of the relevant data indicated that although the processes underlying gestural production are globally autonomous, they are linked to speech production at more than just one computational stage. An account is offered that explicates these links. An experiment was conducted to investigate gesture recognition with a human hand manipulating the DataGlove, an electronically instrumented glove which provides information about finger and hand position. A total of 22 gestures in three classes were investigated. The first class contained gestures which only involved finger flexure. The second class contained gestures which required both finger flexure and hand orientation. The third class of gestures required finger motion in addition to flexure and orientation. Only four sensors were necessary to positively identify specific gestures from groups of up to 15 gestures. The results show the specific number of sensors required to positively identify a gesture from a group. This depends on the number of gestures in a group, as well as the class of gestures. >