Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-modal translation from "in-the-wild'' monologue speech of a single speaker to their hand and arm motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures. The project website with video, code and data can be found at this http URL .
Conditioning gesture prediction on speech is arguably an ambiguous task, since gesture and speech may not be synchronous. While McNeill @cite_6 suggests that gesture and speech originate from a common source and thus should co-occur in time according to well-defined rules, Kendon @cite_10 suggests that gesture starts before the corresponding utterance. Others even argue that the temporal relationships between speech and gesture are not yet clear and that gesture can appear before, after or during an utterance @cite_1 .
@cite_1 @cite_10 @cite_6
2008496867 1989386832 2118163921 1708911126
McNeill (1985) claimed that speech and its accompanying gestures "share a common computational stage" that is located early in the speech production process. This position depends on two crucial arguments that we believe are unsound: first, that gestures "are synchronized with linguistic units in speech" and, second, that gestures "have semantic and pragmatic functions that parallel those of speech." Careful analysis of the relevant data indicated that although the processes underlying gestural production are globally autonomous, they are linked to speech production at more than just one computational stage. An account is offered that explicates these links. Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues. What is the relation between gestures and speech? In terms of symbolic forms, of course, the spontaneous and unwitting gestures we make while talking differ sharply from spoken language itself. Whereas spoken language is linear, segmented, standardized, and arbitrary, gestures are global, synthetic, idiosyncratic, and imagistic. In Hand and Mind, David McNeill presents a bold theory of the essential unity of speech and the gestures that accompany it. This long-awaited, provocative study argues that the unity of gestures and language far exceeds the surface level of speech noted by previous researchers and in fact also includes the semantic and pragmatic levels of language. In effect, the whole concept of language must be altered to take into account the nonsegmented, instantaneous, and holistic images conveyed by gestures. McNeill and his colleagues carefully devised a standard methodology for examining the speech and gesture behavior of individuals engaged in narrative discourse. A research subject is shown a cartoon like the 1950 Canary Row--a classic Sylvester and Tweedy Bird caper that features Sylvester climbing up a downspout, swallowing a bowling ball and slamming into a brick wall. After watching the cartoon, the subject is videotaped recounting the story from memory to a listener who has not seen the cartoon. Painstaking analysis of the videotapes revealed that although the research subjects--children as well as adults, some neurologically impaired--represented a wide variety of linguistic groupings, the gestures of people speaking English and a half dozen other languages manifest the same principles. Relying on data from more than ten years of research, McNeill shows thatgestures do not simply form a part of what is said and meant but have an impact on thought itself. He persuasively argues that because gestures directly transfer mental images to visible forms, conveying ideas that language cannot always express, we must examine language and gesture Gesturing is such an integral yet unconscious part of communication that we are mostly oblivious to it. But if you observe anyone in conversation, you are likely to see his or her fingers, hands, and arms in some form of spontaneous motion. Why? David McNeill, a pioneer in the ongoing study of the relationship between gesture and language, set about answering this question in "Gesture and Thought" with an unlikely accomplice - Tweety Bird. McNeill argues that gestures are active participants in both speaking and thinking. He posits that gestures are key ingredients in an "imagery-language dialectic" that fuels speech and thought; gestures are the "imagery" and also the components of "language," rather than mere consequences. The smallest unit of this dialectic is the "growth point," a snapshot of an utterance at its beginning psychological stage. Enter Tweety Bird. In "Gesture and Thought", the central growth point comes from a cartoon. In his quest to eat Tweety Bird, Sylvester the cat first scales the outside of a rain gutter to reach his prey. Unsuccessful, he makes a second attempt by climbing up the inside of the gutter. Tweety, however, drops a bowling ball down the gutter; Sylvester swallows the ball. Over the course of twenty-five years, McNeill showed this cartoon to numerous subjects who spoke a variety of languages. A fascinating pattern emerged. Those who remembered the exact sequence of the cartoon while retelling it all used the same gesture to describe Sylvester's position inside the gutter. Those who forgot, in the retelling, that Sylvester had first climbed the outside of the gutter did not use this gesture at all. Thus that gesture becomes part of the "growth point" - the building block of language and thought. An ambitious project in the ongoing study of the relationship of how we communicate and its connection to thought, "Gesture and Thought" is a work of such consequence that it will influence all subsequent linguistic and evolutionary theory on the subject.