Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> Decisions like these are made by dialogue participants as often as 2–3 times per second. For a 30 minute conversation that’s over 5000 decisions. And that’s just a fraction of what goes on. How do we do it? Face-to-face dialogue consists of interaction between several complex, dynamic systems — visual and auditory display of information, internal processing, knee-jerk reactions, thought-out rhetoric, learned patterns, social convention, etc. One could postulate that the power of dialogue is a direct result of this fact. However, combining a multitude of systems in one place does not guarantee a coherent outcome such as goal-directed dialogue. For this to happen the systems need to be architected in a way that guides their interaction and ensures that — complex as it may be — the interaction tends towards homeostasis in light of errors and uncertainties, towards the set of goals shared by participants. <s> BIB001 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> Spoken dialogue systems would be more acceptable if they were able to produce backchannel continuers such as mm-hmm in naturalistic locations during the user's utterances. Using the HCRC Map Task Corpus as our data source, we describe models for predicting these locations using only limited processing and features of the user's speech that are commonly available, and which therefore could be used as a low-cost improvement for current systems. The baseline model inserts continuers after a predetermined number of words. One further model correlates back-channel continuers with pause duration, while a second predicts their occurrence using trigram POS frequencies. Combining these two models gives the best results. <s> BIB002 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model or Conditional Random Fields) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules. <s> BIB003 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> We examine BACKCHANNEL-INVITING CUES — distinct prosodic, acoustic and lexical events in the speaker’s speech that tend to precede a short response produced by the interlocutor to convey continued attention —in theColumbiaGames Corpus, a large corpus of task-oriented dialogues. We show that the likelihood of occurrence of a backchannel increases quadratically with the number of cues conjointly displayed by the speaker. Our results are important for improving the coordination of conversational turns in interactive voice-response systems, so that systems can produce backchannels in appropriate places, and so that they can elicit backchannels from users in expected places. Index Terms: dialogue, prosody, turn-taking, backchannels. <s> BIB004 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> During face-to-face conversation, people naturally integrate speech, gestures and higher level language interpretations to predict the right time to start talking or to give backchannel feedback. In this paper we introduce a new model called Latent Mixture of Discriminative Experts which addresses some of the key issues with multimodal language processing: (1) temporal synchrony/asynchrony between modalities, (2) micro dynamics and (3) integration of different levels of interpretation. We present an empirical evaluation on listener nonverbal feedback prediction (e.g., head nod), based on observable behaviors of the speaker. We confirm the importance of combining four types of multimodal features: lexical, syntactic structure, eye gaze, and prosody. We show that our Latent Mixture of Discriminative Experts model outperforms previous approaches based on Conditional Random Fields (CRFs) and Latent-Dynamic CRFs. <s> BIB005 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> Interruptions occur frequently in spontaneous conversations, and they are often associated with changes in the flow of conversation. Predicting interruption is essential in the design of natural human-machine spoken dialog interface. The modeling can bring insights into the dynamics of human-human conversation. This work utilizes Hidden Condition Random Field (HCRF) to predict occurrences of interruption in dyadic spoken interactions by modeling both speakers' behaviors before a turn change takes place. Our prediction model, using both the foreground speaker's acoustic cues and the listener's gestural cues, achieves an F-measure of 0.54, accuracy of 70.68%, and unweighted accuracy of 66.05% on a multimodal database of dyadic interactions. The experimental results also show that listener's behaviors provides an indication of his/her intention of interruption. <s> BIB006 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> Today there are solutions for some specific turn-taking problems, but no general model. We show how turn-taking can be reduced to two more general problems, prediction and selection. We also discuss the value of predicting not only future speech/silence but also prosodic features, thereby handing not only turn-taking but “turn-shaping”. To illustrate how such predictions can be made, we trained a neural network predictor. This was adequate to support some specific turn-taking decisions and was modestly accurate overall. <s> BIB007 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> This paper compares turn-taking in terms of timing and prediction in human-human conversations under the conditions when participants has eye-contact versus when there is no eyecontact, as found in ... <s> BIB008 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Prediction of Communication Dynamics: <s> human, with realistic behaviors and social skills, evoke in users a range of social behaviors normally only seen in human face-to-face interactions. One of the key challenges in creating such virtual humans is to give them human-like conversational skills, such as turn-taking skill. In this paper, we propose a multimodal end-of-turn prediction model. Instead of recording face-to-face conversation data, we collect the turn-taking data using Parasocial Consensus Sampling (PCS) framework. Then we analyze the relationship between verbal and nonverbal features and turn-taking behaviors based on the consensus data and show how these features influence the time people use to take turns. Finally, we present a probabilistic multimodal end-of- turn prediction model, which enables virtual humans to make real-time turn-taking predictions. The result shows that our model achieves a higher accuracy than previous methods did. <s> BIB009
Turn-Taking and Backchannels As discussed in Section 2, synchrony is related to the continuous adaptation of behaviors between interaction partners. Several teams are interested "in developing predictive models of communication dynamics that integrate previous and current actions from all interlocutors to anticipate the most likely next actions of one or all interlocutors" BIB005 . The prediction of turn-taking has been largely studied in the perspective of building fluent dialog systems. The purpose of the prediction is to accurately predict the timing between speaker transitions and the upcoming type of utterance (speaker holding the floor, speaker changes) as it occurs in human-human interactions. For instance, Neiberg and Gustafson BIB008 proposed predicting whether a speaker change will occur or not at the end of an utterance given prosodic, spectral features, and the duration of the previous talkspurt. Huang et al. BIB009 proposed combining features from several modalities to predict the end of a turn. Another option to improve the performance of turn-taking predictors is to look at the behavior of both partners, instead of focusing only on the speaker. For instance, Lee and Narayanan BIB006 proposed predicting interruptions in dialog. They used Hidden Conditional Random Fields and compared three sets of features: 1) interrupter gestural features (mouth, eyebrow, and head), 2) interruptee prosodic features, and 3) an optimized combination of both first sets. They showed that the set combining features from both partners outperformed the performance of individual sets. Ward et al. BIB007 proposed several guidelines for predicting turn-taking in a dialog system. They propose that 1) predictions should be made constantly rather than at certain time points, 2) predictions should be made for several points in the future and not only for the next instant, 3) predictions should be made for the user and the system, instead of predicting the user's behavior and then determining the behavior of the system with some additional reasoning. This framework seems promising in the sense that it could handle any turn-taking pattern. Moreover, the authors propose going beyond turn-taking and predicting "turn-shaping," the prosodic parameters of the next turn (pitch or speaking rate). Back-channel behavior is intrinsically linked to turntaking and includes continuers ("hum," "aha") and regulatory gestures (head nods, shakes, laughter). Back-channel behavior assures the speaker that the listener is paying attention and is in the same state in the conversation BIB001 . Several teams have investigated how the speaker behavior triggered listeners' back-channels. For instance, Cathcart et al. proposed predicting back-channels continuers using part-of-speech tags and pause duration BIB002 . Gravano et al. studied how intonation, intensity level, pitch level, voice quality, and interpausal unit duration yielded to backchannel continuers BIB004 . Morency et al. proposed studying which speaker feature (prosody, pause, spoken words, eye gaze) is important to predict the occurrence and timing of listener's head nods BIB003 . They used sequential probabilistic modeling (Hidden Markov Model and Conditional Random Field) to learn the dynamics from a human-human interaction database. An important aspect of their model is the ability to consider the joint influence of several features to trigger a back-channel.