Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Significance Test: Bootstrap <s> Looking To Understand: The Coupling Between Speakers’ and Listeners’ Eye Movements and its Relationship to Discourse Comprehension Daniel C. Richardson (richardson@psych.stanford.edu) Department of Psychology, Stanford University Stanford, CA 94305, USA Rick Dale (rad28@cornell.edu) Department of Psychology, Cornell University Ithaca, NY 14853, USA Abstract discussing a diagram drawn on a whiteboard, figuring out together how to do something on a computer, or talking during a movie. Uniquely poised between perception and cognition, eye movements can reveal cognitive processes such as speech planning, language comprehension, memory, mental imagery and decision making. The current experiment investigates whether the eye movements of a speaker and a listener to a visual common ground can provide insight into a discourse. While their eye movements were being recorded, participants spoke extemporaneously about a TV show whose cast members they were viewing. Later, other participants listened to these speeches while their eyes were tracked. Within this naturalistic paradigm using spontaneous speech, a number of results linking eye movements to speech comprehension, speech production and memory were replicated. More importantly, a cross- recurrence analysis demonstrated that speaker and listener eye movements were coupled, and that the strength of this relationship positively correlated with listeners’ comprehension. Just as the mental state of a single person can be reflected in patterns of eye movements, the commonality of mental states that is brought about by successful communication is mirrored in a similarity between speaker and listener’s eye movements. Eye movement Research Introduction Imagine standing in front of a painting, discussing it with a friend. As you talk, your eyes will scan across the image, moving approximately three times a second. They will be drawn by characteristics of the image itself, areas of contrast or detail, as well as features of the objects or people portrayed. Eye movements are driven both by properties of the visual world and processes in a person’s mind. Your gaze might also be influenced by what your friend is saying, what you say in reply, what is thought but not said, and where you agree and disagree. If this is so, what is the relationship between your eye movements and those of your friend? How is that relationship related to the flow of conversation between you? Language use often occurs within rich visual contexts such as this, and the interplay between linguistic processes and visual perception is of increasing interest to psycholinguists and vision researchers (Henderson & Ferreira, 2004). As yet, however, such processes have been limited to experiments that examine the eye movements of the speaker or the listener in isolation. Language use, more often than not, occurs within a richer social context as well. Direct eye contact between conversants plays an interesting, crucial role in coordinating a conversation (Bavelas, Coates, & Johnson, 2002), and in conveying various attitudes or social roles (Argyle & Cook, 1976). The focus of the current experiment, however, is cases such as those introduced at the outset, where conversants are not looking at each other, but at some visual scene that is the topic of the conversation. More common examples might be Eye movements of a speaker If a speaker is asked to describe a simple scene, they will fixate the objects in the order in which they are mentioned, around 900ms before naming them (Griffin & Bock, 2000; Meyer, Sleiderink, & Levelt, 1998). Since such pictures can be identified rapidly, it is argued that during this time speakers are not just retrieving words but selecting and planning which to use. Eye movements of a listener Eye movement research has shown that there is a tight interdependence between speech recognition and visual perception. Eye movements to potential referents for a word can provide evidence for a lexical item being recognized before the word is finished being spoken. The link between visual and linguistic processing can also be seen in eye movements that disambiguate syntactic structures (Tanenhaus, Spivey Knowlton, Eberhard, & Sedivy, 1995) and anticipate the future agents of actions (Kamide, Altmann, & Haywood, 2003). Recent studies of the eye- movements of a participant engaged in a conversation with another naive participant reveal a remarkable sensitivity to the referential domains established by the task, the visual context and the preceding conversation (Brown-Schmidt, Campana, & Tanenhaus, 2004). Qualitatively, eye movement research reveals a very close, time-locked integration between visual and linguistic processing (Tanenhaus, Magnuson, Dahan, & Chambers, 2000). Although fixation times are heavily modulated by context, as a very rough quantitative guide, research suggests that listeners will fixate an object around 400-800ms after the name onset. <s> BIB001 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Significance Test: Bootstrap <s> This study examined the influence of sex, social dominance, and context on motion-tracked head movements during dyadic conversations. Windowed cross-correlation analyses found high peak correlation between conversants' head movements over short (≈2-s) intervals and a high degree of nonstationarity. Nonstationarity in head movements was found to be positively related to the number of men in a conversation. Surrogate data analysis offsetting the conversants' time series by a large lag was unable to reject the null hypothesis that the observed high peak correlations were unrelated to short-term coordination between conversants. One way that high peak correlations could be observed when 2 time series are offset by a large time lag is for each time series to exhibit self-similarity over a range of scales. Multifractal analysis found small-scale fluctuations to be persistent, τ(q) 0.5. These results are consistent with a view that symmetry is formed between conversants over short intervals and that this symmetry is broken at longer, irregular intervals. <s> BIB002 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Significance Test: Bootstrap <s> Nonverbal synchrony in face-to-face interaction has been studied in numerous empirical investigations focusing on various communication channels. Furthermore, the pervasiveness of synchrony in physics, chemistry and biology adds to its face-validity. This paper is focused on establishing criteria for a statistical evaluation of synchrony in human interaction. When assessing synchrony in any communication context, it is necessary to distinguish genuine synchrony from pseudosynchrony, which may arise due to random coincidence. By using a bootstrap approach, we demonstrate a way to quantify the amount of synchrony that goes beyond random coincidence, thus establishing an objective measure for the phenomenon. Applying this technique to psychotherapy data, we develop a hypothesis-driven empirical evaluation of nonverbal synchrony. The method of surrogate testing in order to control for chance is suitable to any corpus of empirical data and lends itself to better empirically informed inference. <s> BIB003 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Significance Test: Bootstrap <s> The decisive role of interpersonal coordination in social interactions is well established. As human beings, we are "experts" in decoding and producing social signals, learning since our birth. Though, equipping a machine with the same abilities represents a great challenge to design socially-accepted interfaces. Assessing automatically this coordination requires selecting the most relevant features. In this paper, various audio and visual features are investigated to characterize interactional synchrony in a cooperative task. A hierarchical representation of the similarity between extracted features is proposed. The properties of time-domain (correlation coefficients) and frequency-domain (coherence) synchrony measures are confronted. The presented results support the evidence that synchrony is a multimodal phenomenon. Moreover, it can be assessed both on local (short-length windows of interaction) or global scales (entire interaction). <s> BIB004 </s> Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines <s> Significance Test: Bootstrap <s> Mimicry occurs in conversations both when people agree with each other and when they do not. However, it has been reported that there is more mimicry when people agree than when they disagree: when people want to express shared opinions and attitudes, they do so by displaying behavior that is similar to their interlocutors' behavior. In a conversation, mimicry occurs in order to gain acceptance from an interaction partner by conforming to that person's attitudes, opinions, and behavior. In this paper we describe how visual behavioral information expressed between two interlocutors can be used to detect and identify visual mimicry. We extract and encode visual features that are expected to represent mimicry in a useful way. In order to show that mimicry has indeed occurred, we calculate correlations between visual features extracted from the interactants and compare these against each other and against a baseline. We show that it is possible to visualize the occurrence of visual mimicry during the progress of a conversation which allows us to research in which situations and to what extent mimicry occurs. <s> BIB005
A critical question when attempting to detect dependence relationships between features is where the boundary between scores indicating significant and insignificant synchrony should be. Ashenfelter et al. summarize the issue BIB002 : "...it is reasonable to ask whether these coefficients are due to the coordination between people as they act in a mutual perception-action cycle or if these values might be due to the overall context of the conversation." Consequently, a baseline is needed to compare the scores and determine the significance of the measure. Bernieri and Rosenthal originally proposed a rating method ("the pseudosynchrony experimental paradigm") to evaluate the interactional synchrony that occurred in a dyadic interaction  . The method consists of synthesizing surrogate data (pseudointeractions): Video images of dyadic partners are isolated and recombined in a random order. Judges then rate the original and pseudointeractions videos. Pseudointeraction scores constitute a baseline to judge the scores obtained in the original interaction. The idea of generating surrogate data and comparing the synchrony scores on the genuine and surrogate data sets has been extended to automatic computation of interactional synchrony. First, features are extracted for each dyadic partner. The temporal structure of the first partner's time series is destroyed and reassociated with the second partner's original time series. Synchrony scores are assessed using the original and surrogate data sets. The synchrony scores on the surrogate data set constitute a baseline for judging for the dyad's coordination BIB001 , BIB002 , BIB005 . Ramseyer and Tschacher go beyond the comparison with a single surrogate data set BIB003 . They build N (N = 100) surrogate data sets and estimate the distribution of the surrogate synchrony scores. A statistical test is performed to test the hypothesis that the genuine synchrony score stems from the same distribution as the surrogate synchrony scores. Interactional synchrony scores are considered significant if genuine synchrony scores are above two standard deviations of the pseudosynchrony scores (one-sided z-test, p < 0:05). Various methods have been used to generate the surrogate data sets (offsetting one time series of a large time lag BIB002 , time-shuffling n-second-length windows of one time series BIB003 , BIB004 , BIB001 associating mismatched partners who did not interact with one another BIB001 . Ramseyer and Tschacher BIB003 provide for a methodological description of resampling methods for synchrony assessment. These methods are subject to three main criticisms in the context of studying naturalistic interaction data. First, the evaluation and interpretation of these methods are particularly delicate. On the basis of the discriminative or predictive power of the measure, the authors cannot really know if it is real synchrony that is measured or just a co-occurrence of events without meaning, particularly for features as global as Motion Energy. Moreover, the measures provided by these methods are mostly global and do not shed light on what happened locally during the interaction; they do not provide a local model of the communication dynamics. Second, the importance of speech and multimodality is often concealed in these methods. Third, these methods are suitable for analyzing a database but do not provide direct insights on how to equip a machine with such coordination skills. 