Achieving homophily, or association based on similarity, between a human user and a robot holds a promise of improved perception and task performance. However, no previous studies that address homophily via ethnic similarity with robots exist. In this paper, we discuss the difficulties of evoking ethnic cues in a robot, as opposed to a virtual agent, and an approach to overcome those difficulties based on using ethnically salient behaviors. We outline our methodology for selecting and evaluating such behaviors, and culminate with a study that evaluates our hypotheses of the possibility of ethnic attribution of a robot character through verbal and nonverbal behaviors and of achieving the homophily effect.
The importance of context has motivated several efforts to collect and analyze corpora of context-specific interactions. Iacobelli and Cassell, for example, coded the verbal and nonverbal behaviors, such as gaze, of African American children during spontaneous play @cite_3 . CUBE-G project collected a cross-cultural multimodal corpus of dyadic interactions @cite_12 .
@cite_12 @cite_3
2114030807 2526567570 2157891642 1978650987
Three experiments were conducted to determine the psychometric properties of language in dyadic interactions. Using text-analysis, it was possible to assess the degree to which people coordinate their word use in natural conversations. In Experiments 1 (n = 130) and 2( n = 32), college students interacted in dyadic conversations in laboratory-based private Internet chat rooms. Experiment 3 analyzed the official transcripts of the Watergate tapes involving the dyadic interactions between President Richard Nixon and his aids H. R. Haldeman, John Erlichman, and John Dean. The results of the three studies offer sub- stantial evidence that individuals in dyadic interactions exhibit linguistic style matching (LSM) on both the conversation level as well as on a turn-by-turn level. Furthermore, LSM is unrelated to ratings of the quality of the interaction by both participants and judges. We propose that a coordination-engagement hypothesis is a better description of linguistic behaviors than the coordination-rapport hypothesis that has been proposed in the nonver- bal literature. For years social psychologists have exalted the power of the situa- tion. We comfortably acknowledge that across different situations, with different people, we may act in a range of ways, or even talk using a variety of styles. Aware of this tendency, Gergen (1972) began his explorations of our shifting masks of identity. In writing letters to close friends, he realized that he came across as a "completely different per- son" in each letter. "In one, I was morose, pouring out a philosophy of existential sorrow; in another I was a lusty realist: in a third I was a lighthearted jokester" (p. 32). Based merely on his word choices, Gergen inadvertently varied his style to adapt to the recipients of his letters. This is a prime demonstration of our inherent knowledge of the mutability of our language with respect to varying social contexts. During face-to-face interactions, people naturally integrate nonverbal behaviors such as facial expressions and body postures as part of the conversation to infer the communicative intent or emotional state of their interlocutor. The interpretation of these nonverbal behaviors will often be contextualized by interactional cues such as the previous spoken question, the general discussion topic or the physical environment. A critical step in creating computers able to understand or participate in this type of social face-to-face interactions is to develop a computational platform to synchronously recognize nonverbal behaviors as part of the interactional context. In this platform, information for the acoustic and visual modalities should be carefully synchronized and rapidly processed. At the same time, contextual and interactional cues should be remembered and integrated to better interpret nonverbal (and verbal) behaviors. In this article, we introduce a real-time computational framework, MultiSense, which offers flexible and efficient synchronization approaches for context-based nonverbal behavior analysis. MultiSense is designed to utilize interactional cues from both interlocutors (e.g., from the computer and the human participant) and integrate this contextual information when interpreting nonverbal behaviors. MultiSense can also assimilate behaviors over a full interaction and summarize the observed affective states of the user. We demonstrate the capabilities of the new framework with a concrete use case from the mental health domain where MultiSense is used as part of a decision support tool to assess indicators of psychological distress such as depression and post-traumatic stress disorder (PTSD). In this scenario, MultiSense not only infers psychological distress indicators from nonverbal behaviors but also broadcasts the user state in real-time to a virtual agent (i.e., a digital interviewer) designed to conduct semi-structured interviews with human participants. Our experiments show the added value of our multimodal synchronization approaches and also demonstrate the importance of MultiSense contextual interpretation when inferring distress indicators. Trying to adapt the behavior of an interactive system to the cultural background of the user requires information on how relevant behaviors differ as a function of the user's cultural background. To gain such insights in the interrelation of culture and behavior patterns, the information from the literature is often too anecdotal to serve as the basis for modeling a system's behavior, making it necessary to collect multimodal corpora in a standardized fashion in different cultures. In this chapter, the challenges of such an endeavor are introduced and solutions are presented by examples from a German-Japanese project that aims at modeling culture-specific behaviors for Embodied Conversational Agents. Based on the assumptions that relational messages are multidimensional and that they are frequently communicated by nonverbal cues, this experiment manipulated five nonverbal cues -eye contact, proximity, body lean, smiling, and touch - to determine what meanings they convey along four relational message dimensions. Subjects (N= 150) observed 2 out of 40 videotaped conversational segments in which a male-female dyad presented various combinations of the nonverbal cues. High eye contact, close proximity, forward body lean, and smiling all conveyed greater intimacy, attraction, and trust. Low eye contact, a distal position, backward body lean, and the absence of smiling and touch communicated greater detachment. High eye contact, close proximity, and smiling also communicated less emotional arousal and greater composure, while high eye contact and close proximity alone conveyed greater dominance and control. Effects of combinations of cues and sex-differences are also reported.