We explore the problem of resolving the second person English pronoun you in multi-party dialogue , using a combination of linguistic and visual features . First , we distinguish generic and referential uses , then we classify the referential uses as either plural or singular , and finally , for the latter cases , we identify the addressee . In our first set of experiments , the linguistic and visual features are derived from manual transcriptions and annotations , but in the second set , they are generated through entirely automatic means . Results show that a multimodal system is often preferable to a unimodal one . The English pronoun you is the second most frequent word in unrestricted conversation ( after I and right before it ) . 1 Despite this , with the exception of Gupta et al . ( 2007b ; 2007a ) , its resolution has received very little attention in the literature . This is perhaps not surprising since the vast amount of work on anaphora and reference resolution has focused on text or discourse -mediums where second-person deixis is perhaps not as prominent as it is in dialogue . For spoken dialogue pronoun resolution modules however , resolving you is an essential task that has an important impact on the capabilities of dialogue summarization systems . Besides being important for computational implementations , resolving you is also an interesting and challenging research problem . As for third person pronouns such as it , some uses of you are not strictly referential . These include discourse marker uses such as you know in example ( 1 ) , and generic uses like ( 2 ) , where you does not refer to the addressee as it does in ( 3 ) . ( 1 ) It 's not just , you know , noises like something hitting . ( 2 ) Often , you need to know specific button sequences to get certain functionalities done . ( 3 ) I think it 's good . You 've done a good review . However , unlike it , you is ambiguous between singular and plural interpretations -an issue that is particularly problematic in multi-party conversations . While you clearly has a plural referent in ( 4 ) , in ( 3 ) the number of its referent is ambiguous . 2(4 ) I do n't know if you guys have any questions . When an utterance contains a singular referential you , resolving the you amounts to identifying the individual to whom the utterance is addressed . This is trivial in two-person dialogue since the current listener is always the addressee , but in conversations with multiple participants , it is a complex problem where different kinds of linguistic and visual information play important roles ( Jovanovic , 2007 ) . One of the issues we investigate here is how this applies to the more concrete problem of resolving the second person pronoun you . We approach this issue as a three-step problem . Using the AMI Meeting Corpus ( McCowan et al . , 2005 ) of multi-party dialogues , we first discriminate between referential and generic uses of you . Then , within the referential uses , we distinguish between singular and plural , and finally , we resolve the singular referential instances by identifying the intended addressee . We use multimodal features : initially , we extract discourse features from manual transcriptions and use visual information derived from manual annotations , but then we move to a fully automatic approach , using 1-best transcriptions produced by an automatic speech recognizer ( ASR ) and visual features automatically extracted from raw video . In the next section of this paper , we give a brief overview of related work . We describe our data in Section 3 , and explain how we extract visual and linguistic features in Sections 4 and 5 respectively . Section 6 then presents our experiments with manual transcriptions and annotations , while Section 7 , those with automatically extracted information . We end with conclusions in Section 8 . We have investigated the automatic resolution of the second person English pronoun you in multi-party dialogue , using a combination of linguistic and visual features . We conducted a first set of experiments where our features were derived from manual transcriptions and annotations , and then a second set where they were generated by entirely automatic means . To our knowledge , this is the first attempt at tackling this problem using automatically extracted multimodal information . Our experiments showed that visual information can be highly predictive in resolving the addressee of singular referential uses of you . Visual features significantly improved the performance of both our manual and automatic systems , and the latter achieved an encouraging 75 % accuracy . We also found that our visual features had predictive power for distinguishing between generic and referential uses of you , and between referential singulars and plurals . Indeed , for the latter task , they significantly improved the manual system 's performance . The listeners ' gaze features were useful here : in our data set it was apparently the case that the speaker would often use the whiteboard / projector screen when addressing the group , thus drawing the listeners ' gaze in this direction . Future work will involve expanding our dataset , and investigating new potentially predictive features . In the slightly longer term , we plan to integrate the resulting system into a meeting assistant whose purpose is to automatically extract useful information from multi-party meetings .
Although the word "you" is frequently used and has several possible meanings, such as reference or generic, it is not well studied yet.
They first manually automatically separate the word "you" between generic and referential, then later use a multimodal system for automation.
They show that visual features can help distinguish the word "you" in multi-party conversations.