summary of multimodal refs summary to my query of early january : ) dear linguists , ) ) does anyone know of work - - studies , analyses - - in multimodal reference : ) i . e . how words and picture interact and refer to each other in texts ? ) thanks to all who responsed , among them : susan meredith burt , robert dale , karen emmorey , sharon flank , lisa frumkes , sabine geldof , james a . mcgilvray , georgia green , marti hearst , richard hirsch , sally jacoby , pirjo karvonen , john lee , dick oehrle , toshio ohori , laurent romary , deborah d k ruuskanen , roberta trites , mats wiren . - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - the background to my query is that my group has worked for some time with multimodal input to computer systems , using combinations of text and direct manipulation : point and click gestures . just recently we extended our scope somewhat , and we have just completed a first running version of a speech interface to a 3 - d graphic virtual environment . i myself am looking at reference resolution in the interface . we are currently in the process of experimenting and fooling about with it . i was surprised myself at the strong effects of visual focus and interactivity on referent choice in situations which could be construed as ambiguous , and conversely , how in certain constructions , the textual effects completely override visual and gestural cues . just recently we completed the first stage of the project . i will be happy to mail anyone a copy of our first tentative report on how things turned out . this led me on to a side track : i am currently working on a statistical study of a tintin album ( l ' oreille cassee ) to see how the pictorial mode of the text affects the structure of referential expressions . jussi karlgren , ivan bretan , niklas frost , lars jonsson . 1995 . ` ` interaction models for speech interfaces to virtual environments ' ' , proceedings of second eurographics workshop on virtual environments - - realism and real time , monte carlo . darmstadt : fraunhofer igd . integrating speech and virtual reality technology is being done simultaneously by several research groups , and there are several publications from various sources in the works . naturally , multimodality in the human computer interface can be more than speech , vision , and gestures , and a fair amount of work has been put into investigating effects of adding text input or output to graphical displays or vice versa . there are several studies made on multimodal aspects of human-human discourse , both spoken and in text form , some with an eye on application on human-computer interaction , and some not . several responses encouraged me to look more at studies of children 's literature : the standard reference to that appears to be nodelman , in the following list . the following list only contains the complete references i have been sent so far . they keep arriving , so i may post a revised version if i receive enough more material . naturally , the field is diverse and large , and this list does in no way even approximate the breadth and depth of study in the area , but much of the work in it is new to me . i hope it is of some use ! ( more ) computer oriented work - - system descriptions and empirical studies : edwin bos , carla huls , and wim claassen . 1994 . ` ` edward : full integration of language and action in a multimodal user interface ' ' { \ it international journal of human - computer studies } , { \ bf 40 } : 473-495 . r . chandrasekar and s . ramani . 1989 . ` ` interactive communication of sentential structure and content : an alternative approach to man-machine communication ' ' , { \ it international journal of man - machine studies } { \ bf 30 } : 121-148 . \ item philip r . cohen . 1992 . ` ` the role of natural language in a multimodal interface ' ' , proceedings of the acm symposium on user interface software and technology ( uist ) , monterey , pp . 143-150 . steven k . feiner and kathleen r . mckeown . automating the generation of coordinated multimedia explanations ieee computer 24 ( 10 ) 33-41 october 1991 . interactive spoken dialogue interface in virtual worlds christophe godereaux , korinna diebel , pierre - olivier el guedj , pierre nugues . proc . ling conc and methods in cscw , london , nov 94 . govindaraju , v . ; srihari , s . n . ; sher , d . b . caption - aided face location in newspaper photographs . proceedings of 11th iapr international conference on pattern recognition . vol . 1 . computer vision and applications . the hague , netherlands . 1992 . los alamitos , ca , usa : ieee comput . soc . press , p . 474 - 7 . govindaraju , v . ; srihari , s . n . ; sher , d . a computational model for face location based on cognitive principles . aaai-92 . proceedings tenth national conference on artificial intelligence , san jose , ca , usa . 1992 . menlo park , ca , usa : aaai press , p . 350 - 5 . \ item [ susann luperfoy ] . 1992 . { \ it the representation of multimodal user interface dialogues using discourse pegs } . { \ it proceedings of the 30th annual meeting of the association of computational linguistics } , newark . \ item johanna d . moore and william r . swartout . 1990 . ` ` pointing : a way toward explanation dialogue ' ' , { \ it proceedings of aaai } , boston . dagmar schmauck ' deixis in der mensch - maschine - interaktion multimediala referentenidentifikation durch naturliche und simulierte zeigegesten . ' tubingen : niemeyer . 1991 . wahlster , w . , andre , e . , finkler , w . , profitlich , h . - j . and rist , t . plan - based integration of natural language and graphics generation . artificial intelligence , 63 ( 1 - 2 ) pp 387-427 , october 1993 ( less ) computer based work , mainly human-human interaction studies , although some have an eye on application in human-computer interface design . \ item [ alphonse chapanis , robert b . ochsman , robert n . parrish , and gerald d . weeks ] , ` ` studies in interactive communication : i . the effects of four communication modes on the behavior of teams during cooperative problem solving ' ' , human factors 14 : 6 ( 1972 ) 487-509 \ item [ alphonse chapanis , robert n . parrish , robert b . ochsman , and gerald d . weeks ] , ` ` studies in interactive communication : ii . the effects of four communication modes on the behavior of teams during cooperative problem solving ' ' , human factors 19 : 2 ( 1977 ) 101-126 \ item [ philip r . cohen ] , ` ` the pragmatics of referring and the modality of communication ' ' , computational linguistics 10 : 2 ( 1984 ) 97-147 glenberg , a . & kruley , p . ( 1992 ) . pictures and anaphora : evidence for independent processes . _ memory and cognition _ , 20 ( 5 ) , 461-471 . glenberg , a . & mcdaniel , m . ( 1992 ) . mental models , pictures , and text : integration of spatial and verbal information . _ memory and cognition _ , 20 ( 5 ) , 458-460 . nelson goodman _ languages of art _ and _ routes of reference _ georgia m . green and margaret olsen . interactions of text and illustration in beginning reading . technical report 355 , center for the study of reading , university of illinois , champaign , illinois . hammel , eugene a . ( 1972 ) the myth of structural analysis : levi - strauss and the three bears . current topics in antrhropology , vol . 5 , module 26 , 1972 , paages 1-29 . neilson and lee , " conversations with graphics . . . " in int . journal of human - computer studies , 40 , 1994 . nodelman , perry ( 1988 ) . _ words about pictures _ . athens , georgia : university of georgia press . ochs , e . , jacoby , s . , & gonzales , p . ( 1994 ) interpretive journeys : how physicists talk and travel through graphic space . configurations , 2 ( 1 ) , 151-171 . ochs , e . , gonzales , p . , & jacoby , s . ( forthcoming ) . " when i come down i ' m in the domain state " : grammar and graphic representation in the interpretive activity of physicists . in e . a . schegloff , e . ochs , & s . thompson ( eds . ) , grammar and interaction , cambridge university press . again , thanks to all who responded ! j jussi karlgren , fil . lic . jussi . karlgren @ sics . se sw inst of comp sc ( sics ) spr } kteknologi / natural language processing box 1263 , 164 28 kista ph + 46 8 752 15 00 , fax + 46 8 751 72 30 stockholm , sweden http : / / sics . se / ~ jussi / jussi-karlgren . html
