No
BOOLEAN
A number of the Winograd schemas already published can be “converted” quite easily to this form. Indeed, the above example was constructed in this way; the original form was “Jane knocked on Susan's door, but she did not [answer/get an answer].” Of the 144 schemas in the collection at http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.html there are 33 that can plausibly be translated this way. (#'s 3, 4, 12, 15, 17, 18, 23, 25, 26, 27, 42, 43, 44, 45, 46, 57, 58, 60, 69, 70, 71, 77, 78, 79, 86, 97, 98, 106, 111, 114, 127, 132, 144) — essentially any schema that involves two people as possible referents where the content of the sentences makes sense as applied to groups of people rather than individuals.
1608.01884
Win og rad ĠSc hem as </s> A ĠWin og rad Ġschema Ġ( Le ves que , ĠDavis , Ġand ĠMorg en st ern Ġ2012 ) Ġis Ġa Ġpair Ġof Ġsentences , Ġor Ġof Ġshort Ġtexts , Ġcalled Ġthe Ġelements Ġof Ġthe Ġschema , Ġthat Ġsatisfy Ġthe Ġfollowing Ġconstraints : </s> The Ġfollowing Ġis Ġan Ġexample Ġof Ġa ĠWin og rad Ġschema : </s> Here , Ġthe Ġtwo Ġsentences Ġdiffer Ġonly Ġin Ġthe Ġlast Ġword : Ġ` large ' Ġvs . Ġ` small '. ĠThe Ġambiguous Ġpronoun Ġis Ġ` it '. ĠThe Ġtwo Ġante ced ents Ġare Ġ` tro phy ' Ġand Ġ` brown Ġsuitcase '. ĠA Ġhuman Ġreader Ġwill Ġnaturally Ġinterpret Ġ` it ' Ġas Ġreferring Ġto Ġthe Ġtrophy Ġin Ġthe Ġfirst Ġsentence Ġand Ġto Ġthe Ġsuitcase Ġin Ġthe Ġsecond Ġsentence , Ġusing Ġthe Ġworld Ġknowledge Ġthat Ġa Ġsmall Ġobject Ġcan Ġfit Ġin Ġa Ġlarge Ġcontainer , Ġbut Ġa Ġlarge Ġobject Ġcannot Ġfit Ġin Ġa Ġsmall Ġcontainer Ġ( Davis Ġ2013 ). ĠCondition Ġ4 Ġis Ġsatisfied Ġbecause Ġeither Ġa Ġtrophy Ġor Ġa Ġsuitcase Ġcan Ġbe Ġeither Ġlarge Ġor Ġsmall , Ġand Ġthere Ġis Ġno Ġreason Ġto Ġsuppose Ġthat Ġthere Ġwould Ġbe Ġa Ġmeaningful Ġcorrelation Ġof Ġ` small ' Ġor Ġ` large ' Ġwith Ġ` tro phy ' Ġor Ġ` suit case ' Ġin Ġa Ġtypical Ġcorpus . </s> We Ġwill Ġsay Ġthat Ġan Ġelement Ġof Ġa ĠWin og rad Ġschema Ġis ĠâĢ ľ s olved âĢ Ŀ Ġif Ġthe Ġrefere nt Ġof Ġthe Ġpronoun Ġis Ġidentified . </s> An Ġexample Ġof Ġa Ġpair Ġof Ġsentences Ġsatisfying Ġconditions Ġ1 - 3 Ġbut Ġnot Ġ4 Ġwould Ġbe </s> Since Ġwomen Ġcannot Ġbe Ġcarcin ogenic Ġand Ġpills Ġcannot Ġbe Ġpregnant , Ġthe Ġpronoun Ġ` they ' Ġin Ġthese Ġsentences Ġis Ġeasily Ġdis amb ig uated Ġusing Ġselection al Ġrestrictions . ĠThis Ġpair Ġis Ġtherefore Ġnot Ġa Ġvalid ĠWin og rad Ġschema . </s> The ĠWin og rad ĠSche ma ĠChallenge Ġ( W SC ) Ġis Ġa Ġchallenge Ġfor ĠAI Ġprograms . ĠThe Ġprogram Ġis Ġpresented Ġwith Ġa Ġcollection Ġof Ġsentences , Ġeach Ġof Ġwhich Ġis Ġone Ġelement Ġof Ġa ĠWin og rad Ġschema , Ġand Ġis Ġrequired Ġto Ġfind Ġthe Ġcorrect Ġrefere nt Ġfor Ġthe Ġambiguous Ġpronoun . ĠAn ĠAI Ġprogram Ġpasses Ġthe Ġchallenge Ġif Ġits Ġsuccess Ġrate Ġis Ġcomparable Ġto Ġa Ġhuman Ġreader . ĠThe Ġchallenge Ġis Ġadministered Ġby Ġcommons ense reason ing . org Ġand Ġsponsored Ġby ĠNu ance ĠInc . ĠIt Ġwas Ġoffered Ġfor Ġthe Ġfirst Ġtime Ġat ĠI J CA I - 2016 Ġ( M org en st ern , ĠDavis , Ġand ĠOrtiz , Ġin Ġpreparation ); Ġthe Ġorganizers Ġplan Ġto Ġcontinue Ġto Ġoffer Ġit Ġroughly Ġonce Ġa Ġyear . </s> Win og rad Ġsche mas Ġas Ġthe Ġbasis Ġfor Ġchallenges Ġfor Ġmachine Ġtranslation Ġprograms </s> In Ġmany Ġcases , Ġthe Ġidentification Ġof Ġthe Ġrefere nt Ġof Ġthe Ġp rou n oun Ġin Ġa ĠWin og rad Ġschema Ġis Ġcritical Ġfor Ġfinding Ġthe Ġcorrect Ġtranslation Ġof Ġthat Ġpronoun Ġin Ġa Ġdifferent Ġlanguage . ĠTherefore , ĠWin og rad Ġsche mas Ġcan Ġbe Ġused Ġas Ġa Ġvery Ġdifficult Ġchallenge Ġfor Ġthe Ġdepth Ġof Ġunderstanding Ġachieved Ġby Ġa Ġmachine Ġtranslation Ġprogram . </s> The Ġthird Ġperson Ġplural Ġpronoun Ġ` they ' Ġhas Ġno Ġgender Ġin ĠEnglish Ġand Ġmost Ġother Ġlanguages Ġ( un like Ġthe Ġthird Ġperson Ġsingular Ġpronouns Ġ` he ', Ġ` she ', Ġand Ġ` it '). ĠHowever ĠRomance Ġlanguages Ġsuch Ġas ĠFrench , ĠItalian , Ġand ĠSpanish , Ġand ĠSem itic Ġlanguages Ġsuch Ġas ĠHebrew Ġand ĠArabic Ġdistinguish Ġbetween Ġthe Ġmasculine Ġand Ġthe Ġfeminine Ġthird - person Ġplural Ġpronouns , Ġat Ġleast Ġin Ġsome Ġgram matical Ġcases . ĠFor Ġinstance Ġin ĠFrench , Ġthe Ġmasculine Ġpronoun Ġis Ġ` ils '; Ġthe Ġfeminine Ġpronoun Ġis Ġ` ell es '. ĠIn Ġall Ġof Ġthese Ġcases , Ġthe Ġmasculine Ġpronoun Ġis Ġstandard ly Ġused Ġfor Ġgroups Ġof Ġmixed Ġor Ġunknown Ġgender . </s> In Ġorder Ġto Ġcorrectly Ġtranslate Ġa Ġsentence Ġin ĠEnglish Ġcontaining Ġthe Ġword Ġ` they ' Ġinto Ġone Ġof Ġthese Ġlanguages , Ġit Ġis Ġnecessary Ġto Ġdetermine Ġwhether Ġor Ġnot Ġthe Ġrefere nt Ġis Ġa Ġgroup Ġof Ġfemales . ĠIf Ġit Ġis , Ġthen Ġthe Ġtranslation Ġmust Ġbe Ġthe Ġfeminine Ġpronoun ; Ġotherwise , Ġit Ġmust Ġbe Ġthe Ġmasculine Ġpronoun . ĠTherefore , Ġif Ġone Ġcan Ġcreate Ġa ĠWin og rad Ġschema Ġin ĠEnglish Ġwhere Ġthe Ġambiguous Ġpronoun Ġis Ġ` they ' Ġand Ġthe Ġcorrect Ġrefere nt Ġfor Ġone Ġelement Ġis Ġa Ġcollection Ġof Ġmen Ġand Ġfor Ġthe Ġother Ġis Ġa Ġcollection Ġof Ġwomen , Ġthen Ġto Ġtranslate Ġboth Ġelements Ġcorrectly Ġrequires Ġsolving Ġthe ĠWin og rad Ġschema . </s> As Ġan Ġexample , Ġconsider Ġthe ĠWin og rad Ġschema : </s> If Ġthese Ġsentences Ġare Ġtranslated Ġinto ĠFrench , Ġthen Ġ` they ' Ġin Ġthe Ġfirst Ġsentence Ġshould Ġbe Ġtranslated Ġ` ell es ', Ġas Ġreferring Ġto ĠJane Ġand ĠSusan , Ġand Ġ` they ' Ġin Ġthe Ġsecond Ġsentence Ġshould Ġbe Ġtranslated Ġ` ils ', Ġas Ġreferring Ġto ĠFred Ġand ĠGeorge . </s> A Ġnumber Ġof Ġthe ĠWin og rad Ġsche mas Ġalready Ġpublished Ġcan Ġbe ĠâĢ ľ con verted âĢ Ŀ Ġquite Ġeasily Ġto Ġthis Ġform . ĠIndeed , Ġthe Ġabove Ġexample Ġwas Ġconstructed Ġin Ġthis Ġway ; Ġthe Ġoriginal Ġform Ġwas ĠâĢ ľ Jane Ġknocked Ġon ĠSusan 's Ġdoor , Ġbut Ġshe Ġdid Ġnot Ġ[ answer / get Ġan Ġanswer ]. âĢ Ŀ ĠOf Ġthe Ġ144 Ġsche mas Ġin Ġthe Ġcollection Ġat Ġhttp :// www . cs . ny u . edu / fac ulty / d av ise / papers / Win og rad Sche mas / WS Collection . html Ġthere Ġare Ġ33 Ġthat Ġcan Ġplaus ibly Ġbe Ġtranslated Ġthis Ġway . Ġ(# ' s Ġ3 , Ġ4 , Ġ12 , Ġ15 , Ġ17 , Ġ18 , Ġ23 , Ġ25 , Ġ26 , Ġ27 , Ġ42 , Ġ43 , Ġ44 , Ġ45 , Ġ46 , Ġ57 , Ġ58 , Ġ60 , Ġ69 , Ġ70 , Ġ71 , Ġ77 , Ġ78 , Ġ79 , Ġ86 , Ġ97 , Ġ98 , Ġ106 , Ġ111 , Ġ114 , Ġ127 , Ġ132 , Ġ144 ) ĠâĢĶ Ġessentially Ġany Ġschema Ġthat Ġinvolves Ġtwo Ġpeople Ġas Ġpossible Ġrefere nt s Ġwhere Ġthe Ġcontent Ġof Ġthe Ġsentences Ġmakes Ġsense Ġas Ġapplied Ġto Ġgroups Ġof Ġpeople Ġrather Ġthan Ġindividuals . </s> A Ġsimilar Ġdevice , Ġin Ġthe Ġopposite Ġdirection , Ġrelies Ġon Ġthe Ġfact Ġthat ĠFrench Ġdoes Ġnot Ġdistinguish Ġbetween Ġthe Ġpossess ive Ġpronouns Ġ` his ' Ġand Ġ` her '. ĠThe Ġpronouns Ġ` son ' Ġand Ġ` sa ' Ġare Ġg endered , Ġbut Ġthe Ġgender Ġagrees Ġwith Ġthe Ġpossession , Ġnot Ġthe Ġposs essor . ĠTherefore Ġa ĠWin og rad Ġschema Ġthat Ġrelies Ġon Ġfinding Ġa Ġrefere nt Ġfor Ġa Ġpossess ive Ġpronoun Ġcan Ġbe Ġturned Ġinto Ġa Ġhard Ġpair Ġof ĠFrench - to - English Ġtranslation Ġproblems Ġby Ġmaking Ġone Ġpossible Ġrefere nt Ġmale Ġand Ġthe Ġother Ġone Ġfemale . ĠFor Ġexample , Ġschema Ġ# 124 Ġin Ġthe Ġonline Ġcollection Ġreads ĠâĢ ľ The Ġman Ġlifted Ġthe Ġboy Ġonto Ġhis Ġ[ b unk Ġbed / should ers ]. âĢ Ŀ ĠChanging Ġthe Ġboy Ġto Ġa Ġgirl Ġand Ġtranslating Ġinto ĠFrench Ġgives ĠâĢ ľ L ' hom me Ġle va Ġla Ġfil le Ġsur Ġ[ ses ĠÃ© p aul es / son Ġlit Ġsuper pos Ã© ] âĢ Ŀ . ĠIn Ġthe Ġfirst Ġsentence ĠâĢ ľ ses âĢ Ŀ Ġis Ġtranslated ĠâĢ ľ his âĢ Ŀ , Ġin Ġthe Ġsecond ĠâĢ ľ son âĢ Ŀ Ġis Ġtranslated ĠâĢ ľ her âĢ Ŀ . ĠThe Ġeleven ĠWin og rad Ġsche mas Ġ# ' s Ġ112 , Ġ117 , Ġ118 , Ġ119 , Ġ124 , Ġ125 , Ġ127 , Ġ129 , Ġ130 , Ġ131 , Ġand Ġ143 Ġcan Ġbe Ġconverted Ġthis Ġway . </s> Care Ġmust Ġbe Ġtaken Ġto Ġavoid Ġrelying Ġon , Ġor Ġseeming Ġto Ġrely Ġon , Ġobjectionable Ġstereotypes Ġabout Ġmen Ġand Ġwomen . ĠOne Ġmechanism Ġthat Ġcan Ġsometimes Ġbe Ġused Ġis Ġto Ġinclude Ġa Ġpotentially Ġproblematic Ġsentence Ġin Ġboth Ġdirections . ĠFor Ġinstance , Ġschema Ġ23 Ġfrom Ġthe ĠW SC Ġcollection Ġcan Ġbe Ġtranslated Ġinto Ġboth ĠâĢ ľ The Ġgirls Ġwere Ġbullying Ġthe Ġboys Ġso Ġwe Ġ[ pun ished / res c ued ] Ġthem âĢ Ŀ Ġand ĠâĢ ľ The Ġboys Ġwere Ġbullying Ġthe Ġgirls , Ġso Ġwe Ġ[ pun ished / res c ued ] Ġthem , âĢ Ŀ Ġthus Ġavoiding Ġany Ġpresupp osition Ġof Ġwhether Ġgirls Ġare Ġmore Ġlikely Ġto Ġbully Ġboys Ġor Ġvice Ġversa . </s> A Ġhistorical Ġnote : ĠWin og rad Ġsche mas Ġwere Ġnamed Ġafter ĠTerry ĠWin og rad Ġbecause Ġof Ġa Ġwell - known Ġexample Ġin Ġhis Ġdoctoral Ġthesis Ġ( Win og rad Ġ1970 ). ĠIn Ġhis Ġthesis , Ġthis Ġexample Ġfollowed Ġthe Ġabove Ġform , Ġand Ġthe Ġimportance Ġof Ġthe Ġexample Ġwas Ġjustified Ġin Ġterms Ġof Ġmachine Ġtranslation . ĠWin og rad 's Ġoriginal Ġschema Ġwas : ĠâĢ ľ The Ġcity Ġcouncil men Ġrefused Ġto Ġgive Ġthe Ġwomen Ġa Ġpermit Ġfor Ġa Ġdemonstration Ġbecause Ġthey Ġ[ f eared / adv ocated ] Ġviolence ", Ġand ĠWin og rad Ġexplained Ġthat , Ġin Ġthe Ġsentence Ġwith Ġ` f eared ', Ġ` they ' Ġwould Ġrefer Ġto Ġthe Ġcouncil men Ġand Ġwould Ġbe Ġtranslated Ġas Ġ` ils ' Ġin ĠFrench , Ġwhereas Ġin Ġthe Ġsentence Ġwith Ġ` adv ocated ', Ġ` they ' Ġwould Ġrefer Ġto Ġthe Ġwomen Ġand Ġwould Ġbe Ġtranslated Ġas Ġ` ell es '. ĠIn Ġthe Ġlater Ġversions Ġof Ġthe Ġthesis , Ġpublished Ġas Ġ( Win og rad Ġ1972 ), Ġhe Ġchanged Ġ` women ' Ġto Ġ` demon str ators '; Ġthis Ġmade Ġthe Ġdis amb ig uation Ġclearer , Ġbut Ġlost Ġthe Ġpoint Ġabout Ġtranslation . </s> Current Ġstate Ġof Ġthe Ġart </s> No Ġone Ġfamiliar Ġwith Ġthe Ġstate Ġof Ġthe Ġart Ġin Ġmachine Ġtranslation Ġtechnology Ġor Ġthe Ġstate Ġof Ġthe Ġart Ġof Ġartificial Ġintelligence Ġgenerally Ġwill Ġbe Ġsurprised Ġto Ġlearn Ġthat Ġcurrently Ġmachine Ġtranslation Ġprogram Ġare Ġunable Ġto Ġsolve Ġthese ĠWin og rad Ġschema Ġchallenge Ġproblems . </s> What Ġmay Ġbe Ġmore Ġsurprising Ġis Ġthat Ġcurrently Ġ( July Ġ2016 ), Ġmachine Ġtranslation Ġprograms Ġare Ġunable Ġto Ġchoose Ġthe Ġfeminine Ġplural Ġpronoun Ġeven Ġwhen Ġthe Ġonly Ġpossible Ġante ced ent Ġis Ġa Ġgroup Ġof Ġwomen . ĠFor Ġinstance , Ġthe Ġsentence ĠâĢ ľ The Ġgirls Ġsang Ġa Ġsong Ġand Ġthey Ġdanced âĢ Ŀ Ġis Ġtranslated Ġinto ĠFrench Ġas ĠâĢ ľ Les Ġfill es Ġont Ġchant Ã© Ġune Ġch anson Ġet Ġil s Ġont Ġd ans Ã© âĢ Ŀ Ġby ĠGoogle ĠTrans late Ġ( GT ), ĠBing ĠTrans late , Ġand ĠY and ex . ĠIn Ġfact , ĠI Ġhave Ġbeen Ġunable Ġto Ġconstruct Ġany Ġsentence Ġin ĠEnglish Ġthat Ġis Ġtranslated Ġinto Ġany Ġlanguage Ġusing Ġthe Ġfeminine Ġplural Ġpron un . ĠNote Ġthat , Ġsince Ġthe Ġmasculine Ġplural Ġpronoun Ġis Ġused Ġfor Ġgroups Ġof Ġmixed Ġgender Ġin Ġall Ġthese Ġlanguages , Ġit Ġis Ġalmost Ġcertainly Ġmore Ġcommon Ġin Ġtext Ġthan Ġthe Ġfeminine Ġplural ; Ġhence Ġthis Ġstrategy Ġis Ġreasonable Ġfa ute Ġde Ġm ie ux . Ġ( It Ġalso Ġseems Ġlikely Ġthat Ġthe Ġerroneous Ġuse Ġof Ġa Ġfeminine Ġplural Ġfor Ġa Ġmasculine Ġante ced ent Ġsounds Ġeven Ġmore Ġjarring Ġthan Ġthe Ġreverse .) </s> The Ġsame Ġthing Ġsometimes Ġoccurs Ġin Ġtranslating Ġthe Ġfeminine Ġplural Ġpronoun Ġbetween Ġlanguages Ġthat Ġhave Ġit . ĠGT Ġtranslates Ġthe ĠFrench Ġword Ġ` ell es ' Ġinto ĠSpanish Ġas Ġ` ell os ' Ġ( the Ġmasculine Ġform ). ĠCur iously , Ġin Ġthe Ġopposite Ġdirection , Ġit Ġgets Ġthe Ġright Ġanswer ; Ġthe ĠSpanish Ġ` ell as ' Ġ( f em .) Ġis Ġtranslated Ġinto ĠFrench Ġas Ġ` ell es '. </s> Language - specific Ġissues </s> The Ġmasculine Ġand Ġfeminine Ġplural Ġpronouns Ġare Ġdistinguished Ġin Ġthe ĠRomance Ġlanguages Ġ( French , ĠSpanish , ĠItalian , ĠPortuguese Ġetc .) Ġand Ġin ĠSem itic Ġlanguages Ġ( Arab ic , ĠHebrew , Ġetc .) ĠI Ġhave Ġconsulted Ġwith Ġnative Ġspeakers Ġand Ġexperts Ġin Ġthese Ġlanguages Ġabout Ġthe Ġdegree Ġto Ġwhich Ġthe Ġgender Ġdistinction Ġis Ġobserved Ġin Ġpractice . ĠThe Ġexperts Ġsay Ġthat Ġin ĠFrench , ĠSpanish , ĠItalian , Ġand ĠPortuguese , Ġthe Ġdistinction Ġis Ġvery Ġstrictly Ġobserved ; Ġthe Ġuse Ġof Ġa Ġmasculine Ġpronoun Ġfor Ġa Ġfeminine Ġante ced ent Ġis Ġjarring ly Ġwrong Ġto Ġa Ġnative Ġor Ġfluent Ġspeaker . ĠâĢ ľ Les Ġfill es Ġont Ġchant Ã© Ġune Ġch anson Ġet Ġil s Ġont Ġd ans Ã© âĢ Ŀ Ġsounds Ġas Ġwrong Ġto Ġa ĠFrench Ġspeaker Ġas ĠâĢ ľ The Ġgirl Ġsang Ġa Ġsong Ġand Ġhe Ġdanced âĢ Ŀ Ġsounds Ġto Ġan ĠEnglish Ġspeaker ; Ġin Ġboth Ġcases , Ġthe Ġhe arer Ġwill Ġinterpret Ġthe Ġpronoun Ġas Ġrefer rin ig Ġto Ġsome Ġother Ġpersons Ġor Ġperson , Ġwho Ġis Ġmale . ĠIn ĠHebrew Ġand ĠArabic , Ġthis Ġis Ġmuch Ġless Ġtrue ; Ġin Ġspeech , Ġand Ġeven , Ġincreasingly , Ġin Ġwriting , Ġthe Ġmasculine Ġpronoun Ġis Ġoften Ġused Ġfor Ġa Ġfeminine Ġante ced ent . </s> Looking Ġfurther Ġahead , Ġit Ġis Ġcertainly Ġpossible Ġthat Ġgender Ġdistinctions Ġwill Ġbe Ġabandoned Ġin Ġthe ĠRomance Ġlanguages , Ġor Ġeven Ġthat ĠEnglish Ġwill Ġhave Ġdriven Ġall Ġother Ġlanguages Ġout Ġof Ġexistence , Ġsooner Ġthan ĠAI Ġsystems Ġwill Ġbe Ġable Ġto Ġdo Ġpronoun Ġresolution Ġin ĠWin og rad Ġsche mas ; Ġat Ġthat Ġpoint , Ġthis Ġtest Ġwill Ġno Ġlonger Ġbe Ġuseful . </s> In Ġsome Ġcases , Ġa Ġtranslation Ġprogram Ġcan Ġside - step Ġthe Ġissue Ġby Ġo mitting Ġthe Ġpronoun Ġaltogether . ĠFor Ġexample , ĠGT Ġtranslates Ġthe Ġabove Ġsentence ĠâĢ ľ The Ġgirls Ġsang Ġa Ġsong Ġand Ġthey Ġdanced âĢ Ŀ Ġinto ĠSpanish Ġas ĠâĢ ľ Las Ġch icas Ġcant aron Ġun a Ġcan ci Ã³n Ġy Ġbail aban âĢ Ŀ Ġand Ġinto ĠItalian Ġas ĠâĢ ľ Le Ġrag az ze Ġh ann o Ġcant ato Ġun a Ġcan zone Ġe Ġball av ano . âĢ Ŀ ĠHowever , Ġwith Ġthe Ġmore Ġcomplex Ġsentences Ġof Ġthe ĠWin og rad ĠSc hem as , Ġthis Ġstrategy Ġwill Ġrarely Ġgive Ġa Ġplausible Ġtranslation Ġfor Ġboth Ġelements Ġof Ġthe Ġschema . </s> Other Ġlanguages , Ġother Ġambig u ities </s> Broad ly Ġspeaking , Ġwhenever Ġa Ġtarget Ġlanguage ĠIN LINE FORM 0 Ġrequires Ġsome Ġdistinction Ġthat Ġis Ġoptional Ġor Ġnon - existent Ġin Ġsource Ġlanguage ĠIN LINE FORM 1 Ġ, Ġit Ġis Ġpossible Ġto Ġcreate Ġa Ġsentence ĠIN LINE FORM 2 Ġin ĠIN LINE FORM 3 Ġwhere Ġthe Ġmissing Ġinformation Ġis Ġnot Ġexplicit Ġbut Ġcan Ġbe Ġinferred Ġfrom Ġbackground Ġknowledge . ĠTrans l ating ĠIN LINE FORM 4 Ġfrom ĠIN LINE FORM 5 Ġto ĠIN LINE FORM 6 Ġthus Ġrequires Ġusing Ġthe Ġbackground Ġknowledge Ġto Ġresolve Ġthe Ġambiguity , Ġand Ġwill Ġtherefore Ġbe Ġchallenging Ġfor Ġautomatic Ġmachine Ġtranslation . </s> A Ġcouple Ġof Ġexamples : ĠThe Ġword Ġ` sie ' Ġin ĠGerman Ġserves Ġas Ġboth Ġthe Ġformal Ġsecond Ġperson Ġp rou n oun Ġ( always Ġcapital ized ), Ġthe Ġthird Ġperson Ġfeminine Ġsingular , Ġand Ġthe Ġthird Ġperson Ġplural . ĠTherefore , Ġit Ġcan Ġbe Ġtranslated Ġinto ĠEnglish Ġas Ġeither ĠâĢ ľ you âĢ Ŀ , ĠâĢ ľ she âĢ Ŀ , ĠâĢ ľ it âĢ Ŀ , Ġor ĠâĢ ľ they âĢ Ŀ ; Ġand Ġinto ĠFrench Ġas Ġeither Ġ` vous ', Ġ` il ', Ġ` elle ', Ġ` ils ', Ġor Ġ` ell es '. Ġ( The Ġfeminine Ġthird - person Ġsingular ĠGerman Ġ` sie ' Ġcan Ġbe Ġtranslated Ġas Ġne uter Ġin ĠEnglish Ġand Ġas Ġmasculine Ġin ĠFrench Ġbecause Ġthe Ġthree Ġlanguages Ġdo Ġnot Ġslice Ġup Ġthe Ġworlds Ġinto Ġgenders Ġin Ġthe Ġsame Ġway .) ĠLikewise , Ġthe Ġpossess ive Ġpronoun Ġ` ih r ' Ġin Ġall Ġits Ġdecl ensions Ġcan Ġmean Ġeither Ġ` her ' Ġor Ġ` their '. ĠIn Ġsome Ġcases , Ġthe Ġdis amb ig uation Ġcan Ġbe Ġcarried Ġout Ġon Ġpurely Ġsynt actic Ġground ; Ġe . g . Ġif Ġ` sie ' Ġis Ġthe Ġsubject Ġof Ġa Ġthird - person Ġsingular Ġverb , Ġit Ġmust Ġmean Ġ` she '. ĠHowever , Ġin Ġmany Ġcase , Ġthe Ġdis amb ig uation Ġrequires Ġa Ġdeeper Ġlevel Ġof Ġunderstanding . ĠThus , Ġit Ġshould Ġbe Ġpossible Ġto Ġconstruct ĠGerman ĠWin og rad Ġsche mas Ġbased Ġon Ġthe Ġwords Ġ` sie ' Ġor Ġ` ih r ' Ġthat Ġhave Ġto Ġbe Ġsolved Ġin Ġorder Ġto Ġtranslate Ġthem Ġinto ĠEnglish . ĠFor Ġexample , </s> Marie Ġf and Ġf Ã¼ n f Ġver lass ene ĠK Ã¤ tz chen Ġim ĠKeller . ĠI h re ĠM utter Ġwar Ġgest or ben . </s> ( Marie Ġfound Ġfive Ġabandoned Ġkittens Ġin Ġthe Ġcellar . ĠTheir Ġmother Ġwas Ġdead .) </s> vs . </s> Marie Ġf and Ġf Ã¼ n f Ġver lass ene ĠK Ã¤ tz chen Ġim ĠKeller . ĠI h re ĠM utter Ġwar Ġun z uf ried en . </s> ( Marie Ġfound Ġfive Ġabandoned Ġkittens Ġin Ġthe Ġcellar . ĠHer Ġmother Ġwas Ġdisple ased .) </s> Another Ġexample : ĠFrench Ġdistinguishes Ġbetween Ġa Ġmale Ġfriend Ġ` ami ' Ġand Ġa Ġfemale Ġfriend Ġ` am ie '. ĠTherefore , Ġin Ġtranslating Ġthe Ġword ĠâĢ ľ friend âĢ Ŀ Ġinto ĠFrench , Ġit Ġis Ġnecessary , Ġif Ġpossible , Ġto Ġdetermine Ġthe Ġsex Ġof Ġthe Ġfriend ; Ġand Ġthe Ġclue Ġfor Ġthat Ġcan Ġinvolve Ġan Ġinference Ġthat , Ġas Ġfar Ġas ĠAI Ġprograms Ġare Ġconcerned , Ġis Ġquite Ġremote . ĠHuman Ġreaders Ġare Ġawfully Ġgood Ġat Ġpicking Ġup Ġon Ġthose , Ġof Ġcourse . ĠIn Ġgeneral , Ġwith Ġthis Ġkind Ġof Ġthing , Ġif Ġyou Ġwant Ġto Ġbreak ĠGT Ġor Ġanother Ġtranslation Ġprogram , Ġthe Ġtrick Ġis Ġto Ġplace Ġa Ġlarge Ġseparation Ġbetween Ġthe Ġevidence Ġand Ġthe Ġword Ġbeing Ġdis amb ig uated . ĠIn Ġthis Ġcase , Ġat Ġthe Ġpresent Ġtime , Ġthe Ġseparation Ġcan Ġbe Ġpretty Ġsmall . ĠGT Ġcorrectly Ġtranslates ĠâĢ ľ my Ġfriend ĠPierre âĢ Ŀ Ġand ĠâĢ ľ my Ġfriend ĠMarie âĢ Ŀ Ġas ĠâĢ ľ mon Ġam i ĠPierre âĢ Ŀ Ġand ĠâĢ ľ mon Ġam ie ĠMarie âĢ Ŀ Ġand Ġit Ġtranslates ĠâĢ ľ She Ġis Ġmy Ġfriend âĢ Ŀ Ġas ĠâĢ ľ El le Ġest Ġmon Ġam ie âĢ Ŀ , Ġbut Ġrather Ġsurpr ingly Ġit Ġbreaks Ġdown Ġat ĠâĢ ľ Marie Ġis Ġmy Ġfriend , âĢ Ŀ Ġwhich Ġit Ġtranslates ĠâĢ ľ Marie Ġest Ġmon Ġam i . âĢ Ŀ ĠAs Ġfor Ġsomething Ġlike ĠâĢ ľ Jac ques Ġsaid Ġto ĠMarie , Ġ` You Ġhave Ġalways Ġbeen Ġa Ġtrue Ġfriend ,' ĠâĢ Ŀ Ġthat Ġis Ġquite Ġhopeless . ĠGT Ġcan Ġsurprise Ġone , Ġthough , Ġin Ġboth Ġdirections ; Ġsometimes Ġit Ġmisses Ġa Ġvery Ġclose Ġclue , Ġas Ġin ĠâĢ ľ Marie Ġis Ġmy Ġfriend âĢ Ŀ , Ġbut Ġother Ġtimes Ġit Ġcan Ġcarry Ġa Ġclue Ġfurther Ġthan Ġone Ġwould Ġhave Ġguessed . </s> Acknowled gements </s> Thanks Ġto ĠAri anna ĠBis az za , ĠGer hard ĠBrew ka , ĠAnt oine ĠCer f on , ĠJoseph ĠDavis , ĠGig i ĠD op ico - Black , ĠN iz ar ĠHab ash , ĠLe ora ĠMorg en st ern , ĠO ded ĠRe ge v , ĠFrances ca ĠRossi , ĠVes na ĠSab l j ak ovic - F ritz , Ġand ĠManuel a ĠVel oso Ġfor Ġhelp Ġwith Ġthe Ġvarious Ġlanguages Ġand Ġdiscussion Ġof Ġg endered Ġpronouns . </s> References </s> E . ĠDavis , ĠâĢ ľ Qual itative ĠSp atial ĠReason ing Ġin ĠInter pre ting ĠText Ġand ĠNarr ative âĢ Ŀ ĠSp atial ĠCogn ition Ġand ĠComput ation , Ġ13 : 4 , Ġ2013 , Ġ264 - 294 . </s> H . ĠLe ves que , ĠE . ĠDavis , Ġand ĠL . ĠMorg en st ern , ĠâĢ ľ The ĠWin og rad ĠSche ma ĠChallenge , âĢ Ŀ ĠKR Ġ2012 . </s> L . ĠMorg en st ern , ĠE . ĠDavis , Ġand ĠC . ĠOrtiz , ĠâĢ ľ Report Ġon Ġthe ĠWin og rad ĠSche ma ĠChallenge , Ġ2016 âĢ Ŀ , Ġin Ġpreparation . </s> T . ĠWin og rad , ĠâĢ ľ Pro ced ures Ġas Ġa ĠRepresent ation Ġfor ĠData Ġin Ġa ĠComputer ĠProgram Ġfor ĠUnderstanding ĠNatural ĠLanguage ," ĠPh . D . Ġthesis , ĠDepartment Ġof ĠMathematics , ĠMIT , ĠAugust Ġ1970 . ĠPublished Ġas ĠMIT ĠA IT R - 235 , ĠJanuary Ġ1971 . </s> T . ĠWin og rad , ĠUnderstanding ĠNatural ĠLanguage , ĠAcademic ĠPress , Ġ1972 .
Winograd Schemas A Winograd schema (Levesque, Davis, and Morgenstern 2012) is a pair of sentences, or of short texts, called the elements of the schema, that satisfy the following constraints: The following is an example of a Winograd schema: Here, the two sentences differ only in the last word: `large' vs. `small'. The ambiguous pronoun is `it'. The two antecedents are `trophy' and `brown suitcase'. A human reader will naturally interpret `it' as referring to the trophy in the first sentence and to the suitcase in the second sentence, using the world knowledge that a small object can fit in a large container, but a large object cannot fit in a small container (Davis 2013). Condition 4 is satisfied because either a trophy or a suitcase can be either large or small, and there is no reason to suppose that there would be a meaningful correlation of `small' or `large' with `trophy' or `suitcase' in a typical corpus. We will say that an element of a Winograd schema is “solved” if the referent of the pronoun is identified. An example of a pair of sentences satisfying conditions 1-3 but not 4 would be Since women cannot be carcinogenic and pills cannot be pregnant, the pronoun `they' in these sentences is easily disambiguated using selectional restrictions. This pair is therefore not a valid Winograd schema. The Winograd Schema Challenge (WSC) is a challenge for AI programs. The program is presented with a collection of sentences, each of which is one element of a Winograd schema, and is required to find the correct referent for the ambiguous pronoun. An AI program passes the challenge if its success rate is comparable to a human reader. The challenge is administered by commonsensereasoning.org and sponsored by Nuance Inc. It was offered for the first time at IJCAI-2016 (Morgenstern, Davis, and Ortiz, in preparation); the organizers plan to continue to offer it roughly once a year. Winograd schemas as the basis for challenges for machine translation programs In many cases, the identification of the referent of the prounoun in a Winograd schema is critical for finding the correct translation of that pronoun in a different language. Therefore, Winograd schemas can be used as a very difficult challenge for the depth of understanding achieved by a machine translation program. The third person plural pronoun `they' has no gender in English and most other languages (unlike the third person singular pronouns `he', `she', and `it'). However Romance languages such as French, Italian, and Spanish, and Semitic languages such as Hebrew and Arabic distinguish between the masculine and the feminine third-person plural pronouns, at least in some grammatical cases. For instance in French, the masculine pronoun is `ils'; the feminine pronoun is `elles'. In all of these cases, the masculine pronoun is standardly used for groups of mixed or unknown gender. In order to correctly translate a sentence in English containing the word `they' into one of these languages, it is necessary to determine whether or not the referent is a group of females. If it is, then the translation must be the feminine pronoun; otherwise, it must be the masculine pronoun. Therefore, if one can create a Winograd schema in English where the ambiguous pronoun is `they' and the correct referent for one element is a collection of men and for the other is a collection of women, then to translate both elements correctly requires solving the Winograd schema. As an example, consider the Winograd schema: If these sentences are translated into French, then `they' in the first sentence should be translated `elles', as referring to Jane and Susan, and `they' in the second sentence should be translated `ils', as referring to Fred and George. A number of the Winograd schemas already published can be “converted” quite easily to this form. Indeed, the above example was constructed in this way; the original form was “Jane knocked on Susan's door, but she did not [answer/get an answer].” Of the 144 schemas in the collection at http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.html there are 33 that can plausibly be translated this way. (#'s 3, 4, 12, 15, 17, 18, 23, 25, 26, 27, 42, 43, 44, 45, 46, 57, 58, 60, 69, 70, 71, 77, 78, 79, 86, 97, 98, 106, 111, 114, 127, 132, 144) — essentially any schema that involves two people as possible referents where the content of the sentences makes sense as applied to groups of people rather than individuals. A similar device, in the opposite direction, relies on the fact that French does not distinguish between the possessive pronouns `his' and `her'. The pronouns `son' and `sa' are gendered, but the gender agrees with the possession, not the possessor. Therefore a Winograd schema that relies on finding a referent for a possessive pronoun can be turned into a hard pair of French-to-English translation problems by making one possible referent male and the other one female. For example, schema #124 in the online collection reads “The man lifted the boy onto his [bunk bed/shoulders].” Changing the boy to a girl and translating into French gives “L'homme leva la fille sur [ses épaules/son lit superposé]”. In the first sentence “ses” is translated “his”, in the second “son” is translated “her”. The eleven Winograd schemas #'s 112, 117, 118, 119, 124, 125, 127, 129, 130, 131, and 143 can be converted this way. Care must be taken to avoid relying on, or seeming to rely on, objectionable stereotypes about men and women. One mechanism that can sometimes be used is to include a potentially problematic sentence in both directions. For instance, schema 23 from the WSC collection can be translated into both “The girls were bullying the boys so we [punished/rescued] them” and “The boys were bullying the girls, so we [punished/rescued] them,” thus avoiding any presupposition of whether girls are more likely to bully boys or vice versa. A historical note: Winograd schemas were named after Terry Winograd because of a well-known example in his doctoral thesis (Winograd 1970). In his thesis, this example followed the above form, and the importance of the example was justified in terms of machine translation. Winograd's original schema was: “The city councilmen refused to give the women a permit for a demonstration because they [feared/advocated] violence", and Winograd explained that, in the sentence with `feared', `they' would refer to the councilmen and would be translated as `ils' in French, whereas in the sentence with `advocated', `they' would refer to the women and would be translated as `elles'. In the later versions of the thesis, published as (Winograd 1972), he changed `women' to `demonstrators'; this made the disambiguation clearer, but lost the point about translation. Current state of the art No one familiar with the state of the art in machine translation technology or the state of the art of artificial intelligence generally will be surprised to learn that currently machine translation program are unable to solve these Winograd schema challenge problems. What may be more surprising is that currently (July 2016), machine translation programs are unable to choose the feminine plural pronoun even when the only possible antecedent is a group of women. For instance, the sentence “The girls sang a song and they danced” is translated into French as “Les filles ont chanté une chanson et ils ont dansé” by Google Translate (GT), Bing Translate, and Yandex. In fact, I have been unable to construct any sentence in English that is translated into any language using the feminine plural pronun. Note that, since the masculine plural pronoun is used for groups of mixed gender in all these languages, it is almost certainly more common in text than the feminine plural; hence this strategy is reasonable faute de mieux. (It also seems likely that the erroneous use of a feminine plural for a masculine antecedent sounds even more jarring than the reverse.) The same thing sometimes occurs in translating the feminine plural pronoun between languages that have it. GT translates the French word `elles' into Spanish as `ellos' (the masculine form). Curiously, in the opposite direction, it gets the right answer; the Spanish `ellas' (fem.) is translated into French as `elles'. Language-specific issues The masculine and feminine plural pronouns are distinguished in the Romance languages (French, Spanish, Italian, Portuguese etc.) and in Semitic languages (Arabic, Hebrew, etc.) I have consulted with native speakers and experts in these languages about the degree to which the gender distinction is observed in practice. The experts say that in French, Spanish, Italian, and Portuguese, the distinction is very strictly observed; the use of a masculine pronoun for a feminine antecedent is jarringly wrong to a native or fluent speaker. “Les filles ont chanté une chanson et ils ont dansé” sounds as wrong to a French speaker as “The girl sang a song and he danced” sounds to an English speaker; in both cases, the hearer will interpret the pronoun as referrinig to some other persons or person, who is male. In Hebrew and Arabic, this is much less true; in speech, and even, increasingly, in writing, the masculine pronoun is often used for a feminine antecedent. Looking further ahead, it is certainly possible that gender distinctions will be abandoned in the Romance languages, or even that English will have driven all other languages out of existence, sooner than AI systems will be able to do pronoun resolution in Winograd schemas; at that point, this test will no longer be useful. In some cases, a translation program can side-step the issue by omitting the pronoun altogether. For example, GT translates the above sentence “The girls sang a song and they danced” into Spanish as “Las chicas cantaron una canción y bailaban” and into Italian as “Le ragazze hanno cantato una canzone e ballavano.” However, with the more complex sentences of the Winograd Schemas, this strategy will rarely give a plausible translation for both elements of the schema. Other languages, other ambiguities Broadly speaking, whenever a target language INLINEFORM0 requires some distinction that is optional or non-existent in source language INLINEFORM1 , it is possible to create a sentence INLINEFORM2 in INLINEFORM3 where the missing information is not explicit but can be inferred from background knowledge. Translating INLINEFORM4 from INLINEFORM5 to INLINEFORM6 thus requires using the background knowledge to resolve the ambiguity, and will therefore be challenging for automatic machine translation. A couple of examples: The word `sie' in German serves as both the formal second person prounoun (always capitalized), the third person feminine singular, and the third person plural. Therefore, it can be translated into English as either “you”, “she”, “it”, or “they”; and into French as either `vous', `il', `elle', `ils', or `elles'. (The feminine third-person singular German `sie' can be translated as neuter in English and as masculine in French because the three languages do not slice up the worlds into genders in the same way.) Likewise, the possessive pronoun `ihr' in all its declensions can mean either `her' or `their'. In some cases, the disambiguation can be carried out on purely syntactic ground; e.g. if `sie' is the subject of a third-person singular verb, it must mean `she'. However, in many case, the disambiguation requires a deeper level of understanding. Thus, it should be possible to construct German Winograd schemas based on the words `sie' or `ihr' that have to be solved in order to translate them into English. For example, Marie fand fünf verlassene Kätzchen im Keller. Ihre Mutter war gestorben. (Marie found five abandoned kittens in the cellar. Their mother was dead.) vs. Marie fand fünf verlassene Kätzchen im Keller. Ihre Mutter war unzufrieden. (Marie found five abandoned kittens in the cellar. Her mother was displeased.) Another example: French distinguishes between a male friend `ami' and a female friend `amie'. Therefore, in translating the word “friend” into French, it is necessary, if possible, to determine the sex of the friend; and the clue for that can involve an inference that, as far as AI programs are concerned, is quite remote. Human readers are awfully good at picking up on those, of course. In general, with this kind of thing, if you want to break GT or another translation program, the trick is to place a large separation between the evidence and the word being disambiguated. In this case, at the present time, the separation can be pretty small. GT correctly translates “my friend Pierre” and “my friend Marie” as “mon ami Pierre” and “mon amie Marie” and it translates “She is my friend” as “Elle est mon amie”, but rather surpringly it breaks down at “Marie is my friend,” which it translates “Marie est mon ami.” As for something like “Jacques said to Marie, `You have always been a true friend,' ” that is quite hopeless. GT can surprise one, though, in both directions; sometimes it misses a very close clue, as in “Marie is my friend”, but other times it can carry a clue further than one would have guessed. Acknowledgements Thanks to Arianna Bisazza, Gerhard Brewka, Antoine Cerfon, Joseph Davis, Gigi Dopico-Black, Nizar Habash, Leora Morgenstern, Oded Regev, Francesca Rossi, Vesna Sabljakovic-Fritz, and Manuela Veloso for help with the various languages and discussion of gendered pronouns. References E. Davis, “Qualitative Spatial Reasoning in Interpreting Text and Narrative” Spatial Cognition and Computation, 13:4, 2013, 264-294. H. Levesque, E. Davis, and L. Morgenstern, “The Winograd Schema Challenge,” KR 2012. L. Morgenstern, E. Davis, and C. Ortiz, “Report on the Winograd Schema Challenge, 2016”, in preparation. T. Winograd, “Procedures as a Representation for Data in a Computer Program for Understanding Natural Language," Ph.D. thesis, Department of Mathematics, MIT, August 1970. Published as MIT AITR-235, January 1971. T. Winograd, Understanding Natural Language, Academic Press, 1972.
Did they collect their own datasets?
7fa3c2c0cf7f559d43e84076a9113a390c5ba03a
Did Ġthey Ġcollect Ġtheir Ġown Ġdatasets ?
No
<s> Did Ġthey Ġcollect Ġtheir Ġown Ġdatasets ? </s> Win og rad ĠSc hem as </s> A ĠWin og rad Ġschema Ġ( Le ves que , ĠDavis , Ġand ĠMorg en st ern Ġ2012 ) Ġis Ġa Ġpair Ġof Ġsentences , Ġor Ġof Ġshort Ġtexts , Ġcalled Ġthe Ġelements Ġof Ġthe Ġschema , Ġthat Ġsatisfy Ġthe Ġfollowing Ġconstraints : </s> The Ġfollowing Ġis Ġan Ġexample Ġof Ġa ĠWin og rad Ġschema : </s> Here , Ġthe Ġtwo Ġsentences Ġdiffer Ġonly Ġin Ġthe Ġlast Ġword : Ġ` large ' Ġvs . Ġ` small '. ĠThe Ġambiguous Ġpronoun Ġis Ġ` it '. ĠThe Ġtwo Ġante ced ents Ġare Ġ` tro phy ' Ġand Ġ` brown Ġsuitcase '. ĠA Ġhuman Ġreader Ġwill Ġnaturally Ġinterpret Ġ` it ' Ġas Ġreferring Ġto Ġthe Ġtrophy Ġin Ġthe Ġfirst Ġsentence Ġand Ġto Ġthe Ġsuitcase Ġin Ġthe Ġsecond Ġsentence , Ġusing Ġthe Ġworld Ġknowledge Ġthat Ġa Ġsmall Ġobject Ġcan Ġfit Ġin Ġa Ġlarge Ġcontainer , Ġbut Ġa Ġlarge Ġobject Ġcannot Ġfit Ġin Ġa Ġsmall Ġcontainer Ġ( Davis Ġ2013 ). ĠCondition Ġ4 Ġis Ġsatisfied Ġbecause Ġeither Ġa Ġtrophy Ġor Ġa Ġsuitcase Ġcan Ġbe Ġeither Ġlarge Ġor Ġsmall , Ġand Ġthere Ġis Ġno Ġreason Ġto Ġsuppose Ġthat Ġthere Ġwould Ġbe Ġa Ġmeaningful Ġcorrelation Ġof Ġ` small ' Ġor Ġ` large ' Ġwith Ġ` tro phy ' Ġor Ġ` suit case ' Ġin Ġa Ġtypical Ġcorpus . </s> We Ġwill Ġsay Ġthat Ġan Ġelement Ġof Ġa ĠWin og rad Ġschema Ġis ĠâĢ ľ s olved âĢ Ŀ Ġif Ġthe Ġrefere nt Ġof Ġthe Ġpronoun Ġis Ġidentified . </s> An Ġexample Ġof Ġa Ġpair Ġof Ġsentences Ġsatisfying Ġconditions Ġ1 - 3 Ġbut Ġnot Ġ4 Ġwould Ġbe </s> Since Ġwomen Ġcannot Ġbe Ġcarcin ogenic Ġand Ġpills Ġcannot Ġbe Ġpregnant , Ġthe Ġpronoun Ġ` they ' Ġin Ġthese Ġsentences Ġis Ġeasily Ġdis amb ig uated Ġusing Ġselection al Ġrestrictions . ĠThis Ġpair Ġis Ġtherefore Ġnot Ġa Ġvalid ĠWin og rad Ġschema . </s> The ĠWin og rad ĠSche ma ĠChallenge Ġ( W SC ) Ġis Ġa Ġchallenge Ġfor ĠAI Ġprograms . ĠThe Ġprogram Ġis Ġpresented Ġwith Ġa Ġcollection Ġof Ġsentences , Ġeach Ġof Ġwhich Ġis Ġone Ġelement Ġof Ġa ĠWin og rad Ġschema , Ġand Ġis Ġrequired Ġto Ġfind Ġthe Ġcorrect Ġrefere nt Ġfor Ġthe Ġambiguous Ġpronoun . ĠAn ĠAI Ġprogram Ġpasses Ġthe Ġchallenge Ġif Ġits Ġsuccess Ġrate Ġis Ġcomparable Ġto Ġa Ġhuman Ġreader . ĠThe Ġchallenge Ġis Ġadministered Ġby Ġcommons ense reason ing . org Ġand Ġsponsored Ġby ĠNu ance ĠInc . ĠIt Ġwas Ġoffered Ġfor Ġthe Ġfirst Ġtime Ġat ĠI J CA I - 2016 Ġ( M org en st ern , ĠDavis , Ġand ĠOrtiz , Ġin Ġpreparation ); Ġthe Ġorganizers Ġplan Ġto Ġcontinue Ġto Ġoffer Ġit Ġroughly Ġonce Ġa Ġyear . </s> Win og rad Ġsche mas Ġas Ġthe Ġbasis Ġfor Ġchallenges Ġfor Ġmachine Ġtranslation Ġprograms </s> In Ġmany Ġcases , Ġthe Ġidentification Ġof Ġthe Ġrefere nt Ġof Ġthe Ġp rou n oun Ġin Ġa ĠWin og rad Ġschema Ġis Ġcritical Ġfor Ġfinding Ġthe Ġcorrect Ġtranslation Ġof Ġthat Ġpronoun Ġin Ġa Ġdifferent Ġlanguage . ĠTherefore , ĠWin og rad Ġsche mas Ġcan Ġbe Ġused Ġas Ġa Ġvery Ġdifficult Ġchallenge Ġfor Ġthe Ġdepth Ġof Ġunderstanding Ġachieved Ġby Ġa Ġmachine Ġtranslation Ġprogram . </s> The Ġthird Ġperson Ġplural Ġpronoun Ġ` they ' Ġhas Ġno Ġgender Ġin ĠEnglish Ġand Ġmost Ġother Ġlanguages Ġ( un like Ġthe Ġthird Ġperson Ġsingular Ġpronouns Ġ` he ', Ġ` she ', Ġand Ġ` it '). ĠHowever ĠRomance Ġlanguages Ġsuch Ġas ĠFrench , ĠItalian , Ġand ĠSpanish , Ġand ĠSem itic Ġlanguages Ġsuch Ġas ĠHebrew Ġand ĠArabic Ġdistinguish Ġbetween Ġthe Ġmasculine Ġand Ġthe Ġfeminine Ġthird - person Ġplural Ġpronouns , Ġat Ġleast Ġin Ġsome Ġgram matical Ġcases . ĠFor Ġinstance Ġin ĠFrench , Ġthe Ġmasculine Ġpronoun Ġis Ġ` ils '; Ġthe Ġfeminine Ġpronoun Ġis Ġ` ell es '. ĠIn Ġall Ġof Ġthese Ġcases , Ġthe Ġmasculine Ġpronoun Ġis Ġstandard ly Ġused Ġfor Ġgroups Ġof Ġmixed Ġor Ġunknown Ġgender . </s> In Ġorder Ġto Ġcorrectly Ġtranslate Ġa Ġsentence Ġin ĠEnglish Ġcontaining Ġthe Ġword Ġ` they ' Ġinto Ġone Ġof Ġthese Ġlanguages , Ġit Ġis Ġnecessary Ġto Ġdetermine Ġwhether Ġor Ġnot Ġthe Ġrefere nt Ġis Ġa Ġgroup Ġof Ġfemales . ĠIf Ġit Ġis , Ġthen Ġthe Ġtranslation Ġmust Ġbe Ġthe Ġfeminine Ġpronoun ; Ġotherwise , Ġit Ġmust Ġbe Ġthe Ġmasculine Ġpronoun . ĠTherefore , Ġif Ġone Ġcan Ġcreate Ġa ĠWin og rad Ġschema Ġin ĠEnglish Ġwhere Ġthe Ġambiguous Ġpronoun Ġis Ġ` they ' Ġand Ġthe Ġcorrect Ġrefere nt Ġfor Ġone Ġelement Ġis Ġa Ġcollection Ġof Ġmen Ġand Ġfor Ġthe Ġother Ġis Ġa Ġcollection Ġof Ġwomen , Ġthen Ġto Ġtranslate Ġboth Ġelements Ġcorrectly Ġrequires Ġsolving Ġthe ĠWin og rad Ġschema . </s> As Ġan Ġexample , Ġconsider Ġthe ĠWin og rad Ġschema : </s> If Ġthese Ġsentences Ġare Ġtranslated Ġinto ĠFrench , Ġthen Ġ` they ' Ġin Ġthe Ġfirst Ġsentence Ġshould Ġbe Ġtranslated Ġ` ell es ', Ġas Ġreferring Ġto ĠJane Ġand ĠSusan , Ġand Ġ` they ' Ġin Ġthe Ġsecond Ġsentence Ġshould Ġbe Ġtranslated Ġ` ils ', Ġas Ġreferring Ġto ĠFred Ġand ĠGeorge . </s> A Ġnumber Ġof Ġthe ĠWin og rad Ġsche mas Ġalready Ġpublished Ġcan Ġbe ĠâĢ ľ con verted âĢ Ŀ Ġquite Ġeasily Ġto Ġthis Ġform . ĠIndeed , Ġthe Ġabove Ġexample Ġwas Ġconstructed Ġin Ġthis Ġway ; Ġthe Ġoriginal Ġform Ġwas ĠâĢ ľ Jane Ġknocked Ġon ĠSusan 's Ġdoor , Ġbut Ġshe Ġdid Ġnot Ġ[ answer / get Ġan Ġanswer ]. âĢ Ŀ ĠOf Ġthe Ġ144 Ġsche mas Ġin Ġthe Ġcollection Ġat Ġhttp :// www . cs . ny u . edu / fac ulty / d av ise / papers / Win og rad Sche mas / WS Collection . html Ġthere Ġare Ġ33 Ġthat Ġcan Ġplaus ibly Ġbe Ġtranslated Ġthis Ġway . Ġ(# ' s Ġ3 , Ġ4 , Ġ12 , Ġ15 , Ġ17 , Ġ18 , Ġ23 , Ġ25 , Ġ26 , Ġ27 , Ġ42 , Ġ43 , Ġ44 , Ġ45 , Ġ46 , Ġ57 , Ġ58 , Ġ60 , Ġ69 , Ġ70 , Ġ71 , Ġ77 , Ġ78 , Ġ79 , Ġ86 , Ġ97 , Ġ98 , Ġ106 , Ġ111 , Ġ114 , Ġ127 , Ġ132 , Ġ144 ) ĠâĢĶ Ġessentially Ġany Ġschema Ġthat Ġinvolves Ġtwo Ġpeople Ġas Ġpossible Ġrefere nt s Ġwhere Ġthe Ġcontent Ġof Ġthe Ġsentences Ġmakes Ġsense Ġas Ġapplied Ġto Ġgroups Ġof Ġpeople Ġrather Ġthan Ġindividuals . </s> A Ġsimilar Ġdevice , Ġin Ġthe Ġopposite Ġdirection , Ġrelies Ġon Ġthe Ġfact Ġthat ĠFrench Ġdoes Ġnot Ġdistinguish Ġbetween Ġthe Ġpossess ive Ġpronouns Ġ` his ' Ġand Ġ` her '. ĠThe Ġpronouns Ġ` son ' Ġand Ġ` sa ' Ġare Ġg endered , Ġbut Ġthe Ġgender Ġagrees Ġwith Ġthe Ġpossession , Ġnot Ġthe Ġposs essor . ĠTherefore Ġa ĠWin og rad Ġschema Ġthat Ġrelies Ġon Ġfinding Ġa Ġrefere nt Ġfor Ġa Ġpossess ive Ġpronoun Ġcan Ġbe Ġturned Ġinto Ġa Ġhard Ġpair Ġof ĠFrench - to - English Ġtranslation Ġproblems Ġby Ġmaking Ġone Ġpossible Ġrefere nt Ġmale Ġand Ġthe Ġother Ġone Ġfemale . ĠFor Ġexample , Ġschema Ġ# 124 Ġin Ġthe Ġonline Ġcollection Ġreads ĠâĢ ľ The Ġman Ġlifted Ġthe Ġboy Ġonto Ġhis Ġ[ b unk Ġbed / should ers ]. âĢ Ŀ ĠChanging Ġthe Ġboy Ġto Ġa Ġgirl Ġand Ġtranslating Ġinto ĠFrench Ġgives ĠâĢ ľ L ' hom me Ġle va Ġla Ġfil le Ġsur Ġ[ ses ĠÃ© p aul es / son Ġlit Ġsuper pos Ã© ] âĢ Ŀ . ĠIn Ġthe Ġfirst Ġsentence ĠâĢ ľ ses âĢ Ŀ Ġis Ġtranslated ĠâĢ ľ his âĢ Ŀ , Ġin Ġthe Ġsecond ĠâĢ ľ son âĢ Ŀ Ġis Ġtranslated ĠâĢ ľ her âĢ Ŀ . ĠThe Ġeleven ĠWin og rad Ġsche mas Ġ# ' s Ġ112 , Ġ117 , Ġ118 , Ġ119 , Ġ124 , Ġ125 , Ġ127 , Ġ129 , Ġ130 , Ġ131 , Ġand Ġ143 Ġcan Ġbe Ġconverted Ġthis Ġway . </s> Care Ġmust Ġbe Ġtaken Ġto Ġavoid Ġrelying Ġon , Ġor Ġseeming Ġto Ġrely Ġon , Ġobjectionable Ġstereotypes Ġabout Ġmen Ġand Ġwomen . ĠOne Ġmechanism Ġthat Ġcan Ġsometimes Ġbe Ġused Ġis Ġto Ġinclude Ġa Ġpotentially Ġproblematic Ġsentence Ġin Ġboth Ġdirections . ĠFor Ġinstance , Ġschema Ġ23 Ġfrom Ġthe ĠW SC Ġcollection Ġcan Ġbe Ġtranslated Ġinto Ġboth ĠâĢ ľ The Ġgirls Ġwere Ġbullying Ġthe Ġboys Ġso Ġwe Ġ[ pun ished / res c ued ] Ġthem âĢ Ŀ Ġand ĠâĢ ľ The Ġboys Ġwere Ġbullying Ġthe Ġgirls , Ġso Ġwe Ġ[ pun ished / res c ued ] Ġthem , âĢ Ŀ Ġthus Ġavoiding Ġany Ġpresupp osition Ġof Ġwhether Ġgirls Ġare Ġmore Ġlikely Ġto Ġbully Ġboys Ġor Ġvice Ġversa . </s> A Ġhistorical Ġnote : ĠWin og rad Ġsche mas Ġwere Ġnamed Ġafter ĠTerry ĠWin og rad Ġbecause Ġof Ġa Ġwell - known Ġexample Ġin Ġhis Ġdoctoral Ġthesis Ġ( Win og rad Ġ1970 ). ĠIn Ġhis Ġthesis , Ġthis Ġexample Ġfollowed Ġthe Ġabove Ġform , Ġand Ġthe Ġimportance Ġof Ġthe Ġexample Ġwas Ġjustified Ġin Ġterms Ġof Ġmachine Ġtranslation . ĠWin og rad 's Ġoriginal Ġschema Ġwas : ĠâĢ ľ The Ġcity Ġcouncil men Ġrefused Ġto Ġgive Ġthe Ġwomen Ġa Ġpermit Ġfor Ġa Ġdemonstration Ġbecause Ġthey Ġ[ f eared / adv ocated ] Ġviolence ", Ġand ĠWin og rad Ġexplained Ġthat , Ġin Ġthe Ġsentence Ġwith Ġ` f eared ', Ġ` they ' Ġwould Ġrefer Ġto Ġthe Ġcouncil men Ġand Ġwould Ġbe Ġtranslated Ġas Ġ` ils ' Ġin ĠFrench , Ġwhereas Ġin Ġthe Ġsentence Ġwith Ġ` adv ocated ', Ġ` they ' Ġwould Ġrefer Ġto Ġthe Ġwomen Ġand Ġwould Ġbe Ġtranslated Ġas Ġ` ell es '. ĠIn Ġthe Ġlater Ġversions Ġof Ġthe Ġthesis , Ġpublished Ġas Ġ( Win og rad Ġ1972 ), Ġhe Ġchanged Ġ` women ' Ġto Ġ` demon str ators '; Ġthis Ġmade Ġthe Ġdis amb ig uation Ġclearer , Ġbut Ġlost Ġthe Ġpoint Ġabout Ġtranslation . </s> Current Ġstate Ġof Ġthe Ġart </s> No Ġone Ġfamiliar Ġwith Ġthe Ġstate Ġof Ġthe Ġart Ġin Ġmachine Ġtranslation Ġtechnology Ġor Ġthe Ġstate Ġof Ġthe Ġart Ġof Ġartificial Ġintelligence Ġgenerally Ġwill Ġbe Ġsurprised Ġto Ġlearn Ġthat Ġcurrently Ġmachine Ġtranslation Ġprogram Ġare Ġunable Ġto Ġsolve Ġthese ĠWin og rad Ġschema Ġchallenge Ġproblems . </s> What Ġmay Ġbe Ġmore Ġsurprising Ġis Ġthat Ġcurrently Ġ( July Ġ2016 ), Ġmachine Ġtranslation Ġprograms Ġare Ġunable Ġto Ġchoose Ġthe Ġfeminine Ġplural Ġpronoun Ġeven Ġwhen Ġthe Ġonly Ġpossible Ġante ced ent Ġis Ġa Ġgroup Ġof Ġwomen . ĠFor Ġinstance , Ġthe Ġsentence ĠâĢ ľ The Ġgirls Ġsang Ġa Ġsong Ġand Ġthey Ġdanced âĢ Ŀ Ġis Ġtranslated Ġinto ĠFrench Ġas ĠâĢ ľ Les Ġfill es Ġont Ġchant Ã© Ġune Ġch anson Ġet Ġil s Ġont Ġd ans Ã© âĢ Ŀ Ġby ĠGoogle ĠTrans late Ġ( GT ), ĠBing ĠTrans late , Ġand ĠY and ex . ĠIn Ġfact , ĠI Ġhave Ġbeen Ġunable Ġto Ġconstruct Ġany Ġsentence Ġin ĠEnglish Ġthat Ġis Ġtranslated Ġinto Ġany Ġlanguage Ġusing Ġthe Ġfeminine Ġplural Ġpron un . ĠNote Ġthat , Ġsince Ġthe Ġmasculine Ġplural Ġpronoun Ġis Ġused Ġfor Ġgroups Ġof Ġmixed Ġgender Ġin Ġall Ġthese Ġlanguages , Ġit Ġis Ġalmost Ġcertainly Ġmore Ġcommon Ġin Ġtext Ġthan Ġthe Ġfeminine Ġplural ; Ġhence Ġthis Ġstrategy Ġis Ġreasonable Ġfa ute Ġde Ġm ie ux . Ġ( It Ġalso Ġseems Ġlikely Ġthat Ġthe Ġerroneous Ġuse Ġof Ġa Ġfeminine Ġplural Ġfor Ġa Ġmasculine Ġante ced ent Ġsounds Ġeven Ġmore Ġjarring Ġthan Ġthe Ġreverse .) </s> The Ġsame Ġthing Ġsometimes Ġoccurs Ġin Ġtranslating Ġthe Ġfeminine Ġplural Ġpronoun Ġbetween Ġlanguages Ġthat Ġhave Ġit . ĠGT Ġtranslates Ġthe ĠFrench Ġword Ġ` ell es ' Ġinto ĠSpanish Ġas Ġ` ell os ' Ġ( the Ġmasculine Ġform ). ĠCur iously , Ġin Ġthe Ġopposite Ġdirection , Ġit Ġgets Ġthe Ġright Ġanswer ; Ġthe ĠSpanish Ġ` ell as ' Ġ( f em .) Ġis Ġtranslated Ġinto ĠFrench Ġas Ġ` ell es '. </s> Language - specific Ġissues </s> The Ġmasculine Ġand Ġfeminine Ġplural Ġpronouns Ġare Ġdistinguished Ġin Ġthe ĠRomance Ġlanguages Ġ( French , ĠSpanish , ĠItalian , ĠPortuguese Ġetc .) Ġand Ġin ĠSem itic Ġlanguages Ġ( Arab ic , ĠHebrew , Ġetc .) ĠI Ġhave Ġconsulted Ġwith Ġnative Ġspeakers Ġand Ġexperts Ġin Ġthese Ġlanguages Ġabout Ġthe Ġdegree Ġto Ġwhich Ġthe Ġgender Ġdistinction Ġis Ġobserved Ġin Ġpractice . ĠThe Ġexperts Ġsay Ġthat Ġin ĠFrench , ĠSpanish , ĠItalian , Ġand ĠPortuguese , Ġthe Ġdistinction Ġis Ġvery Ġstrictly Ġobserved ; Ġthe Ġuse Ġof Ġa Ġmasculine Ġpronoun Ġfor Ġa Ġfeminine Ġante ced ent Ġis Ġjarring ly Ġwrong Ġto Ġa Ġnative Ġor Ġfluent Ġspeaker . ĠâĢ ľ Les Ġfill es Ġont Ġchant Ã© Ġune Ġch anson Ġet Ġil s Ġont Ġd ans Ã© âĢ Ŀ Ġsounds Ġas Ġwrong Ġto Ġa ĠFrench Ġspeaker Ġas ĠâĢ ľ The Ġgirl Ġsang Ġa Ġsong Ġand Ġhe Ġdanced âĢ Ŀ Ġsounds Ġto Ġan ĠEnglish Ġspeaker ; Ġin Ġboth Ġcases , Ġthe Ġhe arer Ġwill Ġinterpret Ġthe Ġpronoun Ġas Ġrefer rin ig Ġto Ġsome Ġother Ġpersons Ġor Ġperson , Ġwho Ġis Ġmale . ĠIn ĠHebrew Ġand ĠArabic , Ġthis Ġis Ġmuch Ġless Ġtrue ; Ġin Ġspeech , Ġand Ġeven , Ġincreasingly , Ġin Ġwriting , Ġthe Ġmasculine Ġpronoun Ġis Ġoften Ġused Ġfor Ġa Ġfeminine Ġante ced ent . </s> Looking Ġfurther Ġahead , Ġit Ġis Ġcertainly Ġpossible Ġthat Ġgender Ġdistinctions Ġwill Ġbe Ġabandoned Ġin Ġthe ĠRomance Ġlanguages , Ġor Ġeven Ġthat ĠEnglish Ġwill Ġhave Ġdriven Ġall Ġother Ġlanguages Ġout Ġof Ġexistence , Ġsooner Ġthan ĠAI Ġsystems Ġwill Ġbe Ġable Ġto Ġdo Ġpronoun Ġresolution Ġin ĠWin og rad Ġsche mas ; Ġat Ġthat Ġpoint , Ġthis Ġtest Ġwill Ġno Ġlonger Ġbe Ġuseful . </s> In Ġsome Ġcases , Ġa Ġtranslation Ġprogram Ġcan Ġside - step Ġthe Ġissue Ġby Ġo mitting Ġthe Ġpronoun Ġaltogether . ĠFor Ġexample , ĠGT Ġtranslates Ġthe Ġabove Ġsentence ĠâĢ ľ The Ġgirls Ġsang Ġa Ġsong Ġand Ġthey Ġdanced âĢ Ŀ Ġinto ĠSpanish Ġas ĠâĢ ľ Las Ġch icas Ġcant aron Ġun a Ġcan ci Ã³n Ġy Ġbail aban âĢ Ŀ Ġand Ġinto ĠItalian Ġas ĠâĢ ľ Le Ġrag az ze Ġh ann o Ġcant ato Ġun a Ġcan zone Ġe Ġball av ano . âĢ Ŀ ĠHowever , Ġwith Ġthe Ġmore Ġcomplex Ġsentences Ġof Ġthe ĠWin og rad ĠSc hem as , Ġthis Ġstrategy Ġwill Ġrarely Ġgive Ġa Ġplausible Ġtranslation Ġfor Ġboth Ġelements Ġof Ġthe Ġschema . </s> Other Ġlanguages , Ġother Ġambig u ities </s> Broad ly Ġspeaking , Ġwhenever Ġa Ġtarget Ġlanguage ĠIN LINE FORM 0 Ġrequires Ġsome Ġdistinction Ġthat Ġis Ġoptional Ġor Ġnon - existent Ġin Ġsource Ġlanguage ĠIN LINE FORM 1 Ġ, Ġit Ġis Ġpossible Ġto Ġcreate Ġa Ġsentence ĠIN LINE FORM 2 Ġin ĠIN LINE FORM 3 Ġwhere Ġthe Ġmissing Ġinformation Ġis Ġnot Ġexplicit Ġbut Ġcan Ġbe Ġinferred Ġfrom Ġbackground Ġknowledge . ĠTrans l ating ĠIN LINE FORM 4 Ġfrom ĠIN LINE FORM 5 Ġto ĠIN LINE FORM 6 Ġthus Ġrequires Ġusing Ġthe Ġbackground Ġknowledge Ġto Ġresolve Ġthe Ġambiguity , Ġand Ġwill Ġtherefore Ġbe Ġchallenging Ġfor Ġautomatic Ġmachine Ġtranslation . </s> A Ġcouple Ġof Ġexamples : ĠThe Ġword Ġ` sie ' Ġin ĠGerman Ġserves Ġas Ġboth Ġthe Ġformal Ġsecond Ġperson Ġp rou n oun Ġ( always Ġcapital ized ), Ġthe Ġthird Ġperson Ġfeminine Ġsingular , Ġand Ġthe Ġthird Ġperson Ġplural . ĠTherefore , Ġit Ġcan Ġbe Ġtranslated Ġinto ĠEnglish Ġas Ġeither ĠâĢ ľ you âĢ Ŀ , ĠâĢ ľ she âĢ Ŀ , ĠâĢ ľ it âĢ Ŀ , Ġor ĠâĢ ľ they âĢ Ŀ ; Ġand Ġinto ĠFrench Ġas Ġeither Ġ` vous ', Ġ` il ', Ġ` elle ', Ġ` ils ', Ġor Ġ` ell es '. Ġ( The Ġfeminine Ġthird - person Ġsingular ĠGerman Ġ` sie ' Ġcan Ġbe Ġtranslated Ġas Ġne uter Ġin ĠEnglish Ġand Ġas Ġmasculine Ġin ĠFrench Ġbecause Ġthe Ġthree Ġlanguages Ġdo Ġnot Ġslice Ġup Ġthe Ġworlds Ġinto Ġgenders Ġin Ġthe Ġsame Ġway .) ĠLikewise , Ġthe Ġpossess ive Ġpronoun Ġ` ih r ' Ġin Ġall Ġits Ġdecl ensions Ġcan Ġmean Ġeither Ġ` her ' Ġor Ġ` their '. ĠIn Ġsome Ġcases , Ġthe Ġdis amb ig uation Ġcan Ġbe Ġcarried Ġout Ġon Ġpurely Ġsynt actic Ġground ; Ġe . g . Ġif Ġ` sie ' Ġis Ġthe Ġsubject Ġof Ġa Ġthird - person Ġsingular Ġverb , Ġit Ġmust Ġmean Ġ` she '. ĠHowever , Ġin Ġmany Ġcase , Ġthe Ġdis amb ig uation Ġrequires Ġa Ġdeeper Ġlevel Ġof Ġunderstanding . ĠThus , Ġit Ġshould Ġbe Ġpossible Ġto Ġconstruct ĠGerman ĠWin og rad Ġsche mas Ġbased Ġon Ġthe Ġwords Ġ` sie ' Ġor Ġ` ih r ' Ġthat Ġhave Ġto Ġbe Ġsolved Ġin Ġorder Ġto Ġtranslate Ġthem Ġinto ĠEnglish . ĠFor Ġexample , </s> Marie Ġf and Ġf Ã¼ n f Ġver lass ene ĠK Ã¤ tz chen Ġim ĠKeller . ĠI h re ĠM utter Ġwar Ġgest or ben . </s> ( Marie Ġfound Ġfive Ġabandoned Ġkittens Ġin Ġthe Ġcellar . ĠTheir Ġmother Ġwas Ġdead .) </s> vs . </s> Marie Ġf and Ġf Ã¼ n f Ġver lass ene ĠK Ã¤ tz chen Ġim ĠKeller . ĠI h re ĠM utter Ġwar Ġun z uf ried en . </s> ( Marie Ġfound Ġfive Ġabandoned Ġkittens Ġin Ġthe Ġcellar . ĠHer Ġmother Ġwas Ġdisple ased .) </s> Another Ġexample : ĠFrench Ġdistinguishes Ġbetween Ġa Ġmale Ġfriend Ġ` ami ' Ġand Ġa Ġfemale Ġfriend Ġ` am ie '. ĠTherefore , Ġin Ġtranslating Ġthe Ġword ĠâĢ ľ friend âĢ Ŀ Ġinto ĠFrench , Ġit Ġis Ġnecessary , Ġif Ġpossible , Ġto Ġdetermine Ġthe Ġsex Ġof Ġthe Ġfriend ; Ġand Ġthe Ġclue Ġfor Ġthat Ġcan Ġinvolve Ġan Ġinference Ġthat , Ġas Ġfar Ġas ĠAI Ġprograms Ġare Ġconcerned , Ġis Ġquite Ġremote . ĠHuman Ġreaders Ġare Ġawfully Ġgood Ġat Ġpicking Ġup Ġon Ġthose , Ġof Ġcourse . ĠIn Ġgeneral , Ġwith Ġthis Ġkind Ġof Ġthing , Ġif Ġyou Ġwant Ġto Ġbreak ĠGT Ġor Ġanother Ġtranslation Ġprogram , Ġthe Ġtrick Ġis Ġto Ġplace Ġa Ġlarge Ġseparation Ġbetween Ġthe Ġevidence Ġand Ġthe Ġword Ġbeing Ġdis amb ig uated . ĠIn Ġthis Ġcase , Ġat Ġthe Ġpresent Ġtime , Ġthe Ġseparation Ġcan Ġbe Ġpretty Ġsmall . ĠGT Ġcorrectly Ġtranslates ĠâĢ ľ my Ġfriend ĠPierre âĢ Ŀ Ġand ĠâĢ ľ my Ġfriend ĠMarie âĢ Ŀ Ġas ĠâĢ ľ mon Ġam i ĠPierre âĢ Ŀ Ġand ĠâĢ ľ mon Ġam ie ĠMarie âĢ Ŀ Ġand Ġit Ġtranslates ĠâĢ ľ She Ġis Ġmy Ġfriend âĢ Ŀ Ġas ĠâĢ ľ El le Ġest Ġmon Ġam ie âĢ Ŀ , Ġbut Ġrather Ġsurpr ingly Ġit Ġbreaks Ġdown Ġat ĠâĢ ľ Marie Ġis Ġmy Ġfriend , âĢ Ŀ Ġwhich Ġit Ġtranslates ĠâĢ ľ Marie Ġest Ġmon Ġam i . âĢ Ŀ ĠAs Ġfor Ġsomething Ġlike ĠâĢ ľ Jac ques Ġsaid Ġto ĠMarie , Ġ` You Ġhave Ġalways Ġbeen Ġa Ġtrue Ġfriend ,' ĠâĢ Ŀ Ġthat Ġis Ġquite Ġhopeless . ĠGT Ġcan Ġsurprise Ġone , Ġthough , Ġin Ġboth Ġdirections ; Ġsometimes Ġit Ġmisses Ġa Ġvery Ġclose Ġclue , Ġas Ġin ĠâĢ ľ Marie Ġis Ġmy Ġfriend âĢ Ŀ , Ġbut Ġother Ġtimes Ġit Ġcan Ġcarry Ġa Ġclue Ġfurther Ġthan Ġone Ġwould Ġhave Ġguessed . </s> Acknowled gements </s> Thanks Ġto ĠAri anna ĠBis az za , ĠGer hard ĠBrew ka , ĠAnt oine ĠCer f on , ĠJoseph ĠDavis , ĠGig i ĠD op ico - Black , ĠN iz ar ĠHab ash , ĠLe ora ĠMorg en st ern , ĠO ded ĠRe ge v , ĠFrances ca ĠRossi , ĠVes na ĠSab l j ak ovic - F ritz , Ġand ĠManuel a ĠVel oso Ġfor Ġhelp Ġwith Ġthe Ġvarious Ġlanguages Ġand Ġdiscussion Ġof Ġg endered Ġpronouns . </s> References </s> E . ĠDavis , ĠâĢ ľ Qual itative ĠSp atial ĠReason ing Ġin ĠInter pre ting ĠText Ġand ĠNarr ative âĢ Ŀ ĠSp atial ĠCogn ition Ġand ĠComput ation , Ġ13 : 4 , Ġ2013 , Ġ264 - 294 . </s> H . ĠLe ves que , ĠE . ĠDavis , Ġand ĠL . ĠMorg en st ern , ĠâĢ ľ The ĠWin og rad ĠSche ma ĠChallenge , âĢ Ŀ ĠKR Ġ2012 . </s> L . ĠMorg en st ern , ĠE . ĠDavis , Ġand ĠC . ĠOrtiz , ĠâĢ ľ Report Ġon Ġthe ĠWin og rad ĠSche ma ĠChallenge , Ġ2016 âĢ Ŀ , Ġin Ġpreparation . </s> T . ĠWin og rad , ĠâĢ ľ Pro ced ures Ġas Ġa ĠRepresent ation Ġfor ĠData Ġin Ġa ĠComputer ĠProgram Ġfor ĠUnderstanding ĠNatural ĠLanguage ," ĠPh . D . Ġthesis , ĠDepartment Ġof ĠMathematics , ĠMIT , ĠAugust Ġ1970 . ĠPublished Ġas ĠMIT ĠA IT R - 235 , ĠJanuary Ġ1971 . </s> T . ĠWin og rad , ĠUnderstanding ĠNatural ĠLanguage , ĠAcademic ĠPress , Ġ1972 .