hierarchical phrase-based system BIBREF29, appropriate additional baseline would be to mark translation rules with these indicator functions but without the scores, akin to identifying rules with phrases in them (Baseline + SegOn)
EXTRACTIVE
 English-Spanish MT system 
EXTRACTIVE
To test this hypothesis, we built an English-Spanish MT system using the cdec decoder BIBREF27 for the entire training pipeline (word alignments, phrase extraction, feature weight tuning, and decoding). Corpora from the WMT 2011 evaluation was used to build the translation and language models, and for tuning (on news-test2010) and evaluation (on news-test2011), with scoring done using BLEU BIBREF28 . The baseline is a hierarchical phrase-based system BIBREF29 with a 4-gram language model, with feature weights tuned using MIRA BIBREF30 . For features, each translation rule is decorated with two lexical and phrasal features corresponding to the forward INLINEFORM0 and backward INLINEFORM1 conditional log frequencies, along with the log joint frequency INLINEFORM2 , the log frequency of the source phrase INLINEFORM3 , and whether the phrase pair or the source phrase is a singleton. Weights for the language model, glue rule, and word penalty are also tuned. This setup (Baseline) achieves scores en par with the published WMT results. We added the compositionality score as an additional feature, and also added two binary-valued features: the first indicates if the given translation rule has not been decorated with a compositionality score (either because it consists of non-terminals only or the lexical items in the translation rule are unigrams), and correspondingly the second feature indicates if the translation rule has been scored. Therefore, an appropriate additional baseline would be to mark translation rules with these indicator functions but without the scores, akin to identifying rules with phrases in them (Baseline + SegOn).
To test this hypothesis, we built an English-Spanish MT system using the cdec decoder BIBREF27 for the entire training pipeline (word alignments, phrase extraction, feature weight tuning, and decoding). Corpora from the WMT 2011 evaluation was used to build the translation and language models, and for tuning (on news-test2010) and evaluation (on news-test2011), with scoring done using BLEU BIBREF28 . The baseline is a hierarchical phrase-based system BIBREF29 with a 4-gram language model, with feature weights tuned using MIRA BIBREF30 . For features, each translation rule is decorated with two lexical and phrasal features corresponding to the forward INLINEFORM0 and backward INLINEFORM1 conditional log frequencies, along with the log joint frequency INLINEFORM2 , the log frequency of the source phrase INLINEFORM3 , and whether the phrase pair or the source phrase is a singleton. Weights for the language model, glue rule, and word penalty are also tuned. This setup (Baseline) achieves scores en par with the published WMT results.
1801.10293
Introduction </s> N umerous Ġlex ical Ġsemantic Ġproperties Ġare Ġcaptured Ġby Ġrepresentations Ġencoding Ġdistribution al Ġproperties Ġof Ġwords , Ġas Ġhas Ġbeen Ġdemonstrated Ġin Ġa Ġvariety Ġof Ġtasks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠHowever , Ġthis Ġdistribution al Ġaccount Ġof Ġmeaning Ġdoes Ġnot Ġscale Ġto Ġlarger Ġunits Ġlike Ġphrases Ġand Ġsentences ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġmotivating Ġresearch Ġinto Ġcompos itional Ġmodels Ġthat Ġcombine Ġword Ġrepresentations Ġto Ġproduce Ġrepresentations Ġof Ġthe Ġsemantics Ġof Ġlonger Ġunits ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠPrevious Ġwork Ġhas Ġlearned Ġthese Ġmodels Ġusing Ġaut oen c oder Ġformulations ĠB IB REF 8 Ġor Ġlimited Ġhuman Ġsupervision ĠB IB REF 5 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġexplore Ġthe Ġhypothesis Ġthat Ġthe Ġequivalent Ġknowledge Ġabout Ġhow Ġwords Ġcompose Ġcan Ġbe Ġobtained Ġthrough Ġmon oling ual Ġparaph r ases Ġthat Ġhave Ġbeen Ġextracted Ġusing Ġword Ġalign ments Ġand Ġan Ġintermediate Ġlanguage ĠB IB REF 9 Ġ. ĠConf irming Ġthis Ġhypothesis Ġwould Ġallow Ġthe Ġrapid Ġdevelopment Ġof Ġcompos itional Ġmodels Ġin Ġa Ġlarge Ġnumber Ġof Ġlanguages . </s> As Ġtheir Ġname Ġsuggests , Ġthese Ġmodels Ġalso Ġimpose Ġthe Ġassumption Ġthat Ġlonger Ġunits Ġlike Ġphrases Ġare Ġcompos itional , Ġi . e ., Ġa Ġphrase 's Ġmeaning Ġcan Ġbe Ġunderstood Ġfrom Ġthe Ġliteral Ġmeaning Ġof Ġits Ġparts . ĠHowever , Ġcountless Ġexamples Ġthat Ġrun Ġcontrary Ġto Ġthe Ġassumption Ġexist , Ġand Ġhandling Ġthese Ġnon - com pos itional Ġphrases Ġhas Ġbeen Ġproblematic Ġand Ġof Ġlong - standing Ġinterest Ġin Ġthe Ġcommunity ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. Ġ( Non -) ĠCom position ality Ġdetection Ġcan Ġprovide Ġvital Ġinformation Ġto Ġother Ġlanguage Ġprocessing Ġsystems Ġon Ġwhether Ġa Ġmulti word Ġunit Ġshould Ġbe Ġtreated Ġsem antically Ġas Ġa Ġsingle Ġentity Ġor Ġnot , Ġand Ġscoring Ġthis Ġphenomenon Ġis Ġparticularly Ġrelevant Ġfor Ġdownstream Ġtasks Ġlike Ġmachine Ġtranslation Ġ( MT ) Ġor Ġinformation Ġretrieval . ĠWe Ġexplore Ġthe Ġhypothesis Ġthat Ġcontextual Ġevidence Ġcan Ġbe Ġused Ġto Ġdetermine Ġthe Ġrelative Ġdegree Ġto Ġwhich Ġa Ġphrase Ġis Ġmeant Ġcomposition ally . </s> Rather Ġthan Ġfocusing Ġpurely Ġon Ġintrinsic Ġclean - room Ġevaluations , Ġthe Ġgoal Ġof Ġthis Ġwork Ġis Ġto Ġlearn Ġrelatively Ġaccurate Ġcontext - sensitive Ġcompos itional Ġmodels Ġthat Ġare Ġalso Ġdirectly Ġapplicable Ġin Ġreal - world , Ġnoisy - data Ġscenarios . ĠThis Ġobjective Ġnecess itates Ġcertain Ġdesign Ġdecisions , Ġand Ġto Ġthis Ġend Ġwe Ġpropose Ġa Ġrobust , Ġscalable Ġframework Ġthat Ġlearns Ġcompos itional Ġfunctions Ġand Ġscores Ġrelative Ġphr as al Ġcomposition ality . ĠWe Ġmake Ġthree Ġcontributions : Ġfirst , Ġa Ġnovel Ġway Ġto Ġlearn Ġcompos itional Ġfunctions Ġfor Ġpart - of - speech Ġpairs Ġthat Ġuses Ġsupervision Ġfrom Ġan Ġautomatically - ext racted Ġlist Ġof Ġparaph r ases Ġ( Â§ ĠSEC REF 3 Ġ). ĠSecond , Ġa Ġcontext - dependent Ġscoring Ġmodel Ġthat Ġscores Ġthe Ġrelative Ġcomposition ality Ġof Ġa Ġphrase ĠB IB REF 12 Ġby Ġcomputing Ġthe Ġlikelihood Ġof Ġits Ġcontext Ġgiven Ġits Ġparaph rase - learn ed Ġrepresentation Ġ( Â§ ĠSEC REF 4 Ġ). ĠAnd Ġthird , Ġan Ġevaluation Ġof Ġthe Ġimpact Ġof Ġcomposition ality Ġknowledge Ġin Ġan Ġend - to - end ĠMT Ġsetup . ĠOur Ġexperiments Ġ( Â§ ĠSEC REF 5 Ġ) Ġreveal Ġthat Ġusing Ġsupervision Ġfrom Ġautomatically Ġextracted Ġparaph r ases Ġproduces Ġcompos itional Ġfunctions Ġwith Ġequivalent Ġperformance Ġto Ġprevious Ġapproaches Ġthat Ġhave Ġrelied Ġon Ġhand - annot ated Ġtraining Ġdata . ĠFurthermore , Ġcomposition ality Ġfeatures Ġconsistently Ġimprove Ġthe Ġtranslations Ġproduced Ġby Ġa Ġstrong ĠEnglish âĢĵ Spanish Ġtranslation Ġsystem . </s> Param etric ĠCom position ĠFunctions </s> We Ġformal ize Ġcomposition Ġas Ġa Ġfunction ĠIN LINE FORM 0 Ġthat Ġmaps ĠIN LINE FORM 1 Ġ- dimensional Ġvector Ġrepresentations Ġof Ġphrase Ġconstituents ĠIN LINE FORM 2 Ġto Ġan ĠIN LINE FORM 3 Ġ- dimensional Ġvector Ġrepresentation Ġof Ġthe Ġphrase , Ġi . e ., Ġthe Ġcomposed Ġrepresentation . ĠA Ġphrase Ġis Ġdefined Ġas Ġany Ġcontiguous Ġsequence Ġof Ġwords Ġof Ġlength Ġ2 Ġor Ġgreater , Ġand Ġdoes Ġnot Ġhave Ġto Ġadhere Ġto Ġconstituents Ġin Ġa Ġphrase Ġstructure Ġgrammar . ĠThis Ġdefinition Ġis Ġin Ġline Ġwith Ġour ĠMT Ġapplication Ġand Ġignores ĠâĢ ľ g appy âĢ Ŀ Ġnon cont iguous Ġphrases , Ġbut Ġthis Ġpragmatic Ġchoice Ġdoes Ġexclude Ġmany Ġverb - object Ġrelations ĠB IB REF 13 Ġ. ĠWe Ġassume Ġthe Ġexistence Ġof Ġword - level Ġvector Ġrepresentations Ġfor Ġevery Ġword Ġin Ġour Ġvocabulary Ġof Ġsize ĠIN LINE FORM 4 Ġ. ĠCom position ality Ġis Ġmodeled Ġas Ġa Ġbil in ear Ġmap , Ġand Ġtwo Ġclasses Ġof Ġlinear Ġmodels Ġwith Ġdifferent Ġlevels Ġof Ġparam et riz ation Ġare Ġproposed . ĠUnlike Ġprevious Ġwork ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 14 Ġwhere Ġthe Ġfunctions Ġare Ġword - specific , Ġour Ġcompos itional Ġfunctions Ġoperate Ġon Ġpart - of - speech Ġ( POS ) Ġtag Ġpairs , Ġwhich Ġfacilitates Ġlearning Ġby Ġdrastically Ġreducing Ġthe Ġnumber Ġof Ġparameters , Ġand Ġonly Ġrequires Ġa Ġshallow Ġsynt actic Ġparse Ġof Ġthe Ġinput . </s> Con cat en ation ĠModels </s> Our Ġfirst Ġclass Ġof Ġmodels Ġis Ġa Ġgeneral ization Ġof Ġthe Ġadditive Ġmodels Ġintroduced Ġin ĠMitchell 2008 : ĠDIS PLAY FORM 0 </s> where Ġthe Ġnotation ĠIN LINE FORM 0 Ġrepresents Ġa Ġvertical Ġ( row - wise ) Ġconc aten ation Ġof Ġtwo Ġvectors ; Ġnamely , Ġthe Ġconc aten ation Ġthat Ġresults Ġin Ġa ĠIN LINE FORM 1 Ġ- sized Ġvector . ĠIn Ġaddition Ġto Ġthe ĠIN LINE FORM 2 Ġparameters Ġfor Ġthe Ġword Ġvector Ġrepresentations Ġthat Ġare Ġprovided Ġa Ġprior i , Ġthis Ġmodel Ġintroduces ĠIN LINE FORM 3 Ġparameters , Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġnumber Ġof ĠPOS - tag Ġpairs Ġwe Ġconsider . </s> Mit chell 2008 Ġsignificantly Ġsimplify Ġparameter Ġestimation Ġby Ġassuming Ġa Ġcertain Ġstructure Ġfor Ġthe Ġparameter Ġmatrix ĠIN LINE FORM 0 Ġ, Ġwhich Ġis Ġnecessary Ġgiven Ġthe Ġlimited Ġhuman - annot ated Ġdata Ġthey Ġuse . ĠFor Ġexample , Ġby Ġassuming Ġa Ġblock - di agonal Ġstructure , Ġwe Ġget Ġa Ġscaled Ġelement - wise Ġaddition Ġmodel ĠIN LINE FORM 1 Ġ. ĠWhile Ġnot Ġstrictly Ġin Ġthis Ġcategory Ġdue Ġto Ġthe Ġnon - linear ities Ġinvolved , Ġneural Ġnetwork - based Ġcompos itional Ġmodels ĠB IB REF 7 Ġ, ĠB IB REF 15 Ġcan Ġbe Ġviewed Ġas Ġconc aten ation Ġmodels , Ġalthough Ġthe Ġorder Ġof Ġconc aten ation Ġand Ġmatrix Ġmultiplication Ġis Ġswitched . ĠHowever , Ġthese Ġmodels Ġintroduce Ġmore Ġthan ĠIN LINE FORM 2 Ġparameters . </s> T ensor ĠModels </s> The Ġsecond Ġclass Ġof Ġmodels Ġlever ages Ġpair wise Ġmultipl icative Ġinteractions Ġbetween Ġthe Ġcomponents Ġof Ġthe Ġtwo Ġword Ġvectors : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġcorresponds Ġto Ġa Ġtens or Ġcontraction Ġalong Ġthe ĠIN LINE FORM 1 Ġmode Ġof Ġthe Ġtens or ĠIN LINE FORM 2 Ġ. ĠIn Ġthis Ġcase , Ġwe Ġfirst Ġcompute Ġa Ġcontraction Ġ( t ensor - vector Ġproduct ) Ġbetween ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġalong ĠIN LINE FORM 5 Ġ' s Ġthird Ġmode , Ġcorresponding Ġto Ġinteractions Ġwith Ġthe Ġsecond Ġword Ġvector Ġof Ġa Ġtwo - word Ġphrase Ġand Ġresulting Ġin Ġa Ġmatrix , Ġwhich Ġis Ġthen Ġmultiplied Ġalong Ġits Ġsecond Ġmode Ġ( cor respond ing Ġto Ġtraditional Ġmatrix Ġmultiplication Ġon Ġthe Ġright ) Ġby ĠIN LINE FORM 6 Ġ. ĠThe Ġfinal Ġresult Ġis Ġan ĠIN LINE FORM 7 Ġvector . ĠThis Ġmodel Ġintroduces ĠIN LINE FORM 8 Ġparameters . </s> T ensor Ġmodels Ġare Ġa Ġgeneral ization Ġof Ġthe Ġelement - wise Ġmultipl icative Ġmodel ĠB IB REF 16 Ġ, Ġwhich Ġpermits Ġnon - zero Ġvalues Ġonly Ġon Ġthe Ġtens or Ġdiagonal . ĠOperating Ġat Ġthe Ġvocabulary Ġlevel , Ġthe Ġmodel Ġof ĠBar oni 2010 Ġhas Ġinteresting Ġparallels Ġto Ġour Ġtens or Ġmodel . ĠThey Ġfocus Ġon Ġadjective âĢĵ n oun Ġrelationships Ġand Ġlearn Ġa Ġspecific Ġmatrix Ġfor Ġevery Ġadjective Ġin Ġtheir Ġdataset ; Ġin Ġour Ġcase , Ġthe Ġspecific Ġmatrix Ġfor Ġeach Ġadjective Ġhas Ġa Ġparticular Ġform , Ġnamely Ġthat Ġit Ġcan Ġbe Ġfactor ized Ġinto Ġthe Ġproduct Ġof Ġa Ġtens or Ġand Ġa Ġvector ; Ġthe Ġtens or Ġcorresponds Ġto Ġthe Ġactual Ġadjective âĢĵ n oun Ġcomb iner Ġfunction , Ġand Ġthe Ġvector Ġcorresponds Ġto Ġspecific Ġlex ical Ġinformation Ġthat Ġthe Ġadjective Ġcarries . ĠThis Ġconcept Ġgeneral izes Ġto Ġother ĠPOS Ġpairs : Ġfor Ġexample , Ġmultiplying Ġthe Ġtens or Ġthat Ġrepresents Ġdetermin er - n oun Ġcombinations Ġalong Ġthe Ġsecond Ġmode Ġwith Ġthe Ġvector Ġfor ĠâĢ ľ the âĢ Ŀ Ġresults Ġin Ġa Ġmatrix Ġthat Ġrepresents Ġthe Ġsemantic Ġoperation Ġof Ġdefin it eness . ĠLearning Ġthese Ġparameters Ġjointly Ġis Ġstatistically Ġmore Ġefficient Ġthan Ġseparately Ġlearning Ġversions Ġfor Ġeach Ġword . </s> Long er ĠPh r ases </s> The Ġproposed Ġmodels Ġoperate Ġon Ġpairs Ġof Ġwords Ġat Ġa Ġtime . ĠTo Ġhandle Ġphrases Ġof Ġlength Ġgreater Ġthan Ġtwo , Ġwe Ġgreed ily Ġconstruct Ġa Ġleft - br anch ing Ġtree Ġof Ġthe Ġphrase Ġconstituents Ġthat Ġeventually Ġdictates Ġthe Ġapplication Ġof Ġthe Ġlearned Ġbil in ear Ġmaps . ĠFor Ġeach Ġinternal Ġtree Ġnode , Ġwe Ġconsider Ġthe ĠPOS Ġtags Ġof Ġits Ġchildren : Ġif Ġthe Ġright Ġchild Ġis Ġa Ġnoun , Ġand Ġthe Ġleft Ġchild Ġis Ġeither Ġa Ġnoun , Ġadjective , Ġor Ġdetermin er , Ġthen Ġthe Ġinternal Ġnode Ġis Ġmarked Ġas Ġa Ġnoun , Ġotherwise Ġwe Ġmark Ġit Ġwith Ġa Ġgeneric Ġother Ġtag . ĠAt Ġthe Ġend Ġof Ġthe Ġprocedure , Ġunatt ached Ġnodes Ġ( words ) Ġare Ġattached Ġat Ġthe Ġhighest Ġpoint Ġin Ġthe Ġtree . </s> After Ġthe Ġtree Ġis Ġconstructed , Ġwe Ġcan Ġcompute Ġthe Ġoverall Ġphr as al Ġrepresentation Ġin Ġa Ġbottom - up Ġmanner , Ġguided Ġby Ġthe Ġlabels Ġof Ġleaf Ġand Ġinternal Ġnodes . ĠWe Ġnote Ġthat Ġthe Ġemphasis Ġof Ġthis Ġwork Ġis Ġnot Ġto Ġcompute Ġsentence - level Ġrepresentations . ĠThis Ġgoal Ġhas Ġbeen Ġexplored Ġin Ġrecent Ġresearch ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġand Ġcombining Ġour Ġmodels Ġwith Ġmethods Ġpresented Ġtherein Ġfor Ġsentence - level Ġrepresentations Ġis Ġstraightforward . </s> Learning </s> The Ġmodels Ġdescribed Ġabove Ġrely Ġon Ġparameters ĠIN LINE FORM 0 Ġthat Ġmust Ġbe Ġlearned . ĠIn Ġthis Ġsection , Ġwe Ġargue Ġthat Ġautomatically Ġconstructed Ġdatabases Ġof Ġparaph r ases Ġprovide Ġadequate Ġsupervision Ġfor Ġlearning Ġnotions Ġof Ġcomposition ality . </s> Super vision Ġfrom ĠAutomatic ĠPar aph r ases </s> The ĠPar aph rase ĠDatabase ĠB IB REF 9 Ġis Ġa Ġcollection Ġof Ġranked Ġmon oling ual Ġparaph r ases Ġthat Ġhave Ġbeen Ġextracted Ġfrom Ġword - aligned Ġparallel Ġcorpor a Ġusing Ġthe Ġbilingual Ġpivot Ġmethod ĠB IB REF 19 Ġ. ĠThe Ġunderlying Ġassumption Ġis Ġthat Ġif Ġtwo Ġstrings Ġin Ġthe Ġsame Ġlanguage Ġalign Ġto Ġthe Ġsame Ġstring Ġin Ġanother Ġlanguage , Ġthen Ġthe Ġstrings Ġin Ġthe Ġoriginal Ġlanguage Ġshare Ġthe Ġsame Ġmeaning . ĠPar aph r ases Ġare Ġranked Ġby Ġtheir Ġword Ġalignment Ġscores , Ġand Ġin Ġthis Ġwork Ġwe Ġuse Ġthe Ġpre selected Ġsmall Ġportion Ġof ĠP PD B Ġas Ġour Ġtraining Ġdata . ĠAlthough Ġwe Ġcan Ġdirectly Ġextract Ġphr as al Ġrepresentations Ġof Ġa Ġpre - specified Ġlist Ġof Ġphrases Ġfrom Ġthe Ġcorpus Ġused Ġto Ġcompute Ġword Ġrepresentations ĠB IB REF 6 Ġ, Ġthis Ġapproach Ġis Ġboth Ġcomput ationally Ġand Ġstatistically Ġinefficient : Ġthe Ġnumber Ġof Ġphrases Ġincreases Ġexponentially Ġin Ġthe Ġlength Ġof Ġthe Ġphrase , Ġand Ġcorrespond ingly Ġthe Ġoccurrence Ġof Ġany Ġindividual Ġphrase Ġdecreases Ġexponentially . ĠWe Ġcan Ġthus Ġcircumvent Ġthese Ġcomputational Ġand Ġstatistical Ġissues Ġby Ġusing Ġmon oling ual Ġparaph r ases . </s> The Ġtraining Ġdata Ġis Ġfiltered Ġto Ġprovide Ġonly Ġtwo - to - one Ġword Ġparaph rase Ġm appings , Ġand Ġthe Ġmulti word Ġportion Ġof Ġthe Ġparaph rase Ġis Ġsubsequently ĠPOS - tag ged . ĠTable ĠT AB REF 10 Ġprovides Ġa Ġbreakdown Ġof Ġsuch Ġparaph r ases Ġby Ġtheir ĠPOS Ġpair Ġtype . ĠGiven Ġthe Ġlack Ġof Ġcontext Ġwhen Ġtagging , Ġit Ġis Ġlikely Ġthat Ġthe ĠPOS Ġtag ger Ġyields Ġthe Ġmost Ġprobable Ġtag Ġfor Ġwords Ġand Ġnot Ġthe Ġmost Ġprobable Ġtag Ġgiven Ġthe Ġ( limited ) Ġcontext . ĠFurthermore , Ġeven Ġthe Ġhigher Ġquality Ġportions Ġof ĠP PD B Ġyield Ġparaph r ases Ġof Ġranging Ġquality , Ġranging Ġfrom Ġnon - t riv ial Ġm appings Ġsuch Ġas Ġyoung Ġpeople ĠIN LINE FORM 0 Ġyouth , Ġto Ġredundant Ġones Ġlike Ġthe Ġceasefire ĠIN LINE FORM 1 Ġceasefire . ĠHowever , ĠP PD B - like Ġresources Ġare Ġmore Ġeasily Ġavailable Ġthan Ġhuman - annot ated Ġresources Ġ( in Ġmultiple Ġlanguages Ġtoo : ĠGan it ke vich 2014 ), Ġso Ġit Ġis Ġimperative Ġthat Ġmethods Ġwhich Ġlearn Ġcompos itional Ġfunctions Ġfrom Ġsuch Ġsources Ġhandle Ġnoisy Ġsupervision Ġadequately . </s> Parameter ĠEst imation </s> The Ġparameters ĠIN LINE FORM 0 Ġin ĠE q . ĠEQ REF 4 Ġand ĠEQ REF 6 Ġcan Ġbe Ġestimated Ġthrough Ġstandard Ġlinear Ġregression Ġtechniques Ġin Ġconjunction Ġwith Ġthe Ġdata Ġpresented Ġin ĠÂ§ ĠSEC REF 3 Ġ. ĠThese Ġmethods Ġprovide Ġa Ġnatural Ġway Ġto Ġregular ize ĠIN LINE FORM 1 Ġvia ĠIN LINE FORM 2 Ġ( ridge ) Ġor ĠIN LINE FORM 3 Ġ( L ASS O ) Ġregular ization , Ġwhich Ġalso Ġhelps Ġhandle Ġnoisy Ġparaph r ases . ĠParameters Ġfor Ġthe ĠIN LINE FORM 4 Ġ- regular ized Ġconc aten ation Ġmodel Ġfor Ġselect ĠPOS Ġpairs Ġare Ġdisplayed Ġin ĠFig . ĠFIG REF 12 Ġ. ĠThe Ġheat - maps Ġdisplay Ġthe Ġrelative Ġmagnitude Ġof Ġparameters , Ġwith Ġpositive Ġvalues Ġcolored Ġblue , Ġnegative Ġvalues Ġcolored Ġred , Ġand Ġwhite Ġcells Ġindicating Ġzero Ġvalues . ĠIt Ġis Ġevident Ġthat Ġthe Ġparameters Ġlearned Ġfrom ĠP PD B Ġindicate Ġa Ġnotion Ġof Ġlinguistic Ġheaded ness , Ġnamely Ġthat Ġfor Ġparticular ĠPOS Ġpairs , Ġthe Ġsemantic Ġinformation Ġis Ġprimarily Ġcontained Ġin Ġthe Ġright Ġword , Ġbut Ġfor Ġothers Ġsuch Ġas Ġthe Ġnoun âĢĵ n oun Ġcombination , Ġeach Ġconstituent 's Ġcontribution Ġis Ġrelatively Ġmore Ġequal . </s> Me asuring Ġof ĠCom position ality </s> The Ġconc aten ation Ġand Ġtens or Ġmodels Ġcompute Ġan ĠIN LINE FORM 0 Ġ- dimensional Ġvector Ġrepresentation Ġfor Ġa Ġmulti - word Ġphrase Ġby Ġassuming Ġthe Ġmeaning Ġof Ġthe Ġphrase Ġcan Ġbe Ġexpressed Ġin Ġterms Ġof Ġthe Ġmeaning Ġof Ġits Ġconstituents . ĠThis Ġassumption Ġholds Ġtrue Ġto Ġvarying Ġdegrees ; Ġwhile Ġit Ġclearly Ġholds Ġfor ĠâĢ ľ large Ġamount " Ġand Ġbreaks Ġdown Ġfor ĠâĢ ľ cloud Ġnine ", Ġit Ġis Ġpartially Ġvalid Ġfor Ġphrases Ġsuch Ġas ĠâĢ ľ z ebra Ġcrossing " Ġor ĠâĢ ľ cr ash Ġcourse ". ĠIn Ġline Ġwith Ġprevious Ġwork , Ġwe Ġassume Ġa Ġcomposition ality Ġcontinuum ĠB IB REF 12 Ġ, Ġbut Ġfurther Ġconjecture Ġthat Ġa Ġphrase 's Ġlevel Ġof Ġcomposition ality Ġis Ġdependent Ġon Ġthe Ġspecific Ġcontext Ġin Ġwhich Ġit Ġoccurs , Ġmotivating Ġa Ġcontext - based Ġapproach Ġ( Â§ ĠSEC REF 21 Ġ) Ġwhich Ġscores Ġcomposition ality Ġby Ġcomputing Ġthe Ġlikelihood s Ġof Ġsurrounding Ġcontext Ġwords Ġgiven Ġa Ġphrase Ġrepresentation . ĠThe Ġeffect Ġof Ġcontext Ġis Ġdirectly Ġmeasured Ġthrough Ġa Ġcomparison Ġwith Ġcontext - independent Ġmethods Ġfrom Ġprior Ġwork ĠB IB REF 20 Ġ, ĠB IB REF 21 </s> It Ġis Ġimportant Ġto Ġnote Ġthat Ġmost Ġprior Ġwork Ġon Ġcomposition ality Ġscoring Ġassumes Ġaccess Ġto Ġboth Ġword Ġand Ġphrase Ġvector Ġrepresentations Ġ( for Ġselect Ġphrases Ġthat Ġwill Ġbe Ġevaluated ) Ġa Ġprior i . ĠThe Ġlatter Ġare Ġdistinct Ġfrom Ġrepresentations Ġthat Ġare Ġcomputed Ġfrom Ġlearned Ġcompos itional Ġfunctions Ġas Ġthey Ġare Ġextracted Ġdirectly Ġfrom Ġthe Ġcorpus , Ġwhich Ġis Ġan Ġexpensive Ġprocedure . ĠOur Ġaim Ġis Ġto Ġdevelop Ġcompos itional Ġmodels Ġthat Ġare Ġapplicable Ġin Ġdownstream Ġtasks , Ġand Ġthus Ġassuming Ġpre - existing Ġphrase Ġvectors Ġis Ġunreasonable . ĠHence Ġfor Ġphrases , Ġwe Ġonly Ġrely Ġon Ġrepresentations Ġcomputed Ġfrom Ġour Ġlearned Ġcompos itional Ġfunctions . </s> At Ġthe ĠType ĠLevel </s> Given Ġvector Ġrepresentations Ġfor Ġthe Ġconstituent Ġwords Ġin Ġa Ġphrase Ġand Ġthe Ġphrase Ġitself , Ġthe Ġidea Ġbehind Ġthe Ġtype - based Ġmodel Ġis Ġto Ġcompute Ġsimilarities Ġbetween Ġthe Ġconstituent Ġword Ġrepresentations Ġand Ġthe Ġphr as al Ġrepresentation , Ġand Ġaverage Ġthe Ġsimilarities Ġacross Ġthe Ġconstituents . ĠIf Ġthe Ġcontexts Ġin Ġwhich Ġa Ġconstituent Ġword Ġoccurs , Ġas Ġdictated Ġby Ġits Ġvector Ġrepresentation , Ġare Ġvery Ġdifferent Ġfrom Ġthe Ġcontexts Ġof Ġthe Ġcomposed Ġphrase , Ġas Ġindicated Ġby Ġthe Ġcos ine Ġsimilarity Ġbetween Ġthe Ġword Ġand Ġphrase Ġrepresentations , Ġthen Ġthe Ġphrase Ġis Ġlikely Ġto Ġbe Ġnon - com pos itional . ĠAssuming Ġunit - normal ized Ġword Ġvectors ĠIN LINE FORM 0 Ġand Ġphrase Ġvector ĠIN LINE FORM 1 Ġcomputed Ġfrom Ġone Ġof Ġthe Ġlearned Ġmodels Ġin ĠÂ§ ĠSEC REF 2 Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġhyper param eter Ġthat Ġcontrols Ġthe Ġcontribution Ġof Ġindividual Ġconstituents . ĠThis Ġmodel Ġlever ages Ġthe Ġaverage Ġstatistics Ġcomputed Ġover Ġthe Ġtraining Ġcorpor a Ġ( as Ġencaps ulated Ġin Ġthe Ġword Ġand Ġphrase Ġvectors ) Ġto Ġdetect Ġcomposition ality , Ġand Ġis Ġthe Ġprimary Ġway Ġcomposition ality Ġhas Ġbeen Ġevaluated Ġpreviously ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ. ĠNote Ġthat Ġfor Ġthe Ġsimple Ġadditive Ġmodel ĠIN LINE FORM 1 Ġwith Ġunit - normal ized Ġword Ġvectors , ĠIN LINE FORM 2 Ġis Ġindependent Ġof ĠIN LINE FORM 3 Ġ. </s> At Ġthe ĠToken ĠLevel </s> E q . ĠEQ REF 20 Ġscores Ġphrases Ġfor Ġcomposition ality Ġregardless Ġof Ġthe Ġcontext Ġthat Ġthese Ġphrases Ġoccur Ġin . ĠHowever , Ġphrases Ġsuch Ġas ĠâĢ ľ big Ġfish " Ġor ĠâĢ ľ heavy Ġmetal " Ġmay Ġoccur Ġin Ġboth Ġcompos itional Ġand Ġnon - com pos itional Ġsituations , Ġdepending Ġon Ġthe Ġnature Ġand Ġtopic Ġof Ġthe Ġtexts Ġthey Ġoccur Ġin . ĠHere , Ġwe Ġpropose Ġa Ġcontext - driven Ġmodel Ġfor Ġcomposition ality Ġdetection , Ġinspired Ġby Ġthe Ġskip - gram Ġmodel Ġfor Ġlearning Ġword Ġrepresentations ĠB IB REF 2 Ġ. ĠThe Ġintuition Ġis Ġsimple : Ġif Ġa Ġphrase Ġis Ġcompos itional , Ġit Ġshould Ġbe Ġsufficiently Ġpredictive Ġof Ġthe Ġcontext Ġwords Ġaround Ġit ; Ġotherwise , Ġit Ġis Ġacting Ġin Ġa Ġnon - com pos itional Ġmanner . ĠThus , Ġwe Ġwould Ġlike Ġto Ġcompute Ġthe Ġlikelihood Ġof Ġthe Ġcontext Ġ( ĠIN LINE FORM 0 Ġ) Ġgiven Ġa Ġphr as al Ġrepresentation Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġnormal ization Ġconstant ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> As Ġexplained Ġin ĠGoldberg 2014 , Ġthe Ġcontext Ġrepresentations Ġare Ġdistinct Ġfrom Ġthe Ġword Ġrepresentations . ĠIn Ġpractice , Ġwe Ġcompute Ġthe Ġlog - like lihood Ġaveraged Ġover Ġthe Ġcontext Ġwords Ġor Ġthe Ġperplex ity Ġinstead Ġof Ġthe Ġactual Ġlikelihood . </s> E val uation </s> Our Ġexperiments Ġhad Ġthree Ġaims : Ġfirst , Ġdemonstrate Ġthat Ġthe Ġcompos itional Ġfunctions Ġlearned Ġusing Ġparaph rase Ġsupervision Ġcompute Ġsem antically Ġmeaningful Ġresults Ġfor Ġcompos itional Ġphrases Ġby Ġevaluating Ġon Ġa Ġphrase Ġsimilarity Ġtask Ġ( Â§ ĠSEC REF 29 Ġ); Ġsecond , Ġverify Ġthe Ġhypothesis Ġthat Ġcomposition ality Ġis Ġcontext - dependent Ġby Ġcomparing Ġa Ġtype - based Ġand Ġtoken - based Ġapproach Ġon Ġa Ġcompound Ġnoun Ġevaluation Ġtask Ġ( Â§ ĠSEC REF 36 Ġ); Ġand Ġthird , Ġdetermine Ġif Ġthe Ġcomposition ality - scoring Ġmodels Ġbased Ġon Ġlearned Ġrepresentations Ġimprove Ġthe Ġtranslations Ġproduced Ġby Ġa Ġstate - of - the - art Ġphrase - based ĠMT Ġsystem Ġ( Â§ ĠSEC REF 38 Ġ). </s> The Ġword Ġvectors Ġused Ġin Ġall Ġof Ġour Ġexperiments Ġwere Ġproduced Ġby Ġword 2 vec Ġusing Ġthe Ġskip - gram Ġmodel Ġwith Ġ20 Ġnegative Ġsamples , Ġa Ġcontext Ġwindow Ġsize Ġof Ġ10 , Ġa Ġminimum Ġtoken Ġcount Ġof Ġ3 , Ġand Ġsub - sam pling Ġof Ġfrequent Ġwords Ġwith Ġa Ġparameter Ġof ĠIN LINE FORM 0 Ġ. ĠWe Ġextracted Ġcorpus Ġstatistics Ġfor Ġword 2 vec Ġusing Ġthe ĠAFP Ġportion Ġof Ġthe ĠEnglish ĠGig aw ord , Ġwhich Ġconsists Ġof Ġ8 87 . 5 Ġmillion Ġtokens . ĠThe Ġcode Ġused Ġto Ġgenerate Ġthe Ġresults Ġis Ġavailable Ġat Ġhttp :// www . github . com / xy z , Ġand Ġthe Ġevaluation Ġdatasets Ġare Ġpublicly Ġavailable . </s> Ph ras al ĠSimilar ity </s> For Ġthe Ġphrase Ġsimilarity Ġtask Ġwe Ġfirst Ġcompare Ġour Ġconc aten ation Ġand Ġtens or Ġmodels Ġlearned Ġusing ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġregular ization Ġto Ġthree Ġbas elines : </s> [ no item se p ] </s> add : ĠIN LINE FORM 0 </s> mult 1 : ĠIN LINE FORM 0 </s> mult 2 : ĠIN LINE FORM 0 </s> Other Ġadditive Ġmodels Ġfrom Ġprevious Ġwork ĠB IB REF 5 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġthat Ġimpose Ġvarying Ġamounts Ġof Ġstructural Ġassumptions Ġon Ġthe Ġsemantic Ġinteractions Ġbetween Ġword Ġrepresentations Ġe . g ., ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġare Ġsubs umed Ġby Ġour Ġconc aten ation Ġmodel . ĠThe Ġregular ization Ġstrength Ġhyper param eter Ġfor ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġregular ization Ġwas Ġselected Ġusing Ġ5 - fold Ġcross - valid ation Ġon Ġthe ĠP PD B Ġtraining Ġdata . </s> We Ġevaluated Ġthe Ġphrase Ġcomposition ality Ġmodels Ġon Ġthe Ġadjective âĢĵ n oun Ġand Ġnoun âĢĵ n oun Ġphrase Ġsimilarity Ġtasks Ġcompiled Ġby ĠMitchell 2010 , Ġusing Ġthe Ġsame Ġevaluation Ġscheme Ġas Ġin Ġthe Ġoriginal Ġwork . ĠSpear man 's ĠIN LINE FORM 0 Ġbetween Ġphr as al Ġsimilarities Ġderived Ġfrom Ġour Ġcompos itional Ġfunctions Ġand Ġthe Ġhuman Ġannot ators Ġ( com puted Ġindividually Ġper Ġannot ator Ġand Ġthen Ġaveraged Ġacross Ġall Ġannot ators ) Ġwas Ġthe Ġevaluation Ġmeasure . </s> Figure ĠFIG REF 24 Ġpresents Ġthe Ġcorrelation Ġresults Ġfor Ġthe Ġtwo ĠPOS Ġpair Ġtypes Ġas Ġa Ġfunction Ġof Ġthe Ġdimension ality ĠIN LINE FORM 0 Ġof Ġthe Ġrepresentations Ġfor Ġthe Ġconc aten ation Ġmodels Ġ( and Ġadditive Ġbaseline ) Ġand Ġtens or Ġmodels Ġ( and Ġmultipl icative Ġbas elines ). ĠThe Ġconc aten ation Ġmodels Ġseem Ġmore Ġeffective Ġthan Ġthe Ġtens or Ġmodels Ġin Ġthe Ġadjective âĢĵ n oun Ġcase Ġand Ġgive Ġroughly Ġthe Ġsame Ġperformance Ġon Ġthe Ġnoun âĢĵ n oun Ġdataset , Ġwhich Ġis Ġconsistent Ġwith Ġprevious Ġwork Ġthat Ġuses Ġdense , Ġlow - dimensional Ġrepresentations ĠB IB REF 25 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 26 Ġ. ĠSince Ġthe Ġconc aten ation Ġmodel Ġinvolve Ġfewer Ġparameters , Ġwe Ġuse Ġit Ġas Ġthe Ġcompos itional Ġmodel Ġof Ġchoice Ġfor Ġsubsequent Ġexperiments . ĠThe Ġabsolute Ġresults Ġare Ġalso Ġconsistent Ġwith Ġstate - of - the - art Ġresults Ġon Ġthis Ġdataset ĠB IB REF 24 Ġ, ĠB IB REF 26 Ġ, Ġindicating Ġthat Ġparaph r ases Ġare Ġan Ġexcellent Ġsource Ġof Ġinformation Ġfor Ġlearning Ġcompos itional Ġfunctions Ġand Ġa Ġreasonable Ġalternative Ġto Ġhuman - annot ated Ġtraining Ġsets . ĠFor Ġreference , Ġthe Ġinter - annot ator Ġagreements Ġare Ġ0 . 52 Ġfor Ġthe Ġadjective âĢĵ n oun Ġevaluation Ġand Ġ0 . 51 Ġfor Ġthe Ġnoun âĢĵ n oun Ġone . ĠThe Ġun weight ed Ġadditive Ġbaseline Ġis Ġsurprisingly Ġvery Ġstrong Ġon Ġthe Ġnoun âĢĵ n oun Ġset , Ġso Ġwe Ġalso Ġcompare Ġagainst Ġit Ġin Ġsubsequent Ġexperiments . </s> Com position ality </s> To Ġevaluate Ġthe Ġcomposition ality - scoring Ġmodels , Ġwe Ġused Ġthe Ġcompound Ġnoun Ġcomposition ality Ġdataset Ġintroduced Ġin ĠRed dy 2011 . ĠThis Ġdataset Ġconsists Ġof Ġ26 70 Ġannotations Ġof Ġ90 Ġcompound - n oun Ġphrases Ġexhibiting Ġvarying Ġlevels Ġcomposition ality , Ġwith Ġscores Ġranging Ġfrom Ġ0 Ġto Ġ5 Ġprovided Ġby Ġ30 Ġannot ators . ĠIt Ġalso Ġcontains Ġthree Ġto Ġfive Ġexample Ġsentences Ġof Ġthese Ġphrases Ġthat Ġwere Ġshown Ġto Ġthe Ġannot ators , Ġwhich Ġwe Ġmake Ġuse Ġof Ġin Ġour Ġcontext - dependent Ġmodel . ĠCons istent Ġwith Ġthe Ġoriginal Ġwork , ĠSpear man 's ĠIN LINE FORM 0 Ġis Ġcomputed Ġon Ġthe Ġaveraged Ġcomposition ality Ġscore Ġfor Ġa Ġphrase Ġacross Ġall Ġthe Ġannot ators Ġthat Ġscored Ġthat Ġphrase Ġ( which Ġvaries Ġper Ġphrase ). ĠFor Ġcomputing Ġthe Ġcompos itional Ġfunctions , Ġwe Ġevaluate Ġthree Ġof Ġthe Ġbest Ġperforming Ġsetups Ġfrom ĠÂ§ ĠSEC REF 29 Ġ: Ġthe ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ- regular ized Ġconc aten ation Ġmodels , Ġand Ġthe Ġsimple Ġadditive Ġbaseline . </s> For Ġthe Ġcontext - independent Ġmodel , Ġwe Ġselect Ġthe Ġhyper param eter ĠIN LINE FORM 0 Ġin ĠE q . ĠEQ REF 20 Ġfrom Ġthe Ġvalues ĠIN LINE FORM 1 Ġ. ĠFor Ġthe Ġcontext - dependent Ġmodel , Ġwe Ġvary Ġthe Ġcontext Ġwindow Ġsize ĠIN LINE FORM 2 Ġby Ġselecting Ġfrom Ġthe Ġvalues ĠIN LINE FORM 3 Ġ. ĠTable ĠT AB REF 37 Ġpresents ĠSpear man 's ĠIN LINE FORM 4 Ġfor Ġthese Ġsetups . ĠIn Ġall Ġcases , Ġthe Ġcontext - dependent Ġmodels Ġoutper form Ġthe Ġcontext - independent Ġones , Ġand Ġusing Ġa Ġrelatively Ġsimple Ġtoken - based Ġmodel Ġwe Ġcan Ġapproximately Ġmatch Ġthe Ġperformance Ġof Ġthe ĠBay esian Ġmodel Ġproposed Ġby ĠHerman n 2012 . ĠThe Ġconc aten ation Ġmodels Ġare Ġalso Ġconsistently Ġbetter Ġthan Ġthe Ġadditive Ġcompos itional Ġmodel , Ġindicating Ġthe Ġbenefit Ġof Ġlearning Ġthe Ġcompos itional Ġparameters Ġvia ĠP PD B . </s> Machine ĠTranslation </s> While Ġany Ġtruly Ġsuccessful Ġmodel Ġof Ġsemantics Ġmust Ġmatch Ġhuman Ġintu itions , Ġunderstanding Ġthe Ġapplications Ġof Ġour Ġmodels Ġis Ġlikewise Ġimportant . ĠTo Ġthis Ġend , Ġwe Ġconsider Ġthe Ġproblem Ġof Ġmachine Ġtranslation , Ġoperating Ġunder Ġthe Ġhypothesis Ġthat Ġsentences Ġwhich Ġexpress Ġtheir Ġmeaning Ġnon - com pos itionally Ġshould Ġalso Ġtranslate Ġnon - com pos itionally . </s> Modern Ġphrase - based Ġtranslation Ġsystems Ġare Ġfaced Ġwith Ġa Ġlarge Ġnumber Ġof Ġpossible Ġsegment ations Ġof Ġa Ġsource - language Ġsentence Ġduring Ġdecoding , Ġand Ġall Ġsegment ations Ġare Ġconsidered Ġequally Ġlikely ĠB IB REF 13 Ġ. ĠThus , Ġit Ġwould Ġbe Ġhelpful Ġto Ġprovide Ġguidance Ġon Ġmore Ġlikely Ġsegment ations , Ġas Ġdictated Ġby Ġthe Ġcomposition ality Ġscores Ġof Ġthe Ġphrases Ġextracted Ġfrom Ġa Ġsentence , Ġto Ġthe Ġdec oder . ĠA Ġlow Ġcomposition ality Ġscore Ġwould Ġideally Ġforce Ġthe Ġdec oder Ġto Ġconsider Ġthe Ġentire Ġphrase Ġas Ġa Ġtranslation Ġunit , Ġdue Ġto Ġits Ġunique Ġsemantic Ġcharacteristics . ĠCorrespond ingly , Ġa Ġhigh Ġscore Ġinforms Ġthe Ġdec oder Ġthat Ġit Ġis Ġsafe Ġto Ġrely Ġon Ġword - level Ġtranslations Ġof Ġthe Ġphr as al Ġconstituents . ĠThus , Ġif Ġwe Ġreveal Ġto Ġthe Ġtranslation Ġsystem Ġthat Ġa Ġphrase Ġis Ġnon - com pos itional , Ġit Ġshould Ġbe Ġable Ġto Ġlearn Ġthat Ġtranslation Ġdecisions Ġwhich Ġtranslate Ġit Ġas Ġa Ġunit Ġare Ġto Ġbe Ġfavored , Ġleading Ġto Ġbetter Ġtranslations . </s> To Ġtest Ġthis Ġhypothesis , Ġwe Ġbuilt Ġan ĠEnglish - Spanish ĠMT Ġsystem Ġusing Ġthe Ġc dec Ġdec oder ĠB IB REF 27 Ġfor Ġthe Ġentire Ġtraining Ġpipeline Ġ( word Ġalign ments , Ġphrase Ġextraction , Ġfeature Ġweight Ġtuning , Ġand Ġdecoding ). ĠCorpor a Ġfrom Ġthe ĠW MT Ġ2011 Ġevaluation Ġwas Ġused Ġto Ġbuild Ġthe Ġtranslation Ġand Ġlanguage Ġmodels , Ġand Ġfor Ġtuning Ġ( on Ġnews - test 2010 ) Ġand Ġevaluation Ġ( on Ġnews - test 2011 ), Ġwith Ġscoring Ġdone Ġusing ĠB LE U ĠB IB REF 28 Ġ. ĠThe Ġbaseline Ġis Ġa Ġhierarchical Ġphrase - based Ġsystem ĠB IB REF 29 Ġwith Ġa Ġ4 - gram Ġlanguage Ġmodel , Ġwith Ġfeature Ġweights Ġtuned Ġusing ĠMI RA ĠB IB REF 30 Ġ. ĠFor Ġfeatures , Ġeach Ġtranslation Ġrule Ġis Ġdecorated Ġwith Ġtwo Ġlex ical Ġand Ġphr as al Ġfeatures Ġcorresponding Ġto Ġthe Ġforward ĠIN LINE FORM 0 Ġand Ġbackward ĠIN LINE FORM 1 Ġconditional Ġlog Ġfrequencies , Ġalong Ġwith Ġthe Ġlog Ġjoint Ġfrequency ĠIN LINE FORM 2 Ġ, Ġthe Ġlog Ġfrequency Ġof Ġthe Ġsource Ġphrase ĠIN LINE FORM 3 Ġ, Ġand Ġwhether Ġthe Ġphrase Ġpair Ġor Ġthe Ġsource Ġphrase Ġis Ġa Ġsingle ton . ĠWe ights Ġfor Ġthe Ġlanguage Ġmodel , Ġglue Ġrule , Ġand Ġword Ġpenalty Ġare Ġalso Ġtuned . ĠThis Ġsetup Ġ( Bas eline ) Ġachieves Ġscores Ġen Ġpar Ġwith Ġthe Ġpublished ĠW MT Ġresults . </s> We Ġadded Ġthe Ġcomposition ality Ġscore Ġas Ġan Ġadditional Ġfeature , Ġand Ġalso Ġadded Ġtwo Ġbinary - valued Ġfeatures : Ġthe Ġfirst Ġindicates Ġif Ġthe Ġgiven Ġtranslation Ġrule Ġhas Ġnot Ġbeen Ġdecorated Ġwith Ġa Ġcomposition ality Ġscore Ġ( either Ġbecause Ġit Ġconsists Ġof Ġnon - termin als Ġonly Ġor Ġthe Ġlex ical Ġitems Ġin Ġthe Ġtranslation Ġrule Ġare Ġun ig rams ), Ġand Ġcorrespond ingly Ġthe Ġsecond Ġfeature Ġindicates Ġif Ġthe Ġtranslation Ġrule Ġhas Ġbeen Ġscored . ĠTherefore , Ġan Ġappropriate Ġadditional Ġbaseline Ġwould Ġbe Ġto Ġmark Ġtranslation Ġrules Ġwith Ġthese Ġindicator Ġfunctions Ġbut Ġwithout Ġthe Ġscores , Ġakin Ġto Ġidentifying Ġrules Ġwith Ġphrases Ġin Ġthem Ġ( Bas eline Ġ+ ĠSeg On ). </s> Table ĠT AB REF 40 Ġpresents Ġthe Ġresults Ġof Ġthe ĠMT Ġevaluation , Ġcomparing Ġthe Ġbas elines Ġto Ġthe Ġbest - performing Ġcontext - independent Ġand Ġdependent Ġscoring Ġmodels Ġfrom ĠÂ§ ĠSEC REF 36 Ġ. ĠThe Ġscores Ġhave Ġbeen Ġaveraged Ġover Ġthree Ġtuning Ġruns Ġwith Ġstandard Ġdeviation Ġin Ġparentheses ; Ġbold Ġresults Ġon Ġthe Ġtest Ġset Ġare Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 0 Ġ) Ġwith Ġrespect Ġto Ġthe Ġbaseline . ĠWhile Ġknowledge Ġof Ġrelative Ġcomposition ality Ġconsistently Ġhelps , Ġthe Ġimprovements Ġusing Ġthe Ġcontext - dependent Ġscoring Ġmodels , Ġespecially Ġwith Ġthe ĠIN LINE FORM 1 Ġconc aten ation Ġmodel , Ġare Ġnoticeably Ġbetter . </s> Related ĠWork </s> There Ġhas Ġbeen Ġa Ġlarge Ġamount Ġof Ġwork Ġon Ġcompos itional Ġmodels Ġthat Ġoperate Ġon Ġvector Ġrepresentations Ġof Ġwords . ĠWith Ġsome Ġexceptions ĠB IB REF 16 Ġ, ĠB IB REF 5 Ġ, Ġall Ġof Ġthese Ġapproaches Ġare Ġlex ical ized Ġi . e ., Ġparameters Ġ( gener ally Ġin Ġthe Ġform Ġof Ġvectors , Ġmat rices , Ġor Ġtens ors ) Ġfor Ġspecific Ġwords Ġare Ġlearned , Ġwhich Ġworks Ġwell Ġfor Ġfrequently Ġoccurring Ġwords Ġbut Ġfails Ġwhen Ġdealing Ġwith Ġcompositions Ġof Ġarbitrary Ġword Ġsequences Ġcontaining Ġinf requent Ġwords . ĠThe Ġfunctions Ġare Ġeither Ġlearned Ġwith Ġa Ġneural Ġnetwork Ġarchitecture ĠB IB REF 7 Ġor Ġas Ġa Ġlinear Ġregression ĠB IB REF 6 Ġ; Ġthe Ġlatter Ġrequire Ġphrase Ġrepresentations Ġextracted Ġdirectly Ġfrom Ġthe Ġcorpus Ġfor Ġsupervision , Ġwhich Ġcan Ġbe Ġcomput ationally Ġexpensive Ġand Ġstatistically Ġinefficient . ĠIn Ġcontrast , Ġwe Ġobtain Ġthis Ġinformation Ġthrough Ġmany - to - one ĠP PD B Ġm appings . ĠMost Ġof Ġthese Ġmodels Ġalso Ġrequire Ġadditional Ġsynt actic ĠB IB REF 31 Ġor Ġsemantic ĠB IB REF 15 Ġ, ĠB IB REF 14 Ġresources ; Ġon Ġthe Ġother Ġhand , Ġour Ġproposed Ġapproach Ġonly Ġrequires Ġa Ġshallow Ġsynt actic Ġparse Ġ( POS Ġtags ). ĠRecent Ġefforts Ġto Ġmake Ġthese Ġmodels Ġmore Ġpractical ĠB IB REF 32 Ġattempt Ġto Ġreduce Ġtheir Ġstatistically Ġcomplex Ġand Ġoverly - param et ri zed Ġnature , Ġbut Ġwith Ġthe Ġexception Ġof ĠZ anz otto 2010 , Ġwho Ġpropose Ġa Ġway Ġto Ġextract Ġcompos itional Ġfunction Ġtraining Ġexamples Ġfrom Ġa Ġdictionary , Ġthese Ġmodels Ġgenerally Ġrequire Ġhuman - annot ated Ġdata Ġto Ġwork . </s> Most Ġmodels Ġthat Ġscore Ġthe Ġrelative Ġ( non -) Ġcomposition ality Ġof Ġphrases Ġdo Ġso Ġin Ġa Ġcontext - independent Ġmanner . ĠA Ġcentral Ġidea Ġis Ġto Ġreplace Ġphrase Ġconstituents Ġwith Ġsem antically - related Ġwords Ġand Ġcompute Ġthe Ġsimilarity Ġof Ġthe Ġnew Ġphrase Ġto Ġthe Ġoriginal ĠB IB REF 22 Ġ, ĠB IB REF 33 Ġor Ġmake Ġuse Ġof Ġa Ġvariety Ġof Ġlex ical Ġassociation Ġmeasures ĠB IB REF 10 Ġ, ĠB IB REF 34 Ġ. ĠSp or led er 2009 Ġhowever , Ġdo Ġmake Ġuse Ġof Ġcontext Ġin Ġa Ġtoken - based Ġapproach , Ġwhere Ġthe Ġcontext Ġin Ġwhich Ġa Ġphrase Ġoccurs Ġas Ġwell Ġas Ġthe Ġphrase Ġitself Ġis Ġmodeled Ġas Ġa Ġlex ical Ġchain , Ġand Ġthe Ġcohesion Ġof Ġthe Ġchain Ġis Ġmeasured Ġas Ġan Ġindicator Ġof Ġa Ġphrase 's Ġcomposition ality . ĠCo hesion Ġis Ġcomputed Ġusing Ġa Ġweb Ġsearch Ġengine - based Ġmeasure , Ġwhereas Ġwe Ġuse Ġa Ġprob abil istic Ġmodel Ġof Ġcontext Ġgiven Ġa Ġphrase Ġrepresentation . ĠHerman n 2012 Ġpropose Ġa ĠBay esian Ġgener ative Ġmodel Ġthat Ġis Ġalso Ġcontext - based , Ġbut Ġlearning Ġand Ġinference Ġis Ġdone Ġthrough Ġa Ġrelatively Ġexpensive ĠGibbs Ġsampling Ġscheme . </s> In Ġthe Ġcontext Ġof ĠMT , ĠZhang 2008 b Ġpresent Ġa ĠBay esian Ġmodel Ġthat Ġlearns Ġnon - com pos itional Ġphrases Ġfrom Ġa Ġsynchron ous Ġparse Ġtree Ġof Ġa Ġsentence Ġpair . ĠHowever , Ġthe Ġprimary Ġaim Ġof Ġtheir Ġwork Ġis Ġphrase Ġextraction Ġfor ĠMT , Ġand Ġthe Ġnon - com pos itional Ġconstraints Ġare Ġonly Ġapplied Ġto Ġmake Ġthe Ġspace Ġof Ġphrase Ġpairs Ġmore Ġtract able Ġwhen Ġboot stra pping Ġtheir Ġphr as al Ġparser Ġfrom Ġtheir Ġword - based Ġparser . ĠIn Ġcontrast , Ġwe Ġscore Ġevery Ġphrase Ġthat Ġis Ġextracted Ġwith Ġthe Ġstandard Ġphrase Ġextraction Ġhe ur istics ĠB IB REF 29 Ġ, Ġallowing Ġthe Ġdec oder Ġto Ġmake Ġthe Ġfinal Ġdecision Ġon Ġthe Ġimpact Ġof Ġcomposition ality Ġscores Ġin Ġtranslation . ĠThus , Ġour Ġwork Ġis Ġmore Ġsimilar Ġto ĠX ion g 2010 , Ġwho Ġpropose Ġmaximum Ġentropy Ġclass ifiers Ġthat Ġmark Ġpositions Ġbetween Ġwords Ġin Ġa Ġsentence Ġas Ġbeing Ġa Ġphrase Ġboundary Ġor Ġnot , Ġand Ġintegrate Ġthese Ġscores Ġas Ġadditional Ġfeatures Ġin Ġan ĠMT Ġsystem . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġpresented Ġtwo Ġnew Ġsources Ġof Ġinformation Ġfor Ġcomposition ality Ġmodeling Ġand Ġscoring , Ġparaph rase Ġinformation Ġand Ġcontext . ĠFor Ġmodeling , Ġwe Ġshowed Ġthat Ġthe Ġparaph rase - learn ed Ġcompos itional Ġrepresentations Ġperforms Ġas Ġwell Ġon Ġa Ġphrase Ġsimilarity Ġtask Ġas Ġthe Ġaverage Ġhuman Ġannot ator . ĠFor Ġscoring , Ġthe Ġimportance Ġof Ġcontext Ġwas Ġshown Ġthrough Ġthe Ġcomparison Ġof Ġcontext - independent Ġand Ġdependent Ġmodels . ĠImprovements Ġby Ġthe Ġcontext - dependent Ġmodel Ġon Ġan Ġextr ins ic Ġmachine Ġtranslation Ġtask Ġcorrobor ate Ġthe Ġutility Ġof Ġthese Ġadditional Ġknowledge Ġsources . ĠWe Ġhope Ġthat Ġthis Ġwork Ġencourages Ġfurther Ġresearch Ġin Ġmaking Ġcompos itional Ġsemantic Ġapproaches Ġapplicable Ġin Ġdownstream Ġtasks .
Introduction Numerous lexical semantic properties are captured by representations encoding distributional properties of words, as has been demonstrated in a variety of tasks BIBREF0 , BIBREF1 , BIBREF2 . However, this distributional account of meaning does not scale to larger units like phrases and sentences BIBREF3 , BIBREF4 , motivating research into compositional models that combine word representations to produce representations of the semantics of longer units BIBREF5 , BIBREF6 , BIBREF7 . Previous work has learned these models using autoencoder formulations BIBREF8 or limited human supervision BIBREF5 . In this work, we explore the hypothesis that the equivalent knowledge about how words compose can be obtained through monolingual paraphrases that have been extracted using word alignments and an intermediate language BIBREF9 . Confirming this hypothesis would allow the rapid development of compositional models in a large number of languages. As their name suggests, these models also impose the assumption that longer units like phrases are compositional, i.e., a phrase's meaning can be understood from the literal meaning of its parts. However, countless examples that run contrary to the assumption exist, and handling these non-compositional phrases has been problematic and of long-standing interest in the community BIBREF10 , BIBREF11 . (Non-) Compositionality detection can provide vital information to other language processing systems on whether a multiword unit should be treated semantically as a single entity or not, and scoring this phenomenon is particularly relevant for downstream tasks like machine translation (MT) or information retrieval. We explore the hypothesis that contextual evidence can be used to determine the relative degree to which a phrase is meant compositionally. Rather than focusing purely on intrinsic clean-room evaluations, the goal of this work is to learn relatively accurate context-sensitive compositional models that are also directly applicable in real-world, noisy-data scenarios. This objective necessitates certain design decisions, and to this end we propose a robust, scalable framework that learns compositional functions and scores relative phrasal compositionality. We make three contributions: first, a novel way to learn compositional functions for part-of-speech pairs that uses supervision from an automatically-extracted list of paraphrases (§ SECREF3 ). Second, a context-dependent scoring model that scores the relative compositionality of a phrase BIBREF12 by computing the likelihood of its context given its paraphrase-learned representation (§ SECREF4 ). And third, an evaluation of the impact of compositionality knowledge in an end-to-end MT setup. Our experiments (§ SECREF5 ) reveal that using supervision from automatically extracted paraphrases produces compositional functions with equivalent performance to previous approaches that have relied on hand-annotated training data. Furthermore, compositionality features consistently improve the translations produced by a strong English–Spanish translation system. Parametric Composition Functions We formalize composition as a function INLINEFORM0 that maps INLINEFORM1 -dimensional vector representations of phrase constituents INLINEFORM2 to an INLINEFORM3 -dimensional vector representation of the phrase, i.e., the composed representation. A phrase is defined as any contiguous sequence of words of length 2 or greater, and does not have to adhere to constituents in a phrase structure grammar. This definition is in line with our MT application and ignores “gappy” noncontiguous phrases, but this pragmatic choice does exclude many verb-object relations BIBREF13 . We assume the existence of word-level vector representations for every word in our vocabulary of size INLINEFORM4 . Compositionality is modeled as a bilinear map, and two classes of linear models with different levels of parametrization are proposed. Unlike previous work BIBREF6 , BIBREF7 , BIBREF14 where the functions are word-specific, our compositional functions operate on part-of-speech (POS) tag pairs, which facilitates learning by drastically reducing the number of parameters, and only requires a shallow syntactic parse of the input. Concatenation Models Our first class of models is a generalization of the additive models introduced in Mitchell2008: DISPLAYFORM0 where the notation INLINEFORM0 represents a vertical (row-wise) concatenation of two vectors; namely, the concatenation that results in a INLINEFORM1 -sized vector. In addition to the INLINEFORM2 parameters for the word vector representations that are provided a priori, this model introduces INLINEFORM3 parameters, where INLINEFORM4 is the number of POS-tag pairs we consider. Mitchell2008 significantly simplify parameter estimation by assuming a certain structure for the parameter matrix INLINEFORM0 , which is necessary given the limited human-annotated data they use. For example, by assuming a block-diagonal structure, we get a scaled element-wise addition model INLINEFORM1 . While not strictly in this category due to the non-linearities involved, neural network-based compositional models BIBREF7 , BIBREF15 can be viewed as concatenation models, although the order of concatenation and matrix multiplication is switched. However, these models introduce more than INLINEFORM2 parameters. Tensor Models The second class of models leverages pairwise multiplicative interactions between the components of the two word vectors: DISPLAYFORM0 where INLINEFORM0 corresponds to a tensor contraction along the INLINEFORM1 mode of the tensor INLINEFORM2 . In this case, we first compute a contraction (tensor-vector product) between INLINEFORM3 and INLINEFORM4 along INLINEFORM5 's third mode, corresponding to interactions with the second word vector of a two-word phrase and resulting in a matrix, which is then multiplied along its second mode (corresponding to traditional matrix multiplication on the right) by INLINEFORM6 . The final result is an INLINEFORM7 vector. This model introduces INLINEFORM8 parameters. Tensor models are a generalization of the element-wise multiplicative model BIBREF16 , which permits non-zero values only on the tensor diagonal. Operating at the vocabulary level, the model of Baroni2010 has interesting parallels to our tensor model. They focus on adjective–noun relationships and learn a specific matrix for every adjective in their dataset; in our case, the specific matrix for each adjective has a particular form, namely that it can be factorized into the product of a tensor and a vector; the tensor corresponds to the actual adjective–noun combiner function, and the vector corresponds to specific lexical information that the adjective carries. This concept generalizes to other POS pairs: for example, multiplying the tensor that represents determiner-noun combinations along the second mode with the vector for “the” results in a matrix that represents the semantic operation of definiteness. Learning these parameters jointly is statistically more efficient than separately learning versions for each word. Longer Phrases The proposed models operate on pairs of words at a time. To handle phrases of length greater than two, we greedily construct a left-branching tree of the phrase constituents that eventually dictates the application of the learned bilinear maps. For each internal tree node, we consider the POS tags of its children: if the right child is a noun, and the left child is either a noun, adjective, or determiner, then the internal node is marked as a noun, otherwise we mark it with a generic other tag. At the end of the procedure, unattached nodes (words) are attached at the highest point in the tree. After the tree is constructed, we can compute the overall phrasal representation in a bottom-up manner, guided by the labels of leaf and internal nodes. We note that the emphasis of this work is not to compute sentence-level representations. This goal has been explored in recent research BIBREF17 , BIBREF18 , and combining our models with methods presented therein for sentence-level representations is straightforward. Learning The models described above rely on parameters INLINEFORM0 that must be learned. In this section, we argue that automatically constructed databases of paraphrases provide adequate supervision for learning notions of compositionality. Supervision from Automatic Paraphrases The Paraphrase Database BIBREF9 is a collection of ranked monolingual paraphrases that have been extracted from word-aligned parallel corpora using the bilingual pivot method BIBREF19 . The underlying assumption is that if two strings in the same language align to the same string in another language, then the strings in the original language share the same meaning. Paraphrases are ranked by their word alignment scores, and in this work we use the preselected small portion of PPDB as our training data. Although we can directly extract phrasal representations of a pre-specified list of phrases from the corpus used to compute word representations BIBREF6 , this approach is both computationally and statistically inefficient: the number of phrases increases exponentially in the length of the phrase, and correspondingly the occurrence of any individual phrase decreases exponentially. We can thus circumvent these computational and statistical issues by using monolingual paraphrases. The training data is filtered to provide only two-to-one word paraphrase mappings, and the multiword portion of the paraphrase is subsequently POS-tagged. Table TABREF10 provides a breakdown of such paraphrases by their POS pair type. Given the lack of context when tagging, it is likely that the POS tagger yields the most probable tag for words and not the most probable tag given the (limited) context. Furthermore, even the higher quality portions of PPDB yield paraphrases of ranging quality, ranging from non-trivial mappings such as young people INLINEFORM0 youth, to redundant ones like the ceasefire INLINEFORM1 ceasefire. However, PPDB-like resources are more easily available than human-annotated resources (in multiple languages too: Ganitkevich2014), so it is imperative that methods which learn compositional functions from such sources handle noisy supervision adequately. Parameter Estimation The parameters INLINEFORM0 in Eq. EQREF4 and EQREF6 can be estimated through standard linear regression techniques in conjunction with the data presented in § SECREF3 . These methods provide a natural way to regularize INLINEFORM1 via INLINEFORM2 (ridge) or INLINEFORM3 (LASSO) regularization, which also helps handle noisy paraphrases. Parameters for the INLINEFORM4 -regularized concatenation model for select POS pairs are displayed in Fig. FIGREF12 . The heat-maps display the relative magnitude of parameters, with positive values colored blue, negative values colored red, and white cells indicating zero values. It is evident that the parameters learned from PPDB indicate a notion of linguistic headedness, namely that for particular POS pairs, the semantic information is primarily contained in the right word, but for others such as the noun–noun combination, each constituent's contribution is relatively more equal. Measuring of Compositionality The concatenation and tensor models compute an INLINEFORM0 -dimensional vector representation for a multi-word phrase by assuming the meaning of the phrase can be expressed in terms of the meaning of its constituents. This assumption holds true to varying degrees; while it clearly holds for “large amount" and breaks down for “cloud nine", it is partially valid for phrases such as “zebra crossing" or “crash course". In line with previous work, we assume a compositionality continuum BIBREF12 , but further conjecture that a phrase's level of compositionality is dependent on the specific context in which it occurs, motivating a context-based approach (§ SECREF21 ) which scores compositionality by computing the likelihoods of surrounding context words given a phrase representation. The effect of context is directly measured through a comparison with context-independent methods from prior work BIBREF20 , BIBREF21 It is important to note that most prior work on compositionality scoring assumes access to both word and phrase vector representations (for select phrases that will be evaluated) a priori. The latter are distinct from representations that are computed from learned compositional functions as they are extracted directly from the corpus, which is an expensive procedure. Our aim is to develop compositional models that are applicable in downstream tasks, and thus assuming pre-existing phrase vectors is unreasonable. Hence for phrases, we only rely on representations computed from our learned compositional functions. At the Type Level Given vector representations for the constituent words in a phrase and the phrase itself, the idea behind the type-based model is to compute similarities between the constituent word representations and the phrasal representation, and average the similarities across the constituents. If the contexts in which a constituent word occurs, as dictated by its vector representation, are very different from the contexts of the composed phrase, as indicated by the cosine similarity between the word and phrase representations, then the phrase is likely to be non-compositional. Assuming unit-normalized word vectors INLINEFORM0 and phrase vector INLINEFORM1 computed from one of the learned models in § SECREF2 : DISPLAYFORM0 where INLINEFORM0 is a hyperparameter that controls the contribution of individual constituents. This model leverages the average statistics computed over the training corpora (as encapsulated in the word and phrase vectors) to detect compositionality, and is the primary way compositionality has been evaluated previously BIBREF21 , BIBREF22 . Note that for the simple additive model INLINEFORM1 with unit-normalized word vectors, INLINEFORM2 is independent of INLINEFORM3 . At the Token Level Eq. EQREF20 scores phrases for compositionality regardless of the context that these phrases occur in. However, phrases such as “big fish" or “heavy metal" may occur in both compositional and non-compositional situations, depending on the nature and topic of the texts they occur in. Here, we propose a context-driven model for compositionality detection, inspired by the skip-gram model for learning word representations BIBREF2 . The intuition is simple: if a phrase is compositional, it should be sufficiently predictive of the context words around it; otherwise, it is acting in a non-compositional manner. Thus, we would like to compute the likelihood of the context ( INLINEFORM0 ) given a phrasal representation ( INLINEFORM1 ) and normalization constant INLINEFORM2 : DISPLAYFORM0 As explained in Goldberg2014, the context representations are distinct from the word representations. In practice, we compute the log-likelihood averaged over the context words or the perplexity instead of the actual likelihood. Evaluation Our experiments had three aims: first, demonstrate that the compositional functions learned using paraphrase supervision compute semantically meaningful results for compositional phrases by evaluating on a phrase similarity task (§ SECREF29 ); second, verify the hypothesis that compositionality is context-dependent by comparing a type-based and token-based approach on a compound noun evaluation task (§ SECREF36 ); and third, determine if the compositionality-scoring models based on learned representations improve the translations produced by a state-of-the-art phrase-based MT system (§ SECREF38 ). The word vectors used in all of our experiments were produced by word2vec using the skip-gram model with 20 negative samples, a context window size of 10, a minimum token count of 3, and sub-sampling of frequent words with a parameter of INLINEFORM0 . We extracted corpus statistics for word2vec using the AFP portion of the English Gigaword, which consists of 887.5 million tokens. The code used to generate the results is available at http://www.github.com/xyz, and the evaluation datasets are publicly available. Phrasal Similarity For the phrase similarity task we first compare our concatenation and tensor models learned using INLINEFORM0 and INLINEFORM1 regularization to three baselines: [noitemsep] add: INLINEFORM0 mult1: INLINEFORM0 mult2: INLINEFORM0 Other additive models from previous work BIBREF5 , BIBREF23 , BIBREF24 that impose varying amounts of structural assumptions on the semantic interactions between word representations e.g., INLINEFORM0 or INLINEFORM1 are subsumed by our concatenation model. The regularization strength hyperparameter for INLINEFORM2 and INLINEFORM3 regularization was selected using 5-fold cross-validation on the PPDB training data. We evaluated the phrase compositionality models on the adjective–noun and noun–noun phrase similarity tasks compiled by Mitchell2010, using the same evaluation scheme as in the original work. Spearman's INLINEFORM0 between phrasal similarities derived from our compositional functions and the human annotators (computed individually per annotator and then averaged across all annotators) was the evaluation measure. Figure FIGREF24 presents the correlation results for the two POS pair types as a function of the dimensionality INLINEFORM0 of the representations for the concatenation models (and additive baseline) and tensor models (and multiplicative baselines). The concatenation models seem more effective than the tensor models in the adjective–noun case and give roughly the same performance on the noun–noun dataset, which is consistent with previous work that uses dense, low-dimensional representations BIBREF25 , BIBREF15 , BIBREF26 . Since the concatenation model involve fewer parameters, we use it as the compositional model of choice for subsequent experiments. The absolute results are also consistent with state-of-the-art results on this dataset BIBREF24 , BIBREF26 , indicating that paraphrases are an excellent source of information for learning compositional functions and a reasonable alternative to human-annotated training sets. For reference, the inter-annotator agreements are 0.52 for the adjective–noun evaluation and 0.51 for the noun–noun one. The unweighted additive baseline is surprisingly very strong on the noun–noun set, so we also compare against it in subsequent experiments. Compositionality To evaluate the compositionality-scoring models, we used the compound noun compositionality dataset introduced in Reddy2011. This dataset consists of 2670 annotations of 90 compound-noun phrases exhibiting varying levels compositionality, with scores ranging from 0 to 5 provided by 30 annotators. It also contains three to five example sentences of these phrases that were shown to the annotators, which we make use of in our context-dependent model. Consistent with the original work, Spearman's INLINEFORM0 is computed on the averaged compositionality score for a phrase across all the annotators that scored that phrase (which varies per phrase). For computing the compositional functions, we evaluate three of the best performing setups from § SECREF29 : the INLINEFORM1 and INLINEFORM2 -regularized concatenation models, and the simple additive baseline. For the context-independent model, we select the hyperparameter INLINEFORM0 in Eq. EQREF20 from the values INLINEFORM1 . For the context-dependent model, we vary the context window size INLINEFORM2 by selecting from the values INLINEFORM3 . Table TABREF37 presents Spearman's INLINEFORM4 for these setups. In all cases, the context-dependent models outperform the context-independent ones, and using a relatively simple token-based model we can approximately match the performance of the Bayesian model proposed by Hermann2012. The concatenation models are also consistently better than the additive compositional model, indicating the benefit of learning the compositional parameters via PPDB. Machine Translation While any truly successful model of semantics must match human intuitions, understanding the applications of our models is likewise important. To this end, we consider the problem of machine translation, operating under the hypothesis that sentences which express their meaning non-compositionally should also translate non-compositionally. Modern phrase-based translation systems are faced with a large number of possible segmentations of a source-language sentence during decoding, and all segmentations are considered equally likely BIBREF13 . Thus, it would be helpful to provide guidance on more likely segmentations, as dictated by the compositionality scores of the phrases extracted from a sentence, to the decoder. A low compositionality score would ideally force the decoder to consider the entire phrase as a translation unit, due to its unique semantic characteristics. Correspondingly, a high score informs the decoder that it is safe to rely on word-level translations of the phrasal constituents. Thus, if we reveal to the translation system that a phrase is non-compositional, it should be able to learn that translation decisions which translate it as a unit are to be favored, leading to better translations. To test this hypothesis, we built an English-Spanish MT system using the cdec decoder BIBREF27 for the entire training pipeline (word alignments, phrase extraction, feature weight tuning, and decoding). Corpora from the WMT 2011 evaluation was used to build the translation and language models, and for tuning (on news-test2010) and evaluation (on news-test2011), with scoring done using BLEU BIBREF28 . The baseline is a hierarchical phrase-based system BIBREF29 with a 4-gram language model, with feature weights tuned using MIRA BIBREF30 . For features, each translation rule is decorated with two lexical and phrasal features corresponding to the forward INLINEFORM0 and backward INLINEFORM1 conditional log frequencies, along with the log joint frequency INLINEFORM2 , the log frequency of the source phrase INLINEFORM3 , and whether the phrase pair or the source phrase is a singleton. Weights for the language model, glue rule, and word penalty are also tuned. This setup (Baseline) achieves scores en par with the published WMT results. We added the compositionality score as an additional feature, and also added two binary-valued features: the first indicates if the given translation rule has not been decorated with a compositionality score (either because it consists of non-terminals only or the lexical items in the translation rule are unigrams), and correspondingly the second feature indicates if the translation rule has been scored. Therefore, an appropriate additional baseline would be to mark translation rules with these indicator functions but without the scores, akin to identifying rules with phrases in them (Baseline + SegOn). Table TABREF40 presents the results of the MT evaluation, comparing the baselines to the best-performing context-independent and dependent scoring models from § SECREF36 . The scores have been averaged over three tuning runs with standard deviation in parentheses; bold results on the test set are statistically significant ( INLINEFORM0 ) with respect to the baseline. While knowledge of relative compositionality consistently helps, the improvements using the context-dependent scoring models, especially with the INLINEFORM1 concatenation model, are noticeably better. Related Work There has been a large amount of work on compositional models that operate on vector representations of words. With some exceptions BIBREF16 , BIBREF5 , all of these approaches are lexicalized i.e., parameters (generally in the form of vectors, matrices, or tensors) for specific words are learned, which works well for frequently occurring words but fails when dealing with compositions of arbitrary word sequences containing infrequent words. The functions are either learned with a neural network architecture BIBREF7 or as a linear regression BIBREF6 ; the latter require phrase representations extracted directly from the corpus for supervision, which can be computationally expensive and statistically inefficient. In contrast, we obtain this information through many-to-one PPDB mappings. Most of these models also require additional syntactic BIBREF31 or semantic BIBREF15 , BIBREF14 resources; on the other hand, our proposed approach only requires a shallow syntactic parse (POS tags). Recent efforts to make these models more practical BIBREF32 attempt to reduce their statistically complex and overly-parametrized nature, but with the exception of Zanzotto2010, who propose a way to extract compositional function training examples from a dictionary, these models generally require human-annotated data to work. Most models that score the relative (non-) compositionality of phrases do so in a context-independent manner. A central idea is to replace phrase constituents with semantically-related words and compute the similarity of the new phrase to the original BIBREF22 , BIBREF33 or make use of a variety of lexical association measures BIBREF10 , BIBREF34 . Sporleder2009 however, do make use of context in a token-based approach, where the context in which a phrase occurs as well as the phrase itself is modeled as a lexical chain, and the cohesion of the chain is measured as an indicator of a phrase's compositionality. Cohesion is computed using a web search engine-based measure, whereas we use a probabilistic model of context given a phrase representation. Hermann2012 propose a Bayesian generative model that is also context-based, but learning and inference is done through a relatively expensive Gibbs sampling scheme. In the context of MT, Zhang2008b present a Bayesian model that learns non-compositional phrases from a synchronous parse tree of a sentence pair. However, the primary aim of their work is phrase extraction for MT, and the non-compositional constraints are only applied to make the space of phrase pairs more tractable when bootstrapping their phrasal parser from their word-based parser. In contrast, we score every phrase that is extracted with the standard phrase extraction heuristics BIBREF29 , allowing the decoder to make the final decision on the impact of compositionality scores in translation. Thus, our work is more similar to Xiong2010, who propose maximum entropy classifiers that mark positions between words in a sentence as being a phrase boundary or not, and integrate these scores as additional features in an MT system. Conclusion In this work, we presented two new sources of information for compositionality modeling and scoring, paraphrase information and context. For modeling, we showed that the paraphrase-learned compositional representations performs as well on a phrase similarity task as the average human annotator. For scoring, the importance of context was shown through the comparison of context-independent and dependent models. Improvements by the context-dependent model on an extrinsic machine translation task corroborate the utility of these additional knowledge sources. We hope that this work encourages further research in making compositional semantic approaches applicable in downstream tasks.
Which translation systems do they compare against?
435570723b37ee1f5898c1a34ef86a0b2e8701bb
Which Ġtranslation Ġsystems Ġdo Ġthey Ġcompare Ġagainst ?
h ier arch ical Ġphrase - based Ġsystem ĠB IB REF 29 , Ġappropriate Ġadditional Ġbaseline Ġwould Ġbe Ġto Ġmark Ġtranslation Ġrules Ġwith Ġthese Ġindicator Ġfunctions Ġbut Ġwithout Ġthe Ġscores , Ġakin Ġto Ġidentifying Ġrules Ġwith Ġphrases Ġin Ġthem Ġ( Bas eline Ġ+ ĠSeg On )
<s> Which Ġtranslation Ġsystems Ġdo Ġthey Ġcompare Ġagainst ? </s> Introduction </s> N umerous Ġlex ical Ġsemantic Ġproperties Ġare Ġcaptured Ġby Ġrepresentations Ġencoding Ġdistribution al Ġproperties Ġof Ġwords , Ġas Ġhas Ġbeen Ġdemonstrated Ġin Ġa Ġvariety Ġof Ġtasks ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠHowever , Ġthis Ġdistribution al Ġaccount Ġof Ġmeaning Ġdoes Ġnot Ġscale Ġto Ġlarger Ġunits Ġlike Ġphrases Ġand Ġsentences ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġmotivating Ġresearch Ġinto Ġcompos itional Ġmodels Ġthat Ġcombine Ġword Ġrepresentations Ġto Ġproduce Ġrepresentations Ġof Ġthe Ġsemantics Ġof Ġlonger Ġunits ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠPrevious Ġwork Ġhas Ġlearned Ġthese Ġmodels Ġusing Ġaut oen c oder Ġformulations ĠB IB REF 8 Ġor Ġlimited Ġhuman Ġsupervision ĠB IB REF 5 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġexplore Ġthe Ġhypothesis Ġthat Ġthe Ġequivalent Ġknowledge Ġabout Ġhow Ġwords Ġcompose Ġcan Ġbe Ġobtained Ġthrough Ġmon oling ual Ġparaph r ases Ġthat Ġhave Ġbeen Ġextracted Ġusing Ġword Ġalign ments Ġand Ġan Ġintermediate Ġlanguage ĠB IB REF 9 Ġ. ĠConf irming Ġthis Ġhypothesis Ġwould Ġallow Ġthe Ġrapid Ġdevelopment Ġof Ġcompos itional Ġmodels Ġin Ġa Ġlarge Ġnumber Ġof Ġlanguages . </s> As Ġtheir Ġname Ġsuggests , Ġthese Ġmodels Ġalso Ġimpose Ġthe Ġassumption Ġthat Ġlonger Ġunits Ġlike Ġphrases Ġare Ġcompos itional , Ġi . e ., Ġa Ġphrase 's Ġmeaning Ġcan Ġbe Ġunderstood Ġfrom Ġthe Ġliteral Ġmeaning Ġof Ġits Ġparts . ĠHowever , Ġcountless Ġexamples Ġthat Ġrun Ġcontrary Ġto Ġthe Ġassumption Ġexist , Ġand Ġhandling Ġthese Ġnon - com pos itional Ġphrases Ġhas Ġbeen Ġproblematic Ġand Ġof Ġlong - standing Ġinterest Ġin Ġthe Ġcommunity ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ. Ġ( Non -) ĠCom position ality Ġdetection Ġcan Ġprovide Ġvital Ġinformation Ġto Ġother Ġlanguage Ġprocessing Ġsystems Ġon Ġwhether Ġa Ġmulti word Ġunit Ġshould Ġbe Ġtreated Ġsem antically Ġas Ġa Ġsingle Ġentity Ġor Ġnot , Ġand Ġscoring Ġthis Ġphenomenon Ġis Ġparticularly Ġrelevant Ġfor Ġdownstream Ġtasks Ġlike Ġmachine Ġtranslation Ġ( MT ) Ġor Ġinformation Ġretrieval . ĠWe Ġexplore Ġthe Ġhypothesis Ġthat Ġcontextual Ġevidence Ġcan Ġbe Ġused Ġto Ġdetermine Ġthe Ġrelative Ġdegree Ġto Ġwhich Ġa Ġphrase Ġis Ġmeant Ġcomposition ally . </s> Rather Ġthan Ġfocusing Ġpurely Ġon Ġintrinsic Ġclean - room Ġevaluations , Ġthe Ġgoal Ġof Ġthis Ġwork Ġis Ġto Ġlearn Ġrelatively Ġaccurate Ġcontext - sensitive Ġcompos itional Ġmodels Ġthat Ġare Ġalso Ġdirectly Ġapplicable Ġin Ġreal - world , Ġnoisy - data Ġscenarios . ĠThis Ġobjective Ġnecess itates Ġcertain Ġdesign Ġdecisions , Ġand Ġto Ġthis Ġend Ġwe Ġpropose Ġa Ġrobust , Ġscalable Ġframework Ġthat Ġlearns Ġcompos itional Ġfunctions Ġand Ġscores Ġrelative Ġphr as al Ġcomposition ality . ĠWe Ġmake Ġthree Ġcontributions : Ġfirst , Ġa Ġnovel Ġway Ġto Ġlearn Ġcompos itional Ġfunctions Ġfor Ġpart - of - speech Ġpairs Ġthat Ġuses Ġsupervision Ġfrom Ġan Ġautomatically - ext racted Ġlist Ġof Ġparaph r ases Ġ( Â§ ĠSEC REF 3 Ġ). ĠSecond , Ġa Ġcontext - dependent Ġscoring Ġmodel Ġthat Ġscores Ġthe Ġrelative Ġcomposition ality Ġof Ġa Ġphrase ĠB IB REF 12 Ġby Ġcomputing Ġthe Ġlikelihood Ġof Ġits Ġcontext Ġgiven Ġits Ġparaph rase - learn ed Ġrepresentation Ġ( Â§ ĠSEC REF 4 Ġ). ĠAnd Ġthird , Ġan Ġevaluation Ġof Ġthe Ġimpact Ġof Ġcomposition ality Ġknowledge Ġin Ġan Ġend - to - end ĠMT Ġsetup . ĠOur Ġexperiments Ġ( Â§ ĠSEC REF 5 Ġ) Ġreveal Ġthat Ġusing Ġsupervision Ġfrom Ġautomatically Ġextracted Ġparaph r ases Ġproduces Ġcompos itional Ġfunctions Ġwith Ġequivalent Ġperformance Ġto Ġprevious Ġapproaches Ġthat Ġhave Ġrelied Ġon Ġhand - annot ated Ġtraining Ġdata . ĠFurthermore , Ġcomposition ality Ġfeatures Ġconsistently Ġimprove Ġthe Ġtranslations Ġproduced Ġby Ġa Ġstrong ĠEnglish âĢĵ Spanish Ġtranslation Ġsystem . </s> Param etric ĠCom position ĠFunctions </s> We Ġformal ize Ġcomposition Ġas Ġa Ġfunction ĠIN LINE FORM 0 Ġthat Ġmaps ĠIN LINE FORM 1 Ġ- dimensional Ġvector Ġrepresentations Ġof Ġphrase Ġconstituents ĠIN LINE FORM 2 Ġto Ġan ĠIN LINE FORM 3 Ġ- dimensional Ġvector Ġrepresentation Ġof Ġthe Ġphrase , Ġi . e ., Ġthe Ġcomposed Ġrepresentation . ĠA Ġphrase Ġis Ġdefined Ġas Ġany Ġcontiguous Ġsequence Ġof Ġwords Ġof Ġlength Ġ2 Ġor Ġgreater , Ġand Ġdoes Ġnot Ġhave Ġto Ġadhere Ġto Ġconstituents Ġin Ġa Ġphrase Ġstructure Ġgrammar . ĠThis Ġdefinition Ġis Ġin Ġline Ġwith Ġour ĠMT Ġapplication Ġand Ġignores ĠâĢ ľ g appy âĢ Ŀ Ġnon cont iguous Ġphrases , Ġbut Ġthis Ġpragmatic Ġchoice Ġdoes Ġexclude Ġmany Ġverb - object Ġrelations ĠB IB REF 13 Ġ. ĠWe Ġassume Ġthe Ġexistence Ġof Ġword - level Ġvector Ġrepresentations Ġfor Ġevery Ġword Ġin Ġour Ġvocabulary Ġof Ġsize ĠIN LINE FORM 4 Ġ. ĠCom position ality Ġis Ġmodeled Ġas Ġa Ġbil in ear Ġmap , Ġand Ġtwo Ġclasses Ġof Ġlinear Ġmodels Ġwith Ġdifferent Ġlevels Ġof Ġparam et riz ation Ġare Ġproposed . ĠUnlike Ġprevious Ġwork ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 14 Ġwhere Ġthe Ġfunctions Ġare Ġword - specific , Ġour Ġcompos itional Ġfunctions Ġoperate Ġon Ġpart - of - speech Ġ( POS ) Ġtag Ġpairs , Ġwhich Ġfacilitates Ġlearning Ġby Ġdrastically Ġreducing Ġthe Ġnumber Ġof Ġparameters , Ġand Ġonly Ġrequires Ġa Ġshallow Ġsynt actic Ġparse Ġof Ġthe Ġinput . </s> Con cat en ation ĠModels </s> Our Ġfirst Ġclass Ġof Ġmodels Ġis Ġa Ġgeneral ization Ġof Ġthe Ġadditive Ġmodels Ġintroduced Ġin ĠMitchell 2008 : ĠDIS PLAY FORM 0 </s> where Ġthe Ġnotation ĠIN LINE FORM 0 Ġrepresents Ġa Ġvertical Ġ( row - wise ) Ġconc aten ation Ġof Ġtwo Ġvectors ; Ġnamely , Ġthe Ġconc aten ation Ġthat Ġresults Ġin Ġa ĠIN LINE FORM 1 Ġ- sized Ġvector . ĠIn Ġaddition Ġto Ġthe ĠIN LINE FORM 2 Ġparameters Ġfor Ġthe Ġword Ġvector Ġrepresentations Ġthat Ġare Ġprovided Ġa Ġprior i , Ġthis Ġmodel Ġintroduces ĠIN LINE FORM 3 Ġparameters , Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġnumber Ġof ĠPOS - tag Ġpairs Ġwe Ġconsider . </s> Mit chell 2008 Ġsignificantly Ġsimplify Ġparameter Ġestimation Ġby Ġassuming Ġa Ġcertain Ġstructure Ġfor Ġthe Ġparameter Ġmatrix ĠIN LINE FORM 0 Ġ, Ġwhich Ġis Ġnecessary Ġgiven Ġthe Ġlimited Ġhuman - annot ated Ġdata Ġthey Ġuse . ĠFor Ġexample , Ġby Ġassuming Ġa Ġblock - di agonal Ġstructure , Ġwe Ġget Ġa Ġscaled Ġelement - wise Ġaddition Ġmodel ĠIN LINE FORM 1 Ġ. ĠWhile Ġnot Ġstrictly Ġin Ġthis Ġcategory Ġdue Ġto Ġthe Ġnon - linear ities Ġinvolved , Ġneural Ġnetwork - based Ġcompos itional Ġmodels ĠB IB REF 7 Ġ, ĠB IB REF 15 Ġcan Ġbe Ġviewed Ġas Ġconc aten ation Ġmodels , Ġalthough Ġthe Ġorder Ġof Ġconc aten ation Ġand Ġmatrix Ġmultiplication Ġis Ġswitched . ĠHowever , Ġthese Ġmodels Ġintroduce Ġmore Ġthan ĠIN LINE FORM 2 Ġparameters . </s> T ensor ĠModels </s> The Ġsecond Ġclass Ġof Ġmodels Ġlever ages Ġpair wise Ġmultipl icative Ġinteractions Ġbetween Ġthe Ġcomponents Ġof Ġthe Ġtwo Ġword Ġvectors : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġcorresponds Ġto Ġa Ġtens or Ġcontraction Ġalong Ġthe ĠIN LINE FORM 1 Ġmode Ġof Ġthe Ġtens or ĠIN LINE FORM 2 Ġ. ĠIn Ġthis Ġcase , Ġwe Ġfirst Ġcompute Ġa Ġcontraction Ġ( t ensor - vector Ġproduct ) Ġbetween ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġalong ĠIN LINE FORM 5 Ġ' s Ġthird Ġmode , Ġcorresponding Ġto Ġinteractions Ġwith Ġthe Ġsecond Ġword Ġvector Ġof Ġa Ġtwo - word Ġphrase Ġand Ġresulting Ġin Ġa Ġmatrix , Ġwhich Ġis Ġthen Ġmultiplied Ġalong Ġits Ġsecond Ġmode Ġ( cor respond ing Ġto Ġtraditional Ġmatrix Ġmultiplication Ġon Ġthe Ġright ) Ġby ĠIN LINE FORM 6 Ġ. ĠThe Ġfinal Ġresult Ġis Ġan ĠIN LINE FORM 7 Ġvector . ĠThis Ġmodel Ġintroduces ĠIN LINE FORM 8 Ġparameters . </s> T ensor Ġmodels Ġare Ġa Ġgeneral ization Ġof Ġthe Ġelement - wise Ġmultipl icative Ġmodel ĠB IB REF 16 Ġ, Ġwhich Ġpermits Ġnon - zero Ġvalues Ġonly Ġon Ġthe Ġtens or Ġdiagonal . ĠOperating Ġat Ġthe Ġvocabulary Ġlevel , Ġthe Ġmodel Ġof ĠBar oni 2010 Ġhas Ġinteresting Ġparallels Ġto Ġour Ġtens or Ġmodel . ĠThey Ġfocus Ġon Ġadjective âĢĵ n oun Ġrelationships Ġand Ġlearn Ġa Ġspecific Ġmatrix Ġfor Ġevery Ġadjective Ġin Ġtheir Ġdataset ; Ġin Ġour Ġcase , Ġthe Ġspecific Ġmatrix Ġfor Ġeach Ġadjective Ġhas Ġa Ġparticular Ġform , Ġnamely Ġthat Ġit Ġcan Ġbe Ġfactor ized Ġinto Ġthe Ġproduct Ġof Ġa Ġtens or Ġand Ġa Ġvector ; Ġthe Ġtens or Ġcorresponds Ġto Ġthe Ġactual Ġadjective âĢĵ n oun Ġcomb iner Ġfunction , Ġand Ġthe Ġvector Ġcorresponds Ġto Ġspecific Ġlex ical Ġinformation Ġthat Ġthe Ġadjective Ġcarries . ĠThis Ġconcept Ġgeneral izes Ġto Ġother ĠPOS Ġpairs : Ġfor Ġexample , Ġmultiplying Ġthe Ġtens or Ġthat Ġrepresents Ġdetermin er - n oun Ġcombinations Ġalong Ġthe Ġsecond Ġmode Ġwith Ġthe Ġvector Ġfor ĠâĢ ľ the âĢ Ŀ Ġresults Ġin Ġa Ġmatrix Ġthat Ġrepresents Ġthe Ġsemantic Ġoperation Ġof Ġdefin it eness . ĠLearning Ġthese Ġparameters Ġjointly Ġis Ġstatistically Ġmore Ġefficient Ġthan Ġseparately Ġlearning Ġversions Ġfor Ġeach Ġword . </s> Long er ĠPh r ases </s> The Ġproposed Ġmodels Ġoperate Ġon Ġpairs Ġof Ġwords Ġat Ġa Ġtime . ĠTo Ġhandle Ġphrases Ġof Ġlength Ġgreater Ġthan Ġtwo , Ġwe Ġgreed ily Ġconstruct Ġa Ġleft - br anch ing Ġtree Ġof Ġthe Ġphrase Ġconstituents Ġthat Ġeventually Ġdictates Ġthe Ġapplication Ġof Ġthe Ġlearned Ġbil in ear Ġmaps . ĠFor Ġeach Ġinternal Ġtree Ġnode , Ġwe Ġconsider Ġthe ĠPOS Ġtags Ġof Ġits Ġchildren : Ġif Ġthe Ġright Ġchild Ġis Ġa Ġnoun , Ġand Ġthe Ġleft Ġchild Ġis Ġeither Ġa Ġnoun , Ġadjective , Ġor Ġdetermin er , Ġthen Ġthe Ġinternal Ġnode Ġis Ġmarked Ġas Ġa Ġnoun , Ġotherwise Ġwe Ġmark Ġit Ġwith Ġa Ġgeneric Ġother Ġtag . ĠAt Ġthe Ġend Ġof Ġthe Ġprocedure , Ġunatt ached Ġnodes Ġ( words ) Ġare Ġattached Ġat Ġthe Ġhighest Ġpoint Ġin Ġthe Ġtree . </s> After Ġthe Ġtree Ġis Ġconstructed , Ġwe Ġcan Ġcompute Ġthe Ġoverall Ġphr as al Ġrepresentation Ġin Ġa Ġbottom - up Ġmanner , Ġguided Ġby Ġthe Ġlabels Ġof Ġleaf Ġand Ġinternal Ġnodes . ĠWe Ġnote Ġthat Ġthe Ġemphasis Ġof Ġthis Ġwork Ġis Ġnot Ġto Ġcompute Ġsentence - level Ġrepresentations . ĠThis Ġgoal Ġhas Ġbeen Ġexplored Ġin Ġrecent Ġresearch ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġand Ġcombining Ġour Ġmodels Ġwith Ġmethods Ġpresented Ġtherein Ġfor Ġsentence - level Ġrepresentations Ġis Ġstraightforward . </s> Learning </s> The Ġmodels Ġdescribed Ġabove Ġrely Ġon Ġparameters ĠIN LINE FORM 0 Ġthat Ġmust Ġbe Ġlearned . ĠIn Ġthis Ġsection , Ġwe Ġargue Ġthat Ġautomatically Ġconstructed Ġdatabases Ġof Ġparaph r ases Ġprovide Ġadequate Ġsupervision Ġfor Ġlearning Ġnotions Ġof Ġcomposition ality . </s> Super vision Ġfrom ĠAutomatic ĠPar aph r ases </s> The ĠPar aph rase ĠDatabase ĠB IB REF 9 Ġis Ġa Ġcollection Ġof Ġranked Ġmon oling ual Ġparaph r ases Ġthat Ġhave Ġbeen Ġextracted Ġfrom Ġword - aligned Ġparallel Ġcorpor a Ġusing Ġthe Ġbilingual Ġpivot Ġmethod ĠB IB REF 19 Ġ. ĠThe Ġunderlying Ġassumption Ġis Ġthat Ġif Ġtwo Ġstrings Ġin Ġthe Ġsame Ġlanguage Ġalign Ġto Ġthe Ġsame Ġstring Ġin Ġanother Ġlanguage , Ġthen Ġthe Ġstrings Ġin Ġthe Ġoriginal Ġlanguage Ġshare Ġthe Ġsame Ġmeaning . ĠPar aph r ases Ġare Ġranked Ġby Ġtheir Ġword Ġalignment Ġscores , Ġand Ġin Ġthis Ġwork Ġwe Ġuse Ġthe Ġpre selected Ġsmall Ġportion Ġof ĠP PD B Ġas Ġour Ġtraining Ġdata . ĠAlthough Ġwe Ġcan Ġdirectly Ġextract Ġphr as al Ġrepresentations Ġof Ġa Ġpre - specified Ġlist Ġof Ġphrases Ġfrom Ġthe Ġcorpus Ġused Ġto Ġcompute Ġword Ġrepresentations ĠB IB REF 6 Ġ, Ġthis Ġapproach Ġis Ġboth Ġcomput ationally Ġand Ġstatistically Ġinefficient : Ġthe Ġnumber Ġof Ġphrases Ġincreases Ġexponentially Ġin Ġthe Ġlength Ġof Ġthe Ġphrase , Ġand Ġcorrespond ingly Ġthe Ġoccurrence Ġof Ġany Ġindividual Ġphrase Ġdecreases Ġexponentially . ĠWe Ġcan Ġthus Ġcircumvent Ġthese Ġcomputational Ġand Ġstatistical Ġissues Ġby Ġusing Ġmon oling ual Ġparaph r ases . </s> The Ġtraining Ġdata Ġis Ġfiltered Ġto Ġprovide Ġonly Ġtwo - to - one Ġword Ġparaph rase Ġm appings , Ġand Ġthe Ġmulti word Ġportion Ġof Ġthe Ġparaph rase Ġis Ġsubsequently ĠPOS - tag ged . ĠTable ĠT AB REF 10 Ġprovides Ġa Ġbreakdown Ġof Ġsuch Ġparaph r ases Ġby Ġtheir ĠPOS Ġpair Ġtype . ĠGiven Ġthe Ġlack Ġof Ġcontext Ġwhen Ġtagging , Ġit Ġis Ġlikely Ġthat Ġthe ĠPOS Ġtag ger Ġyields Ġthe Ġmost Ġprobable Ġtag Ġfor Ġwords Ġand Ġnot Ġthe Ġmost Ġprobable Ġtag Ġgiven Ġthe Ġ( limited ) Ġcontext . ĠFurthermore , Ġeven Ġthe Ġhigher Ġquality Ġportions Ġof ĠP PD B Ġyield Ġparaph r ases Ġof Ġranging Ġquality , Ġranging Ġfrom Ġnon - t riv ial Ġm appings Ġsuch Ġas Ġyoung Ġpeople ĠIN LINE FORM 0 Ġyouth , Ġto Ġredundant Ġones Ġlike Ġthe Ġceasefire ĠIN LINE FORM 1 Ġceasefire . ĠHowever , ĠP PD B - like Ġresources Ġare Ġmore Ġeasily Ġavailable Ġthan Ġhuman - annot ated Ġresources Ġ( in Ġmultiple Ġlanguages Ġtoo : ĠGan it ke vich 2014 ), Ġso Ġit Ġis Ġimperative Ġthat Ġmethods Ġwhich Ġlearn Ġcompos itional Ġfunctions Ġfrom Ġsuch Ġsources Ġhandle Ġnoisy Ġsupervision Ġadequately . </s> Parameter ĠEst imation </s> The Ġparameters ĠIN LINE FORM 0 Ġin ĠE q . ĠEQ REF 4 Ġand ĠEQ REF 6 Ġcan Ġbe Ġestimated Ġthrough Ġstandard Ġlinear Ġregression Ġtechniques Ġin Ġconjunction Ġwith Ġthe Ġdata Ġpresented Ġin ĠÂ§ ĠSEC REF 3 Ġ. ĠThese Ġmethods Ġprovide Ġa Ġnatural Ġway Ġto Ġregular ize ĠIN LINE FORM 1 Ġvia ĠIN LINE FORM 2 Ġ( ridge ) Ġor ĠIN LINE FORM 3 Ġ( L ASS O ) Ġregular ization , Ġwhich Ġalso Ġhelps Ġhandle Ġnoisy Ġparaph r ases . ĠParameters Ġfor Ġthe ĠIN LINE FORM 4 Ġ- regular ized Ġconc aten ation Ġmodel Ġfor Ġselect ĠPOS Ġpairs Ġare Ġdisplayed Ġin ĠFig . ĠFIG REF 12 Ġ. ĠThe Ġheat - maps Ġdisplay Ġthe Ġrelative Ġmagnitude Ġof Ġparameters , Ġwith Ġpositive Ġvalues Ġcolored Ġblue , Ġnegative Ġvalues Ġcolored Ġred , Ġand Ġwhite Ġcells Ġindicating Ġzero Ġvalues . ĠIt Ġis Ġevident Ġthat Ġthe Ġparameters Ġlearned Ġfrom ĠP PD B Ġindicate Ġa Ġnotion Ġof Ġlinguistic Ġheaded ness , Ġnamely Ġthat Ġfor Ġparticular ĠPOS Ġpairs , Ġthe Ġsemantic Ġinformation Ġis Ġprimarily Ġcontained Ġin Ġthe Ġright Ġword , Ġbut Ġfor Ġothers Ġsuch Ġas Ġthe Ġnoun âĢĵ n oun Ġcombination , Ġeach Ġconstituent 's Ġcontribution Ġis Ġrelatively Ġmore Ġequal . </s> Me asuring Ġof ĠCom position ality </s> The Ġconc aten ation Ġand Ġtens or Ġmodels Ġcompute Ġan ĠIN LINE FORM 0 Ġ- dimensional Ġvector Ġrepresentation Ġfor Ġa Ġmulti - word Ġphrase Ġby Ġassuming Ġthe Ġmeaning Ġof Ġthe Ġphrase Ġcan Ġbe Ġexpressed Ġin Ġterms Ġof Ġthe Ġmeaning Ġof Ġits Ġconstituents . ĠThis Ġassumption Ġholds Ġtrue Ġto Ġvarying Ġdegrees ; Ġwhile Ġit Ġclearly Ġholds Ġfor ĠâĢ ľ large Ġamount " Ġand Ġbreaks Ġdown Ġfor ĠâĢ ľ cloud Ġnine ", Ġit Ġis Ġpartially Ġvalid Ġfor Ġphrases Ġsuch Ġas ĠâĢ ľ z ebra Ġcrossing " Ġor ĠâĢ ľ cr ash Ġcourse ". ĠIn Ġline Ġwith Ġprevious Ġwork , Ġwe Ġassume Ġa Ġcomposition ality Ġcontinuum ĠB IB REF 12 Ġ, Ġbut Ġfurther Ġconjecture Ġthat Ġa Ġphrase 's Ġlevel Ġof Ġcomposition ality Ġis Ġdependent Ġon Ġthe Ġspecific Ġcontext Ġin Ġwhich Ġit Ġoccurs , Ġmotivating Ġa Ġcontext - based Ġapproach Ġ( Â§ ĠSEC REF 21 Ġ) Ġwhich Ġscores Ġcomposition ality Ġby Ġcomputing Ġthe Ġlikelihood s Ġof Ġsurrounding Ġcontext Ġwords Ġgiven Ġa Ġphrase Ġrepresentation . ĠThe Ġeffect Ġof Ġcontext Ġis Ġdirectly Ġmeasured Ġthrough Ġa Ġcomparison Ġwith Ġcontext - independent Ġmethods Ġfrom Ġprior Ġwork ĠB IB REF 20 Ġ, ĠB IB REF 21 </s> It Ġis Ġimportant Ġto Ġnote Ġthat Ġmost Ġprior Ġwork Ġon Ġcomposition ality Ġscoring Ġassumes Ġaccess Ġto Ġboth Ġword Ġand Ġphrase Ġvector Ġrepresentations Ġ( for Ġselect Ġphrases Ġthat Ġwill Ġbe Ġevaluated ) Ġa Ġprior i . ĠThe Ġlatter Ġare Ġdistinct Ġfrom Ġrepresentations Ġthat Ġare Ġcomputed Ġfrom Ġlearned Ġcompos itional Ġfunctions Ġas Ġthey Ġare Ġextracted Ġdirectly Ġfrom Ġthe Ġcorpus , Ġwhich Ġis Ġan Ġexpensive Ġprocedure . ĠOur Ġaim Ġis Ġto Ġdevelop Ġcompos itional Ġmodels Ġthat Ġare Ġapplicable Ġin Ġdownstream Ġtasks , Ġand Ġthus Ġassuming Ġpre - existing Ġphrase Ġvectors Ġis Ġunreasonable . ĠHence Ġfor Ġphrases , Ġwe Ġonly Ġrely Ġon Ġrepresentations Ġcomputed Ġfrom Ġour Ġlearned Ġcompos itional Ġfunctions . </s> At Ġthe ĠType ĠLevel </s> Given Ġvector Ġrepresentations Ġfor Ġthe Ġconstituent Ġwords Ġin Ġa Ġphrase Ġand Ġthe Ġphrase Ġitself , Ġthe Ġidea Ġbehind Ġthe Ġtype - based Ġmodel Ġis Ġto Ġcompute Ġsimilarities Ġbetween Ġthe Ġconstituent Ġword Ġrepresentations Ġand Ġthe Ġphr as al Ġrepresentation , Ġand Ġaverage Ġthe Ġsimilarities Ġacross Ġthe Ġconstituents . ĠIf Ġthe Ġcontexts Ġin Ġwhich Ġa Ġconstituent Ġword Ġoccurs , Ġas Ġdictated Ġby Ġits Ġvector Ġrepresentation , Ġare Ġvery Ġdifferent Ġfrom Ġthe Ġcontexts Ġof Ġthe Ġcomposed Ġphrase , Ġas Ġindicated Ġby Ġthe Ġcos ine Ġsimilarity Ġbetween Ġthe Ġword Ġand Ġphrase Ġrepresentations , Ġthen Ġthe Ġphrase Ġis Ġlikely Ġto Ġbe Ġnon - com pos itional . ĠAssuming Ġunit - normal ized Ġword Ġvectors ĠIN LINE FORM 0 Ġand Ġphrase Ġvector ĠIN LINE FORM 1 Ġcomputed Ġfrom Ġone Ġof Ġthe Ġlearned Ġmodels Ġin ĠÂ§ ĠSEC REF 2 Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġhyper param eter Ġthat Ġcontrols Ġthe Ġcontribution Ġof Ġindividual Ġconstituents . ĠThis Ġmodel Ġlever ages Ġthe Ġaverage Ġstatistics Ġcomputed Ġover Ġthe Ġtraining Ġcorpor a Ġ( as Ġencaps ulated Ġin Ġthe Ġword Ġand Ġphrase Ġvectors ) Ġto Ġdetect Ġcomposition ality , Ġand Ġis Ġthe Ġprimary Ġway Ġcomposition ality Ġhas Ġbeen Ġevaluated Ġpreviously ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġ. ĠNote Ġthat Ġfor Ġthe Ġsimple Ġadditive Ġmodel ĠIN LINE FORM 1 Ġwith Ġunit - normal ized Ġword Ġvectors , ĠIN LINE FORM 2 Ġis Ġindependent Ġof ĠIN LINE FORM 3 Ġ. </s> At Ġthe ĠToken ĠLevel </s> E q . ĠEQ REF 20 Ġscores Ġphrases Ġfor Ġcomposition ality Ġregardless Ġof Ġthe Ġcontext Ġthat Ġthese Ġphrases Ġoccur Ġin . ĠHowever , Ġphrases Ġsuch Ġas ĠâĢ ľ big Ġfish " Ġor ĠâĢ ľ heavy Ġmetal " Ġmay Ġoccur Ġin Ġboth Ġcompos itional Ġand Ġnon - com pos itional Ġsituations , Ġdepending Ġon Ġthe Ġnature Ġand Ġtopic Ġof Ġthe Ġtexts Ġthey Ġoccur Ġin . ĠHere , Ġwe Ġpropose Ġa Ġcontext - driven Ġmodel Ġfor Ġcomposition ality Ġdetection , Ġinspired Ġby Ġthe Ġskip - gram Ġmodel Ġfor Ġlearning Ġword Ġrepresentations ĠB IB REF 2 Ġ. ĠThe Ġintuition Ġis Ġsimple : Ġif Ġa Ġphrase Ġis Ġcompos itional , Ġit Ġshould Ġbe Ġsufficiently Ġpredictive Ġof Ġthe Ġcontext Ġwords Ġaround Ġit ; Ġotherwise , Ġit Ġis Ġacting Ġin Ġa Ġnon - com pos itional Ġmanner . ĠThus , Ġwe Ġwould Ġlike Ġto Ġcompute Ġthe Ġlikelihood Ġof Ġthe Ġcontext Ġ( ĠIN LINE FORM 0 Ġ) Ġgiven Ġa Ġphr as al Ġrepresentation Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġnormal ization Ġconstant ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> As Ġexplained Ġin ĠGoldberg 2014 , Ġthe Ġcontext Ġrepresentations Ġare Ġdistinct Ġfrom Ġthe Ġword Ġrepresentations . ĠIn Ġpractice , Ġwe Ġcompute Ġthe Ġlog - like lihood Ġaveraged Ġover Ġthe Ġcontext Ġwords Ġor Ġthe Ġperplex ity Ġinstead Ġof Ġthe Ġactual Ġlikelihood . </s> E val uation </s> Our Ġexperiments Ġhad Ġthree Ġaims : Ġfirst , Ġdemonstrate Ġthat Ġthe Ġcompos itional Ġfunctions Ġlearned Ġusing Ġparaph rase Ġsupervision Ġcompute Ġsem antically Ġmeaningful Ġresults Ġfor Ġcompos itional Ġphrases Ġby Ġevaluating Ġon Ġa Ġphrase Ġsimilarity Ġtask Ġ( Â§ ĠSEC REF 29 Ġ); Ġsecond , Ġverify Ġthe Ġhypothesis Ġthat Ġcomposition ality Ġis Ġcontext - dependent Ġby Ġcomparing Ġa Ġtype - based Ġand Ġtoken - based Ġapproach Ġon Ġa Ġcompound Ġnoun Ġevaluation Ġtask Ġ( Â§ ĠSEC REF 36 Ġ); Ġand Ġthird , Ġdetermine Ġif Ġthe Ġcomposition ality - scoring Ġmodels Ġbased Ġon Ġlearned Ġrepresentations Ġimprove Ġthe Ġtranslations Ġproduced Ġby Ġa Ġstate - of - the - art Ġphrase - based ĠMT Ġsystem Ġ( Â§ ĠSEC REF 38 Ġ). </s> The Ġword Ġvectors Ġused Ġin Ġall Ġof Ġour Ġexperiments Ġwere Ġproduced Ġby Ġword 2 vec Ġusing Ġthe Ġskip - gram Ġmodel Ġwith Ġ20 Ġnegative Ġsamples , Ġa Ġcontext Ġwindow Ġsize Ġof Ġ10 , Ġa Ġminimum Ġtoken Ġcount Ġof Ġ3 , Ġand Ġsub - sam pling Ġof Ġfrequent Ġwords Ġwith Ġa Ġparameter Ġof ĠIN LINE FORM 0 Ġ. ĠWe Ġextracted Ġcorpus Ġstatistics Ġfor Ġword 2 vec Ġusing Ġthe ĠAFP Ġportion Ġof Ġthe ĠEnglish ĠGig aw ord , Ġwhich Ġconsists Ġof Ġ8 87 . 5 Ġmillion Ġtokens . ĠThe Ġcode Ġused Ġto Ġgenerate Ġthe Ġresults Ġis Ġavailable Ġat Ġhttp :// www . github . com / xy z , Ġand Ġthe Ġevaluation Ġdatasets Ġare Ġpublicly Ġavailable . </s> Ph ras al ĠSimilar ity </s> For Ġthe Ġphrase Ġsimilarity Ġtask Ġwe Ġfirst Ġcompare Ġour Ġconc aten ation Ġand Ġtens or Ġmodels Ġlearned Ġusing ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġregular ization Ġto Ġthree Ġbas elines : </s> [ no item se p ] </s> add : ĠIN LINE FORM 0 </s> mult 1 : ĠIN LINE FORM 0 </s> mult 2 : ĠIN LINE FORM 0 </s> Other Ġadditive Ġmodels Ġfrom Ġprevious Ġwork ĠB IB REF 5 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġthat Ġimpose Ġvarying Ġamounts Ġof Ġstructural Ġassumptions Ġon Ġthe Ġsemantic Ġinteractions Ġbetween Ġword Ġrepresentations Ġe . g ., ĠIN LINE FORM 0 Ġor ĠIN LINE FORM 1 Ġare Ġsubs umed Ġby Ġour Ġconc aten ation Ġmodel . ĠThe Ġregular ization Ġstrength Ġhyper param eter Ġfor ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġregular ization Ġwas Ġselected Ġusing Ġ5 - fold Ġcross - valid ation Ġon Ġthe ĠP PD B Ġtraining Ġdata . </s> We Ġevaluated Ġthe Ġphrase Ġcomposition ality Ġmodels Ġon Ġthe Ġadjective âĢĵ n oun Ġand Ġnoun âĢĵ n oun Ġphrase Ġsimilarity Ġtasks Ġcompiled Ġby ĠMitchell 2010 , Ġusing Ġthe Ġsame Ġevaluation Ġscheme Ġas Ġin Ġthe Ġoriginal Ġwork . ĠSpear man 's ĠIN LINE FORM 0 Ġbetween Ġphr as al Ġsimilarities Ġderived Ġfrom Ġour Ġcompos itional Ġfunctions Ġand Ġthe Ġhuman Ġannot ators Ġ( com puted Ġindividually Ġper Ġannot ator Ġand Ġthen Ġaveraged Ġacross Ġall Ġannot ators ) Ġwas Ġthe Ġevaluation Ġmeasure . </s> Figure ĠFIG REF 24 Ġpresents Ġthe Ġcorrelation Ġresults Ġfor Ġthe Ġtwo ĠPOS Ġpair Ġtypes Ġas Ġa Ġfunction Ġof Ġthe Ġdimension ality ĠIN LINE FORM 0 Ġof Ġthe Ġrepresentations Ġfor Ġthe Ġconc aten ation Ġmodels Ġ( and Ġadditive Ġbaseline ) Ġand Ġtens or Ġmodels Ġ( and Ġmultipl icative Ġbas elines ). ĠThe Ġconc aten ation Ġmodels Ġseem Ġmore Ġeffective Ġthan Ġthe Ġtens or Ġmodels Ġin Ġthe Ġadjective âĢĵ n oun Ġcase Ġand Ġgive Ġroughly Ġthe Ġsame Ġperformance Ġon Ġthe Ġnoun âĢĵ n oun Ġdataset , Ġwhich Ġis Ġconsistent Ġwith Ġprevious Ġwork Ġthat Ġuses Ġdense , Ġlow - dimensional Ġrepresentations ĠB IB REF 25 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 26 Ġ. ĠSince Ġthe Ġconc aten ation Ġmodel Ġinvolve Ġfewer Ġparameters , Ġwe Ġuse Ġit Ġas Ġthe Ġcompos itional Ġmodel Ġof Ġchoice Ġfor Ġsubsequent Ġexperiments . ĠThe Ġabsolute Ġresults Ġare Ġalso Ġconsistent Ġwith Ġstate - of - the - art Ġresults Ġon Ġthis Ġdataset ĠB IB REF 24 Ġ, ĠB IB REF 26 Ġ, Ġindicating Ġthat Ġparaph r ases Ġare Ġan Ġexcellent Ġsource Ġof Ġinformation Ġfor Ġlearning Ġcompos itional Ġfunctions Ġand Ġa Ġreasonable Ġalternative Ġto Ġhuman - annot ated Ġtraining Ġsets . ĠFor Ġreference , Ġthe Ġinter - annot ator Ġagreements Ġare Ġ0 . 52 Ġfor Ġthe Ġadjective âĢĵ n oun Ġevaluation Ġand Ġ0 . 51 Ġfor Ġthe Ġnoun âĢĵ n oun Ġone . ĠThe Ġun weight ed Ġadditive Ġbaseline Ġis Ġsurprisingly Ġvery Ġstrong Ġon Ġthe Ġnoun âĢĵ n oun Ġset , Ġso Ġwe Ġalso Ġcompare Ġagainst Ġit Ġin Ġsubsequent Ġexperiments . </s> Com position ality </s> To Ġevaluate Ġthe Ġcomposition ality - scoring Ġmodels , Ġwe Ġused Ġthe Ġcompound Ġnoun Ġcomposition ality Ġdataset Ġintroduced Ġin ĠRed dy 2011 . ĠThis Ġdataset Ġconsists Ġof Ġ26 70 Ġannotations Ġof Ġ90 Ġcompound - n oun Ġphrases Ġexhibiting Ġvarying Ġlevels Ġcomposition ality , Ġwith Ġscores Ġranging Ġfrom Ġ0 Ġto Ġ5 Ġprovided Ġby Ġ30 Ġannot ators . ĠIt Ġalso Ġcontains Ġthree Ġto Ġfive Ġexample Ġsentences Ġof Ġthese Ġphrases Ġthat Ġwere Ġshown Ġto Ġthe Ġannot ators , Ġwhich Ġwe Ġmake Ġuse Ġof Ġin Ġour Ġcontext - dependent Ġmodel . ĠCons istent Ġwith Ġthe Ġoriginal Ġwork , ĠSpear man 's ĠIN LINE FORM 0 Ġis Ġcomputed Ġon Ġthe Ġaveraged Ġcomposition ality Ġscore Ġfor Ġa Ġphrase Ġacross Ġall Ġthe Ġannot ators Ġthat Ġscored Ġthat Ġphrase Ġ( which Ġvaries Ġper Ġphrase ). ĠFor Ġcomputing Ġthe Ġcompos itional Ġfunctions , Ġwe Ġevaluate Ġthree Ġof Ġthe Ġbest Ġperforming Ġsetups Ġfrom ĠÂ§ ĠSEC REF 29 Ġ: Ġthe ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġ- regular ized Ġconc aten ation Ġmodels , Ġand Ġthe Ġsimple Ġadditive Ġbaseline . </s> For Ġthe Ġcontext - independent Ġmodel , Ġwe Ġselect Ġthe Ġhyper param eter ĠIN LINE FORM 0 Ġin ĠE q . ĠEQ REF 20 Ġfrom Ġthe Ġvalues ĠIN LINE FORM 1 Ġ. ĠFor Ġthe Ġcontext - dependent Ġmodel , Ġwe Ġvary Ġthe Ġcontext Ġwindow Ġsize ĠIN LINE FORM 2 Ġby Ġselecting Ġfrom Ġthe Ġvalues ĠIN LINE FORM 3 Ġ. ĠTable ĠT AB REF 37 Ġpresents ĠSpear man 's ĠIN LINE FORM 4 Ġfor Ġthese Ġsetups . ĠIn Ġall Ġcases , Ġthe Ġcontext - dependent Ġmodels Ġoutper form Ġthe Ġcontext - independent Ġones , Ġand Ġusing Ġa Ġrelatively Ġsimple Ġtoken - based Ġmodel Ġwe Ġcan Ġapproximately Ġmatch Ġthe Ġperformance Ġof Ġthe ĠBay esian Ġmodel Ġproposed Ġby ĠHerman n 2012 . ĠThe Ġconc aten ation Ġmodels Ġare Ġalso Ġconsistently Ġbetter Ġthan Ġthe Ġadditive Ġcompos itional Ġmodel , Ġindicating Ġthe Ġbenefit Ġof Ġlearning Ġthe Ġcompos itional Ġparameters Ġvia ĠP PD B . </s> Machine ĠTranslation </s> While Ġany Ġtruly Ġsuccessful Ġmodel Ġof Ġsemantics Ġmust Ġmatch Ġhuman Ġintu itions , Ġunderstanding Ġthe Ġapplications Ġof Ġour Ġmodels Ġis Ġlikewise Ġimportant . ĠTo Ġthis Ġend , Ġwe Ġconsider Ġthe Ġproblem Ġof Ġmachine Ġtranslation , Ġoperating Ġunder Ġthe Ġhypothesis Ġthat Ġsentences Ġwhich Ġexpress Ġtheir Ġmeaning Ġnon - com pos itionally Ġshould Ġalso Ġtranslate Ġnon - com pos itionally . </s> Modern Ġphrase - based Ġtranslation Ġsystems Ġare Ġfaced Ġwith Ġa Ġlarge Ġnumber Ġof Ġpossible Ġsegment ations Ġof Ġa Ġsource - language Ġsentence Ġduring Ġdecoding , Ġand Ġall Ġsegment ations Ġare Ġconsidered Ġequally Ġlikely ĠB IB REF 13 Ġ. ĠThus , Ġit Ġwould Ġbe Ġhelpful Ġto Ġprovide Ġguidance Ġon Ġmore Ġlikely Ġsegment ations , Ġas Ġdictated Ġby Ġthe Ġcomposition ality Ġscores Ġof Ġthe Ġphrases Ġextracted Ġfrom Ġa Ġsentence , Ġto Ġthe Ġdec oder . ĠA Ġlow Ġcomposition ality Ġscore Ġwould Ġideally Ġforce Ġthe Ġdec oder Ġto Ġconsider Ġthe Ġentire Ġphrase Ġas Ġa Ġtranslation Ġunit , Ġdue Ġto Ġits Ġunique Ġsemantic Ġcharacteristics . ĠCorrespond ingly , Ġa Ġhigh Ġscore Ġinforms Ġthe Ġdec oder Ġthat Ġit Ġis Ġsafe Ġto Ġrely Ġon Ġword - level Ġtranslations Ġof Ġthe Ġphr as al Ġconstituents . ĠThus , Ġif Ġwe Ġreveal Ġto Ġthe Ġtranslation Ġsystem Ġthat Ġa Ġphrase Ġis Ġnon - com pos itional , Ġit Ġshould Ġbe Ġable Ġto Ġlearn Ġthat Ġtranslation Ġdecisions Ġwhich Ġtranslate Ġit Ġas Ġa Ġunit Ġare Ġto Ġbe Ġfavored , Ġleading Ġto Ġbetter Ġtranslations . </s> To Ġtest Ġthis Ġhypothesis , Ġwe Ġbuilt Ġan ĠEnglish - Spanish ĠMT Ġsystem Ġusing Ġthe Ġc dec Ġdec oder ĠB IB REF 27 Ġfor Ġthe Ġentire Ġtraining Ġpipeline Ġ( word Ġalign ments , Ġphrase Ġextraction , Ġfeature Ġweight Ġtuning , Ġand Ġdecoding ). ĠCorpor a Ġfrom Ġthe ĠW MT Ġ2011 Ġevaluation Ġwas Ġused Ġto Ġbuild Ġthe Ġtranslation Ġand Ġlanguage Ġmodels , Ġand Ġfor Ġtuning Ġ( on Ġnews - test 2010 ) Ġand Ġevaluation Ġ( on Ġnews - test 2011 ), Ġwith Ġscoring Ġdone Ġusing ĠB LE U ĠB IB REF 28 Ġ. ĠThe Ġbaseline Ġis Ġa Ġhierarchical Ġphrase - based Ġsystem ĠB IB REF 29 Ġwith Ġa Ġ4 - gram Ġlanguage Ġmodel , Ġwith Ġfeature Ġweights Ġtuned Ġusing ĠMI RA ĠB IB REF 30 Ġ. ĠFor Ġfeatures , Ġeach Ġtranslation Ġrule Ġis Ġdecorated Ġwith Ġtwo Ġlex ical Ġand Ġphr as al Ġfeatures Ġcorresponding Ġto Ġthe Ġforward ĠIN LINE FORM 0 Ġand Ġbackward ĠIN LINE FORM 1 Ġconditional Ġlog Ġfrequencies , Ġalong Ġwith Ġthe Ġlog Ġjoint Ġfrequency ĠIN LINE FORM 2 Ġ, Ġthe Ġlog Ġfrequency Ġof Ġthe Ġsource Ġphrase ĠIN LINE FORM 3 Ġ, Ġand Ġwhether Ġthe Ġphrase Ġpair Ġor Ġthe Ġsource Ġphrase Ġis Ġa Ġsingle ton . ĠWe ights Ġfor Ġthe Ġlanguage Ġmodel , Ġglue Ġrule , Ġand Ġword Ġpenalty Ġare Ġalso Ġtuned . ĠThis Ġsetup Ġ( Bas eline ) Ġachieves Ġscores Ġen Ġpar Ġwith Ġthe Ġpublished ĠW MT Ġresults . </s> We Ġadded Ġthe Ġcomposition ality Ġscore Ġas Ġan Ġadditional Ġfeature , Ġand Ġalso Ġadded Ġtwo Ġbinary - valued Ġfeatures : Ġthe Ġfirst Ġindicates Ġif Ġthe Ġgiven Ġtranslation Ġrule Ġhas Ġnot Ġbeen Ġdecorated Ġwith Ġa Ġcomposition ality Ġscore Ġ( either Ġbecause Ġit Ġconsists Ġof Ġnon - termin als Ġonly Ġor Ġthe Ġlex ical Ġitems Ġin Ġthe Ġtranslation Ġrule Ġare Ġun ig rams ), Ġand Ġcorrespond ingly Ġthe Ġsecond Ġfeature Ġindicates Ġif Ġthe Ġtranslation Ġrule Ġhas Ġbeen Ġscored . ĠTherefore , Ġan Ġappropriate Ġadditional Ġbaseline Ġwould Ġbe Ġto Ġmark Ġtranslation Ġrules Ġwith Ġthese Ġindicator Ġfunctions Ġbut Ġwithout Ġthe Ġscores , Ġakin Ġto Ġidentifying Ġrules Ġwith Ġphrases Ġin Ġthem Ġ( Bas eline Ġ+ ĠSeg On ). </s> Table ĠT AB REF 40 Ġpresents Ġthe Ġresults Ġof Ġthe ĠMT Ġevaluation , Ġcomparing Ġthe Ġbas elines Ġto Ġthe Ġbest - performing Ġcontext - independent Ġand Ġdependent Ġscoring Ġmodels Ġfrom ĠÂ§ ĠSEC REF 36 Ġ. ĠThe Ġscores Ġhave Ġbeen Ġaveraged Ġover Ġthree Ġtuning Ġruns Ġwith Ġstandard Ġdeviation Ġin Ġparentheses ; Ġbold Ġresults Ġon Ġthe Ġtest Ġset Ġare Ġstatistically Ġsignificant Ġ( ĠIN LINE FORM 0 Ġ) Ġwith Ġrespect Ġto Ġthe Ġbaseline . ĠWhile Ġknowledge Ġof Ġrelative Ġcomposition ality Ġconsistently Ġhelps , Ġthe Ġimprovements Ġusing Ġthe Ġcontext - dependent Ġscoring Ġmodels , Ġespecially Ġwith Ġthe ĠIN LINE FORM 1 Ġconc aten ation Ġmodel , Ġare Ġnoticeably Ġbetter . </s> Related ĠWork </s> There Ġhas Ġbeen Ġa Ġlarge Ġamount Ġof Ġwork Ġon Ġcompos itional Ġmodels Ġthat Ġoperate Ġon Ġvector Ġrepresentations Ġof Ġwords . ĠWith Ġsome Ġexceptions ĠB IB REF 16 Ġ, ĠB IB REF 5 Ġ, Ġall Ġof Ġthese Ġapproaches Ġare Ġlex ical ized Ġi . e ., Ġparameters Ġ( gener ally Ġin Ġthe Ġform Ġof Ġvectors , Ġmat rices , Ġor Ġtens ors ) Ġfor Ġspecific Ġwords Ġare Ġlearned , Ġwhich Ġworks Ġwell Ġfor Ġfrequently Ġoccurring Ġwords Ġbut Ġfails Ġwhen Ġdealing Ġwith Ġcompositions Ġof Ġarbitrary Ġword Ġsequences Ġcontaining Ġinf requent Ġwords . ĠThe Ġfunctions Ġare Ġeither Ġlearned Ġwith Ġa Ġneural Ġnetwork Ġarchitecture ĠB IB REF 7 Ġor Ġas Ġa Ġlinear Ġregression ĠB IB REF 6 Ġ; Ġthe Ġlatter Ġrequire Ġphrase Ġrepresentations Ġextracted Ġdirectly Ġfrom Ġthe Ġcorpus Ġfor Ġsupervision , Ġwhich Ġcan Ġbe Ġcomput ationally Ġexpensive Ġand Ġstatistically Ġinefficient . ĠIn Ġcontrast , Ġwe Ġobtain Ġthis Ġinformation Ġthrough Ġmany - to - one ĠP PD B Ġm appings . ĠMost Ġof Ġthese Ġmodels Ġalso Ġrequire Ġadditional Ġsynt actic ĠB IB REF 31 Ġor Ġsemantic ĠB IB REF 15 Ġ, ĠB IB REF 14 Ġresources ; Ġon Ġthe Ġother Ġhand , Ġour Ġproposed Ġapproach Ġonly Ġrequires Ġa Ġshallow Ġsynt actic Ġparse Ġ( POS Ġtags ). ĠRecent Ġefforts Ġto Ġmake Ġthese Ġmodels Ġmore Ġpractical ĠB IB REF 32 Ġattempt Ġto Ġreduce Ġtheir Ġstatistically Ġcomplex Ġand Ġoverly - param et ri zed Ġnature , Ġbut Ġwith Ġthe Ġexception Ġof ĠZ anz otto 2010 , Ġwho Ġpropose Ġa Ġway Ġto Ġextract Ġcompos itional Ġfunction Ġtraining Ġexamples Ġfrom Ġa Ġdictionary , Ġthese Ġmodels Ġgenerally Ġrequire Ġhuman - annot ated Ġdata Ġto Ġwork . </s> Most Ġmodels Ġthat Ġscore Ġthe Ġrelative Ġ( non -) Ġcomposition ality Ġof Ġphrases Ġdo Ġso Ġin Ġa Ġcontext - independent Ġmanner . ĠA Ġcentral Ġidea Ġis Ġto Ġreplace Ġphrase Ġconstituents Ġwith Ġsem antically - related Ġwords Ġand Ġcompute Ġthe Ġsimilarity Ġof Ġthe Ġnew Ġphrase Ġto Ġthe Ġoriginal ĠB IB REF 22 Ġ, ĠB IB REF 33 Ġor Ġmake Ġuse Ġof Ġa Ġvariety Ġof Ġlex ical Ġassociation Ġmeasures ĠB IB REF 10 Ġ, ĠB IB REF 34 Ġ. ĠSp or led er 2009 Ġhowever , Ġdo Ġmake Ġuse Ġof Ġcontext Ġin Ġa Ġtoken - based Ġapproach , Ġwhere Ġthe Ġcontext Ġin Ġwhich Ġa Ġphrase Ġoccurs Ġas Ġwell Ġas Ġthe Ġphrase Ġitself Ġis Ġmodeled Ġas Ġa Ġlex ical Ġchain , Ġand Ġthe Ġcohesion Ġof Ġthe Ġchain Ġis Ġmeasured Ġas Ġan Ġindicator Ġof Ġa Ġphrase 's Ġcomposition ality . ĠCo hesion Ġis Ġcomputed Ġusing Ġa Ġweb Ġsearch Ġengine - based Ġmeasure , Ġwhereas Ġwe Ġuse Ġa Ġprob abil istic Ġmodel Ġof Ġcontext Ġgiven Ġa Ġphrase Ġrepresentation . ĠHerman n 2012 Ġpropose Ġa ĠBay esian Ġgener ative Ġmodel Ġthat Ġis Ġalso Ġcontext - based , Ġbut Ġlearning Ġand Ġinference Ġis Ġdone Ġthrough Ġa Ġrelatively Ġexpensive ĠGibbs Ġsampling Ġscheme . </s> In Ġthe Ġcontext Ġof ĠMT , ĠZhang 2008 b Ġpresent Ġa ĠBay esian Ġmodel Ġthat Ġlearns Ġnon - com pos itional Ġphrases Ġfrom Ġa Ġsynchron ous Ġparse Ġtree Ġof Ġa Ġsentence Ġpair . ĠHowever , Ġthe Ġprimary Ġaim Ġof Ġtheir Ġwork Ġis Ġphrase Ġextraction Ġfor ĠMT , Ġand Ġthe Ġnon - com pos itional Ġconstraints Ġare Ġonly Ġapplied Ġto Ġmake Ġthe Ġspace Ġof Ġphrase Ġpairs Ġmore Ġtract able Ġwhen Ġboot stra pping Ġtheir Ġphr as al Ġparser Ġfrom Ġtheir Ġword - based Ġparser . ĠIn Ġcontrast , Ġwe Ġscore Ġevery Ġphrase Ġthat Ġis Ġextracted Ġwith Ġthe Ġstandard Ġphrase Ġextraction Ġhe ur istics ĠB IB REF 29 Ġ, Ġallowing Ġthe Ġdec oder Ġto Ġmake Ġthe Ġfinal Ġdecision Ġon Ġthe Ġimpact Ġof Ġcomposition ality Ġscores Ġin Ġtranslation . ĠThus , Ġour Ġwork Ġis Ġmore Ġsimilar Ġto ĠX ion g 2010 , Ġwho Ġpropose Ġmaximum Ġentropy Ġclass ifiers Ġthat Ġmark Ġpositions Ġbetween Ġwords Ġin Ġa Ġsentence Ġas Ġbeing Ġa Ġphrase Ġboundary Ġor Ġnot , Ġand Ġintegrate Ġthese Ġscores Ġas Ġadditional Ġfeatures Ġin Ġan ĠMT Ġsystem . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġpresented Ġtwo Ġnew Ġsources Ġof Ġinformation Ġfor Ġcomposition ality Ġmodeling Ġand Ġscoring , Ġparaph rase Ġinformation Ġand Ġcontext . ĠFor Ġmodeling , Ġwe Ġshowed Ġthat Ġthe Ġparaph rase - learn ed Ġcompos itional Ġrepresentations Ġperforms Ġas Ġwell Ġon Ġa Ġphrase Ġsimilarity Ġtask Ġas Ġthe Ġaverage Ġhuman Ġannot ator . ĠFor Ġscoring , Ġthe Ġimportance Ġof Ġcontext Ġwas Ġshown Ġthrough Ġthe Ġcomparison Ġof Ġcontext - independent Ġand Ġdependent Ġmodels . ĠImprovements Ġby Ġthe Ġcontext - dependent Ġmodel Ġon Ġan Ġextr ins ic Ġmachine Ġtranslation Ġtask Ġcorrobor ate Ġthe Ġutility Ġof Ġthese Ġadditional Ġknowledge Ġsources . ĠWe Ġhope Ġthat Ġthis Ġwork Ġencourages Ġfurther Ġresearch Ġin Ġmaking Ġcompos itional Ġsemantic Ġapproaches Ġapplicable Ġin Ġdownstream Ġtasks .