antonym and synonym pairs, collected from WordNet BIBREF9 and Wordnik
EXTRACTIVE
English Wikipedia dump from June 2016
EXTRACTIVE
For training the models, neural networks require a large amount of training data. We use the existing large-scale antonym and synonym pairs previously used by Nguyen:16. Originally, the data pairs were collected from WordNet BIBREF9 and Wordnik.
We use the English Wikipedia dump from June 2016 as the corpus resource for our methods and baselines. For parsing the corpus, we rely on spaCy. For the lemma embeddings, we rely on the word embeddings of the dLCE model BIBREF10 which is the state-of-the-art vector representation for distinguishing antonyms from synonyms. We re-implemented this cutting-edge model on Wikipedia with 100 dimensions, and then make use of the dLCE word embeddings for initialization the lemma embeddings. The embeddings of POS tags, dependency labels, distance labels, and out-of-vocabulary lemmas are initialized randomly. The number of dimensions is set to 10 for the embeddings of POS tags, dependency labels and distance labels. We use the validation sets to tune the number of dimensions for these labels. For optimization, we rely on the cross-entropy loss function and Stochastic Gradient Descent with the Adadelta update rule BIBREF11 . For training, we use the Theano framework BIBREF12 . Regularization is applied by a dropout of 0.5 on each of component's embeddings (dropout rate is tuned on the validation set). We train the models with 40 epochs and update all embeddings during training.
1701.02962
Introduction </s> Ant ony my Ġand Ġsyn ony my Ġrepresent Ġlex ical Ġsemantic Ġrelations Ġthat Ġare Ġcentral Ġto Ġthe Ġorganization Ġof Ġthe Ġmental Ġlex icon ĠB IB REF 0 Ġ. ĠWhile Ġan ton y my Ġis Ġdefined Ġas Ġthe Ġopp osit eness Ġbetween Ġwords , Ġsyn ony my Ġrefers Ġto Ġwords Ġthat Ġare Ġsimilar Ġin Ġmeaning ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠFrom Ġa Ġcomputational Ġpoint Ġof Ġview , Ġdistinguishing Ġbetween Ġan ton y my Ġand Ġsyn ony my Ġis Ġimportant Ġfor ĠN LP Ġapplications Ġsuch Ġas ĠMachine ĠTranslation Ġand ĠText ual ĠEnt ail ment , Ġwhich Ġgo Ġbeyond Ġa Ġgeneral Ġnotion Ġof Ġsemantic Ġrelated ness Ġand Ġrequire Ġto Ġidentify Ġspecific Ġsemantic Ġrelations . ĠHowever , Ġdue Ġto Ġinterchangeable Ġsubstitution , Ġan ton y ms Ġand Ġsyn onyms Ġoften Ġoccur Ġin Ġsimilar Ġcontexts , Ġwhich Ġmakes Ġit Ġchallenging Ġto Ġautomatically Ġdistinguish Ġbetween Ġthem . </s> Two Ġfamilies Ġof Ġapproaches Ġto Ġdifferentiate Ġbetween Ġan ton y ms Ġand Ġsyn onyms Ġare Ġpredomin ent Ġin ĠN LP . ĠBoth Ġmake Ġuse Ġof Ġdistribution al Ġvector Ġrepresentations , Ġrelying Ġon Ġthe Ġdistribution al Ġhypothesis ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġthat Ġwords Ġwith Ġsimilar Ġdistributions Ġhave Ġrelated Ġmeanings : Ġco - occ urrence Ġmodels Ġand Ġpattern - based Ġmodels . ĠThese Ġdistribution al Ġsemantic Ġmodels Ġ( DS Ms ) Ġoffer Ġa Ġmeans Ġto Ġrepresent Ġmeaning Ġvectors Ġof Ġwords Ġor Ġword Ġpairs , Ġand Ġto Ġdetermine Ġtheir Ġsemantic Ġrelated ness ĠB IB REF 5 Ġ. </s> In Ġco - occ urrence Ġmodels , Ġeach Ġword Ġis Ġrepresented Ġby Ġa Ġweighted Ġfeature Ġvector , Ġwhere Ġfeatures Ġtypically Ġcorrespond Ġto Ġwords Ġthat Ġco - occ ur Ġin Ġparticular Ġcontexts . ĠWhen Ġusing Ġword Ġembed d ings , Ġthese Ġmodels Ġrely Ġon Ġneural Ġmethods Ġto Ġrepresent Ġwords Ġas Ġlow - dimensional Ġvectors . ĠTo Ġcreate Ġthe Ġword Ġembed d ings , Ġthe Ġmodels Ġeither Ġmake Ġuse Ġof Ġneural - based Ġtechniques , Ġsuch Ġas Ġthe Ġskip - gram Ġmodel ĠB IB REF 6 Ġ, Ġor Ġuse Ġmatrix Ġfactor ization ĠB IB REF 7 Ġthat Ġbuilds Ġword Ġembed d ings Ġby Ġfactor izing Ġword - context Ġco - occ urrence Ġmat rices . ĠIn Ġcomparison Ġto Ġstandard Ġco - occ urrence Ġvector Ġrepresentations , Ġword Ġembed d ings Ġaddress Ġthe Ġproblematic Ġsp arsity Ġof Ġword Ġvectors Ġand Ġhave Ġachieved Ġimpressive Ġresults Ġin Ġmany ĠN LP Ġtasks Ġsuch Ġas Ġword Ġsimilarity Ġ( e . g ., ĠPen nington : 14 ), Ġrelation Ġclassification Ġ( e . g ., ĠTh ang V u : 16 ), Ġand Ġan ton ym - syn onym Ġdistinction Ġ( e . g ., ĠNguyen : 16 ). </s> In Ġpattern - based Ġmodels , Ġvector Ġrepresentations Ġmake Ġuse Ġof Ġlex ico - sy nt actic Ġsurface Ġpatterns Ġto Ġdistinguish Ġbetween Ġthe Ġrelations Ġof Ġword Ġpairs . ĠFor Ġexample , ĠJust eson / Kat z : 91 Ġsuggested Ġthat Ġadject ival Ġoppos ites Ġco - occ ur Ġwith Ġeach Ġother Ġin Ġspecific Ġlinear Ġsequences , Ġsuch Ġas Ġbetween ĠX Ġand ĠY . ĠHear st : 92 Ġdetermined Ġsurface Ġpatterns , Ġe . g ., ĠX Ġsuch Ġas ĠY , Ġto Ġidentify Ġnominal Ġhyper ny ms . ĠLin 2003 Ġproposed Ġtwo Ġtextual Ġpatterns Ġindicating Ġsemantic Ġincomp atibility , Ġfrom ĠX Ġto ĠY Ġand Ġeither ĠX Ġor ĠY , Ġto Ġdistinguish Ġoppos ites Ġfrom Ġsem antically Ġsimilar Ġwords . ĠRoth 2014 Ġproposed Ġa Ġmethod Ġthat Ġcombined Ġpatterns Ġwith Ġdiscourse Ġmarkers Ġfor Ġclass ifying Ġparad igmatic Ġrelations Ġincluding Ġan ton y my , Ġsyn ony my , Ġand Ġhyper ny my . ĠRecently , ĠSchwartz : 15 Ġused Ġtwo Ġprominent Ġpatterns Ġfrom ĠLin 2003 Ġto Ġlearn Ġword Ġembed d ings Ġthat Ġdistinguished Ġan ton y ms Ġfrom Ġsimilar Ġwords Ġin Ġdetermining Ġdegrees Ġof Ġsimilarity Ġand Ġword Ġanalogy . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġnovel Ġpattern - based Ġneural Ġmethod ĠAnt Syn NET Ġto Ġdistinguish Ġan ton y ms Ġfrom Ġsyn onyms . ĠWe Ġhypothes ize Ġthat Ġan ton ym ous Ġword Ġpairs Ġco - occ ur Ġwith Ġeach Ġother Ġin Ġlex ico - sy nt actic Ġpatterns Ġwithin Ġa Ġsentence Ġmore Ġoften Ġthan Ġwould Ġbe Ġexpected Ġby Ġsynonymous Ġpairs . ĠThis Ġhypothesis Ġis Ġinspired Ġby Ġcorpus - based Ġstudies Ġon Ġan ton y my Ġand Ġsyn ony my . ĠAmong Ġothers , ĠCharles 1989 Ġsuggested Ġthat Ġadject ival Ġoppos ites Ġco - occ ur Ġin Ġpatterns ; ĠFell baum : 95 Ġstated Ġthat Ġnominal Ġand Ġverbal Ġoppos ites Ġco - occ ur Ġin Ġthe Ġsame Ġsentence Ġsignificantly Ġmore Ġoften Ġthan Ġchance ; ĠLin 2003 Ġargued Ġthat Ġif Ġtwo Ġwords Ġappear Ġin Ġclear Ġan ton ym Ġpatterns , Ġthey Ġare Ġunlikely Ġto Ġrepresent Ġsynonymous Ġpair . </s> We Ġstart Ġout Ġby Ġinducing Ġpatterns Ġbetween ĠX Ġand ĠY Ġfrom Ġa Ġlarge - scale Ġweb Ġcorpus , Ġwhere ĠX Ġand ĠY Ġrepresent Ġtwo Ġwords Ġof Ġan Ġan ton ym Ġor Ġsyn onym Ġword Ġpair , Ġand Ġthe Ġpattern Ġis Ġderived Ġfrom Ġthe Ġsimple Ġpaths Ġbetween ĠX Ġand ĠY Ġin Ġa Ġsynt actic Ġparse Ġtree . ĠEach Ġnode Ġin Ġthe Ġsimple Ġpath Ġcombines Ġlex ical Ġand Ġsynt actic Ġinformation ; Ġin Ġaddition , Ġwe Ġsuggest Ġa Ġnovel Ġfeature Ġfor Ġthe Ġpatterns , Ġi . e ., Ġthe Ġdistance Ġbetween Ġthe Ġtwo Ġwords Ġalong Ġthe Ġsynt actic Ġpath . ĠAll Ġpattern Ġfeatures Ġare Ġfed Ġinto Ġa Ġrecurrent Ġneural Ġnetwork Ġwith Ġlong Ġshort - term Ġmemory Ġ( L ST M ) Ġunits ĠB IB REF 8 Ġ, Ġwhich Ġencode Ġthe Ġpatterns Ġas Ġvector Ġrepresentations . ĠAfterwards , Ġthe Ġvector Ġrepresentations Ġof Ġthe Ġpatterns Ġare Ġused Ġin Ġa Ġclass ifier Ġto Ġdistinguish Ġbetween Ġan ton y ms Ġand Ġsyn onyms . ĠThe Ġresults Ġfrom Ġexperiments Ġshow Ġthat ĠAnt Syn NET Ġimproves Ġthe Ġperformance Ġover Ġprior Ġpattern - based Ġmethods . ĠFurthermore , Ġthe Ġimplementation Ġof Ġour Ġmodels Ġis Ġmade Ġpublicly Ġavailable . </s> The Ġremainder Ġof Ġthis Ġpaper Ġis Ġorganized Ġas Ġfollows : ĠIn ĠSection Ġ" Related ĠWork " Ġ, Ġwe Ġpresent Ġprevious Ġwork Ġdistinguishing Ġan ton y ms Ġand Ġsyn onyms . ĠSection Ġ" Ant Syn NET : ĠL ST M - based ĠAnt onym - Syn onym ĠDist inction " Ġdescribes Ġour Ġproposed ĠAnt Syn NET Ġmodel . ĠWe Ġpresent Ġthe Ġinduction Ġof Ġthe Ġpatterns Ġ( Section Ġ" Bas eline ĠModels " Ġ), Ġdescribe Ġthe Ġrecurrent Ġneural Ġnetwork Ġwith Ġlong Ġshort - term Ġmemory Ġunits Ġwhich Ġis Ġused Ġto Ġencode Ġpatterns Ġwithin Ġa Ġvector Ġrepresentation Ġ( Section Ġ" Rec urrent ĠNeural ĠNetwork Ġwith ĠLong ĠShort - Term ĠMemory ĠUnits " Ġ), Ġand Ġdescribe Ġtwo Ġmodels Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms : Ġthe Ġpure Ġpattern - based Ġmodel Ġ( Section ĠUID 11 Ġ) Ġand Ġthe Ġcombined Ġmodel Ġ( Section ĠUID 13 Ġ). ĠAfter Ġintroducing Ġtwo Ġbas elines Ġin ĠSection Ġ" Bas eline ĠModels " Ġ, Ġwe Ġdescribe Ġour Ġdataset , Ġexperimental Ġsettings , Ġresults Ġof Ġour Ġmethods , Ġthe Ġeffects Ġof Ġthe Ġnewly Ġproposed Ġdistance Ġfeature , Ġand Ġthe Ġeffects Ġof Ġthe Ġvarious Ġtypes Ġof Ġword Ġembed d ings . ĠSection Ġ" Conclusion " Ġconcludes Ġthe Ġpaper . </s> Ant Syn NET : ĠL ST M - based ĠAnt onym - Syn onym ĠDist inction </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe ĠAnt Syn NET Ġmodel , Ġusing Ġa Ġpattern - based ĠL ST M Ġfor Ġdistinguishing Ġan ton y ms Ġfrom Ġsyn onyms . ĠWe Ġfirst Ġpresent Ġthe Ġinduction Ġof Ġpatterns Ġfrom Ġa Ġparsed Ġcorpus Ġ( Section Ġ" Bas eline ĠModels " Ġ). ĠSection Ġ" Rec urrent ĠNeural ĠNetwork Ġwith ĠLong ĠShort - Term ĠMemory ĠUnits " Ġthen Ġdescribes Ġhow Ġwe Ġutilize Ġthe Ġrecurrent Ġneural Ġnetwork Ġwith Ġlong Ġshort - term Ġmemory Ġunits Ġto Ġencode Ġthe Ġpatterns Ġas Ġvector Ġrepresentation . ĠFinally , Ġwe Ġpresent Ġthe ĠAnt Syn NET Ġmodel Ġand Ġtwo Ġapproaches Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms Ġ( Section Ġ" The ĠProp osed ĠAnt Syn NET ĠModel " Ġ). </s> Ind uction Ġof ĠPatterns </s> Corp us - based Ġstudies Ġon Ġan ton y my Ġhave Ġsuggested Ġthat Ġoppos ites Ġco - occ ur Ġwith Ġeach Ġother Ġwithin Ġa Ġsentence Ġsignificantly Ġmore Ġoften Ġthan Ġwould Ġbe Ġexpected Ġby Ġchance . ĠOur Ġmethod Ġthus Ġmakes Ġuse Ġof Ġpatterns Ġas Ġthe Ġmain Ġindicators Ġof Ġword Ġpair Ġco - occ urrence , Ġto Ġenforce Ġa Ġdistinction Ġbetween Ġan ton y ms Ġand Ġsyn onyms . ĠFigure Ġ1 Ġshows Ġa Ġsynt actic Ġparse Ġtree Ġof Ġthe Ġsentence ĠâĢ ľ My Ġold Ġvillage Ġhas Ġbeen Ġprovided Ġwith Ġthe Ġnew Ġservices âĢ Ŀ . ĠFollowing Ġthe Ġcharacter izations Ġof Ġa Ġtree Ġin Ġgraph Ġtheory , Ġany Ġtwo Ġnodes Ġ( vert ices ) Ġof Ġa Ġtree Ġare Ġconnected Ġby Ġa Ġsimple Ġpath Ġ( or Ġone Ġunique Ġpath ). ĠThe Ġsimple Ġpath Ġis Ġthe Ġshortest Ġpath Ġbetween Ġany Ġtwo Ġnodes Ġin Ġa Ġtree Ġand Ġdoes Ġnot Ġcontain Ġrepeated Ġnodes . ĠIn Ġthe Ġexample , Ġthe Ġlex ico - sy nt actic Ġtree Ġpattern Ġof Ġthe Ġan ton ym ous Ġpair Ġold âĢĵ new Ġis Ġdetermined Ġby Ġfinding Ġthe Ġsimple Ġpath Ġ( in Ġred ) Ġfrom Ġthe Ġle mma Ġold Ġto Ġthe Ġle mma Ġnew . ĠIt Ġfocuses Ġon Ġthe Ġmost Ġrelevant Ġinformation Ġand Ġignores Ġirrelevant Ġinformation Ġwhich Ġdoes Ġnot Ġappear Ġin Ġthe Ġsimple Ġpath Ġ( i . e ., Ġhas , Ġbeen ). ĠThe Ġexample Ġpattern Ġbetween ĠX Ġ= Ġold Ġand ĠY Ġ= Ġnew Ġin ĠFigure Ġ1 Ġis Ġrepresented Ġas Ġfollows : ĠX / JJ / am od / 2 ĠâĢĵ Ġvillage / NN / ns ub j / 1 ĠâĢĵ Ġprovide / V BN / RO OT / 0 ĠâĢĵ Ġwith / IN / prep / 1 ĠâĢĵ Ġservice / NN S / p obj / 2 ĠâĢĵ ĠY / JJ / am od / 3 . </s> The Ġpath Ġpatterns Ġmake Ġuse Ġof Ġfour Ġfeatures Ġto Ġrepresent Ġeach Ġnode Ġin Ġthe Ġsyntax Ġtree : Ġle mma , Ġpart - of - speech Ġ( POS ) Ġtag , Ġdependency Ġlabel Ġand Ġdistance Ġlabel . ĠThe Ġle mma Ġfeature Ġcaptures Ġthe Ġlex ical Ġinformation Ġof Ġwords Ġin Ġthe Ġsentence , Ġwhile Ġthe ĠPOS Ġand Ġdependency Ġfeatures Ġcapture Ġthe Ġmorph o - sy nt actic Ġinformation Ġof Ġthe Ġsentence . ĠThe Ġdistance Ġlabel Ġmeasures Ġthe Ġpath Ġdistance Ġbetween Ġthe Ġtarget Ġword Ġnodes Ġin Ġthe Ġsynt actic Ġtree . ĠEach Ġstep Ġbetween Ġa Ġparent Ġand Ġa Ġchild Ġnode Ġrepresents Ġa Ġdistance Ġof Ġ1 ; Ġand Ġthe Ġancestor Ġnodes Ġof Ġthe Ġremaining Ġnodes Ġin Ġthe Ġpath Ġare Ġrepresented Ġby Ġa Ġdistance Ġof Ġ0 . ĠFor Ġexample , Ġthe Ġnode Ġprovided Ġis Ġan Ġancestor Ġnode Ġof Ġthe Ġsimple Ġpath Ġfrom Ġold Ġto Ġnew . ĠThe Ġdistances Ġfrom Ġthe Ġnode Ġprovided Ġto Ġthe Ġnodes Ġvillage Ġand Ġold Ġare Ġ1 Ġand Ġ2 , Ġrespectively . </s> The Ġvector Ġrepresentation Ġof Ġeach Ġnode Ġconc aten ates Ġthe Ġfour - feature Ġvectors Ġas Ġfollows : Ġ$ Ġ\ vec { v } _{ node } Ġ= Ġ[ \ vec { v } _{ lem ma } Ġ\ opl us Ġ\ vec { v } _{ pos } Ġ\ opl us Ġ\ vec { v } _{ dep } Ġ\ opl us Ġ\ vec { v } _{ dist } ] Ġ$ </s> where Ġ$\ vec { v } _{ lem ma }, Ġ\ vec { v } _{ pos }, Ġ\ vec { v } _{ dep }, Ġ\ vec { v } _{ dist } $ Ġrepresent Ġthe Ġembed d ings Ġof Ġthe Ġle mma , ĠPOS Ġtag , Ġdependency Ġlabel Ġand Ġdistance Ġlabel , Ġrespectively ; Ġand Ġthe Ġ$\ opl us Ġ$ Ġdenotes Ġthe Ġconc aten ation Ġoperation . </s> For Ġa Ġpattern Ġ$ p $ Ġwhich Ġis Ġconstructed Ġby Ġthe Ġsequence Ġof Ġnodes Ġ$ n _ 1 , Ġn _ 2 , Ġ... , Ġn _ k $ Ġ, Ġthe Ġpattern Ġrepresentation Ġof Ġ$ p $ Ġis Ġa Ġsequence Ġof Ġvectors : Ġ$ p =[ \ vec { n } _ 1 , Ġ\ vec { n } _ 2 , Ġ... , Ġ\ vec { n } _ k ] $ Ġ. ĠThe Ġpattern Ġvector Ġ$\ vec { v } _ p $ Ġis Ġthen Ġencoded Ġby Ġapplying Ġa Ġrecurrent Ġneural Ġnetwork . </s> Rec urrent ĠNeural ĠNetwork Ġwith ĠLong ĠShort - Term ĠMemory ĠUnits </s> A Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġis Ġsuitable Ġfor Ġmodeling Ġsequential Ġdata Ġby Ġa Ġvector Ġrepresentation . ĠIn Ġour Ġmethods , Ġwe Ġuse Ġa Ġlong Ġshort - term Ġmemory Ġ( L ST M ) Ġnetwork , Ġa Ġvariant Ġof Ġa Ġrecurrent Ġneural Ġnetwork Ġto Ġencode Ġpatterns , Ġfor Ġthe Ġfollowing Ġreasons . ĠGiven Ġa Ġsequence Ġof Ġwords Ġ$ p =[ n _ 1 , n _ 2 ,... , n _ k ] $ Ġas Ġinput Ġdata , Ġan ĠR NN Ġprocesses Ġeach Ġword Ġ$ n _ t $ Ġat Ġa Ġtime , Ġand Ġreturns Ġa Ġvector Ġof Ġstate Ġ$ h _ k $ Ġfor Ġthe Ġcomplete Ġinput Ġsequence . ĠFor Ġeach Ġtime Ġstep Ġ$ t $ Ġ, Ġthe ĠR NN Ġupdates Ġan Ġinternal Ġmemory Ġstate Ġ$ h _ t $ Ġwhich Ġdepends Ġon Ġthe Ġcurrent Ġinput Ġ$ n _ t $ Ġand Ġthe Ġprevious Ġstate Ġ$ h _{ t - 1 } $ Ġ. ĠYet , Ġif Ġthe Ġsequential Ġinput Ġis Ġa Ġlong - term Ġdependency , Ġan ĠR NN Ġfaces Ġthe Ġproblem Ġof Ġgradient Ġvanishing Ġor Ġexploding , Ġleading Ġto Ġdifficulties Ġin Ġtraining Ġthe Ġmodel . </s> L ST M Ġunits Ġaddress Ġthese Ġproblems . ĠThe Ġunderlying Ġidea Ġof Ġan ĠL ST M Ġis Ġto Ġuse Ġan Ġadaptive Ġg ating Ġmechanism Ġto Ġdecide Ġon Ġthe Ġdegree Ġthat ĠL ST M Ġunits Ġkeep Ġthe Ġprevious Ġstate Ġand Ġmemor ize Ġthe Ġextracted Ġfeatures Ġof Ġthe Ġcurrent Ġinput . ĠMore Ġspecifically , Ġan ĠL ST M Ġcomprises Ġfour Ġcomponents : Ġan Ġinput Ġgate Ġ$ i _ t $ Ġ, Ġa Ġforget Ġgate Ġ$ f _ t $ Ġ, Ġan Ġoutput Ġgate Ġ$ o _ t $ Ġ, Ġand Ġa Ġmemory Ġcell Ġ$ c _ t $ Ġ. ĠThe Ġstate Ġof Ġan ĠL ST M Ġat Ġeach Ġtime Ġstep Ġ$ t $ Ġis Ġformal ized Ġas Ġfollows : Ġ$ Ġ\ begin { array }{ l } Ġi _ t Ġ= Ġ\ s igma Ġ( W _ i Ġ\ cd ot Ġx _ t Ġ+ ĠU _ i Ġ\ cd ot Ġh _{ t - 1 } Ġ+ Ġb _ i ) Ġ\\ Ġf _ t Ġ= Ġ\ s igma Ġ( W _ f Ġ\ cd ot Ġx _ t Ġ+ ĠU _ f Ġ\ cd ot Ġh _{ t - 1 } Ġ+ Ġb _ f ) Ġ\\ Ġo _ t Ġ= Ġ\ s igma Ġ( W _ o Ġ\ cd ot Ġx _ t Ġ+ ĠU _ o Ġ\ cd ot Ġh _{ t - 1 } Ġ+ Ġb _ o ) Ġ\\ Ġg _ t Ġ= Ġ\ tan h Ġ( W _ c Ġ\ cd ot Ġx _ t Ġ+ ĠU _ c Ġ\ cd ot Ġh _{ t - 1 } Ġ+ Ġb _ c ) Ġ\\ Ġc _ t Ġ= Ġi _ t Ġ\ ot imes Ġg _ t Ġ+ Ġf _ t Ġ\ ot imes Ġc _{ t - 1 } Ġ\ end { array } Ġ$ </s> $ W $ Ġrefers Ġto Ġa Ġmatrix Ġof Ġweights Ġthat Ġprojects Ġinformation Ġbetween Ġtwo Ġlayers ; Ġ$ b $ Ġis Ġa Ġlayer - specific Ġvector Ġof Ġbias Ġterms ; Ġ$\ s igma Ġ$ Ġdenotes Ġthe Ġs igm oid Ġfunction . ĠThe Ġoutput Ġof Ġan ĠL ST M Ġat Ġa Ġtime Ġstep Ġ$ t $ Ġis Ġcomputed Ġas Ġfollows : Ġ$ Ġh _ t Ġ= Ġo _ t Ġ\ ot imes Ġ\ tan h Ġ( c _ t ) Ġ$ </s> where Ġ$\ ot imes Ġ$ Ġdenotes Ġelement - wise Ġmultiplication . ĠIn Ġour Ġmethods , Ġwe Ġrely Ġon Ġthe Ġlast Ġstate Ġ$ h _ k $ Ġto Ġrepresent Ġthe Ġvector Ġ$\ vec { v } _ p $ Ġof Ġa Ġpattern Ġ$ p Ġ= Ġ[ \ vec { n } _ 1 , Ġ\ vec { n } _ 2 ,... , \ vec { n } _ k ] $ Ġ. </s> The ĠProp osed ĠAnt Syn NET ĠModel </s> In Ġthis Ġsection , Ġwe Ġpresent Ġtwo Ġmodels Ġto Ġdistinguish Ġan ton y ms Ġfrom Ġsyn onyms . ĠThe Ġfirst Ġmodel Ġmakes Ġuse Ġof Ġpatterns Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms , Ġby Ġusing Ġan ĠL ST M Ġto Ġencode Ġpatterns Ġas Ġvector Ġrepresentations Ġand Ġthen Ġfeeding Ġthose Ġvectors Ġto Ġa Ġlog istic Ġregression Ġlayer Ġ( Section ĠUID 11 Ġ). ĠThe Ġsecond Ġmodel Ġcreates Ġcombined Ġvector Ġrepresentations Ġof Ġword Ġpairs , Ġwhich Ġconc aten ate Ġthe Ġvectors Ġof Ġthe Ġwords Ġand Ġthe Ġpatterns Ġ( Section ĠUID 13 Ġ). </s> In Ġthis Ġmodel , Ġwe Ġmake Ġuse Ġof Ġa Ġrecurrent Ġneural Ġnetwork Ġwith ĠL ST M Ġunits Ġto Ġencode Ġpatterns Ġcontaining Ġa Ġsequence Ġof Ġnodes . ĠFigure Ġ2 Ġillustrates Ġthe ĠAnt Syn NET Ġmodel . ĠGiven Ġa Ġword Ġpair Ġ$( x , y ) $ Ġ, Ġwe Ġinduce Ġpatterns Ġfor Ġ$( x , y ) $ Ġfrom Ġa Ġcorpus , Ġwhere Ġeach Ġpattern Ġrepresents Ġa Ġpath Ġfrom Ġ$ x $ Ġto Ġ$ y $ Ġ( cf . ĠSection Ġ" Bas eline ĠModels " Ġ). ĠWe Ġthen Ġfeed Ġeach Ġpattern Ġ$ p $ Ġof Ġthe Ġword Ġpair Ġ$( x , y ) $ Ġinto Ġan ĠL ST M Ġto Ġobtain Ġ$\ vec { v } _ p $ Ġ, Ġthe Ġvector Ġrepresentation Ġof Ġthe Ġpattern Ġ$ p $ Ġ( cf . ĠSection Ġ" Rec urrent ĠNeural ĠNetwork Ġwith ĠLong ĠShort - Term ĠMemory ĠUnits " Ġ). ĠFor Ġeach Ġword Ġpair Ġ$( x , y ) $ Ġ, Ġthe Ġvector Ġrepresentation Ġof Ġ$( x , y ) $ Ġis Ġcomputed Ġas Ġfollows : </s> $$ \ vec { v } _{ xy } Ġ= Ġ\ frac {\ sum Ġ\ n ol im its Ġ_ { p Ġ\ in ĠP ( x , y )} Ġ{\ vec { v } _ p Ġ\ cd ot Ġc _ p }} {\ sum Ġ\ n ol im its Ġ_ { p Ġ\ in ĠP ( x , y )} Ġ{ c _ p }} $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> $ \ vec { v } _{ xy } $ Ġrefers Ġto Ġthe Ġvector Ġof Ġthe Ġword Ġpair Ġ$( x , y ) $ Ġ; Ġ$ P ( x , y ) $ Ġis Ġthe Ġset Ġof Ġpatterns Ġcorresponding Ġto Ġthe Ġpair Ġ$( x , y ) $ Ġ; Ġ$ c _ p $ Ġis Ġthe Ġfrequency Ġof Ġthe Ġpattern Ġ$ p $ Ġ. ĠThe Ġvector Ġ$\ vec { v } _{ xy } $ Ġis Ġthen Ġfed Ġinto Ġa Ġlog istic Ġregression Ġlayer Ġwhose Ġtarget Ġis Ġthe Ġclass Ġlabel Ġassociated Ġwith Ġthe Ġpair Ġ$( x , y ) $ Ġ. ĠFinally , Ġthe Ġpair Ġ$( x , y ) $ Ġis Ġpredicted Ġas Ġpositive Ġ( i . e ., Ġan ton ym ous ) Ġword Ġpair Ġif Ġthe Ġprobability Ġof Ġthe Ġprediction Ġfor Ġ$\ vec { v } _{ xy } $ Ġis Ġlarger Ġthan Ġ0 . 5 . </s> Insp ired Ġby Ġthe Ġsupervised Ġdistribution al Ġconc aten ation Ġmethod Ġin ĠBar oni : 12 Ġand Ġthe Ġintegrated Ġpath - based Ġand Ġdistribution al Ġmethod Ġfor Ġhyper ny my Ġdetection Ġin ĠSh w artz : 16 , Ġwe Ġtake Ġinto Ġaccount Ġthe Ġpatterns Ġand Ġdistribution Ġof Ġtarget Ġpairs Ġto Ġcreate Ġtheir Ġcombined Ġvector Ġrepresentations . ĠGiven Ġa Ġword Ġpair Ġ$( x , y ) $ Ġ, Ġthe Ġcombined Ġvector Ġrepresentation Ġof Ġthe Ġpair Ġ$( x , y ) $ Ġis Ġdetermined Ġby Ġusing Ġboth Ġthe Ġco - occ urrence Ġdistribution Ġof Ġthe Ġwords Ġand Ġthe Ġsynt actic Ġpath Ġpatterns : </s> $$ \ vec { v } _{ comb ( x , y )} Ġ= Ġ[ \ vec { v } _ x Ġ\ opl us Ġ\ vec { v } _{ xy } Ġ\ opl us Ġ\ vec { v } _ y ] $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> $ \ vec { v } _{ comb ( x , y )} $ Ġrefers Ġto Ġthe Ġcombined Ġvector Ġof Ġthe Ġword Ġpair Ġ$( x , y ) $ Ġ; Ġ$\ vec { v } _ x $ Ġand Ġ$\ vec { v } _ y $ Ġare Ġthe Ġvectors Ġof Ġword Ġ$ x $ Ġand Ġword Ġ$ y $ Ġ, Ġrespectively ; Ġ$\ vec { v } _{ xy } $ Ġis Ġthe Ġvector Ġof Ġthe Ġpattern Ġthat Ġcorresponds Ġto Ġthe Ġpair Ġ$( x , y ) $ Ġ, Ġcf . ĠSection ĠUID 11 Ġ. ĠSimilar Ġto Ġthe Ġpattern - based Ġmodel , Ġthe Ġcombined Ġvector Ġ$\ vec { v } _{ comb ( x , y )} $ Ġis Ġfed Ġinto Ġthe Ġlog istic Ġregression Ġlayer Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms . </s> Bas eline ĠModels </s> To Ġcompare ĠAnt Syn NET Ġwith Ġbaseline Ġmodels Ġfor Ġpattern - based Ġclassification Ġof Ġan ton y ms Ġand Ġsyn onyms , Ġwe Ġintroduce Ġtwo Ġpattern - based Ġbaseline Ġmethods : Ġthe Ġdistribution al Ġmethod Ġ( Section Ġ" Dist ribution al ĠBas eline " Ġ), Ġand Ġthe Ġdistributed Ġmethod Ġ( Section Ġ" Dist ributed ĠBas eline " Ġ). </s> Dist ribution al ĠBas eline </s> As Ġa Ġfirst Ġbaseline , Ġwe Ġapply Ġthe Ġapproach Ġby ĠRoth 2014 , Ġhence forth ĠR & Si W . ĠThey Ġused Ġa Ġvector Ġspace Ġmodel Ġto Ġrepresent Ġpairs Ġof Ġwords Ġby Ġa Ġcombination Ġof Ġstandard Ġlex ico - sy nt actic Ġpatterns Ġand Ġdiscourse Ġmarkers . ĠIn Ġaddition Ġto Ġthe Ġpatterns , Ġthe Ġdiscourse Ġmarkers Ġadded Ġinformation Ġto Ġexpress Ġdiscourse Ġrelations , Ġwhich Ġin Ġturn Ġmay Ġindicate Ġthe Ġspecific Ġsemantic Ġrelation Ġbetween Ġthe Ġtwo Ġwords Ġin Ġa Ġword Ġpair . ĠFor Ġexample , Ġcontrast Ġrelations Ġmight Ġindicate Ġan ton y my , Ġwhereas Ġelabor ations Ġmay Ġindicate Ġsyn ony my Ġor Ġhyp ony my . </s> Michael ĠRoth , Ġthe Ġfirst Ġauthor Ġof ĠR & Si W , Ġkindly Ġcomputed Ġthe Ġrelation Ġclassification Ġresults Ġof Ġthe Ġpattern âĢĵ disc ourse Ġmodel Ġfor Ġour Ġtest Ġsets . ĠThe Ġweights Ġbetween Ġmarker - based Ġand Ġpattern - based Ġmodels Ġwere Ġtuned Ġon Ġthe Ġvalidation Ġsets ; Ġother Ġhyper param eters Ġwere Ġset Ġexactly Ġas Ġdescribed Ġby Ġthe ĠR & Si W Ġmethod . </s> Dist ributed ĠBas eline </s> The ĠSP Ġmethod Ġproposed Ġby ĠSchwartz : 15 Ġuses Ġsymm etric Ġpatterns Ġfor Ġgenerating Ġword Ġembed d ings . ĠIn Ġthis Ġwork , Ġthe Ġauthors Ġapplied Ġan Ġun super vised Ġalgorithm Ġfor Ġthe Ġautomatic Ġextraction Ġof Ġsymm etric Ġpatterns Ġfrom Ġplain Ġtext . ĠThe Ġsymm etric Ġpatterns Ġwere Ġdefined Ġas Ġa Ġsequence Ġof Ġ3 - 5 Ġtokens Ġconsisting Ġof Ġexactly Ġtwo Ġwild cards Ġand Ġ1 - 3 Ġwords . ĠThe Ġpatterns Ġwere Ġfiltered Ġbased Ġon Ġtheir Ġfrequencies , Ġsuch Ġthat Ġthe Ġresulting Ġpattern Ġset Ġcontained Ġ11 Ġpatterns . ĠFor Ġgenerating Ġword Ġembed d ings , Ġa Ġmatrix Ġof Ġco - occ urrence Ġcounts Ġbetween Ġpatterns Ġand Ġwords Ġin Ġthe Ġvocabulary Ġwas Ġcomputed , Ġusing Ġpositive Ġpoint - wise Ġmutual Ġinformation . ĠThe Ġsp arsity Ġproblem Ġof Ġvector Ġrepresentations Ġwas Ġaddressed Ġby Ġsmoot hing . ĠFor Ġan ton ym Ġrepresentation , Ġthe Ġauthors Ġrelied Ġon Ġtwo Ġpatterns Ġsuggested Ġby ĠLin 2003 Ġto Ġconstruct Ġword Ġembed d ings Ġcontaining Ġan Ġan ton ym Ġparameter Ġthat Ġcan Ġbe Ġturned Ġon Ġin Ġorder Ġto Ġrepresent Ġan ton y ms Ġas Ġdiss imilar , Ġand Ġthat Ġcan Ġbe Ġturned Ġoff Ġto Ġrepresent Ġan ton y ms Ġas Ġsimilar . </s> To Ġapply Ġthe ĠSP Ġmethod Ġto Ġour Ġdata , Ġwe Ġmake Ġuse Ġof Ġthe Ġpre - trained ĠSP Ġembed d ings Ġwith Ġ500 Ġdimensions . ĠWe Ġcalculate Ġthe Ġcos ine Ġsimilarity Ġof Ġword Ġpairs Ġand Ġthen Ġuse Ġa ĠSupport ĠVector ĠMachine Ġwith ĠRad ial ĠBas is ĠFunction Ġkernel Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms . </s> Dat as et </s> For Ġtraining Ġthe Ġmodels , Ġneural Ġnetworks Ġrequire Ġa Ġlarge Ġamount Ġof Ġtraining Ġdata . ĠWe Ġuse Ġthe Ġexisting Ġlarge - scale Ġan ton ym Ġand Ġsyn onym Ġpairs Ġpreviously Ġused Ġby ĠNguyen : 16 . ĠOriginally , Ġthe Ġdata Ġpairs Ġwere Ġcollected Ġfrom ĠWord Net ĠB IB REF 9 Ġand ĠWord nik . </s> In Ġorder Ġto Ġinduce Ġpatterns Ġfor Ġthe Ġword Ġpairs Ġin Ġthe Ġdataset , Ġwe Ġidentify Ġthe Ġsentences Ġin Ġthe Ġcorpus Ġthat Ġcontain Ġthe Ġword Ġpair . ĠThere after , Ġwe Ġextract Ġall Ġpatterns Ġfor Ġthe Ġword Ġpair . ĠWe Ġfilter Ġout Ġall Ġpatterns Ġwhich Ġoccur Ġless Ġthan Ġfive Ġtimes ; Ġand Ġwe Ġonly Ġtake Ġinto Ġaccount Ġword Ġpairs Ġthat Ġcontain Ġat Ġleast Ġfive Ġpatterns Ġfor Ġtraining , Ġvalid ating Ġand Ġtesting . ĠFor Ġthe Ġproportion Ġof Ġpositive Ġand Ġnegative Ġpairs , Ġwe Ġkeep Ġa Ġratio Ġof Ġ1 : 1 Ġpositive Ġ( ant onym ) Ġto Ġnegative Ġ( syn onym ) Ġpairs Ġin Ġthe Ġdataset . ĠIn Ġorder Ġto Ġcreate Ġthe Ġsets Ġof Ġtraining , Ġtesting Ġand Ġvalidation Ġdata , Ġwe Ġperform Ġrandom Ġsplitting Ġwith Ġ70 % Ġtrain , Ġ25 % Ġtest , Ġand Ġ5 % Ġvalidation Ġsets . ĠThe Ġfinal Ġdataset Ġcontains Ġthe Ġnumber Ġof Ġword Ġpairs Ġaccording Ġto Ġword Ġclasses Ġdescribed Ġin ĠTable Ġ1 Ġ. ĠMoreover , ĠTable Ġ2 Ġshows Ġthe Ġaverage Ġnumber Ġof Ġpatterns Ġfor Ġeach Ġword Ġpair Ġin Ġour Ġdataset . </s> Exper imental ĠSettings </s> We Ġuse Ġthe ĠEnglish ĠWikipedia Ġdump Ġfrom ĠJune Ġ2016 Ġas Ġthe Ġcorpus Ġresource Ġfor Ġour Ġmethods Ġand Ġbas elines . ĠFor Ġparsing Ġthe Ġcorpus , Ġwe Ġrely Ġon Ġspa Cy . ĠFor Ġthe Ġle mma Ġembed d ings , Ġwe Ġrely Ġon Ġthe Ġword Ġembed d ings Ġof Ġthe Ġd L CE Ġmodel ĠB IB REF 10 Ġwhich Ġis Ġthe Ġstate - of - the - art Ġvector Ġrepresentation Ġfor Ġdistinguishing Ġan ton y ms Ġfrom Ġsyn onyms . ĠWe Ġre - im ple mented Ġthis Ġcutting - edge Ġmodel Ġon ĠWikipedia Ġwith Ġ100 Ġdimensions , Ġand Ġthen Ġmake Ġuse Ġof Ġthe Ġd L CE Ġword Ġembed d ings Ġfor Ġinitialization Ġthe Ġle mma Ġembed d ings . ĠThe Ġembed d ings Ġof ĠPOS Ġtags , Ġdependency Ġlabels , Ġdistance Ġlabels , Ġand Ġout - of - voc abulary Ġle mm as Ġare Ġinitialized Ġrandomly . ĠThe Ġnumber Ġof Ġdimensions Ġis Ġset Ġto Ġ10 Ġfor Ġthe Ġembed d ings Ġof ĠPOS Ġtags , Ġdependency Ġlabels Ġand Ġdistance Ġlabels . ĠWe Ġuse Ġthe Ġvalidation Ġsets Ġto Ġtune Ġthe Ġnumber Ġof Ġdimensions Ġfor Ġthese Ġlabels . ĠFor Ġoptimization , Ġwe Ġrely Ġon Ġthe Ġcross - ent ropy Ġloss Ġfunction Ġand ĠSt och astic ĠGrad ient ĠDes cent Ġwith Ġthe ĠAd ad elta Ġupdate Ġrule ĠB IB REF 11 Ġ. ĠFor Ġtraining , Ġwe Ġuse Ġthe ĠThe ano Ġframework ĠB IB REF 12 Ġ. ĠRegular ization Ġis Ġapplied Ġby Ġa Ġdrop out Ġof Ġ0 . 5 Ġon Ġeach Ġof Ġcomponent 's Ġembed d ings Ġ( drop out Ġrate Ġis Ġtuned Ġon Ġthe Ġvalidation Ġset ). ĠWe Ġtrain Ġthe Ġmodels Ġwith Ġ40 Ġepoch s Ġand Ġupdate Ġall Ġembed d ings Ġduring Ġtraining . </s> Overall ĠResults </s> Table Ġ3 Ġshows Ġthe Ġsignificant Ġperformance Ġof Ġour Ġmodels Ġin Ġcomparison Ġto Ġthe Ġbas elines . ĠCon cerning Ġadject ives , Ġthe Ġtwo Ġproposed Ġmodels Ġsignificantly Ġoutper form Ġthe Ġtwo Ġbas elines : ĠThe Ġperformance Ġof Ġthe Ġbas elines Ġis Ġaround Ġ. 72 Ġfor Ġ$ F _ 1 $ Ġ, Ġand Ġthe Ġcorresponding Ġresults Ġfor Ġthe Ġcombined ĠAnt Syn NET Ġmodel Ġachieve Ġan Ġimprovement Ġof Ġ$ > $ Ġ. 06 . ĠRegarding Ġnoun s , Ġthe Ġimprovement Ġof Ġthe Ġnew Ġmethods Ġis Ġjust Ġ. 02 Ġ$ F _ 1 $ Ġin Ġcomparison Ġto Ġthe ĠR & Si W Ġbaseline , Ġbut Ġwe Ġachieve Ġa Ġmuch Ġbetter Ġperformance Ġin Ġcomparison Ġto Ġthe ĠSP Ġbaseline , Ġan Ġincrease Ġof Ġ. 37 Ġ$ F _ 1 $ Ġ. ĠRegarding Ġverbs , Ġwe Ġdo Ġnot Ġoutper form Ġthe Ġmore Ġadvanced ĠR & Si W Ġbaseline Ġin Ġterms Ġof Ġthe Ġ$ F _ 1 $ Ġscore , Ġbut Ġwe Ġobtain Ġhigher Ġrecall Ġscores . ĠIn Ġcomparison Ġto Ġthe ĠSP Ġbaseline , Ġour Ġmodels Ġstill Ġshow Ġa Ġclear Ġ$ F _ 1 $ Ġimprovement . </s> Overall , Ġour Ġproposed Ġmodels Ġachieve Ġcomparatively Ġhigh Ġrecall Ġscores Ġcompared Ġto Ġthe Ġtwo Ġbas elines . ĠThis Ġstrengthens Ġour Ġhypothesis Ġthat Ġthere Ġis Ġa Ġhigher Ġpossibility Ġfor Ġthe Ġco - occ urrence Ġof Ġan ton ym ous Ġpairs Ġin Ġpatterns Ġover Ġsynonymous Ġpairs Ġwithin Ġa Ġsentence . ĠBecause , Ġwhen Ġthe Ġproposed Ġmodels Ġobtain Ġhigh Ġrecall Ġscores , Ġthe Ġmodels Ġare Ġable Ġto Ġretrieve Ġmost Ġrelevant Ġinformation Ġ( ant onymous Ġpairs ) Ġcorresponding Ġto Ġthe Ġpatterns . ĠRegarding Ġthe Ġlow Ġprecision Ġin Ġthe Ġtwo Ġproposed Ġmodels , Ġwe Ġsampled Ġrandomly Ġ5 Ġpairs Ġin Ġeach Ġpopulation : Ġtrue Ġpositive , Ġtrue Ġnegative , Ġfalse Ġpositive , Ġfalse Ġnegative . ĠWe Ġthen Ġcompared Ġthe Ġoverlap Ġof Ġpatterns Ġfor Ġthe Ġtrue Ġpredictions Ġ( true Ġpositive Ġpairs Ġand Ġtrue Ġnegative Ġpairs ) Ġand Ġthe Ġfalse Ġpredictions Ġ( false Ġpositive Ġpairs Ġand Ġfalse Ġnegative Ġpairs ). ĠWe Ġfound Ġout Ġthat Ġthere Ġis Ġno Ġoverlap Ġbetween Ġpatterns Ġof Ġtrue Ġpredictions ; Ġand Ġthe Ġnumber Ġoverlap Ġbetween Ġpatterns Ġof Ġfalse Ġpredictions Ġis Ġ2 , Ġ2 , Ġand Ġ4 Ġpatterns Ġfor Ġnoun , Ġadjective , Ġand Ġverb Ġclasses , Ġrespectively . ĠThis Ġshows Ġthat Ġthe Ġlow Ġprecision Ġof Ġour Ġmodels Ġstems Ġfrom Ġthe Ġpatterns Ġwhich Ġrepresent Ġboth Ġan ton ym ous Ġand Ġsynonymous Ġpairs . </s> Effect Ġof Ġthe ĠDistance ĠFeature </s> In Ġour Ġmodels , Ġthe Ġnovel Ġdistance Ġfeature Ġis Ġsuccessfully Ġintegrated Ġalong Ġthe Ġsynt actic Ġpath Ġto Ġrepresent Ġlex ico - sy nt actic Ġpatterns . ĠThe Ġintuition Ġbehind Ġthe Ġdistance Ġfeature Ġexploits Ġproperties Ġof Ġtrees Ġin Ġgraph Ġtheory , Ġwhich Ġshow Ġthat Ġthere Ġexist Ġdifferences Ġin Ġthe Ġdegree Ġof Ġrelationship Ġbetween Ġthe Ġparent Ġnode Ġand Ġthe Ġchild Ġnodes Ġ( Ġ$ distance Ġ= Ġ1 $ Ġ) Ġand Ġin Ġthe Ġdegree Ġof Ġrelationship Ġbetween Ġthe Ġancestor Ġnode Ġand Ġthe Ġdescendant Ġnodes Ġ( Ġ$ distance Ġ> Ġ1 $ Ġ). ĠHence , Ġwe Ġuse Ġthe Ġdistance Ġfeature Ġto Ġeffectively Ġcapture Ġthese Ġrelationships . </s> In Ġorder Ġto Ġevaluate Ġthe Ġeffect Ġof Ġour Ġnovel Ġdistance Ġfeature , Ġwe Ġcompare Ġthe Ġdistance Ġfeature Ġto Ġthe Ġdirection Ġfeature Ġproposed Ġby ĠSh w artz : 16 . ĠIn Ġtheir Ġapproach , Ġthe Ġauthors Ġcombined Ġle mma , ĠPOS , Ġdependency , Ġand Ġdirection Ġfeatures Ġfor Ġthe Ġtask Ġof Ġhyper ny m Ġdetection . ĠThe Ġdirection Ġfeature Ġrepresented Ġthe Ġdirection Ġof Ġthe Ġdependency Ġlabel Ġbetween Ġtwo Ġnodes Ġin Ġa Ġpath Ġfrom ĠX Ġto ĠY . </s> For Ġevaluation , Ġwe Ġmake Ġuse Ġof Ġthe Ġsame Ġinformation Ġregarding Ġdataset Ġand Ġpatterns Ġas Ġin ĠSection Ġ" Overall ĠResults " Ġ, Ġand Ġthen Ġreplace Ġthe Ġdistance Ġfeature Ġby Ġthe Ġdirection Ġfeature . ĠThe Ġresults Ġare Ġshown Ġin ĠTable Ġ4 Ġ. ĠThe Ġdistance Ġfeature Ġenhances Ġthe Ġperformance Ġof Ġour Ġproposed Ġmodels Ġmore Ġeffectively Ġthan Ġthe Ġdirection Ġfeature Ġdoes , Ġacross Ġall Ġword Ġclasses . </s> Effect Ġof ĠWord ĠEmb edd ings </s> Our Ġmethods Ġrely Ġon Ġthe Ġword Ġembed d ings Ġof Ġthe Ġd L CE Ġmodel , Ġstate - of - the - art Ġword Ġembed d ings Ġfor Ġan ton ym - syn onym Ġdistinction . ĠYet , Ġthe Ġword Ġembed d ings Ġof Ġthe Ġd L CE Ġmodel , Ġi . e ., Ġsupervised Ġword Ġembed d ings , Ġrepresent Ġinformation Ġcollected Ġfrom Ġlex ical Ġresources . ĠIn Ġorder Ġto Ġevaluate Ġthe Ġeffect Ġof Ġthese Ġword Ġembed d ings Ġon Ġthe Ġperformance Ġof Ġour Ġmodels , Ġwe Ġreplace Ġthem Ġby Ġthe Ġpre - trained ĠGl o Ve Ġword Ġembed d ings Ġwith Ġ100 Ġdimensions , Ġand Ġcompare Ġthe Ġeffects Ġof Ġthe ĠGl o Ve Ġword Ġembed d ings Ġand Ġthe Ġd L CE Ġword Ġembed d ings Ġon Ġthe Ġperformance Ġof Ġthe Ġtwo Ġproposed Ġmodels . </s> Table Ġ5 Ġillustrates Ġthe Ġperformance Ġof Ġour Ġtwo Ġmodels Ġon Ġall Ġword Ġclasses . ĠThe Ġtable Ġshows Ġthat Ġthe Ġd L CE Ġword Ġembed d ings Ġare Ġbetter Ġthan Ġthe Ġpre - trained ĠGl o Ve Ġword Ġembed d ings , Ġby Ġaround Ġ. 01 Ġ$ F _ 1 $ Ġfor Ġthe Ġpattern - based ĠAnt Syn NET Ġmodel Ġand Ġthe Ġcombined ĠAnt Syn NET Ġmodel Ġregarding Ġadjective Ġand Ġverb Ġpairs . ĠRegarding Ġnoun Ġpairs , Ġthe Ġimprovements Ġof Ġthe Ġd L CE Ġword Ġembed d ings Ġover Ġpre - trained ĠGl o Ve Ġword Ġembed d ings Ġachieve Ġaround Ġ. 01 Ġand Ġ. 04 Ġ$ F _ 1 $ Ġfor Ġthe Ġpattern - based Ġmodel Ġand Ġthe Ġcombined Ġmodel , Ġrespectively . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġnovel Ġpattern - based Ġneural Ġmethod ĠAnt Syn NET Ġto Ġdistinguish Ġan ton y ms Ġfrom Ġsyn onyms . ĠWe Ġhypothesized Ġthat Ġan ton ym ous Ġword Ġpairs Ġco - occ ur Ġwith Ġeach Ġother Ġin Ġlex ico - sy nt actic Ġpatterns Ġwithin Ġa Ġsentence Ġmore Ġoften Ġthan Ġsynonymous Ġword Ġpairs . </s> The Ġpatterns Ġwere Ġderived Ġfrom Ġthe Ġsimple Ġpaths Ġbetween Ġsem antically Ġrelated Ġwords Ġin Ġa Ġsynt actic Ġparse Ġtree . ĠIn Ġaddition Ġto Ġlex ical Ġand Ġsynt actic Ġinformation , Ġwe Ġsuggested Ġa Ġnovel Ġpath Ġdistance Ġfeature . ĠThe ĠAnt Syn NET Ġmodel Ġconsists Ġof Ġtwo Ġapproaches Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms . ĠIn Ġthe Ġfirst Ġapproach , Ġwe Ġused Ġa Ġrecurrent Ġneural Ġnetwork Ġwith Ġlong Ġshort - term Ġmemory Ġunits Ġto Ġencode Ġthe Ġpatterns Ġas Ġvector Ġrepresentations ; Ġin Ġthe Ġsecond Ġapproach , Ġwe Ġmade Ġuse Ġof Ġthe Ġdistribution Ġand Ġencoded Ġpatterns Ġof Ġthe Ġtarget Ġpairs Ġto Ġgenerate Ġcombined Ġvector Ġrepresentations . ĠThe Ġresulting Ġvectors Ġof Ġpatterns Ġin Ġboth Ġapproaches Ġwere Ġfed Ġinto Ġthe Ġlog istic Ġregression Ġlayer Ġfor Ġclassification . </s> Our Ġproposed Ġmodels Ġsignificantly Ġoutper formed Ġtwo Ġbas elines Ġrelying Ġon Ġprevious Ġwork , Ġmainly Ġin Ġterms Ġof Ġrecall . ĠMoreover , Ġwe Ġdemonstrated Ġthat Ġthe Ġdistance Ġfeature Ġoutper formed Ġa Ġpreviously Ġsuggested Ġdirection Ġfeature , Ġand Ġthat Ġour Ġembed d ings Ġoutper formed Ġthe Ġstate - of - the - art ĠGl o Ve Ġembed d ings . ĠLast Ġbut Ġnot Ġleast , Ġour Ġtwo Ġproposed Ġmodels Ġonly Ġrely Ġon Ġcorpus Ġdata , Ġsuch Ġthat Ġthe Ġmodels Ġare Ġeasily Ġapplicable Ġto Ġother Ġlanguages Ġand Ġrelations . </s> Acknowled gements </s> We Ġwould Ġlike Ġto Ġthank ĠMichael ĠRoth Ġfor Ġhelping Ġus Ġto Ġcompute Ġthe Ġresults Ġof Ġthe ĠR & Si W Ġmodel Ġon Ġour Ġdataset . </s> The Ġresearch Ġwas Ġsupported Ġby Ġthe ĠMinistry Ġof ĠEducation Ġand ĠTraining Ġof Ġthe ĠSocialist ĠRepublic Ġof ĠVietnam Ġ( Sch olars hip Ġ9 77 / Q D - BG DD T ; ĠKim - An h ĠNguyen ), Ġthe ĠD FG ĠCollabor ative ĠResearch ĠCentre ĠSF B Ġ7 32 Ġ( Kim - An h ĠNguyen , ĠNg oc ĠTh ang ĠVu ), Ġand Ġthe ĠD FG ĠHe isen berg ĠFellowship ĠSCH U - 25 80 / 1 Ġ( Sab ine ĠSch ult e Ġim ĠWald e ).
Introduction Antonymy and synonymy represent lexical semantic relations that are central to the organization of the mental lexicon BIBREF0 . While antonymy is defined as the oppositeness between words, synonymy refers to words that are similar in meaning BIBREF1 , BIBREF2 . From a computational point of view, distinguishing between antonymy and synonymy is important for NLP applications such as Machine Translation and Textual Entailment, which go beyond a general notion of semantic relatedness and require to identify specific semantic relations. However, due to interchangeable substitution, antonyms and synonyms often occur in similar contexts, which makes it challenging to automatically distinguish between them. Two families of approaches to differentiate between antonyms and synonyms are predominent in NLP. Both make use of distributional vector representations, relying on the distributional hypothesis BIBREF3 , BIBREF4 , that words with similar distributions have related meanings: co-occurrence models and pattern-based models. These distributional semantic models (DSMs) offer a means to represent meaning vectors of words or word pairs, and to determine their semantic relatedness BIBREF5 . In co-occurrence models, each word is represented by a weighted feature vector, where features typically correspond to words that co-occur in particular contexts. When using word embeddings, these models rely on neural methods to represent words as low-dimensional vectors. To create the word embeddings, the models either make use of neural-based techniques, such as the skip-gram model BIBREF6 , or use matrix factorization BIBREF7 that builds word embeddings by factorizing word-context co-occurrence matrices. In comparison to standard co-occurrence vector representations, word embeddings address the problematic sparsity of word vectors and have achieved impressive results in many NLP tasks such as word similarity (e.g., Pennington:14), relation classification (e.g., ThangVu:16), and antonym-synonym distinction (e.g., Nguyen:16). In pattern-based models, vector representations make use of lexico-syntactic surface patterns to distinguish between the relations of word pairs. For example, Justeson/Katz:91 suggested that adjectival opposites co-occur with each other in specific linear sequences, such as between X and Y. Hearst:92 determined surface patterns, e.g., X such as Y, to identify nominal hypernyms. Lin2003 proposed two textual patterns indicating semantic incompatibility, from X to Y and either X or Y, to distinguish opposites from semantically similar words. Roth2014 proposed a method that combined patterns with discourse markers for classifying paradigmatic relations including antonymy, synonymy, and hypernymy. Recently, Schwartz:15 used two prominent patterns from Lin2003 to learn word embeddings that distinguished antonyms from similar words in determining degrees of similarity and word analogy. In this paper, we present a novel pattern-based neural method AntSynNET to distinguish antonyms from synonyms. We hypothesize that antonymous word pairs co-occur with each other in lexico-syntactic patterns within a sentence more often than would be expected by synonymous pairs. This hypothesis is inspired by corpus-based studies on antonymy and synonymy. Among others, Charles1989 suggested that adjectival opposites co-occur in patterns; Fellbaum:95 stated that nominal and verbal opposites co-occur in the same sentence significantly more often than chance; Lin2003 argued that if two words appear in clear antonym patterns, they are unlikely to represent synonymous pair. We start out by inducing patterns between X and Y from a large-scale web corpus, where X and Y represent two words of an antonym or synonym word pair, and the pattern is derived from the simple paths between X and Y in a syntactic parse tree. Each node in the simple path combines lexical and syntactic information; in addition, we suggest a novel feature for the patterns, i.e., the distance between the two words along the syntactic path. All pattern features are fed into a recurrent neural network with long short-term memory (LSTM) units BIBREF8 , which encode the patterns as vector representations. Afterwards, the vector representations of the patterns are used in a classifier to distinguish between antonyms and synonyms. The results from experiments show that AntSynNET improves the performance over prior pattern-based methods. Furthermore, the implementation of our models is made publicly available. The remainder of this paper is organized as follows: In Section "Related Work" , we present previous work distinguishing antonyms and synonyms. Section "AntSynNET: LSTM-based Antonym-Synonym Distinction" describes our proposed AntSynNET model. We present the induction of the patterns (Section "Baseline Models" ), describe the recurrent neural network with long short-term memory units which is used to encode patterns within a vector representation (Section "Recurrent Neural Network with Long Short-Term Memory Units" ), and describe two models to classify antonyms and synonyms: the pure pattern-based model (Section UID11 ) and the combined model (Section UID13 ). After introducing two baselines in Section "Baseline Models" , we describe our dataset, experimental settings, results of our methods, the effects of the newly proposed distance feature, and the effects of the various types of word embeddings. Section "Conclusion" concludes the paper. AntSynNET: LSTM-based Antonym-Synonym Distinction In this section, we describe the AntSynNET model, using a pattern-based LSTM for distinguishing antonyms from synonyms. We first present the induction of patterns from a parsed corpus (Section "Baseline Models" ). Section "Recurrent Neural Network with Long Short-Term Memory Units" then describes how we utilize the recurrent neural network with long short-term memory units to encode the patterns as vector representation. Finally, we present the AntSynNET model and two approaches to classify antonyms and synonyms (Section "The Proposed AntSynNET Model" ). Induction of Patterns Corpus-based studies on antonymy have suggested that opposites co-occur with each other within a sentence significantly more often than would be expected by chance. Our method thus makes use of patterns as the main indicators of word pair co-occurrence, to enforce a distinction between antonyms and synonyms. Figure 1 shows a syntactic parse tree of the sentence “My old village has been provided with the new services”. Following the characterizations of a tree in graph theory, any two nodes (vertices) of a tree are connected by a simple path (or one unique path). The simple path is the shortest path between any two nodes in a tree and does not contain repeated nodes. In the example, the lexico-syntactic tree pattern of the antonymous pair old–new is determined by finding the simple path (in red) from the lemma old to the lemma new. It focuses on the most relevant information and ignores irrelevant information which does not appear in the simple path (i.e., has, been). The example pattern between X = old and Y = new in Figure 1 is represented as follows: X/JJ/amod/2 – village/NN/nsubj/1 – provide/VBN/ROOT/0 – with/IN/prep/1 – service/NNS/pobj/2 – Y/JJ/amod/3. The path patterns make use of four features to represent each node in the syntax tree: lemma, part-of-speech (POS) tag, dependency label and distance label. The lemma feature captures the lexical information of words in the sentence, while the POS and dependency features capture the morpho-syntactic information of the sentence. The distance label measures the path distance between the target word nodes in the syntactic tree. Each step between a parent and a child node represents a distance of 1; and the ancestor nodes of the remaining nodes in the path are represented by a distance of 0. For example, the node provided is an ancestor node of the simple path from old to new. The distances from the node provided to the nodes village and old are 1 and 2, respectively. The vector representation of each node concatenates the four-feature vectors as follows: $ \vec{v}_{node} = [\vec{v}_{lemma} \oplus \vec{v}_{pos} \oplus \vec{v}_{dep} \oplus \vec{v}_{dist}] $ where $\vec{v}_{lemma}, \vec{v}_{pos}, \vec{v}_{dep}, \vec{v}_{dist}$ represent the embeddings of the lemma, POS tag, dependency label and distance label, respectively; and the $\oplus $ denotes the concatenation operation. For a pattern $p$ which is constructed by the sequence of nodes $n_1, n_2, ..., n_k$ , the pattern representation of $p$ is a sequence of vectors: $p=[\vec{n}_1, \vec{n}_2, ..., \vec{n}_k]$ . The pattern vector $\vec{v}_p$ is then encoded by applying a recurrent neural network. Recurrent Neural Network with Long Short-Term Memory Units A recurrent neural network (RNN) is suitable for modeling sequential data by a vector representation. In our methods, we use a long short-term memory (LSTM) network, a variant of a recurrent neural network to encode patterns, for the following reasons. Given a sequence of words $p=[n_1,n_2,...,n_k]$ as input data, an RNN processes each word $n_t$ at a time, and returns a vector of state $h_k$ for the complete input sequence. For each time step $t$ , the RNN updates an internal memory state $h_t$ which depends on the current input $n_t$ and the previous state $h_{t-1}$ . Yet, if the sequential input is a long-term dependency, an RNN faces the problem of gradient vanishing or exploding, leading to difficulties in training the model. LSTM units address these problems. The underlying idea of an LSTM is to use an adaptive gating mechanism to decide on the degree that LSTM units keep the previous state and memorize the extracted features of the current input. More specifically, an LSTM comprises four components: an input gate $i_t$ , a forget gate $f_t$ , an output gate $o_t$ , and a memory cell $c_t$ . The state of an LSTM at each time step $t$ is formalized as follows: $ \begin{array}{l} i_t = \sigma (W_i \cdot x_t + U_i \cdot h_{t-1} + b_i) \\ f_t = \sigma (W_f \cdot x_t + U_f \cdot h_{t-1} + b_f) \\ o_t = \sigma (W_o \cdot x_t + U_o \cdot h_{t-1} + b_o) \\ g_t = \tanh (W_c \cdot x_t + U_c \cdot h_{t-1} + b_c) \\ c_t = i_t \otimes g_t + f_t \otimes c_{t-1} \end{array} $ $W$ refers to a matrix of weights that projects information between two layers; $b$ is a layer-specific vector of bias terms; $\sigma $ denotes the sigmoid function. The output of an LSTM at a time step $t$ is computed as follows: $ h_t = o_t \otimes \tanh (c_t) $ where $\otimes $ denotes element-wise multiplication. In our methods, we rely on the last state $h_k$ to represent the vector $\vec{v}_p$ of a pattern $p = [\vec{n}_1, \vec{n}_2,...,\vec{n}_k]$ . The Proposed AntSynNET Model In this section, we present two models to distinguish antonyms from synonyms. The first model makes use of patterns to classify antonyms and synonyms, by using an LSTM to encode patterns as vector representations and then feeding those vectors to a logistic regression layer (Section UID11 ). The second model creates combined vector representations of word pairs, which concatenate the vectors of the words and the patterns (Section UID13 ). In this model, we make use of a recurrent neural network with LSTM units to encode patterns containing a sequence of nodes. Figure 2 illustrates the AntSynNET model. Given a word pair $(x,y)$ , we induce patterns for $(x,y)$ from a corpus, where each pattern represents a path from $x$ to $y$ (cf. Section "Baseline Models" ). We then feed each pattern $p$ of the word pair $(x,y)$ into an LSTM to obtain $\vec{v}_p$ , the vector representation of the pattern $p$ (cf. Section "Recurrent Neural Network with Long Short-Term Memory Units" ). For each word pair $(x,y)$ , the vector representation of $(x,y)$ is computed as follows: $$\vec{v}_{xy} = \frac{\sum \nolimits _{p \in P(x,y)} {\vec{v}_p \cdot c_p}}{\sum \nolimits _{p \in P(x,y)} {c_p}}$$   (Eq. 12) $\vec{v}_{xy}$ refers to the vector of the word pair $(x,y)$ ; $P(x,y)$ is the set of patterns corresponding to the pair $(x,y)$ ; $c_p$ is the frequency of the pattern $p$ . The vector $\vec{v}_{xy}$ is then fed into a logistic regression layer whose target is the class label associated with the pair $(x,y)$ . Finally, the pair $(x,y)$ is predicted as positive (i.e., antonymous) word pair if the probability of the prediction for $\vec{v}_{xy}$ is larger than 0.5. Inspired by the supervised distributional concatenation method in Baroni:12 and the integrated path-based and distributional method for hypernymy detection in Shwartz:16, we take into account the patterns and distribution of target pairs to create their combined vector representations. Given a word pair $(x,y)$ , the combined vector representation of the pair $(x,y)$ is determined by using both the co-occurrence distribution of the words and the syntactic path patterns: $$\vec{v}_{comb(x,y)} = [\vec{v}_x \oplus \vec{v}_{xy} \oplus \vec{v}_y]$$   (Eq. 14) $\vec{v}_{comb(x,y)}$ refers to the combined vector of the word pair $(x,y)$ ; $\vec{v}_x$ and $\vec{v}_y$ are the vectors of word $x$ and word $y$ , respectively; $\vec{v}_{xy}$ is the vector of the pattern that corresponds to the pair $(x,y)$ , cf. Section UID11 . Similar to the pattern-based model, the combined vector $\vec{v}_{comb(x,y)}$ is fed into the logistic regression layer to classify antonyms and synonyms. Baseline Models To compare AntSynNET with baseline models for pattern-based classification of antonyms and synonyms, we introduce two pattern-based baseline methods: the distributional method (Section "Distributional Baseline" ), and the distributed method (Section "Distributed Baseline" ). Distributional Baseline As a first baseline, we apply the approach by Roth2014, henceforth R&SiW. They used a vector space model to represent pairs of words by a combination of standard lexico-syntactic patterns and discourse markers. In addition to the patterns, the discourse markers added information to express discourse relations, which in turn may indicate the specific semantic relation between the two words in a word pair. For example, contrast relations might indicate antonymy, whereas elaborations may indicate synonymy or hyponymy. Michael Roth, the first author of R&SiW, kindly computed the relation classification results of the pattern–discourse model for our test sets. The weights between marker-based and pattern-based models were tuned on the validation sets; other hyperparameters were set exactly as described by the R&SiW method. Distributed Baseline The SP method proposed by Schwartz:15 uses symmetric patterns for generating word embeddings. In this work, the authors applied an unsupervised algorithm for the automatic extraction of symmetric patterns from plain text. The symmetric patterns were defined as a sequence of 3-5 tokens consisting of exactly two wildcards and 1-3 words. The patterns were filtered based on their frequencies, such that the resulting pattern set contained 11 patterns. For generating word embeddings, a matrix of co-occurrence counts between patterns and words in the vocabulary was computed, using positive point-wise mutual information. The sparsity problem of vector representations was addressed by smoothing. For antonym representation, the authors relied on two patterns suggested by Lin2003 to construct word embeddings containing an antonym parameter that can be turned on in order to represent antonyms as dissimilar, and that can be turned off to represent antonyms as similar. To apply the SP method to our data, we make use of the pre-trained SP embeddings with 500 dimensions. We calculate the cosine similarity of word pairs and then use a Support Vector Machine with Radial Basis Function kernel to classify antonyms and synonyms. Dataset For training the models, neural networks require a large amount of training data. We use the existing large-scale antonym and synonym pairs previously used by Nguyen:16. Originally, the data pairs were collected from WordNet BIBREF9 and Wordnik. In order to induce patterns for the word pairs in the dataset, we identify the sentences in the corpus that contain the word pair. Thereafter, we extract all patterns for the word pair. We filter out all patterns which occur less than five times; and we only take into account word pairs that contain at least five patterns for training, validating and testing. For the proportion of positive and negative pairs, we keep a ratio of 1:1 positive (antonym) to negative (synonym) pairs in the dataset. In order to create the sets of training, testing and validation data, we perform random splitting with 70% train, 25% test, and 5% validation sets. The final dataset contains the number of word pairs according to word classes described in Table 1 . Moreover, Table 2 shows the average number of patterns for each word pair in our dataset. Experimental Settings We use the English Wikipedia dump from June 2016 as the corpus resource for our methods and baselines. For parsing the corpus, we rely on spaCy. For the lemma embeddings, we rely on the word embeddings of the dLCE model BIBREF10 which is the state-of-the-art vector representation for distinguishing antonyms from synonyms. We re-implemented this cutting-edge model on Wikipedia with 100 dimensions, and then make use of the dLCE word embeddings for initialization the lemma embeddings. The embeddings of POS tags, dependency labels, distance labels, and out-of-vocabulary lemmas are initialized randomly. The number of dimensions is set to 10 for the embeddings of POS tags, dependency labels and distance labels. We use the validation sets to tune the number of dimensions for these labels. For optimization, we rely on the cross-entropy loss function and Stochastic Gradient Descent with the Adadelta update rule BIBREF11 . For training, we use the Theano framework BIBREF12 . Regularization is applied by a dropout of 0.5 on each of component's embeddings (dropout rate is tuned on the validation set). We train the models with 40 epochs and update all embeddings during training. Overall Results Table 3 shows the significant performance of our models in comparison to the baselines. Concerning adjectives, the two proposed models significantly outperform the two baselines: The performance of the baselines is around .72 for $F_1$ , and the corresponding results for the combined AntSynNET model achieve an improvement of $>$ .06. Regarding nouns, the improvement of the new methods is just .02 $F_1$ in comparison to the R&SiW baseline, but we achieve a much better performance in comparison to the SP baseline, an increase of .37 $F_1$ . Regarding verbs, we do not outperform the more advanced R&SiW baseline in terms of the $F_1$ score, but we obtain higher recall scores. In comparison to the SP baseline, our models still show a clear $F_1$ improvement. Overall, our proposed models achieve comparatively high recall scores compared to the two baselines. This strengthens our hypothesis that there is a higher possibility for the co-occurrence of antonymous pairs in patterns over synonymous pairs within a sentence. Because, when the proposed models obtain high recall scores, the models are able to retrieve most relevant information (antonymous pairs) corresponding to the patterns. Regarding the low precision in the two proposed models, we sampled randomly 5 pairs in each population: true positive, true negative, false positive, false negative. We then compared the overlap of patterns for the true predictions (true positive pairs and true negative pairs) and the false predictions (false positive pairs and false negative pairs). We found out that there is no overlap between patterns of true predictions; and the number overlap between patterns of false predictions is 2, 2, and 4 patterns for noun, adjective, and verb classes, respectively. This shows that the low precision of our models stems from the patterns which represent both antonymous and synonymous pairs. Effect of the Distance Feature In our models, the novel distance feature is successfully integrated along the syntactic path to represent lexico-syntactic patterns. The intuition behind the distance feature exploits properties of trees in graph theory, which show that there exist differences in the degree of relationship between the parent node and the child nodes ( $distance = 1$ ) and in the degree of relationship between the ancestor node and the descendant nodes ( $distance > 1$ ). Hence, we use the distance feature to effectively capture these relationships. In order to evaluate the effect of our novel distance feature, we compare the distance feature to the direction feature proposed by Shwartz:16. In their approach, the authors combined lemma, POS, dependency, and direction features for the task of hypernym detection. The direction feature represented the direction of the dependency label between two nodes in a path from X to Y. For evaluation, we make use of the same information regarding dataset and patterns as in Section "Overall Results" , and then replace the distance feature by the direction feature. The results are shown in Table 4 . The distance feature enhances the performance of our proposed models more effectively than the direction feature does, across all word classes. Effect of Word Embeddings Our methods rely on the word embeddings of the dLCE model, state-of-the-art word embeddings for antonym-synonym distinction. Yet, the word embeddings of the dLCE model, i.e., supervised word embeddings, represent information collected from lexical resources. In order to evaluate the effect of these word embeddings on the performance of our models, we replace them by the pre-trained GloVe word embeddings with 100 dimensions, and compare the effects of the GloVe word embeddings and the dLCE word embeddings on the performance of the two proposed models. Table 5 illustrates the performance of our two models on all word classes. The table shows that the dLCE word embeddings are better than the pre-trained GloVe word embeddings, by around .01 $F_1$ for the pattern-based AntSynNET model and the combined AntSynNET model regarding adjective and verb pairs. Regarding noun pairs, the improvements of the dLCE word embeddings over pre-trained GloVe word embeddings achieve around .01 and .04 $F_1$ for the pattern-based model and the combined model, respectively. Conclusion In this paper, we presented a novel pattern-based neural method AntSynNET to distinguish antonyms from synonyms. We hypothesized that antonymous word pairs co-occur with each other in lexico-syntactic patterns within a sentence more often than synonymous word pairs. The patterns were derived from the simple paths between semantically related words in a syntactic parse tree. In addition to lexical and syntactic information, we suggested a novel path distance feature. The AntSynNET model consists of two approaches to classify antonyms and synonyms. In the first approach, we used a recurrent neural network with long short-term memory units to encode the patterns as vector representations; in the second approach, we made use of the distribution and encoded patterns of the target pairs to generate combined vector representations. The resulting vectors of patterns in both approaches were fed into the logistic regression layer for classification. Our proposed models significantly outperformed two baselines relying on previous work, mainly in terms of recall. Moreover, we demonstrated that the distance feature outperformed a previously suggested direction feature, and that our embeddings outperformed the state-of-the-art GloVe embeddings. Last but not least, our two proposed models only rely on corpus data, such that the models are easily applicable to other languages and relations. Acknowledgements We would like to thank Michael Roth for helping us to compute the results of the R&SiW model on our dataset. The research was supported by the Ministry of Education and Training of the Socialist Republic of Vietnam (Scholarship 977/QD-BGDDT; Kim-Anh Nguyen), the DFG Collaborative Research Centre SFB 732 (Kim-Anh Nguyen, Ngoc Thang Vu), and the DFG Heisenberg Fellowship SCHU-2580/1 (Sabine Schulte im Walde).
What dataset do they use to evaluate their method?
2fbb6322e485e7743ec3fb4bb02d44bf4b5ea8a6
What Ġdataset Ġdo Ġthey Ġuse Ġto Ġevaluate Ġtheir Ġmethod ?
ant onym Ġand Ġsyn onym Ġpairs , Ġcollected Ġfrom ĠWord Net ĠB IB REF 9 Ġand ĠWord nik
<s> What Ġdataset Ġdo Ġthey Ġuse Ġto Ġevaluate Ġtheir Ġmethod ? </s> Introduction </s> Ant ony my Ġand Ġsyn ony my Ġrepresent Ġlex ical Ġsemantic Ġrelations Ġthat Ġare Ġcentral Ġto Ġthe Ġorganization Ġof Ġthe Ġmental Ġlex icon ĠB IB REF 0 Ġ. ĠWhile Ġan ton y my Ġis Ġdefined Ġas Ġthe Ġopp osit eness Ġbetween Ġwords , Ġsyn ony my Ġrefers Ġto Ġwords Ġthat Ġare Ġsimilar Ġin Ġmeaning ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ. ĠFrom Ġa Ġcomputational Ġpoint Ġof Ġview , Ġdistinguishing Ġbetween Ġan ton y my Ġand Ġsyn ony my Ġis Ġimportant Ġfor ĠN LP Ġapplications Ġsuch Ġas ĠMachine ĠTranslation Ġand ĠText ual ĠEnt ail ment , Ġwhich Ġgo Ġbeyond Ġa Ġgeneral Ġnotion Ġof Ġsemantic Ġrelated ness Ġand Ġrequire Ġto Ġidentify Ġspecific Ġsemantic Ġrelations . ĠHowever , Ġdue Ġto Ġinterchangeable Ġsubstitution , Ġan ton y ms Ġand Ġsyn onyms Ġoften Ġoccur Ġin Ġsimilar Ġcontexts , Ġwhich Ġmakes Ġit Ġchallenging Ġto Ġautomatically Ġdistinguish Ġbetween Ġthem . </s> Two Ġfamilies Ġof Ġapproaches Ġto Ġdifferentiate Ġbetween Ġan ton y ms Ġand Ġsyn onyms Ġare Ġpredomin ent Ġin ĠN LP . ĠBoth Ġmake Ġuse Ġof Ġdistribution al Ġvector Ġrepresentations , Ġrelying Ġon Ġthe Ġdistribution al Ġhypothesis ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġthat Ġwords Ġwith Ġsimilar Ġdistributions Ġhave Ġrelated Ġmeanings : Ġco - occ urrence Ġmodels Ġand Ġpattern - based Ġmodels . ĠThese Ġdistribution al Ġsemantic Ġmodels Ġ( DS Ms ) Ġoffer Ġa Ġmeans Ġto Ġrepresent Ġmeaning Ġvectors Ġof Ġwords Ġor Ġword Ġpairs , Ġand Ġto Ġdetermine Ġtheir Ġsemantic Ġrelated ness ĠB IB REF 5 Ġ. </s> In Ġco - occ urrence Ġmodels , Ġeach Ġword Ġis Ġrepresented Ġby Ġa Ġweighted Ġfeature Ġvector , Ġwhere Ġfeatures Ġtypically Ġcorrespond Ġto Ġwords Ġthat Ġco - occ ur Ġin Ġparticular Ġcontexts . ĠWhen Ġusing Ġword Ġembed d ings , Ġthese Ġmodels Ġrely Ġon Ġneural Ġmethods Ġto Ġrepresent Ġwords Ġas Ġlow - dimensional Ġvectors . ĠTo Ġcreate Ġthe Ġword Ġembed d ings , Ġthe Ġmodels Ġeither Ġmake Ġuse Ġof Ġneural - based Ġtechniques , Ġsuch Ġas Ġthe Ġskip - gram Ġmodel ĠB IB REF 6 Ġ, Ġor Ġuse Ġmatrix Ġfactor ization ĠB IB REF 7 Ġthat Ġbuilds Ġword Ġembed d ings Ġby Ġfactor izing Ġword - context Ġco - occ urrence Ġmat rices . ĠIn Ġcomparison Ġto Ġstandard Ġco - occ urrence Ġvector Ġrepresentations , Ġword Ġembed d ings Ġaddress Ġthe Ġproblematic Ġsp arsity Ġof Ġword Ġvectors Ġand Ġhave Ġachieved Ġimpressive Ġresults Ġin Ġmany ĠN LP Ġtasks Ġsuch Ġas Ġword Ġsimilarity Ġ( e . g ., ĠPen nington : 14 ), Ġrelation Ġclassification Ġ( e . g ., ĠTh ang V u : 16 ), Ġand Ġan ton ym - syn onym Ġdistinction Ġ( e . g ., ĠNguyen : 16 ). </s> In Ġpattern - based Ġmodels , Ġvector Ġrepresentations Ġmake Ġuse Ġof Ġlex ico - sy nt actic Ġsurface Ġpatterns Ġto Ġdistinguish Ġbetween Ġthe Ġrelations Ġof Ġword Ġpairs . ĠFor Ġexample , ĠJust eson / Kat z : 91 Ġsuggested Ġthat Ġadject ival Ġoppos ites Ġco - occ ur Ġwith Ġeach Ġother Ġin Ġspecific Ġlinear Ġsequences , Ġsuch Ġas Ġbetween ĠX Ġand ĠY . ĠHear st : 92 Ġdetermined Ġsurface Ġpatterns , Ġe . g ., ĠX Ġsuch Ġas ĠY , Ġto Ġidentify Ġnominal Ġhyper ny ms . ĠLin 2003 Ġproposed Ġtwo Ġtextual Ġpatterns Ġindicating Ġsemantic Ġincomp atibility , Ġfrom ĠX Ġto ĠY Ġand Ġeither ĠX Ġor ĠY , Ġto Ġdistinguish Ġoppos ites Ġfrom Ġsem antically Ġsimilar Ġwords . ĠRoth 2014 Ġproposed Ġa Ġmethod Ġthat Ġcombined Ġpatterns Ġwith Ġdiscourse Ġmarkers Ġfor Ġclass ifying Ġparad igmatic Ġrelations Ġincluding Ġan ton y my , Ġsyn ony my , Ġand Ġhyper ny my . ĠRecently , ĠSchwartz : 15 Ġused Ġtwo Ġprominent Ġpatterns Ġfrom ĠLin 2003 Ġto Ġlearn Ġword Ġembed d ings Ġthat Ġdistinguished Ġan ton y ms Ġfrom Ġsimilar Ġwords Ġin Ġdetermining Ġdegrees Ġof Ġsimilarity Ġand Ġword Ġanalogy . </s> In Ġthis Ġpaper , Ġwe Ġpresent Ġa Ġnovel Ġpattern - based Ġneural Ġmethod ĠAnt Syn NET Ġto Ġdistinguish Ġan ton y ms Ġfrom Ġsyn onyms . ĠWe Ġhypothes ize Ġthat Ġan ton ym ous Ġword Ġpairs Ġco - occ ur Ġwith Ġeach Ġother Ġin Ġlex ico - sy nt actic Ġpatterns Ġwithin Ġa Ġsentence Ġmore Ġoften Ġthan Ġwould Ġbe Ġexpected Ġby Ġsynonymous Ġpairs . ĠThis Ġhypothesis Ġis Ġinspired Ġby Ġcorpus - based Ġstudies Ġon Ġan ton y my Ġand Ġsyn ony my . ĠAmong Ġothers , ĠCharles 1989 Ġsuggested Ġthat Ġadject ival Ġoppos ites Ġco - occ ur Ġin Ġpatterns ; ĠFell baum : 95 Ġstated Ġthat Ġnominal Ġand Ġverbal Ġoppos ites Ġco - occ ur Ġin Ġthe Ġsame Ġsentence Ġsignificantly Ġmore Ġoften Ġthan Ġchance ; ĠLin 2003 Ġargued Ġthat Ġif Ġtwo Ġwords Ġappear Ġin Ġclear Ġan ton ym Ġpatterns , Ġthey Ġare Ġunlikely Ġto Ġrepresent Ġsynonymous Ġpair . </s> We Ġstart Ġout Ġby Ġinducing Ġpatterns Ġbetween ĠX Ġand ĠY Ġfrom Ġa Ġlarge - scale Ġweb Ġcorpus , Ġwhere ĠX Ġand ĠY Ġrepresent Ġtwo Ġwords Ġof Ġan Ġan ton ym Ġor Ġsyn onym Ġword Ġpair , Ġand Ġthe Ġpattern Ġis Ġderived Ġfrom Ġthe Ġsimple Ġpaths Ġbetween ĠX Ġand ĠY Ġin Ġa Ġsynt actic Ġparse Ġtree . ĠEach Ġnode Ġin Ġthe Ġsimple Ġpath Ġcombines Ġlex ical Ġand Ġsynt actic Ġinformation ; Ġin Ġaddition , Ġwe Ġsuggest Ġa Ġnovel Ġfeature Ġfor Ġthe Ġpatterns , Ġi . e ., Ġthe Ġdistance Ġbetween Ġthe Ġtwo Ġwords Ġalong Ġthe Ġsynt actic Ġpath . ĠAll Ġpattern Ġfeatures Ġare Ġfed Ġinto Ġa Ġrecurrent Ġneural Ġnetwork Ġwith Ġlong Ġshort - term Ġmemory Ġ( L ST M ) Ġunits ĠB IB REF 8 Ġ, Ġwhich Ġencode Ġthe Ġpatterns Ġas Ġvector Ġrepresentations . ĠAfterwards , Ġthe Ġvector Ġrepresentations Ġof Ġthe Ġpatterns Ġare Ġused Ġin Ġa Ġclass ifier Ġto Ġdistinguish Ġbetween Ġan ton y ms Ġand Ġsyn onyms . ĠThe Ġresults Ġfrom Ġexperiments Ġshow Ġthat ĠAnt Syn NET Ġimproves Ġthe Ġperformance Ġover Ġprior Ġpattern - based Ġmethods . ĠFurthermore , Ġthe Ġimplementation Ġof Ġour Ġmodels Ġis Ġmade Ġpublicly Ġavailable . </s> The Ġremainder Ġof Ġthis Ġpaper Ġis Ġorganized Ġas Ġfollows : ĠIn ĠSection Ġ" Related ĠWork " Ġ, Ġwe Ġpresent Ġprevious Ġwork Ġdistinguishing Ġan ton y ms Ġand Ġsyn onyms . ĠSection Ġ" Ant Syn NET : ĠL ST M - based ĠAnt onym - Syn onym ĠDist inction " Ġdescribes Ġour Ġproposed ĠAnt Syn NET Ġmodel . ĠWe Ġpresent Ġthe Ġinduction Ġof Ġthe Ġpatterns Ġ( Section Ġ" Bas eline ĠModels " Ġ), Ġdescribe Ġthe Ġrecurrent Ġneural Ġnetwork Ġwith Ġlong Ġshort - term Ġmemory Ġunits Ġwhich Ġis Ġused Ġto Ġencode Ġpatterns Ġwithin Ġa Ġvector Ġrepresentation Ġ( Section Ġ" Rec urrent ĠNeural ĠNetwork Ġwith ĠLong ĠShort - Term ĠMemory ĠUnits " Ġ), Ġand Ġdescribe Ġtwo Ġmodels Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms : Ġthe Ġpure Ġpattern - based Ġmodel Ġ( Section ĠUID 11 Ġ) Ġand Ġthe Ġcombined Ġmodel Ġ( Section ĠUID 13 Ġ). ĠAfter Ġintroducing Ġtwo Ġbas elines Ġin ĠSection Ġ" Bas eline ĠModels " Ġ, Ġwe Ġdescribe Ġour Ġdataset , Ġexperimental Ġsettings , Ġresults Ġof Ġour Ġmethods , Ġthe Ġeffects Ġof Ġthe Ġnewly Ġproposed Ġdistance Ġfeature , Ġand Ġthe Ġeffects Ġof Ġthe Ġvarious Ġtypes Ġof Ġword Ġembed d ings . ĠSection Ġ" Conclusion " Ġconcludes Ġthe Ġpaper . </s> Ant Syn NET : ĠL ST M - based ĠAnt onym - Syn onym ĠDist inction </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe ĠAnt Syn NET Ġmodel , Ġusing Ġa Ġpattern - based ĠL ST M Ġfor Ġdistinguishing Ġan ton y ms Ġfrom Ġsyn onyms . ĠWe Ġfirst Ġpresent Ġthe Ġinduction Ġof Ġpatterns Ġfrom Ġa Ġparsed Ġcorpus Ġ( Section Ġ" Bas eline ĠModels " Ġ). ĠSection Ġ" Rec urrent ĠNeural ĠNetwork Ġwith ĠLong ĠShort - Term ĠMemory ĠUnits " Ġthen Ġdescribes Ġhow Ġwe Ġutilize Ġthe Ġrecurrent Ġneural Ġnetwork Ġwith Ġlong Ġshort - term Ġmemory Ġunits Ġto Ġencode Ġthe Ġpatterns Ġas Ġvector Ġrepresentation . ĠFinally , Ġwe Ġpresent Ġthe ĠAnt Syn NET Ġmodel Ġand Ġtwo Ġapproaches Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms Ġ( Section Ġ" The ĠProp osed ĠAnt Syn NET ĠModel " Ġ). </s> Ind uction Ġof ĠPatterns </s> Corp us - based Ġstudies Ġon Ġan ton y my Ġhave Ġsuggested Ġthat Ġoppos ites Ġco - occ ur Ġwith Ġeach Ġother Ġwithin Ġa Ġsentence Ġsignificantly Ġmore Ġoften Ġthan Ġwould Ġbe Ġexpected Ġby Ġchance . ĠOur Ġmethod Ġthus Ġmakes Ġuse Ġof Ġpatterns Ġas Ġthe Ġmain Ġindicators Ġof Ġword Ġpair Ġco - occ urrence , Ġto Ġenforce Ġa Ġdistinction Ġbetween Ġan ton y ms Ġand Ġsyn onyms . ĠFigure Ġ1 Ġshows Ġa Ġsynt actic Ġparse Ġtree Ġof Ġthe Ġsentence ĠâĢ ľ My Ġold Ġvillage Ġhas Ġbeen Ġprovided Ġwith Ġthe Ġnew Ġservices âĢ Ŀ . ĠFollowing Ġthe Ġcharacter izations Ġof Ġa Ġtree Ġin Ġgraph Ġtheory , Ġany Ġtwo Ġnodes Ġ( vert ices ) Ġof Ġa Ġtree Ġare Ġconnected Ġby Ġa Ġsimple Ġpath Ġ( or Ġone Ġunique Ġpath ). ĠThe Ġsimple Ġpath Ġis Ġthe Ġshortest Ġpath Ġbetween Ġany Ġtwo Ġnodes Ġin Ġa Ġtree Ġand Ġdoes Ġnot Ġcontain Ġrepeated Ġnodes . ĠIn Ġthe Ġexample , Ġthe Ġlex ico - sy nt actic Ġtree Ġpattern Ġof Ġthe Ġan ton ym ous Ġpair Ġold âĢĵ new Ġis Ġdetermined Ġby Ġfinding Ġthe Ġsimple Ġpath Ġ( in Ġred ) Ġfrom Ġthe Ġle mma Ġold Ġto Ġthe Ġle mma Ġnew . ĠIt Ġfocuses Ġon Ġthe Ġmost Ġrelevant Ġinformation Ġand Ġignores Ġirrelevant Ġinformation Ġwhich Ġdoes Ġnot Ġappear Ġin Ġthe Ġsimple Ġpath Ġ( i . e ., Ġhas , Ġbeen ). ĠThe Ġexample Ġpattern Ġbetween ĠX Ġ= Ġold Ġand ĠY Ġ= Ġnew Ġin ĠFigure Ġ1 Ġis Ġrepresented Ġas Ġfollows : ĠX / JJ / am od / 2 ĠâĢĵ Ġvillage / NN / ns ub j / 1 ĠâĢĵ Ġprovide / V BN / RO OT / 0 ĠâĢĵ Ġwith / IN / prep / 1 ĠâĢĵ Ġservice / NN S / p obj / 2 ĠâĢĵ ĠY / JJ / am od / 3 . </s> The Ġpath Ġpatterns Ġmake Ġuse Ġof Ġfour Ġfeatures Ġto Ġrepresent Ġeach Ġnode Ġin Ġthe Ġsyntax Ġtree : Ġle mma , Ġpart - of - speech Ġ( POS ) Ġtag , Ġdependency Ġlabel Ġand Ġdistance Ġlabel . ĠThe Ġle mma Ġfeature Ġcaptures Ġthe Ġlex ical Ġinformation Ġof Ġwords Ġin Ġthe Ġsentence , Ġwhile Ġthe ĠPOS Ġand Ġdependency Ġfeatures Ġcapture Ġthe Ġmorph o - sy nt actic Ġinformation Ġof Ġthe Ġsentence . ĠThe Ġdistance Ġlabel Ġmeasures Ġthe Ġpath Ġdistance Ġbetween Ġthe Ġtarget Ġword Ġnodes Ġin Ġthe Ġsynt actic Ġtree . ĠEach Ġstep Ġbetween Ġa Ġparent Ġand Ġa Ġchild Ġnode Ġrepresents Ġa Ġdistance Ġof Ġ1 ; Ġand Ġthe Ġancestor Ġnodes Ġof Ġthe Ġremaining Ġnodes Ġin Ġthe Ġpath Ġare Ġrepresented Ġby Ġa Ġdistance Ġof Ġ0 . ĠFor Ġexample , Ġthe Ġnode Ġprovided Ġis Ġan Ġancestor Ġnode Ġof Ġthe Ġsimple Ġpath Ġfrom Ġold Ġto Ġnew . ĠThe Ġdistances Ġfrom Ġthe Ġnode Ġprovided Ġto Ġthe Ġnodes Ġvillage Ġand Ġold Ġare Ġ1 Ġand Ġ2 , Ġrespectively . </s> The Ġvector Ġrepresentation Ġof Ġeach Ġnode Ġconc aten ates Ġthe Ġfour - feature Ġvectors Ġas Ġfollows : Ġ$ Ġ\ vec { v } _{ node } Ġ= Ġ[ \ vec { v } _{ lem ma } Ġ\ opl us Ġ\ vec { v } _{ pos } Ġ\ opl us Ġ\ vec { v } _{ dep } Ġ\ opl us Ġ\ vec { v } _{ dist } ] Ġ$ </s> where Ġ$\ vec { v } _{ lem ma }, Ġ\ vec { v } _{ pos }, Ġ\ vec { v } _{ dep }, Ġ\ vec { v } _{ dist } $ Ġrepresent Ġthe Ġembed d ings Ġof Ġthe Ġle mma , ĠPOS Ġtag , Ġdependency Ġlabel Ġand Ġdistance Ġlabel , Ġrespectively ; Ġand Ġthe Ġ$\ opl us Ġ$ Ġdenotes Ġthe Ġconc aten ation Ġoperation . </s> For Ġa Ġpattern Ġ$ p $ Ġwhich Ġis Ġconstructed Ġby Ġthe Ġsequence Ġof Ġnodes Ġ$ n _ 1 , Ġn _ 2 , Ġ... , Ġn _ k $ Ġ, Ġthe Ġpattern Ġrepresentation Ġof Ġ$ p $ Ġis Ġa Ġsequence Ġof Ġvectors : Ġ$ p =[ \ vec { n } _ 1 , Ġ\ vec { n } _ 2 , Ġ... , Ġ\ vec { n } _ k ] $ Ġ. ĠThe Ġpattern Ġvector Ġ$\ vec { v } _ p $ Ġis Ġthen Ġencoded Ġby Ġapplying Ġa Ġrecurrent Ġneural Ġnetwork . </s> Rec urrent ĠNeural ĠNetwork Ġwith ĠLong ĠShort - Term ĠMemory ĠUnits </s> A Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġis Ġsuitable Ġfor Ġmodeling Ġsequential Ġdata Ġby Ġa Ġvector Ġrepresentation . ĠIn Ġour Ġmethods , Ġwe Ġuse Ġa Ġlong Ġshort - term Ġmemory Ġ( L ST M ) Ġnetwork , Ġa Ġvariant Ġof Ġa Ġrecurrent Ġneural Ġnetwork Ġto Ġencode Ġpatterns , Ġfor Ġthe Ġfollowing Ġreasons . ĠGiven Ġa Ġsequence Ġof Ġwords Ġ$ p =[ n _ 1 , n _ 2 ,... , n _ k ] $ Ġas Ġinput Ġdata , Ġan ĠR NN Ġprocesses Ġeach Ġword Ġ$ n _ t $ Ġat Ġa Ġtime , Ġand Ġreturns Ġa Ġvector Ġof Ġstate Ġ$ h _ k $ Ġfor Ġthe Ġcomplete Ġinput Ġsequence . ĠFor Ġeach Ġtime Ġstep Ġ$ t $ Ġ, Ġthe ĠR NN Ġupdates Ġan Ġinternal Ġmemory Ġstate Ġ$ h _ t $ Ġwhich Ġdepends Ġon Ġthe Ġcurrent Ġinput Ġ$ n _ t $ Ġand Ġthe Ġprevious Ġstate Ġ$ h _{ t - 1 } $ Ġ. ĠYet , Ġif Ġthe Ġsequential Ġinput Ġis Ġa Ġlong - term Ġdependency , Ġan ĠR NN Ġfaces Ġthe Ġproblem Ġof Ġgradient Ġvanishing Ġor Ġexploding , Ġleading Ġto Ġdifficulties Ġin Ġtraining Ġthe Ġmodel . </s> L ST M Ġunits Ġaddress Ġthese Ġproblems . ĠThe Ġunderlying Ġidea Ġof Ġan ĠL ST M Ġis Ġto Ġuse Ġan Ġadaptive Ġg ating Ġmechanism Ġto Ġdecide Ġon Ġthe Ġdegree Ġthat ĠL ST M Ġunits Ġkeep Ġthe Ġprevious Ġstate Ġand Ġmemor ize Ġthe Ġextracted Ġfeatures Ġof Ġthe Ġcurrent Ġinput . ĠMore Ġspecifically , Ġan ĠL ST M Ġcomprises Ġfour Ġcomponents : Ġan Ġinput Ġgate Ġ$ i _ t $ Ġ, Ġa Ġforget Ġgate Ġ$ f _ t $ Ġ, Ġan Ġoutput Ġgate Ġ$ o _ t $ Ġ, Ġand Ġa Ġmemory Ġcell Ġ$ c _ t $ Ġ. ĠThe Ġstate Ġof Ġan ĠL ST M Ġat Ġeach Ġtime Ġstep Ġ$ t $ Ġis Ġformal ized Ġas Ġfollows : Ġ$ Ġ\ begin { array }{ l } Ġi _ t Ġ= Ġ\ s igma Ġ( W _ i Ġ\ cd ot Ġx _ t Ġ+ ĠU _ i Ġ\ cd ot Ġh _{ t - 1 } Ġ+ Ġb _ i ) Ġ\\ Ġf _ t Ġ= Ġ\ s igma Ġ( W _ f Ġ\ cd ot Ġx _ t Ġ+ ĠU _ f Ġ\ cd ot Ġh _{ t - 1 } Ġ+ Ġb _ f ) Ġ\\ Ġo _ t Ġ= Ġ\ s igma Ġ( W _ o Ġ\ cd ot Ġx _ t Ġ+ ĠU _ o Ġ\ cd ot Ġh _{ t - 1 } Ġ+ Ġb _ o ) Ġ\\ Ġg _ t Ġ= Ġ\ tan h Ġ( W _ c Ġ\ cd ot Ġx _ t Ġ+ ĠU _ c Ġ\ cd ot Ġh _{ t - 1 } Ġ+ Ġb _ c ) Ġ\\ Ġc _ t Ġ= Ġi _ t Ġ\ ot imes Ġg _ t Ġ+ Ġf _ t Ġ\ ot imes Ġc _{ t - 1 } Ġ\ end { array } Ġ$ </s> $ W $ Ġrefers Ġto Ġa Ġmatrix Ġof Ġweights Ġthat Ġprojects Ġinformation Ġbetween Ġtwo Ġlayers ; Ġ$ b $ Ġis Ġa Ġlayer - specific Ġvector Ġof Ġbias Ġterms ; Ġ$\ s igma Ġ$ Ġdenotes Ġthe Ġs igm oid Ġfunction . ĠThe Ġoutput Ġof Ġan ĠL ST M Ġat Ġa Ġtime Ġstep Ġ$ t $ Ġis Ġcomputed Ġas Ġfollows : Ġ$ Ġh _ t Ġ= Ġo _ t Ġ\ ot imes Ġ\ tan h Ġ( c _ t ) Ġ$ </s> where Ġ$\ ot imes Ġ$ Ġdenotes Ġelement - wise Ġmultiplication . ĠIn Ġour Ġmethods , Ġwe Ġrely Ġon Ġthe Ġlast Ġstate Ġ$ h _ k $ Ġto Ġrepresent Ġthe Ġvector Ġ$\ vec { v } _ p $ Ġof Ġa Ġpattern Ġ$ p Ġ= Ġ[ \ vec { n } _ 1 , Ġ\ vec { n } _ 2 ,... , \ vec { n } _ k ] $ Ġ. </s> The ĠProp osed ĠAnt Syn NET ĠModel </s> In Ġthis Ġsection , Ġwe Ġpresent Ġtwo Ġmodels Ġto Ġdistinguish Ġan ton y ms Ġfrom Ġsyn onyms . ĠThe Ġfirst Ġmodel Ġmakes Ġuse Ġof Ġpatterns Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms , Ġby Ġusing Ġan ĠL ST M Ġto Ġencode Ġpatterns Ġas Ġvector Ġrepresentations Ġand Ġthen Ġfeeding Ġthose Ġvectors Ġto Ġa Ġlog istic Ġregression Ġlayer Ġ( Section ĠUID 11 Ġ). ĠThe Ġsecond Ġmodel Ġcreates Ġcombined Ġvector Ġrepresentations Ġof Ġword Ġpairs , Ġwhich Ġconc aten ate Ġthe Ġvectors Ġof Ġthe Ġwords Ġand Ġthe Ġpatterns Ġ( Section ĠUID 13 Ġ). </s> In Ġthis Ġmodel , Ġwe Ġmake Ġuse Ġof Ġa Ġrecurrent Ġneural Ġnetwork Ġwith ĠL ST M Ġunits Ġto Ġencode Ġpatterns Ġcontaining Ġa Ġsequence Ġof Ġnodes . ĠFigure Ġ2 Ġillustrates Ġthe ĠAnt Syn NET Ġmodel . ĠGiven Ġa Ġword Ġpair Ġ$( x , y ) $ Ġ, Ġwe Ġinduce Ġpatterns Ġfor Ġ$( x , y ) $ Ġfrom Ġa Ġcorpus , Ġwhere Ġeach Ġpattern Ġrepresents Ġa Ġpath Ġfrom Ġ$ x $ Ġto Ġ$ y $ Ġ( cf . ĠSection Ġ" Bas eline ĠModels " Ġ). ĠWe Ġthen Ġfeed Ġeach Ġpattern Ġ$ p $ Ġof Ġthe Ġword Ġpair Ġ$( x , y ) $ Ġinto Ġan ĠL ST M Ġto Ġobtain Ġ$\ vec { v } _ p $ Ġ, Ġthe Ġvector Ġrepresentation Ġof Ġthe Ġpattern Ġ$ p $ Ġ( cf . ĠSection Ġ" Rec urrent ĠNeural ĠNetwork Ġwith ĠLong ĠShort - Term ĠMemory ĠUnits " Ġ). ĠFor Ġeach Ġword Ġpair Ġ$( x , y ) $ Ġ, Ġthe Ġvector Ġrepresentation Ġof Ġ$( x , y ) $ Ġis Ġcomputed Ġas Ġfollows : </s> $$ \ vec { v } _{ xy } Ġ= Ġ\ frac {\ sum Ġ\ n ol im its Ġ_ { p Ġ\ in ĠP ( x , y )} Ġ{\ vec { v } _ p Ġ\ cd ot Ġc _ p }} {\ sum Ġ\ n ol im its Ġ_ { p Ġ\ in ĠP ( x , y )} Ġ{ c _ p }} $$ Ġ Ġ Ġ( E q . Ġ12 ) </s> $ \ vec { v } _{ xy } $ Ġrefers Ġto Ġthe Ġvector Ġof Ġthe Ġword Ġpair Ġ$( x , y ) $ Ġ; Ġ$ P ( x , y ) $ Ġis Ġthe Ġset Ġof Ġpatterns Ġcorresponding Ġto Ġthe Ġpair Ġ$( x , y ) $ Ġ; Ġ$ c _ p $ Ġis Ġthe Ġfrequency Ġof Ġthe Ġpattern Ġ$ p $ Ġ. ĠThe Ġvector Ġ$\ vec { v } _{ xy } $ Ġis Ġthen Ġfed Ġinto Ġa Ġlog istic Ġregression Ġlayer Ġwhose Ġtarget Ġis Ġthe Ġclass Ġlabel Ġassociated Ġwith Ġthe Ġpair Ġ$( x , y ) $ Ġ. ĠFinally , Ġthe Ġpair Ġ$( x , y ) $ Ġis Ġpredicted Ġas Ġpositive Ġ( i . e ., Ġan ton ym ous ) Ġword Ġpair Ġif Ġthe Ġprobability Ġof Ġthe Ġprediction Ġfor Ġ$\ vec { v } _{ xy } $ Ġis Ġlarger Ġthan Ġ0 . 5 . </s> Insp ired Ġby Ġthe Ġsupervised Ġdistribution al Ġconc aten ation Ġmethod Ġin ĠBar oni : 12 Ġand Ġthe Ġintegrated Ġpath - based Ġand Ġdistribution al Ġmethod Ġfor Ġhyper ny my Ġdetection Ġin ĠSh w artz : 16 , Ġwe Ġtake Ġinto Ġaccount Ġthe Ġpatterns Ġand Ġdistribution Ġof Ġtarget Ġpairs Ġto Ġcreate Ġtheir Ġcombined Ġvector Ġrepresentations . ĠGiven Ġa Ġword Ġpair Ġ$( x , y ) $ Ġ, Ġthe Ġcombined Ġvector Ġrepresentation Ġof Ġthe Ġpair Ġ$( x , y ) $ Ġis Ġdetermined Ġby Ġusing Ġboth Ġthe Ġco - occ urrence Ġdistribution Ġof Ġthe Ġwords Ġand Ġthe Ġsynt actic Ġpath Ġpatterns : </s> $$ \ vec { v } _{ comb ( x , y )} Ġ= Ġ[ \ vec { v } _ x Ġ\ opl us Ġ\ vec { v } _{ xy } Ġ\ opl us Ġ\ vec { v } _ y ] $$ Ġ Ġ Ġ( E q . Ġ14 ) </s> $ \ vec { v } _{ comb ( x , y )} $ Ġrefers Ġto Ġthe Ġcombined Ġvector Ġof Ġthe Ġword Ġpair Ġ$( x , y ) $ Ġ; Ġ$\ vec { v } _ x $ Ġand Ġ$\ vec { v } _ y $ Ġare Ġthe Ġvectors Ġof Ġword Ġ$ x $ Ġand Ġword Ġ$ y $ Ġ, Ġrespectively ; Ġ$\ vec { v } _{ xy } $ Ġis Ġthe Ġvector Ġof Ġthe Ġpattern Ġthat Ġcorresponds Ġto Ġthe Ġpair Ġ$( x , y ) $ Ġ, Ġcf . ĠSection ĠUID 11 Ġ. ĠSimilar Ġto Ġthe Ġpattern - based Ġmodel , Ġthe Ġcombined Ġvector Ġ$\ vec { v } _{ comb ( x , y )} $ Ġis Ġfed Ġinto Ġthe Ġlog istic Ġregression Ġlayer Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms . </s> Bas eline ĠModels </s> To Ġcompare ĠAnt Syn NET Ġwith Ġbaseline Ġmodels Ġfor Ġpattern - based Ġclassification Ġof Ġan ton y ms Ġand Ġsyn onyms , Ġwe Ġintroduce Ġtwo Ġpattern - based Ġbaseline Ġmethods : Ġthe Ġdistribution al Ġmethod Ġ( Section Ġ" Dist ribution al ĠBas eline " Ġ), Ġand Ġthe Ġdistributed Ġmethod Ġ( Section Ġ" Dist ributed ĠBas eline " Ġ). </s> Dist ribution al ĠBas eline </s> As Ġa Ġfirst Ġbaseline , Ġwe Ġapply Ġthe Ġapproach Ġby ĠRoth 2014 , Ġhence forth ĠR & Si W . ĠThey Ġused Ġa Ġvector Ġspace Ġmodel Ġto Ġrepresent Ġpairs Ġof Ġwords Ġby Ġa Ġcombination Ġof Ġstandard Ġlex ico - sy nt actic Ġpatterns Ġand Ġdiscourse Ġmarkers . ĠIn Ġaddition Ġto Ġthe Ġpatterns , Ġthe Ġdiscourse Ġmarkers Ġadded Ġinformation Ġto Ġexpress Ġdiscourse Ġrelations , Ġwhich Ġin Ġturn Ġmay Ġindicate Ġthe Ġspecific Ġsemantic Ġrelation Ġbetween Ġthe Ġtwo Ġwords Ġin Ġa Ġword Ġpair . ĠFor Ġexample , Ġcontrast Ġrelations Ġmight Ġindicate Ġan ton y my , Ġwhereas Ġelabor ations Ġmay Ġindicate Ġsyn ony my Ġor Ġhyp ony my . </s> Michael ĠRoth , Ġthe Ġfirst Ġauthor Ġof ĠR & Si W , Ġkindly Ġcomputed Ġthe Ġrelation Ġclassification Ġresults Ġof Ġthe Ġpattern âĢĵ disc ourse Ġmodel Ġfor Ġour Ġtest Ġsets . ĠThe Ġweights Ġbetween Ġmarker - based Ġand Ġpattern - based Ġmodels Ġwere Ġtuned Ġon Ġthe Ġvalidation Ġsets ; Ġother Ġhyper param eters Ġwere Ġset Ġexactly Ġas Ġdescribed Ġby Ġthe ĠR & Si W Ġmethod . </s> Dist ributed ĠBas eline </s> The ĠSP Ġmethod Ġproposed Ġby ĠSchwartz : 15 Ġuses Ġsymm etric Ġpatterns Ġfor Ġgenerating Ġword Ġembed d ings . ĠIn Ġthis Ġwork , Ġthe Ġauthors Ġapplied Ġan Ġun super vised Ġalgorithm Ġfor Ġthe Ġautomatic Ġextraction Ġof Ġsymm etric Ġpatterns Ġfrom Ġplain Ġtext . ĠThe Ġsymm etric Ġpatterns Ġwere Ġdefined Ġas Ġa Ġsequence Ġof Ġ3 - 5 Ġtokens Ġconsisting Ġof Ġexactly Ġtwo Ġwild cards Ġand Ġ1 - 3 Ġwords . ĠThe Ġpatterns Ġwere Ġfiltered Ġbased Ġon Ġtheir Ġfrequencies , Ġsuch Ġthat Ġthe Ġresulting Ġpattern Ġset Ġcontained Ġ11 Ġpatterns . ĠFor Ġgenerating Ġword Ġembed d ings , Ġa Ġmatrix Ġof Ġco - occ urrence Ġcounts Ġbetween Ġpatterns Ġand Ġwords Ġin Ġthe Ġvocabulary Ġwas Ġcomputed , Ġusing Ġpositive Ġpoint - wise Ġmutual Ġinformation . ĠThe Ġsp arsity Ġproblem Ġof Ġvector Ġrepresentations Ġwas Ġaddressed Ġby Ġsmoot hing . ĠFor Ġan ton ym Ġrepresentation , Ġthe Ġauthors Ġrelied Ġon Ġtwo Ġpatterns Ġsuggested Ġby ĠLin 2003 Ġto Ġconstruct Ġword Ġembed d ings Ġcontaining Ġan Ġan ton ym Ġparameter Ġthat Ġcan Ġbe Ġturned Ġon Ġin Ġorder Ġto Ġrepresent Ġan ton y ms Ġas Ġdiss imilar , Ġand Ġthat Ġcan Ġbe Ġturned Ġoff Ġto Ġrepresent Ġan ton y ms Ġas Ġsimilar . </s> To Ġapply Ġthe ĠSP Ġmethod Ġto Ġour Ġdata , Ġwe Ġmake Ġuse Ġof Ġthe Ġpre - trained ĠSP Ġembed d ings Ġwith Ġ500 Ġdimensions . ĠWe Ġcalculate Ġthe Ġcos ine Ġsimilarity Ġof Ġword Ġpairs Ġand Ġthen Ġuse Ġa ĠSupport ĠVector ĠMachine Ġwith ĠRad ial ĠBas is ĠFunction Ġkernel Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms . </s> Dat as et </s> For Ġtraining Ġthe Ġmodels , Ġneural Ġnetworks Ġrequire Ġa Ġlarge Ġamount Ġof Ġtraining Ġdata . ĠWe Ġuse Ġthe Ġexisting Ġlarge - scale Ġan ton ym Ġand Ġsyn onym Ġpairs Ġpreviously Ġused Ġby ĠNguyen : 16 . ĠOriginally , Ġthe Ġdata Ġpairs Ġwere Ġcollected Ġfrom ĠWord Net ĠB IB REF 9 Ġand ĠWord nik . </s> In Ġorder Ġto Ġinduce Ġpatterns Ġfor Ġthe Ġword Ġpairs Ġin Ġthe Ġdataset , Ġwe Ġidentify Ġthe Ġsentences Ġin Ġthe Ġcorpus Ġthat Ġcontain Ġthe Ġword Ġpair . ĠThere after , Ġwe Ġextract Ġall Ġpatterns Ġfor Ġthe Ġword Ġpair . ĠWe Ġfilter Ġout Ġall Ġpatterns Ġwhich Ġoccur Ġless Ġthan Ġfive Ġtimes ; Ġand Ġwe Ġonly Ġtake Ġinto Ġaccount Ġword Ġpairs Ġthat Ġcontain Ġat Ġleast Ġfive Ġpatterns Ġfor Ġtraining , Ġvalid ating Ġand Ġtesting . ĠFor Ġthe Ġproportion Ġof Ġpositive Ġand Ġnegative Ġpairs , Ġwe Ġkeep Ġa Ġratio Ġof Ġ1 : 1 Ġpositive Ġ( ant onym ) Ġto Ġnegative Ġ( syn onym ) Ġpairs Ġin Ġthe Ġdataset . ĠIn Ġorder Ġto Ġcreate Ġthe Ġsets Ġof Ġtraining , Ġtesting Ġand Ġvalidation Ġdata , Ġwe Ġperform Ġrandom Ġsplitting Ġwith Ġ70 % Ġtrain , Ġ25 % Ġtest , Ġand Ġ5 % Ġvalidation Ġsets . ĠThe Ġfinal Ġdataset Ġcontains Ġthe Ġnumber Ġof Ġword Ġpairs Ġaccording Ġto Ġword Ġclasses Ġdescribed Ġin ĠTable Ġ1 Ġ. ĠMoreover , ĠTable Ġ2 Ġshows Ġthe Ġaverage Ġnumber Ġof Ġpatterns Ġfor Ġeach Ġword Ġpair Ġin Ġour Ġdataset . </s> Exper imental ĠSettings </s> We Ġuse Ġthe ĠEnglish ĠWikipedia Ġdump Ġfrom ĠJune Ġ2016 Ġas Ġthe Ġcorpus Ġresource Ġfor Ġour Ġmethods Ġand Ġbas elines . ĠFor Ġparsing Ġthe Ġcorpus , Ġwe Ġrely Ġon Ġspa Cy . ĠFor Ġthe Ġle mma Ġembed d ings , Ġwe Ġrely Ġon Ġthe Ġword Ġembed d ings Ġof Ġthe Ġd L CE Ġmodel ĠB IB REF 10 Ġwhich Ġis Ġthe Ġstate - of - the - art Ġvector Ġrepresentation Ġfor Ġdistinguishing Ġan ton y ms Ġfrom Ġsyn onyms . ĠWe Ġre - im ple mented Ġthis Ġcutting - edge Ġmodel Ġon ĠWikipedia Ġwith Ġ100 Ġdimensions , Ġand Ġthen Ġmake Ġuse Ġof Ġthe Ġd L CE Ġword Ġembed d ings Ġfor Ġinitialization Ġthe Ġle mma Ġembed d ings . ĠThe Ġembed d ings Ġof ĠPOS Ġtags , Ġdependency Ġlabels , Ġdistance Ġlabels , Ġand Ġout - of - voc abulary Ġle mm as Ġare Ġinitialized Ġrandomly . ĠThe Ġnumber Ġof Ġdimensions Ġis Ġset Ġto Ġ10 Ġfor Ġthe Ġembed d ings Ġof ĠPOS Ġtags , Ġdependency Ġlabels Ġand Ġdistance Ġlabels . ĠWe Ġuse Ġthe Ġvalidation Ġsets Ġto Ġtune Ġthe Ġnumber Ġof Ġdimensions Ġfor Ġthese Ġlabels . ĠFor Ġoptimization , Ġwe Ġrely Ġon Ġthe Ġcross - ent ropy Ġloss Ġfunction Ġand ĠSt och astic ĠGrad ient ĠDes cent Ġwith Ġthe ĠAd ad elta Ġupdate Ġrule ĠB IB REF 11 Ġ. ĠFor Ġtraining , Ġwe Ġuse Ġthe ĠThe ano Ġframework ĠB IB REF 12 Ġ. ĠRegular ization Ġis Ġapplied Ġby Ġa Ġdrop out Ġof Ġ0 . 5 Ġon Ġeach Ġof Ġcomponent 's Ġembed d ings Ġ( drop out Ġrate Ġis Ġtuned Ġon Ġthe Ġvalidation Ġset ). ĠWe Ġtrain Ġthe Ġmodels Ġwith Ġ40 Ġepoch s Ġand Ġupdate Ġall Ġembed d ings Ġduring Ġtraining . </s> Overall ĠResults </s> Table Ġ3 Ġshows Ġthe Ġsignificant Ġperformance Ġof Ġour Ġmodels Ġin Ġcomparison Ġto Ġthe Ġbas elines . ĠCon cerning Ġadject ives , Ġthe Ġtwo Ġproposed Ġmodels Ġsignificantly Ġoutper form Ġthe Ġtwo Ġbas elines : ĠThe Ġperformance Ġof Ġthe Ġbas elines Ġis Ġaround Ġ. 72 Ġfor Ġ$ F _ 1 $ Ġ, Ġand Ġthe Ġcorresponding Ġresults Ġfor Ġthe Ġcombined ĠAnt Syn NET Ġmodel Ġachieve Ġan Ġimprovement Ġof Ġ$ > $ Ġ. 06 . ĠRegarding Ġnoun s , Ġthe Ġimprovement Ġof Ġthe Ġnew Ġmethods Ġis Ġjust Ġ. 02 Ġ$ F _ 1 $ Ġin Ġcomparison Ġto Ġthe ĠR & Si W Ġbaseline , Ġbut Ġwe Ġachieve Ġa Ġmuch Ġbetter Ġperformance Ġin Ġcomparison Ġto Ġthe ĠSP Ġbaseline , Ġan Ġincrease Ġof Ġ. 37 Ġ$ F _ 1 $ Ġ. ĠRegarding Ġverbs , Ġwe Ġdo Ġnot Ġoutper form Ġthe Ġmore Ġadvanced ĠR & Si W Ġbaseline Ġin Ġterms Ġof Ġthe Ġ$ F _ 1 $ Ġscore , Ġbut Ġwe Ġobtain Ġhigher Ġrecall Ġscores . ĠIn Ġcomparison Ġto Ġthe ĠSP Ġbaseline , Ġour Ġmodels Ġstill Ġshow Ġa Ġclear Ġ$ F _ 1 $ Ġimprovement . </s> Overall , Ġour Ġproposed Ġmodels Ġachieve Ġcomparatively Ġhigh Ġrecall Ġscores Ġcompared Ġto Ġthe Ġtwo Ġbas elines . ĠThis Ġstrengthens Ġour Ġhypothesis Ġthat Ġthere Ġis Ġa Ġhigher Ġpossibility Ġfor Ġthe Ġco - occ urrence Ġof Ġan ton ym ous Ġpairs Ġin Ġpatterns Ġover Ġsynonymous Ġpairs Ġwithin Ġa Ġsentence . ĠBecause , Ġwhen Ġthe Ġproposed Ġmodels Ġobtain Ġhigh Ġrecall Ġscores , Ġthe Ġmodels Ġare Ġable Ġto Ġretrieve Ġmost Ġrelevant Ġinformation Ġ( ant onymous Ġpairs ) Ġcorresponding Ġto Ġthe Ġpatterns . ĠRegarding Ġthe Ġlow Ġprecision Ġin Ġthe Ġtwo Ġproposed Ġmodels , Ġwe Ġsampled Ġrandomly Ġ5 Ġpairs Ġin Ġeach Ġpopulation : Ġtrue Ġpositive , Ġtrue Ġnegative , Ġfalse Ġpositive , Ġfalse Ġnegative . ĠWe Ġthen Ġcompared Ġthe Ġoverlap Ġof Ġpatterns Ġfor Ġthe Ġtrue Ġpredictions Ġ( true Ġpositive Ġpairs Ġand Ġtrue Ġnegative Ġpairs ) Ġand Ġthe Ġfalse Ġpredictions Ġ( false Ġpositive Ġpairs Ġand Ġfalse Ġnegative Ġpairs ). ĠWe Ġfound Ġout Ġthat Ġthere Ġis Ġno Ġoverlap Ġbetween Ġpatterns Ġof Ġtrue Ġpredictions ; Ġand Ġthe Ġnumber Ġoverlap Ġbetween Ġpatterns Ġof Ġfalse Ġpredictions Ġis Ġ2 , Ġ2 , Ġand Ġ4 Ġpatterns Ġfor Ġnoun , Ġadjective , Ġand Ġverb Ġclasses , Ġrespectively . ĠThis Ġshows Ġthat Ġthe Ġlow Ġprecision Ġof Ġour Ġmodels Ġstems Ġfrom Ġthe Ġpatterns Ġwhich Ġrepresent Ġboth Ġan ton ym ous Ġand Ġsynonymous Ġpairs . </s> Effect Ġof Ġthe ĠDistance ĠFeature </s> In Ġour Ġmodels , Ġthe Ġnovel Ġdistance Ġfeature Ġis Ġsuccessfully Ġintegrated Ġalong Ġthe Ġsynt actic Ġpath Ġto Ġrepresent Ġlex ico - sy nt actic Ġpatterns . ĠThe Ġintuition Ġbehind Ġthe Ġdistance Ġfeature Ġexploits Ġproperties Ġof Ġtrees Ġin Ġgraph Ġtheory , Ġwhich Ġshow Ġthat Ġthere Ġexist Ġdifferences Ġin Ġthe Ġdegree Ġof Ġrelationship Ġbetween Ġthe Ġparent Ġnode Ġand Ġthe Ġchild Ġnodes Ġ( Ġ$ distance Ġ= Ġ1 $ Ġ) Ġand Ġin Ġthe Ġdegree Ġof Ġrelationship Ġbetween Ġthe Ġancestor Ġnode Ġand Ġthe Ġdescendant Ġnodes Ġ( Ġ$ distance Ġ> Ġ1 $ Ġ). ĠHence , Ġwe Ġuse Ġthe Ġdistance Ġfeature Ġto Ġeffectively Ġcapture Ġthese Ġrelationships . </s> In Ġorder Ġto Ġevaluate Ġthe Ġeffect Ġof Ġour Ġnovel Ġdistance Ġfeature , Ġwe Ġcompare Ġthe Ġdistance Ġfeature Ġto Ġthe Ġdirection Ġfeature Ġproposed Ġby ĠSh w artz : 16 . ĠIn Ġtheir Ġapproach , Ġthe Ġauthors Ġcombined Ġle mma , ĠPOS , Ġdependency , Ġand Ġdirection Ġfeatures Ġfor Ġthe Ġtask Ġof Ġhyper ny m Ġdetection . ĠThe Ġdirection Ġfeature Ġrepresented Ġthe Ġdirection Ġof Ġthe Ġdependency Ġlabel Ġbetween Ġtwo Ġnodes Ġin Ġa Ġpath Ġfrom ĠX Ġto ĠY . </s> For Ġevaluation , Ġwe Ġmake Ġuse Ġof Ġthe Ġsame Ġinformation Ġregarding Ġdataset Ġand Ġpatterns Ġas Ġin ĠSection Ġ" Overall ĠResults " Ġ, Ġand Ġthen Ġreplace Ġthe Ġdistance Ġfeature Ġby Ġthe Ġdirection Ġfeature . ĠThe Ġresults Ġare Ġshown Ġin ĠTable Ġ4 Ġ. ĠThe Ġdistance Ġfeature Ġenhances Ġthe Ġperformance Ġof Ġour Ġproposed Ġmodels Ġmore Ġeffectively Ġthan Ġthe Ġdirection Ġfeature Ġdoes , Ġacross Ġall Ġword Ġclasses . </s> Effect Ġof ĠWord ĠEmb edd ings </s> Our Ġmethods Ġrely Ġon Ġthe Ġword Ġembed d ings Ġof Ġthe Ġd L CE Ġmodel , Ġstate - of - the - art Ġword Ġembed d ings Ġfor Ġan ton ym - syn onym Ġdistinction . ĠYet , Ġthe Ġword Ġembed d ings Ġof Ġthe Ġd L CE Ġmodel , Ġi . e ., Ġsupervised Ġword Ġembed d ings , Ġrepresent Ġinformation Ġcollected Ġfrom Ġlex ical Ġresources . ĠIn Ġorder Ġto Ġevaluate Ġthe Ġeffect Ġof Ġthese Ġword Ġembed d ings Ġon Ġthe Ġperformance Ġof Ġour Ġmodels , Ġwe Ġreplace Ġthem Ġby Ġthe Ġpre - trained ĠGl o Ve Ġword Ġembed d ings Ġwith Ġ100 Ġdimensions , Ġand Ġcompare Ġthe Ġeffects Ġof Ġthe ĠGl o Ve Ġword Ġembed d ings Ġand Ġthe Ġd L CE Ġword Ġembed d ings Ġon Ġthe Ġperformance Ġof Ġthe Ġtwo Ġproposed Ġmodels . </s> Table Ġ5 Ġillustrates Ġthe Ġperformance Ġof Ġour Ġtwo Ġmodels Ġon Ġall Ġword Ġclasses . ĠThe Ġtable Ġshows Ġthat Ġthe Ġd L CE Ġword Ġembed d ings Ġare Ġbetter Ġthan Ġthe Ġpre - trained ĠGl o Ve Ġword Ġembed d ings , Ġby Ġaround Ġ. 01 Ġ$ F _ 1 $ Ġfor Ġthe Ġpattern - based ĠAnt Syn NET Ġmodel Ġand Ġthe Ġcombined ĠAnt Syn NET Ġmodel Ġregarding Ġadjective Ġand Ġverb Ġpairs . ĠRegarding Ġnoun Ġpairs , Ġthe Ġimprovements Ġof Ġthe Ġd L CE Ġword Ġembed d ings Ġover Ġpre - trained ĠGl o Ve Ġword Ġembed d ings Ġachieve Ġaround Ġ. 01 Ġand Ġ. 04 Ġ$ F _ 1 $ Ġfor Ġthe Ġpattern - based Ġmodel Ġand Ġthe Ġcombined Ġmodel , Ġrespectively . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġnovel Ġpattern - based Ġneural Ġmethod ĠAnt Syn NET Ġto Ġdistinguish Ġan ton y ms Ġfrom Ġsyn onyms . ĠWe Ġhypothesized Ġthat Ġan ton ym ous Ġword Ġpairs Ġco - occ ur Ġwith Ġeach Ġother Ġin Ġlex ico - sy nt actic Ġpatterns Ġwithin Ġa Ġsentence Ġmore Ġoften Ġthan Ġsynonymous Ġword Ġpairs . </s> The Ġpatterns Ġwere Ġderived Ġfrom Ġthe Ġsimple Ġpaths Ġbetween Ġsem antically Ġrelated Ġwords Ġin Ġa Ġsynt actic Ġparse Ġtree . ĠIn Ġaddition Ġto Ġlex ical Ġand Ġsynt actic Ġinformation , Ġwe Ġsuggested Ġa Ġnovel Ġpath Ġdistance Ġfeature . ĠThe ĠAnt Syn NET Ġmodel Ġconsists Ġof Ġtwo Ġapproaches Ġto Ġclassify Ġan ton y ms Ġand Ġsyn onyms . ĠIn Ġthe Ġfirst Ġapproach , Ġwe Ġused Ġa Ġrecurrent Ġneural Ġnetwork Ġwith Ġlong Ġshort - term Ġmemory Ġunits Ġto Ġencode Ġthe Ġpatterns Ġas Ġvector Ġrepresentations ; Ġin Ġthe Ġsecond Ġapproach , Ġwe Ġmade Ġuse Ġof Ġthe Ġdistribution Ġand Ġencoded Ġpatterns Ġof Ġthe Ġtarget Ġpairs Ġto Ġgenerate Ġcombined Ġvector Ġrepresentations . ĠThe Ġresulting Ġvectors Ġof Ġpatterns Ġin Ġboth Ġapproaches Ġwere Ġfed Ġinto Ġthe Ġlog istic Ġregression Ġlayer Ġfor Ġclassification . </s> Our Ġproposed Ġmodels Ġsignificantly Ġoutper formed Ġtwo Ġbas elines Ġrelying Ġon Ġprevious Ġwork , Ġmainly Ġin Ġterms Ġof Ġrecall . ĠMoreover , Ġwe Ġdemonstrated Ġthat Ġthe Ġdistance Ġfeature Ġoutper formed Ġa Ġpreviously Ġsuggested Ġdirection Ġfeature , Ġand Ġthat Ġour Ġembed d ings Ġoutper formed Ġthe Ġstate - of - the - art ĠGl o Ve Ġembed d ings . ĠLast Ġbut Ġnot Ġleast , Ġour Ġtwo Ġproposed Ġmodels Ġonly Ġrely Ġon Ġcorpus Ġdata , Ġsuch Ġthat Ġthe Ġmodels Ġare Ġeasily Ġapplicable Ġto Ġother Ġlanguages Ġand Ġrelations . </s> Acknowled gements </s> We Ġwould Ġlike Ġto Ġthank ĠMichael ĠRoth Ġfor Ġhelping Ġus Ġto Ġcompute Ġthe Ġresults Ġof Ġthe ĠR & Si W Ġmodel Ġon Ġour Ġdataset . </s> The Ġresearch Ġwas Ġsupported Ġby Ġthe ĠMinistry Ġof ĠEducation Ġand ĠTraining Ġof Ġthe ĠSocialist ĠRepublic Ġof ĠVietnam Ġ( Sch olars hip Ġ9 77 / Q D - BG DD T ; ĠKim - An h ĠNguyen ), Ġthe ĠD FG ĠCollabor ative ĠResearch ĠCentre ĠSF B Ġ7 32 Ġ( Kim - An h ĠNguyen , ĠNg oc ĠTh ang ĠVu ), Ġand Ġthe ĠD FG ĠHe isen berg ĠFellowship ĠSCH U - 25 80 / 1 Ġ( Sab ine ĠSch ult e Ġim ĠWald e ).