we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes, In case of a tie, we pick one of the most frequent classes randomly
EXTRACTIVE
Among all the classes predicted by several models, for each test sentence, class with most votes are picked. In case of a tie, one of the most frequent classes are picked randomly.
ABSTRACTIVE
Finally, we combine our CNN and RNN models using a voting process. For each sentence in the test set, we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes. In case of a tie, we pick one of the most frequent classes randomly. The combination achieves an F1 score of 84.9 which is better than the performance of the two NN types alone. It, thus, confirms our assumption that the networks provide complementary information: while the RNN computes a weighted combination of all words in the sentence, the CNN extracts the most informative n-grams for the relation and only considers their resulting activations.
Finally, we combine our CNN and RNN models using a voting process. For each sentence in the test set, we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes. In case of a tie, we pick one of the most frequent classes randomly. The combination achieves an F1 score of 84.9 which is better than the performance of the two NN types alone. It, thus, confirms our assumption that the networks provide complementary information: while the RNN computes a weighted combination of all words in the sentence, the CNN extracts the most informative n-grams for the relation and only considers their resulting activations.
1605.07333
Introduction </s> Rel ation Ġclassification Ġis Ġthe Ġtask Ġof Ġassigning Ġsentences Ġwith Ġtwo Ġmarked Ġentities Ġto Ġa Ġpred efined Ġset Ġof Ġrelations . ĠThe Ġsentence ĠâĢ ľ We Ġpoured Ġthe Ġ< e 1 > mil k </ e 1 > Ġinto Ġthe Ġ< e 2 > p ump kin Ġmixture </ e 2 >. âĢ Ŀ , Ġfor Ġexample , Ġexpresses Ġthe Ġrelation ĠEntity - Dest ination ( e 1 , e 2 ). ĠWhile Ġearly Ġresearch Ġmostly Ġfocused Ġon Ġsupport Ġvector Ġmachines Ġor Ġmaximum Ġentropy Ġclass ifiers ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġrecent Ġresearch Ġshowed Ġperformance Ġimprovements Ġby Ġapplying Ġneural Ġnetworks Ġ( NN s ) ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġon Ġthe Ġbenchmark Ġdata Ġfrom ĠSem E val Ġ2010 Ġshared Ġtask Ġ8 ĠB IB REF 8 Ġ. </s> This Ġstudy Ġinvestigates Ġtwo Ġdifferent Ġtypes Ġof ĠN Ns : Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ) Ġand Ġconv olution al Ġneural Ġnetworks Ġ( CNN s ) Ġas Ġwell Ġas Ġtheir Ġcombination . ĠWe Ġmake Ġthe Ġfollowing Ġcontributions : </s> ( 1 ) ĠWe Ġpropose Ġextended Ġmiddle Ġcontext , Ġa Ġnew Ġcontext Ġrepresentation Ġfor ĠCNN s Ġfor Ġrelation Ġclassification . ĠThe Ġextended Ġmiddle Ġcontext Ġuses Ġall Ġparts Ġof Ġthe Ġsentence Ġ( the Ġrelation Ġarguments , Ġleft Ġof Ġthe Ġrelation Ġarguments , Ġbetween Ġthe Ġarguments , Ġright Ġof Ġthe Ġarguments ) Ġand Ġpays Ġspecial Ġattention Ġto Ġthe Ġmiddle Ġpart . </s> ( 2 ) ĠWe Ġpresent Ġconnection ist Ġbi - direction al ĠR NN Ġmodels Ġwhich Ġare Ġespecially Ġsuited Ġfor Ġsentence Ġclassification Ġtasks Ġsince Ġthey Ġcombine Ġall Ġintermediate Ġhidden Ġlayers Ġfor Ġtheir Ġfinal Ġdecision . ĠFurthermore , Ġthe Ġranking Ġloss Ġfunction Ġis Ġintroduced Ġfor Ġthe ĠR NN Ġmodel Ġoptimization Ġwhich Ġhas Ġnot Ġbeen Ġinvestigated Ġin Ġthe Ġliterature Ġfor Ġrelation Ġclassification Ġbefore . </s> ( 3 ) ĠFinally , Ġwe Ġcombine ĠCNN s Ġand ĠR NN s Ġusing Ġa Ġsimple Ġvoting Ġscheme Ġand Ġachieve Ġnew Ġstate - of - the - art Ġresults Ġon Ġthe ĠSem E val Ġ2010 Ġbenchmark Ġdataset . </s> Related ĠWork </s> In Ġ2010 , Ġmanually Ġannot ated Ġdata Ġfor Ġrelation Ġclassification Ġwas Ġreleased Ġin Ġthe Ġcontext Ġof Ġa ĠSem E val Ġshared Ġtask ĠB IB REF 8 Ġ. ĠShared Ġtask Ġparticipants Ġused , Ġi . a ., Ġsupport Ġvector Ġmachines Ġor Ġmaximum Ġentropy Ġclass ifiers ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠRecently , Ġtheir Ġresults Ġon Ġthis Ġdata Ġset Ġwere Ġoutper formed Ġby Ġapplying ĠN Ns ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> z eng 2014 Ġbuilt Ġa ĠCNN Ġbased Ġonly Ġon Ġthe Ġcontext Ġbetween Ġthe Ġrelation Ġarguments Ġand Ġextended Ġit Ġwith Ġseveral Ġlex ical Ġfeatures . Ġk im 2014 Ġand Ġothers Ġused Ġconv olution al Ġfilters Ġof Ġdifferent Ġsizes Ġfor ĠCNN s . Ġn guyen Ġapplied Ġthis Ġto Ġrelation Ġclassification Ġand Ġobtained Ġimprovements Ġover Ġsingle Ġfilter Ġsizes . Ġde S ant os 2015 Ġreplaced Ġthe Ġsoft max Ġlayer Ġof Ġthe ĠCNN Ġwith Ġa Ġranking Ġlayer . ĠThey Ġshowed Ġimprovements Ġand Ġpublished Ġthe Ġbest Ġresult Ġso Ġfar Ġon Ġthe ĠSem E val Ġdataset , Ġto Ġour Ġknowledge . </s> soc her Ġused Ġanother ĠN N Ġarchitecture Ġfor Ġrelation Ġclassification : Ġrecursive Ġneural Ġnetworks Ġthat Ġbuilt Ġrecursive Ġsentence Ġrepresentations Ġbased Ġon Ġsynt actic Ġparsing . ĠIn Ġcontrast , Ġz hang Ġinvestigated Ġa Ġtemporal Ġstructured ĠR NN Ġwith Ġonly Ġwords Ġas Ġinput . ĠThey Ġused Ġa Ġbi - direction al Ġmodel Ġwith Ġa Ġpool ing Ġlayer Ġon Ġtop . </s> Con v olution al ĠNeural ĠNetworks Ġ( CNN ) </s> CNN s Ġperform Ġa Ġdiscrete Ġconv olution Ġon Ġan Ġinput Ġmatrix Ġwith Ġa Ġset Ġof Ġdifferent Ġfilters . ĠFor ĠN LP Ġtasks , Ġthe Ġinput Ġmatrix Ġrepresents Ġa Ġsentence : ĠEach Ġcolumn Ġof Ġthe Ġmatrix Ġstores Ġthe Ġword Ġembed ding Ġof Ġthe Ġcorresponding Ġword . ĠBy Ġapplying Ġa Ġfilter Ġwith Ġa Ġwidth Ġof , Ġe . g ., Ġthree Ġcolumns , Ġthree Ġneighboring Ġwords Ġ( tr ig ram ) Ġare Ġconv olved . ĠAfterwards , Ġthe Ġresults Ġof Ġthe Ġconv olution Ġare Ġpooled . ĠFollowing Ġcoll ober t West on , Ġwe Ġperform Ġmax - pool ing Ġwhich Ġextracts Ġthe Ġmaximum Ġvalue Ġfor Ġeach Ġfilter Ġand , Ġthus , Ġthe Ġmost Ġinformative Ġn - gram Ġfor Ġthe Ġfollowing Ġsteps . ĠFinally , Ġthe Ġresulting Ġvalues Ġare Ġconc aten ated Ġand Ġused Ġfor Ġclass ifying Ġthe Ġrelation Ġexpressed Ġin Ġthe Ġsentence . </s> Input : ĠExtended ĠMiddle ĠContext </s> One Ġof Ġour Ġcontributions Ġis Ġa Ġnew Ġinput Ġrepresentation Ġespecially Ġdesigned Ġfor Ġrelation Ġclassification . ĠThe Ġcontexts Ġare Ġsplit Ġinto Ġthree Ġdis j oint Ġregions Ġbased Ġon Ġthe Ġtwo Ġrelation Ġarguments : Ġthe Ġleft Ġcontext , Ġthe Ġmiddle Ġcontext Ġand Ġthe Ġright Ġcontext . ĠSince Ġin Ġmost Ġcases Ġthe Ġmiddle Ġcontext Ġcontains Ġthe Ġmost Ġrelevant Ġinformation Ġfor Ġthe Ġrelation , Ġwe Ġwant Ġto Ġfocus Ġon Ġit Ġbut Ġnot Ġignore Ġthe Ġother Ġregions Ġcompletely . ĠHence , Ġwe Ġpropose Ġto Ġuse Ġtwo Ġcontexts : Ġ( 1 ) Ġa Ġcombination Ġof Ġthe Ġleft Ġcontext , Ġthe Ġleft Ġentity Ġand Ġthe Ġmiddle Ġcontext ; Ġand Ġ( 2 ) Ġa Ġcombination Ġof Ġthe Ġmiddle Ġcontext , Ġthe Ġright Ġentity Ġand Ġthe Ġright Ġcontext . ĠDue Ġto Ġthe Ġrepetition Ġof Ġthe Ġmiddle Ġcontext , Ġwe Ġforce Ġthe Ġnetwork Ġto Ġpay Ġspecial Ġattention Ġto Ġit . ĠThe Ġtwo Ġcontexts Ġare Ġprocessed Ġby Ġtwo Ġindependent Ġconv olution al Ġand Ġmax - pool ing Ġlayers . ĠAfter Ġpool ing , Ġthe Ġresults Ġare Ġconc aten ated Ġto Ġform Ġthe Ġsentence Ġrepresentation . ĠFigure ĠFIG REF 3 Ġdepicts Ġthis Ġprocedure . ĠIt Ġshows Ġan Ġexam pl ary Ġsentence : ĠâĢ ľ He Ġhad Ġchest Ġpain Ġand Ġ< e 1 > head aches </ e 1 > Ġfrom Ġ< e 2 > m old </ e 2 > Ġin Ġthe Ġbedroom . âĢ Ŀ ĠIf Ġwe Ġonly Ġconsidered Ġthe Ġmiddle Ġcontext ĠâĢ ľ from âĢ Ŀ , Ġthe Ġnetwork Ġmight Ġbe Ġtempted Ġto Ġpredict Ġa Ġrelation Ġlike ĠEntity - Origin ( e 1 , e 2 ). ĠHowever , Ġby Ġalso Ġtaking Ġthe Ġleft Ġand Ġright Ġcontext Ġinto Ġaccount , Ġthe Ġmodel Ġcan Ġdetect Ġthe Ġrelation ĠCause - Effect ( e 2 , e 1 ). ĠWhile Ġthis Ġcould Ġalso Ġbe Ġachieved Ġby Ġintegrating Ġthe Ġwhole Ġcontext Ġinto Ġthe Ġmodel , Ġusing Ġthe Ġwhole Ġcontext Ġcan Ġhave Ġdisadvantages Ġfor Ġlonger Ġsentences : ĠThe Ġmax Ġpool ing Ġstep Ġcan Ġeasily Ġchoose Ġa Ġvalue Ġfrom Ġa Ġpart Ġof Ġthe Ġsentence Ġwhich Ġis Ġfar Ġaway Ġfrom Ġthe Ġmention Ġof Ġthe Ġrelation . ĠWith Ġsplitting Ġthe Ġcontext Ġinto Ġtwo Ġparts , Ġwe Ġreduce Ġthis Ġdanger . ĠRepe ating Ġthe Ġmiddle Ġcontext Ġincreases Ġthe Ġchance Ġfor Ġthe Ġmax Ġpool ing Ġstep Ġto Ġpick Ġa Ġvalue Ġfrom Ġthe Ġmiddle Ġcontext . </s> Con v olution al ĠLayer </s> Following Ġprevious Ġwork Ġ( e . g ., ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ), Ġwe Ġuse Ġ2 D Ġfilters Ġspanning Ġall Ġembed ding Ġdimensions . ĠAfter Ġconv olution , Ġa Ġmax Ġpool ing Ġoperation Ġis Ġapplied Ġthat Ġstores Ġonly Ġthe Ġhighest Ġactivation Ġof Ġeach Ġfilter . ĠWe Ġapply Ġfilters Ġwith Ġdifferent Ġwindow Ġsizes Ġ2 - 5 Ġ( multi - windows ) Ġas Ġin ĠB IB REF 5 Ġ, Ġi . e . Ġspanning Ġa Ġdifferent Ġnumber Ġof Ġinput Ġwords . </s> Rec urrent ĠNeural ĠNetworks Ġ( R NN ) </s> Traditional ĠR NN s Ġconsist Ġof Ġan Ġinput Ġvector , Ġa Ġhistory Ġvector Ġand Ġan Ġoutput Ġvector . ĠBased Ġon Ġthe Ġrepresentation Ġof Ġthe Ġcurrent Ġinput Ġword Ġand Ġthe Ġprevious Ġhistory Ġvector , Ġa Ġnew Ġhistory Ġis Ġcomputed . ĠThen , Ġan Ġoutput Ġis Ġpredicted Ġ( e . g ., Ġusing Ġa Ġsoft max Ġlayer ). ĠIn Ġcontrast Ġto Ġmost Ġtraditional ĠR NN Ġarchitectures , Ġwe Ġuse Ġthe ĠR NN Ġfor Ġsentence Ġmodeling , Ġi . e ., Ġwe Ġpredict Ġan Ġoutput Ġvector Ġonly Ġafter Ġprocessing Ġthe Ġwhole Ġsentence Ġand Ġnot Ġafter Ġeach Ġword . ĠTraining Ġis Ġperformed Ġusing Ġback prop ag ation Ġthrough Ġtime ĠB IB REF 9 Ġwhich Ġunfolds Ġthe Ġrecurrent Ġcomput ations Ġof Ġthe Ġhistory Ġvector Ġfor Ġa Ġcertain Ġnumber Ġof Ġtime Ġsteps . ĠTo Ġavoid Ġexploding Ġgrad ients , Ġwe Ġuse Ġgradient Ġclipping Ġwith Ġa Ġthreshold Ġof Ġ10 ĠB IB REF 10 Ġ. </s> Input Ġof Ġthe ĠR NN s </s> Initial Ġexperiments Ġshowed Ġthat Ġusing Ġtrig rams Ġas Ġinput Ġinstead Ġof Ġsingle Ġwords Ġled Ġto Ġsuperior Ġresults . ĠHence , Ġat Ġtim est ep ĠIN LINE FORM 0 Ġwe Ġdo Ġnot Ġonly Ġgive Ġword ĠIN LINE FORM 1 Ġto Ġthe Ġmodel Ġbut Ġthe Ġtrig ram ĠIN LINE FORM 2 Ġby Ġconc aten ating Ġthe Ġcorresponding Ġword Ġembed d ings . </s> Connection ist ĠBi - direction al ĠR NN s </s> Especially Ġfor Ġrelation Ġclassification , Ġthe Ġprocessing Ġof Ġthe Ġrelation Ġarguments Ġmight Ġbe Ġeasier Ġwith Ġknowledge Ġof Ġthe Ġsucceeding Ġwords . ĠTherefore Ġin Ġbi - direction al ĠR NN s , Ġnot Ġonly Ġa Ġhistory Ġvector Ġof Ġword ĠIN LINE FORM 0 Ġis Ġregarded Ġbut Ġalso Ġa Ġfuture Ġvector . ĠThis Ġleads Ġto Ġthe Ġfollowing Ġconditioned Ġprobability Ġfor Ġthe Ġhistory ĠIN LINE FORM 1 Ġat Ġtime Ġstep ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> Thus , Ġthe Ġnetwork Ġcan Ġbe Ġsplit Ġinto Ġthree Ġparts : Ġa Ġforward Ġpass Ġwhich Ġprocesses Ġthe Ġoriginal Ġsentence Ġword Ġby Ġword Ġ( Equ ation ĠEQ REF 6 Ġ); Ġa Ġbackward Ġpass Ġwhich Ġprocesses Ġthe Ġreversed Ġsentence Ġword Ġby Ġword Ġ( Equ ation Ġ); Ġand Ġa Ġcombination Ġof Ġboth Ġ( Equ ation Ġ). ĠAll Ġthree Ġparts Ġare Ġtrained Ġjointly . ĠThis Ġis Ġalso Ġdepicted Ġin ĠFigure ĠFIG REF 7 Ġ. </s> Comb ining Ġforward Ġand Ġbackward Ġpass Ġby Ġadding Ġtheir Ġhidden Ġlayer Ġis Ġsimilar Ġto ĠB IB REF 7 Ġ. ĠWe , Ġhowever , Ġalso Ġadd Ġa Ġconnection Ġto Ġthe Ġprevious Ġcombined Ġhidden Ġlayer Ġwith Ġweight ĠIN LINE FORM 0 Ġto Ġbe Ġable Ġto Ġinclude Ġall Ġintermediate Ġhidden Ġlayers Ġinto Ġthe Ġfinal Ġdecision Ġof Ġthe Ġnetwork Ġ( see ĠEqu ation Ġ). ĠWe Ġcall Ġthis ĠâĢ ľ connection ist Ġbi - direction al ĠR NN âĢ Ŀ . </s> In Ġour Ġexperiments , Ġwe Ġcompare Ġthis ĠR NN Ġwith Ġun i - direction al ĠR NN s Ġand Ġbi - direction al ĠR NN s Ġwithout Ġadditional Ġhidden Ġlayer Ġconnections . </s> Word ĠRepresent ations </s> Words Ġare Ġrepresented Ġby Ġconc aten ated Ġvectors : Ġa Ġword Ġembed ding Ġand Ġa Ġposition Ġfeature Ġvector . </s> P ret rained Ġword Ġembed d ings . ĠIn Ġthis Ġstudy , Ġwe Ġused Ġthe Ġword 2 vec Ġtool kit ĠB IB REF 11 Ġto Ġtrain Ġembed d ings Ġon Ġan ĠEnglish ĠWikipedia Ġfrom ĠMay Ġ2014 . ĠWe Ġonly Ġconsidered Ġwords Ġappearing Ġmore Ġthan Ġ100 Ġtimes Ġand Ġadded Ġa Ġspecial ĠP ADD ING Ġtoken Ġfor Ġconv olution . ĠThis Ġresults Ġin Ġan Ġembed ding Ġtraining Ġtext Ġof Ġabout Ġ48 5 , 000 Ġterms Ġand ĠIN LINE FORM 0 Ġtokens . ĠDuring Ġmodel Ġtraining , Ġthe Ġembed d ings Ġare Ġupdated . </s> Position Ġfeatures . ĠWe Ġincorporate Ġrandomly Ġinitialized Ġposition Ġembed d ings Ġsimilar Ġto Ġz eng 2014 , Ġn guyen Ġand Ġde S ant os 2015 . ĠIn Ġour ĠR NN Ġexperiments , Ġwe Ġinvestigate Ġdifferent Ġpossibilities Ġof Ġintegrating Ġposition Ġinformation : Ġposition Ġembed d ings , Ġposition Ġembed d ings Ġwith Ġentity Ġpresence Ġflags Ġ( flags Ġindicating Ġwhether Ġthe Ġcurrent Ġword Ġis Ġone Ġof Ġthe Ġrelation Ġarguments ), Ġand Ġposition Ġindicators ĠB IB REF 7 Ġ. </s> Object ive ĠFunction : ĠRanking ĠLoss </s> R anking . ĠWe Ġapplied Ġthe Ġranking Ġloss Ġfunction Ġproposed Ġin Ġde S ant os 2015 Ġto Ġtrain Ġour Ġmodels . ĠIt Ġmaxim izes Ġthe Ġdistance Ġbetween Ġthe Ġtrue Ġlabel ĠIN LINE FORM 0 Ġand Ġthe Ġbest Ġcompetitive Ġlabel ĠIN LINE FORM 1 Ġgiven Ġa Ġdata Ġpoint ĠIN LINE FORM 2 Ġ. ĠThe Ġobjective Ġfunction Ġis ĠDIS PLAY FORM 0 </s> with ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġbeing Ġthe Ġscores Ġfor Ġthe Ġclasses ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġrespectively . ĠThe Ġparameter ĠIN LINE FORM 4 Ġcontrols Ġthe Ġpenal ization Ġof Ġthe Ġprediction Ġerrors Ġand ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġare Ġmargins Ġfor Ġthe Ġcorrect Ġand Ġincorrect Ġclasses . ĠFollowing Ġde S ant os 2015 , Ġwe Ġset ĠIN LINE FORM 7 Ġ. ĠWe Ġdo Ġnot Ġlearn Ġa Ġpattern Ġfor Ġthe Ġclass ĠOther Ġbut Ġincrease Ġits Ġdifference Ġto Ġthe Ġbest Ġcompetitive Ġlabel Ġby Ġusing Ġonly Ġthe Ġsecond Ġsum mand Ġin ĠEqu ation ĠEQ REF 10 Ġduring Ġtraining . </s> Exper iments Ġand ĠResults </s> We Ġused Ġthe Ġrelation Ġclassification Ġdataset Ġof Ġthe ĠSem E val Ġ2010 Ġtask Ġ8 ĠB IB REF 8 Ġ. ĠIt Ġconsists Ġof Ġsentences Ġwhich Ġhave Ġbeen Ġmanually Ġlabeled Ġwith Ġ19 Ġrelations Ġ( 9 Ġdirected Ġrelations Ġand Ġone Ġartificial Ġclass ĠOther ). Ġ8 , 000 Ġsentences Ġhave Ġbeen Ġdistributed Ġas Ġtraining Ġset Ġand Ġ2 , 7 17 Ġsentences Ġserved Ġas Ġtest Ġset . ĠFor Ġevaluation , Ġwe Ġapplied Ġthe Ġofficial Ġscoring Ġscript Ġand Ġreport Ġthe Ġmacro ĠF 1 Ġscore Ġwhich Ġalso Ġserved Ġas Ġthe Ġofficial Ġresult Ġof Ġthe Ġshared Ġtask . </s> R NN Ġand ĠCNN Ġmodels Ġwere Ġimplemented Ġwith Ġthe ano ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠFor Ġall Ġour Ġmodels , Ġwe Ġuse ĠL 2 Ġregular ization Ġwith Ġa Ġweight Ġof Ġ0 . 0001 . ĠFor ĠCNN Ġtraining , Ġwe Ġuse Ġmini Ġbatches Ġof Ġ25 Ġtraining Ġexamples Ġwhile Ġwe Ġperform Ġsto ch astic Ġgradient Ġdescent Ġfor Ġthe ĠR NN . ĠThe Ġinitial Ġlearning Ġrates Ġare Ġ0 . 2 Ġfor Ġthe ĠCNN Ġand Ġ0 . 01 Ġfor Ġthe ĠR NN . ĠWe Ġtrain Ġthe Ġmodels Ġfor Ġ10 Ġ( CNN ) Ġand Ġ50 Ġ( R NN ) Ġepoch s Ġwithout Ġearly Ġstopping . ĠAs Ġactivation Ġfunction , Ġwe Ġapply Ġtan h Ġfor Ġthe ĠCNN Ġand Ġcapped ĠRe LU Ġfor Ġthe ĠR NN . ĠFor Ġtuning Ġthe Ġhyper param eters , Ġwe Ġsplit Ġthe Ġtraining Ġdata Ġinto Ġtwo Ġparts : Ġ6 . 5 k Ġ( training ) Ġand Ġ1 . 5 k Ġ( development ) Ġsentences . ĠWe Ġalso Ġtuned Ġthe Ġlearning Ġrate Ġschedule Ġon Ġdev . </s> B es ide Ġof Ġtraining Ġsingle Ġmodels , Ġwe Ġalso Ġreport Ġensemble Ġresults Ġfor Ġwhich Ġwe Ġcombined Ġthe Ġpresented Ġsingle Ġmodels Ġwith Ġa Ġvoting Ġprocess . </s> Performance Ġof ĠCNN s </s> As Ġa Ġbaseline Ġsystem , Ġwe Ġimplemented Ġa ĠCNN Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġby Ġz eng 2014 . ĠIt Ġconsists Ġof Ġa Ġstandard Ġconv olution al Ġlayer Ġwith Ġfilters Ġwith Ġonly Ġone Ġwindow Ġsize , Ġfollowed Ġby Ġa Ġsoft max Ġlayer . ĠAs Ġinput Ġit Ġuses Ġthe Ġmiddle Ġcontext . ĠIn Ġcontrast Ġto Ġz eng 2014 , Ġour ĠCNN Ġdoes Ġnot Ġhave Ġan Ġadditional Ġfully Ġconnected Ġhidden Ġlayer . ĠTherefore , Ġwe Ġincreased Ġthe Ġnumber Ġof Ġconv olution al Ġfilters Ġto Ġ1200 Ġto Ġkeep Ġthe Ġnumber Ġof Ġparameters Ġcomparable . ĠWith Ġthis , Ġwe Ġobtain Ġa Ġbaseline Ġresult Ġof Ġ73 . 0 . ĠAfter Ġincluding Ġ5 Ġdimensional Ġposition Ġfeatures , Ġthe Ġperformance Ġwas Ġimproved Ġto Ġ78 . 6 Ġ( com parable Ġto Ġ78 . 9 Ġas Ġreported Ġby Ġz eng 2014 Ġwithout Ġlinguistic Ġfeatures ). </s> In Ġthe Ġnext Ġstep , Ġwe Ġinvestigate Ġhow Ġthis Ġresult Ġchanges Ġif Ġwe Ġsuccess ively Ġadd Ġfurther Ġfeatures Ġto Ġour ĠCNN : Ġmulti - windows Ġfor Ġconv olution Ġ( window Ġsizes : Ġ2 , 3 , 4 , 5 Ġand Ġ300 Ġfeature Ġmaps Ġeach ), Ġranking Ġlayer Ġinstead Ġof Ġsoft max Ġand Ġour Ġproposed Ġextended Ġmiddle Ġcontext . ĠTable ĠT AB REF 12 Ġshows Ġthe Ġresults . ĠNote Ġthat Ġall Ġnumbers Ġare Ġproduced Ġby ĠCNN s Ġwith Ġa Ġcomparable Ġnumber Ġof Ġparameters . ĠWe Ġalso Ġreport ĠF 1 Ġfor Ġincreasing Ġthe Ġword Ġembed ding Ġdimension ality Ġfrom Ġ50 Ġto Ġ400 . ĠThe Ġposition Ġembed ding Ġdimension ality Ġis Ġ5 Ġin Ġcombination Ġwith Ġ50 Ġdimensional Ġword Ġembed d ings Ġand Ġ35 Ġwith Ġ400 Ġdimensional Ġword Ġembed d ings . ĠOur Ġresults Ġshow Ġthat Ġespecially Ġthe Ġranking Ġlayer Ġand Ġthe Ġembed ding Ġsize Ġhave Ġan Ġimportant Ġimpact Ġon Ġthe Ġperformance . </s> Performance Ġof ĠR NN s </s> As Ġa Ġbaseline Ġfor Ġthe ĠR NN Ġmodels , Ġwe Ġapply Ġa Ġun i - direction al ĠR NN Ġwhich Ġpredicts Ġthe Ġrelation Ġafter Ġprocessing Ġthe Ġwhole Ġsentence . ĠWith Ġthis Ġmodel , Ġwe Ġachieve Ġan ĠF 1 Ġscore Ġof Ġ61 . 2 Ġon Ġthe ĠSem E val Ġtest Ġset . </s> After wards , Ġwe Ġinvestigate Ġthe Ġimpact Ġof Ġdifferent Ġposition Ġfeatures Ġon Ġthe Ġperformance Ġof Ġun i - direction al ĠR NN s Ġ( position Ġembed d ings , Ġposition Ġembed d ings Ġconc aten ated Ġwith Ġa Ġflag Ġindicating Ġwhether Ġthe Ġcurrent Ġword Ġis Ġan Ġentity Ġor Ġnot , Ġand Ġposition Ġindicators ĠB IB REF 7 Ġ). ĠThe Ġresults Ġindicate Ġthat Ġposition Ġindicators Ġ( i . e . Ġartificial Ġwords Ġthat Ġindicate Ġthe Ġentity Ġpresence ) Ġperform Ġthe Ġbest Ġon Ġthe ĠSem E val Ġdata . ĠWe Ġachieve Ġan ĠF 1 Ġscore Ġof Ġ73 . 4 Ġwith Ġthem . ĠHowever , Ġthe Ġdifference Ġto Ġusing Ġposition Ġembed d ings Ġwith Ġentity Ġflags Ġis Ġnot Ġstatistically Ġsignificant . </s> Similar Ġto Ġour ĠCNN Ġexperiments , Ġwe Ġsuccess ively Ġvary Ġthe ĠR NN Ġmodels Ġby Ġusing Ġbi - direction ality , Ġby Ġadding Ġconnections Ġbetween Ġthe Ġhidden Ġlayers Ġ( âĢ ľ connection ist âĢ Ŀ ), Ġby Ġapplying Ġranking Ġinstead Ġof Ġsoft max Ġto Ġpredict Ġthe Ġrelation Ġand Ġby Ġincreasing Ġthe Ġword Ġembed ding Ġdimension Ġto Ġ400 . </s> The Ġresults Ġin ĠTable ĠT AB REF 14 Ġshow Ġthat Ġall Ġof Ġthese Ġvariations Ġlead Ġto Ġstatistically Ġsignificant Ġimprovements . ĠEspecially Ġthe Ġadditional Ġhidden Ġlayer Ġconnections Ġand Ġthe Ġintegration Ġof Ġthe Ġranking Ġlayer Ġhave Ġa Ġlarge Ġimpact Ġon Ġthe Ġperformance . </s> Comb ination Ġof ĠCNN s Ġand ĠR NN s </s> Finally , Ġwe Ġcombine Ġour ĠCNN Ġand ĠR NN Ġmodels Ġusing Ġa Ġvoting Ġprocess . ĠFor Ġeach Ġsentence Ġin Ġthe Ġtest Ġset , Ġwe Ġapply Ġseveral ĠCNN Ġand ĠR NN Ġmodels Ġpresented Ġin ĠTables ĠT AB REF 12 Ġand ĠT AB REF 14 Ġand Ġpredict Ġthe Ġclass Ġwith Ġthe Ġmost Ġvotes . ĠIn Ġcase Ġof Ġa Ġtie , Ġwe Ġpick Ġone Ġof Ġthe Ġmost Ġfrequent Ġclasses Ġrandomly . ĠThe Ġcombination Ġachieves Ġan ĠF 1 Ġscore Ġof Ġ84 . 9 Ġwhich Ġis Ġbetter Ġthan Ġthe Ġperformance Ġof Ġthe Ġtwo ĠN N Ġtypes Ġalone . ĠIt , Ġthus , Ġconfirms Ġour Ġassumption Ġthat Ġthe Ġnetworks Ġprovide Ġcomplementary Ġinformation : Ġwhile Ġthe ĠR NN Ġcomp utes Ġa Ġweighted Ġcombination Ġof Ġall Ġwords Ġin Ġthe Ġsentence , Ġthe ĠCNN Ġextracts Ġthe Ġmost Ġinformative Ġn - gram s Ġfor Ġthe Ġrelation Ġand Ġonly Ġconsiders Ġtheir Ġresulting Ġactiv ations . </s> Compar ison Ġwith ĠState Ġof Ġthe ĠArt </s> Table ĠT AB REF 16 Ġshows Ġthe Ġresults Ġof Ġour Ġmodels ĠER - CNN Ġ( ext ended Ġranking ĠCNN ) Ġand ĠR - R NN Ġ( ranking ĠR NN ) Ġin Ġthe Ġcontext Ġof Ġother Ġstate - of - the - art Ġmodels . ĠOur Ġproposed Ġmodels Ġobtain Ġstate - of - the - art Ġresults Ġon Ġthe ĠSem E val Ġ2010 Ġtask Ġ8 Ġdata Ġset Ġwithout Ġmaking Ġuse Ġof Ġany Ġlinguistic Ġfeatures . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġinvestigated Ġdifferent Ġfeatures Ġand Ġarchitectural Ġchoices Ġfor Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks Ġfor Ġrelation Ġclassification Ġwithout Ġusing Ġany Ġlinguistic Ġfeatures . ĠFor Ġconv olution al Ġneural Ġnetworks , Ġwe Ġpresented Ġa Ġnew Ġcontext Ġrepresentation Ġfor Ġrelation Ġclassification . ĠFurthermore , Ġwe Ġintroduced Ġconnection ist Ġrecurrent Ġneural Ġnetworks Ġfor Ġsentence Ġclassification Ġtasks Ġand Ġperformed Ġthe Ġfirst Ġexperiments Ġwith Ġranking Ġrecurrent Ġneural Ġnetworks . ĠFinally , Ġwe Ġshowed Ġthat Ġeven Ġa Ġsimple Ġcombination Ġof Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks Ġimproved Ġresults . ĠWith Ġour Ġneural Ġmodels , Ġwe Ġachieved Ġnew Ġstate - of - the - art Ġresults Ġon Ġthe ĠSem E val Ġ2010 Ġtask Ġ8 Ġbenchmark Ġdata . </s> Acknowled gments </s> He ike ĠAd el Ġis Ġa Ġrecipient Ġof Ġthe ĠGoogle ĠEuropean ĠDoctor al ĠFellowship Ġin ĠNatural ĠLanguage ĠProcessing Ġand Ġthis Ġresearch Ġis Ġsupported Ġby Ġthis Ġfellowship . </s> This Ġresearch Ġwas Ġalso Ġsupported Ġby ĠDeutsche ĠFors ch ung s gem e ins cha ft : Ġgrant ĠSCH U Ġ224 6 / 4 - 2 .
Introduction Relation classification is the task of assigning sentences with two marked entities to a predefined set of relations. The sentence “We poured the <e1>milk</e1> into the <e2>pumpkin mixture</e2>.”, for example, expresses the relation Entity-Destination(e1,e2). While early research mostly focused on support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1 , recent research showed performance improvements by applying neural networks (NNs) BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 on the benchmark data from SemEval 2010 shared task 8 BIBREF8 . This study investigates two different types of NNs: recurrent neural networks (RNNs) and convolutional neural networks (CNNs) as well as their combination. We make the following contributions: (1) We propose extended middle context, a new context representation for CNNs for relation classification. The extended middle context uses all parts of the sentence (the relation arguments, left of the relation arguments, between the arguments, right of the arguments) and pays special attention to the middle part. (2) We present connectionist bi-directional RNN models which are especially suited for sentence classification tasks since they combine all intermediate hidden layers for their final decision. Furthermore, the ranking loss function is introduced for the RNN model optimization which has not been investigated in the literature for relation classification before. (3) Finally, we combine CNNs and RNNs using a simple voting scheme and achieve new state-of-the-art results on the SemEval 2010 benchmark dataset. Related Work In 2010, manually annotated data for relation classification was released in the context of a SemEval shared task BIBREF8 . Shared task participants used, i.a., support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1 . Recently, their results on this data set were outperformed by applying NNs BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . zeng2014 built a CNN based only on the context between the relation arguments and extended it with several lexical features. kim2014 and others used convolutional filters of different sizes for CNNs. nguyen applied this to relation classification and obtained improvements over single filter sizes. deSantos2015 replaced the softmax layer of the CNN with a ranking layer. They showed improvements and published the best result so far on the SemEval dataset, to our knowledge. socher used another NN architecture for relation classification: recursive neural networks that built recursive sentence representations based on syntactic parsing. In contrast, zhang investigated a temporal structured RNN with only words as input. They used a bi-directional model with a pooling layer on top. Convolutional Neural Networks (CNN) CNNs perform a discrete convolution on an input matrix with a set of different filters. For NLP tasks, the input matrix represents a sentence: Each column of the matrix stores the word embedding of the corresponding word. By applying a filter with a width of, e.g., three columns, three neighboring words (trigram) are convolved. Afterwards, the results of the convolution are pooled. Following collobertWeston, we perform max-pooling which extracts the maximum value for each filter and, thus, the most informative n-gram for the following steps. Finally, the resulting values are concatenated and used for classifying the relation expressed in the sentence. Input: Extended Middle Context One of our contributions is a new input representation especially designed for relation classification. The contexts are split into three disjoint regions based on the two relation arguments: the left context, the middle context and the right context. Since in most cases the middle context contains the most relevant information for the relation, we want to focus on it but not ignore the other regions completely. Hence, we propose to use two contexts: (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context. Due to the repetition of the middle context, we force the network to pay special attention to it. The two contexts are processed by two independent convolutional and max-pooling layers. After pooling, the results are concatenated to form the sentence representation. Figure FIGREF3 depicts this procedure. It shows an examplary sentence: “He had chest pain and <e1>headaches</e1> from <e2>mold</e2> in the bedroom.” If we only considered the middle context “from”, the network might be tempted to predict a relation like Entity-Origin(e1,e2). However, by also taking the left and right context into account, the model can detect the relation Cause-Effect(e2,e1). While this could also be achieved by integrating the whole context into the model, using the whole context can have disadvantages for longer sentences: The max pooling step can easily choose a value from a part of the sentence which is far away from the mention of the relation. With splitting the context into two parts, we reduce this danger. Repeating the middle context increases the chance for the max pooling step to pick a value from the middle context. Convolutional Layer Following previous work (e.g., BIBREF5 , BIBREF6 ), we use 2D filters spanning all embedding dimensions. After convolution, a max pooling operation is applied that stores only the highest activation of each filter. We apply filters with different window sizes 2-5 (multi-windows) as in BIBREF5 , i.e. spanning a different number of input words. Recurrent Neural Networks (RNN) Traditional RNNs consist of an input vector, a history vector and an output vector. Based on the representation of the current input word and the previous history vector, a new history is computed. Then, an output is predicted (e.g., using a softmax layer). In contrast to most traditional RNN architectures, we use the RNN for sentence modeling, i.e., we predict an output vector only after processing the whole sentence and not after each word. Training is performed using backpropagation through time BIBREF9 which unfolds the recurrent computations of the history vector for a certain number of time steps. To avoid exploding gradients, we use gradient clipping with a threshold of 10 BIBREF10 . Input of the RNNs Initial experiments showed that using trigrams as input instead of single words led to superior results. Hence, at timestep INLINEFORM0 we do not only give word INLINEFORM1 to the model but the trigram INLINEFORM2 by concatenating the corresponding word embeddings. Connectionist Bi-directional RNNs Especially for relation classification, the processing of the relation arguments might be easier with knowledge of the succeeding words. Therefore in bi-directional RNNs, not only a history vector of word INLINEFORM0 is regarded but also a future vector. This leads to the following conditioned probability for the history INLINEFORM1 at time step INLINEFORM2 : DISPLAYFORM0 Thus, the network can be split into three parts: a forward pass which processes the original sentence word by word (Equation EQREF6 ); a backward pass which processes the reversed sentence word by word (Equation ); and a combination of both (Equation ). All three parts are trained jointly. This is also depicted in Figure FIGREF7 . Combining forward and backward pass by adding their hidden layer is similar to BIBREF7 . We, however, also add a connection to the previous combined hidden layer with weight INLINEFORM0 to be able to include all intermediate hidden layers into the final decision of the network (see Equation ). We call this “connectionist bi-directional RNN”. In our experiments, we compare this RNN with uni-directional RNNs and bi-directional RNNs without additional hidden layer connections. Word Representations Words are represented by concatenated vectors: a word embedding and a position feature vector. Pretrained word embeddings. In this study, we used the word2vec toolkit BIBREF11 to train embeddings on an English Wikipedia from May 2014. We only considered words appearing more than 100 times and added a special PADDING token for convolution. This results in an embedding training text of about 485,000 terms and INLINEFORM0 tokens. During model training, the embeddings are updated. Position features. We incorporate randomly initialized position embeddings similar to zeng2014, nguyen and deSantos2015. In our RNN experiments, we investigate different possibilities of integrating position information: position embeddings, position embeddings with entity presence flags (flags indicating whether the current word is one of the relation arguments), and position indicators BIBREF7 . Objective Function: Ranking Loss Ranking. We applied the ranking loss function proposed in deSantos2015 to train our models. It maximizes the distance between the true label INLINEFORM0 and the best competitive label INLINEFORM1 given a data point INLINEFORM2 . The objective function is DISPLAYFORM0 with INLINEFORM0 and INLINEFORM1 being the scores for the classes INLINEFORM2 and INLINEFORM3 respectively. The parameter INLINEFORM4 controls the penalization of the prediction errors and INLINEFORM5 and INLINEFORM6 are margins for the correct and incorrect classes. Following deSantos2015, we set INLINEFORM7 . We do not learn a pattern for the class Other but increase its difference to the best competitive label by using only the second summand in Equation EQREF10 during training. Experiments and Results We used the relation classification dataset of the SemEval 2010 task 8 BIBREF8 . It consists of sentences which have been manually labeled with 19 relations (9 directed relations and one artificial class Other). 8,000 sentences have been distributed as training set and 2,717 sentences served as test set. For evaluation, we applied the official scoring script and report the macro F1 score which also served as the official result of the shared task. RNN and CNN models were implemented with theano BIBREF12 , BIBREF13 . For all our models, we use L2 regularization with a weight of 0.0001. For CNN training, we use mini batches of 25 training examples while we perform stochastic gradient descent for the RNN. The initial learning rates are 0.2 for the CNN and 0.01 for the RNN. We train the models for 10 (CNN) and 50 (RNN) epochs without early stopping. As activation function, we apply tanh for the CNN and capped ReLU for the RNN. For tuning the hyperparameters, we split the training data into two parts: 6.5k (training) and 1.5k (development) sentences. We also tuned the learning rate schedule on dev. Beside of training single models, we also report ensemble results for which we combined the presented single models with a voting process. Performance of CNNs As a baseline system, we implemented a CNN similar to the one described by zeng2014. It consists of a standard convolutional layer with filters with only one window size, followed by a softmax layer. As input it uses the middle context. In contrast to zeng2014, our CNN does not have an additional fully connected hidden layer. Therefore, we increased the number of convolutional filters to 1200 to keep the number of parameters comparable. With this, we obtain a baseline result of 73.0. After including 5 dimensional position features, the performance was improved to 78.6 (comparable to 78.9 as reported by zeng2014 without linguistic features). In the next step, we investigate how this result changes if we successively add further features to our CNN: multi-windows for convolution (window sizes: 2,3,4,5 and 300 feature maps each), ranking layer instead of softmax and our proposed extended middle context. Table TABREF12 shows the results. Note that all numbers are produced by CNNs with a comparable number of parameters. We also report F1 for increasing the word embedding dimensionality from 50 to 400. The position embedding dimensionality is 5 in combination with 50 dimensional word embeddings and 35 with 400 dimensional word embeddings. Our results show that especially the ranking layer and the embedding size have an important impact on the performance. Performance of RNNs As a baseline for the RNN models, we apply a uni-directional RNN which predicts the relation after processing the whole sentence. With this model, we achieve an F1 score of 61.2 on the SemEval test set. Afterwards, we investigate the impact of different position features on the performance of uni-directional RNNs (position embeddings, position embeddings concatenated with a flag indicating whether the current word is an entity or not, and position indicators BIBREF7 ). The results indicate that position indicators (i.e. artificial words that indicate the entity presence) perform the best on the SemEval data. We achieve an F1 score of 73.4 with them. However, the difference to using position embeddings with entity flags is not statistically significant. Similar to our CNN experiments, we successively vary the RNN models by using bi-directionality, by adding connections between the hidden layers (“connectionist”), by applying ranking instead of softmax to predict the relation and by increasing the word embedding dimension to 400. The results in Table TABREF14 show that all of these variations lead to statistically significant improvements. Especially the additional hidden layer connections and the integration of the ranking layer have a large impact on the performance. Combination of CNNs and RNNs Finally, we combine our CNN and RNN models using a voting process. For each sentence in the test set, we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes. In case of a tie, we pick one of the most frequent classes randomly. The combination achieves an F1 score of 84.9 which is better than the performance of the two NN types alone. It, thus, confirms our assumption that the networks provide complementary information: while the RNN computes a weighted combination of all words in the sentence, the CNN extracts the most informative n-grams for the relation and only considers their resulting activations. Comparison with State of the Art Table TABREF16 shows the results of our models ER-CNN (extended ranking CNN) and R-RNN (ranking RNN) in the context of other state-of-the-art models. Our proposed models obtain state-of-the-art results on the SemEval 2010 task 8 data set without making use of any linguistic features. Conclusion In this paper, we investigated different features and architectural choices for convolutional and recurrent neural networks for relation classification without using any linguistic features. For convolutional neural networks, we presented a new context representation for relation classification. Furthermore, we introduced connectionist recurrent neural networks for sentence classification tasks and performed the first experiments with ranking recurrent neural networks. Finally, we showed that even a simple combination of convolutional and recurrent neural networks improved results. With our neural models, we achieved new state-of-the-art results on the SemEval 2010 task 8 benchmark data. Acknowledgments Heike Adel is a recipient of the Google European Doctoral Fellowship in Natural Language Processing and this research is supported by this fellowship. This research was also supported by Deutsche Forschungsgemeinschaft: grant SCHU 2246/4-2.
How does their simple voting scheme work?
0f7867f888109b9e000ef68965df4dde2511a55f
How Ġdoes Ġtheir Ġsimple Ġvoting Ġscheme Ġwork ?
we Ġapply Ġseveral ĠCNN Ġand ĠR NN Ġmodels Ġpresented Ġin ĠTables ĠT AB REF 12 Ġand ĠT AB REF 14 Ġand Ġpredict Ġthe Ġclass Ġwith Ġthe Ġmost Ġvotes , ĠIn Ġcase Ġof Ġa Ġtie , Ġwe Ġpick Ġone Ġof Ġthe Ġmost Ġfrequent Ġclasses Ġrandomly
<s> How Ġdoes Ġtheir Ġsimple Ġvoting Ġscheme Ġwork ? </s> Introduction </s> Rel ation Ġclassification Ġis Ġthe Ġtask Ġof Ġassigning Ġsentences Ġwith Ġtwo Ġmarked Ġentities Ġto Ġa Ġpred efined Ġset Ġof Ġrelations . ĠThe Ġsentence ĠâĢ ľ We Ġpoured Ġthe Ġ< e 1 > mil k </ e 1 > Ġinto Ġthe Ġ< e 2 > p ump kin Ġmixture </ e 2 >. âĢ Ŀ , Ġfor Ġexample , Ġexpresses Ġthe Ġrelation ĠEntity - Dest ination ( e 1 , e 2 ). ĠWhile Ġearly Ġresearch Ġmostly Ġfocused Ġon Ġsupport Ġvector Ġmachines Ġor Ġmaximum Ġentropy Ġclass ifiers ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġrecent Ġresearch Ġshowed Ġperformance Ġimprovements Ġby Ġapplying Ġneural Ġnetworks Ġ( NN s ) ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġon Ġthe Ġbenchmark Ġdata Ġfrom ĠSem E val Ġ2010 Ġshared Ġtask Ġ8 ĠB IB REF 8 Ġ. </s> This Ġstudy Ġinvestigates Ġtwo Ġdifferent Ġtypes Ġof ĠN Ns : Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ) Ġand Ġconv olution al Ġneural Ġnetworks Ġ( CNN s ) Ġas Ġwell Ġas Ġtheir Ġcombination . ĠWe Ġmake Ġthe Ġfollowing Ġcontributions : </s> ( 1 ) ĠWe Ġpropose Ġextended Ġmiddle Ġcontext , Ġa Ġnew Ġcontext Ġrepresentation Ġfor ĠCNN s Ġfor Ġrelation Ġclassification . ĠThe Ġextended Ġmiddle Ġcontext Ġuses Ġall Ġparts Ġof Ġthe Ġsentence Ġ( the Ġrelation Ġarguments , Ġleft Ġof Ġthe Ġrelation Ġarguments , Ġbetween Ġthe Ġarguments , Ġright Ġof Ġthe Ġarguments ) Ġand Ġpays Ġspecial Ġattention Ġto Ġthe Ġmiddle Ġpart . </s> ( 2 ) ĠWe Ġpresent Ġconnection ist Ġbi - direction al ĠR NN Ġmodels Ġwhich Ġare Ġespecially Ġsuited Ġfor Ġsentence Ġclassification Ġtasks Ġsince Ġthey Ġcombine Ġall Ġintermediate Ġhidden Ġlayers Ġfor Ġtheir Ġfinal Ġdecision . ĠFurthermore , Ġthe Ġranking Ġloss Ġfunction Ġis Ġintroduced Ġfor Ġthe ĠR NN Ġmodel Ġoptimization Ġwhich Ġhas Ġnot Ġbeen Ġinvestigated Ġin Ġthe Ġliterature Ġfor Ġrelation Ġclassification Ġbefore . </s> ( 3 ) ĠFinally , Ġwe Ġcombine ĠCNN s Ġand ĠR NN s Ġusing Ġa Ġsimple Ġvoting Ġscheme Ġand Ġachieve Ġnew Ġstate - of - the - art Ġresults Ġon Ġthe ĠSem E val Ġ2010 Ġbenchmark Ġdataset . </s> Related ĠWork </s> In Ġ2010 , Ġmanually Ġannot ated Ġdata Ġfor Ġrelation Ġclassification Ġwas Ġreleased Ġin Ġthe Ġcontext Ġof Ġa ĠSem E val Ġshared Ġtask ĠB IB REF 8 Ġ. ĠShared Ġtask Ġparticipants Ġused , Ġi . a ., Ġsupport Ġvector Ġmachines Ġor Ġmaximum Ġentropy Ġclass ifiers ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠRecently , Ġtheir Ġresults Ġon Ġthis Ġdata Ġset Ġwere Ġoutper formed Ġby Ġapplying ĠN Ns ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. </s> z eng 2014 Ġbuilt Ġa ĠCNN Ġbased Ġonly Ġon Ġthe Ġcontext Ġbetween Ġthe Ġrelation Ġarguments Ġand Ġextended Ġit Ġwith Ġseveral Ġlex ical Ġfeatures . Ġk im 2014 Ġand Ġothers Ġused Ġconv olution al Ġfilters Ġof Ġdifferent Ġsizes Ġfor ĠCNN s . Ġn guyen Ġapplied Ġthis Ġto Ġrelation Ġclassification Ġand Ġobtained Ġimprovements Ġover Ġsingle Ġfilter Ġsizes . Ġde S ant os 2015 Ġreplaced Ġthe Ġsoft max Ġlayer Ġof Ġthe ĠCNN Ġwith Ġa Ġranking Ġlayer . ĠThey Ġshowed Ġimprovements Ġand Ġpublished Ġthe Ġbest Ġresult Ġso Ġfar Ġon Ġthe ĠSem E val Ġdataset , Ġto Ġour Ġknowledge . </s> soc her Ġused Ġanother ĠN N Ġarchitecture Ġfor Ġrelation Ġclassification : Ġrecursive Ġneural Ġnetworks Ġthat Ġbuilt Ġrecursive Ġsentence Ġrepresentations Ġbased Ġon Ġsynt actic Ġparsing . ĠIn Ġcontrast , Ġz hang Ġinvestigated Ġa Ġtemporal Ġstructured ĠR NN Ġwith Ġonly Ġwords Ġas Ġinput . ĠThey Ġused Ġa Ġbi - direction al Ġmodel Ġwith Ġa Ġpool ing Ġlayer Ġon Ġtop . </s> Con v olution al ĠNeural ĠNetworks Ġ( CNN ) </s> CNN s Ġperform Ġa Ġdiscrete Ġconv olution Ġon Ġan Ġinput Ġmatrix Ġwith Ġa Ġset Ġof Ġdifferent Ġfilters . ĠFor ĠN LP Ġtasks , Ġthe Ġinput Ġmatrix Ġrepresents Ġa Ġsentence : ĠEach Ġcolumn Ġof Ġthe Ġmatrix Ġstores Ġthe Ġword Ġembed ding Ġof Ġthe Ġcorresponding Ġword . ĠBy Ġapplying Ġa Ġfilter Ġwith Ġa Ġwidth Ġof , Ġe . g ., Ġthree Ġcolumns , Ġthree Ġneighboring Ġwords Ġ( tr ig ram ) Ġare Ġconv olved . ĠAfterwards , Ġthe Ġresults Ġof Ġthe Ġconv olution Ġare Ġpooled . ĠFollowing Ġcoll ober t West on , Ġwe Ġperform Ġmax - pool ing Ġwhich Ġextracts Ġthe Ġmaximum Ġvalue Ġfor Ġeach Ġfilter Ġand , Ġthus , Ġthe Ġmost Ġinformative Ġn - gram Ġfor Ġthe Ġfollowing Ġsteps . ĠFinally , Ġthe Ġresulting Ġvalues Ġare Ġconc aten ated Ġand Ġused Ġfor Ġclass ifying Ġthe Ġrelation Ġexpressed Ġin Ġthe Ġsentence . </s> Input : ĠExtended ĠMiddle ĠContext </s> One Ġof Ġour Ġcontributions Ġis Ġa Ġnew Ġinput Ġrepresentation Ġespecially Ġdesigned Ġfor Ġrelation Ġclassification . ĠThe Ġcontexts Ġare Ġsplit Ġinto Ġthree Ġdis j oint Ġregions Ġbased Ġon Ġthe Ġtwo Ġrelation Ġarguments : Ġthe Ġleft Ġcontext , Ġthe Ġmiddle Ġcontext Ġand Ġthe Ġright Ġcontext . ĠSince Ġin Ġmost Ġcases Ġthe Ġmiddle Ġcontext Ġcontains Ġthe Ġmost Ġrelevant Ġinformation Ġfor Ġthe Ġrelation , Ġwe Ġwant Ġto Ġfocus Ġon Ġit Ġbut Ġnot Ġignore Ġthe Ġother Ġregions Ġcompletely . ĠHence , Ġwe Ġpropose Ġto Ġuse Ġtwo Ġcontexts : Ġ( 1 ) Ġa Ġcombination Ġof Ġthe Ġleft Ġcontext , Ġthe Ġleft Ġentity Ġand Ġthe Ġmiddle Ġcontext ; Ġand Ġ( 2 ) Ġa Ġcombination Ġof Ġthe Ġmiddle Ġcontext , Ġthe Ġright Ġentity Ġand Ġthe Ġright Ġcontext . ĠDue Ġto Ġthe Ġrepetition Ġof Ġthe Ġmiddle Ġcontext , Ġwe Ġforce Ġthe Ġnetwork Ġto Ġpay Ġspecial Ġattention Ġto Ġit . ĠThe Ġtwo Ġcontexts Ġare Ġprocessed Ġby Ġtwo Ġindependent Ġconv olution al Ġand Ġmax - pool ing Ġlayers . ĠAfter Ġpool ing , Ġthe Ġresults Ġare Ġconc aten ated Ġto Ġform Ġthe Ġsentence Ġrepresentation . ĠFigure ĠFIG REF 3 Ġdepicts Ġthis Ġprocedure . ĠIt Ġshows Ġan Ġexam pl ary Ġsentence : ĠâĢ ľ He Ġhad Ġchest Ġpain Ġand Ġ< e 1 > head aches </ e 1 > Ġfrom Ġ< e 2 > m old </ e 2 > Ġin Ġthe Ġbedroom . âĢ Ŀ ĠIf Ġwe Ġonly Ġconsidered Ġthe Ġmiddle Ġcontext ĠâĢ ľ from âĢ Ŀ , Ġthe Ġnetwork Ġmight Ġbe Ġtempted Ġto Ġpredict Ġa Ġrelation Ġlike ĠEntity - Origin ( e 1 , e 2 ). ĠHowever , Ġby Ġalso Ġtaking Ġthe Ġleft Ġand Ġright Ġcontext Ġinto Ġaccount , Ġthe Ġmodel Ġcan Ġdetect Ġthe Ġrelation ĠCause - Effect ( e 2 , e 1 ). ĠWhile Ġthis Ġcould Ġalso Ġbe Ġachieved Ġby Ġintegrating Ġthe Ġwhole Ġcontext Ġinto Ġthe Ġmodel , Ġusing Ġthe Ġwhole Ġcontext Ġcan Ġhave Ġdisadvantages Ġfor Ġlonger Ġsentences : ĠThe Ġmax Ġpool ing Ġstep Ġcan Ġeasily Ġchoose Ġa Ġvalue Ġfrom Ġa Ġpart Ġof Ġthe Ġsentence Ġwhich Ġis Ġfar Ġaway Ġfrom Ġthe Ġmention Ġof Ġthe Ġrelation . ĠWith Ġsplitting Ġthe Ġcontext Ġinto Ġtwo Ġparts , Ġwe Ġreduce Ġthis Ġdanger . ĠRepe ating Ġthe Ġmiddle Ġcontext Ġincreases Ġthe Ġchance Ġfor Ġthe Ġmax Ġpool ing Ġstep Ġto Ġpick Ġa Ġvalue Ġfrom Ġthe Ġmiddle Ġcontext . </s> Con v olution al ĠLayer </s> Following Ġprevious Ġwork Ġ( e . g ., ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ), Ġwe Ġuse Ġ2 D Ġfilters Ġspanning Ġall Ġembed ding Ġdimensions . ĠAfter Ġconv olution , Ġa Ġmax Ġpool ing Ġoperation Ġis Ġapplied Ġthat Ġstores Ġonly Ġthe Ġhighest Ġactivation Ġof Ġeach Ġfilter . ĠWe Ġapply Ġfilters Ġwith Ġdifferent Ġwindow Ġsizes Ġ2 - 5 Ġ( multi - windows ) Ġas Ġin ĠB IB REF 5 Ġ, Ġi . e . Ġspanning Ġa Ġdifferent Ġnumber Ġof Ġinput Ġwords . </s> Rec urrent ĠNeural ĠNetworks Ġ( R NN ) </s> Traditional ĠR NN s Ġconsist Ġof Ġan Ġinput Ġvector , Ġa Ġhistory Ġvector Ġand Ġan Ġoutput Ġvector . ĠBased Ġon Ġthe Ġrepresentation Ġof Ġthe Ġcurrent Ġinput Ġword Ġand Ġthe Ġprevious Ġhistory Ġvector , Ġa Ġnew Ġhistory Ġis Ġcomputed . ĠThen , Ġan Ġoutput Ġis Ġpredicted Ġ( e . g ., Ġusing Ġa Ġsoft max Ġlayer ). ĠIn Ġcontrast Ġto Ġmost Ġtraditional ĠR NN Ġarchitectures , Ġwe Ġuse Ġthe ĠR NN Ġfor Ġsentence Ġmodeling , Ġi . e ., Ġwe Ġpredict Ġan Ġoutput Ġvector Ġonly Ġafter Ġprocessing Ġthe Ġwhole Ġsentence Ġand Ġnot Ġafter Ġeach Ġword . ĠTraining Ġis Ġperformed Ġusing Ġback prop ag ation Ġthrough Ġtime ĠB IB REF 9 Ġwhich Ġunfolds Ġthe Ġrecurrent Ġcomput ations Ġof Ġthe Ġhistory Ġvector Ġfor Ġa Ġcertain Ġnumber Ġof Ġtime Ġsteps . ĠTo Ġavoid Ġexploding Ġgrad ients , Ġwe Ġuse Ġgradient Ġclipping Ġwith Ġa Ġthreshold Ġof Ġ10 ĠB IB REF 10 Ġ. </s> Input Ġof Ġthe ĠR NN s </s> Initial Ġexperiments Ġshowed Ġthat Ġusing Ġtrig rams Ġas Ġinput Ġinstead Ġof Ġsingle Ġwords Ġled Ġto Ġsuperior Ġresults . ĠHence , Ġat Ġtim est ep ĠIN LINE FORM 0 Ġwe Ġdo Ġnot Ġonly Ġgive Ġword ĠIN LINE FORM 1 Ġto Ġthe Ġmodel Ġbut Ġthe Ġtrig ram ĠIN LINE FORM 2 Ġby Ġconc aten ating Ġthe Ġcorresponding Ġword Ġembed d ings . </s> Connection ist ĠBi - direction al ĠR NN s </s> Especially Ġfor Ġrelation Ġclassification , Ġthe Ġprocessing Ġof Ġthe Ġrelation Ġarguments Ġmight Ġbe Ġeasier Ġwith Ġknowledge Ġof Ġthe Ġsucceeding Ġwords . ĠTherefore Ġin Ġbi - direction al ĠR NN s , Ġnot Ġonly Ġa Ġhistory Ġvector Ġof Ġword ĠIN LINE FORM 0 Ġis Ġregarded Ġbut Ġalso Ġa Ġfuture Ġvector . ĠThis Ġleads Ġto Ġthe Ġfollowing Ġconditioned Ġprobability Ġfor Ġthe Ġhistory ĠIN LINE FORM 1 Ġat Ġtime Ġstep ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> Thus , Ġthe Ġnetwork Ġcan Ġbe Ġsplit Ġinto Ġthree Ġparts : Ġa Ġforward Ġpass Ġwhich Ġprocesses Ġthe Ġoriginal Ġsentence Ġword Ġby Ġword Ġ( Equ ation ĠEQ REF 6 Ġ); Ġa Ġbackward Ġpass Ġwhich Ġprocesses Ġthe Ġreversed Ġsentence Ġword Ġby Ġword Ġ( Equ ation Ġ); Ġand Ġa Ġcombination Ġof Ġboth Ġ( Equ ation Ġ). ĠAll Ġthree Ġparts Ġare Ġtrained Ġjointly . ĠThis Ġis Ġalso Ġdepicted Ġin ĠFigure ĠFIG REF 7 Ġ. </s> Comb ining Ġforward Ġand Ġbackward Ġpass Ġby Ġadding Ġtheir Ġhidden Ġlayer Ġis Ġsimilar Ġto ĠB IB REF 7 Ġ. ĠWe , Ġhowever , Ġalso Ġadd Ġa Ġconnection Ġto Ġthe Ġprevious Ġcombined Ġhidden Ġlayer Ġwith Ġweight ĠIN LINE FORM 0 Ġto Ġbe Ġable Ġto Ġinclude Ġall Ġintermediate Ġhidden Ġlayers Ġinto Ġthe Ġfinal Ġdecision Ġof Ġthe Ġnetwork Ġ( see ĠEqu ation Ġ). ĠWe Ġcall Ġthis ĠâĢ ľ connection ist Ġbi - direction al ĠR NN âĢ Ŀ . </s> In Ġour Ġexperiments , Ġwe Ġcompare Ġthis ĠR NN Ġwith Ġun i - direction al ĠR NN s Ġand Ġbi - direction al ĠR NN s Ġwithout Ġadditional Ġhidden Ġlayer Ġconnections . </s> Word ĠRepresent ations </s> Words Ġare Ġrepresented Ġby Ġconc aten ated Ġvectors : Ġa Ġword Ġembed ding Ġand Ġa Ġposition Ġfeature Ġvector . </s> P ret rained Ġword Ġembed d ings . ĠIn Ġthis Ġstudy , Ġwe Ġused Ġthe Ġword 2 vec Ġtool kit ĠB IB REF 11 Ġto Ġtrain Ġembed d ings Ġon Ġan ĠEnglish ĠWikipedia Ġfrom ĠMay Ġ2014 . ĠWe Ġonly Ġconsidered Ġwords Ġappearing Ġmore Ġthan Ġ100 Ġtimes Ġand Ġadded Ġa Ġspecial ĠP ADD ING Ġtoken Ġfor Ġconv olution . ĠThis Ġresults Ġin Ġan Ġembed ding Ġtraining Ġtext Ġof Ġabout Ġ48 5 , 000 Ġterms Ġand ĠIN LINE FORM 0 Ġtokens . ĠDuring Ġmodel Ġtraining , Ġthe Ġembed d ings Ġare Ġupdated . </s> Position Ġfeatures . ĠWe Ġincorporate Ġrandomly Ġinitialized Ġposition Ġembed d ings Ġsimilar Ġto Ġz eng 2014 , Ġn guyen Ġand Ġde S ant os 2015 . ĠIn Ġour ĠR NN Ġexperiments , Ġwe Ġinvestigate Ġdifferent Ġpossibilities Ġof Ġintegrating Ġposition Ġinformation : Ġposition Ġembed d ings , Ġposition Ġembed d ings Ġwith Ġentity Ġpresence Ġflags Ġ( flags Ġindicating Ġwhether Ġthe Ġcurrent Ġword Ġis Ġone Ġof Ġthe Ġrelation Ġarguments ), Ġand Ġposition Ġindicators ĠB IB REF 7 Ġ. </s> Object ive ĠFunction : ĠRanking ĠLoss </s> R anking . ĠWe Ġapplied Ġthe Ġranking Ġloss Ġfunction Ġproposed Ġin Ġde S ant os 2015 Ġto Ġtrain Ġour Ġmodels . ĠIt Ġmaxim izes Ġthe Ġdistance Ġbetween Ġthe Ġtrue Ġlabel ĠIN LINE FORM 0 Ġand Ġthe Ġbest Ġcompetitive Ġlabel ĠIN LINE FORM 1 Ġgiven Ġa Ġdata Ġpoint ĠIN LINE FORM 2 Ġ. ĠThe Ġobjective Ġfunction Ġis ĠDIS PLAY FORM 0 </s> with ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġbeing Ġthe Ġscores Ġfor Ġthe Ġclasses ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġrespectively . ĠThe Ġparameter ĠIN LINE FORM 4 Ġcontrols Ġthe Ġpenal ization Ġof Ġthe Ġprediction Ġerrors Ġand ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġare Ġmargins Ġfor Ġthe Ġcorrect Ġand Ġincorrect Ġclasses . ĠFollowing Ġde S ant os 2015 , Ġwe Ġset ĠIN LINE FORM 7 Ġ. ĠWe Ġdo Ġnot Ġlearn Ġa Ġpattern Ġfor Ġthe Ġclass ĠOther Ġbut Ġincrease Ġits Ġdifference Ġto Ġthe Ġbest Ġcompetitive Ġlabel Ġby Ġusing Ġonly Ġthe Ġsecond Ġsum mand Ġin ĠEqu ation ĠEQ REF 10 Ġduring Ġtraining . </s> Exper iments Ġand ĠResults </s> We Ġused Ġthe Ġrelation Ġclassification Ġdataset Ġof Ġthe ĠSem E val Ġ2010 Ġtask Ġ8 ĠB IB REF 8 Ġ. ĠIt Ġconsists Ġof Ġsentences Ġwhich Ġhave Ġbeen Ġmanually Ġlabeled Ġwith Ġ19 Ġrelations Ġ( 9 Ġdirected Ġrelations Ġand Ġone Ġartificial Ġclass ĠOther ). Ġ8 , 000 Ġsentences Ġhave Ġbeen Ġdistributed Ġas Ġtraining Ġset Ġand Ġ2 , 7 17 Ġsentences Ġserved Ġas Ġtest Ġset . ĠFor Ġevaluation , Ġwe Ġapplied Ġthe Ġofficial Ġscoring Ġscript Ġand Ġreport Ġthe Ġmacro ĠF 1 Ġscore Ġwhich Ġalso Ġserved Ġas Ġthe Ġofficial Ġresult Ġof Ġthe Ġshared Ġtask . </s> R NN Ġand ĠCNN Ġmodels Ġwere Ġimplemented Ġwith Ġthe ano ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠFor Ġall Ġour Ġmodels , Ġwe Ġuse ĠL 2 Ġregular ization Ġwith Ġa Ġweight Ġof Ġ0 . 0001 . ĠFor ĠCNN Ġtraining , Ġwe Ġuse Ġmini Ġbatches Ġof Ġ25 Ġtraining Ġexamples Ġwhile Ġwe Ġperform Ġsto ch astic Ġgradient Ġdescent Ġfor Ġthe ĠR NN . ĠThe Ġinitial Ġlearning Ġrates Ġare Ġ0 . 2 Ġfor Ġthe ĠCNN Ġand Ġ0 . 01 Ġfor Ġthe ĠR NN . ĠWe Ġtrain Ġthe Ġmodels Ġfor Ġ10 Ġ( CNN ) Ġand Ġ50 Ġ( R NN ) Ġepoch s Ġwithout Ġearly Ġstopping . ĠAs Ġactivation Ġfunction , Ġwe Ġapply Ġtan h Ġfor Ġthe ĠCNN Ġand Ġcapped ĠRe LU Ġfor Ġthe ĠR NN . ĠFor Ġtuning Ġthe Ġhyper param eters , Ġwe Ġsplit Ġthe Ġtraining Ġdata Ġinto Ġtwo Ġparts : Ġ6 . 5 k Ġ( training ) Ġand Ġ1 . 5 k Ġ( development ) Ġsentences . ĠWe Ġalso Ġtuned Ġthe Ġlearning Ġrate Ġschedule Ġon Ġdev . </s> B es ide Ġof Ġtraining Ġsingle Ġmodels , Ġwe Ġalso Ġreport Ġensemble Ġresults Ġfor Ġwhich Ġwe Ġcombined Ġthe Ġpresented Ġsingle Ġmodels Ġwith Ġa Ġvoting Ġprocess . </s> Performance Ġof ĠCNN s </s> As Ġa Ġbaseline Ġsystem , Ġwe Ġimplemented Ġa ĠCNN Ġsimilar Ġto Ġthe Ġone Ġdescribed Ġby Ġz eng 2014 . ĠIt Ġconsists Ġof Ġa Ġstandard Ġconv olution al Ġlayer Ġwith Ġfilters Ġwith Ġonly Ġone Ġwindow Ġsize , Ġfollowed Ġby Ġa Ġsoft max Ġlayer . ĠAs Ġinput Ġit Ġuses Ġthe Ġmiddle Ġcontext . ĠIn Ġcontrast Ġto Ġz eng 2014 , Ġour ĠCNN Ġdoes Ġnot Ġhave Ġan Ġadditional Ġfully Ġconnected Ġhidden Ġlayer . ĠTherefore , Ġwe Ġincreased Ġthe Ġnumber Ġof Ġconv olution al Ġfilters Ġto Ġ1200 Ġto Ġkeep Ġthe Ġnumber Ġof Ġparameters Ġcomparable . ĠWith Ġthis , Ġwe Ġobtain Ġa Ġbaseline Ġresult Ġof Ġ73 . 0 . ĠAfter Ġincluding Ġ5 Ġdimensional Ġposition Ġfeatures , Ġthe Ġperformance Ġwas Ġimproved Ġto Ġ78 . 6 Ġ( com parable Ġto Ġ78 . 9 Ġas Ġreported Ġby Ġz eng 2014 Ġwithout Ġlinguistic Ġfeatures ). </s> In Ġthe Ġnext Ġstep , Ġwe Ġinvestigate Ġhow Ġthis Ġresult Ġchanges Ġif Ġwe Ġsuccess ively Ġadd Ġfurther Ġfeatures Ġto Ġour ĠCNN : Ġmulti - windows Ġfor Ġconv olution Ġ( window Ġsizes : Ġ2 , 3 , 4 , 5 Ġand Ġ300 Ġfeature Ġmaps Ġeach ), Ġranking Ġlayer Ġinstead Ġof Ġsoft max Ġand Ġour Ġproposed Ġextended Ġmiddle Ġcontext . ĠTable ĠT AB REF 12 Ġshows Ġthe Ġresults . ĠNote Ġthat Ġall Ġnumbers Ġare Ġproduced Ġby ĠCNN s Ġwith Ġa Ġcomparable Ġnumber Ġof Ġparameters . ĠWe Ġalso Ġreport ĠF 1 Ġfor Ġincreasing Ġthe Ġword Ġembed ding Ġdimension ality Ġfrom Ġ50 Ġto Ġ400 . ĠThe Ġposition Ġembed ding Ġdimension ality Ġis Ġ5 Ġin Ġcombination Ġwith Ġ50 Ġdimensional Ġword Ġembed d ings Ġand Ġ35 Ġwith Ġ400 Ġdimensional Ġword Ġembed d ings . ĠOur Ġresults Ġshow Ġthat Ġespecially Ġthe Ġranking Ġlayer Ġand Ġthe Ġembed ding Ġsize Ġhave Ġan Ġimportant Ġimpact Ġon Ġthe Ġperformance . </s> Performance Ġof ĠR NN s </s> As Ġa Ġbaseline Ġfor Ġthe ĠR NN Ġmodels , Ġwe Ġapply Ġa Ġun i - direction al ĠR NN Ġwhich Ġpredicts Ġthe Ġrelation Ġafter Ġprocessing Ġthe Ġwhole Ġsentence . ĠWith Ġthis Ġmodel , Ġwe Ġachieve Ġan ĠF 1 Ġscore Ġof Ġ61 . 2 Ġon Ġthe ĠSem E val Ġtest Ġset . </s> After wards , Ġwe Ġinvestigate Ġthe Ġimpact Ġof Ġdifferent Ġposition Ġfeatures Ġon Ġthe Ġperformance Ġof Ġun i - direction al ĠR NN s Ġ( position Ġembed d ings , Ġposition Ġembed d ings Ġconc aten ated Ġwith Ġa Ġflag Ġindicating Ġwhether Ġthe Ġcurrent Ġword Ġis Ġan Ġentity Ġor Ġnot , Ġand Ġposition Ġindicators ĠB IB REF 7 Ġ). ĠThe Ġresults Ġindicate Ġthat Ġposition Ġindicators Ġ( i . e . Ġartificial Ġwords Ġthat Ġindicate Ġthe Ġentity Ġpresence ) Ġperform Ġthe Ġbest Ġon Ġthe ĠSem E val Ġdata . ĠWe Ġachieve Ġan ĠF 1 Ġscore Ġof Ġ73 . 4 Ġwith Ġthem . ĠHowever , Ġthe Ġdifference Ġto Ġusing Ġposition Ġembed d ings Ġwith Ġentity Ġflags Ġis Ġnot Ġstatistically Ġsignificant . </s> Similar Ġto Ġour ĠCNN Ġexperiments , Ġwe Ġsuccess ively Ġvary Ġthe ĠR NN Ġmodels Ġby Ġusing Ġbi - direction ality , Ġby Ġadding Ġconnections Ġbetween Ġthe Ġhidden Ġlayers Ġ( âĢ ľ connection ist âĢ Ŀ ), Ġby Ġapplying Ġranking Ġinstead Ġof Ġsoft max Ġto Ġpredict Ġthe Ġrelation Ġand Ġby Ġincreasing Ġthe Ġword Ġembed ding Ġdimension Ġto Ġ400 . </s> The Ġresults Ġin ĠTable ĠT AB REF 14 Ġshow Ġthat Ġall Ġof Ġthese Ġvariations Ġlead Ġto Ġstatistically Ġsignificant Ġimprovements . ĠEspecially Ġthe Ġadditional Ġhidden Ġlayer Ġconnections Ġand Ġthe Ġintegration Ġof Ġthe Ġranking Ġlayer Ġhave Ġa Ġlarge Ġimpact Ġon Ġthe Ġperformance . </s> Comb ination Ġof ĠCNN s Ġand ĠR NN s </s> Finally , Ġwe Ġcombine Ġour ĠCNN Ġand ĠR NN Ġmodels Ġusing Ġa Ġvoting Ġprocess . ĠFor Ġeach Ġsentence Ġin Ġthe Ġtest Ġset , Ġwe Ġapply Ġseveral ĠCNN Ġand ĠR NN Ġmodels Ġpresented Ġin ĠTables ĠT AB REF 12 Ġand ĠT AB REF 14 Ġand Ġpredict Ġthe Ġclass Ġwith Ġthe Ġmost Ġvotes . ĠIn Ġcase Ġof Ġa Ġtie , Ġwe Ġpick Ġone Ġof Ġthe Ġmost Ġfrequent Ġclasses Ġrandomly . ĠThe Ġcombination Ġachieves Ġan ĠF 1 Ġscore Ġof Ġ84 . 9 Ġwhich Ġis Ġbetter Ġthan Ġthe Ġperformance Ġof Ġthe Ġtwo ĠN N Ġtypes Ġalone . ĠIt , Ġthus , Ġconfirms Ġour Ġassumption Ġthat Ġthe Ġnetworks Ġprovide Ġcomplementary Ġinformation : Ġwhile Ġthe ĠR NN Ġcomp utes Ġa Ġweighted Ġcombination Ġof Ġall Ġwords Ġin Ġthe Ġsentence , Ġthe ĠCNN Ġextracts Ġthe Ġmost Ġinformative Ġn - gram s Ġfor Ġthe Ġrelation Ġand Ġonly Ġconsiders Ġtheir Ġresulting Ġactiv ations . </s> Compar ison Ġwith ĠState Ġof Ġthe ĠArt </s> Table ĠT AB REF 16 Ġshows Ġthe Ġresults Ġof Ġour Ġmodels ĠER - CNN Ġ( ext ended Ġranking ĠCNN ) Ġand ĠR - R NN Ġ( ranking ĠR NN ) Ġin Ġthe Ġcontext Ġof Ġother Ġstate - of - the - art Ġmodels . ĠOur Ġproposed Ġmodels Ġobtain Ġstate - of - the - art Ġresults Ġon Ġthe ĠSem E val Ġ2010 Ġtask Ġ8 Ġdata Ġset Ġwithout Ġmaking Ġuse Ġof Ġany Ġlinguistic Ġfeatures . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġinvestigated Ġdifferent Ġfeatures Ġand Ġarchitectural Ġchoices Ġfor Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks Ġfor Ġrelation Ġclassification Ġwithout Ġusing Ġany Ġlinguistic Ġfeatures . ĠFor Ġconv olution al Ġneural Ġnetworks , Ġwe Ġpresented Ġa Ġnew Ġcontext Ġrepresentation Ġfor Ġrelation Ġclassification . ĠFurthermore , Ġwe Ġintroduced Ġconnection ist Ġrecurrent Ġneural Ġnetworks Ġfor Ġsentence Ġclassification Ġtasks Ġand Ġperformed Ġthe Ġfirst Ġexperiments Ġwith Ġranking Ġrecurrent Ġneural Ġnetworks . ĠFinally , Ġwe Ġshowed Ġthat Ġeven Ġa Ġsimple Ġcombination Ġof Ġconv olution al Ġand Ġrecurrent Ġneural Ġnetworks Ġimproved Ġresults . ĠWith Ġour Ġneural Ġmodels , Ġwe Ġachieved Ġnew Ġstate - of - the - art Ġresults Ġon Ġthe ĠSem E val Ġ2010 Ġtask Ġ8 Ġbenchmark Ġdata . </s> Acknowled gments </s> He ike ĠAd el Ġis Ġa Ġrecipient Ġof Ġthe ĠGoogle ĠEuropean ĠDoctor al ĠFellowship Ġin ĠNatural ĠLanguage ĠProcessing Ġand Ġthis Ġresearch Ġis Ġsupported Ġby Ġthis Ġfellowship . </s> This Ġresearch Ġwas Ġalso Ġsupported Ġby ĠDeutsche ĠFors ch ung s gem e ins cha ft : Ġgrant ĠSCH U Ġ224 6 / 4 - 2 .