No
BOOLEAN
No
BOOLEAN
FLOAT SELECTED: Table 1: Summary of representative neural models for sentence pair modeling. The upper half contains sentence encoding models, and the lower half contains sentence pair interaction models.
1806.04330
Introduction </s> This Ġwork Ġis Ġlicensed Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLicense . ĠLicense Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 /. </s> Sent ence Ġpair Ġmodeling Ġis Ġa Ġfundamental Ġtechnique Ġunderlying Ġmany ĠN LP Ġtasks , Ġincluding Ġthe Ġfollowing : </s> Tr ad itionally , Ġresearchers Ġhad Ġto Ġdevelop Ġdifferent Ġmethods Ġspecific Ġfor Ġeach Ġtask . ĠNow Ġneural Ġnetworks Ġcan Ġperform Ġall Ġthe Ġabove Ġtasks Ġwith Ġthe Ġsame Ġarchitecture Ġby Ġtraining Ġend Ġto Ġend . ĠVarious Ġneural Ġmodels ĠB IB REF 1 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġhave Ġdeclared Ġstate - of - the - art Ġresults Ġfor Ġsentence Ġpair Ġmodeling Ġtasks ; Ġhowever , Ġthey Ġwere Ġcarefully Ġdesigned Ġand Ġevaluated Ġon Ġselected Ġ( often Ġone Ġor Ġtwo ) Ġdatasets Ġthat Ġcan Ġdemonstrate Ġthe Ġsuperiority Ġof Ġthe Ġmodel . ĠThe Ġresearch Ġquestions Ġare Ġas Ġfollows : ĠDo Ġthey Ġperform Ġwell Ġon Ġother Ġtasks Ġand Ġdatasets ? ĠHow Ġmuch Ġperformance Ġgain Ġis Ġdue Ġto Ġcertain Ġsystem Ġdesign Ġchoices Ġand Ġhyper param eter Ġoptimizations ? </s> To Ġanswer Ġthese Ġquestions Ġand Ġbetter Ġunderstand Ġdifferent Ġnetwork Ġdesigns , Ġwe Ġsystematically Ġanalyze Ġand Ġcompare Ġthe Ġstate - of - the - art Ġneural Ġmodels Ġacross Ġmultiple Ġtasks Ġand Ġmultiple Ġdomains . ĠName ly , Ġwe Ġimplement Ġfive Ġmodels Ġand Ġtheir Ġvariations Ġon Ġthe Ġsame ĠPy Tor ch Ġplatform : ĠIn fer Sent Ġmodel ĠB IB REF 16 Ġ, ĠShort cut - st acked ĠSent ence ĠEnc oder ĠModel ĠB IB REF 17 Ġ, ĠPair wise ĠWord ĠInter action ĠModel ĠB IB REF 1 Ġ, ĠDec om pos able ĠAttention ĠModel ĠB IB REF 10 Ġ, Ġand ĠEnhanced ĠSequ ential ĠIn ference ĠModel ĠB IB REF 0 Ġ. ĠThey Ġare Ġrepresentative Ġof Ġthe Ġtwo Ġmost Ġcommon Ġapproaches : Ġsentence Ġencoding Ġmodels Ġthat Ġlearn Ġvector Ġrepresentations Ġof Ġindividual Ġsentences Ġand Ġthen Ġcalculate Ġthe Ġsemantic Ġrelationship Ġbetween Ġsentences Ġbased Ġon Ġvector Ġdistance Ġand Ġsentence Ġpair Ġinteraction Ġmodels Ġthat Ġuse Ġsome Ġsorts Ġof Ġword Ġalignment Ġmechanisms Ġ( e . g ., Ġattention ) Ġthen Ġaggregate Ġinter - sent ence Ġinteractions . ĠWe Ġfocus Ġon Ġidentifying Ġimportant Ġnetwork Ġdesigns Ġand Ġpresent Ġa Ġseries Ġof Ġfindings Ġwith Ġquantitative Ġmeasurements Ġand Ġin - depth Ġanalyses , Ġincluding Ġ( i ) Ġincorporating Ġinter - sent ence Ġinteractions Ġis Ġcritical ; Ġ( ii ) ĠTree - L ST M Ġdoes Ġnot Ġhelp Ġas Ġmuch Ġas Ġpreviously Ġclaimed Ġbut Ġsurprisingly Ġimproves Ġperformance Ġon ĠTwitter Ġdata ; Ġ( iii ) ĠEnhanced ĠSequ ential ĠIn ference ĠModel Ġhas Ġthe Ġmost Ġconsistent Ġhigh Ġperformance Ġfor Ġlarger Ġdatasets , Ġwhile ĠPair wise ĠWord ĠInter action ĠModel Ġperforms Ġbetter Ġon Ġsmaller Ġdatasets Ġand ĠShort cut - St acked ĠSent ence ĠEnc oder ĠModel Ġis Ġthe Ġbest Ġperform aning Ġmodel Ġon Ġthe ĠQu ora Ġcorpus . ĠWe Ġrelease Ġour Ġimplementations Ġas Ġa Ġtool kit Ġto Ġthe Ġresearch Ġcommunity . </s> General ĠFramework Ġfor ĠSent ence ĠPair ĠModel ing </s> Various Ġneural Ġnetworks Ġhave Ġbeen Ġproposed Ġfor Ġsentence Ġpair Ġmodeling , Ġall Ġof Ġwhich Ġfall Ġinto Ġtwo Ġtypes Ġof Ġapproaches . ĠThe Ġsentence Ġencoding Ġapproach Ġenc odes Ġeach Ġsentence Ġinto Ġa Ġfixed - length Ġvector Ġand Ġthen Ġcomp utes Ġsentence Ġsimilarity Ġdirectly . ĠThe Ġmodel Ġof Ġthis Ġtype Ġhas Ġadvantages Ġin Ġthe Ġsimplicity Ġof Ġthe Ġnetwork Ġdesign Ġand Ġgeneral ization Ġto Ġother ĠN LP Ġtasks . ĠThe Ġsentence Ġpair Ġinteraction Ġapproach Ġtakes Ġword Ġalignment Ġand Ġinteractions Ġbetween Ġthe Ġsentence Ġpair Ġinto Ġaccount Ġand Ġoften Ġshow Ġbetter Ġperformance Ġwhen Ġtrained Ġon Ġin - domain Ġdata . ĠHere Ġwe Ġoutline Ġthe Ġtwo Ġtypes Ġof Ġneural Ġnetworks Ġunder Ġthe Ġsame Ġgeneral Ġframework : </s> Represent ative ĠModels Ġfor ĠSent ence ĠPair ĠModel ing </s> Table Ġ1 Ġgives Ġa Ġsummary Ġof Ġtypical Ġmodels Ġfor Ġsentence Ġpair Ġmodeling Ġin Ġrecent Ġyears . ĠIn Ġparticular , Ġwe Ġinvestigate Ġfive Ġmodels Ġin Ġdepth : Ġtwo Ġare Ġrepresentative Ġof Ġthe Ġsentence Ġencoding Ġtype Ġof Ġmodel , Ġand Ġthree Ġare Ġrepresentative Ġof Ġthe Ġinteraction - agg regation Ġtype Ġof Ġmodel . ĠThese Ġmodels Ġhave Ġreported Ġstate - or - the - art Ġresults Ġwith Ġvaried Ġarchitecture Ġdesign Ġ( this Ġsection ) Ġand Ġimplementation Ġdetails Ġ( Section Ġ" Im plementation ĠDetails " Ġ). </s> The ĠBi - L ST M ĠMax - pool ing ĠNetwork Ġ( In fer Sent ) </s> We Ġchoose Ġthe Ġsimple ĠBi - L ST M Ġmax - pool ing Ġnetwork Ġfrom ĠIn fer Sent ĠB IB REF 16 Ġ: </s> $$ & \ over left right arrow {\ math bf Ġ{ h }} _{ i } Ġ= ĠBi L ST M (\ math bf Ġ{ x } _{ i }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i - 1 }) Ġ\\ Ġ& \ math bf Ġ{ v } = max (\ over left right arrow {\ math bf Ġ{ h }} _{ 1 }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ 2 }, Ġ... , Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ n }) $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> where Ġ$\ over left right arrow {\ math bf Ġ{ h }} _{ i } $ Ġrepresents Ġthe Ġconc aten ation Ġof Ġhidden Ġstates Ġin Ġboth Ġdirect ons . ĠIt Ġhas Ġshown Ġbetter Ġtransfer Ġlearning Ġcapabilities Ġthan Ġseveral Ġother Ġsentence Ġembed ding Ġmodels , Ġincluding ĠSkip Th ought ĠB IB REF 31 Ġand ĠFast Sent ĠB IB REF 32 Ġ, Ġwhen Ġtrained Ġon Ġthe Ġnatural Ġlanguage Ġinference Ġdatasets . </s> The ĠShort cut - St acked ĠSent ence ĠEnc oder ĠModel Ġ( S SE ) </s> The ĠShort cut - St acked ĠSent ence ĠEnc oder Ġmodel ĠB IB REF 17 Ġis Ġa Ġsentence - based Ġembed ding Ġmodel , Ġwhich Ġenhances Ġmulti - layer ĠBi - L ST M Ġwith Ġskip Ġconnection Ġto Ġavoid Ġtraining Ġerror Ġaccumulation , Ġand Ġcalculates Ġeach Ġlayer Ġas Ġfollows : </s> $$ & \ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ k } Ġ= ĠBi L ST M (\ math bf Ġ{ x } _{ i } ^{ k }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i - 1 } ^{ k }) Ġ\\ Ġ& \ math bf Ġ{ x } _{ i } ^{ 1 } = \ math bf Ġ{ w } _{ i } Ġ\ quad Ġ( k = 1 ), Ġ\ q quad Ġ\ math bf Ġ{ x } _{ i } ^{ k } =[ \ math bf Ġ{ w } _{ i }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ k - 1 }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ k - 2 }, Ġ... , Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ 1 } ] Ġ\ quad Ġ( k > 1 ) Ġ\\ Ġ& \ math bf Ġ{ v } = max (\ over left right arrow {\ math bf Ġ{ h }} _{ 1 } ^{ m }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ 2 } ^{ m }, Ġ... , Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ n } ^{ m }) $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ math bf Ġ{ x } _{ i } ^{ k } $ Ġis Ġthe Ġinput Ġof Ġthe Ġ$ k $ Ġth ĠBi - L ST M Ġlayer Ġat Ġtime Ġstep Ġ$ i $ Ġ, Ġwhich Ġis Ġthe Ġcombination Ġof Ġoutputs Ġfrom Ġall Ġprevious Ġlayers , Ġ$\ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ k } $ Ġrepresents Ġthe Ġhidden Ġstate Ġof Ġthe Ġ$ k $ Ġth ĠBi - L ST M Ġlayer Ġin Ġboth Ġdirections . ĠThe Ġfinal Ġsentence Ġembed ding Ġ$\ math bf Ġ{ v } $ Ġis Ġthe Ġrow - based Ġmax Ġpool ing Ġover Ġthe Ġoutput Ġof Ġthe Ġlast ĠBi - L ST M Ġlayer , Ġwhere Ġ$ n $ Ġdenotes Ġthe Ġnumber Ġof Ġwords Ġwithin Ġa Ġsentence Ġand Ġ$ m $ Ġis Ġthe Ġnumber Ġof ĠBi - L ST M Ġlayers Ġ( Ġ$ m Ġ= Ġ3 $ Ġin ĠS SE ). </s> The ĠPair wise ĠWord ĠInter action ĠModel Ġ( P W IM ) </s> In Ġthe ĠPair wise ĠWord ĠInter action Ġmodel ĠB IB REF 1 Ġ, Ġeach Ġword Ġvector Ġ$\ math bf Ġ{ w } _{ i } $ Ġis Ġencoded Ġwith Ġcontext Ġthrough Ġforward Ġand Ġbackward ĠL ST Ms : Ġ$\ over right arrow {\ math bf Ġ{ h }} _{ i } Ġ= ĠL ST M ^{ f } (\ math bf Ġ{ w } _{ i }, Ġ\ over right arrow {\ math bf Ġ{ h }} _{ i - 1 }) $ Ġand Ġ$\ over left arrow {\ math bf Ġ{ h }} _{ i } Ġ= ĠL ST M ^{ b } (\ math bf Ġ{ w } _{ i }, Ġ\ over left arrow {\ math bf Ġ{ h }} _{ i + 1 }) $ Ġ. ĠFor Ġevery Ġword Ġpair Ġ$( \ math bf Ġ{ w } ^ a _{ i }, Ġ\ math bf Ġ{ w } ^ b _{ j }) $ Ġacross Ġsentences , Ġthe Ġmodel Ġdirectly Ġcalculates Ġword Ġpair Ġinteractions Ġusing Ġcos ine Ġsimilarity , ĠEucl ide an Ġdistance , Ġand Ġdot Ġproduct Ġover Ġthe Ġoutputs Ġof Ġthe Ġencoding Ġlayer : </s> $$ D (\ over right arrow {\ math bf Ġ{ h }} _{ i }, Ġ\ over right arrow {\ math bf Ġ{ h }} _{ j }) Ġ& Ġ= Ġ[ cos (\ over right arrow {\ math bf Ġ{ h }} _{ i }, Ġ\ over right arrow {\ math bf Ġ{ h }} _{ j } ), Ġ\ Vert Ġ\ over right arrow {\ math bf Ġ{ h }} _{ i } Ġ- Ġ\ over right arrow {\ math bf Ġ{ h }} _{ j }\ Vert Ġ, Ġ\ over right arrow {\ math bf Ġ{ h }} _{ i } Ġ\ cd ot Ġ\ over right arrow {\ math bf Ġ{ h }} _{ j } ] $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> The Ġabove Ġequation Ġnot Ġonly Ġapplies Ġto Ġforward Ġhidden Ġstate Ġ$\ over right arrow {\ math bf Ġ{ h }} _{ i } $ Ġand Ġbackward Ġhidden Ġstate Ġ$\ over left arrow {\ math bf Ġ{ h }} _{ i } $ Ġ, Ġbut Ġalso Ġto Ġthe Ġconc aten ation Ġ$\ over left right arrow {\ math bf Ġ{ h }} _{ i } = Ġ[ \ over right arrow {\ math bf Ġ{ h }} _{ i }, Ġ\ over left arrow {\ math bf Ġ{ h }} _{ i } ] $ Ġand Ġsumm ation Ġ$\ math bf Ġ{ h } ^{ + } _{ i } = Ġ\ over right arrow {\ math bf Ġ{ h }} _{ i } Ġ+ Ġ\ over left arrow {\ math bf Ġ{ h }} _{ i } $ Ġ, Ġresulting Ġin Ġa Ġtens or Ġ$\ math bf Ġ{ D } ^{ 13 Ġ\ times Ġ| sent 1 | Ġ\ times Ġ| sent 2 | } $ Ġafter Ġpadding Ġone Ġextra Ġbias Ġterm . ĠA ĠâĢ ľ hard âĢ Ŀ Ġattention Ġis Ġapplied Ġto Ġthe Ġinteraction Ġtens or Ġto Ġbuild Ġword Ġalignment : Ġselecting Ġthe Ġmost Ġrelated Ġword Ġpairs Ġand Ġincreasing Ġthe Ġcorresponding Ġweights Ġby Ġ10 Ġtimes . ĠThen Ġa Ġ19 - layer Ġdeep ĠCNN Ġis Ġapplied Ġto Ġaggregate Ġthe Ġword Ġinteraction Ġfeatures Ġfor Ġfinal Ġclassification . </s> The ĠDec om pos able ĠAttention ĠModel Ġ( Dec Att ) </s> The ĠDec om pos able ĠAttention Ġmodel ĠB IB REF 10 Ġis Ġone Ġof Ġthe Ġearliest Ġmodels Ġto Ġintroduce Ġattention - based Ġalignment Ġfor Ġsentence Ġpair Ġmodeling , Ġand Ġit Ġachieved Ġstate - of - the - art Ġresults Ġon Ġthe ĠSN LI Ġdataset Ġwith Ġabout Ġan Ġorder Ġof Ġmagnitude Ġfewer Ġparameters Ġthan Ġother Ġmodels Ġ( see Ġmore Ġin ĠTable Ġ5 Ġ) Ġwithout Ġrelying Ġon Ġword Ġorder Ġinformation . ĠIt Ġcomp utes Ġthe Ġword Ġpair Ġinteraction Ġbetween Ġ$\ math bf Ġ{ w } _{ i } ^{ a } $ Ġand Ġ$\ math bf Ġ{ w } _{ j } ^{ b } $ Ġ( from Ġinput Ġsentences Ġ$ s _ a $ Ġand Ġ$ s _ b $ Ġ, Ġeach Ġwith Ġ$ m $ Ġand Ġ$ n $ Ġwords , Ġrespectively ) Ġas Ġ${ e } _{ ij } Ġ= Ġ{ F (\ math bf Ġ{ w } _{ i } ^{ a }) } ^{ T } ĠF (\ math bf Ġ{ w } _{ j } ^{ b }) $ Ġ, Ġwhere Ġ$ F $ Ġis Ġa Ġfeed forward Ġnetwork ; Ġthen Ġalignment Ġis Ġdetermined Ġas Ġfollows : </s> $$ & \ math bf Ġ{\ beta Ġ} _{ i } Ġ= Ġ\ sum Ġ_ { j = 1 } ^{ n } Ġ\ frac { exp ({ e } _{ ij }) }{ \ sum Ġ_ { k = 1 } ^{ n } Ġexp ({ e } _{ ik }) } Ġ\ math bf Ġ{ w } _{ j } ^{ b } Ġ\ quad Ġ& \ math bf Ġ{\ alpha Ġ} _{ j } Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ m } Ġ\ frac { exp ({ e } _{ ij }) }{ \ sum Ġ_ { k = 1 } ^{ m } Ġexp ({ e } _{ kj }) } Ġ\ math bf Ġ{ w } _{ i } ^{ a } Ġ$$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$\ math bf Ġ{\ beta Ġ} _{ i } $ Ġis Ġthe Ġsoft Ġalignment Ġbetween Ġ$\ math bf Ġ{ w } _{ i } ^{ a } $ Ġand Ġsub ph r ases Ġ$\ math bf Ġ{ w } _{ j } ^{ b } $ Ġin Ġsentence Ġ$ s _ b $ Ġ, Ġand Ġvice Ġversa Ġfor Ġ$\ math bf Ġ{\ alpha Ġ} _{ j } $ Ġ. ĠThe Ġaligned Ġphrases Ġare Ġfed Ġinto Ġanother Ġfeed forward Ġnetwork Ġ$ G $ Ġ: Ġ$\ math bf Ġ{ v } _{ i } ^{ a } Ġ= ĠG ([ \ math bf Ġ{ w } _{ i } ^{ a }; Ġ\ math bf Ġ{\ beta Ġ} _{ i } ]) Ġ$ Ġand Ġ$\ math bf Ġ{ v } _{ j } ^{ b } Ġ= ĠG ([ \ math bf Ġ{ w } _{ j } ^{ b }; Ġ\ math bf Ġ{\ alpha Ġ} _{ j } ]) Ġ$ Ġto Ġgenerate Ġsets Ġ$\ l brace Ġ\ math bf Ġ{ v } _{ i } ^{ a }\ r brace Ġ$ Ġand Ġ$\ l brace Ġ\ math bf Ġ{ v } _{ j } ^{ b }\ r brace Ġ$ Ġ, Ġwhich Ġare Ġaggreg ated Ġby Ġsumm ation Ġand Ġthen Ġconc aten ated Ġtogether Ġfor Ġclassification . </s> The ĠEnhanced ĠSequ ential ĠIn ference ĠModel Ġ( ES IM ) </s> The ĠEnhanced ĠSequ ential ĠIn ference ĠModel ĠB IB REF 0 Ġis Ġclosely Ġrelated Ġto Ġthe ĠDec Att Ġmodel , Ġbut Ġit Ġdiffers Ġin Ġa Ġfew Ġaspects . ĠFirst , ĠChen Ġet Ġal . ĠChen - Q ian : 2017 : AC L Ġdemonstrated Ġthat Ġusing ĠBi - L ST M Ġto Ġencode Ġsequential Ġcontexts Ġis Ġimportant Ġfor Ġperformance Ġimprovement . ĠThey Ġused Ġthe Ġconc aten ation Ġ$\ over line {\ math bf Ġ{ w }} _{ i } Ġ= Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i } Ġ= Ġ[ \ over right arrow {\ math bf Ġ{ h }} _{ i }, Ġ\ over left arrow {\ math bf Ġ{ h }} _{ i } ] $ Ġof Ġboth Ġdirections Ġas Ġin Ġthe ĠPW IM Ġmodel . ĠThe Ġword Ġalignment Ġ$\ math bf Ġ{\ beta Ġ} _{ i } $ Ġand Ġ$\ math bf Ġ{\ alpha Ġ} _{ j } $ Ġbetween Ġ$\ over line {\ math bf Ġ{ w }} ^{ a } $ Ġand Ġ$\ over line {\ math bf Ġ{ w }} ^{ b } $ Ġare Ġcalculated Ġthe Ġsame Ġway Ġas Ġin ĠDec Att . ĠSecond , Ġthey Ġshowed Ġthe Ġcompetitive Ġperformance Ġof Ġrecursive Ġarchitecture Ġwith Ġconstituency Ġparsing , Ġwhich Ġcomple ments Ġwith Ġsequential ĠL ST M . ĠThe Ġfeed forward Ġfunction Ġ$ G $ Ġin ĠDec Att Ġis Ġreplaced Ġwith ĠTree - L ST M : </s> $$ & \ math bf Ġ{ v } _{ i } ^{ a } Ġ= ĠTree L ST M ([ \ over line {\ math bf Ġ{ w }} _{ i } ^{ a }; Ġ\ math bf Ġ{\ beta Ġ} _{ i }; Ġ\ over line {\ math bf Ġ{ w }} _{ i } ^{ a } - \ math bf Ġ{\ beta Ġ} _{ i }; Ġ\ over line {\ math bf Ġ{ w }} _{ i } ^{ a } Ġ\ od ot Ġ\ math bf Ġ{\ beta Ġ} _{ i } ]) \\ Ġ& \ math bf Ġ{ v } _{ j } ^{ b } Ġ= ĠTree L ST M ([ \ over line {\ math bf Ġ{ w }} _{ j } ^{ b }; Ġ\ math bf Ġ{\ alpha Ġ} _{ j }; Ġ\ over line {\ math bf Ġ{ w }} _{ j } ^{ b } - \ math bf Ġ{\ alpha Ġ} _{ j }; Ġ\ over line {\ math bf Ġ{ w }} _{ j } ^{ b } Ġ\ od ot Ġ\ math bf Ġ{\ alpha Ġ} _{ j } ]) $$ Ġ Ġ Ġ( E q . Ġ25 ) </s> Third , Ġinstead Ġof Ġusing Ġsumm ation Ġin Ġaggregation , ĠES IM Ġadapt s Ġthe Ġaverage Ġand Ġmax Ġpool ing Ġand Ġconc aten ation Ġ$\ math bf Ġ{ v } = Ġ[ \ math bf Ġ{ v } _{ ave } ^{ a }; Ġ\ math bf Ġ{ v } _{ max } ^{ a }; Ġ\ math bf Ġ{ v } _{ ave } ^{ b }; Ġ\ math bf Ġ{ v } _{ max } ^{ b } ] $ Ġbefore Ġpassing Ġthrough Ġmulti - layer Ġpercept ron Ġ( ML P ) Ġfor Ġclassification : </s> $$ & Ġ\ math bf Ġ{ v } _{ ave } ^{ a } Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ m } Ġ\ frac {\ math bf Ġ{ v } _{ i } ^{ a }} { m }, Ġ\ q quad Ġ\ math bf Ġ{ v } _{ max } ^{ a } Ġ= Ġ\ max Ġ_ { i = 1 } ^{ m }\ math bf Ġ{ v } _{ i } ^{ a } Ġ, Ġ\ q quad Ġ\ math bf Ġ{ v } _{ ave } ^{ b } Ġ= Ġ\ sum Ġ_ { j = 1 } ^{ n } Ġ\ frac {\ math bf Ġ{ v } _{ j } ^{ b }} { n }, Ġ\ q quad Ġ\ math bf Ġ{ v } _{ max } ^{ b } Ġ= Ġ\ max Ġ_ { j = 1 } ^{ n }\ math bf Ġ{ v } _{ j } ^{ b } $$ Ġ Ġ Ġ( E q . Ġ26 ) </s> Dat as ets </s> We Ġconducted Ġsentence Ġpair Ġmodeling Ġexperiments Ġon Ġeight Ġpopular Ġdatasets : Ġtwo ĠNL I Ġdatasets , Ġthree ĠPI Ġdatasets , Ġone ĠST S Ġdataset Ġand Ġtwo ĠQ A Ġdatasets . ĠTable Ġ2 Ġgives Ġa Ġcomparison Ġof Ġthese Ġdatasets : </s> SN LI ĠB IB REF 7 Ġcontains Ġ570 k Ġhypotheses Ġwritten Ġby Ġcrowds ourcing Ġworkers Ġgiven Ġthe Ġpremises . ĠIt Ġfocuses Ġon Ġthree Ġsemantic Ġrelations : Ġthe Ġpremise Ġentails Ġthe Ġhypothesis Ġ( ent ail ment ), Ġthey Ġcontradict Ġeach Ġother Ġ( cont rad iction ), Ġor Ġthey Ġare Ġunrelated Ġ( neutral ). </s> Multi - N LI ĠB IB REF 33 Ġextends Ġthe ĠSN LI Ġcorpus Ġto Ġmultiple Ġgenres Ġof Ġwritten Ġand Ġspoken Ġtexts Ġwith Ġ433 k Ġsentence Ġpairs . </s> Qu ora ĠB IB REF 34 Ġcontains Ġ400 k Ġquestion Ġpairs Ġcollected Ġfrom Ġthe ĠQu ora Ġwebsite . ĠThis Ġdataset Ġhas Ġbalanced Ġpositive Ġand Ġnegative Ġlabels Ġindicating Ġwhether Ġthe Ġquestions Ġare Ġdupl icated Ġor Ġnot . </s> Twitter - URL ĠB IB REF 35 Ġincludes Ġ50 k Ġsentence Ġpairs Ġcollected Ġfrom Ġtweets Ġthat Ġshare Ġthe Ġsame ĠURL Ġof Ġnews Ġarticles . ĠThis Ġdataset Ġcontains Ġboth Ġformal Ġand Ġinformal Ġlanguage . </s> P IT - 2015 ĠB IB REF 5 Ġcomes Ġfrom ĠSem E val - 2015 Ġand Ġwas Ġcollected Ġfrom Ġtweets Ġunder Ġthe Ġsame Ġtrending Ġtopic . ĠIt Ġcontains Ġnaturally Ġoccurred Ġ( i . e . Ġwritten Ġby Ġindependent ĠTwitter Ġusers Ġspontaneously ) Ġparaph r ases Ġand Ġnon - par aph r ases Ġwith Ġvaried Ġtopics Ġand Ġlanguage Ġstyles . </s> ST S - 2014 ĠB IB REF 36 Ġis Ġfrom ĠSem E val - 2014 , Ġconstructed Ġfrom Ġimage Ġdescriptions , Ġnews Ġheadlines , Ġtweet Ġnews , Ġdiscussion Ġforums , Ġand ĠOnt o Notes ĠB IB REF 37 Ġ. </s> Wiki Q A ĠB IB REF 8 Ġis Ġan Ġopen - domain Ġquestion - ans w ering Ġdataset . ĠFollowing ĠHe Ġand ĠLin Ġhe - lin : 2016 : N 16 - 1 , Ġquestions Ġwithout Ġcorrect Ġcandidate Ġanswer Ġsentences Ġare Ġexcluded , Ġand Ġanswer Ġsentences Ġare Ġtrunc ated Ġto Ġ40 Ġtokens , Ġresulting Ġin Ġ12 k Ġquestion - answer Ġpairs Ġfor Ġour Ġexperiments . </s> T rec Q A ĠB IB REF 38 Ġis Ġan Ġanswer Ġselection Ġtask Ġof Ġ56 k Ġquestion - answer Ġpairs Ġand Ġcreated Ġin ĠText ĠRet ri eval ĠConf erences Ġ( T REC ). ĠFor Ġboth ĠWiki Q A Ġand ĠTre c Q A Ġdatasets , Ġthe Ġbest Ġanswer Ġis Ġselected Ġaccording Ġto Ġthe Ġsemantic Ġrelated ness Ġwith Ġthe Ġquestion . </s> Im plementation ĠDetails </s> We Ġimplement Ġall Ġthe Ġmodels Ġwith Ġthe Ġsame ĠPy Tor ch Ġframework . ĠBelow , Ġwe Ġsummarize Ġthe Ġimplementation Ġdetails Ġthat Ġare Ġkey Ġfor Ġreprodu cing Ġresults Ġfor Ġeach Ġmodel : </s> S SE : ĠThis Ġmodel Ġcan Ġconverge Ġvery Ġfast , Ġfor Ġexample , Ġ2 Ġor Ġ3 Ġepoch s Ġfor Ġthe ĠSN LI Ġdataset . ĠWe Ġcontrol Ġthe Ġconvergence Ġspeed Ġby Ġupdating Ġthe Ġlearning Ġrate Ġfor Ġeach Ġepoch : Ġspecifically , Ġ$ lr = \ frac { 1 }{ 2 ^ {\ frac { ep och \ _ i }{ 2 }}} * { init \ _ lr } $ Ġ, Ġwhere Ġ$ init \ _ lr $ Ġis Ġthe Ġinitial Ġlearning Ġrate Ġand Ġ$ ep och \ _ i $ Ġis Ġthe Ġindex Ġof Ġcurrent Ġepoch . </s> Dec Att : ĠIt Ġis Ġimportant Ġto Ġuse Ġgradient Ġclipping Ġfor Ġthis Ġmodel : Ġfor Ġeach Ġgradient Ġupdate , Ġwe Ġcheck Ġthe ĠL 2 Ġnorm Ġof Ġall Ġthe Ġgradient Ġvalues , Ġif Ġit Ġis Ġgreater Ġthan Ġa Ġthreshold Ġ$ b $ Ġ, Ġwe Ġscale Ġthe Ġgradient Ġby Ġa Ġfactor Ġ$\ alpha Ġ= Ġb / L 2 \ _ norm $ Ġ. ĠAnother Ġuseful Ġprocedure Ġis Ġto Ġassemble Ġbatches Ġof Ġsentences Ġwith Ġsimilar Ġlength . </s> ES IM : ĠSimilar Ġbut Ġdifferent Ġfrom ĠDec Att , ĠES IM Ġbatches Ġsentences Ġwith Ġvaried Ġlength Ġand Ġuses Ġmasks Ġto Ġfilter Ġout Ġpadding Ġinformation . ĠIn Ġorder Ġto Ġbatch Ġthe Ġparse Ġtrees Ġwithin ĠTree - L ST M Ġrec ursion , Ġwe Ġfollow ĠBowman Ġet Ġal .' s Ġbow man - E t Al : 2016 : P 16 - 1 Ġprocedure Ġthat Ġconverts Ġtree Ġstructures Ġinto Ġthe Ġlinear Ġsequential Ġstructure Ġof Ġa Ġshift Ġreduce Ġparser . ĠTwo Ġadditional Ġmasks Ġare Ġused Ġfor Ġproducing Ġleft Ġand Ġright Ġchildren Ġof Ġa Ġtree Ġnode . </s> P W IM : ĠThe Ġcos ine Ġand ĠEucl ide an Ġdistances Ġused Ġin Ġthe Ġword Ġinteraction Ġlayer Ġhave Ġsmaller Ġvalues Ġfor Ġsimilar Ġvectors Ġwhile Ġdot Ġproducts Ġhave Ġlarger Ġvalues . ĠThe Ġperformance Ġincreases Ġif Ġwe Ġadd Ġa Ġnegative Ġsign Ġto Ġmake Ġall Ġthe Ġvector Ġsimilarity Ġmeasurements Ġbehave Ġconsistently . </s> Analysis </s> Table Ġ3 Ġand Ġ4 Ġshow Ġthe Ġresults Ġreported Ġin Ġthe Ġoriginal Ġpapers Ġand Ġthe Ġreplicated Ġresults Ġwith Ġour Ġimplementation . ĠWe Ġuse Ġaccuracy , ĠF 1 Ġscore , ĠPearson 's Ġ$ r $ Ġ, ĠMean ĠAverage ĠPrecision Ġ( MAP ), Ġand ĠMean ĠRec ip rocal ĠRank Ġ( MR R ) Ġfor Ġevaluation Ġon Ġdifferent Ġdatasets Ġfollowing Ġthe Ġliterature . ĠOur Ġreproduced Ġresults Ġare Ġslightly Ġlower Ġthan Ġthe Ġoriginal Ġresults Ġby Ġ0 . 5 Ġ$\ sim Ġ$ Ġ1 . 5 Ġpoints Ġon Ġaccuracy . ĠWe Ġsuspect Ġthe Ġfollowing Ġpotential Ġreasons : Ġ( i ) Ġless Ġextensive Ġhyper param eter Ġtuning Ġfor Ġeach Ġindividual Ġdataset ; Ġ( ii ) Ġonly Ġone Ġrun Ġwith Ġrandom Ġse eding Ġto Ġreport Ġresults ; Ġand Ġ( iii ) Ġuse Ġof Ġdifferent Ġneural Ġnetwork Ġtool k its : Ġfor Ġexample , Ġthe Ġoriginal ĠES IM Ġmodel Ġwas Ġimplemented Ġwith ĠThe ano , Ġand ĠPW IM Ġmodel Ġwas Ġin ĠTorch . </s> Here in , Ġwe Ġexamine Ġthe Ġmain Ġcomponents Ġthat Ġaccount Ġfor Ġperformance Ġin Ġsentence Ġpair Ġmodeling . </s> How Ġimportant Ġis ĠL ST M Ġencoded Ġcontext Ġinformation Ġfor Ġsentence Ġpair Ġmodeling ? </s> Regarding ĠDec Att , ĠPar ikh Ġet Ġal . Ġpar ikh - E t Al : 2016 : EM N LP 2016 Ġmentioned Ġthat ĠâĢ ľ int ra - sent ence Ġattention Ġis Ġoptional "; Ġthey Ġcan Ġachieve Ġcompetitive Ġresults Ġwithout Ġconsidering Ġcontext Ġinformation . ĠHowever , Ġnot Ġsurprisingly , Ġour Ġexperiments Ġconsistently Ġshow Ġthat Ġencoding Ġsequential Ġcontext Ġinformation Ġwith ĠL ST M Ġis Ġcritical . ĠCompared Ġto ĠDec Att , ĠES IM Ġshows Ġbetter Ġperformance Ġon Ġevery Ġdataset Ġ( see ĠTable Ġ4 Ġand ĠFigure Ġ3 Ġ). ĠThe Ġmain Ġdifference Ġbetween ĠES IM Ġand ĠDec Att Ġthat Ġcontributes Ġto Ġperformance Ġimprovement , Ġwe Ġfound , Ġis Ġthe Ġuse Ġof ĠBi - L ST M Ġand ĠTree - L ST M Ġfor Ġsentence Ġencoding , Ġrather Ġthan Ġthe Ġdifferent Ġchoices Ġof Ġaggregation Ġfunctions . </s> Why Ġdoes ĠTree - L ST M Ġhelp Ġwith ĠTwitter Ġdata ? </s> C hen Ġet Ġal . ĠChen - Q ian : 2017 : AC L Ġoffered Ġa Ġsimple Ġcombination Ġ( ES IM Ġ$ _{ seq + tree } $ Ġ) Ġby Ġaveraging Ġthe Ġprediction Ġprobabilities Ġof Ġtwo ĠES IM Ġvariants Ġthat Ġuse Ġsequential ĠBi - L ST M Ġand ĠTree - L ST M Ġrespectively , Ġand Ġsuggested ĠâĢ ľ p ars ing Ġinformation Ġcomple ments Ġvery Ġwell Ġwith ĠES IM Ġand Ġfurther Ġimproves Ġthe Ġperformance ". ĠHowever , Ġwe Ġfound Ġthat Ġadding ĠTree - L ST M Ġonly Ġhelps Ġslightly Ġor Ġnot Ġat Ġall Ġfor Ġmost Ġdatasets , Ġbut Ġit Ġhelps Ġnot icably Ġwith Ġthe Ġtwo ĠTwitter Ġparaph rase Ġdatasets . ĠWe Ġhypothes ize Ġthe Ġreason Ġis Ġthat Ġthese Ġtwo Ġdatasets Ġcome Ġfrom Ġreal - world Ġtweets Ġwhich Ġoften Ġcontain Ġextr aneous Ġtext Ġfragments , Ġin Ġcontrast Ġto ĠSN LI Ġand Ġother Ġdatasets Ġthat Ġhave Ġsentences Ġwritten Ġby Ġcrowds ourcing Ġworkers . ĠFor Ġexample , Ġthe Ġsegment ĠâĢ ľ ever Ġwondered Ġ, âĢ Ŀ Ġin Ġthe Ġsentence Ġpair Ġever Ġwondered Ġ, Ġwhy Ġyour Ġrecorded Ġ# voice Ġsounds Ġweird Ġto Ġyou ? Ġand Ġwhy Ġdo Ġour Ġrecorded Ġvoices Ġsound Ġso Ġweird Ġto Ġus ? Ġintroduces Ġa Ġdisruptive Ġcontext Ġinto Ġthe ĠBi - L ST M Ġenc oder , Ġwhile ĠTree - L ST M Ġcan Ġput Ġit Ġin Ġa Ġless Ġimportant Ġposition Ġafter Ġconstituency Ġparsing . </s> How Ġimportant Ġis Ġattentive Ġinteraction Ġfor Ġsentence Ġpair Ġmodeling ? ĠWhy Ġdoes ĠS SE Ġexcel Ġon ĠQu ora ? </s> Both ĠES IM Ġand ĠDec Att Ġ( E q . Ġ23 Ġ) Ġcalculate Ġan Ġattention - based Ġsoft Ġalignment Ġbetween Ġa Ġsentence Ġpair , Ġwhich Ġwas Ġalso Ġproposed Ġin ĠB IB REF 27 Ġand ĠB IB REF 29 Ġfor Ġsentence Ġpair Ġmodeling , Ġwhereas ĠPW IM Ġutilizes Ġa Ġhard Ġattention Ġmechanism . ĠBoth Ġattention Ġstrategies Ġare Ġcritical Ġfor Ġmodel Ġperformance . ĠIn ĠPW IM Ġmodel ĠB IB REF 1 Ġ, Ġwe Ġobserved Ġa Ġ1 Ġ$\ sim Ġ$ Ġ2 Ġpoint Ġperformance Ġdrop Ġafter Ġremoving Ġthe Ġhard Ġattention , Ġ0 Ġ$\ sim Ġ$ Ġ3 Ġpoint Ġperformance Ġdrop Ġand Ġ$\ sim Ġ$ Ġ25 % Ġtraining Ġtime Ġreduction Ġafter Ġremoving Ġthe Ġ19 - layer ĠCNN Ġaggregation . ĠLikely Ġwithout Ġeven Ġthe Ġauthors Ġof ĠS SE Ġknowing , Ġthe ĠS SE Ġmodel Ġperforms Ġextraordinarily Ġwell Ġon Ġthe ĠQu ora Ġcorpus , Ġperhaps Ġbecause ĠQu ora Ġcontains Ġmany Ġsentence Ġpairs Ġwith Ġless Ġcomplicated Ġinter - sent ence Ġinteractions Ġ( e . g ., Ġmany Ġidentical Ġwords Ġin Ġthe Ġtwo Ġsentences ) Ġand Ġincorrect Ġground Ġtruth Ġlabels Ġ( e . g ., ĠWhat Ġis Ġyour Ġbiggest Ġregret Ġin Ġlife ? Ġand ĠWhat 's Ġthe Ġbiggest Ġregret Ġyou 've Ġhad Ġin Ġlife ? Ġare Ġlabeled Ġas Ġnon - du pl icate Ġquestions Ġby Ġmistake ). </s> Figure Ġ3 Ġshows Ġthe Ġlearning Ġcurves . ĠThe ĠDec Att Ġmodel Ġconver ges Ġquickly Ġand Ġperforms Ġwell Ġon Ġlarge ĠNL I Ġdatasets Ġdue Ġto Ġits Ġdesign Ġsimplicity . ĠPW IM Ġis Ġthe Ġslow est Ġmodel Ġ( see Ġtime Ġcomparison Ġin ĠTable Ġ5 Ġ) Ġbut Ġshows Ġvery Ġstrong Ġperformance Ġon Ġsemantic Ġsimilarity Ġand Ġparaph rase Ġidentification Ġdatasets . ĠES IM Ġand ĠS SE Ġkeep Ġa Ġgood Ġbalance Ġbetween Ġtraining Ġtime Ġand Ġperformance . </s> [ 3 ] This Ġnumber Ġwas Ġreported Ġin ĠB IB REF 12 Ġby Ġco - authors Ġof ĠDec Att ĠB IB REF 10 Ġ. Ġ[ 4 ] This Ġnumber Ġwas Ġreproduced Ġby ĠWilliams Ġet Ġal . Ġwill iam s 2017 broad . Ġ[ 5 ] This Ġnumber Ġwas Ġgenerated Ġby ĠIn fer Sent Ġtrain d Ġon ĠSN LI Ġand ĠMulti - N LI Ġdatasets . </s> As Ġshown Ġin ĠFigure Ġ4 Ġ, Ġwe Ġexperimented Ġwith Ġdifferent Ġtraining Ġsizes Ġof Ġthe Ġlargest ĠSN LI Ġdataset . ĠAll Ġthe Ġmodels Ġshow Ġimproved Ġperformance Ġas Ġwe Ġincrease Ġthe Ġtraining Ġsize . ĠES IM Ġand ĠS SE Ġhave Ġvery Ġsimilar Ġtrends Ġand Ġclearly Ġoutper form ĠPW IM Ġon Ġthe ĠSN LI Ġdataset . ĠDec Att Ġshows Ġa Ġperformance Ġjump Ġwhen Ġthe Ġtraining Ġsize Ġexceeds Ġa Ġthreshold . </s> We Ġconducted Ġan Ġin - depth Ġanalysis Ġof Ġmodel Ġperformance Ġon Ġthe ĠMulti - domain ĠNL I Ġdataset Ġbased Ġon Ġdifferent Ġcategories : Ġtext Ġgenre , Ġsentence Ġpair Ġoverlap , Ġand Ġsentence Ġlength . ĠAs Ġshown Ġin ĠTable Ġ6 Ġ, Ġall Ġmodels Ġhave Ġcomparable Ġperformance Ġbetween Ġmatched Ġgenre Ġand Ġunmatched Ġgenre . ĠSent ence Ġlength Ġand Ġoverlap Ġturn Ġout Ġto Ġbe Ġtwo Ġimportant Ġfactors ĠâĢĵ Ġthe Ġlonger Ġthe Ġsentences Ġand Ġthe Ġfewer Ġtokens Ġin Ġcommon , Ġthe Ġmore Ġchallenging Ġit Ġis Ġto Ġdetermine Ġtheir Ġsemantic Ġrelationship . ĠThese Ġphenomena Ġshared Ġby Ġthe Ġstate - of - the - art Ġsystems Ġreflect Ġtheir Ġsimilar Ġdesign Ġframework Ġwhich Ġis Ġsymm etric Ġat Ġprocessing Ġboth Ġsentences Ġin Ġthe Ġpair , Ġwhile Ġquestion Ġanswering Ġand Ġnatural Ġlanguage Ġinference Ġtasks Ġare Ġdirectional ĠB IB REF 30 Ġ. ĠHow Ġto Ġincorporate Ġasymm etry Ġinto Ġmodel Ġdesign Ġwill Ġbe Ġworth Ġmore Ġexploration Ġin Ġfuture Ġresearch . </s> In Ġaddition Ġto Ġthe Ġcross - domain Ġstudy Ġ( Table Ġ6 Ġ), Ġwe Ġconducted Ġtransfer Ġlearning Ġexperiments Ġon Ġthree Ġparaph rase Ġidentification Ġdatasets Ġ( Table Ġ4 Ġ). ĠThe Ġmost Ġnoteworthy Ġphenomenon Ġis Ġthat Ġthe ĠS SE Ġmodel Ġperforms Ġbetter Ġon ĠTwitter - URL Ġand ĠPIT - 2015 Ġwhen Ġtrained Ġon Ġthe Ġlarge Ġout - of - domain ĠQu ora Ġdata Ġthan Ġthe Ġsmall Ġin - domain Ġtraining Ġdata . ĠTwo Ġlikely Ġreasons Ġare : Ġ( i ) Ġthe ĠS SE Ġmodel Ġwith Ġover Ġ29 Ġmillion Ġparameters Ġis Ġdata Ġhungry Ġand Ġ( ii ) ĠS SE Ġmodel Ġis Ġa Ġsentence Ġencoding Ġmodel , Ġwhich Ġgeneral izes Ġbetter Ġacross Ġdomains / t asks Ġthan Ġsentence Ġpair Ġinteraction Ġmodels . ĠSent ence Ġpair Ġinteraction Ġmodels Ġmay Ġencounter Ġdifficulties Ġon ĠQu ora , Ġwhich Ġcontains Ġsentence Ġpairs Ġwith Ġthe Ġhighest Ġword Ġoverlap Ġ( 51 . 5 %) Ġamong Ġall Ġdatasets Ġand Ġoften Ġcauses Ġthe Ġinteraction Ġpatterns Ġto Ġfocus Ġon Ġa Ġfew Ġkey Ġwords Ġthat Ġdiffer . ĠIn Ġcontrast , Ġthe ĠTwitter - URL Ġdataset Ġhas Ġthe Ġlowest Ġoverlap Ġ( 23 . 0 %) Ġwith Ġa Ġsemantic Ġrelationship Ġthat Ġis Ġmainly Ġbased Ġon Ġthe Ġintention Ġof Ġthe Ġtweets . </s> Conclusion </s> We Ġanalyzed Ġfive Ġdifferent Ġneural Ġmodels Ġ( and Ġtheir Ġvariations ) Ġfor Ġsentence Ġpair Ġmodeling Ġand Ġconducted Ġa Ġseries Ġof Ġexperiments Ġwith Ġeight Ġrepresentative Ġdatasets Ġfor Ġdifferent ĠN LP Ġtasks . ĠWe Ġquant ified Ġthe Ġimportance Ġof Ġthe ĠL ST M Ġenc oder Ġand Ġattentive Ġalignment Ġfor Ġinter - sent ence Ġinteraction , Ġas Ġwell Ġas Ġthe Ġtransfer Ġlearning Ġability Ġof Ġsentence Ġencoding Ġbased Ġmodels . ĠWe Ġshowed Ġthat Ġthe ĠSN LI Ġcorpus Ġof Ġover Ġ550 k Ġsentence Ġpairs Ġcannot Ġsatur ate Ġthe Ġlearning Ġcurve . ĠWe Ġsystematically Ġcompared Ġthe Ġstrengths Ġand Ġweaknesses Ġof Ġdifferent Ġnetwork Ġdesigns Ġand Ġprovided Ġinsights Ġfor Ġfuture Ġwork . </s> Acknowled gements </s> We Ġthank ĠOhio ĠSuper computer ĠCenter ĠB IB REF 39 Ġfor Ġcomputing Ġresources . ĠThis Ġwork Ġwas Ġsupported Ġin Ġpart Ġby ĠNS F ĠCR II Ġaward Ġ( RI - 17 558 98 ) Ġand ĠDAR PA Ġthrough Ġthe ĠAR O Ġ( W 911 NF - 17 - C - 00 95 ). ĠThe Ġcontent Ġof Ġthe Ġinformation Ġin Ġthis Ġdocument Ġdoes Ġnot Ġnecessarily Ġreflect Ġthe Ġposition Ġor Ġthe Ġpolicy Ġof Ġthe ĠU . S . ĠGovernment , Ġand Ġno Ġofficial Ġendorsement Ġshould Ġbe Ġinferred . </s> P ret rained ĠWord ĠEmb edd ings </s> We Ġused Ġthe Ġ200 - dimensional ĠGl o Ve Ġword Ġvectors ĠB IB REF 18 Ġ, Ġtrained Ġon Ġ27 Ġbillion Ġwords Ġfrom ĠTwitter Ġ( voc abulary Ġsize Ġof Ġ1 . 2 Ġmil ion Ġwords ) Ġfor ĠTwitter ĠURL ĠB IB REF 35 Ġand ĠPIT - 2015 ĠB IB REF 5 Ġdatasets , Ġand Ġthe Ġ300 - dimensional ĠGl o Ve Ġvectors , Ġtrained Ġon Ġ840 Ġbillion Ġwords Ġ( voc abulary Ġsize Ġof Ġ2 . 2 Ġmil ion Ġwords ) Ġfrom ĠCommon ĠC rawl Ġfor Ġall Ġother Ġdatasets . ĠFor Ġout - of - voc abulary Ġwords , Ġwe Ġinitialized Ġthe Ġword Ġvectors Ġusing Ġnormal Ġdistribution Ġwith Ġmean Ġ0 Ġand Ġdeviation Ġ1 . </s> Hyper - param eter ĠSettings </s> We Ġfollowed Ġoriginal Ġpapers Ġor Ġcode Ġimplementations Ġto Ġset Ġhyper - param eters Ġfor Ġthese Ġmodels . ĠIn ĠInf ers ent Ġmodel ĠB IB REF 16 Ġ, Ġthe Ġhidden Ġdimension Ġsize Ġfor ĠBi - L ST M Ġis Ġ2048 , Ġand Ġthe Ġfully Ġconnected Ġlayers Ġhave Ġ512 Ġhidden Ġunits . ĠIn ĠS SE Ġmodel ĠB IB REF 17 Ġ, Ġthe Ġhidden Ġsize Ġfor Ġthree ĠBi - L ST Ms Ġis Ġ512 , Ġ2014 Ġand Ġ2048 , Ġrespectively . ĠThe Ġfully Ġconnected Ġlayers Ġhave Ġ1600 Ġunits . ĠPW IM ĠB IB REF 1 Ġand ĠES IM ĠB IB REF 0 Ġboth Ġuse ĠBi - L ST M Ġfor Ġcontext Ġencoding , Ġhaving Ġ200 Ġhidden Ġunits Ġand Ġ300 Ġhidden Ġunits Ġrespectively . ĠThe ĠDec Att Ġmodel ĠB IB REF 10 Ġuses Ġthree Ġkinds Ġof Ġfeed Ġforward Ġnetworks , Ġall Ġof Ġwhich Ġhave Ġ300 Ġhidden Ġunits . ĠOther Ġparameters Ġlike Ġlearning Ġrate , Ġbatch Ġsize , Ġdrop out Ġrate , Ġand Ġall Ġof Ġthem Ġuse Ġthe Ġsame Ġsettings Ġas Ġin Ġoriginal Ġpapers . </s> Fine - tun ing Ġthe ĠModels </s> It Ġis Ġnot Ġpractical Ġto Ġfine Ġtune Ġevery Ġhyper - param eter Ġin Ġevery Ġmodel Ġand Ġevery Ġdataset , Ġsince Ġwe Ġwant Ġto Ġshow Ġhow Ġthese Ġmodels Ġcan Ġgeneral ize Ġwell Ġon Ġother Ġdatasets , Ġwe Ġneed Ġtry Ġto Ġavoid Ġfine - tun ing Ġthese Ġparameters Ġon Ġsome Ġspecific Ġdatasets , Ġotherwise Ġwe Ġcan Ġeasily Ġget Ġover - fitted Ġmodels . ĠTherefore , Ġwe Ġkeep Ġthe Ġhyper - param eters Ġunchanged Ġacross Ġdifferent Ġdatasets , Ġto Ġdemonstrate Ġthe Ġgeneral ization Ġcapability Ġof Ġeach Ġmodel . ĠThe Ġdefault Ġnumber Ġof Ġepoch s Ġfor Ġtraining Ġthese Ġmodels Ġis Ġset Ġto Ġ20 , Ġif Ġsome Ġmodels Ġcould Ġconverge Ġearlier Ġ( no Ġmore Ġperformance Ġgain Ġon Ġdevelopment Ġset ), Ġwe Ġwould Ġstop Ġrunning Ġthem Ġbefore Ġthey Ġapproached Ġepoch Ġ20 . ĠThe Ġ20 Ġepoch s Ġcan Ġguarantee Ġevery Ġmodel Ġget Ġconver ged Ġon Ġevery Ġdataset .
Introduction This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/. Sentence pair modeling is a fundamental technique underlying many NLP tasks, including the following: Traditionally, researchers had to develop different methods specific for each task. Now neural networks can perform all the above tasks with the same architecture by training end to end. Various neural models BIBREF1 , BIBREF0 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 have declared state-of-the-art results for sentence pair modeling tasks; however, they were carefully designed and evaluated on selected (often one or two) datasets that can demonstrate the superiority of the model. The research questions are as follows: Do they perform well on other tasks and datasets? How much performance gain is due to certain system design choices and hyperparameter optimizations? To answer these questions and better understand different network designs, we systematically analyze and compare the state-of-the-art neural models across multiple tasks and multiple domains. Namely, we implement five models and their variations on the same PyTorch platform: InferSent model BIBREF16 , Shortcut-stacked Sentence Encoder Model BIBREF17 , Pairwise Word Interaction Model BIBREF1 , Decomposable Attention Model BIBREF10 , and Enhanced Sequential Inference Model BIBREF0 . They are representative of the two most common approaches: sentence encoding models that learn vector representations of individual sentences and then calculate the semantic relationship between sentences based on vector distance and sentence pair interaction models that use some sorts of word alignment mechanisms (e.g., attention) then aggregate inter-sentence interactions. We focus on identifying important network designs and present a series of findings with quantitative measurements and in-depth analyses, including (i) incorporating inter-sentence interactions is critical; (ii) Tree-LSTM does not help as much as previously claimed but surprisingly improves performance on Twitter data; (iii) Enhanced Sequential Inference Model has the most consistent high performance for larger datasets, while Pairwise Word Interaction Model performs better on smaller datasets and Shortcut-Stacked Sentence Encoder Model is the best performaning model on the Quora corpus. We release our implementations as a toolkit to the research community. General Framework for Sentence Pair Modeling Various neural networks have been proposed for sentence pair modeling, all of which fall into two types of approaches. The sentence encoding approach encodes each sentence into a fixed-length vector and then computes sentence similarity directly. The model of this type has advantages in the simplicity of the network design and generalization to other NLP tasks. The sentence pair interaction approach takes word alignment and interactions between the sentence pair into account and often show better performance when trained on in-domain data. Here we outline the two types of neural networks under the same general framework: Representative Models for Sentence Pair Modeling Table 1 gives a summary of typical models for sentence pair modeling in recent years. In particular, we investigate five models in depth: two are representative of the sentence encoding type of model, and three are representative of the interaction-aggregation type of model. These models have reported state-or-the-art results with varied architecture design (this section) and implementation details (Section "Implementation Details" ). The Bi-LSTM Max-pooling Network (InferSent) We choose the simple Bi-LSTM max-pooling network from InferSent BIBREF16 : $$&\overleftrightarrow{\mathbf {h}}_{i} = BiLSTM(\mathbf {x}_{i}, \overleftrightarrow{\mathbf {h}}_{i-1}) \\ &\mathbf {v}=max(\overleftrightarrow{\mathbf {h}}_{1}, \overleftrightarrow{\mathbf {h}}_{2}, ..., \overleftrightarrow{\mathbf {h}}_{n})$$   (Eq. 17) where $\overleftrightarrow{\mathbf {h}}_{i}$ represents the concatenation of hidden states in both directons. It has shown better transfer learning capabilities than several other sentence embedding models, including SkipThought BIBREF31 and FastSent BIBREF32 , when trained on the natural language inference datasets. The Shortcut-Stacked Sentence Encoder Model (SSE) The Shortcut-Stacked Sentence Encoder model BIBREF17 is a sentence-based embedding model, which enhances multi-layer Bi-LSTM with skip connection to avoid training error accumulation, and calculates each layer as follows: $$&\overleftrightarrow{\mathbf {h}}_{i}^{k} = BiLSTM(\mathbf {x}_{i}^{k}, \overleftrightarrow{\mathbf {h}}_{i-1}^{k}) \\ &\mathbf {x}_{i}^{1}=\mathbf {w}_{i} \quad (k=1), \qquad \mathbf {x}_{i}^{k}=[\mathbf {w}_{i}, \overleftrightarrow{\mathbf {h}}_{i}^{k-1}, \overleftrightarrow{\mathbf {h}}_{i}^{k-2}, ..., \overleftrightarrow{\mathbf {h}}_{i}^{1}] \quad (k>1) \\ &\mathbf {v}=max(\overleftrightarrow{\mathbf {h}}_{1}^{m}, \overleftrightarrow{\mathbf {h}}_{2}^{m}, ..., \overleftrightarrow{\mathbf {h}}_{n}^{m})$$   (Eq. 19) where $\mathbf {x}_{i}^{k}$ is the input of the $k$ th Bi-LSTM layer at time step $i$ , which is the combination of outputs from all previous layers, $\overleftrightarrow{\mathbf {h}}_{i}^{k}$ represents the hidden state of the $k$ th Bi-LSTM layer in both directions. The final sentence embedding $\mathbf {v}$ is the row-based max pooling over the output of the last Bi-LSTM layer, where $n$ denotes the number of words within a sentence and $m$ is the number of Bi-LSTM layers ( $m = 3$ in SSE). The Pairwise Word Interaction Model (PWIM) In the Pairwise Word Interaction model BIBREF1 , each word vector $\mathbf {w}_{i}$ is encoded with context through forward and backward LSTMs: $\overrightarrow{\mathbf {h}}_{i} = LSTM^{f}(\mathbf {w}_{i}, \overrightarrow{\mathbf {h}}_{i-1})$ and $\overleftarrow{\mathbf {h}}_{i} = LSTM^{b}(\mathbf {w}_{i}, \overleftarrow{\mathbf {h}}_{i+1})$ . For every word pair $(\mathbf {w}^a_{i}, \mathbf {w}^b_{j})$ across sentences, the model directly calculates word pair interactions using cosine similarity, Euclidean distance, and dot product over the outputs of the encoding layer: $$D(\overrightarrow{\mathbf {h}}_{i}, \overrightarrow{\mathbf {h}}_{j}) & = [cos(\overrightarrow{\mathbf {h}}_{i}, \overrightarrow{\mathbf {h}}_{j}), \Vert \overrightarrow{\mathbf {h}}_{i} - \overrightarrow{\mathbf {h}}_{j}\Vert , \overrightarrow{\mathbf {h}}_{i} \cdot \overrightarrow{\mathbf {h}}_{j}]$$   (Eq. 21) The above equation not only applies to forward hidden state $\overrightarrow{\mathbf {h}}_{i}$ and backward hidden state $\overleftarrow{\mathbf {h}}_{i}$ , but also to the concatenation $\overleftrightarrow{\mathbf {h}}_{i}= [\overrightarrow{\mathbf {h}}_{i}, \overleftarrow{\mathbf {h}}_{i}]$ and summation $\mathbf {h}^{+}_{i}= \overrightarrow{\mathbf {h}}_{i} + \overleftarrow{\mathbf {h}}_{i}$ , resulting in a tensor $\mathbf {D}^{13 \times |sent1| \times |sent2|}$ after padding one extra bias term. A “hard” attention is applied to the interaction tensor to build word alignment: selecting the most related word pairs and increasing the corresponding weights by 10 times. Then a 19-layer deep CNN is applied to aggregate the word interaction features for final classification. The Decomposable Attention Model (DecAtt) The Decomposable Attention model BIBREF10 is one of the earliest models to introduce attention-based alignment for sentence pair modeling, and it achieved state-of-the-art results on the SNLI dataset with about an order of magnitude fewer parameters than other models (see more in Table 5 ) without relying on word order information. It computes the word pair interaction between $\mathbf {w}_{i}^{a}$ and $\mathbf {w}_{j}^{b}$ (from input sentences $s_a$ and $s_b$ , each with $m$ and $n$ words, respectively) as ${e}_{ij} = {F(\mathbf {w}_{i}^{a})}^{T} F(\mathbf {w}_{j}^{b})$ , where $F$ is a feedforward network; then alignment is determined as follows: $$&\mathbf {\beta }_{i} = \sum _{j=1}^{n} \frac{exp({e}_{ij})}{\sum _{k=1}^{n} exp({e}_{ik})} \mathbf {w}_{j}^{b} \quad &\mathbf {\alpha }_{j} = \sum _{i=1}^{m} \frac{exp({e}_{ij})}{\sum _{k=1}^{m} exp({e}_{kj})} \mathbf {w}_{i}^{a} $$   (Eq. 23) where $\mathbf {\beta }_{i}$ is the soft alignment between $\mathbf {w}_{i}^{a}$ and subphrases $\mathbf {w}_{j}^{b}$ in sentence $s_b$ , and vice versa for $\mathbf {\alpha }_{j}$ . The aligned phrases are fed into another feedforward network $G$ : $\mathbf {v}_{i}^{a} = G([\mathbf {w}_{i}^{a}; \mathbf {\beta }_{i}]) $ and $\mathbf {v}_{j}^{b} = G([\mathbf {w}_{j}^{b}; \mathbf {\alpha }_{j}]) $ to generate sets $\lbrace \mathbf {v}_{i}^{a}\rbrace $ and $\lbrace \mathbf {v}_{j}^{b}\rbrace $ , which are aggregated by summation and then concatenated together for classification. The Enhanced Sequential Inference Model (ESIM) The Enhanced Sequential Inference Model BIBREF0 is closely related to the DecAtt model, but it differs in a few aspects. First, Chen et al. Chen-Qian:2017:ACL demonstrated that using Bi-LSTM to encode sequential contexts is important for performance improvement. They used the concatenation $\overline{\mathbf {w}}_{i} = \overleftrightarrow{\mathbf {h}}_{i} = [\overrightarrow{\mathbf {h}}_{i}, \overleftarrow{\mathbf {h}}_{i}]$ of both directions as in the PWIM model. The word alignment $\mathbf {\beta }_{i}$ and $\mathbf {\alpha }_{j}$ between $\overline{\mathbf {w}}^{a}$ and $\overline{\mathbf {w}}^{b}$ are calculated the same way as in DecAtt. Second, they showed the competitive performance of recursive architecture with constituency parsing, which complements with sequential LSTM. The feedforward function $G$ in DecAtt is replaced with Tree-LSTM: $$&\mathbf {v}_{i}^{a} = TreeLSTM([\overline{\mathbf {w}}_{i}^{a}; \mathbf {\beta }_{i}; \overline{\mathbf {w}}_{i}^{a}-\mathbf {\beta }_{i}; \overline{\mathbf {w}}_{i}^{a} \odot \mathbf {\beta }_{i}])\\ &\mathbf {v}_{j}^{b} = TreeLSTM([\overline{\mathbf {w}}_{j}^{b}; \mathbf {\alpha }_{j}; \overline{\mathbf {w}}_{j}^{b}-\mathbf {\alpha }_{j}; \overline{\mathbf {w}}_{j}^{b} \odot \mathbf {\alpha }_{j}])$$   (Eq. 25) Third, instead of using summation in aggregation, ESIM adapts the average and max pooling and concatenation $\mathbf {v}= [\mathbf {v}_{ave}^{a}; \mathbf {v}_{max}^{a}; \mathbf {v}_{ave}^{b}; \mathbf {v}_{max}^{b}]$ before passing through multi-layer perceptron (MLP) for classification: $$& \mathbf {v}_{ave}^{a} = \sum _{i=1}^{m} \frac{\mathbf {v}_{i}^{a}}{m}, \qquad \mathbf {v}_{max}^{a} = \max _{i=1}^{m}\mathbf {v}_{i}^{a} , \qquad \mathbf {v}_{ave}^{b} = \sum _{j=1}^{n} \frac{\mathbf {v}_{j}^{b}}{n}, \qquad \mathbf {v}_{max}^{b} = \max _{j=1}^{n}\mathbf {v}_{j}^{b}$$   (Eq. 26) Datasets We conducted sentence pair modeling experiments on eight popular datasets: two NLI datasets, three PI datasets, one STS dataset and two QA datasets. Table 2 gives a comparison of these datasets: SNLI BIBREF7 contains 570k hypotheses written by crowdsourcing workers given the premises. It focuses on three semantic relations: the premise entails the hypothesis (entailment), they contradict each other (contradiction), or they are unrelated (neutral). Multi-NLI BIBREF33 extends the SNLI corpus to multiple genres of written and spoken texts with 433k sentence pairs. Quora BIBREF34 contains 400k question pairs collected from the Quora website. This dataset has balanced positive and negative labels indicating whether the questions are duplicated or not. Twitter-URL BIBREF35 includes 50k sentence pairs collected from tweets that share the same URL of news articles. This dataset contains both formal and informal language. PIT-2015 BIBREF5 comes from SemEval-2015 and was collected from tweets under the same trending topic. It contains naturally occurred (i.e. written by independent Twitter users spontaneously) paraphrases and non-paraphrases with varied topics and language styles. STS-2014 BIBREF36 is from SemEval-2014, constructed from image descriptions, news headlines, tweet news, discussion forums, and OntoNotes BIBREF37 . WikiQA BIBREF8 is an open-domain question-answering dataset. Following He and Lin he-lin:2016:N16-1, questions without correct candidate answer sentences are excluded, and answer sentences are truncated to 40 tokens, resulting in 12k question-answer pairs for our experiments. TrecQA BIBREF38 is an answer selection task of 56k question-answer pairs and created in Text Retrieval Conferences (TREC). For both WikiQA and TrecQA datasets, the best answer is selected according to the semantic relatedness with the question. Implementation Details We implement all the models with the same PyTorch framework. Below, we summarize the implementation details that are key for reproducing results for each model: SSE: This model can converge very fast, for example, 2 or 3 epochs for the SNLI dataset. We control the convergence speed by updating the learning rate for each epoch: specifically, $lr=\frac{1}{2^{\frac{epoch\_i}{2}}}*{init\_lr}$ , where $init\_lr$ is the initial learning rate and $epoch\_i$ is the index of current epoch. DecAtt: It is important to use gradient clipping for this model: for each gradient update, we check the L2 norm of all the gradient values, if it is greater than a threshold $b$ , we scale the gradient by a factor $\alpha = b/L2\_norm$ . Another useful procedure is to assemble batches of sentences with similar length. ESIM: Similar but different from DecAtt, ESIM batches sentences with varied length and uses masks to filter out padding information. In order to batch the parse trees within Tree-LSTM recursion, we follow Bowman et al.'s bowman-EtAl:2016:P16-1 procedure that converts tree structures into the linear sequential structure of a shift reduce parser. Two additional masks are used for producing left and right children of a tree node. PWIM: The cosine and Euclidean distances used in the word interaction layer have smaller values for similar vectors while dot products have larger values. The performance increases if we add a negative sign to make all the vector similarity measurements behave consistently. Analysis Table 3 and 4 show the results reported in the original papers and the replicated results with our implementation. We use accuracy, F1 score, Pearson's $r$ , Mean Average Precision (MAP), and Mean Reciprocal Rank (MRR) for evaluation on different datasets following the literature. Our reproduced results are slightly lower than the original results by 0.5 $\sim $ 1.5 points on accuracy. We suspect the following potential reasons: (i) less extensive hyperparameter tuning for each individual dataset; (ii) only one run with random seeding to report results; and (iii) use of different neural network toolkits: for example, the original ESIM model was implemented with Theano, and PWIM model was in Torch. Herein, we examine the main components that account for performance in sentence pair modeling. How important is LSTM encoded context information for sentence pair modeling? Regarding DecAtt, Parikh et al. parikh-EtAl:2016:EMNLP2016 mentioned that “intra-sentence attention is optional"; they can achieve competitive results without considering context information. However, not surprisingly, our experiments consistently show that encoding sequential context information with LSTM is critical. Compared to DecAtt, ESIM shows better performance on every dataset (see Table 4 and Figure 3 ). The main difference between ESIM and DecAtt that contributes to performance improvement, we found, is the use of Bi-LSTM and Tree-LSTM for sentence encoding, rather than the different choices of aggregation functions. Why does Tree-LSTM help with Twitter data? Chen et al. Chen-Qian:2017:ACL offered a simple combination (ESIM $_{seq+tree}$ ) by averaging the prediction probabilities of two ESIM variants that use sequential Bi-LSTM and Tree-LSTM respectively, and suggested “parsing information complements very well with ESIM and further improves the performance". However, we found that adding Tree-LSTM only helps slightly or not at all for most datasets, but it helps noticably with the two Twitter paraphrase datasets. We hypothesize the reason is that these two datasets come from real-world tweets which often contain extraneous text fragments, in contrast to SNLI and other datasets that have sentences written by crowdsourcing workers. For example, the segment “ever wondered ,” in the sentence pair ever wondered , why your recorded #voice sounds weird to you? and why do our recorded voices sound so weird to us? introduces a disruptive context into the Bi-LSTM encoder, while Tree-LSTM can put it in a less important position after constituency parsing. How important is attentive interaction for sentence pair modeling? Why does SSE excel on Quora? Both ESIM and DecAtt (Eq. 23 ) calculate an attention-based soft alignment between a sentence pair, which was also proposed in BIBREF27 and BIBREF29 for sentence pair modeling, whereas PWIM utilizes a hard attention mechanism. Both attention strategies are critical for model performance. In PWIM model BIBREF1 , we observed a 1 $\sim $ 2 point performance drop after removing the hard attention, 0 $\sim $ 3 point performance drop and $\sim $ 25% training time reduction after removing the 19-layer CNN aggregation. Likely without even the authors of SSE knowing, the SSE model performs extraordinarily well on the Quora corpus, perhaps because Quora contains many sentence pairs with less complicated inter-sentence interactions (e.g., many identical words in the two sentences) and incorrect ground truth labels (e.g., What is your biggest regret in life? and What's the biggest regret you've had in life? are labeled as non-duplicate questions by mistake). Figure 3 shows the learning curves. The DecAtt model converges quickly and performs well on large NLI datasets due to its design simplicity. PWIM is the slowest model (see time comparison in Table 5 ) but shows very strong performance on semantic similarity and paraphrase identification datasets. ESIM and SSE keep a good balance between training time and performance. [3]This number was reported in BIBREF12 by co-authors of DecAtt BIBREF10 . [4]This number was reproduced by Williams et al. williams2017broad. [5]This number was generated by InferSent traind on SNLI and Multi-NLI datasets. As shown in Figure 4 , we experimented with different training sizes of the largest SNLI dataset. All the models show improved performance as we increase the training size. ESIM and SSE have very similar trends and clearly outperform PWIM on the SNLI dataset. DecAtt shows a performance jump when the training size exceeds a threshold. We conducted an in-depth analysis of model performance on the Multi-domain NLI dataset based on different categories: text genre, sentence pair overlap, and sentence length. As shown in Table 6 , all models have comparable performance between matched genre and unmatched genre. Sentence length and overlap turn out to be two important factors – the longer the sentences and the fewer tokens in common, the more challenging it is to determine their semantic relationship. These phenomena shared by the state-of-the-art systems reflect their similar design framework which is symmetric at processing both sentences in the pair, while question answering and natural language inference tasks are directional BIBREF30 . How to incorporate asymmetry into model design will be worth more exploration in future research. In addition to the cross-domain study (Table 6 ), we conducted transfer learning experiments on three paraphrase identification datasets (Table 4 ). The most noteworthy phenomenon is that the SSE model performs better on Twitter-URL and PIT-2015 when trained on the large out-of-domain Quora data than the small in-domain training data. Two likely reasons are: (i) the SSE model with over 29 million parameters is data hungry and (ii) SSE model is a sentence encoding model, which generalizes better across domains/tasks than sentence pair interaction models. Sentence pair interaction models may encounter difficulties on Quora, which contains sentence pairs with the highest word overlap (51.5%) among all datasets and often causes the interaction patterns to focus on a few key words that differ. In contrast, the Twitter-URL dataset has the lowest overlap (23.0%) with a semantic relationship that is mainly based on the intention of the tweets. Conclusion We analyzed five different neural models (and their variations) for sentence pair modeling and conducted a series of experiments with eight representative datasets for different NLP tasks. We quantified the importance of the LSTM encoder and attentive alignment for inter-sentence interaction, as well as the transfer learning ability of sentence encoding based models. We showed that the SNLI corpus of over 550k sentence pairs cannot saturate the learning curve. We systematically compared the strengths and weaknesses of different network designs and provided insights for future work. Acknowledgements We thank Ohio Supercomputer Center BIBREF39 for computing resources. This work was supported in part by NSF CRII award (RI-1755898) and DARPA through the ARO (W911NF-17-C-0095). The content of the information in this document does not necessarily reflect the position or the policy of the U.S. Government, and no official endorsement should be inferred. Pretrained Word Embeddings We used the 200-dimensional GloVe word vectors BIBREF18 , trained on 27 billion words from Twitter (vocabulary size of 1.2 milion words) for Twitter URL BIBREF35 and PIT-2015 BIBREF5 datasets, and the 300-dimensional GloVe vectors, trained on 840 billion words (vocabulary size of 2.2 milion words) from Common Crawl for all other datasets. For out-of-vocabulary words, we initialized the word vectors using normal distribution with mean 0 and deviation 1. Hyper-parameter Settings We followed original papers or code implementations to set hyper-parameters for these models. In Infersent model BIBREF16 , the hidden dimension size for Bi-LSTM is 2048, and the fully connected layers have 512 hidden units. In SSE model BIBREF17 , the hidden size for three Bi-LSTMs is 512, 2014 and 2048, respectively. The fully connected layers have 1600 units. PWIM BIBREF1 and ESIM BIBREF0 both use Bi-LSTM for context encoding, having 200 hidden units and 300 hidden units respectively. The DecAtt model BIBREF10 uses three kinds of feed forward networks, all of which have 300 hidden units. Other parameters like learning rate, batch size, dropout rate, and all of them use the same settings as in original papers. Fine-tuning the Models It is not practical to fine tune every hyper-parameter in every model and every dataset, since we want to show how these models can generalize well on other datasets, we need try to avoid fine-tuning these parameters on some specific datasets, otherwise we can easily get over-fitted models. Therefore, we keep the hyper-parameters unchanged across different datasets, to demonstrate the generalization capability of each model. The default number of epochs for training these models is set to 20, if some models could converge earlier (no more performance gain on development set), we would stop running them before they approached epoch 20. The 20 epochs can guarantee every model get converged on every dataset.
Do the authors also analyze transformer-based architectures?
8bf7f1f93d0a2816234d36395ab40c481be9a0e0
Do Ġthe Ġauthors Ġalso Ġanalyze Ġtransformer - based Ġarchitectures ?
No
<s> Do Ġthe Ġauthors Ġalso Ġanalyze Ġtransformer - based Ġarchitectures ? </s> Introduction </s> This Ġwork Ġis Ġlicensed Ġunder Ġa ĠCreative ĠCommons ĠAttribution Ġ4 . 0 ĠInternational ĠLicense . ĠLicense Ġdetails : Ġhttp :// creat ive comm ons . org / lic enses / by / 4 . 0 /. </s> Sent ence Ġpair Ġmodeling Ġis Ġa Ġfundamental Ġtechnique Ġunderlying Ġmany ĠN LP Ġtasks , Ġincluding Ġthe Ġfollowing : </s> Tr ad itionally , Ġresearchers Ġhad Ġto Ġdevelop Ġdifferent Ġmethods Ġspecific Ġfor Ġeach Ġtask . ĠNow Ġneural Ġnetworks Ġcan Ġperform Ġall Ġthe Ġabove Ġtasks Ġwith Ġthe Ġsame Ġarchitecture Ġby Ġtraining Ġend Ġto Ġend . ĠVarious Ġneural Ġmodels ĠB IB REF 1 Ġ, ĠB IB REF 0 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġhave Ġdeclared Ġstate - of - the - art Ġresults Ġfor Ġsentence Ġpair Ġmodeling Ġtasks ; Ġhowever , Ġthey Ġwere Ġcarefully Ġdesigned Ġand Ġevaluated Ġon Ġselected Ġ( often Ġone Ġor Ġtwo ) Ġdatasets Ġthat Ġcan Ġdemonstrate Ġthe Ġsuperiority Ġof Ġthe Ġmodel . ĠThe Ġresearch Ġquestions Ġare Ġas Ġfollows : ĠDo Ġthey Ġperform Ġwell Ġon Ġother Ġtasks Ġand Ġdatasets ? ĠHow Ġmuch Ġperformance Ġgain Ġis Ġdue Ġto Ġcertain Ġsystem Ġdesign Ġchoices Ġand Ġhyper param eter Ġoptimizations ? </s> To Ġanswer Ġthese Ġquestions Ġand Ġbetter Ġunderstand Ġdifferent Ġnetwork Ġdesigns , Ġwe Ġsystematically Ġanalyze Ġand Ġcompare Ġthe Ġstate - of - the - art Ġneural Ġmodels Ġacross Ġmultiple Ġtasks Ġand Ġmultiple Ġdomains . ĠName ly , Ġwe Ġimplement Ġfive Ġmodels Ġand Ġtheir Ġvariations Ġon Ġthe Ġsame ĠPy Tor ch Ġplatform : ĠIn fer Sent Ġmodel ĠB IB REF 16 Ġ, ĠShort cut - st acked ĠSent ence ĠEnc oder ĠModel ĠB IB REF 17 Ġ, ĠPair wise ĠWord ĠInter action ĠModel ĠB IB REF 1 Ġ, ĠDec om pos able ĠAttention ĠModel ĠB IB REF 10 Ġ, Ġand ĠEnhanced ĠSequ ential ĠIn ference ĠModel ĠB IB REF 0 Ġ. ĠThey Ġare Ġrepresentative Ġof Ġthe Ġtwo Ġmost Ġcommon Ġapproaches : Ġsentence Ġencoding Ġmodels Ġthat Ġlearn Ġvector Ġrepresentations Ġof Ġindividual Ġsentences Ġand Ġthen Ġcalculate Ġthe Ġsemantic Ġrelationship Ġbetween Ġsentences Ġbased Ġon Ġvector Ġdistance Ġand Ġsentence Ġpair Ġinteraction Ġmodels Ġthat Ġuse Ġsome Ġsorts Ġof Ġword Ġalignment Ġmechanisms Ġ( e . g ., Ġattention ) Ġthen Ġaggregate Ġinter - sent ence Ġinteractions . ĠWe Ġfocus Ġon Ġidentifying Ġimportant Ġnetwork Ġdesigns Ġand Ġpresent Ġa Ġseries Ġof Ġfindings Ġwith Ġquantitative Ġmeasurements Ġand Ġin - depth Ġanalyses , Ġincluding Ġ( i ) Ġincorporating Ġinter - sent ence Ġinteractions Ġis Ġcritical ; Ġ( ii ) ĠTree - L ST M Ġdoes Ġnot Ġhelp Ġas Ġmuch Ġas Ġpreviously Ġclaimed Ġbut Ġsurprisingly Ġimproves Ġperformance Ġon ĠTwitter Ġdata ; Ġ( iii ) ĠEnhanced ĠSequ ential ĠIn ference ĠModel Ġhas Ġthe Ġmost Ġconsistent Ġhigh Ġperformance Ġfor Ġlarger Ġdatasets , Ġwhile ĠPair wise ĠWord ĠInter action ĠModel Ġperforms Ġbetter Ġon Ġsmaller Ġdatasets Ġand ĠShort cut - St acked ĠSent ence ĠEnc oder ĠModel Ġis Ġthe Ġbest Ġperform aning Ġmodel Ġon Ġthe ĠQu ora Ġcorpus . ĠWe Ġrelease Ġour Ġimplementations Ġas Ġa Ġtool kit Ġto Ġthe Ġresearch Ġcommunity . </s> General ĠFramework Ġfor ĠSent ence ĠPair ĠModel ing </s> Various Ġneural Ġnetworks Ġhave Ġbeen Ġproposed Ġfor Ġsentence Ġpair Ġmodeling , Ġall Ġof Ġwhich Ġfall Ġinto Ġtwo Ġtypes Ġof Ġapproaches . ĠThe Ġsentence Ġencoding Ġapproach Ġenc odes Ġeach Ġsentence Ġinto Ġa Ġfixed - length Ġvector Ġand Ġthen Ġcomp utes Ġsentence Ġsimilarity Ġdirectly . ĠThe Ġmodel Ġof Ġthis Ġtype Ġhas Ġadvantages Ġin Ġthe Ġsimplicity Ġof Ġthe Ġnetwork Ġdesign Ġand Ġgeneral ization Ġto Ġother ĠN LP Ġtasks . ĠThe Ġsentence Ġpair Ġinteraction Ġapproach Ġtakes Ġword Ġalignment Ġand Ġinteractions Ġbetween Ġthe Ġsentence Ġpair Ġinto Ġaccount Ġand Ġoften Ġshow Ġbetter Ġperformance Ġwhen Ġtrained Ġon Ġin - domain Ġdata . ĠHere Ġwe Ġoutline Ġthe Ġtwo Ġtypes Ġof Ġneural Ġnetworks Ġunder Ġthe Ġsame Ġgeneral Ġframework : </s> Represent ative ĠModels Ġfor ĠSent ence ĠPair ĠModel ing </s> Table Ġ1 Ġgives Ġa Ġsummary Ġof Ġtypical Ġmodels Ġfor Ġsentence Ġpair Ġmodeling Ġin Ġrecent Ġyears . ĠIn Ġparticular , Ġwe Ġinvestigate Ġfive Ġmodels Ġin Ġdepth : Ġtwo Ġare Ġrepresentative Ġof Ġthe Ġsentence Ġencoding Ġtype Ġof Ġmodel , Ġand Ġthree Ġare Ġrepresentative Ġof Ġthe Ġinteraction - agg regation Ġtype Ġof Ġmodel . ĠThese Ġmodels Ġhave Ġreported Ġstate - or - the - art Ġresults Ġwith Ġvaried Ġarchitecture Ġdesign Ġ( this Ġsection ) Ġand Ġimplementation Ġdetails Ġ( Section Ġ" Im plementation ĠDetails " Ġ). </s> The ĠBi - L ST M ĠMax - pool ing ĠNetwork Ġ( In fer Sent ) </s> We Ġchoose Ġthe Ġsimple ĠBi - L ST M Ġmax - pool ing Ġnetwork Ġfrom ĠIn fer Sent ĠB IB REF 16 Ġ: </s> $$ & \ over left right arrow {\ math bf Ġ{ h }} _{ i } Ġ= ĠBi L ST M (\ math bf Ġ{ x } _{ i }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i - 1 }) Ġ\\ Ġ& \ math bf Ġ{ v } = max (\ over left right arrow {\ math bf Ġ{ h }} _{ 1 }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ 2 }, Ġ... , Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ n }) $$ Ġ Ġ Ġ( E q . Ġ17 ) </s> where Ġ$\ over left right arrow {\ math bf Ġ{ h }} _{ i } $ Ġrepresents Ġthe Ġconc aten ation Ġof Ġhidden Ġstates Ġin Ġboth Ġdirect ons . ĠIt Ġhas Ġshown Ġbetter Ġtransfer Ġlearning Ġcapabilities Ġthan Ġseveral Ġother Ġsentence Ġembed ding Ġmodels , Ġincluding ĠSkip Th ought ĠB IB REF 31 Ġand ĠFast Sent ĠB IB REF 32 Ġ, Ġwhen Ġtrained Ġon Ġthe Ġnatural Ġlanguage Ġinference Ġdatasets . </s> The ĠShort cut - St acked ĠSent ence ĠEnc oder ĠModel Ġ( S SE ) </s> The ĠShort cut - St acked ĠSent ence ĠEnc oder Ġmodel ĠB IB REF 17 Ġis Ġa Ġsentence - based Ġembed ding Ġmodel , Ġwhich Ġenhances Ġmulti - layer ĠBi - L ST M Ġwith Ġskip Ġconnection Ġto Ġavoid Ġtraining Ġerror Ġaccumulation , Ġand Ġcalculates Ġeach Ġlayer Ġas Ġfollows : </s> $$ & \ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ k } Ġ= ĠBi L ST M (\ math bf Ġ{ x } _{ i } ^{ k }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i - 1 } ^{ k }) Ġ\\ Ġ& \ math bf Ġ{ x } _{ i } ^{ 1 } = \ math bf Ġ{ w } _{ i } Ġ\ quad Ġ( k = 1 ), Ġ\ q quad Ġ\ math bf Ġ{ x } _{ i } ^{ k } =[ \ math bf Ġ{ w } _{ i }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ k - 1 }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ k - 2 }, Ġ... , Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ 1 } ] Ġ\ quad Ġ( k > 1 ) Ġ\\ Ġ& \ math bf Ġ{ v } = max (\ over left right arrow {\ math bf Ġ{ h }} _{ 1 } ^{ m }, Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ 2 } ^{ m }, Ġ... , Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ n } ^{ m }) $$ Ġ Ġ Ġ( E q . Ġ19 ) </s> where Ġ$\ math bf Ġ{ x } _{ i } ^{ k } $ Ġis Ġthe Ġinput Ġof Ġthe Ġ$ k $ Ġth ĠBi - L ST M Ġlayer Ġat Ġtime Ġstep Ġ$ i $ Ġ, Ġwhich Ġis Ġthe Ġcombination Ġof Ġoutputs Ġfrom Ġall Ġprevious Ġlayers , Ġ$\ over left right arrow {\ math bf Ġ{ h }} _{ i } ^{ k } $ Ġrepresents Ġthe Ġhidden Ġstate Ġof Ġthe Ġ$ k $ Ġth ĠBi - L ST M Ġlayer Ġin Ġboth Ġdirections . ĠThe Ġfinal Ġsentence Ġembed ding Ġ$\ math bf Ġ{ v } $ Ġis Ġthe Ġrow - based Ġmax Ġpool ing Ġover Ġthe Ġoutput Ġof Ġthe Ġlast ĠBi - L ST M Ġlayer , Ġwhere Ġ$ n $ Ġdenotes Ġthe Ġnumber Ġof Ġwords Ġwithin Ġa Ġsentence Ġand Ġ$ m $ Ġis Ġthe Ġnumber Ġof ĠBi - L ST M Ġlayers Ġ( Ġ$ m Ġ= Ġ3 $ Ġin ĠS SE ). </s> The ĠPair wise ĠWord ĠInter action ĠModel Ġ( P W IM ) </s> In Ġthe ĠPair wise ĠWord ĠInter action Ġmodel ĠB IB REF 1 Ġ, Ġeach Ġword Ġvector Ġ$\ math bf Ġ{ w } _{ i } $ Ġis Ġencoded Ġwith Ġcontext Ġthrough Ġforward Ġand Ġbackward ĠL ST Ms : Ġ$\ over right arrow {\ math bf Ġ{ h }} _{ i } Ġ= ĠL ST M ^{ f } (\ math bf Ġ{ w } _{ i }, Ġ\ over right arrow {\ math bf Ġ{ h }} _{ i - 1 }) $ Ġand Ġ$\ over left arrow {\ math bf Ġ{ h }} _{ i } Ġ= ĠL ST M ^{ b } (\ math bf Ġ{ w } _{ i }, Ġ\ over left arrow {\ math bf Ġ{ h }} _{ i + 1 }) $ Ġ. ĠFor Ġevery Ġword Ġpair Ġ$( \ math bf Ġ{ w } ^ a _{ i }, Ġ\ math bf Ġ{ w } ^ b _{ j }) $ Ġacross Ġsentences , Ġthe Ġmodel Ġdirectly Ġcalculates Ġword Ġpair Ġinteractions Ġusing Ġcos ine Ġsimilarity , ĠEucl ide an Ġdistance , Ġand Ġdot Ġproduct Ġover Ġthe Ġoutputs Ġof Ġthe Ġencoding Ġlayer : </s> $$ D (\ over right arrow {\ math bf Ġ{ h }} _{ i }, Ġ\ over right arrow {\ math bf Ġ{ h }} _{ j }) Ġ& Ġ= Ġ[ cos (\ over right arrow {\ math bf Ġ{ h }} _{ i }, Ġ\ over right arrow {\ math bf Ġ{ h }} _{ j } ), Ġ\ Vert Ġ\ over right arrow {\ math bf Ġ{ h }} _{ i } Ġ- Ġ\ over right arrow {\ math bf Ġ{ h }} _{ j }\ Vert Ġ, Ġ\ over right arrow {\ math bf Ġ{ h }} _{ i } Ġ\ cd ot Ġ\ over right arrow {\ math bf Ġ{ h }} _{ j } ] $$ Ġ Ġ Ġ( E q . Ġ21 ) </s> The Ġabove Ġequation Ġnot Ġonly Ġapplies Ġto Ġforward Ġhidden Ġstate Ġ$\ over right arrow {\ math bf Ġ{ h }} _{ i } $ Ġand Ġbackward Ġhidden Ġstate Ġ$\ over left arrow {\ math bf Ġ{ h }} _{ i } $ Ġ, Ġbut Ġalso Ġto Ġthe Ġconc aten ation Ġ$\ over left right arrow {\ math bf Ġ{ h }} _{ i } = Ġ[ \ over right arrow {\ math bf Ġ{ h }} _{ i }, Ġ\ over left arrow {\ math bf Ġ{ h }} _{ i } ] $ Ġand Ġsumm ation Ġ$\ math bf Ġ{ h } ^{ + } _{ i } = Ġ\ over right arrow {\ math bf Ġ{ h }} _{ i } Ġ+ Ġ\ over left arrow {\ math bf Ġ{ h }} _{ i } $ Ġ, Ġresulting Ġin Ġa Ġtens or Ġ$\ math bf Ġ{ D } ^{ 13 Ġ\ times Ġ| sent 1 | Ġ\ times Ġ| sent 2 | } $ Ġafter Ġpadding Ġone Ġextra Ġbias Ġterm . ĠA ĠâĢ ľ hard âĢ Ŀ Ġattention Ġis Ġapplied Ġto Ġthe Ġinteraction Ġtens or Ġto Ġbuild Ġword Ġalignment : Ġselecting Ġthe Ġmost Ġrelated Ġword Ġpairs Ġand Ġincreasing Ġthe Ġcorresponding Ġweights Ġby Ġ10 Ġtimes . ĠThen Ġa Ġ19 - layer Ġdeep ĠCNN Ġis Ġapplied Ġto Ġaggregate Ġthe Ġword Ġinteraction Ġfeatures Ġfor Ġfinal Ġclassification . </s> The ĠDec om pos able ĠAttention ĠModel Ġ( Dec Att ) </s> The ĠDec om pos able ĠAttention Ġmodel ĠB IB REF 10 Ġis Ġone Ġof Ġthe Ġearliest Ġmodels Ġto Ġintroduce Ġattention - based Ġalignment Ġfor Ġsentence Ġpair Ġmodeling , Ġand Ġit Ġachieved Ġstate - of - the - art Ġresults Ġon Ġthe ĠSN LI Ġdataset Ġwith Ġabout Ġan Ġorder Ġof Ġmagnitude Ġfewer Ġparameters Ġthan Ġother Ġmodels Ġ( see Ġmore Ġin ĠTable Ġ5 Ġ) Ġwithout Ġrelying Ġon Ġword Ġorder Ġinformation . ĠIt Ġcomp utes Ġthe Ġword Ġpair Ġinteraction Ġbetween Ġ$\ math bf Ġ{ w } _{ i } ^{ a } $ Ġand Ġ$\ math bf Ġ{ w } _{ j } ^{ b } $ Ġ( from Ġinput Ġsentences Ġ$ s _ a $ Ġand Ġ$ s _ b $ Ġ, Ġeach Ġwith Ġ$ m $ Ġand Ġ$ n $ Ġwords , Ġrespectively ) Ġas Ġ${ e } _{ ij } Ġ= Ġ{ F (\ math bf Ġ{ w } _{ i } ^{ a }) } ^{ T } ĠF (\ math bf Ġ{ w } _{ j } ^{ b }) $ Ġ, Ġwhere Ġ$ F $ Ġis Ġa Ġfeed forward Ġnetwork ; Ġthen Ġalignment Ġis Ġdetermined Ġas Ġfollows : </s> $$ & \ math bf Ġ{\ beta Ġ} _{ i } Ġ= Ġ\ sum Ġ_ { j = 1 } ^{ n } Ġ\ frac { exp ({ e } _{ ij }) }{ \ sum Ġ_ { k = 1 } ^{ n } Ġexp ({ e } _{ ik }) } Ġ\ math bf Ġ{ w } _{ j } ^{ b } Ġ\ quad Ġ& \ math bf Ġ{\ alpha Ġ} _{ j } Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ m } Ġ\ frac { exp ({ e } _{ ij }) }{ \ sum Ġ_ { k = 1 } ^{ m } Ġexp ({ e } _{ kj }) } Ġ\ math bf Ġ{ w } _{ i } ^{ a } Ġ$$ Ġ Ġ Ġ( E q . Ġ23 ) </s> where Ġ$\ math bf Ġ{\ beta Ġ} _{ i } $ Ġis Ġthe Ġsoft Ġalignment Ġbetween Ġ$\ math bf Ġ{ w } _{ i } ^{ a } $ Ġand Ġsub ph r ases Ġ$\ math bf Ġ{ w } _{ j } ^{ b } $ Ġin Ġsentence Ġ$ s _ b $ Ġ, Ġand Ġvice Ġversa Ġfor Ġ$\ math bf Ġ{\ alpha Ġ} _{ j } $ Ġ. ĠThe Ġaligned Ġphrases Ġare Ġfed Ġinto Ġanother Ġfeed forward Ġnetwork Ġ$ G $ Ġ: Ġ$\ math bf Ġ{ v } _{ i } ^{ a } Ġ= ĠG ([ \ math bf Ġ{ w } _{ i } ^{ a }; Ġ\ math bf Ġ{\ beta Ġ} _{ i } ]) Ġ$ Ġand Ġ$\ math bf Ġ{ v } _{ j } ^{ b } Ġ= ĠG ([ \ math bf Ġ{ w } _{ j } ^{ b }; Ġ\ math bf Ġ{\ alpha Ġ} _{ j } ]) Ġ$ Ġto Ġgenerate Ġsets Ġ$\ l brace Ġ\ math bf Ġ{ v } _{ i } ^{ a }\ r brace Ġ$ Ġand Ġ$\ l brace Ġ\ math bf Ġ{ v } _{ j } ^{ b }\ r brace Ġ$ Ġ, Ġwhich Ġare Ġaggreg ated Ġby Ġsumm ation Ġand Ġthen Ġconc aten ated Ġtogether Ġfor Ġclassification . </s> The ĠEnhanced ĠSequ ential ĠIn ference ĠModel Ġ( ES IM ) </s> The ĠEnhanced ĠSequ ential ĠIn ference ĠModel ĠB IB REF 0 Ġis Ġclosely Ġrelated Ġto Ġthe ĠDec Att Ġmodel , Ġbut Ġit Ġdiffers Ġin Ġa Ġfew Ġaspects . ĠFirst , ĠChen Ġet Ġal . ĠChen - Q ian : 2017 : AC L Ġdemonstrated Ġthat Ġusing ĠBi - L ST M Ġto Ġencode Ġsequential Ġcontexts Ġis Ġimportant Ġfor Ġperformance Ġimprovement . ĠThey Ġused Ġthe Ġconc aten ation Ġ$\ over line {\ math bf Ġ{ w }} _{ i } Ġ= Ġ\ over left right arrow {\ math bf Ġ{ h }} _{ i } Ġ= Ġ[ \ over right arrow {\ math bf Ġ{ h }} _{ i }, Ġ\ over left arrow {\ math bf Ġ{ h }} _{ i } ] $ Ġof Ġboth Ġdirections Ġas Ġin Ġthe ĠPW IM Ġmodel . ĠThe Ġword Ġalignment Ġ$\ math bf Ġ{\ beta Ġ} _{ i } $ Ġand Ġ$\ math bf Ġ{\ alpha Ġ} _{ j } $ Ġbetween Ġ$\ over line {\ math bf Ġ{ w }} ^{ a } $ Ġand Ġ$\ over line {\ math bf Ġ{ w }} ^{ b } $ Ġare Ġcalculated Ġthe Ġsame Ġway Ġas Ġin ĠDec Att . ĠSecond , Ġthey Ġshowed Ġthe Ġcompetitive Ġperformance Ġof Ġrecursive Ġarchitecture Ġwith Ġconstituency Ġparsing , Ġwhich Ġcomple ments Ġwith Ġsequential ĠL ST M . ĠThe Ġfeed forward Ġfunction Ġ$ G $ Ġin ĠDec Att Ġis Ġreplaced Ġwith ĠTree - L ST M : </s> $$ & \ math bf Ġ{ v } _{ i } ^{ a } Ġ= ĠTree L ST M ([ \ over line {\ math bf Ġ{ w }} _{ i } ^{ a }; Ġ\ math bf Ġ{\ beta Ġ} _{ i }; Ġ\ over line {\ math bf Ġ{ w }} _{ i } ^{ a } - \ math bf Ġ{\ beta Ġ} _{ i }; Ġ\ over line {\ math bf Ġ{ w }} _{ i } ^{ a } Ġ\ od ot Ġ\ math bf Ġ{\ beta Ġ} _{ i } ]) \\ Ġ& \ math bf Ġ{ v } _{ j } ^{ b } Ġ= ĠTree L ST M ([ \ over line {\ math bf Ġ{ w }} _{ j } ^{ b }; Ġ\ math bf Ġ{\ alpha Ġ} _{ j }; Ġ\ over line {\ math bf Ġ{ w }} _{ j } ^{ b } - \ math bf Ġ{\ alpha Ġ} _{ j }; Ġ\ over line {\ math bf Ġ{ w }} _{ j } ^{ b } Ġ\ od ot Ġ\ math bf Ġ{\ alpha Ġ} _{ j } ]) $$ Ġ Ġ Ġ( E q . Ġ25 ) </s> Third , Ġinstead Ġof Ġusing Ġsumm ation Ġin Ġaggregation , ĠES IM Ġadapt s Ġthe Ġaverage Ġand Ġmax Ġpool ing Ġand Ġconc aten ation Ġ$\ math bf Ġ{ v } = Ġ[ \ math bf Ġ{ v } _{ ave } ^{ a }; Ġ\ math bf Ġ{ v } _{ max } ^{ a }; Ġ\ math bf Ġ{ v } _{ ave } ^{ b }; Ġ\ math bf Ġ{ v } _{ max } ^{ b } ] $ Ġbefore Ġpassing Ġthrough Ġmulti - layer Ġpercept ron Ġ( ML P ) Ġfor Ġclassification : </s> $$ & Ġ\ math bf Ġ{ v } _{ ave } ^{ a } Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ m } Ġ\ frac {\ math bf Ġ{ v } _{ i } ^{ a }} { m }, Ġ\ q quad Ġ\ math bf Ġ{ v } _{ max } ^{ a } Ġ= Ġ\ max Ġ_ { i = 1 } ^{ m }\ math bf Ġ{ v } _{ i } ^{ a } Ġ, Ġ\ q quad Ġ\ math bf Ġ{ v } _{ ave } ^{ b } Ġ= Ġ\ sum Ġ_ { j = 1 } ^{ n } Ġ\ frac {\ math bf Ġ{ v } _{ j } ^{ b }} { n }, Ġ\ q quad Ġ\ math bf Ġ{ v } _{ max } ^{ b } Ġ= Ġ\ max Ġ_ { j = 1 } ^{ n }\ math bf Ġ{ v } _{ j } ^{ b } $$ Ġ Ġ Ġ( E q . Ġ26 ) </s> Dat as ets </s> We Ġconducted Ġsentence Ġpair Ġmodeling Ġexperiments Ġon Ġeight Ġpopular Ġdatasets : Ġtwo ĠNL I Ġdatasets , Ġthree ĠPI Ġdatasets , Ġone ĠST S Ġdataset Ġand Ġtwo ĠQ A Ġdatasets . ĠTable Ġ2 Ġgives Ġa Ġcomparison Ġof Ġthese Ġdatasets : </s> SN LI ĠB IB REF 7 Ġcontains Ġ570 k Ġhypotheses Ġwritten Ġby Ġcrowds ourcing Ġworkers Ġgiven Ġthe Ġpremises . ĠIt Ġfocuses Ġon Ġthree Ġsemantic Ġrelations : Ġthe Ġpremise Ġentails Ġthe Ġhypothesis Ġ( ent ail ment ), Ġthey Ġcontradict Ġeach Ġother Ġ( cont rad iction ), Ġor Ġthey Ġare Ġunrelated Ġ( neutral ). </s> Multi - N LI ĠB IB REF 33 Ġextends Ġthe ĠSN LI Ġcorpus Ġto Ġmultiple Ġgenres Ġof Ġwritten Ġand Ġspoken Ġtexts Ġwith Ġ433 k Ġsentence Ġpairs . </s> Qu ora ĠB IB REF 34 Ġcontains Ġ400 k Ġquestion Ġpairs Ġcollected Ġfrom Ġthe ĠQu ora Ġwebsite . ĠThis Ġdataset Ġhas Ġbalanced Ġpositive Ġand Ġnegative Ġlabels Ġindicating Ġwhether Ġthe Ġquestions Ġare Ġdupl icated Ġor Ġnot . </s> Twitter - URL ĠB IB REF 35 Ġincludes Ġ50 k Ġsentence Ġpairs Ġcollected Ġfrom Ġtweets Ġthat Ġshare Ġthe Ġsame ĠURL Ġof Ġnews Ġarticles . ĠThis Ġdataset Ġcontains Ġboth Ġformal Ġand Ġinformal Ġlanguage . </s> P IT - 2015 ĠB IB REF 5 Ġcomes Ġfrom ĠSem E val - 2015 Ġand Ġwas Ġcollected Ġfrom Ġtweets Ġunder Ġthe Ġsame Ġtrending Ġtopic . ĠIt Ġcontains Ġnaturally Ġoccurred Ġ( i . e . Ġwritten Ġby Ġindependent ĠTwitter Ġusers Ġspontaneously ) Ġparaph r ases Ġand Ġnon - par aph r ases Ġwith Ġvaried Ġtopics Ġand Ġlanguage Ġstyles . </s> ST S - 2014 ĠB IB REF 36 Ġis Ġfrom ĠSem E val - 2014 , Ġconstructed Ġfrom Ġimage Ġdescriptions , Ġnews Ġheadlines , Ġtweet Ġnews , Ġdiscussion Ġforums , Ġand ĠOnt o Notes ĠB IB REF 37 Ġ. </s> Wiki Q A ĠB IB REF 8 Ġis Ġan Ġopen - domain Ġquestion - ans w ering Ġdataset . ĠFollowing ĠHe Ġand ĠLin Ġhe - lin : 2016 : N 16 - 1 , Ġquestions Ġwithout Ġcorrect Ġcandidate Ġanswer Ġsentences Ġare Ġexcluded , Ġand Ġanswer Ġsentences Ġare Ġtrunc ated Ġto Ġ40 Ġtokens , Ġresulting Ġin Ġ12 k Ġquestion - answer Ġpairs Ġfor Ġour Ġexperiments . </s> T rec Q A ĠB IB REF 38 Ġis Ġan Ġanswer Ġselection Ġtask Ġof Ġ56 k Ġquestion - answer Ġpairs Ġand Ġcreated Ġin ĠText ĠRet ri eval ĠConf erences Ġ( T REC ). ĠFor Ġboth ĠWiki Q A Ġand ĠTre c Q A Ġdatasets , Ġthe Ġbest Ġanswer Ġis Ġselected Ġaccording Ġto Ġthe Ġsemantic Ġrelated ness Ġwith Ġthe Ġquestion . </s> Im plementation ĠDetails </s> We Ġimplement Ġall Ġthe Ġmodels Ġwith Ġthe Ġsame ĠPy Tor ch Ġframework . ĠBelow , Ġwe Ġsummarize Ġthe Ġimplementation Ġdetails Ġthat Ġare Ġkey Ġfor Ġreprodu cing Ġresults Ġfor Ġeach Ġmodel : </s> S SE : ĠThis Ġmodel Ġcan Ġconverge Ġvery Ġfast , Ġfor Ġexample , Ġ2 Ġor Ġ3 Ġepoch s Ġfor Ġthe ĠSN LI Ġdataset . ĠWe Ġcontrol Ġthe Ġconvergence Ġspeed Ġby Ġupdating Ġthe Ġlearning Ġrate Ġfor Ġeach Ġepoch : Ġspecifically , Ġ$ lr = \ frac { 1 }{ 2 ^ {\ frac { ep och \ _ i }{ 2 }}} * { init \ _ lr } $ Ġ, Ġwhere Ġ$ init \ _ lr $ Ġis Ġthe Ġinitial Ġlearning Ġrate Ġand Ġ$ ep och \ _ i $ Ġis Ġthe Ġindex Ġof Ġcurrent Ġepoch . </s> Dec Att : ĠIt Ġis Ġimportant Ġto Ġuse Ġgradient Ġclipping Ġfor Ġthis Ġmodel : Ġfor Ġeach Ġgradient Ġupdate , Ġwe Ġcheck Ġthe ĠL 2 Ġnorm Ġof Ġall Ġthe Ġgradient Ġvalues , Ġif Ġit Ġis Ġgreater Ġthan Ġa Ġthreshold Ġ$ b $ Ġ, Ġwe Ġscale Ġthe Ġgradient Ġby Ġa Ġfactor Ġ$\ alpha Ġ= Ġb / L 2 \ _ norm $ Ġ. ĠAnother Ġuseful Ġprocedure Ġis Ġto Ġassemble Ġbatches Ġof Ġsentences Ġwith Ġsimilar Ġlength . </s> ES IM : ĠSimilar Ġbut Ġdifferent Ġfrom ĠDec Att , ĠES IM Ġbatches Ġsentences Ġwith Ġvaried Ġlength Ġand Ġuses Ġmasks Ġto Ġfilter Ġout Ġpadding Ġinformation . ĠIn Ġorder Ġto Ġbatch Ġthe Ġparse Ġtrees Ġwithin ĠTree - L ST M Ġrec ursion , Ġwe Ġfollow ĠBowman Ġet Ġal .' s Ġbow man - E t Al : 2016 : P 16 - 1 Ġprocedure Ġthat Ġconverts Ġtree Ġstructures Ġinto Ġthe Ġlinear Ġsequential Ġstructure Ġof Ġa Ġshift Ġreduce Ġparser . ĠTwo Ġadditional Ġmasks Ġare Ġused Ġfor Ġproducing Ġleft Ġand Ġright Ġchildren Ġof Ġa Ġtree Ġnode . </s> P W IM : ĠThe Ġcos ine Ġand ĠEucl ide an Ġdistances Ġused Ġin Ġthe Ġword Ġinteraction Ġlayer Ġhave Ġsmaller Ġvalues Ġfor Ġsimilar Ġvectors Ġwhile Ġdot Ġproducts Ġhave Ġlarger Ġvalues . ĠThe Ġperformance Ġincreases Ġif Ġwe Ġadd Ġa Ġnegative Ġsign Ġto Ġmake Ġall Ġthe Ġvector Ġsimilarity Ġmeasurements Ġbehave Ġconsistently . </s> Analysis </s> Table Ġ3 Ġand Ġ4 Ġshow Ġthe Ġresults Ġreported Ġin Ġthe Ġoriginal Ġpapers Ġand Ġthe Ġreplicated Ġresults Ġwith Ġour Ġimplementation . ĠWe Ġuse Ġaccuracy , ĠF 1 Ġscore , ĠPearson 's Ġ$ r $ Ġ, ĠMean ĠAverage ĠPrecision Ġ( MAP ), Ġand ĠMean ĠRec ip rocal ĠRank Ġ( MR R ) Ġfor Ġevaluation Ġon Ġdifferent Ġdatasets Ġfollowing Ġthe Ġliterature . ĠOur Ġreproduced Ġresults Ġare Ġslightly Ġlower Ġthan Ġthe Ġoriginal Ġresults Ġby Ġ0 . 5 Ġ$\ sim Ġ$ Ġ1 . 5 Ġpoints Ġon Ġaccuracy . ĠWe Ġsuspect Ġthe Ġfollowing Ġpotential Ġreasons : Ġ( i ) Ġless Ġextensive Ġhyper param eter Ġtuning Ġfor Ġeach Ġindividual Ġdataset ; Ġ( ii ) Ġonly Ġone Ġrun Ġwith Ġrandom Ġse eding Ġto Ġreport Ġresults ; Ġand Ġ( iii ) Ġuse Ġof Ġdifferent Ġneural Ġnetwork Ġtool k its : Ġfor Ġexample , Ġthe Ġoriginal ĠES IM Ġmodel Ġwas Ġimplemented Ġwith ĠThe ano , Ġand ĠPW IM Ġmodel Ġwas Ġin ĠTorch . </s> Here in , Ġwe Ġexamine Ġthe Ġmain Ġcomponents Ġthat Ġaccount Ġfor Ġperformance Ġin Ġsentence Ġpair Ġmodeling . </s> How Ġimportant Ġis ĠL ST M Ġencoded Ġcontext Ġinformation Ġfor Ġsentence Ġpair Ġmodeling ? </s> Regarding ĠDec Att , ĠPar ikh Ġet Ġal . Ġpar ikh - E t Al : 2016 : EM N LP 2016 Ġmentioned Ġthat ĠâĢ ľ int ra - sent ence Ġattention Ġis Ġoptional "; Ġthey Ġcan Ġachieve Ġcompetitive Ġresults Ġwithout Ġconsidering Ġcontext Ġinformation . ĠHowever , Ġnot Ġsurprisingly , Ġour Ġexperiments Ġconsistently Ġshow Ġthat Ġencoding Ġsequential Ġcontext Ġinformation Ġwith ĠL ST M Ġis Ġcritical . ĠCompared Ġto ĠDec Att , ĠES IM Ġshows Ġbetter Ġperformance Ġon Ġevery Ġdataset Ġ( see ĠTable Ġ4 Ġand ĠFigure Ġ3 Ġ). ĠThe Ġmain Ġdifference Ġbetween ĠES IM Ġand ĠDec Att Ġthat Ġcontributes Ġto Ġperformance Ġimprovement , Ġwe Ġfound , Ġis Ġthe Ġuse Ġof ĠBi - L ST M Ġand ĠTree - L ST M Ġfor Ġsentence Ġencoding , Ġrather Ġthan Ġthe Ġdifferent Ġchoices Ġof Ġaggregation Ġfunctions . </s> Why Ġdoes ĠTree - L ST M Ġhelp Ġwith ĠTwitter Ġdata ? </s> C hen Ġet Ġal . ĠChen - Q ian : 2017 : AC L Ġoffered Ġa Ġsimple Ġcombination Ġ( ES IM Ġ$ _{ seq + tree } $ Ġ) Ġby Ġaveraging Ġthe Ġprediction Ġprobabilities Ġof Ġtwo ĠES IM Ġvariants Ġthat Ġuse Ġsequential ĠBi - L ST M Ġand ĠTree - L ST M Ġrespectively , Ġand Ġsuggested ĠâĢ ľ p ars ing Ġinformation Ġcomple ments Ġvery Ġwell Ġwith ĠES IM Ġand Ġfurther Ġimproves Ġthe Ġperformance ". ĠHowever , Ġwe Ġfound Ġthat Ġadding ĠTree - L ST M Ġonly Ġhelps Ġslightly Ġor Ġnot Ġat Ġall Ġfor Ġmost Ġdatasets , Ġbut Ġit Ġhelps Ġnot icably Ġwith Ġthe Ġtwo ĠTwitter Ġparaph rase Ġdatasets . ĠWe Ġhypothes ize Ġthe Ġreason Ġis Ġthat Ġthese Ġtwo Ġdatasets Ġcome Ġfrom Ġreal - world Ġtweets Ġwhich Ġoften Ġcontain Ġextr aneous Ġtext Ġfragments , Ġin Ġcontrast Ġto ĠSN LI Ġand Ġother Ġdatasets Ġthat Ġhave Ġsentences Ġwritten Ġby Ġcrowds ourcing Ġworkers . ĠFor Ġexample , Ġthe Ġsegment ĠâĢ ľ ever Ġwondered Ġ, âĢ Ŀ Ġin Ġthe Ġsentence Ġpair Ġever Ġwondered Ġ, Ġwhy Ġyour Ġrecorded Ġ# voice Ġsounds Ġweird Ġto Ġyou ? Ġand Ġwhy Ġdo Ġour Ġrecorded Ġvoices Ġsound Ġso Ġweird Ġto Ġus ? Ġintroduces Ġa Ġdisruptive Ġcontext Ġinto Ġthe ĠBi - L ST M Ġenc oder , Ġwhile ĠTree - L ST M Ġcan Ġput Ġit Ġin Ġa Ġless Ġimportant Ġposition Ġafter Ġconstituency Ġparsing . </s> How Ġimportant Ġis Ġattentive Ġinteraction Ġfor Ġsentence Ġpair Ġmodeling ? ĠWhy Ġdoes ĠS SE Ġexcel Ġon ĠQu ora ? </s> Both ĠES IM Ġand ĠDec Att Ġ( E q . Ġ23 Ġ) Ġcalculate Ġan Ġattention - based Ġsoft Ġalignment Ġbetween Ġa Ġsentence Ġpair , Ġwhich Ġwas Ġalso Ġproposed Ġin ĠB IB REF 27 Ġand ĠB IB REF 29 Ġfor Ġsentence Ġpair Ġmodeling , Ġwhereas ĠPW IM Ġutilizes Ġa Ġhard Ġattention Ġmechanism . ĠBoth Ġattention Ġstrategies Ġare Ġcritical Ġfor Ġmodel Ġperformance . ĠIn ĠPW IM Ġmodel ĠB IB REF 1 Ġ, Ġwe Ġobserved Ġa Ġ1 Ġ$\ sim Ġ$ Ġ2 Ġpoint Ġperformance Ġdrop Ġafter Ġremoving Ġthe Ġhard Ġattention , Ġ0 Ġ$\ sim Ġ$ Ġ3 Ġpoint Ġperformance Ġdrop Ġand Ġ$\ sim Ġ$ Ġ25 % Ġtraining Ġtime Ġreduction Ġafter Ġremoving Ġthe Ġ19 - layer ĠCNN Ġaggregation . ĠLikely Ġwithout Ġeven Ġthe Ġauthors Ġof ĠS SE Ġknowing , Ġthe ĠS SE Ġmodel Ġperforms Ġextraordinarily Ġwell Ġon Ġthe ĠQu ora Ġcorpus , Ġperhaps Ġbecause ĠQu ora Ġcontains Ġmany Ġsentence Ġpairs Ġwith Ġless Ġcomplicated Ġinter - sent ence Ġinteractions Ġ( e . g ., Ġmany Ġidentical Ġwords Ġin Ġthe Ġtwo Ġsentences ) Ġand Ġincorrect Ġground Ġtruth Ġlabels Ġ( e . g ., ĠWhat Ġis Ġyour Ġbiggest Ġregret Ġin Ġlife ? Ġand ĠWhat 's Ġthe Ġbiggest Ġregret Ġyou 've Ġhad Ġin Ġlife ? Ġare Ġlabeled Ġas Ġnon - du pl icate Ġquestions Ġby Ġmistake ). </s> Figure Ġ3 Ġshows Ġthe Ġlearning Ġcurves . ĠThe ĠDec Att Ġmodel Ġconver ges Ġquickly Ġand Ġperforms Ġwell Ġon Ġlarge ĠNL I Ġdatasets Ġdue Ġto Ġits Ġdesign Ġsimplicity . ĠPW IM Ġis Ġthe Ġslow est Ġmodel Ġ( see Ġtime Ġcomparison Ġin ĠTable Ġ5 Ġ) Ġbut Ġshows Ġvery Ġstrong Ġperformance Ġon Ġsemantic Ġsimilarity Ġand Ġparaph rase Ġidentification Ġdatasets . ĠES IM Ġand ĠS SE Ġkeep Ġa Ġgood Ġbalance Ġbetween Ġtraining Ġtime Ġand Ġperformance . </s> [ 3 ] This Ġnumber Ġwas Ġreported Ġin ĠB IB REF 12 Ġby Ġco - authors Ġof ĠDec Att ĠB IB REF 10 Ġ. Ġ[ 4 ] This Ġnumber Ġwas Ġreproduced Ġby ĠWilliams Ġet Ġal . Ġwill iam s 2017 broad . Ġ[ 5 ] This Ġnumber Ġwas Ġgenerated Ġby ĠIn fer Sent Ġtrain d Ġon ĠSN LI Ġand ĠMulti - N LI Ġdatasets . </s> As Ġshown Ġin ĠFigure Ġ4 Ġ, Ġwe Ġexperimented Ġwith Ġdifferent Ġtraining Ġsizes Ġof Ġthe Ġlargest ĠSN LI Ġdataset . ĠAll Ġthe Ġmodels Ġshow Ġimproved Ġperformance Ġas Ġwe Ġincrease Ġthe Ġtraining Ġsize . ĠES IM Ġand ĠS SE Ġhave Ġvery Ġsimilar Ġtrends Ġand Ġclearly Ġoutper form ĠPW IM Ġon Ġthe ĠSN LI Ġdataset . ĠDec Att Ġshows Ġa Ġperformance Ġjump Ġwhen Ġthe Ġtraining Ġsize Ġexceeds Ġa Ġthreshold . </s> We Ġconducted Ġan Ġin - depth Ġanalysis Ġof Ġmodel Ġperformance Ġon Ġthe ĠMulti - domain ĠNL I Ġdataset Ġbased Ġon Ġdifferent Ġcategories : Ġtext Ġgenre , Ġsentence Ġpair Ġoverlap , Ġand Ġsentence Ġlength . ĠAs Ġshown Ġin ĠTable Ġ6 Ġ, Ġall Ġmodels Ġhave Ġcomparable Ġperformance Ġbetween Ġmatched Ġgenre Ġand Ġunmatched Ġgenre . ĠSent ence Ġlength Ġand Ġoverlap Ġturn Ġout Ġto Ġbe Ġtwo Ġimportant Ġfactors ĠâĢĵ Ġthe Ġlonger Ġthe Ġsentences Ġand Ġthe Ġfewer Ġtokens Ġin Ġcommon , Ġthe Ġmore Ġchallenging Ġit Ġis Ġto Ġdetermine Ġtheir Ġsemantic Ġrelationship . ĠThese Ġphenomena Ġshared Ġby Ġthe Ġstate - of - the - art Ġsystems Ġreflect Ġtheir Ġsimilar Ġdesign Ġframework Ġwhich Ġis Ġsymm etric Ġat Ġprocessing Ġboth Ġsentences Ġin Ġthe Ġpair , Ġwhile Ġquestion Ġanswering Ġand Ġnatural Ġlanguage Ġinference Ġtasks Ġare Ġdirectional ĠB IB REF 30 Ġ. ĠHow Ġto Ġincorporate Ġasymm etry Ġinto Ġmodel Ġdesign Ġwill Ġbe Ġworth Ġmore Ġexploration Ġin Ġfuture Ġresearch . </s> In Ġaddition Ġto Ġthe Ġcross - domain Ġstudy Ġ( Table Ġ6 Ġ), Ġwe Ġconducted Ġtransfer Ġlearning Ġexperiments Ġon Ġthree Ġparaph rase Ġidentification Ġdatasets Ġ( Table Ġ4 Ġ). ĠThe Ġmost Ġnoteworthy Ġphenomenon Ġis Ġthat Ġthe ĠS SE Ġmodel Ġperforms Ġbetter Ġon ĠTwitter - URL Ġand ĠPIT - 2015 Ġwhen Ġtrained Ġon Ġthe Ġlarge Ġout - of - domain ĠQu ora Ġdata Ġthan Ġthe Ġsmall Ġin - domain Ġtraining Ġdata . ĠTwo Ġlikely Ġreasons Ġare : Ġ( i ) Ġthe ĠS SE Ġmodel Ġwith Ġover Ġ29 Ġmillion Ġparameters Ġis Ġdata Ġhungry Ġand Ġ( ii ) ĠS SE Ġmodel Ġis Ġa Ġsentence Ġencoding Ġmodel , Ġwhich Ġgeneral izes Ġbetter Ġacross Ġdomains / t asks Ġthan Ġsentence Ġpair Ġinteraction Ġmodels . ĠSent ence Ġpair Ġinteraction Ġmodels Ġmay Ġencounter Ġdifficulties Ġon ĠQu ora , Ġwhich Ġcontains Ġsentence Ġpairs Ġwith Ġthe Ġhighest Ġword Ġoverlap Ġ( 51 . 5 %) Ġamong Ġall Ġdatasets Ġand Ġoften Ġcauses Ġthe Ġinteraction Ġpatterns Ġto Ġfocus Ġon Ġa Ġfew Ġkey Ġwords Ġthat Ġdiffer . ĠIn Ġcontrast , Ġthe ĠTwitter - URL Ġdataset Ġhas Ġthe Ġlowest Ġoverlap Ġ( 23 . 0 %) Ġwith Ġa Ġsemantic Ġrelationship Ġthat Ġis Ġmainly Ġbased Ġon Ġthe Ġintention Ġof Ġthe Ġtweets . </s> Conclusion </s> We Ġanalyzed Ġfive Ġdifferent Ġneural Ġmodels Ġ( and Ġtheir Ġvariations ) Ġfor Ġsentence Ġpair Ġmodeling Ġand Ġconducted Ġa Ġseries Ġof Ġexperiments Ġwith Ġeight Ġrepresentative Ġdatasets Ġfor Ġdifferent ĠN LP Ġtasks . ĠWe Ġquant ified Ġthe Ġimportance Ġof Ġthe ĠL ST M Ġenc oder Ġand Ġattentive Ġalignment Ġfor Ġinter - sent ence Ġinteraction , Ġas Ġwell Ġas Ġthe Ġtransfer Ġlearning Ġability Ġof Ġsentence Ġencoding Ġbased Ġmodels . ĠWe Ġshowed Ġthat Ġthe ĠSN LI Ġcorpus Ġof Ġover Ġ550 k Ġsentence Ġpairs Ġcannot Ġsatur ate Ġthe Ġlearning Ġcurve . ĠWe Ġsystematically Ġcompared Ġthe Ġstrengths Ġand Ġweaknesses Ġof Ġdifferent Ġnetwork Ġdesigns Ġand Ġprovided Ġinsights Ġfor Ġfuture Ġwork . </s> Acknowled gements </s> We Ġthank ĠOhio ĠSuper computer ĠCenter ĠB IB REF 39 Ġfor Ġcomputing Ġresources . ĠThis Ġwork Ġwas Ġsupported Ġin Ġpart Ġby ĠNS F ĠCR II Ġaward Ġ( RI - 17 558 98 ) Ġand ĠDAR PA Ġthrough Ġthe ĠAR O Ġ( W 911 NF - 17 - C - 00 95 ). ĠThe Ġcontent Ġof Ġthe Ġinformation Ġin Ġthis Ġdocument Ġdoes Ġnot Ġnecessarily Ġreflect Ġthe Ġposition Ġor Ġthe Ġpolicy Ġof Ġthe ĠU . S . ĠGovernment , Ġand Ġno Ġofficial Ġendorsement Ġshould Ġbe Ġinferred . </s> P ret rained ĠWord ĠEmb edd ings </s> We Ġused Ġthe Ġ200 - dimensional ĠGl o Ve Ġword Ġvectors ĠB IB REF 18 Ġ, Ġtrained Ġon Ġ27 Ġbillion Ġwords Ġfrom ĠTwitter Ġ( voc abulary Ġsize Ġof Ġ1 . 2 Ġmil ion Ġwords ) Ġfor ĠTwitter ĠURL ĠB IB REF 35 Ġand ĠPIT - 2015 ĠB IB REF 5 Ġdatasets , Ġand Ġthe Ġ300 - dimensional ĠGl o Ve Ġvectors , Ġtrained Ġon Ġ840 Ġbillion Ġwords Ġ( voc abulary Ġsize Ġof Ġ2 . 2 Ġmil ion Ġwords ) Ġfrom ĠCommon ĠC rawl Ġfor Ġall Ġother Ġdatasets . ĠFor Ġout - of - voc abulary Ġwords , Ġwe Ġinitialized Ġthe Ġword Ġvectors Ġusing Ġnormal Ġdistribution Ġwith Ġmean Ġ0 Ġand Ġdeviation Ġ1 . </s> Hyper - param eter ĠSettings </s> We Ġfollowed Ġoriginal Ġpapers Ġor Ġcode Ġimplementations Ġto Ġset Ġhyper - param eters Ġfor Ġthese Ġmodels . ĠIn ĠInf ers ent Ġmodel ĠB IB REF 16 Ġ, Ġthe Ġhidden Ġdimension Ġsize Ġfor ĠBi - L ST M Ġis Ġ2048 , Ġand Ġthe Ġfully Ġconnected Ġlayers Ġhave Ġ512 Ġhidden Ġunits . ĠIn ĠS SE Ġmodel ĠB IB REF 17 Ġ, Ġthe Ġhidden Ġsize Ġfor Ġthree ĠBi - L ST Ms Ġis Ġ512 , Ġ2014 Ġand Ġ2048 , Ġrespectively . ĠThe Ġfully Ġconnected Ġlayers Ġhave Ġ1600 Ġunits . ĠPW IM ĠB IB REF 1 Ġand ĠES IM ĠB IB REF 0 Ġboth Ġuse ĠBi - L ST M Ġfor Ġcontext Ġencoding , Ġhaving Ġ200 Ġhidden Ġunits Ġand Ġ300 Ġhidden Ġunits Ġrespectively . ĠThe ĠDec Att Ġmodel ĠB IB REF 10 Ġuses Ġthree Ġkinds Ġof Ġfeed Ġforward Ġnetworks , Ġall Ġof Ġwhich Ġhave Ġ300 Ġhidden Ġunits . ĠOther Ġparameters Ġlike Ġlearning Ġrate , Ġbatch Ġsize , Ġdrop out Ġrate , Ġand Ġall Ġof Ġthem Ġuse Ġthe Ġsame Ġsettings Ġas Ġin Ġoriginal Ġpapers . </s> Fine - tun ing Ġthe ĠModels </s> It Ġis Ġnot Ġpractical Ġto Ġfine Ġtune Ġevery Ġhyper - param eter Ġin Ġevery Ġmodel Ġand Ġevery Ġdataset , Ġsince Ġwe Ġwant Ġto Ġshow Ġhow Ġthese Ġmodels Ġcan Ġgeneral ize Ġwell Ġon Ġother Ġdatasets , Ġwe Ġneed Ġtry Ġto Ġavoid Ġfine - tun ing Ġthese Ġparameters Ġon Ġsome Ġspecific Ġdatasets , Ġotherwise Ġwe Ġcan Ġeasily Ġget Ġover - fitted Ġmodels . ĠTherefore , Ġwe Ġkeep Ġthe Ġhyper - param eters Ġunchanged Ġacross Ġdifferent Ġdatasets , Ġto Ġdemonstrate Ġthe Ġgeneral ization Ġcapability Ġof Ġeach Ġmodel . ĠThe Ġdefault Ġnumber Ġof Ġepoch s Ġfor Ġtraining Ġthese Ġmodels Ġis Ġset Ġto Ġ20 , Ġif Ġsome Ġmodels Ġcould Ġconverge Ġearlier Ġ( no Ġmore Ġperformance Ġgain Ġon Ġdevelopment Ġset ), Ġwe Ġwould Ġstop Ġrunning Ġthem Ġbefore Ġthey Ġapproached Ġepoch Ġ20 . ĠThe Ġ20 Ġepoch s Ġcan Ġguarantee Ġevery Ġmodel Ġget Ġconver ged Ġon Ġevery Ġdataset .