Task 1: Quora Duplicate Question Pair Detection, Task 2: Ranking questions
EXTRACTIVE
Quora Duplicate Question Pair Detection, Ranking questions in Bing's People Also Ask
EXTRACTIVE
Task 1: Quora Duplicate Question Pair Detection Task 2: Ranking questions in Bing's People Also Ask
Task 1: Quora Duplicate Question Pair Detection Task 2: Ranking questions in Bing's People Also Ask
1906.08593
Introduction </s> Mod elling Ġthe Ġrelationship Ġbetween Ġsequences Ġis Ġextremely Ġsignificant Ġin Ġmost Ġretrieval Ġor Ġclassification Ġproblems Ġinvolving Ġtwo Ġsequences . ĠTrad itionally , Ġin ĠSi ames e Ġnetworks , ĠHad am ard Ġproduct Ġor Ġconc aten ation Ġhave Ġbeen Ġused Ġto Ġfuse Ġtwo Ġvector Ġrepresentations Ġof Ġtwo Ġinput Ġsequences Ġto Ġform Ġa Ġfinal Ġrepresentation Ġfor Ġtasks Ġlike Ġsemantic Ġsimilarity , Ġpassage Ġretrieval . ĠThis Ġrepresentation , Ġsubsequently , Ġhas Ġbeen Ġused Ġto Ġcompute Ġsimilarity Ġscores Ġwhich Ġhas Ġbeen Ġused Ġin Ġa Ġvariety Ġof Ġtraining Ġobjectives Ġlike Ġmargin Ġloss Ġfor Ġranking Ġor Ġcross - ent ropy Ġerror Ġin Ġclassification . </s> We Ġhave Ġalso Ġwitnessed Ġword Ġor Ġphrase Ġlevel Ġsimilarity Ġto Ġcreate Ġalignment Ġmat rices Ġbetween Ġtwo Ġsequences ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠThese Ġalignment Ġmat rices Ġhas Ġproved Ġto Ġbe Ġvery Ġuseful Ġto Ġmodel Ġthe Ġrelationship Ġbetween Ġtwo Ġword Ġrepresentations Ġas Ġwell Ġfuse Ġthe Ġrelevant Ġinformation Ġof Ġone Ġsequence Ġinto Ġanother . ĠEm pir ical Ġevid ences Ġhave Ġshown Ġthis Ġalignment Ġprocedures Ġhave Ġsignificantly Ġperformed Ġbetter Ġthen Ġsimple Ġconc aten ation Ġor Ġelement - wise Ġmultiplication , Ġespecially Ġfor Ġlong Ġsentences Ġor Ġparagraphs . </s> Att ention Ġworks Ġon Ġcreating Ġneural Ġalignment Ġmatrix Ġusing Ġlearnt Ġweights Ġwithout Ġpre - com puting Ġalignment Ġmatrix Ġand Ġusing Ġthem Ġas Ġfeatures . ĠThe Ġmain Ġobjective Ġof Ġany Ġattentive Ġor Ġalignment Ġprocess Ġis Ġto Ġlook Ġfor Ġmatching Ġwords Ġor Ġphrases Ġbetween Ġtwo Ġsequences Ġand Ġassign Ġa Ġhigh Ġweight Ġto Ġthe Ġmost Ġsimilar Ġpairs Ġand Ġvice - vers a . ĠThe Ġnotion Ġof Ġmatching Ġor Ġsimilarity Ġmaybe Ġnot Ġsemantic Ġsimilarity Ġbut Ġbased Ġon Ġwhatever Ġtask Ġwe Ġhave Ġat Ġhand . ĠFor Ġexample , Ġfor Ġa Ġtask Ġthat Ġrequires Ġcapturing Ġsemantic Ġsimilarity Ġbetween Ġtwo Ġsequences Ġlike Ġ" how Ġrich Ġis Ġtom Ġcruise " Ġand Ġ" how Ġmuch Ġwealth Ġdoes Ġtom Ġcruise Ġhave ", Ġan Ġattentive Ġmodel Ġshall Ġdiscover Ġthe Ġhigh Ġsimilarity Ġbetween Ġ" rich " Ġand Ġ" wealth y " Ġand Ġassign Ġa Ġhigh Ġweight Ġvalue Ġto Ġthe Ġpair . ĠLikewise , Ġfor Ġa Ġdifferent Ġtask Ġlike Ġquestion Ġanswering , Ġa Ġword Ġ" long " Ġin Ġa Ġquestion Ġlike Ġ" how Ġlong Ġdoes Ġit Ġtake Ġto Ġrecover Ġfrom Ġa Ġmild Ġfever " Ġmight Ġbe Ġaligned Ġwith Ġthe Ġphrase Ġ" a Ġweek " Ġfrom Ġthe Ġcandidate Ġanswer Ġ" it Ġtakes Ġalmost Ġa Ġweek Ġto Ġrecover Ġfully Ġfrom Ġa Ġfever ". ĠThus , Ġattention Ġsignificantly Ġaids Ġin Ġbetter Ġunderstanding Ġthe Ġrelevance Ġof Ġa Ġsimilar Ġuser Ġquery Ġin Ġa Ġsimilar Ġmeasurement Ġtask Ġor Ġa Ġcandidate Ġanswer Ġin Ġa Ġquestion Ġanswering Ġtask . ĠThe Ġfinal Ġprediction Ġscore Ġis Ġdependent Ġon Ġhow Ġwell Ġthe Ġrelationship Ġbetween Ġtwo Ġsequences Ġare Ġmodeled Ġand Ġestablished . </s> The Ġgeneral Ġprocess Ġof Ġmatching Ġone Ġsequence Ġwith Ġanother Ġthrough Ġattention Ġincludes Ġcomputing Ġthe Ġalignment Ġmatrix Ġcontaining Ġweight Ġvalue Ġbetween Ġevery Ġpair Ġof Ġword Ġrepresentations Ġbelonging Ġto Ġboth Ġof Ġthe Ġsequences . ĠSub sequently , Ġsoft max Ġfunction Ġis Ġapplied Ġon Ġall Ġthe Ġelements Ġof Ġone Ġof Ġthe Ġtwo Ġdimensions Ġof Ġthe Ġmatrix Ġto Ġrepresent Ġthe Ġmatching Ġprobabilities Ġof Ġall Ġthe Ġword Ġof Ġa Ġsequence Ġwith Ġrespect Ġto Ġone Ġparticular Ġword Ġin Ġthe Ġother Ġsequence . </s> Since Ġattention Ġalways Ġlooks Ġfor Ġmatching Ġword Ġrepresentations , Ġit Ġoperates Ġunder Ġthe Ġassumption Ġthat Ġthere Ġis Ġalways Ġa Ġmatch Ġto Ġbe Ġfound Ġinside Ġthe Ġsequences . ĠWe Ġprovide Ġa Ġtheoretical Ġlimitation Ġto Ġit Ġand Ġpropose Ġanother Ġtechnique Ġcalled Ġconflict Ġthat Ġlooks Ġfor Ġcontrasting Ġrelationship Ġbetween Ġwords Ġin Ġtwo Ġsequences . ĠWe Ġempir ically Ġverify Ġthat Ġour Ġproposed Ġconflict Ġmechanism Ġcombined Ġwith Ġattention Ġcan Ġoutper form Ġthe Ġperformance Ġof Ġattention Ġworking Ġsolely . </s> Related ĠWork </s> Bah dan au Ġet Ġal . ĠB IB REF 2 Ġintroduced Ġattention Ġfirst Ġin Ġneural Ġmachine Ġtranslation . ĠIt Ġused Ġa Ġfeed - forward Ġnetwork Ġover Ġaddition Ġof Ġenc oder Ġand Ġdec oder Ġstates Ġto Ġcompute Ġalignment Ġscore . ĠOur Ġwork Ġis Ġvery Ġsimilar Ġto Ġthis Ġexcept Ġwe Ġuse Ġelement Ġwise Ġdifference Ġinstead Ġof Ġaddition Ġto Ġbuild Ġour Ġconflict Ġfunction . ĠB IB REF 3 Ġcame Ġup Ġwith Ġa Ġscaled Ġdot - product Ġattention Ġin Ġtheir ĠTrans former Ġmodel Ġwhich Ġis Ġfast Ġand Ġmemory - efficient . ĠDue Ġto Ġthe Ġscaling Ġfactor , Ġit Ġdidn 't Ġhave Ġthe Ġissue Ġof Ġgrad ients Ġzero ing Ġout . ĠOn Ġthe Ġother Ġhand , ĠB IB REF 4 Ġhas Ġexperimented Ġwith Ġglobal Ġand Ġlocal Ġattention Ġbased Ġon Ġthe Ġhow Ġmany Ġhidden Ġstates Ġthe Ġattention Ġfunction Ġtakes Ġinto Ġaccount . ĠTheir Ġexperiments Ġhave Ġrev olved Ġaround Ġthree Ġattention Ġfunctions Ġ- Ġdot , Ġconc at Ġand Ġgeneral . ĠTheir Ġfindings Ġinclude Ġthat Ġdot Ġproduct Ġworks Ġbest Ġfor Ġglobal Ġattention . ĠOur Ġwork Ġalso Ġbelongs Ġto Ġthe Ġglobal Ġattention Ġfamily Ġas Ġwe Ġconsider Ġall Ġthe Ġhidden Ġstates Ġof Ġthe Ġsequence . </s> Att ention Ġhas Ġbeen Ġwidely Ġused Ġin Ġpair - class ification Ġproblems Ġlike Ġnatural Ġlanguage Ġinference . ĠWang Ġet Ġal . ĠB IB REF 5 Ġintroduced ĠB IM PM Ġwhich Ġmatched Ġone Ġsequence Ġwith Ġanother Ġin Ġfour Ġdifferent Ġfashion Ġbut Ġone Ġsingle Ġmatching Ġfunction Ġwhich Ġthey Ġused Ġas Ġcos ine . ĠLiu Ġet Ġal . ĠB IB REF 6 Ġproposed ĠSAN Ġfor Ġlanguage Ġinference Ġwhich Ġalso Ġused Ġdot - product Ġattention Ġbetween Ġthe Ġsequences . </s> Sum mar izing , Ġattention Ġhas Ġhelped Ġin Ġachieving Ġstate - of - the - art Ġresults Ġin ĠNL I Ġand ĠQ A . ĠPrior Ġwork Ġin Ġattention Ġhas Ġbeen Ġmostly Ġin Ġsimilarity Ġbased Ġapproaches Ġwhile Ġour Ġwork Ġfocuses Ġon Ġnon - match ing Ġsequences . </s> How Ġattention Ġworks </s> Let Ġus Ġconsider Ġthat Ġwe Ġhave Ġtwo Ġsequences ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġeach Ġwith ĠM Ġand ĠN Ġwords Ġrespectively . ĠThe Ġobjective Ġof Ġattention Ġis Ġtwo - fold : Ġcompute Ġalignment Ġscores Ġ( or Ġweight ) Ġbetween Ġevery Ġword Ġrepresentation Ġpairs Ġfrom ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġand Ġfuse Ġthe Ġmatching Ġinformation Ġof ĠIN LINE FORM 4 Ġwith ĠIN LINE FORM 5 Ġthus Ġcomputing Ġa Ġnew Ġrepresentation Ġof ĠIN LINE FORM 6 Ġconditioned Ġon ĠIN LINE FORM 7 Ġ. </s> The Ġword Ġrepresentations Ġthat Ġattention Ġoperates Ġon Ġcan Ġbe Ġeither Ġembed d ings Ġlike ĠGl o Ve Ġor Ġhidden Ġstates Ġfrom Ġany Ġrecurrent Ġneural Ġnetwork . ĠWe Ġdenote Ġthese Ġrepresentations Ġas Ġu Ġ= ĠIN LINE FORM 0 Ġand Ġv Ġ= ĠIN LINE FORM 1 Ġ. ĠWe Ġprovide Ġa Ġmathematical Ġworking Ġof Ġhow Ġa Ġgeneral Ġattention Ġmechanism Ġworks Ġbetween Ġtwo Ġsequences , Ġfollowed Ġby Ġa Ġexplanation Ġin Ġwords : ĠDIS PLAY FORM 0 </s> Ex plan ation : ĠBoth Ġare Ġsequences Ġare Ġnon - lin early Ġprojected Ġinto Ġtwo Ġdifferent Ġspaces Ġ( eq n . 1 ) Ġand Ġeach Ġword Ġrepresentation Ġin ĠIN LINE FORM 0 Ġis Ġmatched Ġwith Ġthat Ġin ĠIN LINE FORM 1 Ġby Ġcomputing Ġa Ġdot - product Ġ( eq n . 2 ). ĠIN LINE FORM 2 Ġis Ġa ĠM ĠX ĠN Ġmatrix Ġthat Ġstores Ġthe Ġalignment Ġscores Ġbetween Ġword ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ( eq n . 2 ). ĠSince , Ġthe Ġscores Ġare Ġnot Ġnormalized , Ġa Ġsoft max Ġfunction Ġis Ġapplied Ġon Ġeach Ġrow Ġto Ġconvert Ġthem Ġto Ġprobabilities Ġ( eq n . Ġ3 ). ĠThus , Ġeach Ġrow Ġcontains Ġrelative Ġimportance Ġof Ġwords Ġin ĠIN LINE FORM 5 Ġto Ġa Ġparticular Ġword ĠIN LINE FORM 6 Ġ. ĠWeight ed Ġsum Ġof ĠIN LINE FORM 7 Ġis Ġtaken Ġ( eq n . Ġ4 ) Ġand Ġfused Ġwith Ġthe Ġword Ġrepresentation ĠIN LINE FORM 8 Ġusing Ġconc aten ation Ġ( eq n . 5 ). </s> Lim its Ġof Ġusing Ġonly ĠAttention </s> Att ention Ġoperates Ġby Ġusing Ġdot Ġproduct Ġor Ġsometimes Ġaddition Ġfollowed Ġby Ġlinear Ġprojection Ġto Ġa Ġscal ar Ġwhich Ġmodels Ġthe Ġsimilarity Ġbetween Ġtwo Ġvectors . ĠSub sequently , Ġsoft max Ġis Ġapplied Ġwhich Ġgives Ġhigh Ġprobabilities Ġto Ġmost Ġmatching Ġword Ġrepresentations . ĠThis Ġassumes Ġthat Ġthere Ġis Ġsome Ġhighly Ġmatched Ġword Ġpairs Ġalready Ġexisting Ġand Ġhigh Ġscores Ġwill Ġbe Ġassigned Ġto Ġthem . ĠGiven Ġa Ġvector ĠIN LINE FORM 0 Ġ= ( ĠIN LINE FORM 1 Ġ, ... , ĠIN LINE FORM 2 Ġ) Ġon Ġwhich Ġsoft max Ġfunction Ġis Ġapplied , Ġeach ĠIN LINE FORM 3 ĠIN LINE FORM 4 Ġ( 0 , Ġ1 ). ĠIt Ġis Ġobservable Ġthat Ġthe Ġaverage Ġvalue Ġof ĠIN LINE FORM 5 Ġis Ġalways ĠIN LINE FORM 6 Ġ. ĠIn Ġother Ġwords , Ġit Ġis Ġimpossible Ġto Ġproduce Ġa Ġvector Ġhaving Ġall ĠIN LINE FORM 7 Ġ< ĠIN LINE FORM 8 Ġwhen Ġtwo Ġsequences Ġhave Ġno Ġmatching Ġat Ġall . </s> In Ġcases , Ġwhere Ġone Ġor Ġmore Ġword Ġpairs Ġfrom Ġtwo Ġdifferent Ġsequences Ġare Ġhighly Ġdiss imilar , Ġit Ġis Ġimpossible Ġto Ġassign Ġa Ġvery Ġlow Ġprobability Ġto Ġit Ġwithout Ġincreasing Ġthe Ġprobability Ġof Ġsome Ġother Ġpair Ġsomewhere Ġelse Ġsince ĠIN LINE FORM 0 Ġ= Ġ1 . </s> For Ġexample , Ġwhen Ġwe Ġconsider Ġtwo Ġsequences Ġ" height Ġof Ġtom Ġcruise " Ġand Ġ" age Ġof Ġsun ", Ġwhile Ġcomputing Ġthe Ġattention Ġweights Ġbetween Ġthe Ġword Ġ" height " Ġand Ġall Ġthe Ġwords Ġin Ġthe Ġsecond Ġsequence Ġit Ġcan Ġbe Ġobserved Ġthat Ġtheir Ġno Ġmatching Ġword Ġin Ġthe Ġlatter . ĠIn Ġthis Ġcase , Ġa Ġstandard Ġdot - product Ġbased Ġattention Ġwith Ġsoft max Ġwon 't Ġbe Ġable Ġto Ġproduce Ġweights Ġwhich Ġis Ġbelow Ġ0 . 33 Ġ(= 1 / 3 ) Ġfor Ġall Ġthe Ġwords Ġin Ġthe Ġsecond Ġsequence Ġwith Ġrespect Ġto Ġthe Ġword Ġ" height " Ġin Ġthe Ġfirst Ġsequence . </s> Conf lict Ġmodel </s> We Ġpropose Ġa Ġdifferent Ġmechanism Ġthat Ġdoes Ġthe Ġopposite Ġof Ġwhat Ġattention Ġdoes Ġthat Ġis Ġcomputing Ġhow Ġmuch Ġtwo Ġsequences Ġrep el Ġeach Ġother . ĠThis Ġworks Ġvery Ġsimilar Ġto Ġhow Ġattention Ġworks Ġbut Ġin versely . </s> We Ġdemonstrate Ġa Ġgeneral Ġmodel Ġbut Ġwe Ġalso Ġrealize Ġthat Ġthere Ġcan Ġbe Ġother Ġvariants Ġof Ġit Ġwhich Ġmay Ġbe Ġworked Ġout Ġto Ġperform Ġbetter . ĠOur Ġapproach Ġconsists Ġof Ġusing Ġelement Ġwise Ġdifference Ġbetween Ġtwo Ġvectors Ġfollowed Ġby Ġa Ġlinear Ġtransformation Ġto Ġproduce Ġa Ġscal ar Ġweight . ĠThe Ġremaining Ġof Ġthe Ġprocess Ġacts Ġsimilar Ġto Ġhow Ġattention Ġworks . ĠMat hematically , Ġwe Ġcan Ġexpress Ġit Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 ĠIN LINE FORM 1 ĠIN LINE FORM 2 Ġis Ġa Ġparameter Ġthat Ġwe Ġintroduce Ġto Ġprovide Ġa Ġweight Ġfor Ġthe Ġpair . ĠThe Ġtwo Ġword Ġrepresentations ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġprojected Ġto Ġa Ġspace Ġwhere Ġtheir Ġelement Ġwise Ġdifference Ġcan Ġbe Ġused Ġto Ġmodel Ġtheir Ġdiss imilar ity Ġand Ġsoft max Ġapplied Ġon Ġthem Ġcan Ġproduce Ġhigh Ġprobability Ġto Ġmore Ġdiss imilar Ġword Ġpairs . </s> It Ġis Ġgood Ġto Ġnote Ġthat Ġconflict Ġsuffers Ġfrom Ġthe Ġsame Ġlimitation Ġthat Ġattention Ġsuffers Ġfrom . ĠThis Ġis Ġwhen Ġa Ġpair Ġof Ġsentences Ġare Ġhighly Ġmatching Ġespecially Ġwith Ġmultiple Ġassociations . ĠBut Ġwhen Ġthe Ġtwo Ġmethods Ġwork Ġtogether , Ġeach Ġcompens ates Ġfor Ġthe Ġother 's Ġshortcomings . </s> Comb ination Ġof Ġattention Ġand Ġconflict </s> We Ġused Ġtwo Ġweighted Ġrepresentations Ġof ĠIN LINE FORM 0 Ġusing Ġweights Ġof Ġattention Ġand Ġconflict Ġas Ġcomputed Ġin ĠE q n . Ġ( 4 ) Ġand Ġ( 8 ) Ġrespectively . ĠOur Ġfinal Ġrepresentation Ġof Ġa Ġword Ġrepresentation ĠIN LINE FORM 1 Ġconditioned Ġon ĠIN LINE FORM 2 Ġcan Ġbe Ġexpressed Ġas : ĠDIS PLAY FORM 0 </s> where ĠA Ġand ĠC Ġdenote Ġthat Ġthey Ġare Ġfrom Ġattention Ġand Ġconflict Ġmodels Ġrespectively . </s> Rel ation Ġto ĠMulti - Head Ġattention </s> Multi - head Ġattention , Ġas Ġintroduced Ġin ĠB IB REF 3 Ġ, Ġcomp utes Ġmultiple Ġidentical Ġattention Ġmechanism Ġparalle lly Ġon Ġmultiple Ġlinear Ġprojections Ġof Ġsame Ġinputs . ĠThe Ġparameters Ġof Ġeach Ġattention Ġand Ġprojections Ġare Ġdifferent Ġin Ġeach Ġhead . ĠFinally , Ġthey Ġconc aten ate Ġall Ġthe Ġattent ions Ġwhich Ġis Ġsimilar Ġto Ġhow Ġwe Ġconc aten ate Ġconflict Ġand Ġattention . ĠHowever , Ġthey Ġuse Ġdot - product Ġto Ġcompute Ġeach Ġof Ġthe Ġattention . </s> Our Ġcombined Ġmodel Ġthat Ġcontains Ġboth Ġattention Ġand Ġconflict Ġcan Ġbe Ġthought Ġof Ġas Ġa Ġ2 - head Ġattention Ġmodel Ġbut Ġboth Ġheads Ġare Ġdifferent . ĠOur Ġconflict Ġhead Ġexplicitly Ġcaptures Ġdifference Ġbetween Ġthe Ġinputs . </s> Visual izing Ġattention Ġand Ġconflict </s> We Ġobserve Ġhow Ġour Ġconflict Ġmodel Ġlearns Ġthe Ġdiss imilar ities Ġbetween Ġword Ġrepresentations . ĠWe Ġachieve Ġthat Ġby Ġvisual izing Ġthe Ġheat map Ġof Ġthe Ġweight Ġmatrix ĠIN LINE FORM 0 Ġfor Ġboth Ġattention Ġand Ġconflict Ġfrom Ġeq ns . Ġ( 3 ) Ġand Ġ( 8 ). ĠWhile Ġattention Ġsuccessfully Ġlearns Ġthe Ġalign ments , Ġconflict Ġmatrix Ġalso Ġshows Ġthat Ġour Ġapproach Ġmodels Ġthe Ġcontradict ing Ġassociations Ġlike Ġ" animal " Ġand Ġ" lake " Ġor Ġ" a ust ral ia " Ġand Ġ" world ". ĠThese Ġtwo Ġassociations Ġare Ġthe Ġunique Ġpairs Ġwhich Ġare Ġinstrumental Ġin Ġdetermining Ġthat Ġthe Ġtwo Ġqueries Ġare Ġnot Ġsimilar . </s> The Ġmodel </s> We Ġcreate Ġtwo Ġmodels Ġboth Ġof Ġwhich Ġconstitutes Ġof Ġthree Ġmain Ġparts : Ġenc oder , Ġinteraction Ġand Ġclass ifier Ġand Ġtake Ġtwo Ġsequences Ġas Ġinput . ĠExcept Ġinteraction , Ġall Ġthe Ġother Ġparts Ġare Ġexactly Ġidentical Ġbetween Ġthe Ġtwo Ġmodels . ĠThe Ġenc oder Ġis Ġshared Ġamong Ġthe Ġsequences Ġsimply Ġuses Ġtwo Ġstacked ĠGR U Ġlayers . ĠThe Ġinteraction Ġpart Ġconsists Ġof Ġonly Ġattention Ġfor Ġone Ġmodel Ġwhile Ġfor Ġthe Ġanother Ġone Ġit Ġconsists Ġof Ġattention Ġand Ġconflict Ġcombined Ġas Ġshown Ġin Ġ( eq n . 11 ) Ġ. ĠThe Ġclass ifier Ġpart Ġis Ġsimply Ġstacked Ġfully - connected Ġlayers . ĠFigure Ġ3 Ġshows Ġa Ġblock Ġdiagram Ġof Ġhow Ġour Ġmodel Ġlooks Ġlike . </s> Task Ġ1 : ĠQu ora ĠDupl icate ĠQuestion ĠPair ĠDetection </s> The Ġdataset Ġincludes Ġpairs Ġof Ġquestions Ġlabelled Ġas Ġ1 Ġor Ġ0 Ġdepending Ġon Ġwhether Ġa Ġpair Ġis Ġduplicate Ġor Ġnot Ġrespectively . ĠThis Ġis Ġa Ġpopular Ġpair - level Ġclassification Ġtask Ġon Ġwhich Ġextensive Ġwork Ġhas Ġalready Ġbeen Ġdone Ġbefore Ġlike ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠFor Ġthis Ġtask , Ġwe Ġmake Ġthe Ġoutput Ġlayer Ġof Ġour Ġmodel Ġto Ġpredict Ġtwo Ġprobabilities Ġfor Ġnon - du pl icate Ġand Ġduplicate . ĠWe Ġsample Ġthe Ġdata Ġfrom Ġthe Ġoriginal Ġdataset Ġso Ġthat Ġit Ġcontains Ġequal Ġpositive Ġand Ġnegative Ġclasses . ĠOriginal Ġdataset Ġhas Ġsome Ġclass Ġimbalance Ġbut Ġfor Ġsake Ġsimplicity Ġwe Ġdon 't Ġconsider Ġit . ĠThe Ġfinal Ġdata Ġthat Ġwe Ġuse Ġhas Ġroughly Ġ400 , 000 Ġquestion Ġpairs Ġand Ġwe Ġsplit Ġthis Ġdata Ġinto Ġtrain Ġand Ġtest Ġusing Ġ8 : 2 Ġratio . </s> We Ġtrain Ġall Ġour Ġmodels Ġfor Ġroughly Ġ2 Ġepoch s Ġwith Ġa Ġbatch Ġsize Ġof Ġ64 . ĠWe Ġuse Ġa Ġhidden Ġdimension Ġof Ġ150 Ġthroughout Ġthe Ġmodel . ĠThe Ġembed ding Ġlayer Ġuses ĠEL MO ĠB IB REF 9 Ġwhich Ġhas Ġproven Ġto Ġbe Ġvery Ġuseful Ġin Ġvarious Ġdownstream Ġlanguage Ġunderstanding Ġtasks . ĠOur ĠFC Ġlayers Ġconsists Ġof Ġfour Ġdense Ġlayers Ġwith ĠIN LINE FORM 0 Ġactivation Ġafter Ġeach Ġlayer . ĠThe Ġdrop out Ġrate Ġis Ġkept Ġas Ġ0 . 2 Ġfor Ġevery Ġrecurrent Ġand ĠFC Ġlinear Ġlayers . ĠWe Ġuse ĠAdam Ġoptim izer Ġin Ġour Ġexperiment Ġwith Ġe ps ilon = 1 e - 8 , Ġbeta = 0 . 9 Ġand Ġlearning Ġrate = 1 e - 3 . </s> Task Ġ2 : ĠRanking Ġquestions Ġin ĠBing 's ĠPeople ĠAlso ĠAsk </s> People ĠAlso ĠAsk Ġis Ġa Ġfeature Ġin ĠBing Ġsearch Ġresult Ġpage Ġwhere Ġrelated Ġquestions Ġare Ġrecommended Ġto Ġthe Ġuser . ĠUser Ġmay Ġclick Ġon Ġa Ġquestion Ġto Ġview Ġthe Ġanswer . ĠClick ing Ġis Ġa Ġpositive Ġfeedback Ġthat Ġshows Ġuser 's Ġinterest Ġin Ġthe Ġquestion . ĠWe Ġuse Ġthis Ġclick Ġlogs Ġto Ġbuild Ġa Ġquestion Ġclass ifier Ġusing Ġthe Ġsame Ġmodel Ġin ĠFigure Ġ3 . ĠThe Ġproblem Ġstatement Ġis Ġvery Ġsimilar Ġto ĠB IB REF 10 Ġwhere Ġthey Ġuse Ġlog istic Ġregression Ġto Ġpredict Ġwhether Ġan Ġuser Ġwould Ġclick Ġon Ġad . ĠOur Ġgoal Ġis Ġto Ġclassify Ġif Ġa Ġquestion Ġis Ġpotential Ġhigh - click Ġquestion Ġor Ġnot Ġfor Ġa Ġgiven Ġquery . ĠFor Ġthis , Ġwe Ġfirst Ġcreate Ġa Ġlabelled Ġdata Ġset Ġusing Ġthe Ġclick Ġlogs Ġwhere Ġany Ġquestion Ġhaving ĠCTR Ġlower Ġthan Ġ0 . 3 Ġis Ġlabelled Ġas Ġ0 Ġand Ġa Ġquestion Ġhaving ĠCTR Ġmore Ġthan Ġ0 . 7 Ġas Ġ1 . </s> Our Ġfinal Ġdata Ġresembles Ġthat Ġof Ġa Ġpair - level Ġclass ifier , Ġas Ġin ĠTask Ġ1 , Ġwhere Ġuser Ġquery Ġand Ġcandidate Ġquestions Ġare Ġinput . ĠWith Ġthese Ġdata Ġset , Ġwe Ġtrain Ġa Ġbinary Ġclass ifier Ġto Ġdetect Ġhigh - click Ġand Ġlow - click Ġquestions . </s> Quant itative ĠAnalysis </s> For Ġboth Ġtasks , Ġwe Ġcompute Ġclassification Ġaccuracy Ġusing Ġthree Ġmodel Ġvariants Ġand Ġreport Ġthe Ġresults Ġin ĠTable Ġ1 Ġand ĠTable Ġ2 . ĠWe Ġobserve Ġthat Ġmodel Ġwith Ġboth Ġattention Ġand Ġconflict Ġcombined Ġgives Ġthe Ġbest Ġresults . </s> We Ġalso Ġshow Ġthe Ġtraining Ġloss Ġcurve Ġfor Ġboth Ġthe Ġmodels Ġhaving Ġattention Ġand Ġattention Ġcombined Ġwith Ġconflict Ġrespectively . ĠFigure Ġ4 Ġand Ġ5 Ġshows Ġthese Ġcurves Ġfor ĠTask Ġ1 Ġand ĠTask Ġ2 Ġrespectively . ĠThe Ġcurves Ġare Ġsmoot hed Ġusing Ġmoving Ġaverage Ġhaving Ġan Ġwindow Ġsize Ġof Ġ8 . ĠWe Ġnotice Ġthat Ġthe Ġconflict Ġmodel Ġhas Ġmuch Ġst eeper Ġslope Ġand Ġconver ges Ġto Ġa Ġmuch Ġbetter Ġmin ima Ġin Ġboth Ġthe Ġtasks . ĠIt Ġcan Ġalso Ġbe Ġnoticed Ġthat Ġin Ġthe Ġtraining Ġprocedure Ġfor Ġthe Ġmodel Ġwhich Ġhas Ġboth Ġattention Ġand Ġconflict , Ġthe Ġupdates Ġare Ġmuch Ġsmoother . </s> Qual itative ĠComparison </s> We Ġalso Ġshow Ġqualitative Ġresults Ġwhere Ġwe Ġcan Ġobserve Ġthat Ġour Ġmodel Ġwith Ġattention Ġand Ġconflict Ġcombined Ġdoes Ġbetter Ġon Ġcases Ġwhere Ġpairs Ġare Ġnon - du pl icate Ġand Ġhas Ġvery Ġsmall Ġdifference . ĠWe Ġhave Ġobserved Ġthat Ġthe Ġconflict Ġmodel Ġis Ġvery Ġsensitive Ġto Ġeven Ġminor Ġdifferences Ġand Ġcompens ates Ġin Ġsuch Ġcases Ġwhere Ġattention Ġposes Ġhigh Ġbias Ġtowards Ġsimilarities Ġalready Ġthere Ġin Ġthe Ġsequences . </s> Sequ ence Ġ1 : ĠWhat Ġare Ġthe Ġbest Ġways Ġto Ġlearn ĠFrench Ġ? </s> Sequ ence Ġ2 : ĠHow Ġdo ĠI Ġlearn Ġfrench Ġgenders Ġ? </s> Att ention Ġonly : Ġ1 </s> Att ention + Conf lict : Ġ0 </s> Ground ĠTruth : Ġ0 </s> Sequ ence Ġ1 : ĠHow Ġdo ĠI Ġprevent Ġbreast Ġcancer Ġ? </s> Sequ ence Ġ2 : ĠIs Ġbreast Ġcancer Ġprevent able Ġ? </s> Att ention Ġonly : Ġ1 </s> Att ention + Conf lict : Ġ0 </s> Ground ĠTruth : Ġ0 </s> We Ġprovide Ġtwo Ġexamples Ġwith Ġpredictions Ġfrom Ġthe Ġmodels Ġwith Ġonly Ġattention Ġand Ġcombination Ġof Ġattention Ġand Ġconflict . ĠEach Ġexample Ġis Ġaccompanied Ġby Ġthe Ġground Ġtruth Ġin Ġour Ġdata . </s> Analy zing Ġthe Ġgains </s> We Ġanalyzed Ġthe Ġgains Ġin ĠTask Ġ1 Ġwhich Ġwe Ġget Ġfrom Ġthe Ġattention - conf lict Ġmodel Ġin Ġorder Ġto Ġensure Ġthat Ġthey Ġare Ġnot Ġdue Ġto Ġrandom ness Ġin Ġweight Ġinitialization Ġor Ġsimply Ġadditional Ġparameters . ĠWe Ġparticularly Ġfocused Ġon Ġthe Ġexamples Ġwhich Ġwere Ġincorrectly Ġmarked Ġin Ġattention Ġmodel Ġbut Ġcorrectly Ġin Ġattention - conf lict Ġmodel . ĠWe Ġsaw Ġthat Ġ70 % Ġof Ġthose Ġcases Ġare Ġthe Ġones Ġwhere Ġthe Ġpair Ġwas Ġincorrectly Ġmarked Ġas Ġduplicate Ġin Ġthe Ġprevious Ġmodel Ġbut Ġour Ġcombined Ġmodel Ġcorrectly Ġmarked Ġthem Ġas Ġnon - du pl icate . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġhighlighted Ġthe Ġlimits Ġof Ġattention Ġespecially Ġin Ġcases Ġwhere Ġtwo Ġsequences Ġhave Ġa Ġcontradict ing Ġrelationship Ġbased Ġon Ġthe Ġtask Ġit Ġperforms . ĠTo Ġalleviate Ġthis Ġproblem Ġand Ġfurther Ġimprove Ġthe Ġperformance , Ġwe Ġpropose Ġa Ġconflict Ġmechanism Ġthat Ġtries Ġto Ġcapture Ġhow Ġtwo Ġsequences Ġrep el Ġeach Ġother . ĠThis Ġacts Ġlike Ġthe Ġinverse Ġof Ġattention Ġand , Ġempir ically , Ġwe Ġshow Ġthat Ġhow Ġconflict Ġand Ġattention Ġtogether Ġcan Ġimprove Ġthe Ġperformance . </s> Future Ġresearch Ġwork Ġshould Ġbe Ġbased Ġon Ġalternative Ġdesign Ġof Ġconflict Ġmechanism Ġusing Ġother Ġdifference Ġoperators Ġother Ġthan Ġelement Ġwise Ġdifference Ġwhich Ġwe Ġuse .
Introduction Modelling the relationship between sequences is extremely significant in most retrieval or classification problems involving two sequences. Traditionally, in Siamese networks, Hadamard product or concatenation have been used to fuse two vector representations of two input sequences to form a final representation for tasks like semantic similarity, passage retrieval. This representation, subsequently, has been used to compute similarity scores which has been used in a variety of training objectives like margin loss for ranking or cross-entropy error in classification. We have also witnessed word or phrase level similarity to create alignment matrices between two sequences BIBREF0 , BIBREF1 . These alignment matrices has proved to be very useful to model the relationship between two word representations as well fuse the relevant information of one sequence into another. Empirical evidences have shown this alignment procedures have significantly performed better then simple concatenation or element-wise multiplication, especially for long sentences or paragraphs. Attention works on creating neural alignment matrix using learnt weights without pre-computing alignment matrix and using them as features. The main objective of any attentive or alignment process is to look for matching words or phrases between two sequences and assign a high weight to the most similar pairs and vice-versa. The notion of matching or similarity maybe not semantic similarity but based on whatever task we have at hand. For example, for a task that requires capturing semantic similarity between two sequences like "how rich is tom cruise" and "how much wealth does tom cruise have", an attentive model shall discover the high similarity between "rich" and "wealthy" and assign a high weight value to the pair. Likewise, for a different task like question answering, a word "long" in a question like "how long does it take to recover from a mild fever" might be aligned with the phrase "a week" from the candidate answer "it takes almost a week to recover fully from a fever". Thus, attention significantly aids in better understanding the relevance of a similar user query in a similar measurement task or a candidate answer in a question answering task. The final prediction score is dependent on how well the relationship between two sequences are modeled and established. The general process of matching one sequence with another through attention includes computing the alignment matrix containing weight value between every pair of word representations belonging to both of the sequences. Subsequently, softmax function is applied on all the elements of one of the two dimensions of the matrix to represent the matching probabilities of all the word of a sequence with respect to one particular word in the other sequence. Since attention always looks for matching word representations, it operates under the assumption that there is always a match to be found inside the sequences. We provide a theoretical limitation to it and propose another technique called conflict that looks for contrasting relationship between words in two sequences. We empirically verify that our proposed conflict mechanism combined with attention can outperform the performance of attention working solely. Related Work Bahdanau et al. BIBREF2 introduced attention first in neural machine translation. It used a feed-forward network over addition of encoder and decoder states to compute alignment score. Our work is very similar to this except we use element wise difference instead of addition to build our conflict function. BIBREF3 came up with a scaled dot-product attention in their Transformer model which is fast and memory-efficient. Due to the scaling factor, it didn't have the issue of gradients zeroing out. On the other hand, BIBREF4 has experimented with global and local attention based on the how many hidden states the attention function takes into account. Their experiments have revolved around three attention functions - dot, concat and general. Their findings include that dot product works best for global attention. Our work also belongs to the global attention family as we consider all the hidden states of the sequence. Attention has been widely used in pair-classification problems like natural language inference. Wang et al. BIBREF5 introduced BIMPM which matched one sequence with another in four different fashion but one single matching function which they used as cosine. Liu et al. BIBREF6 proposed SAN for language inference which also used dot-product attention between the sequences. Summarizing, attention has helped in achieving state-of-the-art results in NLI and QA. Prior work in attention has been mostly in similarity based approaches while our work focuses on non-matching sequences. How attention works Let us consider that we have two sequences INLINEFORM0 and INLINEFORM1 each with M and N words respectively. The objective of attention is two-fold: compute alignment scores (or weight) between every word representation pairs from INLINEFORM2 and INLINEFORM3 and fuse the matching information of INLINEFORM4 with INLINEFORM5 thus computing a new representation of INLINEFORM6 conditioned on INLINEFORM7 . The word representations that attention operates on can be either embeddings like GloVe or hidden states from any recurrent neural network. We denote these representations as u = INLINEFORM0 and v = INLINEFORM1 . We provide a mathematical working of how a general attention mechanism works between two sequences, followed by a explanation in words: DISPLAYFORM0 Explanation: Both are sequences are non-linearly projected into two different spaces (eqn.1) and each word representation in INLINEFORM0 is matched with that in INLINEFORM1 by computing a dot-product (eqn.2). INLINEFORM2 is a M X N matrix that stores the alignment scores between word INLINEFORM3 and INLINEFORM4 (eqn.2). Since, the scores are not normalized, a softmax function is applied on each row to convert them to probabilities (eqn. 3). Thus, each row contains relative importance of words in INLINEFORM5 to a particular word INLINEFORM6 . Weighted sum of INLINEFORM7 is taken (eqn. 4) and fused with the word representation INLINEFORM8 using concatenation (eqn.5). Limits of using only Attention Attention operates by using dot product or sometimes addition followed by linear projection to a scalar which models the similarity between two vectors. Subsequently, softmax is applied which gives high probabilities to most matching word representations. This assumes that there is some highly matched word pairs already existing and high scores will be assigned to them. Given a vector INLINEFORM0 =( INLINEFORM1 ,..., INLINEFORM2 ) on which softmax function is applied, each INLINEFORM3 INLINEFORM4 (0, 1). It is observable that the average value of INLINEFORM5 is always INLINEFORM6 . In other words, it is impossible to produce a vector having all INLINEFORM7 < INLINEFORM8 when two sequences have no matching at all. In cases, where one or more word pairs from two different sequences are highly dissimilar, it is impossible to assign a very low probability to it without increasing the probability of some other pair somewhere else since INLINEFORM0 = 1. For example, when we consider two sequences "height of tom cruise" and "age of sun", while computing the attention weights between the word "height" and all the words in the second sequence it can be observed that their no matching word in the latter. In this case, a standard dot-product based attention with softmax won't be able to produce weights which is below 0.33 (=1/3) for all the words in the second sequence with respect to the word "height" in the first sequence. Conflict model We propose a different mechanism that does the opposite of what attention does that is computing how much two sequences repel each other. This works very similar to how attention works but inversely. We demonstrate a general model but we also realize that there can be other variants of it which may be worked out to perform better. Our approach consists of using element wise difference between two vectors followed by a linear transformation to produce a scalar weight. The remaining of the process acts similar to how attention works. Mathematically, we can express it as: DISPLAYFORM0 where INLINEFORM0 INLINEFORM1 INLINEFORM2 is a parameter that we introduce to provide a weight for the pair. The two word representations INLINEFORM3 and INLINEFORM4 are projected to a space where their element wise difference can be used to model their dissimilarity and softmax applied on them can produce high probability to more dissimilar word pairs. It is good to note that conflict suffers from the same limitation that attention suffers from. This is when a pair of sentences are highly matching especially with multiple associations. But when the two methods work together, each compensates for the other's shortcomings. Combination of attention and conflict We used two weighted representations of INLINEFORM0 using weights of attention and conflict as computed in Eqn. (4) and (8) respectively. Our final representation of a word representation INLINEFORM1 conditioned on INLINEFORM2 can be expressed as: DISPLAYFORM0 where A and C denote that they are from attention and conflict models respectively. Relation to Multi-Head attention Multi-head attention, as introduced in BIBREF3 , computes multiple identical attention mechanism parallelly on multiple linear projections of same inputs. The parameters of each attention and projections are different in each head. Finally, they concatenate all the attentions which is similar to how we concatenate conflict and attention. However, they use dot-product to compute each of the attention. Our combined model that contains both attention and conflict can be thought of as a 2-head attention model but both heads are different. Our conflict head explicitly captures difference between the inputs. Visualizing attention and conflict We observe how our conflict model learns the dissimilarities between word representations. We achieve that by visualizing the heatmap of the weight matrix INLINEFORM0 for both attention and conflict from eqns. (3) and (8). While attention successfully learns the alignments, conflict matrix also shows that our approach models the contradicting associations like "animal" and "lake" or "australia" and "world". These two associations are the unique pairs which are instrumental in determining that the two queries are not similar. The model We create two models both of which constitutes of three main parts: encoder, interaction and classifier and take two sequences as input. Except interaction, all the other parts are exactly identical between the two models. The encoder is shared among the sequences simply uses two stacked GRU layers. The interaction part consists of only attention for one model while for the another one it consists of attention and conflict combined as shown in (eqn.11) . The classifier part is simply stacked fully-connected layers. Figure 3 shows a block diagram of how our model looks like. Task 1: Quora Duplicate Question Pair Detection The dataset includes pairs of questions labelled as 1 or 0 depending on whether a pair is duplicate or not respectively. This is a popular pair-level classification task on which extensive work has already been done before like BIBREF7 , BIBREF8 . For this task, we make the output layer of our model to predict two probabilities for non-duplicate and duplicate. We sample the data from the original dataset so that it contains equal positive and negative classes. Original dataset has some class imbalance but for sake simplicity we don't consider it. The final data that we use has roughly 400,000 question pairs and we split this data into train and test using 8:2 ratio. We train all our models for roughly 2 epochs with a batch size of 64. We use a hidden dimension of 150 throughout the model. The embedding layer uses ELMO BIBREF9 which has proven to be very useful in various downstream language understanding tasks. Our FC layers consists of four dense layers with INLINEFORM0 activation after each layer. The dropout rate is kept as 0.2 for every recurrent and FC linear layers. We use Adam optimizer in our experiment with epsilon=1e-8, beta=0.9 and learning rate=1e-3. Task 2: Ranking questions in Bing's People Also Ask People Also Ask is a feature in Bing search result page where related questions are recommended to the user. User may click on a question to view the answer. Clicking is a positive feedback that shows user's interest in the question. We use this click logs to build a question classifier using the same model in Figure 3. The problem statement is very similar to BIBREF10 where they use logistic regression to predict whether an user would click on ad. Our goal is to classify if a question is potential high-click question or not for a given query. For this, we first create a labelled data set using the click logs where any question having CTR lower than 0.3 is labelled as 0 and a question having CTR more than 0.7 as 1. Our final data resembles that of a pair-level classifier, as in Task 1, where user query and candidate questions are input. With these data set, we train a binary classifier to detect high-click and low-click questions. Quantitative Analysis For both tasks, we compute classification accuracy using three model variants and report the results in Table 1 and Table 2. We observe that model with both attention and conflict combined gives the best results. We also show the training loss curve for both the models having attention and attention combined with conflict respectively. Figure 4 and 5 shows these curves for Task 1 and Task 2 respectively. The curves are smoothed using moving average having an window size of 8. We notice that the conflict model has much steeper slope and converges to a much better minima in both the tasks. It can also be noticed that in the training procedure for the model which has both attention and conflict, the updates are much smoother. Qualitative Comparison We also show qualitative results where we can observe that our model with attention and conflict combined does better on cases where pairs are non-duplicate and has very small difference. We have observed that the conflict model is very sensitive to even minor differences and compensates in such cases where attention poses high bias towards similarities already there in the sequences. Sequence 1: What are the best ways to learn French ? Sequence 2: How do I learn french genders ? Attention only: 1 Attention+Conflict: 0 Ground Truth: 0 Sequence 1: How do I prevent breast cancer ? Sequence 2: Is breast cancer preventable ? Attention only: 1 Attention+Conflict: 0 Ground Truth: 0 We provide two examples with predictions from the models with only attention and combination of attention and conflict. Each example is accompanied by the ground truth in our data. Analyzing the gains We analyzed the gains in Task 1 which we get from the attention-conflict model in order to ensure that they are not due to randomness in weight initialization or simply additional parameters. We particularly focused on the examples which were incorrectly marked in attention model but correctly in attention-conflict model. We saw that 70% of those cases are the ones where the pair was incorrectly marked as duplicate in the previous model but our combined model correctly marked them as non-duplicate. Conclusion In this work, we highlighted the limits of attention especially in cases where two sequences have a contradicting relationship based on the task it performs. To alleviate this problem and further improve the performance, we propose a conflict mechanism that tries to capture how two sequences repel each other. This acts like the inverse of attention and, empirically, we show that how conflict and attention together can improve the performance. Future research work should be based on alternative design of conflict mechanism using other difference operators other than element wise difference which we use.
On which tasks do they test their conflict method?
b14217978ad9c3c9b6b1ce393b1b5c6e7f49ecab
On Ġwhich Ġtasks Ġdo Ġthey Ġtest Ġtheir Ġconflict Ġmethod ?
Task Ġ1 : ĠQu ora ĠDupl icate ĠQuestion ĠPair ĠDetection , ĠTask Ġ2 : ĠRanking Ġquestions
<s> On Ġwhich Ġtasks Ġdo Ġthey Ġtest Ġtheir Ġconflict Ġmethod ? </s> Introduction </s> Mod elling Ġthe Ġrelationship Ġbetween Ġsequences Ġis Ġextremely Ġsignificant Ġin Ġmost Ġretrieval Ġor Ġclassification Ġproblems Ġinvolving Ġtwo Ġsequences . ĠTrad itionally , Ġin ĠSi ames e Ġnetworks , ĠHad am ard Ġproduct Ġor Ġconc aten ation Ġhave Ġbeen Ġused Ġto Ġfuse Ġtwo Ġvector Ġrepresentations Ġof Ġtwo Ġinput Ġsequences Ġto Ġform Ġa Ġfinal Ġrepresentation Ġfor Ġtasks Ġlike Ġsemantic Ġsimilarity , Ġpassage Ġretrieval . ĠThis Ġrepresentation , Ġsubsequently , Ġhas Ġbeen Ġused Ġto Ġcompute Ġsimilarity Ġscores Ġwhich Ġhas Ġbeen Ġused Ġin Ġa Ġvariety Ġof Ġtraining Ġobjectives Ġlike Ġmargin Ġloss Ġfor Ġranking Ġor Ġcross - ent ropy Ġerror Ġin Ġclassification . </s> We Ġhave Ġalso Ġwitnessed Ġword Ġor Ġphrase Ġlevel Ġsimilarity Ġto Ġcreate Ġalignment Ġmat rices Ġbetween Ġtwo Ġsequences ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠThese Ġalignment Ġmat rices Ġhas Ġproved Ġto Ġbe Ġvery Ġuseful Ġto Ġmodel Ġthe Ġrelationship Ġbetween Ġtwo Ġword Ġrepresentations Ġas Ġwell Ġfuse Ġthe Ġrelevant Ġinformation Ġof Ġone Ġsequence Ġinto Ġanother . ĠEm pir ical Ġevid ences Ġhave Ġshown Ġthis Ġalignment Ġprocedures Ġhave Ġsignificantly Ġperformed Ġbetter Ġthen Ġsimple Ġconc aten ation Ġor Ġelement - wise Ġmultiplication , Ġespecially Ġfor Ġlong Ġsentences Ġor Ġparagraphs . </s> Att ention Ġworks Ġon Ġcreating Ġneural Ġalignment Ġmatrix Ġusing Ġlearnt Ġweights Ġwithout Ġpre - com puting Ġalignment Ġmatrix Ġand Ġusing Ġthem Ġas Ġfeatures . ĠThe Ġmain Ġobjective Ġof Ġany Ġattentive Ġor Ġalignment Ġprocess Ġis Ġto Ġlook Ġfor Ġmatching Ġwords Ġor Ġphrases Ġbetween Ġtwo Ġsequences Ġand Ġassign Ġa Ġhigh Ġweight Ġto Ġthe Ġmost Ġsimilar Ġpairs Ġand Ġvice - vers a . ĠThe Ġnotion Ġof Ġmatching Ġor Ġsimilarity Ġmaybe Ġnot Ġsemantic Ġsimilarity Ġbut Ġbased Ġon Ġwhatever Ġtask Ġwe Ġhave Ġat Ġhand . ĠFor Ġexample , Ġfor Ġa Ġtask Ġthat Ġrequires Ġcapturing Ġsemantic Ġsimilarity Ġbetween Ġtwo Ġsequences Ġlike Ġ" how Ġrich Ġis Ġtom Ġcruise " Ġand Ġ" how Ġmuch Ġwealth Ġdoes Ġtom Ġcruise Ġhave ", Ġan Ġattentive Ġmodel Ġshall Ġdiscover Ġthe Ġhigh Ġsimilarity Ġbetween Ġ" rich " Ġand Ġ" wealth y " Ġand Ġassign Ġa Ġhigh Ġweight Ġvalue Ġto Ġthe Ġpair . ĠLikewise , Ġfor Ġa Ġdifferent Ġtask Ġlike Ġquestion Ġanswering , Ġa Ġword Ġ" long " Ġin Ġa Ġquestion Ġlike Ġ" how Ġlong Ġdoes Ġit Ġtake Ġto Ġrecover Ġfrom Ġa Ġmild Ġfever " Ġmight Ġbe Ġaligned Ġwith Ġthe Ġphrase Ġ" a Ġweek " Ġfrom Ġthe Ġcandidate Ġanswer Ġ" it Ġtakes Ġalmost Ġa Ġweek Ġto Ġrecover Ġfully Ġfrom Ġa Ġfever ". ĠThus , Ġattention Ġsignificantly Ġaids Ġin Ġbetter Ġunderstanding Ġthe Ġrelevance Ġof Ġa Ġsimilar Ġuser Ġquery Ġin Ġa Ġsimilar Ġmeasurement Ġtask Ġor Ġa Ġcandidate Ġanswer Ġin Ġa Ġquestion Ġanswering Ġtask . ĠThe Ġfinal Ġprediction Ġscore Ġis Ġdependent Ġon Ġhow Ġwell Ġthe Ġrelationship Ġbetween Ġtwo Ġsequences Ġare Ġmodeled Ġand Ġestablished . </s> The Ġgeneral Ġprocess Ġof Ġmatching Ġone Ġsequence Ġwith Ġanother Ġthrough Ġattention Ġincludes Ġcomputing Ġthe Ġalignment Ġmatrix Ġcontaining Ġweight Ġvalue Ġbetween Ġevery Ġpair Ġof Ġword Ġrepresentations Ġbelonging Ġto Ġboth Ġof Ġthe Ġsequences . ĠSub sequently , Ġsoft max Ġfunction Ġis Ġapplied Ġon Ġall Ġthe Ġelements Ġof Ġone Ġof Ġthe Ġtwo Ġdimensions Ġof Ġthe Ġmatrix Ġto Ġrepresent Ġthe Ġmatching Ġprobabilities Ġof Ġall Ġthe Ġword Ġof Ġa Ġsequence Ġwith Ġrespect Ġto Ġone Ġparticular Ġword Ġin Ġthe Ġother Ġsequence . </s> Since Ġattention Ġalways Ġlooks Ġfor Ġmatching Ġword Ġrepresentations , Ġit Ġoperates Ġunder Ġthe Ġassumption Ġthat Ġthere Ġis Ġalways Ġa Ġmatch Ġto Ġbe Ġfound Ġinside Ġthe Ġsequences . ĠWe Ġprovide Ġa Ġtheoretical Ġlimitation Ġto Ġit Ġand Ġpropose Ġanother Ġtechnique Ġcalled Ġconflict Ġthat Ġlooks Ġfor Ġcontrasting Ġrelationship Ġbetween Ġwords Ġin Ġtwo Ġsequences . ĠWe Ġempir ically Ġverify Ġthat Ġour Ġproposed Ġconflict Ġmechanism Ġcombined Ġwith Ġattention Ġcan Ġoutper form Ġthe Ġperformance Ġof Ġattention Ġworking Ġsolely . </s> Related ĠWork </s> Bah dan au Ġet Ġal . ĠB IB REF 2 Ġintroduced Ġattention Ġfirst Ġin Ġneural Ġmachine Ġtranslation . ĠIt Ġused Ġa Ġfeed - forward Ġnetwork Ġover Ġaddition Ġof Ġenc oder Ġand Ġdec oder Ġstates Ġto Ġcompute Ġalignment Ġscore . ĠOur Ġwork Ġis Ġvery Ġsimilar Ġto Ġthis Ġexcept Ġwe Ġuse Ġelement Ġwise Ġdifference Ġinstead Ġof Ġaddition Ġto Ġbuild Ġour Ġconflict Ġfunction . ĠB IB REF 3 Ġcame Ġup Ġwith Ġa Ġscaled Ġdot - product Ġattention Ġin Ġtheir ĠTrans former Ġmodel Ġwhich Ġis Ġfast Ġand Ġmemory - efficient . ĠDue Ġto Ġthe Ġscaling Ġfactor , Ġit Ġdidn 't Ġhave Ġthe Ġissue Ġof Ġgrad ients Ġzero ing Ġout . ĠOn Ġthe Ġother Ġhand , ĠB IB REF 4 Ġhas Ġexperimented Ġwith Ġglobal Ġand Ġlocal Ġattention Ġbased Ġon Ġthe Ġhow Ġmany Ġhidden Ġstates Ġthe Ġattention Ġfunction Ġtakes Ġinto Ġaccount . ĠTheir Ġexperiments Ġhave Ġrev olved Ġaround Ġthree Ġattention Ġfunctions Ġ- Ġdot , Ġconc at Ġand Ġgeneral . ĠTheir Ġfindings Ġinclude Ġthat Ġdot Ġproduct Ġworks Ġbest Ġfor Ġglobal Ġattention . ĠOur Ġwork Ġalso Ġbelongs Ġto Ġthe Ġglobal Ġattention Ġfamily Ġas Ġwe Ġconsider Ġall Ġthe Ġhidden Ġstates Ġof Ġthe Ġsequence . </s> Att ention Ġhas Ġbeen Ġwidely Ġused Ġin Ġpair - class ification Ġproblems Ġlike Ġnatural Ġlanguage Ġinference . ĠWang Ġet Ġal . ĠB IB REF 5 Ġintroduced ĠB IM PM Ġwhich Ġmatched Ġone Ġsequence Ġwith Ġanother Ġin Ġfour Ġdifferent Ġfashion Ġbut Ġone Ġsingle Ġmatching Ġfunction Ġwhich Ġthey Ġused Ġas Ġcos ine . ĠLiu Ġet Ġal . ĠB IB REF 6 Ġproposed ĠSAN Ġfor Ġlanguage Ġinference Ġwhich Ġalso Ġused Ġdot - product Ġattention Ġbetween Ġthe Ġsequences . </s> Sum mar izing , Ġattention Ġhas Ġhelped Ġin Ġachieving Ġstate - of - the - art Ġresults Ġin ĠNL I Ġand ĠQ A . ĠPrior Ġwork Ġin Ġattention Ġhas Ġbeen Ġmostly Ġin Ġsimilarity Ġbased Ġapproaches Ġwhile Ġour Ġwork Ġfocuses Ġon Ġnon - match ing Ġsequences . </s> How Ġattention Ġworks </s> Let Ġus Ġconsider Ġthat Ġwe Ġhave Ġtwo Ġsequences ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġeach Ġwith ĠM Ġand ĠN Ġwords Ġrespectively . ĠThe Ġobjective Ġof Ġattention Ġis Ġtwo - fold : Ġcompute Ġalignment Ġscores Ġ( or Ġweight ) Ġbetween Ġevery Ġword Ġrepresentation Ġpairs Ġfrom ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġand Ġfuse Ġthe Ġmatching Ġinformation Ġof ĠIN LINE FORM 4 Ġwith ĠIN LINE FORM 5 Ġthus Ġcomputing Ġa Ġnew Ġrepresentation Ġof ĠIN LINE FORM 6 Ġconditioned Ġon ĠIN LINE FORM 7 Ġ. </s> The Ġword Ġrepresentations Ġthat Ġattention Ġoperates Ġon Ġcan Ġbe Ġeither Ġembed d ings Ġlike ĠGl o Ve Ġor Ġhidden Ġstates Ġfrom Ġany Ġrecurrent Ġneural Ġnetwork . ĠWe Ġdenote Ġthese Ġrepresentations Ġas Ġu Ġ= ĠIN LINE FORM 0 Ġand Ġv Ġ= ĠIN LINE FORM 1 Ġ. ĠWe Ġprovide Ġa Ġmathematical Ġworking Ġof Ġhow Ġa Ġgeneral Ġattention Ġmechanism Ġworks Ġbetween Ġtwo Ġsequences , Ġfollowed Ġby Ġa Ġexplanation Ġin Ġwords : ĠDIS PLAY FORM 0 </s> Ex plan ation : ĠBoth Ġare Ġsequences Ġare Ġnon - lin early Ġprojected Ġinto Ġtwo Ġdifferent Ġspaces Ġ( eq n . 1 ) Ġand Ġeach Ġword Ġrepresentation Ġin ĠIN LINE FORM 0 Ġis Ġmatched Ġwith Ġthat Ġin ĠIN LINE FORM 1 Ġby Ġcomputing Ġa Ġdot - product Ġ( eq n . 2 ). ĠIN LINE FORM 2 Ġis Ġa ĠM ĠX ĠN Ġmatrix Ġthat Ġstores Ġthe Ġalignment Ġscores Ġbetween Ġword ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġ( eq n . 2 ). ĠSince , Ġthe Ġscores Ġare Ġnot Ġnormalized , Ġa Ġsoft max Ġfunction Ġis Ġapplied Ġon Ġeach Ġrow Ġto Ġconvert Ġthem Ġto Ġprobabilities Ġ( eq n . Ġ3 ). ĠThus , Ġeach Ġrow Ġcontains Ġrelative Ġimportance Ġof Ġwords Ġin ĠIN LINE FORM 5 Ġto Ġa Ġparticular Ġword ĠIN LINE FORM 6 Ġ. ĠWeight ed Ġsum Ġof ĠIN LINE FORM 7 Ġis Ġtaken Ġ( eq n . Ġ4 ) Ġand Ġfused Ġwith Ġthe Ġword Ġrepresentation ĠIN LINE FORM 8 Ġusing Ġconc aten ation Ġ( eq n . 5 ). </s> Lim its Ġof Ġusing Ġonly ĠAttention </s> Att ention Ġoperates Ġby Ġusing Ġdot Ġproduct Ġor Ġsometimes Ġaddition Ġfollowed Ġby Ġlinear Ġprojection Ġto Ġa Ġscal ar Ġwhich Ġmodels Ġthe Ġsimilarity Ġbetween Ġtwo Ġvectors . ĠSub sequently , Ġsoft max Ġis Ġapplied Ġwhich Ġgives Ġhigh Ġprobabilities Ġto Ġmost Ġmatching Ġword Ġrepresentations . ĠThis Ġassumes Ġthat Ġthere Ġis Ġsome Ġhighly Ġmatched Ġword Ġpairs Ġalready Ġexisting Ġand Ġhigh Ġscores Ġwill Ġbe Ġassigned Ġto Ġthem . ĠGiven Ġa Ġvector ĠIN LINE FORM 0 Ġ= ( ĠIN LINE FORM 1 Ġ, ... , ĠIN LINE FORM 2 Ġ) Ġon Ġwhich Ġsoft max Ġfunction Ġis Ġapplied , Ġeach ĠIN LINE FORM 3 ĠIN LINE FORM 4 Ġ( 0 , Ġ1 ). ĠIt Ġis Ġobservable Ġthat Ġthe Ġaverage Ġvalue Ġof ĠIN LINE FORM 5 Ġis Ġalways ĠIN LINE FORM 6 Ġ. ĠIn Ġother Ġwords , Ġit Ġis Ġimpossible Ġto Ġproduce Ġa Ġvector Ġhaving Ġall ĠIN LINE FORM 7 Ġ< ĠIN LINE FORM 8 Ġwhen Ġtwo Ġsequences Ġhave Ġno Ġmatching Ġat Ġall . </s> In Ġcases , Ġwhere Ġone Ġor Ġmore Ġword Ġpairs Ġfrom Ġtwo Ġdifferent Ġsequences Ġare Ġhighly Ġdiss imilar , Ġit Ġis Ġimpossible Ġto Ġassign Ġa Ġvery Ġlow Ġprobability Ġto Ġit Ġwithout Ġincreasing Ġthe Ġprobability Ġof Ġsome Ġother Ġpair Ġsomewhere Ġelse Ġsince ĠIN LINE FORM 0 Ġ= Ġ1 . </s> For Ġexample , Ġwhen Ġwe Ġconsider Ġtwo Ġsequences Ġ" height Ġof Ġtom Ġcruise " Ġand Ġ" age Ġof Ġsun ", Ġwhile Ġcomputing Ġthe Ġattention Ġweights Ġbetween Ġthe Ġword Ġ" height " Ġand Ġall Ġthe Ġwords Ġin Ġthe Ġsecond Ġsequence Ġit Ġcan Ġbe Ġobserved Ġthat Ġtheir Ġno Ġmatching Ġword Ġin Ġthe Ġlatter . ĠIn Ġthis Ġcase , Ġa Ġstandard Ġdot - product Ġbased Ġattention Ġwith Ġsoft max Ġwon 't Ġbe Ġable Ġto Ġproduce Ġweights Ġwhich Ġis Ġbelow Ġ0 . 33 Ġ(= 1 / 3 ) Ġfor Ġall Ġthe Ġwords Ġin Ġthe Ġsecond Ġsequence Ġwith Ġrespect Ġto Ġthe Ġword Ġ" height " Ġin Ġthe Ġfirst Ġsequence . </s> Conf lict Ġmodel </s> We Ġpropose Ġa Ġdifferent Ġmechanism Ġthat Ġdoes Ġthe Ġopposite Ġof Ġwhat Ġattention Ġdoes Ġthat Ġis Ġcomputing Ġhow Ġmuch Ġtwo Ġsequences Ġrep el Ġeach Ġother . ĠThis Ġworks Ġvery Ġsimilar Ġto Ġhow Ġattention Ġworks Ġbut Ġin versely . </s> We Ġdemonstrate Ġa Ġgeneral Ġmodel Ġbut Ġwe Ġalso Ġrealize Ġthat Ġthere Ġcan Ġbe Ġother Ġvariants Ġof Ġit Ġwhich Ġmay Ġbe Ġworked Ġout Ġto Ġperform Ġbetter . ĠOur Ġapproach Ġconsists Ġof Ġusing Ġelement Ġwise Ġdifference Ġbetween Ġtwo Ġvectors Ġfollowed Ġby Ġa Ġlinear Ġtransformation Ġto Ġproduce Ġa Ġscal ar Ġweight . ĠThe Ġremaining Ġof Ġthe Ġprocess Ġacts Ġsimilar Ġto Ġhow Ġattention Ġworks . ĠMat hematically , Ġwe Ġcan Ġexpress Ġit Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 ĠIN LINE FORM 1 ĠIN LINE FORM 2 Ġis Ġa Ġparameter Ġthat Ġwe Ġintroduce Ġto Ġprovide Ġa Ġweight Ġfor Ġthe Ġpair . ĠThe Ġtwo Ġword Ġrepresentations ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġare Ġprojected Ġto Ġa Ġspace Ġwhere Ġtheir Ġelement Ġwise Ġdifference Ġcan Ġbe Ġused Ġto Ġmodel Ġtheir Ġdiss imilar ity Ġand Ġsoft max Ġapplied Ġon Ġthem Ġcan Ġproduce Ġhigh Ġprobability Ġto Ġmore Ġdiss imilar Ġword Ġpairs . </s> It Ġis Ġgood Ġto Ġnote Ġthat Ġconflict Ġsuffers Ġfrom Ġthe Ġsame Ġlimitation Ġthat Ġattention Ġsuffers Ġfrom . ĠThis Ġis Ġwhen Ġa Ġpair Ġof Ġsentences Ġare Ġhighly Ġmatching Ġespecially Ġwith Ġmultiple Ġassociations . ĠBut Ġwhen Ġthe Ġtwo Ġmethods Ġwork Ġtogether , Ġeach Ġcompens ates Ġfor Ġthe Ġother 's Ġshortcomings . </s> Comb ination Ġof Ġattention Ġand Ġconflict </s> We Ġused Ġtwo Ġweighted Ġrepresentations Ġof ĠIN LINE FORM 0 Ġusing Ġweights Ġof Ġattention Ġand Ġconflict Ġas Ġcomputed Ġin ĠE q n . Ġ( 4 ) Ġand Ġ( 8 ) Ġrespectively . ĠOur Ġfinal Ġrepresentation Ġof Ġa Ġword Ġrepresentation ĠIN LINE FORM 1 Ġconditioned Ġon ĠIN LINE FORM 2 Ġcan Ġbe Ġexpressed Ġas : ĠDIS PLAY FORM 0 </s> where ĠA Ġand ĠC Ġdenote Ġthat Ġthey Ġare Ġfrom Ġattention Ġand Ġconflict Ġmodels Ġrespectively . </s> Rel ation Ġto ĠMulti - Head Ġattention </s> Multi - head Ġattention , Ġas Ġintroduced Ġin ĠB IB REF 3 Ġ, Ġcomp utes Ġmultiple Ġidentical Ġattention Ġmechanism Ġparalle lly Ġon Ġmultiple Ġlinear Ġprojections Ġof Ġsame Ġinputs . ĠThe Ġparameters Ġof Ġeach Ġattention Ġand Ġprojections Ġare Ġdifferent Ġin Ġeach Ġhead . ĠFinally , Ġthey Ġconc aten ate Ġall Ġthe Ġattent ions Ġwhich Ġis Ġsimilar Ġto Ġhow Ġwe Ġconc aten ate Ġconflict Ġand Ġattention . ĠHowever , Ġthey Ġuse Ġdot - product Ġto Ġcompute Ġeach Ġof Ġthe Ġattention . </s> Our Ġcombined Ġmodel Ġthat Ġcontains Ġboth Ġattention Ġand Ġconflict Ġcan Ġbe Ġthought Ġof Ġas Ġa Ġ2 - head Ġattention Ġmodel Ġbut Ġboth Ġheads Ġare Ġdifferent . ĠOur Ġconflict Ġhead Ġexplicitly Ġcaptures Ġdifference Ġbetween Ġthe Ġinputs . </s> Visual izing Ġattention Ġand Ġconflict </s> We Ġobserve Ġhow Ġour Ġconflict Ġmodel Ġlearns Ġthe Ġdiss imilar ities Ġbetween Ġword Ġrepresentations . ĠWe Ġachieve Ġthat Ġby Ġvisual izing Ġthe Ġheat map Ġof Ġthe Ġweight Ġmatrix ĠIN LINE FORM 0 Ġfor Ġboth Ġattention Ġand Ġconflict Ġfrom Ġeq ns . Ġ( 3 ) Ġand Ġ( 8 ). ĠWhile Ġattention Ġsuccessfully Ġlearns Ġthe Ġalign ments , Ġconflict Ġmatrix Ġalso Ġshows Ġthat Ġour Ġapproach Ġmodels Ġthe Ġcontradict ing Ġassociations Ġlike Ġ" animal " Ġand Ġ" lake " Ġor Ġ" a ust ral ia " Ġand Ġ" world ". ĠThese Ġtwo Ġassociations Ġare Ġthe Ġunique Ġpairs Ġwhich Ġare Ġinstrumental Ġin Ġdetermining Ġthat Ġthe Ġtwo Ġqueries Ġare Ġnot Ġsimilar . </s> The Ġmodel </s> We Ġcreate Ġtwo Ġmodels Ġboth Ġof Ġwhich Ġconstitutes Ġof Ġthree Ġmain Ġparts : Ġenc oder , Ġinteraction Ġand Ġclass ifier Ġand Ġtake Ġtwo Ġsequences Ġas Ġinput . ĠExcept Ġinteraction , Ġall Ġthe Ġother Ġparts Ġare Ġexactly Ġidentical Ġbetween Ġthe Ġtwo Ġmodels . ĠThe Ġenc oder Ġis Ġshared Ġamong Ġthe Ġsequences Ġsimply Ġuses Ġtwo Ġstacked ĠGR U Ġlayers . ĠThe Ġinteraction Ġpart Ġconsists Ġof Ġonly Ġattention Ġfor Ġone Ġmodel Ġwhile Ġfor Ġthe Ġanother Ġone Ġit Ġconsists Ġof Ġattention Ġand Ġconflict Ġcombined Ġas Ġshown Ġin Ġ( eq n . 11 ) Ġ. ĠThe Ġclass ifier Ġpart Ġis Ġsimply Ġstacked Ġfully - connected Ġlayers . ĠFigure Ġ3 Ġshows Ġa Ġblock Ġdiagram Ġof Ġhow Ġour Ġmodel Ġlooks Ġlike . </s> Task Ġ1 : ĠQu ora ĠDupl icate ĠQuestion ĠPair ĠDetection </s> The Ġdataset Ġincludes Ġpairs Ġof Ġquestions Ġlabelled Ġas Ġ1 Ġor Ġ0 Ġdepending Ġon Ġwhether Ġa Ġpair Ġis Ġduplicate Ġor Ġnot Ġrespectively . ĠThis Ġis Ġa Ġpopular Ġpair - level Ġclassification Ġtask Ġon Ġwhich Ġextensive Ġwork Ġhas Ġalready Ġbeen Ġdone Ġbefore Ġlike ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠFor Ġthis Ġtask , Ġwe Ġmake Ġthe Ġoutput Ġlayer Ġof Ġour Ġmodel Ġto Ġpredict Ġtwo Ġprobabilities Ġfor Ġnon - du pl icate Ġand Ġduplicate . ĠWe Ġsample Ġthe Ġdata Ġfrom Ġthe Ġoriginal Ġdataset Ġso Ġthat Ġit Ġcontains Ġequal Ġpositive Ġand Ġnegative Ġclasses . ĠOriginal Ġdataset Ġhas Ġsome Ġclass Ġimbalance Ġbut Ġfor Ġsake Ġsimplicity Ġwe Ġdon 't Ġconsider Ġit . ĠThe Ġfinal Ġdata Ġthat Ġwe Ġuse Ġhas Ġroughly Ġ400 , 000 Ġquestion Ġpairs Ġand Ġwe Ġsplit Ġthis Ġdata Ġinto Ġtrain Ġand Ġtest Ġusing Ġ8 : 2 Ġratio . </s> We Ġtrain Ġall Ġour Ġmodels Ġfor Ġroughly Ġ2 Ġepoch s Ġwith Ġa Ġbatch Ġsize Ġof Ġ64 . ĠWe Ġuse Ġa Ġhidden Ġdimension Ġof Ġ150 Ġthroughout Ġthe Ġmodel . ĠThe Ġembed ding Ġlayer Ġuses ĠEL MO ĠB IB REF 9 Ġwhich Ġhas Ġproven Ġto Ġbe Ġvery Ġuseful Ġin Ġvarious Ġdownstream Ġlanguage Ġunderstanding Ġtasks . ĠOur ĠFC Ġlayers Ġconsists Ġof Ġfour Ġdense Ġlayers Ġwith ĠIN LINE FORM 0 Ġactivation Ġafter Ġeach Ġlayer . ĠThe Ġdrop out Ġrate Ġis Ġkept Ġas Ġ0 . 2 Ġfor Ġevery Ġrecurrent Ġand ĠFC Ġlinear Ġlayers . ĠWe Ġuse ĠAdam Ġoptim izer Ġin Ġour Ġexperiment Ġwith Ġe ps ilon = 1 e - 8 , Ġbeta = 0 . 9 Ġand Ġlearning Ġrate = 1 e - 3 . </s> Task Ġ2 : ĠRanking Ġquestions Ġin ĠBing 's ĠPeople ĠAlso ĠAsk </s> People ĠAlso ĠAsk Ġis Ġa Ġfeature Ġin ĠBing Ġsearch Ġresult Ġpage Ġwhere Ġrelated Ġquestions Ġare Ġrecommended Ġto Ġthe Ġuser . ĠUser Ġmay Ġclick Ġon Ġa Ġquestion Ġto Ġview Ġthe Ġanswer . ĠClick ing Ġis Ġa Ġpositive Ġfeedback Ġthat Ġshows Ġuser 's Ġinterest Ġin Ġthe Ġquestion . ĠWe Ġuse Ġthis Ġclick Ġlogs Ġto Ġbuild Ġa Ġquestion Ġclass ifier Ġusing Ġthe Ġsame Ġmodel Ġin ĠFigure Ġ3 . ĠThe Ġproblem Ġstatement Ġis Ġvery Ġsimilar Ġto ĠB IB REF 10 Ġwhere Ġthey Ġuse Ġlog istic Ġregression Ġto Ġpredict Ġwhether Ġan Ġuser Ġwould Ġclick Ġon Ġad . ĠOur Ġgoal Ġis Ġto Ġclassify Ġif Ġa Ġquestion Ġis Ġpotential Ġhigh - click Ġquestion Ġor Ġnot Ġfor Ġa Ġgiven Ġquery . ĠFor Ġthis , Ġwe Ġfirst Ġcreate Ġa Ġlabelled Ġdata Ġset Ġusing Ġthe Ġclick Ġlogs Ġwhere Ġany Ġquestion Ġhaving ĠCTR Ġlower Ġthan Ġ0 . 3 Ġis Ġlabelled Ġas Ġ0 Ġand Ġa Ġquestion Ġhaving ĠCTR Ġmore Ġthan Ġ0 . 7 Ġas Ġ1 . </s> Our Ġfinal Ġdata Ġresembles Ġthat Ġof Ġa Ġpair - level Ġclass ifier , Ġas Ġin ĠTask Ġ1 , Ġwhere Ġuser Ġquery Ġand Ġcandidate Ġquestions Ġare Ġinput . ĠWith Ġthese Ġdata Ġset , Ġwe Ġtrain Ġa Ġbinary Ġclass ifier Ġto Ġdetect Ġhigh - click Ġand Ġlow - click Ġquestions . </s> Quant itative ĠAnalysis </s> For Ġboth Ġtasks , Ġwe Ġcompute Ġclassification Ġaccuracy Ġusing Ġthree Ġmodel Ġvariants Ġand Ġreport Ġthe Ġresults Ġin ĠTable Ġ1 Ġand ĠTable Ġ2 . ĠWe Ġobserve Ġthat Ġmodel Ġwith Ġboth Ġattention Ġand Ġconflict Ġcombined Ġgives Ġthe Ġbest Ġresults . </s> We Ġalso Ġshow Ġthe Ġtraining Ġloss Ġcurve Ġfor Ġboth Ġthe Ġmodels Ġhaving Ġattention Ġand Ġattention Ġcombined Ġwith Ġconflict Ġrespectively . ĠFigure Ġ4 Ġand Ġ5 Ġshows Ġthese Ġcurves Ġfor ĠTask Ġ1 Ġand ĠTask Ġ2 Ġrespectively . ĠThe Ġcurves Ġare Ġsmoot hed Ġusing Ġmoving Ġaverage Ġhaving Ġan Ġwindow Ġsize Ġof Ġ8 . ĠWe Ġnotice Ġthat Ġthe Ġconflict Ġmodel Ġhas Ġmuch Ġst eeper Ġslope Ġand Ġconver ges Ġto Ġa Ġmuch Ġbetter Ġmin ima Ġin Ġboth Ġthe Ġtasks . ĠIt Ġcan Ġalso Ġbe Ġnoticed Ġthat Ġin Ġthe Ġtraining Ġprocedure Ġfor Ġthe Ġmodel Ġwhich Ġhas Ġboth Ġattention Ġand Ġconflict , Ġthe Ġupdates Ġare Ġmuch Ġsmoother . </s> Qual itative ĠComparison </s> We Ġalso Ġshow Ġqualitative Ġresults Ġwhere Ġwe Ġcan Ġobserve Ġthat Ġour Ġmodel Ġwith Ġattention Ġand Ġconflict Ġcombined Ġdoes Ġbetter Ġon Ġcases Ġwhere Ġpairs Ġare Ġnon - du pl icate Ġand Ġhas Ġvery Ġsmall Ġdifference . ĠWe Ġhave Ġobserved Ġthat Ġthe Ġconflict Ġmodel Ġis Ġvery Ġsensitive Ġto Ġeven Ġminor Ġdifferences Ġand Ġcompens ates Ġin Ġsuch Ġcases Ġwhere Ġattention Ġposes Ġhigh Ġbias Ġtowards Ġsimilarities Ġalready Ġthere Ġin Ġthe Ġsequences . </s> Sequ ence Ġ1 : ĠWhat Ġare Ġthe Ġbest Ġways Ġto Ġlearn ĠFrench Ġ? </s> Sequ ence Ġ2 : ĠHow Ġdo ĠI Ġlearn Ġfrench Ġgenders Ġ? </s> Att ention Ġonly : Ġ1 </s> Att ention + Conf lict : Ġ0 </s> Ground ĠTruth : Ġ0 </s> Sequ ence Ġ1 : ĠHow Ġdo ĠI Ġprevent Ġbreast Ġcancer Ġ? </s> Sequ ence Ġ2 : ĠIs Ġbreast Ġcancer Ġprevent able Ġ? </s> Att ention Ġonly : Ġ1 </s> Att ention + Conf lict : Ġ0 </s> Ground ĠTruth : Ġ0 </s> We Ġprovide Ġtwo Ġexamples Ġwith Ġpredictions Ġfrom Ġthe Ġmodels Ġwith Ġonly Ġattention Ġand Ġcombination Ġof Ġattention Ġand Ġconflict . ĠEach Ġexample Ġis Ġaccompanied Ġby Ġthe Ġground Ġtruth Ġin Ġour Ġdata . </s> Analy zing Ġthe Ġgains </s> We Ġanalyzed Ġthe Ġgains Ġin ĠTask Ġ1 Ġwhich Ġwe Ġget Ġfrom Ġthe Ġattention - conf lict Ġmodel Ġin Ġorder Ġto Ġensure Ġthat Ġthey Ġare Ġnot Ġdue Ġto Ġrandom ness Ġin Ġweight Ġinitialization Ġor Ġsimply Ġadditional Ġparameters . ĠWe Ġparticularly Ġfocused Ġon Ġthe Ġexamples Ġwhich Ġwere Ġincorrectly Ġmarked Ġin Ġattention Ġmodel Ġbut Ġcorrectly Ġin Ġattention - conf lict Ġmodel . ĠWe Ġsaw Ġthat Ġ70 % Ġof Ġthose Ġcases Ġare Ġthe Ġones Ġwhere Ġthe Ġpair Ġwas Ġincorrectly Ġmarked Ġas Ġduplicate Ġin Ġthe Ġprevious Ġmodel Ġbut Ġour Ġcombined Ġmodel Ġcorrectly Ġmarked Ġthem Ġas Ġnon - du pl icate . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġhighlighted Ġthe Ġlimits Ġof Ġattention Ġespecially Ġin Ġcases Ġwhere Ġtwo Ġsequences Ġhave Ġa Ġcontradict ing Ġrelationship Ġbased Ġon Ġthe Ġtask Ġit Ġperforms . ĠTo Ġalleviate Ġthis Ġproblem Ġand Ġfurther Ġimprove Ġthe Ġperformance , Ġwe Ġpropose Ġa Ġconflict Ġmechanism Ġthat Ġtries Ġto Ġcapture Ġhow Ġtwo Ġsequences Ġrep el Ġeach Ġother . ĠThis Ġacts Ġlike Ġthe Ġinverse Ġof Ġattention Ġand , Ġempir ically , Ġwe Ġshow Ġthat Ġhow Ġconflict Ġand Ġattention Ġtogether Ġcan Ġimprove Ġthe Ġperformance . </s> Future Ġresearch Ġwork Ġshould Ġbe Ġbased Ġon Ġalternative Ġdesign Ġof Ġconflict Ġmechanism Ġusing Ġother Ġdifference Ġoperators Ġother Ġthan Ġelement Ġwise Ġdifference Ġwhich Ġwe Ġuse .