20 minutes
EXTRACTIVE
Previous neural sentence embedding methods started the training from a random initialization. In this publication, we use the pre-trained BERT and RoBERTa network and only fine-tune it to yield useful sentence embeddings. This reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods.
1908.10084
Introduction </s> In Ġthis Ġpublication , Ġwe Ġpresent ĠSent ence - BER T Ġ( S BER T ), Ġa Ġmodification Ġof Ġthe ĠB ERT Ġnetwork Ġusing Ġsi ames e Ġand Ġtriple t Ġnetworks Ġthat Ġis Ġable Ġto Ġderive Ġsem antically Ġmeaningful Ġsentence Ġembed d ings . ĠThis Ġenables ĠB ERT Ġto Ġbe Ġused Ġfor Ġcertain Ġnew Ġtasks , Ġwhich Ġup - to - now Ġwere Ġnot Ġapplicable Ġfor ĠB ERT . ĠThese Ġtasks Ġinclude Ġlarge - scale Ġsemantic Ġsimilarity Ġcomparison , Ġclust ering , Ġand Ġinformation Ġretrieval Ġvia Ġsemantic Ġsearch . </s> BER T Ġset Ġnew Ġstate - of - the - art Ġperformance Ġon Ġvarious Ġsentence Ġclassification Ġand Ġsentence - pair Ġregression Ġtasks . ĠB ERT Ġuses Ġa Ġcross - enc oder : ĠTwo Ġsentences Ġare Ġpassed Ġto Ġthe Ġtransformer Ġnetwork Ġand Ġthe Ġtarget Ġvalue Ġis Ġpredicted . ĠHowever , Ġthis Ġsetup Ġis Ġunsu itable Ġfor Ġvarious Ġpair Ġregression Ġtasks Ġdue Ġto Ġtoo Ġmany Ġpossible Ġcombinations . ĠFinding Ġin Ġa Ġcollection Ġof Ġ$ n = 10 \ , 000 $ Ġsentences Ġthe Ġpair Ġwith Ġthe Ġhighest Ġsimilarity Ġrequires Ġwith ĠB ERT Ġ$ n \ cd ot Ġ( n - 1 )/ 2 = 49 \ , 995 \ , 000 $ Ġinference Ġcomput ations . ĠOn Ġa Ġmodern ĠV 100 ĠGPU , Ġthis Ġrequires Ġabout Ġ65 Ġhours . ĠSimilar , Ġfinding Ġwhich Ġof Ġthe Ġover Ġ40 Ġmillion Ġexist ent Ġquestions Ġof ĠQu ora Ġis Ġthe Ġmost Ġsimilar Ġfor Ġa Ġnew Ġquestion Ġcould Ġbe Ġmodeled Ġas Ġa Ġpair - wise Ġcomparison Ġwith ĠB ERT , Ġhowever , Ġanswering Ġa Ġsingle Ġquery Ġwould Ġrequire Ġover Ġ50 Ġhours . </s> A Ġcommon Ġmethod Ġto Ġaddress Ġclust ering Ġand Ġsemantic Ġsearch Ġis Ġto Ġmap Ġeach Ġsentence Ġto Ġa Ġvector Ġspace Ġsuch Ġthat Ġsem antically Ġsimilar Ġsentences Ġare Ġclose . ĠResearchers Ġhave Ġstarted Ġto Ġinput Ġindividual Ġsentences Ġinto ĠB ERT Ġand Ġto Ġderive Ġfixed - size Ġsentence Ġembed d ings . ĠThe Ġmost Ġcommonly Ġused Ġapproach Ġis Ġto Ġaverage Ġthe ĠB ERT Ġoutput Ġlayer Ġ( known Ġas ĠB ERT Ġembed d ings ) Ġor Ġby Ġusing Ġthe Ġoutput Ġof Ġthe Ġfirst Ġtoken Ġ( the Ġ[ CL S ] Ġtoken ). ĠAs Ġwe Ġwill Ġshow , Ġthis Ġcommon Ġpractice Ġyields Ġrather Ġbad Ġsentence Ġembed d ings , Ġoften Ġworse Ġthan Ġaveraging ĠGl o Ve Ġembed d ings ĠB IB REF 2 . </s> To Ġalleviate Ġthis Ġissue , Ġwe Ġdeveloped ĠS BER T . ĠThe Ġsi ames e Ġnetwork Ġarchitecture Ġenables Ġthat Ġfixed - sized Ġvectors Ġfor Ġinput Ġsentences Ġcan Ġbe Ġderived . ĠUsing Ġa Ġsimilarity Ġmeasure Ġlike Ġcos ine - similar ity Ġor ĠMan hatt en Ġ/ ĠEucl ide an Ġdistance , Ġsem antically Ġsimilar Ġsentences Ġcan Ġbe Ġfound . ĠThese Ġsimilarity Ġmeasures Ġcan Ġbe Ġperformed Ġextremely Ġefficient Ġon Ġmodern Ġhardware , Ġallowing ĠS BER T Ġto Ġbe Ġused Ġfor Ġsemantic Ġsimilarity Ġsearch Ġas Ġwell Ġas Ġfor Ġclust ering . ĠThe Ġcomplexity Ġfor Ġfinding Ġthe Ġmost Ġsimilar Ġsentence Ġpair Ġin Ġa Ġcollection Ġof Ġ10 , 000 Ġsentences Ġis Ġreduced Ġfrom Ġ65 Ġhours Ġwith ĠB ERT Ġto Ġthe Ġcomputation Ġof Ġ10 , 000 Ġsentence Ġembed d ings Ġ( 5 Ġseconds Ġwith ĠS BER T ) Ġand Ġcomputing Ġcos ine - similar ity Ġ( 0 . 01 Ġseconds ). ĠBy Ġusing Ġoptimized Ġindex Ġstructures , Ġfinding Ġthe Ġmost Ġsimilar ĠQu ora Ġquestion Ġcan Ġbe Ġreduced Ġfrom Ġ50 Ġhours Ġto Ġa Ġfew Ġmilliseconds ĠB IB REF 3 . </s> We Ġfine - t une ĠS BER T Ġon ĠNL I Ġdata , Ġwhich Ġcreates Ġsentence Ġembed d ings Ġthat Ġsignificantly Ġoutper form Ġother Ġstate - of - the - art Ġsentence Ġembed ding Ġmethods Ġlike ĠIn fer Sent ĠB IB REF 4 Ġand ĠUniversal ĠSent ence ĠEnc oder ĠB IB REF 5 . ĠOn Ġseven ĠSem antic ĠText ual ĠSimilar ity Ġ( ST S ) Ġtasks , ĠS BER T Ġachieves Ġan Ġimprovement Ġof Ġ11 . 7 Ġpoints Ġcompared Ġto ĠIn fer Sent Ġand Ġ5 . 5 Ġpoints Ġcompared Ġto ĠUniversal ĠSent ence ĠEnc oder . ĠOn ĠSent E val ĠB IB REF 6 , Ġan Ġevaluation Ġtool kit Ġfor Ġsentence Ġembed d ings , Ġwe Ġachieve Ġan Ġimprovement Ġof Ġ2 . 1 Ġand Ġ2 . 6 Ġpoints , Ġrespectively . </s> S BER T Ġcan Ġbe Ġadapted Ġto Ġa Ġspecific Ġtask . ĠIt Ġsets Ġnew Ġstate - of - the - art Ġperformance Ġon Ġa Ġchallenging Ġargument Ġsimilarity Ġdataset ĠB IB REF 7 Ġand Ġon Ġa Ġtriple t Ġdataset Ġto Ġdistinguish Ġsentences Ġfrom Ġdifferent Ġsections Ġof Ġa ĠWikipedia Ġarticle ĠB IB REF 8 . </s> The Ġpaper Ġis Ġstructured Ġin Ġthe Ġfollowing Ġway : ĠSection ĠSEC REF 3 Ġpresents ĠS BER T , Ġsection ĠSEC REF 4 Ġevaluates ĠS BER T Ġon Ġcommon ĠST S Ġtasks Ġand Ġon Ġthe Ġchallenging ĠArgument ĠFac et ĠSimilar ity Ġ( AF S ) Ġcorpus ĠB IB REF 7 . ĠSection ĠSEC REF 5 Ġevaluates ĠS BER T Ġon ĠSent E val . ĠIn Ġsection ĠSEC REF 6 , Ġwe Ġperform Ġan Ġab lation Ġstudy Ġto Ġtest Ġsome Ġdesign Ġaspect Ġof ĠS BER T . ĠIn Ġsection ĠSEC REF 7 , Ġwe Ġcompare Ġthe Ġcomputational Ġefficiency Ġof ĠS BER T Ġsentence Ġembed d ings Ġin Ġcontrast Ġto Ġother Ġstate - of - the - art Ġsentence Ġembed ding Ġmethods . </s> Related ĠWork </s> We Ġfirst Ġintroduce ĠB ERT , Ġthen , Ġwe Ġdiscuss Ġstate - of - the - art Ġsentence Ġembed ding Ġmethods . </s> BER T ĠB IB REF 0 Ġis Ġa Ġpre - trained Ġtransformer Ġnetwork ĠB IB REF 9 , Ġwhich Ġset Ġfor Ġvarious ĠN LP Ġtasks Ġnew Ġstate - of - the - art Ġresults , Ġincluding Ġquestion Ġanswering , Ġsentence Ġclassification , Ġand Ġsentence - pair Ġregression . ĠThe Ġinput Ġfor ĠB ERT Ġfor Ġsentence - pair Ġregression Ġconsists Ġof Ġthe Ġtwo Ġsentences , Ġseparated Ġby Ġa Ġspecial Ġ[ SE P ] Ġtoken . ĠMulti - head Ġattention Ġover Ġ12 Ġ( base - model ) Ġor Ġ24 Ġlayers Ġ( large - model ) Ġis Ġapplied Ġand Ġthe Ġoutput Ġis Ġpassed Ġto Ġa Ġsimple Ġregression Ġfunction Ġto Ġderive Ġthe Ġfinal Ġlabel . ĠUsing Ġthis Ġsetup , ĠB ERT Ġset Ġa Ġnew Ġstate - of - the - art Ġperformance Ġon Ġthe ĠSem antic ĠText ual ĠSem ilar ity Ġ( ST S ) Ġbenchmark ĠB IB REF 10 . ĠRo BER Ta ĠB IB REF 1 Ġshowed , Ġthat Ġthe Ġperformance Ġof ĠB ERT Ġcan Ġfurther Ġimproved Ġby Ġsmall Ġadaptations Ġto Ġthe Ġpre - training Ġprocess . ĠWe Ġalso Ġtested ĠXL Net ĠB IB REF 11 , Ġbut Ġit Ġled Ġin Ġgeneral Ġto Ġworse Ġresults Ġthan ĠB ERT . </s> A Ġlarge Ġdisadvantage Ġof Ġthe ĠB ERT Ġnetwork Ġstructure Ġis Ġthat Ġno Ġindependent Ġsentence Ġembed d ings Ġare Ġcomputed , Ġwhich Ġmakes Ġit Ġdifficult Ġto Ġderive Ġsentence Ġembed d ings Ġfrom ĠB ERT . ĠTo Ġbypass Ġthis Ġlimitations , Ġresearchers Ġpassed Ġsingle Ġsentences Ġthrough ĠB ERT Ġand Ġthen Ġderive Ġa Ġfixed Ġsized Ġvector Ġby Ġeither Ġaveraging Ġthe Ġoutputs Ġ( similar Ġto Ġaverage Ġword Ġembed d ings ) Ġor Ġby Ġusing Ġthe Ġoutput Ġof Ġthe Ġspecial ĠCLS Ġtoken Ġ( for Ġexample : Ġb ert sent ence embed d ings 1 , ber ts ent ence embed d ings 2 , ber ts ent ence embed d ings 3 ). ĠThese Ġtwo Ġoptions Ġare Ġalso Ġprovided Ġby Ġthe Ġpopular Ġb ert - as - a - service - re pos itory . ĠUp Ġto Ġour Ġknowledge , Ġthere Ġis Ġso Ġfar Ġno Ġevaluation Ġif Ġthese Ġmethods Ġlead Ġto Ġuseful Ġsentence Ġembed d ings . </s> Sent ence Ġembed d ings Ġare Ġa Ġwell Ġstudied Ġarea Ġwith Ġdozens Ġof Ġproposed Ġmethods . ĠSkip - Th ought ĠB IB REF 12 Ġtrains Ġan Ġenc oder - dec oder Ġarchitecture Ġto Ġpredict Ġthe Ġsurrounding Ġsentences . ĠIn fer Sent ĠB IB REF 4 Ġuses Ġlabeled Ġdata Ġof Ġthe ĠStanford ĠNatural ĠLanguage ĠIn ference Ġdataset ĠB IB REF 13 Ġand Ġthe ĠMulti - Gen re ĠNL I Ġdataset ĠB IB REF 14 Ġto Ġtrain Ġa Ġsi ames e ĠBi L ST M Ġnetwork Ġwith Ġmax - pool ing Ġover Ġthe Ġoutput . ĠCon neau Ġet Ġal . Ġshowed , Ġthat ĠIn fer Sent Ġconsistently Ġoutper forms Ġun super vised Ġmethods Ġlike ĠSkip Th ought . ĠUniversal ĠSent ence ĠEnc oder ĠB IB REF 5 Ġtrains Ġa Ġtransformer Ġnetwork Ġand Ġaug ments Ġun super vised Ġlearning Ġwith Ġtraining Ġon ĠSN LI . Ġhill - et al - 2016 - learning Ġshowed , Ġthat Ġthe Ġtask Ġon Ġwhich Ġsentence Ġembed d ings Ġare Ġtrained Ġsignificantly Ġimpacts Ġtheir Ġquality . ĠPrevious Ġwork ĠB IB REF 4 , ĠB IB REF 5 Ġfound Ġthat Ġthe ĠSN LI Ġdatasets Ġare Ġsuitable Ġfor Ġtraining Ġsentence Ġembed d ings . Ġy ang - 2018 - learning Ġpresented Ġa Ġmethod Ġto Ġtrain Ġon Ġconversations Ġfrom ĠReddit Ġusing Ġsi ames e ĠD AN Ġand Ġsi ames e Ġtransformer Ġnetworks , Ġwhich Ġyielded Ġgood Ġresults Ġon Ġthe ĠST S Ġbenchmark Ġdataset . </s> poly enc od ers Ġaddresses Ġthe Ġrun - time Ġoverhead Ġof Ġthe Ġcross - enc oder Ġfrom ĠB ERT Ġand Ġpresent Ġa Ġmethod Ġ( poly - enc od ers ) Ġto Ġcompute Ġa Ġscore Ġbetween Ġ$ m $ Ġcontext Ġvectors Ġand Ġpre - com puted Ġcandidate Ġembed d ings Ġusing Ġattention . ĠThis Ġidea Ġworks Ġfor Ġfinding Ġthe Ġhighest Ġscoring Ġsentence Ġin Ġa Ġlarger Ġcollection . ĠHowever , Ġpoly - enc od ers Ġhave Ġthe Ġdrawback Ġthat Ġthe Ġscore Ġfunction Ġis Ġnot Ġsymm etric Ġand Ġthe Ġcomputational Ġoverhead Ġis Ġtoo Ġlarge Ġfor Ġuse - cases Ġlike Ġclust ering , Ġwhich Ġwould Ġrequire Ġ$ O ( n ^ 2 ) $ Ġscore Ġcomput ations . </s> Previous Ġneural Ġsentence Ġembed ding Ġmethods Ġstarted Ġthe Ġtraining Ġfrom Ġa Ġrandom Ġinitialization . ĠIn Ġthis Ġpublication , Ġwe Ġuse Ġthe Ġpre - trained ĠB ERT Ġand ĠRo BER Ta Ġnetwork Ġand Ġonly Ġfine - t une Ġit Ġto Ġyield Ġuseful Ġsentence Ġembed d ings . ĠThis Ġreduces Ġsignificantly Ġthe Ġneeded Ġtraining Ġtime : ĠS BER T Ġcan Ġbe Ġtuned Ġin Ġless Ġthan Ġ20 Ġminutes , Ġwhile Ġyielding Ġbetter Ġresults Ġthan Ġcomparable Ġsentence Ġembed ding Ġmethods . </s> Model </s> S BER T Ġadds Ġa Ġpool ing Ġoperation Ġto Ġthe Ġoutput Ġof ĠB ERT Ġ/ ĠRo BER Ta Ġto Ġderive Ġa Ġfixed Ġsized Ġsentence Ġembed ding . ĠWe Ġexperiment Ġwith Ġthree Ġpool ing Ġstrategies : ĠUsing Ġthe Ġoutput Ġof Ġthe ĠCLS - token , Ġcomputing Ġthe Ġmean Ġof Ġall Ġoutput Ġvectors Ġ( ME AN - str ategy ), Ġand Ġcomputing Ġa Ġmax - over - time Ġof Ġthe Ġoutput Ġvectors Ġ( MAX - str ategy ). ĠThe Ġdefault Ġconfiguration Ġis ĠME AN . </s> In Ġorder Ġto Ġfine - t une ĠB ERT Ġ/ ĠRo BER Ta , Ġwe Ġcreate Ġsi ames e Ġand Ġtriple t Ġnetworks ĠB IB REF 15 Ġto Ġupdate Ġthe Ġweights Ġsuch Ġthat Ġthe Ġproduced Ġsentence Ġembed d ings Ġare Ġsem antically Ġmeaningful Ġand Ġcan Ġbe Ġcompared Ġwith Ġcos ine - similar ity . </s> The Ġnetwork Ġstructure Ġdepends Ġon Ġthe Ġavailable Ġtraining Ġdata . ĠWe Ġexperiment Ġwith Ġthe Ġfollowing Ġstructures Ġand Ġobjective Ġfunctions . </s> Class ification ĠObjective ĠFunction . ĠWe Ġconc aten ate Ġthe Ġsentence Ġembed d ings Ġ$ u $ Ġand Ġ$ v $ Ġwith Ġthe Ġelement - wise Ġdifference Ġ$ | u - v | $ Ġand Ġmultiply Ġit Ġwith Ġthe Ġtrain able Ġweight Ġ$ W _ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ 3 n Ġ\ times Ġk } $ : </s> where Ġ$ n $ Ġis Ġthe Ġdimension Ġof Ġthe Ġsentence Ġembed d ings Ġand Ġ$ k $ Ġthe Ġnumber Ġof Ġlabels . ĠWe Ġoptimize Ġcross - ent ropy Ġloss . ĠThis Ġstructure Ġis Ġdepicted Ġin ĠFigure ĠFIG REF 4 . </s> Reg ression ĠObjective ĠFunction . ĠThe Ġcos ine - similar ity Ġbetween Ġthe Ġtwo Ġsentence Ġembed d ings Ġ$ u $ Ġand Ġ$ v $ Ġis Ġcomputed Ġ( Figure ĠFIG REF 5 ). ĠWe Ġuse Ġmean - squ ared - error Ġloss Ġas Ġthe Ġobjective Ġfunction . </s> Tri plet ĠObjective ĠFunction . ĠGiven Ġan Ġanchor Ġsentence Ġ$ a $, Ġa Ġpositive Ġsentence Ġ$ p $, Ġand Ġa Ġnegative Ġsentence Ġ$ n $, Ġtriple t Ġloss Ġtunes Ġthe Ġnetwork Ġsuch Ġthat Ġthe Ġdistance Ġbetween Ġ$ a $ Ġand Ġ$ p $ Ġis Ġsmaller Ġthan Ġthe Ġdistance Ġbetween Ġ$ a $ Ġand Ġ$ n $. ĠMat hematically , Ġwe Ġminimize Ġthe Ġfollowing Ġloss Ġfunction : </s> with Ġ$ s _ x $ Ġthe Ġsentence Ġembed ding Ġfor Ġ$ a $ /$ n $ /$ p $, Ġ$ || \ cd ot Ġ|| $ Ġa Ġdistance Ġmetric Ġand Ġmargin Ġ$\ ep sil on Ġ$ . ĠMarg in Ġ$\ ep sil on Ġ$ Ġensures Ġthat Ġ$ s _ p $ Ġis Ġat Ġleast Ġ$\ ep sil on Ġ$ Ġcloser Ġto Ġ$ s _ a $ Ġthan Ġ$ s _ n $. ĠAs Ġmetric Ġwe Ġuse ĠEucl ide an Ġdistance Ġand Ġwe Ġset Ġ$\ ep sil on Ġ= 1 $ Ġin Ġour Ġexperiments . </s> Model Ġ: :: ĠTraining ĠDetails </s> We Ġtrain ĠS BER T Ġon Ġthe Ġcombination Ġof Ġthe ĠSN LI ĠB IB REF 13 Ġand Ġthe ĠMulti - Gen re ĠNL I ĠB IB REF 14 Ġdataset . ĠThe ĠSN LI Ġis Ġa Ġcollection Ġof Ġ570 , 000 Ġsentence Ġpairs Ġannot ated Ġwith Ġthe Ġlabels Ġcontradiction , Ġe int ail ment , Ġand Ġneutral . ĠMulti N LI Ġcontains Ġ430 , 000 Ġsentence Ġpairs Ġand Ġcovers Ġa Ġrange Ġof Ġgenres Ġof Ġspoken Ġand Ġwritten Ġtext . ĠWe Ġfine - t une ĠS BER T Ġwith Ġa Ġ3 - way Ġsoft max - class ifier Ġobjective Ġfunction Ġfor Ġone Ġepoch . ĠWe Ġused Ġa Ġbatch - size Ġof Ġ16 , ĠAdam Ġoptim izer Ġwith Ġlearning Ġrate Ġ$ 2 \ math rm Ġ{ e }{ - 5 } $, Ġand Ġa Ġlinear Ġlearning Ġrate Ġwarm - up Ġover Ġ10 % Ġof Ġthe Ġtraining Ġdata . ĠOur Ġdefault Ġpool ing Ġstrategy Ġis ĠME AN . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity </s> We Ġevaluate Ġthe Ġperformance Ġof ĠS BER T Ġfor Ġcommon ĠSem antic ĠText ual ĠSimilar ity Ġ( ST S ) Ġtasks . ĠState - of - the - art Ġmethods Ġoften Ġlearn Ġa Ġ( complex ) Ġregression Ġfunction Ġthat Ġmaps Ġsentence Ġembed d ings Ġto Ġa Ġsimilarity Ġscore . ĠHowever , Ġthese Ġregression Ġfunctions Ġwork Ġpair - wise Ġand Ġdue Ġto Ġthe Ġcomb inator ial Ġexplosion Ġthose Ġare Ġoften Ġnot Ġscalable Ġif Ġthe Ġcollection Ġof Ġsentences Ġreaches Ġa Ġcertain Ġsize . ĠInstead , Ġwe Ġalways Ġuse Ġcos ine - similar ity Ġto Ġcompare Ġthe Ġsimilarity Ġbetween Ġtwo Ġsentence Ġembed d ings . ĠWe Ġran Ġour Ġexperiments Ġalso Ġwith Ġnegative ĠMan hatt en Ġand Ġnegative ĠEucl ide an Ġdistances Ġas Ġsimilarity Ġmeasures , Ġbut Ġthe Ġresults Ġfor Ġall Ġapproaches Ġremained Ġroughly Ġthe Ġsame . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity Ġ: :: ĠUn super vised ĠST S </s> We Ġevaluate Ġthe Ġperformance Ġof ĠS BER T Ġfor ĠST S Ġwithout Ġusing Ġany ĠST S Ġspecific Ġtraining Ġdata . ĠWe Ġuse Ġthe ĠST S Ġtasks Ġ2012 Ġ- Ġ2016 ĠB IB REF 16 , ĠB IB REF 17 , ĠB IB REF 18 , ĠB IB REF 19 , ĠB IB REF 20 , Ġthe ĠST S Ġbenchmark ĠB IB REF 10 , Ġand Ġthe ĠS ICK - Related ness Ġdataset ĠB IB REF 21 . ĠThese Ġdatasets Ġprovide Ġlabels Ġbetween Ġ0 Ġand Ġ5 Ġon Ġthe Ġsemantic Ġrelated ness Ġof Ġsentence Ġpairs . ĠWe Ġshowed Ġin ĠB IB REF 22 Ġthat ĠPearson Ġcorrelation Ġis Ġbadly Ġsuited Ġfor ĠST S . ĠInstead , Ġwe Ġcompute Ġthe ĠSpear man 's Ġrank Ġcorrelation Ġbetween Ġthe Ġcos ine - similar ity Ġof Ġthe Ġsentence Ġembed d ings Ġand Ġthe Ġgold Ġlabels . ĠThe Ġsetup Ġfor Ġthe Ġother Ġsentence Ġembed ding Ġmethods Ġis Ġequivalent , Ġthe Ġsimilarity Ġis Ġcomputed Ġby Ġcos ine - similar ity . ĠThe Ġresults Ġare Ġdepicted Ġin ĠTable ĠT AB REF 6 . </s> The Ġresults Ġshows Ġthat Ġdirectly Ġusing Ġthe Ġoutput Ġof ĠB ERT Ġleads Ġto Ġrather Ġpoor Ġperformances . ĠA ver aging Ġthe ĠB ERT Ġembed d ings Ġachieves Ġan Ġaverage Ġcorrelation Ġof Ġonly Ġ54 . 81 , Ġand Ġusing Ġthe ĠCLS - token Ġoutput Ġonly Ġachieves Ġan Ġaverage Ġcorrelation Ġof Ġ29 . 19 . ĠBoth Ġare Ġworse Ġthan Ġcomputing Ġaverage ĠGl o Ve Ġembed d ings . </s> Using Ġthe Ġdescribed Ġsi ames e Ġnetwork Ġstructure Ġand Ġfine - tun ing Ġmechanism Ġsubstantially Ġimproves Ġthe Ġcorrelation , Ġoutper forming Ġboth ĠIn fer Sent Ġand ĠUniversal ĠSent ence ĠEnc oder Ġsubstantially . ĠThe Ġonly Ġdataset Ġwhere ĠS BER T Ġperforms Ġworse Ġthan ĠUniversal ĠSent ence ĠEnc oder Ġis ĠS ICK - R . ĠUniversal ĠSent ence ĠEnc oder Ġwas Ġtrained Ġon Ġvarious Ġdatasets , Ġincluding Ġnews , Ġquestion - answer Ġpages Ġand Ġdiscussion Ġforums , Ġwhich Ġappears Ġto Ġbe Ġmore Ġsuitable Ġto Ġthe Ġdata Ġof ĠS ICK - R . ĠIn Ġcontrast , ĠS BER T Ġwas Ġpre - trained Ġonly Ġon ĠWikipedia Ġ( via ĠB ERT ) Ġand Ġon ĠNL I Ġdata . </s> While ĠRo BER Ta Ġwas Ġable Ġto Ġimprove Ġthe Ġperformance Ġfor Ġseveral Ġsupervised Ġtasks , Ġwe Ġonly Ġobserve Ġminor Ġdifference Ġbetween ĠS BER T Ġand ĠS Ro BER Ta Ġfor Ġgenerating Ġsentence Ġembed d ings . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity Ġ: :: ĠSuper vised ĠST S </s> The ĠST S Ġbenchmark Ġ( ST S b ) ĠB IB REF 10 Ġprovides Ġis Ġa Ġpopular Ġdataset Ġto Ġevaluate Ġsupervised ĠST S Ġsystems . ĠThe Ġdata Ġincludes Ġ8 , 628 Ġsentence Ġpairs Ġfrom Ġthe Ġthree Ġcategories Ġcapt ions , Ġnews , Ġand Ġforums . ĠIt Ġis Ġdivided Ġinto Ġtrain Ġ( 5 , 7 49 ), Ġdev Ġ( 1 , 500 ) Ġand Ġtest Ġ( 1 , 379 ). ĠB ERT Ġset Ġa Ġnew Ġstate - of - the - art Ġperformance Ġon Ġthis Ġdataset Ġby Ġpassing Ġboth Ġsentences Ġto Ġthe Ġnetwork Ġand Ġusing Ġa Ġsimple Ġregression Ġmethod Ġfor Ġthe Ġoutput . </s> We Ġuse Ġthe Ġtraining Ġset Ġto Ġfine - t une ĠS BER T Ġusing Ġthe Ġregression Ġobjective Ġfunction . ĠAt Ġprediction Ġtime , Ġwe Ġcompute Ġthe Ġcos ine - similar ity Ġbetween Ġthe Ġsentence Ġembed d ings . ĠAll Ġsystems Ġare Ġtrained Ġwith Ġ10 Ġrandom Ġseeds Ġto Ġcounter Ġvar iances ĠB IB REF 23 . </s> The Ġresults Ġare Ġdepicted Ġin ĠTable ĠT AB REF 10 . ĠWe Ġexperimented Ġwith Ġtwo Ġsetups : ĠOnly Ġtraining Ġon ĠST S b , Ġand Ġfirst Ġtraining Ġon ĠNL I , Ġthen Ġtraining Ġon ĠST S b . ĠWe Ġobserve Ġthat Ġthe Ġlater Ġstrategy Ġleads Ġto Ġa Ġslight Ġimprovement Ġof Ġ1 - 2 Ġpoints . ĠThis Ġtwo - step Ġapproach Ġhad Ġan Ġespecially Ġlarge Ġimpact Ġfor Ġthe ĠB ERT Ġcross - enc oder , Ġwhich Ġimproved Ġthe Ġperformance Ġby Ġ3 - 4 Ġpoints . ĠWe Ġdo Ġnot Ġobserve Ġa Ġsignificant Ġdifference Ġbetween ĠB ERT Ġand ĠRo BER Ta . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity Ġ: :: ĠArgument ĠFac et ĠSimilar ity </s> We Ġevaluate ĠS BER T Ġon Ġthe ĠArgument ĠFac et ĠSimilar ity Ġ( AF S ) Ġcorpus Ġby ĠMis ra EW 16 . ĠThe ĠA FS Ġcorpus Ġannot ated Ġ6 , 000 Ġsent ential Ġargument Ġpairs Ġfrom Ġsocial Ġmedia Ġdialog s Ġon Ġthree Ġcontroversial Ġtopics : Ġgun Ġcontrol , Ġgay Ġmarriage , Ġand Ġdeath Ġpenalty . ĠThe Ġdata Ġwas Ġannot ated Ġon Ġa Ġscale Ġfrom Ġ0 Ġ( âĢ ľ different Ġtopic ") Ġto Ġ5 Ġ( âĢ ľ completely Ġequivalent "). ĠThe Ġsimilarity Ġnotion Ġin Ġthe ĠA FS Ġcorpus Ġis Ġfairly Ġdifferent Ġto Ġthe Ġsimilarity Ġnotion Ġin Ġthe ĠST S Ġdatasets Ġfrom ĠSem E val . ĠST S Ġdata Ġis Ġusually Ġdescriptive , Ġwhile ĠA FS Ġdata Ġare Ġargument ative Ġexcerpts Ġfrom Ġdialog s . ĠTo Ġbe Ġconsidered Ġsimilar , Ġarguments Ġmust Ġnot Ġonly Ġmake Ġsimilar Ġclaims , Ġbut Ġalso Ġprovide Ġa Ġsimilar Ġreasoning . ĠFurther , Ġthe Ġlex ical Ġgap Ġbetween Ġthe Ġsentences Ġin ĠA FS Ġis Ġmuch Ġlarger . ĠHence , Ġsimple Ġun super vised Ġmethods Ġas Ġwell Ġas Ġstate - of - the - art ĠST S Ġsystems Ġperform Ġbadly Ġon Ġthis Ġdataset ĠB IB REF 24 . </s> We Ġevaluate ĠS BER T Ġon Ġthis Ġdataset Ġin Ġtwo Ġscenarios : Ġ1 ) ĠAs Ġproposed Ġby ĠMis ra Ġet Ġal ., Ġwe Ġevaluate ĠS BER T Ġusing Ġ10 - fold Ġcross - valid ation . ĠA Ġdraw - back Ġof Ġthis Ġevaluation Ġsetup Ġis Ġthat Ġit Ġis Ġnot Ġclear Ġhow Ġwell Ġapproaches Ġgeneral ize Ġto Ġdifferent Ġtopics . ĠHence , Ġ2 ) Ġwe Ġevaluate ĠS BER T Ġin Ġa Ġcross - topic Ġsetup . ĠTwo Ġtopics Ġserve Ġfor Ġtraining Ġand Ġthe Ġapproach Ġis Ġevaluated Ġon Ġthe Ġleft - out Ġtopic . ĠWe Ġrepeat Ġthis Ġfor Ġall Ġthree Ġtopics Ġand Ġaverage Ġthe Ġresults . </s> S BER T Ġis Ġfine - tun ed Ġusing Ġthe ĠReg ression ĠObjective ĠFunction . ĠThe Ġsimilarity Ġscore Ġis Ġcomputed Ġusing Ġcos ine - similar ity Ġbased Ġon Ġthe Ġsentence Ġembed d ings . ĠWe Ġalso Ġprovide Ġthe ĠPearson Ġcorrelation Ġ$ r $ Ġto Ġmake Ġthe Ġresults Ġcomparable Ġto ĠMis ra Ġet Ġal . ĠHowever , Ġwe Ġshowed ĠB IB REF 22 Ġthat ĠPearson Ġcorrelation Ġhas Ġsome Ġserious Ġdrawbacks Ġand Ġshould Ġbe Ġavoided Ġfor Ġcomparing ĠST S Ġsystems . ĠThe Ġresults Ġare Ġdepicted Ġin ĠTable ĠT AB REF 12 . </s> Un super vised Ġmethods Ġlike Ġtf - id f , Ġaverage ĠGl o Ve Ġembed d ings Ġor ĠIn fer Sent Ġperform Ġrather Ġbadly Ġon Ġthis Ġdataset Ġwith Ġlow Ġscores . ĠTraining ĠS BER T Ġin Ġthe Ġ10 - fold Ġcross - valid ation Ġsetup Ġgives Ġa Ġperformance Ġthat Ġis Ġnearly Ġon - par Ġwith ĠB ERT . </s> However , Ġin Ġthe Ġcross - topic Ġevaluation , Ġwe Ġobserve Ġa Ġperformance Ġdrop Ġof ĠS BER T Ġby Ġabout Ġ7 Ġpoints ĠSpear man Ġcorrelation . ĠTo Ġbe Ġconsidered Ġsimilar , Ġarguments Ġshould Ġaddress Ġthe Ġsame Ġclaims Ġand Ġprovide Ġthe Ġsame Ġreasoning . ĠB ERT Ġis Ġable Ġto Ġuse Ġattention Ġto Ġcompare Ġdirectly Ġboth Ġsentences Ġ( e . g . Ġword - by - word Ġcomparison ), Ġwhile ĠS BER T Ġmust Ġmap Ġindividual Ġsentences Ġfrom Ġan Ġunseen Ġtopic Ġto Ġa Ġvector Ġspace Ġsuch Ġthat Ġarguments Ġwith Ġsimilar Ġclaims Ġand Ġreasons Ġare Ġclose . ĠThis Ġis Ġa Ġmuch Ġmore Ġchallenging Ġtask , Ġwhich Ġappears Ġto Ġrequire Ġmore Ġthan Ġjust Ġtwo Ġtopics Ġfor Ġtraining Ġto Ġwork Ġon - par Ġwith ĠB ERT . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity Ġ: :: ĠWikipedia ĠSections ĠDist inction </s> e in - dor - et al - 2018 - learning Ġuse ĠWikipedia Ġto Ġcreate Ġa Ġthem atically Ġfine - gr ained Ġtrain , Ġdev Ġand Ġtest Ġset Ġfor Ġsentence Ġembed d ings Ġmethods . ĠWikipedia Ġarticles Ġare Ġseparated Ġinto Ġdistinct Ġsections Ġfocusing Ġon Ġcertain Ġaspects . ĠDor Ġet Ġal . Ġassume Ġthat Ġsentences Ġin Ġthe Ġsame Ġsection Ġare Ġthem atically Ġcloser Ġthan Ġsentences Ġin Ġdifferent Ġsections . ĠThey Ġuse Ġthis Ġto Ġcreate Ġa Ġlarge Ġdataset Ġof Ġweak ly Ġlabeled Ġsentence Ġtriple ts : ĠThe Ġanchor Ġand Ġthe Ġpositive Ġexample Ġcome Ġfrom Ġthe Ġsame Ġsection , Ġwhile Ġthe Ġnegative Ġexample Ġcomes Ġfrom Ġa Ġdifferent Ġsection Ġof Ġthe Ġsame Ġarticle . ĠFor Ġexample , Ġfrom Ġthe ĠAlice ĠArnold Ġarticle : ĠAnch or : ĠArnold Ġjoined Ġthe ĠBBC ĠRadio ĠDrama ĠCompany Ġin Ġ1988 ., Ġpositive : ĠArnold Ġgained Ġmedia Ġattention Ġin ĠMay Ġ2012 ., Ġnegative : ĠBald ing Ġand ĠArnold Ġare Ġkeen Ġamateur Ġgolf ers . </s> We Ġuse Ġthe Ġdataset Ġfrom ĠDor Ġet Ġal . ĠWe Ġuse Ġthe ĠTriple t ĠObjective , Ġtrain ĠS BER T Ġfor Ġone Ġepoch Ġon Ġthe Ġabout Ġ1 . 8 ĠMillion Ġtraining Ġtriple ts Ġand Ġevaluate Ġit Ġon Ġthe Ġ222 , 9 57 Ġtest Ġtriple ts . ĠTest Ġtriple ts Ġare Ġfrom Ġa Ġdistinct Ġset Ġof ĠWikipedia Ġarticles . ĠAs Ġevaluation Ġmetric , Ġwe Ġuse Ġaccuracy : ĠIs Ġthe Ġpositive Ġexample Ġcloser Ġto Ġthe Ġanchor Ġthan Ġthe Ġnegative Ġexample ? </s> Results Ġare Ġpresented Ġin ĠTable ĠT AB REF 14 . ĠDor Ġet Ġal . Ġfine - tun ed Ġa ĠBi L ST M Ġarchitecture Ġwith Ġtriple t Ġloss Ġto Ġderive Ġsentence Ġembed d ings Ġfor Ġthis Ġdataset . ĠAs Ġthe Ġtable Ġshows , ĠS BER T Ġclearly Ġoutper forms Ġthe ĠBi L ST M Ġapproach Ġby ĠDor Ġet Ġal . </s> E val uation Ġ- ĠSent E val </s> Sent E val ĠB IB REF 6 Ġis Ġa Ġpopular Ġtool kit Ġto Ġevaluate Ġthe Ġquality Ġof Ġsentence Ġembed d ings . ĠSent ence Ġembed d ings Ġare Ġused Ġas Ġfeatures Ġfor Ġa Ġlog istic Ġregression Ġclass ifier . ĠThe Ġlog istic Ġregression Ġclass ifier Ġis Ġtrained Ġon Ġvarious Ġtasks Ġin Ġa Ġ10 - fold Ġcross - valid ation Ġsetup Ġand Ġthe Ġprediction Ġaccuracy Ġis Ġcomputed Ġfor Ġthe Ġtest - fold . </s> The Ġpurpose Ġof ĠS BER T Ġsentence Ġembed d ings Ġare Ġnot Ġto Ġbe Ġused Ġfor Ġtransfer Ġlearning Ġfor Ġother Ġtasks . ĠHere , Ġwe Ġthink Ġfine - tun ing ĠB ERT Ġas Ġdescribed Ġby Ġdev lin 2018 bert Ġfor Ġnew Ġtasks Ġis Ġthe Ġmore Ġsuitable Ġmethod , Ġas Ġit Ġupdates Ġall Ġlayers Ġof Ġthe ĠB ERT Ġnetwork . ĠHowever , ĠSent E val Ġcan Ġstill Ġgive Ġan Ġimpression Ġon Ġthe Ġquality Ġof Ġour Ġsentence Ġembed d ings Ġfor Ġvarious Ġtasks . </s> We Ġcompare Ġthe ĠS BER T Ġsentence Ġembed d ings Ġto Ġother Ġsentence Ġembed d ings Ġmethods Ġon Ġthe Ġfollowing Ġseven ĠSent E val Ġtransfer Ġtasks : </s> MR : ĠSent iment Ġprediction Ġfor Ġmovie Ġreviews Ġsnippets Ġon Ġa Ġfive Ġstart Ġscale ĠB IB REF 25 . </s> CR : ĠSent iment Ġprediction Ġof Ġcustomer Ġproduct Ġreviews ĠB IB REF 26 . </s> S UB J : ĠSubject ivity Ġprediction Ġof Ġsentences Ġfrom Ġmovie Ġreviews Ġand Ġplot Ġsumm aries ĠB IB REF 27 . </s> MP Q A : ĠPh rase Ġlevel Ġopinion Ġpol arity Ġclassification Ġfrom Ġnews wire ĠB IB REF 28 . </s> S ST : ĠStanford ĠSent iment ĠTree bank Ġwith Ġbinary Ġlabels ĠB IB REF 29 . </s> T REC : ĠFine Ġgr ained Ġquestion - type Ġclassification Ġfrom ĠT REC ĠB IB REF 30 . </s> MR PC : ĠMicrosoft ĠResearch ĠPar aph rase ĠCorpus Ġfrom Ġparallel Ġnews Ġsources ĠB IB REF 31 . </s> The Ġresults Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 15 . ĠS BER T Ġis Ġable Ġto Ġachieve Ġthe Ġbest Ġperformance Ġin Ġ5 Ġout Ġof Ġ7 Ġtasks . ĠThe Ġaverage Ġperformance Ġincreases Ġby Ġabout Ġ2 Ġpercentage Ġpoints Ġcompared Ġto ĠIn fer Sent Ġas Ġwell Ġas Ġthe ĠUniversal ĠSent ence ĠEnc oder . ĠEven Ġthough Ġtransfer Ġlearning Ġis Ġnot Ġthe Ġpurpose Ġof ĠS BER T , Ġit Ġoutper forms Ġother Ġstate - of - the - art Ġsentence Ġembed d ings Ġmethods Ġon Ġthis Ġtask . </s> It Ġappears Ġthat Ġthe Ġsentence Ġembed d ings Ġfrom ĠS BER T Ġcapture Ġwell Ġsentiment Ġinformation : ĠWe Ġobserve Ġlarge Ġimprovements Ġfor Ġall Ġsentiment Ġtasks Ġ( MR , ĠCR , Ġand ĠS ST ) Ġfrom ĠSent E val Ġin Ġcomparison Ġto ĠIn fer Sent Ġand ĠUniversal ĠSent ence ĠEnc oder . </s> The Ġonly Ġdataset Ġwhere ĠS BER T Ġis Ġsignificantly Ġworse Ġthan ĠUniversal ĠSent ence ĠEnc oder Ġis Ġthe ĠT REC Ġdataset . ĠUniversal ĠSent ence ĠEnc oder Ġwas Ġpre - trained Ġon Ġquestion - ans w ering Ġdata , Ġwhich Ġappears Ġto Ġbe Ġbeneficial Ġfor Ġthe Ġquestion - type Ġclassification Ġtask Ġof Ġthe ĠT REC Ġdataset . </s> Average ĠB ERT Ġembed d ings Ġor Ġusing Ġthe ĠCLS - token Ġoutput Ġfrom Ġa ĠB ERT Ġnetwork Ġachieved Ġbad Ġresults Ġfor Ġvarious ĠST S Ġtasks Ġ( Table ĠT AB REF 6 ), Ġworse Ġthan Ġaverage ĠGl o Ve Ġembed d ings . ĠHowever , Ġfor ĠSent E val , Ġaverage ĠB ERT Ġembed d ings Ġand Ġthe ĠB ERT ĠCLS - token Ġoutput Ġachieves Ġdecent Ġresults Ġ( Table ĠT AB REF 15 ), Ġoutper forming Ġaverage ĠGl o Ve Ġembed d ings . ĠThe Ġreason Ġfor Ġthis Ġare Ġthe Ġdifferent Ġsetups . ĠFor Ġthe ĠST S Ġtasks , Ġwe Ġused Ġcos ine - similar ity Ġto Ġestimate Ġthe Ġsimilarities Ġbetween Ġsentence Ġembed d ings . ĠCos ine - similar ity Ġtreats Ġall Ġdimensions Ġequally . ĠIn Ġcontrast , ĠSent E val Ġfits Ġa Ġlog istic Ġregression Ġclass ifier Ġto Ġthe Ġsentence Ġembed d ings . ĠThis Ġallows Ġthat Ġcertain Ġdimensions Ġcan Ġhave Ġhigher Ġor Ġlower Ġimpact Ġon Ġthe Ġclassification Ġresult . </s> We Ġconclude Ġthat Ġaverage ĠB ERT Ġembed d ings Ġ/ ĠCLS - token Ġoutput Ġfrom ĠB ERT Ġreturn Ġsentence Ġembed d ings Ġthat Ġare Ġinf eas ible Ġto Ġbe Ġused Ġwith Ġcos ine - similar ity Ġor Ġwith ĠMan hatt en Ġ/ ĠEucl ide an Ġdistance . ĠFor Ġtransfer Ġlearning , Ġthey Ġyield Ġslightly Ġworse Ġresults Ġthan ĠIn fer Sent Ġor ĠUniversal ĠSent ence ĠEnc oder . ĠHowever , Ġusing Ġthe Ġdescribed Ġfine - tun ing Ġsetup Ġwith Ġa Ġsi ames e Ġnetwork Ġstructure Ġon ĠNL I Ġdatasets Ġyields Ġsentence Ġembed d ings Ġthat Ġachieve Ġa Ġnew Ġstate - of - the - art Ġfor Ġthe ĠSent E val Ġtool kit . </s> A bl ation ĠStudy </s> We Ġhave Ġdemonstrated Ġstrong Ġempirical Ġresults Ġfor Ġthe Ġquality Ġof ĠS BER T Ġsentence Ġembed d ings . ĠIn Ġthis Ġsection , Ġwe Ġperform Ġan Ġab lation Ġstudy Ġof Ġdifferent Ġaspects Ġof ĠS BER T Ġin Ġorder Ġto Ġget Ġa Ġbetter Ġunderstanding Ġof Ġtheir Ġrelative Ġimportance . </s> We Ġevaluated Ġdifferent Ġpool ing Ġstrategies Ġ( ME AN , ĠMAX , Ġand ĠCLS ). ĠFor Ġthe Ġclassification Ġobjective Ġfunction , Ġwe Ġevaluate Ġdifferent Ġconc aten ation Ġmethods . ĠFor Ġeach Ġpossible Ġconfiguration , Ġwe Ġtrain ĠS BER T Ġwith Ġ10 Ġdifferent Ġrandom Ġseeds Ġand Ġaverage Ġthe Ġperformances . </s> The Ġobjective Ġfunction Ġ( class ification Ġvs . Ġregression ) Ġdepends Ġon Ġthe Ġannot ated Ġdataset . ĠFor Ġthe Ġclassification Ġobjective Ġfunction , Ġwe Ġtrain ĠS BER T - base Ġon Ġthe ĠSN LI Ġand Ġthe ĠMulti - N LI Ġdataset . ĠFor Ġthe Ġregression Ġobjective Ġfunction , Ġwe Ġtrain Ġon Ġthe Ġtraining Ġset Ġof Ġthe ĠST S Ġbenchmark Ġdataset . ĠPerform ances Ġare Ġmeasured Ġon Ġthe Ġdevelopment Ġsplit Ġof Ġthe ĠST S Ġbenchmark Ġdataset . ĠResults Ġare Ġshown Ġin ĠTable ĠT AB REF 23 . </s> When Ġtrained Ġwith Ġthe Ġclassification Ġobjective Ġfunction Ġon ĠNL I Ġdata , Ġthe Ġpool ing Ġstrategy Ġhas Ġa Ġrather Ġminor Ġimpact . ĠThe Ġimpact Ġof Ġthe Ġconc aten ation Ġmode Ġis Ġmuch Ġlarger . ĠIn fer Sent ĠB IB REF 4 Ġand ĠUniversal ĠSent ence ĠEnc oder ĠB IB REF 5 Ġboth Ġuse Ġ$( u , Ġv , Ġ| u - v | , Ġu * v ) $ Ġas Ġinput Ġfor Ġa Ġsoft max Ġclass ifier . ĠHowever , Ġin Ġour Ġarchitecture , Ġadding Ġthe Ġelement - wise Ġ$ u * v $ Ġdecreased Ġthe Ġperformance . </s> The Ġmost Ġimportant Ġcomponent Ġis Ġthe Ġelement - wise Ġdifference Ġ$ | u - v | $. ĠNote , Ġthat Ġthe Ġconc aten ation Ġmode Ġis Ġonly Ġrelevant Ġfor Ġtraining Ġthe Ġsoft max Ġclass ifier . ĠAt Ġinference , Ġwhen Ġpredicting Ġsimilarities Ġfor Ġthe ĠST S Ġbenchmark Ġdataset , Ġonly Ġthe Ġsentence Ġembed d ings Ġ$ u $ Ġand Ġ$ v $ Ġare Ġused Ġin Ġcombination Ġwith Ġcos ine - similar ity . ĠThe Ġelement - wise Ġdifference Ġmeasures Ġthe Ġdistance Ġbetween Ġthe Ġdimensions Ġof Ġthe Ġtwo Ġsentence Ġembed d ings , Ġensuring Ġthat Ġsimilar Ġpairs Ġare Ġcloser Ġand Ġdiss imilar Ġpairs Ġare Ġfurther Ġapart . </s> When Ġtrained Ġwith Ġthe Ġregression Ġobjective Ġfunction , Ġwe Ġobserve Ġthat Ġthe Ġpool ing Ġstrategy Ġhas Ġa Ġlarge Ġimpact . ĠThere , Ġthe ĠMAX Ġstrategy Ġperform Ġsignificantly Ġworse Ġthan ĠME AN Ġor ĠCLS - token Ġstrategy . ĠThis Ġis Ġin Ġcontrast Ġto ĠB IB REF 4 , Ġwho Ġfound Ġit Ġbeneficial Ġfor Ġthe ĠBi L ST M - layer Ġof ĠIn fer Sent Ġto Ġuse ĠMAX Ġinstead Ġof ĠME AN Ġpool ing . </s> Com put ational ĠEfficiency </s> Sent ence Ġembed d ings Ġneed Ġpotentially Ġbe Ġcomputed Ġfor ĠMillions Ġof Ġsentences , Ġhence , Ġa Ġhigh Ġcomputation Ġspeed Ġis Ġdesired . ĠIn Ġthis Ġsection , Ġwe Ġcompare ĠS BER T Ġto Ġaverage ĠGl o Ve Ġembed d ings , ĠIn fer Sent ĠB IB REF 4 , Ġand ĠUniversal ĠSent ence ĠEnc oder ĠB IB REF 5 . </s> For Ġour Ġcomparison Ġwe Ġuse Ġthe Ġsentences Ġfrom Ġthe ĠST S Ġbenchmark ĠB IB REF 10 . ĠWe Ġcompute Ġaverage ĠGl o Ve Ġembed d ings Ġusing Ġa Ġsimple Ġfor - loop Ġwith Ġpython Ġdictionary Ġlook ups Ġand ĠNum Py . ĠIn fer Sent Ġis Ġbased Ġon ĠPy Tor ch . ĠFor ĠUniversal ĠSent ence ĠEnc oder , Ġwe Ġuse Ġthe ĠT ensor Flow ĠHub Ġversion , Ġwhich Ġis Ġbased Ġon ĠT ensor Flow . ĠS BER T Ġis Ġbased Ġon ĠPy Tor ch . ĠFor Ġimproved Ġcomputation Ġof Ġsentence Ġembed d ings , Ġwe Ġimplemented Ġa Ġsmart Ġbatch ing Ġstrategy : ĠSent ences Ġwith Ġsimilar Ġlengths Ġare Ġgrouped Ġtogether Ġand Ġare Ġonly Ġpadded Ġto Ġthe Ġlongest Ġelement Ġin Ġa Ġmini - batch . ĠThis Ġdrastically Ġreduces Ġcomputational Ġoverhead Ġfrom Ġpadding Ġtokens . </s> Per form ances Ġwere Ġmeasured Ġon Ġa Ġserver Ġwith ĠIntel Ġi 7 - 58 20 K ĠCPU Ġ@ Ġ3 . 30 GHz , ĠNvidia ĠTesla ĠV 100 ĠGPU , ĠCU DA Ġ9 . 2 Ġand Ġcu D NN . ĠThe Ġresults Ġare Ġdepicted Ġin ĠTable ĠT AB REF 26 . </s> On ĠCPU , ĠIn fer Sent Ġis Ġabout Ġ65 % Ġfaster Ġthan ĠS BER T . ĠThis Ġis Ġdue Ġto Ġthe Ġmuch Ġsimpler Ġnetwork Ġarchitecture . ĠIn fer Sent Ġuses Ġa Ġsingle ĠBi L ST M Ġlayer , Ġwhile ĠB ERT Ġuses Ġ12 Ġstacked Ġtransformer Ġlayers . ĠHowever , Ġan Ġadvantage Ġof Ġtransformer Ġnetworks Ġis Ġthe Ġcomputational Ġefficiency Ġon ĠGPUs . ĠThere , ĠS BER T Ġwith Ġsmart Ġbatch ing Ġis Ġabout Ġ9 % Ġfaster Ġthan ĠIn fer Sent Ġand Ġabout Ġ55 % Ġfaster Ġthan ĠUniversal ĠSent ence ĠEnc oder . ĠSmart Ġbatch ing Ġachieves Ġa Ġspeed - up Ġof Ġ89 % Ġon ĠCPU Ġand Ġ48 % Ġon ĠGPU . ĠAverage ĠGl o Ve Ġembed d ings Ġis Ġobviously Ġby Ġa Ġlarge Ġmargin Ġthe Ġfastest Ġmethod Ġto Ġcompute Ġsentence Ġembed d ings . </s> Conclusion </s> We Ġshowed Ġthat ĠB ERT Ġout - of - the - box Ġmaps Ġsentences Ġto Ġa Ġvector Ġspace Ġthat Ġis Ġrather Ġunsu itable Ġto Ġbe Ġused Ġwith Ġcommon Ġsimilarity Ġmeasures Ġlike Ġcos ine - similar ity . ĠThe Ġperformance Ġfor Ġseven ĠST S Ġtasks Ġwas Ġbelow Ġthe Ġperformance Ġof Ġaverage ĠGl o Ve Ġembed d ings . </s> To Ġovercome Ġthis Ġshort coming , Ġwe Ġpresented ĠSent ence - BER T Ġ( S BER T ). ĠS BER T Ġfine - t unes ĠB ERT Ġin Ġa Ġsi ames e Ġ/ Ġtriple t Ġnetwork Ġarchitecture . ĠWe Ġevaluated Ġthe Ġquality Ġon Ġvarious Ġcommon Ġbenchmarks , Ġwhere Ġit Ġcould Ġachieve Ġa Ġsignificant Ġimprovement Ġover Ġstate - of - the - art Ġsentence Ġembed d ings Ġmethods . ĠRepl acing ĠB ERT Ġwith ĠRo BER Ta Ġdid Ġnot Ġyield Ġa Ġsignificant Ġimprovement Ġin Ġour Ġexperiments . </s> S BER T Ġis Ġcomput ationally Ġefficient . ĠOn Ġa ĠGPU , Ġit Ġis Ġabout Ġ9 % Ġfaster Ġthan ĠIn fer Sent Ġand Ġabout Ġ55 % Ġfaster Ġthan ĠUniversal ĠSent ence ĠEnc oder . ĠS BER T Ġcan Ġbe Ġused Ġfor Ġtasks Ġwhich Ġare Ġcomput ationally Ġnot Ġfeasible Ġto Ġbe Ġmodeled Ġwith ĠB ERT . ĠFor Ġexample , Ġclust ering Ġof Ġ10 , 000 Ġsentences Ġwith Ġhierarchical Ġclust ering Ġrequires Ġwith ĠB ERT Ġabout Ġ65 Ġhours , Ġas Ġaround Ġ50 ĠMillion Ġsentence Ġcombinations Ġmust Ġbe Ġcomputed . ĠWith ĠS BER T , Ġwe Ġwere Ġable Ġto Ġreduce Ġthe Ġeffort Ġto Ġabout Ġ5 Ġseconds . </s> Acknowled gments </s> This Ġwork Ġhas Ġbeen Ġsupported Ġby Ġthe ĠGerman ĠResearch ĠFoundation Ġthrough Ġthe ĠGerman - Israeli ĠProject ĠCooperation Ġ( D IP , Ġgrant ĠDA Ġ1600 / 1 - 1 Ġand Ġgrant ĠGU Ġ7 98 / 17 - 1 ). ĠIt Ġhas Ġbeen Ġco - funded Ġby Ġthe ĠGerman ĠFederal ĠMinistry Ġof ĠEducation Ġand ĠResearch Ġ( B MB F ) Ġunder Ġthe Ġpromotional Ġreferences Ġ03 VP 025 40 Ġ( Arg umen Text ).
Introduction In this publication, we present Sentence-BERT (SBERT), a modification of the BERT network using siamese and triplet networks that is able to derive semantically meaningful sentence embeddings. This enables BERT to be used for certain new tasks, which up-to-now were not applicable for BERT. These tasks include large-scale semantic similarity comparison, clustering, and information retrieval via semantic search. BERT set new state-of-the-art performance on various sentence classification and sentence-pair regression tasks. BERT uses a cross-encoder: Two sentences are passed to the transformer network and the target value is predicted. However, this setup is unsuitable for various pair regression tasks due to too many possible combinations. Finding in a collection of $n=10\,000$ sentences the pair with the highest similarity requires with BERT $n\cdot (n-1)/2=49\,995\,000$ inference computations. On a modern V100 GPU, this requires about 65 hours. Similar, finding which of the over 40 million existent questions of Quora is the most similar for a new question could be modeled as a pair-wise comparison with BERT, however, answering a single query would require over 50 hours. A common method to address clustering and semantic search is to map each sentence to a vector space such that semantically similar sentences are close. Researchers have started to input individual sentences into BERT and to derive fixed-size sentence embeddings. The most commonly used approach is to average the BERT output layer (known as BERT embeddings) or by using the output of the first token (the [CLS] token). As we will show, this common practice yields rather bad sentence embeddings, often worse than averaging GloVe embeddings BIBREF2. To alleviate this issue, we developed SBERT. The siamese network architecture enables that fixed-sized vectors for input sentences can be derived. Using a similarity measure like cosine-similarity or Manhatten / Euclidean distance, semantically similar sentences can be found. These similarity measures can be performed extremely efficient on modern hardware, allowing SBERT to be used for semantic similarity search as well as for clustering. The complexity for finding the most similar sentence pair in a collection of 10,000 sentences is reduced from 65 hours with BERT to the computation of 10,000 sentence embeddings (5 seconds with SBERT) and computing cosine-similarity (0.01 seconds). By using optimized index structures, finding the most similar Quora question can be reduced from 50 hours to a few milliseconds BIBREF3. We fine-tune SBERT on NLI data, which creates sentence embeddings that significantly outperform other state-of-the-art sentence embedding methods like InferSent BIBREF4 and Universal Sentence Encoder BIBREF5. On seven Semantic Textual Similarity (STS) tasks, SBERT achieves an improvement of 11.7 points compared to InferSent and 5.5 points compared to Universal Sentence Encoder. On SentEval BIBREF6, an evaluation toolkit for sentence embeddings, we achieve an improvement of 2.1 and 2.6 points, respectively. SBERT can be adapted to a specific task. It sets new state-of-the-art performance on a challenging argument similarity dataset BIBREF7 and on a triplet dataset to distinguish sentences from different sections of a Wikipedia article BIBREF8. The paper is structured in the following way: Section SECREF3 presents SBERT, section SECREF4 evaluates SBERT on common STS tasks and on the challenging Argument Facet Similarity (AFS) corpus BIBREF7. Section SECREF5 evaluates SBERT on SentEval. In section SECREF6, we perform an ablation study to test some design aspect of SBERT. In section SECREF7, we compare the computational efficiency of SBERT sentence embeddings in contrast to other state-of-the-art sentence embedding methods. Related Work We first introduce BERT, then, we discuss state-of-the-art sentence embedding methods. BERT BIBREF0 is a pre-trained transformer network BIBREF9, which set for various NLP tasks new state-of-the-art results, including question answering, sentence classification, and sentence-pair regression. The input for BERT for sentence-pair regression consists of the two sentences, separated by a special [SEP] token. Multi-head attention over 12 (base-model) or 24 layers (large-model) is applied and the output is passed to a simple regression function to derive the final label. Using this setup, BERT set a new state-of-the-art performance on the Semantic Textual Semilarity (STS) benchmark BIBREF10. RoBERTa BIBREF1 showed, that the performance of BERT can further improved by small adaptations to the pre-training process. We also tested XLNet BIBREF11, but it led in general to worse results than BERT. A large disadvantage of the BERT network structure is that no independent sentence embeddings are computed, which makes it difficult to derive sentence embeddings from BERT. To bypass this limitations, researchers passed single sentences through BERT and then derive a fixed sized vector by either averaging the outputs (similar to average word embeddings) or by using the output of the special CLS token (for example: bertsentenceembeddings1,bertsentenceembeddings2,bertsentenceembeddings3). These two options are also provided by the popular bert-as-a-service-repository. Up to our knowledge, there is so far no evaluation if these methods lead to useful sentence embeddings. Sentence embeddings are a well studied area with dozens of proposed methods. Skip-Thought BIBREF12 trains an encoder-decoder architecture to predict the surrounding sentences. InferSent BIBREF4 uses labeled data of the Stanford Natural Language Inference dataset BIBREF13 and the Multi-Genre NLI dataset BIBREF14 to train a siamese BiLSTM network with max-pooling over the output. Conneau et al. showed, that InferSent consistently outperforms unsupervised methods like SkipThought. Universal Sentence Encoder BIBREF5 trains a transformer network and augments unsupervised learning with training on SNLI. hill-etal-2016-learning showed, that the task on which sentence embeddings are trained significantly impacts their quality. Previous work BIBREF4, BIBREF5 found that the SNLI datasets are suitable for training sentence embeddings. yang-2018-learning presented a method to train on conversations from Reddit using siamese DAN and siamese transformer networks, which yielded good results on the STS benchmark dataset. polyencoders addresses the run-time overhead of the cross-encoder from BERT and present a method (poly-encoders) to compute a score between $m$ context vectors and pre-computed candidate embeddings using attention. This idea works for finding the highest scoring sentence in a larger collection. However, poly-encoders have the drawback that the score function is not symmetric and the computational overhead is too large for use-cases like clustering, which would require $O(n^2)$ score computations. Previous neural sentence embedding methods started the training from a random initialization. In this publication, we use the pre-trained BERT and RoBERTa network and only fine-tune it to yield useful sentence embeddings. This reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods. Model SBERT adds a pooling operation to the output of BERT / RoBERTa to derive a fixed sized sentence embedding. We experiment with three pooling strategies: Using the output of the CLS-token, computing the mean of all output vectors (MEAN-strategy), and computing a max-over-time of the output vectors (MAX-strategy). The default configuration is MEAN. In order to fine-tune BERT / RoBERTa, we create siamese and triplet networks BIBREF15 to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity. The network structure depends on the available training data. We experiment with the following structures and objective functions. Classification Objective Function. We concatenate the sentence embeddings $u$ and $v$ with the element-wise difference $|u-v|$ and multiply it with the trainable weight $W_t \in \mathbb {R}^{3n \times k}$: where $n$ is the dimension of the sentence embeddings and $k$ the number of labels. We optimize cross-entropy loss. This structure is depicted in Figure FIGREF4. Regression Objective Function. The cosine-similarity between the two sentence embeddings $u$ and $v$ is computed (Figure FIGREF5). We use mean-squared-error loss as the objective function. Triplet Objective Function. Given an anchor sentence $a$, a positive sentence $p$, and a negative sentence $n$, triplet loss tunes the network such that the distance between $a$ and $p$ is smaller than the distance between $a$ and $n$. Mathematically, we minimize the following loss function: with $s_x$ the sentence embedding for $a$/$n$/$p$, $||\cdot ||$ a distance metric and margin $\epsilon $. Margin $\epsilon $ ensures that $s_p$ is at least $\epsilon $ closer to $s_a$ than $s_n$. As metric we use Euclidean distance and we set $\epsilon =1$ in our experiments. Model ::: Training Details We train SBERT on the combination of the SNLI BIBREF13 and the Multi-Genre NLI BIBREF14 dataset. The SNLI is a collection of 570,000 sentence pairs annotated with the labels contradiction, eintailment, and neutral. MultiNLI contains 430,000 sentence pairs and covers a range of genres of spoken and written text. We fine-tune SBERT with a 3-way softmax-classifier objective function for one epoch. We used a batch-size of 16, Adam optimizer with learning rate $2\mathrm {e}{-5}$, and a linear learning rate warm-up over 10% of the training data. Our default pooling strategy is MEAN. Evaluation - Semantic Textual Similarity We evaluate the performance of SBERT for common Semantic Textual Similarity (STS) tasks. State-of-the-art methods often learn a (complex) regression function that maps sentence embeddings to a similarity score. However, these regression functions work pair-wise and due to the combinatorial explosion those are often not scalable if the collection of sentences reaches a certain size. Instead, we always use cosine-similarity to compare the similarity between two sentence embeddings. We ran our experiments also with negative Manhatten and negative Euclidean distances as similarity measures, but the results for all approaches remained roughly the same. Evaluation - Semantic Textual Similarity ::: Unsupervised STS We evaluate the performance of SBERT for STS without using any STS specific training data. We use the STS tasks 2012 - 2016 BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF20, the STS benchmark BIBREF10, and the SICK-Relatedness dataset BIBREF21. These datasets provide labels between 0 and 5 on the semantic relatedness of sentence pairs. We showed in BIBREF22 that Pearson correlation is badly suited for STS. Instead, we compute the Spearman's rank correlation between the cosine-similarity of the sentence embeddings and the gold labels. The setup for the other sentence embedding methods is equivalent, the similarity is computed by cosine-similarity. The results are depicted in Table TABREF6. The results shows that directly using the output of BERT leads to rather poor performances. Averaging the BERT embeddings achieves an average correlation of only 54.81, and using the CLS-token output only achieves an average correlation of 29.19. Both are worse than computing average GloVe embeddings. Using the described siamese network structure and fine-tuning mechanism substantially improves the correlation, outperforming both InferSent and Universal Sentence Encoder substantially. The only dataset where SBERT performs worse than Universal Sentence Encoder is SICK-R. Universal Sentence Encoder was trained on various datasets, including news, question-answer pages and discussion forums, which appears to be more suitable to the data of SICK-R. In contrast, SBERT was pre-trained only on Wikipedia (via BERT) and on NLI data. While RoBERTa was able to improve the performance for several supervised tasks, we only observe minor difference between SBERT and SRoBERTa for generating sentence embeddings. Evaluation - Semantic Textual Similarity ::: Supervised STS The STS benchmark (STSb) BIBREF10 provides is a popular dataset to evaluate supervised STS systems. The data includes 8,628 sentence pairs from the three categories captions, news, and forums. It is divided into train (5,749), dev (1,500) and test (1,379). BERT set a new state-of-the-art performance on this dataset by passing both sentences to the network and using a simple regression method for the output. We use the training set to fine-tune SBERT using the regression objective function. At prediction time, we compute the cosine-similarity between the sentence embeddings. All systems are trained with 10 random seeds to counter variances BIBREF23. The results are depicted in Table TABREF10. We experimented with two setups: Only training on STSb, and first training on NLI, then training on STSb. We observe that the later strategy leads to a slight improvement of 1-2 points. This two-step approach had an especially large impact for the BERT cross-encoder, which improved the performance by 3-4 points. We do not observe a significant difference between BERT and RoBERTa. Evaluation - Semantic Textual Similarity ::: Argument Facet Similarity We evaluate SBERT on the Argument Facet Similarity (AFS) corpus by MisraEW16. The AFS corpus annotated 6,000 sentential argument pairs from social media dialogs on three controversial topics: gun control, gay marriage, and death penalty. The data was annotated on a scale from 0 (“different topic") to 5 (“completely equivalent"). The similarity notion in the AFS corpus is fairly different to the similarity notion in the STS datasets from SemEval. STS data is usually descriptive, while AFS data are argumentative excerpts from dialogs. To be considered similar, arguments must not only make similar claims, but also provide a similar reasoning. Further, the lexical gap between the sentences in AFS is much larger. Hence, simple unsupervised methods as well as state-of-the-art STS systems perform badly on this dataset BIBREF24. We evaluate SBERT on this dataset in two scenarios: 1) As proposed by Misra et al., we evaluate SBERT using 10-fold cross-validation. A draw-back of this evaluation setup is that it is not clear how well approaches generalize to different topics. Hence, 2) we evaluate SBERT in a cross-topic setup. Two topics serve for training and the approach is evaluated on the left-out topic. We repeat this for all three topics and average the results. SBERT is fine-tuned using the Regression Objective Function. The similarity score is computed using cosine-similarity based on the sentence embeddings. We also provide the Pearson correlation $r$ to make the results comparable to Misra et al. However, we showed BIBREF22 that Pearson correlation has some serious drawbacks and should be avoided for comparing STS systems. The results are depicted in Table TABREF12. Unsupervised methods like tf-idf, average GloVe embeddings or InferSent perform rather badly on this dataset with low scores. Training SBERT in the 10-fold cross-validation setup gives a performance that is nearly on-par with BERT. However, in the cross-topic evaluation, we observe a performance drop of SBERT by about 7 points Spearman correlation. To be considered similar, arguments should address the same claims and provide the same reasoning. BERT is able to use attention to compare directly both sentences (e.g. word-by-word comparison), while SBERT must map individual sentences from an unseen topic to a vector space such that arguments with similar claims and reasons are close. This is a much more challenging task, which appears to require more than just two topics for training to work on-par with BERT. Evaluation - Semantic Textual Similarity ::: Wikipedia Sections Distinction ein-dor-etal-2018-learning use Wikipedia to create a thematically fine-grained train, dev and test set for sentence embeddings methods. Wikipedia articles are separated into distinct sections focusing on certain aspects. Dor et al. assume that sentences in the same section are thematically closer than sentences in different sections. They use this to create a large dataset of weakly labeled sentence triplets: The anchor and the positive example come from the same section, while the negative example comes from a different section of the same article. For example, from the Alice Arnold article: Anchor: Arnold joined the BBC Radio Drama Company in 1988., positive: Arnold gained media attention in May 2012., negative: Balding and Arnold are keen amateur golfers. We use the dataset from Dor et al. We use the Triplet Objective, train SBERT for one epoch on the about 1.8 Million training triplets and evaluate it on the 222,957 test triplets. Test triplets are from a distinct set of Wikipedia articles. As evaluation metric, we use accuracy: Is the positive example closer to the anchor than the negative example? Results are presented in Table TABREF14. Dor et al. fine-tuned a BiLSTM architecture with triplet loss to derive sentence embeddings for this dataset. As the table shows, SBERT clearly outperforms the BiLSTM approach by Dor et al. Evaluation - SentEval SentEval BIBREF6 is a popular toolkit to evaluate the quality of sentence embeddings. Sentence embeddings are used as features for a logistic regression classifier. The logistic regression classifier is trained on various tasks in a 10-fold cross-validation setup and the prediction accuracy is computed for the test-fold. The purpose of SBERT sentence embeddings are not to be used for transfer learning for other tasks. Here, we think fine-tuning BERT as described by devlin2018bert for new tasks is the more suitable method, as it updates all layers of the BERT network. However, SentEval can still give an impression on the quality of our sentence embeddings for various tasks. We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks: MR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25. CR: Sentiment prediction of customer product reviews BIBREF26. SUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27. MPQA: Phrase level opinion polarity classification from newswire BIBREF28. SST: Stanford Sentiment Treebank with binary labels BIBREF29. TREC: Fine grained question-type classification from TREC BIBREF30. MRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31. The results can be found in Table TABREF15. SBERT is able to achieve the best performance in 5 out of 7 tasks. The average performance increases by about 2 percentage points compared to InferSent as well as the Universal Sentence Encoder. Even though transfer learning is not the purpose of SBERT, it outperforms other state-of-the-art sentence embeddings methods on this task. It appears that the sentence embeddings from SBERT capture well sentiment information: We observe large improvements for all sentiment tasks (MR, CR, and SST) from SentEval in comparison to InferSent and Universal Sentence Encoder. The only dataset where SBERT is significantly worse than Universal Sentence Encoder is the TREC dataset. Universal Sentence Encoder was pre-trained on question-answering data, which appears to be beneficial for the question-type classification task of the TREC dataset. Average BERT embeddings or using the CLS-token output from a BERT network achieved bad results for various STS tasks (Table TABREF6), worse than average GloVe embeddings. However, for SentEval, average BERT embeddings and the BERT CLS-token output achieves decent results (Table TABREF15), outperforming average GloVe embeddings. The reason for this are the different setups. For the STS tasks, we used cosine-similarity to estimate the similarities between sentence embeddings. Cosine-similarity treats all dimensions equally. In contrast, SentEval fits a logistic regression classifier to the sentence embeddings. This allows that certain dimensions can have higher or lower impact on the classification result. We conclude that average BERT embeddings / CLS-token output from BERT return sentence embeddings that are infeasible to be used with cosine-similarity or with Manhatten / Euclidean distance. For transfer learning, they yield slightly worse results than InferSent or Universal Sentence Encoder. However, using the described fine-tuning setup with a siamese network structure on NLI datasets yields sentence embeddings that achieve a new state-of-the-art for the SentEval toolkit. Ablation Study We have demonstrated strong empirical results for the quality of SBERT sentence embeddings. In this section, we perform an ablation study of different aspects of SBERT in order to get a better understanding of their relative importance. We evaluated different pooling strategies (MEAN, MAX, and CLS). For the classification objective function, we evaluate different concatenation methods. For each possible configuration, we train SBERT with 10 different random seeds and average the performances. The objective function (classification vs. regression) depends on the annotated dataset. For the classification objective function, we train SBERT-base on the SNLI and the Multi-NLI dataset. For the regression objective function, we train on the training set of the STS benchmark dataset. Performances are measured on the development split of the STS benchmark dataset. Results are shown in Table TABREF23. When trained with the classification objective function on NLI data, the pooling strategy has a rather minor impact. The impact of the concatenation mode is much larger. InferSent BIBREF4 and Universal Sentence Encoder BIBREF5 both use $(u, v, |u-v|, u*v)$ as input for a softmax classifier. However, in our architecture, adding the element-wise $u*v$ decreased the performance. The most important component is the element-wise difference $|u-v|$. Note, that the concatenation mode is only relevant for training the softmax classifier. At inference, when predicting similarities for the STS benchmark dataset, only the sentence embeddings $u$ and $v$ are used in combination with cosine-similarity. The element-wise difference measures the distance between the dimensions of the two sentence embeddings, ensuring that similar pairs are closer and dissimilar pairs are further apart. When trained with the regression objective function, we observe that the pooling strategy has a large impact. There, the MAX strategy perform significantly worse than MEAN or CLS-token strategy. This is in contrast to BIBREF4, who found it beneficial for the BiLSTM-layer of InferSent to use MAX instead of MEAN pooling. Computational Efficiency Sentence embeddings need potentially be computed for Millions of sentences, hence, a high computation speed is desired. In this section, we compare SBERT to average GloVe embeddings, InferSent BIBREF4, and Universal Sentence Encoder BIBREF5. For our comparison we use the sentences from the STS benchmark BIBREF10. We compute average GloVe embeddings using a simple for-loop with python dictionary lookups and NumPy. InferSent is based on PyTorch. For Universal Sentence Encoder, we use the TensorFlow Hub version, which is based on TensorFlow. SBERT is based on PyTorch. For improved computation of sentence embeddings, we implemented a smart batching strategy: Sentences with similar lengths are grouped together and are only padded to the longest element in a mini-batch. This drastically reduces computational overhead from padding tokens. Performances were measured on a server with Intel i7-5820K CPU @ 3.30GHz, Nvidia Tesla V100 GPU, CUDA 9.2 and cuDNN. The results are depicted in Table TABREF26. On CPU, InferSent is about 65% faster than SBERT. This is due to the much simpler network architecture. InferSent uses a single BiLSTM layer, while BERT uses 12 stacked transformer layers. However, an advantage of transformer networks is the computational efficiency on GPUs. There, SBERT with smart batching is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder. Smart batching achieves a speed-up of 89% on CPU and 48% on GPU. Average GloVe embeddings is obviously by a large margin the fastest method to compute sentence embeddings. Conclusion We showed that BERT out-of-the-box maps sentences to a vector space that is rather unsuitable to be used with common similarity measures like cosine-similarity. The performance for seven STS tasks was below the performance of average GloVe embeddings. To overcome this shortcoming, we presented Sentence-BERT (SBERT). SBERT fine-tunes BERT in a siamese / triplet network architecture. We evaluated the quality on various common benchmarks, where it could achieve a significant improvement over state-of-the-art sentence embeddings methods. Replacing BERT with RoBERTa did not yield a significant improvement in our experiments. SBERT is computationally efficient. On a GPU, it is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder. SBERT can be used for tasks which are computationally not feasible to be modeled with BERT. For example, clustering of 10,000 sentences with hierarchical clustering requires with BERT about 65 hours, as around 50 Million sentence combinations must be computed. With SBERT, we were able to reduce the effort to about 5 seconds. Acknowledgments This work has been supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1). It has been co-funded by the German Federal Ministry of Education and Research (BMBF) under the promotional references 03VP02540 (ArgumenText).
How much time takes its training?
7f207549c75f5c4388efc15ed28822672b845663
How Ġmuch Ġtime Ġtakes Ġits Ġtraining ?
20 Ġminutes
<s> How Ġmuch Ġtime Ġtakes Ġits Ġtraining ? </s> Introduction </s> In Ġthis Ġpublication , Ġwe Ġpresent ĠSent ence - BER T Ġ( S BER T ), Ġa Ġmodification Ġof Ġthe ĠB ERT Ġnetwork Ġusing Ġsi ames e Ġand Ġtriple t Ġnetworks Ġthat Ġis Ġable Ġto Ġderive Ġsem antically Ġmeaningful Ġsentence Ġembed d ings . ĠThis Ġenables ĠB ERT Ġto Ġbe Ġused Ġfor Ġcertain Ġnew Ġtasks , Ġwhich Ġup - to - now Ġwere Ġnot Ġapplicable Ġfor ĠB ERT . ĠThese Ġtasks Ġinclude Ġlarge - scale Ġsemantic Ġsimilarity Ġcomparison , Ġclust ering , Ġand Ġinformation Ġretrieval Ġvia Ġsemantic Ġsearch . </s> BER T Ġset Ġnew Ġstate - of - the - art Ġperformance Ġon Ġvarious Ġsentence Ġclassification Ġand Ġsentence - pair Ġregression Ġtasks . ĠB ERT Ġuses Ġa Ġcross - enc oder : ĠTwo Ġsentences Ġare Ġpassed Ġto Ġthe Ġtransformer Ġnetwork Ġand Ġthe Ġtarget Ġvalue Ġis Ġpredicted . ĠHowever , Ġthis Ġsetup Ġis Ġunsu itable Ġfor Ġvarious Ġpair Ġregression Ġtasks Ġdue Ġto Ġtoo Ġmany Ġpossible Ġcombinations . ĠFinding Ġin Ġa Ġcollection Ġof Ġ$ n = 10 \ , 000 $ Ġsentences Ġthe Ġpair Ġwith Ġthe Ġhighest Ġsimilarity Ġrequires Ġwith ĠB ERT Ġ$ n \ cd ot Ġ( n - 1 )/ 2 = 49 \ , 995 \ , 000 $ Ġinference Ġcomput ations . ĠOn Ġa Ġmodern ĠV 100 ĠGPU , Ġthis Ġrequires Ġabout Ġ65 Ġhours . ĠSimilar , Ġfinding Ġwhich Ġof Ġthe Ġover Ġ40 Ġmillion Ġexist ent Ġquestions Ġof ĠQu ora Ġis Ġthe Ġmost Ġsimilar Ġfor Ġa Ġnew Ġquestion Ġcould Ġbe Ġmodeled Ġas Ġa Ġpair - wise Ġcomparison Ġwith ĠB ERT , Ġhowever , Ġanswering Ġa Ġsingle Ġquery Ġwould Ġrequire Ġover Ġ50 Ġhours . </s> A Ġcommon Ġmethod Ġto Ġaddress Ġclust ering Ġand Ġsemantic Ġsearch Ġis Ġto Ġmap Ġeach Ġsentence Ġto Ġa Ġvector Ġspace Ġsuch Ġthat Ġsem antically Ġsimilar Ġsentences Ġare Ġclose . ĠResearchers Ġhave Ġstarted Ġto Ġinput Ġindividual Ġsentences Ġinto ĠB ERT Ġand Ġto Ġderive Ġfixed - size Ġsentence Ġembed d ings . ĠThe Ġmost Ġcommonly Ġused Ġapproach Ġis Ġto Ġaverage Ġthe ĠB ERT Ġoutput Ġlayer Ġ( known Ġas ĠB ERT Ġembed d ings ) Ġor Ġby Ġusing Ġthe Ġoutput Ġof Ġthe Ġfirst Ġtoken Ġ( the Ġ[ CL S ] Ġtoken ). ĠAs Ġwe Ġwill Ġshow , Ġthis Ġcommon Ġpractice Ġyields Ġrather Ġbad Ġsentence Ġembed d ings , Ġoften Ġworse Ġthan Ġaveraging ĠGl o Ve Ġembed d ings ĠB IB REF 2 . </s> To Ġalleviate Ġthis Ġissue , Ġwe Ġdeveloped ĠS BER T . ĠThe Ġsi ames e Ġnetwork Ġarchitecture Ġenables Ġthat Ġfixed - sized Ġvectors Ġfor Ġinput Ġsentences Ġcan Ġbe Ġderived . ĠUsing Ġa Ġsimilarity Ġmeasure Ġlike Ġcos ine - similar ity Ġor ĠMan hatt en Ġ/ ĠEucl ide an Ġdistance , Ġsem antically Ġsimilar Ġsentences Ġcan Ġbe Ġfound . ĠThese Ġsimilarity Ġmeasures Ġcan Ġbe Ġperformed Ġextremely Ġefficient Ġon Ġmodern Ġhardware , Ġallowing ĠS BER T Ġto Ġbe Ġused Ġfor Ġsemantic Ġsimilarity Ġsearch Ġas Ġwell Ġas Ġfor Ġclust ering . ĠThe Ġcomplexity Ġfor Ġfinding Ġthe Ġmost Ġsimilar Ġsentence Ġpair Ġin Ġa Ġcollection Ġof Ġ10 , 000 Ġsentences Ġis Ġreduced Ġfrom Ġ65 Ġhours Ġwith ĠB ERT Ġto Ġthe Ġcomputation Ġof Ġ10 , 000 Ġsentence Ġembed d ings Ġ( 5 Ġseconds Ġwith ĠS BER T ) Ġand Ġcomputing Ġcos ine - similar ity Ġ( 0 . 01 Ġseconds ). ĠBy Ġusing Ġoptimized Ġindex Ġstructures , Ġfinding Ġthe Ġmost Ġsimilar ĠQu ora Ġquestion Ġcan Ġbe Ġreduced Ġfrom Ġ50 Ġhours Ġto Ġa Ġfew Ġmilliseconds ĠB IB REF 3 . </s> We Ġfine - t une ĠS BER T Ġon ĠNL I Ġdata , Ġwhich Ġcreates Ġsentence Ġembed d ings Ġthat Ġsignificantly Ġoutper form Ġother Ġstate - of - the - art Ġsentence Ġembed ding Ġmethods Ġlike ĠIn fer Sent ĠB IB REF 4 Ġand ĠUniversal ĠSent ence ĠEnc oder ĠB IB REF 5 . ĠOn Ġseven ĠSem antic ĠText ual ĠSimilar ity Ġ( ST S ) Ġtasks , ĠS BER T Ġachieves Ġan Ġimprovement Ġof Ġ11 . 7 Ġpoints Ġcompared Ġto ĠIn fer Sent Ġand Ġ5 . 5 Ġpoints Ġcompared Ġto ĠUniversal ĠSent ence ĠEnc oder . ĠOn ĠSent E val ĠB IB REF 6 , Ġan Ġevaluation Ġtool kit Ġfor Ġsentence Ġembed d ings , Ġwe Ġachieve Ġan Ġimprovement Ġof Ġ2 . 1 Ġand Ġ2 . 6 Ġpoints , Ġrespectively . </s> S BER T Ġcan Ġbe Ġadapted Ġto Ġa Ġspecific Ġtask . ĠIt Ġsets Ġnew Ġstate - of - the - art Ġperformance Ġon Ġa Ġchallenging Ġargument Ġsimilarity Ġdataset ĠB IB REF 7 Ġand Ġon Ġa Ġtriple t Ġdataset Ġto Ġdistinguish Ġsentences Ġfrom Ġdifferent Ġsections Ġof Ġa ĠWikipedia Ġarticle ĠB IB REF 8 . </s> The Ġpaper Ġis Ġstructured Ġin Ġthe Ġfollowing Ġway : ĠSection ĠSEC REF 3 Ġpresents ĠS BER T , Ġsection ĠSEC REF 4 Ġevaluates ĠS BER T Ġon Ġcommon ĠST S Ġtasks Ġand Ġon Ġthe Ġchallenging ĠArgument ĠFac et ĠSimilar ity Ġ( AF S ) Ġcorpus ĠB IB REF 7 . ĠSection ĠSEC REF 5 Ġevaluates ĠS BER T Ġon ĠSent E val . ĠIn Ġsection ĠSEC REF 6 , Ġwe Ġperform Ġan Ġab lation Ġstudy Ġto Ġtest Ġsome Ġdesign Ġaspect Ġof ĠS BER T . ĠIn Ġsection ĠSEC REF 7 , Ġwe Ġcompare Ġthe Ġcomputational Ġefficiency Ġof ĠS BER T Ġsentence Ġembed d ings Ġin Ġcontrast Ġto Ġother Ġstate - of - the - art Ġsentence Ġembed ding Ġmethods . </s> Related ĠWork </s> We Ġfirst Ġintroduce ĠB ERT , Ġthen , Ġwe Ġdiscuss Ġstate - of - the - art Ġsentence Ġembed ding Ġmethods . </s> BER T ĠB IB REF 0 Ġis Ġa Ġpre - trained Ġtransformer Ġnetwork ĠB IB REF 9 , Ġwhich Ġset Ġfor Ġvarious ĠN LP Ġtasks Ġnew Ġstate - of - the - art Ġresults , Ġincluding Ġquestion Ġanswering , Ġsentence Ġclassification , Ġand Ġsentence - pair Ġregression . ĠThe Ġinput Ġfor ĠB ERT Ġfor Ġsentence - pair Ġregression Ġconsists Ġof Ġthe Ġtwo Ġsentences , Ġseparated Ġby Ġa Ġspecial Ġ[ SE P ] Ġtoken . ĠMulti - head Ġattention Ġover Ġ12 Ġ( base - model ) Ġor Ġ24 Ġlayers Ġ( large - model ) Ġis Ġapplied Ġand Ġthe Ġoutput Ġis Ġpassed Ġto Ġa Ġsimple Ġregression Ġfunction Ġto Ġderive Ġthe Ġfinal Ġlabel . ĠUsing Ġthis Ġsetup , ĠB ERT Ġset Ġa Ġnew Ġstate - of - the - art Ġperformance Ġon Ġthe ĠSem antic ĠText ual ĠSem ilar ity Ġ( ST S ) Ġbenchmark ĠB IB REF 10 . ĠRo BER Ta ĠB IB REF 1 Ġshowed , Ġthat Ġthe Ġperformance Ġof ĠB ERT Ġcan Ġfurther Ġimproved Ġby Ġsmall Ġadaptations Ġto Ġthe Ġpre - training Ġprocess . ĠWe Ġalso Ġtested ĠXL Net ĠB IB REF 11 , Ġbut Ġit Ġled Ġin Ġgeneral Ġto Ġworse Ġresults Ġthan ĠB ERT . </s> A Ġlarge Ġdisadvantage Ġof Ġthe ĠB ERT Ġnetwork Ġstructure Ġis Ġthat Ġno Ġindependent Ġsentence Ġembed d ings Ġare Ġcomputed , Ġwhich Ġmakes Ġit Ġdifficult Ġto Ġderive Ġsentence Ġembed d ings Ġfrom ĠB ERT . ĠTo Ġbypass Ġthis Ġlimitations , Ġresearchers Ġpassed Ġsingle Ġsentences Ġthrough ĠB ERT Ġand Ġthen Ġderive Ġa Ġfixed Ġsized Ġvector Ġby Ġeither Ġaveraging Ġthe Ġoutputs Ġ( similar Ġto Ġaverage Ġword Ġembed d ings ) Ġor Ġby Ġusing Ġthe Ġoutput Ġof Ġthe Ġspecial ĠCLS Ġtoken Ġ( for Ġexample : Ġb ert sent ence embed d ings 1 , ber ts ent ence embed d ings 2 , ber ts ent ence embed d ings 3 ). ĠThese Ġtwo Ġoptions Ġare Ġalso Ġprovided Ġby Ġthe Ġpopular Ġb ert - as - a - service - re pos itory . ĠUp Ġto Ġour Ġknowledge , Ġthere Ġis Ġso Ġfar Ġno Ġevaluation Ġif Ġthese Ġmethods Ġlead Ġto Ġuseful Ġsentence Ġembed d ings . </s> Sent ence Ġembed d ings Ġare Ġa Ġwell Ġstudied Ġarea Ġwith Ġdozens Ġof Ġproposed Ġmethods . ĠSkip - Th ought ĠB IB REF 12 Ġtrains Ġan Ġenc oder - dec oder Ġarchitecture Ġto Ġpredict Ġthe Ġsurrounding Ġsentences . ĠIn fer Sent ĠB IB REF 4 Ġuses Ġlabeled Ġdata Ġof Ġthe ĠStanford ĠNatural ĠLanguage ĠIn ference Ġdataset ĠB IB REF 13 Ġand Ġthe ĠMulti - Gen re ĠNL I Ġdataset ĠB IB REF 14 Ġto Ġtrain Ġa Ġsi ames e ĠBi L ST M Ġnetwork Ġwith Ġmax - pool ing Ġover Ġthe Ġoutput . ĠCon neau Ġet Ġal . Ġshowed , Ġthat ĠIn fer Sent Ġconsistently Ġoutper forms Ġun super vised Ġmethods Ġlike ĠSkip Th ought . ĠUniversal ĠSent ence ĠEnc oder ĠB IB REF 5 Ġtrains Ġa Ġtransformer Ġnetwork Ġand Ġaug ments Ġun super vised Ġlearning Ġwith Ġtraining Ġon ĠSN LI . Ġhill - et al - 2016 - learning Ġshowed , Ġthat Ġthe Ġtask Ġon Ġwhich Ġsentence Ġembed d ings Ġare Ġtrained Ġsignificantly Ġimpacts Ġtheir Ġquality . ĠPrevious Ġwork ĠB IB REF 4 , ĠB IB REF 5 Ġfound Ġthat Ġthe ĠSN LI Ġdatasets Ġare Ġsuitable Ġfor Ġtraining Ġsentence Ġembed d ings . Ġy ang - 2018 - learning Ġpresented Ġa Ġmethod Ġto Ġtrain Ġon Ġconversations Ġfrom ĠReddit Ġusing Ġsi ames e ĠD AN Ġand Ġsi ames e Ġtransformer Ġnetworks , Ġwhich Ġyielded Ġgood Ġresults Ġon Ġthe ĠST S Ġbenchmark Ġdataset . </s> poly enc od ers Ġaddresses Ġthe Ġrun - time Ġoverhead Ġof Ġthe Ġcross - enc oder Ġfrom ĠB ERT Ġand Ġpresent Ġa Ġmethod Ġ( poly - enc od ers ) Ġto Ġcompute Ġa Ġscore Ġbetween Ġ$ m $ Ġcontext Ġvectors Ġand Ġpre - com puted Ġcandidate Ġembed d ings Ġusing Ġattention . ĠThis Ġidea Ġworks Ġfor Ġfinding Ġthe Ġhighest Ġscoring Ġsentence Ġin Ġa Ġlarger Ġcollection . ĠHowever , Ġpoly - enc od ers Ġhave Ġthe Ġdrawback Ġthat Ġthe Ġscore Ġfunction Ġis Ġnot Ġsymm etric Ġand Ġthe Ġcomputational Ġoverhead Ġis Ġtoo Ġlarge Ġfor Ġuse - cases Ġlike Ġclust ering , Ġwhich Ġwould Ġrequire Ġ$ O ( n ^ 2 ) $ Ġscore Ġcomput ations . </s> Previous Ġneural Ġsentence Ġembed ding Ġmethods Ġstarted Ġthe Ġtraining Ġfrom Ġa Ġrandom Ġinitialization . ĠIn Ġthis Ġpublication , Ġwe Ġuse Ġthe Ġpre - trained ĠB ERT Ġand ĠRo BER Ta Ġnetwork Ġand Ġonly Ġfine - t une Ġit Ġto Ġyield Ġuseful Ġsentence Ġembed d ings . ĠThis Ġreduces Ġsignificantly Ġthe Ġneeded Ġtraining Ġtime : ĠS BER T Ġcan Ġbe Ġtuned Ġin Ġless Ġthan Ġ20 Ġminutes , Ġwhile Ġyielding Ġbetter Ġresults Ġthan Ġcomparable Ġsentence Ġembed ding Ġmethods . </s> Model </s> S BER T Ġadds Ġa Ġpool ing Ġoperation Ġto Ġthe Ġoutput Ġof ĠB ERT Ġ/ ĠRo BER Ta Ġto Ġderive Ġa Ġfixed Ġsized Ġsentence Ġembed ding . ĠWe Ġexperiment Ġwith Ġthree Ġpool ing Ġstrategies : ĠUsing Ġthe Ġoutput Ġof Ġthe ĠCLS - token , Ġcomputing Ġthe Ġmean Ġof Ġall Ġoutput Ġvectors Ġ( ME AN - str ategy ), Ġand Ġcomputing Ġa Ġmax - over - time Ġof Ġthe Ġoutput Ġvectors Ġ( MAX - str ategy ). ĠThe Ġdefault Ġconfiguration Ġis ĠME AN . </s> In Ġorder Ġto Ġfine - t une ĠB ERT Ġ/ ĠRo BER Ta , Ġwe Ġcreate Ġsi ames e Ġand Ġtriple t Ġnetworks ĠB IB REF 15 Ġto Ġupdate Ġthe Ġweights Ġsuch Ġthat Ġthe Ġproduced Ġsentence Ġembed d ings Ġare Ġsem antically Ġmeaningful Ġand Ġcan Ġbe Ġcompared Ġwith Ġcos ine - similar ity . </s> The Ġnetwork Ġstructure Ġdepends Ġon Ġthe Ġavailable Ġtraining Ġdata . ĠWe Ġexperiment Ġwith Ġthe Ġfollowing Ġstructures Ġand Ġobjective Ġfunctions . </s> Class ification ĠObjective ĠFunction . ĠWe Ġconc aten ate Ġthe Ġsentence Ġembed d ings Ġ$ u $ Ġand Ġ$ v $ Ġwith Ġthe Ġelement - wise Ġdifference Ġ$ | u - v | $ Ġand Ġmultiply Ġit Ġwith Ġthe Ġtrain able Ġweight Ġ$ W _ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ 3 n Ġ\ times Ġk } $ : </s> where Ġ$ n $ Ġis Ġthe Ġdimension Ġof Ġthe Ġsentence Ġembed d ings Ġand Ġ$ k $ Ġthe Ġnumber Ġof Ġlabels . ĠWe Ġoptimize Ġcross - ent ropy Ġloss . ĠThis Ġstructure Ġis Ġdepicted Ġin ĠFigure ĠFIG REF 4 . </s> Reg ression ĠObjective ĠFunction . ĠThe Ġcos ine - similar ity Ġbetween Ġthe Ġtwo Ġsentence Ġembed d ings Ġ$ u $ Ġand Ġ$ v $ Ġis Ġcomputed Ġ( Figure ĠFIG REF 5 ). ĠWe Ġuse Ġmean - squ ared - error Ġloss Ġas Ġthe Ġobjective Ġfunction . </s> Tri plet ĠObjective ĠFunction . ĠGiven Ġan Ġanchor Ġsentence Ġ$ a $, Ġa Ġpositive Ġsentence Ġ$ p $, Ġand Ġa Ġnegative Ġsentence Ġ$ n $, Ġtriple t Ġloss Ġtunes Ġthe Ġnetwork Ġsuch Ġthat Ġthe Ġdistance Ġbetween Ġ$ a $ Ġand Ġ$ p $ Ġis Ġsmaller Ġthan Ġthe Ġdistance Ġbetween Ġ$ a $ Ġand Ġ$ n $. ĠMat hematically , Ġwe Ġminimize Ġthe Ġfollowing Ġloss Ġfunction : </s> with Ġ$ s _ x $ Ġthe Ġsentence Ġembed ding Ġfor Ġ$ a $ /$ n $ /$ p $, Ġ$ || \ cd ot Ġ|| $ Ġa Ġdistance Ġmetric Ġand Ġmargin Ġ$\ ep sil on Ġ$ . ĠMarg in Ġ$\ ep sil on Ġ$ Ġensures Ġthat Ġ$ s _ p $ Ġis Ġat Ġleast Ġ$\ ep sil on Ġ$ Ġcloser Ġto Ġ$ s _ a $ Ġthan Ġ$ s _ n $. ĠAs Ġmetric Ġwe Ġuse ĠEucl ide an Ġdistance Ġand Ġwe Ġset Ġ$\ ep sil on Ġ= 1 $ Ġin Ġour Ġexperiments . </s> Model Ġ: :: ĠTraining ĠDetails </s> We Ġtrain ĠS BER T Ġon Ġthe Ġcombination Ġof Ġthe ĠSN LI ĠB IB REF 13 Ġand Ġthe ĠMulti - Gen re ĠNL I ĠB IB REF 14 Ġdataset . ĠThe ĠSN LI Ġis Ġa Ġcollection Ġof Ġ570 , 000 Ġsentence Ġpairs Ġannot ated Ġwith Ġthe Ġlabels Ġcontradiction , Ġe int ail ment , Ġand Ġneutral . ĠMulti N LI Ġcontains Ġ430 , 000 Ġsentence Ġpairs Ġand Ġcovers Ġa Ġrange Ġof Ġgenres Ġof Ġspoken Ġand Ġwritten Ġtext . ĠWe Ġfine - t une ĠS BER T Ġwith Ġa Ġ3 - way Ġsoft max - class ifier Ġobjective Ġfunction Ġfor Ġone Ġepoch . ĠWe Ġused Ġa Ġbatch - size Ġof Ġ16 , ĠAdam Ġoptim izer Ġwith Ġlearning Ġrate Ġ$ 2 \ math rm Ġ{ e }{ - 5 } $, Ġand Ġa Ġlinear Ġlearning Ġrate Ġwarm - up Ġover Ġ10 % Ġof Ġthe Ġtraining Ġdata . ĠOur Ġdefault Ġpool ing Ġstrategy Ġis ĠME AN . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity </s> We Ġevaluate Ġthe Ġperformance Ġof ĠS BER T Ġfor Ġcommon ĠSem antic ĠText ual ĠSimilar ity Ġ( ST S ) Ġtasks . ĠState - of - the - art Ġmethods Ġoften Ġlearn Ġa Ġ( complex ) Ġregression Ġfunction Ġthat Ġmaps Ġsentence Ġembed d ings Ġto Ġa Ġsimilarity Ġscore . ĠHowever , Ġthese Ġregression Ġfunctions Ġwork Ġpair - wise Ġand Ġdue Ġto Ġthe Ġcomb inator ial Ġexplosion Ġthose Ġare Ġoften Ġnot Ġscalable Ġif Ġthe Ġcollection Ġof Ġsentences Ġreaches Ġa Ġcertain Ġsize . ĠInstead , Ġwe Ġalways Ġuse Ġcos ine - similar ity Ġto Ġcompare Ġthe Ġsimilarity Ġbetween Ġtwo Ġsentence Ġembed d ings . ĠWe Ġran Ġour Ġexperiments Ġalso Ġwith Ġnegative ĠMan hatt en Ġand Ġnegative ĠEucl ide an Ġdistances Ġas Ġsimilarity Ġmeasures , Ġbut Ġthe Ġresults Ġfor Ġall Ġapproaches Ġremained Ġroughly Ġthe Ġsame . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity Ġ: :: ĠUn super vised ĠST S </s> We Ġevaluate Ġthe Ġperformance Ġof ĠS BER T Ġfor ĠST S Ġwithout Ġusing Ġany ĠST S Ġspecific Ġtraining Ġdata . ĠWe Ġuse Ġthe ĠST S Ġtasks Ġ2012 Ġ- Ġ2016 ĠB IB REF 16 , ĠB IB REF 17 , ĠB IB REF 18 , ĠB IB REF 19 , ĠB IB REF 20 , Ġthe ĠST S Ġbenchmark ĠB IB REF 10 , Ġand Ġthe ĠS ICK - Related ness Ġdataset ĠB IB REF 21 . ĠThese Ġdatasets Ġprovide Ġlabels Ġbetween Ġ0 Ġand Ġ5 Ġon Ġthe Ġsemantic Ġrelated ness Ġof Ġsentence Ġpairs . ĠWe Ġshowed Ġin ĠB IB REF 22 Ġthat ĠPearson Ġcorrelation Ġis Ġbadly Ġsuited Ġfor ĠST S . ĠInstead , Ġwe Ġcompute Ġthe ĠSpear man 's Ġrank Ġcorrelation Ġbetween Ġthe Ġcos ine - similar ity Ġof Ġthe Ġsentence Ġembed d ings Ġand Ġthe Ġgold Ġlabels . ĠThe Ġsetup Ġfor Ġthe Ġother Ġsentence Ġembed ding Ġmethods Ġis Ġequivalent , Ġthe Ġsimilarity Ġis Ġcomputed Ġby Ġcos ine - similar ity . ĠThe Ġresults Ġare Ġdepicted Ġin ĠTable ĠT AB REF 6 . </s> The Ġresults Ġshows Ġthat Ġdirectly Ġusing Ġthe Ġoutput Ġof ĠB ERT Ġleads Ġto Ġrather Ġpoor Ġperformances . ĠA ver aging Ġthe ĠB ERT Ġembed d ings Ġachieves Ġan Ġaverage Ġcorrelation Ġof Ġonly Ġ54 . 81 , Ġand Ġusing Ġthe ĠCLS - token Ġoutput Ġonly Ġachieves Ġan Ġaverage Ġcorrelation Ġof Ġ29 . 19 . ĠBoth Ġare Ġworse Ġthan Ġcomputing Ġaverage ĠGl o Ve Ġembed d ings . </s> Using Ġthe Ġdescribed Ġsi ames e Ġnetwork Ġstructure Ġand Ġfine - tun ing Ġmechanism Ġsubstantially Ġimproves Ġthe Ġcorrelation , Ġoutper forming Ġboth ĠIn fer Sent Ġand ĠUniversal ĠSent ence ĠEnc oder Ġsubstantially . ĠThe Ġonly Ġdataset Ġwhere ĠS BER T Ġperforms Ġworse Ġthan ĠUniversal ĠSent ence ĠEnc oder Ġis ĠS ICK - R . ĠUniversal ĠSent ence ĠEnc oder Ġwas Ġtrained Ġon Ġvarious Ġdatasets , Ġincluding Ġnews , Ġquestion - answer Ġpages Ġand Ġdiscussion Ġforums , Ġwhich Ġappears Ġto Ġbe Ġmore Ġsuitable Ġto Ġthe Ġdata Ġof ĠS ICK - R . ĠIn Ġcontrast , ĠS BER T Ġwas Ġpre - trained Ġonly Ġon ĠWikipedia Ġ( via ĠB ERT ) Ġand Ġon ĠNL I Ġdata . </s> While ĠRo BER Ta Ġwas Ġable Ġto Ġimprove Ġthe Ġperformance Ġfor Ġseveral Ġsupervised Ġtasks , Ġwe Ġonly Ġobserve Ġminor Ġdifference Ġbetween ĠS BER T Ġand ĠS Ro BER Ta Ġfor Ġgenerating Ġsentence Ġembed d ings . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity Ġ: :: ĠSuper vised ĠST S </s> The ĠST S Ġbenchmark Ġ( ST S b ) ĠB IB REF 10 Ġprovides Ġis Ġa Ġpopular Ġdataset Ġto Ġevaluate Ġsupervised ĠST S Ġsystems . ĠThe Ġdata Ġincludes Ġ8 , 628 Ġsentence Ġpairs Ġfrom Ġthe Ġthree Ġcategories Ġcapt ions , Ġnews , Ġand Ġforums . ĠIt Ġis Ġdivided Ġinto Ġtrain Ġ( 5 , 7 49 ), Ġdev Ġ( 1 , 500 ) Ġand Ġtest Ġ( 1 , 379 ). ĠB ERT Ġset Ġa Ġnew Ġstate - of - the - art Ġperformance Ġon Ġthis Ġdataset Ġby Ġpassing Ġboth Ġsentences Ġto Ġthe Ġnetwork Ġand Ġusing Ġa Ġsimple Ġregression Ġmethod Ġfor Ġthe Ġoutput . </s> We Ġuse Ġthe Ġtraining Ġset Ġto Ġfine - t une ĠS BER T Ġusing Ġthe Ġregression Ġobjective Ġfunction . ĠAt Ġprediction Ġtime , Ġwe Ġcompute Ġthe Ġcos ine - similar ity Ġbetween Ġthe Ġsentence Ġembed d ings . ĠAll Ġsystems Ġare Ġtrained Ġwith Ġ10 Ġrandom Ġseeds Ġto Ġcounter Ġvar iances ĠB IB REF 23 . </s> The Ġresults Ġare Ġdepicted Ġin ĠTable ĠT AB REF 10 . ĠWe Ġexperimented Ġwith Ġtwo Ġsetups : ĠOnly Ġtraining Ġon ĠST S b , Ġand Ġfirst Ġtraining Ġon ĠNL I , Ġthen Ġtraining Ġon ĠST S b . ĠWe Ġobserve Ġthat Ġthe Ġlater Ġstrategy Ġleads Ġto Ġa Ġslight Ġimprovement Ġof Ġ1 - 2 Ġpoints . ĠThis Ġtwo - step Ġapproach Ġhad Ġan Ġespecially Ġlarge Ġimpact Ġfor Ġthe ĠB ERT Ġcross - enc oder , Ġwhich Ġimproved Ġthe Ġperformance Ġby Ġ3 - 4 Ġpoints . ĠWe Ġdo Ġnot Ġobserve Ġa Ġsignificant Ġdifference Ġbetween ĠB ERT Ġand ĠRo BER Ta . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity Ġ: :: ĠArgument ĠFac et ĠSimilar ity </s> We Ġevaluate ĠS BER T Ġon Ġthe ĠArgument ĠFac et ĠSimilar ity Ġ( AF S ) Ġcorpus Ġby ĠMis ra EW 16 . ĠThe ĠA FS Ġcorpus Ġannot ated Ġ6 , 000 Ġsent ential Ġargument Ġpairs Ġfrom Ġsocial Ġmedia Ġdialog s Ġon Ġthree Ġcontroversial Ġtopics : Ġgun Ġcontrol , Ġgay Ġmarriage , Ġand Ġdeath Ġpenalty . ĠThe Ġdata Ġwas Ġannot ated Ġon Ġa Ġscale Ġfrom Ġ0 Ġ( âĢ ľ different Ġtopic ") Ġto Ġ5 Ġ( âĢ ľ completely Ġequivalent "). ĠThe Ġsimilarity Ġnotion Ġin Ġthe ĠA FS Ġcorpus Ġis Ġfairly Ġdifferent Ġto Ġthe Ġsimilarity Ġnotion Ġin Ġthe ĠST S Ġdatasets Ġfrom ĠSem E val . ĠST S Ġdata Ġis Ġusually Ġdescriptive , Ġwhile ĠA FS Ġdata Ġare Ġargument ative Ġexcerpts Ġfrom Ġdialog s . ĠTo Ġbe Ġconsidered Ġsimilar , Ġarguments Ġmust Ġnot Ġonly Ġmake Ġsimilar Ġclaims , Ġbut Ġalso Ġprovide Ġa Ġsimilar Ġreasoning . ĠFurther , Ġthe Ġlex ical Ġgap Ġbetween Ġthe Ġsentences Ġin ĠA FS Ġis Ġmuch Ġlarger . ĠHence , Ġsimple Ġun super vised Ġmethods Ġas Ġwell Ġas Ġstate - of - the - art ĠST S Ġsystems Ġperform Ġbadly Ġon Ġthis Ġdataset ĠB IB REF 24 . </s> We Ġevaluate ĠS BER T Ġon Ġthis Ġdataset Ġin Ġtwo Ġscenarios : Ġ1 ) ĠAs Ġproposed Ġby ĠMis ra Ġet Ġal ., Ġwe Ġevaluate ĠS BER T Ġusing Ġ10 - fold Ġcross - valid ation . ĠA Ġdraw - back Ġof Ġthis Ġevaluation Ġsetup Ġis Ġthat Ġit Ġis Ġnot Ġclear Ġhow Ġwell Ġapproaches Ġgeneral ize Ġto Ġdifferent Ġtopics . ĠHence , Ġ2 ) Ġwe Ġevaluate ĠS BER T Ġin Ġa Ġcross - topic Ġsetup . ĠTwo Ġtopics Ġserve Ġfor Ġtraining Ġand Ġthe Ġapproach Ġis Ġevaluated Ġon Ġthe Ġleft - out Ġtopic . ĠWe Ġrepeat Ġthis Ġfor Ġall Ġthree Ġtopics Ġand Ġaverage Ġthe Ġresults . </s> S BER T Ġis Ġfine - tun ed Ġusing Ġthe ĠReg ression ĠObjective ĠFunction . ĠThe Ġsimilarity Ġscore Ġis Ġcomputed Ġusing Ġcos ine - similar ity Ġbased Ġon Ġthe Ġsentence Ġembed d ings . ĠWe Ġalso Ġprovide Ġthe ĠPearson Ġcorrelation Ġ$ r $ Ġto Ġmake Ġthe Ġresults Ġcomparable Ġto ĠMis ra Ġet Ġal . ĠHowever , Ġwe Ġshowed ĠB IB REF 22 Ġthat ĠPearson Ġcorrelation Ġhas Ġsome Ġserious Ġdrawbacks Ġand Ġshould Ġbe Ġavoided Ġfor Ġcomparing ĠST S Ġsystems . ĠThe Ġresults Ġare Ġdepicted Ġin ĠTable ĠT AB REF 12 . </s> Un super vised Ġmethods Ġlike Ġtf - id f , Ġaverage ĠGl o Ve Ġembed d ings Ġor ĠIn fer Sent Ġperform Ġrather Ġbadly Ġon Ġthis Ġdataset Ġwith Ġlow Ġscores . ĠTraining ĠS BER T Ġin Ġthe Ġ10 - fold Ġcross - valid ation Ġsetup Ġgives Ġa Ġperformance Ġthat Ġis Ġnearly Ġon - par Ġwith ĠB ERT . </s> However , Ġin Ġthe Ġcross - topic Ġevaluation , Ġwe Ġobserve Ġa Ġperformance Ġdrop Ġof ĠS BER T Ġby Ġabout Ġ7 Ġpoints ĠSpear man Ġcorrelation . ĠTo Ġbe Ġconsidered Ġsimilar , Ġarguments Ġshould Ġaddress Ġthe Ġsame Ġclaims Ġand Ġprovide Ġthe Ġsame Ġreasoning . ĠB ERT Ġis Ġable Ġto Ġuse Ġattention Ġto Ġcompare Ġdirectly Ġboth Ġsentences Ġ( e . g . Ġword - by - word Ġcomparison ), Ġwhile ĠS BER T Ġmust Ġmap Ġindividual Ġsentences Ġfrom Ġan Ġunseen Ġtopic Ġto Ġa Ġvector Ġspace Ġsuch Ġthat Ġarguments Ġwith Ġsimilar Ġclaims Ġand Ġreasons Ġare Ġclose . ĠThis Ġis Ġa Ġmuch Ġmore Ġchallenging Ġtask , Ġwhich Ġappears Ġto Ġrequire Ġmore Ġthan Ġjust Ġtwo Ġtopics Ġfor Ġtraining Ġto Ġwork Ġon - par Ġwith ĠB ERT . </s> E val uation Ġ- ĠSem antic ĠText ual ĠSimilar ity Ġ: :: ĠWikipedia ĠSections ĠDist inction </s> e in - dor - et al - 2018 - learning Ġuse ĠWikipedia Ġto Ġcreate Ġa Ġthem atically Ġfine - gr ained Ġtrain , Ġdev Ġand Ġtest Ġset Ġfor Ġsentence Ġembed d ings Ġmethods . ĠWikipedia Ġarticles Ġare Ġseparated Ġinto Ġdistinct Ġsections Ġfocusing Ġon Ġcertain Ġaspects . ĠDor Ġet Ġal . Ġassume Ġthat Ġsentences Ġin Ġthe Ġsame Ġsection Ġare Ġthem atically Ġcloser Ġthan Ġsentences Ġin Ġdifferent Ġsections . ĠThey Ġuse Ġthis Ġto Ġcreate Ġa Ġlarge Ġdataset Ġof Ġweak ly Ġlabeled Ġsentence Ġtriple ts : ĠThe Ġanchor Ġand Ġthe Ġpositive Ġexample Ġcome Ġfrom Ġthe Ġsame Ġsection , Ġwhile Ġthe Ġnegative Ġexample Ġcomes Ġfrom Ġa Ġdifferent Ġsection Ġof Ġthe Ġsame Ġarticle . ĠFor Ġexample , Ġfrom Ġthe ĠAlice ĠArnold Ġarticle : ĠAnch or : ĠArnold Ġjoined Ġthe ĠBBC ĠRadio ĠDrama ĠCompany Ġin Ġ1988 ., Ġpositive : ĠArnold Ġgained Ġmedia Ġattention Ġin ĠMay Ġ2012 ., Ġnegative : ĠBald ing Ġand ĠArnold Ġare Ġkeen Ġamateur Ġgolf ers . </s> We Ġuse Ġthe Ġdataset Ġfrom ĠDor Ġet Ġal . ĠWe Ġuse Ġthe ĠTriple t ĠObjective , Ġtrain ĠS BER T Ġfor Ġone Ġepoch Ġon Ġthe Ġabout Ġ1 . 8 ĠMillion Ġtraining Ġtriple ts Ġand Ġevaluate Ġit Ġon Ġthe Ġ222 , 9 57 Ġtest Ġtriple ts . ĠTest Ġtriple ts Ġare Ġfrom Ġa Ġdistinct Ġset Ġof ĠWikipedia Ġarticles . ĠAs Ġevaluation Ġmetric , Ġwe Ġuse Ġaccuracy : ĠIs Ġthe Ġpositive Ġexample Ġcloser Ġto Ġthe Ġanchor Ġthan Ġthe Ġnegative Ġexample ? </s> Results Ġare Ġpresented Ġin ĠTable ĠT AB REF 14 . ĠDor Ġet Ġal . Ġfine - tun ed Ġa ĠBi L ST M Ġarchitecture Ġwith Ġtriple t Ġloss Ġto Ġderive Ġsentence Ġembed d ings Ġfor Ġthis Ġdataset . ĠAs Ġthe Ġtable Ġshows , ĠS BER T Ġclearly Ġoutper forms Ġthe ĠBi L ST M Ġapproach Ġby ĠDor Ġet Ġal . </s> E val uation Ġ- ĠSent E val </s> Sent E val ĠB IB REF 6 Ġis Ġa Ġpopular Ġtool kit Ġto Ġevaluate Ġthe Ġquality Ġof Ġsentence Ġembed d ings . ĠSent ence Ġembed d ings Ġare Ġused Ġas Ġfeatures Ġfor Ġa Ġlog istic Ġregression Ġclass ifier . ĠThe Ġlog istic Ġregression Ġclass ifier Ġis Ġtrained Ġon Ġvarious Ġtasks Ġin Ġa Ġ10 - fold Ġcross - valid ation Ġsetup Ġand Ġthe Ġprediction Ġaccuracy Ġis Ġcomputed Ġfor Ġthe Ġtest - fold . </s> The Ġpurpose Ġof ĠS BER T Ġsentence Ġembed d ings Ġare Ġnot Ġto Ġbe Ġused Ġfor Ġtransfer Ġlearning Ġfor Ġother Ġtasks . ĠHere , Ġwe Ġthink Ġfine - tun ing ĠB ERT Ġas Ġdescribed Ġby Ġdev lin 2018 bert Ġfor Ġnew Ġtasks Ġis Ġthe Ġmore Ġsuitable Ġmethod , Ġas Ġit Ġupdates Ġall Ġlayers Ġof Ġthe ĠB ERT Ġnetwork . ĠHowever , ĠSent E val Ġcan Ġstill Ġgive Ġan Ġimpression Ġon Ġthe Ġquality Ġof Ġour Ġsentence Ġembed d ings Ġfor Ġvarious Ġtasks . </s> We Ġcompare Ġthe ĠS BER T Ġsentence Ġembed d ings Ġto Ġother Ġsentence Ġembed d ings Ġmethods Ġon Ġthe Ġfollowing Ġseven ĠSent E val Ġtransfer Ġtasks : </s> MR : ĠSent iment Ġprediction Ġfor Ġmovie Ġreviews Ġsnippets Ġon Ġa Ġfive Ġstart Ġscale ĠB IB REF 25 . </s> CR : ĠSent iment Ġprediction Ġof Ġcustomer Ġproduct Ġreviews ĠB IB REF 26 . </s> S UB J : ĠSubject ivity Ġprediction Ġof Ġsentences Ġfrom Ġmovie Ġreviews Ġand Ġplot Ġsumm aries ĠB IB REF 27 . </s> MP Q A : ĠPh rase Ġlevel Ġopinion Ġpol arity Ġclassification Ġfrom Ġnews wire ĠB IB REF 28 . </s> S ST : ĠStanford ĠSent iment ĠTree bank Ġwith Ġbinary Ġlabels ĠB IB REF 29 . </s> T REC : ĠFine Ġgr ained Ġquestion - type Ġclassification Ġfrom ĠT REC ĠB IB REF 30 . </s> MR PC : ĠMicrosoft ĠResearch ĠPar aph rase ĠCorpus Ġfrom Ġparallel Ġnews Ġsources ĠB IB REF 31 . </s> The Ġresults Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 15 . ĠS BER T Ġis Ġable Ġto Ġachieve Ġthe Ġbest Ġperformance Ġin Ġ5 Ġout Ġof Ġ7 Ġtasks . ĠThe Ġaverage Ġperformance Ġincreases Ġby Ġabout Ġ2 Ġpercentage Ġpoints Ġcompared Ġto ĠIn fer Sent Ġas Ġwell Ġas Ġthe ĠUniversal ĠSent ence ĠEnc oder . ĠEven Ġthough Ġtransfer Ġlearning Ġis Ġnot Ġthe Ġpurpose Ġof ĠS BER T , Ġit Ġoutper forms Ġother Ġstate - of - the - art Ġsentence Ġembed d ings Ġmethods Ġon Ġthis Ġtask . </s> It Ġappears Ġthat Ġthe Ġsentence Ġembed d ings Ġfrom ĠS BER T Ġcapture Ġwell Ġsentiment Ġinformation : ĠWe Ġobserve Ġlarge Ġimprovements Ġfor Ġall Ġsentiment Ġtasks Ġ( MR , ĠCR , Ġand ĠS ST ) Ġfrom ĠSent E val Ġin Ġcomparison Ġto ĠIn fer Sent Ġand ĠUniversal ĠSent ence ĠEnc oder . </s> The Ġonly Ġdataset Ġwhere ĠS BER T Ġis Ġsignificantly Ġworse Ġthan ĠUniversal ĠSent ence ĠEnc oder Ġis Ġthe ĠT REC Ġdataset . ĠUniversal ĠSent ence ĠEnc oder Ġwas Ġpre - trained Ġon Ġquestion - ans w ering Ġdata , Ġwhich Ġappears Ġto Ġbe Ġbeneficial Ġfor Ġthe Ġquestion - type Ġclassification Ġtask Ġof Ġthe ĠT REC Ġdataset . </s> Average ĠB ERT Ġembed d ings Ġor Ġusing Ġthe ĠCLS - token Ġoutput Ġfrom Ġa ĠB ERT Ġnetwork Ġachieved Ġbad Ġresults Ġfor Ġvarious ĠST S Ġtasks Ġ( Table ĠT AB REF 6 ), Ġworse Ġthan Ġaverage ĠGl o Ve Ġembed d ings . ĠHowever , Ġfor ĠSent E val , Ġaverage ĠB ERT Ġembed d ings Ġand Ġthe ĠB ERT ĠCLS - token Ġoutput Ġachieves Ġdecent Ġresults Ġ( Table ĠT AB REF 15 ), Ġoutper forming Ġaverage ĠGl o Ve Ġembed d ings . ĠThe Ġreason Ġfor Ġthis Ġare Ġthe Ġdifferent Ġsetups . ĠFor Ġthe ĠST S Ġtasks , Ġwe Ġused Ġcos ine - similar ity Ġto Ġestimate Ġthe Ġsimilarities Ġbetween Ġsentence Ġembed d ings . ĠCos ine - similar ity Ġtreats Ġall Ġdimensions Ġequally . ĠIn Ġcontrast , ĠSent E val Ġfits Ġa Ġlog istic Ġregression Ġclass ifier Ġto Ġthe Ġsentence Ġembed d ings . ĠThis Ġallows Ġthat Ġcertain Ġdimensions Ġcan Ġhave Ġhigher Ġor Ġlower Ġimpact Ġon Ġthe Ġclassification Ġresult . </s> We Ġconclude Ġthat Ġaverage ĠB ERT Ġembed d ings Ġ/ ĠCLS - token Ġoutput Ġfrom ĠB ERT Ġreturn Ġsentence Ġembed d ings Ġthat Ġare Ġinf eas ible Ġto Ġbe Ġused Ġwith Ġcos ine - similar ity Ġor Ġwith ĠMan hatt en Ġ/ ĠEucl ide an Ġdistance . ĠFor Ġtransfer Ġlearning , Ġthey Ġyield Ġslightly Ġworse Ġresults Ġthan ĠIn fer Sent Ġor ĠUniversal ĠSent ence ĠEnc oder . ĠHowever , Ġusing Ġthe Ġdescribed Ġfine - tun ing Ġsetup Ġwith Ġa Ġsi ames e Ġnetwork Ġstructure Ġon ĠNL I Ġdatasets Ġyields Ġsentence Ġembed d ings Ġthat Ġachieve Ġa Ġnew Ġstate - of - the - art Ġfor Ġthe ĠSent E val Ġtool kit . </s> A bl ation ĠStudy </s> We Ġhave Ġdemonstrated Ġstrong Ġempirical Ġresults Ġfor Ġthe Ġquality Ġof ĠS BER T Ġsentence Ġembed d ings . ĠIn Ġthis Ġsection , Ġwe Ġperform Ġan Ġab lation Ġstudy Ġof Ġdifferent Ġaspects Ġof ĠS BER T Ġin Ġorder Ġto Ġget Ġa Ġbetter Ġunderstanding Ġof Ġtheir Ġrelative Ġimportance . </s> We Ġevaluated Ġdifferent Ġpool ing Ġstrategies Ġ( ME AN , ĠMAX , Ġand ĠCLS ). ĠFor Ġthe Ġclassification Ġobjective Ġfunction , Ġwe Ġevaluate Ġdifferent Ġconc aten ation Ġmethods . ĠFor Ġeach Ġpossible Ġconfiguration , Ġwe Ġtrain ĠS BER T Ġwith Ġ10 Ġdifferent Ġrandom Ġseeds Ġand Ġaverage Ġthe Ġperformances . </s> The Ġobjective Ġfunction Ġ( class ification Ġvs . Ġregression ) Ġdepends Ġon Ġthe Ġannot ated Ġdataset . ĠFor Ġthe Ġclassification Ġobjective Ġfunction , Ġwe Ġtrain ĠS BER T - base Ġon Ġthe ĠSN LI Ġand Ġthe ĠMulti - N LI Ġdataset . ĠFor Ġthe Ġregression Ġobjective Ġfunction , Ġwe Ġtrain Ġon Ġthe Ġtraining Ġset Ġof Ġthe ĠST S Ġbenchmark Ġdataset . ĠPerform ances Ġare Ġmeasured Ġon Ġthe Ġdevelopment Ġsplit Ġof Ġthe ĠST S Ġbenchmark Ġdataset . ĠResults Ġare Ġshown Ġin ĠTable ĠT AB REF 23 . </s> When Ġtrained Ġwith Ġthe Ġclassification Ġobjective Ġfunction Ġon ĠNL I Ġdata , Ġthe Ġpool ing Ġstrategy Ġhas Ġa Ġrather Ġminor Ġimpact . ĠThe Ġimpact Ġof Ġthe Ġconc aten ation Ġmode Ġis Ġmuch Ġlarger . ĠIn fer Sent ĠB IB REF 4 Ġand ĠUniversal ĠSent ence ĠEnc oder ĠB IB REF 5 Ġboth Ġuse Ġ$( u , Ġv , Ġ| u - v | , Ġu * v ) $ Ġas Ġinput Ġfor Ġa Ġsoft max Ġclass ifier . ĠHowever , Ġin Ġour Ġarchitecture , Ġadding Ġthe Ġelement - wise Ġ$ u * v $ Ġdecreased Ġthe Ġperformance . </s> The Ġmost Ġimportant Ġcomponent Ġis Ġthe Ġelement - wise Ġdifference Ġ$ | u - v | $. ĠNote , Ġthat Ġthe Ġconc aten ation Ġmode Ġis Ġonly Ġrelevant Ġfor Ġtraining Ġthe Ġsoft max Ġclass ifier . ĠAt Ġinference , Ġwhen Ġpredicting Ġsimilarities Ġfor Ġthe ĠST S Ġbenchmark Ġdataset , Ġonly Ġthe Ġsentence Ġembed d ings Ġ$ u $ Ġand Ġ$ v $ Ġare Ġused Ġin Ġcombination Ġwith Ġcos ine - similar ity . ĠThe Ġelement - wise Ġdifference Ġmeasures Ġthe Ġdistance Ġbetween Ġthe Ġdimensions Ġof Ġthe Ġtwo Ġsentence Ġembed d ings , Ġensuring Ġthat Ġsimilar Ġpairs Ġare Ġcloser Ġand Ġdiss imilar Ġpairs Ġare Ġfurther Ġapart . </s> When Ġtrained Ġwith Ġthe Ġregression Ġobjective Ġfunction , Ġwe Ġobserve Ġthat Ġthe Ġpool ing Ġstrategy Ġhas Ġa Ġlarge Ġimpact . ĠThere , Ġthe ĠMAX Ġstrategy Ġperform Ġsignificantly Ġworse Ġthan ĠME AN Ġor ĠCLS - token Ġstrategy . ĠThis Ġis Ġin Ġcontrast Ġto ĠB IB REF 4 , Ġwho Ġfound Ġit Ġbeneficial Ġfor Ġthe ĠBi L ST M - layer Ġof ĠIn fer Sent Ġto Ġuse ĠMAX Ġinstead Ġof ĠME AN Ġpool ing . </s> Com put ational ĠEfficiency </s> Sent ence Ġembed d ings Ġneed Ġpotentially Ġbe Ġcomputed Ġfor ĠMillions Ġof Ġsentences , Ġhence , Ġa Ġhigh Ġcomputation Ġspeed Ġis Ġdesired . ĠIn Ġthis Ġsection , Ġwe Ġcompare ĠS BER T Ġto Ġaverage ĠGl o Ve Ġembed d ings , ĠIn fer Sent ĠB IB REF 4 , Ġand ĠUniversal ĠSent ence ĠEnc oder ĠB IB REF 5 . </s> For Ġour Ġcomparison Ġwe Ġuse Ġthe Ġsentences Ġfrom Ġthe ĠST S Ġbenchmark ĠB IB REF 10 . ĠWe Ġcompute Ġaverage ĠGl o Ve Ġembed d ings Ġusing Ġa Ġsimple Ġfor - loop Ġwith Ġpython Ġdictionary Ġlook ups Ġand ĠNum Py . ĠIn fer Sent Ġis Ġbased Ġon ĠPy Tor ch . ĠFor ĠUniversal ĠSent ence ĠEnc oder , Ġwe Ġuse Ġthe ĠT ensor Flow ĠHub Ġversion , Ġwhich Ġis Ġbased Ġon ĠT ensor Flow . ĠS BER T Ġis Ġbased Ġon ĠPy Tor ch . ĠFor Ġimproved Ġcomputation Ġof Ġsentence Ġembed d ings , Ġwe Ġimplemented Ġa Ġsmart Ġbatch ing Ġstrategy : ĠSent ences Ġwith Ġsimilar Ġlengths Ġare Ġgrouped Ġtogether Ġand Ġare Ġonly Ġpadded Ġto Ġthe Ġlongest Ġelement Ġin Ġa Ġmini - batch . ĠThis Ġdrastically Ġreduces Ġcomputational Ġoverhead Ġfrom Ġpadding Ġtokens . </s> Per form ances Ġwere Ġmeasured Ġon Ġa Ġserver Ġwith ĠIntel Ġi 7 - 58 20 K ĠCPU Ġ@ Ġ3 . 30 GHz , ĠNvidia ĠTesla ĠV 100 ĠGPU , ĠCU DA Ġ9 . 2 Ġand Ġcu D NN . ĠThe Ġresults Ġare Ġdepicted Ġin ĠTable ĠT AB REF 26 . </s> On ĠCPU , ĠIn fer Sent Ġis Ġabout Ġ65 % Ġfaster Ġthan ĠS BER T . ĠThis Ġis Ġdue Ġto Ġthe Ġmuch Ġsimpler Ġnetwork Ġarchitecture . ĠIn fer Sent Ġuses Ġa Ġsingle ĠBi L ST M Ġlayer , Ġwhile ĠB ERT Ġuses Ġ12 Ġstacked Ġtransformer Ġlayers . ĠHowever , Ġan Ġadvantage Ġof Ġtransformer Ġnetworks Ġis Ġthe Ġcomputational Ġefficiency Ġon ĠGPUs . ĠThere , ĠS BER T Ġwith Ġsmart Ġbatch ing Ġis Ġabout Ġ9 % Ġfaster Ġthan ĠIn fer Sent Ġand Ġabout Ġ55 % Ġfaster Ġthan ĠUniversal ĠSent ence ĠEnc oder . ĠSmart Ġbatch ing Ġachieves Ġa Ġspeed - up Ġof Ġ89 % Ġon ĠCPU Ġand Ġ48 % Ġon ĠGPU . ĠAverage ĠGl o Ve Ġembed d ings Ġis Ġobviously Ġby Ġa Ġlarge Ġmargin Ġthe Ġfastest Ġmethod Ġto Ġcompute Ġsentence Ġembed d ings . </s> Conclusion </s> We Ġshowed Ġthat ĠB ERT Ġout - of - the - box Ġmaps Ġsentences Ġto Ġa Ġvector Ġspace Ġthat Ġis Ġrather Ġunsu itable Ġto Ġbe Ġused Ġwith Ġcommon Ġsimilarity Ġmeasures Ġlike Ġcos ine - similar ity . ĠThe Ġperformance Ġfor Ġseven ĠST S Ġtasks Ġwas Ġbelow Ġthe Ġperformance Ġof Ġaverage ĠGl o Ve Ġembed d ings . </s> To Ġovercome Ġthis Ġshort coming , Ġwe Ġpresented ĠSent ence - BER T Ġ( S BER T ). ĠS BER T Ġfine - t unes ĠB ERT Ġin Ġa Ġsi ames e Ġ/ Ġtriple t Ġnetwork Ġarchitecture . ĠWe Ġevaluated Ġthe Ġquality Ġon Ġvarious Ġcommon Ġbenchmarks , Ġwhere Ġit Ġcould Ġachieve Ġa Ġsignificant Ġimprovement Ġover Ġstate - of - the - art Ġsentence Ġembed d ings Ġmethods . ĠRepl acing ĠB ERT Ġwith ĠRo BER Ta Ġdid Ġnot Ġyield Ġa Ġsignificant Ġimprovement Ġin Ġour Ġexperiments . </s> S BER T Ġis Ġcomput ationally Ġefficient . ĠOn Ġa ĠGPU , Ġit Ġis Ġabout Ġ9 % Ġfaster Ġthan ĠIn fer Sent Ġand Ġabout Ġ55 % Ġfaster Ġthan ĠUniversal ĠSent ence ĠEnc oder . ĠS BER T Ġcan Ġbe Ġused Ġfor Ġtasks Ġwhich Ġare Ġcomput ationally Ġnot Ġfeasible Ġto Ġbe Ġmodeled Ġwith ĠB ERT . ĠFor Ġexample , Ġclust ering Ġof Ġ10 , 000 Ġsentences Ġwith Ġhierarchical Ġclust ering Ġrequires Ġwith ĠB ERT Ġabout Ġ65 Ġhours , Ġas Ġaround Ġ50 ĠMillion Ġsentence Ġcombinations Ġmust Ġbe Ġcomputed . ĠWith ĠS BER T , Ġwe Ġwere Ġable Ġto Ġreduce Ġthe Ġeffort Ġto Ġabout Ġ5 Ġseconds . </s> Acknowled gments </s> This Ġwork Ġhas Ġbeen Ġsupported Ġby Ġthe ĠGerman ĠResearch ĠFoundation Ġthrough Ġthe ĠGerman - Israeli ĠProject ĠCooperation Ġ( D IP , Ġgrant ĠDA Ġ1600 / 1 - 1 Ġand Ġgrant ĠGU Ġ7 98 / 17 - 1 ). ĠIt Ġhas Ġbeen Ġco - funded Ġby Ġthe ĠGerman ĠFederal ĠMinistry Ġof ĠEducation Ġand ĠResearch Ġ( B MB F ) Ġunder Ġthe Ġpromotional Ġreferences Ġ03 VP 025 40 Ġ( Arg umen Text ).