No
BOOLEAN
No
BOOLEAN
This section gives detailed experiment settings. The method is evaluated on the BERTbase model, which has 12 layers, 12 self-attention heads with a hidden size of 768. To accelerate the training speed, two-phase training BIBREF1 is adopted. The first phase uses a maximal sentence length of 128, and 512 for the second phase. The numbers of training steps of two phases are 50K and 40K for the BERTBase model. We used AdamW BIBREF13 optimizer with a learning rate of 1e-4, a $\beta _1$ of 0.9, a $\beta _2$ of 0.999 and a L2 weight decay rate of $0.01$. The first 10% of the total steps are used for learning rate warming up, followed by the linear decay schema. We used a dropout probability of 0.1 on all layers. The data used for pre-training is the same as BERT, i.e., English Wikipedia (2500M words) and BookCorpus (800M words) BIBREF14. For the Masked LM task, we followed the same masking rate and settings as in BERT.
This section gives detailed experiment settings. The method is evaluated on the BERTbase model, which has 12 layers, 12 self-attention heads with a hidden size of 768. To accelerate the training speed, two-phase training BIBREF1 is adopted. The first phase uses a maximal sentence length of 128, and 512 for the second phase. The numbers of training steps of two phases are 50K and 40K for the BERTBase model. We used AdamW BIBREF13 optimizer with a learning rate of 1e-4, a $\beta _1$ of 0.9, a $\beta _2$ of 0.999 and a L2 weight decay rate of $0.01$. The first 10% of the total steps are used for learning rate warming up, followed by the linear decay schema. We used a dropout probability of 0.1 on all layers. The data used for pre-training is the same as BERT, i.e., English Wikipedia (2500M words) and BookCorpus (800M words) BIBREF14. For the Masked LM task, we followed the same masking rate and settings as in BERT.
1909.03405
Introduction </s> The Ġability Ġof Ġsemantic Ġreasoning Ġis Ġessential Ġfor Ġadvanced Ġnatural Ġlanguage Ġunderstanding Ġ( NL U ) Ġsystems . ĠMany ĠNL U Ġtasks Ġthat Ġtake Ġsentence Ġpairs Ġas Ġinput , Ġsuch Ġas Ġnatural Ġlanguage Ġinference Ġ( N LI ) Ġand Ġmachine Ġreading Ġcomprehension Ġ( M RC ), Ġheavily Ġrely Ġon Ġthe Ġability Ġof Ġsophisticated Ġsemantic Ġreasoning . ĠFor Ġinstance , Ġthe ĠNL I Ġtask Ġaims Ġto Ġdetermine Ġwhether Ġthe Ġhypothesis Ġsentence Ġ( e . g ., Ġa Ġwoman Ġis Ġsleeping ) Ġcan Ġbe Ġinferred Ġfrom Ġthe Ġpremise Ġsentence Ġ( e . g ., Ġa Ġwoman Ġis Ġtalking Ġon Ġthe Ġphone ). ĠThis Ġrequires Ġthe Ġmodel Ġto Ġread Ġand Ġunderstand Ġsentence Ġpairs Ġto Ġmake Ġthe Ġspecific Ġsemantic Ġinference . </s> B id irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTrans former Ġ( BER T ) ĠB IB REF 1 Ġhas Ġshown Ġstrong Ġability Ġin Ġsemantic Ġreasoning . ĠIt Ġwas Ġrecently Ġproposed Ġand Ġobtained Ġimpressive Ġresults Ġon Ġmany Ġtasks , Ġranging Ġfrom Ġtext Ġclassification , Ġnatural Ġlanguage Ġinference , Ġand Ġmachine Ġreading Ġcomprehension . ĠB ERT Ġachieves Ġthis Ġby Ġemploying Ġtwo Ġobjectives Ġin Ġthe Ġpre - training , Ġi . e ., Ġthe Ġmasked Ġlanguage Ġmodeling Ġ( Mask ed ĠLM ) Ġand Ġthe Ġnext Ġsentence Ġprediction Ġ( N SP ). ĠInt uitive ly , Ġthe ĠMask ed ĠLM Ġtask Ġconcerns Ġword - level Ġknowledge , Ġand Ġthe ĠN SP Ġtask Ġcaptures Ġthe Ġglobal Ġdocument - level Ġinformation . ĠThe Ġgoal Ġof ĠN SP Ġis Ġto Ġidentify Ġwhether Ġan Ġinput Ġsentence Ġis Ġnext Ġto Ġanother Ġinput Ġsentence . ĠFrom Ġthe Ġab lation Ġstudy ĠB IB REF 1 , Ġthe ĠN SP Ġtask Ġis Ġquite Ġuseful Ġfor Ġthe Ġdownstream ĠNL I Ġand ĠM RC Ġtasks Ġ( e . g ., Ġ+ 3 . 5 % Ġabsolute Ġgain Ġon Ġthe ĠQuestion ĠNL I Ġ( Q N LI ) ĠB IB REF 2 Ġtask ). </s> Despite Ġits Ġusefulness , Ġwe Ġsuggest Ġthat ĠB ERT Ġhas Ġnot Ġmade Ġfull Ġuse Ġof Ġthe Ġdocument - level Ġknowledge . ĠThe Ġsentences Ġin Ġthe Ġnegative Ġsamples Ġused Ġin ĠN SP Ġare Ġrandomly Ġdrawn Ġfrom Ġother Ġdocuments . ĠTherefore , Ġto Ġdiscriminate Ġagainst Ġthese Ġsentences , ĠB ERT Ġis Ġprone Ġto Ġaggreg ating Ġthe Ġshallow Ġsemantic , Ġe . g ., Ġtopic , Ġneglect ing Ġcontext Ġclues Ġuseful Ġfor Ġdetailed Ġreasoning . ĠIn Ġother Ġwords , Ġthe Ġcanonical ĠN SP Ġtask Ġwould Ġencourage Ġthe Ġmodel Ġto Ġrecognize Ġthe Ġcorrelation Ġbetween Ġsentences , Ġrather Ġthan Ġobtaining Ġthe Ġability Ġof Ġsemantic Ġentail ment . ĠThis Ġsetting Ġweak ens Ġthe ĠB ERT Ġmodel Ġfrom Ġlearning Ġspecific Ġsemantic Ġfor Ġinference . ĠAnother Ġissue Ġthat Ġrenders ĠN SP Ġless Ġeffective Ġis Ġthat ĠB ERT Ġis Ġorder - sensitive . ĠPerformance Ġdegradation Ġwas Ġobserved Ġon Ġtypical ĠNL I Ġtasks Ġwhen Ġthe Ġorder Ġof Ġtwo Ġinput Ġsentences Ġare Ġreversed Ġduring Ġthe ĠB ERT Ġfine - tun ing Ġphase . ĠIt Ġis Ġreasonable Ġas Ġthe ĠN SP Ġtask Ġcan Ġbe Ġroughly Ġanalogy Ġto Ġthe ĠNL I Ġtask Ġwhen Ġthe Ġinput Ġcomes Ġas Ġ( prem ise , Ġhypothesis ), Ġconsidering Ġthe Ġcausal Ġorder Ġamong Ġsentences . ĠHowever , Ġthis Ġidentity Ġbetween ĠN SP Ġand ĠNL I Ġis Ġcompromised Ġwhen Ġthe Ġsentences Ġare Ġswapped . </s> Based Ġon Ġthese Ġconsiderations , Ġwe Ġpropose Ġa Ġsimple Ġyet Ġeffective Ġmethod , Ġi . e ., Ġintroducing Ġa ĠIs Prev Ġcategory Ġto Ġthe Ġclassification Ġtask , Ġwhich Ġis Ġa Ġsymm etric Ġlabel Ġof ĠIs Next Ġof ĠN SP . ĠThe Ġinput Ġof Ġsamples Ġwith ĠIs Prev Ġis Ġthe Ġreverse Ġof Ġthose Ġwith ĠIs Next Ġlabel . ĠThe Ġadvantages Ġof Ġusing Ġthis Ġprevious Ġsentence Ġprediction Ġ( PS P ) Ġare Ġthree Ġfolds . Ġ( 1 ) ĠLearning Ġthe Ġcontrast Ġbetween ĠN SP Ġand ĠPSP Ġforces Ġthe Ġmodel Ġto Ġextract Ġmore Ġdetailed Ġsemantic , Ġthereby Ġthe Ġmodel Ġis Ġmore Ġcapable Ġof Ġdiscriminating Ġthe Ġcorrelation Ġand Ġentail ment . Ġ( 2 ) ĠN SP Ġand ĠPSP Ġare Ġsymm etric . ĠThis Ġsymm etric Ġregular ization Ġallev iates Ġthe Ġinfluence Ġof Ġthe Ġorder Ġof Ġthe Ġinput Ġpair . Ġ( 3 ) ĠEm pir ical Ġresults Ġindicate Ġthat Ġour Ġmethod Ġis Ġbeneficial Ġfor Ġall Ġthe Ġsemantic Ġreasoning Ġtasks Ġthat Ġtake Ġsentence Ġpair Ġas Ġinput . </s> In Ġaddition , Ġto Ġfurther Ġincorporating Ġthe Ġdocument - level Ġknowledge , ĠN SP Ġand ĠPSP Ġare Ġextended Ġwith Ġnon - success ive Ġsentences , Ġwhere Ġthe Ġlabel Ġsmoot hing Ġtechnique Ġis Ġadopted . ĠThe Ġproposed Ġmethod Ġyields Ġa Ġconsiderable Ġimprovement Ġin Ġour Ġexperiments . ĠWe Ġevaluate Ġthe Ġability Ġof Ġsemantic Ġreasoning Ġon Ġstandard ĠNL I Ġand ĠM RC Ġbenchmarks , Ġincluding Ġthe Ġchallenging ĠH ANS Ġdataset ĠB IB REF 0 . ĠAnaly tical Ġwork Ġon Ġthe ĠH ANS Ġdataset Ġprovides Ġa Ġmore Ġcomprehens ible Ġperspective Ġtowards Ġthe Ġproposed Ġmethod . ĠFurthermore , Ġthe Ġresults Ġon Ġthe ĠChinese Ġbenchmarks Ġare Ġprovided Ġto Ġdemonstrate Ġits Ġgener ality . </s> In Ġsummary , Ġthis Ġwork Ġmakes Ġthe Ġfollowing Ġcontributions : </s> The Ġsupervision Ġsignal Ġfrom Ġthe Ġoriginal ĠN SP Ġtask Ġis Ġweak Ġfor Ġsemantic Ġinference . ĠTherefore , Ġa Ġnovel Ġmethod Ġis Ġproposed Ġto Ġremedy Ġthe Ġasymm etric Ġissue Ġand Ġenhance Ġthe Ġreasoning Ġability . </s> Both Ġempirical Ġand Ġanalytical Ġevaluations Ġare Ġprovided Ġon Ġthe ĠNL I Ġand ĠM RC Ġdatasets , Ġwhich Ġver ifies Ġthe Ġeffectiveness Ġof Ġusing Ġmore Ġdocument - level Ġknowledge . </s> Related ĠWork Ġ: :: ĠPair - wise Ġsemantic Ġreasoning </s> Many ĠNL U Ġtasks Ġseek Ġto Ġmodel Ġthe Ġrelationship Ġbetween Ġtwo Ġsentences . ĠSem antic Ġreasoning Ġis Ġperformed Ġon Ġthe Ġsentence Ġpair Ġfor Ġthe Ġtask - specific Ġinference . ĠPair - wise Ġsemantic Ġreasoning Ġtasks Ġhave Ġdrawn Ġa Ġlot Ġof Ġattention Ġfrom Ġthe ĠN LP Ġcommunity Ġas Ġthey Ġlargely Ġrequire Ġthe Ġcomprehension Ġability Ġof Ġthe Ġlearning Ġsystems . ĠRecently , Ġthe Ġsignificant Ġimprovement Ġon Ġthese Ġbenchmarks Ġcomes Ġfrom Ġthe Ġpre - training Ġmodels , Ġe . g ., ĠB ERT , ĠStruct BER T ĠB IB REF 3 , ĠER N IE ĠB IB REF 4 , ĠB IB REF 5 , ĠRo BER Ta ĠB IB REF 6 Ġand ĠXL Net ĠB IB REF 7 . ĠThese Ġmodels Ġlearn Ġfrom Ġun super vised / self - super vised Ġobjectives Ġand Ġperform Ġex cell ently Ġin Ġthe Ġdownstream Ġtasks . ĠAmong Ġthese Ġmodels , ĠB ERT Ġadop ts ĠN SP Ġas Ġone Ġof Ġthe Ġobjectives Ġin Ġthe Ġpre - training Ġand Ġshows Ġthat Ġthe ĠN SP Ġtask Ġhas Ġa Ġpositive Ġeffect Ġon Ġthe ĠNL I Ġand ĠM RC Ġtasks . ĠAlthough Ġthe Ġprimary Ġstudy Ġof ĠXL Net Ġand ĠRo BER Ta Ġsuggests Ġthat ĠN SP Ġis Ġineffective Ġwhen Ġthe Ġmodel Ġis Ġtrained Ġwith Ġa Ġlarge Ġsequence Ġlength Ġof Ġ512 , Ġthe Ġeffect Ġof ĠN SP Ġon Ġthe ĠNL I Ġproblems Ġshould Ġstill Ġbe Ġemphasized . ĠThe Ġin efficiency Ġof ĠN SP Ġis Ġlikely Ġbecause Ġthe Ġexpected Ġcontext Ġlength Ġwill Ġbe Ġhal ved Ġfor ĠMask ed ĠLM Ġwhen Ġtaking Ġa Ġsentence Ġpair Ġas Ġthe Ġinput . ĠThe Ġmodels Ġderived Ġfrom ĠB ERT , Ġe . g ., ĠStruct BER T Ġand ĠER N IE Ġ1 . 0 / 2 . 0 , Ġaim Ġto Ġincorporating Ġmore Ġknowledge Ġby Ġelabor ating Ġpre - training Ġobjectives . ĠThis Ġwork Ġaims Ġto Ġenhance Ġthe ĠN SP Ġtask Ġand Ġver ifies Ġwhether Ġdocument - level Ġinformation Ġis Ġhelpful Ġfor Ġthe Ġpre - training . ĠTo Ġprobe Ġwhether Ġour Ġmethod Ġachieves Ġa Ġbetter Ġregular ization Ġability , Ġour Ġapproach Ġis Ġalso Ġevaluated Ġon Ġthe ĠH ANS ĠB IB REF 0 Ġdataset , Ġwhich Ġcontains Ġhard Ġdata Ġsamples Ġconstructed Ġby Ġthree Ġhe ur istics . ĠPrevious Ġadvanced Ġmodels Ġsuch Ġas ĠB ERT Ġfail Ġon Ġthe ĠH ANS Ġdataset , Ġand Ġthe Ġtest Ġaccuracy Ġcan Ġbarely Ġexceed Ġ0 % Ġin Ġthe Ġsubset Ġof Ġtest Ġexamples . </s> Related ĠWork Ġ: :: ĠUn super vised Ġlearning Ġfrom Ġdocument </s> In Ġrecent Ġyears , Ġmany Ġun super vised Ġpre - training Ġmethods Ġhave Ġbeen Ġproposed Ġin Ġthe ĠN LP Ġfields Ġto Ġextract Ġknowledge Ġamong Ġsentences ĠD BL P : conf / n ips / K i ros Z S Z UTF 15 , D BL P : conf / em nl p / Con neau KS BB 17 , D BL P : conf / ic lr / Log es war an L 18 , D BL P : j ournals / cor r / abs - 19 03 - 09 424 . ĠThe Ġprediction Ġof Ġsurrounding Ġsentences Ġend ows Ġthe Ġmodel Ġwith Ġthe Ġability Ġto Ġmodel Ġthe Ġsentence - level Ġco herence . ĠSkip - Th ought ĠB IB REF 8 Ġconsists Ġof Ġan Ġenc oder Ġand Ġtwo Ġdec od ers . ĠWhen Ġa Ġsentence Ġis Ġgiven Ġand Ġencoded Ġinto Ġa Ġvector Ġby Ġthe Ġenc oder , Ġthe Ġdec od ers Ġare Ġtrained Ġto Ġpredict Ġthe Ġnext Ġsentence Ġand Ġthe Ġprevious Ġsentence . ĠThe Ġgoal Ġis Ġto Ġobtain Ġa Ġbetter Ġsentence Ġrepresentation Ġthat Ġis Ġuseful Ġfor Ġreconstruct ing Ġthe Ġsurrounding Ġcontext . ĠConsidering Ġthat Ġthe Ġestimation Ġof Ġthe Ġlikelihood Ġof Ġsequences Ġis Ġcomput ationally Ġexpensive Ġand Ġtime - consuming , Ġthe ĠQuick - Th ought Ġmethod ĠB IB REF 9 Ġsimpl ifies Ġthis Ġin Ġa Ġmanner Ġsimilar Ġto Ġsampled Ġsoft max ĠB IB REF 10 , Ġwhich Ġclass ifies Ġthe Ġinput Ġsentences Ġbetween Ġsurrounding Ġsentences Ġand Ġthe Ġother . ĠNote Ġthat ĠQuick - Th ought Ġdoes Ġnot Ġdistinguish Ġbetween Ġthe Ġprevious Ġand Ġnext Ġsentence Ġas Ġit Ġis Ġfunctionally Ġrotation Ġinvari ant . ĠHowever , ĠB ERT Ġis Ġorder - dependent , Ġand Ġthe Ġdiscrimination Ġcan Ġprovide Ġmore Ġsupervision Ġsignal Ġfor Ġsemantic Ġlearning . ĠIn fer Sent ĠB IB REF 11 Ġinstead Ġpre - tr ains Ġthe Ġmodel Ġin Ġa Ġmanner Ġof Ġsupervised Ġlearning . ĠIt Ġuses Ġa Ġlarge - scale ĠNL I Ġdataset Ġas Ġthe Ġpre - training Ġtask Ġto Ġlearn Ġthe Ġsentence Ġrepresentation . ĠIn Ġour Ġwork , Ġwe Ġfocus Ġon Ġdesigning Ġa Ġmore Ġeffective Ġdocument - level Ġobjective , Ġextended Ġfrom Ġthe ĠN SP Ġtask . ĠThe Ġproposed Ġmethod Ġwill Ġbe Ġdescribed Ġin Ġthe Ġfollowing Ġsection Ġand Ġvalidated Ġby Ġproviding Ġextensive Ġexperimental Ġresults Ġin Ġthe Ġexperiment Ġpart . </s> Method </s> Our Ġmethod Ġfollows Ġthe Ġsame Ġinput Ġformat Ġand Ġthe Ġmodel Ġarchitecture Ġwith Ġoriginal ĠB ERT . ĠThe Ġproposed Ġmethod Ġsolely Ġconcerns Ġthe ĠN SP Ġtask . ĠThe ĠN SP Ġtask Ġis Ġa Ġbinary Ġclassification Ġtask , Ġwhich Ġtakes Ġtwo Ġsentences Ġ( A Ġand ĠB ) Ġas Ġinput Ġand Ġdetermines Ġwhether ĠB Ġis Ġthe Ġnext Ġsentence Ġof ĠA . ĠAlthough Ġit Ġhas Ġbeen Ġproven Ġto Ġbe Ġvery Ġeffective Ġfor ĠB ERT , Ġthere Ġare Ġtwo Ġmajor Ġdeficiencies . Ġ( 1 ) ĠDiscrimination Ġbetween ĠIs Next Ġand ĠDiff Doc Ġ( the Ġlabel Ġof Ġthe Ġsentences Ġdrawn Ġfrom Ġdifferent Ġdocuments Ġvia Ġnegative Ġsampling ) Ġis Ġsem antically Ġshallow Ġas Ġthe Ġsignal Ġof Ġsentence Ġorder Ġis Ġabsent . ĠThe Ġcorrelation Ġbetween Ġtwo Ġsuccessive Ġsentences Ġcould Ġbe Ġobvious , Ġdue Ġto , Ġfor Ġexample , Ġlex ical Ġoverlap Ġor Ġthe Ġconjunction Ġused Ġat Ġthe Ġbeginning Ġof Ġthe Ġsecond Ġsentence . ĠAs Ġreported ĠB IB REF 1 , Ġthe Ġfinal Ġpre - trained Ġmodel Ġis Ġable Ġto Ġachieve Ġ97 %- 98 % Ġaccuracy Ġon Ġthe ĠN SP Ġtask . Ġ( 2 ) ĠB ERT Ġis Ġorder - sensitive , Ġi . e ., Ġ$ f _{ \ text { BER T }} ( Ġ\ text tt Ġ{ A }, Ġ\ text tt Ġ{ B }) Ġ\ ne Ġf _{ \ text { BER T }} (\ text tt Ġ{ B }, Ġ\ text tt Ġ{ A }) $, Ġwhile ĠN SP Ġis Ġun i - direction al . ĠWhen Ġthe Ġorder Ġof Ġthe Ġinput ĠNL I Ġpair Ġis Ġreversed , Ġthe Ġperformance Ġwill Ġdegrade . ĠFor Ġinstance , Ġthe Ġaccuracy Ġdecreases Ġby Ġabout Ġ0 . 5 % Ġon ĠMN LI ĠB IB REF 12 Ġand Ġ0 . 4 % Ġon ĠQ N LI Ġafter Ġswapping Ġthe Ġsentences Ġin Ġour Ġexperiments Ġ. </s> Mot ivated Ġby Ġthese Ġproblems , Ġwe Ġpropose Ġto Ġextend Ġthe ĠN SP Ġtask Ġwith Ġprevious Ġsentence Ġprediction Ġ( PS P ). ĠDespite Ġits Ġsimplicity , Ġempirical Ġresults Ġshow Ġthat Ġthis Ġis Ġbeneficial Ġfor Ġdownstream Ġtasks , Ġincluding Ġboth ĠNL I Ġand ĠM RC Ġtasks . ĠTo Ġfurther Ġincorporate Ġthe Ġdocument - level Ġinformation , Ġthe Ġscope Ġis Ġalso Ġexpanded Ġto Ġinclude Ġmore Ġsurrounding Ġsentences , Ġnot Ġjust Ġthe Ġadjacent . ĠThe Ġmethod Ġis Ġbriefly Ġillustrated Ġin ĠFig . ĠFIG REF 6 . </s> Method Ġ: :: ĠPrevious ĠSent ence ĠPrediction </s> Learning Ġto Ġrecognize Ġthe Ġprevious Ġsentence Ġenables Ġthe Ġmodel Ġto Ġcapture Ġmore Ġcompact Ġcontext Ġinformation . ĠOne Ġwould Ġargue Ġthat ĠIs Prev Ġ( the Ġlabel Ġof ĠPSP ) Ġis Ġredundant Ġas Ġit Ġplays Ġa Ġsimilar Ġrole Ġof ĠIs Next Ġ( the Ġlabel Ġof ĠN SP ). ĠIn Ġfact , ĠQuick - Th ought Ġuses Ġthe Ġsampled Ġsoft max Ġto Ġapproximate Ġthe Ġsentence Ġlikelihood Ġestimation Ġof ĠSkip - Th ought , Ġand Ġit Ġactually Ġdoes Ġnot Ġdifferentiate Ġbetween Ġthe Ġprevious Ġand Ġnext Ġsentences . ĠHowever , Ġwe Ġsuggest Ġthe Ġorder Ġdiscrimination Ġis Ġessential Ġfor ĠB ERT Ġpre - training . ĠQuick - Th ought Ġaims Ġat Ġextracting Ġsentence Ġembed ding , Ġand Ġit Ġuses Ġa Ġrotating Ġsymm etric Ġfunction , Ġwhich Ġmakes ĠIs Prev Ġredundant Ġin ĠQuick - Th ought . ĠIn Ġcontrast , ĠB ERT Ġis Ġorder - sensitive , Ġand Ġlearning Ġthe Ġsymm etric Ġregular ization Ġis Ġrather Ġnecessary . ĠAnother Ġadvantage Ġof ĠPSP Ġis Ġto Ġenhance Ġdocument - level Ġsupervision . ĠIn Ġorder Ġto Ġtell Ġthe Ġdifference Ġbetween ĠN SP Ġand ĠPSP , Ġthe Ġmodel Ġhas Ġto Ġextract Ġthe Ġdetailed Ġsemantic Ġfor Ġinference . </s> Method Ġ: :: ĠGathering ĠMore ĠDocument - level ĠInformation </s> Beyond ĠN SP Ġand ĠPSP , Ġwhich Ġenable Ġthe Ġmodel Ġto Ġlearn Ġthe Ġshort - term Ġdependency Ġbetween Ġsentences , Ġwe Ġalso Ġpropose Ġto Ġexpand Ġthe Ġscope Ġof Ġdiscrimination Ġtask Ġto Ġfurther Ġincorporate Ġthe Ġdocument - level Ġinformation . </s> Specifically , Ġwe Ġalso Ġinclude Ġthe Ġin - adj acent Ġsentences Ġin Ġthe Ġsentence - pair Ġclassification Ġtask . ĠThe Ġin - adj acent Ġsentences Ġnext Ġto Ġthe ĠIs Prev Ġand ĠIs Next Ġsentences Ġare Ġsampled , Ġlabeled Ġas ĠIs Prev In adj Ġand ĠIs Next In adj Ġ( cf . Ġthe Ġbottom Ġof ĠFig . ĠFIG REF 6 ). ĠNote Ġthat Ġthese Ġin - adj acent Ġsentences Ġwill Ġintroduce Ġmuch Ġmore Ġtraining Ġnoise Ġto Ġthe Ġmodel . ĠTherefore , Ġthe Ġlabel Ġsmoot hing Ġtechnique Ġis Ġadopted Ġto Ġreduce Ġthe Ġnoise Ġof Ġthese Ġadditional Ġsamples . ĠIt Ġachieves Ġthis Ġby Ġrelaxing Ġour Ġconfidence Ġon Ġthe Ġlabels , Ġe . g ., Ġtransforming Ġthe Ġtarget Ġprobability Ġfrom Ġ( 1 . 0 , Ġ0 . 0 ) Ġto Ġ( 0 . 8 , Ġ0 . 2 ) Ġin Ġa Ġbinary Ġclassification Ġproblem . </s> In Ġsummary , Ġwhen ĠA Ġis Ġgiven , Ġthe Ġpre - training Ġexample Ġfor Ġeach Ġlabel Ġis Ġconstructed Ġas Ġfollows : </s> Is Next : ĠCho osing Ġthe Ġadjacent Ġfollowing Ġsentence Ġas ĠB . </s> Is Prev : ĠCho osing Ġthe Ġadjacent Ġprevious Ġsentence Ġas ĠB . </s> Is Next In adj : ĠCho osing Ġthe Ġin - adj acent Ġfollowing Ġsentence Ġas ĠB . ĠThere Ġis Ġa Ġsentence Ġbetween ĠA Ġand ĠB . </s> Is Prev In adj : ĠCho osing Ġthe Ġin - adj acent Ġprevious Ġsentence Ġas ĠB . ĠThere Ġis Ġa Ġsentence Ġbetween ĠA Ġand ĠB . </s> Diff Doc : ĠDrawing ĠB Ġrandomly Ġfrom Ġa Ġdifferent Ġdocument . </s> Exper iment ĠSettings </s> This Ġsection Ġgives Ġdetailed Ġexperiment Ġsettings . ĠThe Ġmethod Ġis Ġevaluated Ġon Ġthe ĠB ERT base Ġmodel , Ġwhich Ġhas Ġ12 Ġlayers , Ġ12 Ġself - att ention Ġheads Ġwith Ġa Ġhidden Ġsize Ġof Ġ768 . </s> To Ġaccelerate Ġthe Ġtraining Ġspeed , Ġtwo - phase Ġtraining ĠB IB REF 1 Ġis Ġadopted . ĠThe Ġfirst Ġphase Ġuses Ġa Ġmaximal Ġsentence Ġlength Ġof Ġ128 , Ġand Ġ512 Ġfor Ġthe Ġsecond Ġphase . ĠThe Ġnumbers Ġof Ġtraining Ġsteps Ġof Ġtwo Ġphases Ġare Ġ50 K Ġand Ġ40 K Ġfor Ġthe ĠB ERT Base Ġmodel . ĠWe Ġused ĠAdam W ĠB IB REF 13 Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ1 e - 4 , Ġa Ġ$\ beta Ġ_ 1 $ Ġof Ġ0 . 9 , Ġa Ġ$\ beta Ġ_ 2 $ Ġof Ġ0 . 999 Ġand Ġa ĠL 2 Ġweight Ġdecay Ġrate Ġof Ġ$ 0 . 01 $. ĠThe Ġfirst Ġ10 % Ġof Ġthe Ġtotal Ġsteps Ġare Ġused Ġfor Ġlearning Ġrate Ġwarming Ġup , Ġfollowed Ġby Ġthe Ġlinear Ġdecay Ġschema . ĠWe Ġused Ġa Ġdrop out Ġprobability Ġof Ġ0 . 1 Ġon Ġall Ġlayers . ĠThe Ġdata Ġused Ġfor Ġpre - training Ġis Ġthe Ġsame Ġas ĠB ERT , Ġi . e ., ĠEnglish ĠWikipedia Ġ( 2500 M Ġwords ) Ġand ĠBook Corp us Ġ( 800 M Ġwords ) ĠB IB REF 14 . ĠFor Ġthe ĠMask ed ĠLM Ġtask , Ġwe Ġfollowed Ġthe Ġsame Ġmask ing Ġrate Ġand Ġsettings Ġas Ġin ĠB ERT . </s> We Ġexplore Ġthree Ġmethod Ġsettings Ġfor Ġcomparison . </s> BER T - PN : ĠThe ĠN SP Ġtask Ġin ĠB ERT Ġis Ġreplaced Ġby Ġa Ġ3 - class Ġtask Ġwith ĠIs Next , ĠIs Prev Ġand ĠDiff Doc . ĠThe Ġlabel Ġdistribution Ġis Ġ1 : 1 : 1 . </s> BER T - PN 5 cl s : ĠThe ĠN SP Ġtask Ġin ĠB ERT Ġis Ġreplaced Ġby Ġa Ġ5 - class Ġtask Ġwith Ġtwo Ġadditional Ġlabels ĠIs Next In adj , ĠIs Prev In adj . ĠThe Ġlabel Ġdistribution Ġis Ġ1 : 1 : 1 : 1 : 1 . </s> BER T - PN sm th : ĠIt Ġuses Ġthe Ġsame Ġdata Ġwith ĠB ERT - PN 5 cl s , Ġexcept Ġthat Ġthe ĠIs Prev In adj Ġ( Is Next In adj ) Ġlabel Ġis Ġmapped Ġto ĠIs Prev Ġ( Is Next ) Ġwith Ġa Ġlabel Ġsmoot hing Ġfactor Ġof Ġ0 . 8 . </s> BER T - PN Ġis Ġused Ġto Ġverify Ġthe Ġfeasibility Ġof ĠPSP . ĠThe Ġcomparison Ġwith ĠB ERT - PN 5 cl s Ġillustrates Ġwhether Ġmore Ġdocument - level Ġinformation Ġhelps . ĠB ERT - PN sm th , Ġwhich Ġis Ġthe Ġlabel - sm oot hed Ġversion Ġof ĠB ERT - PN 5 cl s , Ġis Ġused Ġto Ġcompare Ġwith ĠB ERT - PN 5 cl s Ġto Ġsee Ġwhether Ġthe Ġnoise Ġreduction Ġis Ġnecessary . </s> In Ġthe Ġfollowing , Ġwe Ġfirst Ġshow Ġthat ĠB ERT Ġis Ġorder - sensitive Ġand Ġthe Ġuse Ġof ĠPSP Ġremedies Ġthis Ġproblem . ĠThen Ġwe Ġprovide Ġexperimental Ġresults Ġon Ġthe ĠNL I Ġand ĠM RC Ġtasks Ġto Ġverify Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġmethod . ĠAt Ġlast , Ġthe Ġproposed Ġmethod Ġis Ġevaluated Ġon Ġseveral ĠChinese Ġdatasets . </s> Order - inv ari ant Ġwith ĠPSP </s> N SP Ġin Ġthe Ġpre - training Ġis Ġuseful Ġfor ĠNL I Ġand ĠM RC Ġtask ĠB IB REF 1 . ĠHowever , Ġwe Ġsuggested Ġthat ĠB ERT Ġtrained Ġwith ĠN SP Ġis Ġorder - sensitive , Ġi . e ., Ġthe Ġperformance Ġof ĠB ERT Ġdepends Ġon Ġthe Ġorder Ġof Ġthe Ġinput Ġsentence Ġpair . ĠTo Ġverify Ġour Ġassumption , Ġa Ġprimary Ġexperiment Ġwas Ġconducted . ĠThe Ġorder Ġof Ġthe Ġinput Ġpair Ġof ĠNL I Ġsamples Ġis Ġreversed Ġin Ġthe Ġfine - tun ing Ġphase , Ġand Ġother Ġhyper - param eters Ġand Ġsettings Ġkeep Ġthe Ġsame Ġwith Ġthe ĠB ERT Ġpaper . ĠTable ĠT AB REF 19 Ġshows Ġthe Ġaccuracy Ġon Ġthe Ġvalidation Ġset Ġof Ġthe ĠMN LI Ġand ĠQ N LI Ġdatasets . ĠFor Ġthe ĠB ERT Base Ġmodel , Ġwhen Ġthe Ġsentences Ġare Ġswapped , Ġthe Ġaccuracy Ġdecreases Ġby Ġ0 . 5 % Ġon Ġthe ĠMN LI Ġtask Ġand Ġ0 . 4 % Ġon Ġthe ĠQ N LI Ġtask . ĠThese Ġresults Ġconfirm Ġthat ĠB ERT Ġtrained Ġwith ĠN SP Ġonly Ġis Ġindeed Ġaffected Ġby Ġthe Ġinput Ġorder . ĠThis Ġphenomenon Ġmotiv ates Ġus Ġto Ġmake Ġthe ĠN SP Ġtask Ġsymm etric . ĠThe Ġresults Ġof ĠB ERT - PN Ġverify Ġthat ĠB ERT - PN Ġis Ġorder - inv ari ant . ĠWhen Ġthe Ġinput Ġorder Ġis Ġreversed , Ġthe Ġperformance Ġof ĠB ERT - PN Ġremains Ġstable . ĠThese Ġresults Ġindicate Ġthat Ġour Ġmethod Ġis Ġable Ġto Ġremedy Ġthe Ġorder - s ensitivity Ġproblem . </s> Results Ġof ĠNL I ĠT asks Ġ: :: ĠGL UE </s> A Ġpopular Ġbenchmark Ġfor Ġevaluation Ġof Ġlanguage Ġunderstanding Ġis ĠGL UE ĠB IB REF 2 , Ġwhich Ġis Ġa Ġcollection Ġof Ġthree ĠNL I Ġtasks Ġ( MN LI , ĠQ N LI Ġand ĠR TE ), Ġthree Ġsemantic Ġtextual Ġsimilarity Ġ( ST S ) Ġtasks Ġ( Q Q P , ĠST S - B Ġand ĠMR PC ), Ġtwo Ġtext Ġclassification Ġ( TC ) Ġtasks Ġ( S ST - 2 Ġand ĠCo LA ). ĠAlthough Ġthe Ġmethod Ġis Ġmotivated Ġfor Ġpair - wise Ġreasoning , Ġthe Ġresults Ġof Ġother Ġproblems Ġare Ġalso Ġlisted . </s> Our Ġimplementation Ġfollows Ġthe Ġsame Ġway Ġthat ĠB ERT Ġperforms Ġin Ġthese Ġtasks . ĠThe Ġfine - tun ing Ġwas Ġconducted Ġfor Ġ3 Ġepoch s Ġfor Ġall Ġthe Ġtasks , Ġwith Ġa Ġlearning Ġrate Ġof Ġ2 e - 5 . ĠThe Ġpredictions Ġwere Ġobtained Ġby Ġevaluating Ġthe Ġtraining Ġcheckpoint Ġwith Ġthe Ġbest Ġvalidation Ġperformance . </s> Table ĠT AB REF 21 Ġillustrates Ġthe Ġexperimental Ġresults , Ġshowing Ġthat Ġour Ġmethod Ġis Ġbeneficial Ġfor Ġall Ġof ĠNL I Ġtasks . ĠThe Ġimprovement Ġon Ġthe ĠR TE Ġdataset Ġis Ġsignificant , Ġi . e ., Ġ4 % Ġabsolute Ġgain Ġover Ġthe ĠB ERT Base . ĠBesides ĠNL I , Ġour Ġmodel Ġalso Ġperforms Ġbetter Ġthan ĠB ERT Base Ġin Ġthe ĠST S Ġtask . ĠThe ĠST S Ġtasks Ġare Ġsem antically Ġsimilar Ġto Ġthe ĠNL I Ġtasks , Ġand Ġhence Ġable Ġto Ġtake Ġadvantage Ġof ĠPSP Ġas Ġwell . ĠActually , Ġthe Ġproposed Ġmethod Ġhas Ġa Ġpositive Ġeffect Ġwhenever Ġthe Ġinput Ġis Ġa Ġsentence Ġpair . ĠThe Ġimprovements Ġsuggest Ġthat Ġthe ĠPSP Ġtask Ġencourages Ġthe Ġmodel Ġto Ġlearn Ġmore Ġdetailed Ġsemantics Ġin Ġthe Ġpre - training , Ġwhich Ġimproves Ġthe Ġmodel Ġon Ġthe Ġdownstream Ġlearning Ġtasks . ĠMoreover , Ġour Ġmethod Ġis Ġsurprisingly Ġable Ġto Ġachieve Ġslightly Ġbetter Ġresults Ġin Ġthe Ġsingle - sent ence Ġproblem . ĠThe Ġimprovement Ġshould Ġbe Ġattributed Ġto Ġbetter Ġsemantic Ġrepresentation . </s> When Ġcomparing Ġbetween ĠP N Ġand ĠP N 5 cl s , ĠP N 5 cl s Ġachieves Ġbetter Ġresults Ġthan ĠP N . ĠThis Ġindicates Ġthat Ġincluding Ġa Ġbroader Ġrange Ġof Ġthe Ġcontext Ġis Ġeffective Ġfor Ġimproving Ġinference Ġability . ĠConsidering Ġthat Ġthe Ġrepresentation Ġof ĠIs Next Ġand ĠIs Next In adj Ġshould Ġbe Ġcoherent , Ġwe Ġpropose ĠB ERT Base - PN sm th Ġto Ġmitigate Ġthis Ġproblem . ĠP N sm th Ġfurther Ġimproves Ġthe Ġperformance Ġand Ġob tains Ġan Ġaveraged Ġscore Ġof Ġ81 . 0 . </s> Results Ġof ĠNL I ĠT asks Ġ: :: ĠH ANS </s> Although ĠB ERT Ġhas Ġshown Ġits Ġeffectiveness Ġin Ġthe ĠNL I Ġtasks . ĠB IB REF 0 Ġpointed Ġout Ġthat ĠB ERT Ġis Ġstill Ġvulnerable Ġin Ġthe ĠNL I Ġtask Ġas Ġit Ġis Ġprone Ġto Ġadopting Ġfall ible Ġhe ur istics . ĠTherefore , Ġthey Ġreleased Ġa Ġdataset , Ġcalled ĠThe ĠHe uristic ĠAnalysis Ġfor ĠNL I ĠSystems Ġ( H ANS ), Ġto Ġprobe Ġwhether Ġthe Ġmodel Ġlearns Ġinappropriate Ġinduct ive Ġbias Ġfrom Ġthe Ġtraining Ġset . ĠIt Ġis Ġconstructed Ġby Ġthree Ġhe ur istics , Ġi . e ., Ġlex ical Ġoverlap Ġhe uristic , Ġsub - sequence Ġhe uristic , Ġand Ġconstituent Ġhe uristic . ĠThe Ġfirst Ġhe uristic Ġassumes Ġthat Ġa Ġpremise Ġentails Ġall Ġhypotheses Ġconstructed Ġfrom Ġwords Ġin Ġthe Ġpremise , Ġthe Ġsecond Ġassumes Ġthat Ġa Ġpremise Ġentails Ġall Ġof Ġits Ġcontiguous Ġsub - sequ ences Ġand Ġthe Ġthird Ġassumes Ġthat Ġa Ġpremise Ġentails Ġall Ġcomplete Ġsub - t rees Ġin Ġits Ġparse Ġtree . ĠB ERT Ġand Ġother Ġadvanced Ġmodels Ġfail Ġon Ġthis Ġdataset Ġand Ġbarely Ġexceeds Ġ0 % Ġaccuracy Ġin Ġmost Ġcases ĠB IB REF 0 . </s> Fig . ĠFIG REF 23 Ġillustrates Ġthe Ġaccuracy Ġof ĠB ERT Base Ġand ĠB ERT Base - PN sm th Ġon Ġthe ĠH ANS Ġdataset . ĠThe Ġevaluation Ġis Ġmade Ġupon Ġthe Ġmodel Ġtrained Ġon Ġthe ĠMN LI Ġdataset Ġand Ġthe Ġpredicted Ġneutral Ġand Ġcontradiction Ġlabels Ġare Ġmapped Ġinto Ġnon - ent ail ment . ĠThe ĠB ERT Base - PN sm th Ġevidently Ġoutper forms Ġthe ĠB ERT Base Ġwith Ġthe Ġnon - ent ail ment Ġexamples . ĠFor Ġthe Ġnon - ent ail ment Ġsamples Ġconstructed Ġusing Ġthe Ġlex ical Ġoverlap Ġhe uristic , Ġour Ġmodel Ġachieves Ġ160 % Ġrelative Ġimprovement Ġover Ġthe ĠB ERT Base Ġmodel . ĠSome Ġsamples Ġare Ġconstructed Ġby Ġswapping Ġthe Ġentities Ġin Ġthe Ġsentence Ġ( e . g ., ĠThe Ġdoctor Ġsaw Ġthe Ġlawyer Ġ$\ n right arrow Ġ$ ĠThe Ġlawyer Ġsaw Ġthe Ġdoctor ) Ġand Ġour Ġmethod Ġoutper forms ĠB ERT Base Ġby Ġ20 % Ġin Ġaccuracy . ĠWe Ġsuggest Ġthat Ġthe ĠMask ed ĠLM Ġtask Ġcan Ġhardly Ġmodel Ġthe Ġrelationship Ġbetween Ġtwo Ġentities Ġand ĠN SP Ġonly Ġis Ġtoo Ġsem antically Ġshallow Ġto Ġcapture Ġthe Ġprecise Ġmeaning . ĠHowever , Ġthe Ġdiscrimination Ġbetween ĠN SP Ġand ĠPSP Ġenhances Ġthe Ġmodel Ġto Ġrealize Ġthe Ġrole Ġof Ġentities Ġin Ġa Ġgiven Ġsentence . ĠFor Ġexample , Ġto Ġdetermine Ġthat ĠA Ġ( X Ġis Ġbeautiful ) Ġrather Ġthan Ġ$\ bar {\ text tt Ġ{ A }} $ Ġ( Y Ġis Ġbeautiful ) Ġis Ġthe Ġprevious Ġsentence Ġof ĠB Ġ( Y Ġloves ĠX ), Ġthe Ġmodel Ġhave Ġto Ġrecognize Ġthe Ġrelationship Ġbetween ĠX Ġand ĠY . ĠIn Ġcontrast , Ġwhen ĠPSP Ġis Ġabsent , ĠN SP Ġcan Ġbe Ġprobably Ġinferred Ġby Ġlearning Ġthe Ġoccurrence Ġbetween Ġbeautiful Ġand Ġloves , Ġregardless Ġof Ġthe Ġsentence Ġstructure . ĠThe Ġdetailed Ġperformance Ġof Ġthe Ġproposed Ġmethod Ġon Ġthe ĠH ANS Ġdataset Ġis Ġillustrated Ġin ĠFig . ĠFIG REF 24 . ĠThe Ġdefinition Ġof Ġeach Ġhe uristic Ġrules Ġcan Ġbe Ġfound Ġin ĠB IB REF 0 . </s> Results Ġof ĠM RC ĠT asks Ġ: :: ĠS Qu AD Ġv 1 . 1 Ġand Ġv 2 . 0 </s> We Ġalso Ġevaluate Ġour Ġmethod Ġon Ġthe ĠM RC Ġtasks . ĠThe ĠStanford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD Ġv 1 . 1 ) Ġis Ġa Ġquestion Ġanswering Ġ( Q A ) Ġdataset , Ġwhich Ġconsists Ġof Ġ100 K Ġsamples ĠB IB REF 15 . ĠEach Ġdata Ġsample Ġhas Ġa Ġquestion Ġand Ġa Ġcorresponding ĠWikipedia Ġpassage Ġthat Ġcontains Ġthe Ġanswer . ĠThe Ġgoal Ġis Ġto Ġextract Ġthe Ġanswer Ġfrom Ġthe Ġpassage Ġfor Ġthe Ġgiven Ġquestion . </s> In Ġthe Ġfine - tun ing Ġprocedure , Ġwe Ġfollow Ġthe Ġexact Ġway Ġthe ĠB ERT Ġperformed . ĠThe Ġoutput Ġvectors Ġare Ġused Ġto Ġcompute Ġthe Ġscore Ġof Ġtokens Ġbeing Ġstart Ġand Ġend Ġof Ġthe Ġanswer Ġspan . ĠThe Ġvalid Ġspan Ġthat Ġhas Ġthe Ġmaximum Ġscore Ġis Ġselected Ġas Ġthe Ġprediction . ĠAnd Ġsimilarly , Ġthe Ġfine - tun ing Ġtraining Ġwas Ġperformed Ġfor Ġ3 Ġepoch s Ġwith Ġa Ġlearning Ġrate Ġof Ġ3 e - 5 . </s> Table ĠT AB REF 26 Ġdemonstrates Ġthe Ġresults Ġon Ġthe ĠS Qu AD Ġv 1 . 1 Ġdataset . ĠThe Ġcomparison Ġbetween ĠB ERT Base - PN Ġand ĠB ERT Base Ġindicates Ġthat Ġthe Ġinclusion Ġof Ġthe ĠPSP Ġsubt ask Ġis Ġbeneficial Ġ( 2 . 4 % Ġabsolute Ġimprovement ). ĠWhen Ġusing ĠB ERT Base - PN sm th , Ġanother Ġ0 . 3 % Ġincrease Ġin ĠEM Ġcan Ġbe Ġobtained . ĠThe Ġexperimental Ġresults Ġon Ġthe ĠS Qu AD Ġv 2 . 0 ĠB IB REF 16 Ġare Ġalso Ġshown Ġin ĠTable . ĠT AB REF 26 . ĠThe ĠS Qu AD Ġv 2 . 0 Ġdiffers Ġfrom ĠS Qu AD Ġv 1 . 1 Ġby Ġallowing Ġthe Ġquestion - paragraph Ġpairs Ġthat Ġhave Ġno Ġanswer . ĠFor ĠS Qu AD Ġv 2 . 0 , Ġour Ġmethod Ġalso Ġachieved Ġabout Ġ4 % Ġabsolute Ġimprovement Ġin Ġboth ĠEM Ġand ĠF 1 Ġagainst ĠB ERT Base . </s> Results Ġof ĠM RC ĠT asks Ġ: :: ĠR ACE </s> The ĠRe Ad ing ĠComp rehens ion Ġfrom ĠEx amin ations Ġ( R ACE ) Ġdataset ĠB IB REF 17 Ġconsists Ġof Ġ100 K Ġquestions Ġtaken Ġfrom ĠEnglish Ġexams , Ġand Ġthe Ġanswers Ġare Ġgenerated Ġby Ġhuman Ġexperts . ĠThis Ġis Ġone Ġof Ġthe Ġmost Ġchallenging ĠM RC Ġdatasets Ġthat Ġrequire Ġsophisticated Ġreasoning . </s> In Ġour Ġimplementation , Ġthe Ġquestion , Ġdocument , Ġand Ġoption Ġare Ġconc aten ated Ġas Ġa Ġsingle Ġsequence , Ġseparated Ġby Ġ[ SE P ] Ġtoken . ĠAnd Ġeach Ġpart Ġis Ġtrunc ated Ġby Ġa Ġmaximal Ġlength Ġof Ġ40 / 432 / 40 , Ġrespectively . ĠThe Ġmodel Ġcomp utes Ġfor Ġa Ġconc aten ation Ġa Ġscal ar Ġas Ġthe Ġscore , Ġwhich Ġis Ġthen Ġused Ġin Ġa Ġsoft max Ġlayer Ġfor Ġthe Ġfinal Ġprediction . ĠThe Ġfine - tun ing Ġwas Ġconducted Ġfor Ġ5 Ġepoch s , Ġwith Ġa Ġbatch Ġsize Ġof Ġ32 Ġand Ġa Ġlearning Ġrate Ġof Ġ5 e - 5 . ĠAs Ġshown Ġin ĠTable ĠT AB REF 28 , Ġthe Ġproposed Ġmethod Ġsignificantly Ġimprove Ġthe Ġperformance Ġon Ġthe ĠR ACE Ġdataset . ĠB ERT Base - PN Ġob tains Ġ2 . 6 % Ġaccuracy Ġimprovement , Ġand ĠB ERT Base - PN 5 cl s Ġfurther Ġbrings Ġ0 . 4 % Ġabsolute Ġgain . </s> The Ġcomparisons Ġon Ġthe ĠS Qu AD Ġv 1 . 1 , ĠS Qu AD Ġv 2 . 0 , Ġand ĠR ACE Ġdataset Ġdemonstrate Ġthat Ġthe Ġinvolvement Ġof Ġadditional Ġsentence Ġand Ġdiscourse Ġinformation Ġis Ġnot Ġonly Ġbeneficial Ġfor Ġthe ĠNL I Ġtask Ġbut Ġalso Ġthe ĠM RC Ġtask . ĠThis Ġis Ġreasonable Ġas Ġthese Ġtasks Ġheavily Ġrely Ġon Ġthe Ġglobal Ġsemantic Ġunderstanding Ġand Ġsophisticated Ġreasoning Ġamong Ġsentences . ĠAnd Ġthis Ġability Ġcan Ġbe Ġeffectively Ġenhanced Ġby Ġour Ġmethod . </s> Results Ġof ĠChinese ĠN LP ĠT asks </s> The Ġexperiments Ġare Ġalso Ġconducted Ġon ĠChinese ĠN LP Ġtasks : </s> X N LI ĠB IB REF 19 Ġa Ġmulti - ling ual Ġdataset . ĠThe Ġdata Ġsample Ġin ĠX N LI Ġis Ġa Ġsentence Ġpair Ġannot ated Ġwith Ġtextual Ġentail ment . ĠThe ĠChinese Ġpart Ġis Ġused . </s> LC Q MC ĠB IB REF 20 Ġis Ġa Ġdataset Ġfor Ġsequence Ġmatching . ĠA Ġbinary Ġlabel Ġis Ġannot ated Ġfor Ġa Ġsentence Ġpair Ġin Ġthe Ġdataset Ġto Ġindicate Ġwhether Ġthese Ġtwo Ġsentences Ġhave Ġthe Ġsame Ġintention . </s> N LP CC - DB Q A ĠB IB REF 21 Ġform ulates Ġthe Ġdomain - based Ġquestion Ġanswering Ġas Ġa Ġbinary Ġclassification Ġtask . ĠEach Ġdata Ġsample Ġis Ġa Ġquestion - sent ence Ġpair . ĠThe Ġgoal Ġis Ġto Ġidentify Ġwhether Ġthe Ġsentence Ġcontains Ġthe Ġanswer Ġto Ġthe Ġquestion . </s> CM RC - 2018 Ġis Ġthe ĠChinese ĠMachine ĠReading ĠComp rehens ion Ġdataset . ĠSimilar Ġto ĠS Qu AD , Ġthe Ġsystem Ġneeds Ġto Ġextract Ġfragments Ġfrom Ġthe Ġtext Ġas Ġthe Ġanswer . </s> D RC D ĠB IB REF 22 Ġis Ġalso Ġa ĠChinese ĠM RC Ġdata Ġset . ĠThe Ġdata Ġfollows Ġthe Ġformat Ġof ĠS Qu AD . </s> For ĠChinese ĠN LP Ġtasks , Ġwe Ġpre - train Ġthe Ġmodel Ġusing ĠChinese Ġcorpus . ĠWe Ġcollected Ġtextual Ġdata Ġ( 108 79 M Ġtokens Ġin Ġtotal ) Ġfrom Ġthe Ġwebsite , Ġconsisting Ġof ĠHud ong ĠBa ike Ġdata Ġ( 60 84 M Ġtokens ) Ġ, ĠZh ihu Ġdata ( 465 M Ġtokens ) Ġ, ĠS oh u ĠNews ( 39 37 M Ġtokens ) Ġand ĠWikipedia Ġdata Ġ( 393 M Ġtokens ). </s> For Ġthe Ġfirst Ġ3 ĠChinese Ġtasks , Ġwe Ġfollow Ġthe Ġsettings Ġas Ġin ĠER N IE ĠB IB REF 4 . ĠThe Ġexperimental Ġresults Ġare Ġgiven Ġin ĠTable ĠT AB REF 29 . ĠThe Ġproposed Ġmethod Ġis Ġcompared Ġwith Ġfour Ġmodels , Ġi . e ., ĠB ERT Base ĠB IB REF 1 , ĠB ERT Base Ġwith Ġwhole Ġword Ġmask ing ĠB IB REF 18 , ĠER N IE ĠB IB REF 4 Ġand ĠER N IE Ġ2 . 0 ĠB IB REF 5 . ĠOur Ġmethod Ġachieves Ġcomparable Ġor Ġeven Ġbetter Ġresults Ġagainst ĠER N IE Ġ2 . 0 ĠB IB REF 5 . ĠNote Ġthat Ġthe ĠChinese ĠER N IE Ġ2 . 0 Ġis Ġequipped Ġwith Ġ5 Ġdifferent Ġobjectives Ġand Ġit Ġuses Ġmore Ġtraining Ġdata Ġ( 149 88 M Ġtokens Ġin Ġtotal ) Ġthan Ġours . ĠThe Ġresults Ġindicate Ġthat Ġthe Ġproposed Ġmethod Ġis Ġquite Ġeffective Ġfor Ġthe Ġpair - wise Ġsemantic Ġreasoning Ġas Ġsimply Ġincluding ĠPSP Ġcan Ġachieve Ġthe Ġresults Ġon Ġpar Ġwith Ġmultiple Ġobjectives . </s> The Ġresults Ġof ĠCM RC - 2018 Ġand ĠD RC D Ġdatasets Ġare Ġgiven Ġin ĠTable ĠT AB REF 30 . ĠSince Ġthe ĠCM RC - 2018 Ġcompetition Ġdoes Ġnot Ġrelease Ġthe Ġtest Ġset , Ġthe Ġcomparison Ġon Ġthe Ġtest Ġset Ġis Ġabsent . ĠOur Ġresults Ġare Ġobtained Ġusing Ġthe Ġopen - s ourced Ġcode Ġof ĠB ERT - ww m Ġ. ĠWe Ġkeep Ġthe Ġhyper - param eters Ġthe Ġsame Ġwith Ġthat Ġin ĠER N IE ĠB IB REF 4 , Ġexcept Ġthat Ġthe Ġbatch Ġsize Ġis Ġ12 Ġinstead Ġof Ġ64 Ġdue Ġto Ġthe Ġmemory Ġlimit . ĠUnder Ġthis Ġsetting , Ġwe Ġachieved Ġsimilar Ġresults Ġof ĠB ERT Base Ġin Ġthe ĠB ERT - ww m Ġpaper ĠB IB REF 18 . ĠHowever , Ġthis Ġis Ġworse Ġthan Ġthe Ġresults Ġof ĠB ERT Base Ġreported Ġin Ġthe ĠER N IE Ġ2 . 0 Ġpaper ĠB IB REF 5 Ġby Ġabout Ġ1 % Ġin ĠF 1 . ĠThis Ġsuggests Ġthat Ġour Ġresults Ġare Ġcurrently Ġincom parable Ġwith ĠER N IE Ġ2 . 0 . ĠOverall , Ġthe Ġresults Ġin ĠTable ĠT AB REF 30 Ġillustrate Ġthat Ġour Ġmethod Ġis Ġalso Ġeffective Ġfor Ġthe ĠChinese ĠQ A Ġtasks . </s> Conclusion </s> This Ġpaper Ġaims Ġto Ġenrich Ġthe ĠN SP Ġtask Ġto Ġprovide Ġmore Ġdocument - level Ġinformation Ġin Ġthe Ġpre - training . ĠMot ivated Ġby Ġthe Ġin - sy mm etric Ġproperty Ġof ĠN SP , Ġwe Ġpropose Ġto Ġdifferentiate Ġbetween Ġdifferent Ġsentence Ġorders Ġby Ġincluding ĠPSP . ĠDespite Ġthe Ġsimplicity , Ġextensive Ġexperiments Ġdemonstrate Ġthat Ġthe Ġmodel Ġob tains Ġa Ġbetter Ġability Ġin Ġpair - wise Ġsemantic Ġreasoning . ĠOur Ġwork Ġsuggests Ġthat Ġthe Ġdocument - level Ġobjective Ġis Ġeffective , Ġat Ġleast Ġfor Ġthe ĠB ERT base Ġmodel . ĠIn Ġthe Ġfuture , Ġwe Ġwill Ġinvestigate Ġthe Ġway Ġto Ġtake Ġadvantages Ġof Ġboth Ġlarge - scale Ġtraining Ġand Ġour Ġmethod .
Introduction The ability of semantic reasoning is essential for advanced natural language understanding (NLU) systems. Many NLU tasks that take sentence pairs as input, such as natural language inference (NLI) and machine reading comprehension (MRC), heavily rely on the ability of sophisticated semantic reasoning. For instance, the NLI task aims to determine whether the hypothesis sentence (e.g., a woman is sleeping) can be inferred from the premise sentence (e.g., a woman is talking on the phone). This requires the model to read and understand sentence pairs to make the specific semantic inference. Bidirectional Encoder Representations from Transformer (BERT) BIBREF1 has shown strong ability in semantic reasoning. It was recently proposed and obtained impressive results on many tasks, ranging from text classification, natural language inference, and machine reading comprehension. BERT achieves this by employing two objectives in the pre-training, i.e., the masked language modeling (Masked LM) and the next sentence prediction (NSP). Intuitively, the Masked LM task concerns word-level knowledge, and the NSP task captures the global document-level information. The goal of NSP is to identify whether an input sentence is next to another input sentence. From the ablation study BIBREF1, the NSP task is quite useful for the downstream NLI and MRC tasks (e.g., +3.5% absolute gain on the Question NLI (QNLI) BIBREF2 task). Despite its usefulness, we suggest that BERT has not made full use of the document-level knowledge. The sentences in the negative samples used in NSP are randomly drawn from other documents. Therefore, to discriminate against these sentences, BERT is prone to aggregating the shallow semantic, e.g., topic, neglecting context clues useful for detailed reasoning. In other words, the canonical NSP task would encourage the model to recognize the correlation between sentences, rather than obtaining the ability of semantic entailment. This setting weakens the BERT model from learning specific semantic for inference. Another issue that renders NSP less effective is that BERT is order-sensitive. Performance degradation was observed on typical NLI tasks when the order of two input sentences are reversed during the BERT fine-tuning phase. It is reasonable as the NSP task can be roughly analogy to the NLI task when the input comes as (premise, hypothesis), considering the causal order among sentences. However, this identity between NSP and NLI is compromised when the sentences are swapped. Based on these considerations, we propose a simple yet effective method, i.e., introducing a IsPrev category to the classification task, which is a symmetric label of IsNext of NSP. The input of samples with IsPrev is the reverse of those with IsNext label. The advantages of using this previous sentence prediction (PSP) are three folds. (1) Learning the contrast between NSP and PSP forces the model to extract more detailed semantic, thereby the model is more capable of discriminating the correlation and entailment. (2) NSP and PSP are symmetric. This symmetric regularization alleviates the influence of the order of the input pair. (3) Empirical results indicate that our method is beneficial for all the semantic reasoning tasks that take sentence pair as input. In addition, to further incorporating the document-level knowledge, NSP and PSP are extended with non-successive sentences, where the label smoothing technique is adopted. The proposed method yields a considerable improvement in our experiments. We evaluate the ability of semantic reasoning on standard NLI and MRC benchmarks, including the challenging HANS dataset BIBREF0. Analytical work on the HANS dataset provides a more comprehensible perspective towards the proposed method. Furthermore, the results on the Chinese benchmarks are provided to demonstrate its generality. In summary, this work makes the following contributions: The supervision signal from the original NSP task is weak for semantic inference. Therefore, a novel method is proposed to remedy the asymmetric issue and enhance the reasoning ability. Both empirical and analytical evaluations are provided on the NLI and MRC datasets, which verifies the effectiveness of using more document-level knowledge. Related Work ::: Pair-wise semantic reasoning Many NLU tasks seek to model the relationship between two sentences. Semantic reasoning is performed on the sentence pair for the task-specific inference. Pair-wise semantic reasoning tasks have drawn a lot of attention from the NLP community as they largely require the comprehension ability of the learning systems. Recently, the significant improvement on these benchmarks comes from the pre-training models, e.g., BERT, StructBERT BIBREF3, ERNIE BIBREF4, BIBREF5, RoBERTa BIBREF6 and XLNet BIBREF7. These models learn from unsupervised/self-supervised objectives and perform excellently in the downstream tasks. Among these models, BERT adopts NSP as one of the objectives in the pre-training and shows that the NSP task has a positive effect on the NLI and MRC tasks. Although the primary study of XLNet and RoBERTa suggests that NSP is ineffective when the model is trained with a large sequence length of 512, the effect of NSP on the NLI problems should still be emphasized. The inefficiency of NSP is likely because the expected context length will be halved for Masked LM when taking a sentence pair as the input. The models derived from BERT, e.g., StructBERT and ERNIE 1.0/2.0, aim to incorporating more knowledge by elaborating pre-training objectives. This work aims to enhance the NSP task and verifies whether document-level information is helpful for the pre-training. To probe whether our method achieves a better regularization ability, our approach is also evaluated on the HANS BIBREF0 dataset, which contains hard data samples constructed by three heuristics. Previous advanced models such as BERT fail on the HANS dataset, and the test accuracy can barely exceed 0% in the subset of test examples. Related Work ::: Unsupervised learning from document In recent years, many unsupervised pre-training methods have been proposed in the NLP fields to extract knowledge among sentences DBLP:conf/nips/KirosZSZUTF15,DBLP:conf/emnlp/ConneauKSBB17,DBLP:conf/iclr/LogeswaranL18,DBLP:journals/corr/abs-1903-09424. The prediction of surrounding sentences endows the model with the ability to model the sentence-level coherence. Skip-Thought BIBREF8 consists of an encoder and two decoders. When a sentence is given and encoded into a vector by the encoder, the decoders are trained to predict the next sentence and the previous sentence. The goal is to obtain a better sentence representation that is useful for reconstructing the surrounding context. Considering that the estimation of the likelihood of sequences is computationally expensive and time-consuming, the Quick-Thought method BIBREF9 simplifies this in a manner similar to sampled softmax BIBREF10, which classifies the input sentences between surrounding sentences and the other. Note that Quick-Thought does not distinguish between the previous and next sentence as it is functionally rotation invariant. However, BERT is order-dependent, and the discrimination can provide more supervision signal for semantic learning. InferSent BIBREF11 instead pre-trains the model in a manner of supervised learning. It uses a large-scale NLI dataset as the pre-training task to learn the sentence representation. In our work, we focus on designing a more effective document-level objective, extended from the NSP task. The proposed method will be described in the following section and validated by providing extensive experimental results in the experiment part. Method Our method follows the same input format and the model architecture with original BERT. The proposed method solely concerns the NSP task. The NSP task is a binary classification task, which takes two sentences (A and B) as input and determines whether B is the next sentence of A. Although it has been proven to be very effective for BERT, there are two major deficiencies. (1) Discrimination between IsNext and DiffDoc (the label of the sentences drawn from different documents via negative sampling) is semantically shallow as the signal of sentence order is absent. The correlation between two successive sentences could be obvious, due to, for example, lexical overlap or the conjunction used at the beginning of the second sentence. As reported BIBREF1, the final pre-trained model is able to achieve 97%-98% accuracy on the NSP task. (2) BERT is order-sensitive, i.e., $f_{\text{BERT}}( \texttt {A}, \texttt {B}) \ne f_{\text{BERT}}(\texttt {B}, \texttt {A})$, while NSP is uni-directional. When the order of the input NLI pair is reversed, the performance will degrade. For instance, the accuracy decreases by about 0.5% on MNLI BIBREF12 and 0.4% on QNLI after swapping the sentences in our experiments . Motivated by these problems, we propose to extend the NSP task with previous sentence prediction (PSP). Despite its simplicity, empirical results show that this is beneficial for downstream tasks, including both NLI and MRC tasks. To further incorporate the document-level information, the scope is also expanded to include more surrounding sentences, not just the adjacent. The method is briefly illustrated in Fig. FIGREF6. Method ::: Previous Sentence Prediction Learning to recognize the previous sentence enables the model to capture more compact context information. One would argue that IsPrev (the label of PSP) is redundant as it plays a similar role of IsNext (the label of NSP). In fact, Quick-Thought uses the sampled softmax to approximate the sentence likelihood estimation of Skip-Thought, and it actually does not differentiate between the previous and next sentences. However, we suggest the order discrimination is essential for BERT pre-training. Quick-Thought aims at extracting sentence embedding, and it uses a rotating symmetric function, which makes IsPrev redundant in Quick-Thought. In contrast, BERT is order-sensitive, and learning the symmetric regularization is rather necessary. Another advantage of PSP is to enhance document-level supervision. In order to tell the difference between NSP and PSP, the model has to extract the detailed semantic for inference. Method ::: Gathering More Document-level Information Beyond NSP and PSP, which enable the model to learn the short-term dependency between sentences, we also propose to expand the scope of discrimination task to further incorporate the document-level information. Specifically, we also include the in-adjacent sentences in the sentence-pair classification task. The in-adjacent sentences next to the IsPrev and IsNext sentences are sampled, labeled as IsPrevInadj and IsNextInadj (cf. the bottom of Fig. FIGREF6). Note that these in-adjacent sentences will introduce much more training noise to the model. Therefore, the label smoothing technique is adopted to reduce the noise of these additional samples. It achieves this by relaxing our confidence on the labels, e.g., transforming the target probability from (1.0, 0.0) to (0.8, 0.2) in a binary classification problem. In summary, when A is given, the pre-training example for each label is constructed as follows: IsNext: Choosing the adjacent following sentence as B. IsPrev: Choosing the adjacent previous sentence as B. IsNextInadj: Choosing the in-adjacent following sentence as B. There is a sentence between A and B. IsPrevInadj: Choosing the in-adjacent previous sentence as B. There is a sentence between A and B. DiffDoc: Drawing B randomly from a different document. Experiment Settings This section gives detailed experiment settings. The method is evaluated on the BERTbase model, which has 12 layers, 12 self-attention heads with a hidden size of 768. To accelerate the training speed, two-phase training BIBREF1 is adopted. The first phase uses a maximal sentence length of 128, and 512 for the second phase. The numbers of training steps of two phases are 50K and 40K for the BERTBase model. We used AdamW BIBREF13 optimizer with a learning rate of 1e-4, a $\beta _1$ of 0.9, a $\beta _2$ of 0.999 and a L2 weight decay rate of $0.01$. The first 10% of the total steps are used for learning rate warming up, followed by the linear decay schema. We used a dropout probability of 0.1 on all layers. The data used for pre-training is the same as BERT, i.e., English Wikipedia (2500M words) and BookCorpus (800M words) BIBREF14. For the Masked LM task, we followed the same masking rate and settings as in BERT. We explore three method settings for comparison. BERT-PN: The NSP task in BERT is replaced by a 3-class task with IsNext, IsPrev and DiffDoc. The label distribution is 1:1:1. BERT-PN5cls: The NSP task in BERT is replaced by a 5-class task with two additional labels IsNextInadj, IsPrevInadj. The label distribution is 1:1:1:1:1. BERT-PNsmth: It uses the same data with BERT-PN5cls, except that the IsPrevInadj (IsNextInadj) label is mapped to IsPrev (IsNext) with a label smoothing factor of 0.8. BERT-PN is used to verify the feasibility of PSP. The comparison with BERT-PN5cls illustrates whether more document-level information helps. BERT-PNsmth, which is the label-smoothed version of BERT-PN5cls, is used to compare with BERT-PN5cls to see whether the noise reduction is necessary. In the following, we first show that BERT is order-sensitive and the use of PSP remedies this problem. Then we provide experimental results on the NLI and MRC tasks to verify the effectiveness of the proposed method. At last, the proposed method is evaluated on several Chinese datasets. Order-invariant with PSP NSP in the pre-training is useful for NLI and MRC task BIBREF1. However, we suggested that BERT trained with NSP is order-sensitive, i.e., the performance of BERT depends on the order of the input sentence pair. To verify our assumption, a primary experiment was conducted. The order of the input pair of NLI samples is reversed in the fine-tuning phase, and other hyper-parameters and settings keep the same with the BERT paper. Table TABREF19 shows the accuracy on the validation set of the MNLI and QNLI datasets. For the BERTBase model, when the sentences are swapped, the accuracy decreases by 0.5% on the MNLI task and 0.4% on the QNLI task. These results confirm that BERT trained with NSP only is indeed affected by the input order. This phenomenon motivates us to make the NSP task symmetric. The results of BERT-PN verify that BERT-PN is order-invariant. When the input order is reversed, the performance of BERT-PN remains stable. These results indicate that our method is able to remedy the order-sensitivity problem. Results of NLI Tasks ::: GLUE A popular benchmark for evaluation of language understanding is GLUE BIBREF2, which is a collection of three NLI tasks (MNLI, QNLI and RTE), three semantic textual similarity (STS) tasks (QQP, STS-B and MRPC), two text classification (TC) tasks (SST-2 and CoLA). Although the method is motivated for pair-wise reasoning, the results of other problems are also listed. Our implementation follows the same way that BERT performs in these tasks. The fine-tuning was conducted for 3 epochs for all the tasks, with a learning rate of 2e-5. The predictions were obtained by evaluating the training checkpoint with the best validation performance. Table TABREF21 illustrates the experimental results, showing that our method is beneficial for all of NLI tasks. The improvement on the RTE dataset is significant, i.e., 4% absolute gain over the BERTBase. Besides NLI, our model also performs better than BERTBase in the STS task. The STS tasks are semantically similar to the NLI tasks, and hence able to take advantage of PSP as well. Actually, the proposed method has a positive effect whenever the input is a sentence pair. The improvements suggest that the PSP task encourages the model to learn more detailed semantics in the pre-training, which improves the model on the downstream learning tasks. Moreover, our method is surprisingly able to achieve slightly better results in the single-sentence problem. The improvement should be attributed to better semantic representation. When comparing between PN and PN5cls, PN5cls achieves better results than PN. This indicates that including a broader range of the context is effective for improving inference ability. Considering that the representation of IsNext and IsNextInadj should be coherent, we propose BERTBase-PNsmth to mitigate this problem. PNsmth further improves the performance and obtains an averaged score of 81.0. Results of NLI Tasks ::: HANS Although BERT has shown its effectiveness in the NLI tasks. BIBREF0 pointed out that BERT is still vulnerable in the NLI task as it is prone to adopting fallible heuristics. Therefore, they released a dataset, called The Heuristic Analysis for NLI Systems (HANS), to probe whether the model learns inappropriate inductive bias from the training set. It is constructed by three heuristics, i.e., lexical overlap heuristic, sub-sequence heuristic, and constituent heuristic. The first heuristic assumes that a premise entails all hypotheses constructed from words in the premise, the second assumes that a premise entails all of its contiguous sub-sequences and the third assumes that a premise entails all complete sub-trees in its parse tree. BERT and other advanced models fail on this dataset and barely exceeds 0% accuracy in most cases BIBREF0. Fig. FIGREF23 illustrates the accuracy of BERTBase and BERTBase-PNsmth on the HANS dataset. The evaluation is made upon the model trained on the MNLI dataset and the predicted neutral and contradiction labels are mapped into non-entailment. The BERTBase-PNsmth evidently outperforms the BERTBase with the non-entailment examples. For the non-entailment samples constructed using the lexical overlap heuristic, our model achieves 160% relative improvement over the BERTBase model. Some samples are constructed by swapping the entities in the sentence (e.g., The doctor saw the lawyer $\nrightarrow $ The lawyer saw the doctor) and our method outperforms BERTBase by 20% in accuracy. We suggest that the Masked LM task can hardly model the relationship between two entities and NSP only is too semantically shallow to capture the precise meaning. However, the discrimination between NSP and PSP enhances the model to realize the role of entities in a given sentence. For example, to determine that A (X is beautiful) rather than $\bar{\texttt {A}}$ (Y is beautiful) is the previous sentence of B (Y loves X), the model have to recognize the relationship between X and Y. In contrast, when PSP is absent, NSP can be probably inferred by learning the occurrence between beautiful and loves, regardless of the sentence structure. The detailed performance of the proposed method on the HANS dataset is illustrated in Fig. FIGREF24. The definition of each heuristic rules can be found in BIBREF0. Results of MRC Tasks ::: SQuAD v1.1 and v2.0 We also evaluate our method on the MRC tasks. The Stanford Question Answering Dataset (SQuAD v1.1) is a question answering (QA) dataset, which consists of 100K samples BIBREF15. Each data sample has a question and a corresponding Wikipedia passage that contains the answer. The goal is to extract the answer from the passage for the given question. In the fine-tuning procedure, we follow the exact way the BERT performed. The output vectors are used to compute the score of tokens being start and end of the answer span. The valid span that has the maximum score is selected as the prediction. And similarly, the fine-tuning training was performed for 3 epochs with a learning rate of 3e-5. Table TABREF26 demonstrates the results on the SQuAD v1.1 dataset. The comparison between BERTBase-PN and BERTBase indicates that the inclusion of the PSP subtask is beneficial (2.4% absolute improvement). When using BERTBase-PNsmth, another 0.3% increase in EM can be obtained. The experimental results on the SQuAD v2.0 BIBREF16 are also shown in Table. TABREF26. The SQuAD v2.0 differs from SQuAD v1.1 by allowing the question-paragraph pairs that have no answer. For SQuAD v2.0, our method also achieved about 4% absolute improvement in both EM and F1 against BERTBase. Results of MRC Tasks ::: RACE The ReAding Comprehension from Examinations (RACE) dataset BIBREF17 consists of 100K questions taken from English exams, and the answers are generated by human experts. This is one of the most challenging MRC datasets that require sophisticated reasoning. In our implementation, the question, document, and option are concatenated as a single sequence, separated by [SEP] token. And each part is truncated by a maximal length of 40/432/40, respectively. The model computes for a concatenation a scalar as the score, which is then used in a softmax layer for the final prediction. The fine-tuning was conducted for 5 epochs, with a batch size of 32 and a learning rate of 5e-5. As shown in Table TABREF28, the proposed method significantly improve the performance on the RACE dataset. BERTBase-PN obtains 2.6% accuracy improvement, and BERTBase-PN5cls further brings 0.4% absolute gain. The comparisons on the SQuAD v1.1, SQuAD v2.0, and RACE dataset demonstrate that the involvement of additional sentence and discourse information is not only beneficial for the NLI task but also the MRC task. This is reasonable as these tasks heavily rely on the global semantic understanding and sophisticated reasoning among sentences. And this ability can be effectively enhanced by our method. Results of Chinese NLP Tasks The experiments are also conducted on Chinese NLP tasks: XNLI BIBREF19 a multi-lingual dataset. The data sample in XNLI is a sentence pair annotated with textual entailment. The Chinese part is used. LCQMC BIBREF20 is a dataset for sequence matching. A binary label is annotated for a sentence pair in the dataset to indicate whether these two sentences have the same intention. NLPCC-DBQA BIBREF21 formulates the domain-based question answering as a binary classification task. Each data sample is a question-sentence pair. The goal is to identify whether the sentence contains the answer to the question. CMRC-2018 is the Chinese Machine Reading Comprehension dataset. Similar to SQuAD, the system needs to extract fragments from the text as the answer. DRCD BIBREF22 is also a Chinese MRC data set. The data follows the format of SQuAD. For Chinese NLP tasks, we pre-train the model using Chinese corpus. We collected textual data (10879M tokens in total) from the website, consisting of Hudong Baike data (6084M tokens) , Zhihu data(465M tokens) , Sohu News(3937M tokens) and Wikipedia data (393M tokens). For the first 3 Chinese tasks, we follow the settings as in ERNIE BIBREF4. The experimental results are given in Table TABREF29. The proposed method is compared with four models, i.e., BERTBase BIBREF1, BERTBase with whole word masking BIBREF18, ERNIE BIBREF4 and ERNIE 2.0 BIBREF5. Our method achieves comparable or even better results against ERNIE 2.0 BIBREF5. Note that the Chinese ERNIE 2.0 is equipped with 5 different objectives and it uses more training data (14988M tokens in total) than ours. The results indicate that the proposed method is quite effective for the pair-wise semantic reasoning as simply including PSP can achieve the results on par with multiple objectives. The results of CMRC-2018 and DRCD datasets are given in Table TABREF30. Since the CMRC-2018 competition does not release the test set, the comparison on the test set is absent. Our results are obtained using the open-sourced code of BERT-wwm . We keep the hyper-parameters the same with that in ERNIE BIBREF4, except that the batch size is 12 instead of 64 due to the memory limit. Under this setting, we achieved similar results of BERTBase in the BERT-wwm paper BIBREF18. However, this is worse than the results of BERTBase reported in the ERNIE 2.0 paper BIBREF5 by about 1% in F1. This suggests that our results are currently incomparable with ERNIE 2.0. Overall, the results in Table TABREF30 illustrate that our method is also effective for the Chinese QA tasks. Conclusion This paper aims to enrich the NSP task to provide more document-level information in the pre-training. Motivated by the in-symmetric property of NSP, we propose to differentiate between different sentence orders by including PSP. Despite the simplicity, extensive experiments demonstrate that the model obtains a better ability in pair-wise semantic reasoning. Our work suggests that the document-level objective is effective, at least for the BERTbase model. In the future, we will investigate the way to take advantages of both large-scale training and our method.
Do they train their model starting from a checkpoint?
7b4fb6da74e6bd1baea556788a02969134cf0800
Do Ġthey Ġtrain Ġtheir Ġmodel Ġstarting Ġfrom Ġa Ġcheckpoint ?
No
<s> Do Ġthey Ġtrain Ġtheir Ġmodel Ġstarting Ġfrom Ġa Ġcheckpoint ? </s> Introduction </s> The Ġability Ġof Ġsemantic Ġreasoning Ġis Ġessential Ġfor Ġadvanced Ġnatural Ġlanguage Ġunderstanding Ġ( NL U ) Ġsystems . ĠMany ĠNL U Ġtasks Ġthat Ġtake Ġsentence Ġpairs Ġas Ġinput , Ġsuch Ġas Ġnatural Ġlanguage Ġinference Ġ( N LI ) Ġand Ġmachine Ġreading Ġcomprehension Ġ( M RC ), Ġheavily Ġrely Ġon Ġthe Ġability Ġof Ġsophisticated Ġsemantic Ġreasoning . ĠFor Ġinstance , Ġthe ĠNL I Ġtask Ġaims Ġto Ġdetermine Ġwhether Ġthe Ġhypothesis Ġsentence Ġ( e . g ., Ġa Ġwoman Ġis Ġsleeping ) Ġcan Ġbe Ġinferred Ġfrom Ġthe Ġpremise Ġsentence Ġ( e . g ., Ġa Ġwoman Ġis Ġtalking Ġon Ġthe Ġphone ). ĠThis Ġrequires Ġthe Ġmodel Ġto Ġread Ġand Ġunderstand Ġsentence Ġpairs Ġto Ġmake Ġthe Ġspecific Ġsemantic Ġinference . </s> B id irection al ĠEnc oder ĠRepresent ations Ġfrom ĠTrans former Ġ( BER T ) ĠB IB REF 1 Ġhas Ġshown Ġstrong Ġability Ġin Ġsemantic Ġreasoning . ĠIt Ġwas Ġrecently Ġproposed Ġand Ġobtained Ġimpressive Ġresults Ġon Ġmany Ġtasks , Ġranging Ġfrom Ġtext Ġclassification , Ġnatural Ġlanguage Ġinference , Ġand Ġmachine Ġreading Ġcomprehension . ĠB ERT Ġachieves Ġthis Ġby Ġemploying Ġtwo Ġobjectives Ġin Ġthe Ġpre - training , Ġi . e ., Ġthe Ġmasked Ġlanguage Ġmodeling Ġ( Mask ed ĠLM ) Ġand Ġthe Ġnext Ġsentence Ġprediction Ġ( N SP ). ĠInt uitive ly , Ġthe ĠMask ed ĠLM Ġtask Ġconcerns Ġword - level Ġknowledge , Ġand Ġthe ĠN SP Ġtask Ġcaptures Ġthe Ġglobal Ġdocument - level Ġinformation . ĠThe Ġgoal Ġof ĠN SP Ġis Ġto Ġidentify Ġwhether Ġan Ġinput Ġsentence Ġis Ġnext Ġto Ġanother Ġinput Ġsentence . ĠFrom Ġthe Ġab lation Ġstudy ĠB IB REF 1 , Ġthe ĠN SP Ġtask Ġis Ġquite Ġuseful Ġfor Ġthe Ġdownstream ĠNL I Ġand ĠM RC Ġtasks Ġ( e . g ., Ġ+ 3 . 5 % Ġabsolute Ġgain Ġon Ġthe ĠQuestion ĠNL I Ġ( Q N LI ) ĠB IB REF 2 Ġtask ). </s> Despite Ġits Ġusefulness , Ġwe Ġsuggest Ġthat ĠB ERT Ġhas Ġnot Ġmade Ġfull Ġuse Ġof Ġthe Ġdocument - level Ġknowledge . ĠThe Ġsentences Ġin Ġthe Ġnegative Ġsamples Ġused Ġin ĠN SP Ġare Ġrandomly Ġdrawn Ġfrom Ġother Ġdocuments . ĠTherefore , Ġto Ġdiscriminate Ġagainst Ġthese Ġsentences , ĠB ERT Ġis Ġprone Ġto Ġaggreg ating Ġthe Ġshallow Ġsemantic , Ġe . g ., Ġtopic , Ġneglect ing Ġcontext Ġclues Ġuseful Ġfor Ġdetailed Ġreasoning . ĠIn Ġother Ġwords , Ġthe Ġcanonical ĠN SP Ġtask Ġwould Ġencourage Ġthe Ġmodel Ġto Ġrecognize Ġthe Ġcorrelation Ġbetween Ġsentences , Ġrather Ġthan Ġobtaining Ġthe Ġability Ġof Ġsemantic Ġentail ment . ĠThis Ġsetting Ġweak ens Ġthe ĠB ERT Ġmodel Ġfrom Ġlearning Ġspecific Ġsemantic Ġfor Ġinference . ĠAnother Ġissue Ġthat Ġrenders ĠN SP Ġless Ġeffective Ġis Ġthat ĠB ERT Ġis Ġorder - sensitive . ĠPerformance Ġdegradation Ġwas Ġobserved Ġon Ġtypical ĠNL I Ġtasks Ġwhen Ġthe Ġorder Ġof Ġtwo Ġinput Ġsentences Ġare Ġreversed Ġduring Ġthe ĠB ERT Ġfine - tun ing Ġphase . ĠIt Ġis Ġreasonable Ġas Ġthe ĠN SP Ġtask Ġcan Ġbe Ġroughly Ġanalogy Ġto Ġthe ĠNL I Ġtask Ġwhen Ġthe Ġinput Ġcomes Ġas Ġ( prem ise , Ġhypothesis ), Ġconsidering Ġthe Ġcausal Ġorder Ġamong Ġsentences . ĠHowever , Ġthis Ġidentity Ġbetween ĠN SP Ġand ĠNL I Ġis Ġcompromised Ġwhen Ġthe Ġsentences Ġare Ġswapped . </s> Based Ġon Ġthese Ġconsiderations , Ġwe Ġpropose Ġa Ġsimple Ġyet Ġeffective Ġmethod , Ġi . e ., Ġintroducing Ġa ĠIs Prev Ġcategory Ġto Ġthe Ġclassification Ġtask , Ġwhich Ġis Ġa Ġsymm etric Ġlabel Ġof ĠIs Next Ġof ĠN SP . ĠThe Ġinput Ġof Ġsamples Ġwith ĠIs Prev Ġis Ġthe Ġreverse Ġof Ġthose Ġwith ĠIs Next Ġlabel . ĠThe Ġadvantages Ġof Ġusing Ġthis Ġprevious Ġsentence Ġprediction Ġ( PS P ) Ġare Ġthree Ġfolds . Ġ( 1 ) ĠLearning Ġthe Ġcontrast Ġbetween ĠN SP Ġand ĠPSP Ġforces Ġthe Ġmodel Ġto Ġextract Ġmore Ġdetailed Ġsemantic , Ġthereby Ġthe Ġmodel Ġis Ġmore Ġcapable Ġof Ġdiscriminating Ġthe Ġcorrelation Ġand Ġentail ment . Ġ( 2 ) ĠN SP Ġand ĠPSP Ġare Ġsymm etric . ĠThis Ġsymm etric Ġregular ization Ġallev iates Ġthe Ġinfluence Ġof Ġthe Ġorder Ġof Ġthe Ġinput Ġpair . Ġ( 3 ) ĠEm pir ical Ġresults Ġindicate Ġthat Ġour Ġmethod Ġis Ġbeneficial Ġfor Ġall Ġthe Ġsemantic Ġreasoning Ġtasks Ġthat Ġtake Ġsentence Ġpair Ġas Ġinput . </s> In Ġaddition , Ġto Ġfurther Ġincorporating Ġthe Ġdocument - level Ġknowledge , ĠN SP Ġand ĠPSP Ġare Ġextended Ġwith Ġnon - success ive Ġsentences , Ġwhere Ġthe Ġlabel Ġsmoot hing Ġtechnique Ġis Ġadopted . ĠThe Ġproposed Ġmethod Ġyields Ġa Ġconsiderable Ġimprovement Ġin Ġour Ġexperiments . ĠWe Ġevaluate Ġthe Ġability Ġof Ġsemantic Ġreasoning Ġon Ġstandard ĠNL I Ġand ĠM RC Ġbenchmarks , Ġincluding Ġthe Ġchallenging ĠH ANS Ġdataset ĠB IB REF 0 . ĠAnaly tical Ġwork Ġon Ġthe ĠH ANS Ġdataset Ġprovides Ġa Ġmore Ġcomprehens ible Ġperspective Ġtowards Ġthe Ġproposed Ġmethod . ĠFurthermore , Ġthe Ġresults Ġon Ġthe ĠChinese Ġbenchmarks Ġare Ġprovided Ġto Ġdemonstrate Ġits Ġgener ality . </s> In Ġsummary , Ġthis Ġwork Ġmakes Ġthe Ġfollowing Ġcontributions : </s> The Ġsupervision Ġsignal Ġfrom Ġthe Ġoriginal ĠN SP Ġtask Ġis Ġweak Ġfor Ġsemantic Ġinference . ĠTherefore , Ġa Ġnovel Ġmethod Ġis Ġproposed Ġto Ġremedy Ġthe Ġasymm etric Ġissue Ġand Ġenhance Ġthe Ġreasoning Ġability . </s> Both Ġempirical Ġand Ġanalytical Ġevaluations Ġare Ġprovided Ġon Ġthe ĠNL I Ġand ĠM RC Ġdatasets , Ġwhich Ġver ifies Ġthe Ġeffectiveness Ġof Ġusing Ġmore Ġdocument - level Ġknowledge . </s> Related ĠWork Ġ: :: ĠPair - wise Ġsemantic Ġreasoning </s> Many ĠNL U Ġtasks Ġseek Ġto Ġmodel Ġthe Ġrelationship Ġbetween Ġtwo Ġsentences . ĠSem antic Ġreasoning Ġis Ġperformed Ġon Ġthe Ġsentence Ġpair Ġfor Ġthe Ġtask - specific Ġinference . ĠPair - wise Ġsemantic Ġreasoning Ġtasks Ġhave Ġdrawn Ġa Ġlot Ġof Ġattention Ġfrom Ġthe ĠN LP Ġcommunity Ġas Ġthey Ġlargely Ġrequire Ġthe Ġcomprehension Ġability Ġof Ġthe Ġlearning Ġsystems . ĠRecently , Ġthe Ġsignificant Ġimprovement Ġon Ġthese Ġbenchmarks Ġcomes Ġfrom Ġthe Ġpre - training Ġmodels , Ġe . g ., ĠB ERT , ĠStruct BER T ĠB IB REF 3 , ĠER N IE ĠB IB REF 4 , ĠB IB REF 5 , ĠRo BER Ta ĠB IB REF 6 Ġand ĠXL Net ĠB IB REF 7 . ĠThese Ġmodels Ġlearn Ġfrom Ġun super vised / self - super vised Ġobjectives Ġand Ġperform Ġex cell ently Ġin Ġthe Ġdownstream Ġtasks . ĠAmong Ġthese Ġmodels , ĠB ERT Ġadop ts ĠN SP Ġas Ġone Ġof Ġthe Ġobjectives Ġin Ġthe Ġpre - training Ġand Ġshows Ġthat Ġthe ĠN SP Ġtask Ġhas Ġa Ġpositive Ġeffect Ġon Ġthe ĠNL I Ġand ĠM RC Ġtasks . ĠAlthough Ġthe Ġprimary Ġstudy Ġof ĠXL Net Ġand ĠRo BER Ta Ġsuggests Ġthat ĠN SP Ġis Ġineffective Ġwhen Ġthe Ġmodel Ġis Ġtrained Ġwith Ġa Ġlarge Ġsequence Ġlength Ġof Ġ512 , Ġthe Ġeffect Ġof ĠN SP Ġon Ġthe ĠNL I Ġproblems Ġshould Ġstill Ġbe Ġemphasized . ĠThe Ġin efficiency Ġof ĠN SP Ġis Ġlikely Ġbecause Ġthe Ġexpected Ġcontext Ġlength Ġwill Ġbe Ġhal ved Ġfor ĠMask ed ĠLM Ġwhen Ġtaking Ġa Ġsentence Ġpair Ġas Ġthe Ġinput . ĠThe Ġmodels Ġderived Ġfrom ĠB ERT , Ġe . g ., ĠStruct BER T Ġand ĠER N IE Ġ1 . 0 / 2 . 0 , Ġaim Ġto Ġincorporating Ġmore Ġknowledge Ġby Ġelabor ating Ġpre - training Ġobjectives . ĠThis Ġwork Ġaims Ġto Ġenhance Ġthe ĠN SP Ġtask Ġand Ġver ifies Ġwhether Ġdocument - level Ġinformation Ġis Ġhelpful Ġfor Ġthe Ġpre - training . ĠTo Ġprobe Ġwhether Ġour Ġmethod Ġachieves Ġa Ġbetter Ġregular ization Ġability , Ġour Ġapproach Ġis Ġalso Ġevaluated Ġon Ġthe ĠH ANS ĠB IB REF 0 Ġdataset , Ġwhich Ġcontains Ġhard Ġdata Ġsamples Ġconstructed Ġby Ġthree Ġhe ur istics . ĠPrevious Ġadvanced Ġmodels Ġsuch Ġas ĠB ERT Ġfail Ġon Ġthe ĠH ANS Ġdataset , Ġand Ġthe Ġtest Ġaccuracy Ġcan Ġbarely Ġexceed Ġ0 % Ġin Ġthe Ġsubset Ġof Ġtest Ġexamples . </s> Related ĠWork Ġ: :: ĠUn super vised Ġlearning Ġfrom Ġdocument </s> In Ġrecent Ġyears , Ġmany Ġun super vised Ġpre - training Ġmethods Ġhave Ġbeen Ġproposed Ġin Ġthe ĠN LP Ġfields Ġto Ġextract Ġknowledge Ġamong Ġsentences ĠD BL P : conf / n ips / K i ros Z S Z UTF 15 , D BL P : conf / em nl p / Con neau KS BB 17 , D BL P : conf / ic lr / Log es war an L 18 , D BL P : j ournals / cor r / abs - 19 03 - 09 424 . ĠThe Ġprediction Ġof Ġsurrounding Ġsentences Ġend ows Ġthe Ġmodel Ġwith Ġthe Ġability Ġto Ġmodel Ġthe Ġsentence - level Ġco herence . ĠSkip - Th ought ĠB IB REF 8 Ġconsists Ġof Ġan Ġenc oder Ġand Ġtwo Ġdec od ers . ĠWhen Ġa Ġsentence Ġis Ġgiven Ġand Ġencoded Ġinto Ġa Ġvector Ġby Ġthe Ġenc oder , Ġthe Ġdec od ers Ġare Ġtrained Ġto Ġpredict Ġthe Ġnext Ġsentence Ġand Ġthe Ġprevious Ġsentence . ĠThe Ġgoal Ġis Ġto Ġobtain Ġa Ġbetter Ġsentence Ġrepresentation Ġthat Ġis Ġuseful Ġfor Ġreconstruct ing Ġthe Ġsurrounding Ġcontext . ĠConsidering Ġthat Ġthe Ġestimation Ġof Ġthe Ġlikelihood Ġof Ġsequences Ġis Ġcomput ationally Ġexpensive Ġand Ġtime - consuming , Ġthe ĠQuick - Th ought Ġmethod ĠB IB REF 9 Ġsimpl ifies Ġthis Ġin Ġa Ġmanner Ġsimilar Ġto Ġsampled Ġsoft max ĠB IB REF 10 , Ġwhich Ġclass ifies Ġthe Ġinput Ġsentences Ġbetween Ġsurrounding Ġsentences Ġand Ġthe Ġother . ĠNote Ġthat ĠQuick - Th ought Ġdoes Ġnot Ġdistinguish Ġbetween Ġthe Ġprevious Ġand Ġnext Ġsentence Ġas Ġit Ġis Ġfunctionally Ġrotation Ġinvari ant . ĠHowever , ĠB ERT Ġis Ġorder - dependent , Ġand Ġthe Ġdiscrimination Ġcan Ġprovide Ġmore Ġsupervision Ġsignal Ġfor Ġsemantic Ġlearning . ĠIn fer Sent ĠB IB REF 11 Ġinstead Ġpre - tr ains Ġthe Ġmodel Ġin Ġa Ġmanner Ġof Ġsupervised Ġlearning . ĠIt Ġuses Ġa Ġlarge - scale ĠNL I Ġdataset Ġas Ġthe Ġpre - training Ġtask Ġto Ġlearn Ġthe Ġsentence Ġrepresentation . ĠIn Ġour Ġwork , Ġwe Ġfocus Ġon Ġdesigning Ġa Ġmore Ġeffective Ġdocument - level Ġobjective , Ġextended Ġfrom Ġthe ĠN SP Ġtask . ĠThe Ġproposed Ġmethod Ġwill Ġbe Ġdescribed Ġin Ġthe Ġfollowing Ġsection Ġand Ġvalidated Ġby Ġproviding Ġextensive Ġexperimental Ġresults Ġin Ġthe Ġexperiment Ġpart . </s> Method </s> Our Ġmethod Ġfollows Ġthe Ġsame Ġinput Ġformat Ġand Ġthe Ġmodel Ġarchitecture Ġwith Ġoriginal ĠB ERT . ĠThe Ġproposed Ġmethod Ġsolely Ġconcerns Ġthe ĠN SP Ġtask . ĠThe ĠN SP Ġtask Ġis Ġa Ġbinary Ġclassification Ġtask , Ġwhich Ġtakes Ġtwo Ġsentences Ġ( A Ġand ĠB ) Ġas Ġinput Ġand Ġdetermines Ġwhether ĠB Ġis Ġthe Ġnext Ġsentence Ġof ĠA . ĠAlthough Ġit Ġhas Ġbeen Ġproven Ġto Ġbe Ġvery Ġeffective Ġfor ĠB ERT , Ġthere Ġare Ġtwo Ġmajor Ġdeficiencies . Ġ( 1 ) ĠDiscrimination Ġbetween ĠIs Next Ġand ĠDiff Doc Ġ( the Ġlabel Ġof Ġthe Ġsentences Ġdrawn Ġfrom Ġdifferent Ġdocuments Ġvia Ġnegative Ġsampling ) Ġis Ġsem antically Ġshallow Ġas Ġthe Ġsignal Ġof Ġsentence Ġorder Ġis Ġabsent . ĠThe Ġcorrelation Ġbetween Ġtwo Ġsuccessive Ġsentences Ġcould Ġbe Ġobvious , Ġdue Ġto , Ġfor Ġexample , Ġlex ical Ġoverlap Ġor Ġthe Ġconjunction Ġused Ġat Ġthe Ġbeginning Ġof Ġthe Ġsecond Ġsentence . ĠAs Ġreported ĠB IB REF 1 , Ġthe Ġfinal Ġpre - trained Ġmodel Ġis Ġable Ġto Ġachieve Ġ97 %- 98 % Ġaccuracy Ġon Ġthe ĠN SP Ġtask . Ġ( 2 ) ĠB ERT Ġis Ġorder - sensitive , Ġi . e ., Ġ$ f _{ \ text { BER T }} ( Ġ\ text tt Ġ{ A }, Ġ\ text tt Ġ{ B }) Ġ\ ne Ġf _{ \ text { BER T }} (\ text tt Ġ{ B }, Ġ\ text tt Ġ{ A }) $, Ġwhile ĠN SP Ġis Ġun i - direction al . ĠWhen Ġthe Ġorder Ġof Ġthe Ġinput ĠNL I Ġpair Ġis Ġreversed , Ġthe Ġperformance Ġwill Ġdegrade . ĠFor Ġinstance , Ġthe Ġaccuracy Ġdecreases Ġby Ġabout Ġ0 . 5 % Ġon ĠMN LI ĠB IB REF 12 Ġand Ġ0 . 4 % Ġon ĠQ N LI Ġafter Ġswapping Ġthe Ġsentences Ġin Ġour Ġexperiments Ġ. </s> Mot ivated Ġby Ġthese Ġproblems , Ġwe Ġpropose Ġto Ġextend Ġthe ĠN SP Ġtask Ġwith Ġprevious Ġsentence Ġprediction Ġ( PS P ). ĠDespite Ġits Ġsimplicity , Ġempirical Ġresults Ġshow Ġthat Ġthis Ġis Ġbeneficial Ġfor Ġdownstream Ġtasks , Ġincluding Ġboth ĠNL I Ġand ĠM RC Ġtasks . ĠTo Ġfurther Ġincorporate Ġthe Ġdocument - level Ġinformation , Ġthe Ġscope Ġis Ġalso Ġexpanded Ġto Ġinclude Ġmore Ġsurrounding Ġsentences , Ġnot Ġjust Ġthe Ġadjacent . ĠThe Ġmethod Ġis Ġbriefly Ġillustrated Ġin ĠFig . ĠFIG REF 6 . </s> Method Ġ: :: ĠPrevious ĠSent ence ĠPrediction </s> Learning Ġto Ġrecognize Ġthe Ġprevious Ġsentence Ġenables Ġthe Ġmodel Ġto Ġcapture Ġmore Ġcompact Ġcontext Ġinformation . ĠOne Ġwould Ġargue Ġthat ĠIs Prev Ġ( the Ġlabel Ġof ĠPSP ) Ġis Ġredundant Ġas Ġit Ġplays Ġa Ġsimilar Ġrole Ġof ĠIs Next Ġ( the Ġlabel Ġof ĠN SP ). ĠIn Ġfact , ĠQuick - Th ought Ġuses Ġthe Ġsampled Ġsoft max Ġto Ġapproximate Ġthe Ġsentence Ġlikelihood Ġestimation Ġof ĠSkip - Th ought , Ġand Ġit Ġactually Ġdoes Ġnot Ġdifferentiate Ġbetween Ġthe Ġprevious Ġand Ġnext Ġsentences . ĠHowever , Ġwe Ġsuggest Ġthe Ġorder Ġdiscrimination Ġis Ġessential Ġfor ĠB ERT Ġpre - training . ĠQuick - Th ought Ġaims Ġat Ġextracting Ġsentence Ġembed ding , Ġand Ġit Ġuses Ġa Ġrotating Ġsymm etric Ġfunction , Ġwhich Ġmakes ĠIs Prev Ġredundant Ġin ĠQuick - Th ought . ĠIn Ġcontrast , ĠB ERT Ġis Ġorder - sensitive , Ġand Ġlearning Ġthe Ġsymm etric Ġregular ization Ġis Ġrather Ġnecessary . ĠAnother Ġadvantage Ġof ĠPSP Ġis Ġto Ġenhance Ġdocument - level Ġsupervision . ĠIn Ġorder Ġto Ġtell Ġthe Ġdifference Ġbetween ĠN SP Ġand ĠPSP , Ġthe Ġmodel Ġhas Ġto Ġextract Ġthe Ġdetailed Ġsemantic Ġfor Ġinference . </s> Method Ġ: :: ĠGathering ĠMore ĠDocument - level ĠInformation </s> Beyond ĠN SP Ġand ĠPSP , Ġwhich Ġenable Ġthe Ġmodel Ġto Ġlearn Ġthe Ġshort - term Ġdependency Ġbetween Ġsentences , Ġwe Ġalso Ġpropose Ġto Ġexpand Ġthe Ġscope Ġof Ġdiscrimination Ġtask Ġto Ġfurther Ġincorporate Ġthe Ġdocument - level Ġinformation . </s> Specifically , Ġwe Ġalso Ġinclude Ġthe Ġin - adj acent Ġsentences Ġin Ġthe Ġsentence - pair Ġclassification Ġtask . ĠThe Ġin - adj acent Ġsentences Ġnext Ġto Ġthe ĠIs Prev Ġand ĠIs Next Ġsentences Ġare Ġsampled , Ġlabeled Ġas ĠIs Prev In adj Ġand ĠIs Next In adj Ġ( cf . Ġthe Ġbottom Ġof ĠFig . ĠFIG REF 6 ). ĠNote Ġthat Ġthese Ġin - adj acent Ġsentences Ġwill Ġintroduce Ġmuch Ġmore Ġtraining Ġnoise Ġto Ġthe Ġmodel . ĠTherefore , Ġthe Ġlabel Ġsmoot hing Ġtechnique Ġis Ġadopted Ġto Ġreduce Ġthe Ġnoise Ġof Ġthese Ġadditional Ġsamples . ĠIt Ġachieves Ġthis Ġby Ġrelaxing Ġour Ġconfidence Ġon Ġthe Ġlabels , Ġe . g ., Ġtransforming Ġthe Ġtarget Ġprobability Ġfrom Ġ( 1 . 0 , Ġ0 . 0 ) Ġto Ġ( 0 . 8 , Ġ0 . 2 ) Ġin Ġa Ġbinary Ġclassification Ġproblem . </s> In Ġsummary , Ġwhen ĠA Ġis Ġgiven , Ġthe Ġpre - training Ġexample Ġfor Ġeach Ġlabel Ġis Ġconstructed Ġas Ġfollows : </s> Is Next : ĠCho osing Ġthe Ġadjacent Ġfollowing Ġsentence Ġas ĠB . </s> Is Prev : ĠCho osing Ġthe Ġadjacent Ġprevious Ġsentence Ġas ĠB . </s> Is Next In adj : ĠCho osing Ġthe Ġin - adj acent Ġfollowing Ġsentence Ġas ĠB . ĠThere Ġis Ġa Ġsentence Ġbetween ĠA Ġand ĠB . </s> Is Prev In adj : ĠCho osing Ġthe Ġin - adj acent Ġprevious Ġsentence Ġas ĠB . ĠThere Ġis Ġa Ġsentence Ġbetween ĠA Ġand ĠB . </s> Diff Doc : ĠDrawing ĠB Ġrandomly Ġfrom Ġa Ġdifferent Ġdocument . </s> Exper iment ĠSettings </s> This Ġsection Ġgives Ġdetailed Ġexperiment Ġsettings . ĠThe Ġmethod Ġis Ġevaluated Ġon Ġthe ĠB ERT base Ġmodel , Ġwhich Ġhas Ġ12 Ġlayers , Ġ12 Ġself - att ention Ġheads Ġwith Ġa Ġhidden Ġsize Ġof Ġ768 . </s> To Ġaccelerate Ġthe Ġtraining Ġspeed , Ġtwo - phase Ġtraining ĠB IB REF 1 Ġis Ġadopted . ĠThe Ġfirst Ġphase Ġuses Ġa Ġmaximal Ġsentence Ġlength Ġof Ġ128 , Ġand Ġ512 Ġfor Ġthe Ġsecond Ġphase . ĠThe Ġnumbers Ġof Ġtraining Ġsteps Ġof Ġtwo Ġphases Ġare Ġ50 K Ġand Ġ40 K Ġfor Ġthe ĠB ERT Base Ġmodel . ĠWe Ġused ĠAdam W ĠB IB REF 13 Ġoptim izer Ġwith Ġa Ġlearning Ġrate Ġof Ġ1 e - 4 , Ġa Ġ$\ beta Ġ_ 1 $ Ġof Ġ0 . 9 , Ġa Ġ$\ beta Ġ_ 2 $ Ġof Ġ0 . 999 Ġand Ġa ĠL 2 Ġweight Ġdecay Ġrate Ġof Ġ$ 0 . 01 $. ĠThe Ġfirst Ġ10 % Ġof Ġthe Ġtotal Ġsteps Ġare Ġused Ġfor Ġlearning Ġrate Ġwarming Ġup , Ġfollowed Ġby Ġthe Ġlinear Ġdecay Ġschema . ĠWe Ġused Ġa Ġdrop out Ġprobability Ġof Ġ0 . 1 Ġon Ġall Ġlayers . ĠThe Ġdata Ġused Ġfor Ġpre - training Ġis Ġthe Ġsame Ġas ĠB ERT , Ġi . e ., ĠEnglish ĠWikipedia Ġ( 2500 M Ġwords ) Ġand ĠBook Corp us Ġ( 800 M Ġwords ) ĠB IB REF 14 . ĠFor Ġthe ĠMask ed ĠLM Ġtask , Ġwe Ġfollowed Ġthe Ġsame Ġmask ing Ġrate Ġand Ġsettings Ġas Ġin ĠB ERT . </s> We Ġexplore Ġthree Ġmethod Ġsettings Ġfor Ġcomparison . </s> BER T - PN : ĠThe ĠN SP Ġtask Ġin ĠB ERT Ġis Ġreplaced Ġby Ġa Ġ3 - class Ġtask Ġwith ĠIs Next , ĠIs Prev Ġand ĠDiff Doc . ĠThe Ġlabel Ġdistribution Ġis Ġ1 : 1 : 1 . </s> BER T - PN 5 cl s : ĠThe ĠN SP Ġtask Ġin ĠB ERT Ġis Ġreplaced Ġby Ġa Ġ5 - class Ġtask Ġwith Ġtwo Ġadditional Ġlabels ĠIs Next In adj , ĠIs Prev In adj . ĠThe Ġlabel Ġdistribution Ġis Ġ1 : 1 : 1 : 1 : 1 . </s> BER T - PN sm th : ĠIt Ġuses Ġthe Ġsame Ġdata Ġwith ĠB ERT - PN 5 cl s , Ġexcept Ġthat Ġthe ĠIs Prev In adj Ġ( Is Next In adj ) Ġlabel Ġis Ġmapped Ġto ĠIs Prev Ġ( Is Next ) Ġwith Ġa Ġlabel Ġsmoot hing Ġfactor Ġof Ġ0 . 8 . </s> BER T - PN Ġis Ġused Ġto Ġverify Ġthe Ġfeasibility Ġof ĠPSP . ĠThe Ġcomparison Ġwith ĠB ERT - PN 5 cl s Ġillustrates Ġwhether Ġmore Ġdocument - level Ġinformation Ġhelps . ĠB ERT - PN sm th , Ġwhich Ġis Ġthe Ġlabel - sm oot hed Ġversion Ġof ĠB ERT - PN 5 cl s , Ġis Ġused Ġto Ġcompare Ġwith ĠB ERT - PN 5 cl s Ġto Ġsee Ġwhether Ġthe Ġnoise Ġreduction Ġis Ġnecessary . </s> In Ġthe Ġfollowing , Ġwe Ġfirst Ġshow Ġthat ĠB ERT Ġis Ġorder - sensitive Ġand Ġthe Ġuse Ġof ĠPSP Ġremedies Ġthis Ġproblem . ĠThen Ġwe Ġprovide Ġexperimental Ġresults Ġon Ġthe ĠNL I Ġand ĠM RC Ġtasks Ġto Ġverify Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġmethod . ĠAt Ġlast , Ġthe Ġproposed Ġmethod Ġis Ġevaluated Ġon Ġseveral ĠChinese Ġdatasets . </s> Order - inv ari ant Ġwith ĠPSP </s> N SP Ġin Ġthe Ġpre - training Ġis Ġuseful Ġfor ĠNL I Ġand ĠM RC Ġtask ĠB IB REF 1 . ĠHowever , Ġwe Ġsuggested Ġthat ĠB ERT Ġtrained Ġwith ĠN SP Ġis Ġorder - sensitive , Ġi . e ., Ġthe Ġperformance Ġof ĠB ERT Ġdepends Ġon Ġthe Ġorder Ġof Ġthe Ġinput Ġsentence Ġpair . ĠTo Ġverify Ġour Ġassumption , Ġa Ġprimary Ġexperiment Ġwas Ġconducted . ĠThe Ġorder Ġof Ġthe Ġinput Ġpair Ġof ĠNL I Ġsamples Ġis Ġreversed Ġin Ġthe Ġfine - tun ing Ġphase , Ġand Ġother Ġhyper - param eters Ġand Ġsettings Ġkeep Ġthe Ġsame Ġwith Ġthe ĠB ERT Ġpaper . ĠTable ĠT AB REF 19 Ġshows Ġthe Ġaccuracy Ġon Ġthe Ġvalidation Ġset Ġof Ġthe ĠMN LI Ġand ĠQ N LI Ġdatasets . ĠFor Ġthe ĠB ERT Base Ġmodel , Ġwhen Ġthe Ġsentences Ġare Ġswapped , Ġthe Ġaccuracy Ġdecreases Ġby Ġ0 . 5 % Ġon Ġthe ĠMN LI Ġtask Ġand Ġ0 . 4 % Ġon Ġthe ĠQ N LI Ġtask . ĠThese Ġresults Ġconfirm Ġthat ĠB ERT Ġtrained Ġwith ĠN SP Ġonly Ġis Ġindeed Ġaffected Ġby Ġthe Ġinput Ġorder . ĠThis Ġphenomenon Ġmotiv ates Ġus Ġto Ġmake Ġthe ĠN SP Ġtask Ġsymm etric . ĠThe Ġresults Ġof ĠB ERT - PN Ġverify Ġthat ĠB ERT - PN Ġis Ġorder - inv ari ant . ĠWhen Ġthe Ġinput Ġorder Ġis Ġreversed , Ġthe Ġperformance Ġof ĠB ERT - PN Ġremains Ġstable . ĠThese Ġresults Ġindicate Ġthat Ġour Ġmethod Ġis Ġable Ġto Ġremedy Ġthe Ġorder - s ensitivity Ġproblem . </s> Results Ġof ĠNL I ĠT asks Ġ: :: ĠGL UE </s> A Ġpopular Ġbenchmark Ġfor Ġevaluation Ġof Ġlanguage Ġunderstanding Ġis ĠGL UE ĠB IB REF 2 , Ġwhich Ġis Ġa Ġcollection Ġof Ġthree ĠNL I Ġtasks Ġ( MN LI , ĠQ N LI Ġand ĠR TE ), Ġthree Ġsemantic Ġtextual Ġsimilarity Ġ( ST S ) Ġtasks Ġ( Q Q P , ĠST S - B Ġand ĠMR PC ), Ġtwo Ġtext Ġclassification Ġ( TC ) Ġtasks Ġ( S ST - 2 Ġand ĠCo LA ). ĠAlthough Ġthe Ġmethod Ġis Ġmotivated Ġfor Ġpair - wise Ġreasoning , Ġthe Ġresults Ġof Ġother Ġproblems Ġare Ġalso Ġlisted . </s> Our Ġimplementation Ġfollows Ġthe Ġsame Ġway Ġthat ĠB ERT Ġperforms Ġin Ġthese Ġtasks . ĠThe Ġfine - tun ing Ġwas Ġconducted Ġfor Ġ3 Ġepoch s Ġfor Ġall Ġthe Ġtasks , Ġwith Ġa Ġlearning Ġrate Ġof Ġ2 e - 5 . ĠThe Ġpredictions Ġwere Ġobtained Ġby Ġevaluating Ġthe Ġtraining Ġcheckpoint Ġwith Ġthe Ġbest Ġvalidation Ġperformance . </s> Table ĠT AB REF 21 Ġillustrates Ġthe Ġexperimental Ġresults , Ġshowing Ġthat Ġour Ġmethod Ġis Ġbeneficial Ġfor Ġall Ġof ĠNL I Ġtasks . ĠThe Ġimprovement Ġon Ġthe ĠR TE Ġdataset Ġis Ġsignificant , Ġi . e ., Ġ4 % Ġabsolute Ġgain Ġover Ġthe ĠB ERT Base . ĠBesides ĠNL I , Ġour Ġmodel Ġalso Ġperforms Ġbetter Ġthan ĠB ERT Base Ġin Ġthe ĠST S Ġtask . ĠThe ĠST S Ġtasks Ġare Ġsem antically Ġsimilar Ġto Ġthe ĠNL I Ġtasks , Ġand Ġhence Ġable Ġto Ġtake Ġadvantage Ġof ĠPSP Ġas Ġwell . ĠActually , Ġthe Ġproposed Ġmethod Ġhas Ġa Ġpositive Ġeffect Ġwhenever Ġthe Ġinput Ġis Ġa Ġsentence Ġpair . ĠThe Ġimprovements Ġsuggest Ġthat Ġthe ĠPSP Ġtask Ġencourages Ġthe Ġmodel Ġto Ġlearn Ġmore Ġdetailed Ġsemantics Ġin Ġthe Ġpre - training , Ġwhich Ġimproves Ġthe Ġmodel Ġon Ġthe Ġdownstream Ġlearning Ġtasks . ĠMoreover , Ġour Ġmethod Ġis Ġsurprisingly Ġable Ġto Ġachieve Ġslightly Ġbetter Ġresults Ġin Ġthe Ġsingle - sent ence Ġproblem . ĠThe Ġimprovement Ġshould Ġbe Ġattributed Ġto Ġbetter Ġsemantic Ġrepresentation . </s> When Ġcomparing Ġbetween ĠP N Ġand ĠP N 5 cl s , ĠP N 5 cl s Ġachieves Ġbetter Ġresults Ġthan ĠP N . ĠThis Ġindicates Ġthat Ġincluding Ġa Ġbroader Ġrange Ġof Ġthe Ġcontext Ġis Ġeffective Ġfor Ġimproving Ġinference Ġability . ĠConsidering Ġthat Ġthe Ġrepresentation Ġof ĠIs Next Ġand ĠIs Next In adj Ġshould Ġbe Ġcoherent , Ġwe Ġpropose ĠB ERT Base - PN sm th Ġto Ġmitigate Ġthis Ġproblem . ĠP N sm th Ġfurther Ġimproves Ġthe Ġperformance Ġand Ġob tains Ġan Ġaveraged Ġscore Ġof Ġ81 . 0 . </s> Results Ġof ĠNL I ĠT asks Ġ: :: ĠH ANS </s> Although ĠB ERT Ġhas Ġshown Ġits Ġeffectiveness Ġin Ġthe ĠNL I Ġtasks . ĠB IB REF 0 Ġpointed Ġout Ġthat ĠB ERT Ġis Ġstill Ġvulnerable Ġin Ġthe ĠNL I Ġtask Ġas Ġit Ġis Ġprone Ġto Ġadopting Ġfall ible Ġhe ur istics . ĠTherefore , Ġthey Ġreleased Ġa Ġdataset , Ġcalled ĠThe ĠHe uristic ĠAnalysis Ġfor ĠNL I ĠSystems Ġ( H ANS ), Ġto Ġprobe Ġwhether Ġthe Ġmodel Ġlearns Ġinappropriate Ġinduct ive Ġbias Ġfrom Ġthe Ġtraining Ġset . ĠIt Ġis Ġconstructed Ġby Ġthree Ġhe ur istics , Ġi . e ., Ġlex ical Ġoverlap Ġhe uristic , Ġsub - sequence Ġhe uristic , Ġand Ġconstituent Ġhe uristic . ĠThe Ġfirst Ġhe uristic Ġassumes Ġthat Ġa Ġpremise Ġentails Ġall Ġhypotheses Ġconstructed Ġfrom Ġwords Ġin Ġthe Ġpremise , Ġthe Ġsecond Ġassumes Ġthat Ġa Ġpremise Ġentails Ġall Ġof Ġits Ġcontiguous Ġsub - sequ ences Ġand Ġthe Ġthird Ġassumes Ġthat Ġa Ġpremise Ġentails Ġall Ġcomplete Ġsub - t rees Ġin Ġits Ġparse Ġtree . ĠB ERT Ġand Ġother Ġadvanced Ġmodels Ġfail Ġon Ġthis Ġdataset Ġand Ġbarely Ġexceeds Ġ0 % Ġaccuracy Ġin Ġmost Ġcases ĠB IB REF 0 . </s> Fig . ĠFIG REF 23 Ġillustrates Ġthe Ġaccuracy Ġof ĠB ERT Base Ġand ĠB ERT Base - PN sm th Ġon Ġthe ĠH ANS Ġdataset . ĠThe Ġevaluation Ġis Ġmade Ġupon Ġthe Ġmodel Ġtrained Ġon Ġthe ĠMN LI Ġdataset Ġand Ġthe Ġpredicted Ġneutral Ġand Ġcontradiction Ġlabels Ġare Ġmapped Ġinto Ġnon - ent ail ment . ĠThe ĠB ERT Base - PN sm th Ġevidently Ġoutper forms Ġthe ĠB ERT Base Ġwith Ġthe Ġnon - ent ail ment Ġexamples . ĠFor Ġthe Ġnon - ent ail ment Ġsamples Ġconstructed Ġusing Ġthe Ġlex ical Ġoverlap Ġhe uristic , Ġour Ġmodel Ġachieves Ġ160 % Ġrelative Ġimprovement Ġover Ġthe ĠB ERT Base Ġmodel . ĠSome Ġsamples Ġare Ġconstructed Ġby Ġswapping Ġthe Ġentities Ġin Ġthe Ġsentence Ġ( e . g ., ĠThe Ġdoctor Ġsaw Ġthe Ġlawyer Ġ$\ n right arrow Ġ$ ĠThe Ġlawyer Ġsaw Ġthe Ġdoctor ) Ġand Ġour Ġmethod Ġoutper forms ĠB ERT Base Ġby Ġ20 % Ġin Ġaccuracy . ĠWe Ġsuggest Ġthat Ġthe ĠMask ed ĠLM Ġtask Ġcan Ġhardly Ġmodel Ġthe Ġrelationship Ġbetween Ġtwo Ġentities Ġand ĠN SP Ġonly Ġis Ġtoo Ġsem antically Ġshallow Ġto Ġcapture Ġthe Ġprecise Ġmeaning . ĠHowever , Ġthe Ġdiscrimination Ġbetween ĠN SP Ġand ĠPSP Ġenhances Ġthe Ġmodel Ġto Ġrealize Ġthe Ġrole Ġof Ġentities Ġin Ġa Ġgiven Ġsentence . ĠFor Ġexample , Ġto Ġdetermine Ġthat ĠA Ġ( X Ġis Ġbeautiful ) Ġrather Ġthan Ġ$\ bar {\ text tt Ġ{ A }} $ Ġ( Y Ġis Ġbeautiful ) Ġis Ġthe Ġprevious Ġsentence Ġof ĠB Ġ( Y Ġloves ĠX ), Ġthe Ġmodel Ġhave Ġto Ġrecognize Ġthe Ġrelationship Ġbetween ĠX Ġand ĠY . ĠIn Ġcontrast , Ġwhen ĠPSP Ġis Ġabsent , ĠN SP Ġcan Ġbe Ġprobably Ġinferred Ġby Ġlearning Ġthe Ġoccurrence Ġbetween Ġbeautiful Ġand Ġloves , Ġregardless Ġof Ġthe Ġsentence Ġstructure . ĠThe Ġdetailed Ġperformance Ġof Ġthe Ġproposed Ġmethod Ġon Ġthe ĠH ANS Ġdataset Ġis Ġillustrated Ġin ĠFig . ĠFIG REF 24 . ĠThe Ġdefinition Ġof Ġeach Ġhe uristic Ġrules Ġcan Ġbe Ġfound Ġin ĠB IB REF 0 . </s> Results Ġof ĠM RC ĠT asks Ġ: :: ĠS Qu AD Ġv 1 . 1 Ġand Ġv 2 . 0 </s> We Ġalso Ġevaluate Ġour Ġmethod Ġon Ġthe ĠM RC Ġtasks . ĠThe ĠStanford ĠQuestion ĠAn sw ering ĠDat as et Ġ( S Qu AD Ġv 1 . 1 ) Ġis Ġa Ġquestion Ġanswering Ġ( Q A ) Ġdataset , Ġwhich Ġconsists Ġof Ġ100 K Ġsamples ĠB IB REF 15 . ĠEach Ġdata Ġsample Ġhas Ġa Ġquestion Ġand Ġa Ġcorresponding ĠWikipedia Ġpassage Ġthat Ġcontains Ġthe Ġanswer . ĠThe Ġgoal Ġis Ġto Ġextract Ġthe Ġanswer Ġfrom Ġthe Ġpassage Ġfor Ġthe Ġgiven Ġquestion . </s> In Ġthe Ġfine - tun ing Ġprocedure , Ġwe Ġfollow Ġthe Ġexact Ġway Ġthe ĠB ERT Ġperformed . ĠThe Ġoutput Ġvectors Ġare Ġused Ġto Ġcompute Ġthe Ġscore Ġof Ġtokens Ġbeing Ġstart Ġand Ġend Ġof Ġthe Ġanswer Ġspan . ĠThe Ġvalid Ġspan Ġthat Ġhas Ġthe Ġmaximum Ġscore Ġis Ġselected Ġas Ġthe Ġprediction . ĠAnd Ġsimilarly , Ġthe Ġfine - tun ing Ġtraining Ġwas Ġperformed Ġfor Ġ3 Ġepoch s Ġwith Ġa Ġlearning Ġrate Ġof Ġ3 e - 5 . </s> Table ĠT AB REF 26 Ġdemonstrates Ġthe Ġresults Ġon Ġthe ĠS Qu AD Ġv 1 . 1 Ġdataset . ĠThe Ġcomparison Ġbetween ĠB ERT Base - PN Ġand ĠB ERT Base Ġindicates Ġthat Ġthe Ġinclusion Ġof Ġthe ĠPSP Ġsubt ask Ġis Ġbeneficial Ġ( 2 . 4 % Ġabsolute Ġimprovement ). ĠWhen Ġusing ĠB ERT Base - PN sm th , Ġanother Ġ0 . 3 % Ġincrease Ġin ĠEM Ġcan Ġbe Ġobtained . ĠThe Ġexperimental Ġresults Ġon Ġthe ĠS Qu AD Ġv 2 . 0 ĠB IB REF 16 Ġare Ġalso Ġshown Ġin ĠTable . ĠT AB REF 26 . ĠThe ĠS Qu AD Ġv 2 . 0 Ġdiffers Ġfrom ĠS Qu AD Ġv 1 . 1 Ġby Ġallowing Ġthe Ġquestion - paragraph Ġpairs Ġthat Ġhave Ġno Ġanswer . ĠFor ĠS Qu AD Ġv 2 . 0 , Ġour Ġmethod Ġalso Ġachieved Ġabout Ġ4 % Ġabsolute Ġimprovement Ġin Ġboth ĠEM Ġand ĠF 1 Ġagainst ĠB ERT Base . </s> Results Ġof ĠM RC ĠT asks Ġ: :: ĠR ACE </s> The ĠRe Ad ing ĠComp rehens ion Ġfrom ĠEx amin ations Ġ( R ACE ) Ġdataset ĠB IB REF 17 Ġconsists Ġof Ġ100 K Ġquestions Ġtaken Ġfrom ĠEnglish Ġexams , Ġand Ġthe Ġanswers Ġare Ġgenerated Ġby Ġhuman Ġexperts . ĠThis Ġis Ġone Ġof Ġthe Ġmost Ġchallenging ĠM RC Ġdatasets Ġthat Ġrequire Ġsophisticated Ġreasoning . </s> In Ġour Ġimplementation , Ġthe Ġquestion , Ġdocument , Ġand Ġoption Ġare Ġconc aten ated Ġas Ġa Ġsingle Ġsequence , Ġseparated Ġby Ġ[ SE P ] Ġtoken . ĠAnd Ġeach Ġpart Ġis Ġtrunc ated Ġby Ġa Ġmaximal Ġlength Ġof Ġ40 / 432 / 40 , Ġrespectively . ĠThe Ġmodel Ġcomp utes Ġfor Ġa Ġconc aten ation Ġa Ġscal ar Ġas Ġthe Ġscore , Ġwhich Ġis Ġthen Ġused Ġin Ġa Ġsoft max Ġlayer Ġfor Ġthe Ġfinal Ġprediction . ĠThe Ġfine - tun ing Ġwas Ġconducted Ġfor Ġ5 Ġepoch s , Ġwith Ġa Ġbatch Ġsize Ġof Ġ32 Ġand Ġa Ġlearning Ġrate Ġof Ġ5 e - 5 . ĠAs Ġshown Ġin ĠTable ĠT AB REF 28 , Ġthe Ġproposed Ġmethod Ġsignificantly Ġimprove Ġthe Ġperformance Ġon Ġthe ĠR ACE Ġdataset . ĠB ERT Base - PN Ġob tains Ġ2 . 6 % Ġaccuracy Ġimprovement , Ġand ĠB ERT Base - PN 5 cl s Ġfurther Ġbrings Ġ0 . 4 % Ġabsolute Ġgain . </s> The Ġcomparisons Ġon Ġthe ĠS Qu AD Ġv 1 . 1 , ĠS Qu AD Ġv 2 . 0 , Ġand ĠR ACE Ġdataset Ġdemonstrate Ġthat Ġthe Ġinvolvement Ġof Ġadditional Ġsentence Ġand Ġdiscourse Ġinformation Ġis Ġnot Ġonly Ġbeneficial Ġfor Ġthe ĠNL I Ġtask Ġbut Ġalso Ġthe ĠM RC Ġtask . ĠThis Ġis Ġreasonable Ġas Ġthese Ġtasks Ġheavily Ġrely Ġon Ġthe Ġglobal Ġsemantic Ġunderstanding Ġand Ġsophisticated Ġreasoning Ġamong Ġsentences . ĠAnd Ġthis Ġability Ġcan Ġbe Ġeffectively Ġenhanced Ġby Ġour Ġmethod . </s> Results Ġof ĠChinese ĠN LP ĠT asks </s> The Ġexperiments Ġare Ġalso Ġconducted Ġon ĠChinese ĠN LP Ġtasks : </s> X N LI ĠB IB REF 19 Ġa Ġmulti - ling ual Ġdataset . ĠThe Ġdata Ġsample Ġin ĠX N LI Ġis Ġa Ġsentence Ġpair Ġannot ated Ġwith Ġtextual Ġentail ment . ĠThe ĠChinese Ġpart Ġis Ġused . </s> LC Q MC ĠB IB REF 20 Ġis Ġa Ġdataset Ġfor Ġsequence Ġmatching . ĠA Ġbinary Ġlabel Ġis Ġannot ated Ġfor Ġa Ġsentence Ġpair Ġin Ġthe Ġdataset Ġto Ġindicate Ġwhether Ġthese Ġtwo Ġsentences Ġhave Ġthe Ġsame Ġintention . </s> N LP CC - DB Q A ĠB IB REF 21 Ġform ulates Ġthe Ġdomain - based Ġquestion Ġanswering Ġas Ġa Ġbinary Ġclassification Ġtask . ĠEach Ġdata Ġsample Ġis Ġa Ġquestion - sent ence Ġpair . ĠThe Ġgoal Ġis Ġto Ġidentify Ġwhether Ġthe Ġsentence Ġcontains Ġthe Ġanswer Ġto Ġthe Ġquestion . </s> CM RC - 2018 Ġis Ġthe ĠChinese ĠMachine ĠReading ĠComp rehens ion Ġdataset . ĠSimilar Ġto ĠS Qu AD , Ġthe Ġsystem Ġneeds Ġto Ġextract Ġfragments Ġfrom Ġthe Ġtext Ġas Ġthe Ġanswer . </s> D RC D ĠB IB REF 22 Ġis Ġalso Ġa ĠChinese ĠM RC Ġdata Ġset . ĠThe Ġdata Ġfollows Ġthe Ġformat Ġof ĠS Qu AD . </s> For ĠChinese ĠN LP Ġtasks , Ġwe Ġpre - train Ġthe Ġmodel Ġusing ĠChinese Ġcorpus . ĠWe Ġcollected Ġtextual Ġdata Ġ( 108 79 M Ġtokens Ġin Ġtotal ) Ġfrom Ġthe Ġwebsite , Ġconsisting Ġof ĠHud ong ĠBa ike Ġdata Ġ( 60 84 M Ġtokens ) Ġ, ĠZh ihu Ġdata ( 465 M Ġtokens ) Ġ, ĠS oh u ĠNews ( 39 37 M Ġtokens ) Ġand ĠWikipedia Ġdata Ġ( 393 M Ġtokens ). </s> For Ġthe Ġfirst Ġ3 ĠChinese Ġtasks , Ġwe Ġfollow Ġthe Ġsettings Ġas Ġin ĠER N IE ĠB IB REF 4 . ĠThe Ġexperimental Ġresults Ġare Ġgiven Ġin ĠTable ĠT AB REF 29 . ĠThe Ġproposed Ġmethod Ġis Ġcompared Ġwith Ġfour Ġmodels , Ġi . e ., ĠB ERT Base ĠB IB REF 1 , ĠB ERT Base Ġwith Ġwhole Ġword Ġmask ing ĠB IB REF 18 , ĠER N IE ĠB IB REF 4 Ġand ĠER N IE Ġ2 . 0 ĠB IB REF 5 . ĠOur Ġmethod Ġachieves Ġcomparable Ġor Ġeven Ġbetter Ġresults Ġagainst ĠER N IE Ġ2 . 0 ĠB IB REF 5 . ĠNote Ġthat Ġthe ĠChinese ĠER N IE Ġ2 . 0 Ġis Ġequipped Ġwith Ġ5 Ġdifferent Ġobjectives Ġand Ġit Ġuses Ġmore Ġtraining Ġdata Ġ( 149 88 M Ġtokens Ġin Ġtotal ) Ġthan Ġours . ĠThe Ġresults Ġindicate Ġthat Ġthe Ġproposed Ġmethod Ġis Ġquite Ġeffective Ġfor Ġthe Ġpair - wise Ġsemantic Ġreasoning Ġas Ġsimply Ġincluding ĠPSP Ġcan Ġachieve Ġthe Ġresults Ġon Ġpar Ġwith Ġmultiple Ġobjectives . </s> The Ġresults Ġof ĠCM RC - 2018 Ġand ĠD RC D Ġdatasets Ġare Ġgiven Ġin ĠTable ĠT AB REF 30 . ĠSince Ġthe ĠCM RC - 2018 Ġcompetition Ġdoes Ġnot Ġrelease Ġthe Ġtest Ġset , Ġthe Ġcomparison Ġon Ġthe Ġtest Ġset Ġis Ġabsent . ĠOur Ġresults Ġare Ġobtained Ġusing Ġthe Ġopen - s ourced Ġcode Ġof ĠB ERT - ww m Ġ. ĠWe Ġkeep Ġthe Ġhyper - param eters Ġthe Ġsame Ġwith Ġthat Ġin ĠER N IE ĠB IB REF 4 , Ġexcept Ġthat Ġthe Ġbatch Ġsize Ġis Ġ12 Ġinstead Ġof Ġ64 Ġdue Ġto Ġthe Ġmemory Ġlimit . ĠUnder Ġthis Ġsetting , Ġwe Ġachieved Ġsimilar Ġresults Ġof ĠB ERT Base Ġin Ġthe ĠB ERT - ww m Ġpaper ĠB IB REF 18 . ĠHowever , Ġthis Ġis Ġworse Ġthan Ġthe Ġresults Ġof ĠB ERT Base Ġreported Ġin Ġthe ĠER N IE Ġ2 . 0 Ġpaper ĠB IB REF 5 Ġby Ġabout Ġ1 % Ġin ĠF 1 . ĠThis Ġsuggests Ġthat Ġour Ġresults Ġare Ġcurrently Ġincom parable Ġwith ĠER N IE Ġ2 . 0 . ĠOverall , Ġthe Ġresults Ġin ĠTable ĠT AB REF 30 Ġillustrate Ġthat Ġour Ġmethod Ġis Ġalso Ġeffective Ġfor Ġthe ĠChinese ĠQ A Ġtasks . </s> Conclusion </s> This Ġpaper Ġaims Ġto Ġenrich Ġthe ĠN SP Ġtask Ġto Ġprovide Ġmore Ġdocument - level Ġinformation Ġin Ġthe Ġpre - training . ĠMot ivated Ġby Ġthe Ġin - sy mm etric Ġproperty Ġof ĠN SP , Ġwe Ġpropose Ġto Ġdifferentiate Ġbetween Ġdifferent Ġsentence Ġorders Ġby Ġincluding ĠPSP . ĠDespite Ġthe Ġsimplicity , Ġextensive Ġexperiments Ġdemonstrate Ġthat Ġthe Ġmodel Ġob tains Ġa Ġbetter Ġability Ġin Ġpair - wise Ġsemantic Ġreasoning . ĠOur Ġwork Ġsuggests Ġthat Ġthe Ġdocument - level Ġobjective Ġis Ġeffective , Ġat Ġleast Ġfor Ġthe ĠB ERT base Ġmodel . ĠIn Ġthe Ġfuture , Ġwe Ġwill Ġinvestigate Ġthe Ġway Ġto Ġtake Ġadvantages Ġof Ġboth Ġlarge - scale Ġtraining Ġand Ġour Ġmethod .