+1.86 in terms of F1 score on CTB5, +1.80 on CTB6, +2.19 on UD1.4
EXTRACTIVE
 +1.86
EXTRACTIVE
Table presents the experimental results on the POS task. As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i.e., outperforming BERT-tagger by +1.86 in terms of F1 score on CTB5, +1.80 on CTB6 and +2.19 on UD1.4. As far as we are concerned, we are achieving SOTA performances on the three datasets. Weighted cross entropy and focal loss only gain a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in resolving the data imbalance issue. The proposed DSC loss performs robustly on all the three datasets.
Table presents the experimental results on the POS task. As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i.e., outperforming BERT-tagger by +1.86 in terms of F1 score on CTB5, +1.80 on CTB6 and +2.19 on UD1.4. As far as we are concerned, we are achieving SOTA performances on the three datasets. Weighted cross entropy and focal loss only gain a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in resolving the data imbalance issue. The proposed DSC loss performs robustly on all the three datasets.
1911.02855
Introduction </s> Data Ġimbalance Ġis Ġa Ġcommon Ġissue Ġin Ġa Ġvariety Ġof ĠN LP Ġtasks Ġsuch Ġas Ġtagging Ġand Ġmachine Ġreading Ġcomprehension . ĠTable ĠT AB REF 3 Ġgives Ġconcrete Ġexamples : Ġfor Ġthe ĠNamed ĠEntity ĠRecogn ition Ġ( NER ) Ġtask ĠB IB REF 2 , ĠB IB REF 3 , Ġmost Ġtokens Ġare Ġbackgrounds Ġwith Ġtagging Ġclass Ġ$ O $. ĠSpecifically , Ġthe Ġnumber Ġof Ġtokens Ġtagging Ġclass Ġ$ O $ Ġis Ġ5 Ġtimes Ġas Ġmany Ġas Ġthose Ġwith Ġentity Ġlabels Ġfor Ġthe ĠCo N LL 03 Ġdataset Ġand Ġ8 Ġtimes Ġfor Ġthe ĠOnt o Notes 5 . 0 Ġdataset ; ĠData - im balanced Ġissue Ġis Ġmore Ġsevere Ġfor ĠM RC Ġtasks ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 Ġwith Ġthe Ġvalue Ġof Ġnegative - positive Ġratio Ġbeing Ġ50 - 200 . </s> Data Ġimbalance Ġresults Ġin Ġthe Ġfollowing Ġtwo Ġissues : Ġ( 1 ) Ġthe Ġtraining - test Ġdiscrepancy : ĠWithout Ġbalancing Ġthe Ġlabels , Ġthe Ġlearning Ġprocess Ġtends Ġto Ġconverge Ġto Ġa Ġpoint Ġthat Ġstrongly Ġbiases Ġtowards Ġclass Ġwith Ġthe Ġmajority Ġlabel . ĠThis Ġactually Ġcreates Ġa Ġdiscrepancy Ġbetween Ġtraining Ġand Ġtest : Ġat Ġtraining Ġtime , Ġeach Ġtraining Ġinstance Ġcontributes Ġequally Ġto Ġthe Ġobjective Ġfunction Ġwhile Ġat Ġtest Ġtime , ĠF 1 Ġscore Ġconcerns Ġmore Ġabout Ġpositive Ġexamples ; Ġ( 2 ) Ġthe Ġoverwhelming Ġeffect Ġof Ġeasy - negative Ġexamples . ĠAs Ġpointed Ġout Ġby Ġmen g 2019 ds reg , Ġsignificantly Ġlarge Ġnumber Ġof Ġnegative Ġexamples Ġalso Ġmeans Ġthat Ġthe Ġnumber Ġof Ġeasy - negative Ġexample Ġis Ġlarge . ĠThe Ġhuge Ġnumber Ġof Ġeasy Ġexamples Ġtends Ġto Ġoverwhelm Ġthe Ġtraining , Ġmaking Ġthe Ġmodel Ġnot Ġsufficiently Ġlearned Ġto Ġdistinguish Ġbetween Ġpositive Ġexamples Ġand Ġhard - negative Ġexamples . ĠThe Ġcross - ent ropy Ġobjective Ġ( CE Ġfor Ġshort ) Ġor Ġmaximum Ġlikelihood Ġ( M LE ) Ġobjective , Ġwhich Ġis Ġwidely Ġadopted Ġas Ġthe Ġtraining Ġobjective Ġfor Ġdata - im balanced ĠN LP Ġtasks ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 , ĠB IB REF 15 , Ġhandles Ġneither Ġof Ġthe Ġissues . </s> To Ġhandle Ġthe Ġfirst Ġissue , Ġwe Ġpropose Ġto Ġreplace ĠCE Ġor ĠM LE Ġwith Ġlosses Ġbased Ġon Ġthe ĠS Ã¸ ren sen âĢĵ D ice Ġcoefficient ĠB IB REF 0 Ġor ĠT vers ky Ġindex ĠB IB REF 1 . ĠThe ĠS Ã¸ ren sen âĢĵ D ice Ġcoefficient , Ġdice Ġloss Ġfor Ġshort , Ġis Ġthe Ġharmonic Ġmean Ġof Ġprecision Ġand Ġrecall . ĠIt Ġattaches Ġequal Ġimportance Ġto Ġfalse Ġpositives Ġ( FP s ) Ġand Ġfalse Ġnegatives Ġ( FN s ) Ġand Ġis Ġthus Ġmore Ġimmune Ġto Ġdata - im balanced Ġdatasets . ĠT vers ky Ġindex Ġextends Ġdice Ġloss Ġby Ġusing Ġa Ġweight Ġthat Ġtrades Ġprecision Ġand Ġrecall , Ġwhich Ġcan Ġbe Ġthought Ġas Ġthe Ġapproximation Ġof Ġthe Ġ$ F _{ \ beta Ġ} $ Ġscore , Ġand Ġthus Ġcomes Ġwith Ġmore Ġflexibility . ĠTherefore , ĠWe Ġuse Ġdice Ġloss Ġor ĠT vers ky Ġindex Ġto Ġreplace ĠCE Ġloss Ġto Ġaddress Ġthe Ġfirst Ġissue . </s> Only Ġusing Ġdice Ġloss Ġor ĠT vers ky Ġindex Ġis Ġnot Ġenough Ġsince Ġthey Ġare Ġunable Ġto Ġaddress Ġthe Ġdominating Ġinfluence Ġof Ġeasy - negative Ġexamples . ĠThis Ġis Ġintrinsically Ġbecause Ġdice Ġloss Ġis Ġactually Ġa Ġhard Ġversion Ġof Ġthe ĠF 1 Ġscore . ĠTaking Ġthe Ġbinary Ġclassification Ġtask Ġas Ġan Ġexample , Ġat Ġtest Ġtime , Ġan Ġexample Ġwill Ġbe Ġclassified Ġas Ġnegative Ġas Ġlong Ġas Ġits Ġprobability Ġis Ġsmaller Ġthan Ġ0 . 5 , Ġbut Ġtraining Ġwill Ġpush Ġthe Ġvalue Ġto Ġ0 Ġas Ġmuch Ġas Ġpossible . ĠThis Ġgap Ġisn 't Ġa Ġbig Ġissue Ġfor Ġbalanced Ġdatasets , Ġbut Ġis Ġextremely Ġdetrimental Ġif Ġa Ġbig Ġproportion Ġof Ġtraining Ġexamples Ġare Ġeasy - negative Ġones : Ġeasy - negative Ġexamples Ġcan Ġeasily Ġdominate Ġtraining Ġsince Ġtheir Ġprobabilities Ġcan Ġbe Ġpushed Ġto Ġ0 Ġfairly Ġeasily . ĠMeanwhile , Ġthe Ġmodel Ġcan Ġhardly Ġdistinguish Ġbetween Ġhard - negative Ġexamples Ġand Ġpositive Ġones . ĠInspired Ġby Ġthe Ġidea Ġof Ġfocal Ġloss ĠB IB REF 16 Ġin Ġcomputer Ġvision , Ġwe Ġpropose Ġa Ġdynamic Ġweight Ġadjusting Ġstrategy , Ġwhich Ġassociates Ġeach Ġtraining Ġexample Ġwith Ġa Ġweight Ġin Ġproportion Ġto Ġ$( 1 - p ) $, Ġand Ġthis Ġweight Ġdynamically Ġchanges Ġas Ġtraining Ġproceeds . ĠThis Ġstrategy Ġhelps Ġto Ġdeem phas ize Ġconfident Ġexamples Ġduring Ġtraining Ġas Ġtheir Ġ$ p $ Ġapproaches Ġthe Ġvalue Ġof Ġ1 , Ġmakes Ġthe Ġmodel Ġattentive Ġto Ġhard - negative Ġexamples , Ġand Ġthus Ġallev iates Ġthe Ġdominating Ġeffect Ġof Ġeasy - negative Ġexamples . </s> Comb ing Ġboth Ġstrategies , Ġwe Ġobserve Ġsignificant Ġperformance Ġboosts Ġon Ġa Ġwide Ġrange Ġof Ġdata Ġim balanced ĠN LP Ġtasks . ĠNotably , Ġwe Ġare Ġable Ġto Ġachieve ĠS OTA Ġresults Ġon ĠCT B 5 Ġ( 97 . 92 , Ġ+ 1 . 86 ), ĠCT B 6 Ġ( 96 . 57 , Ġ+ 1 . 80 ) Ġand ĠUD 1 . 4 Ġ( 96 . 98 , Ġ+ 2 . 19 ) Ġfor Ġthe ĠPOS Ġtask ; ĠS OTA Ġresults Ġon ĠCo N LL 03 Ġ( 93 . 33 , Ġ+ 0 . 29 ), ĠOnt o Notes 5 . 0 Ġ( 92 . 07 , Ġ+ 0 . 96 )), ĠMS RA Ġ96 . 72 ( + 0 . 97 ) Ġand ĠOnt o Notes 4 . 0 Ġ( 84 . 47 , + 2 . 36 ) Ġfor Ġthe ĠN ER Ġtask ; Ġalong Ġwith Ġcompetitive Ġresults Ġon Ġthe Ġtasks Ġof Ġmachine Ġreading Ġcomprehension Ġand Ġparaph rase Ġidentification . </s> The Ġrest Ġof Ġthis Ġpaper Ġis Ġorganized Ġas Ġfollows : Ġrelated Ġwork Ġis Ġpresented Ġin ĠSection Ġ2 . ĠWe Ġdescribe Ġdifferent Ġtraining Ġobjectives Ġin ĠSection Ġ3 . ĠExperimental Ġresults Ġare Ġpresented Ġin ĠSection Ġ4 . ĠWe Ġperform Ġab lation Ġstudies Ġin ĠSection Ġ5 , Ġfollowed Ġby Ġa Ġbrief Ġconclusion Ġin ĠSection Ġ6 . </s> Related ĠWork Ġ: :: ĠData ĠRes ample </s> The Ġidea Ġof Ġweight ing Ġtraining Ġexamples Ġhas Ġa Ġlong Ġhistory . ĠImport ance Ġsampling ĠB IB REF 17 Ġassigns Ġweights Ġto Ġdifferent Ġsamples Ġand Ġchanges Ġthe Ġdata Ġdistribution . ĠBoost ing Ġalgorithms Ġsuch Ġas ĠAda Boost ĠB IB REF 18 Ġselect Ġharder Ġexamples Ġto Ġtrain Ġsubsequent Ġclass ifiers . ĠSimilarly , Ġhard Ġexample Ġmining ĠB IB REF 19 Ġdowns amples Ġthe Ġmajority Ġclass Ġand Ġexploits Ġthe Ġmost Ġdifficult Ġexamples . ĠOvers am pling ĠB IB REF 20 , ĠB IB REF 21 Ġis Ġused Ġto Ġbalance Ġthe Ġdata Ġdistribution . ĠAnother Ġline Ġof Ġdata Ġres am pling Ġis Ġto Ġdynamically Ġcontrol Ġthe Ġweights Ġof Ġexamples Ġas Ġtraining Ġproceeds . ĠFor Ġexample , Ġfocal Ġloss ĠB IB REF 16 Ġused Ġa Ġsoft Ġweight ing Ġscheme Ġthat Ġemphasizes Ġharder Ġexamples Ġduring Ġtraining . ĠIn Ġself - paced Ġlearning ĠB IB REF 22 , Ġexample Ġweights Ġare Ġobtained Ġthrough Ġoptimizing Ġthe Ġweighted Ġtraining Ġloss Ġwhich Ġencourages Ġlearning Ġeasier Ġexamples Ġfirst . ĠAt Ġeach Ġtraining Ġstep , Ġself - paced Ġlearning Ġalgorithm Ġoptim izes Ġmodel Ġparameters Ġand Ġexample Ġweights Ġjointly . ĠOther Ġworks ĠB IB REF 23 , ĠB IB REF 24 Ġadjusted Ġthe Ġweights Ġof Ġdifferent Ġtraining Ġexamples Ġbased Ġon Ġtraining Ġloss . ĠBesides , Ġrecent Ġwork ĠB IB REF 25 , ĠB IB REF 26 Ġproposed Ġto Ġlearn Ġa Ġseparate Ġnetwork Ġto Ġpredict Ġsample Ġweights . </s> Related ĠWork Ġ: :: ĠData ĠIm balance ĠIssue Ġin ĠObject ĠDetection </s> The Ġbackground - object Ġlabel Ġimbalance Ġissue Ġis Ġsevere Ġand Ġthus Ġwell Ġstudied Ġin Ġthe Ġfield Ġof Ġobject Ġdetection ĠB IB REF 27 , ĠB IB REF 28 , ĠB IB REF 29 , ĠB IB REF 30 , ĠB IB REF 31 . ĠThe Ġidea Ġof Ġhard Ġnegative Ġmining Ġ( H NM ) ĠB IB REF 30 Ġhas Ġgained Ġmuch Ġattention Ġrecently . Ġsh riv ast ava 2016 o hem Ġproposed Ġthe Ġonline Ġhard Ġexample Ġmining Ġ( OH EM ) Ġalgorithm Ġin Ġan Ġiter ative Ġmanner Ġthat Ġmakes Ġtraining Ġprogressively Ġmore Ġdifficult , Ġand Ġpushes Ġthe Ġmodel Ġto Ġlearn Ġbetter . Ġs sd 2016 li u Ġsorted Ġall Ġof Ġthe Ġnegative Ġsamples Ġbased Ġon Ġthe Ġconfidence Ġloss Ġand Ġpicking Ġthe Ġtraining Ġexamples Ġwith Ġthe Ġnegative - positive Ġratio Ġat Ġ3 : 1 . Ġp ang 2019 rc nn Ġproposed Ġa Ġnovel Ġmethod Ġcalled ĠIo U - balanced Ġsampling Ġand Ġa pl oss 2019 chen Ġdesigned Ġa Ġranking Ġmodel Ġto Ġreplace Ġthe Ġconventional Ġclassification Ġtask Ġwith Ġa Ġaverage - pre cision Ġloss Ġto Ġalleviate Ġthe Ġclass Ġimbalance Ġissue . ĠThe Ġefforts Ġmade Ġon Ġobject Ġdetection Ġhave Ġgreatly Ġinspired Ġus Ġto Ġsolve Ġthe Ġdata Ġimbalance Ġissue Ġin ĠN LP . </s> L oss es Ġ: :: ĠNot ation </s> For Ġillustration Ġpurposes , Ġwe Ġuse Ġthe Ġbinary Ġclassification Ġtask Ġto Ġdemonstrate Ġhow Ġdifferent Ġlosses Ġwork . ĠThe Ġmechanism Ġcan Ġbe Ġeasily Ġextended Ġto Ġmulti - class Ġclassification . </s> Let Ġ$\ l brace Ġx _ i \ r brace Ġ$ Ġdenote Ġa Ġset Ġof Ġinstances . ĠEach Ġ$ x _ i $ Ġis Ġassociated Ġwith Ġa Ġgolden Ġlabel Ġvector Ġ$ y _ i Ġ= Ġ[ y _{ i 0 }, y _{ i 1 } Ġ] $, Ġwhere Ġ$ y _{ i 1 }\ in Ġ\ l brace Ġ0 , 1 \ r brace Ġ$ Ġand Ġ$ y _{ i 0 }\ in Ġ\ l brace Ġ0 , 1 \ r brace Ġ$ Ġrespectively Ġdenote Ġthe Ġpositive Ġand Ġnegative Ġclasses , Ġand Ġthus Ġ$ y _ i $ Ġcan Ġbe Ġeither Ġ$ [ 0 , 1 ] $ Ġor Ġ$ [ 0 , 1 ] $. ĠLet Ġ$ p _ i Ġ= Ġ[ p _{ i 0 }, p _{ i 1 } Ġ] $ Ġdenote Ġthe Ġprobability Ġvector , Ġand Ġ$ p _{ i 1 } $ Ġand Ġ$ p _{ i 0 } $ Ġrespectively Ġdenote Ġthe Ġprobability Ġthat Ġa Ġmodel Ġassigns Ġthe Ġpositive Ġand Ġnegative Ġlabel Ġto Ġ$ x _ i $. </s> L oss es Ġ: :: ĠCross ĠEnt ropy ĠLoss </s> The Ġvanilla Ġcross Ġentropy Ġ( CE ) Ġloss Ġis Ġgiven Ġby : </s> As Ġcan Ġbe Ġseen Ġfrom ĠE q . DIS PLAY _ FORM 8 , Ġeach Ġ$ x _ i $ Ġcontributes Ġequally Ġto Ġthe Ġfinal Ġobjective . ĠTwo Ġstrategies Ġare Ġnormally Ġused Ġto Ġaddress Ġthe Ġthe Ġcase Ġwhere Ġwe Ġwish Ġthat Ġnot Ġall Ġ$ x _ i $ Ġare Ġtreated Ġequal : Ġassoci ating Ġdifferent Ġclasses Ġwith Ġdifferent Ġweight ing Ġfactor Ġ$\ alpha Ġ$ Ġor Ġres am pling Ġthe Ġdatasets . ĠFor Ġthe Ġformer , ĠE q . DIS PLAY _ FORM 8 Ġis Ġadjusted Ġas Ġfollows : </s> where Ġ$\ alpha Ġ_ i \ in Ġ[ 0 , 1 ] $ Ġmay Ġbe Ġset Ġby Ġthe Ġinverse Ġclass Ġfrequency Ġor Ġtreated Ġas Ġa Ġhyper param eter Ġto Ġset Ġby Ġcross Ġvalidation . ĠIn Ġthis Ġwork , Ġwe Ġuse Ġ$\ l g Ġ( \ frac { n - n _ t }{ n _ t } + K ) $ Ġto Ġcalculate Ġthe Ġcoefficient Ġ$\ alpha Ġ$ , Ġwhere Ġ$ n _ t $ Ġis Ġthe Ġnumber Ġof Ġsamples Ġwith Ġclass Ġ$ t $ Ġand Ġ$ n $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġsamples Ġin Ġthe Ġtraining Ġset . Ġ$ K $ Ġis Ġa Ġhyper param eter Ġto Ġtune . ĠThe Ġdata Ġres am pling Ġstrategy Ġconstructs Ġa Ġnew Ġdataset Ġby Ġsampling Ġtraining Ġexamples Ġfrom Ġthe Ġoriginal Ġdataset Ġbased Ġon Ġhuman - designed Ġcriteria , Ġe . g ., Ġextract Ġequal Ġtraining Ġsamples Ġfrom Ġeach Ġclass . ĠBoth Ġstrategies Ġare Ġequivalent Ġto Ġchanging Ġthe Ġdata Ġdistribution Ġand Ġthus Ġare Ġof Ġthe Ġsame Ġnature . ĠEm pir ically , Ġthese Ġtwo Ġmethods Ġare Ġnot Ġwidely Ġused Ġdue Ġto Ġthe Ġtrick iness Ġof Ġselecting Ġ$\ alpha Ġ$ Ġespecially Ġfor Ġmulti - class Ġclassification Ġtasks Ġand Ġthat Ġinappropriate Ġselection Ġcan Ġeasily Ġbias Ġtowards Ġrare Ġclasses ĠB IB REF 32 . </s> L oss es Ġ: :: ĠDice Ġcoefficient Ġand ĠT vers ky Ġindex </s> S Ã¸ ren sen âĢĵ D ice Ġcoefficient ĠB IB REF 0 , ĠB IB REF 33 , Ġdice Ġcoefficient Ġ( DS C ) Ġfor Ġshort , Ġis Ġa ĠF 1 - oriented Ġstatistic Ġused Ġto Ġgauge Ġthe Ġsimilarity Ġof Ġtwo Ġsets . ĠGiven Ġtwo Ġsets Ġ$ A $ Ġand Ġ$ B $, Ġthe Ġdice Ġcoefficient Ġbetween Ġthem Ġis Ġgiven Ġas Ġfollows : </s> In Ġour Ġcase , Ġ$ A $ Ġis Ġthe Ġset Ġthat Ġcontains Ġof Ġall Ġpositive Ġexamples Ġpredicted Ġby Ġa Ġspecific Ġmodel , Ġand Ġ$ B $ Ġis Ġthe Ġset Ġof Ġall Ġgolden Ġpositive Ġexamples Ġin Ġthe Ġdataset . ĠWhen Ġapplied Ġto Ġboolean Ġdata Ġwith Ġthe Ġdefinition Ġof Ġtrue Ġpositive Ġ( TP ), Ġfalse Ġpositive Ġ( FP ), Ġand Ġfalse Ġnegative Ġ( FN ), Ġit Ġcan Ġbe Ġthen Ġwritten Ġas Ġfollows : </s> For Ġan Ġindividual Ġexample Ġ$ x _ i $, Ġits Ġcorresponding ĠD SC Ġloss Ġis Ġgiven Ġas Ġfollows : </s> As Ġcan Ġbe Ġseen , Ġfor Ġa Ġnegative Ġexample Ġwith Ġ$ y _{ i 1 } = 0 $, Ġit Ġdoes Ġnot Ġcontribute Ġto Ġthe Ġobjective . ĠFor Ġsmoot hing Ġpurposes , Ġit Ġis Ġcommon Ġto Ġadd Ġa Ġ$\ gam ma Ġ$ Ġfactor Ġto Ġboth Ġthe Ġnom inator Ġand Ġthe Ġdenomin ator , Ġmaking Ġthe Ġform Ġto Ġbe Ġas Ġfollows : </s> As Ġcan Ġbe Ġseen , Ġnegative Ġexamples , Ġwith Ġ$ y _{ i 1 } $ Ġbeing Ġ0 Ġand ĠD SC Ġbeing Ġ$\ frac {\ gam ma Ġ} { Ġp _{ i 1 } + \ gam ma Ġ} $, Ġalso Ġcontribute Ġto Ġthe Ġtraining . ĠAdditionally , Ġmil let ari 2016 v Ġproposed Ġto Ġchange Ġthe Ġdenomin ator Ġto Ġthe Ġsquare Ġform Ġfor Ġfaster Ġconvergence , Ġwhich Ġleads Ġto Ġthe Ġfollowing Ġdice Ġloss Ġ( DL ): </s> Another Ġversion Ġof ĠDL Ġis Ġto Ġdirectly Ġcompute Ġset - level Ġdice Ġcoefficient Ġinstead Ġof Ġthe Ġsum Ġof Ġindividual Ġdice Ġcoefficient . ĠWe Ġchoose Ġthe Ġlatter Ġdue Ġto Ġease Ġof Ġoptimization . </s> T vers ky Ġindex Ġ( TI ), Ġwhich Ġcan Ġbe Ġthought Ġas Ġthe Ġapproximation Ġof Ġthe Ġ$ F _{ \ beta Ġ} $ Ġscore , Ġextends Ġdice Ġcoefficient Ġto Ġa Ġmore Ġgeneral Ġcase . ĠGiven Ġtwo Ġsets Ġ$ A $ Ġand Ġ$ B $, Ġt vers ky Ġindex Ġis Ġcomputed Ġas Ġfollows : </s> T vers ky Ġindex Ġoffers Ġthe Ġflexibility Ġin Ġcontrolling Ġthe Ġtrade off Ġbetween Ġfalse - neg atives Ġand Ġfalse - pos itives . ĠIt Ġdegener ates Ġto ĠD SC Ġif Ġ$\ alpha Ġ= \ beta Ġ= 0 . 5 $. ĠThe ĠT vers ky Ġloss Ġ( TL ) Ġfor Ġthe Ġtraining Ġset Ġ$\ l brace Ġx _ i , y _ i \ r brace Ġ$ Ġis Ġthus Ġas Ġfollows : </s> L oss es Ġ: :: ĠSelf - ad usting ĠDice ĠLoss </s> Consider Ġa Ġsimple Ġcase Ġwhere Ġthe Ġdataset Ġconsists Ġof Ġonly Ġone Ġexample Ġ$ x _ i $, Ġwhich Ġis Ġclassified Ġas Ġpositive Ġas Ġlong Ġas Ġ$ p _{ i 1 } $ Ġis Ġlarger Ġthan Ġ0 . 5 . ĠThe Ġcomputation Ġof Ġ$ F 1 $ Ġscore Ġis Ġactually Ġas Ġfollows : </s> Comp aring ĠE q . DIS PLAY _ FORM 14 Ġwith ĠE q . DIS PLAY _ FORM 22 , Ġwe Ġcan Ġsee Ġthat ĠE q . DIS PLAY _ FORM 14 Ġis Ġactually Ġa Ġsoft Ġform Ġof Ġ$ F 1 $, Ġusing Ġa Ġcontinuous Ġ$ p $ Ġrather Ġthan Ġthe Ġbinary Ġ$\ math bb Ġ{ I } ( Ġp _{ i 1 } > 0 . 5 ) $. ĠThis Ġgap Ġisn 't Ġa Ġbig Ġissue Ġfor Ġbalanced Ġdatasets , Ġbut Ġis Ġextremely Ġdetrimental Ġif Ġa Ġbig Ġproportion Ġof Ġtraining Ġexamples Ġare Ġeasy - negative Ġones : Ġeasy - negative Ġexamples Ġcan Ġeasily Ġdominate Ġtraining Ġsince Ġtheir Ġprobabilities Ġcan Ġbe Ġpushed Ġto Ġ0 Ġfairly Ġeasily . ĠMeanwhile , Ġthe Ġmodel Ġcan Ġhardly Ġdistinguish Ġbetween Ġhard - negative Ġexamples Ġand Ġpositive Ġones , Ġwhich Ġhas Ġa Ġhuge Ġnegative Ġeffect Ġon Ġthe Ġfinal ĠF 1 Ġperformance . </s> To Ġaddress Ġthis Ġissue , Ġwe Ġpropose Ġto Ġmultiply Ġthe Ġsoft Ġprobability Ġ$ p $ Ġwith Ġa Ġdecaying Ġfactor Ġ$( 1 - p ) $, Ġchanging ĠE q . DIS PLAY _ FORM 22 Ġto Ġthe Ġfollowing Ġform : </s> One Ġcan Ġthink Ġ$( 1 - p _{ i 1 }) $ Ġas Ġa Ġweight Ġassociated Ġwith Ġeach Ġexample , Ġwhich Ġchanges Ġas Ġtraining Ġproceeds . ĠThe Ġintuition Ġof Ġchanging Ġ$ p _{ i 1 } $ Ġto Ġ$( 1 - p _{ i 1 }) Ġp _{ i 1 } $ Ġis Ġto Ġpush Ġdown Ġthe Ġweight Ġof Ġeasy Ġexamples . ĠFor Ġeasy Ġexamples Ġwhose Ġprobability Ġare Ġapproaching Ġ0 Ġor Ġ1 , Ġ$( 1 - p _{ i 1 }) Ġp _{ i 1 } $ Ġmakes Ġthe Ġmodel Ġattach Ġsignificantly Ġless Ġfocus Ġto Ġthem . ĠFigure ĠFIG REF 23 Ġgives Ġgives Ġan Ġexplanation Ġfrom Ġthe Ġperspective Ġin Ġderivative : Ġthe Ġderivative Ġof Ġ$\ frac { ( 1 - p ) p }{ 1 +( 1 - p ) p } $ Ġwith Ġrespect Ġto Ġ$ p $ Ġapproaches Ġ0 Ġimmediately Ġafter Ġ$ p $ Ġapproaches Ġ0 , Ġwhich Ġmeans Ġthe Ġmodel Ġattends Ġless Ġto Ġexamples Ġonce Ġthey Ġare Ġcorrectly Ġclassified . </s> A Ġclose Ġlook Ġat ĠE q . DIS PLAY _ FORM 14 Ġreveals Ġthat Ġit Ġactually Ġmim ics Ġthe Ġidea Ġof Ġfocal Ġloss Ġ( FL Ġfor Ġshort ) ĠB IB REF 16 Ġfor Ġobject Ġdetection Ġin Ġvision . ĠF ocal Ġloss Ġwas Ġproposed Ġfor Ġone - stage Ġobject Ġdetector Ġto Ġhandle Ġforeground - background Ġtrade off Ġencountered Ġduring Ġtraining . ĠIt Ġdown - weights Ġthe Ġloss Ġassigned Ġto Ġwell - classified Ġexamples Ġby Ġadding Ġa Ġ$( 1 - p ) ^ {\ beta Ġ} $ Ġfactor , Ġleading Ġthe Ġfinal Ġloss Ġto Ġbe Ġ$( 1 - p ) ^ {\ beta Ġ} \ log Ġp $. </s> In ĠTable ĠT AB REF 18 , Ġwe Ġshow Ġthe Ġlosses Ġused Ġin Ġour Ġexperiments , Ġwhich Ġis Ġdescribed Ġin Ġthe Ġnext Ġsection . </s> Exper iments </s> We Ġevaluate Ġthe Ġproposed Ġmethod Ġon Ġfour ĠN LP Ġtasks : Ġpart - of - speech Ġtagging , Ġnamed Ġentity Ġrecognition , Ġmachine Ġreading Ġcomprehension Ġand Ġparaph rase Ġidentification . ĠBas elines Ġin Ġour Ġexperiments Ġare Ġoptimized Ġby Ġusing Ġthe Ġstandard Ġcross - ent ropy Ġtraining Ġobjective . </s> Exper iments Ġ: :: ĠPart - of - Spe ech ĠT agging </s> Part - of - speech Ġtagging Ġ( POS ) Ġis Ġthe Ġtask Ġof Ġassigning Ġa Ġlabel Ġ( e . g ., Ġnoun , Ġverb , Ġadjective ) Ġto Ġeach Ġword Ġin Ġa Ġgiven Ġtext . ĠIn Ġthis Ġpaper , Ġwe Ġchoose ĠB ERT Ġas Ġthe Ġbackbone Ġand Ġconduct Ġexperiments Ġon Ġthree ĠChinese ĠPOS Ġdatasets . ĠWe Ġreport Ġthe Ġspan - level Ġmicro - aver aged Ġprecision , Ġrecall Ġand ĠF 1 Ġfor Ġevaluation . ĠHyper param eters Ġare Ġtuned Ġon Ġthe Ġcorresponding Ġdevelopment Ġset Ġof Ġeach Ġdataset . </s> Exper iments Ġ: :: ĠPart - of - Spe ech ĠT agging Ġ: :: ĠDat as ets </s> We Ġconduct Ġexperiments Ġon Ġthe Ġwidely Ġused ĠChinese ĠTree bank Ġ5 . 0 , Ġ6 . 0 Ġas Ġwell Ġas ĠUD 1 . 4 . </s> CT B 5 Ġis Ġa ĠChinese Ġdataset Ġfor Ġtagging Ġand Ġparsing , Ġwhich Ġcontains Ġ50 7 , 222 Ġwords , Ġ8 24 , 98 3 Ġcharacters Ġand Ġ18 , 782 Ġsentences Ġextracted Ġfrom Ġnews wire Ġsources . </s> CT B 6 Ġis Ġan Ġextension Ġof ĠCT B 5 , Ġcontaining Ġ7 81 , 351 Ġwords , Ġ1 , 285 , 149 Ġcharacters Ġand Ġ28 , 295 Ġsentences . </s> UD Ġis Ġthe Ġabbre viation Ġof ĠUniversal ĠDepend encies , Ġwhich Ġis Ġa Ġframework Ġfor Ġconsistent Ġannotation Ġof Ġgrammar Ġ( parts Ġof Ġspeech , Ġmorph ological Ġfeatures , Ġand Ġsynt actic Ġdependencies ) Ġacross Ġdifferent Ġhuman Ġlanguages . ĠIn Ġthis Ġwork , Ġwe Ġuse ĠUD 1 . 4 Ġfor ĠChinese ĠPOS Ġtagging . </s> Exper iments Ġ: :: ĠPart - of - Spe ech ĠT agging Ġ: :: ĠBas elines </s> We Ġuse Ġthe Ġfollowing Ġbas elines : </s> J oint - POS : Ġsh ao 2017 character Ġjointly Ġlearns ĠChinese Ġword Ġsegment ation Ġand ĠPOS . </s> L att ice - L ST M : Ġlatt ice 2018 zh ang Ġconstructs Ġa Ġword - character Ġlatt ice . </s> B ert - T agger : Ġdev lin 2018 bert Ġtreats Ġpart - of - speech Ġas Ġa Ġtagging Ġtask . </s> Exper iments Ġ: :: ĠPart - of - Spe ech ĠT agging Ġ: :: ĠResults </s> Table Ġpresents Ġthe Ġexperimental Ġresults Ġon Ġthe ĠPOS Ġtask . ĠAs Ġcan Ġbe Ġseen , Ġthe Ġproposed ĠD SC Ġloss Ġoutper forms Ġthe Ġbest Ġbaseline Ġresults Ġby Ġa Ġlarge Ġmargin , Ġi . e ., Ġoutper forming ĠB ERT - t agger Ġby Ġ+ 1 . 86 Ġin Ġterms Ġof ĠF 1 Ġscore Ġon ĠCT B 5 , Ġ+ 1 . 80 Ġon ĠCT B 6 Ġand Ġ+ 2 . 19 Ġon ĠUD 1 . 4 . ĠAs Ġfar Ġas Ġwe Ġare Ġconcerned , Ġwe Ġare Ġachieving ĠS OTA Ġperformances Ġon Ġthe Ġthree Ġdatasets . ĠWeight ed Ġcross Ġentropy Ġand Ġfocal Ġloss Ġonly Ġgain Ġa Ġlittle Ġperformance Ġimprovement Ġon ĠCT B 5 Ġand ĠCT B 6 , Ġand Ġthe Ġdice Ġloss Ġob tains Ġhuge Ġgain Ġon ĠCT B 5 Ġbut Ġnot Ġon ĠCT B 6 , Ġwhich Ġindicates Ġthe Ġthree Ġlosses Ġare Ġnot Ġconsistently Ġrobust Ġin Ġresolving Ġthe Ġdata Ġimbalance Ġissue . ĠThe Ġproposed ĠD SC Ġloss Ġperforms Ġrobust ly Ġon Ġall Ġthe Ġthree Ġdatasets . </s> Exper iments Ġ: :: ĠNamed ĠEntity ĠRecogn ition </s> N amed Ġentity Ġrecognition Ġ( NER ) Ġrefers Ġto Ġthe Ġtask Ġof Ġdetecting Ġthe Ġspan Ġand Ġsemantic Ġcategory Ġof Ġentities Ġfrom Ġa Ġchunk Ġof Ġtext . ĠOur Ġimplementation Ġuses Ġthe Ġcurrent Ġstate - of - the - art ĠB ERT - M RC Ġmodel Ġproposed Ġby Ġx ia oya 2019 ner Ġas Ġa Ġbackbone . ĠFor ĠEnglish Ġdatasets , Ġwe Ġuse ĠB ERT $ _ \ text { Large } $ ĠEnglish Ġcheckpoints , Ġwhile Ġfor ĠChinese Ġwe Ġuse Ġthe Ġofficial ĠChinese Ġcheckpoints . ĠWe Ġreport Ġspan - level Ġmicro - aver aged Ġprecision , Ġrecall Ġand ĠF 1 - score . ĠHyper param eters Ġare Ġtuned Ġon Ġthe Ġdevelopment Ġset Ġof Ġeach Ġdataset . </s> Exper iments Ġ: :: ĠNamed ĠEntity ĠRecogn ition Ġ: :: ĠDat as ets </s> For Ġthe ĠN ER Ġtask , Ġwe Ġconsider Ġboth ĠChinese Ġdatasets , Ġi . e ., ĠOnt o Notes 4 . 0 ĠB IB REF 34 Ġand ĠMS RA ĠB IB REF 35 , Ġand ĠEnglish Ġdatasets , Ġi . e ., ĠCo N LL 2003 ĠB IB REF 36 Ġand ĠOnt o Notes 5 . 0 ĠB IB REF 37 . </s> Co N LL 2003 Ġis Ġan ĠEnglish Ġdataset Ġwith Ġ4 Ġentity Ġtypes : ĠLocation , ĠOrganization , ĠPerson Ġand ĠMiscellaneous . ĠWe Ġfollowed Ġdata Ġprocessing Ġprotocols Ġin ĠB IB REF 14 . </s> English ĠOnt o Notes 5 . 0 Ġconsists Ġof Ġtexts Ġfrom Ġa Ġwide Ġvariety Ġof Ġsources Ġand Ġcontains Ġ18 Ġentity Ġtypes . ĠWe Ġuse Ġthe Ġstandard Ġtrain / dev / test Ġsplit Ġof ĠCo N LL 2012 Ġshared Ġtask . </s> Chinese ĠMS RA Ġperforms Ġas Ġa ĠChinese Ġbenchmark Ġdataset Ġcontaining Ġ3 Ġentity Ġtypes . ĠData Ġin ĠMS RA Ġis Ġcollected Ġfrom Ġnews Ġdomain . ĠSince Ġthe Ġdevelopment Ġset Ġis Ġnot Ġprovided Ġin Ġthe Ġoriginal ĠMS RA Ġdataset , Ġwe Ġrandomly Ġsplit Ġthe Ġtraining Ġset Ġinto Ġtraining Ġand Ġdevelopment Ġsplits Ġby Ġ9 : 1 . ĠWe Ġuse Ġthe Ġofficial Ġtest Ġset Ġfor Ġevaluation . </s> Chinese ĠOnt o Notes 4 . 0 Ġis Ġa ĠChinese Ġdataset Ġand Ġconsists Ġof Ġtexts Ġfrom Ġnews Ġdomain , Ġwhich Ġhas Ġ18 Ġentity Ġtypes . ĠIn Ġthis Ġpaper , Ġwe Ġtake Ġthe Ġsame Ġdata Ġsplit Ġas Ġw u 2019 gly ce Ġdid . </s> Exper iments Ġ: :: ĠNamed ĠEntity ĠRecogn ition Ġ: :: ĠBas elines </s> We Ġuse Ġthe Ġfollowing Ġbas elines : </s> EL Mo : Ġa Ġtagging Ġmodel Ġfrom Ġpet ers 2018 deep . </s> L att ice - L ST M : Ġlatt ice 2018 zh ang Ġconstructs Ġa Ġword - character Ġlatt ice , Ġonly Ġused Ġin ĠChinese Ġdatasets . </s> CV T : Ġfrom Ġke vin 2018 cross , Ġwhich Ġuses ĠCross - View ĠTraining ( CV T ) Ġto Ġimprove Ġthe Ġrepresentations Ġof Ġa ĠBi - L ST M Ġenc oder . </s> B ert - T agger : Ġdev lin 2018 bert Ġtreats ĠN ER Ġas Ġa Ġtagging Ġtask . </s> G ly ce - BER T : Ġw u 2019 gly ce Ġcombines Ġglyph Ġinformation Ġwith ĠB ERT Ġpret raining . </s> BER T - M RC : ĠThe Ġcurrent ĠS OTA Ġmodel Ġfor Ġboth ĠChinese Ġand ĠEnglish ĠN ER Ġdatasets Ġproposed Ġby Ġx ia oya 2019 ner , Ġwhich Ġformulate ĠN ER Ġas Ġmachine Ġreading Ġcomprehension Ġtask . </s> Exper iments Ġ: :: ĠNamed ĠEntity ĠRecogn ition Ġ: :: ĠResults </s> Table Ġshows Ġexperimental Ġresults Ġon ĠN ER Ġdatasets . ĠFor ĠEnglish Ġdatasets Ġincluding ĠCo N LL 2003 Ġand ĠOnt o Notes 5 . 0 , Ġour Ġproposed Ġmethod Ġoutper forms ĠB ERT - M RC BI B REF 38 Ġby Ġ+ 0 . 29 Ġand Ġ+ 0 . 96 Ġrespectively . ĠWe Ġobserve Ġhuge Ġperformance Ġboosts Ġon ĠChinese Ġdatasets , Ġachieving ĠF 1 Ġimprovements Ġby Ġ+ 0 . 97 Ġand Ġ+ 2 . 36 Ġon ĠMS RA Ġand ĠOnt o Notes 4 . 0 , Ġrespectively . ĠAs Ġfar Ġas Ġwe Ġare Ġconcerned , Ġwe Ġare Ġsetting Ġnew ĠS OTA Ġperformances Ġon Ġall Ġof Ġthe Ġfour ĠN ER Ġdatasets . </s> Exper iments Ġ: :: ĠMachine ĠReading ĠComp rehens ion </s> Machine Ġreading Ġcomprehension Ġ( M RC ) ĠB IB REF 39 , ĠB IB REF 40 , ĠB IB REF 41 , ĠB IB REF 40 , ĠB IB REF 42 , ĠB IB REF 15 Ġhas Ġbecome Ġa Ġcentral Ġtask Ġin Ġnatural Ġlanguage Ġunderstanding . ĠM RC Ġin Ġthe ĠS Qu AD - style Ġis Ġto Ġpredict Ġthe Ġanswer Ġspan Ġin Ġthe Ġpassage Ġgiven Ġa Ġquestion Ġand Ġthe Ġpassage . ĠIn Ġthis Ġpaper , Ġwe Ġchoose Ġthe ĠS Qu AD - style ĠM RC Ġtask Ġand Ġreport ĠExtract ĠMatch Ġ( EM ) Ġin Ġaddition Ġto ĠF 1 Ġscore Ġon Ġvalidation Ġset . ĠAll Ġhyper param eters Ġare Ġtuned Ġon Ġthe Ġdevelopment Ġset Ġof Ġeach Ġdataset . </s> Exper iments Ġ: :: ĠMachine ĠReading ĠComp rehens ion Ġ: :: ĠDat as ets </s> The Ġfollowing Ġfive Ġdatasets Ġare Ġused Ġfor ĠM RC Ġtask : ĠS Qu AD Ġv 1 . 1 , ĠS Qu AD Ġv 2 . 0 ĠB IB REF 4 , ĠB IB REF 6 Ġand ĠQu ore f ĠB IB REF 8 . </s> S Qu AD Ġv 1 . 1 Ġand ĠS Qu AD Ġv 2 . 0 Ġare Ġthe Ġmost Ġwidely Ġused ĠQ A Ġbenchmarks . ĠS Qu AD 1 . 1 Ġis Ġa Ġcollection Ġof Ġ100 K Ġcrowds ourced Ġquestion - answer Ġpairs , Ġand ĠS Qu AD 2 . 0 Ġextends ĠS Qu AD 1 . 1 Ġallowing Ġno Ġshort Ġanswer Ġexists Ġin Ġthe Ġprovided Ġpassage . </s> Qu ore f Ġis Ġa ĠQ A Ġdataset Ġwhich Ġtests Ġthe Ġcore f erential Ġreasoning Ġcapability Ġof Ġreading Ġcomprehension Ġsystems , Ġcontaining Ġ24 K Ġquestions Ġover Ġ4 . 7 K Ġparagraphs Ġfrom ĠWikipedia . </s> Exper iments Ġ: :: ĠMachine ĠReading ĠComp rehens ion Ġ: :: ĠBas elines </s> We Ġuse Ġthe Ġfollowing Ġbas elines : </s> Q AN et : Ġq an et 2018 Ġbuilds Ġa Ġmodel Ġbased Ġon Ġconv olutions Ġand Ġself - att ention . ĠConv olution Ġto Ġmodel Ġlocal Ġinteractions Ġand Ġself - att ention Ġto Ġmodel Ġglobal Ġinteractions . </s> BER T : Ġdev lin 2018 bert Ġtreats ĠN ER Ġas Ġa Ġtagging Ġtask . </s> XL Net : Ġx l net 2019 Ġproposes Ġa Ġgeneralized Ġaut ore gressive Ġpret raining Ġmethod Ġthat Ġenables Ġlearning Ġbid irection al Ġcontexts . </s> Exper iments Ġ: :: ĠMachine ĠReading ĠComp rehens ion Ġ: :: ĠResults </s> Table Ġshows Ġthe Ġexperimental Ġresults Ġfor ĠM RC Ġtasks . ĠWith Ġeither ĠB ERT Ġor ĠXL Net , Ġour Ġproposed ĠD SC Ġloss Ġob tains Ġsignificant Ġperformance Ġboost Ġon Ġboth ĠEM Ġand ĠF 1 . ĠFor ĠS Qu AD v 1 . 1 , Ġour Ġproposed Ġmethod Ġoutper forms ĠXL Net Ġby Ġ+ 1 . 25 Ġin Ġterms Ġof ĠF 1 Ġscore Ġand Ġ+ 0 . 84 Ġin Ġterms Ġof ĠEM Ġand Ġachieves Ġ87 . 65 Ġon ĠEM Ġand Ġ89 . 51 Ġon ĠF 1 Ġfor ĠS Qu AD Ġv 2 . 0 . ĠMoreover , Ġon ĠQu o Ref , Ġthe Ġproposed Ġmethod Ġsurpass es ĠXL Net Ġresults Ġby Ġ+ 1 . 46 Ġon ĠEM Ġand Ġ+ 1 . 41 Ġon ĠF 1 . ĠAnother Ġobservation Ġis Ġthat , ĠXL Net Ġoutper forms ĠB ERT Ġby Ġa Ġhuge Ġmargin , Ġand Ġthe Ġproposed ĠD SC Ġloss Ġcan Ġobtain Ġfurther Ġperformance Ġimprovement Ġby Ġan Ġaverage Ġscore Ġabove Ġ1 . 0 Ġin Ġterms Ġof Ġboth ĠEM Ġand ĠF 1 , Ġwhich Ġindicates Ġthe ĠD SC Ġloss Ġis Ġcomplementary Ġto Ġthe Ġmodel Ġstructures . </s> Exper iments Ġ: :: ĠPar aph rase ĠIdentification </s> Par aph r ases Ġare Ġtextual Ġexpressions Ġthat Ġhave Ġthe Ġsame Ġsemantic Ġmeaning Ġusing Ġdifferent Ġsurface Ġwords . ĠPar aph rase Ġidentification Ġ( PI ) Ġis Ġthe Ġtask Ġof Ġidentifying Ġwhether Ġtwo Ġsentences Ġhave Ġthe Ġsame Ġmeaning Ġor Ġnot . ĠWe Ġuse ĠB ERT ĠB IB REF 11 Ġand ĠXL Net ĠB IB REF 43 Ġas Ġback bones Ġand Ġreport ĠF 1 Ġscore Ġfor Ġcomparison . ĠHyper param eters Ġare Ġtuned Ġon Ġthe Ġdevelopment Ġset Ġof Ġeach Ġdataset . </s> Exper iments Ġ: :: ĠPar aph rase ĠIdentification Ġ: :: ĠDat as ets </s> We Ġconduct Ġexperiments Ġon Ġtwo Ġwidely Ġused Ġdatasets Ġfor ĠPI Ġtask : ĠMR PC ĠB IB REF 44 Ġand ĠQ Q P . </s> MR PC Ġis Ġa Ġcorpus Ġof Ġsentence Ġpairs Ġautomatically Ġextracted Ġfrom Ġonline Ġnews Ġsources , Ġwith Ġhuman Ġannotations Ġof Ġwhether Ġthe Ġsentence Ġpairs Ġare Ġsem antically Ġequivalent . ĠThe ĠMR PC Ġdataset Ġhas Ġim balanced Ġclasses Ġ( 68 % Ġpositive , Ġ32 % Ġfor Ġnegative ). </s> Q Q P Ġis Ġa Ġcollection Ġof Ġquestion Ġpairs Ġfrom Ġthe Ġcommunity Ġquestion - ans w ering Ġwebsite ĠQu ora . ĠThe Ġclass Ġdistribution Ġin ĠQ Q P Ġis Ġalso Ġun balanced Ġ( 37 % Ġpositive , Ġ63 % Ġnegative ). </s> Exper iments Ġ: :: ĠPar aph rase ĠIdentification Ġ: :: ĠResults </s> Table Ġshows Ġthe Ġresults Ġfor ĠPI Ġtask . ĠWe Ġfind Ġthat Ġreplacing Ġthe Ġtraining Ġobjective Ġwith ĠD SC Ġintroduces Ġperformance Ġboost Ġfor Ġboth ĠB ERT Ġand ĠXL Net . ĠUsing ĠD SC Ġloss Ġimproves Ġthe ĠF 1 Ġscore Ġby Ġ+ 0 . 58 Ġfor ĠMR PC Ġand Ġ+ 0 . 73 Ġfor ĠQ Q P . </s> A bl ation ĠStudies Ġ: :: ĠThe ĠEffect Ġof ĠDice ĠLoss Ġon ĠAccuracy - oriented ĠT asks </s> We Ġargue Ġthat Ġthe Ġmost Ġcommonly Ġused Ġcross - ent ropy Ġobjective Ġis Ġactually Ġaccuracy - oriented , Ġwhereas Ġthe Ġproposed Ġdice Ġloss Ġ( DL ) Ġperforms Ġas Ġa Ġhard Ġversion Ġof ĠF 1 - score . ĠTo Ġexplore Ġthe Ġeffect Ġof Ġthe Ġdice Ġloss Ġon Ġaccuracy - oriented Ġtasks Ġsuch Ġas Ġtext Ġclassification , Ġwe Ġconduct Ġexperiments Ġon Ġthe ĠStanford ĠSent iment ĠTree bank Ġsentiment Ġclassification Ġdatasets Ġincluding ĠS ST - 2 Ġand ĠS ST - 5 . ĠWe Ġfine - t une ĠB ERT $ _ \ text { Large } $ Ġwith Ġdifferent Ġtraining Ġobjectives . ĠExperiment Ġresults Ġfor ĠS ST Ġare Ġshown Ġin Ġ. ĠFor ĠS ST - 5 , ĠB ERT Ġwith ĠCE Ġachieves Ġ55 . 57 Ġin Ġterms Ġof Ġaccuracy , Ġwith ĠDL Ġand ĠD SC Ġlosses Ġslightly Ġdegrade Ġthe Ġaccuracy Ġperformance Ġand Ġachieve Ġ54 . 63 Ġand Ġ55 . 19 , Ġrespectively . ĠFor ĠS ST - 2 , ĠB ERT Ġwith ĠCE Ġachieves Ġ94 . 9 Ġin Ġterms Ġof Ġaccuracy . ĠThe Ġsame Ġas ĠS ST - 5 , Ġwe Ġobserve Ġa Ġslight Ġperformance Ġdrop Ġwith ĠDL Ġand ĠD SC , Ġwhich Ġmeans Ġthat Ġthe Ġdice Ġloss Ġactually Ġworks Ġwell Ġfor ĠF 1 Ġbut Ġnot Ġfor Ġaccuracy . </s> A bl ation ĠStudies Ġ: :: ĠThe ĠEffect Ġof ĠHyper param eters Ġin ĠT vers ky Ġindex </s> As Ġmentioned Ġin ĠSection ĠSEC REF 10 , ĠT vers ky Ġindex Ġ( TI ) Ġoffers Ġthe Ġflexibility Ġin Ġcontrolling Ġthe Ġtrade off Ġbetween Ġfalse - neg atives Ġand Ġfalse - pos itives . ĠIn Ġthis Ġsubsection , Ġwe Ġexplore Ġthe Ġeffect Ġof Ġhyper param eters Ġ( i . e ., Ġ$\ alpha Ġ$ Ġand Ġ$\ beta Ġ$ ) Ġin ĠTI Ġto Ġtest Ġhow Ġthey Ġmanipulate Ġthe Ġtrade off . ĠWe Ġconduct Ġexperiments Ġon Ġthe ĠChinese ĠOnt o Notes 4 . 0 ĠN ER Ġdataset Ġand ĠEnglish ĠQu o Ref ĠM RC Ġdataset Ġto Ġexamine Ġthe Ġinfluence Ġof Ġtrade off Ġbetween Ġprecision Ġand Ġrecall . ĠExperiment Ġresults Ġare Ġshown Ġin ĠTable Ġ. ĠThe Ġhighest ĠF 1 Ġfor ĠChinese ĠOnt o Notes 4 . 0 Ġis Ġ84 . 67 Ġwhen Ġ$\ alpha Ġ$ Ġis Ġset Ġto Ġ0 . 6 Ġwhile Ġfor ĠQu o Ref , Ġthe Ġhighest ĠF 1 Ġis Ġ68 . 44 Ġwhen Ġ$\ alpha Ġ$ Ġis Ġset Ġto Ġ0 . 4 . ĠIn Ġaddition , Ġwe Ġcan Ġobserve Ġthat Ġthe Ġperformance Ġvaries Ġa Ġlot Ġas Ġ$\ alpha Ġ$ Ġchanges Ġin Ġdistinct Ġdatasets , Ġwhich Ġshows Ġthat Ġthe Ġhyper param eters Ġ$\ alpha Ġ, \ beta Ġ$ Ġplay Ġan Ġimportant Ġrole Ġin Ġthe Ġproposed Ġmethod . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġalleviate Ġthe Ġsevere Ġdata Ġimbalance Ġissue Ġin ĠN LP Ġtasks . ĠWe Ġpropose Ġto Ġuse Ġdice Ġloss Ġin Ġreplacement Ġof Ġthe Ġstandard Ġcross - ent ropy Ġloss , Ġwhich Ġperforms Ġas Ġa Ġsoft Ġversion Ġof ĠF 1 Ġscore . ĠUsing Ġdice Ġloss Ġcan Ġhelp Ġnarrow Ġthe Ġgap Ġbetween Ġtraining Ġobjectives Ġand Ġevaluation Ġmetrics . ĠEm pir ically , Ġwe Ġshow Ġthat Ġthe Ġproposed Ġtraining Ġobjective Ġleads Ġto Ġsignificant Ġperformance Ġboost Ġfor Ġpart - of - speech , Ġnamed Ġentity Ġrecognition , Ġmachine Ġreading Ġcomprehension Ġand Ġparaph rase Ġidentification Ġtasks .
Introduction Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine reading comprehension. Table TABREF3 gives concrete examples: for the Named Entity Recognition (NER) task BIBREF2, BIBREF3, most tokens are backgrounds with tagging class $O$. Specifically, the number of tokens tagging class $O$ is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5.0 dataset; Data-imbalanced issue is more severe for MRC tasks BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8 with the value of negative-positive ratio being 50-200. Data imbalance results in the following two issues: (1) the training-test discrepancy: Without balancing the labels, the learning process tends to converge to a point that strongly biases towards class with the majority label. This actually creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function while at test time, F1 score concerns more about positive examples; (2) the overwhelming effect of easy-negative examples. As pointed out by meng2019dsreg, significantly large number of negative examples also means that the number of easy-negative example is large. The huge number of easy examples tends to overwhelm the training, making the model not sufficiently learned to distinguish between positive examples and hard-negative examples. The cross-entropy objective (CE for short) or maximum likelihood (MLE) objective, which is widely adopted as the training objective for data-imbalanced NLP tasks BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, handles neither of the issues. To handle the first issue, we propose to replace CE or MLE with losses based on the Sørensen–Dice coefficient BIBREF0 or Tversky index BIBREF1. The Sørensen–Dice coefficient, dice loss for short, is the harmonic mean of precision and recall. It attaches equal importance to false positives (FPs) and false negatives (FNs) and is thus more immune to data-imbalanced datasets. Tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the $F_{\beta }$ score, and thus comes with more flexibility. Therefore, We use dice loss or Tversky index to replace CE loss to address the first issue. Only using dice loss or Tversky index is not enough since they are unable to address the dominating influence of easy-negative examples. This is intrinsically because dice loss is actually a hard version of the F1 score. Taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0.5, but training will push the value to 0 as much as possible. This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones. Inspired by the idea of focal loss BIBREF16 in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to $(1-p)$, and this weight dynamically changes as training proceeds. This strategy helps to deemphasize confident examples during training as their $p$ approaches the value of 1, makes the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative examples. Combing both strategies, we observe significant performance boosts on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5 (97.92, +1.86), CTB6 (96.57, +1.80) and UD1.4 (96.98, +2.19) for the POS task; SOTA results on CoNLL03 (93.33, +0.29), OntoNotes5.0 (92.07, +0.96)), MSRA 96.72(+0.97) and OntoNotes4.0 (84.47,+2.36) for the NER task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification. The rest of this paper is organized as follows: related work is presented in Section 2. We describe different training objectives in Section 3. Experimental results are presented in Section 4. We perform ablation studies in Section 5, followed by a brief conclusion in Section 6. Related Work ::: Data Resample The idea of weighting training examples has a long history. Importance sampling BIBREF17 assigns weights to different samples and changes the data distribution. Boosting algorithms such as AdaBoost BIBREF18 select harder examples to train subsequent classifiers. Similarly, hard example mining BIBREF19 downsamples the majority class and exploits the most difficult examples. Oversampling BIBREF20, BIBREF21 is used to balance the data distribution. Another line of data resampling is to dynamically control the weights of examples as training proceeds. For example, focal loss BIBREF16 used a soft weighting scheme that emphasizes harder examples during training. In self-paced learning BIBREF22, example weights are obtained through optimizing the weighted training loss which encourages learning easier examples first. At each training step, self-paced learning algorithm optimizes model parameters and example weights jointly. Other works BIBREF23, BIBREF24 adjusted the weights of different training examples based on training loss. Besides, recent work BIBREF25, BIBREF26 proposed to learn a separate network to predict sample weights. Related Work ::: Data Imbalance Issue in Object Detection The background-object label imbalance issue is severe and thus well studied in the field of object detection BIBREF27, BIBREF28, BIBREF29, BIBREF30, BIBREF31. The idea of hard negative mining (HNM) BIBREF30 has gained much attention recently. shrivastava2016ohem proposed the online hard example mining (OHEM) algorithm in an iterative manner that makes training progressively more difficult, and pushes the model to learn better. ssd2016liu sorted all of the negative samples based on the confidence loss and picking the training examples with the negative-positive ratio at 3:1. pang2019rcnn proposed a novel method called IoU-balanced sampling and aploss2019chen designed a ranking model to replace the conventional classification task with a average-precision loss to alleviate the class imbalance issue. The efforts made on object detection have greatly inspired us to solve the data imbalance issue in NLP. Losses ::: Notation For illustration purposes, we use the binary classification task to demonstrate how different losses work. The mechanism can be easily extended to multi-class classification. Let $\lbrace x_i\rbrace $ denote a set of instances. Each $x_i$ is associated with a golden label vector $y_i = [y_{i0},y_{i1} ]$, where $y_{i1}\in \lbrace 0,1\rbrace $ and $y_{i0}\in \lbrace 0,1\rbrace $ respectively denote the positive and negative classes, and thus $y_i$ can be either $[0,1]$ or $[0,1]$. Let $p_i = [p_{i0},p_{i1} ]$ denote the probability vector, and $p_{i1}$ and $p_{i0}$ respectively denote the probability that a model assigns the positive and negative label to $x_i$. Losses ::: Cross Entropy Loss The vanilla cross entropy (CE) loss is given by: As can be seen from Eq.DISPLAY_FORM8, each $x_i$ contributes equally to the final objective. Two strategies are normally used to address the the case where we wish that not all $x_i$ are treated equal: associating different classes with different weighting factor $\alpha $ or resampling the datasets. For the former, Eq.DISPLAY_FORM8 is adjusted as follows: where $\alpha _i\in [0,1]$ may be set by the inverse class frequency or treated as a hyperparameter to set by cross validation. In this work, we use $\lg (\frac{n-n_t}{n_t}+K)$ to calculate the coefficient $\alpha $, where $n_t$ is the number of samples with class $t$ and $n$ is the total number of samples in the training set. $K$ is a hyperparameter to tune. The data resampling strategy constructs a new dataset by sampling training examples from the original dataset based on human-designed criteria, e.g., extract equal training samples from each class. Both strategies are equivalent to changing the data distribution and thus are of the same nature. Empirically, these two methods are not widely used due to the trickiness of selecting $\alpha $ especially for multi-class classification tasks and that inappropriate selection can easily bias towards rare classes BIBREF32. Losses ::: Dice coefficient and Tversky index Sørensen–Dice coefficient BIBREF0, BIBREF33, dice coefficient (DSC) for short, is a F1-oriented statistic used to gauge the similarity of two sets. Given two sets $A$ and $B$, the dice coefficient between them is given as follows: In our case, $A$ is the set that contains of all positive examples predicted by a specific model, and $B$ is the set of all golden positive examples in the dataset. When applied to boolean data with the definition of true positive (TP), false positive (FP), and false negative (FN), it can be then written as follows: For an individual example $x_i$, its corresponding DSC loss is given as follows: As can be seen, for a negative example with $y_{i1}=0$, it does not contribute to the objective. For smoothing purposes, it is common to add a $\gamma $ factor to both the nominator and the denominator, making the form to be as follows: As can be seen, negative examples, with $y_{i1}$ being 0 and DSC being $\frac{\gamma }{ p_{i1}+\gamma }$, also contribute to the training. Additionally, milletari2016v proposed to change the denominator to the square form for faster convergence, which leads to the following dice loss (DL): Another version of DL is to directly compute set-level dice coefficient instead of the sum of individual dice coefficient. We choose the latter due to ease of optimization. Tversky index (TI), which can be thought as the approximation of the $F_{\beta }$ score, extends dice coefficient to a more general case. Given two sets $A$ and $B$, tversky index is computed as follows: Tversky index offers the flexibility in controlling the tradeoff between false-negatives and false-positives. It degenerates to DSC if $\alpha =\beta =0.5$. The Tversky loss (TL) for the training set $\lbrace x_i,y_i\rbrace $ is thus as follows: Losses ::: Self-adusting Dice Loss Consider a simple case where the dataset consists of only one example $x_i$, which is classified as positive as long as $p_{i1}$ is larger than 0.5. The computation of $F1$ score is actually as follows: Comparing Eq.DISPLAY_FORM14 with Eq.DISPLAY_FORM22, we can see that Eq.DISPLAY_FORM14 is actually a soft form of $F1$, using a continuous $p$ rather than the binary $\mathbb {I}( p_{i1}>0.5)$. This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance. To address this issue, we propose to multiply the soft probability $p$ with a decaying factor $(1-p)$, changing Eq.DISPLAY_FORM22 to the following form: One can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds. The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples. For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them. Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified. A close look at Eq.DISPLAY_FORM14 reveals that it actually mimics the idea of focal loss (FL for short) BIBREF16 for object detection in vision. Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training. It down-weights the loss assigned to well-classified examples by adding a $(1-p)^{\beta }$ factor, leading the final loss to be $(1-p)^{\beta }\log p$. In Table TABREF18, we show the losses used in our experiments, which is described in the next section. Experiments We evaluate the proposed method on four NLP tasks: part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification. Baselines in our experiments are optimized by using the standard cross-entropy training objective. Experiments ::: Part-of-Speech Tagging Part-of-speech tagging (POS) is the task of assigning a label (e.g., noun, verb, adjective) to each word in a given text. In this paper, we choose BERT as the backbone and conduct experiments on three Chinese POS datasets. We report the span-level micro-averaged precision, recall and F1 for evaluation. Hyperparameters are tuned on the corresponding development set of each dataset. Experiments ::: Part-of-Speech Tagging ::: Datasets We conduct experiments on the widely used Chinese Treebank 5.0, 6.0 as well as UD1.4. CTB5 is a Chinese dataset for tagging and parsing, which contains 507,222 words, 824,983 characters and 18,782 sentences extracted from newswire sources. CTB6 is an extension of CTB5, containing 781,351 words, 1,285,149 characters and 28,295 sentences. UD is the abbreviation of Universal Dependencies, which is a framework for consistent annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages. In this work, we use UD1.4 for Chinese POS tagging. Experiments ::: Part-of-Speech Tagging ::: Baselines We use the following baselines: Joint-POS: shao2017character jointly learns Chinese word segmentation and POS. Lattice-LSTM: lattice2018zhang constructs a word-character lattice. Bert-Tagger: devlin2018bert treats part-of-speech as a tagging task. Experiments ::: Part-of-Speech Tagging ::: Results Table presents the experimental results on the POS task. As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i.e., outperforming BERT-tagger by +1.86 in terms of F1 score on CTB5, +1.80 on CTB6 and +2.19 on UD1.4. As far as we are concerned, we are achieving SOTA performances on the three datasets. Weighted cross entropy and focal loss only gain a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in resolving the data imbalance issue. The proposed DSC loss performs robustly on all the three datasets. Experiments ::: Named Entity Recognition Named entity recognition (NER) refers to the task of detecting the span and semantic category of entities from a chunk of text. Our implementation uses the current state-of-the-art BERT-MRC model proposed by xiaoya2019ner as a backbone. For English datasets, we use BERT$_\text{Large}$ English checkpoints, while for Chinese we use the official Chinese checkpoints. We report span-level micro-averaged precision, recall and F1-score. Hyperparameters are tuned on the development set of each dataset. Experiments ::: Named Entity Recognition ::: Datasets For the NER task, we consider both Chinese datasets, i.e., OntoNotes4.0 BIBREF34 and MSRA BIBREF35, and English datasets, i.e., CoNLL2003 BIBREF36 and OntoNotes5.0 BIBREF37. CoNLL2003 is an English dataset with 4 entity types: Location, Organization, Person and Miscellaneous. We followed data processing protocols in BIBREF14. English OntoNotes5.0 consists of texts from a wide variety of sources and contains 18 entity types. We use the standard train/dev/test split of CoNLL2012 shared task. Chinese MSRA performs as a Chinese benchmark dataset containing 3 entity types. Data in MSRA is collected from news domain. Since the development set is not provided in the original MSRA dataset, we randomly split the training set into training and development splits by 9:1. We use the official test set for evaluation. Chinese OntoNotes4.0 is a Chinese dataset and consists of texts from news domain, which has 18 entity types. In this paper, we take the same data split as wu2019glyce did. Experiments ::: Named Entity Recognition ::: Baselines We use the following baselines: ELMo: a tagging model from peters2018deep. Lattice-LSTM: lattice2018zhang constructs a word-character lattice, only used in Chinese datasets. CVT: from kevin2018cross, which uses Cross-View Training(CVT) to improve the representations of a Bi-LSTM encoder. Bert-Tagger: devlin2018bert treats NER as a tagging task. Glyce-BERT: wu2019glyce combines glyph information with BERT pretraining. BERT-MRC: The current SOTA model for both Chinese and English NER datasets proposed by xiaoya2019ner, which formulate NER as machine reading comprehension task. Experiments ::: Named Entity Recognition ::: Results Table shows experimental results on NER datasets. For English datasets including CoNLL2003 and OntoNotes5.0, our proposed method outperforms BERT-MRCBIBREF38 by +0.29 and +0.96 respectively. We observe huge performance boosts on Chinese datasets, achieving F1 improvements by +0.97 and +2.36 on MSRA and OntoNotes4.0, respectively. As far as we are concerned, we are setting new SOTA performances on all of the four NER datasets. Experiments ::: Machine Reading Comprehension Machine reading comprehension (MRC) BIBREF39, BIBREF40, BIBREF41, BIBREF40, BIBREF42, BIBREF15 has become a central task in natural language understanding. MRC in the SQuAD-style is to predict the answer span in the passage given a question and the passage. In this paper, we choose the SQuAD-style MRC task and report Extract Match (EM) in addition to F1 score on validation set. All hyperparameters are tuned on the development set of each dataset. Experiments ::: Machine Reading Comprehension ::: Datasets The following five datasets are used for MRC task: SQuAD v1.1, SQuAD v2.0 BIBREF4, BIBREF6 and Quoref BIBREF8. SQuAD v1.1 and SQuAD v2.0 are the most widely used QA benchmarks. SQuAD1.1 is a collection of 100K crowdsourced question-answer pairs, and SQuAD2.0 extends SQuAD1.1 allowing no short answer exists in the provided passage. Quoref is a QA dataset which tests the coreferential reasoning capability of reading comprehension systems, containing 24K questions over 4.7K paragraphs from Wikipedia. Experiments ::: Machine Reading Comprehension ::: Baselines We use the following baselines: QANet: qanet2018 builds a model based on convolutions and self-attention. Convolution to model local interactions and self-attention to model global interactions. BERT: devlin2018bert treats NER as a tagging task. XLNet: xlnet2019 proposes a generalized autoregressive pretraining method that enables learning bidirectional contexts. Experiments ::: Machine Reading Comprehension ::: Results Table shows the experimental results for MRC tasks. With either BERT or XLNet, our proposed DSC loss obtains significant performance boost on both EM and F1. For SQuADv1.1, our proposed method outperforms XLNet by +1.25 in terms of F1 score and +0.84 in terms of EM and achieves 87.65 on EM and 89.51 on F1 for SQuAD v2.0. Moreover, on QuoRef, the proposed method surpasses XLNet results by +1.46 on EM and +1.41 on F1. Another observation is that, XLNet outperforms BERT by a huge margin, and the proposed DSC loss can obtain further performance improvement by an average score above 1.0 in terms of both EM and F1, which indicates the DSC loss is complementary to the model structures. Experiments ::: Paraphrase Identification Paraphrases are textual expressions that have the same semantic meaning using different surface words. Paraphrase identification (PI) is the task of identifying whether two sentences have the same meaning or not. We use BERT BIBREF11 and XLNet BIBREF43 as backbones and report F1 score for comparison. Hyperparameters are tuned on the development set of each dataset. Experiments ::: Paraphrase Identification ::: Datasets We conduct experiments on two widely used datasets for PI task: MRPC BIBREF44 and QQP. MRPC is a corpus of sentence pairs automatically extracted from online news sources, with human annotations of whether the sentence pairs are semantically equivalent. The MRPC dataset has imbalanced classes (68% positive, 32% for negative). QQP is a collection of question pairs from the community question-answering website Quora. The class distribution in QQP is also unbalanced (37% positive, 63% negative). Experiments ::: Paraphrase Identification ::: Results Table shows the results for PI task. We find that replacing the training objective with DSC introduces performance boost for both BERT and XLNet. Using DSC loss improves the F1 score by +0.58 for MRPC and +0.73 for QQP. Ablation Studies ::: The Effect of Dice Loss on Accuracy-oriented Tasks We argue that the most commonly used cross-entropy objective is actually accuracy-oriented, whereas the proposed dice loss (DL) performs as a hard version of F1-score. To explore the effect of the dice loss on accuracy-oriented tasks such as text classification, we conduct experiments on the Stanford Sentiment Treebank sentiment classification datasets including SST-2 and SST-5. We fine-tune BERT$_\text{Large}$ with different training objectives. Experiment results for SST are shown in . For SST-5, BERT with CE achieves 55.57 in terms of accuracy, with DL and DSC losses slightly degrade the accuracy performance and achieve 54.63 and 55.19, respectively. For SST-2, BERT with CE achieves 94.9 in terms of accuracy. The same as SST-5, we observe a slight performance drop with DL and DSC, which means that the dice loss actually works well for F1 but not for accuracy. Ablation Studies ::: The Effect of Hyperparameters in Tversky index As mentioned in Section SECREF10, Tversky index (TI) offers the flexibility in controlling the tradeoff between false-negatives and false-positives. In this subsection, we explore the effect of hyperparameters (i.e., $\alpha $ and $\beta $) in TI to test how they manipulate the tradeoff. We conduct experiments on the Chinese OntoNotes4.0 NER dataset and English QuoRef MRC dataset to examine the influence of tradeoff between precision and recall. Experiment results are shown in Table . The highest F1 for Chinese OntoNotes4.0 is 84.67 when $\alpha $ is set to 0.6 while for QuoRef, the highest F1 is 68.44 when $\alpha $ is set to 0.4. In addition, we can observe that the performance varies a lot as $\alpha $ changes in distinct datasets, which shows that the hyperparameters $\alpha ,\beta $ play an important role in the proposed method. Conclusion In this paper, we alleviate the severe data imbalance issue in NLP tasks. We propose to use dice loss in replacement of the standard cross-entropy loss, which performs as a soft version of F1 score. Using dice loss can help narrow the gap between training objectives and evaluation metrics. Empirically, we show that the proposed training objective leads to significant performance boost for part-of-speech, named entity recognition, machine reading comprehension and paraphrase identification tasks.
What are method's improvements of F1 w.r.t. baseline BERT tagger for Chinese POS datasets?
72f7ef55e150e16dcf97fe443aff9971a32414ef
What Ġare Ġmethod 's Ġimprovements Ġof ĠF 1 Ġw . r . t . Ġbaseline ĠB ERT Ġtag ger Ġfor ĠChinese ĠPOS Ġdatasets ?
+ 1 . 86 Ġin Ġterms Ġof ĠF 1 Ġscore Ġon ĠCT B 5 , Ġ+ 1 . 80 Ġon ĠCT B 6 , Ġ+ 2 . 19 Ġon ĠUD 1 . 4
<s> What Ġare Ġmethod 's Ġimprovements Ġof ĠF 1 Ġw . r . t . Ġbaseline ĠB ERT Ġtag ger Ġfor ĠChinese ĠPOS Ġdatasets ? </s> Introduction </s> Data Ġimbalance Ġis Ġa Ġcommon Ġissue Ġin Ġa Ġvariety Ġof ĠN LP Ġtasks Ġsuch Ġas Ġtagging Ġand Ġmachine Ġreading Ġcomprehension . ĠTable ĠT AB REF 3 Ġgives Ġconcrete Ġexamples : Ġfor Ġthe ĠNamed ĠEntity ĠRecogn ition Ġ( NER ) Ġtask ĠB IB REF 2 , ĠB IB REF 3 , Ġmost Ġtokens Ġare Ġbackgrounds Ġwith Ġtagging Ġclass Ġ$ O $. ĠSpecifically , Ġthe Ġnumber Ġof Ġtokens Ġtagging Ġclass Ġ$ O $ Ġis Ġ5 Ġtimes Ġas Ġmany Ġas Ġthose Ġwith Ġentity Ġlabels Ġfor Ġthe ĠCo N LL 03 Ġdataset Ġand Ġ8 Ġtimes Ġfor Ġthe ĠOnt o Notes 5 . 0 Ġdataset ; ĠData - im balanced Ġissue Ġis Ġmore Ġsevere Ġfor ĠM RC Ġtasks ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 Ġwith Ġthe Ġvalue Ġof Ġnegative - positive Ġratio Ġbeing Ġ50 - 200 . </s> Data Ġimbalance Ġresults Ġin Ġthe Ġfollowing Ġtwo Ġissues : Ġ( 1 ) Ġthe Ġtraining - test Ġdiscrepancy : ĠWithout Ġbalancing Ġthe Ġlabels , Ġthe Ġlearning Ġprocess Ġtends Ġto Ġconverge Ġto Ġa Ġpoint Ġthat Ġstrongly Ġbiases Ġtowards Ġclass Ġwith Ġthe Ġmajority Ġlabel . ĠThis Ġactually Ġcreates Ġa Ġdiscrepancy Ġbetween Ġtraining Ġand Ġtest : Ġat Ġtraining Ġtime , Ġeach Ġtraining Ġinstance Ġcontributes Ġequally Ġto Ġthe Ġobjective Ġfunction Ġwhile Ġat Ġtest Ġtime , ĠF 1 Ġscore Ġconcerns Ġmore Ġabout Ġpositive Ġexamples ; Ġ( 2 ) Ġthe Ġoverwhelming Ġeffect Ġof Ġeasy - negative Ġexamples . ĠAs Ġpointed Ġout Ġby Ġmen g 2019 ds reg , Ġsignificantly Ġlarge Ġnumber Ġof Ġnegative Ġexamples Ġalso Ġmeans Ġthat Ġthe Ġnumber Ġof Ġeasy - negative Ġexample Ġis Ġlarge . ĠThe Ġhuge Ġnumber Ġof Ġeasy Ġexamples Ġtends Ġto Ġoverwhelm Ġthe Ġtraining , Ġmaking Ġthe Ġmodel Ġnot Ġsufficiently Ġlearned Ġto Ġdistinguish Ġbetween Ġpositive Ġexamples Ġand Ġhard - negative Ġexamples . ĠThe Ġcross - ent ropy Ġobjective Ġ( CE Ġfor Ġshort ) Ġor Ġmaximum Ġlikelihood Ġ( M LE ) Ġobjective , Ġwhich Ġis Ġwidely Ġadopted Ġas Ġthe Ġtraining Ġobjective Ġfor Ġdata - im balanced ĠN LP Ġtasks ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 , ĠB IB REF 12 , ĠB IB REF 13 , ĠB IB REF 14 , ĠB IB REF 15 , Ġhandles Ġneither Ġof Ġthe Ġissues . </s> To Ġhandle Ġthe Ġfirst Ġissue , Ġwe Ġpropose Ġto Ġreplace ĠCE Ġor ĠM LE Ġwith Ġlosses Ġbased Ġon Ġthe ĠS Ã¸ ren sen âĢĵ D ice Ġcoefficient ĠB IB REF 0 Ġor ĠT vers ky Ġindex ĠB IB REF 1 . ĠThe ĠS Ã¸ ren sen âĢĵ D ice Ġcoefficient , Ġdice Ġloss Ġfor Ġshort , Ġis Ġthe Ġharmonic Ġmean Ġof Ġprecision Ġand Ġrecall . ĠIt Ġattaches Ġequal Ġimportance Ġto Ġfalse Ġpositives Ġ( FP s ) Ġand Ġfalse Ġnegatives Ġ( FN s ) Ġand Ġis Ġthus Ġmore Ġimmune Ġto Ġdata - im balanced Ġdatasets . ĠT vers ky Ġindex Ġextends Ġdice Ġloss Ġby Ġusing Ġa Ġweight Ġthat Ġtrades Ġprecision Ġand Ġrecall , Ġwhich Ġcan Ġbe Ġthought Ġas Ġthe Ġapproximation Ġof Ġthe Ġ$ F _{ \ beta Ġ} $ Ġscore , Ġand Ġthus Ġcomes Ġwith Ġmore Ġflexibility . ĠTherefore , ĠWe Ġuse Ġdice Ġloss Ġor ĠT vers ky Ġindex Ġto Ġreplace ĠCE Ġloss Ġto Ġaddress Ġthe Ġfirst Ġissue . </s> Only Ġusing Ġdice Ġloss Ġor ĠT vers ky Ġindex Ġis Ġnot Ġenough Ġsince Ġthey Ġare Ġunable Ġto Ġaddress Ġthe Ġdominating Ġinfluence Ġof Ġeasy - negative Ġexamples . ĠThis Ġis Ġintrinsically Ġbecause Ġdice Ġloss Ġis Ġactually Ġa Ġhard Ġversion Ġof Ġthe ĠF 1 Ġscore . ĠTaking Ġthe Ġbinary Ġclassification Ġtask Ġas Ġan Ġexample , Ġat Ġtest Ġtime , Ġan Ġexample Ġwill Ġbe Ġclassified Ġas Ġnegative Ġas Ġlong Ġas Ġits Ġprobability Ġis Ġsmaller Ġthan Ġ0 . 5 , Ġbut Ġtraining Ġwill Ġpush Ġthe Ġvalue Ġto Ġ0 Ġas Ġmuch Ġas Ġpossible . ĠThis Ġgap Ġisn 't Ġa Ġbig Ġissue Ġfor Ġbalanced Ġdatasets , Ġbut Ġis Ġextremely Ġdetrimental Ġif Ġa Ġbig Ġproportion Ġof Ġtraining Ġexamples Ġare Ġeasy - negative Ġones : Ġeasy - negative Ġexamples Ġcan Ġeasily Ġdominate Ġtraining Ġsince Ġtheir Ġprobabilities Ġcan Ġbe Ġpushed Ġto Ġ0 Ġfairly Ġeasily . ĠMeanwhile , Ġthe Ġmodel Ġcan Ġhardly Ġdistinguish Ġbetween Ġhard - negative Ġexamples Ġand Ġpositive Ġones . ĠInspired Ġby Ġthe Ġidea Ġof Ġfocal Ġloss ĠB IB REF 16 Ġin Ġcomputer Ġvision , Ġwe Ġpropose Ġa Ġdynamic Ġweight Ġadjusting Ġstrategy , Ġwhich Ġassociates Ġeach Ġtraining Ġexample Ġwith Ġa Ġweight Ġin Ġproportion Ġto Ġ$( 1 - p ) $, Ġand Ġthis Ġweight Ġdynamically Ġchanges Ġas Ġtraining Ġproceeds . ĠThis Ġstrategy Ġhelps Ġto Ġdeem phas ize Ġconfident Ġexamples Ġduring Ġtraining Ġas Ġtheir Ġ$ p $ Ġapproaches Ġthe Ġvalue Ġof Ġ1 , Ġmakes Ġthe Ġmodel Ġattentive Ġto Ġhard - negative Ġexamples , Ġand Ġthus Ġallev iates Ġthe Ġdominating Ġeffect Ġof Ġeasy - negative Ġexamples . </s> Comb ing Ġboth Ġstrategies , Ġwe Ġobserve Ġsignificant Ġperformance Ġboosts Ġon Ġa Ġwide Ġrange Ġof Ġdata Ġim balanced ĠN LP Ġtasks . ĠNotably , Ġwe Ġare Ġable Ġto Ġachieve ĠS OTA Ġresults Ġon ĠCT B 5 Ġ( 97 . 92 , Ġ+ 1 . 86 ), ĠCT B 6 Ġ( 96 . 57 , Ġ+ 1 . 80 ) Ġand ĠUD 1 . 4 Ġ( 96 . 98 , Ġ+ 2 . 19 ) Ġfor Ġthe ĠPOS Ġtask ; ĠS OTA Ġresults Ġon ĠCo N LL 03 Ġ( 93 . 33 , Ġ+ 0 . 29 ), ĠOnt o Notes 5 . 0 Ġ( 92 . 07 , Ġ+ 0 . 96 )), ĠMS RA Ġ96 . 72 ( + 0 . 97 ) Ġand ĠOnt o Notes 4 . 0 Ġ( 84 . 47 , + 2 . 36 ) Ġfor Ġthe ĠN ER Ġtask ; Ġalong Ġwith Ġcompetitive Ġresults Ġon Ġthe Ġtasks Ġof Ġmachine Ġreading Ġcomprehension Ġand Ġparaph rase Ġidentification . </s> The Ġrest Ġof Ġthis Ġpaper Ġis Ġorganized Ġas Ġfollows : Ġrelated Ġwork Ġis Ġpresented Ġin ĠSection Ġ2 . ĠWe Ġdescribe Ġdifferent Ġtraining Ġobjectives Ġin ĠSection Ġ3 . ĠExperimental Ġresults Ġare Ġpresented Ġin ĠSection Ġ4 . ĠWe Ġperform Ġab lation Ġstudies Ġin ĠSection Ġ5 , Ġfollowed Ġby Ġa Ġbrief Ġconclusion Ġin ĠSection Ġ6 . </s> Related ĠWork Ġ: :: ĠData ĠRes ample </s> The Ġidea Ġof Ġweight ing Ġtraining Ġexamples Ġhas Ġa Ġlong Ġhistory . ĠImport ance Ġsampling ĠB IB REF 17 Ġassigns Ġweights Ġto Ġdifferent Ġsamples Ġand Ġchanges Ġthe Ġdata Ġdistribution . ĠBoost ing Ġalgorithms Ġsuch Ġas ĠAda Boost ĠB IB REF 18 Ġselect Ġharder Ġexamples Ġto Ġtrain Ġsubsequent Ġclass ifiers . ĠSimilarly , Ġhard Ġexample Ġmining ĠB IB REF 19 Ġdowns amples Ġthe Ġmajority Ġclass Ġand Ġexploits Ġthe Ġmost Ġdifficult Ġexamples . ĠOvers am pling ĠB IB REF 20 , ĠB IB REF 21 Ġis Ġused Ġto Ġbalance Ġthe Ġdata Ġdistribution . ĠAnother Ġline Ġof Ġdata Ġres am pling Ġis Ġto Ġdynamically Ġcontrol Ġthe Ġweights Ġof Ġexamples Ġas Ġtraining Ġproceeds . ĠFor Ġexample , Ġfocal Ġloss ĠB IB REF 16 Ġused Ġa Ġsoft Ġweight ing Ġscheme Ġthat Ġemphasizes Ġharder Ġexamples Ġduring Ġtraining . ĠIn Ġself - paced Ġlearning ĠB IB REF 22 , Ġexample Ġweights Ġare Ġobtained Ġthrough Ġoptimizing Ġthe Ġweighted Ġtraining Ġloss Ġwhich Ġencourages Ġlearning Ġeasier Ġexamples Ġfirst . ĠAt Ġeach Ġtraining Ġstep , Ġself - paced Ġlearning Ġalgorithm Ġoptim izes Ġmodel Ġparameters Ġand Ġexample Ġweights Ġjointly . ĠOther Ġworks ĠB IB REF 23 , ĠB IB REF 24 Ġadjusted Ġthe Ġweights Ġof Ġdifferent Ġtraining Ġexamples Ġbased Ġon Ġtraining Ġloss . ĠBesides , Ġrecent Ġwork ĠB IB REF 25 , ĠB IB REF 26 Ġproposed Ġto Ġlearn Ġa Ġseparate Ġnetwork Ġto Ġpredict Ġsample Ġweights . </s> Related ĠWork Ġ: :: ĠData ĠIm balance ĠIssue Ġin ĠObject ĠDetection </s> The Ġbackground - object Ġlabel Ġimbalance Ġissue Ġis Ġsevere Ġand Ġthus Ġwell Ġstudied Ġin Ġthe Ġfield Ġof Ġobject Ġdetection ĠB IB REF 27 , ĠB IB REF 28 , ĠB IB REF 29 , ĠB IB REF 30 , ĠB IB REF 31 . ĠThe Ġidea Ġof Ġhard Ġnegative Ġmining Ġ( H NM ) ĠB IB REF 30 Ġhas Ġgained Ġmuch Ġattention Ġrecently . Ġsh riv ast ava 2016 o hem Ġproposed Ġthe Ġonline Ġhard Ġexample Ġmining Ġ( OH EM ) Ġalgorithm Ġin Ġan Ġiter ative Ġmanner Ġthat Ġmakes Ġtraining Ġprogressively Ġmore Ġdifficult , Ġand Ġpushes Ġthe Ġmodel Ġto Ġlearn Ġbetter . Ġs sd 2016 li u Ġsorted Ġall Ġof Ġthe Ġnegative Ġsamples Ġbased Ġon Ġthe Ġconfidence Ġloss Ġand Ġpicking Ġthe Ġtraining Ġexamples Ġwith Ġthe Ġnegative - positive Ġratio Ġat Ġ3 : 1 . Ġp ang 2019 rc nn Ġproposed Ġa Ġnovel Ġmethod Ġcalled ĠIo U - balanced Ġsampling Ġand Ġa pl oss 2019 chen Ġdesigned Ġa Ġranking Ġmodel Ġto Ġreplace Ġthe Ġconventional Ġclassification Ġtask Ġwith Ġa Ġaverage - pre cision Ġloss Ġto Ġalleviate Ġthe Ġclass Ġimbalance Ġissue . ĠThe Ġefforts Ġmade Ġon Ġobject Ġdetection Ġhave Ġgreatly Ġinspired Ġus Ġto Ġsolve Ġthe Ġdata Ġimbalance Ġissue Ġin ĠN LP . </s> L oss es Ġ: :: ĠNot ation </s> For Ġillustration Ġpurposes , Ġwe Ġuse Ġthe Ġbinary Ġclassification Ġtask Ġto Ġdemonstrate Ġhow Ġdifferent Ġlosses Ġwork . ĠThe Ġmechanism Ġcan Ġbe Ġeasily Ġextended Ġto Ġmulti - class Ġclassification . </s> Let Ġ$\ l brace Ġx _ i \ r brace Ġ$ Ġdenote Ġa Ġset Ġof Ġinstances . ĠEach Ġ$ x _ i $ Ġis Ġassociated Ġwith Ġa Ġgolden Ġlabel Ġvector Ġ$ y _ i Ġ= Ġ[ y _{ i 0 }, y _{ i 1 } Ġ] $, Ġwhere Ġ$ y _{ i 1 }\ in Ġ\ l brace Ġ0 , 1 \ r brace Ġ$ Ġand Ġ$ y _{ i 0 }\ in Ġ\ l brace Ġ0 , 1 \ r brace Ġ$ Ġrespectively Ġdenote Ġthe Ġpositive Ġand Ġnegative Ġclasses , Ġand Ġthus Ġ$ y _ i $ Ġcan Ġbe Ġeither Ġ$ [ 0 , 1 ] $ Ġor Ġ$ [ 0 , 1 ] $. ĠLet Ġ$ p _ i Ġ= Ġ[ p _{ i 0 }, p _{ i 1 } Ġ] $ Ġdenote Ġthe Ġprobability Ġvector , Ġand Ġ$ p _{ i 1 } $ Ġand Ġ$ p _{ i 0 } $ Ġrespectively Ġdenote Ġthe Ġprobability Ġthat Ġa Ġmodel Ġassigns Ġthe Ġpositive Ġand Ġnegative Ġlabel Ġto Ġ$ x _ i $. </s> L oss es Ġ: :: ĠCross ĠEnt ropy ĠLoss </s> The Ġvanilla Ġcross Ġentropy Ġ( CE ) Ġloss Ġis Ġgiven Ġby : </s> As Ġcan Ġbe Ġseen Ġfrom ĠE q . DIS PLAY _ FORM 8 , Ġeach Ġ$ x _ i $ Ġcontributes Ġequally Ġto Ġthe Ġfinal Ġobjective . ĠTwo Ġstrategies Ġare Ġnormally Ġused Ġto Ġaddress Ġthe Ġthe Ġcase Ġwhere Ġwe Ġwish Ġthat Ġnot Ġall Ġ$ x _ i $ Ġare Ġtreated Ġequal : Ġassoci ating Ġdifferent Ġclasses Ġwith Ġdifferent Ġweight ing Ġfactor Ġ$\ alpha Ġ$ Ġor Ġres am pling Ġthe Ġdatasets . ĠFor Ġthe Ġformer , ĠE q . DIS PLAY _ FORM 8 Ġis Ġadjusted Ġas Ġfollows : </s> where Ġ$\ alpha Ġ_ i \ in Ġ[ 0 , 1 ] $ Ġmay Ġbe Ġset Ġby Ġthe Ġinverse Ġclass Ġfrequency Ġor Ġtreated Ġas Ġa Ġhyper param eter Ġto Ġset Ġby Ġcross Ġvalidation . ĠIn Ġthis Ġwork , Ġwe Ġuse Ġ$\ l g Ġ( \ frac { n - n _ t }{ n _ t } + K ) $ Ġto Ġcalculate Ġthe Ġcoefficient Ġ$\ alpha Ġ$ , Ġwhere Ġ$ n _ t $ Ġis Ġthe Ġnumber Ġof Ġsamples Ġwith Ġclass Ġ$ t $ Ġand Ġ$ n $ Ġis Ġthe Ġtotal Ġnumber Ġof Ġsamples Ġin Ġthe Ġtraining Ġset . Ġ$ K $ Ġis Ġa Ġhyper param eter Ġto Ġtune . ĠThe Ġdata Ġres am pling Ġstrategy Ġconstructs Ġa Ġnew Ġdataset Ġby Ġsampling Ġtraining Ġexamples Ġfrom Ġthe Ġoriginal Ġdataset Ġbased Ġon Ġhuman - designed Ġcriteria , Ġe . g ., Ġextract Ġequal Ġtraining Ġsamples Ġfrom Ġeach Ġclass . ĠBoth Ġstrategies Ġare Ġequivalent Ġto Ġchanging Ġthe Ġdata Ġdistribution Ġand Ġthus Ġare Ġof Ġthe Ġsame Ġnature . ĠEm pir ically , Ġthese Ġtwo Ġmethods Ġare Ġnot Ġwidely Ġused Ġdue Ġto Ġthe Ġtrick iness Ġof Ġselecting Ġ$\ alpha Ġ$ Ġespecially Ġfor Ġmulti - class Ġclassification Ġtasks Ġand Ġthat Ġinappropriate Ġselection Ġcan Ġeasily Ġbias Ġtowards Ġrare Ġclasses ĠB IB REF 32 . </s> L oss es Ġ: :: ĠDice Ġcoefficient Ġand ĠT vers ky Ġindex </s> S Ã¸ ren sen âĢĵ D ice Ġcoefficient ĠB IB REF 0 , ĠB IB REF 33 , Ġdice Ġcoefficient Ġ( DS C ) Ġfor Ġshort , Ġis Ġa ĠF 1 - oriented Ġstatistic Ġused Ġto Ġgauge Ġthe Ġsimilarity Ġof Ġtwo Ġsets . ĠGiven Ġtwo Ġsets Ġ$ A $ Ġand Ġ$ B $, Ġthe Ġdice Ġcoefficient Ġbetween Ġthem Ġis Ġgiven Ġas Ġfollows : </s> In Ġour Ġcase , Ġ$ A $ Ġis Ġthe Ġset Ġthat Ġcontains Ġof Ġall Ġpositive Ġexamples Ġpredicted Ġby Ġa Ġspecific Ġmodel , Ġand Ġ$ B $ Ġis Ġthe Ġset Ġof Ġall Ġgolden Ġpositive Ġexamples Ġin Ġthe Ġdataset . ĠWhen Ġapplied Ġto Ġboolean Ġdata Ġwith Ġthe Ġdefinition Ġof Ġtrue Ġpositive Ġ( TP ), Ġfalse Ġpositive Ġ( FP ), Ġand Ġfalse Ġnegative Ġ( FN ), Ġit Ġcan Ġbe Ġthen Ġwritten Ġas Ġfollows : </s> For Ġan Ġindividual Ġexample Ġ$ x _ i $, Ġits Ġcorresponding ĠD SC Ġloss Ġis Ġgiven Ġas Ġfollows : </s> As Ġcan Ġbe Ġseen , Ġfor Ġa Ġnegative Ġexample Ġwith Ġ$ y _{ i 1 } = 0 $, Ġit Ġdoes Ġnot Ġcontribute Ġto Ġthe Ġobjective . ĠFor Ġsmoot hing Ġpurposes , Ġit Ġis Ġcommon Ġto Ġadd Ġa Ġ$\ gam ma Ġ$ Ġfactor Ġto Ġboth Ġthe Ġnom inator Ġand Ġthe Ġdenomin ator , Ġmaking Ġthe Ġform Ġto Ġbe Ġas Ġfollows : </s> As Ġcan Ġbe Ġseen , Ġnegative Ġexamples , Ġwith Ġ$ y _{ i 1 } $ Ġbeing Ġ0 Ġand ĠD SC Ġbeing Ġ$\ frac {\ gam ma Ġ} { Ġp _{ i 1 } + \ gam ma Ġ} $, Ġalso Ġcontribute Ġto Ġthe Ġtraining . ĠAdditionally , Ġmil let ari 2016 v Ġproposed Ġto Ġchange Ġthe Ġdenomin ator Ġto Ġthe Ġsquare Ġform Ġfor Ġfaster Ġconvergence , Ġwhich Ġleads Ġto Ġthe Ġfollowing Ġdice Ġloss Ġ( DL ): </s> Another Ġversion Ġof ĠDL Ġis Ġto Ġdirectly Ġcompute Ġset - level Ġdice Ġcoefficient Ġinstead Ġof Ġthe Ġsum Ġof Ġindividual Ġdice Ġcoefficient . ĠWe Ġchoose Ġthe Ġlatter Ġdue Ġto Ġease Ġof Ġoptimization . </s> T vers ky Ġindex Ġ( TI ), Ġwhich Ġcan Ġbe Ġthought Ġas Ġthe Ġapproximation Ġof Ġthe Ġ$ F _{ \ beta Ġ} $ Ġscore , Ġextends Ġdice Ġcoefficient Ġto Ġa Ġmore Ġgeneral Ġcase . ĠGiven Ġtwo Ġsets Ġ$ A $ Ġand Ġ$ B $, Ġt vers ky Ġindex Ġis Ġcomputed Ġas Ġfollows : </s> T vers ky Ġindex Ġoffers Ġthe Ġflexibility Ġin Ġcontrolling Ġthe Ġtrade off Ġbetween Ġfalse - neg atives Ġand Ġfalse - pos itives . ĠIt Ġdegener ates Ġto ĠD SC Ġif Ġ$\ alpha Ġ= \ beta Ġ= 0 . 5 $. ĠThe ĠT vers ky Ġloss Ġ( TL ) Ġfor Ġthe Ġtraining Ġset Ġ$\ l brace Ġx _ i , y _ i \ r brace Ġ$ Ġis Ġthus Ġas Ġfollows : </s> L oss es Ġ: :: ĠSelf - ad usting ĠDice ĠLoss </s> Consider Ġa Ġsimple Ġcase Ġwhere Ġthe Ġdataset Ġconsists Ġof Ġonly Ġone Ġexample Ġ$ x _ i $, Ġwhich Ġis Ġclassified Ġas Ġpositive Ġas Ġlong Ġas Ġ$ p _{ i 1 } $ Ġis Ġlarger Ġthan Ġ0 . 5 . ĠThe Ġcomputation Ġof Ġ$ F 1 $ Ġscore Ġis Ġactually Ġas Ġfollows : </s> Comp aring ĠE q . DIS PLAY _ FORM 14 Ġwith ĠE q . DIS PLAY _ FORM 22 , Ġwe Ġcan Ġsee Ġthat ĠE q . DIS PLAY _ FORM 14 Ġis Ġactually Ġa Ġsoft Ġform Ġof Ġ$ F 1 $, Ġusing Ġa Ġcontinuous Ġ$ p $ Ġrather Ġthan Ġthe Ġbinary Ġ$\ math bb Ġ{ I } ( Ġp _{ i 1 } > 0 . 5 ) $. ĠThis Ġgap Ġisn 't Ġa Ġbig Ġissue Ġfor Ġbalanced Ġdatasets , Ġbut Ġis Ġextremely Ġdetrimental Ġif Ġa Ġbig Ġproportion Ġof Ġtraining Ġexamples Ġare Ġeasy - negative Ġones : Ġeasy - negative Ġexamples Ġcan Ġeasily Ġdominate Ġtraining Ġsince Ġtheir Ġprobabilities Ġcan Ġbe Ġpushed Ġto Ġ0 Ġfairly Ġeasily . ĠMeanwhile , Ġthe Ġmodel Ġcan Ġhardly Ġdistinguish Ġbetween Ġhard - negative Ġexamples Ġand Ġpositive Ġones , Ġwhich Ġhas Ġa Ġhuge Ġnegative Ġeffect Ġon Ġthe Ġfinal ĠF 1 Ġperformance . </s> To Ġaddress Ġthis Ġissue , Ġwe Ġpropose Ġto Ġmultiply Ġthe Ġsoft Ġprobability Ġ$ p $ Ġwith Ġa Ġdecaying Ġfactor Ġ$( 1 - p ) $, Ġchanging ĠE q . DIS PLAY _ FORM 22 Ġto Ġthe Ġfollowing Ġform : </s> One Ġcan Ġthink Ġ$( 1 - p _{ i 1 }) $ Ġas Ġa Ġweight Ġassociated Ġwith Ġeach Ġexample , Ġwhich Ġchanges Ġas Ġtraining Ġproceeds . ĠThe Ġintuition Ġof Ġchanging Ġ$ p _{ i 1 } $ Ġto Ġ$( 1 - p _{ i 1 }) Ġp _{ i 1 } $ Ġis Ġto Ġpush Ġdown Ġthe Ġweight Ġof Ġeasy Ġexamples . ĠFor Ġeasy Ġexamples Ġwhose Ġprobability Ġare Ġapproaching Ġ0 Ġor Ġ1 , Ġ$( 1 - p _{ i 1 }) Ġp _{ i 1 } $ Ġmakes Ġthe Ġmodel Ġattach Ġsignificantly Ġless Ġfocus Ġto Ġthem . ĠFigure ĠFIG REF 23 Ġgives Ġgives Ġan Ġexplanation Ġfrom Ġthe Ġperspective Ġin Ġderivative : Ġthe Ġderivative Ġof Ġ$\ frac { ( 1 - p ) p }{ 1 +( 1 - p ) p } $ Ġwith Ġrespect Ġto Ġ$ p $ Ġapproaches Ġ0 Ġimmediately Ġafter Ġ$ p $ Ġapproaches Ġ0 , Ġwhich Ġmeans Ġthe Ġmodel Ġattends Ġless Ġto Ġexamples Ġonce Ġthey Ġare Ġcorrectly Ġclassified . </s> A Ġclose Ġlook Ġat ĠE q . DIS PLAY _ FORM 14 Ġreveals Ġthat Ġit Ġactually Ġmim ics Ġthe Ġidea Ġof Ġfocal Ġloss Ġ( FL Ġfor Ġshort ) ĠB IB REF 16 Ġfor Ġobject Ġdetection Ġin Ġvision . ĠF ocal Ġloss Ġwas Ġproposed Ġfor Ġone - stage Ġobject Ġdetector Ġto Ġhandle Ġforeground - background Ġtrade off Ġencountered Ġduring Ġtraining . ĠIt Ġdown - weights Ġthe Ġloss Ġassigned Ġto Ġwell - classified Ġexamples Ġby Ġadding Ġa Ġ$( 1 - p ) ^ {\ beta Ġ} $ Ġfactor , Ġleading Ġthe Ġfinal Ġloss Ġto Ġbe Ġ$( 1 - p ) ^ {\ beta Ġ} \ log Ġp $. </s> In ĠTable ĠT AB REF 18 , Ġwe Ġshow Ġthe Ġlosses Ġused Ġin Ġour Ġexperiments , Ġwhich Ġis Ġdescribed Ġin Ġthe Ġnext Ġsection . </s> Exper iments </s> We Ġevaluate Ġthe Ġproposed Ġmethod Ġon Ġfour ĠN LP Ġtasks : Ġpart - of - speech Ġtagging , Ġnamed Ġentity Ġrecognition , Ġmachine Ġreading Ġcomprehension Ġand Ġparaph rase Ġidentification . ĠBas elines Ġin Ġour Ġexperiments Ġare Ġoptimized Ġby Ġusing Ġthe Ġstandard Ġcross - ent ropy Ġtraining Ġobjective . </s> Exper iments Ġ: :: ĠPart - of - Spe ech ĠT agging </s> Part - of - speech Ġtagging Ġ( POS ) Ġis Ġthe Ġtask Ġof Ġassigning Ġa Ġlabel Ġ( e . g ., Ġnoun , Ġverb , Ġadjective ) Ġto Ġeach Ġword Ġin Ġa Ġgiven Ġtext . ĠIn Ġthis Ġpaper , Ġwe Ġchoose ĠB ERT Ġas Ġthe Ġbackbone Ġand Ġconduct Ġexperiments Ġon Ġthree ĠChinese ĠPOS Ġdatasets . ĠWe Ġreport Ġthe Ġspan - level Ġmicro - aver aged Ġprecision , Ġrecall Ġand ĠF 1 Ġfor Ġevaluation . ĠHyper param eters Ġare Ġtuned Ġon Ġthe Ġcorresponding Ġdevelopment Ġset Ġof Ġeach Ġdataset . </s> Exper iments Ġ: :: ĠPart - of - Spe ech ĠT agging Ġ: :: ĠDat as ets </s> We Ġconduct Ġexperiments Ġon Ġthe Ġwidely Ġused ĠChinese ĠTree bank Ġ5 . 0 , Ġ6 . 0 Ġas Ġwell Ġas ĠUD 1 . 4 . </s> CT B 5 Ġis Ġa ĠChinese Ġdataset Ġfor Ġtagging Ġand Ġparsing , Ġwhich Ġcontains Ġ50 7 , 222 Ġwords , Ġ8 24 , 98 3 Ġcharacters Ġand Ġ18 , 782 Ġsentences Ġextracted Ġfrom Ġnews wire Ġsources . </s> CT B 6 Ġis Ġan Ġextension Ġof ĠCT B 5 , Ġcontaining Ġ7 81 , 351 Ġwords , Ġ1 , 285 , 149 Ġcharacters Ġand Ġ28 , 295 Ġsentences . </s> UD Ġis Ġthe Ġabbre viation Ġof ĠUniversal ĠDepend encies , Ġwhich Ġis Ġa Ġframework Ġfor Ġconsistent Ġannotation Ġof Ġgrammar Ġ( parts Ġof Ġspeech , Ġmorph ological Ġfeatures , Ġand Ġsynt actic Ġdependencies ) Ġacross Ġdifferent Ġhuman Ġlanguages . ĠIn Ġthis Ġwork , Ġwe Ġuse ĠUD 1 . 4 Ġfor ĠChinese ĠPOS Ġtagging . </s> Exper iments Ġ: :: ĠPart - of - Spe ech ĠT agging Ġ: :: ĠBas elines </s> We Ġuse Ġthe Ġfollowing Ġbas elines : </s> J oint - POS : Ġsh ao 2017 character Ġjointly Ġlearns ĠChinese Ġword Ġsegment ation Ġand ĠPOS . </s> L att ice - L ST M : Ġlatt ice 2018 zh ang Ġconstructs Ġa Ġword - character Ġlatt ice . </s> B ert - T agger : Ġdev lin 2018 bert Ġtreats Ġpart - of - speech Ġas Ġa Ġtagging Ġtask . </s> Exper iments Ġ: :: ĠPart - of - Spe ech ĠT agging Ġ: :: ĠResults </s> Table Ġpresents Ġthe Ġexperimental Ġresults Ġon Ġthe ĠPOS Ġtask . ĠAs Ġcan Ġbe Ġseen , Ġthe Ġproposed ĠD SC Ġloss Ġoutper forms Ġthe Ġbest Ġbaseline Ġresults Ġby Ġa Ġlarge Ġmargin , Ġi . e ., Ġoutper forming ĠB ERT - t agger Ġby Ġ+ 1 . 86 Ġin Ġterms Ġof ĠF 1 Ġscore Ġon ĠCT B 5 , Ġ+ 1 . 80 Ġon ĠCT B 6 Ġand Ġ+ 2 . 19 Ġon ĠUD 1 . 4 . ĠAs Ġfar Ġas Ġwe Ġare Ġconcerned , Ġwe Ġare Ġachieving ĠS OTA Ġperformances Ġon Ġthe Ġthree Ġdatasets . ĠWeight ed Ġcross Ġentropy Ġand Ġfocal Ġloss Ġonly Ġgain Ġa Ġlittle Ġperformance Ġimprovement Ġon ĠCT B 5 Ġand ĠCT B 6 , Ġand Ġthe Ġdice Ġloss Ġob tains Ġhuge Ġgain Ġon ĠCT B 5 Ġbut Ġnot Ġon ĠCT B 6 , Ġwhich Ġindicates Ġthe Ġthree Ġlosses Ġare Ġnot Ġconsistently Ġrobust Ġin Ġresolving Ġthe Ġdata Ġimbalance Ġissue . ĠThe Ġproposed ĠD SC Ġloss Ġperforms Ġrobust ly Ġon Ġall Ġthe Ġthree Ġdatasets . </s> Exper iments Ġ: :: ĠNamed ĠEntity ĠRecogn ition </s> N amed Ġentity Ġrecognition Ġ( NER ) Ġrefers Ġto Ġthe Ġtask Ġof Ġdetecting Ġthe Ġspan Ġand Ġsemantic Ġcategory Ġof Ġentities Ġfrom Ġa Ġchunk Ġof Ġtext . ĠOur Ġimplementation Ġuses Ġthe Ġcurrent Ġstate - of - the - art ĠB ERT - M RC Ġmodel Ġproposed Ġby Ġx ia oya 2019 ner Ġas Ġa Ġbackbone . ĠFor ĠEnglish Ġdatasets , Ġwe Ġuse ĠB ERT $ _ \ text { Large } $ ĠEnglish Ġcheckpoints , Ġwhile Ġfor ĠChinese Ġwe Ġuse Ġthe Ġofficial ĠChinese Ġcheckpoints . ĠWe Ġreport Ġspan - level Ġmicro - aver aged Ġprecision , Ġrecall Ġand ĠF 1 - score . ĠHyper param eters Ġare Ġtuned Ġon Ġthe Ġdevelopment Ġset Ġof Ġeach Ġdataset . </s> Exper iments Ġ: :: ĠNamed ĠEntity ĠRecogn ition Ġ: :: ĠDat as ets </s> For Ġthe ĠN ER Ġtask , Ġwe Ġconsider Ġboth ĠChinese Ġdatasets , Ġi . e ., ĠOnt o Notes 4 . 0 ĠB IB REF 34 Ġand ĠMS RA ĠB IB REF 35 , Ġand ĠEnglish Ġdatasets , Ġi . e ., ĠCo N LL 2003 ĠB IB REF 36 Ġand ĠOnt o Notes 5 . 0 ĠB IB REF 37 . </s> Co N LL 2003 Ġis Ġan ĠEnglish Ġdataset Ġwith Ġ4 Ġentity Ġtypes : ĠLocation , ĠOrganization , ĠPerson Ġand ĠMiscellaneous . ĠWe Ġfollowed Ġdata Ġprocessing Ġprotocols Ġin ĠB IB REF 14 . </s> English ĠOnt o Notes 5 . 0 Ġconsists Ġof Ġtexts Ġfrom Ġa Ġwide Ġvariety Ġof Ġsources Ġand Ġcontains Ġ18 Ġentity Ġtypes . ĠWe Ġuse Ġthe Ġstandard Ġtrain / dev / test Ġsplit Ġof ĠCo N LL 2012 Ġshared Ġtask . </s> Chinese ĠMS RA Ġperforms Ġas Ġa ĠChinese Ġbenchmark Ġdataset Ġcontaining Ġ3 Ġentity Ġtypes . ĠData Ġin ĠMS RA Ġis Ġcollected Ġfrom Ġnews Ġdomain . ĠSince Ġthe Ġdevelopment Ġset Ġis Ġnot Ġprovided Ġin Ġthe Ġoriginal ĠMS RA Ġdataset , Ġwe Ġrandomly Ġsplit Ġthe Ġtraining Ġset Ġinto Ġtraining Ġand Ġdevelopment Ġsplits Ġby Ġ9 : 1 . ĠWe Ġuse Ġthe Ġofficial Ġtest Ġset Ġfor Ġevaluation . </s> Chinese ĠOnt o Notes 4 . 0 Ġis Ġa ĠChinese Ġdataset Ġand Ġconsists Ġof Ġtexts Ġfrom Ġnews Ġdomain , Ġwhich Ġhas Ġ18 Ġentity Ġtypes . ĠIn Ġthis Ġpaper , Ġwe Ġtake Ġthe Ġsame Ġdata Ġsplit Ġas Ġw u 2019 gly ce Ġdid . </s> Exper iments Ġ: :: ĠNamed ĠEntity ĠRecogn ition Ġ: :: ĠBas elines </s> We Ġuse Ġthe Ġfollowing Ġbas elines : </s> EL Mo : Ġa Ġtagging Ġmodel Ġfrom Ġpet ers 2018 deep . </s> L att ice - L ST M : Ġlatt ice 2018 zh ang Ġconstructs Ġa Ġword - character Ġlatt ice , Ġonly Ġused Ġin ĠChinese Ġdatasets . </s> CV T : Ġfrom Ġke vin 2018 cross , Ġwhich Ġuses ĠCross - View ĠTraining ( CV T ) Ġto Ġimprove Ġthe Ġrepresentations Ġof Ġa ĠBi - L ST M Ġenc oder . </s> B ert - T agger : Ġdev lin 2018 bert Ġtreats ĠN ER Ġas Ġa Ġtagging Ġtask . </s> G ly ce - BER T : Ġw u 2019 gly ce Ġcombines Ġglyph Ġinformation Ġwith ĠB ERT Ġpret raining . </s> BER T - M RC : ĠThe Ġcurrent ĠS OTA Ġmodel Ġfor Ġboth ĠChinese Ġand ĠEnglish ĠN ER Ġdatasets Ġproposed Ġby Ġx ia oya 2019 ner , Ġwhich Ġformulate ĠN ER Ġas Ġmachine Ġreading Ġcomprehension Ġtask . </s> Exper iments Ġ: :: ĠNamed ĠEntity ĠRecogn ition Ġ: :: ĠResults </s> Table Ġshows Ġexperimental Ġresults Ġon ĠN ER Ġdatasets . ĠFor ĠEnglish Ġdatasets Ġincluding ĠCo N LL 2003 Ġand ĠOnt o Notes 5 . 0 , Ġour Ġproposed Ġmethod Ġoutper forms ĠB ERT - M RC BI B REF 38 Ġby Ġ+ 0 . 29 Ġand Ġ+ 0 . 96 Ġrespectively . ĠWe Ġobserve Ġhuge Ġperformance Ġboosts Ġon ĠChinese Ġdatasets , Ġachieving ĠF 1 Ġimprovements Ġby Ġ+ 0 . 97 Ġand Ġ+ 2 . 36 Ġon ĠMS RA Ġand ĠOnt o Notes 4 . 0 , Ġrespectively . ĠAs Ġfar Ġas Ġwe Ġare Ġconcerned , Ġwe Ġare Ġsetting Ġnew ĠS OTA Ġperformances Ġon Ġall Ġof Ġthe Ġfour ĠN ER Ġdatasets . </s> Exper iments Ġ: :: ĠMachine ĠReading ĠComp rehens ion </s> Machine Ġreading Ġcomprehension Ġ( M RC ) ĠB IB REF 39 , ĠB IB REF 40 , ĠB IB REF 41 , ĠB IB REF 40 , ĠB IB REF 42 , ĠB IB REF 15 Ġhas Ġbecome Ġa Ġcentral Ġtask Ġin Ġnatural Ġlanguage Ġunderstanding . ĠM RC Ġin Ġthe ĠS Qu AD - style Ġis Ġto Ġpredict Ġthe Ġanswer Ġspan Ġin Ġthe Ġpassage Ġgiven Ġa Ġquestion Ġand Ġthe Ġpassage . ĠIn Ġthis Ġpaper , Ġwe Ġchoose Ġthe ĠS Qu AD - style ĠM RC Ġtask Ġand Ġreport ĠExtract ĠMatch Ġ( EM ) Ġin Ġaddition Ġto ĠF 1 Ġscore Ġon Ġvalidation Ġset . ĠAll Ġhyper param eters Ġare Ġtuned Ġon Ġthe Ġdevelopment Ġset Ġof Ġeach Ġdataset . </s> Exper iments Ġ: :: ĠMachine ĠReading ĠComp rehens ion Ġ: :: ĠDat as ets </s> The Ġfollowing Ġfive Ġdatasets Ġare Ġused Ġfor ĠM RC Ġtask : ĠS Qu AD Ġv 1 . 1 , ĠS Qu AD Ġv 2 . 0 ĠB IB REF 4 , ĠB IB REF 6 Ġand ĠQu ore f ĠB IB REF 8 . </s> S Qu AD Ġv 1 . 1 Ġand ĠS Qu AD Ġv 2 . 0 Ġare Ġthe Ġmost Ġwidely Ġused ĠQ A Ġbenchmarks . ĠS Qu AD 1 . 1 Ġis Ġa Ġcollection Ġof Ġ100 K Ġcrowds ourced Ġquestion - answer Ġpairs , Ġand ĠS Qu AD 2 . 0 Ġextends ĠS Qu AD 1 . 1 Ġallowing Ġno Ġshort Ġanswer Ġexists Ġin Ġthe Ġprovided Ġpassage . </s> Qu ore f Ġis Ġa ĠQ A Ġdataset Ġwhich Ġtests Ġthe Ġcore f erential Ġreasoning Ġcapability Ġof Ġreading Ġcomprehension Ġsystems , Ġcontaining Ġ24 K Ġquestions Ġover Ġ4 . 7 K Ġparagraphs Ġfrom ĠWikipedia . </s> Exper iments Ġ: :: ĠMachine ĠReading ĠComp rehens ion Ġ: :: ĠBas elines </s> We Ġuse Ġthe Ġfollowing Ġbas elines : </s> Q AN et : Ġq an et 2018 Ġbuilds Ġa Ġmodel Ġbased Ġon Ġconv olutions Ġand Ġself - att ention . ĠConv olution Ġto Ġmodel Ġlocal Ġinteractions Ġand Ġself - att ention Ġto Ġmodel Ġglobal Ġinteractions . </s> BER T : Ġdev lin 2018 bert Ġtreats ĠN ER Ġas Ġa Ġtagging Ġtask . </s> XL Net : Ġx l net 2019 Ġproposes Ġa Ġgeneralized Ġaut ore gressive Ġpret raining Ġmethod Ġthat Ġenables Ġlearning Ġbid irection al Ġcontexts . </s> Exper iments Ġ: :: ĠMachine ĠReading ĠComp rehens ion Ġ: :: ĠResults </s> Table Ġshows Ġthe Ġexperimental Ġresults Ġfor ĠM RC Ġtasks . ĠWith Ġeither ĠB ERT Ġor ĠXL Net , Ġour Ġproposed ĠD SC Ġloss Ġob tains Ġsignificant Ġperformance Ġboost Ġon Ġboth ĠEM Ġand ĠF 1 . ĠFor ĠS Qu AD v 1 . 1 , Ġour Ġproposed Ġmethod Ġoutper forms ĠXL Net Ġby Ġ+ 1 . 25 Ġin Ġterms Ġof ĠF 1 Ġscore Ġand Ġ+ 0 . 84 Ġin Ġterms Ġof ĠEM Ġand Ġachieves Ġ87 . 65 Ġon ĠEM Ġand Ġ89 . 51 Ġon ĠF 1 Ġfor ĠS Qu AD Ġv 2 . 0 . ĠMoreover , Ġon ĠQu o Ref , Ġthe Ġproposed Ġmethod Ġsurpass es ĠXL Net Ġresults Ġby Ġ+ 1 . 46 Ġon ĠEM Ġand Ġ+ 1 . 41 Ġon ĠF 1 . ĠAnother Ġobservation Ġis Ġthat , ĠXL Net Ġoutper forms ĠB ERT Ġby Ġa Ġhuge Ġmargin , Ġand Ġthe Ġproposed ĠD SC Ġloss Ġcan Ġobtain Ġfurther Ġperformance Ġimprovement Ġby Ġan Ġaverage Ġscore Ġabove Ġ1 . 0 Ġin Ġterms Ġof Ġboth ĠEM Ġand ĠF 1 , Ġwhich Ġindicates Ġthe ĠD SC Ġloss Ġis Ġcomplementary Ġto Ġthe Ġmodel Ġstructures . </s> Exper iments Ġ: :: ĠPar aph rase ĠIdentification </s> Par aph r ases Ġare Ġtextual Ġexpressions Ġthat Ġhave Ġthe Ġsame Ġsemantic Ġmeaning Ġusing Ġdifferent Ġsurface Ġwords . ĠPar aph rase Ġidentification Ġ( PI ) Ġis Ġthe Ġtask Ġof Ġidentifying Ġwhether Ġtwo Ġsentences Ġhave Ġthe Ġsame Ġmeaning Ġor Ġnot . ĠWe Ġuse ĠB ERT ĠB IB REF 11 Ġand ĠXL Net ĠB IB REF 43 Ġas Ġback bones Ġand Ġreport ĠF 1 Ġscore Ġfor Ġcomparison . ĠHyper param eters Ġare Ġtuned Ġon Ġthe Ġdevelopment Ġset Ġof Ġeach Ġdataset . </s> Exper iments Ġ: :: ĠPar aph rase ĠIdentification Ġ: :: ĠDat as ets </s> We Ġconduct Ġexperiments Ġon Ġtwo Ġwidely Ġused Ġdatasets Ġfor ĠPI Ġtask : ĠMR PC ĠB IB REF 44 Ġand ĠQ Q P . </s> MR PC Ġis Ġa Ġcorpus Ġof Ġsentence Ġpairs Ġautomatically Ġextracted Ġfrom Ġonline Ġnews Ġsources , Ġwith Ġhuman Ġannotations Ġof Ġwhether Ġthe Ġsentence Ġpairs Ġare Ġsem antically Ġequivalent . ĠThe ĠMR PC Ġdataset Ġhas Ġim balanced Ġclasses Ġ( 68 % Ġpositive , Ġ32 % Ġfor Ġnegative ). </s> Q Q P Ġis Ġa Ġcollection Ġof Ġquestion Ġpairs Ġfrom Ġthe Ġcommunity Ġquestion - ans w ering Ġwebsite ĠQu ora . ĠThe Ġclass Ġdistribution Ġin ĠQ Q P Ġis Ġalso Ġun balanced Ġ( 37 % Ġpositive , Ġ63 % Ġnegative ). </s> Exper iments Ġ: :: ĠPar aph rase ĠIdentification Ġ: :: ĠResults </s> Table Ġshows Ġthe Ġresults Ġfor ĠPI Ġtask . ĠWe Ġfind Ġthat Ġreplacing Ġthe Ġtraining Ġobjective Ġwith ĠD SC Ġintroduces Ġperformance Ġboost Ġfor Ġboth ĠB ERT Ġand ĠXL Net . ĠUsing ĠD SC Ġloss Ġimproves Ġthe ĠF 1 Ġscore Ġby Ġ+ 0 . 58 Ġfor ĠMR PC Ġand Ġ+ 0 . 73 Ġfor ĠQ Q P . </s> A bl ation ĠStudies Ġ: :: ĠThe ĠEffect Ġof ĠDice ĠLoss Ġon ĠAccuracy - oriented ĠT asks </s> We Ġargue Ġthat Ġthe Ġmost Ġcommonly Ġused Ġcross - ent ropy Ġobjective Ġis Ġactually Ġaccuracy - oriented , Ġwhereas Ġthe Ġproposed Ġdice Ġloss Ġ( DL ) Ġperforms Ġas Ġa Ġhard Ġversion Ġof ĠF 1 - score . ĠTo Ġexplore Ġthe Ġeffect Ġof Ġthe Ġdice Ġloss Ġon Ġaccuracy - oriented Ġtasks Ġsuch Ġas Ġtext Ġclassification , Ġwe Ġconduct Ġexperiments Ġon Ġthe ĠStanford ĠSent iment ĠTree bank Ġsentiment Ġclassification Ġdatasets Ġincluding ĠS ST - 2 Ġand ĠS ST - 5 . ĠWe Ġfine - t une ĠB ERT $ _ \ text { Large } $ Ġwith Ġdifferent Ġtraining Ġobjectives . ĠExperiment Ġresults Ġfor ĠS ST Ġare Ġshown Ġin Ġ. ĠFor ĠS ST - 5 , ĠB ERT Ġwith ĠCE Ġachieves Ġ55 . 57 Ġin Ġterms Ġof Ġaccuracy , Ġwith ĠDL Ġand ĠD SC Ġlosses Ġslightly Ġdegrade Ġthe Ġaccuracy Ġperformance Ġand Ġachieve Ġ54 . 63 Ġand Ġ55 . 19 , Ġrespectively . ĠFor ĠS ST - 2 , ĠB ERT Ġwith ĠCE Ġachieves Ġ94 . 9 Ġin Ġterms Ġof Ġaccuracy . ĠThe Ġsame Ġas ĠS ST - 5 , Ġwe Ġobserve Ġa Ġslight Ġperformance Ġdrop Ġwith ĠDL Ġand ĠD SC , Ġwhich Ġmeans Ġthat Ġthe Ġdice Ġloss Ġactually Ġworks Ġwell Ġfor ĠF 1 Ġbut Ġnot Ġfor Ġaccuracy . </s> A bl ation ĠStudies Ġ: :: ĠThe ĠEffect Ġof ĠHyper param eters Ġin ĠT vers ky Ġindex </s> As Ġmentioned Ġin ĠSection ĠSEC REF 10 , ĠT vers ky Ġindex Ġ( TI ) Ġoffers Ġthe Ġflexibility Ġin Ġcontrolling Ġthe Ġtrade off Ġbetween Ġfalse - neg atives Ġand Ġfalse - pos itives . ĠIn Ġthis Ġsubsection , Ġwe Ġexplore Ġthe Ġeffect Ġof Ġhyper param eters Ġ( i . e ., Ġ$\ alpha Ġ$ Ġand Ġ$\ beta Ġ$ ) Ġin ĠTI Ġto Ġtest Ġhow Ġthey Ġmanipulate Ġthe Ġtrade off . ĠWe Ġconduct Ġexperiments Ġon Ġthe ĠChinese ĠOnt o Notes 4 . 0 ĠN ER Ġdataset Ġand ĠEnglish ĠQu o Ref ĠM RC Ġdataset Ġto Ġexamine Ġthe Ġinfluence Ġof Ġtrade off Ġbetween Ġprecision Ġand Ġrecall . ĠExperiment Ġresults Ġare Ġshown Ġin ĠTable Ġ. ĠThe Ġhighest ĠF 1 Ġfor ĠChinese ĠOnt o Notes 4 . 0 Ġis Ġ84 . 67 Ġwhen Ġ$\ alpha Ġ$ Ġis Ġset Ġto Ġ0 . 6 Ġwhile Ġfor ĠQu o Ref , Ġthe Ġhighest ĠF 1 Ġis Ġ68 . 44 Ġwhen Ġ$\ alpha Ġ$ Ġis Ġset Ġto Ġ0 . 4 . ĠIn Ġaddition , Ġwe Ġcan Ġobserve Ġthat Ġthe Ġperformance Ġvaries Ġa Ġlot Ġas Ġ$\ alpha Ġ$ Ġchanges Ġin Ġdistinct Ġdatasets , Ġwhich Ġshows Ġthat Ġthe Ġhyper param eters Ġ$\ alpha Ġ, \ beta Ġ$ Ġplay Ġan Ġimportant Ġrole Ġin Ġthe Ġproposed Ġmethod . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġalleviate Ġthe Ġsevere Ġdata Ġimbalance Ġissue Ġin ĠN LP Ġtasks . ĠWe Ġpropose Ġto Ġuse Ġdice Ġloss Ġin Ġreplacement Ġof Ġthe Ġstandard Ġcross - ent ropy Ġloss , Ġwhich Ġperforms Ġas Ġa Ġsoft Ġversion Ġof ĠF 1 Ġscore . ĠUsing Ġdice Ġloss Ġcan Ġhelp Ġnarrow Ġthe Ġgap Ġbetween Ġtraining Ġobjectives Ġand Ġevaluation Ġmetrics . ĠEm pir ically , Ġwe Ġshow Ġthat Ġthe Ġproposed Ġtraining Ġobjective Ġleads Ġto Ġsignificant Ġperformance Ġboost Ġfor Ġpart - of - speech , Ġnamed Ġentity Ġrecognition , Ġmachine Ġreading Ġcomprehension Ġand Ġparaph rase Ġidentification Ġtasks .