closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation
EXTRACTIVE
closed test limits all the data for learning should not be beyond the given training set
EXTRACTIVE
External data and pre-trained embedding. Whereas both encoder and graph model are about exploring a way to get better performance only by improving the model strength itself. Using external resource such as pre-trained embeddings or language representation is an alternative for the same purpose BIBREF22, BIBREF23. SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21. In this work, we will focus on the closed test setting by finding a better model design for further CWS performance improvement.
External data and pre-trained embedding. Whereas both encoder and graph model are about exploring a way to get better performance only by improving the model strength itself. Using external resource such as pre-trained embeddings or language representation is an alternative for the same purpose BIBREF22, BIBREF23. SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21. In this work, we will focus on the closed test setting by finding a better model design for further CWS performance improvement.
1910.14537
Introduction </s> Chinese Ġword Ġsegment ation Ġ( C WS ) Ġis Ġa Ġtask Ġfor ĠChinese Ġnatural Ġlanguage Ġprocess Ġto Ġdelim it Ġword Ġboundary . ĠC WS Ġis Ġa Ġbasic Ġand Ġessential Ġtask Ġfor ĠChinese Ġwhich Ġis Ġwritten Ġwithout Ġexplicit Ġword Ġdelim it ers Ġand Ġdifferent Ġfrom Ġalphabet ical Ġlanguages Ġlike ĠEnglish . ĠB IB REF 0 Ġtreats ĠChinese Ġword Ġsegment ation Ġ( C WS ) Ġas Ġa Ġsequence Ġlabeling Ġtask Ġwith Ġcharacter Ġposition Ġtags , Ġwhich Ġis Ġfollowed Ġby ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 . ĠTraditional ĠC WS Ġmodels Ġdepend Ġon Ġthe Ġdesign Ġof Ġfeatures Ġheavily Ġwhich Ġeffects Ġthe Ġperformance Ġof Ġmodel . ĠTo Ġminimize Ġthe Ġeffort Ġin Ġfeature Ġengineering , Ġsome ĠC WS Ġmodels ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 Ġare Ġdeveloped Ġfollowing Ġneural Ġnetwork Ġarchitecture Ġfor Ġsequence Ġlabeling Ġtasks ĠB IB REF 12 . ĠNeural ĠC WS Ġmodels Ġperform Ġstrong Ġability Ġof Ġfeature Ġrepresentation , Ġemploying Ġun ig ram Ġand Ġbig ram Ġcharacter Ġembed ding Ġas Ġinput Ġand Ġapproach Ġgood Ġperformance . </s> The ĠC WS Ġtask Ġis Ġoften Ġmodeled Ġas Ġone Ġgraph Ġmodel Ġbased Ġon Ġa Ġscoring Ġmodel Ġthat Ġmeans Ġit Ġis Ġcomposed Ġof Ġtwo Ġparts , Ġone Ġpart Ġis Ġan Ġenc oder Ġwhich Ġis Ġused Ġto Ġgenerate Ġthe Ġrepresentation Ġof Ġcharacters Ġfrom Ġthe Ġinput Ġsequence , Ġthe Ġother Ġpart Ġis Ġa Ġdec oder Ġwhich Ġperforms Ġsegment ation Ġaccording Ġto Ġthe Ġenc oder Ġscoring . ĠTable ĠT AB REF 1 Ġsummarizes Ġtypical ĠC WS Ġmodels Ġaccording Ġto Ġtheir Ġdecoding Ġways Ġfor Ġboth Ġtraditional Ġand Ġneural Ġmodels . ĠMark ov Ġmodels Ġsuch Ġas ĠB IB REF 13 Ġand ĠB IB REF 4 Ġdepend Ġon Ġthe Ġmaximum Ġentropy Ġmodel Ġor Ġmaximum Ġentropy ĠMark ov Ġmodel Ġboth Ġwith Ġa ĠV iter bi Ġdec oder . ĠBesides , Ġconditional Ġrandom Ġfield Ġ( CR F ) Ġor ĠSemi - CR F Ġfor Ġsequence Ġlabeling Ġhas Ġbeen Ġused Ġfor Ġboth Ġtraditional Ġand Ġneural Ġmodels Ġthough Ġwith Ġdifferent Ġrepresentations ĠB IB REF 2 , ĠB IB REF 15 , ĠB IB REF 10 , ĠB IB REF 17 , ĠB IB REF 18 . ĠGenerally Ġspeaking , Ġthe Ġmajor Ġdifference Ġbetween Ġtraditional Ġand Ġneural Ġnetwork Ġmodels Ġis Ġabout Ġthe Ġway Ġto Ġrepresent Ġinput Ġsentences . </s> Recent Ġworks Ġabout Ġneural ĠC WS Ġwhich Ġfocus Ġon Ġbenchmark Ġdataset , Ġnamely ĠS IGH AN ĠBake off ĠB IB REF 21 , Ġmay Ġbe Ġput Ġinto Ġthe Ġfollowing Ġthree Ġcategories Ġroughly . </s> Enc oder . ĠPractice Ġin Ġvarious Ġnatural Ġlanguage Ġprocessing Ġtasks Ġhas Ġbeen Ġshown Ġthat Ġeffective Ġrepresentation Ġis Ġessential Ġto Ġthe Ġperformance Ġimprovement . ĠThus Ġfor Ġbetter ĠC WS , Ġit Ġis Ġcrucial Ġto Ġencode Ġthe Ġinput Ġcharacter , Ġword Ġor Ġsentence Ġinto Ġeffective Ġrepresentation . ĠTable ĠT AB REF 2 Ġsummarizes Ġregular Ġfeature Ġsets Ġfor Ġtypical ĠC WS Ġmodels Ġincluding Ġours Ġas Ġwell . ĠThe Ġbuilding Ġblocks Ġthat Ġenc od ers Ġuse Ġinclude Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġand Ġconv olution al Ġneural Ġnetwork Ġ( CNN ), Ġand Ġlong - term Ġmemory Ġnetwork Ġ( L ST M ). </s> Graph Ġmodel . ĠAs ĠC WS Ġis Ġa Ġkind Ġof Ġstructure Ġlearning Ġtask , Ġthe Ġgraph Ġmodel Ġdetermines Ġwhich Ġtype Ġof Ġdec oder Ġshould Ġbe Ġadopted Ġfor Ġsegment ation , Ġalso Ġit Ġmay Ġlimit Ġthe Ġcapability Ġof Ġdefining Ġfeature , Ġas Ġshown Ġin ĠTable Ġ2 , Ġnot Ġall Ġgraph Ġmodels Ġcan Ġsupport Ġthe Ġword Ġfeatures . ĠThus Ġrecent Ġwork Ġfocused Ġon Ġfinding Ġmore Ġgeneral Ġor Ġflexible Ġgraph Ġmodel Ġto Ġmake Ġmodel Ġlearn Ġthe Ġrepresentation Ġof Ġsegment ation Ġmore Ġeffective Ġas ĠB IB REF 9 , ĠB IB REF 11 . </s> External Ġdata Ġand Ġpre - trained Ġembed ding . ĠWhereas Ġboth Ġenc oder Ġand Ġgraph Ġmodel Ġare Ġabout Ġexploring Ġa Ġway Ġto Ġget Ġbetter Ġperformance Ġonly Ġby Ġimproving Ġthe Ġmodel Ġstrength Ġitself . ĠUsing Ġexternal Ġresource Ġsuch Ġas Ġpre - trained Ġembed d ings Ġor Ġlanguage Ġrepresentation Ġis Ġan Ġalternative Ġfor Ġthe Ġsame Ġpurpose ĠB IB REF 22 , ĠB IB REF 23 . ĠS IGH AN ĠBake off Ġdefines Ġtwo Ġtypes Ġof Ġevaluation Ġsettings , Ġclosed Ġtest Ġlimits Ġall Ġthe Ġdata Ġfor Ġlearning Ġshould Ġnot Ġbe Ġbeyond Ġthe Ġgiven Ġtraining Ġset , Ġwhile Ġopen Ġtest Ġdoes Ġnot Ġtake Ġthis Ġlimitation ĠB IB REF 21 . ĠIn Ġthis Ġwork , Ġwe Ġwill Ġfocus Ġon Ġthe Ġclosed Ġtest Ġsetting Ġby Ġfinding Ġa Ġbetter Ġmodel Ġdesign Ġfor Ġfurther ĠC WS Ġperformance Ġimprovement . </s> Sh own Ġin ĠTable ĠT AB REF 1 , Ġdifferent Ġdec od ers Ġhave Ġparticular Ġdecoding Ġalgorithms Ġto Ġmatch Ġthe Ġrespective ĠC WS Ġmodels . ĠMark ov Ġmodels Ġand ĠCR F - based Ġmodels Ġoften Ġuse ĠV iter bi Ġdec od ers Ġwith Ġpo lyn omial Ġtime Ġcomplexity . ĠIn Ġgeneral Ġgraph Ġmodel , Ġsearch Ġspace Ġmay Ġbe Ġtoo Ġlarge Ġfor Ġmodel Ġto Ġsearch . ĠThus Ġit Ġforces Ġgraph Ġmodels Ġto Ġuse Ġan Ġapproximate Ġbeam Ġsearch Ġstrategy . ĠBeam Ġsearch Ġalgorithm Ġhas Ġa Ġkind Ġlow - order Ġpo lyn omial Ġtime Ġcomplexity . ĠEspecially , Ġwhen Ġbeam Ġwidth Ġ$ b $ = 1 , Ġthe Ġbeam Ġsearch Ġalgorithm Ġwill Ġreduce Ġto Ġgreedy Ġalgorithm Ġwith Ġa Ġbetter Ġtime Ġcomplexity Ġ$ O ( M n ) $ Ġagainst Ġthe Ġgeneral Ġbeam Ġsearch Ġtime Ġcomplexity Ġ$ O ( M nb ^ 2 ) $, Ġwhere Ġ$ n $ Ġis Ġthe Ġnumber Ġof Ġunits Ġin Ġone Ġsentences , Ġ$ M $ Ġis Ġa Ġconstant Ġrepresenting Ġthe Ġmodel Ġcomplexity . ĠGre edy Ġdecoding Ġalgorithm Ġcan Ġbring Ġthe Ġfastest Ġspeed Ġof Ġdecoding Ġwhile Ġit Ġis Ġnot Ġeasy Ġto Ġguarantee Ġthe Ġprecision Ġof Ġdecoding Ġwhen Ġthe Ġenc oder Ġis Ġnot Ġstrong Ġenough . </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġmore Ġeffective Ġenc oder Ġdesign Ġwhich Ġis Ġcapable Ġof Ġoffering Ġfast Ġand Ġaccurate ĠChinese Ġword Ġsegment ation Ġwith Ġonly Ġun ig ram Ġfeature Ġand Ġgreedy Ġdecoding . ĠOur Ġproposed Ġenc oder Ġwill Ġonly Ġconsist Ġof Ġattention Ġmechanisms Ġas Ġbuilding Ġblocks Ġbut Ġnothing Ġelse . ĠMot ivated Ġby Ġthe ĠTrans former ĠB IB REF 24 Ġand Ġits Ġstrength Ġof Ġcapturing Ġlong - range Ġdependencies Ġof Ġinput Ġsentences , Ġwe Ġuse Ġa Ġself - att ention Ġnetwork Ġto Ġgenerate Ġthe Ġrepresentation Ġof Ġinput Ġwhich Ġmakes Ġthe Ġmodel Ġencode Ġsentences Ġat Ġonce Ġwithout Ġfeeding Ġinput Ġiter atively . ĠConsidering Ġthe Ġweakness Ġof Ġthe ĠTrans former Ġto Ġmodel Ġrelative Ġand Ġabsolute Ġposition Ġinformation Ġdirectly ĠB IB REF 25 Ġand Ġthe Ġimportance Ġof Ġlocal ness Ġinformation , Ġposition Ġinformation Ġand Ġdirectional Ġinformation Ġfor ĠC WS , Ġwe Ġfurther Ġimprove Ġthe Ġarchitecture Ġof Ġstandard Ġmulti - head Ġself - att ention Ġof Ġthe ĠTrans former Ġwith Ġa Ġdirectional ĠGa ussian Ġmask Ġand Ġget Ġa Ġvariant Ġcalled ĠGa ussian - mask ed Ġdirectional Ġmulti - head Ġattention . ĠBased Ġon Ġthe Ġnewly Ġimproved Ġattention Ġmechanism , Ġwe Ġexpand Ġthe Ġenc oder Ġof Ġthe ĠTrans former Ġto Ġcapture Ġdifferent Ġdirectional Ġinformation . ĠWith Ġour Ġpowerful Ġenc oder , Ġour Ġmodel Ġuses Ġonly Ġsimple Ġun ig ram Ġfeatures Ġto Ġgenerate Ġrepresentation Ġof Ġsentences . </s> For Ġdec oder Ġwhich Ġdirectly Ġperforms Ġthe Ġsegment ation , Ġwe Ġuse Ġthe Ġbi - aff inal Ġattention Ġscorer , Ġwhich Ġhas Ġbeen Ġused Ġin Ġdependency Ġparsing ĠB IB REF 26 Ġand Ġsemantic Ġrole Ġlabeling ĠB IB REF 27 , Ġto Ġimplement Ġgreedy Ġdecoding Ġon Ġfinding Ġthe Ġboundaries Ġof Ġwords . ĠIn Ġour Ġproposed Ġmodel , Ġgreedy Ġdecoding Ġensures Ġa Ġfast Ġsegment ation Ġwhile Ġpowerful Ġenc oder Ġdesign Ġensures Ġa Ġgood Ġenough Ġsegment ation Ġperformance Ġeven Ġworking Ġwith Ġgreedy Ġdec oder Ġtogether . ĠOur Ġmodel Ġwill Ġbe Ġstrictly Ġevaluated Ġon Ġbenchmark Ġdatasets Ġfrom ĠS IGH AN ĠBake off Ġshared Ġtask Ġon ĠC WS Ġin Ġterms Ġof Ġclosed Ġtest Ġsetting , Ġand Ġthe Ġexperimental Ġresults Ġshow Ġthat Ġour Ġproposed Ġmodel Ġachieves Ġnew Ġstate - of - the - art . </s> The Ġtechnical Ġcontributions Ġof Ġthis Ġpaper Ġcan Ġbe Ġsummarized Ġas Ġfollows . </s> We Ġpropose Ġa ĠC WS Ġmodel Ġwith Ġonly Ġattention Ġstructure . ĠThe Ġenc oder Ġand Ġdec oder Ġare Ġboth Ġbased Ġon Ġattention Ġstructure . </s> With Ġa Ġpowerful Ġenough Ġenc oder , Ġwe Ġfor Ġthe Ġfirst Ġtime Ġshow Ġthat Ġun ig ram Ġ( character ) Ġfeat ues Ġcan Ġhelp Ġyield Ġstrong Ġperformance Ġinstead Ġof Ġdiverse Ġ$ n $ - gram Ġ( character Ġand Ġword ) Ġfeatures Ġin Ġmost Ġof Ġprevious Ġwork . </s> To Ġcapture Ġthe Ġrepresentation Ġof Ġlocal ness Ġinformation Ġand Ġdirectional Ġinformation , Ġwe Ġpropose Ġa Ġvariant Ġof Ġdirectional Ġmulti - head Ġself - att ention Ġto Ġfurther Ġenhance Ġthe Ġstate - of - the - art ĠTrans former Ġenc oder . </s> Mod els </s> The ĠC WS Ġtask Ġis Ġoften Ġmod elled Ġas Ġone Ġgraph Ġmodel Ġbased Ġon Ġan Ġenc oder - based Ġscoring Ġmodel . ĠThe Ġmodel Ġfor ĠC WS Ġtask Ġis Ġcomposed Ġof Ġan Ġenc oder Ġto Ġrepresent Ġthe Ġinput Ġand Ġa Ġdec oder Ġbased Ġon Ġthe Ġenc oder Ġto Ġperform Ġactual Ġsegment ation . ĠFigure ĠFIG REF 6 Ġis Ġthe Ġarchitecture Ġof Ġour Ġmodel . ĠThe Ġmodel Ġfeeds Ġsentence Ġinto Ġenc oder . ĠEmb edd ing Ġcaptures Ġthe Ġvector Ġ$ e =( e _ 1 ,... , e _ n ) $ Ġof Ġthe Ġinput Ġcharacter Ġsequences Ġof Ġ$ c =( c _ 1 ,... , c _ n ) $. ĠThe Ġenc oder Ġmaps Ġvector Ġsequences Ġof Ġ$ Ġ{ e } =( e _ 1 , .. , e _ n ) $ Ġto Ġtwo Ġsequences Ġof Ġvector Ġwhich Ġare Ġ$ Ġ{ v ^ b } =( v _ 1 ^ b ,... , v _ n ^ b ) $ Ġand Ġ${ v ^ f } =( v _ 1 ^ f ,... v _ n ^ f ) $ Ġas Ġthe Ġrepresentation Ġof Ġsentences . ĠWith Ġ$ v ^ b $ Ġand Ġ$ v ^ f $, Ġthe Ġbi - aff inal Ġscorer Ġcalculates Ġthe Ġprobability Ġof Ġeach Ġsegment ation Ġgaps Ġand Ġpredicts Ġthe Ġword Ġboundaries Ġof Ġinput . ĠSimilar Ġas Ġthe ĠTrans former , Ġthe Ġenc oder Ġis Ġan Ġattention Ġnetwork Ġwith Ġstacked Ġself - att ention Ġand Ġpoint - wise , Ġfully Ġconnected Ġlayers Ġwhile Ġour Ġenc oder Ġincludes Ġthree Ġindependent Ġdirectional Ġenc od ers . </s> Mod els Ġ: :: ĠEnc oder ĠSt acks </s> In Ġthe ĠTrans former , Ġthe Ġenc oder Ġis Ġcomposed Ġof Ġa Ġstack Ġof ĠN Ġidentical Ġlayers Ġand Ġeach Ġlayer Ġhas Ġone Ġmulti - head Ġself - att ention Ġlayer Ġand Ġone Ġposition - wise Ġfully Ġconnected Ġfeed - forward Ġlayer . ĠOne Ġresidual Ġconnection Ġis Ġaround Ġtwo Ġsub - l ayers Ġand Ġfollowed Ġby Ġlayer Ġnormal ization ĠB IB REF 24 . ĠThis Ġarchitecture Ġprovides Ġthe ĠTrans former Ġa Ġgood Ġability Ġto Ġgenerate Ġrepresentation Ġof Ġsentence . </s> With Ġthe Ġvariant Ġof Ġmulti - head Ġself - att ention , Ġwe Ġdesign Ġa ĠGa ussian - mask ed Ġdirectional Ġenc oder Ġto Ġcapture Ġrepresentation Ġof Ġdifferent Ġdirections Ġto Ġimprove Ġthe Ġability Ġof Ġcapturing Ġthe Ġlocal ness Ġinformation Ġand Ġposition Ġinformation Ġfor Ġthe Ġimportance Ġof Ġadjacent Ġcharacters . ĠOne Ġun id irection al Ġenc oder Ġcan Ġcapture Ġinformation Ġof Ġone Ġparticular Ġdirection . </s> For ĠC WS Ġtasks , Ġone Ġgap Ġof Ġcharacters , Ġwhich Ġis Ġfrom Ġa Ġword Ġboundary , Ġcan Ġdivide Ġone Ġsequence Ġinto Ġtwo Ġparts , Ġone Ġpart Ġin Ġfront Ġof Ġthe Ġgap Ġand Ġone Ġpart Ġin Ġthe Ġrear Ġof Ġit . ĠThe Ġforward Ġenc oder Ġand Ġbackward Ġenc oder Ġare Ġused Ġto Ġcapture Ġinformation Ġof Ġtwo Ġdirections Ġwhich Ġcorrespond Ġto Ġtwo Ġparts Ġdivided Ġby Ġthe Ġgap . </s> One Ġcentral Ġenc oder Ġis Ġparalle led Ġwith Ġforward Ġand Ġbackward Ġenc od ers Ġto Ġcapture Ġthe Ġinformation Ġof Ġentire Ġsentences . ĠThe Ġcentral Ġenc oder Ġis Ġa Ġspecial Ġdirectional Ġenc oder Ġfor Ġforward Ġand Ġbackward Ġinformation Ġof Ġsentences . ĠThe Ġcentral Ġenc oder Ġcan Ġfuse Ġthe Ġinformation Ġand Ġenable Ġthe Ġenc oder Ġto Ġcapture Ġthe Ġglobal Ġinformation . </s> The Ġenc oder Ġoutputs Ġone Ġforward Ġinformation Ġand Ġone Ġbackward Ġinformation Ġof Ġeach Ġpositions . ĠThe Ġrepresentation Ġof Ġsentence Ġgenerated Ġby Ġcenter Ġenc oder Ġwill Ġbe Ġadded Ġto Ġthese Ġinformation Ġdirectly : </s> where Ġ$ v ^{ b } =( v ^ b _ 1 ,... , v ^ b _ n ) $ Ġis Ġthe Ġbackward Ġinformation , Ġ$ v ^{ f } =( v ^ f _ 1 ,... , v ^ f _ n ) $ Ġis Ġthe Ġforward Ġinformation , Ġ$ r ^{ b } =( r ^ b _ 1 ,... , r ^ b _ n ) $ Ġis Ġthe Ġoutput Ġof Ġbackward Ġenc oder , Ġ$ r ^{ c } =( r ^ c _ 1 ,... , r ^ c _ n ) $ Ġis Ġthe Ġoutput Ġof Ġcenter Ġenc oder Ġand Ġ$ r ^{ f } =( r ^ f _ 1 ,... , r ^ f _ n ) $ Ġis Ġthe Ġoutput Ġof Ġforward Ġenc oder . </s> Mod els Ġ: :: ĠGa ussian - Mask ed ĠDirection al ĠMulti - Head ĠAttention </s> Similar Ġas Ġscaled Ġdot - product Ġattention ĠB IB REF 24 , ĠGa ussian - mask ed Ġdirectional Ġattention Ġcan Ġbe Ġdescribed Ġas Ġa Ġfunction Ġto Ġmap Ġqueries Ġand Ġkey - value Ġpairs Ġto Ġthe Ġrepresentation Ġof Ġinput . ĠHere Ġqueries , Ġkeys Ġand Ġvalues Ġare Ġall Ġvectors . ĠStandard Ġscaled Ġdot - product Ġattention Ġis Ġcalculated Ġby Ġdot ting Ġquery Ġ$ Q $ Ġwith Ġall Ġkeys Ġ$ K $, Ġdividing Ġeach Ġvalues Ġby Ġ$\ sq rt { d _ k } $, Ġwhere Ġ$\ sq rt { d _ k } $ Ġis Ġthe Ġdimension Ġof Ġkeys , Ġand Ġapply Ġa Ġsoft max Ġfunction Ġto Ġgenerate Ġthe Ġweights Ġin Ġthe Ġattention : </s> Different Ġfrom Ġscaled Ġdot - product Ġattention , ĠGa ussian - mask ed Ġdirectional Ġattention Ġexpects Ġto Ġpay Ġattention Ġto Ġthe Ġadjacent Ġcharacters Ġof Ġeach Ġpositions Ġand Ġcast Ġthe Ġlocal ness Ġrelationship Ġbetween Ġcharacters Ġas Ġa Ġfix ĠGa ussian Ġweight Ġfor Ġattention . ĠWe Ġassume Ġthat Ġthe ĠGa ussian Ġweight Ġonly Ġrely s Ġon Ġthe Ġdistance Ġbetween Ġcharacters . </s> Firstly Ġwe Ġintroduce Ġthe ĠGa ussian Ġweight Ġmatrix Ġ$ G $ Ġwhich Ġpresents Ġthe Ġlocal ness Ġrelationship Ġbetween Ġeach Ġtwo Ġcharacters : </s> where Ġ$ g _{ ij } $ Ġis Ġthe ĠGa ussian Ġweight Ġbetween Ġcharacter Ġ$ i $ Ġand Ġ$ j $, Ġ$ dis _{ ij } $ Ġis Ġthe Ġdistance Ġbetween Ġcharacter Ġ$ i $ Ġand Ġ$ j $, Ġ$\ Ph i Ġ( x ) $ Ġis Ġthe Ġcumulative Ġdistribution Ġfunction Ġof ĠGa ussian , Ġ$\ s igma Ġ$ Ġis Ġthe Ġstandard Ġdeviation Ġof ĠGa ussian Ġfunction Ġand Ġit Ġis Ġa Ġhyper param eter Ġin Ġour Ġmethod . ĠEqu ation Ġ( DIS PLAY _ FORM 13 ) Ġcan Ġensure Ġthe ĠGa ussian Ġweight Ġequals Ġ1 Ġwhen Ġ$ dis _{ ij } $ Ġis Ġ0 . ĠThe Ġlarger Ġdistance Ġbetween Ġcharacters is , Ġthe Ġsmaller Ġthe Ġweight Ġis , Ġwhich Ġmakes Ġone Ġcharacter Ġcan Ġaffect Ġits Ġadjacent Ġcharacters Ġmore Ġcompared Ġwith Ġother Ġcharacters . </s> To Ġcombine Ġthe ĠGa ussian Ġweight Ġto Ġthe Ġself - att ention , Ġwe Ġproduce Ġthe ĠHad am ard Ġproduct Ġof ĠGa ussian Ġweight Ġmatrix Ġ$ G $ Ġand Ġthe Ġscore Ġmatrix Ġproduced Ġby Ġ$ Q { K ^{ T }} $ </s> where Ġ$ AG $ Ġis Ġthe ĠGa ussian - mask ed Ġattention . ĠIt Ġensures Ġthat Ġthe Ġrelationship Ġbetween Ġtwo Ġcharacters Ġwith Ġlong Ġdistances Ġis Ġweaker Ġthan Ġadjacent Ġcharacters . </s> The Ġscaled Ġdot - product Ġattention Ġmodels Ġthe Ġrelationship Ġbetween Ġtwo Ġcharacters Ġwithout Ġregard Ġto Ġtheir Ġdistances Ġin Ġone Ġsequence . ĠFor ĠC WS Ġtask , Ġthe Ġweight Ġbetween Ġadjacent Ġcharacters Ġshould Ġbe Ġmore Ġimportant Ġwhile Ġit Ġis Ġhard Ġfor Ġself - att ention Ġto Ġachieve Ġthe Ġeffect Ġexplicitly Ġbecause Ġthe Ġself - att ention Ġcannot Ġget Ġthe Ġorder Ġof Ġsentences Ġdirectly . ĠThe ĠGa ussian - mask ed Ġattention Ġadjusts Ġthe Ġweight Ġbetween Ġcharacters Ġand Ġtheir Ġadjacent Ġcharacter Ġto Ġa Ġlarger Ġvalue Ġwhich Ġstands Ġfor Ġthe Ġeffect Ġof Ġadjacent Ġcharacters . </s> For Ġforward Ġand Ġbackward Ġenc oder , Ġthe Ġself - att ention Ġsub layer Ġneeds Ġto Ġuse Ġa Ġtriangular Ġmatrix Ġmask Ġto Ġlet Ġthe Ġself - att ention Ġfocus Ġon Ġdifferent Ġweights : </s> where Ġ$ pos _ i $ Ġis Ġthe Ġposition Ġof Ġcharacter Ġ$ c _ i $. ĠThe Ġtriangular Ġmatrix Ġfor Ġforward Ġand Ġbackward Ġencode Ġare : </s> $ \ left [ Ġ\ begin { mat rix } Ġ1 Ġ& Ġ0 Ġ& Ġ0 Ġ& Ġ\ cd ots Ġ& 0 \\ Ġ1 Ġ& Ġ1 Ġ& Ġ0 Ġ& Ġ\ cd ots Ġ& 0 \\ Ġ1 Ġ& Ġ1 Ġ& Ġ1 Ġ& Ġ\ cd ots Ġ& 0 \\ Ġ\ vd ots Ġ& \ vd ots Ġ& \ vd ots Ġ& \ dd ots Ġ& \ vd ots Ġ\\ Ġ1 Ġ& Ġ1 Ġ& Ġ1 Ġ& Ġ\ cd ots Ġ& Ġ1 \\ Ġ\ end { mat rix } Ġ\ right ] $ Ġ$\ left [ Ġ\ begin { mat rix } Ġ1 Ġ& Ġ1 Ġ& Ġ1 Ġ& Ġ\ cd ots Ġ& 1 Ġ\\ Ġ0 Ġ& Ġ1 Ġ& Ġ1 Ġ& Ġ\ cd ots Ġ& 1 Ġ\\ Ġ0 Ġ& Ġ0 & Ġ1 Ġ& Ġ\ cd ots Ġ& 1 Ġ\\ Ġ\ vd ots Ġ& \ vd ots Ġ& \ vd ots Ġ& \ dd ots Ġ& \ vd ots Ġ\\ Ġ0 Ġ& Ġ0 Ġ& Ġ0 Ġ& Ġ\ cd ots Ġ& Ġ1 \\ Ġ\ end { mat rix }\ right ] $ </s> Similar Ġas ĠB IB REF 24 , Ġwe Ġuse Ġmulti - head Ġattention Ġto Ġcapture Ġinformation Ġfrom Ġdifferent Ġdimension Ġpositions Ġas ĠFigure ĠFIG REF 16 Ġand Ġget ĠGa ussian - mask ed Ġdirectional Ġmulti - head Ġattention . ĠWith Ġmulti - head Ġattention Ġarchitecture , Ġthe Ġrepresentation Ġof Ġinput Ġcan Ġbe Ġcaptured Ġby </s> where Ġ$ MH $ Ġis Ġthe ĠGa ussian - mask ed Ġmulti - head Ġattention , Ġ${ W _ i ^ q , ĠW _ i ^ k , W _ i ^ v } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ k Ġ\ times Ġd _ h } $ Ġis Ġthe Ġparameter Ġmat rices Ġto Ġgenerate Ġheads , Ġ$ d _ k $ Ġis Ġthe Ġdimension Ġof Ġmodel Ġand Ġ$ d _ h $ Ġis Ġthe Ġdimension Ġof Ġone Ġhead . </s> Mod els Ġ: :: ĠBi - aff inal ĠAttention ĠSc orer </s> Regarding Ġword Ġboundaries Ġas Ġgaps Ġbetween Ġany Ġadjacent Ġwords Ġconverts Ġthe Ġcharacter Ġlabeling Ġtask Ġto Ġthe Ġgap Ġlabeling Ġtask . ĠDifferent Ġfrom Ġcharacter Ġlabeling Ġtask , Ġgap Ġlabeling Ġtask Ġrequires Ġinformation Ġof Ġtwo Ġadjacent Ġcharacters . ĠThe Ġrelationship Ġbetween Ġadjacent Ġcharacters Ġcan Ġbe Ġrepresented Ġas Ġthe Ġtype Ġof Ġgap . ĠThe Ġcharacteristic Ġof Ġword Ġboundaries Ġmakes Ġbi - aff ine Ġattention Ġan Ġappropriate Ġscorer Ġfor ĠC WS Ġtask . </s> Bi - aff inal Ġattention Ġscorer Ġis Ġthe Ġcomponent Ġthat Ġwe Ġuse Ġto Ġlabel Ġthe Ġgap . ĠBi - aff inal Ġattention Ġis Ġdeveloped Ġfrom Ġbil in ear Ġattention Ġwhich Ġhas Ġbeen Ġused Ġin Ġdependency Ġparsing ĠB IB REF 26 Ġand ĠS RL ĠB IB REF 27 . ĠThe Ġdistribution Ġof Ġlabels Ġin Ġa Ġlabeling Ġtask Ġis Ġoften Ġuneven Ġwhich Ġmakes Ġthe Ġoutput Ġlayer Ġoften Ġinclude Ġa Ġfixed Ġbias Ġterm Ġfor Ġthe Ġprior Ġprobability Ġof Ġdifferent Ġlabels ĠB IB REF 27 . ĠBi - aff ine Ġattention Ġuses Ġbias Ġterms Ġto Ġalleviate Ġthe Ġburden Ġof Ġthe Ġfixed Ġbias Ġterm Ġand Ġget Ġthe Ġprior Ġprobability Ġwhich Ġmakes Ġit Ġdifferent Ġfrom Ġbil in ear Ġattention . ĠThe Ġdistribution Ġof Ġthe Ġgap Ġis Ġuneven Ġthat Ġis Ġsimilar Ġas Ġother Ġlabeling Ġtask Ġwhich Ġfits Ġbi - aff ine . </s> Bi - aff inal Ġattention Ġscorer Ġlabels Ġthe Ġtarget Ġdepending Ġon Ġinformation Ġof Ġindependent Ġunit Ġand Ġthe Ġjoint Ġinformation Ġof Ġtwo Ġunits . ĠIn Ġbi - aff inal Ġattention , Ġthe Ġscore Ġ$ s _{ ij } $ Ġof Ġcharacters Ġ$ c _ i $ Ġand Ġ$ c _ j $ Ġ$( i Ġ< Ġj ) $ Ġis Ġcalculated Ġby : </s> where Ġ$ v _ i ^ f $ Ġis Ġthe Ġforward Ġinformation Ġof Ġ$ c _ i $ Ġand Ġ$ v _ i ^ b $ Ġis Ġthe Ġbackward Ġinformation Ġof Ġ$ c _ j $. ĠIn ĠEqu ation Ġ( DIS PLAY _ FORM 21 ), Ġ$ W $, Ġ$ U $ Ġand Ġ$ b $ Ġare Ġall Ġparameters Ġthat Ġcan Ġbe Ġupdated Ġin Ġtraining . Ġ$ W $ Ġis Ġa Ġmatrix Ġwith Ġshape Ġ$( d _ i Ġ\ times ĠN \ times Ġd _ j ) $ Ġand Ġ$ U $ Ġis Ġa Ġ$( N \ times Ġ( d _ i Ġ+ Ġd _ j )) $ Ġmatrix Ġwhere Ġ$ d _ i $ Ġis Ġthe Ġdimension Ġof Ġvector Ġ$ v _ i ^ f $ Ġand Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġlabels . </s> In Ġour Ġmodel , Ġthe Ġb ia ff ine Ġscorer Ġuses Ġthe Ġforward Ġinformation Ġof Ġcharacter Ġin Ġfront Ġof Ġthe Ġgap Ġand Ġthe Ġbackward Ġinformation Ġof Ġthe Ġcharacter Ġbehind Ġthe Ġgap Ġto Ġdistinguish Ġthe Ġposition Ġof Ġcharacters . ĠFigure ĠFIG REF 22 Ġis Ġan Ġexample Ġof Ġlabeling Ġgap . ĠThe Ġmethod Ġof Ġusing Ġb ia ff ine Ġscorer Ġensures Ġthat Ġthe Ġboundaries Ġof Ġwords Ġcan Ġbe Ġdetermined Ġby Ġadjacent Ġcharacters Ġwith Ġdifferent Ġdirectional Ġinformation . ĠThe Ġscore Ġvector Ġof Ġthe Ġgap Ġis Ġformed Ġby Ġthe Ġprobability Ġof Ġbeing Ġa Ġboundary Ġof Ġword . ĠFurther , Ġthe Ġmodel Ġgenerates Ġall Ġboundaries Ġusing Ġactivation Ġfunction Ġin Ġa Ġgreedy Ġdecoding Ġway . </s> Exper iments Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠData </s> We Ġtrain Ġand Ġevaluate Ġour Ġmodel Ġon Ġdatasets Ġfrom ĠS IGH AN ĠBake off Ġ2005 ĠB IB REF 21 Ġwhich Ġhas Ġfour Ġdatasets , ĠPK U , ĠMS R , ĠAS Ġand ĠCITY U . ĠTable ĠT AB REF 23 Ġshows Ġthe Ġstatistics Ġof Ġtrain Ġdata . ĠWe Ġuse ĠF - score Ġto Ġevaluate ĠC WS Ġmodels . ĠTo Ġtrain Ġmodel Ġwith Ġpre - trained Ġembed d ings Ġin ĠAS Ġand ĠCITY U , Ġwe Ġuse ĠOpen CC Ġto Ġtransfer Ġdata Ġfrom Ġtraditional ĠChinese Ġto Ġsimplified ĠChinese . </s> Exper iments Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠPre - trained ĠEmb edd ing </s> We Ġonly Ġuse Ġun ig ram Ġfeature Ġso Ġwe Ġonly Ġtrained Ġcharacter Ġembed d ings . ĠOur Ġpre - trained Ġembed ding Ġare Ġpre - trained Ġon ĠChinese ĠWikipedia Ġcorpus Ġby Ġword 2 vec ĠB IB REF 29 Ġtool kit . ĠThe Ġcorpus Ġused Ġfor Ġpre - trained Ġembed ding Ġis Ġall Ġtransferred Ġto Ġsimplified ĠChinese Ġand Ġnot Ġsegment ed . ĠOn Ġclosed Ġtest , Ġwe Ġuse Ġembed d ings Ġinitialized Ġrandomly . </s> Exper iments Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠHyper param eters </s> For Ġdifferent Ġdatasets , Ġwe Ġuse Ġtwo Ġkinds Ġof Ġhyper param eters Ġwhich Ġare Ġpresented Ġin ĠTable ĠT AB REF 24 . ĠWe Ġuse Ġhyper param eters Ġin ĠTable ĠT AB REF 24 Ġfor Ġsmall Ġcorpor a Ġ( PK U Ġand ĠCITY U ) Ġand Ġnormal Ġcorpor a Ġ( MS R Ġand ĠAS ). ĠWe Ġset Ġthe Ġstandard Ġdeviation Ġof ĠGa ussian Ġfunction Ġin ĠEqu ation Ġ( DIS PLAY _ FORM 13 ) Ġto Ġ2 . ĠEach Ġtraining Ġbatch Ġcontains Ġsentences Ġwith Ġat Ġmost Ġ4096 Ġtokens . </s> Exper iments Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠOptim izer </s> To Ġtrain Ġour Ġmodel , Ġwe Ġuse Ġthe ĠAdam ĠB IB REF 30 Ġoptim izer Ġwith Ġ$\ beta Ġ_ 1 = 0 . 9 $, Ġ$\ beta Ġ_ 2 = 0 . 98 $ Ġand Ġ$\ ep sil on Ġ= 10 ^{ - 9 } $. ĠThe Ġlearning Ġrate Ġschedule Ġis Ġthe Ġsame Ġas ĠB IB REF 24 : </s> where Ġ$ d $ Ġis Ġthe Ġdimension Ġof Ġembed d ings , Ġ$ step $ Ġis Ġthe Ġstep Ġnumber Ġof Ġtraining Ġand Ġ$ warm up _ step $ Ġis Ġthe Ġstep Ġnumber Ġof Ġwarm up . ĠWhen Ġthe Ġnumber Ġof Ġsteps Ġis Ġsmaller Ġthan Ġthe Ġstep Ġof Ġwarm up , Ġthe Ġlearning Ġrate Ġincreases Ġlin early Ġand Ġthen Ġdecreases . </s> Exper iments Ġ: :: ĠHardware Ġand ĠIm ple ments </s> We Ġtrained Ġour Ġmodels Ġon Ġa Ġsingle ĠCPU Ġ( Intel Ġi 7 - 59 60 X ) Ġwith Ġan Ġn V idia Ġ1080 ĠTi ĠGPU . ĠWe Ġimplement Ġour Ġmodel Ġin ĠPython Ġwith ĠPy tor ch Ġ1 . 0 . </s> Exper iments Ġ: :: ĠResults </s> T ables ĠT AB REF 25 Ġand ĠT AB REF 26 Ġreports Ġthe Ġperformance Ġof Ġrecent Ġmodels Ġand Ġours Ġin Ġterms Ġof Ġclosed Ġtest Ġsetting . ĠWithout Ġthe Ġassistance Ġof Ġun super vised Ġsegment ation Ġfeatures Ġuser d Ġin ĠB IB REF 20 , Ġour Ġmodel Ġoutper forms Ġall Ġthe Ġother Ġmodels Ġin ĠMS R Ġand ĠAS Ġexcept ĠB IB REF 18 Ġand Ġget Ġcomparable Ġperformance Ġin ĠPK U Ġand ĠCITY U . ĠNote Ġthat Ġall Ġthe Ġother Ġmodels Ġfor Ġthis Ġcomparison Ġadopt Ġvarious Ġ$ n $ - gram Ġfeatures Ġwhile Ġonly Ġour Ġmodel Ġtakes Ġun ig ram Ġones . </s> With Ġun super vised Ġsegment ation Ġfeatures Ġintroduced Ġby ĠB IB REF 20 , Ġour Ġmodel Ġgets Ġa Ġhigher Ġresult . ĠS pecially , Ġthe Ġresults Ġin ĠMS R Ġand ĠAS Ġachieve Ġnew Ġstate - of - the - art Ġand Ġapproaching Ġprevious Ġstate - of - the - art Ġin ĠCITY U Ġand ĠPK U . ĠThe Ġun super vised Ġsegment ation Ġfeatures Ġare Ġderived Ġfrom Ġthe Ġgiven Ġtraining Ġdataset , Ġthus Ġusing Ġthem Ġdoes Ġnot Ġviolate Ġthe Ġrule Ġof Ġclosed Ġtest Ġof ĠS IGH AN ĠBake off . </s> Table ĠT AB REF 36 Ġcompares Ġour Ġmodel Ġand Ġrecent Ġneural Ġmodels Ġin Ġterms Ġof Ġopen Ġtest Ġsetting Ġin Ġwhich Ġany Ġexternal Ġresources , Ġespecially Ġpre - trained Ġembed d ings Ġor Ġlanguage Ġmodels Ġcan Ġbe Ġused . ĠIn ĠMS R Ġand ĠAS , Ġour Ġmodel Ġgets Ġa Ġcomparable Ġresult Ġwhile Ġour Ġresults Ġin ĠCITY U Ġand ĠPK U Ġare Ġnot Ġremarkable . </s> However , Ġit Ġis Ġwell Ġknown Ġthat Ġit Ġis Ġalways Ġhard Ġto Ġcompare Ġmodels Ġwhen Ġusing Ġopen Ġtest Ġsetting , Ġespecially Ġwith Ġpre - trained Ġembed ding . ĠNot Ġall Ġmodels Ġmay Ġuse Ġthe Ġsame Ġmethod Ġand Ġdata Ġto Ġpre - train . ĠThough Ġpre - trained Ġembed ding Ġor Ġlanguage Ġmodel Ġcan Ġimprove Ġthe Ġperformance , Ġthe Ġperformance Ġimprovement Ġitself Ġmay Ġbe Ġfrom Ġmultiple Ġsources . ĠIt Ġoften Ġthat Ġthere Ġis Ġa Ġsuccess Ġof Ġpre - trained Ġembed ding Ġto Ġimprove Ġthe Ġperformance , Ġwhile Ġit Ġcannot Ġprove Ġthat Ġthe Ġmodel Ġis Ġbetter . </s> Compared Ġwith Ġother ĠL ST M Ġmodels , Ġour Ġmodel Ġperforms Ġbetter Ġin ĠAS Ġand ĠMS R Ġthan Ġin ĠCITY U Ġand ĠPK U . ĠConsidering Ġthe Ġscale Ġof Ġdifferent Ġcorpor a , Ġwe Ġbelieve Ġthat Ġthe Ġsize Ġof Ġcorpus Ġaffects Ġour Ġmodel Ġand Ġthe Ġlarger Ġsize Ġis , Ġthe Ġbetter Ġmodel Ġperforms . ĠFor Ġsmall Ġcorpus , Ġthe Ġmodel Ġtends Ġto Ġbe Ġover fitting . </s> T ables ĠT AB REF 25 Ġand ĠT AB REF 26 Ġalso Ġshow Ġthe Ġdecoding Ġtime Ġin Ġdifferent Ġdatasets . ĠOur Ġmodel Ġfinishes Ġthe Ġsegment ation Ġwith Ġthe Ġleast Ġdecoding Ġtime Ġin Ġall Ġfour Ġdatasets , Ġthanks Ġto Ġthe Ġarchitecture Ġof Ġmodel Ġwhich Ġonly Ġtakes Ġattention Ġmechanism Ġas Ġbasic Ġblock . </s> Related ĠWork Ġ: :: ĠChinese ĠWord ĠSe gment ation </s> C WS Ġis Ġa Ġtask Ġfor ĠChinese Ġnatural Ġlanguage Ġprocess Ġto Ġdelim it Ġword Ġboundary . ĠB IB REF 0 Ġfor Ġthe Ġfirst Ġtime Ġform ul ize ĠC WS Ġas Ġa Ġsequence Ġlabeling Ġtask . ĠB IB REF 3 Ġshow Ġthat Ġdifferent Ġcharacter Ġtag Ġsets Ġcan Ġmake Ġessential Ġimpact Ġfor ĠC WS . ĠB IB REF 2 Ġuse ĠCR Fs Ġas Ġa Ġmodel Ġfor ĠC WS , Ġachieving Ġnew Ġstate - of - the - art . ĠWorks Ġof Ġstatistical ĠC WS Ġhas Ġbuilt Ġthe Ġbasis Ġfor Ġneural ĠC WS . </s> Ne ural Ġword Ġsegment ation Ġhas Ġbeen Ġwidely Ġused Ġto Ġminimize Ġthe Ġefforts Ġin Ġfeature Ġengineering Ġwhich Ġwas Ġimportant Ġin Ġstatistical ĠC WS . ĠB IB REF 4 Ġintroduce Ġthe Ġneural Ġmodel Ġwith Ġsliding - window Ġbased Ġsequence Ġlabeling . ĠB IB REF 6 Ġpropose Ġa Ġg ated Ġrecursive Ġneural Ġnetwork Ġ( GR NN ) Ġfor ĠC WS Ġto Ġincorporate Ġcomplicated Ġcombination Ġof Ġcontextual Ġcharacter Ġand Ġn - gram Ġfeatures . ĠB IB REF 7 Ġuse ĠL ST M Ġto Ġlearn Ġlong Ġdistance Ġinformation . ĠB IB REF 9 Ġpropose Ġa Ġneural Ġframework Ġthat Ġeliminates Ġcontext Ġwindows Ġand Ġutilize Ġcomplete Ġsegment ation Ġhistory . ĠB IB REF 33 Ġexplore Ġa Ġjoint Ġmodel Ġthat Ġperforms Ġsegment ation , ĠPOS - T agging Ġand Ġchunk ing Ġsimultaneously . ĠB IB REF 34 Ġpropose Ġa Ġfeature - en riched Ġneural Ġmodel Ġfor Ġjoint ĠC WS Ġand Ġpart - of - speech Ġtagging . ĠB IB REF 35 Ġpresent Ġa Ġjoint Ġmodel Ġto Ġenhance Ġthe Ġsegment ation Ġof ĠChinese Ġmicro text Ġby Ġperforming ĠC WS Ġand Ġinformal Ġword Ġdetection Ġsimultaneously . ĠB IB REF 17 Ġpropose Ġa Ġcharacter - based Ġconv olution al Ġneural Ġmodel Ġto Ġcapture Ġ$ n $ - gram Ġfeatures Ġautomatically Ġand Ġan Ġeffective Ġapproach Ġto Ġincorporate Ġword Ġembed d ings . ĠB IB REF 11 Ġimprove Ġthe Ġmodel Ġin ĠB IB REF 9 Ġand Ġpropose Ġa Ġgreedy Ġneural Ġword Ġsegment er Ġwith Ġbalanced Ġword Ġand Ġcharacter Ġembed ding Ġinputs . ĠB IB REF 23 Ġpropose Ġa Ġnovel Ġneural Ġnetwork Ġmodel Ġto Ġincorporate Ġunl abel ed Ġand Ġpartially - label ed Ġdata . ĠB IB REF 36 Ġpropose Ġtwo Ġmethods Ġthat Ġextend Ġthe ĠBi - L ST M Ġto Ġperform Ġincorporating Ġdiction aries Ġinto Ġneural Ġnetworks Ġfor ĠC WS . ĠB IB REF 37 Ġpropose ĠSwitch - L ST Ms Ġto Ġsegment Ġwords Ġand Ġprovided Ġa Ġmore Ġflexible Ġsolution Ġfor Ġmulti - crit eria ĠC WS Ġwhich Ġis Ġeasy Ġto Ġtransfer Ġthe Ġlearned Ġknowledge Ġto Ġnew Ġcriteria . </s> Related ĠWork Ġ: :: ĠTrans former </s> Trans former ĠB IB REF 24 Ġis Ġan Ġattention - based Ġneural Ġmachine Ġtranslation Ġmodel . ĠThe ĠTrans former Ġis Ġone Ġkind Ġof Ġself - att ention Ġnetworks Ġ( SAN s ) Ġwhich Ġis Ġproposed Ġin ĠB IB REF 38 . ĠEnc oder Ġof Ġthe ĠTrans former Ġconsists Ġof Ġone Ġself - att ention Ġlayer Ġand Ġa Ġposition - wise Ġfeed - forward Ġlayer . ĠDecoder Ġof Ġthe ĠTrans former Ġcontains Ġone Ġself - att ention Ġlayer , Ġone Ġenc oder - dec oder Ġattention Ġlayer Ġand Ġone Ġposition - wise Ġfeed - forward Ġlayer . ĠThe ĠTrans former Ġuses Ġresidual Ġconnections Ġaround Ġthe Ġsub l ayers Ġand Ġthen Ġfollowed Ġby Ġa Ġlayer Ġnormal ization Ġlayer . </s> Sc aled Ġdot - product Ġattention Ġis Ġthe Ġkey Ġcomponent Ġin Ġthe ĠTrans former . ĠThe Ġinput Ġof Ġattention Ġcontains Ġqueries , Ġkeys , Ġand Ġvalues Ġof Ġinput Ġsequences . ĠThe Ġattention Ġis Ġgenerated Ġusing Ġqueries Ġand Ġkeys Ġlike ĠEqu ation Ġ( DIS PLAY _ FORM 11 ). ĠStructure Ġof Ġscaled Ġdot - product Ġattention Ġallows Ġthe Ġself - att ention Ġlayer Ġgenerate Ġthe Ġrepresentation Ġof Ġsentences Ġat Ġonce Ġand Ġcontain Ġthe Ġinformation Ġof Ġthe Ġsentence Ġwhich Ġis Ġdifferent Ġfrom ĠR NN Ġthat Ġprocess Ġcharacters Ġof Ġsentences Ġone Ġby Ġone . ĠStandard Ġself - att ention Ġis Ġsimilar Ġas ĠGa ussian - mask ed Ġdirection Ġattention Ġwhile Ġit Ġdoes Ġnot Ġhave Ġdirectional Ġmask Ġand Ġga ussian Ġmask . ĠB IB REF 24 Ġalso Ġpropose Ġmulti - head Ġattention Ġwhich Ġis Ġbetter Ġto Ġgenerate Ġrepresentation Ġof Ġsentence Ġby Ġdividing Ġqueries , Ġkeys Ġand Ġvalues Ġto Ġdifferent Ġheads Ġand Ġget Ġinformation Ġfrom Ġdifferent Ġsub sp aces . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġan Ġattention Ġmechanism Ġonly Ġbased ĠChinese Ġword Ġsegment ation Ġmodel . ĠOur Ġmodel Ġuses Ġself - att ention Ġfrom Ġthe ĠTrans former Ġenc oder Ġto Ġtake Ġsequence Ġinput Ġand Ġbi - aff ine Ġattention Ġscorer Ġto Ġpredict Ġthe Ġlabel Ġof Ġgaps . ĠTo Ġimprove Ġthe Ġability Ġof Ġcapturing Ġthe Ġlocal ness Ġand Ġdirectional Ġinformation Ġof Ġself - att ention Ġbased Ġenc oder , Ġwe Ġpropose Ġa Ġvariant Ġof Ġself - att ention Ġcalled ĠGa ussian - mask ed Ġdirectional Ġmulti - head Ġattention Ġto Ġreplace Ġthe Ġstandard Ġself - att ention . ĠWe Ġalso Ġextend Ġthe ĠTrans former Ġenc oder Ġto Ġcapture Ġdirectional Ġfeatures . ĠOur Ġmodel Ġuses Ġonly Ġun ig ram Ġfeatures Ġinstead Ġof Ġmultiple Ġ$ n $ - gram Ġfeatures Ġin Ġprevious Ġwork . ĠOur Ġmodel Ġis Ġevaluated Ġon Ġstandard Ġbenchmark Ġdataset , ĠS IGH AN ĠBake off Ġ2005 , Ġwhich Ġshows Ġnot Ġonly Ġour Ġmodel Ġperforms Ġsegment ation Ġfaster Ġthan Ġany Ġprevious Ġmodels Ġbut Ġalso Ġgives Ġnew Ġhigher Ġor Ġcomparable Ġsegment ation Ġperformance Ġagainst Ġprevious Ġstate - of - the - art Ġmodels .
Introduction Chinese word segmentation (CWS) is a task for Chinese natural language process to delimit word boundary. CWS is a basic and essential task for Chinese which is written without explicit word delimiters and different from alphabetical languages like English. BIBREF0 treats Chinese word segmentation (CWS) as a sequence labeling task with character position tags, which is followed by BIBREF1, BIBREF2, BIBREF3. Traditional CWS models depend on the design of features heavily which effects the performance of model. To minimize the effort in feature engineering, some CWS models BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11 are developed following neural network architecture for sequence labeling tasks BIBREF12. Neural CWS models perform strong ability of feature representation, employing unigram and bigram character embedding as input and approach good performance. The CWS task is often modeled as one graph model based on a scoring model that means it is composed of two parts, one part is an encoder which is used to generate the representation of characters from the input sequence, the other part is a decoder which performs segmentation according to the encoder scoring. Table TABREF1 summarizes typical CWS models according to their decoding ways for both traditional and neural models. Markov models such as BIBREF13 and BIBREF4 depend on the maximum entropy model or maximum entropy Markov model both with a Viterbi decoder. Besides, conditional random field (CRF) or Semi-CRF for sequence labeling has been used for both traditional and neural models though with different representations BIBREF2, BIBREF15, BIBREF10, BIBREF17, BIBREF18. Generally speaking, the major difference between traditional and neural network models is about the way to represent input sentences. Recent works about neural CWS which focus on benchmark dataset, namely SIGHAN Bakeoff BIBREF21, may be put into the following three categories roughly. Encoder. Practice in various natural language processing tasks has been shown that effective representation is essential to the performance improvement. Thus for better CWS, it is crucial to encode the input character, word or sentence into effective representation. Table TABREF2 summarizes regular feature sets for typical CWS models including ours as well. The building blocks that encoders use include recurrent neural network (RNN) and convolutional neural network (CNN), and long-term memory network (LSTM). Graph model. As CWS is a kind of structure learning task, the graph model determines which type of decoder should be adopted for segmentation, also it may limit the capability of defining feature, as shown in Table 2, not all graph models can support the word features. Thus recent work focused on finding more general or flexible graph model to make model learn the representation of segmentation more effective as BIBREF9, BIBREF11. External data and pre-trained embedding. Whereas both encoder and graph model are about exploring a way to get better performance only by improving the model strength itself. Using external resource such as pre-trained embeddings or language representation is an alternative for the same purpose BIBREF22, BIBREF23. SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21. In this work, we will focus on the closed test setting by finding a better model design for further CWS performance improvement. Shown in Table TABREF1, different decoders have particular decoding algorithms to match the respective CWS models. Markov models and CRF-based models often use Viterbi decoders with polynomial time complexity. In general graph model, search space may be too large for model to search. Thus it forces graph models to use an approximate beam search strategy. Beam search algorithm has a kind low-order polynomial time complexity. Especially, when beam width $b$=1, the beam search algorithm will reduce to greedy algorithm with a better time complexity $O(Mn)$ against the general beam search time complexity $O(Mnb^2)$, where $n$ is the number of units in one sentences, $M$ is a constant representing the model complexity. Greedy decoding algorithm can bring the fastest speed of decoding while it is not easy to guarantee the precision of decoding when the encoder is not strong enough. In this paper, we focus on more effective encoder design which is capable of offering fast and accurate Chinese word segmentation with only unigram feature and greedy decoding. Our proposed encoder will only consist of attention mechanisms as building blocks but nothing else. Motivated by the Transformer BIBREF24 and its strength of capturing long-range dependencies of input sentences, we use a self-attention network to generate the representation of input which makes the model encode sentences at once without feeding input iteratively. Considering the weakness of the Transformer to model relative and absolute position information directly BIBREF25 and the importance of localness information, position information and directional information for CWS, we further improve the architecture of standard multi-head self-attention of the Transformer with a directional Gaussian mask and get a variant called Gaussian-masked directional multi-head attention. Based on the newly improved attention mechanism, we expand the encoder of the Transformer to capture different directional information. With our powerful encoder, our model uses only simple unigram features to generate representation of sentences. For decoder which directly performs the segmentation, we use the bi-affinal attention scorer, which has been used in dependency parsing BIBREF26 and semantic role labeling BIBREF27, to implement greedy decoding on finding the boundaries of words. In our proposed model, greedy decoding ensures a fast segmentation while powerful encoder design ensures a good enough segmentation performance even working with greedy decoder together. Our model will be strictly evaluated on benchmark datasets from SIGHAN Bakeoff shared task on CWS in terms of closed test setting, and the experimental results show that our proposed model achieves new state-of-the-art. The technical contributions of this paper can be summarized as follows. We propose a CWS model with only attention structure. The encoder and decoder are both based on attention structure. With a powerful enough encoder, we for the first time show that unigram (character) featues can help yield strong performance instead of diverse $n$-gram (character and word) features in most of previous work. To capture the representation of localness information and directional information, we propose a variant of directional multi-head self-attention to further enhance the state-of-the-art Transformer encoder. Models The CWS task is often modelled as one graph model based on an encoder-based scoring model. The model for CWS task is composed of an encoder to represent the input and a decoder based on the encoder to perform actual segmentation. Figure FIGREF6 is the architecture of our model. The model feeds sentence into encoder. Embedding captures the vector $e=(e_1,...,e_n)$ of the input character sequences of $c=(c_1,...,c_n)$. The encoder maps vector sequences of $ {e}=(e_1,..,e_n)$ to two sequences of vector which are $ {v^b}=(v_1^b,...,v_n^b)$ and ${v^f}=(v_1^f,...v_n^f)$ as the representation of sentences. With $v^b$ and $v^f$, the bi-affinal scorer calculates the probability of each segmentation gaps and predicts the word boundaries of input. Similar as the Transformer, the encoder is an attention network with stacked self-attention and point-wise, fully connected layers while our encoder includes three independent directional encoders. Models ::: Encoder Stacks In the Transformer, the encoder is composed of a stack of N identical layers and each layer has one multi-head self-attention layer and one position-wise fully connected feed-forward layer. One residual connection is around two sub-layers and followed by layer normalization BIBREF24. This architecture provides the Transformer a good ability to generate representation of sentence. With the variant of multi-head self-attention, we design a Gaussian-masked directional encoder to capture representation of different directions to improve the ability of capturing the localness information and position information for the importance of adjacent characters. One unidirectional encoder can capture information of one particular direction. For CWS tasks, one gap of characters, which is from a word boundary, can divide one sequence into two parts, one part in front of the gap and one part in the rear of it. The forward encoder and backward encoder are used to capture information of two directions which correspond to two parts divided by the gap. One central encoder is paralleled with forward and backward encoders to capture the information of entire sentences. The central encoder is a special directional encoder for forward and backward information of sentences. The central encoder can fuse the information and enable the encoder to capture the global information. The encoder outputs one forward information and one backward information of each positions. The representation of sentence generated by center encoder will be added to these information directly: where $v^{b}=(v^b_1,...,v^b_n)$ is the backward information, $v^{f}=(v^f_1,...,v^f_n)$ is the forward information, $r^{b}=(r^b_1,...,r^b_n)$ is the output of backward encoder, $r^{c}=(r^c_1,...,r^c_n)$ is the output of center encoder and $r^{f}=(r^f_1,...,r^f_n)$ is the output of forward encoder. Models ::: Gaussian-Masked Directional Multi-Head Attention Similar as scaled dot-product attention BIBREF24, Gaussian-masked directional attention can be described as a function to map queries and key-value pairs to the representation of input. Here queries, keys and values are all vectors. Standard scaled dot-product attention is calculated by dotting query $Q$ with all keys $K$, dividing each values by $\sqrt{d_k}$, where $\sqrt{d_k}$ is the dimension of keys, and apply a softmax function to generate the weights in the attention: Different from scaled dot-product attention, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention. We assume that the Gaussian weight only relys on the distance between characters. Firstly we introduce the Gaussian weight matrix $G$ which presents the localness relationship between each two characters: where $g_{ij}$ is the Gaussian weight between character $i$ and $j$, $dis_{ij}$ is the distance between character $i$ and $j$, $\Phi (x)$ is the cumulative distribution function of Gaussian, $\sigma $ is the standard deviation of Gaussian function and it is a hyperparameter in our method. Equation (DISPLAY_FORM13) can ensure the Gaussian weight equals 1 when $dis_{ij}$ is 0. The larger distance between charactersis, the smaller the weight is, which makes one character can affect its adjacent characters more compared with other characters. To combine the Gaussian weight to the self-attention, we produce the Hadamard product of Gaussian weight matrix $G$ and the score matrix produced by $Q{K^{T}}$ where $AG$ is the Gaussian-masked attention. It ensures that the relationship between two characters with long distances is weaker than adjacent characters. The scaled dot-product attention models the relationship between two characters without regard to their distances in one sequence. For CWS task, the weight between adjacent characters should be more important while it is hard for self-attention to achieve the effect explicitly because the self-attention cannot get the order of sentences directly. The Gaussian-masked attention adjusts the weight between characters and their adjacent character to a larger value which stands for the effect of adjacent characters. For forward and backward encoder, the self-attention sublayer needs to use a triangular matrix mask to let the self-attention focus on different weights: where $pos_i$ is the position of character $c_i$. The triangular matrix for forward and backward encode are: $\left[ \begin{matrix} 1 & 0 & 0 & \cdots &0\\ 1 & 1 & 0 & \cdots &0\\ 1 & 1 & 1 & \cdots &0\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 1 & 1 & 1 & \cdots & 1\\ \end{matrix} \right]$ $\left[ \begin{matrix} 1 & 1 & 1 & \cdots &1 \\ 0 & 1 & 1 & \cdots &1 \\ 0 & 0& 1 & \cdots &1 \\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 0 & 0 & 0 & \cdots & 1\\ \end{matrix}\right]$ Similar as BIBREF24, we use multi-head attention to capture information from different dimension positions as Figure FIGREF16 and get Gaussian-masked directional multi-head attention. With multi-head attention architecture, the representation of input can be captured by where $MH$ is the Gaussian-masked multi-head attention, ${W_i^q, W_i^k,W_i^v} \in \mathbb {R}^{d_k \times d_h}$ is the parameter matrices to generate heads, $d_k$ is the dimension of model and $d_h$ is the dimension of one head. Models ::: Bi-affinal Attention Scorer Regarding word boundaries as gaps between any adjacent words converts the character labeling task to the gap labeling task. Different from character labeling task, gap labeling task requires information of two adjacent characters. The relationship between adjacent characters can be represented as the type of gap. The characteristic of word boundaries makes bi-affine attention an appropriate scorer for CWS task. Bi-affinal attention scorer is the component that we use to label the gap. Bi-affinal attention is developed from bilinear attention which has been used in dependency parsing BIBREF26 and SRL BIBREF27. The distribution of labels in a labeling task is often uneven which makes the output layer often include a fixed bias term for the prior probability of different labels BIBREF27. Bi-affine attention uses bias terms to alleviate the burden of the fixed bias term and get the prior probability which makes it different from bilinear attention. The distribution of the gap is uneven that is similar as other labeling task which fits bi-affine. Bi-affinal attention scorer labels the target depending on information of independent unit and the joint information of two units. In bi-affinal attention, the score $s_{ij}$ of characters $c_i$ and $c_j$ $(i < j)$ is calculated by: where $v_i^f$ is the forward information of $c_i$ and $v_i^b$ is the backward information of $c_j$. In Equation (DISPLAY_FORM21), $W$, $U$ and $b$ are all parameters that can be updated in training. $W$ is a matrix with shape $(d_i \times N\times d_j)$ and $U$ is a $(N\times (d_i + d_j))$ matrix where $d_i$ is the dimension of vector $v_i^f$ and $N$ is the number of labels. In our model, the biaffine scorer uses the forward information of character in front of the gap and the backward information of the character behind the gap to distinguish the position of characters. Figure FIGREF22 is an example of labeling gap. The method of using biaffine scorer ensures that the boundaries of words can be determined by adjacent characters with different directional information. The score vector of the gap is formed by the probability of being a boundary of word. Further, the model generates all boundaries using activation function in a greedy decoding way. Experiments ::: Experimental Settings ::: Data We train and evaluate our model on datasets from SIGHAN Bakeoff 2005 BIBREF21 which has four datasets, PKU, MSR, AS and CITYU. Table TABREF23 shows the statistics of train data. We use F-score to evaluate CWS models. To train model with pre-trained embeddings in AS and CITYU, we use OpenCC to transfer data from traditional Chinese to simplified Chinese. Experiments ::: Experimental Settings ::: Pre-trained Embedding We only use unigram feature so we only trained character embeddings. Our pre-trained embedding are pre-trained on Chinese Wikipedia corpus by word2vec BIBREF29 toolkit. The corpus used for pre-trained embedding is all transferred to simplified Chinese and not segmented. On closed test, we use embeddings initialized randomly. Experiments ::: Experimental Settings ::: Hyperparameters For different datasets, we use two kinds of hyperparameters which are presented in Table TABREF24. We use hyperparameters in Table TABREF24 for small corpora (PKU and CITYU) and normal corpora (MSR and AS). We set the standard deviation of Gaussian function in Equation (DISPLAY_FORM13) to 2. Each training batch contains sentences with at most 4096 tokens. Experiments ::: Experimental Settings ::: Optimizer To train our model, we use the Adam BIBREF30 optimizer with $\beta _1=0.9$, $\beta _2=0.98$ and $\epsilon =10^{-9}$. The learning rate schedule is the same as BIBREF24: where $d$ is the dimension of embeddings, $step$ is the step number of training and $warmup_step$ is the step number of warmup. When the number of steps is smaller than the step of warmup, the learning rate increases linearly and then decreases. Experiments ::: Hardware and Implements We trained our models on a single CPU (Intel i7-5960X) with an nVidia 1080 Ti GPU. We implement our model in Python with Pytorch 1.0. Experiments ::: Results Tables TABREF25 and TABREF26 reports the performance of recent models and ours in terms of closed test setting. Without the assistance of unsupervised segmentation features userd in BIBREF20, our model outperforms all the other models in MSR and AS except BIBREF18 and get comparable performance in PKU and CITYU. Note that all the other models for this comparison adopt various $n$-gram features while only our model takes unigram ones. With unsupervised segmentation features introduced by BIBREF20, our model gets a higher result. Specially, the results in MSR and AS achieve new state-of-the-art and approaching previous state-of-the-art in CITYU and PKU. The unsupervised segmentation features are derived from the given training dataset, thus using them does not violate the rule of closed test of SIGHAN Bakeoff. Table TABREF36 compares our model and recent neural models in terms of open test setting in which any external resources, especially pre-trained embeddings or language models can be used. In MSR and AS, our model gets a comparable result while our results in CITYU and PKU are not remarkable. However, it is well known that it is always hard to compare models when using open test setting, especially with pre-trained embedding. Not all models may use the same method and data to pre-train. Though pre-trained embedding or language model can improve the performance, the performance improvement itself may be from multiple sources. It often that there is a success of pre-trained embedding to improve the performance, while it cannot prove that the model is better. Compared with other LSTM models, our model performs better in AS and MSR than in CITYU and PKU. Considering the scale of different corpora, we believe that the size of corpus affects our model and the larger size is, the better model performs. For small corpus, the model tends to be overfitting. Tables TABREF25 and TABREF26 also show the decoding time in different datasets. Our model finishes the segmentation with the least decoding time in all four datasets, thanks to the architecture of model which only takes attention mechanism as basic block. Related Work ::: Chinese Word Segmentation CWS is a task for Chinese natural language process to delimit word boundary. BIBREF0 for the first time formulize CWS as a sequence labeling task. BIBREF3 show that different character tag sets can make essential impact for CWS. BIBREF2 use CRFs as a model for CWS, achieving new state-of-the-art. Works of statistical CWS has built the basis for neural CWS. Neural word segmentation has been widely used to minimize the efforts in feature engineering which was important in statistical CWS. BIBREF4 introduce the neural model with sliding-window based sequence labeling. BIBREF6 propose a gated recursive neural network (GRNN) for CWS to incorporate complicated combination of contextual character and n-gram features. BIBREF7 use LSTM to learn long distance information. BIBREF9 propose a neural framework that eliminates context windows and utilize complete segmentation history. BIBREF33 explore a joint model that performs segmentation, POS-Tagging and chunking simultaneously. BIBREF34 propose a feature-enriched neural model for joint CWS and part-of-speech tagging. BIBREF35 present a joint model to enhance the segmentation of Chinese microtext by performing CWS and informal word detection simultaneously. BIBREF17 propose a character-based convolutional neural model to capture $n$-gram features automatically and an effective approach to incorporate word embeddings. BIBREF11 improve the model in BIBREF9 and propose a greedy neural word segmenter with balanced word and character embedding inputs. BIBREF23 propose a novel neural network model to incorporate unlabeled and partially-labeled data. BIBREF36 propose two methods that extend the Bi-LSTM to perform incorporating dictionaries into neural networks for CWS. BIBREF37 propose Switch-LSTMs to segment words and provided a more flexible solution for multi-criteria CWS which is easy to transfer the learned knowledge to new criteria. Related Work ::: Transformer Transformer BIBREF24 is an attention-based neural machine translation model. The Transformer is one kind of self-attention networks (SANs) which is proposed in BIBREF38. Encoder of the Transformer consists of one self-attention layer and a position-wise feed-forward layer. Decoder of the Transformer contains one self-attention layer, one encoder-decoder attention layer and one position-wise feed-forward layer. The Transformer uses residual connections around the sublayers and then followed by a layer normalization layer. Scaled dot-product attention is the key component in the Transformer. The input of attention contains queries, keys, and values of input sequences. The attention is generated using queries and keys like Equation (DISPLAY_FORM11). Structure of scaled dot-product attention allows the self-attention layer generate the representation of sentences at once and contain the information of the sentence which is different from RNN that process characters of sentences one by one. Standard self-attention is similar as Gaussian-masked direction attention while it does not have directional mask and gaussian mask. BIBREF24 also propose multi-head attention which is better to generate representation of sentence by dividing queries, keys and values to different heads and get information from different subspaces. Conclusion In this paper, we propose an attention mechanism only based Chinese word segmentation model. Our model uses self-attention from the Transformer encoder to take sequence input and bi-affine attention scorer to predict the label of gaps. To improve the ability of capturing the localness and directional information of self-attention based encoder, we propose a variant of self-attention called Gaussian-masked directional multi-head attention to replace the standard self-attention. We also extend the Transformer encoder to capture directional features. Our model uses only unigram features instead of multiple $n$-gram features in previous work. Our model is evaluated on standard benchmark dataset, SIGHAN Bakeoff 2005, which shows not only our model performs segmentation faster than any previous models but also gives new higher or comparable segmentation performance against previous state-of-the-art models.
What is meant by closed test setting?
186b7978ee33b563a37139adff1da7d51a60f581
What Ġis Ġmeant Ġby Ġclosed Ġtest Ġsetting ?
closed Ġtest Ġlimits Ġall Ġthe Ġdata Ġfor Ġlearning Ġshould Ġnot Ġbe Ġbeyond Ġthe Ġgiven Ġtraining Ġset , Ġwhile Ġopen Ġtest Ġdoes Ġnot Ġtake Ġthis Ġlimitation
<s> What Ġis Ġmeant Ġby Ġclosed Ġtest Ġsetting ? </s> Introduction </s> Chinese Ġword Ġsegment ation Ġ( C WS ) Ġis Ġa Ġtask Ġfor ĠChinese Ġnatural Ġlanguage Ġprocess Ġto Ġdelim it Ġword Ġboundary . ĠC WS Ġis Ġa Ġbasic Ġand Ġessential Ġtask Ġfor ĠChinese Ġwhich Ġis Ġwritten Ġwithout Ġexplicit Ġword Ġdelim it ers Ġand Ġdifferent Ġfrom Ġalphabet ical Ġlanguages Ġlike ĠEnglish . ĠB IB REF 0 Ġtreats ĠChinese Ġword Ġsegment ation Ġ( C WS ) Ġas Ġa Ġsequence Ġlabeling Ġtask Ġwith Ġcharacter Ġposition Ġtags , Ġwhich Ġis Ġfollowed Ġby ĠB IB REF 1 , ĠB IB REF 2 , ĠB IB REF 3 . ĠTraditional ĠC WS Ġmodels Ġdepend Ġon Ġthe Ġdesign Ġof Ġfeatures Ġheavily Ġwhich Ġeffects Ġthe Ġperformance Ġof Ġmodel . ĠTo Ġminimize Ġthe Ġeffort Ġin Ġfeature Ġengineering , Ġsome ĠC WS Ġmodels ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 , ĠB IB REF 7 , ĠB IB REF 8 , ĠB IB REF 9 , ĠB IB REF 10 , ĠB IB REF 11 Ġare Ġdeveloped Ġfollowing Ġneural Ġnetwork Ġarchitecture Ġfor Ġsequence Ġlabeling Ġtasks ĠB IB REF 12 . ĠNeural ĠC WS Ġmodels Ġperform Ġstrong Ġability Ġof Ġfeature Ġrepresentation , Ġemploying Ġun ig ram Ġand Ġbig ram Ġcharacter Ġembed ding Ġas Ġinput Ġand Ġapproach Ġgood Ġperformance . </s> The ĠC WS Ġtask Ġis Ġoften Ġmodeled Ġas Ġone Ġgraph Ġmodel Ġbased Ġon Ġa Ġscoring Ġmodel Ġthat Ġmeans Ġit Ġis Ġcomposed Ġof Ġtwo Ġparts , Ġone Ġpart Ġis Ġan Ġenc oder Ġwhich Ġis Ġused Ġto Ġgenerate Ġthe Ġrepresentation Ġof Ġcharacters Ġfrom Ġthe Ġinput Ġsequence , Ġthe Ġother Ġpart Ġis Ġa Ġdec oder Ġwhich Ġperforms Ġsegment ation Ġaccording Ġto Ġthe Ġenc oder Ġscoring . ĠTable ĠT AB REF 1 Ġsummarizes Ġtypical ĠC WS Ġmodels Ġaccording Ġto Ġtheir Ġdecoding Ġways Ġfor Ġboth Ġtraditional Ġand Ġneural Ġmodels . ĠMark ov Ġmodels Ġsuch Ġas ĠB IB REF 13 Ġand ĠB IB REF 4 Ġdepend Ġon Ġthe Ġmaximum Ġentropy Ġmodel Ġor Ġmaximum Ġentropy ĠMark ov Ġmodel Ġboth Ġwith Ġa ĠV iter bi Ġdec oder . ĠBesides , Ġconditional Ġrandom Ġfield Ġ( CR F ) Ġor ĠSemi - CR F Ġfor Ġsequence Ġlabeling Ġhas Ġbeen Ġused Ġfor Ġboth Ġtraditional Ġand Ġneural Ġmodels Ġthough Ġwith Ġdifferent Ġrepresentations ĠB IB REF 2 , ĠB IB REF 15 , ĠB IB REF 10 , ĠB IB REF 17 , ĠB IB REF 18 . ĠGenerally Ġspeaking , Ġthe Ġmajor Ġdifference Ġbetween Ġtraditional Ġand Ġneural Ġnetwork Ġmodels Ġis Ġabout Ġthe Ġway Ġto Ġrepresent Ġinput Ġsentences . </s> Recent Ġworks Ġabout Ġneural ĠC WS Ġwhich Ġfocus Ġon Ġbenchmark Ġdataset , Ġnamely ĠS IGH AN ĠBake off ĠB IB REF 21 , Ġmay Ġbe Ġput Ġinto Ġthe Ġfollowing Ġthree Ġcategories Ġroughly . </s> Enc oder . ĠPractice Ġin Ġvarious Ġnatural Ġlanguage Ġprocessing Ġtasks Ġhas Ġbeen Ġshown Ġthat Ġeffective Ġrepresentation Ġis Ġessential Ġto Ġthe Ġperformance Ġimprovement . ĠThus Ġfor Ġbetter ĠC WS , Ġit Ġis Ġcrucial Ġto Ġencode Ġthe Ġinput Ġcharacter , Ġword Ġor Ġsentence Ġinto Ġeffective Ġrepresentation . ĠTable ĠT AB REF 2 Ġsummarizes Ġregular Ġfeature Ġsets Ġfor Ġtypical ĠC WS Ġmodels Ġincluding Ġours Ġas Ġwell . ĠThe Ġbuilding Ġblocks Ġthat Ġenc od ers Ġuse Ġinclude Ġrecurrent Ġneural Ġnetwork Ġ( R NN ) Ġand Ġconv olution al Ġneural Ġnetwork Ġ( CNN ), Ġand Ġlong - term Ġmemory Ġnetwork Ġ( L ST M ). </s> Graph Ġmodel . ĠAs ĠC WS Ġis Ġa Ġkind Ġof Ġstructure Ġlearning Ġtask , Ġthe Ġgraph Ġmodel Ġdetermines Ġwhich Ġtype Ġof Ġdec oder Ġshould Ġbe Ġadopted Ġfor Ġsegment ation , Ġalso Ġit Ġmay Ġlimit Ġthe Ġcapability Ġof Ġdefining Ġfeature , Ġas Ġshown Ġin ĠTable Ġ2 , Ġnot Ġall Ġgraph Ġmodels Ġcan Ġsupport Ġthe Ġword Ġfeatures . ĠThus Ġrecent Ġwork Ġfocused Ġon Ġfinding Ġmore Ġgeneral Ġor Ġflexible Ġgraph Ġmodel Ġto Ġmake Ġmodel Ġlearn Ġthe Ġrepresentation Ġof Ġsegment ation Ġmore Ġeffective Ġas ĠB IB REF 9 , ĠB IB REF 11 . </s> External Ġdata Ġand Ġpre - trained Ġembed ding . ĠWhereas Ġboth Ġenc oder Ġand Ġgraph Ġmodel Ġare Ġabout Ġexploring Ġa Ġway Ġto Ġget Ġbetter Ġperformance Ġonly Ġby Ġimproving Ġthe Ġmodel Ġstrength Ġitself . ĠUsing Ġexternal Ġresource Ġsuch Ġas Ġpre - trained Ġembed d ings Ġor Ġlanguage Ġrepresentation Ġis Ġan Ġalternative Ġfor Ġthe Ġsame Ġpurpose ĠB IB REF 22 , ĠB IB REF 23 . ĠS IGH AN ĠBake off Ġdefines Ġtwo Ġtypes Ġof Ġevaluation Ġsettings , Ġclosed Ġtest Ġlimits Ġall Ġthe Ġdata Ġfor Ġlearning Ġshould Ġnot Ġbe Ġbeyond Ġthe Ġgiven Ġtraining Ġset , Ġwhile Ġopen Ġtest Ġdoes Ġnot Ġtake Ġthis Ġlimitation ĠB IB REF 21 . ĠIn Ġthis Ġwork , Ġwe Ġwill Ġfocus Ġon Ġthe Ġclosed Ġtest Ġsetting Ġby Ġfinding Ġa Ġbetter Ġmodel Ġdesign Ġfor Ġfurther ĠC WS Ġperformance Ġimprovement . </s> Sh own Ġin ĠTable ĠT AB REF 1 , Ġdifferent Ġdec od ers Ġhave Ġparticular Ġdecoding Ġalgorithms Ġto Ġmatch Ġthe Ġrespective ĠC WS Ġmodels . ĠMark ov Ġmodels Ġand ĠCR F - based Ġmodels Ġoften Ġuse ĠV iter bi Ġdec od ers Ġwith Ġpo lyn omial Ġtime Ġcomplexity . ĠIn Ġgeneral Ġgraph Ġmodel , Ġsearch Ġspace Ġmay Ġbe Ġtoo Ġlarge Ġfor Ġmodel Ġto Ġsearch . ĠThus Ġit Ġforces Ġgraph Ġmodels Ġto Ġuse Ġan Ġapproximate Ġbeam Ġsearch Ġstrategy . ĠBeam Ġsearch Ġalgorithm Ġhas Ġa Ġkind Ġlow - order Ġpo lyn omial Ġtime Ġcomplexity . ĠEspecially , Ġwhen Ġbeam Ġwidth Ġ$ b $ = 1 , Ġthe Ġbeam Ġsearch Ġalgorithm Ġwill Ġreduce Ġto Ġgreedy Ġalgorithm Ġwith Ġa Ġbetter Ġtime Ġcomplexity Ġ$ O ( M n ) $ Ġagainst Ġthe Ġgeneral Ġbeam Ġsearch Ġtime Ġcomplexity Ġ$ O ( M nb ^ 2 ) $, Ġwhere Ġ$ n $ Ġis Ġthe Ġnumber Ġof Ġunits Ġin Ġone Ġsentences , Ġ$ M $ Ġis Ġa Ġconstant Ġrepresenting Ġthe Ġmodel Ġcomplexity . ĠGre edy Ġdecoding Ġalgorithm Ġcan Ġbring Ġthe Ġfastest Ġspeed Ġof Ġdecoding Ġwhile Ġit Ġis Ġnot Ġeasy Ġto Ġguarantee Ġthe Ġprecision Ġof Ġdecoding Ġwhen Ġthe Ġenc oder Ġis Ġnot Ġstrong Ġenough . </s> In Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġmore Ġeffective Ġenc oder Ġdesign Ġwhich Ġis Ġcapable Ġof Ġoffering Ġfast Ġand Ġaccurate ĠChinese Ġword Ġsegment ation Ġwith Ġonly Ġun ig ram Ġfeature Ġand Ġgreedy Ġdecoding . ĠOur Ġproposed Ġenc oder Ġwill Ġonly Ġconsist Ġof Ġattention Ġmechanisms Ġas Ġbuilding Ġblocks Ġbut Ġnothing Ġelse . ĠMot ivated Ġby Ġthe ĠTrans former ĠB IB REF 24 Ġand Ġits Ġstrength Ġof Ġcapturing Ġlong - range Ġdependencies Ġof Ġinput Ġsentences , Ġwe Ġuse Ġa Ġself - att ention Ġnetwork Ġto Ġgenerate Ġthe Ġrepresentation Ġof Ġinput Ġwhich Ġmakes Ġthe Ġmodel Ġencode Ġsentences Ġat Ġonce Ġwithout Ġfeeding Ġinput Ġiter atively . ĠConsidering Ġthe Ġweakness Ġof Ġthe ĠTrans former Ġto Ġmodel Ġrelative Ġand Ġabsolute Ġposition Ġinformation Ġdirectly ĠB IB REF 25 Ġand Ġthe Ġimportance Ġof Ġlocal ness Ġinformation , Ġposition Ġinformation Ġand Ġdirectional Ġinformation Ġfor ĠC WS , Ġwe Ġfurther Ġimprove Ġthe Ġarchitecture Ġof Ġstandard Ġmulti - head Ġself - att ention Ġof Ġthe ĠTrans former Ġwith Ġa Ġdirectional ĠGa ussian Ġmask Ġand Ġget Ġa Ġvariant Ġcalled ĠGa ussian - mask ed Ġdirectional Ġmulti - head Ġattention . ĠBased Ġon Ġthe Ġnewly Ġimproved Ġattention Ġmechanism , Ġwe Ġexpand Ġthe Ġenc oder Ġof Ġthe ĠTrans former Ġto Ġcapture Ġdifferent Ġdirectional Ġinformation . ĠWith Ġour Ġpowerful Ġenc oder , Ġour Ġmodel Ġuses Ġonly Ġsimple Ġun ig ram Ġfeatures Ġto Ġgenerate Ġrepresentation Ġof Ġsentences . </s> For Ġdec oder Ġwhich Ġdirectly Ġperforms Ġthe Ġsegment ation , Ġwe Ġuse Ġthe Ġbi - aff inal Ġattention Ġscorer , Ġwhich Ġhas Ġbeen Ġused Ġin Ġdependency Ġparsing ĠB IB REF 26 Ġand Ġsemantic Ġrole Ġlabeling ĠB IB REF 27 , Ġto Ġimplement Ġgreedy Ġdecoding Ġon Ġfinding Ġthe Ġboundaries Ġof Ġwords . ĠIn Ġour Ġproposed Ġmodel , Ġgreedy Ġdecoding Ġensures Ġa Ġfast Ġsegment ation Ġwhile Ġpowerful Ġenc oder Ġdesign Ġensures Ġa Ġgood Ġenough Ġsegment ation Ġperformance Ġeven Ġworking Ġwith Ġgreedy Ġdec oder Ġtogether . ĠOur Ġmodel Ġwill Ġbe Ġstrictly Ġevaluated Ġon Ġbenchmark Ġdatasets Ġfrom ĠS IGH AN ĠBake off Ġshared Ġtask Ġon ĠC WS Ġin Ġterms Ġof Ġclosed Ġtest Ġsetting , Ġand Ġthe Ġexperimental Ġresults Ġshow Ġthat Ġour Ġproposed Ġmodel Ġachieves Ġnew Ġstate - of - the - art . </s> The Ġtechnical Ġcontributions Ġof Ġthis Ġpaper Ġcan Ġbe Ġsummarized Ġas Ġfollows . </s> We Ġpropose Ġa ĠC WS Ġmodel Ġwith Ġonly Ġattention Ġstructure . ĠThe Ġenc oder Ġand Ġdec oder Ġare Ġboth Ġbased Ġon Ġattention Ġstructure . </s> With Ġa Ġpowerful Ġenough Ġenc oder , Ġwe Ġfor Ġthe Ġfirst Ġtime Ġshow Ġthat Ġun ig ram Ġ( character ) Ġfeat ues Ġcan Ġhelp Ġyield Ġstrong Ġperformance Ġinstead Ġof Ġdiverse Ġ$ n $ - gram Ġ( character Ġand Ġword ) Ġfeatures Ġin Ġmost Ġof Ġprevious Ġwork . </s> To Ġcapture Ġthe Ġrepresentation Ġof Ġlocal ness Ġinformation Ġand Ġdirectional Ġinformation , Ġwe Ġpropose Ġa Ġvariant Ġof Ġdirectional Ġmulti - head Ġself - att ention Ġto Ġfurther Ġenhance Ġthe Ġstate - of - the - art ĠTrans former Ġenc oder . </s> Mod els </s> The ĠC WS Ġtask Ġis Ġoften Ġmod elled Ġas Ġone Ġgraph Ġmodel Ġbased Ġon Ġan Ġenc oder - based Ġscoring Ġmodel . ĠThe Ġmodel Ġfor ĠC WS Ġtask Ġis Ġcomposed Ġof Ġan Ġenc oder Ġto Ġrepresent Ġthe Ġinput Ġand Ġa Ġdec oder Ġbased Ġon Ġthe Ġenc oder Ġto Ġperform Ġactual Ġsegment ation . ĠFigure ĠFIG REF 6 Ġis Ġthe Ġarchitecture Ġof Ġour Ġmodel . ĠThe Ġmodel Ġfeeds Ġsentence Ġinto Ġenc oder . ĠEmb edd ing Ġcaptures Ġthe Ġvector Ġ$ e =( e _ 1 ,... , e _ n ) $ Ġof Ġthe Ġinput Ġcharacter Ġsequences Ġof Ġ$ c =( c _ 1 ,... , c _ n ) $. ĠThe Ġenc oder Ġmaps Ġvector Ġsequences Ġof Ġ$ Ġ{ e } =( e _ 1 , .. , e _ n ) $ Ġto Ġtwo Ġsequences Ġof Ġvector Ġwhich Ġare Ġ$ Ġ{ v ^ b } =( v _ 1 ^ b ,... , v _ n ^ b ) $ Ġand Ġ${ v ^ f } =( v _ 1 ^ f ,... v _ n ^ f ) $ Ġas Ġthe Ġrepresentation Ġof Ġsentences . ĠWith Ġ$ v ^ b $ Ġand Ġ$ v ^ f $, Ġthe Ġbi - aff inal Ġscorer Ġcalculates Ġthe Ġprobability Ġof Ġeach Ġsegment ation Ġgaps Ġand Ġpredicts Ġthe Ġword Ġboundaries Ġof Ġinput . ĠSimilar Ġas Ġthe ĠTrans former , Ġthe Ġenc oder Ġis Ġan Ġattention Ġnetwork Ġwith Ġstacked Ġself - att ention Ġand Ġpoint - wise , Ġfully Ġconnected Ġlayers Ġwhile Ġour Ġenc oder Ġincludes Ġthree Ġindependent Ġdirectional Ġenc od ers . </s> Mod els Ġ: :: ĠEnc oder ĠSt acks </s> In Ġthe ĠTrans former , Ġthe Ġenc oder Ġis Ġcomposed Ġof Ġa Ġstack Ġof ĠN Ġidentical Ġlayers Ġand Ġeach Ġlayer Ġhas Ġone Ġmulti - head Ġself - att ention Ġlayer Ġand Ġone Ġposition - wise Ġfully Ġconnected Ġfeed - forward Ġlayer . ĠOne Ġresidual Ġconnection Ġis Ġaround Ġtwo Ġsub - l ayers Ġand Ġfollowed Ġby Ġlayer Ġnormal ization ĠB IB REF 24 . ĠThis Ġarchitecture Ġprovides Ġthe ĠTrans former Ġa Ġgood Ġability Ġto Ġgenerate Ġrepresentation Ġof Ġsentence . </s> With Ġthe Ġvariant Ġof Ġmulti - head Ġself - att ention , Ġwe Ġdesign Ġa ĠGa ussian - mask ed Ġdirectional Ġenc oder Ġto Ġcapture Ġrepresentation Ġof Ġdifferent Ġdirections Ġto Ġimprove Ġthe Ġability Ġof Ġcapturing Ġthe Ġlocal ness Ġinformation Ġand Ġposition Ġinformation Ġfor Ġthe Ġimportance Ġof Ġadjacent Ġcharacters . ĠOne Ġun id irection al Ġenc oder Ġcan Ġcapture Ġinformation Ġof Ġone Ġparticular Ġdirection . </s> For ĠC WS Ġtasks , Ġone Ġgap Ġof Ġcharacters , Ġwhich Ġis Ġfrom Ġa Ġword Ġboundary , Ġcan Ġdivide Ġone Ġsequence Ġinto Ġtwo Ġparts , Ġone Ġpart Ġin Ġfront Ġof Ġthe Ġgap Ġand Ġone Ġpart Ġin Ġthe Ġrear Ġof Ġit . ĠThe Ġforward Ġenc oder Ġand Ġbackward Ġenc oder Ġare Ġused Ġto Ġcapture Ġinformation Ġof Ġtwo Ġdirections Ġwhich Ġcorrespond Ġto Ġtwo Ġparts Ġdivided Ġby Ġthe Ġgap . </s> One Ġcentral Ġenc oder Ġis Ġparalle led Ġwith Ġforward Ġand Ġbackward Ġenc od ers Ġto Ġcapture Ġthe Ġinformation Ġof Ġentire Ġsentences . ĠThe Ġcentral Ġenc oder Ġis Ġa Ġspecial Ġdirectional Ġenc oder Ġfor Ġforward Ġand Ġbackward Ġinformation Ġof Ġsentences . ĠThe Ġcentral Ġenc oder Ġcan Ġfuse Ġthe Ġinformation Ġand Ġenable Ġthe Ġenc oder Ġto Ġcapture Ġthe Ġglobal Ġinformation . </s> The Ġenc oder Ġoutputs Ġone Ġforward Ġinformation Ġand Ġone Ġbackward Ġinformation Ġof Ġeach Ġpositions . ĠThe Ġrepresentation Ġof Ġsentence Ġgenerated Ġby Ġcenter Ġenc oder Ġwill Ġbe Ġadded Ġto Ġthese Ġinformation Ġdirectly : </s> where Ġ$ v ^{ b } =( v ^ b _ 1 ,... , v ^ b _ n ) $ Ġis Ġthe Ġbackward Ġinformation , Ġ$ v ^{ f } =( v ^ f _ 1 ,... , v ^ f _ n ) $ Ġis Ġthe Ġforward Ġinformation , Ġ$ r ^{ b } =( r ^ b _ 1 ,... , r ^ b _ n ) $ Ġis Ġthe Ġoutput Ġof Ġbackward Ġenc oder , Ġ$ r ^{ c } =( r ^ c _ 1 ,... , r ^ c _ n ) $ Ġis Ġthe Ġoutput Ġof Ġcenter Ġenc oder Ġand Ġ$ r ^{ f } =( r ^ f _ 1 ,... , r ^ f _ n ) $ Ġis Ġthe Ġoutput Ġof Ġforward Ġenc oder . </s> Mod els Ġ: :: ĠGa ussian - Mask ed ĠDirection al ĠMulti - Head ĠAttention </s> Similar Ġas Ġscaled Ġdot - product Ġattention ĠB IB REF 24 , ĠGa ussian - mask ed Ġdirectional Ġattention Ġcan Ġbe Ġdescribed Ġas Ġa Ġfunction Ġto Ġmap Ġqueries Ġand Ġkey - value Ġpairs Ġto Ġthe Ġrepresentation Ġof Ġinput . ĠHere Ġqueries , Ġkeys Ġand Ġvalues Ġare Ġall Ġvectors . ĠStandard Ġscaled Ġdot - product Ġattention Ġis Ġcalculated Ġby Ġdot ting Ġquery Ġ$ Q $ Ġwith Ġall Ġkeys Ġ$ K $, Ġdividing Ġeach Ġvalues Ġby Ġ$\ sq rt { d _ k } $, Ġwhere Ġ$\ sq rt { d _ k } $ Ġis Ġthe Ġdimension Ġof Ġkeys , Ġand Ġapply Ġa Ġsoft max Ġfunction Ġto Ġgenerate Ġthe Ġweights Ġin Ġthe Ġattention : </s> Different Ġfrom Ġscaled Ġdot - product Ġattention , ĠGa ussian - mask ed Ġdirectional Ġattention Ġexpects Ġto Ġpay Ġattention Ġto Ġthe Ġadjacent Ġcharacters Ġof Ġeach Ġpositions Ġand Ġcast Ġthe Ġlocal ness Ġrelationship Ġbetween Ġcharacters Ġas Ġa Ġfix ĠGa ussian Ġweight Ġfor Ġattention . ĠWe Ġassume Ġthat Ġthe ĠGa ussian Ġweight Ġonly Ġrely s Ġon Ġthe Ġdistance Ġbetween Ġcharacters . </s> Firstly Ġwe Ġintroduce Ġthe ĠGa ussian Ġweight Ġmatrix Ġ$ G $ Ġwhich Ġpresents Ġthe Ġlocal ness Ġrelationship Ġbetween Ġeach Ġtwo Ġcharacters : </s> where Ġ$ g _{ ij } $ Ġis Ġthe ĠGa ussian Ġweight Ġbetween Ġcharacter Ġ$ i $ Ġand Ġ$ j $, Ġ$ dis _{ ij } $ Ġis Ġthe Ġdistance Ġbetween Ġcharacter Ġ$ i $ Ġand Ġ$ j $, Ġ$\ Ph i Ġ( x ) $ Ġis Ġthe Ġcumulative Ġdistribution Ġfunction Ġof ĠGa ussian , Ġ$\ s igma Ġ$ Ġis Ġthe Ġstandard Ġdeviation Ġof ĠGa ussian Ġfunction Ġand Ġit Ġis Ġa Ġhyper param eter Ġin Ġour Ġmethod . ĠEqu ation Ġ( DIS PLAY _ FORM 13 ) Ġcan Ġensure Ġthe ĠGa ussian Ġweight Ġequals Ġ1 Ġwhen Ġ$ dis _{ ij } $ Ġis Ġ0 . ĠThe Ġlarger Ġdistance Ġbetween Ġcharacters is , Ġthe Ġsmaller Ġthe Ġweight Ġis , Ġwhich Ġmakes Ġone Ġcharacter Ġcan Ġaffect Ġits Ġadjacent Ġcharacters Ġmore Ġcompared Ġwith Ġother Ġcharacters . </s> To Ġcombine Ġthe ĠGa ussian Ġweight Ġto Ġthe Ġself - att ention , Ġwe Ġproduce Ġthe ĠHad am ard Ġproduct Ġof ĠGa ussian Ġweight Ġmatrix Ġ$ G $ Ġand Ġthe Ġscore Ġmatrix Ġproduced Ġby Ġ$ Q { K ^{ T }} $ </s> where Ġ$ AG $ Ġis Ġthe ĠGa ussian - mask ed Ġattention . ĠIt Ġensures Ġthat Ġthe Ġrelationship Ġbetween Ġtwo Ġcharacters Ġwith Ġlong Ġdistances Ġis Ġweaker Ġthan Ġadjacent Ġcharacters . </s> The Ġscaled Ġdot - product Ġattention Ġmodels Ġthe Ġrelationship Ġbetween Ġtwo Ġcharacters Ġwithout Ġregard Ġto Ġtheir Ġdistances Ġin Ġone Ġsequence . ĠFor ĠC WS Ġtask , Ġthe Ġweight Ġbetween Ġadjacent Ġcharacters Ġshould Ġbe Ġmore Ġimportant Ġwhile Ġit Ġis Ġhard Ġfor Ġself - att ention Ġto Ġachieve Ġthe Ġeffect Ġexplicitly Ġbecause Ġthe Ġself - att ention Ġcannot Ġget Ġthe Ġorder Ġof Ġsentences Ġdirectly . ĠThe ĠGa ussian - mask ed Ġattention Ġadjusts Ġthe Ġweight Ġbetween Ġcharacters Ġand Ġtheir Ġadjacent Ġcharacter Ġto Ġa Ġlarger Ġvalue Ġwhich Ġstands Ġfor Ġthe Ġeffect Ġof Ġadjacent Ġcharacters . </s> For Ġforward Ġand Ġbackward Ġenc oder , Ġthe Ġself - att ention Ġsub layer Ġneeds Ġto Ġuse Ġa Ġtriangular Ġmatrix Ġmask Ġto Ġlet Ġthe Ġself - att ention Ġfocus Ġon Ġdifferent Ġweights : </s> where Ġ$ pos _ i $ Ġis Ġthe Ġposition Ġof Ġcharacter Ġ$ c _ i $. ĠThe Ġtriangular Ġmatrix Ġfor Ġforward Ġand Ġbackward Ġencode Ġare : </s> $ \ left [ Ġ\ begin { mat rix } Ġ1 Ġ& Ġ0 Ġ& Ġ0 Ġ& Ġ\ cd ots Ġ& 0 \\ Ġ1 Ġ& Ġ1 Ġ& Ġ0 Ġ& Ġ\ cd ots Ġ& 0 \\ Ġ1 Ġ& Ġ1 Ġ& Ġ1 Ġ& Ġ\ cd ots Ġ& 0 \\ Ġ\ vd ots Ġ& \ vd ots Ġ& \ vd ots Ġ& \ dd ots Ġ& \ vd ots Ġ\\ Ġ1 Ġ& Ġ1 Ġ& Ġ1 Ġ& Ġ\ cd ots Ġ& Ġ1 \\ Ġ\ end { mat rix } Ġ\ right ] $ Ġ$\ left [ Ġ\ begin { mat rix } Ġ1 Ġ& Ġ1 Ġ& Ġ1 Ġ& Ġ\ cd ots Ġ& 1 Ġ\\ Ġ0 Ġ& Ġ1 Ġ& Ġ1 Ġ& Ġ\ cd ots Ġ& 1 Ġ\\ Ġ0 Ġ& Ġ0 & Ġ1 Ġ& Ġ\ cd ots Ġ& 1 Ġ\\ Ġ\ vd ots Ġ& \ vd ots Ġ& \ vd ots Ġ& \ dd ots Ġ& \ vd ots Ġ\\ Ġ0 Ġ& Ġ0 Ġ& Ġ0 Ġ& Ġ\ cd ots Ġ& Ġ1 \\ Ġ\ end { mat rix }\ right ] $ </s> Similar Ġas ĠB IB REF 24 , Ġwe Ġuse Ġmulti - head Ġattention Ġto Ġcapture Ġinformation Ġfrom Ġdifferent Ġdimension Ġpositions Ġas ĠFigure ĠFIG REF 16 Ġand Ġget ĠGa ussian - mask ed Ġdirectional Ġmulti - head Ġattention . ĠWith Ġmulti - head Ġattention Ġarchitecture , Ġthe Ġrepresentation Ġof Ġinput Ġcan Ġbe Ġcaptured Ġby </s> where Ġ$ MH $ Ġis Ġthe ĠGa ussian - mask ed Ġmulti - head Ġattention , Ġ${ W _ i ^ q , ĠW _ i ^ k , W _ i ^ v } Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ k Ġ\ times Ġd _ h } $ Ġis Ġthe Ġparameter Ġmat rices Ġto Ġgenerate Ġheads , Ġ$ d _ k $ Ġis Ġthe Ġdimension Ġof Ġmodel Ġand Ġ$ d _ h $ Ġis Ġthe Ġdimension Ġof Ġone Ġhead . </s> Mod els Ġ: :: ĠBi - aff inal ĠAttention ĠSc orer </s> Regarding Ġword Ġboundaries Ġas Ġgaps Ġbetween Ġany Ġadjacent Ġwords Ġconverts Ġthe Ġcharacter Ġlabeling Ġtask Ġto Ġthe Ġgap Ġlabeling Ġtask . ĠDifferent Ġfrom Ġcharacter Ġlabeling Ġtask , Ġgap Ġlabeling Ġtask Ġrequires Ġinformation Ġof Ġtwo Ġadjacent Ġcharacters . ĠThe Ġrelationship Ġbetween Ġadjacent Ġcharacters Ġcan Ġbe Ġrepresented Ġas Ġthe Ġtype Ġof Ġgap . ĠThe Ġcharacteristic Ġof Ġword Ġboundaries Ġmakes Ġbi - aff ine Ġattention Ġan Ġappropriate Ġscorer Ġfor ĠC WS Ġtask . </s> Bi - aff inal Ġattention Ġscorer Ġis Ġthe Ġcomponent Ġthat Ġwe Ġuse Ġto Ġlabel Ġthe Ġgap . ĠBi - aff inal Ġattention Ġis Ġdeveloped Ġfrom Ġbil in ear Ġattention Ġwhich Ġhas Ġbeen Ġused Ġin Ġdependency Ġparsing ĠB IB REF 26 Ġand ĠS RL ĠB IB REF 27 . ĠThe Ġdistribution Ġof Ġlabels Ġin Ġa Ġlabeling Ġtask Ġis Ġoften Ġuneven Ġwhich Ġmakes Ġthe Ġoutput Ġlayer Ġoften Ġinclude Ġa Ġfixed Ġbias Ġterm Ġfor Ġthe Ġprior Ġprobability Ġof Ġdifferent Ġlabels ĠB IB REF 27 . ĠBi - aff ine Ġattention Ġuses Ġbias Ġterms Ġto Ġalleviate Ġthe Ġburden Ġof Ġthe Ġfixed Ġbias Ġterm Ġand Ġget Ġthe Ġprior Ġprobability Ġwhich Ġmakes Ġit Ġdifferent Ġfrom Ġbil in ear Ġattention . ĠThe Ġdistribution Ġof Ġthe Ġgap Ġis Ġuneven Ġthat Ġis Ġsimilar Ġas Ġother Ġlabeling Ġtask Ġwhich Ġfits Ġbi - aff ine . </s> Bi - aff inal Ġattention Ġscorer Ġlabels Ġthe Ġtarget Ġdepending Ġon Ġinformation Ġof Ġindependent Ġunit Ġand Ġthe Ġjoint Ġinformation Ġof Ġtwo Ġunits . ĠIn Ġbi - aff inal Ġattention , Ġthe Ġscore Ġ$ s _{ ij } $ Ġof Ġcharacters Ġ$ c _ i $ Ġand Ġ$ c _ j $ Ġ$( i Ġ< Ġj ) $ Ġis Ġcalculated Ġby : </s> where Ġ$ v _ i ^ f $ Ġis Ġthe Ġforward Ġinformation Ġof Ġ$ c _ i $ Ġand Ġ$ v _ i ^ b $ Ġis Ġthe Ġbackward Ġinformation Ġof Ġ$ c _ j $. ĠIn ĠEqu ation Ġ( DIS PLAY _ FORM 21 ), Ġ$ W $, Ġ$ U $ Ġand Ġ$ b $ Ġare Ġall Ġparameters Ġthat Ġcan Ġbe Ġupdated Ġin Ġtraining . Ġ$ W $ Ġis Ġa Ġmatrix Ġwith Ġshape Ġ$( d _ i Ġ\ times ĠN \ times Ġd _ j ) $ Ġand Ġ$ U $ Ġis Ġa Ġ$( N \ times Ġ( d _ i Ġ+ Ġd _ j )) $ Ġmatrix Ġwhere Ġ$ d _ i $ Ġis Ġthe Ġdimension Ġof Ġvector Ġ$ v _ i ^ f $ Ġand Ġ$ N $ Ġis Ġthe Ġnumber Ġof Ġlabels . </s> In Ġour Ġmodel , Ġthe Ġb ia ff ine Ġscorer Ġuses Ġthe Ġforward Ġinformation Ġof Ġcharacter Ġin Ġfront Ġof Ġthe Ġgap Ġand Ġthe Ġbackward Ġinformation Ġof Ġthe Ġcharacter Ġbehind Ġthe Ġgap Ġto Ġdistinguish Ġthe Ġposition Ġof Ġcharacters . ĠFigure ĠFIG REF 22 Ġis Ġan Ġexample Ġof Ġlabeling Ġgap . ĠThe Ġmethod Ġof Ġusing Ġb ia ff ine Ġscorer Ġensures Ġthat Ġthe Ġboundaries Ġof Ġwords Ġcan Ġbe Ġdetermined Ġby Ġadjacent Ġcharacters Ġwith Ġdifferent Ġdirectional Ġinformation . ĠThe Ġscore Ġvector Ġof Ġthe Ġgap Ġis Ġformed Ġby Ġthe Ġprobability Ġof Ġbeing Ġa Ġboundary Ġof Ġword . ĠFurther , Ġthe Ġmodel Ġgenerates Ġall Ġboundaries Ġusing Ġactivation Ġfunction Ġin Ġa Ġgreedy Ġdecoding Ġway . </s> Exper iments Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠData </s> We Ġtrain Ġand Ġevaluate Ġour Ġmodel Ġon Ġdatasets Ġfrom ĠS IGH AN ĠBake off Ġ2005 ĠB IB REF 21 Ġwhich Ġhas Ġfour Ġdatasets , ĠPK U , ĠMS R , ĠAS Ġand ĠCITY U . ĠTable ĠT AB REF 23 Ġshows Ġthe Ġstatistics Ġof Ġtrain Ġdata . ĠWe Ġuse ĠF - score Ġto Ġevaluate ĠC WS Ġmodels . ĠTo Ġtrain Ġmodel Ġwith Ġpre - trained Ġembed d ings Ġin ĠAS Ġand ĠCITY U , Ġwe Ġuse ĠOpen CC Ġto Ġtransfer Ġdata Ġfrom Ġtraditional ĠChinese Ġto Ġsimplified ĠChinese . </s> Exper iments Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠPre - trained ĠEmb edd ing </s> We Ġonly Ġuse Ġun ig ram Ġfeature Ġso Ġwe Ġonly Ġtrained Ġcharacter Ġembed d ings . ĠOur Ġpre - trained Ġembed ding Ġare Ġpre - trained Ġon ĠChinese ĠWikipedia Ġcorpus Ġby Ġword 2 vec ĠB IB REF 29 Ġtool kit . ĠThe Ġcorpus Ġused Ġfor Ġpre - trained Ġembed ding Ġis Ġall Ġtransferred Ġto Ġsimplified ĠChinese Ġand Ġnot Ġsegment ed . ĠOn Ġclosed Ġtest , Ġwe Ġuse Ġembed d ings Ġinitialized Ġrandomly . </s> Exper iments Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠHyper param eters </s> For Ġdifferent Ġdatasets , Ġwe Ġuse Ġtwo Ġkinds Ġof Ġhyper param eters Ġwhich Ġare Ġpresented Ġin ĠTable ĠT AB REF 24 . ĠWe Ġuse Ġhyper param eters Ġin ĠTable ĠT AB REF 24 Ġfor Ġsmall Ġcorpor a Ġ( PK U Ġand ĠCITY U ) Ġand Ġnormal Ġcorpor a Ġ( MS R Ġand ĠAS ). ĠWe Ġset Ġthe Ġstandard Ġdeviation Ġof ĠGa ussian Ġfunction Ġin ĠEqu ation Ġ( DIS PLAY _ FORM 13 ) Ġto Ġ2 . ĠEach Ġtraining Ġbatch Ġcontains Ġsentences Ġwith Ġat Ġmost Ġ4096 Ġtokens . </s> Exper iments Ġ: :: ĠExperimental ĠSettings Ġ: :: ĠOptim izer </s> To Ġtrain Ġour Ġmodel , Ġwe Ġuse Ġthe ĠAdam ĠB IB REF 30 Ġoptim izer Ġwith Ġ$\ beta Ġ_ 1 = 0 . 9 $, Ġ$\ beta Ġ_ 2 = 0 . 98 $ Ġand Ġ$\ ep sil on Ġ= 10 ^{ - 9 } $. ĠThe Ġlearning Ġrate Ġschedule Ġis Ġthe Ġsame Ġas ĠB IB REF 24 : </s> where Ġ$ d $ Ġis Ġthe Ġdimension Ġof Ġembed d ings , Ġ$ step $ Ġis Ġthe Ġstep Ġnumber Ġof Ġtraining Ġand Ġ$ warm up _ step $ Ġis Ġthe Ġstep Ġnumber Ġof Ġwarm up . ĠWhen Ġthe Ġnumber Ġof Ġsteps Ġis Ġsmaller Ġthan Ġthe Ġstep Ġof Ġwarm up , Ġthe Ġlearning Ġrate Ġincreases Ġlin early Ġand Ġthen Ġdecreases . </s> Exper iments Ġ: :: ĠHardware Ġand ĠIm ple ments </s> We Ġtrained Ġour Ġmodels Ġon Ġa Ġsingle ĠCPU Ġ( Intel Ġi 7 - 59 60 X ) Ġwith Ġan Ġn V idia Ġ1080 ĠTi ĠGPU . ĠWe Ġimplement Ġour Ġmodel Ġin ĠPython Ġwith ĠPy tor ch Ġ1 . 0 . </s> Exper iments Ġ: :: ĠResults </s> T ables ĠT AB REF 25 Ġand ĠT AB REF 26 Ġreports Ġthe Ġperformance Ġof Ġrecent Ġmodels Ġand Ġours Ġin Ġterms Ġof Ġclosed Ġtest Ġsetting . ĠWithout Ġthe Ġassistance Ġof Ġun super vised Ġsegment ation Ġfeatures Ġuser d Ġin ĠB IB REF 20 , Ġour Ġmodel Ġoutper forms Ġall Ġthe Ġother Ġmodels Ġin ĠMS R Ġand ĠAS Ġexcept ĠB IB REF 18 Ġand Ġget Ġcomparable Ġperformance Ġin ĠPK U Ġand ĠCITY U . ĠNote Ġthat Ġall Ġthe Ġother Ġmodels Ġfor Ġthis Ġcomparison Ġadopt Ġvarious Ġ$ n $ - gram Ġfeatures Ġwhile Ġonly Ġour Ġmodel Ġtakes Ġun ig ram Ġones . </s> With Ġun super vised Ġsegment ation Ġfeatures Ġintroduced Ġby ĠB IB REF 20 , Ġour Ġmodel Ġgets Ġa Ġhigher Ġresult . ĠS pecially , Ġthe Ġresults Ġin ĠMS R Ġand ĠAS Ġachieve Ġnew Ġstate - of - the - art Ġand Ġapproaching Ġprevious Ġstate - of - the - art Ġin ĠCITY U Ġand ĠPK U . ĠThe Ġun super vised Ġsegment ation Ġfeatures Ġare Ġderived Ġfrom Ġthe Ġgiven Ġtraining Ġdataset , Ġthus Ġusing Ġthem Ġdoes Ġnot Ġviolate Ġthe Ġrule Ġof Ġclosed Ġtest Ġof ĠS IGH AN ĠBake off . </s> Table ĠT AB REF 36 Ġcompares Ġour Ġmodel Ġand Ġrecent Ġneural Ġmodels Ġin Ġterms Ġof Ġopen Ġtest Ġsetting Ġin Ġwhich Ġany Ġexternal Ġresources , Ġespecially Ġpre - trained Ġembed d ings Ġor Ġlanguage Ġmodels Ġcan Ġbe Ġused . ĠIn ĠMS R Ġand ĠAS , Ġour Ġmodel Ġgets Ġa Ġcomparable Ġresult Ġwhile Ġour Ġresults Ġin ĠCITY U Ġand ĠPK U Ġare Ġnot Ġremarkable . </s> However , Ġit Ġis Ġwell Ġknown Ġthat Ġit Ġis Ġalways Ġhard Ġto Ġcompare Ġmodels Ġwhen Ġusing Ġopen Ġtest Ġsetting , Ġespecially Ġwith Ġpre - trained Ġembed ding . ĠNot Ġall Ġmodels Ġmay Ġuse Ġthe Ġsame Ġmethod Ġand Ġdata Ġto Ġpre - train . ĠThough Ġpre - trained Ġembed ding Ġor Ġlanguage Ġmodel Ġcan Ġimprove Ġthe Ġperformance , Ġthe Ġperformance Ġimprovement Ġitself Ġmay Ġbe Ġfrom Ġmultiple Ġsources . ĠIt Ġoften Ġthat Ġthere Ġis Ġa Ġsuccess Ġof Ġpre - trained Ġembed ding Ġto Ġimprove Ġthe Ġperformance , Ġwhile Ġit Ġcannot Ġprove Ġthat Ġthe Ġmodel Ġis Ġbetter . </s> Compared Ġwith Ġother ĠL ST M Ġmodels , Ġour Ġmodel Ġperforms Ġbetter Ġin ĠAS Ġand ĠMS R Ġthan Ġin ĠCITY U Ġand ĠPK U . ĠConsidering Ġthe Ġscale Ġof Ġdifferent Ġcorpor a , Ġwe Ġbelieve Ġthat Ġthe Ġsize Ġof Ġcorpus Ġaffects Ġour Ġmodel Ġand Ġthe Ġlarger Ġsize Ġis , Ġthe Ġbetter Ġmodel Ġperforms . ĠFor Ġsmall Ġcorpus , Ġthe Ġmodel Ġtends Ġto Ġbe Ġover fitting . </s> T ables ĠT AB REF 25 Ġand ĠT AB REF 26 Ġalso Ġshow Ġthe Ġdecoding Ġtime Ġin Ġdifferent Ġdatasets . ĠOur Ġmodel Ġfinishes Ġthe Ġsegment ation Ġwith Ġthe Ġleast Ġdecoding Ġtime Ġin Ġall Ġfour Ġdatasets , Ġthanks Ġto Ġthe Ġarchitecture Ġof Ġmodel Ġwhich Ġonly Ġtakes Ġattention Ġmechanism Ġas Ġbasic Ġblock . </s> Related ĠWork Ġ: :: ĠChinese ĠWord ĠSe gment ation </s> C WS Ġis Ġa Ġtask Ġfor ĠChinese Ġnatural Ġlanguage Ġprocess Ġto Ġdelim it Ġword Ġboundary . ĠB IB REF 0 Ġfor Ġthe Ġfirst Ġtime Ġform ul ize ĠC WS Ġas Ġa Ġsequence Ġlabeling Ġtask . ĠB IB REF 3 Ġshow Ġthat Ġdifferent Ġcharacter Ġtag Ġsets Ġcan Ġmake Ġessential Ġimpact Ġfor ĠC WS . ĠB IB REF 2 Ġuse ĠCR Fs Ġas Ġa Ġmodel Ġfor ĠC WS , Ġachieving Ġnew Ġstate - of - the - art . ĠWorks Ġof Ġstatistical ĠC WS Ġhas Ġbuilt Ġthe Ġbasis Ġfor Ġneural ĠC WS . </s> Ne ural Ġword Ġsegment ation Ġhas Ġbeen Ġwidely Ġused Ġto Ġminimize Ġthe Ġefforts Ġin Ġfeature Ġengineering Ġwhich Ġwas Ġimportant Ġin Ġstatistical ĠC WS . ĠB IB REF 4 Ġintroduce Ġthe Ġneural Ġmodel Ġwith Ġsliding - window Ġbased Ġsequence Ġlabeling . ĠB IB REF 6 Ġpropose Ġa Ġg ated Ġrecursive Ġneural Ġnetwork Ġ( GR NN ) Ġfor ĠC WS Ġto Ġincorporate Ġcomplicated Ġcombination Ġof Ġcontextual Ġcharacter Ġand Ġn - gram Ġfeatures . ĠB IB REF 7 Ġuse ĠL ST M Ġto Ġlearn Ġlong Ġdistance Ġinformation . ĠB IB REF 9 Ġpropose Ġa Ġneural Ġframework Ġthat Ġeliminates Ġcontext Ġwindows Ġand Ġutilize Ġcomplete Ġsegment ation Ġhistory . ĠB IB REF 33 Ġexplore Ġa Ġjoint Ġmodel Ġthat Ġperforms Ġsegment ation , ĠPOS - T agging Ġand Ġchunk ing Ġsimultaneously . ĠB IB REF 34 Ġpropose Ġa Ġfeature - en riched Ġneural Ġmodel Ġfor Ġjoint ĠC WS Ġand Ġpart - of - speech Ġtagging . ĠB IB REF 35 Ġpresent Ġa Ġjoint Ġmodel Ġto Ġenhance Ġthe Ġsegment ation Ġof ĠChinese Ġmicro text Ġby Ġperforming ĠC WS Ġand Ġinformal Ġword Ġdetection Ġsimultaneously . ĠB IB REF 17 Ġpropose Ġa Ġcharacter - based Ġconv olution al Ġneural Ġmodel Ġto Ġcapture Ġ$ n $ - gram Ġfeatures Ġautomatically Ġand Ġan Ġeffective Ġapproach Ġto Ġincorporate Ġword Ġembed d ings . ĠB IB REF 11 Ġimprove Ġthe Ġmodel Ġin ĠB IB REF 9 Ġand Ġpropose Ġa Ġgreedy Ġneural Ġword Ġsegment er Ġwith Ġbalanced Ġword Ġand Ġcharacter Ġembed ding Ġinputs . ĠB IB REF 23 Ġpropose Ġa Ġnovel Ġneural Ġnetwork Ġmodel Ġto Ġincorporate Ġunl abel ed Ġand Ġpartially - label ed Ġdata . ĠB IB REF 36 Ġpropose Ġtwo Ġmethods Ġthat Ġextend Ġthe ĠBi - L ST M Ġto Ġperform Ġincorporating Ġdiction aries Ġinto Ġneural Ġnetworks Ġfor ĠC WS . ĠB IB REF 37 Ġpropose ĠSwitch - L ST Ms Ġto Ġsegment Ġwords Ġand Ġprovided Ġa Ġmore Ġflexible Ġsolution Ġfor Ġmulti - crit eria ĠC WS Ġwhich Ġis Ġeasy Ġto Ġtransfer Ġthe Ġlearned Ġknowledge Ġto Ġnew Ġcriteria . </s> Related ĠWork Ġ: :: ĠTrans former </s> Trans former ĠB IB REF 24 Ġis Ġan Ġattention - based Ġneural Ġmachine Ġtranslation Ġmodel . ĠThe ĠTrans former Ġis Ġone Ġkind Ġof Ġself - att ention Ġnetworks Ġ( SAN s ) Ġwhich Ġis Ġproposed Ġin ĠB IB REF 38 . ĠEnc oder Ġof Ġthe ĠTrans former Ġconsists Ġof Ġone Ġself - att ention Ġlayer Ġand Ġa Ġposition - wise Ġfeed - forward Ġlayer . ĠDecoder Ġof Ġthe ĠTrans former Ġcontains Ġone Ġself - att ention Ġlayer , Ġone Ġenc oder - dec oder Ġattention Ġlayer Ġand Ġone Ġposition - wise Ġfeed - forward Ġlayer . ĠThe ĠTrans former Ġuses Ġresidual Ġconnections Ġaround Ġthe Ġsub l ayers Ġand Ġthen Ġfollowed Ġby Ġa Ġlayer Ġnormal ization Ġlayer . </s> Sc aled Ġdot - product Ġattention Ġis Ġthe Ġkey Ġcomponent Ġin Ġthe ĠTrans former . ĠThe Ġinput Ġof Ġattention Ġcontains Ġqueries , Ġkeys , Ġand Ġvalues Ġof Ġinput Ġsequences . ĠThe Ġattention Ġis Ġgenerated Ġusing Ġqueries Ġand Ġkeys Ġlike ĠEqu ation Ġ( DIS PLAY _ FORM 11 ). ĠStructure Ġof Ġscaled Ġdot - product Ġattention Ġallows Ġthe Ġself - att ention Ġlayer Ġgenerate Ġthe Ġrepresentation Ġof Ġsentences Ġat Ġonce Ġand Ġcontain Ġthe Ġinformation Ġof Ġthe Ġsentence Ġwhich Ġis Ġdifferent Ġfrom ĠR NN Ġthat Ġprocess Ġcharacters Ġof Ġsentences Ġone Ġby Ġone . ĠStandard Ġself - att ention Ġis Ġsimilar Ġas ĠGa ussian - mask ed Ġdirection Ġattention Ġwhile Ġit Ġdoes Ġnot Ġhave Ġdirectional Ġmask Ġand Ġga ussian Ġmask . ĠB IB REF 24 Ġalso Ġpropose Ġmulti - head Ġattention Ġwhich Ġis Ġbetter Ġto Ġgenerate Ġrepresentation Ġof Ġsentence Ġby Ġdividing Ġqueries , Ġkeys Ġand Ġvalues Ġto Ġdifferent Ġheads Ġand Ġget Ġinformation Ġfrom Ġdifferent Ġsub sp aces . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġan Ġattention Ġmechanism Ġonly Ġbased ĠChinese Ġword Ġsegment ation Ġmodel . ĠOur Ġmodel Ġuses Ġself - att ention Ġfrom Ġthe ĠTrans former Ġenc oder Ġto Ġtake Ġsequence Ġinput Ġand Ġbi - aff ine Ġattention Ġscorer Ġto Ġpredict Ġthe Ġlabel Ġof Ġgaps . ĠTo Ġimprove Ġthe Ġability Ġof Ġcapturing Ġthe Ġlocal ness Ġand Ġdirectional Ġinformation Ġof Ġself - att ention Ġbased Ġenc oder , Ġwe Ġpropose Ġa Ġvariant Ġof Ġself - att ention Ġcalled ĠGa ussian - mask ed Ġdirectional Ġmulti - head Ġattention Ġto Ġreplace Ġthe Ġstandard Ġself - att ention . ĠWe Ġalso Ġextend Ġthe ĠTrans former Ġenc oder Ġto Ġcapture Ġdirectional Ġfeatures . ĠOur Ġmodel Ġuses Ġonly Ġun ig ram Ġfeatures Ġinstead Ġof Ġmultiple Ġ$ n $ - gram Ġfeatures Ġin Ġprevious Ġwork . ĠOur Ġmodel Ġis Ġevaluated Ġon Ġstandard Ġbenchmark Ġdataset , ĠS IGH AN ĠBake off Ġ2005 , Ġwhich Ġshows Ġnot Ġonly Ġour Ġmodel Ġperforms Ġsegment ation Ġfaster Ġthan Ġany Ġprevious Ġmodels Ġbut Ġalso Ġgives Ġnew Ġhigher Ġor Ġcomparable Ġsegment ation Ġperformance Ġagainst Ġprevious Ġstate - of - the - art Ġmodels .