 ambiguous words, unknown words
EXTRACTIVE
lacks of complete review approaches, datasets and toolkits 
EXTRACTIVE
Maximum matching (MM) is one of the most popular fundamental and structural segmentation algorithms for word segmentation BIBREF19 . This method is also considered as the Longest Matching (LM) in several research BIBREF9 , BIBREF3 . It is used for identifying word boundary in languages like Chinese, Vietnamese and Thai. This method is a greedy algorithm, which simply chooses longest words based on the dictionary. Segmentation may start from either end of the line without any difference in segmentation results. If the dictionary is sufficient BIBREF19 , the expected segmentation accuracy is over 90%, so it is a major advantage of maximum matching . However, it does not solve the problem of ambiguous words and unknown words that do not exist in the dictionary.
According to our observation, we found that is lacks of complete review approaches, datasets and toolkits which we recently used in Vietnamese word segmentation. A all sided review of word segmentation will help next studies on Vietnamese natural language processing tasks have an up-to-date guideline and choose the most suitable solution for the task. The remaining part of the paper is organized as follows. Section II discusses building corpus in Vietnamese, containing linguistic issues and the building progress. Section III briefly mentions methods to model sentences and text in machine learning systems. Next, learning models and approaches for labeling and segmenting sequence data will be presented in Section IV. Section V mainly addresses two existing toolkits, vnTokenizer and JVnSegmenter, for Vietnamese word segmentation. Several experiments based on mentioned approaches and toolkits are described in Section VI. Finally, conclusions and future works are given in Section VII.
1906.07662
Introduction </s> Lex ical Ġanalysis , Ġsynt actic Ġanalysis , Ġsemantic Ġanalysis , Ġdisclosure Ġanalysis Ġand Ġpragmatic Ġanalysis Ġare Ġfive Ġmain Ġsteps Ġin Ġnatural Ġlanguage Ġprocessing ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠWhile Ġmorphology Ġis Ġa Ġbasic Ġtask Ġin Ġlex ical Ġanalysis Ġof ĠEnglish , Ġword Ġsegment ation Ġis Ġconsidered Ġa Ġbasic Ġtask Ġin Ġlex ical Ġanalysis Ġof ĠVietnamese Ġand Ġother ĠEast ĠAsian Ġlanguages Ġprocessing . ĠThis Ġtask Ġis Ġto Ġdetermine Ġborders Ġbetween Ġwords Ġin Ġa Ġsentence . ĠIn Ġother Ġwords , Ġit Ġis Ġsegment ing Ġa Ġlist Ġof Ġtokens Ġinto Ġa Ġlist Ġof Ġwords Ġsuch Ġthat Ġwords Ġare Ġmeaningful . </s> Word Ġsegment ation Ġis Ġthe Ġprimary Ġstep Ġin Ġprior Ġto Ġother Ġnatural Ġlanguage Ġprocessing Ġtasks Ġi . Ġe ., Ġterm Ġextraction Ġand Ġlinguistic Ġanalysis Ġ( as Ġshown Ġin ĠFigure Ġ1 ). ĠIt Ġidentifies Ġthe Ġbasic Ġmeaningful Ġunits Ġin Ġinput Ġtexts Ġwhich Ġwill Ġbe Ġprocessed Ġin Ġthe Ġnext Ġsteps Ġof Ġseveral Ġapplications . ĠFor Ġnamed Ġentity Ġrecogn ization ĠB IB REF 2 Ġ, Ġword Ġsegment ation Ġchunks Ġsentences Ġin Ġinput Ġdocuments Ġinto Ġsequences Ġof Ġwords Ġbefore Ġthey Ġare Ġfurther Ġclassified Ġin Ġto Ġnamed Ġentity Ġclasses . ĠFor ĠVietnamese Ġlanguage , Ġwords Ġand Ġcandidate Ġterms Ġcan Ġbe Ġextracted Ġfrom ĠVietnamese Ġcop ora Ġ( such Ġas Ġbooks , Ġnovels , Ġnews , Ġand Ġso Ġon ) Ġby Ġusing Ġa Ġword Ġsegment ation Ġtool . ĠCon formed Ġfeatures Ġand Ġcontext Ġof Ġthese Ġwords Ġand Ġterms Ġare Ġused Ġto Ġidentify Ġnamed Ġentity Ġtags , Ġtopic Ġof Ġdocuments , Ġor Ġfunction Ġwords . ĠFor Ġlinguistic Ġanalysis , Ġseveral Ġlinguistic Ġfeatures Ġfrom Ġdiction aries Ġcan Ġbe Ġused Ġeither Ġto Ġannot ating ĠPOS Ġtags Ġor Ġto Ġidentifying Ġthe Ġanswer Ġsentences . ĠMoreover , Ġlanguage Ġmodels Ġcan Ġbe Ġtrained Ġby Ġusing Ġmachine Ġlearning Ġapproaches Ġand Ġbe Ġused Ġin Ġtagging Ġsystems , Ġlike Ġthe Ġnamed Ġentity Ġrecogn ization Ġsystem Ġof ĠTr an Ġet Ġal . ĠB IB REF 2 Ġ. </s> Many Ġstudies Ġfor cus Ġon Ġword Ġsegment ation Ġfor ĠAsian Ġlanguages , Ġsuch Ġas : ĠChinese , ĠJapanese , ĠBur m ese Ġ( My anmar ) Ġand ĠThai ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠAppro aches Ġfor Ġword Ġsegment ation Ġtask Ġare Ġvariety , Ġfrom Ġlex icon - based Ġto Ġmachine Ġlearning - based Ġmethods . ĠRecently , Ġmachine Ġlearning - based Ġmethods Ġare Ġused Ġwidely Ġto Ġsolve Ġthis Ġissue , Ġsuch Ġas : ĠSupport ĠVector ĠMachine Ġor ĠCond itional ĠRandom ĠFields ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠIn Ġgeneral , ĠChinese Ġis Ġa Ġlanguage Ġwhich Ġhas Ġthe Ġmost Ġstudies Ġon Ġthe Ġword Ġsegment ation Ġissue . ĠHowever , Ġthere Ġis Ġa Ġlack Ġof Ġsurvey Ġof Ġword Ġsegment ation Ġstudies Ġon ĠAsian Ġlanguages Ġand ĠVietnamese Ġas Ġwell . ĠThis Ġpaper Ġaims Ġreviewing Ġstate - of - the - art Ġword Ġsegment ation Ġapproaches Ġand Ġsystems Ġapplying Ġfor ĠVietnamese . ĠThis Ġstudy Ġwill Ġbe Ġa Ġfoundation Ġfor Ġstudies Ġon ĠVietnamese Ġword Ġsegment ation Ġand Ġother Ġfollowing ĠVietnamese Ġtasks Ġas Ġwell , Ġsuch Ġas Ġpart - of - speech Ġtag ger , Ġchunk er , Ġor Ġparser Ġsystems . </s> There Ġare Ġseveral Ġstudies Ġabout Ġthe ĠVietnamese Ġword Ġsegment ation Ġtask Ġover Ġthe Ġlast Ġdecade . ĠDin h Ġet Ġal . Ġstarted Ġthis Ġtask Ġwith ĠWeight ed ĠFin ite ĠState ĠTrans du cer Ġ( WF ST ) Ġapproach Ġand ĠNeural ĠNetwork Ġapproach ĠB IB REF 9 Ġ. ĠIn Ġaddition , Ġmachine Ġlearning Ġapproaches Ġare Ġstudied Ġand Ġwidely Ġapplied Ġto Ġnatural Ġlanguage Ġprocessing Ġand Ġword Ġsegment ation Ġas Ġwell . ĠIn Ġfact , Ġseveral Ġstudies Ġused Ġsupport Ġvector Ġmachines Ġ( S VM ) Ġand Ġconditional Ġrandom Ġfields Ġ( CR F ) Ġfor Ġthe Ġword Ġsegment ation Ġtask ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠBased Ġon Ġannot ated Ġcorpor a Ġand Ġtoken - based Ġfeatures , Ġstudies Ġused Ġmachine Ġlearning Ġapproaches Ġto Ġbuild Ġword Ġsegment ation Ġsystems Ġwith Ġaccuracy Ġabout Ġ94 %- 97 %. </s> According Ġto Ġour Ġobservation , Ġwe Ġfound Ġthat Ġis Ġlacks Ġof Ġcomplete Ġreview Ġapproaches , Ġdatasets Ġand Ġtool k its Ġwhich Ġwe Ġrecently Ġused Ġin ĠVietnamese Ġword Ġsegment ation . ĠA Ġall Ġsided Ġreview Ġof Ġword Ġsegment ation Ġwill Ġhelp Ġnext Ġstudies Ġon ĠVietnamese Ġnatural Ġlanguage Ġprocessing Ġtasks Ġhave Ġan Ġup - to - date Ġguideline Ġand Ġchoose Ġthe Ġmost Ġsuitable Ġsolution Ġfor Ġthe Ġtask . ĠThe Ġremaining Ġpart Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠSection ĠII Ġdiscusses Ġbuilding Ġcorpus Ġin ĠVietnamese , Ġcontaining Ġlinguistic Ġissues Ġand Ġthe Ġbuilding Ġprogress . ĠSection ĠIII Ġbriefly Ġmentions Ġmethods Ġto Ġmodel Ġsentences Ġand Ġtext Ġin Ġmachine Ġlearning Ġsystems . ĠNext , Ġlearning Ġmodels Ġand Ġapproaches Ġfor Ġlabeling Ġand Ġsegment ing Ġsequence Ġdata Ġwill Ġbe Ġpresented Ġin ĠSection ĠIV . ĠSection ĠV Ġmainly Ġaddresses Ġtwo Ġexisting Ġtool k its , Ġv n Token izer Ġand ĠJ V n Seg ment er , Ġfor ĠVietnamese Ġword Ġsegment ation . ĠSeveral Ġexperiments Ġbased Ġon Ġmentioned Ġapproaches Ġand Ġtool k its Ġare Ġdescribed Ġin ĠSection ĠVI . ĠFinally , Ġconclusions Ġand Ġfuture Ġworks Ġare Ġgiven Ġin ĠSection ĠVII . </s> Language ĠDefinition </s> V iet namese , Ġlike Ġmany Ġlanguages Ġin Ġcontinental ĠEast ĠAsia , Ġis Ġan Ġisol ating Ġlanguage Ġand Ġone Ġbranch Ġof ĠMon - Kh mer Ġlanguage Ġgroup . ĠThe Ġmost Ġbasic Ġlinguistic Ġunit Ġin ĠVietnamese Ġis Ġmor phe me , Ġsimilar Ġwith Ġsyll able Ġor Ġtoken Ġin ĠEnglish Ġand ĠâĢ ľ h Ã ¬ n h Ġv á » ĭ âĢ Ŀ Ġ( ph on eme ) Ġor ĠâĢ ľ ti á º ¿ ng âĢ Ŀ Ġ( sy ll able ) Ġin ĠVietnamese . ĠAccording Ġto Ġthe Ġstructured Ġrule Ġof Ġits , ĠVietnamese Ġcan Ġhave Ġabout Ġ20 , 000 Ġdifferent Ġsyll ables Ġ( t ok ens ). ĠHowever , Ġthere Ġare Ġabout Ġ8 , 000 Ġsyll ables Ġused Ġthe ĠVietnamese Ġdiction aries . ĠThere Ġare Ġthree Ġmethods Ġto Ġidentify Ġmorp hem es Ġin ĠVietnamese Ġtext ĠB IB REF 10 Ġ. </s> M orp he me Ġis Ġthe Ġsmallest Ġmeaningful Ġunit Ġof ĠVietnamese . </s> M orp he me Ġis Ġthe Ġbasic Ġunit Ġof ĠVietnamese . </s> M orp he me Ġis Ġthe Ġsmallest Ġmeaningful Ġunit Ġand Ġis Ġnot Ġused Ġindependently Ġin Ġthe Ġsyntax Ġfactor . </s> In Ġcomputational Ġlingu istics , Ġmor phe me Ġis Ġthe Ġbasic Ġunit Ġof Ġlanguages Ġas ĠLeonard ĠBloom field Ġmentioned Ġfor ĠEnglish ĠB IB REF 11 Ġ. ĠIn Ġour Ġresearch Ġfor ĠVietnamese , Ġwe Ġconsider Ġthe Ġmor phe me Ġas Ġsyll able , Ġcalled ĠâĢ ľ ti á º ¿ ng âĢ Ŀ Ġin ĠVietnamese Ġ( as ĠNguyen âĢ Ļ s Ġdefinition ĠB IB REF 12 Ġ). </s> The Ġnext Ġconcept Ġin Ġlingu istics Ġis Ġword Ġwhich Ġhas Ġfully Ġgrammar Ġand Ġmeaning Ġfunction Ġin Ġsentences . ĠFor ĠVietnamese , Ġword Ġis Ġa Ġsingle Ġmor phe me Ġor Ġa Ġgroup Ġof Ġmorp hem es , Ġwhich Ġare Ġfixed Ġand Ġhave Ġfull Ġmeaning ĠB IB REF 12 Ġ. ĠAccording Ġto ĠNguyen , ĠVietnamese Ġwords Ġare Ġable Ġclassified Ġinto Ġtwo Ġtypes , Ġ( 1 ) Ġ1 - Ġsyll able Ġwords Ġwith Ġfully Ġmeaning Ġand Ġ( 2 ) Ġn - sy ll ables Ġwords Ġwhereas Ġthese Ġgroup Ġof Ġtokens Ġare Ġfixed . ĠVietnamese Ġsyll able Ġis Ġnot Ġfully Ġmeaningful . ĠHowever , Ġit Ġis Ġalso Ġexplained Ġin Ġthe Ġmeaning Ġand Ġstructure Ġcharacteristics . ĠFor Ġexample , Ġthe Ġtoken ĠâĢ ľ k á » ³ âĢ Ŀ Ġin ĠâĢ ľ qu á » ĳ c Ġk á » ³ âĢ Ŀ Ġwhereas ĠâĢ ľ qu á » ĳ c âĢ Ŀ Ġmeans Ġnational , ĠâĢ ľ k á » ³ âĢ Ŀ Ġmeans Ġflag . ĠTherefore , ĠâĢ ľ qu á » ĳ c Ġk á » ³ âĢ Ŀ Ġmeans Ġnational Ġflag . </s> Consider Ġdictionary Ġused Ġfor Ġevaluating Ġthe Ġcorpus , Ġextracting Ġfeatures Ġfor Ġmodels , Ġand Ġevaluating Ġthe Ġsystems , Ġthere Ġare Ġmany ĠVietnamese Ġdiction aries , Ġhowever Ġwe Ġrecommend Ġthe ĠVietnamese Ġdictionary Ġof ĠHo ang ĠP he , Ġso Ġcalled ĠHo ang ĠP he ĠDictionary . ĠThis Ġdictionary Ġhas Ġbeen Ġbuilt Ġby Ġa Ġgroup Ġof Ġlingu istical Ġscientists Ġat Ġthe ĠL ingu istic ĠInstitute , ĠVietnam . ĠIt Ġwas Ġfirst ly Ġpublished Ġin Ġ1988 , Ġreprinted Ġand Ġextended Ġin Ġ2000 , Ġ2005 Ġand Ġ2010 . ĠThe Ġdictionary Ġcurrently Ġhas Ġ45 , 757 Ġword Ġitems Ġwith Ġ15 , 901 ĠS ino - V iet namese Ġword Ġitems Ġ( account ing Ġfor Ġ34 . 75 %) ĠB IB REF 13 Ġ. </s> Name ĠEntity ĠIssue </s> In ĠVietnamese , Ġnot Ġall Ġof Ġmeaningful Ġproper Ġnames Ġare Ġin Ġthe Ġdictionary . ĠIdent ifying Ġproper Ġnames Ġin Ġinput Ġtext Ġare Ġalso Ġimportant Ġissue Ġin Ġword Ġsegment ation . ĠThis Ġissue Ġis Ġsometimes Ġincluded Ġinto Ġunknown Ġword Ġissue Ġto Ġbe Ġsolved . ĠIn Ġaddition , Ġnamed Ġentity Ġrecognition Ġhas Ġto Ġclassify Ġit Ġinto Ġseveral Ġtypes Ġsuch Ġas Ġperson , Ġlocation , Ġorganization , Ġtime , Ġmoney , Ġnumber , Ġand Ġso Ġon . </s> Pro per Ġname Ġidentification Ġcan Ġbe Ġsolved Ġby Ġcharacteristics . ĠFor Ġexample , Ġsystems Ġuse Ġbeginning Ġcharacters Ġof Ġproper Ġnames Ġwhich Ġare Ġu pperc ase Ġcharacters . ĠMoreover , Ġa Ġlist Ġof Ġproper Ġnames Ġis Ġalso Ġused Ġto Ġidentify Ġnames Ġin Ġthe Ġtext . ĠIn Ġparticular , Ġa Ġlist Ġof Ġ2000 Ġpersonal Ġnames Ġextracted Ġfrom ĠVietnam G ia Ph a , Ġand Ġa Ġlist Ġof Ġ7 07 Ġnames Ġof Ġlocations Ġin ĠVietnam Ġextracted Ġfrom Ġvi . wikipedia . org Ġare Ġused Ġin Ġthe Ġstudy Ġof ĠNguyen Ġet Ġal . Ġfor ĠVietnamese Ġword Ġsegment ation ĠB IB REF 7 Ġ. </s> Building ĠCorpus </s> In Ġgeneral , Ġbuilding Ġcorpus Ġis Ġcarried Ġout Ġthrough Ġfour Ġstages : Ġ( 1 ) Ġchoose Ġtarget Ġof Ġcorpus Ġand Ġsource Ġof Ġraw Ġdata ; Ġ( 2 ) Ġbuilding Ġa Ġguideline Ġbased Ġon Ġlingu istics Ġknowledge Ġfor Ġannotation ; Ġ( 3 ) Ġannot ating Ġor Ġtagging Ġcorpus Ġbased Ġon Ġrule Ġset Ġin Ġthe Ġguideline ; Ġand Ġ( 4 ) Ġreviewing Ġcorpus Ġto Ġcheck Ġthe Ġconsistency Ġissue . </s> Enc oding Ġword Ġsegment ation Ġcorpus Ġusing ĠB - I - O Ġtag set Ġcan Ġbe Ġapplied , Ġwhere ĠB , ĠI , Ġand ĠO Ġden oted Ġbegin Ġof Ġword , Ġinside Ġof Ġword , Ġand Ġothers , Ġrespectively . ĠFor Ġexample , Ġthe Ġsentence ĠâĢ ľ Meg abit Ġtr Ãª n Ġg i Ã¢ y Ġl Ãł ĠÄ ĳ Æ ¡ n Ġv á » ĭ ĠÄ ĳ o Ġt á » ĳ c ĠÄ ĳ á » į Ġtr uy á » ģ n Ġd á º « n Ġd á » ¯ Ġli á » ĩ u Ġ." Ġ( âĢ Ŀ Meg abit Ġper Ġsecond Ġis Ġa Ġunit Ġto Ġmeasure Ġthe Ġnetwork Ġtraffic . âĢ Ŀ Ġin ĠEnglish ) Ġwith Ġthe Ġword Ġboundary Ġresult ĠâĢ ľ Meg abit Ġtr Ãª n Ġg i Ã¢ y Ġl Ãł ĠÄ ĳ Æ ¡ n _ v á » ĭ ĠÄ ĳ o Ġt á » ĳ c _ Ä ĳ á » Ļ Ġtr uy á » ģ n _ d á º « n Ġd á » ¯ _ li á » ĩ u Ġ." Ġis Ġencoded Ġas ĠâĢ ľ Meg abit / B Ġtr Ãª n / B Ġg i Ã¢ y / B Ġl Ãł / B ĠÄ ĳ Æ ¡ n / B Ġv á » ĭ / I ĠÄ ĳ o / B Ġt á » ĳ c / B ĠÄ ĳ á » Ļ / I Ġtr uy á » ģ n / B Ġd á º « n / I Ġd á » ¯ / B Ġli á » ĩ u / I Ġ./ O " Ġ. </s> An notation Ġguidelines Ġcan Ġbe Ġapplied Ġto Ġensure Ġthat Ġannot ated Ġcorpus Ġhas Ġless Ġerrors Ġbecause Ġthe Ġmanual Ġannotation Ġis Ġapplied . ĠEven Ġthough Ġthere Ġare Ġguidelines Ġfor Ġannot ating , Ġthe Ġavailable Ġoutput Ġcorpor a Ġare Ġstill Ġinconsistent . ĠFor Ġexample , Ġfor Ġthe ĠVietnamese ĠTree bank Ġcorpus Ġof Ġthe ĠV L SP Ġproject , ĠNguyen Ġet Ġal . Ġlisted Ġout Ġseveral ĠVietnamese Ġword Ġsegment ation Ġinconsistencies Ġin Ġthe Ġcorpus Ġbased Ġon ĠPOS Ġinformation Ġand Ġn - gram Ġsequences ĠB IB REF 14 Ġ. </s> Currently , Ġthere Ġare Ġat Ġleast Ġthree Ġavailable Ġword Ġsegment ation Ġcorpus Ġused Ġin ĠVietnamese Ġword Ġsegment ation Ġstudies Ġand Ġsystems . ĠFirstly , ĠDin h Ġet Ġal . Ġbuilt Ġthe ĠCAD AS A Ġcorpus Ġfrom ĠCAD AS A âĢ Ļ s Ġbooks ĠB IB REF 15 Ġ. ĠSecondly , ĠNguyen Ġet Ġal . Ġbuilt Ġv n Q TAG Ġcorpus Ġfrom Ġgeneral Ġnews Ġarticles ĠB IB REF 7 Ġ. ĠMore Ġrecently , ĠN go Ġet Ġal . Ġintroduced Ġthe ĠEV BC orp us Ġcorpus , Ġwhich Ġis Ġcollected Ġfrom Ġfour Ġsources , Ġnews Ġarticles , Ġbooks , Ġlaw Ġdocuments , Ġand Ġnovels . ĠAs Ġa Ġpart Ġof ĠEV BC orp us , ĠEV B News , Ġwas Ġannot ated Ġcommon Ġtags Ġin ĠN LP , Ġsuch Ġas Ġword Ġsegment ation , Ġchunk er , Ġand Ġnamed Ġentity ĠB IB REF 16 Ġ. ĠAll Ġof Ġthese Ġcorpor a Ġare Ġcollected Ġfrom Ġnews Ġarticles Ġor Ġbook Ġstories , Ġand Ġthey Ġare Ġmanually Ġannot ated Ġthe Ġword Ġboundary Ġtags Ġ( as Ġshown Ġin ĠTable ĠI ). </s> TEXT ĠM ODE LL ING ĠAND ĠFE ATURES </s> To Ġunderstand Ġnatural Ġlanguage Ġand Ġanalyze Ġdocuments Ġand Ġtext , Ġcomputers Ġneed Ġto Ġrepresent Ġnatural Ġlanguages Ġas Ġlingu istics Ġmodels . ĠThese Ġmodels Ġcan Ġbe Ġgenerated Ġby Ġusing Ġmachine Ġlearning Ġmethods Ġ( as Ġshow Ġin ĠFigure Ġ2 ). </s> There Ġare Ġtwo Ġcommon Ġmodeling Ġmethods Ġfor Ġbasic ĠN LP Ġtasks , Ġincluding Ġn - gram Ġmodel Ġand Ġbag - of - words Ġmodel . ĠThe Ġn - gram Ġmodel Ġis Ġwidely Ġused Ġin Ġnatural Ġlanguage Ġprocessing Ġwhile Ġthe Ġbag - of - words Ġmodel Ġis Ġa Ġsimplified Ġrepresentation Ġused Ġin Ġnatural Ġlanguage Ġprocessing Ġand Ġinformation Ġretrieval ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠAccording Ġto Ġthe Ġbag - of - words Ġmodel , Ġthe Ġrepresentative Ġvector Ġof Ġsentences Ġin Ġthe Ġdocument Ġdoes Ġnot Ġpreserve Ġthe Ġorder Ġof Ġthe Ġwords Ġin Ġthe Ġoriginal Ġsentences . ĠIt Ġrepresents Ġthe Ġword Ġusing Ġterm Ġfrequency Ġcollected Ġfrom Ġthe Ġdocument Ġrather Ġthan Ġthe Ġorder Ġof Ġwords Ġor Ġthe Ġstructure Ġof Ġsentences Ġin Ġthe Ġdocument . ĠThe Ġbag - of - words Ġmodel Ġis Ġcommonly Ġused Ġin Ġmethods Ġof Ġdocument Ġclassification , Ġwhere Ġthe Ġfrequency Ġof Ġoccurrence Ġof Ġeach Ġword Ġis Ġused Ġas Ġan Ġattribute Ġfeature Ġfor Ġtraining Ġa Ġclass ifier . ĠIn Ġcontrast , Ġan Ġn - gram Ġis Ġa Ġcontiguous Ġsequence Ġof Ġn Ġitems Ġfrom Ġa Ġgiven Ġsequence Ġof Ġtext . ĠAn Ġn - gram Ġmodel Ġis Ġa Ġtype Ġof Ġprob abil istic Ġlanguage Ġmodel Ġfor Ġpredicting Ġthe Ġnext Ġitem Ġin Ġa Ġgiven Ġsequence Ġin Ġform Ġof Ġa ĠMark ov Ġmodel . ĠTo Ġaddress Ġword Ġsegment ation Ġissue , Ġthe Ġn - gram Ġmodel Ġis Ġusually Ġused Ġfor Ġapproaches Ġbecause Ġit Ġconsiders Ġthe Ġorder Ġof Ġtokens Ġin Ġthe Ġoriginal Ġsentences . ĠThe Ġsequence Ġis Ġalso Ġkept Ġthe Ġoriginal Ġorder Ġas Ġinput Ġand Ġoutput Ġsentences . </s> BU ILD ING ĠMOD EL ĠM ETHOD S </s> There Ġare Ġseveral Ġstudies Ġfor ĠVietnamese ĠWord ĠSe gment ation Ġduring Ġlast Ġdecade . ĠFor Ġinstance , ĠDin h Ġet Ġal . Ġstarted Ġthe Ġword Ġsegment ation Ġtask Ġfor ĠVietnamese Ġwith ĠNeural ĠNetwork Ġand ĠWeight ed ĠFin ite ĠState ĠTrans du cer Ġ( WF ST ) ĠB IB REF 9 Ġ. ĠNguyen Ġet Ġal . Ġcontinued Ġwith Ġmachine Ġlearning Ġapproaches , ĠCond itional ĠRandom ĠFields Ġand ĠSupport ĠVector ĠMachine ĠB IB REF 7 Ġ. ĠMost Ġof Ġstatistical Ġapproaches Ġare Ġbased Ġon Ġthe Ġarchitecture Ġas Ġshown Ġin ĠFigure Ġ2 . ĠAccording Ġto Ġthe Ġarchitecture , Ġrecent Ġstudies Ġand Ġsystems Ġfocus Ġon Ġeither Ġimproving Ġor Ġmodifying Ġdifference Ġlearning Ġmodels Ġto Ġget Ġthe Ġhighest Ġaccuracy . ĠFeatures Ġused Ġin Ġword Ġsegment ation Ġsystems Ġare Ġsyll able , Ġdictionary , Ġand Ġentity Ġname . ĠThe Ġdetail Ġof Ġall Ġwidely Ġused Ġtechniques Ġapplied Ġare Ġcollected Ġand Ġdescribed Ġin Ġfollowing Ġsubsections . </s> Maximum ĠMatch ing </s> Maximum Ġmatching Ġ( MM ) Ġis Ġone Ġof Ġthe Ġmost Ġpopular Ġfundamental Ġand Ġstructural Ġsegment ation Ġalgorithms Ġfor Ġword Ġsegment ation ĠB IB REF 19 Ġ. ĠThis Ġmethod Ġis Ġalso Ġconsidered Ġas Ġthe ĠLong est ĠMatch ing Ġ( LM ) Ġin Ġseveral Ġresearch ĠB IB REF 9 Ġ, ĠB IB REF 3 Ġ. ĠIt Ġis Ġused Ġfor Ġidentifying Ġword Ġboundary Ġin Ġlanguages Ġlike ĠChinese , ĠVietnamese Ġand ĠThai . ĠThis Ġmethod Ġis Ġa Ġgreedy Ġalgorithm , Ġwhich Ġsimply Ġchooses Ġlongest Ġwords Ġbased Ġon Ġthe Ġdictionary . ĠSe gment ation Ġmay Ġstart Ġfrom Ġeither Ġend Ġof Ġthe Ġline Ġwithout Ġany Ġdifference Ġin Ġsegment ation Ġresults . ĠIf Ġthe Ġdictionary Ġis Ġsufficient ĠB IB REF 19 Ġ, Ġthe Ġexpected Ġsegment ation Ġaccuracy Ġis Ġover Ġ90 %, Ġso Ġit Ġis Ġa Ġmajor Ġadvantage Ġof Ġmaximum Ġmatching Ġ. ĠHowever , Ġit Ġdoes Ġnot Ġsolve Ġthe Ġproblem Ġof Ġambiguous Ġwords Ġand Ġunknown Ġwords Ġthat Ġdo Ġnot Ġexist Ġin Ġthe Ġdictionary . </s> There Ġare Ġtwo Ġtypes Ġof Ġthe Ġmaximum Ġmatching Ġapproach : Ġforward ĠMM Ġ( F MM ) Ġand Ġbackward ĠMM Ġ( B MM ). ĠF MM Ġstarts Ġfrom Ġthe Ġbeginning Ġtoken Ġof Ġthe Ġsentence Ġwhile ĠB MM Ġstarts Ġfrom Ġthe Ġend . ĠIf Ġthe Ġsentence Ġhas Ġword Ġboundary Ġambig u ities , Ġthe Ġoutput Ġof ĠF MM Ġand ĠB MM Ġwill Ġbe Ġdifferent . ĠWhen Ġapplying ĠF MM Ġand ĠB MM , Ġthere Ġare Ġtwo Ġtypes Ġof Ġcommon Ġerrors Ġdue Ġto Ġtwo Ġambig u ities : Ġoverlapping Ġambig u ities Ġand Ġcombination Ġambiguity . ĠOver l apping Ġambig u ities Ġoccur Ġwhen Ġthe Ġtext ĠAB Ġhas Ġboth Ġword ĠA , ĠB Ġand ĠAB , Ġwhich Ġare Ġin Ġthe Ġdictionary Ġwhile Ġthe Ġtext ĠABC Ġhas Ġword ĠAB Ġand ĠBC , Ġwhich Ġare Ġin Ġthe Ġdictionary . ĠFor Ġexample , Ġ" c á » ¥ Ġg i Ãł ĠÄ ĳ i Ġn han h Ġqu Ã¡ " Ġ( there Ġtwo Ġmeanings : ĠâĢ Ŀ the Ġold Ġman Ġgoes Ġvery Ġfast âĢ Ŀ Ġor ĠâĢ Ŀ the Ġold Ġman Ġdied Ġsuddenly âĢ Ŀ ) Ġis Ġa Ġcase Ġof Ġthe Ġoverlapping Ġambiguity Ġwhile Ġ" t á » ĳ c ĠÄ ĳ á » Ļ Ġtr uy á » ģ n Ġth Ã´ ng Ġtin " Ġis Ġa Ġcase Ġof Ġthe Ġcombination Ġambiguity . </s> As Ġshown Ġin ĠFigure Ġ2 , Ġthe Ġmethod Ġsimpl ification Ġambig u ities , Ġmaximum Ġmatching Ġis Ġthe Ġfirst Ġstep Ġto Ġget Ġfeatures Ġfor Ġthe Ġmodelling Ġstage Ġin Ġmachine Ġlearning Ġsystems , Ġlike ĠCond itional ĠRandom ĠFields Ġor ĠSupport ĠVector ĠMachines . </s> Hidden ĠMark ov ĠModel Ġ( H MM ) </s> In ĠMark ov Ġchain Ġmodel Ġis Ġrepresented Ġas Ġa Ġchain Ġof Ġtokens Ġwhich Ġare Ġobservations , Ġand Ġword Ġtag gers Ġare Ġrepresented Ġas Ġpredicted Ġlabels . ĠMany Ġresearchers Ġapplied ĠHidden ĠMark ov Ġmodel Ġto Ġsolve ĠVietnamese Ġword Ġsegment ation Ġsuch Ġas Ġin ĠB IB REF 8 Ġ, ĠB IB REF 20 Ġand Ġso Ġon . </s> N - gram Ġlanguage Ġmodeling Ġapplied Ġto Ġestimate Ġprobabilities Ġfor Ġeach Ġword Ġsegment ation Ġsolution ĠB IB REF 21 Ġ. ĠThe Ġresult Ġof Ġthis Ġmethod Ġdepends Ġon Ġcop ora Ġand Ġis Ġbased Ġmaximal Ġmatching Ġstrategy . ĠSo , Ġthey Ġdo Ġnot Ġsolve Ġmissing Ġword Ġissue . ĠLet ĠIN LINE FORM 0 Ġis Ġa Ġproduct Ġof Ġprobabilities Ġof Ġwords Ġcreated Ġfrom Ġsentence Ġs Ġ( 1 ) Ġwith Ġlength ĠIN LINE FORM 1 Ġ: ĠDIS PLAY FORM 0 </s> Each Ġconditional Ġprobability Ġof Ġword Ġis Ġbased Ġon Ġthe Ġlast Ġn - 1 Ġwords Ġ( n - gram ) Ġin Ġthe Ġsentence Ġs . ĠIt Ġis Ġestimated Ġby ĠMark ov Ġchain Ġmodel Ġfor Ġword Ġw Ġfrom Ġposition Ġi - n + 1 Ġto Ġi - 1 Ġwith Ġprobability Ġ( 2 ) ĠDIS PLAY FORM 0 </s> We Ġhave Ġequation Ġ( 3 ) ĠDIS PLAY FORM 0 </s> Maximum ĠEnt ropy Ġ( ME ) </s> Maximum ĠEnt ropy Ġtheory Ġis Ġapplied Ġto Ġsolve ĠVietnamese Ġword Ġsegment ation ĠB IB REF 15 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ. ĠSome Ġresearchers Ġdo Ġnot Ġwant Ġthe Ġlimit Ġin ĠMark ov Ġchain Ġmodel . ĠSo , Ġthey Ġuse Ġthe Ġcontext Ġaround Ġof Ġthe Ġword Ġneeded Ġto Ġbe Ġsegment ed . ĠLet Ġh Ġis Ġa Ġcontext , Ġw Ġis Ġa Ġlist Ġof Ġwords Ġand Ġt Ġis Ġa Ġlist Ġof Ġtag gers , ĠLe ĠB IB REF 15 Ġ, ĠB IB REF 22 Ġused ĠDIS PLAY FORM 0 </s> P ( s ) Ġis Ġalso Ġa Ġproduct Ġof Ġprobabilities Ġof Ġwords Ġcreated Ġfrom Ġsentence ĠIN LINE FORM 0 Ġ( 1 ). ĠEach Ġconditional Ġprobability Ġof Ġword Ġis Ġbased Ġon Ġcontext Ġh Ġof Ġthe Ġlast Ġn Ġword Ġin Ġthe Ġsentence Ġs . </s> Cond itional ĠRandom ĠFields </s> To Ġtoken ize Ġa ĠVietnamese Ġword , Ġin ĠH MM Ġor ĠME , Ġauthors Ġonly Ġrely Ġon Ġfeatures Ġaround Ġa Ġword Ġsegment Ġposition . ĠSome Ġother Ġfeatures Ġare Ġalso Ġaffected Ġby Ġadding Ġmore Ġspecial Ġattributes , Ġsuch Ġas , Ġin Ġcase ĠâĢ Ļ ? âĢ Ļ Ġquestion Ġmark Ġat Ġend Ġof Ġsentence , ĠPart Ġof ĠSpeech Ġ( POS ), Ġand Ġso Ġon . ĠCond itional ĠRandom ĠFields Ġis Ġone Ġof Ġmethods Ġthat Ġuses Ġadditional Ġfeatures Ġto Ġimprove Ġthe Ġselection Ġstrategy ĠB IB REF 7 Ġ. </s> There Ġare Ġseveral ĠCR F Ġlibraries , Ġsuch Ġas ĠCR F ++ , ĠCR F su ite . ĠThese Ġmachine Ġlearning Ġtool k its Ġcan Ġbe Ġused Ġto Ġsolve Ġthe Ġtask Ġby Ġproviding Ġan Ġannot ated Ġcorpus Ġwith Ġextracted Ġfeatures . ĠThe Ġtool kit Ġwill Ġbe Ġused Ġto Ġtrain Ġa Ġmodel Ġbased Ġon Ġthe Ġcorpus Ġand Ġextract Ġa Ġtagging Ġmodel . ĠThe Ġtagging Ġmodel Ġwill Ġthen Ġbe Ġused Ġto Ġtag Ġon Ġinput Ġtext Ġwithout Ġannot ated Ġcorpus . ĠIn Ġthe Ġtraining Ġand Ġtagging Ġstages , Ġextracting Ġfeatures Ġfrom Ġthe Ġcorpus Ġand Ġthe Ġinput Ġtext Ġis Ġnecessary Ġfor Ġboth Ġstages . </s> Support ĠVector ĠMachines </s> Support ĠVector ĠMachines Ġ( S VM ) Ġis Ġa Ġsupervised Ġmachine Ġlearning Ġmethod Ġwhich Ġconsiders Ġdataset Ġas Ġa Ġset Ġof Ġvectors Ġand Ġtries Ġto Ġclassify Ġthem Ġinto Ġspecific Ġclasses . ĠBasically , ĠS VM Ġis Ġa Ġbinary Ġclass ifier . Ġhowever , Ġmost Ġclassification Ġtasks Ġare Ġmulti - class Ġclass ifiers . ĠWhen Ġapplying ĠSV Ms , Ġthe Ġmethod Ġhas Ġbeen Ġextended Ġto Ġclassify Ġthree Ġor Ġmore Ġclasses . ĠPart icular ĠN LP Ġtasks , Ġlike Ġword Ġsegment ation Ġand ĠPart - of - speech Ġtask , Ġeach Ġtoken / word Ġin Ġdocuments Ġwill Ġbe Ġused Ġas Ġa Ġfeature Ġvector . ĠFor Ġthe Ġword Ġsegment ation Ġtask , Ġeach Ġtoken Ġand Ġits Ġfeatures Ġare Ġconsidered Ġas Ġa Ġvector Ġfor Ġthe Ġwhole Ġdocument , Ġand Ġthe ĠS VM Ġmodel Ġwill Ġclassify Ġthis Ġvector Ġinto Ġone Ġof Ġthe Ġthree Ġtags Ġ( B - IO ). </s> This Ġtechnique Ġis Ġapplied Ġfor ĠVietnamese Ġword Ġsegment ation Ġin Ġseveral Ġstudies ĠB IB REF 7 Ġ, ĠB IB REF 24 Ġ. ĠNguyen Ġet Ġal . Ġapplied Ġon Ġa Ġsegment ed Ġcorpus Ġof Ġ8 , 000 Ġsentences Ġand Ġgot Ġthe Ġresult Ġat Ġ94 . 05 % Ġwhile ĠN go Ġet Ġal . Ġused Ġit Ġwith Ġ45 , 5 31 Ġsegment ed Ġsentences Ġand Ġget Ġthe Ġresult Ġat Ġ97 . 2 %. ĠIt Ġis Ġworth Ġto Ġmention Ġthat Ġgeneral ĠS VM Ġlibraries Ġ( such Ġas ĠLI BS VM , ĠLI BL INE AR , ĠS VM light , ĠNode - S VM , Ġand ĠTree S VM Ġ), ĠYam Ch a Ġis Ġan Ġopened Ġsource ĠS VM Ġlibrary Ġthat Ġserves Ġseveral ĠN LP Ġtasks : ĠPOS Ġtagging , ĠNamed ĠEntity ĠRecogn ition , Ġbase ĠNP Ġchunk ing , ĠText ĠCh unk ing , ĠText ĠClassification Ġand Ġevent ĠWord ĠSe gment ation . </s> TO OL K ITS </s> v n Token izer Ġand ĠJ V n Seg ment er Ġare Ġtwo Ġfamous Ġsegment ation Ġtool k its Ġfor ĠVietnamese Ġword Ġsegment ation . ĠBoth Ġtwo Ġword Ġsegment ation Ġtool k its Ġare Ġimplemented Ġthe Ġword Ġsegment ation Ġdata Ġprocess Ġin ĠFigure Ġ2 . ĠThis Ġsection Ġgives Ġmore Ġdetails Ġof Ġthese ĠVietnamese Ġword Ġtool k its . </s> Program ming ĠLanguages </s> In Ġgeneral , ĠJava Ġand ĠC ++ Ġare Ġthe Ġmost Ġcommon Ġlanguage Ġin Ġdeveloping Ġtool k its Ġand Ġsystems Ġfor Ġnatural Ġlanguage Ġprocessing Ġtasks . ĠFor Ġexample , ĠG ATE , ĠOpen N LP , ĠStanford ĠCore N LP Ġand ĠLing P ipe Ġplatforms Ġare Ġdeveloped Ġby ĠJ AV A Ġwhile Ġfoundation Ġtasks Ġand Ġmachine Ġlearning Ġtool k its Ġare Ġdeveloped Ġby ĠC ++ . ĠCR F ++ , ĠSV ML ight Ġand ĠY AM CHA Ġ. ĠRecently , ĠPython Ġbecomes Ġpopular Ġamong Ġthe ĠN LP Ġcommunity . ĠIn Ġfact , Ġmany Ġtool k its Ġand Ġplatforms Ġhave Ġbeen Ġdeveloped Ġby Ġthis Ġlanguage , Ġsuch Ġas ĠNL T K , ĠPy NL Pl Ġlibrary Ġfor ĠNatural ĠLanguage ĠProcessing . </s> JV n Seg ment er </s> JV n Seg ment er Ġis Ġa ĠJava - based ĠVietnamese ĠWord ĠSe gment ation ĠTool Ġdeveloped Ġby ĠNguyen Ġand ĠPh an . ĠThe Ġsegment ation Ġmodel Ġin Ġthis Ġtool Ġwas Ġtrained Ġon Ġabout Ġ8 , 000 Ġtagged ĠVietnamese Ġtext Ġsentences Ġbased Ġon ĠCR F Ġmodel Ġand Ġthe Ġmodel Ġextracted Ġover Ġ151 , 000 Ġwords Ġfrom Ġthe Ġtraining Ġcorpus . ĠIn Ġaddition , Ġthis Ġis Ġused Ġin Ġbuilding Ġthe ĠEnglish V iet namese ĠTranslation ĠSystem ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠVietnamese Ġtext Ġclassification ĠB IB REF 28 Ġand Ġbuilding ĠVietnamese Ġcorpus ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ. </s> v n Token izer </s> v n Token izer Ġis Ġimplemented Ġin ĠJava Ġand Ġbundled Ġas ĠEclipse Ġplug - in , Ġand Ġit Ġhas Ġalready Ġbeen Ġintegrated Ġinto Ġv n Tool kit , Ġan ĠEclipse ĠRich ĠClient Ġapplication , Ġwhich Ġis Ġintended Ġto Ġbe Ġa Ġgeneral Ġframework Ġintegrating Ġtools Ġfor Ġprocessing Ġof ĠVietnamese Ġtext . Ġv n Token izer Ġplug - in , Ġv n Tool kit Ġand Ġrelated Ġresources , Ġincluding Ġthe Ġlex icon Ġand Ġtest Ġcorpus Ġare Ġfreely Ġavailable Ġfor Ġdownload . ĠAccording Ġto Ġour Ġobservation , Ġmany Ġresearch Ġcited Ġv n Token izer Ġto Ġuse Ġword Ġsegment ation Ġresults Ġfor Ġapplications Ġas Ġbuilding Ġa Ġlarge ĠVietnamese Ġcorpus ĠB IB REF 31 Ġ, Ġbuilding Ġan ĠEnglish - V iet namese ĠB ilingual ĠCorpus Ġfor ĠMachine ĠTranslation ĠB IB REF 32 Ġ, ĠVietnamese Ġtext Ġclassification ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, Ġetc . </s> EV AL U ATION ĠAND ĠRES ULTS </s> This Ġresearch Ġgathers Ġthe Ġresults Ġof ĠVietnamese Ġword Ġsegment ation Ġof Ġseveral Ġmethods Ġinto Ġone Ġtable Ġas Ġshow Ġin ĠTable ĠII . ĠIt Ġis Ġnoted Ġthat Ġthey Ġare Ġnot Ġevaluated Ġon Ġa Ġsame Ġcorpus . ĠThe Ġpurpose Ġof Ġthe Ġresult Ġillustration Ġis Ġto Ġprovide Ġan Ġoverview Ġof Ġthe Ġresults Ġof Ġcurrent ĠVietnamese Ġword Ġsegment ation Ġsystems Ġbased Ġon Ġtheir Ġindividual Ġfeatures . ĠAll Ġstudies Ġmentioned Ġin Ġthe Ġtable Ġhave Ġaccuracy Ġaround Ġ94 - 97 % Ġbased Ġon Ġtheir Ġprovided Ġcorpus . </s> This Ġstudy Ġalso Ġevaluates Ġthe ĠVietnamese Ġword Ġsegment ation Ġbased Ġon Ġexisting Ġtool k its Ġusing Ġthe Ġsame Ġannot ated ĠVietnamese Ġword Ġsegment ation Ġcorpus . ĠThere Ġare Ġtwo Ġavailable Ġtool k its Ġto Ġevaluate Ġand Ġto Ġsegment . ĠTo Ġbe Ġneutral Ġto Ġboth Ġtool k its , Ġwe Ġuse Ġthe ĠEV B News ĠVietnamese Ġcorpus , Ġa Ġpart Ġof ĠEV BC orp us , Ġto Ġevaluate ĠVietnamese Ġword Ġsegment ation . ĠThe ĠEV B News Ġcorpus Ġcontains Ġover Ġ45 , 000 Ġsegment ed ĠVietnamese Ġsentences Ġextracted Ġfrom Ġ1 , 000 Ġgeneral Ġnews Ġarticles Ġ( as Ġshown Ġin ĠTable ĠIII ) ĠB IB REF 16 Ġ. ĠWe Ġused Ġthe Ġsame Ġtraining Ġset Ġwhich Ġhas Ġ1000 Ġfiles Ġand Ġ45 , 5 31 Ġsentences . Ġv n Token izer Ġoutputs Ġ8 31 , 455 ĠVietnamese Ġwords Ġand Ġ1 , 206 , 475 Ġtokens . ĠJ V n Seg ment er Ġoutputs Ġ840 , 387 Ġwords Ġand Ġ1 , 201 , 683 . ĠWe Ġcorrect Ġtags Ġ( BI O ), Ġand Ġcompare Ġto Ġprevious Ġoutputs , Ġwe Ġhave Ġrate Ġfrom Ġv n Token izer Ġis Ġ95 . 6 % Ġand Ġfrom ĠJ V n se gment er Ġis Ġ93 . 4 %. ĠThe Ġresult Ġof Ġboth Ġv n Token izer Ġand ĠJ V n Seg ment er Ġtesting Ġon Ġthe ĠEV B News ĠVietnamese ĠCorpus Ġare Ġprovided Ġin ĠTable ĠIV . </s> CONCLUS IONS ĠAND ĠF UT URE ĠWORK S </s> This Ġstudy Ġreviewed Ġstate - of - the - art Ġapproaches Ġand Ġsystems Ġof ĠVietnamese Ġword Ġsegment ation . ĠThe Ġreview Ġpointed Ġout Ġcommon Ġfeatures Ġand Ġmethods Ġused Ġin ĠVietnamese Ġword Ġsegment ation Ġstudies . ĠThis Ġstudy Ġalso Ġhad Ġan Ġevaluation Ġof Ġthe Ġexisting ĠVietnamese Ġword Ġsegment ation Ġtool k its Ġbased Ġon Ġa Ġsame Ġcorpus Ġto Ġshow Ġadvantages Ġand Ġdisadvantages Ġas Ġto Ġshed Ġsome Ġlights Ġon Ġsystem Ġenhancement . </s> There Ġare Ġseveral Ġchallenges Ġon Ġsupervised Ġlearning Ġapproaches Ġin Ġfuture Ġwork . ĠThe Ġfirst Ġchallenge Ġis Ġto Ġacquire Ġvery Ġlarge ĠVietnamese Ġcorpus Ġand Ġto Ġuse Ġthem Ġin Ġbuilding Ġa Ġclass ifier , Ġwhich Ġcould Ġfurther Ġimprove Ġaccuracy . ĠIn Ġaddition , Ġapplying Ġlingu istics Ġknowledge Ġon Ġword Ġcontext Ġto Ġextract Ġuseful Ġfeatures Ġalso Ġenhances Ġprediction Ġperformance . ĠThe Ġsecond Ġchallenge Ġis Ġdesign Ġand Ġdevelopment Ġof Ġbig Ġdata Ġwarehouse Ġand Ġanalytic Ġframework Ġfor ĠVietnamese Ġdocuments , Ġwhich Ġcorresponds Ġto Ġthe Ġrapid Ġand Ġcontinuous Ġgrowth Ġof Ġgigantic Ġvolume Ġof Ġarticles Ġand / or Ġdocuments Ġfrom ĠWeb Ġ2 . 0 Ġapplications , Ġsuch Ġas , ĠFacebook , ĠTwitter , Ġand Ġso Ġon . ĠIt Ġshould Ġbe Ġaddressed Ġthat Ġthere Ġare Ġmany Ġkinds Ġof ĠVietnamese Ġdocuments , Ġfor Ġexample , ĠHan Ġ- ĠNom Ġdocuments Ġand Ġold Ġand Ġmodern ĠVietnamese Ġdocuments Ġthat Ġare Ġessential Ġand Ġstill Ġneeds Ġfurther Ġanalysis . ĠAccording Ġto Ġour Ġstudy , Ġthere Ġis Ġno Ġa Ġpowerful ĠVietnamese Ġlanguage Ġprocessing Ġused Ġfor Ġprocessing ĠVietnamese Ġbig Ġdata Ġas Ġwell Ġas Ġunderstanding Ġsuch Ġlanguage . ĠThe Ġfinal Ġchallenge Ġrelates Ġto Ġbuilding Ġa Ġsystem , Ġwhich Ġis Ġable Ġto Ġincrement ally Ġlearn Ġnew Ġcorpor a Ġand Ġinteract ively Ġprocess Ġfeedback . ĠIn Ġparticular , Ġit Ġis Ġfeasible Ġto Ġbuild Ġan Ġadvance ĠN LP Ġsystem Ġfor ĠVietnamese Ġbased Ġon ĠH ado op Ġplatform Ġto Ġimprove Ġsystem Ġperformance Ġand Ġto Ġaddress Ġexisting Ġlimitations .
Introduction Lexical analysis, syntactic analysis, semantic analysis, disclosure analysis and pragmatic analysis are five main steps in natural language processing BIBREF0 , BIBREF1 . While morphology is a basic task in lexical analysis of English, word segmentation is considered a basic task in lexical analysis of Vietnamese and other East Asian languages processing. This task is to determine borders between words in a sentence. In other words, it is segmenting a list of tokens into a list of words such that words are meaningful. Word segmentation is the primary step in prior to other natural language processing tasks i. e., term extraction and linguistic analysis (as shown in Figure 1). It identifies the basic meaningful units in input texts which will be processed in the next steps of several applications. For named entity recognization BIBREF2 , word segmentation chunks sentences in input documents into sequences of words before they are further classified in to named entity classes. For Vietnamese language, words and candidate terms can be extracted from Vietnamese copora (such as books, novels, news, and so on) by using a word segmentation tool. Conformed features and context of these words and terms are used to identify named entity tags, topic of documents, or function words. For linguistic analysis, several linguistic features from dictionaries can be used either to annotating POS tags or to identifying the answer sentences. Moreover, language models can be trained by using machine learning approaches and be used in tagging systems, like the named entity recognization system of Tran et al. BIBREF2 . Many studies forcus on word segmentation for Asian languages, such as: Chinese, Japanese, Burmese (Myanmar) and Thai BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . Approaches for word segmentation task are variety, from lexicon-based to machine learning-based methods. Recently, machine learning-based methods are used widely to solve this issue, such as: Support Vector Machine or Conditional Random Fields BIBREF7 , BIBREF8 . In general, Chinese is a language which has the most studies on the word segmentation issue. However, there is a lack of survey of word segmentation studies on Asian languages and Vietnamese as well. This paper aims reviewing state-of-the-art word segmentation approaches and systems applying for Vietnamese. This study will be a foundation for studies on Vietnamese word segmentation and other following Vietnamese tasks as well, such as part-of-speech tagger, chunker, or parser systems. There are several studies about the Vietnamese word segmentation task over the last decade. Dinh et al. started this task with Weighted Finite State Transducer (WFST) approach and Neural Network approach BIBREF9 . In addition, machine learning approaches are studied and widely applied to natural language processing and word segmentation as well. In fact, several studies used support vector machines (SVM) and conditional random fields (CRF) for the word segmentation task BIBREF7 , BIBREF8 . Based on annotated corpora and token-based features, studies used machine learning approaches to build word segmentation systems with accuracy about 94%-97%. According to our observation, we found that is lacks of complete review approaches, datasets and toolkits which we recently used in Vietnamese word segmentation. A all sided review of word segmentation will help next studies on Vietnamese natural language processing tasks have an up-to-date guideline and choose the most suitable solution for the task. The remaining part of the paper is organized as follows. Section II discusses building corpus in Vietnamese, containing linguistic issues and the building progress. Section III briefly mentions methods to model sentences and text in machine learning systems. Next, learning models and approaches for labeling and segmenting sequence data will be presented in Section IV. Section V mainly addresses two existing toolkits, vnTokenizer and JVnSegmenter, for Vietnamese word segmentation. Several experiments based on mentioned approaches and toolkits are described in Section VI. Finally, conclusions and future works are given in Section VII. Language Definition Vietnamese, like many languages in continental East Asia, is an isolating language and one branch of Mon-Khmer language group. The most basic linguistic unit in Vietnamese is morpheme, similar with syllable or token in English and “hình vị” (phoneme) or “tiếng” (syllable) in Vietnamese. According to the structured rule of its, Vietnamese can have about 20,000 different syllables (tokens). However, there are about 8,000 syllables used the Vietnamese dictionaries. There are three methods to identify morphemes in Vietnamese text BIBREF10 . Morpheme is the smallest meaningful unit of Vietnamese. Morpheme is the basic unit of Vietnamese. Morpheme is the smallest meaningful unit and is not used independently in the syntax factor. In computational linguistics, morpheme is the basic unit of languages as Leonard Bloomfield mentioned for English BIBREF11 . In our research for Vietnamese, we consider the morpheme as syllable, called “tiếng” in Vietnamese (as Nguyen’s definition BIBREF12 ). The next concept in linguistics is word which has fully grammar and meaning function in sentences. For Vietnamese, word is a single morpheme or a group of morphemes, which are fixed and have full meaning BIBREF12 . According to Nguyen, Vietnamese words are able classified into two types, (1) 1- syllable words with fully meaning and (2) n-syllables words whereas these group of tokens are fixed. Vietnamese syllable is not fully meaningful. However, it is also explained in the meaning and structure characteristics. For example, the token “kỳ” in “quốc kỳ” whereas “quốc” means national, “kỳ” means flag. Therefore, “quốc kỳ” means national flag. Consider dictionary used for evaluating the corpus, extracting features for models, and evaluating the systems, there are many Vietnamese dictionaries, however we recommend the Vietnamese dictionary of Hoang Phe, so called Hoang Phe Dictionary. This dictionary has been built by a group of linguistical scientists at the Linguistic Institute, Vietnam. It was firstly published in 1988, reprinted and extended in 2000, 2005 and 2010. The dictionary currently has 45,757 word items with 15,901 Sino-Vietnamese word items (accounting for 34.75%) BIBREF13 . Name Entity Issue In Vietnamese, not all of meaningful proper names are in the dictionary. Identifying proper names in input text are also important issue in word segmentation. This issue is sometimes included into unknown word issue to be solved. In addition, named entity recognition has to classify it into several types such as person, location, organization, time, money, number, and so on. Proper name identification can be solved by characteristics. For example, systems use beginning characters of proper names which are uppercase characters. Moreover, a list of proper names is also used to identify names in the text. In particular, a list of 2000 personal names extracted from VietnamGiaPha, and a list of 707 names of locations in Vietnam extracted from vi.wikipedia.org are used in the study of Nguyen et al. for Vietnamese word segmentation BIBREF7 . Building Corpus In general, building corpus is carried out through four stages: (1) choose target of corpus and source of raw data; (2) building a guideline based on linguistics knowledge for annotation; (3) annotating or tagging corpus based on rule set in the guideline; and (4) reviewing corpus to check the consistency issue. Encoding word segmentation corpus using B-I-O tagset can be applied, where B, I, and O denoted begin of word, inside of word, and others, respectively. For example, the sentence “Megabit trên giây là đơn vị đo tốc đọ truyền dẫn dữ liệu ." (”Megabit per second is a unit to measure the network traffic.” in English) with the word boundary result “Megabit trên giây là đơn_vị đo tốc_độ truyền_dẫn dữ_liệu ." is encoded as “Megabit/B trên/B giây/B là/B đơn/B vị/I đo/B tốc/B độ/I truyền/B dẫn/I dữ/B liệu/I ./O" . Annotation guidelines can be applied to ensure that annotated corpus has less errors because the manual annotation is applied. Even though there are guidelines for annotating, the available output corpora are still inconsistent. For example, for the Vietnamese Treebank corpus of the VLSP project, Nguyen et al. listed out several Vietnamese word segmentation inconsistencies in the corpus based on POS information and n-gram sequences BIBREF14 . Currently, there are at least three available word segmentation corpus used in Vietnamese word segmentation studies and systems. Firstly, Dinh et al. built the CADASA corpus from CADASA’s books BIBREF15 . Secondly, Nguyen et al. built vnQTAG corpus from general news articles BIBREF7 . More recently, Ngo et al. introduced the EVBCorpus corpus, which is collected from four sources, news articles, books, law documents, and novels. As a part of EVBCorpus, EVBNews, was annotated common tags in NLP, such as word segmentation, chunker, and named entity BIBREF16 . All of these corpora are collected from news articles or book stories, and they are manually annotated the word boundary tags (as shown in Table I). TEXT MODELLING AND FEATURES To understand natural language and analyze documents and text, computers need to represent natural languages as linguistics models. These models can be generated by using machine learning methods (as show in Figure 2). There are two common modeling methods for basic NLP tasks, including n-gram model and bag-of-words model. The n-gram model is widely used in natural language processing while the bag-of-words model is a simplified representation used in natural language processing and information retrieval BIBREF17 , BIBREF18 . According to the bag-of-words model, the representative vector of sentences in the document does not preserve the order of the words in the original sentences. It represents the word using term frequency collected from the document rather than the order of words or the structure of sentences in the document. The bag-of-words model is commonly used in methods of document classification, where the frequency of occurrence of each word is used as an attribute feature for training a classifier. In contrast, an n-gram is a contiguous sequence of n items from a given sequence of text. An n-gram model is a type of probabilistic language model for predicting the next item in a given sequence in form of a Markov model. To address word segmentation issue, the n-gram model is usually used for approaches because it considers the order of tokens in the original sentences. The sequence is also kept the original order as input and output sentences. BUILDING MODEL METHODS There are several studies for Vietnamese Word Segmentation during last decade. For instance, Dinh et al. started the word segmentation task for Vietnamese with Neural Network and Weighted Finite State Transducer (WFST) BIBREF9 . Nguyen et al. continued with machine learning approaches, Conditional Random Fields and Support Vector Machine BIBREF7 . Most of statistical approaches are based on the architecture as shown in Figure 2. According to the architecture, recent studies and systems focus on either improving or modifying difference learning models to get the highest accuracy. Features used in word segmentation systems are syllable, dictionary, and entity name. The detail of all widely used techniques applied are collected and described in following subsections. Maximum Matching Maximum matching (MM) is one of the most popular fundamental and structural segmentation algorithms for word segmentation BIBREF19 . This method is also considered as the Longest Matching (LM) in several research BIBREF9 , BIBREF3 . It is used for identifying word boundary in languages like Chinese, Vietnamese and Thai. This method is a greedy algorithm, which simply chooses longest words based on the dictionary. Segmentation may start from either end of the line without any difference in segmentation results. If the dictionary is sufficient BIBREF19 , the expected segmentation accuracy is over 90%, so it is a major advantage of maximum matching . However, it does not solve the problem of ambiguous words and unknown words that do not exist in the dictionary. There are two types of the maximum matching approach: forward MM (FMM) and backward MM (BMM). FMM starts from the beginning token of the sentence while BMM starts from the end. If the sentence has word boundary ambiguities, the output of FMM and BMM will be different. When applying FMM and BMM, there are two types of common errors due to two ambiguities: overlapping ambiguities and combination ambiguity. Overlapping ambiguities occur when the text AB has both word A, B and AB, which are in the dictionary while the text ABC has word AB and BC, which are in the dictionary. For example, "cụ già đi nhanh quá" (there two meanings: ”the old man goes very fast” or ”the old man died suddenly”) is a case of the overlapping ambiguity while "tốc độ truyền thông tin" is a case of the combination ambiguity. As shown in Figure 2, the method simplification ambiguities, maximum matching is the first step to get features for the modelling stage in machine learning systems, like Conditional Random Fields or Support Vector Machines. Hidden Markov Model (HMM) In Markov chain model is represented as a chain of tokens which are observations, and word taggers are represented as predicted labels. Many researchers applied Hidden Markov model to solve Vietnamese word segmentation such as in BIBREF8 , BIBREF20 and so on. N-gram language modeling applied to estimate probabilities for each word segmentation solution BIBREF21 . The result of this method depends on copora and is based maximal matching strategy. So, they do not solve missing word issue. Let INLINEFORM0 is a product of probabilities of words created from sentence s (1) with length INLINEFORM1 : DISPLAYFORM0 Each conditional probability of word is based on the last n-1 words (n-gram) in the sentence s. It is estimated by Markov chain model for word w from position i-n+1 to i-1 with probability (2) DISPLAYFORM0 We have equation (3) DISPLAYFORM0 Maximum Entropy (ME) Maximum Entropy theory is applied to solve Vietnamese word segmentation BIBREF15 , BIBREF22 , BIBREF23 . Some researchers do not want the limit in Markov chain model. So, they use the context around of the word needed to be segmented. Let h is a context, w is a list of words and t is a list of taggers, Le BIBREF15 , BIBREF22 used DISPLAYFORM0 P(s) is also a product of probabilities of words created from sentence INLINEFORM0 (1). Each conditional probability of word is based on context h of the last n word in the sentence s. Conditional Random Fields To tokenize a Vietnamese word, in HMM or ME, authors only rely on features around a word segment position. Some other features are also affected by adding more special attributes, such as, in case ’?’ question mark at end of sentence, Part of Speech (POS), and so on. Conditional Random Fields is one of methods that uses additional features to improve the selection strategy BIBREF7 . There are several CRF libraries, such as CRF++, CRFsuite. These machine learning toolkits can be used to solve the task by providing an annotated corpus with extracted features. The toolkit will be used to train a model based on the corpus and extract a tagging model. The tagging model will then be used to tag on input text without annotated corpus. In the training and tagging stages, extracting features from the corpus and the input text is necessary for both stages. Support Vector Machines Support Vector Machines (SVM) is a supervised machine learning method which considers dataset as a set of vectors and tries to classify them into specific classes. Basically, SVM is a binary classifier. however, most classification tasks are multi-class classifiers. When applying SVMs, the method has been extended to classify three or more classes. Particular NLP tasks, like word segmentation and Part-of-speech task, each token/word in documents will be used as a feature vector. For the word segmentation task, each token and its features are considered as a vector for the whole document, and the SVM model will classify this vector into one of the three tags (B-IO). This technique is applied for Vietnamese word segmentation in several studies BIBREF7 , BIBREF24 . Nguyen et al. applied on a segmented corpus of 8,000 sentences and got the result at 94.05% while Ngo et al. used it with 45,531 segmented sentences and get the result at 97.2%. It is worth to mention that general SVM libraries (such as LIBSVM, LIBLINEAR, SVMlight, Node-SVM, and TreeSVM ), YamCha is an opened source SVM library that serves several NLP tasks: POS tagging, Named Entity Recognition, base NP chunking, Text Chunking, Text Classification and event Word Segmentation. TOOLKITS vnTokenizer and JVnSegmenter are two famous segmentation toolkits for Vietnamese word segmentation. Both two word segmentation toolkits are implemented the word segmentation data process in Figure 2. This section gives more details of these Vietnamese word toolkits. Programming Languages In general, Java and C++ are the most common language in developing toolkits and systems for natural language processing tasks. For example, GATE, OpenNLP, Stanford CoreNLP and LingPipe platforms are developed by JAVA while foundation tasks and machine learning toolkits are developed by C++. CRF++, SVMLight and YAMCHA . Recently, Python becomes popular among the NLP community. In fact, many toolkits and platforms have been developed by this language, such as NLTK, PyNLPl library for Natural Language Processing. JVnSegmenter JVnSegmenter is a Java-based Vietnamese Word Segmentation Tool developed by Nguyen and Phan. The segmentation model in this tool was trained on about 8,000 tagged Vietnamese text sentences based on CRF model and the model extracted over 151,000 words from the training corpus. In addition, this is used in building the EnglishVietnamese Translation System BIBREF25 , BIBREF26 , BIBREF27 . Vietnamese text classification BIBREF28 and building Vietnamese corpus BIBREF29 , BIBREF30 . vnTokenizer vnTokenizer is implemented in Java and bundled as Eclipse plug-in, and it has already been integrated into vnToolkit, an Eclipse Rich Client application, which is intended to be a general framework integrating tools for processing of Vietnamese text. vnTokenizer plug-in, vnToolkit and related resources, including the lexicon and test corpus are freely available for download. According to our observation, many research cited vnTokenizer to use word segmentation results for applications as building a large Vietnamese corpus BIBREF31 , building an English-Vietnamese Bilingual Corpus for Machine Translation BIBREF32 , Vietnamese text classification BIBREF33 , BIBREF34 , etc. EVALUATION AND RESULTS This research gathers the results of Vietnamese word segmentation of several methods into one table as show in Table II. It is noted that they are not evaluated on a same corpus. The purpose of the result illustration is to provide an overview of the results of current Vietnamese word segmentation systems based on their individual features. All studies mentioned in the table have accuracy around 94-97% based on their provided corpus. This study also evaluates the Vietnamese word segmentation based on existing toolkits using the same annotated Vietnamese word segmentation corpus. There are two available toolkits to evaluate and to segment. To be neutral to both toolkits, we use the EVBNews Vietnamese corpus, a part of EVBCorpus, to evaluate Vietnamese word segmentation. The EVBNews corpus contains over 45,000 segmented Vietnamese sentences extracted from 1,000 general news articles (as shown in Table III) BIBREF16 . We used the same training set which has 1000 files and 45,531 sentences. vnTokenizer outputs 831,455 Vietnamese words and 1,206,475 tokens. JVnSegmenter outputs 840,387 words and 1,201,683. We correct tags (BIO), and compare to previous outputs, we have rate from vnTokenizer is 95.6% and from JVnsegmenter is 93.4%. The result of both vnTokenizer and JVnSegmenter testing on the EVBNews Vietnamese Corpus are provided in Table IV. CONCLUSIONS AND FUTURE WORKS This study reviewed state-of-the-art approaches and systems of Vietnamese word segmentation. The review pointed out common features and methods used in Vietnamese word segmentation studies. This study also had an evaluation of the existing Vietnamese word segmentation toolkits based on a same corpus to show advantages and disadvantages as to shed some lights on system enhancement. There are several challenges on supervised learning approaches in future work. The first challenge is to acquire very large Vietnamese corpus and to use them in building a classifier, which could further improve accuracy. In addition, applying linguistics knowledge on word context to extract useful features also enhances prediction performance. The second challenge is design and development of big data warehouse and analytic framework for Vietnamese documents, which corresponds to the rapid and continuous growth of gigantic volume of articles and/or documents from Web 2.0 applications, such as, Facebook, Twitter, and so on. It should be addressed that there are many kinds of Vietnamese documents, for example, Han - Nom documents and old and modern Vietnamese documents that are essential and still needs further analysis. According to our study, there is no a powerful Vietnamese language processing used for processing Vietnamese big data as well as understanding such language. The final challenge relates to building a system, which is able to incrementally learn new corpora and interactively process feedback. In particular, it is feasible to build an advance NLP system for Vietnamese based on Hadoop platform to improve system performance and to address existing limitations.
What are the limitations of existing Vietnamese word segmentation systems?
8d074aabf4f51c8455618c5bf7689d3f62c4da1d
What Ġare Ġthe Ġlimitations Ġof Ġexisting ĠVietnamese Ġword Ġsegment ation Ġsystems ?
Ġambiguous Ġwords , Ġunknown Ġwords
<s> What Ġare Ġthe Ġlimitations Ġof Ġexisting ĠVietnamese Ġword Ġsegment ation Ġsystems ? </s> Introduction </s> Lex ical Ġanalysis , Ġsynt actic Ġanalysis , Ġsemantic Ġanalysis , Ġdisclosure Ġanalysis Ġand Ġpragmatic Ġanalysis Ġare Ġfive Ġmain Ġsteps Ġin Ġnatural Ġlanguage Ġprocessing ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠWhile Ġmorphology Ġis Ġa Ġbasic Ġtask Ġin Ġlex ical Ġanalysis Ġof ĠEnglish , Ġword Ġsegment ation Ġis Ġconsidered Ġa Ġbasic Ġtask Ġin Ġlex ical Ġanalysis Ġof ĠVietnamese Ġand Ġother ĠEast ĠAsian Ġlanguages Ġprocessing . ĠThis Ġtask Ġis Ġto Ġdetermine Ġborders Ġbetween Ġwords Ġin Ġa Ġsentence . ĠIn Ġother Ġwords , Ġit Ġis Ġsegment ing Ġa Ġlist Ġof Ġtokens Ġinto Ġa Ġlist Ġof Ġwords Ġsuch Ġthat Ġwords Ġare Ġmeaningful . </s> Word Ġsegment ation Ġis Ġthe Ġprimary Ġstep Ġin Ġprior Ġto Ġother Ġnatural Ġlanguage Ġprocessing Ġtasks Ġi . Ġe ., Ġterm Ġextraction Ġand Ġlinguistic Ġanalysis Ġ( as Ġshown Ġin ĠFigure Ġ1 ). ĠIt Ġidentifies Ġthe Ġbasic Ġmeaningful Ġunits Ġin Ġinput Ġtexts Ġwhich Ġwill Ġbe Ġprocessed Ġin Ġthe Ġnext Ġsteps Ġof Ġseveral Ġapplications . ĠFor Ġnamed Ġentity Ġrecogn ization ĠB IB REF 2 Ġ, Ġword Ġsegment ation Ġchunks Ġsentences Ġin Ġinput Ġdocuments Ġinto Ġsequences Ġof Ġwords Ġbefore Ġthey Ġare Ġfurther Ġclassified Ġin Ġto Ġnamed Ġentity Ġclasses . ĠFor ĠVietnamese Ġlanguage , Ġwords Ġand Ġcandidate Ġterms Ġcan Ġbe Ġextracted Ġfrom ĠVietnamese Ġcop ora Ġ( such Ġas Ġbooks , Ġnovels , Ġnews , Ġand Ġso Ġon ) Ġby Ġusing Ġa Ġword Ġsegment ation Ġtool . ĠCon formed Ġfeatures Ġand Ġcontext Ġof Ġthese Ġwords Ġand Ġterms Ġare Ġused Ġto Ġidentify Ġnamed Ġentity Ġtags , Ġtopic Ġof Ġdocuments , Ġor Ġfunction Ġwords . ĠFor Ġlinguistic Ġanalysis , Ġseveral Ġlinguistic Ġfeatures Ġfrom Ġdiction aries Ġcan Ġbe Ġused Ġeither Ġto Ġannot ating ĠPOS Ġtags Ġor Ġto Ġidentifying Ġthe Ġanswer Ġsentences . ĠMoreover , Ġlanguage Ġmodels Ġcan Ġbe Ġtrained Ġby Ġusing Ġmachine Ġlearning Ġapproaches Ġand Ġbe Ġused Ġin Ġtagging Ġsystems , Ġlike Ġthe Ġnamed Ġentity Ġrecogn ization Ġsystem Ġof ĠTr an Ġet Ġal . ĠB IB REF 2 Ġ. </s> Many Ġstudies Ġfor cus Ġon Ġword Ġsegment ation Ġfor ĠAsian Ġlanguages , Ġsuch Ġas : ĠChinese , ĠJapanese , ĠBur m ese Ġ( My anmar ) Ġand ĠThai ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠAppro aches Ġfor Ġword Ġsegment ation Ġtask Ġare Ġvariety , Ġfrom Ġlex icon - based Ġto Ġmachine Ġlearning - based Ġmethods . ĠRecently , Ġmachine Ġlearning - based Ġmethods Ġare Ġused Ġwidely Ġto Ġsolve Ġthis Ġissue , Ġsuch Ġas : ĠSupport ĠVector ĠMachine Ġor ĠCond itional ĠRandom ĠFields ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠIn Ġgeneral , ĠChinese Ġis Ġa Ġlanguage Ġwhich Ġhas Ġthe Ġmost Ġstudies Ġon Ġthe Ġword Ġsegment ation Ġissue . ĠHowever , Ġthere Ġis Ġa Ġlack Ġof Ġsurvey Ġof Ġword Ġsegment ation Ġstudies Ġon ĠAsian Ġlanguages Ġand ĠVietnamese Ġas Ġwell . ĠThis Ġpaper Ġaims Ġreviewing Ġstate - of - the - art Ġword Ġsegment ation Ġapproaches Ġand Ġsystems Ġapplying Ġfor ĠVietnamese . ĠThis Ġstudy Ġwill Ġbe Ġa Ġfoundation Ġfor Ġstudies Ġon ĠVietnamese Ġword Ġsegment ation Ġand Ġother Ġfollowing ĠVietnamese Ġtasks Ġas Ġwell , Ġsuch Ġas Ġpart - of - speech Ġtag ger , Ġchunk er , Ġor Ġparser Ġsystems . </s> There Ġare Ġseveral Ġstudies Ġabout Ġthe ĠVietnamese Ġword Ġsegment ation Ġtask Ġover Ġthe Ġlast Ġdecade . ĠDin h Ġet Ġal . Ġstarted Ġthis Ġtask Ġwith ĠWeight ed ĠFin ite ĠState ĠTrans du cer Ġ( WF ST ) Ġapproach Ġand ĠNeural ĠNetwork Ġapproach ĠB IB REF 9 Ġ. ĠIn Ġaddition , Ġmachine Ġlearning Ġapproaches Ġare Ġstudied Ġand Ġwidely Ġapplied Ġto Ġnatural Ġlanguage Ġprocessing Ġand Ġword Ġsegment ation Ġas Ġwell . ĠIn Ġfact , Ġseveral Ġstudies Ġused Ġsupport Ġvector Ġmachines Ġ( S VM ) Ġand Ġconditional Ġrandom Ġfields Ġ( CR F ) Ġfor Ġthe Ġword Ġsegment ation Ġtask ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠBased Ġon Ġannot ated Ġcorpor a Ġand Ġtoken - based Ġfeatures , Ġstudies Ġused Ġmachine Ġlearning Ġapproaches Ġto Ġbuild Ġword Ġsegment ation Ġsystems Ġwith Ġaccuracy Ġabout Ġ94 %- 97 %. </s> According Ġto Ġour Ġobservation , Ġwe Ġfound Ġthat Ġis Ġlacks Ġof Ġcomplete Ġreview Ġapproaches , Ġdatasets Ġand Ġtool k its Ġwhich Ġwe Ġrecently Ġused Ġin ĠVietnamese Ġword Ġsegment ation . ĠA Ġall Ġsided Ġreview Ġof Ġword Ġsegment ation Ġwill Ġhelp Ġnext Ġstudies Ġon ĠVietnamese Ġnatural Ġlanguage Ġprocessing Ġtasks Ġhave Ġan Ġup - to - date Ġguideline Ġand Ġchoose Ġthe Ġmost Ġsuitable Ġsolution Ġfor Ġthe Ġtask . ĠThe Ġremaining Ġpart Ġof Ġthe Ġpaper Ġis Ġorganized Ġas Ġfollows . ĠSection ĠII Ġdiscusses Ġbuilding Ġcorpus Ġin ĠVietnamese , Ġcontaining Ġlinguistic Ġissues Ġand Ġthe Ġbuilding Ġprogress . ĠSection ĠIII Ġbriefly Ġmentions Ġmethods Ġto Ġmodel Ġsentences Ġand Ġtext Ġin Ġmachine Ġlearning Ġsystems . ĠNext , Ġlearning Ġmodels Ġand Ġapproaches Ġfor Ġlabeling Ġand Ġsegment ing Ġsequence Ġdata Ġwill Ġbe Ġpresented Ġin ĠSection ĠIV . ĠSection ĠV Ġmainly Ġaddresses Ġtwo Ġexisting Ġtool k its , Ġv n Token izer Ġand ĠJ V n Seg ment er , Ġfor ĠVietnamese Ġword Ġsegment ation . ĠSeveral Ġexperiments Ġbased Ġon Ġmentioned Ġapproaches Ġand Ġtool k its Ġare Ġdescribed Ġin ĠSection ĠVI . ĠFinally , Ġconclusions Ġand Ġfuture Ġworks Ġare Ġgiven Ġin ĠSection ĠVII . </s> Language ĠDefinition </s> V iet namese , Ġlike Ġmany Ġlanguages Ġin Ġcontinental ĠEast ĠAsia , Ġis Ġan Ġisol ating Ġlanguage Ġand Ġone Ġbranch Ġof ĠMon - Kh mer Ġlanguage Ġgroup . ĠThe Ġmost Ġbasic Ġlinguistic Ġunit Ġin ĠVietnamese Ġis Ġmor phe me , Ġsimilar Ġwith Ġsyll able Ġor Ġtoken Ġin ĠEnglish Ġand ĠâĢ ľ h Ã ¬ n h Ġv á » ĭ âĢ Ŀ Ġ( ph on eme ) Ġor ĠâĢ ľ ti á º ¿ ng âĢ Ŀ Ġ( sy ll able ) Ġin ĠVietnamese . ĠAccording Ġto Ġthe Ġstructured Ġrule Ġof Ġits , ĠVietnamese Ġcan Ġhave Ġabout Ġ20 , 000 Ġdifferent Ġsyll ables Ġ( t ok ens ). ĠHowever , Ġthere Ġare Ġabout Ġ8 , 000 Ġsyll ables Ġused Ġthe ĠVietnamese Ġdiction aries . ĠThere Ġare Ġthree Ġmethods Ġto Ġidentify Ġmorp hem es Ġin ĠVietnamese Ġtext ĠB IB REF 10 Ġ. </s> M orp he me Ġis Ġthe Ġsmallest Ġmeaningful Ġunit Ġof ĠVietnamese . </s> M orp he me Ġis Ġthe Ġbasic Ġunit Ġof ĠVietnamese . </s> M orp he me Ġis Ġthe Ġsmallest Ġmeaningful Ġunit Ġand Ġis Ġnot Ġused Ġindependently Ġin Ġthe Ġsyntax Ġfactor . </s> In Ġcomputational Ġlingu istics , Ġmor phe me Ġis Ġthe Ġbasic Ġunit Ġof Ġlanguages Ġas ĠLeonard ĠBloom field Ġmentioned Ġfor ĠEnglish ĠB IB REF 11 Ġ. ĠIn Ġour Ġresearch Ġfor ĠVietnamese , Ġwe Ġconsider Ġthe Ġmor phe me Ġas Ġsyll able , Ġcalled ĠâĢ ľ ti á º ¿ ng âĢ Ŀ Ġin ĠVietnamese Ġ( as ĠNguyen âĢ Ļ s Ġdefinition ĠB IB REF 12 Ġ). </s> The Ġnext Ġconcept Ġin Ġlingu istics Ġis Ġword Ġwhich Ġhas Ġfully Ġgrammar Ġand Ġmeaning Ġfunction Ġin Ġsentences . ĠFor ĠVietnamese , Ġword Ġis Ġa Ġsingle Ġmor phe me Ġor Ġa Ġgroup Ġof Ġmorp hem es , Ġwhich Ġare Ġfixed Ġand Ġhave Ġfull Ġmeaning ĠB IB REF 12 Ġ. ĠAccording Ġto ĠNguyen , ĠVietnamese Ġwords Ġare Ġable Ġclassified Ġinto Ġtwo Ġtypes , Ġ( 1 ) Ġ1 - Ġsyll able Ġwords Ġwith Ġfully Ġmeaning Ġand Ġ( 2 ) Ġn - sy ll ables Ġwords Ġwhereas Ġthese Ġgroup Ġof Ġtokens Ġare Ġfixed . ĠVietnamese Ġsyll able Ġis Ġnot Ġfully Ġmeaningful . ĠHowever , Ġit Ġis Ġalso Ġexplained Ġin Ġthe Ġmeaning Ġand Ġstructure Ġcharacteristics . ĠFor Ġexample , Ġthe Ġtoken ĠâĢ ľ k á » ³ âĢ Ŀ Ġin ĠâĢ ľ qu á » ĳ c Ġk á » ³ âĢ Ŀ Ġwhereas ĠâĢ ľ qu á » ĳ c âĢ Ŀ Ġmeans Ġnational , ĠâĢ ľ k á » ³ âĢ Ŀ Ġmeans Ġflag . ĠTherefore , ĠâĢ ľ qu á » ĳ c Ġk á » ³ âĢ Ŀ Ġmeans Ġnational Ġflag . </s> Consider Ġdictionary Ġused Ġfor Ġevaluating Ġthe Ġcorpus , Ġextracting Ġfeatures Ġfor Ġmodels , Ġand Ġevaluating Ġthe Ġsystems , Ġthere Ġare Ġmany ĠVietnamese Ġdiction aries , Ġhowever Ġwe Ġrecommend Ġthe ĠVietnamese Ġdictionary Ġof ĠHo ang ĠP he , Ġso Ġcalled ĠHo ang ĠP he ĠDictionary . ĠThis Ġdictionary Ġhas Ġbeen Ġbuilt Ġby Ġa Ġgroup Ġof Ġlingu istical Ġscientists Ġat Ġthe ĠL ingu istic ĠInstitute , ĠVietnam . ĠIt Ġwas Ġfirst ly Ġpublished Ġin Ġ1988 , Ġreprinted Ġand Ġextended Ġin Ġ2000 , Ġ2005 Ġand Ġ2010 . ĠThe Ġdictionary Ġcurrently Ġhas Ġ45 , 757 Ġword Ġitems Ġwith Ġ15 , 901 ĠS ino - V iet namese Ġword Ġitems Ġ( account ing Ġfor Ġ34 . 75 %) ĠB IB REF 13 Ġ. </s> Name ĠEntity ĠIssue </s> In ĠVietnamese , Ġnot Ġall Ġof Ġmeaningful Ġproper Ġnames Ġare Ġin Ġthe Ġdictionary . ĠIdent ifying Ġproper Ġnames Ġin Ġinput Ġtext Ġare Ġalso Ġimportant Ġissue Ġin Ġword Ġsegment ation . ĠThis Ġissue Ġis Ġsometimes Ġincluded Ġinto Ġunknown Ġword Ġissue Ġto Ġbe Ġsolved . ĠIn Ġaddition , Ġnamed Ġentity Ġrecognition Ġhas Ġto Ġclassify Ġit Ġinto Ġseveral Ġtypes Ġsuch Ġas Ġperson , Ġlocation , Ġorganization , Ġtime , Ġmoney , Ġnumber , Ġand Ġso Ġon . </s> Pro per Ġname Ġidentification Ġcan Ġbe Ġsolved Ġby Ġcharacteristics . ĠFor Ġexample , Ġsystems Ġuse Ġbeginning Ġcharacters Ġof Ġproper Ġnames Ġwhich Ġare Ġu pperc ase Ġcharacters . ĠMoreover , Ġa Ġlist Ġof Ġproper Ġnames Ġis Ġalso Ġused Ġto Ġidentify Ġnames Ġin Ġthe Ġtext . ĠIn Ġparticular , Ġa Ġlist Ġof Ġ2000 Ġpersonal Ġnames Ġextracted Ġfrom ĠVietnam G ia Ph a , Ġand Ġa Ġlist Ġof Ġ7 07 Ġnames Ġof Ġlocations Ġin ĠVietnam Ġextracted Ġfrom Ġvi . wikipedia . org Ġare Ġused Ġin Ġthe Ġstudy Ġof ĠNguyen Ġet Ġal . Ġfor ĠVietnamese Ġword Ġsegment ation ĠB IB REF 7 Ġ. </s> Building ĠCorpus </s> In Ġgeneral , Ġbuilding Ġcorpus Ġis Ġcarried Ġout Ġthrough Ġfour Ġstages : Ġ( 1 ) Ġchoose Ġtarget Ġof Ġcorpus Ġand Ġsource Ġof Ġraw Ġdata ; Ġ( 2 ) Ġbuilding Ġa Ġguideline Ġbased Ġon Ġlingu istics Ġknowledge Ġfor Ġannotation ; Ġ( 3 ) Ġannot ating Ġor Ġtagging Ġcorpus Ġbased Ġon Ġrule Ġset Ġin Ġthe Ġguideline ; Ġand Ġ( 4 ) Ġreviewing Ġcorpus Ġto Ġcheck Ġthe Ġconsistency Ġissue . </s> Enc oding Ġword Ġsegment ation Ġcorpus Ġusing ĠB - I - O Ġtag set Ġcan Ġbe Ġapplied , Ġwhere ĠB , ĠI , Ġand ĠO Ġden oted Ġbegin Ġof Ġword , Ġinside Ġof Ġword , Ġand Ġothers , Ġrespectively . ĠFor Ġexample , Ġthe Ġsentence ĠâĢ ľ Meg abit Ġtr Ãª n Ġg i Ã¢ y Ġl Ãł ĠÄ ĳ Æ ¡ n Ġv á » ĭ ĠÄ ĳ o Ġt á » ĳ c ĠÄ ĳ á » į Ġtr uy á » ģ n Ġd á º « n Ġd á » ¯ Ġli á » ĩ u Ġ." Ġ( âĢ Ŀ Meg abit Ġper Ġsecond Ġis Ġa Ġunit Ġto Ġmeasure Ġthe Ġnetwork Ġtraffic . âĢ Ŀ Ġin ĠEnglish ) Ġwith Ġthe Ġword Ġboundary Ġresult ĠâĢ ľ Meg abit Ġtr Ãª n Ġg i Ã¢ y Ġl Ãł ĠÄ ĳ Æ ¡ n _ v á » ĭ ĠÄ ĳ o Ġt á » ĳ c _ Ä ĳ á » Ļ Ġtr uy á » ģ n _ d á º « n Ġd á » ¯ _ li á » ĩ u Ġ." Ġis Ġencoded Ġas ĠâĢ ľ Meg abit / B Ġtr Ãª n / B Ġg i Ã¢ y / B Ġl Ãł / B ĠÄ ĳ Æ ¡ n / B Ġv á » ĭ / I ĠÄ ĳ o / B Ġt á » ĳ c / B ĠÄ ĳ á » Ļ / I Ġtr uy á » ģ n / B Ġd á º « n / I Ġd á » ¯ / B Ġli á » ĩ u / I Ġ./ O " Ġ. </s> An notation Ġguidelines Ġcan Ġbe Ġapplied Ġto Ġensure Ġthat Ġannot ated Ġcorpus Ġhas Ġless Ġerrors Ġbecause Ġthe Ġmanual Ġannotation Ġis Ġapplied . ĠEven Ġthough Ġthere Ġare Ġguidelines Ġfor Ġannot ating , Ġthe Ġavailable Ġoutput Ġcorpor a Ġare Ġstill Ġinconsistent . ĠFor Ġexample , Ġfor Ġthe ĠVietnamese ĠTree bank Ġcorpus Ġof Ġthe ĠV L SP Ġproject , ĠNguyen Ġet Ġal . Ġlisted Ġout Ġseveral ĠVietnamese Ġword Ġsegment ation Ġinconsistencies Ġin Ġthe Ġcorpus Ġbased Ġon ĠPOS Ġinformation Ġand Ġn - gram Ġsequences ĠB IB REF 14 Ġ. </s> Currently , Ġthere Ġare Ġat Ġleast Ġthree Ġavailable Ġword Ġsegment ation Ġcorpus Ġused Ġin ĠVietnamese Ġword Ġsegment ation Ġstudies Ġand Ġsystems . ĠFirstly , ĠDin h Ġet Ġal . Ġbuilt Ġthe ĠCAD AS A Ġcorpus Ġfrom ĠCAD AS A âĢ Ļ s Ġbooks ĠB IB REF 15 Ġ. ĠSecondly , ĠNguyen Ġet Ġal . Ġbuilt Ġv n Q TAG Ġcorpus Ġfrom Ġgeneral Ġnews Ġarticles ĠB IB REF 7 Ġ. ĠMore Ġrecently , ĠN go Ġet Ġal . Ġintroduced Ġthe ĠEV BC orp us Ġcorpus , Ġwhich Ġis Ġcollected Ġfrom Ġfour Ġsources , Ġnews Ġarticles , Ġbooks , Ġlaw Ġdocuments , Ġand Ġnovels . ĠAs Ġa Ġpart Ġof ĠEV BC orp us , ĠEV B News , Ġwas Ġannot ated Ġcommon Ġtags Ġin ĠN LP , Ġsuch Ġas Ġword Ġsegment ation , Ġchunk er , Ġand Ġnamed Ġentity ĠB IB REF 16 Ġ. ĠAll Ġof Ġthese Ġcorpor a Ġare Ġcollected Ġfrom Ġnews Ġarticles Ġor Ġbook Ġstories , Ġand Ġthey Ġare Ġmanually Ġannot ated Ġthe Ġword Ġboundary Ġtags Ġ( as Ġshown Ġin ĠTable ĠI ). </s> TEXT ĠM ODE LL ING ĠAND ĠFE ATURES </s> To Ġunderstand Ġnatural Ġlanguage Ġand Ġanalyze Ġdocuments Ġand Ġtext , Ġcomputers Ġneed Ġto Ġrepresent Ġnatural Ġlanguages Ġas Ġlingu istics Ġmodels . ĠThese Ġmodels Ġcan Ġbe Ġgenerated Ġby Ġusing Ġmachine Ġlearning Ġmethods Ġ( as Ġshow Ġin ĠFigure Ġ2 ). </s> There Ġare Ġtwo Ġcommon Ġmodeling Ġmethods Ġfor Ġbasic ĠN LP Ġtasks , Ġincluding Ġn - gram Ġmodel Ġand Ġbag - of - words Ġmodel . ĠThe Ġn - gram Ġmodel Ġis Ġwidely Ġused Ġin Ġnatural Ġlanguage Ġprocessing Ġwhile Ġthe Ġbag - of - words Ġmodel Ġis Ġa Ġsimplified Ġrepresentation Ġused Ġin Ġnatural Ġlanguage Ġprocessing Ġand Ġinformation Ġretrieval ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠAccording Ġto Ġthe Ġbag - of - words Ġmodel , Ġthe Ġrepresentative Ġvector Ġof Ġsentences Ġin Ġthe Ġdocument Ġdoes Ġnot Ġpreserve Ġthe Ġorder Ġof Ġthe Ġwords Ġin Ġthe Ġoriginal Ġsentences . ĠIt Ġrepresents Ġthe Ġword Ġusing Ġterm Ġfrequency Ġcollected Ġfrom Ġthe Ġdocument Ġrather Ġthan Ġthe Ġorder Ġof Ġwords Ġor Ġthe Ġstructure Ġof Ġsentences Ġin Ġthe Ġdocument . ĠThe Ġbag - of - words Ġmodel Ġis Ġcommonly Ġused Ġin Ġmethods Ġof Ġdocument Ġclassification , Ġwhere Ġthe Ġfrequency Ġof Ġoccurrence Ġof Ġeach Ġword Ġis Ġused Ġas Ġan Ġattribute Ġfeature Ġfor Ġtraining Ġa Ġclass ifier . ĠIn Ġcontrast , Ġan Ġn - gram Ġis Ġa Ġcontiguous Ġsequence Ġof Ġn Ġitems Ġfrom Ġa Ġgiven Ġsequence Ġof Ġtext . ĠAn Ġn - gram Ġmodel Ġis Ġa Ġtype Ġof Ġprob abil istic Ġlanguage Ġmodel Ġfor Ġpredicting Ġthe Ġnext Ġitem Ġin Ġa Ġgiven Ġsequence Ġin Ġform Ġof Ġa ĠMark ov Ġmodel . ĠTo Ġaddress Ġword Ġsegment ation Ġissue , Ġthe Ġn - gram Ġmodel Ġis Ġusually Ġused Ġfor Ġapproaches Ġbecause Ġit Ġconsiders Ġthe Ġorder Ġof Ġtokens Ġin Ġthe Ġoriginal Ġsentences . ĠThe Ġsequence Ġis Ġalso Ġkept Ġthe Ġoriginal Ġorder Ġas Ġinput Ġand Ġoutput Ġsentences . </s> BU ILD ING ĠMOD EL ĠM ETHOD S </s> There Ġare Ġseveral Ġstudies Ġfor ĠVietnamese ĠWord ĠSe gment ation Ġduring Ġlast Ġdecade . ĠFor Ġinstance , ĠDin h Ġet Ġal . Ġstarted Ġthe Ġword Ġsegment ation Ġtask Ġfor ĠVietnamese Ġwith ĠNeural ĠNetwork Ġand ĠWeight ed ĠFin ite ĠState ĠTrans du cer Ġ( WF ST ) ĠB IB REF 9 Ġ. ĠNguyen Ġet Ġal . Ġcontinued Ġwith Ġmachine Ġlearning Ġapproaches , ĠCond itional ĠRandom ĠFields Ġand ĠSupport ĠVector ĠMachine ĠB IB REF 7 Ġ. ĠMost Ġof Ġstatistical Ġapproaches Ġare Ġbased Ġon Ġthe Ġarchitecture Ġas Ġshown Ġin ĠFigure Ġ2 . ĠAccording Ġto Ġthe Ġarchitecture , Ġrecent Ġstudies Ġand Ġsystems Ġfocus Ġon Ġeither Ġimproving Ġor Ġmodifying Ġdifference Ġlearning Ġmodels Ġto Ġget Ġthe Ġhighest Ġaccuracy . ĠFeatures Ġused Ġin Ġword Ġsegment ation Ġsystems Ġare Ġsyll able , Ġdictionary , Ġand Ġentity Ġname . ĠThe Ġdetail Ġof Ġall Ġwidely Ġused Ġtechniques Ġapplied Ġare Ġcollected Ġand Ġdescribed Ġin Ġfollowing Ġsubsections . </s> Maximum ĠMatch ing </s> Maximum Ġmatching Ġ( MM ) Ġis Ġone Ġof Ġthe Ġmost Ġpopular Ġfundamental Ġand Ġstructural Ġsegment ation Ġalgorithms Ġfor Ġword Ġsegment ation ĠB IB REF 19 Ġ. ĠThis Ġmethod Ġis Ġalso Ġconsidered Ġas Ġthe ĠLong est ĠMatch ing Ġ( LM ) Ġin Ġseveral Ġresearch ĠB IB REF 9 Ġ, ĠB IB REF 3 Ġ. ĠIt Ġis Ġused Ġfor Ġidentifying Ġword Ġboundary Ġin Ġlanguages Ġlike ĠChinese , ĠVietnamese Ġand ĠThai . ĠThis Ġmethod Ġis Ġa Ġgreedy Ġalgorithm , Ġwhich Ġsimply Ġchooses Ġlongest Ġwords Ġbased Ġon Ġthe Ġdictionary . ĠSe gment ation Ġmay Ġstart Ġfrom Ġeither Ġend Ġof Ġthe Ġline Ġwithout Ġany Ġdifference Ġin Ġsegment ation Ġresults . ĠIf Ġthe Ġdictionary Ġis Ġsufficient ĠB IB REF 19 Ġ, Ġthe Ġexpected Ġsegment ation Ġaccuracy Ġis Ġover Ġ90 %, Ġso Ġit Ġis Ġa Ġmajor Ġadvantage Ġof Ġmaximum Ġmatching Ġ. ĠHowever , Ġit Ġdoes Ġnot Ġsolve Ġthe Ġproblem Ġof Ġambiguous Ġwords Ġand Ġunknown Ġwords Ġthat Ġdo Ġnot Ġexist Ġin Ġthe Ġdictionary . </s> There Ġare Ġtwo Ġtypes Ġof Ġthe Ġmaximum Ġmatching Ġapproach : Ġforward ĠMM Ġ( F MM ) Ġand Ġbackward ĠMM Ġ( B MM ). ĠF MM Ġstarts Ġfrom Ġthe Ġbeginning Ġtoken Ġof Ġthe Ġsentence Ġwhile ĠB MM Ġstarts Ġfrom Ġthe Ġend . ĠIf Ġthe Ġsentence Ġhas Ġword Ġboundary Ġambig u ities , Ġthe Ġoutput Ġof ĠF MM Ġand ĠB MM Ġwill Ġbe Ġdifferent . ĠWhen Ġapplying ĠF MM Ġand ĠB MM , Ġthere Ġare Ġtwo Ġtypes Ġof Ġcommon Ġerrors Ġdue Ġto Ġtwo Ġambig u ities : Ġoverlapping Ġambig u ities Ġand Ġcombination Ġambiguity . ĠOver l apping Ġambig u ities Ġoccur Ġwhen Ġthe Ġtext ĠAB Ġhas Ġboth Ġword ĠA , ĠB Ġand ĠAB , Ġwhich Ġare Ġin Ġthe Ġdictionary Ġwhile Ġthe Ġtext ĠABC Ġhas Ġword ĠAB Ġand ĠBC , Ġwhich Ġare Ġin Ġthe Ġdictionary . ĠFor Ġexample , Ġ" c á » ¥ Ġg i Ãł ĠÄ ĳ i Ġn han h Ġqu Ã¡ " Ġ( there Ġtwo Ġmeanings : ĠâĢ Ŀ the Ġold Ġman Ġgoes Ġvery Ġfast âĢ Ŀ Ġor ĠâĢ Ŀ the Ġold Ġman Ġdied Ġsuddenly âĢ Ŀ ) Ġis Ġa Ġcase Ġof Ġthe Ġoverlapping Ġambiguity Ġwhile Ġ" t á » ĳ c ĠÄ ĳ á » Ļ Ġtr uy á » ģ n Ġth Ã´ ng Ġtin " Ġis Ġa Ġcase Ġof Ġthe Ġcombination Ġambiguity . </s> As Ġshown Ġin ĠFigure Ġ2 , Ġthe Ġmethod Ġsimpl ification Ġambig u ities , Ġmaximum Ġmatching Ġis Ġthe Ġfirst Ġstep Ġto Ġget Ġfeatures Ġfor Ġthe Ġmodelling Ġstage Ġin Ġmachine Ġlearning Ġsystems , Ġlike ĠCond itional ĠRandom ĠFields Ġor ĠSupport ĠVector ĠMachines . </s> Hidden ĠMark ov ĠModel Ġ( H MM ) </s> In ĠMark ov Ġchain Ġmodel Ġis Ġrepresented Ġas Ġa Ġchain Ġof Ġtokens Ġwhich Ġare Ġobservations , Ġand Ġword Ġtag gers Ġare Ġrepresented Ġas Ġpredicted Ġlabels . ĠMany Ġresearchers Ġapplied ĠHidden ĠMark ov Ġmodel Ġto Ġsolve ĠVietnamese Ġword Ġsegment ation Ġsuch Ġas Ġin ĠB IB REF 8 Ġ, ĠB IB REF 20 Ġand Ġso Ġon . </s> N - gram Ġlanguage Ġmodeling Ġapplied Ġto Ġestimate Ġprobabilities Ġfor Ġeach Ġword Ġsegment ation Ġsolution ĠB IB REF 21 Ġ. ĠThe Ġresult Ġof Ġthis Ġmethod Ġdepends Ġon Ġcop ora Ġand Ġis Ġbased Ġmaximal Ġmatching Ġstrategy . ĠSo , Ġthey Ġdo Ġnot Ġsolve Ġmissing Ġword Ġissue . ĠLet ĠIN LINE FORM 0 Ġis Ġa Ġproduct Ġof Ġprobabilities Ġof Ġwords Ġcreated Ġfrom Ġsentence Ġs Ġ( 1 ) Ġwith Ġlength ĠIN LINE FORM 1 Ġ: ĠDIS PLAY FORM 0 </s> Each Ġconditional Ġprobability Ġof Ġword Ġis Ġbased Ġon Ġthe Ġlast Ġn - 1 Ġwords Ġ( n - gram ) Ġin Ġthe Ġsentence Ġs . ĠIt Ġis Ġestimated Ġby ĠMark ov Ġchain Ġmodel Ġfor Ġword Ġw Ġfrom Ġposition Ġi - n + 1 Ġto Ġi - 1 Ġwith Ġprobability Ġ( 2 ) ĠDIS PLAY FORM 0 </s> We Ġhave Ġequation Ġ( 3 ) ĠDIS PLAY FORM 0 </s> Maximum ĠEnt ropy Ġ( ME ) </s> Maximum ĠEnt ropy Ġtheory Ġis Ġapplied Ġto Ġsolve ĠVietnamese Ġword Ġsegment ation ĠB IB REF 15 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ. ĠSome Ġresearchers Ġdo Ġnot Ġwant Ġthe Ġlimit Ġin ĠMark ov Ġchain Ġmodel . ĠSo , Ġthey Ġuse Ġthe Ġcontext Ġaround Ġof Ġthe Ġword Ġneeded Ġto Ġbe Ġsegment ed . ĠLet Ġh Ġis Ġa Ġcontext , Ġw Ġis Ġa Ġlist Ġof Ġwords Ġand Ġt Ġis Ġa Ġlist Ġof Ġtag gers , ĠLe ĠB IB REF 15 Ġ, ĠB IB REF 22 Ġused ĠDIS PLAY FORM 0 </s> P ( s ) Ġis Ġalso Ġa Ġproduct Ġof Ġprobabilities Ġof Ġwords Ġcreated Ġfrom Ġsentence ĠIN LINE FORM 0 Ġ( 1 ). ĠEach Ġconditional Ġprobability Ġof Ġword Ġis Ġbased Ġon Ġcontext Ġh Ġof Ġthe Ġlast Ġn Ġword Ġin Ġthe Ġsentence Ġs . </s> Cond itional ĠRandom ĠFields </s> To Ġtoken ize Ġa ĠVietnamese Ġword , Ġin ĠH MM Ġor ĠME , Ġauthors Ġonly Ġrely Ġon Ġfeatures Ġaround Ġa Ġword Ġsegment Ġposition . ĠSome Ġother Ġfeatures Ġare Ġalso Ġaffected Ġby Ġadding Ġmore Ġspecial Ġattributes , Ġsuch Ġas , Ġin Ġcase ĠâĢ Ļ ? âĢ Ļ Ġquestion Ġmark Ġat Ġend Ġof Ġsentence , ĠPart Ġof ĠSpeech Ġ( POS ), Ġand Ġso Ġon . ĠCond itional ĠRandom ĠFields Ġis Ġone Ġof Ġmethods Ġthat Ġuses Ġadditional Ġfeatures Ġto Ġimprove Ġthe Ġselection Ġstrategy ĠB IB REF 7 Ġ. </s> There Ġare Ġseveral ĠCR F Ġlibraries , Ġsuch Ġas ĠCR F ++ , ĠCR F su ite . ĠThese Ġmachine Ġlearning Ġtool k its Ġcan Ġbe Ġused Ġto Ġsolve Ġthe Ġtask Ġby Ġproviding Ġan Ġannot ated Ġcorpus Ġwith Ġextracted Ġfeatures . ĠThe Ġtool kit Ġwill Ġbe Ġused Ġto Ġtrain Ġa Ġmodel Ġbased Ġon Ġthe Ġcorpus Ġand Ġextract Ġa Ġtagging Ġmodel . ĠThe Ġtagging Ġmodel Ġwill Ġthen Ġbe Ġused Ġto Ġtag Ġon Ġinput Ġtext Ġwithout Ġannot ated Ġcorpus . ĠIn Ġthe Ġtraining Ġand Ġtagging Ġstages , Ġextracting Ġfeatures Ġfrom Ġthe Ġcorpus Ġand Ġthe Ġinput Ġtext Ġis Ġnecessary Ġfor Ġboth Ġstages . </s> Support ĠVector ĠMachines </s> Support ĠVector ĠMachines Ġ( S VM ) Ġis Ġa Ġsupervised Ġmachine Ġlearning Ġmethod Ġwhich Ġconsiders Ġdataset Ġas Ġa Ġset Ġof Ġvectors Ġand Ġtries Ġto Ġclassify Ġthem Ġinto Ġspecific Ġclasses . ĠBasically , ĠS VM Ġis Ġa Ġbinary Ġclass ifier . Ġhowever , Ġmost Ġclassification Ġtasks Ġare Ġmulti - class Ġclass ifiers . ĠWhen Ġapplying ĠSV Ms , Ġthe Ġmethod Ġhas Ġbeen Ġextended Ġto Ġclassify Ġthree Ġor Ġmore Ġclasses . ĠPart icular ĠN LP Ġtasks , Ġlike Ġword Ġsegment ation Ġand ĠPart - of - speech Ġtask , Ġeach Ġtoken / word Ġin Ġdocuments Ġwill Ġbe Ġused Ġas Ġa Ġfeature Ġvector . ĠFor Ġthe Ġword Ġsegment ation Ġtask , Ġeach Ġtoken Ġand Ġits Ġfeatures Ġare Ġconsidered Ġas Ġa Ġvector Ġfor Ġthe Ġwhole Ġdocument , Ġand Ġthe ĠS VM Ġmodel Ġwill Ġclassify Ġthis Ġvector Ġinto Ġone Ġof Ġthe Ġthree Ġtags Ġ( B - IO ). </s> This Ġtechnique Ġis Ġapplied Ġfor ĠVietnamese Ġword Ġsegment ation Ġin Ġseveral Ġstudies ĠB IB REF 7 Ġ, ĠB IB REF 24 Ġ. ĠNguyen Ġet Ġal . Ġapplied Ġon Ġa Ġsegment ed Ġcorpus Ġof Ġ8 , 000 Ġsentences Ġand Ġgot Ġthe Ġresult Ġat Ġ94 . 05 % Ġwhile ĠN go Ġet Ġal . Ġused Ġit Ġwith Ġ45 , 5 31 Ġsegment ed Ġsentences Ġand Ġget Ġthe Ġresult Ġat Ġ97 . 2 %. ĠIt Ġis Ġworth Ġto Ġmention Ġthat Ġgeneral ĠS VM Ġlibraries Ġ( such Ġas ĠLI BS VM , ĠLI BL INE AR , ĠS VM light , ĠNode - S VM , Ġand ĠTree S VM Ġ), ĠYam Ch a Ġis Ġan Ġopened Ġsource ĠS VM Ġlibrary Ġthat Ġserves Ġseveral ĠN LP Ġtasks : ĠPOS Ġtagging , ĠNamed ĠEntity ĠRecogn ition , Ġbase ĠNP Ġchunk ing , ĠText ĠCh unk ing , ĠText ĠClassification Ġand Ġevent ĠWord ĠSe gment ation . </s> TO OL K ITS </s> v n Token izer Ġand ĠJ V n Seg ment er Ġare Ġtwo Ġfamous Ġsegment ation Ġtool k its Ġfor ĠVietnamese Ġword Ġsegment ation . ĠBoth Ġtwo Ġword Ġsegment ation Ġtool k its Ġare Ġimplemented Ġthe Ġword Ġsegment ation Ġdata Ġprocess Ġin ĠFigure Ġ2 . ĠThis Ġsection Ġgives Ġmore Ġdetails Ġof Ġthese ĠVietnamese Ġword Ġtool k its . </s> Program ming ĠLanguages </s> In Ġgeneral , ĠJava Ġand ĠC ++ Ġare Ġthe Ġmost Ġcommon Ġlanguage Ġin Ġdeveloping Ġtool k its Ġand Ġsystems Ġfor Ġnatural Ġlanguage Ġprocessing Ġtasks . ĠFor Ġexample , ĠG ATE , ĠOpen N LP , ĠStanford ĠCore N LP Ġand ĠLing P ipe Ġplatforms Ġare Ġdeveloped Ġby ĠJ AV A Ġwhile Ġfoundation Ġtasks Ġand Ġmachine Ġlearning Ġtool k its Ġare Ġdeveloped Ġby ĠC ++ . ĠCR F ++ , ĠSV ML ight Ġand ĠY AM CHA Ġ. ĠRecently , ĠPython Ġbecomes Ġpopular Ġamong Ġthe ĠN LP Ġcommunity . ĠIn Ġfact , Ġmany Ġtool k its Ġand Ġplatforms Ġhave Ġbeen Ġdeveloped Ġby Ġthis Ġlanguage , Ġsuch Ġas ĠNL T K , ĠPy NL Pl Ġlibrary Ġfor ĠNatural ĠLanguage ĠProcessing . </s> JV n Seg ment er </s> JV n Seg ment er Ġis Ġa ĠJava - based ĠVietnamese ĠWord ĠSe gment ation ĠTool Ġdeveloped Ġby ĠNguyen Ġand ĠPh an . ĠThe Ġsegment ation Ġmodel Ġin Ġthis Ġtool Ġwas Ġtrained Ġon Ġabout Ġ8 , 000 Ġtagged ĠVietnamese Ġtext Ġsentences Ġbased Ġon ĠCR F Ġmodel Ġand Ġthe Ġmodel Ġextracted Ġover Ġ151 , 000 Ġwords Ġfrom Ġthe Ġtraining Ġcorpus . ĠIn Ġaddition , Ġthis Ġis Ġused Ġin Ġbuilding Ġthe ĠEnglish V iet namese ĠTranslation ĠSystem ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠVietnamese Ġtext Ġclassification ĠB IB REF 28 Ġand Ġbuilding ĠVietnamese Ġcorpus ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ. </s> v n Token izer </s> v n Token izer Ġis Ġimplemented Ġin ĠJava Ġand Ġbundled Ġas ĠEclipse Ġplug - in , Ġand Ġit Ġhas Ġalready Ġbeen Ġintegrated Ġinto Ġv n Tool kit , Ġan ĠEclipse ĠRich ĠClient Ġapplication , Ġwhich Ġis Ġintended Ġto Ġbe Ġa Ġgeneral Ġframework Ġintegrating Ġtools Ġfor Ġprocessing Ġof ĠVietnamese Ġtext . Ġv n Token izer Ġplug - in , Ġv n Tool kit Ġand Ġrelated Ġresources , Ġincluding Ġthe Ġlex icon Ġand Ġtest Ġcorpus Ġare Ġfreely Ġavailable Ġfor Ġdownload . ĠAccording Ġto Ġour Ġobservation , Ġmany Ġresearch Ġcited Ġv n Token izer Ġto Ġuse Ġword Ġsegment ation Ġresults Ġfor Ġapplications Ġas Ġbuilding Ġa Ġlarge ĠVietnamese Ġcorpus ĠB IB REF 31 Ġ, Ġbuilding Ġan ĠEnglish - V iet namese ĠB ilingual ĠCorpus Ġfor ĠMachine ĠTranslation ĠB IB REF 32 Ġ, ĠVietnamese Ġtext Ġclassification ĠB IB REF 33 Ġ, ĠB IB REF 34 Ġ, Ġetc . </s> EV AL U ATION ĠAND ĠRES ULTS </s> This Ġresearch Ġgathers Ġthe Ġresults Ġof ĠVietnamese Ġword Ġsegment ation Ġof Ġseveral Ġmethods Ġinto Ġone Ġtable Ġas Ġshow Ġin ĠTable ĠII . ĠIt Ġis Ġnoted Ġthat Ġthey Ġare Ġnot Ġevaluated Ġon Ġa Ġsame Ġcorpus . ĠThe Ġpurpose Ġof Ġthe Ġresult Ġillustration Ġis Ġto Ġprovide Ġan Ġoverview Ġof Ġthe Ġresults Ġof Ġcurrent ĠVietnamese Ġword Ġsegment ation Ġsystems Ġbased Ġon Ġtheir Ġindividual Ġfeatures . ĠAll Ġstudies Ġmentioned Ġin Ġthe Ġtable Ġhave Ġaccuracy Ġaround Ġ94 - 97 % Ġbased Ġon Ġtheir Ġprovided Ġcorpus . </s> This Ġstudy Ġalso Ġevaluates Ġthe ĠVietnamese Ġword Ġsegment ation Ġbased Ġon Ġexisting Ġtool k its Ġusing Ġthe Ġsame Ġannot ated ĠVietnamese Ġword Ġsegment ation Ġcorpus . ĠThere Ġare Ġtwo Ġavailable Ġtool k its Ġto Ġevaluate Ġand Ġto Ġsegment . ĠTo Ġbe Ġneutral Ġto Ġboth Ġtool k its , Ġwe Ġuse Ġthe ĠEV B News ĠVietnamese Ġcorpus , Ġa Ġpart Ġof ĠEV BC orp us , Ġto Ġevaluate ĠVietnamese Ġword Ġsegment ation . ĠThe ĠEV B News Ġcorpus Ġcontains Ġover Ġ45 , 000 Ġsegment ed ĠVietnamese Ġsentences Ġextracted Ġfrom Ġ1 , 000 Ġgeneral Ġnews Ġarticles Ġ( as Ġshown Ġin ĠTable ĠIII ) ĠB IB REF 16 Ġ. ĠWe Ġused Ġthe Ġsame Ġtraining Ġset Ġwhich Ġhas Ġ1000 Ġfiles Ġand Ġ45 , 5 31 Ġsentences . Ġv n Token izer Ġoutputs Ġ8 31 , 455 ĠVietnamese Ġwords Ġand Ġ1 , 206 , 475 Ġtokens . ĠJ V n Seg ment er Ġoutputs Ġ840 , 387 Ġwords Ġand Ġ1 , 201 , 683 . ĠWe Ġcorrect Ġtags Ġ( BI O ), Ġand Ġcompare Ġto Ġprevious Ġoutputs , Ġwe Ġhave Ġrate Ġfrom Ġv n Token izer Ġis Ġ95 . 6 % Ġand Ġfrom ĠJ V n se gment er Ġis Ġ93 . 4 %. ĠThe Ġresult Ġof Ġboth Ġv n Token izer Ġand ĠJ V n Seg ment er Ġtesting Ġon Ġthe ĠEV B News ĠVietnamese ĠCorpus Ġare Ġprovided Ġin ĠTable ĠIV . </s> CONCLUS IONS ĠAND ĠF UT URE ĠWORK S </s> This Ġstudy Ġreviewed Ġstate - of - the - art Ġapproaches Ġand Ġsystems Ġof ĠVietnamese Ġword Ġsegment ation . ĠThe Ġreview Ġpointed Ġout Ġcommon Ġfeatures Ġand Ġmethods Ġused Ġin ĠVietnamese Ġword Ġsegment ation Ġstudies . ĠThis Ġstudy Ġalso Ġhad Ġan Ġevaluation Ġof Ġthe Ġexisting ĠVietnamese Ġword Ġsegment ation Ġtool k its Ġbased Ġon Ġa Ġsame Ġcorpus Ġto Ġshow Ġadvantages Ġand Ġdisadvantages Ġas Ġto Ġshed Ġsome Ġlights Ġon Ġsystem Ġenhancement . </s> There Ġare Ġseveral Ġchallenges Ġon Ġsupervised Ġlearning Ġapproaches Ġin Ġfuture Ġwork . ĠThe Ġfirst Ġchallenge Ġis Ġto Ġacquire Ġvery Ġlarge ĠVietnamese Ġcorpus Ġand Ġto Ġuse Ġthem Ġin Ġbuilding Ġa Ġclass ifier , Ġwhich Ġcould Ġfurther Ġimprove Ġaccuracy . ĠIn Ġaddition , Ġapplying Ġlingu istics Ġknowledge Ġon Ġword Ġcontext Ġto Ġextract Ġuseful Ġfeatures Ġalso Ġenhances Ġprediction Ġperformance . ĠThe Ġsecond Ġchallenge Ġis Ġdesign Ġand Ġdevelopment Ġof Ġbig Ġdata Ġwarehouse Ġand Ġanalytic Ġframework Ġfor ĠVietnamese Ġdocuments , Ġwhich Ġcorresponds Ġto Ġthe Ġrapid Ġand Ġcontinuous Ġgrowth Ġof Ġgigantic Ġvolume Ġof Ġarticles Ġand / or Ġdocuments Ġfrom ĠWeb Ġ2 . 0 Ġapplications , Ġsuch Ġas , ĠFacebook , ĠTwitter , Ġand Ġso Ġon . ĠIt Ġshould Ġbe Ġaddressed Ġthat Ġthere Ġare Ġmany Ġkinds Ġof ĠVietnamese Ġdocuments , Ġfor Ġexample , ĠHan Ġ- ĠNom Ġdocuments Ġand Ġold Ġand Ġmodern ĠVietnamese Ġdocuments Ġthat Ġare Ġessential Ġand Ġstill Ġneeds Ġfurther Ġanalysis . ĠAccording Ġto Ġour Ġstudy , Ġthere Ġis Ġno Ġa Ġpowerful ĠVietnamese Ġlanguage Ġprocessing Ġused Ġfor Ġprocessing ĠVietnamese Ġbig Ġdata Ġas Ġwell Ġas Ġunderstanding Ġsuch Ġlanguage . ĠThe Ġfinal Ġchallenge Ġrelates Ġto Ġbuilding Ġa Ġsystem , Ġwhich Ġis Ġable Ġto Ġincrement ally Ġlearn Ġnew Ġcorpor a Ġand Ġinteract ively Ġprocess Ġfeedback . ĠIn Ġparticular , Ġit Ġis Ġfeasible Ġto Ġbuild Ġan Ġadvance ĠN LP Ġsystem Ġfor ĠVietnamese Ġbased Ġon ĠH ado op Ġplatform Ġto Ġimprove Ġsystem Ġperformance Ġand Ġto Ġaddress Ġexisting Ġlimitations .