Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish
EXTRACTIVE
Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish , Swedish
EXTRACTIVE
As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1.2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results. We considered UD1.2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish. Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian). Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse.
As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1.2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results. We considered UD1.2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish. Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian). Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse.
1606.03676
Introduction </s> Part - of - speech Ġtagging Ġis Ġnow Ġa Ġclassic Ġtask Ġin Ġnatural Ġlanguage Ġprocessing , Ġfor Ġwhich Ġmany Ġsystems Ġhave Ġbeen Ġdeveloped Ġor Ġadapted Ġfor Ġa Ġlarge Ġvariety Ġof Ġlanguages . ĠIts Ġaim Ġis Ġto Ġassociate Ġeach ĠâĢ ľ word âĢ Ŀ Ġwith Ġa Ġmorph os ynt actic Ġtag , Ġwhose Ġgran ularity Ġcan Ġrange Ġfrom Ġa Ġsimple Ġmorph os ynt actic Ġcategory , Ġor Ġpart - of - speech Ġ( here after ĠPo S ), Ġto Ġfiner Ġcategories Ġenriched Ġwith Ġmorph ological Ġfeatures Ġ( gender , Ġnumber , Ġcase , Ġtense , Ġmood , Ġetc .). </s> The Ġuse Ġof Ġmachine Ġlearning Ġalgorithms Ġtrained Ġon Ġmanually Ġannot ated Ġcorpor a Ġhas Ġlong Ġbecome Ġthe Ġstandard Ġway Ġto Ġdevelop ĠPo S Ġtag gers . ĠA Ġlarge Ġvariety Ġof Ġalgorithms Ġhave Ġbeen Ġused , Ġsuch Ġas Ġ( in Ġapprox im ative Ġchronological Ġorder ) Ġbig ram Ġand Ġtrig ram Ġhidden ĠMark ov Ġmodels ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġdecision Ġtrees ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġmaximum Ġentropy ĠMark ov Ġmodels Ġ( M EM Ms ) ĠB IB REF 5 Ġand ĠCond itional ĠRandom ĠFields Ġ( CR Fs ) ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠWith Ġsuch Ġmachine Ġlearning Ġalgorithms , Ġit Ġis Ġpossible Ġto Ġbuild ĠPo S Ġtag gers Ġfor Ġany Ġlanguage , Ġprovided Ġadequate Ġtraining Ġdata Ġis Ġavailable . </s> As Ġa Ġcomplement Ġto Ġannot ated Ġcorpor a , Ġit Ġhas Ġpreviously Ġbeen Ġshown Ġthat Ġexternal Ġlex icons Ġare Ġvaluable Ġsources Ġof Ġinformation , Ġin Ġparticular Ġmorph os ynt actic Ġlex icons , Ġwhich Ġprovide Ġa Ġlarge Ġinventory Ġof Ġ( word , ĠPo S ) Ġpairs . ĠSuch Ġlex ical Ġinformation Ġcan Ġbe Ġused Ġin Ġthe Ġform Ġof Ġconstraints Ġat Ġtagging Ġtime ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġor Ġduring Ġthe Ġtraining Ġprocess Ġas Ġadditional Ġfeatures Ġcombined Ġwith Ġstandard Ġfeatures Ġextracted Ġfrom Ġthe Ġtraining Ġcorpus ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. </s> In Ġrecent Ġyears , Ġa Ġdifferent Ġapproach Ġto Ġmodelling Ġlex ical Ġinformation Ġand Ġintegrating Ġit Ġinto Ġnatural Ġlanguage Ġprocessing Ġsystems Ġhas Ġemerged , Ġnamely Ġthe Ġuse Ġof Ġvector Ġrepresentations Ġfor Ġwords Ġor Ġword Ġsequences ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠSuch Ġrepresentations , Ġwhich Ġare Ġgenerally Ġextracted Ġfrom Ġlarge Ġamounts Ġof Ġraw Ġtext , Ġhave Ġproved Ġvery Ġuseful Ġfor Ġnumerous Ġtasks Ġincluding ĠPo S Ġtagging , Ġin Ġparticular Ġwhen Ġused Ġin Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ) Ġand Ġmore Ġspecifically Ġin Ġmono - Ġor Ġbi - direction al , Ġword - level Ġand / or Ġcharacter - level Ġlong Ġshort - term Ġmemory Ġnetworks Ġ( L ST Ms ) ĠB IB REF 19 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 20 Ġ. </s> Both Ġapproaches Ġto Ġrepresenting Ġlex ical Ġproperties Ġand Ġto Ġintegrating Ġthem Ġinto Ġa ĠPo S Ġtag ger Ġimprove Ġtagging Ġresults . ĠYet Ġthey Ġrely Ġon Ġresources Ġof Ġdifferent Ġn atures . ĠThe Ġmain Ġadvantage Ġof Ġword Ġvectors Ġis Ġthat Ġthey Ġare Ġbuilt Ġin Ġan Ġun super vised Ġway , Ġonly Ġrequiring Ġlarge Ġamounts Ġof Ġraw Ġtextual Ġdata . ĠThey Ġalso Ġencode Ġfiner - gr ained Ġinformation Ġthan Ġusual Ġmorph os ynt actic Ġlex icons , Ġmost Ġof Ġwhich Ġdo Ġnot Ġinclude Ġany Ġquantitative Ġdata , Ġnot Ġeven Ġsimple Ġfrequency Ġinformation . ĠConversely , Ġlex ical Ġresources Ġoften Ġprovide Ġinformation Ġabout Ġscarcely Ġatt ested Ġwords , Ġfor Ġwhich Ġcorpus - based Ġapproaches Ġsuch Ġas Ġword Ġvector Ġrepresentations Ġare Ġof Ġlimited Ġrelevance . ĠMoreover , Ġmorph ological Ġor Ġmorph os ynt actic Ġlex icons Ġalready Ġexist Ġfor Ġa Ġnumber Ġof Ġlanguages , Ġincluding Ġless - res ourced Ġlang au ges Ġfor Ġwhich Ġit Ġmight Ġbe Ġdifficult Ġto Ġobtain Ġthe Ġlarge Ġamounts Ġof Ġraw Ġdata Ġnecessary Ġto Ġextract Ġword Ġvector Ġrepresentations . </s> Our Ġmain Ġgoal Ġis Ġtherefore Ġto Ġcompare Ġthe Ġrespective Ġimpact Ġof Ġexternal Ġlex icons Ġand Ġword Ġvector Ġrepresentations Ġon Ġthe Ġaccuracy Ġof ĠPo S Ġmodels . ĠThis Ġquestion Ġhas Ġalready Ġbeen Ġinvestigated Ġfor Ġ6 Ġlanguages Ġby ĠB IB REF 18 Ġusing Ġthe Ġstate - of - the - art ĠCR F - based Ġtagging Ġsystem ĠMar Mo T . ĠThe Ġauthors Ġfound Ġthat Ġtheir Ġbest - performing Ġword - vector - based ĠPo S Ġtagging Ġmodels Ġoutper form Ġtheir Ġmodels Ġthat Ġrely Ġon Ġmorph os ynt actic Ġresources Ġ( lex icons Ġor Ġmorph ological Ġanalys ers ). ĠIn Ġthis Ġpaper , Ġwe Ġreport Ġon Ġlarger Ġcomparison , Ġcarried Ġout Ġin Ġa Ġlarger Ġmult ilingual Ġsetting Ġand Ġcomparing Ġdifferent Ġtagging Ġmodels . ĠUsing Ġdifferent Ġ16 Ġdatasets , Ġwe Ġcompare Ġthe Ġperformances Ġof Ġtwo Ġfeature - based Ġmodels Ġenriched Ġwith Ġexternal Ġlex icons Ġand Ġof Ġtwo ĠL ST M - based Ġmodels Ġenriched Ġwith Ġword Ġvector Ġrepresentations . ĠA Ġsecondary Ġgoal Ġof Ġour Ġwork Ġis Ġto Ġcompare Ġthe Ġrelative Ġimprovements Ġlinked Ġto Ġthe Ġuse Ġof Ġexternal Ġlex ical Ġinformation Ġin Ġthe Ġtwo Ġfeature - based Ġmodels , Ġwhich Ġuse Ġdifferent Ġmodels Ġ( M EM M Ġvs . ĠCR F ) Ġand Ġfeature Ġsets . </s> More Ġspecifically , Ġour Ġstarting Ġpoint Ġis Ġthe ĠME lt Ġsystem ĠB IB REF 12 Ġ, Ġan ĠMEM M Ġtagging Ġsystem . ĠWe Ġfirst Ġbriefly Ġdescribe Ġthis Ġsystem Ġand Ġthe Ġway Ġwe Ġadapted Ġit Ġby Ġintegrating Ġour Ġown Ġset Ġof Ġcorpus - based Ġand Ġlex ical Ġfeatures . ĠWe Ġthen Ġintroduce Ġthe Ġtagging Ġmodels Ġwe Ġhave Ġtrained Ġfor Ġ16 Ġdifferent Ġlanguages Ġusing Ġour Ġadapted Ġversion Ġof ĠME lt . ĠThese Ġmodels Ġare Ġtrained Ġon Ġthe ĠUniversal ĠDepend encies Ġ( v 1 . 2 ) Ġcorpus Ġset ĠB IB REF 21 Ġ, Ġcomple mented Ġby Ġmorph os ynt actic Ġlex icons . ĠWe Ġcompare Ġthe Ġaccuracy Ġof Ġour Ġmodels Ġwith Ġthe Ġscores Ġobtained Ġby Ġthe ĠCR F - based Ġsystem ĠMar Mo T ĠB IB REF 22 Ġ, ĠB IB REF 18 Ġ, Ġret rained Ġon Ġthe Ġsame Ġcorpor a Ġand Ġthe Ġsame Ġexternal Ġmorph os ynt actic Ġlex icons . ĠWe Ġalso Ġcompare Ġour Ġresults Ġto Ġthose Ġobtained Ġby Ġthe Ġbest Ġbid irection al ĠL ST M Ġmodels Ġdescribed Ġby ĠB IB REF 20 Ġ, Ġwhich Ġboth Ġmake Ġuse Ġof ĠPoly gl ot Ġword Ġvector Ġrepresentations Ġpublished Ġby ĠB IB REF 23 Ġ. ĠWe Ġwill Ġshow Ġthat Ġan Ġoptim ised Ġenrichment Ġof Ġfeature - based Ġmodels Ġwith Ġmorph os ynt actic Ġlex icon Ġresults Ġin Ġsignificant Ġaccuracy Ġgains . ĠThe Ġmacro - aver aged Ġaccuracy Ġof Ġour Ġenriched ĠME lt Ġmodels Ġis Ġabove Ġthat Ġof Ġenriched ĠMar Mo T Ġmodels Ġand Ġvirtually Ġidentical Ġto Ġthat Ġof ĠL ST Ms Ġenriched Ġwith Ġword Ġvector Ġrepresentations . ĠMore Ġprecisely , Ġper - language Ġresults Ġindicate Ġthat Ġlex icons Ġprovide Ġmore Ġuseful Ġinformation Ġfor Ġlanguages Ġwith Ġa Ġhigh Ġlex ical Ġvariability Ġ( such Ġas Ġmorph ologically Ġrich Ġlanguages ), Ġwhereas Ġword Ġvectors Ġare Ġmore Ġinformative Ġfor Ġlanguages Ġwith Ġa Ġlower Ġlex ical Ġvariability Ġ( such Ġas ĠEnglish ). </s> ME lt </s> ME lt ĠB IB REF 12 Ġis Ġa Ġtagging Ġsystem Ġbased Ġon Ġmaximum Ġentropy ĠMark ov Ġmodels Ġ( M EM M ) ĠB IB REF 5 Ġ, Ġa Ġclass Ġof Ġdiscrim in ative Ġmodels Ġthat Ġare Ġsuitable Ġfor Ġsequence Ġlab elling ĠB IB REF 5 Ġ. ĠThe Ġbasic Ġset Ġof Ġfeatures Ġused Ġby ĠME lt Ġis Ġgiven Ġin ĠB IB REF 12 Ġ. ĠIt Ġis Ġa Ġsupers et Ġof Ġthe Ġfeature Ġsets Ġused Ġby ĠB IB REF 5 Ġand ĠB IB REF 24 Ġand Ġincludes Ġboth Ġlocal Ġstandard Ġfeatures Ġ( for Ġexample Ġthe Ġcurrent Ġword Ġitself Ġand Ġits Ġprefix es Ġand Ġsuffix es Ġof Ġlength Ġ1 Ġto Ġ4 ) Ġand Ġcontextual Ġstandard Ġfeatures Ġ( for Ġexample Ġthe Ġtag Ġjust Ġassigned Ġto Ġthe Ġpreceding Ġword ). ĠIn Ġparticular , Ġwith Ġrespect Ġto ĠRat n ap ark hi 's Ġfeature Ġset , ĠME lt 's Ġbasic Ġfeature Ġset Ġlifts Ġthe Ġrestriction Ġthat Ġlocal Ġstandard Ġfeatures Ġused Ġto Ġanalyse Ġthe Ġinternal Ġcomposition Ġof Ġthe Ġcurrent Ġword Ġshould Ġonly Ġapply Ġto Ġrare Ġwords . </s> One Ġof Ġthe Ġadvantages Ġof Ġfeature - based Ġmodels Ġsuch Ġas ĠMEM Ms Ġand ĠCR Fs Ġis Ġthat Ġcomplementary Ġinformation Ġcan Ġbe Ġeasily Ġadded Ġin Ġthe Ġform Ġof Ġadditional Ġfeatures . ĠThis Ġwas Ġinvestigated Ġfor Ġinstance Ġby ĠB IB REF 25 Ġ, Ġwhose Ġbest - performing Ġmodel Ġfor ĠPo S Ġtagging Ġdialog ues Ġwas Ġobtained Ġwith Ġa Ġversion Ġof ĠME lt Ġextended Ġwith Ġdialogue - specific Ġfeatures . ĠYet Ġthe Ġmotivation Ġof ĠME lt 's Ġdevelopers Ġwas Ġfirst Ġand Ġforemost Ġto Ġinvestigate Ġthe Ġbest Ġway Ġto Ġintegrate Ġlex ical Ġinformation Ġextracted Ġfrom Ġlarge - scale Ġmorph os ynt actic Ġlex ical Ġresources Ġinto Ġtheir Ġmodels , Ġon Ġtop Ġof Ġthe Ġtraining Ġdata ĠB IB REF 12 Ġ. ĠThey Ġshowed Ġthat Ġperformances Ġare Ġbetter Ġwhen Ġthis Ġexternal Ġlex ical Ġinformation Ġis Ġintegrated Ġin Ġthe Ġform Ġof Ġadditional Ġlex ical Ġfeatures Ġthan Ġwhen Ġthe Ġexternal Ġlex icon Ġis Ġused Ġas Ġconstraints Ġat Ġtagging Ġtime . ĠThese Ġlex ical Ġfeatures Ġcan Ġalso Ġbe Ġdivided Ġinto Ġlocal Ġlex ical Ġfeatures Ġ( for Ġexample Ġthe Ġlist Ġof Ġpossible Ġtags Ġknown Ġto Ġthe Ġexternal Ġlex icon Ġfor Ġthe Ġcurrent Ġword ) Ġand Ġcontextual Ġlex ical Ġfeatures Ġ( for Ġexample Ġthe Ġlist Ġof Ġpossible Ġtags Ġknown Ġto Ġthe Ġexternal Ġlex icon Ġfor Ġsurrounding Ġwords ). ĠIn Ġparticular , Ġlex ical Ġcontextual Ġfeatures Ġprovide Ġa Ġmeans Ġto Ġmodel Ġthe Ġright Ġcontext Ġof Ġthe Ġcurrent Ġword , Ġmade Ġof Ġwords Ġthat Ġhave Ġnot Ġyet Ġbeen Ġtagged Ġby Ġthe Ġsystem Ġbut Ġfor Ġwhich Ġthe Ġlex icon Ġoften Ġprovides Ġa Ġlist Ġof Ġpossible Ġtags . ĠMoreover , Ġtagging Ġaccuracy Ġfor Ġout - of - voc abulary Ġ( OO V ) Ġwords Ġis Ġimproved , Ġas Ġa Ġresult Ġof Ġthe Ġfact Ġthat Ġwords Ġunknown Ġto Ġthe Ġtraining Ġcorpus Ġmight Ġbe Ġknown Ġto Ġthe Ġexternal Ġlex icon . </s> Despite Ġa Ġfew Ġexperiments Ġpublished Ġwith ĠME lt Ġon Ġlanguages Ġother Ġthan ĠFrench ĠB IB REF 12 Ġ, ĠB IB REF 40 Ġ, ĠB IB REF 41 Ġ, Ġthe Ġoriginal Ġfeature Ġset Ġused Ġby ĠME lt Ġ( standard Ġand Ġlex ical Ġfeatures ) Ġwas Ġdesigned Ġand Ġtested Ġmostly Ġon Ġthis Ġlanguage , Ġby Ġbuilding Ġand Ġevaluating Ġtagging Ġmodels Ġon Ġa Ġvariant Ġof Ġthe ĠFrench ĠTree Bank . ĠSince Ġour Ġgoal Ġwas Ġto Ġcarry Ġout Ġexperiments Ġin Ġa Ġmult ilingual Ġsetting , Ġwe Ġhave Ġdecided Ġto Ġdesign Ġour Ġown Ġset Ġof Ġfeatures , Ġusing Ġthe Ġstandard ĠME lt Ġfeatures Ġas Ġa Ġstarting Ġpoint . ĠWith Ġrespect Ġto Ġthe Ġoriginal ĠME lt Ġfeature Ġset , Ġwe Ġhave Ġadded Ġnew Ġones , Ġsuch Ġas Ġprefix es Ġand Ġsuffix es Ġof Ġthe Ġfollowing Ġword , Ġas Ġwell Ġas Ġa Ġhybrid Ġcontextual Ġfeature Ġobtained Ġby Ġconc aten ating Ġthe Ġtag Ġpredicted Ġfor Ġthe Ġpreceding Ġword Ġand Ġthe Ġtag ( s ) Ġprovided Ġby Ġthe Ġexternal Ġlex icon Ġfor Ġthe Ġfollowing Ġword . </s> In Ġorder Ġto Ġselect Ġthe Ġbest Ġperforming Ġfeature Ġset , Ġwe Ġcarried Ġout Ġa Ġseries Ġof Ġexperiments Ġusing Ġthe Ġmult ilingual Ġdataset Ġprovided Ġduring Ġthe ĠS PM RL Ġparsing Ġshared Ġtask ĠB IB REF 42 Ġ. ĠThis Ġincluded Ġdisc arding Ġuseless Ġor Ġharmful Ġfeatures Ġand Ġselecting Ġthe Ġmaximal Ġlength Ġof Ġthe Ġprefix es Ġand Ġsuffix es Ġto Ġbe Ġused Ġas Ġfeatures , Ġboth Ġfor Ġthe Ġcurrent Ġword Ġand Ġfor Ġthe Ġfollowing Ġword . </s> We Ġincorporated Ġin ĠME lt Ġthe Ġbest Ġperforming Ġfeature Ġset , Ġdescribed Ġin ĠTable ĠT AB REF 1 Ġ. ĠAll Ġmodels Ġdiscussed Ġin Ġthis Ġpaper Ġare Ġbased Ġon Ġthis Ġfeature Ġset . </s> Cor pora </s> We Ġcarried Ġout Ġour Ġexperiments Ġon Ġthe ĠUniversal ĠDepend encies Ġv 1 . 2 Ġtree banks ĠB IB REF 21 Ġ, Ġhereafter ĠUD 1 . 2 , Ġfrom Ġwhich Ġmorph os ynt act ically Ġannot ated Ġcorpor a Ġcan Ġbe Ġtriv ially Ġextracted . ĠAll ĠUD 1 . 2 Ġcorpor a Ġuse Ġa Ġcommon Ġtag Ġset , Ġthe Ġ17 Ġuniversal ĠPo S Ġtags , Ġwhich Ġis Ġan Ġextension Ġof Ġthe Ġtag set Ġproposed Ġby ĠB IB REF 43 Ġ. </s> As Ġour Ġgoal Ġis Ġto Ġstudy Ġthe Ġimpact Ġof Ġlex ical Ġinformation Ġfor ĠPo S Ġtagging , Ġwe Ġhave Ġrestricted Ġour Ġexperiments Ġto ĠUD 1 . 2 Ġcorpor a Ġthat Ġcover Ġlanguages Ġfor Ġwhich Ġwe Ġhave Ġmorph os ynt actic Ġlex icons Ġat Ġour Ġdisposal , Ġand Ġfor Ġwhich ĠB IB REF 20 Ġprovide Ġresults . ĠWe Ġconsidered ĠUD 1 . 2 Ġcorpor a Ġfor Ġthe Ġfollowing Ġ16 Ġlanguages : ĠBulgarian , ĠCroatian , ĠCzech , ĠDanish , ĠEnglish , ĠFrench , ĠGerman , ĠIndonesian , ĠItalian , ĠNorwegian , ĠPersian , ĠPolish , ĠPortuguese , ĠSloven ian , ĠSpanish Ġand ĠSwedish . ĠAlthough Ġthis Ġlanguage Ġlist Ġcontains Ġonly Ġone Ġnon - Ind o - European Ġ( Ind ones ian ), Ġfour Ġmajor ĠIndo - European Ġsub - fam ilies Ġare Ġrepresented Ġ( German ic , ĠRomance , ĠSlav ic , ĠIndo - Iran ian ). ĠOverall , Ġthe Ġ16 Ġlanguages Ġconsidered Ġin Ġour Ġexperiments Ġare Ġtyp ologically , Ġmorph ologically Ġand Ġsynt act ically Ġfairly Ġdiverse . </s> Lex icons </s> We Ġgenerate Ġour Ġexternal Ġlex icons Ġusing Ġthe Ġset Ġof Ġsource Ġlex icons Ġlisted Ġin ĠTable ĠT AB REF 3 Ġ. ĠSince Ġexternal Ġlex ical Ġinformation Ġis Ġexploited Ġvia Ġfeatures , Ġthere Ġis Ġno Ġneed Ġfor Ġthe Ġexternal Ġlex icons Ġand Ġthe Ġannot ated Ġcorpor a Ġto Ġuse Ġthe Ġsame ĠPo S Ġinventory . ĠTherefore , Ġfor Ġeach Ġlanguage , Ġwe Ġsimply Ġextracted Ġfrom Ġthe Ġcorresponding Ġlex icon Ġthe ĠPo S Ġof Ġeach Ġword Ġbased Ġon Ġits Ġmorph ological Ġtags , Ġby Ġremoving Ġall Ġinformation Ġprovided Ġexcept Ġfor Ġits Ġco ars est - level Ġcategory . ĠWe Ġalso Ġadded Ġentries Ġfor Ġpunct uations Ġwhen Ġthe Ġsource Ġlex icons Ġdid Ġnot Ġcontain Ġany . </s> We Ġalso Ġperformed Ġexperiments Ġin Ġwhich Ġwe Ġretained Ġthe Ġfull Ġoriginal Ġtags Ġprovided Ġby Ġthe Ġlex icons , Ġwith Ġall Ġmorph ological Ġfeatures Ġincluded . ĠOn Ġaverage , Ġresults Ġwere Ġslightly Ġbetter Ġthan Ġthose Ġpresented Ġin Ġthe Ġpaper , Ġalthough Ġnot Ġstatistically Ġsignificantly . ĠMoreover , Ġthe Ġgran ularity Ġof Ġtag Ġinvent ories Ġin Ġthe Ġlex icons Ġis Ġdiverse , Ġwhich Ġmakes Ġit Ġdifficult Ġto Ġdraw Ġgeneral Ġconclusions Ġabout Ġresults Ġbased Ġon Ġfull Ġtags . ĠThis Ġis Ġwhy Ġwe Ġonly Ġreport Ġresults Ġbased Ġon Ġ( co arse ) ĠPo S Ġextracted Ġfrom Ġthe Ġoriginal Ġlex icons . </s> Bas eline Ġmodels </s> In Ġorder Ġto Ġassess Ġthe Ġrespective Ġcontributions Ġof Ġexternal Ġlex icons Ġand Ġword Ġvector Ġrepresentations , Ġwe Ġfirst Ġcompared Ġthe Ġresults Ġof Ġthe Ġthree Ġabove - mentioned Ġsystems Ġwhen Ġtrained Ġwithout Ġsuch Ġadditional Ġlex ical Ġinformation . ĠTable ĠT AB REF 11 Ġprovides Ġthe Ġresults Ġof ĠME lt Ġand ĠMar Mo T Ġret rained Ġon ĠUD 1 . 2 Ġcorpor a , Ġtogether Ġwith Ġthe Ġresults Ġpub l ised Ġon Ġthe Ġsame Ġcorpor a Ġby ĠB IB REF 20 Ġ, Ġusing Ġtheir Ġbest Ġmodel Ġnot Ġenhanced Ġby Ġexternal Ġword Ġvector Ġrepresentations ĠâĢĶ i . e . Ġthe Ġmodel Ġthey Ġcall ĠIN LINE FORM 0 Ġ, Ġwhich Ġis Ġa Ġbid irection al ĠL ST M Ġthat Ġcombines Ġboth Ġword Ġand Ġcharacter Ġembed d ings . </s> These Ġresults Ġshow Ġthat ĠPl ank Ġet Ġal .' s Ġ( 2016 ) Ġbi - L ST M Ġperforms Ġextremely Ġwell , Ġsurpassed Ġby ĠMar Mo T Ġon Ġonly Ġ3 Ġout Ġof Ġ16 Ġdatasets Ġ( C zech , ĠFrench Ġand ĠItalian ), Ġand Ġby ĠME lt Ġonly Ġonce Ġ( Ind ones ian ). </s> Mod els Ġenriched Ġwith Ġexternal Ġlex ical Ġinformation </s> Table ĠT AB REF 13 Ġprovides Ġthe Ġresults Ġof Ġfour Ġsystems Ġenriched Ġwith Ġlex ical Ġinformation . ĠThe Ġfeature - based Ġsystems ĠME lt Ġand ĠMar Mo T , Ġrespectively Ġbased Ġon ĠMEM Ms Ġand ĠCR Fs , Ġare Ġextended Ġwith Ġthe Ġlex ical Ġinformation Ġprovided Ġby Ġour Ġmorph os ynt actic Ġlex icons . ĠThis Ġextension Ġtakes Ġthe Ġform Ġof Ġadditional Ġfeatures , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 2 Ġfor ĠME lt . ĠThe Ġresults Ġreported Ġby ĠB IB REF 20 Ġfor Ġtheir Ġbid irection al ĠL ST M Ġwhen Ġinitial ised Ġwith ĠPoly gl ot Ġembed d ings Ġtrained Ġon Ġfull Ġwik iped ias Ġare Ġalso Ġincluded , Ġtogether Ġwith Ġtheir Ġnew Ġsystem ĠFRE QB IN , Ġalso Ġinitial ised Ġwith ĠPoly gl ot Ġembed d ings . ĠFRE QB IN Ġtrains Ġbi - L ST Ms Ġto Ġpredict Ġfor Ġeach Ġinput Ġword Ġboth Ġa ĠPo S Ġand Ġa Ġlabel Ġthat Ġrepresents Ġits Ġlog Ġfrequency Ġin Ġthe Ġtraining Ġdata . ĠAs Ġthey Ġword Ġit , ĠâĢ ľ the Ġidea Ġbehind Ġthis Ġmodel Ġis Ġto Ġmake Ġthe Ġrepresentation Ġpredictive Ġfor Ġfrequency , Ġwhich Ġencourages Ġthe Ġmodel Ġnot Ġto Ġshare Ġrepresentations Ġbetween Ġcommon Ġand Ġrare Ġwords , Ġthus Ġbenefiting Ġthe Ġhandling Ġof Ġrare Ġtokens . âĢ Ŀ </s> The Ġresults , Ġwhich Ġare Ġalso Ġdisplayed Ġin ĠFigures ĠFIG REF 14 Ġand ĠFIG REF 15 Ġ, Ġshow Ġthat Ġall Ġsystems Ġreach Ġvery Ġsimilar Ġresults Ġon Ġaverage , Ġalthough Ġdiscrepancies Ġcan Ġbe Ġobserved Ġfrom Ġone Ġdataset Ġto Ġanother , Ġon Ġwhich Ġwe Ġshall Ġcomment Ġshortly . ĠThe Ġbest Ġperforming Ġsystem Ġin Ġterms Ġof Ġmacro - average Ġis ĠME lt Ġ( 96 . 60 %). ĠBoth Ġbi - L ST M Ġsystems Ġreach Ġthe Ġsame Ġscore Ġ( 96 . 58 %), Ġthe Ġdifference Ġwith ĠME lt 's Ġresults Ġbeing Ġnon Ġsignificant , Ġwhereas ĠMar Mo T Ġis Ġonly Ġ0 . 14 % Ġbehind Ġ( 96 . 46 %). ĠGiven Ġthe Ġbetter Ġbaseline Ġscores Ġof Ġthe Ġneural Ġapproaches , Ġthese Ġresults Ġshow Ġthat Ġthe Ġbenefit Ġof Ġusing Ġexternal Ġlex icons Ġin Ġthe Ġfeature - based Ġmodels ĠME lt Ġand ĠMar Mo T Ġare Ġmuch Ġhigher Ġthan Ġthose Ġusing ĠPoly gl ot Ġword Ġvector Ġrepresentations Ġas Ġinitial isations Ġfor Ġbi - L ST Ms . </s> Yet Ġthese Ġvery Ġsimilar Ġoverall Ġresults Ġreflect Ġa Ġdifferent Ġpicture Ġwhen Ġfocusing Ġon ĠO OV Ġtagging Ġaccuracy . ĠThe Ġbest Ġmodels Ġfor ĠO OV Ġtagging Ġaccuracy Ġare , Ġby Ġfar , ĠFRE QB IN Ġmodels , Ġwhich Ġare Ġbeaten Ġby ĠMar Mo T Ġand Ġby ĠME lt Ġonly Ġonce Ġeach Ġ( on ĠEnglish Ġand ĠDanish Ġrespectively ). ĠThe Ġcomparison Ġon ĠO OV Ġtagging Ġbetween ĠME lt Ġand ĠMar Mo T Ġshows Ġthat ĠME lt Ġperforms Ġbetter Ġon Ġaverage Ġthan ĠMar Mo T , Ġdespite Ġthe Ġfact Ġthat ĠMar Mo T 's Ġbaseline Ġresults Ġwere Ġbetter Ġthan Ġthose Ġreached Ġby ĠME lt . ĠThis Ġshows Ġthat Ġthe Ġinformation Ġprovided Ġby Ġexternal Ġmorph os ynt actic Ġlex icons Ġis Ġbetter Ġexploited Ġby ĠME lt 's Ġlex ical Ġfeatures Ġthan Ġby Ġthose Ġused Ġby ĠMar Mo T . ĠOn Ġthe Ġother Ġhand , Ġthe Ġcomparison Ġof Ġboth Ġbi - L ST M - based Ġapproaches Ġconfirm Ġthat Ġthe ĠFRE QB IN Ġmodels Ġis Ġbetter Ġby Ġover Ġ10 % Ġabsolute Ġon ĠO OV Ġtagging Ġaccuracy Ġ( 94 . 28 % Ġvs . Ġ83 . 59 %), Ġwith Ġ65 % Ġlower Ġerror Ġrate . </s> One Ġof Ġthe Ġimportant Ġdifferences Ġbetween Ġthe Ġlex ical Ġinformation Ġprovided Ġby Ġan Ġexternal Ġlex icon Ġand Ġword Ġvectors Ġbuilt Ġfrom Ġraw Ġcorpor a , Ġapart Ġfrom Ġthe Ġvery Ġnature Ġof Ġthe Ġlex ical Ġinformation Ġprovided , Ġis Ġthe Ġcoverage Ġand Ġaccuracy Ġof Ġthis Ġlex ical Ġinformation Ġon Ġrare Ġwords . ĠAll Ġwords Ġin Ġa Ġmorph os ynt actic Ġlex icon Ġare Ġassociated Ġwith Ġinformation Ġof Ġa Ġsame Ġgran ularity Ġand Ġquality , Ġwhich Ġis Ġnot Ġthe Ġcase Ġwith Ġword Ġrepresentations Ġsuch Ġas Ġprovided Ġby ĠPoly gl ot . ĠModels Ġthat Ġtake Ġadvantage Ġof Ġexternal Ġlex icons Ġshould Ġtherefore Ġperform Ġcomparatively Ġbetter Ġon Ġdatasets Ġcontaining Ġa Ġhigher Ġproportion Ġof Ġrare r Ġwords , Ġprovided Ġthe Ġlex icons ' Ġcoverage Ġis Ġhigh . ĠIn Ġorder Ġto Ġconfirm Ġthis Ġintuition , Ġwe Ġhave Ġused Ġa Ġlex ical Ġrichness Ġmetric Ġbased Ġon Ġthe Ġtype / token Ġratio . ĠSince Ġthis Ġratio Ġis Ġwell - known Ġfor Ġbeing Ġsensitive Ġto Ġcorpus Ġlength , Ġwe Ġnormal ised Ġit Ġby Ġcomputing Ġit Ġover Ġthe Ġ60 , 000 Ġfirst Ġtokens Ġof Ġeach Ġtraining Ġset . ĠWhen Ġthis Ġnormal ised Ġtype / token Ġratio Ġis Ġplotted Ġagainst Ġthe Ġdifference Ġbetween Ġthe Ġresults Ġof ĠME lt Ġand Ġboth Ġbi - L ST M - based Ġmodels , Ġthe Ġexpected Ġcorrelation Ġis Ġclearly Ġvisible Ġ( see ĠFigure ĠFIG REF 16 Ġ). ĠThis Ġexplains Ġwhy ĠME lt Ġob tains Ġbetter Ġresults Ġon Ġthe Ġmorph ologically Ġricher ĠSlav ic Ġdatasets Ġ( average Ġnormal ised Ġtype / token Ġratio : Ġ0 . 28 , Ġaverage Ġaccuracy Ġdifference : Ġ0 . 32 Ġcompared Ġto Ġboth Ġbi - L ST M + Poly gl ot Ġand ĠFRE QB IN + Poly gl ot ) Ġand , Ġat Ġthe Ġother Ġend Ġof Ġthe Ġspectrum , Ġsignificantly Ġworse Ġresults Ġon Ġthe ĠEnglish Ġdataset Ġ( normal ised Ġtype / token Ġratio : Ġ0 . 15 , Ġaverage Ġaccuracy Ġdifference : Ġ- 0 . 56 Ġcompared Ġto Ġbi - L ST M + Poly gl ot , Ġ- 0 . 57 Ġcompared Ġto ĠFRE QB IN + Poly gl ot ). </s> Conclusion </s> Two Ġmain Ġconclusions Ġcan Ġbe Ġdrawn Ġfrom Ġour Ġcomparative Ġresults . ĠFirst , Ġfeature - based Ġtagging Ġmodels Ġadequately Ġenriched Ġwith Ġexternal Ġmorph os ynt actic Ġlex icons Ġperform , Ġon Ġaverage , Ġas Ġwell Ġas Ġbi - L ST Ms Ġenriched Ġwith Ġword Ġembed d ings . ĠPer - language Ġresults Ġshow Ġthat Ġthe Ġbest Ġaccuracy Ġlevels Ġare Ġreached Ġby Ġfeature - based Ġmodels , Ġand Ġin Ġparticular Ġby Ġour Ġimproved Ġversion Ġof Ġthe ĠMEM M - based Ġsystem ĠME lt , Ġon Ġdatasets Ġwith Ġhigh Ġlex ical Ġvariability Ġ( in Ġshort , Ġfor Ġmorph ologically Ġrich Ġlanguages ), Ġwhereas Ġneural - based Ġresults Ġperform Ġbetter Ġon Ġdat ats ets Ġwith Ġlower Ġlex ical Ġvariability Ġ( e . g . Ġfor ĠEnglish ). </s> We Ġhave Ġonly Ġcompared Ġthe Ġcontribution Ġof Ġmorph os ynt actic Ġlex icons Ġto Ġfeature - based Ġmodels Ġ( M EM Ms , ĠCR Fs ) Ġand Ġthat Ġof Ġword Ġvector Ġrepresentations Ġto Ġbi - L ST M - based Ġmodels Ġas Ġreported Ġby ĠB IB REF 20 Ġ. ĠAs Ġmentioned Ġabove , Ġwork Ġon Ġthe Ġcontribution Ġof Ġword Ġvector Ġrepresentations Ġto Ġfeature - based Ġapproaches Ġhas Ġbeen Ġcarried Ġout Ġby ĠB IB REF 18 Ġ. ĠHowever , Ġthe Ġexploitation Ġof Ġexisting Ġmorph os ynt actic Ġor Ġmorph ological Ġlex icons Ġin Ġneural Ġmodels Ġis Ġa Ġless Ġstudied Ġquestion . ĠImprovements Ġover Ġthe Ġstate Ġof Ġthe Ġart Ġmight Ġbe Ġachieved Ġby Ġintegrating Ġlex ical Ġinformation Ġboth Ġfrom Ġan Ġexternal Ġlex icon Ġand Ġfrom Ġword Ġvector Ġrepresentations Ġinto Ġtagging Ġmodels . </s> In Ġthat Ġregard , Ġfurther Ġwork Ġwill Ġbe Ġrequired Ġto Ġunderstand Ġwhich Ġclass Ġof Ġmodels Ġperform Ġthe Ġbest . ĠAn Ġoption Ġwould Ġbe Ġto Ġintegrate Ġfeature - based Ġmodels Ġsuch Ġas Ġa ĠCR F Ġwith Ġan ĠL ST M - based Ġlayer , Ġfollowing Ġrecent Ġproposals Ġsuch Ġas Ġthe Ġone Ġproposed Ġby ĠB IB REF 45 Ġfor Ġnamed Ġentity Ġrecognition .
Introduction Part-of-speech tagging is now a classic task in natural language processing, for which many systems have been developed or adapted for a large variety of languages. Its aim is to associate each “word” with a morphosyntactic tag, whose granularity can range from a simple morphosyntactic category, or part-of-speech (hereafter PoS), to finer categories enriched with morphological features (gender, number, case, tense, mood, etc.). The use of machine learning algorithms trained on manually annotated corpora has long become the standard way to develop PoS taggers. A large variety of algorithms have been used, such as (in approximative chronological order) bigram and trigram hidden Markov models BIBREF0 , BIBREF1 , BIBREF2 , decision trees BIBREF3 , BIBREF4 , maximum entropy Markov models (MEMMs) BIBREF5 and Conditional Random Fields (CRFs) BIBREF6 , BIBREF7 . With such machine learning algorithms, it is possible to build PoS taggers for any language, provided adequate training data is available. As a complement to annotated corpora, it has previously been shown that external lexicons are valuable sources of information, in particular morphosyntactic lexicons, which provide a large inventory of (word, PoS) pairs. Such lexical information can be used in the form of constraints at tagging time BIBREF8 , BIBREF9 or during the training process as additional features combined with standard features extracted from the training corpus BIBREF10 , BIBREF11 , BIBREF12 . In recent years, a different approach to modelling lexical information and integrating it into natural language processing systems has emerged, namely the use of vector representations for words or word sequences BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 . Such representations, which are generally extracted from large amounts of raw text, have proved very useful for numerous tasks including PoS tagging, in particular when used in recurrent neural networks (RNNs) and more specifically in mono- or bi-directional, word-level and/or character-level long short-term memory networks (LSTMs) BIBREF19 , BIBREF16 , BIBREF17 , BIBREF20 . Both approaches to representing lexical properties and to integrating them into a PoS tagger improve tagging results. Yet they rely on resources of different natures. The main advantage of word vectors is that they are built in an unsupervised way, only requiring large amounts of raw textual data. They also encode finer-grained information than usual morphosyntactic lexicons, most of which do not include any quantitative data, not even simple frequency information. Conversely, lexical resources often provide information about scarcely attested words, for which corpus-based approaches such as word vector representations are of limited relevance. Moreover, morphological or morphosyntactic lexicons already exist for a number of languages, including less-resourced langauges for which it might be difficult to obtain the large amounts of raw data necessary to extract word vector representations. Our main goal is therefore to compare the respective impact of external lexicons and word vector representations on the accuracy of PoS models. This question has already been investigated for 6 languages by BIBREF18 using the state-of-the-art CRF-based tagging system MarMoT. The authors found that their best-performing word-vector-based PoS tagging models outperform their models that rely on morphosyntactic resources (lexicons or morphological analysers). In this paper, we report on larger comparison, carried out in a larger multilingual setting and comparing different tagging models. Using different 16 datasets, we compare the performances of two feature-based models enriched with external lexicons and of two LSTM-based models enriched with word vector representations. A secondary goal of our work is to compare the relative improvements linked to the use of external lexical information in the two feature-based models, which use different models (MEMM vs. CRF) and feature sets. More specifically, our starting point is the MElt system BIBREF12 , an MEMM tagging system. We first briefly describe this system and the way we adapted it by integrating our own set of corpus-based and lexical features. We then introduce the tagging models we have trained for 16 different languages using our adapted version of MElt. These models are trained on the Universal Dependencies (v1.2) corpus set BIBREF21 , complemented by morphosyntactic lexicons. We compare the accuracy of our models with the scores obtained by the CRF-based system MarMoT BIBREF22 , BIBREF18 , retrained on the same corpora and the same external morphosyntactic lexicons. We also compare our results to those obtained by the best bidirectional LSTM models described by BIBREF20 , which both make use of Polyglot word vector representations published by BIBREF23 . We will show that an optimised enrichment of feature-based models with morphosyntactic lexicon results in significant accuracy gains. The macro-averaged accuracy of our enriched MElt models is above that of enriched MarMoT models and virtually identical to that of LSTMs enriched with word vector representations. More precisely, per-language results indicate that lexicons provide more useful information for languages with a high lexical variability (such as morphologically rich languages), whereas word vectors are more informative for languages with a lower lexical variability (such as English). MElt MElt BIBREF12 is a tagging system based on maximum entropy Markov models (MEMM) BIBREF5 , a class of discriminative models that are suitable for sequence labelling BIBREF5 . The basic set of features used by MElt is given in BIBREF12 . It is a superset of the feature sets used by BIBREF5 and BIBREF24 and includes both local standard features (for example the current word itself and its prefixes and suffixes of length 1 to 4) and contextual standard features (for example the tag just assigned to the preceding word). In particular, with respect to Ratnaparkhi's feature set, MElt's basic feature set lifts the restriction that local standard features used to analyse the internal composition of the current word should only apply to rare words. One of the advantages of feature-based models such as MEMMs and CRFs is that complementary information can be easily added in the form of additional features. This was investigated for instance by BIBREF25 , whose best-performing model for PoS tagging dialogues was obtained with a version of MElt extended with dialogue-specific features. Yet the motivation of MElt's developers was first and foremost to investigate the best way to integrate lexical information extracted from large-scale morphosyntactic lexical resources into their models, on top of the training data BIBREF12 . They showed that performances are better when this external lexical information is integrated in the form of additional lexical features than when the external lexicon is used as constraints at tagging time. These lexical features can also be divided into local lexical features (for example the list of possible tags known to the external lexicon for the current word) and contextual lexical features (for example the list of possible tags known to the external lexicon for surrounding words). In particular, lexical contextual features provide a means to model the right context of the current word, made of words that have not yet been tagged by the system but for which the lexicon often provides a list of possible tags. Moreover, tagging accuracy for out-of-vocabulary (OOV) words is improved, as a result of the fact that words unknown to the training corpus might be known to the external lexicon. Despite a few experiments published with MElt on languages other than French BIBREF12 , BIBREF40 , BIBREF41 , the original feature set used by MElt (standard and lexical features) was designed and tested mostly on this language, by building and evaluating tagging models on a variant of the French TreeBank. Since our goal was to carry out experiments in a multilingual setting, we have decided to design our own set of features, using the standard MElt features as a starting point. With respect to the original MElt feature set, we have added new ones, such as prefixes and suffixes of the following word, as well as a hybrid contextual feature obtained by concatenating the tag predicted for the preceding word and the tag(s) provided by the external lexicon for the following word. In order to select the best performing feature set, we carried out a series of experiments using the multilingual dataset provided during the SPMRL parsing shared task BIBREF42 . This included discarding useless or harmful features and selecting the maximal length of the prefixes and suffixes to be used as features, both for the current word and for the following word. We incorporated in MElt the best performing feature set, described in Table TABREF1 . All models discussed in this paper are based on this feature set. Corpora We carried out our experiments on the Universal Dependencies v1.2 treebanks BIBREF21 , hereafter UD1.2, from which morphosyntactically annotated corpora can be trivially extracted. All UD1.2 corpora use a common tag set, the 17 universal PoS tags, which is an extension of the tagset proposed by BIBREF43 . As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1.2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results. We considered UD1.2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish. Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian). Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse. Lexicons We generate our external lexicons using the set of source lexicons listed in Table TABREF3 . Since external lexical information is exploited via features, there is no need for the external lexicons and the annotated corpora to use the same PoS inventory. Therefore, for each language, we simply extracted from the corresponding lexicon the PoS of each word based on its morphological tags, by removing all information provided except for its coarsest-level category. We also added entries for punctuations when the source lexicons did not contain any. We also performed experiments in which we retained the full original tags provided by the lexicons, with all morphological features included. On average, results were slightly better than those presented in the paper, although not statistically significantly. Moreover, the granularity of tag inventories in the lexicons is diverse, which makes it difficult to draw general conclusions about results based on full tags. This is why we only report results based on (coarse) PoS extracted from the original lexicons. Baseline models In order to assess the respective contributions of external lexicons and word vector representations, we first compared the results of the three above-mentioned systems when trained without such additional lexical information. Table TABREF11 provides the results of MElt and MarMoT retrained on UD1.2 corpora, together with the results publised on the same corpora by BIBREF20 , using their best model not enhanced by external word vector representations —i.e. the model they call INLINEFORM0 , which is a bidirectional LSTM that combines both word and character embeddings. These results show that Plank et al.'s (2016) bi-LSTM performs extremely well, surpassed by MarMoT on only 3 out of 16 datasets (Czech, French and Italian), and by MElt only once (Indonesian). Models enriched with external lexical information Table TABREF13 provides the results of four systems enriched with lexical information. The feature-based systems MElt and MarMoT, respectively based on MEMMs and CRFs, are extended with the lexical information provided by our morphosyntactic lexicons. This extension takes the form of additional features, as described in Section SECREF2 for MElt. The results reported by BIBREF20 for their bidirectional LSTM when initialised with Polyglot embeddings trained on full wikipedias are also included, together with their new system FREQBIN, also initialised with Polyglot embeddings. FREQBIN trains bi-LSTMs to predict for each input word both a PoS and a label that represents its log frequency in the training data. As they word it, “the idea behind this model is to make the representation predictive for frequency, which encourages the model not to share representations between common and rare words, thus benefiting the handling of rare tokens.” The results, which are also displayed in Figures FIGREF14 and FIGREF15 , show that all systems reach very similar results on average, although discrepancies can be observed from one dataset to another, on which we shall comment shortly. The best performing system in terms of macro-average is MElt (96.60%). Both bi-LSTM systems reach the same score (96.58%), the difference with MElt's results being non significant, whereas MarMoT is only 0.14% behind (96.46%). Given the better baseline scores of the neural approaches, these results show that the benefit of using external lexicons in the feature-based models MElt and MarMoT are much higher than those using Polyglot word vector representations as initialisations for bi-LSTMs. Yet these very similar overall results reflect a different picture when focusing on OOV tagging accuracy. The best models for OOV tagging accuracy are, by far, FREQBIN models, which are beaten by MarMoT and by MElt only once each (on English and Danish respectively). The comparison on OOV tagging between MElt and MarMoT shows that MElt performs better on average than MarMoT, despite the fact that MarMoT's baseline results were better than those reached by MElt. This shows that the information provided by external morphosyntactic lexicons is better exploited by MElt's lexical features than by those used by MarMoT. On the other hand, the comparison of both bi-LSTM-based approaches confirm that the FREQBIN models is better by over 10% absolute on OOV tagging accuracy (94.28% vs. 83.59%), with 65% lower error rate. One of the important differences between the lexical information provided by an external lexicon and word vectors built from raw corpora, apart from the very nature of the lexical information provided, is the coverage and accuracy of this lexical information on rare words. All words in a morphosyntactic lexicon are associated with information of a same granularity and quality, which is not the case with word representations such as provided by Polyglot. Models that take advantage of external lexicons should therefore perform comparatively better on datasets containing a higher proportion of rarer words, provided the lexicons' coverage is high. In order to confirm this intuition, we have used a lexical richness metric based on the type/token ratio. Since this ratio is well-known for being sensitive to corpus length, we normalised it by computing it over the 60,000 first tokens of each training set. When this normalised type/token ratio is plotted against the difference between the results of MElt and both bi-LSTM-based models, the expected correlation is clearly visible (see Figure FIGREF16 ). This explains why MElt obtains better results on the morphologically richer Slavic datasets (average normalised type/token ratio: 0.28, average accuracy difference: 0.32 compared to both bi-LSTM+Polyglot and FREQBIN+Polyglot) and, at the other end of the spectrum, significantly worse results on the English dataset (normalised type/token ratio: 0.15, average accuracy difference: -0.56 compared to bi-LSTM+Polyglot, -0.57 compared to FREQBIN+Polyglot). Conclusion Two main conclusions can be drawn from our comparative results. First, feature-based tagging models adequately enriched with external morphosyntactic lexicons perform, on average, as well as bi-LSTMs enriched with word embeddings. Per-language results show that the best accuracy levels are reached by feature-based models, and in particular by our improved version of the MEMM-based system MElt, on datasets with high lexical variability (in short, for morphologically rich languages), whereas neural-based results perform better on datatsets with lower lexical variability (e.g. for English). We have only compared the contribution of morphosyntactic lexicons to feature-based models (MEMMs, CRFs) and that of word vector representations to bi-LSTM-based models as reported by BIBREF20 . As mentioned above, work on the contribution of word vector representations to feature-based approaches has been carried out by BIBREF18 . However, the exploitation of existing morphosyntactic or morphological lexicons in neural models is a less studied question. Improvements over the state of the art might be achieved by integrating lexical information both from an external lexicon and from word vector representations into tagging models. In that regard, further work will be required to understand which class of models perform the best. An option would be to integrate feature-based models such as a CRF with an LSTM-based layer, following recent proposals such as the one proposed by BIBREF45 for named entity recognition.
which languages are explored?
0a5ffe4697913a57fda1fd5a188cd5ed59bdc5c7
which Ġlanguages Ġare Ġexplored ?
Bul g arian , ĠCroatian , ĠCzech , ĠDanish , ĠEnglish , ĠFrench , ĠGerman , ĠIndonesian , ĠItalian , ĠNorwegian , ĠPersian , ĠPolish , ĠPortuguese , ĠSloven ian , ĠSpanish Ġand ĠSwedish
<s> which Ġlanguages Ġare Ġexplored ? </s> Introduction </s> Part - of - speech Ġtagging Ġis Ġnow Ġa Ġclassic Ġtask Ġin Ġnatural Ġlanguage Ġprocessing , Ġfor Ġwhich Ġmany Ġsystems Ġhave Ġbeen Ġdeveloped Ġor Ġadapted Ġfor Ġa Ġlarge Ġvariety Ġof Ġlanguages . ĠIts Ġaim Ġis Ġto Ġassociate Ġeach ĠâĢ ľ word âĢ Ŀ Ġwith Ġa Ġmorph os ynt actic Ġtag , Ġwhose Ġgran ularity Ġcan Ġrange Ġfrom Ġa Ġsimple Ġmorph os ynt actic Ġcategory , Ġor Ġpart - of - speech Ġ( here after ĠPo S ), Ġto Ġfiner Ġcategories Ġenriched Ġwith Ġmorph ological Ġfeatures Ġ( gender , Ġnumber , Ġcase , Ġtense , Ġmood , Ġetc .). </s> The Ġuse Ġof Ġmachine Ġlearning Ġalgorithms Ġtrained Ġon Ġmanually Ġannot ated Ġcorpor a Ġhas Ġlong Ġbecome Ġthe Ġstandard Ġway Ġto Ġdevelop ĠPo S Ġtag gers . ĠA Ġlarge Ġvariety Ġof Ġalgorithms Ġhave Ġbeen Ġused , Ġsuch Ġas Ġ( in Ġapprox im ative Ġchronological Ġorder ) Ġbig ram Ġand Ġtrig ram Ġhidden ĠMark ov Ġmodels ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, Ġdecision Ġtrees ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, Ġmaximum Ġentropy ĠMark ov Ġmodels Ġ( M EM Ms ) ĠB IB REF 5 Ġand ĠCond itional ĠRandom ĠFields Ġ( CR Fs ) ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠWith Ġsuch Ġmachine Ġlearning Ġalgorithms , Ġit Ġis Ġpossible Ġto Ġbuild ĠPo S Ġtag gers Ġfor Ġany Ġlanguage , Ġprovided Ġadequate Ġtraining Ġdata Ġis Ġavailable . </s> As Ġa Ġcomplement Ġto Ġannot ated Ġcorpor a , Ġit Ġhas Ġpreviously Ġbeen Ġshown Ġthat Ġexternal Ġlex icons Ġare Ġvaluable Ġsources Ġof Ġinformation , Ġin Ġparticular Ġmorph os ynt actic Ġlex icons , Ġwhich Ġprovide Ġa Ġlarge Ġinventory Ġof Ġ( word , ĠPo S ) Ġpairs . ĠSuch Ġlex ical Ġinformation Ġcan Ġbe Ġused Ġin Ġthe Ġform Ġof Ġconstraints Ġat Ġtagging Ġtime ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġor Ġduring Ġthe Ġtraining Ġprocess Ġas Ġadditional Ġfeatures Ġcombined Ġwith Ġstandard Ġfeatures Ġextracted Ġfrom Ġthe Ġtraining Ġcorpus ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. </s> In Ġrecent Ġyears , Ġa Ġdifferent Ġapproach Ġto Ġmodelling Ġlex ical Ġinformation Ġand Ġintegrating Ġit Ġinto Ġnatural Ġlanguage Ġprocessing Ġsystems Ġhas Ġemerged , Ġnamely Ġthe Ġuse Ġof Ġvector Ġrepresentations Ġfor Ġwords Ġor Ġword Ġsequences ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠSuch Ġrepresentations , Ġwhich Ġare Ġgenerally Ġextracted Ġfrom Ġlarge Ġamounts Ġof Ġraw Ġtext , Ġhave Ġproved Ġvery Ġuseful Ġfor Ġnumerous Ġtasks Ġincluding ĠPo S Ġtagging , Ġin Ġparticular Ġwhen Ġused Ġin Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ) Ġand Ġmore Ġspecifically Ġin Ġmono - Ġor Ġbi - direction al , Ġword - level Ġand / or Ġcharacter - level Ġlong Ġshort - term Ġmemory Ġnetworks Ġ( L ST Ms ) ĠB IB REF 19 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 20 Ġ. </s> Both Ġapproaches Ġto Ġrepresenting Ġlex ical Ġproperties Ġand Ġto Ġintegrating Ġthem Ġinto Ġa ĠPo S Ġtag ger Ġimprove Ġtagging Ġresults . ĠYet Ġthey Ġrely Ġon Ġresources Ġof Ġdifferent Ġn atures . ĠThe Ġmain Ġadvantage Ġof Ġword Ġvectors Ġis Ġthat Ġthey Ġare Ġbuilt Ġin Ġan Ġun super vised Ġway , Ġonly Ġrequiring Ġlarge Ġamounts Ġof Ġraw Ġtextual Ġdata . ĠThey Ġalso Ġencode Ġfiner - gr ained Ġinformation Ġthan Ġusual Ġmorph os ynt actic Ġlex icons , Ġmost Ġof Ġwhich Ġdo Ġnot Ġinclude Ġany Ġquantitative Ġdata , Ġnot Ġeven Ġsimple Ġfrequency Ġinformation . ĠConversely , Ġlex ical Ġresources Ġoften Ġprovide Ġinformation Ġabout Ġscarcely Ġatt ested Ġwords , Ġfor Ġwhich Ġcorpus - based Ġapproaches Ġsuch Ġas Ġword Ġvector Ġrepresentations Ġare Ġof Ġlimited Ġrelevance . ĠMoreover , Ġmorph ological Ġor Ġmorph os ynt actic Ġlex icons Ġalready Ġexist Ġfor Ġa Ġnumber Ġof Ġlanguages , Ġincluding Ġless - res ourced Ġlang au ges Ġfor Ġwhich Ġit Ġmight Ġbe Ġdifficult Ġto Ġobtain Ġthe Ġlarge Ġamounts Ġof Ġraw Ġdata Ġnecessary Ġto Ġextract Ġword Ġvector Ġrepresentations . </s> Our Ġmain Ġgoal Ġis Ġtherefore Ġto Ġcompare Ġthe Ġrespective Ġimpact Ġof Ġexternal Ġlex icons Ġand Ġword Ġvector Ġrepresentations Ġon Ġthe Ġaccuracy Ġof ĠPo S Ġmodels . ĠThis Ġquestion Ġhas Ġalready Ġbeen Ġinvestigated Ġfor Ġ6 Ġlanguages Ġby ĠB IB REF 18 Ġusing Ġthe Ġstate - of - the - art ĠCR F - based Ġtagging Ġsystem ĠMar Mo T . ĠThe Ġauthors Ġfound Ġthat Ġtheir Ġbest - performing Ġword - vector - based ĠPo S Ġtagging Ġmodels Ġoutper form Ġtheir Ġmodels Ġthat Ġrely Ġon Ġmorph os ynt actic Ġresources Ġ( lex icons Ġor Ġmorph ological Ġanalys ers ). ĠIn Ġthis Ġpaper , Ġwe Ġreport Ġon Ġlarger Ġcomparison , Ġcarried Ġout Ġin Ġa Ġlarger Ġmult ilingual Ġsetting Ġand Ġcomparing Ġdifferent Ġtagging Ġmodels . ĠUsing Ġdifferent Ġ16 Ġdatasets , Ġwe Ġcompare Ġthe Ġperformances Ġof Ġtwo Ġfeature - based Ġmodels Ġenriched Ġwith Ġexternal Ġlex icons Ġand Ġof Ġtwo ĠL ST M - based Ġmodels Ġenriched Ġwith Ġword Ġvector Ġrepresentations . ĠA Ġsecondary Ġgoal Ġof Ġour Ġwork Ġis Ġto Ġcompare Ġthe Ġrelative Ġimprovements Ġlinked Ġto Ġthe Ġuse Ġof Ġexternal Ġlex ical Ġinformation Ġin Ġthe Ġtwo Ġfeature - based Ġmodels , Ġwhich Ġuse Ġdifferent Ġmodels Ġ( M EM M Ġvs . ĠCR F ) Ġand Ġfeature Ġsets . </s> More Ġspecifically , Ġour Ġstarting Ġpoint Ġis Ġthe ĠME lt Ġsystem ĠB IB REF 12 Ġ, Ġan ĠMEM M Ġtagging Ġsystem . ĠWe Ġfirst Ġbriefly Ġdescribe Ġthis Ġsystem Ġand Ġthe Ġway Ġwe Ġadapted Ġit Ġby Ġintegrating Ġour Ġown Ġset Ġof Ġcorpus - based Ġand Ġlex ical Ġfeatures . ĠWe Ġthen Ġintroduce Ġthe Ġtagging Ġmodels Ġwe Ġhave Ġtrained Ġfor Ġ16 Ġdifferent Ġlanguages Ġusing Ġour Ġadapted Ġversion Ġof ĠME lt . ĠThese Ġmodels Ġare Ġtrained Ġon Ġthe ĠUniversal ĠDepend encies Ġ( v 1 . 2 ) Ġcorpus Ġset ĠB IB REF 21 Ġ, Ġcomple mented Ġby Ġmorph os ynt actic Ġlex icons . ĠWe Ġcompare Ġthe Ġaccuracy Ġof Ġour Ġmodels Ġwith Ġthe Ġscores Ġobtained Ġby Ġthe ĠCR F - based Ġsystem ĠMar Mo T ĠB IB REF 22 Ġ, ĠB IB REF 18 Ġ, Ġret rained Ġon Ġthe Ġsame Ġcorpor a Ġand Ġthe Ġsame Ġexternal Ġmorph os ynt actic Ġlex icons . ĠWe Ġalso Ġcompare Ġour Ġresults Ġto Ġthose Ġobtained Ġby Ġthe Ġbest Ġbid irection al ĠL ST M Ġmodels Ġdescribed Ġby ĠB IB REF 20 Ġ, Ġwhich Ġboth Ġmake Ġuse Ġof ĠPoly gl ot Ġword Ġvector Ġrepresentations Ġpublished Ġby ĠB IB REF 23 Ġ. ĠWe Ġwill Ġshow Ġthat Ġan Ġoptim ised Ġenrichment Ġof Ġfeature - based Ġmodels Ġwith Ġmorph os ynt actic Ġlex icon Ġresults Ġin Ġsignificant Ġaccuracy Ġgains . ĠThe Ġmacro - aver aged Ġaccuracy Ġof Ġour Ġenriched ĠME lt Ġmodels Ġis Ġabove Ġthat Ġof Ġenriched ĠMar Mo T Ġmodels Ġand Ġvirtually Ġidentical Ġto Ġthat Ġof ĠL ST Ms Ġenriched Ġwith Ġword Ġvector Ġrepresentations . ĠMore Ġprecisely , Ġper - language Ġresults Ġindicate Ġthat Ġlex icons Ġprovide Ġmore Ġuseful Ġinformation Ġfor Ġlanguages Ġwith Ġa Ġhigh Ġlex ical Ġvariability Ġ( such Ġas Ġmorph ologically Ġrich Ġlanguages ), Ġwhereas Ġword Ġvectors Ġare Ġmore Ġinformative Ġfor Ġlanguages Ġwith Ġa Ġlower Ġlex ical Ġvariability Ġ( such Ġas ĠEnglish ). </s> ME lt </s> ME lt ĠB IB REF 12 Ġis Ġa Ġtagging Ġsystem Ġbased Ġon Ġmaximum Ġentropy ĠMark ov Ġmodels Ġ( M EM M ) ĠB IB REF 5 Ġ, Ġa Ġclass Ġof Ġdiscrim in ative Ġmodels Ġthat Ġare Ġsuitable Ġfor Ġsequence Ġlab elling ĠB IB REF 5 Ġ. ĠThe Ġbasic Ġset Ġof Ġfeatures Ġused Ġby ĠME lt Ġis Ġgiven Ġin ĠB IB REF 12 Ġ. ĠIt Ġis Ġa Ġsupers et Ġof Ġthe Ġfeature Ġsets Ġused Ġby ĠB IB REF 5 Ġand ĠB IB REF 24 Ġand Ġincludes Ġboth Ġlocal Ġstandard Ġfeatures Ġ( for Ġexample Ġthe Ġcurrent Ġword Ġitself Ġand Ġits Ġprefix es Ġand Ġsuffix es Ġof Ġlength Ġ1 Ġto Ġ4 ) Ġand Ġcontextual Ġstandard Ġfeatures Ġ( for Ġexample Ġthe Ġtag Ġjust Ġassigned Ġto Ġthe Ġpreceding Ġword ). ĠIn Ġparticular , Ġwith Ġrespect Ġto ĠRat n ap ark hi 's Ġfeature Ġset , ĠME lt 's Ġbasic Ġfeature Ġset Ġlifts Ġthe Ġrestriction Ġthat Ġlocal Ġstandard Ġfeatures Ġused Ġto Ġanalyse Ġthe Ġinternal Ġcomposition Ġof Ġthe Ġcurrent Ġword Ġshould Ġonly Ġapply Ġto Ġrare Ġwords . </s> One Ġof Ġthe Ġadvantages Ġof Ġfeature - based Ġmodels Ġsuch Ġas ĠMEM Ms Ġand ĠCR Fs Ġis Ġthat Ġcomplementary Ġinformation Ġcan Ġbe Ġeasily Ġadded Ġin Ġthe Ġform Ġof Ġadditional Ġfeatures . ĠThis Ġwas Ġinvestigated Ġfor Ġinstance Ġby ĠB IB REF 25 Ġ, Ġwhose Ġbest - performing Ġmodel Ġfor ĠPo S Ġtagging Ġdialog ues Ġwas Ġobtained Ġwith Ġa Ġversion Ġof ĠME lt Ġextended Ġwith Ġdialogue - specific Ġfeatures . ĠYet Ġthe Ġmotivation Ġof ĠME lt 's Ġdevelopers Ġwas Ġfirst Ġand Ġforemost Ġto Ġinvestigate Ġthe Ġbest Ġway Ġto Ġintegrate Ġlex ical Ġinformation Ġextracted Ġfrom Ġlarge - scale Ġmorph os ynt actic Ġlex ical Ġresources Ġinto Ġtheir Ġmodels , Ġon Ġtop Ġof Ġthe Ġtraining Ġdata ĠB IB REF 12 Ġ. ĠThey Ġshowed Ġthat Ġperformances Ġare Ġbetter Ġwhen Ġthis Ġexternal Ġlex ical Ġinformation Ġis Ġintegrated Ġin Ġthe Ġform Ġof Ġadditional Ġlex ical Ġfeatures Ġthan Ġwhen Ġthe Ġexternal Ġlex icon Ġis Ġused Ġas Ġconstraints Ġat Ġtagging Ġtime . ĠThese Ġlex ical Ġfeatures Ġcan Ġalso Ġbe Ġdivided Ġinto Ġlocal Ġlex ical Ġfeatures Ġ( for Ġexample Ġthe Ġlist Ġof Ġpossible Ġtags Ġknown Ġto Ġthe Ġexternal Ġlex icon Ġfor Ġthe Ġcurrent Ġword ) Ġand Ġcontextual Ġlex ical Ġfeatures Ġ( for Ġexample Ġthe Ġlist Ġof Ġpossible Ġtags Ġknown Ġto Ġthe Ġexternal Ġlex icon Ġfor Ġsurrounding Ġwords ). ĠIn Ġparticular , Ġlex ical Ġcontextual Ġfeatures Ġprovide Ġa Ġmeans Ġto Ġmodel Ġthe Ġright Ġcontext Ġof Ġthe Ġcurrent Ġword , Ġmade Ġof Ġwords Ġthat Ġhave Ġnot Ġyet Ġbeen Ġtagged Ġby Ġthe Ġsystem Ġbut Ġfor Ġwhich Ġthe Ġlex icon Ġoften Ġprovides Ġa Ġlist Ġof Ġpossible Ġtags . ĠMoreover , Ġtagging Ġaccuracy Ġfor Ġout - of - voc abulary Ġ( OO V ) Ġwords Ġis Ġimproved , Ġas Ġa Ġresult Ġof Ġthe Ġfact Ġthat Ġwords Ġunknown Ġto Ġthe Ġtraining Ġcorpus Ġmight Ġbe Ġknown Ġto Ġthe Ġexternal Ġlex icon . </s> Despite Ġa Ġfew Ġexperiments Ġpublished Ġwith ĠME lt Ġon Ġlanguages Ġother Ġthan ĠFrench ĠB IB REF 12 Ġ, ĠB IB REF 40 Ġ, ĠB IB REF 41 Ġ, Ġthe Ġoriginal Ġfeature Ġset Ġused Ġby ĠME lt Ġ( standard Ġand Ġlex ical Ġfeatures ) Ġwas Ġdesigned Ġand Ġtested Ġmostly Ġon Ġthis Ġlanguage , Ġby Ġbuilding Ġand Ġevaluating Ġtagging Ġmodels Ġon Ġa Ġvariant Ġof Ġthe ĠFrench ĠTree Bank . ĠSince Ġour Ġgoal Ġwas Ġto Ġcarry Ġout Ġexperiments Ġin Ġa Ġmult ilingual Ġsetting , Ġwe Ġhave Ġdecided Ġto Ġdesign Ġour Ġown Ġset Ġof Ġfeatures , Ġusing Ġthe Ġstandard ĠME lt Ġfeatures Ġas Ġa Ġstarting Ġpoint . ĠWith Ġrespect Ġto Ġthe Ġoriginal ĠME lt Ġfeature Ġset , Ġwe Ġhave Ġadded Ġnew Ġones , Ġsuch Ġas Ġprefix es Ġand Ġsuffix es Ġof Ġthe Ġfollowing Ġword , Ġas Ġwell Ġas Ġa Ġhybrid Ġcontextual Ġfeature Ġobtained Ġby Ġconc aten ating Ġthe Ġtag Ġpredicted Ġfor Ġthe Ġpreceding Ġword Ġand Ġthe Ġtag ( s ) Ġprovided Ġby Ġthe Ġexternal Ġlex icon Ġfor Ġthe Ġfollowing Ġword . </s> In Ġorder Ġto Ġselect Ġthe Ġbest Ġperforming Ġfeature Ġset , Ġwe Ġcarried Ġout Ġa Ġseries Ġof Ġexperiments Ġusing Ġthe Ġmult ilingual Ġdataset Ġprovided Ġduring Ġthe ĠS PM RL Ġparsing Ġshared Ġtask ĠB IB REF 42 Ġ. ĠThis Ġincluded Ġdisc arding Ġuseless Ġor Ġharmful Ġfeatures Ġand Ġselecting Ġthe Ġmaximal Ġlength Ġof Ġthe Ġprefix es Ġand Ġsuffix es Ġto Ġbe Ġused Ġas Ġfeatures , Ġboth Ġfor Ġthe Ġcurrent Ġword Ġand Ġfor Ġthe Ġfollowing Ġword . </s> We Ġincorporated Ġin ĠME lt Ġthe Ġbest Ġperforming Ġfeature Ġset , Ġdescribed Ġin ĠTable ĠT AB REF 1 Ġ. ĠAll Ġmodels Ġdiscussed Ġin Ġthis Ġpaper Ġare Ġbased Ġon Ġthis Ġfeature Ġset . </s> Cor pora </s> We Ġcarried Ġout Ġour Ġexperiments Ġon Ġthe ĠUniversal ĠDepend encies Ġv 1 . 2 Ġtree banks ĠB IB REF 21 Ġ, Ġhereafter ĠUD 1 . 2 , Ġfrom Ġwhich Ġmorph os ynt act ically Ġannot ated Ġcorpor a Ġcan Ġbe Ġtriv ially Ġextracted . ĠAll ĠUD 1 . 2 Ġcorpor a Ġuse Ġa Ġcommon Ġtag Ġset , Ġthe Ġ17 Ġuniversal ĠPo S Ġtags , Ġwhich Ġis Ġan Ġextension Ġof Ġthe Ġtag set Ġproposed Ġby ĠB IB REF 43 Ġ. </s> As Ġour Ġgoal Ġis Ġto Ġstudy Ġthe Ġimpact Ġof Ġlex ical Ġinformation Ġfor ĠPo S Ġtagging , Ġwe Ġhave Ġrestricted Ġour Ġexperiments Ġto ĠUD 1 . 2 Ġcorpor a Ġthat Ġcover Ġlanguages Ġfor Ġwhich Ġwe Ġhave Ġmorph os ynt actic Ġlex icons Ġat Ġour Ġdisposal , Ġand Ġfor Ġwhich ĠB IB REF 20 Ġprovide Ġresults . ĠWe Ġconsidered ĠUD 1 . 2 Ġcorpor a Ġfor Ġthe Ġfollowing Ġ16 Ġlanguages : ĠBulgarian , ĠCroatian , ĠCzech , ĠDanish , ĠEnglish , ĠFrench , ĠGerman , ĠIndonesian , ĠItalian , ĠNorwegian , ĠPersian , ĠPolish , ĠPortuguese , ĠSloven ian , ĠSpanish Ġand ĠSwedish . ĠAlthough Ġthis Ġlanguage Ġlist Ġcontains Ġonly Ġone Ġnon - Ind o - European Ġ( Ind ones ian ), Ġfour Ġmajor ĠIndo - European Ġsub - fam ilies Ġare Ġrepresented Ġ( German ic , ĠRomance , ĠSlav ic , ĠIndo - Iran ian ). ĠOverall , Ġthe Ġ16 Ġlanguages Ġconsidered Ġin Ġour Ġexperiments Ġare Ġtyp ologically , Ġmorph ologically Ġand Ġsynt act ically Ġfairly Ġdiverse . </s> Lex icons </s> We Ġgenerate Ġour Ġexternal Ġlex icons Ġusing Ġthe Ġset Ġof Ġsource Ġlex icons Ġlisted Ġin ĠTable ĠT AB REF 3 Ġ. ĠSince Ġexternal Ġlex ical Ġinformation Ġis Ġexploited Ġvia Ġfeatures , Ġthere Ġis Ġno Ġneed Ġfor Ġthe Ġexternal Ġlex icons Ġand Ġthe Ġannot ated Ġcorpor a Ġto Ġuse Ġthe Ġsame ĠPo S Ġinventory . ĠTherefore , Ġfor Ġeach Ġlanguage , Ġwe Ġsimply Ġextracted Ġfrom Ġthe Ġcorresponding Ġlex icon Ġthe ĠPo S Ġof Ġeach Ġword Ġbased Ġon Ġits Ġmorph ological Ġtags , Ġby Ġremoving Ġall Ġinformation Ġprovided Ġexcept Ġfor Ġits Ġco ars est - level Ġcategory . ĠWe Ġalso Ġadded Ġentries Ġfor Ġpunct uations Ġwhen Ġthe Ġsource Ġlex icons Ġdid Ġnot Ġcontain Ġany . </s> We Ġalso Ġperformed Ġexperiments Ġin Ġwhich Ġwe Ġretained Ġthe Ġfull Ġoriginal Ġtags Ġprovided Ġby Ġthe Ġlex icons , Ġwith Ġall Ġmorph ological Ġfeatures Ġincluded . ĠOn Ġaverage , Ġresults Ġwere Ġslightly Ġbetter Ġthan Ġthose Ġpresented Ġin Ġthe Ġpaper , Ġalthough Ġnot Ġstatistically Ġsignificantly . ĠMoreover , Ġthe Ġgran ularity Ġof Ġtag Ġinvent ories Ġin Ġthe Ġlex icons Ġis Ġdiverse , Ġwhich Ġmakes Ġit Ġdifficult Ġto Ġdraw Ġgeneral Ġconclusions Ġabout Ġresults Ġbased Ġon Ġfull Ġtags . ĠThis Ġis Ġwhy Ġwe Ġonly Ġreport Ġresults Ġbased Ġon Ġ( co arse ) ĠPo S Ġextracted Ġfrom Ġthe Ġoriginal Ġlex icons . </s> Bas eline Ġmodels </s> In Ġorder Ġto Ġassess Ġthe Ġrespective Ġcontributions Ġof Ġexternal Ġlex icons Ġand Ġword Ġvector Ġrepresentations , Ġwe Ġfirst Ġcompared Ġthe Ġresults Ġof Ġthe Ġthree Ġabove - mentioned Ġsystems Ġwhen Ġtrained Ġwithout Ġsuch Ġadditional Ġlex ical Ġinformation . ĠTable ĠT AB REF 11 Ġprovides Ġthe Ġresults Ġof ĠME lt Ġand ĠMar Mo T Ġret rained Ġon ĠUD 1 . 2 Ġcorpor a , Ġtogether Ġwith Ġthe Ġresults Ġpub l ised Ġon Ġthe Ġsame Ġcorpor a Ġby ĠB IB REF 20 Ġ, Ġusing Ġtheir Ġbest Ġmodel Ġnot Ġenhanced Ġby Ġexternal Ġword Ġvector Ġrepresentations ĠâĢĶ i . e . Ġthe Ġmodel Ġthey Ġcall ĠIN LINE FORM 0 Ġ, Ġwhich Ġis Ġa Ġbid irection al ĠL ST M Ġthat Ġcombines Ġboth Ġword Ġand Ġcharacter Ġembed d ings . </s> These Ġresults Ġshow Ġthat ĠPl ank Ġet Ġal .' s Ġ( 2016 ) Ġbi - L ST M Ġperforms Ġextremely Ġwell , Ġsurpassed Ġby ĠMar Mo T Ġon Ġonly Ġ3 Ġout Ġof Ġ16 Ġdatasets Ġ( C zech , ĠFrench Ġand ĠItalian ), Ġand Ġby ĠME lt Ġonly Ġonce Ġ( Ind ones ian ). </s> Mod els Ġenriched Ġwith Ġexternal Ġlex ical Ġinformation </s> Table ĠT AB REF 13 Ġprovides Ġthe Ġresults Ġof Ġfour Ġsystems Ġenriched Ġwith Ġlex ical Ġinformation . ĠThe Ġfeature - based Ġsystems ĠME lt Ġand ĠMar Mo T , Ġrespectively Ġbased Ġon ĠMEM Ms Ġand ĠCR Fs , Ġare Ġextended Ġwith Ġthe Ġlex ical Ġinformation Ġprovided Ġby Ġour Ġmorph os ynt actic Ġlex icons . ĠThis Ġextension Ġtakes Ġthe Ġform Ġof Ġadditional Ġfeatures , Ġas Ġdescribed Ġin ĠSection ĠSEC REF 2 Ġfor ĠME lt . ĠThe Ġresults Ġreported Ġby ĠB IB REF 20 Ġfor Ġtheir Ġbid irection al ĠL ST M Ġwhen Ġinitial ised Ġwith ĠPoly gl ot Ġembed d ings Ġtrained Ġon Ġfull Ġwik iped ias Ġare Ġalso Ġincluded , Ġtogether Ġwith Ġtheir Ġnew Ġsystem ĠFRE QB IN , Ġalso Ġinitial ised Ġwith ĠPoly gl ot Ġembed d ings . ĠFRE QB IN Ġtrains Ġbi - L ST Ms Ġto Ġpredict Ġfor Ġeach Ġinput Ġword Ġboth Ġa ĠPo S Ġand Ġa Ġlabel Ġthat Ġrepresents Ġits Ġlog Ġfrequency Ġin Ġthe Ġtraining Ġdata . ĠAs Ġthey Ġword Ġit , ĠâĢ ľ the Ġidea Ġbehind Ġthis Ġmodel Ġis Ġto Ġmake Ġthe Ġrepresentation Ġpredictive Ġfor Ġfrequency , Ġwhich Ġencourages Ġthe Ġmodel Ġnot Ġto Ġshare Ġrepresentations Ġbetween Ġcommon Ġand Ġrare Ġwords , Ġthus Ġbenefiting Ġthe Ġhandling Ġof Ġrare Ġtokens . âĢ Ŀ </s> The Ġresults , Ġwhich Ġare Ġalso Ġdisplayed Ġin ĠFigures ĠFIG REF 14 Ġand ĠFIG REF 15 Ġ, Ġshow Ġthat Ġall Ġsystems Ġreach Ġvery Ġsimilar Ġresults Ġon Ġaverage , Ġalthough Ġdiscrepancies Ġcan Ġbe Ġobserved Ġfrom Ġone Ġdataset Ġto Ġanother , Ġon Ġwhich Ġwe Ġshall Ġcomment Ġshortly . ĠThe Ġbest Ġperforming Ġsystem Ġin Ġterms Ġof Ġmacro - average Ġis ĠME lt Ġ( 96 . 60 %). ĠBoth Ġbi - L ST M Ġsystems Ġreach Ġthe Ġsame Ġscore Ġ( 96 . 58 %), Ġthe Ġdifference Ġwith ĠME lt 's Ġresults Ġbeing Ġnon Ġsignificant , Ġwhereas ĠMar Mo T Ġis Ġonly Ġ0 . 14 % Ġbehind Ġ( 96 . 46 %). ĠGiven Ġthe Ġbetter Ġbaseline Ġscores Ġof Ġthe Ġneural Ġapproaches , Ġthese Ġresults Ġshow Ġthat Ġthe Ġbenefit Ġof Ġusing Ġexternal Ġlex icons Ġin Ġthe Ġfeature - based Ġmodels ĠME lt Ġand ĠMar Mo T Ġare Ġmuch Ġhigher Ġthan Ġthose Ġusing ĠPoly gl ot Ġword Ġvector Ġrepresentations Ġas Ġinitial isations Ġfor Ġbi - L ST Ms . </s> Yet Ġthese Ġvery Ġsimilar Ġoverall Ġresults Ġreflect Ġa Ġdifferent Ġpicture Ġwhen Ġfocusing Ġon ĠO OV Ġtagging Ġaccuracy . ĠThe Ġbest Ġmodels Ġfor ĠO OV Ġtagging Ġaccuracy Ġare , Ġby Ġfar , ĠFRE QB IN Ġmodels , Ġwhich Ġare Ġbeaten Ġby ĠMar Mo T Ġand Ġby ĠME lt Ġonly Ġonce Ġeach Ġ( on ĠEnglish Ġand ĠDanish Ġrespectively ). ĠThe Ġcomparison Ġon ĠO OV Ġtagging Ġbetween ĠME lt Ġand ĠMar Mo T Ġshows Ġthat ĠME lt Ġperforms Ġbetter Ġon Ġaverage Ġthan ĠMar Mo T , Ġdespite Ġthe Ġfact Ġthat ĠMar Mo T 's Ġbaseline Ġresults Ġwere Ġbetter Ġthan Ġthose Ġreached Ġby ĠME lt . ĠThis Ġshows Ġthat Ġthe Ġinformation Ġprovided Ġby Ġexternal Ġmorph os ynt actic Ġlex icons Ġis Ġbetter Ġexploited Ġby ĠME lt 's Ġlex ical Ġfeatures Ġthan Ġby Ġthose Ġused Ġby ĠMar Mo T . ĠOn Ġthe Ġother Ġhand , Ġthe Ġcomparison Ġof Ġboth Ġbi - L ST M - based Ġapproaches Ġconfirm Ġthat Ġthe ĠFRE QB IN Ġmodels Ġis Ġbetter Ġby Ġover Ġ10 % Ġabsolute Ġon ĠO OV Ġtagging Ġaccuracy Ġ( 94 . 28 % Ġvs . Ġ83 . 59 %), Ġwith Ġ65 % Ġlower Ġerror Ġrate . </s> One Ġof Ġthe Ġimportant Ġdifferences Ġbetween Ġthe Ġlex ical Ġinformation Ġprovided Ġby Ġan Ġexternal Ġlex icon Ġand Ġword Ġvectors Ġbuilt Ġfrom Ġraw Ġcorpor a , Ġapart Ġfrom Ġthe Ġvery Ġnature Ġof Ġthe Ġlex ical Ġinformation Ġprovided , Ġis Ġthe Ġcoverage Ġand Ġaccuracy Ġof Ġthis Ġlex ical Ġinformation Ġon Ġrare Ġwords . ĠAll Ġwords Ġin Ġa Ġmorph os ynt actic Ġlex icon Ġare Ġassociated Ġwith Ġinformation Ġof Ġa Ġsame Ġgran ularity Ġand Ġquality , Ġwhich Ġis Ġnot Ġthe Ġcase Ġwith Ġword Ġrepresentations Ġsuch Ġas Ġprovided Ġby ĠPoly gl ot . ĠModels Ġthat Ġtake Ġadvantage Ġof Ġexternal Ġlex icons Ġshould Ġtherefore Ġperform Ġcomparatively Ġbetter Ġon Ġdatasets Ġcontaining Ġa Ġhigher Ġproportion Ġof Ġrare r Ġwords , Ġprovided Ġthe Ġlex icons ' Ġcoverage Ġis Ġhigh . ĠIn Ġorder Ġto Ġconfirm Ġthis Ġintuition , Ġwe Ġhave Ġused Ġa Ġlex ical Ġrichness Ġmetric Ġbased Ġon Ġthe Ġtype / token Ġratio . ĠSince Ġthis Ġratio Ġis Ġwell - known Ġfor Ġbeing Ġsensitive Ġto Ġcorpus Ġlength , Ġwe Ġnormal ised Ġit Ġby Ġcomputing Ġit Ġover Ġthe Ġ60 , 000 Ġfirst Ġtokens Ġof Ġeach Ġtraining Ġset . ĠWhen Ġthis Ġnormal ised Ġtype / token Ġratio Ġis Ġplotted Ġagainst Ġthe Ġdifference Ġbetween Ġthe Ġresults Ġof ĠME lt Ġand Ġboth Ġbi - L ST M - based Ġmodels , Ġthe Ġexpected Ġcorrelation Ġis Ġclearly Ġvisible Ġ( see ĠFigure ĠFIG REF 16 Ġ). ĠThis Ġexplains Ġwhy ĠME lt Ġob tains Ġbetter Ġresults Ġon Ġthe Ġmorph ologically Ġricher ĠSlav ic Ġdatasets Ġ( average Ġnormal ised Ġtype / token Ġratio : Ġ0 . 28 , Ġaverage Ġaccuracy Ġdifference : Ġ0 . 32 Ġcompared Ġto Ġboth Ġbi - L ST M + Poly gl ot Ġand ĠFRE QB IN + Poly gl ot ) Ġand , Ġat Ġthe Ġother Ġend Ġof Ġthe Ġspectrum , Ġsignificantly Ġworse Ġresults Ġon Ġthe ĠEnglish Ġdataset Ġ( normal ised Ġtype / token Ġratio : Ġ0 . 15 , Ġaverage Ġaccuracy Ġdifference : Ġ- 0 . 56 Ġcompared Ġto Ġbi - L ST M + Poly gl ot , Ġ- 0 . 57 Ġcompared Ġto ĠFRE QB IN + Poly gl ot ). </s> Conclusion </s> Two Ġmain Ġconclusions Ġcan Ġbe Ġdrawn Ġfrom Ġour Ġcomparative Ġresults . ĠFirst , Ġfeature - based Ġtagging Ġmodels Ġadequately Ġenriched Ġwith Ġexternal Ġmorph os ynt actic Ġlex icons Ġperform , Ġon Ġaverage , Ġas Ġwell Ġas Ġbi - L ST Ms Ġenriched Ġwith Ġword Ġembed d ings . ĠPer - language Ġresults Ġshow Ġthat Ġthe Ġbest Ġaccuracy Ġlevels Ġare Ġreached Ġby Ġfeature - based Ġmodels , Ġand Ġin Ġparticular Ġby Ġour Ġimproved Ġversion Ġof Ġthe ĠMEM M - based Ġsystem ĠME lt , Ġon Ġdatasets Ġwith Ġhigh Ġlex ical Ġvariability Ġ( in Ġshort , Ġfor Ġmorph ologically Ġrich Ġlanguages ), Ġwhereas Ġneural - based Ġresults Ġperform Ġbetter Ġon Ġdat ats ets Ġwith Ġlower Ġlex ical Ġvariability Ġ( e . g . Ġfor ĠEnglish ). </s> We Ġhave Ġonly Ġcompared Ġthe Ġcontribution Ġof Ġmorph os ynt actic Ġlex icons Ġto Ġfeature - based Ġmodels Ġ( M EM Ms , ĠCR Fs ) Ġand Ġthat Ġof Ġword Ġvector Ġrepresentations Ġto Ġbi - L ST M - based Ġmodels Ġas Ġreported Ġby ĠB IB REF 20 Ġ. ĠAs Ġmentioned Ġabove , Ġwork Ġon Ġthe Ġcontribution Ġof Ġword Ġvector Ġrepresentations Ġto Ġfeature - based Ġapproaches Ġhas Ġbeen Ġcarried Ġout Ġby ĠB IB REF 18 Ġ. ĠHowever , Ġthe Ġexploitation Ġof Ġexisting Ġmorph os ynt actic Ġor Ġmorph ological Ġlex icons Ġin Ġneural Ġmodels Ġis Ġa Ġless Ġstudied Ġquestion . ĠImprovements Ġover Ġthe Ġstate Ġof Ġthe Ġart Ġmight Ġbe Ġachieved Ġby Ġintegrating Ġlex ical Ġinformation Ġboth Ġfrom Ġan Ġexternal Ġlex icon Ġand Ġfrom Ġword Ġvector Ġrepresentations Ġinto Ġtagging Ġmodels . </s> In Ġthat Ġregard , Ġfurther Ġwork Ġwill Ġbe Ġrequired Ġto Ġunderstand Ġwhich Ġclass Ġof Ġmodels Ġperform Ġthe Ġbest . ĠAn Ġoption Ġwould Ġbe Ġto Ġintegrate Ġfeature - based Ġmodels Ġsuch Ġas Ġa ĠCR F Ġwith Ġan ĠL ST M - based Ġlayer , Ġfollowing Ġrecent Ġproposals Ġsuch Ġas Ġthe Ġone Ġproposed Ġby ĠB IB REF 45 Ġfor Ġnamed Ġentity Ġrecognition .