Yes
BOOLEAN
Yes
BOOLEAN
The lexicon based classifier is then used to predict the specific language within a language group. For the South African languages this is done for the Nguni and Sotho groups. If the lexicon prediction of the specific language has high confidence then its result is used as the final label else the naive Bayesian classifier's specific language prediction is used as the final result. The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets.
The lexicon based classifier is then used to predict the specific language within a language group. For the South African languages this is done for the Nguni and Sotho groups. If the lexicon prediction of the specific language has high confidence then its result is used as the final label else the naive Bayesian classifier's specific language prediction is used as the final result. The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets.
1911.07555
Introduction </s> Acc urate Ġlanguage Ġidentification Ġ( L ID ) Ġis Ġthe Ġfirst Ġstep Ġin Ġmany Ġnatural Ġlanguage Ġprocessing Ġand Ġmachine Ġcomprehension Ġpipelines . ĠIf Ġthe Ġlanguage Ġof Ġa Ġpiece Ġof Ġtext Ġis Ġknown Ġthen Ġthe Ġappropriate Ġdownstream Ġmodels Ġlike Ġparts Ġof Ġspeech Ġtag gers Ġand Ġlanguage Ġmodels Ġcan Ġbe Ġapplied Ġas Ġrequired . </s> L ID Ġis Ġfurther Ġalso Ġan Ġimportant Ġstep Ġin Ġharvesting Ġscarce Ġlanguage Ġresources . ĠHarvest ed Ġdata Ġcan Ġbe Ġused Ġto Ġboot strap Ġmore Ġaccurate ĠL ID Ġmodels Ġand Ġin Ġdoing Ġso Ġcontinually Ġimprove Ġthe Ġquality Ġof Ġthe Ġharvested Ġdata . ĠAvailability Ġof Ġdata Ġis Ġstill Ġone Ġof Ġthe Ġbig Ġroad blocks Ġfor Ġapplying Ġdata Ġdriven Ġapproaches Ġlike Ġsupervised Ġmachine Ġlearning Ġin Ġdeveloping Ġcountries . </s> Having Ġ11 Ġofficial Ġlanguages Ġof ĠSouth ĠAfrica Ġhas Ġlead Ġto Ġinitiatives Ġ( disc ussed Ġin Ġthe Ġnext Ġsection ) Ġthat Ġhave Ġhad Ġpositive Ġeffect Ġon Ġthe Ġavailability Ġof Ġlanguage Ġresources Ġfor Ġresearch . ĠHowever , Ġmany Ġof Ġthe ĠSouth ĠAfrican Ġlanguages Ġare Ġstill Ġunder Ġres ourced Ġfrom Ġthe Ġpoint Ġof Ġview Ġof Ġbuilding Ġdata Ġdriven Ġmodels Ġfor Ġmachine Ġcomprehension Ġand Ġprocess Ġautomation . </s> Table ĠT AB REF 2 Ġshows Ġthe Ġpercentages Ġof Ġfirst Ġlanguage Ġspeakers Ġfor Ġeach Ġof Ġthe Ġofficial Ġlanguages Ġof ĠSouth ĠAfrica . ĠThese Ġare Ġfour Ġconj unct ively Ġwritten ĠN gun i Ġlanguages Ġ( z ul , Ġx ho , Ġn bl , Ġs sw ), ĠAf rika ans Ġ( af r ) Ġand ĠEnglish Ġ( eng ), Ġthree Ġdis j unct ively Ġwritten ĠS oth o Ġlanguages Ġ( n so , Ġs ot , Ġt sn ), Ġas Ġwell Ġas Ġt shi V enda Ġ( ven ) Ġand ĠX its onga Ġ( t so ). ĠThe ĠN gun i Ġlanguages Ġare Ġsimilar Ġto Ġeach Ġother Ġand Ġharder Ġto Ġdistinguish . ĠThe Ġsame Ġis Ġtrue Ġof Ġthe ĠS oth o Ġlanguages . </s> This Ġpaper Ġpresents Ġa Ġhierarchical Ġnaive ĠBay esian Ġand Ġlex icon Ġbased Ġclass ifier Ġfor ĠL ID Ġof Ġshort Ġpieces Ġof Ġtext Ġof Ġ15 - 20 Ġcharacters Ġlong . ĠThe Ġalgorithm Ġis Ġevaluated Ġagainst Ġrecent Ġapproaches Ġusing Ġexisting Ġtest Ġsets Ġfrom Ġprevious Ġworks Ġon ĠSouth ĠAfrican Ġlanguages Ġas Ġwell Ġas Ġthe ĠDisc rim inating Ġbetween ĠSimilar ĠLanguages Ġ( DS L ) Ġ2015 Ġand Ġ2017 Ġshared Ġtasks . </s> Section ĠSEC REF 2 Ġreviews Ġexisting Ġworks Ġon Ġthe Ġtopic Ġand Ġsummar ises Ġthe Ġremaining Ġresearch Ġproblems . ĠSection ĠSEC REF 3 Ġof Ġthe Ġpaper Ġdiscusses Ġthe Ġproposed Ġalgorithm Ġand ĠSection ĠSEC REF 4 Ġpresents Ġcomparative Ġresults . </s> Related ĠWorks </s> The Ġfocus Ġof Ġthis Ġsection Ġis Ġon Ġrecently Ġpublished Ġdatasets Ġand ĠL ID Ġresearch Ġapplicable Ġto Ġthe ĠSouth ĠAfrican Ġcontext . ĠAn Ġin Ġdepth Ġsurvey Ġof Ġalgorithms , Ġfeatures , Ġdatasets , Ġshared Ġtasks Ġand Ġevaluation Ġmethods Ġmay Ġbe Ġfound Ġin ĠB IB REF 0 . </s> The Ġdatasets Ġfor Ġthe ĠDSL Ġ2015 Ġ& ĠDSL Ġ2017 Ġshared Ġtasks ĠB IB REF 1 Ġare Ġoften Ġused Ġin ĠL ID Ġbenchmarks Ġand Ġalso Ġavailable Ġon ĠK agg le Ġ. ĠThe ĠDSL Ġdatasets , Ġlike Ġother ĠL ID Ġdatasets , Ġconsists Ġof Ġtext Ġsentences Ġlabelled Ġby Ġlanguage . ĠThe Ġ2017 Ġdataset , Ġfor Ġexample , Ġcontains Ġ14 Ġlanguages Ġover Ġ6 Ġlanguage Ġgroups Ġwith Ġ18 000 Ġtraining Ġsamples Ġand Ġ1000 Ġtesting Ġsamples Ġper Ġlanguage . </s> The Ġrecently Ġpublished ĠJ W 300 Ġparallel Ġcorpus ĠB IB REF 2 Ġcovers Ġover Ġ300 Ġlanguages Ġwith Ġaround Ġ100 Ġthousand Ġparallel Ġsentences Ġper Ġlanguage Ġpair Ġon Ġaverage . ĠIn ĠSouth ĠAfrica , Ġa Ġmult ilingual Ġcorpus Ġof Ġacademic Ġtexts Ġproduced Ġby Ġuniversity Ġstudents Ġwith Ġdifferent Ġmother Ġtongues Ġis Ġbeing Ġdeveloped ĠB IB REF 3 . ĠThe ĠWi LI - 2018 Ġbenchmark Ġdataset ĠB IB REF 4 Ġfor Ġmon oling ual Ġwritten Ġnatural Ġlanguage Ġidentification Ġincludes Ġaround Ġ1000 Ġparagraphs Ġof Ġ235 Ġlanguages . ĠA Ġpossibly Ġuseful Ġlink Ġcan Ġalso Ġbe Ġmade ĠB IB REF 5 Ġbetween ĠNative ĠLanguage ĠIdentification Ġ( N LI ) Ġ( d etermin ing Ġthe Ġnative Ġlanguage Ġof Ġthe Ġauthor Ġof Ġa Ġtext ) Ġand ĠLanguage ĠVariety ĠIdentification Ġ( L VI ) Ġ( class ification Ġof Ġdifferent Ġvarieties Ġof Ġa Ġsingle Ġlanguage ) Ġwhich Ġopens Ġup Ġmore Ġdatasets . ĠThe ĠLe ip zig ĠCorpor a ĠCollection ĠB IB REF 6 , Ġthe ĠUniversal ĠDeclaration Ġof ĠHuman ĠRights Ġand ĠT ato eb a Ġare Ġalso Ġoften Ġused Ġsources Ġof Ġdata . </s> The ĠN CH LT Ġtext Ġcorpor a ĠB IB REF 7 Ġis Ġlikely Ġa Ġgood Ġstarting Ġpoint Ġfor Ġa Ġshared ĠL ID Ġtask Ġdataset Ġfor Ġthe ĠSouth ĠAfrican Ġlanguages ĠB IB REF 8 . ĠThe ĠN CH LT Ġtext Ġcorpor a Ġcontains Ġenough Ġdata Ġto Ġhave Ġ35 00 Ġtraining Ġsamples Ġand Ġ600 Ġtesting Ġsamples Ġof Ġ300 + Ġcharacter Ġsentences Ġper Ġlanguage . ĠResearchers Ġhave Ġrecently Ġstarted Ġapplying Ġexisting Ġalgorithms Ġfor Ġtasks Ġlike Ġneural Ġmachine Ġtranslation Ġin Ġearnest Ġto Ġsuch ĠSouth ĠAfrican Ġlanguage Ġdatasets ĠB IB REF 9 . </s> Ex isting ĠN LP Ġdatasets , Ġmodels Ġand Ġservices ĠB IB REF 10 Ġare Ġavailable Ġfor ĠSouth ĠAfrican Ġlanguages . ĠThese Ġinclude Ġan ĠL ID Ġalgorithm ĠB IB REF 11 Ġthat Ġuses Ġa Ġcharacter Ġlevel Ġn - gram Ġlanguage Ġmodel . ĠMultiple Ġpapers Ġhave Ġshown Ġthat Ġ' sh allow ' Ġnaive ĠBay es Ġclass ifiers ĠB IB REF 12 , ĠB IB REF 8 , ĠB IB REF 13 , ĠB IB REF 14 , ĠSV Ms ĠB IB REF 15 Ġand Ġsimilar Ġmodels Ġwork Ġvery Ġwell Ġfor Ġdoing ĠL ID . ĠThe ĠDSL Ġ2017 Ġpaper ĠB IB REF 1 , Ġfor Ġexample , Ġgives Ġan Ġoverview Ġof Ġthe Ġsolutions Ġof Ġall Ġof Ġthe Ġteams Ġthat Ġcompeted Ġon Ġthe Ġshared Ġtask Ġand Ġthe Ġwinning Ġapproach ĠB IB REF 16 Ġused Ġan ĠS VM Ġwith Ġcharacter Ġn - gram , Ġparts Ġof Ġspeech Ġtag Ġfeatures Ġand Ġsome Ġother Ġengineered Ġfeatures . ĠThe Ġwinning Ġapproach Ġfor ĠDSL Ġ2015 Ġused Ġan Ġensemble Ġnaive ĠBay es Ġclass ifier . ĠThe Ġfast text Ġclass ifier ĠB IB REF 17 Ġis Ġperhaps Ġone Ġof Ġthe Ġbest Ġknown Ġefficient Ġ' sh allow ' Ġtext Ġclass ifiers Ġthat Ġhave Ġbeen Ġused Ġfor ĠL ID Ġ. </s> Multiple Ġpapers Ġhave Ġproposed Ġhierarchical Ġstacked Ġclass ifiers Ġ( including Ġlex icons ) Ġthat Ġwould Ġfor Ġexample Ġfirst Ġclassify Ġa Ġpiece Ġof Ġtext Ġby Ġlanguage Ġgroup Ġand Ġthen Ġby Ġexact Ġlanguage ĠB IB REF 18 , ĠB IB REF 19 , ĠB IB REF 8 , ĠB IB REF 0 . ĠSome Ġwork Ġhas Ġalso Ġbeen Ġdone Ġon Ġclass ifying Ġsurn ames Ġbetween ĠT sh iv enda , ĠX its onga Ġand ĠSep edi ĠB IB REF 20 . ĠAdditionally , Ġdata Ġaug mentation ĠB IB REF 21 Ġand Ġadvers arial Ġtraining ĠB IB REF 22 Ġapproaches Ġare Ġpotentially Ġvery Ġuseful Ġto Ġreduce Ġthe Ġrequirement Ġfor Ġdata . </s> Researchers Ġhave Ġinvestigated Ġdeeper ĠL ID Ġmodels Ġlike Ġbid irection al Ġrecurrent Ġneural Ġnetworks ĠB IB REF 23 Ġor Ġen semb les Ġof Ġrecurrent Ġneural Ġnetworks ĠB IB REF 24 . ĠThe Ġlatter Ġis Ġreported Ġto Ġachieve Ġ95 . 12 % Ġin Ġthe ĠDSL Ġ2015 Ġshared Ġtask . ĠIn Ġthese Ġmodels Ġtext Ġfeatures Ġcan Ġinclude Ġcharacter Ġand Ġword Ġn - gram s Ġas Ġwell Ġas Ġinformative Ġcharacter Ġand Ġword - level Ġfeatures Ġlearnt ĠB IB REF 25 Ġfrom Ġthe Ġtraining Ġdata . ĠThe Ġneural Ġmethods Ġseem Ġto Ġwork Ġwell Ġin Ġtasks Ġwhere Ġmore Ġtraining Ġdata Ġis Ġavailable . </s> In Ġsummary , ĠL ID Ġof Ġshort Ġtexts , Ġinformal Ġstyles Ġand Ġsimilar Ġlanguages Ġremains Ġa Ġdifficult Ġproblem Ġwhich Ġis Ġactively Ġbeing Ġresearched . ĠIncreased Ġconfusion Ġcan Ġin Ġgeneral Ġbe Ġexpected Ġbetween Ġshorter Ġpieces Ġof Ġtext Ġand Ġlanguages Ġthat Ġare Ġmore Ġclosely Ġrelated . ĠSh allow Ġmethods Ġstill Ġseem Ġto Ġwork Ġwell Ġcompared Ġto Ġdeeper Ġmodels Ġfor ĠL ID . ĠOther Ġremaining Ġresearch Ġopportunities Ġseem Ġto Ġbe Ġdata Ġharvesting , Ġbuilding Ġstandard ised Ġdatasets Ġand Ġcreating Ġshared Ġtasks Ġfor ĠSouth ĠAfrica Ġand ĠAfrica . ĠSupport Ġfor Ġlanguage Ġcodes Ġthat Ġinclude Ġmore Ġlanguages Ġseems Ġto Ġbe Ġgrowing Ġand Ġdiscover ability Ġof Ġresearch Ġis Ġimproving Ġwith Ġmore Ġsurvey Ġpapers Ġcoming Ġout . ĠPay w alls Ġalso Ġseem Ġto Ġno Ġlonger Ġbe Ġa Ġproblem ; Ġthe Ġreferences Ġused Ġin Ġthis Ġpaper Ġwas Ġeither Ġopenly Ġpublished Ġor Ġavailable Ġas Ġpre print Ġpapers . </s> Method ology </s> The Ġproposed ĠL ID Ġalgorithm Ġbuilds Ġon Ġthe Ġwork Ġin ĠB IB REF 8 Ġand ĠB IB REF 26 . ĠWe Ġapply Ġa Ġnaive ĠBay esian Ġclass ifier Ġwith Ġcharacter Ġ( 2 , Ġ4 Ġ& Ġ6 )- gram s , Ġword Ġun ig ram Ġand Ġword Ġbig ram Ġfeatures Ġwith Ġa Ġhierarchical Ġlex icon Ġbased Ġclass ifier . </s> The Ġnaive ĠBay esian Ġclass ifier Ġis Ġtrained Ġto Ġpredict Ġthe Ġspecific Ġlanguage Ġlabel Ġof Ġa Ġpiece Ġof Ġtext , Ġbut Ġused Ġto Ġfirst Ġclassify Ġtext Ġas Ġbelonging Ġto Ġeither Ġthe ĠN gun i Ġfamily , Ġthe ĠS oth o Ġfamily , ĠEnglish , ĠAf rika ans , ĠX its onga Ġor ĠT sh iv enda . ĠThe Ġsc ik it - learn Ġmult in omial Ġnaive ĠBay es Ġclass ifier Ġis Ġused Ġfor Ġthe Ġimplementation Ġwith Ġan Ġalpha Ġsmoot hing Ġvalue Ġof Ġ0 . 01 Ġand Ġhas hed Ġtext Ġfeatures . </s> The Ġlex icon Ġbased Ġclass ifier Ġis Ġthen Ġused Ġto Ġpredict Ġthe Ġspecific Ġlanguage Ġwithin Ġa Ġlanguage Ġgroup . ĠFor Ġthe ĠSouth ĠAfrican Ġlanguages Ġthis Ġis Ġdone Ġfor Ġthe ĠN gun i Ġand ĠS oth o Ġgroups . ĠIf Ġthe Ġlex icon Ġprediction Ġof Ġthe Ġspecific Ġlanguage Ġhas Ġhigh Ġconfidence Ġthen Ġits Ġresult Ġis Ġused Ġas Ġthe Ġfinal Ġlabel Ġelse Ġthe Ġnaive ĠBay esian Ġclass ifier 's Ġspecific Ġlanguage Ġprediction Ġis Ġused Ġas Ġthe Ġfinal Ġresult . ĠThe Ġlex icon Ġis Ġbuilt Ġover Ġall Ġthe Ġdata Ġand Ġtherefore Ġincludes Ġthe Ġvocabulary Ġfrom Ġboth Ġthe Ġtraining Ġand Ġtesting Ġsets . </s> The Ġlex icon Ġbased Ġclass ifier Ġis Ġdesigned Ġto Ġtrade Ġhigher Ġprecision Ġfor Ġlower Ġrecall . ĠThe Ġproposed Ġimplementation Ġis Ġconsidered Ġconfident Ġif Ġthe Ġnumber Ġof Ġwords Ġfrom Ġthe Ġwinning Ġlanguage Ġis Ġat Ġleast Ġone Ġmore Ġthan Ġthe Ġnumber Ġof Ġwords Ġconsidered Ġto Ġbe Ġfrom Ġthe Ġlanguage Ġscored Ġin Ġsecond Ġplace . </s> The Ġstacked Ġclass ifier Ġis Ġtested Ġagainst Ġthree Ġpublic ĠL ID Ġimplementations ĠB IB REF 17 , ĠB IB REF 23 , ĠB IB REF 8 . ĠThe ĠL ID Ġimplementation Ġdescribed Ġin ĠB IB REF 17 Ġis Ġavailable Ġon ĠGitHub Ġand Ġis Ġtrained Ġand Ġtested Ġaccording Ġto Ġa Ġpost Ġon Ġthe Ġfast text Ġblog . ĠCharacter Ġ( 5 - 6 )- gram Ġfeatures Ġwith Ġ16 Ġdimensional Ġvectors Ġworked Ġthe Ġbest . ĠThe Ġimplementation Ġdiscussed Ġin ĠB IB REF 23 Ġis Ġavailable Ġfrom Ġhttps :// github . com / tom k oc mi / L an ide NN . ĠFollowing Ġthe Ġinstructions Ġfor Ġan ĠOS X Ġpip Ġinstall Ġof Ġan Ġold Ġr 0 . 8 Ġrelease Ġof ĠT ensor Flow , Ġthe ĠLan ide NN Ġcode Ġcould Ġbe Ġexecuted Ġin ĠPython Ġ3 . 7 . 4 . ĠSettings Ġwere Ġleft Ġat Ġtheir Ġdefaults Ġand Ġa Ġlearning Ġrate Ġof Ġ0 . 001 Ġwas Ġused Ġfollowed Ġby Ġa Ġrefinement Ġwith Ġlearning Ġrate Ġof Ġ0 . 0001 . ĠOnly Ġone Ġcode Ġmodification Ġwas Ġapplied Ġto Ġreturn Ġthe Ġresults Ġfrom Ġa Ġmethod Ġthat Ġpreviously Ġjust Ġprinted Ġto Ġscreen . ĠThe ĠL ID Ġalgorithm Ġdescribed Ġin ĠB IB REF 8 Ġis Ġalso Ġavailable Ġon ĠGitHub . </s> The Ġstacked Ġclass ifier Ġis Ġalso Ġtested Ġagainst Ġthe Ġresults Ġreported Ġfor Ġfour Ġother Ġalgorithms ĠB IB REF 16 , ĠB IB REF 26 , ĠB IB REF 24 , ĠB IB REF 15 . ĠAll Ġthe Ġcomparisons Ġare Ġdone Ġusing Ġthe ĠN CH LT ĠB IB REF 7 , ĠDSL Ġ2015 ĠB IB REF 19 Ġand ĠDSL Ġ2017 ĠB IB REF 1 Ġdatasets Ġdiscussed Ġin ĠSection ĠSEC REF 2 . </s> Results Ġand ĠAnalysis </s> The Ġaverage Ġclassification Ġaccuracy Ġresults Ġare Ġsummar ised Ġin ĠTable ĠT AB REF 9 . ĠThe Ġaccur acies Ġreported Ġare Ġfor Ġclass ifying Ġa Ġpiece Ġof Ġtext Ġby Ġits Ġspecific Ġlanguage Ġlabel . ĠClass ifying Ġtext Ġonly Ġby Ġlanguage Ġgroup Ġor Ġfamily Ġis Ġa Ġmuch Ġeasier Ġtask Ġas Ġreported Ġin ĠB IB REF 8 . </s> Different Ġvariations Ġof Ġthe Ġproposed Ġclass ifier Ġwere Ġevaluated . ĠA Ġsingle ĠNB Ġclass ifier Ġ( NB ), Ġa Ġstack Ġof Ġtwo ĠNB Ġclass ifiers Ġ( NB + NB ), Ġa Ġstack Ġof Ġa ĠNB Ġclass ifier Ġand Ġlex icon Ġ( NB + Lex ) Ġand Ġa Ġlex icon Ġ( Lex ) Ġby Ġitself . ĠA Ġlex icon Ġwith Ġa Ġ50 % Ġtraining Ġtoken Ġdrop out Ġis Ġalso Ġlisted Ġto Ġshow Ġthe Ġimpact Ġof Ġthe Ġlex icon Ġsupport Ġon Ġthe Ġaccuracy . </s> From Ġthe Ġresults Ġit Ġseems Ġthat Ġthe ĠDSL Ġ2017 Ġtask Ġmight Ġbe Ġharder Ġthan Ġthe ĠDSL Ġ2015 Ġand ĠN CH LT Ġtasks . ĠAlso , Ġthe Ġresults Ġfor Ġthe Ġimplementation Ġdiscussed Ġin ĠB IB REF 23 Ġmight Ġseem Ġlow , Ġbut Ġthe Ġresults Ġreported Ġin Ġthat Ġpaper Ġis Ġgenerated Ġon Ġlonger Ġpieces Ġof Ġtext Ġso Ġlower Ġscores Ġon Ġthe Ġshorter Ġpieces Ġof Ġtext Ġderived Ġfrom Ġthe ĠN CH LT Ġcorpor a Ġis Ġexpected . </s> The Ġaccuracy Ġof Ġthe Ġproposed Ġalgorithm Ġseems Ġto Ġbe Ġdependent Ġon Ġthe Ġsupport Ġof Ġthe Ġlex icon . ĠWithout Ġa Ġgood Ġlex icon Ġa Ġnon - st acked Ġnaive ĠBay esian Ġclass ifier Ġmight Ġeven Ġperform Ġbetter . </s> The Ġexecution Ġperformance Ġof Ġsome Ġof Ġthe ĠL ID Ġimplementations Ġare Ġshown Ġin ĠTable ĠT AB REF 10 . ĠResults Ġwere Ġgenerated Ġon Ġan Ġearly Ġ2015 Ġ13 - inch ĠRet ina ĠMacBook ĠPro Ġwith Ġa Ġ2 . 9 ĠGHz ĠCPU Ġ( Tur bo ĠBoost ed Ġto Ġ3 . 4 ĠGHz ) Ġand Ġ8 GB ĠRAM . ĠThe ĠC ++ Ġimplementation Ġin ĠB IB REF 17 Ġis Ġthe Ġfastest . ĠThe Ġimplementation Ġin ĠB IB REF 8 Ġmakes Ġuse Ġof Ġun - h ashed Ġfeature Ġrepresentations Ġwhich Ġcauses Ġit Ġto Ġbe Ġslower Ġthan Ġthe Ġproposed Ġsk learn Ġimplementation . ĠThe Ġexecution Ġperformance Ġof ĠB IB REF 23 Ġmight Ġimprove Ġby Ġa Ġfactor Ġof Ġfive Ġto Ġten Ġwhen Ġexecuted Ġon Ġa ĠGPU . </s> Conclusion </s> L ID Ġof Ġshort Ġtexts , Ġinformal Ġstyles Ġand Ġsimilar Ġlanguages Ġremains Ġa Ġdifficult Ġproblem Ġwhich Ġis Ġactively Ġbeing Ġresearched . ĠThe Ġproposed Ġalgorithm Ġwas Ġevaluated Ġon Ġthree Ġexisting Ġdatasets Ġand Ġcompared Ġto Ġthe Ġimplementations Ġof Ġthree Ġpublic ĠL ID Ġimplementations Ġas Ġwell Ġas Ġto Ġreported Ġresults Ġof Ġfour Ġother Ġalgorithms . ĠIt Ġperformed Ġwell Ġrelative Ġto Ġthe Ġother Ġmethods Ġbeating Ġtheir Ġresults . ĠHowever , Ġthe Ġperformance Ġis Ġdependent Ġon Ġthe Ġsupport Ġof Ġthe Ġlex icon . </s> We Ġwould Ġlike Ġto Ġinvestigate Ġthe Ġvalue Ġof Ġa Ġlex icon Ġin Ġa Ġproduction Ġsystem Ġand Ġhow Ġto Ġpossibly Ġmaintain Ġit Ġusing Ġself - super vised Ġlearning . ĠWe Ġare Ġinvestigating Ġthe Ġapplication Ġof Ġdeeper Ġlanguage Ġmodels Ġsome Ġof Ġwhich Ġhave Ġbeen Ġused Ġin Ġmore Ġrecent ĠDSL Ġshared Ġtasks . ĠWe Ġwould Ġalso Ġlike Ġto Ġinvestigate Ġdata Ġaug mentation Ġstrategies Ġto Ġreduce Ġthe Ġamount Ġof Ġtraining Ġdata Ġthat Ġis Ġrequired . </s> Further Ġresearch Ġopportunities Ġinclude Ġdata Ġharvesting , Ġbuilding Ġstandard ised Ġdatasets Ġand Ġshared Ġtasks Ġfor ĠSouth ĠAfrica Ġas Ġwell Ġas Ġthe Ġrest Ġof ĠAfrica . ĠIn Ġgeneral , Ġthe Ġsupport Ġfor Ġlanguage Ġcodes Ġthat Ġinclude Ġmore Ġlanguages Ġseems Ġto Ġbe Ġgrowing , Ġdiscover ability Ġof Ġresearch Ġis Ġimproving Ġand Ġpay w alls Ġseem Ġto Ġno Ġlonger Ġbe Ġa Ġbig Ġproblem Ġin Ġgetting Ġaccess Ġto Ġpublished Ġresearch .
Introduction Accurate language identification (LID) is the first step in many natural language processing and machine comprehension pipelines. If the language of a piece of text is known then the appropriate downstream models like parts of speech taggers and language models can be applied as required. LID is further also an important step in harvesting scarce language resources. Harvested data can be used to bootstrap more accurate LID models and in doing so continually improve the quality of the harvested data. Availability of data is still one of the big roadblocks for applying data driven approaches like supervised machine learning in developing countries. Having 11 official languages of South Africa has lead to initiatives (discussed in the next section) that have had positive effect on the availability of language resources for research. However, many of the South African languages are still under resourced from the point of view of building data driven models for machine comprehension and process automation. Table TABREF2 shows the percentages of first language speakers for each of the official languages of South Africa. These are four conjunctively written Nguni languages (zul, xho, nbl, ssw), Afrikaans (afr) and English (eng), three disjunctively written Sotho languages (nso, sot, tsn), as well as tshiVenda (ven) and Xitsonga (tso). The Nguni languages are similar to each other and harder to distinguish. The same is true of the Sotho languages. This paper presents a hierarchical naive Bayesian and lexicon based classifier for LID of short pieces of text of 15-20 characters long. The algorithm is evaluated against recent approaches using existing test sets from previous works on South African languages as well as the Discriminating between Similar Languages (DSL) 2015 and 2017 shared tasks. Section SECREF2 reviews existing works on the topic and summarises the remaining research problems. Section SECREF3 of the paper discusses the proposed algorithm and Section SECREF4 presents comparative results. Related Works The focus of this section is on recently published datasets and LID research applicable to the South African context. An in depth survey of algorithms, features, datasets, shared tasks and evaluation methods may be found in BIBREF0. The datasets for the DSL 2015 & DSL 2017 shared tasks BIBREF1 are often used in LID benchmarks and also available on Kaggle . The DSL datasets, like other LID datasets, consists of text sentences labelled by language. The 2017 dataset, for example, contains 14 languages over 6 language groups with 18000 training samples and 1000 testing samples per language. The recently published JW300 parallel corpus BIBREF2 covers over 300 languages with around 100 thousand parallel sentences per language pair on average. In South Africa, a multilingual corpus of academic texts produced by university students with different mother tongues is being developed BIBREF3. The WiLI-2018 benchmark dataset BIBREF4 for monolingual written natural language identification includes around 1000 paragraphs of 235 languages. A possibly useful link can also be made BIBREF5 between Native Language Identification (NLI) (determining the native language of the author of a text) and Language Variety Identification (LVI) (classification of different varieties of a single language) which opens up more datasets. The Leipzig Corpora Collection BIBREF6, the Universal Declaration of Human Rights and Tatoeba are also often used sources of data. The NCHLT text corpora BIBREF7 is likely a good starting point for a shared LID task dataset for the South African languages BIBREF8. The NCHLT text corpora contains enough data to have 3500 training samples and 600 testing samples of 300+ character sentences per language. Researchers have recently started applying existing algorithms for tasks like neural machine translation in earnest to such South African language datasets BIBREF9. Existing NLP datasets, models and services BIBREF10 are available for South African languages. These include an LID algorithm BIBREF11 that uses a character level n-gram language model. Multiple papers have shown that 'shallow' naive Bayes classifiers BIBREF12, BIBREF8, BIBREF13, BIBREF14, SVMs BIBREF15 and similar models work very well for doing LID. The DSL 2017 paper BIBREF1, for example, gives an overview of the solutions of all of the teams that competed on the shared task and the winning approach BIBREF16 used an SVM with character n-gram, parts of speech tag features and some other engineered features. The winning approach for DSL 2015 used an ensemble naive Bayes classifier. The fasttext classifier BIBREF17 is perhaps one of the best known efficient 'shallow' text classifiers that have been used for LID . Multiple papers have proposed hierarchical stacked classifiers (including lexicons) that would for example first classify a piece of text by language group and then by exact language BIBREF18, BIBREF19, BIBREF8, BIBREF0. Some work has also been done on classifying surnames between Tshivenda, Xitsonga and Sepedi BIBREF20. Additionally, data augmentation BIBREF21 and adversarial training BIBREF22 approaches are potentially very useful to reduce the requirement for data. Researchers have investigated deeper LID models like bidirectional recurrent neural networks BIBREF23 or ensembles of recurrent neural networks BIBREF24. The latter is reported to achieve 95.12% in the DSL 2015 shared task. In these models text features can include character and word n-grams as well as informative character and word-level features learnt BIBREF25 from the training data. The neural methods seem to work well in tasks where more training data is available. In summary, LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched. Increased confusion can in general be expected between shorter pieces of text and languages that are more closely related. Shallow methods still seem to work well compared to deeper models for LID. Other remaining research opportunities seem to be data harvesting, building standardised datasets and creating shared tasks for South Africa and Africa. Support for language codes that include more languages seems to be growing and discoverability of research is improving with more survey papers coming out. Paywalls also seem to no longer be a problem; the references used in this paper was either openly published or available as preprint papers. Methodology The proposed LID algorithm builds on the work in BIBREF8 and BIBREF26. We apply a naive Bayesian classifier with character (2, 4 & 6)-grams, word unigram and word bigram features with a hierarchical lexicon based classifier. The naive Bayesian classifier is trained to predict the specific language label of a piece of text, but used to first classify text as belonging to either the Nguni family, the Sotho family, English, Afrikaans, Xitsonga or Tshivenda. The scikit-learn multinomial naive Bayes classifier is used for the implementation with an alpha smoothing value of 0.01 and hashed text features. The lexicon based classifier is then used to predict the specific language within a language group. For the South African languages this is done for the Nguni and Sotho groups. If the lexicon prediction of the specific language has high confidence then its result is used as the final label else the naive Bayesian classifier's specific language prediction is used as the final result. The lexicon is built over all the data and therefore includes the vocabulary from both the training and testing sets. The lexicon based classifier is designed to trade higher precision for lower recall. The proposed implementation is considered confident if the number of words from the winning language is at least one more than the number of words considered to be from the language scored in second place. The stacked classifier is tested against three public LID implementations BIBREF17, BIBREF23, BIBREF8. The LID implementation described in BIBREF17 is available on GitHub and is trained and tested according to a post on the fasttext blog. Character (5-6)-gram features with 16 dimensional vectors worked the best. The implementation discussed in BIBREF23 is available from https://github.com/tomkocmi/LanideNN. Following the instructions for an OSX pip install of an old r0.8 release of TensorFlow, the LanideNN code could be executed in Python 3.7.4. Settings were left at their defaults and a learning rate of 0.001 was used followed by a refinement with learning rate of 0.0001. Only one code modification was applied to return the results from a method that previously just printed to screen. The LID algorithm described in BIBREF8 is also available on GitHub. The stacked classifier is also tested against the results reported for four other algorithms BIBREF16, BIBREF26, BIBREF24, BIBREF15. All the comparisons are done using the NCHLT BIBREF7, DSL 2015 BIBREF19 and DSL 2017 BIBREF1 datasets discussed in Section SECREF2. Results and Analysis The average classification accuracy results are summarised in Table TABREF9. The accuracies reported are for classifying a piece of text by its specific language label. Classifying text only by language group or family is a much easier task as reported in BIBREF8. Different variations of the proposed classifier were evaluated. A single NB classifier (NB), a stack of two NB classifiers (NB+NB), a stack of a NB classifier and lexicon (NB+Lex) and a lexicon (Lex) by itself. A lexicon with a 50% training token dropout is also listed to show the impact of the lexicon support on the accuracy. From the results it seems that the DSL 2017 task might be harder than the DSL 2015 and NCHLT tasks. Also, the results for the implementation discussed in BIBREF23 might seem low, but the results reported in that paper is generated on longer pieces of text so lower scores on the shorter pieces of text derived from the NCHLT corpora is expected. The accuracy of the proposed algorithm seems to be dependent on the support of the lexicon. Without a good lexicon a non-stacked naive Bayesian classifier might even perform better. The execution performance of some of the LID implementations are shown in Table TABREF10. Results were generated on an early 2015 13-inch Retina MacBook Pro with a 2.9 GHz CPU (Turbo Boosted to 3.4 GHz) and 8GB RAM. The C++ implementation in BIBREF17 is the fastest. The implementation in BIBREF8 makes use of un-hashed feature representations which causes it to be slower than the proposed sklearn implementation. The execution performance of BIBREF23 might improve by a factor of five to ten when executed on a GPU. Conclusion LID of short texts, informal styles and similar languages remains a difficult problem which is actively being researched. The proposed algorithm was evaluated on three existing datasets and compared to the implementations of three public LID implementations as well as to reported results of four other algorithms. It performed well relative to the other methods beating their results. However, the performance is dependent on the support of the lexicon. We would like to investigate the value of a lexicon in a production system and how to possibly maintain it using self-supervised learning. We are investigating the application of deeper language models some of which have been used in more recent DSL shared tasks. We would also like to investigate data augmentation strategies to reduce the amount of training data that is required. Further research opportunities include data harvesting, building standardised datasets and shared tasks for South Africa as well as the rest of Africa. In general, the support for language codes that include more languages seems to be growing, discoverability of research is improving and paywalls seem to no longer be a big problem in getting access to published research.
Is the lexicon the same for all languages?
c598028815066089cc1e131b96d6966d2610467a
Is Ġthe Ġlex icon Ġthe Ġsame Ġfor Ġall Ġlanguages ?
Yes
<s> Is Ġthe Ġlex icon Ġthe Ġsame Ġfor Ġall Ġlanguages ? </s> Introduction </s> Acc urate Ġlanguage Ġidentification Ġ( L ID ) Ġis Ġthe Ġfirst Ġstep Ġin Ġmany Ġnatural Ġlanguage Ġprocessing Ġand Ġmachine Ġcomprehension Ġpipelines . ĠIf Ġthe Ġlanguage Ġof Ġa Ġpiece Ġof Ġtext Ġis Ġknown Ġthen Ġthe Ġappropriate Ġdownstream Ġmodels Ġlike Ġparts Ġof Ġspeech Ġtag gers Ġand Ġlanguage Ġmodels Ġcan Ġbe Ġapplied Ġas Ġrequired . </s> L ID Ġis Ġfurther Ġalso Ġan Ġimportant Ġstep Ġin Ġharvesting Ġscarce Ġlanguage Ġresources . ĠHarvest ed Ġdata Ġcan Ġbe Ġused Ġto Ġboot strap Ġmore Ġaccurate ĠL ID Ġmodels Ġand Ġin Ġdoing Ġso Ġcontinually Ġimprove Ġthe Ġquality Ġof Ġthe Ġharvested Ġdata . ĠAvailability Ġof Ġdata Ġis Ġstill Ġone Ġof Ġthe Ġbig Ġroad blocks Ġfor Ġapplying Ġdata Ġdriven Ġapproaches Ġlike Ġsupervised Ġmachine Ġlearning Ġin Ġdeveloping Ġcountries . </s> Having Ġ11 Ġofficial Ġlanguages Ġof ĠSouth ĠAfrica Ġhas Ġlead Ġto Ġinitiatives Ġ( disc ussed Ġin Ġthe Ġnext Ġsection ) Ġthat Ġhave Ġhad Ġpositive Ġeffect Ġon Ġthe Ġavailability Ġof Ġlanguage Ġresources Ġfor Ġresearch . ĠHowever , Ġmany Ġof Ġthe ĠSouth ĠAfrican Ġlanguages Ġare Ġstill Ġunder Ġres ourced Ġfrom Ġthe Ġpoint Ġof Ġview Ġof Ġbuilding Ġdata Ġdriven Ġmodels Ġfor Ġmachine Ġcomprehension Ġand Ġprocess Ġautomation . </s> Table ĠT AB REF 2 Ġshows Ġthe Ġpercentages Ġof Ġfirst Ġlanguage Ġspeakers Ġfor Ġeach Ġof Ġthe Ġofficial Ġlanguages Ġof ĠSouth ĠAfrica . ĠThese Ġare Ġfour Ġconj unct ively Ġwritten ĠN gun i Ġlanguages Ġ( z ul , Ġx ho , Ġn bl , Ġs sw ), ĠAf rika ans Ġ( af r ) Ġand ĠEnglish Ġ( eng ), Ġthree Ġdis j unct ively Ġwritten ĠS oth o Ġlanguages Ġ( n so , Ġs ot , Ġt sn ), Ġas Ġwell Ġas Ġt shi V enda Ġ( ven ) Ġand ĠX its onga Ġ( t so ). ĠThe ĠN gun i Ġlanguages Ġare Ġsimilar Ġto Ġeach Ġother Ġand Ġharder Ġto Ġdistinguish . ĠThe Ġsame Ġis Ġtrue Ġof Ġthe ĠS oth o Ġlanguages . </s> This Ġpaper Ġpresents Ġa Ġhierarchical Ġnaive ĠBay esian Ġand Ġlex icon Ġbased Ġclass ifier Ġfor ĠL ID Ġof Ġshort Ġpieces Ġof Ġtext Ġof Ġ15 - 20 Ġcharacters Ġlong . ĠThe Ġalgorithm Ġis Ġevaluated Ġagainst Ġrecent Ġapproaches Ġusing Ġexisting Ġtest Ġsets Ġfrom Ġprevious Ġworks Ġon ĠSouth ĠAfrican Ġlanguages Ġas Ġwell Ġas Ġthe ĠDisc rim inating Ġbetween ĠSimilar ĠLanguages Ġ( DS L ) Ġ2015 Ġand Ġ2017 Ġshared Ġtasks . </s> Section ĠSEC REF 2 Ġreviews Ġexisting Ġworks Ġon Ġthe Ġtopic Ġand Ġsummar ises Ġthe Ġremaining Ġresearch Ġproblems . ĠSection ĠSEC REF 3 Ġof Ġthe Ġpaper Ġdiscusses Ġthe Ġproposed Ġalgorithm Ġand ĠSection ĠSEC REF 4 Ġpresents Ġcomparative Ġresults . </s> Related ĠWorks </s> The Ġfocus Ġof Ġthis Ġsection Ġis Ġon Ġrecently Ġpublished Ġdatasets Ġand ĠL ID Ġresearch Ġapplicable Ġto Ġthe ĠSouth ĠAfrican Ġcontext . ĠAn Ġin Ġdepth Ġsurvey Ġof Ġalgorithms , Ġfeatures , Ġdatasets , Ġshared Ġtasks Ġand Ġevaluation Ġmethods Ġmay Ġbe Ġfound Ġin ĠB IB REF 0 . </s> The Ġdatasets Ġfor Ġthe ĠDSL Ġ2015 Ġ& ĠDSL Ġ2017 Ġshared Ġtasks ĠB IB REF 1 Ġare Ġoften Ġused Ġin ĠL ID Ġbenchmarks Ġand Ġalso Ġavailable Ġon ĠK agg le Ġ. ĠThe ĠDSL Ġdatasets , Ġlike Ġother ĠL ID Ġdatasets , Ġconsists Ġof Ġtext Ġsentences Ġlabelled Ġby Ġlanguage . ĠThe Ġ2017 Ġdataset , Ġfor Ġexample , Ġcontains Ġ14 Ġlanguages Ġover Ġ6 Ġlanguage Ġgroups Ġwith Ġ18 000 Ġtraining Ġsamples Ġand Ġ1000 Ġtesting Ġsamples Ġper Ġlanguage . </s> The Ġrecently Ġpublished ĠJ W 300 Ġparallel Ġcorpus ĠB IB REF 2 Ġcovers Ġover Ġ300 Ġlanguages Ġwith Ġaround Ġ100 Ġthousand Ġparallel Ġsentences Ġper Ġlanguage Ġpair Ġon Ġaverage . ĠIn ĠSouth ĠAfrica , Ġa Ġmult ilingual Ġcorpus Ġof Ġacademic Ġtexts Ġproduced Ġby Ġuniversity Ġstudents Ġwith Ġdifferent Ġmother Ġtongues Ġis Ġbeing Ġdeveloped ĠB IB REF 3 . ĠThe ĠWi LI - 2018 Ġbenchmark Ġdataset ĠB IB REF 4 Ġfor Ġmon oling ual Ġwritten Ġnatural Ġlanguage Ġidentification Ġincludes Ġaround Ġ1000 Ġparagraphs Ġof Ġ235 Ġlanguages . ĠA Ġpossibly Ġuseful Ġlink Ġcan Ġalso Ġbe Ġmade ĠB IB REF 5 Ġbetween ĠNative ĠLanguage ĠIdentification Ġ( N LI ) Ġ( d etermin ing Ġthe Ġnative Ġlanguage Ġof Ġthe Ġauthor Ġof Ġa Ġtext ) Ġand ĠLanguage ĠVariety ĠIdentification Ġ( L VI ) Ġ( class ification Ġof Ġdifferent Ġvarieties Ġof Ġa Ġsingle Ġlanguage ) Ġwhich Ġopens Ġup Ġmore Ġdatasets . ĠThe ĠLe ip zig ĠCorpor a ĠCollection ĠB IB REF 6 , Ġthe ĠUniversal ĠDeclaration Ġof ĠHuman ĠRights Ġand ĠT ato eb a Ġare Ġalso Ġoften Ġused Ġsources Ġof Ġdata . </s> The ĠN CH LT Ġtext Ġcorpor a ĠB IB REF 7 Ġis Ġlikely Ġa Ġgood Ġstarting Ġpoint Ġfor Ġa Ġshared ĠL ID Ġtask Ġdataset Ġfor Ġthe ĠSouth ĠAfrican Ġlanguages ĠB IB REF 8 . ĠThe ĠN CH LT Ġtext Ġcorpor a Ġcontains Ġenough Ġdata Ġto Ġhave Ġ35 00 Ġtraining Ġsamples Ġand Ġ600 Ġtesting Ġsamples Ġof Ġ300 + Ġcharacter Ġsentences Ġper Ġlanguage . ĠResearchers Ġhave Ġrecently Ġstarted Ġapplying Ġexisting Ġalgorithms Ġfor Ġtasks Ġlike Ġneural Ġmachine Ġtranslation Ġin Ġearnest Ġto Ġsuch ĠSouth ĠAfrican Ġlanguage Ġdatasets ĠB IB REF 9 . </s> Ex isting ĠN LP Ġdatasets , Ġmodels Ġand Ġservices ĠB IB REF 10 Ġare Ġavailable Ġfor ĠSouth ĠAfrican Ġlanguages . ĠThese Ġinclude Ġan ĠL ID Ġalgorithm ĠB IB REF 11 Ġthat Ġuses Ġa Ġcharacter Ġlevel Ġn - gram Ġlanguage Ġmodel . ĠMultiple Ġpapers Ġhave Ġshown Ġthat Ġ' sh allow ' Ġnaive ĠBay es Ġclass ifiers ĠB IB REF 12 , ĠB IB REF 8 , ĠB IB REF 13 , ĠB IB REF 14 , ĠSV Ms ĠB IB REF 15 Ġand Ġsimilar Ġmodels Ġwork Ġvery Ġwell Ġfor Ġdoing ĠL ID . ĠThe ĠDSL Ġ2017 Ġpaper ĠB IB REF 1 , Ġfor Ġexample , Ġgives Ġan Ġoverview Ġof Ġthe Ġsolutions Ġof Ġall Ġof Ġthe Ġteams Ġthat Ġcompeted Ġon Ġthe Ġshared Ġtask Ġand Ġthe Ġwinning Ġapproach ĠB IB REF 16 Ġused Ġan ĠS VM Ġwith Ġcharacter Ġn - gram , Ġparts Ġof Ġspeech Ġtag Ġfeatures Ġand Ġsome Ġother Ġengineered Ġfeatures . ĠThe Ġwinning Ġapproach Ġfor ĠDSL Ġ2015 Ġused Ġan Ġensemble Ġnaive ĠBay es Ġclass ifier . ĠThe Ġfast text Ġclass ifier ĠB IB REF 17 Ġis Ġperhaps Ġone Ġof Ġthe Ġbest Ġknown Ġefficient Ġ' sh allow ' Ġtext Ġclass ifiers Ġthat Ġhave Ġbeen Ġused Ġfor ĠL ID Ġ. </s> Multiple Ġpapers Ġhave Ġproposed Ġhierarchical Ġstacked Ġclass ifiers Ġ( including Ġlex icons ) Ġthat Ġwould Ġfor Ġexample Ġfirst Ġclassify Ġa Ġpiece Ġof Ġtext Ġby Ġlanguage Ġgroup Ġand Ġthen Ġby Ġexact Ġlanguage ĠB IB REF 18 , ĠB IB REF 19 , ĠB IB REF 8 , ĠB IB REF 0 . ĠSome Ġwork Ġhas Ġalso Ġbeen Ġdone Ġon Ġclass ifying Ġsurn ames Ġbetween ĠT sh iv enda , ĠX its onga Ġand ĠSep edi ĠB IB REF 20 . ĠAdditionally , Ġdata Ġaug mentation ĠB IB REF 21 Ġand Ġadvers arial Ġtraining ĠB IB REF 22 Ġapproaches Ġare Ġpotentially Ġvery Ġuseful Ġto Ġreduce Ġthe Ġrequirement Ġfor Ġdata . </s> Researchers Ġhave Ġinvestigated Ġdeeper ĠL ID Ġmodels Ġlike Ġbid irection al Ġrecurrent Ġneural Ġnetworks ĠB IB REF 23 Ġor Ġen semb les Ġof Ġrecurrent Ġneural Ġnetworks ĠB IB REF 24 . ĠThe Ġlatter Ġis Ġreported Ġto Ġachieve Ġ95 . 12 % Ġin Ġthe ĠDSL Ġ2015 Ġshared Ġtask . ĠIn Ġthese Ġmodels Ġtext Ġfeatures Ġcan Ġinclude Ġcharacter Ġand Ġword Ġn - gram s Ġas Ġwell Ġas Ġinformative Ġcharacter Ġand Ġword - level Ġfeatures Ġlearnt ĠB IB REF 25 Ġfrom Ġthe Ġtraining Ġdata . ĠThe Ġneural Ġmethods Ġseem Ġto Ġwork Ġwell Ġin Ġtasks Ġwhere Ġmore Ġtraining Ġdata Ġis Ġavailable . </s> In Ġsummary , ĠL ID Ġof Ġshort Ġtexts , Ġinformal Ġstyles Ġand Ġsimilar Ġlanguages Ġremains Ġa Ġdifficult Ġproblem Ġwhich Ġis Ġactively Ġbeing Ġresearched . ĠIncreased Ġconfusion Ġcan Ġin Ġgeneral Ġbe Ġexpected Ġbetween Ġshorter Ġpieces Ġof Ġtext Ġand Ġlanguages Ġthat Ġare Ġmore Ġclosely Ġrelated . ĠSh allow Ġmethods Ġstill Ġseem Ġto Ġwork Ġwell Ġcompared Ġto Ġdeeper Ġmodels Ġfor ĠL ID . ĠOther Ġremaining Ġresearch Ġopportunities Ġseem Ġto Ġbe Ġdata Ġharvesting , Ġbuilding Ġstandard ised Ġdatasets Ġand Ġcreating Ġshared Ġtasks Ġfor ĠSouth ĠAfrica Ġand ĠAfrica . ĠSupport Ġfor Ġlanguage Ġcodes Ġthat Ġinclude Ġmore Ġlanguages Ġseems Ġto Ġbe Ġgrowing Ġand Ġdiscover ability Ġof Ġresearch Ġis Ġimproving Ġwith Ġmore Ġsurvey Ġpapers Ġcoming Ġout . ĠPay w alls Ġalso Ġseem Ġto Ġno Ġlonger Ġbe Ġa Ġproblem ; Ġthe Ġreferences Ġused Ġin Ġthis Ġpaper Ġwas Ġeither Ġopenly Ġpublished Ġor Ġavailable Ġas Ġpre print Ġpapers . </s> Method ology </s> The Ġproposed ĠL ID Ġalgorithm Ġbuilds Ġon Ġthe Ġwork Ġin ĠB IB REF 8 Ġand ĠB IB REF 26 . ĠWe Ġapply Ġa Ġnaive ĠBay esian Ġclass ifier Ġwith Ġcharacter Ġ( 2 , Ġ4 Ġ& Ġ6 )- gram s , Ġword Ġun ig ram Ġand Ġword Ġbig ram Ġfeatures Ġwith Ġa Ġhierarchical Ġlex icon Ġbased Ġclass ifier . </s> The Ġnaive ĠBay esian Ġclass ifier Ġis Ġtrained Ġto Ġpredict Ġthe Ġspecific Ġlanguage Ġlabel Ġof Ġa Ġpiece Ġof Ġtext , Ġbut Ġused Ġto Ġfirst Ġclassify Ġtext Ġas Ġbelonging Ġto Ġeither Ġthe ĠN gun i Ġfamily , Ġthe ĠS oth o Ġfamily , ĠEnglish , ĠAf rika ans , ĠX its onga Ġor ĠT sh iv enda . ĠThe Ġsc ik it - learn Ġmult in omial Ġnaive ĠBay es Ġclass ifier Ġis Ġused Ġfor Ġthe Ġimplementation Ġwith Ġan Ġalpha Ġsmoot hing Ġvalue Ġof Ġ0 . 01 Ġand Ġhas hed Ġtext Ġfeatures . </s> The Ġlex icon Ġbased Ġclass ifier Ġis Ġthen Ġused Ġto Ġpredict Ġthe Ġspecific Ġlanguage Ġwithin Ġa Ġlanguage Ġgroup . ĠFor Ġthe ĠSouth ĠAfrican Ġlanguages Ġthis Ġis Ġdone Ġfor Ġthe ĠN gun i Ġand ĠS oth o Ġgroups . ĠIf Ġthe Ġlex icon Ġprediction Ġof Ġthe Ġspecific Ġlanguage Ġhas Ġhigh Ġconfidence Ġthen Ġits Ġresult Ġis Ġused Ġas Ġthe Ġfinal Ġlabel Ġelse Ġthe Ġnaive ĠBay esian Ġclass ifier 's Ġspecific Ġlanguage Ġprediction Ġis Ġused Ġas Ġthe Ġfinal Ġresult . ĠThe Ġlex icon Ġis Ġbuilt Ġover Ġall Ġthe Ġdata Ġand Ġtherefore Ġincludes Ġthe Ġvocabulary Ġfrom Ġboth Ġthe Ġtraining Ġand Ġtesting Ġsets . </s> The Ġlex icon Ġbased Ġclass ifier Ġis Ġdesigned Ġto Ġtrade Ġhigher Ġprecision Ġfor Ġlower Ġrecall . ĠThe Ġproposed Ġimplementation Ġis Ġconsidered Ġconfident Ġif Ġthe Ġnumber Ġof Ġwords Ġfrom Ġthe Ġwinning Ġlanguage Ġis Ġat Ġleast Ġone Ġmore Ġthan Ġthe Ġnumber Ġof Ġwords Ġconsidered Ġto Ġbe Ġfrom Ġthe Ġlanguage Ġscored Ġin Ġsecond Ġplace . </s> The Ġstacked Ġclass ifier Ġis Ġtested Ġagainst Ġthree Ġpublic ĠL ID Ġimplementations ĠB IB REF 17 , ĠB IB REF 23 , ĠB IB REF 8 . ĠThe ĠL ID Ġimplementation Ġdescribed Ġin ĠB IB REF 17 Ġis Ġavailable Ġon ĠGitHub Ġand Ġis Ġtrained Ġand Ġtested Ġaccording Ġto Ġa Ġpost Ġon Ġthe Ġfast text Ġblog . ĠCharacter Ġ( 5 - 6 )- gram Ġfeatures Ġwith Ġ16 Ġdimensional Ġvectors Ġworked Ġthe Ġbest . ĠThe Ġimplementation Ġdiscussed Ġin ĠB IB REF 23 Ġis Ġavailable Ġfrom Ġhttps :// github . com / tom k oc mi / L an ide NN . ĠFollowing Ġthe Ġinstructions Ġfor Ġan ĠOS X Ġpip Ġinstall Ġof Ġan Ġold Ġr 0 . 8 Ġrelease Ġof ĠT ensor Flow , Ġthe ĠLan ide NN Ġcode Ġcould Ġbe Ġexecuted Ġin ĠPython Ġ3 . 7 . 4 . ĠSettings Ġwere Ġleft Ġat Ġtheir Ġdefaults Ġand Ġa Ġlearning Ġrate Ġof Ġ0 . 001 Ġwas Ġused Ġfollowed Ġby Ġa Ġrefinement Ġwith Ġlearning Ġrate Ġof Ġ0 . 0001 . ĠOnly Ġone Ġcode Ġmodification Ġwas Ġapplied Ġto Ġreturn Ġthe Ġresults Ġfrom Ġa Ġmethod Ġthat Ġpreviously Ġjust Ġprinted Ġto Ġscreen . ĠThe ĠL ID Ġalgorithm Ġdescribed Ġin ĠB IB REF 8 Ġis Ġalso Ġavailable Ġon ĠGitHub . </s> The Ġstacked Ġclass ifier Ġis Ġalso Ġtested Ġagainst Ġthe Ġresults Ġreported Ġfor Ġfour Ġother Ġalgorithms ĠB IB REF 16 , ĠB IB REF 26 , ĠB IB REF 24 , ĠB IB REF 15 . ĠAll Ġthe Ġcomparisons Ġare Ġdone Ġusing Ġthe ĠN CH LT ĠB IB REF 7 , ĠDSL Ġ2015 ĠB IB REF 19 Ġand ĠDSL Ġ2017 ĠB IB REF 1 Ġdatasets Ġdiscussed Ġin ĠSection ĠSEC REF 2 . </s> Results Ġand ĠAnalysis </s> The Ġaverage Ġclassification Ġaccuracy Ġresults Ġare Ġsummar ised Ġin ĠTable ĠT AB REF 9 . ĠThe Ġaccur acies Ġreported Ġare Ġfor Ġclass ifying Ġa Ġpiece Ġof Ġtext Ġby Ġits Ġspecific Ġlanguage Ġlabel . ĠClass ifying Ġtext Ġonly Ġby Ġlanguage Ġgroup Ġor Ġfamily Ġis Ġa Ġmuch Ġeasier Ġtask Ġas Ġreported Ġin ĠB IB REF 8 . </s> Different Ġvariations Ġof Ġthe Ġproposed Ġclass ifier Ġwere Ġevaluated . ĠA Ġsingle ĠNB Ġclass ifier Ġ( NB ), Ġa Ġstack Ġof Ġtwo ĠNB Ġclass ifiers Ġ( NB + NB ), Ġa Ġstack Ġof Ġa ĠNB Ġclass ifier Ġand Ġlex icon Ġ( NB + Lex ) Ġand Ġa Ġlex icon Ġ( Lex ) Ġby Ġitself . ĠA Ġlex icon Ġwith Ġa Ġ50 % Ġtraining Ġtoken Ġdrop out Ġis Ġalso Ġlisted Ġto Ġshow Ġthe Ġimpact Ġof Ġthe Ġlex icon Ġsupport Ġon Ġthe Ġaccuracy . </s> From Ġthe Ġresults Ġit Ġseems Ġthat Ġthe ĠDSL Ġ2017 Ġtask Ġmight Ġbe Ġharder Ġthan Ġthe ĠDSL Ġ2015 Ġand ĠN CH LT Ġtasks . ĠAlso , Ġthe Ġresults Ġfor Ġthe Ġimplementation Ġdiscussed Ġin ĠB IB REF 23 Ġmight Ġseem Ġlow , Ġbut Ġthe Ġresults Ġreported Ġin Ġthat Ġpaper Ġis Ġgenerated Ġon Ġlonger Ġpieces Ġof Ġtext Ġso Ġlower Ġscores Ġon Ġthe Ġshorter Ġpieces Ġof Ġtext Ġderived Ġfrom Ġthe ĠN CH LT Ġcorpor a Ġis Ġexpected . </s> The Ġaccuracy Ġof Ġthe Ġproposed Ġalgorithm Ġseems Ġto Ġbe Ġdependent Ġon Ġthe Ġsupport Ġof Ġthe Ġlex icon . ĠWithout Ġa Ġgood Ġlex icon Ġa Ġnon - st acked Ġnaive ĠBay esian Ġclass ifier Ġmight Ġeven Ġperform Ġbetter . </s> The Ġexecution Ġperformance Ġof Ġsome Ġof Ġthe ĠL ID Ġimplementations Ġare Ġshown Ġin ĠTable ĠT AB REF 10 . ĠResults Ġwere Ġgenerated Ġon Ġan Ġearly Ġ2015 Ġ13 - inch ĠRet ina ĠMacBook ĠPro Ġwith Ġa Ġ2 . 9 ĠGHz ĠCPU Ġ( Tur bo ĠBoost ed Ġto Ġ3 . 4 ĠGHz ) Ġand Ġ8 GB ĠRAM . ĠThe ĠC ++ Ġimplementation Ġin ĠB IB REF 17 Ġis Ġthe Ġfastest . ĠThe Ġimplementation Ġin ĠB IB REF 8 Ġmakes Ġuse Ġof Ġun - h ashed Ġfeature Ġrepresentations Ġwhich Ġcauses Ġit Ġto Ġbe Ġslower Ġthan Ġthe Ġproposed Ġsk learn Ġimplementation . ĠThe Ġexecution Ġperformance Ġof ĠB IB REF 23 Ġmight Ġimprove Ġby Ġa Ġfactor Ġof Ġfive Ġto Ġten Ġwhen Ġexecuted Ġon Ġa ĠGPU . </s> Conclusion </s> L ID Ġof Ġshort Ġtexts , Ġinformal Ġstyles Ġand Ġsimilar Ġlanguages Ġremains Ġa Ġdifficult Ġproblem Ġwhich Ġis Ġactively Ġbeing Ġresearched . ĠThe Ġproposed Ġalgorithm Ġwas Ġevaluated Ġon Ġthree Ġexisting Ġdatasets Ġand Ġcompared Ġto Ġthe Ġimplementations Ġof Ġthree Ġpublic ĠL ID Ġimplementations Ġas Ġwell Ġas Ġto Ġreported Ġresults Ġof Ġfour Ġother Ġalgorithms . ĠIt Ġperformed Ġwell Ġrelative Ġto Ġthe Ġother Ġmethods Ġbeating Ġtheir Ġresults . ĠHowever , Ġthe Ġperformance Ġis Ġdependent Ġon Ġthe Ġsupport Ġof Ġthe Ġlex icon . </s> We Ġwould Ġlike Ġto Ġinvestigate Ġthe Ġvalue Ġof Ġa Ġlex icon Ġin Ġa Ġproduction Ġsystem Ġand Ġhow Ġto Ġpossibly Ġmaintain Ġit Ġusing Ġself - super vised Ġlearning . ĠWe Ġare Ġinvestigating Ġthe Ġapplication Ġof Ġdeeper Ġlanguage Ġmodels Ġsome Ġof Ġwhich Ġhave Ġbeen Ġused Ġin Ġmore Ġrecent ĠDSL Ġshared Ġtasks . ĠWe Ġwould Ġalso Ġlike Ġto Ġinvestigate Ġdata Ġaug mentation Ġstrategies Ġto Ġreduce Ġthe Ġamount Ġof Ġtraining Ġdata Ġthat Ġis Ġrequired . </s> Further Ġresearch Ġopportunities Ġinclude Ġdata Ġharvesting , Ġbuilding Ġstandard ised Ġdatasets Ġand Ġshared Ġtasks Ġfor ĠSouth ĠAfrica Ġas Ġwell Ġas Ġthe Ġrest Ġof ĠAfrica . ĠIn Ġgeneral , Ġthe Ġsupport Ġfor Ġlanguage Ġcodes Ġthat Ġinclude Ġmore Ġlanguages Ġseems Ġto Ġbe Ġgrowing , Ġdiscover ability Ġof Ġresearch Ġis Ġimproving Ġand Ġpay w alls Ġseem Ġto Ġno Ġlonger Ġbe Ġa Ġbig Ġproblem Ġin Ġgetting Ġaccess Ġto Ġpublished Ġresearch .