POS, gender/number and stem POS
ABSTRACTIVE
Table TABREF17 lists the features that we used for CE recovery. We used Farasa to perform segmentation and POS tagging and to determine stem-templates BIBREF31. Farasa has a reported POS accuracy of 96% on the WikiNews dataset BIBREF31. Though the Farasa diacritizer utilizes a combination of some the features presented herein, namely segmentation, POS tagging, and stem templates, Farasa's SVM-ranking approach requires explicit specification of feature combinations (ex. $Prob(CE\Vert current\_word, prev\_word, prev\_CE)$). Manual exploration of the feature space is undesirable, and ideally we would want our learning algorithm to do so automatically. The flexibility of the DNN model allowed us to include many more surface level features such as affixes, leading and trailing characters in words and stems, and the presence of words in large gazetteers of named entities. As we show later, these additional features significantly lowered CEER. FLOAT SELECTED: Table 1. Features with examples and motivation.
2002.01207
Introduction </s> Modern ĠStandard ĠArabic Ġ( M SA ) Ġand ĠClassical ĠArabic Ġ( CA ) Ġhave Ġtwo Ġtypes Ġof Ġvow els , Ġnamely Ġlong Ġvow els , Ġwhich Ġare Ġexplicitly Ġwritten , Ġand Ġshort Ġvow els , Ġaka Ġdi ac rit ics , Ġwhich Ġare Ġtypically Ġomitted Ġin Ġwriting Ġbut Ġare Ġreintrodu ced Ġby Ġreaders Ġto Ġproperly Ġpronounce Ġwords . ĠSince Ġdi ac rit ics Ġdis amb ig uate Ġthe Ġsense Ġof Ġthe Ġwords Ġin Ġcontext Ġand Ġtheir Ġsynt actic Ġroles Ġin Ġsentences , Ġautomatic Ġdi ac ritic Ġrecovery Ġis Ġessential Ġfor Ġapplications Ġsuch Ġas Ġtext - to - speech Ġand Ġeducational Ġtools Ġfor Ġlanguage Ġlearners , Ġwho Ġmay Ġnot Ġknow Ġhow Ġto Ġproperly Ġverbal ize Ġwords . ĠDi ac rit ics Ġhave Ġtwo Ġtypes , Ġnamely : Ġcore - word Ġ( CW ) Ġdi ac rit ics , Ġwhich Ġare Ġinternal Ġto Ġwords Ġand Ġspecify Ġlex ical Ġselection ; Ġand Ġcase - end ings Ġ( CE ), Ġwhich Ġappear Ġon Ġthe Ġlast Ġletter Ġof Ġword Ġstems , Ġtypically Ġspecifying Ġtheir Ġsynt actic Ġrole . ĠFor Ġexample , Ġthe Ġword ĠâĢ ľ kt b âĢ Ŀ Ġ( Ù ĥ Øª Ø¨ >) Ġcan Ġhave Ġmultiple Ġdi ac rit ized Ġforms Ġsuch Ġas ĠâĢ ľ kat ab âĢ Ŀ Ġ( Ù ĥ Ùİ Øª Ùİ Ø¨ > ĠâĢĵ Ġmeaning ĠâĢ ľ he Ġwrote âĢ Ŀ ) ĠâĢ ľ k ut ub âĢ Ŀ Ġ( Ù ĥ Ù ı Øª Ù ı Ø¨ > ĠâĢĵ ĠâĢ ľ books âĢ Ŀ ). ĠWhile ĠâĢ ľ kat ab âĢ Ŀ Ġcan Ġonly Ġassume Ġone ĠCE , Ġnamely ĠâĢ ľ fat Ha âĢ Ŀ Ġ( âĢ ľ a âĢ Ŀ ), ĠâĢ ľ k ut ub âĢ Ŀ Ġcan Ġaccept Ġthe ĠCE s : ĠâĢ ľ dam ma âĢ Ŀ Ġ( âĢ ľ u âĢ Ŀ ) Ġ( nom inal ĠâĢĵ Ġex . Ġsubject ), ĠâĢ ľ a âĢ Ŀ Ġ( acc us ative ĠâĢĵ Ġex . Ġobject ), ĠâĢ ľ kas ra âĢ Ŀ Ġ( âĢ ľ i âĢ Ŀ ) Ġ( gen itive ĠâĢĵ Ġex . ĠPP Ġpredicate ), Ġor Ġtheir Ġnun ations . ĠThere Ġare Ġ14 Ġdi ac ritic Ġcombinations . ĠWhen Ġused Ġas ĠCE s , Ġthey Ġtypically Ġconvey Ġspecific Ġsynt actic Ġinformation , Ġnamely : Ġfat Ha ĠâĢ ľ a âĢ Ŀ Ġfor Ġaccus ative Ġnoun s , Ġpast Ġverbs Ġand Ġsub jun ctive Ġpresent Ġverbs ; Ġk as ra ĠâĢ ľ i âĢ Ŀ Ġfor Ġgen itive Ġnoun s ; Ġdam ma ĠâĢ ľ u âĢ Ŀ Ġfor Ġnom in ative Ġnoun s Ġand Ġindicative Ġpresent Ġverbs ; Ġsu kun ĠâĢ ľ o âĢ Ŀ Ġfor Ġj uss ive Ġpresent Ġverbs Ġand Ġimperative Ġverbs . ĠFat Ha , Ġk as ra Ġand Ġdam ma Ġcan Ġbe Ġpreceded Ġby Ġsh add a ĠâĢ ľ $ \ sim Ġ$ âĢ Ŀ Ġfor Ġgem ination Ġ( cons on ant Ġdoubling ) Ġand / or Ġconverted Ġto Ġnun ation Ġforms Ġfollowing Ġsome Ġgrammar Ġrules . ĠIn Ġaddition , Ġaccording Ġto ĠArabic Ġorth ography Ġand Ġphon ology , Ġsome Ġwords Ġtake Ġa Ġvirtual Ġ( null ) ĠâĢ ľ # âĢ Ŀ Ġmarker Ġwhen Ġthey Ġend Ġwith Ġcertain Ġcharacters Ġ( ex : Ġlong Ġvow els ). ĠThis Ġapplies Ġalso Ġto Ġall Ġnon - Arab ic Ġwords Ġ( ex : Ġpunct uation , Ġdigits , ĠLatin Ġwords , Ġetc .). ĠGenerally , Ġfunction Ġwords , Ġad verbs Ġand Ġforeign Ġnamed Ġentities Ġ( NE s ) Ġhave Ġset ĠCE s Ġ( su kun , Ġfat Ha Ġor Ġvirtual ). ĠSimilar Ġto Ġother ĠSem itic Ġlanguages , ĠArabic Ġallows Ġflexible ĠVerb - Subject - Object Ġas Ġwell Ġas ĠVerb - Object - Subject Ġconstructs ĠB IB REF 1 . ĠSuch Ġflexibility Ġcreates Ġinherent Ġambiguity , Ġwhich Ġis Ġresolved Ġby Ġdi ac rit ics Ġas Ġin ĠâĢ ľ r $ > $ Y ĠEm r ĠEly âĢ Ŀ Ġ( Ø± Ø £ Ù ī ĠØ ¹ Ùħ Ø± ĠØ ¹ ÙĦ ÙĬ > ĠOmar Ġsaw ĠAli / Ali Ġsaw ĠOmar ). ĠIn Ġthe Ġabsence Ġof Ġdi ac rit ics Ġit Ġis Ġnot Ġclear Ġwho Ġsaw Ġwhom . ĠSimilarly , Ġin Ġthe Ġsub - sent ence ĠâĢ ľ k An ĠAlm & tm r ĠAlt As E âĢ Ŀ Ġ( Ù ĥ Ø§ ÙĨ ĠØ§ÙĦ Ùħ Ø ¤ Øª Ùħ Ø± ĠØ§ÙĦ Øª Ø§Ø ³ Ø¹ > ), Ġif Ġthe Ġlast Ġword , Ġis Ġa Ġpredicate Ġof Ġthe Ġverb ĠâĢ ľ k An âĢ Ŀ , Ġthen Ġthe Ġsentence Ġwould Ġmean ĠâĢ ľ this Ġconference Ġwas Ġthe Ġninth âĢ Ŀ Ġand Ġwould Ġreceive Ġa Ġfat Ha Ġ( a ) Ġas Ġa Ġcase Ġending . ĠConversely , Ġif Ġit Ġwas Ġan Ġadjective Ġto Ġthe ĠâĢ ľ conference âĢ Ŀ , Ġthen Ġthe Ġsentence Ġwould Ġmean ĠâĢ ľ the Ġninth Ġconference Ġwas Ġ... âĢ Ŀ Ġand Ġwould Ġreceive Ġa Ġdam ma Ġ( u ) Ġas Ġa Ġcase Ġending . ĠThus , Ġa Ġconsideration Ġof Ġcontext Ġis Ġrequired Ġfor Ġproper Ġdis amb ig uation . ĠDue Ġto Ġthe Ġinter - word Ġdependence Ġof ĠCE s , Ġthey Ġare Ġtypically Ġharder Ġto Ġpredict Ġcompared Ġto Ġcore - word Ġdi ac rit ics ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , Ġwith ĠCE ER Ġof Ġstate - of - the - art Ġsystems Ġbeing Ġin Ġdouble Ġdigits Ġcompared Ġto Ġnearly Ġ3 % Ġfor Ġword - c ores . ĠSince Ġrecovering ĠCE s Ġis Ġakin Ġto Ġshallow Ġparsing ĠB IB REF 6 Ġand Ġrequires Ġmorph ological Ġand Ġsynt actic Ġprocessing , Ġit Ġis Ġa Ġdifficult Ġproblem Ġin ĠArabic ĠN LP . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġrecovering Ġboth ĠCW Ġdi ac rit ics Ġand ĠCE s . ĠWe Ġemploy Ġtwo Ġseparate ĠDeep ĠNeural ĠNetwork Ġ( D NN ) Ġarchitectures Ġfor Ġrecovering Ġboth Ġkinds Ġof Ġdi ac ritic Ġtypes . ĠWe Ġuse Ġcharacter - level Ġand Ġword - level Ġbid irection al ĠLong - Short ĠTerm ĠMemory Ġ( bi L ST M ) Ġbased Ġrecurrent Ġneural Ġmodels Ġfor ĠCW Ġdi ac ritic Ġand ĠCE Ġrecovery Ġrespectively . ĠWe Ġtrain Ġmodels Ġfor Ġboth ĠModern ĠStandard ĠArabic Ġ( M SA ) Ġand ĠClassical ĠArabic Ġ( CA ). ĠFor ĠCW Ġdi ac rit ics , Ġthe Ġmodel Ġis Ġinformed Ġusing Ġword Ġsegment ation Ġinformation Ġand Ġa Ġun ig ram Ġlanguage Ġmodel . ĠWe Ġalso Ġemploy Ġa Ġun ig ram Ġlanguage Ġmodel Ġto Ġperform Ġpost Ġcorrection Ġon Ġthe Ġmodel Ġoutput . ĠWe Ġachieve Ġword Ġerror Ġrates Ġfor ĠCW Ġdi ac rit ics Ġof Ġ2 . 9 % Ġand Ġ2 . 2 % Ġfor ĠM SA Ġand ĠCA . ĠThe ĠM SA Ġword Ġerror Ġrate Ġis Ġ6 % Ġlower Ġthan Ġthe Ġbest Ġresults Ġin Ġthe Ġliterature Ġ( the ĠR DI Ġdi ac rit izer ĠB IB REF 7 ). ĠThe ĠCE Ġmodel Ġis Ġtrained Ġwith Ġa Ġrich Ġset Ġof Ġsurface , Ġmorph ological , Ġand Ġsynt actic Ġfeatures . ĠThe Ġproposed Ġfeatures Ġwould Ġaid Ġthe Ġbi L ST M Ġmodel Ġin Ġcapturing Ġsynt actic Ġdependencies Ġindicated Ġby ĠPart - Of - Spe ech Ġ( POS ) Ġtags , Ġgender Ġand Ġnumber Ġfeatures , Ġmorph ological Ġpatterns , Ġand Ġaff ix es . ĠWe Ġshow Ġthat Ġour Ġmodel Ġachieves Ġa Ġcase Ġending Ġerror Ġrate Ġ( CE ER ) Ġof Ġ3 . 7 % Ġfor ĠM SA Ġand Ġ2 . 5 % Ġfor ĠCA . ĠFor ĠM SA , Ġthis ĠCE ER Ġis Ġmore Ġthan Ġ60 % Ġlower Ġthan Ġother Ġstate - of - the - art Ġsystems Ġsuch Ġas ĠFar asa Ġand Ġthe ĠR DI Ġdi ac rit izer , Ġwhich Ġare Ġtrained Ġon Ġthe Ġsame Ġdataset Ġand Ġachieve ĠCE ER s Ġof Ġ10 . 7 % Ġand Ġ14 . 4 % Ġrespectively . ĠThe Ġcontributions Ġof Ġthis Ġpaper Ġare Ġas Ġfollows : </s> We Ġemploy Ġa Ġcharacter - level ĠR NN Ġmodel Ġthat Ġis Ġinformed Ġusing Ġword Ġmorph ological Ġinformation Ġand Ġa Ġword Ġun ig ram Ġlanguage Ġmodel Ġto Ġrecover ĠCW Ġdi ac rit ics . ĠOur Ġmodel Ġbeats Ġthe Ġbest Ġstate - of - the - art Ġsystem Ġby Ġ6 % Ġfor ĠM SA . </s> We Ġintroduce Ġa Ġnew Ġfeature Ġrich ĠR NN - based ĠCE Ġrecovery Ġmodel Ġthat Ġachieves Ġerrors Ġrates Ġthat Ġare Ġ60 % Ġlower Ġthan Ġthe Ġcurrent Ġstate - of - the - art Ġfor ĠM SA . </s> We Ġexplore Ġthe Ġeffect Ġof Ġdifferent Ġfeatures , Ġwhich Ġmay Ġpotentially Ġbe Ġexploited Ġfor ĠArabic Ġparsing . </s> We Ġshow Ġthe Ġeffectiveness Ġof Ġour Ġapproach Ġfor Ġboth ĠM SA Ġand ĠCA . </s> Background </s> Aut omatic Ġdi ac rit ics Ġrestoration Ġhas Ġbeen Ġinvestigated Ġfor Ġmany Ġdifferent Ġlanguage Ġsuch Ġas ĠEuropean Ġlanguages Ġ( e . g . ĠRomanian ĠB IB REF 8 , ĠB IB REF 9 , ĠFrench ĠB IB REF 10 , Ġand ĠCroatian ĠB IB REF 11 ), ĠAfrican Ġlanguages Ġ( e . g . ĠYor ba ĠB IB REF 12 ), ĠSoutheast ĠAsian Ġlanguages Ġ( e . g . ĠVietnamese ĠB IB REF 13 ), ĠSem itic Ġlanguage Ġ( e . g . ĠArabic Ġand ĠHebrew ĠB IB REF 14 ), Ġand Ġmany Ġothers ĠB IB REF 15 . ĠFor Ġmany Ġlanguages , Ġdi ac ritic Ġ( or Ġaccent Ġrestoration ) Ġis Ġlimited Ġto Ġa Ġhandful Ġof Ġletters . ĠHowever , Ġfor ĠSem itic Ġlanguages , Ġdi ac ritic Ġrecovery Ġextends Ġto Ġmost Ġletters . ĠMany Ġgeneral Ġapproaches Ġhave Ġbeen Ġexplored Ġfor Ġthis Ġproblem Ġincluding Ġlingu istically Ġmotivated Ġrule - based Ġapproaches , Ġmachine Ġlearning Ġapproaches , Ġsuch Ġas ĠHidden ĠMark ov ĠModels Ġ( H MM ) ĠB IB REF 14 Ġand ĠCond itional ĠRandom ĠFields Ġ( CR F ) ĠB IB REF 16 , Ġand Ġlately Ġdeep Ġlearning Ġapproaches Ġsuch Ġas ĠArabic ĠB IB REF 17 , ĠB IB REF 18 , ĠB IB REF 19 , ĠSlov ak ĠB IB REF 20 , Ġand ĠYor ba ĠB IB REF 12 . ĠAside Ġfrom Ġrule - based Ġapproaches ĠB IB REF 21 , Ġdifferent Ġmethods Ġwere Ġused Ġto Ġrecover Ġdi ac rit ics Ġin ĠArabic Ġtext . ĠUsing Ġa Ġhidden ĠMark ov Ġmodel Ġ( H MM ) ĠB IB REF 14 , ĠB IB REF 22 Ġwith Ġan Ġinput Ġcharacter Ġsequence , Ġthe Ġmodel Ġattempts Ġto Ġfind Ġthe Ġbest Ġstate Ġsequence Ġgiven Ġprevious Ġobservations . ĠB IB REF 14 Ġreported Ġa Ġ14 % Ġword Ġerror Ġrate Ġ( WER ) Ġwhile ĠB IB REF 22 Ġachieved Ġa Ġ4 . 1 % Ġdi ac ritic Ġerror Ġrate Ġ( DER ) Ġon Ġthe ĠQuran Ġ( CA ). ĠB IB REF 23 Ġcombined Ġboth Ġmorph ological , Ġacoustic , Ġand Ġcontextual Ġfeatures Ġto Ġbuild Ġa Ġdi ac rit izer Ġtrained Ġon ĠFB IS Ġand ĠL DC ĠCall Home ĠE CA Ġcollections . ĠThey Ġreported Ġa Ġ9 % Ġ( DER ) Ġwithout ĠCE , Ġand Ġ28 % ĠD ER Ġwith ĠCE . ĠB IB REF 24 Ġemployed Ġa Ġcascade Ġof Ġa Ġfinite Ġstate Ġtrans ducers . ĠThe Ġcascade Ġstacked Ġa Ġword Ġlanguage Ġmodel Ġ( LM ), Ġa Ġchar ach ter ĠLM , Ġand Ġa Ġmorph ological Ġmodel . ĠThe Ġmodel Ġachieved Ġan Ġaccuracy Ġof Ġ7 . 33 % ĠW ER Ġwithout ĠCE Ġand Ġand Ġ23 . 61 % ĠW ER Ġwith ĠCE . ĠB IB REF 25 Ġemployed Ġa Ġmaximum Ġentropy Ġmodel Ġfor Ġsequence Ġclassification . ĠThe Ġsystem Ġwas Ġtrained Ġon Ġthe ĠL DC âĢ Ļ s ĠArabic ĠTree bank Ġ( AT B ) Ġand Ġevaluated Ġon Ġa Ġ600 Ġarticles Ġfrom ĠAn - N ahar ĠNewspaper Ġ( 340 K Ġwords ) Ġand Ġachieved Ġ5 . 5 % ĠD ER Ġand Ġ18 % ĠW ER Ġon Ġwords Ġwithout ĠCE . </s> BI B REF 26 Ġused Ġa Ġhybrid Ġapproach Ġthat Ġutilizes Ġthe Ġoutput Ġof ĠAl k hal il Ġmorph ological ĠAnaly zer ĠB IB REF 27 Ġto Ġgenerate Ġall Ġpossible Ġout Ġof Ġcontext Ġdi ac rit izations Ġof Ġa Ġword . ĠThen , Ġan ĠH MM Ġguesses Ġthe Ġcorrect Ġdi ac rit ized Ġform . ĠSimilarly , ĠMicrosoft ĠArabic ĠTool kit ĠServices Ġ( AT KS ) Ġdi ac rit izer ĠB IB REF 28 Ġuses Ġa Ġrule - based Ġmorph ological Ġanaly zer Ġthat Ġproduces Ġpossible Ġanalyses Ġand Ġan ĠH MM Ġin Ġconjunction Ġwith Ġrules Ġto Ġguess Ġthe Ġmost Ġlikely Ġanalysis . ĠThey Ġreport ĠW ER Ġof Ġ11 . 4 % Ġand Ġ4 . 4 % Ġwith Ġand Ġwithout ĠCE . ĠMAD AMI RA ĠB IB REF 29 Ġuses Ġa Ġcombinations Ġof Ġmorph o - sy nt actic Ġfeatures Ġto Ġrank Ġa Ġlist Ġof Ġpotential Ġanalyses Ġprovided Ġby Ġthe ĠBuck wal ter ĠArabic ĠMorph ological ĠAnaly zer Ġ( B AMA ) ĠB IB REF 30 . ĠAn ĠS VM Ġtrained Ġon ĠAT B Ġselects Ġthe Ġmost Ġprobable Ġanalysis , Ġincluding Ġthe Ġdi ac rit ized Ġform . ĠMAD AMI RA Ġachieves Ġ19 . 0 % Ġand Ġ6 . 7 % ĠW ER Ġwith Ġand Ġwithout ĠCE Ġrespectively ĠB IB REF 31 . ĠFar asa ĠB IB REF 31 Ġuses Ġan ĠH MM Ġto Ġguess ĠCW Ġdi ac rit ics Ġand Ġan ĠS VM - rank Ġbased Ġmodel Ġtrained Ġon Ġmorph ological Ġand Ġsynt actic Ġfeatures Ġto Ġguess ĠCE s . ĠFar asa Ġachieves ĠW ER Ġof Ġ12 . 8 % Ġand Ġ3 . 3 % Ġwith Ġand Ġwithout ĠCE s . </s> More Ġrecent Ġwork Ġemployed Ġdifferent Ġneural Ġarchitectures Ġto Ġmodel Ġthe Ġdi ac rit ization Ġproblem . ĠB IB REF 17 Ġused Ġa Ġbi L ST M Ġrecurrent Ġneural Ġnetwork Ġtrained Ġon Ġthe Ġsame Ġdataset Ġas ĠB IB REF 25 . ĠThey Ġexplored Ġone , Ġtwo Ġand Ġthree ĠBi L ST M Ġlayers Ġwith Ġ250 Ġnodes Ġin Ġeach Ġlayers , Ġachieving ĠW ER Ġof Ġ9 . 1 % Ġincluding ĠCE Ġon ĠAT B . ĠSimilar Ġarchitectures Ġwere Ġused Ġbut Ġachieved Ġlower Ġresults ĠB IB REF 7 , ĠB IB REF 32 . ĠB IB REF 33 Ġprovide Ġa Ġcomprehensive Ġsurvey Ġon ĠArabic Ġdi ac rit ization . ĠA Ġmore Ġrecent Ġsurvey Ġby ĠB IB REF 34 Ġconcluded Ġthat Ġreported Ġresults Ġare Ġoften Ġincom parable Ġdue Ġto Ġthe Ġusage Ġof Ġdifferent Ġtest Ġsets . ĠThey Ġconcluded Ġthat Ġa Ġlarge Ġun ig ram ĠLM Ġfor ĠCW Ġdi ac ritic Ġrecovery Ġis Ġcompetitive Ġwith Ġmany Ġof Ġthe Ġsystems Ġin Ġthe Ġliterature , Ġwhich Ġprompted Ġus Ġto Ġutilize Ġa Ġun ig ram Ġlanguage Ġmodel Ġfor Ġpost Ġcorrection . ĠAs Ġmentioned Ġearlier , Ġtwo Ġconclusions Ġcan Ġbe Ġdrawn , Ġnamely : Ġrestoring ĠCE s Ġis Ġmore Ġchallenging Ġthan ĠCW Ġdi ac ritic Ġrestoration ; Ġand Ġcombining Ġmultiple Ġfeatures Ġtypically Ġimproves ĠCE Ġrestoration . </s> In Ġthis Ġpaper , Ġwe Ġexpand Ġupon Ġthe Ġwork Ġin Ġthe Ġliterature Ġby Ġintroducing Ġfeature - rich ĠD NN Ġmodels Ġfor Ġrestoring Ġboth ĠCW Ġand ĠCE Ġdi ac rit ics . ĠWe Ġcompare Ġour Ġmodels Ġto Ġmultiple Ġsystems Ġon Ġthe Ġsame Ġtest Ġset . ĠWe Ġachieve Ġresults Ġthat Ġreduce Ġdi ac rit ization Ġerror Ġrates Ġby Ġmore Ġthan Ġhalf Ġcompared Ġto Ġthe Ġbest ĠS OTA Ġsystems . ĠWe Ġfurther Ġconduct Ġan Ġab lation Ġstudy Ġto Ġdetermine Ġthe Ġrelative Ġeffect Ġof Ġthe Ġdifferent Ġfeatures . ĠAs Ġfor ĠArabic , Ġit Ġis Ġa ĠSem itic Ġlanguage Ġwith Ġderiv ational Ġmorphology . ĠArabic Ġnoun s , Ġadject ives , Ġad verbs , Ġand Ġverbs Ġare Ġtypically Ġderived Ġfrom Ġa Ġclosed Ġset Ġof Ġ10 , 000 Ġroots Ġof Ġlength Ġ3 , Ġ4 , Ġor Ġrarely Ġ5 . ĠArabic Ġnoun s Ġand Ġverbs Ġare Ġderived Ġfrom Ġroots Ġby Ġapplying Ġtemplates Ġto Ġthe Ġroots Ġto Ġgenerate Ġstems . ĠSuch Ġtemplates Ġmay Ġcarry Ġinformation Ġthat Ġindicate Ġmorph ological Ġfeatures Ġof Ġwords Ġsuch ĠPOS Ġtag , Ġgender , Ġand Ġnumber . ĠFor Ġexample , Ġgiven Ġa Ġ3 - letter Ġroot Ġwith Ġ3 Ġconson ants ĠC CC , Ġa Ġvalid Ġtemplate Ġmay Ġbe ĠC w ACC Ġ, Ġwhere Ġthe Ġinf ix ĠâĢ ľ w A âĢ Ŀ Ġ( ÙĪ Ø§ >) Ġis Ġinserted , Ġthis Ġtemplate Ġtypically Ġindicates Ġan ĠArabic Ġbroken , Ġor Ġirregular , Ġplural Ġtemplate Ġfor Ġa Ġnoun Ġof Ġtemplate ĠC ACC Ġor ĠC ACC p Ġif Ġmasculine Ġor Ġfeminine Ġrespectively . ĠFurther , Ġstems Ġmay Ġaccept Ġprefix es Ġand / or Ġsuffix es Ġto Ġform Ġwords . ĠPre fixes Ġinclude Ġcoordinating Ġconj unctions , Ġdetermin er , Ġand Ġpre pos itions , Ġand Ġsuffix es Ġinclude Ġattached Ġpronouns Ġand Ġgender Ġand Ġnumber Ġmarkers . </s> Our ĠDi ac rit izer Ġ: :: ĠTraining Ġand ĠTest ĠCorpor a </s> For ĠM SA , Ġwe Ġacquired Ġthe Ġdi ac rit ized Ġcorpus Ġthat Ġwas Ġused Ġto Ġtrain Ġthe ĠR DI ĠB IB REF 7 Ġdi ac rit izer Ġand Ġthe ĠFar asa Ġdi ac rit izer ĠB IB REF 31 . ĠThe Ġcorpus Ġcontains Ġ9 . 7 M Ġtokens Ġwith Ġapproximately Ġ194 K Ġunique Ġsurface Ġforms Ġ( excluding Ġnumbers Ġand Ġpunct uation Ġmarks ). ĠThe Ġcorpus Ġcovers Ġmultiple Ġgenres Ġsuch Ġas Ġpolitics Ġand Ġsports Ġand Ġis Ġa Ġmix Ġof ĠM SA Ġand ĠCA . ĠThis Ġcorpus Ġis Ġconsiderably Ġlarger Ġthan Ġthe ĠArabic ĠTree bank ĠB IB REF 35 Ġand Ġis Ġmore Ġconsistent Ġin Ġits Ġdi ac rit ization . ĠFor Ġtesting , Ġwe Ġused Ġthe Ġfreely Ġavailable ĠWiki News Ġtest Ġset ĠB IB REF 31 , Ġwhich Ġis Ġcomposed Ġof Ġ70 ĠM SA ĠWiki News Ġarticles Ġ( 18 , 300 Ġtokens ) Ġand Ġevenly Ġcovers Ġa Ġvariety Ġof Ġgenres Ġincluding Ġpolitics , Ġeconomics , Ġhealth , Ġscience Ġand Ġtechnology , Ġsports , Ġarts Ġand Ġculture . </s> For ĠCA , Ġwe Ġobtained Ġa Ġlarge Ġcollection Ġof Ġfully Ġdi ac rit ized Ġclassical Ġtexts Ġ( 2 . 7 M Ġtokens ) Ġfrom Ġa Ġbook Ġpublisher , Ġand Ġwe Ġheld - out Ġa Ġsmall Ġsubset Ġof Ġ5 , 000 Ġsentences Ġ( approximately Ġ400 k Ġwords ) Ġfor Ġtesting . ĠThen , Ġwe Ġused Ġthe Ġremaining Ġsentences Ġto Ġtrain Ġthe ĠCA Ġmodels . </s> Our ĠDi ac rit izer Ġ: :: ĠCore ĠWord ĠDi ac rit ization Ġ: :: ĠFeatures . </s> Arab ic Ġwords Ġare Ġtypically Ġderived Ġfrom Ġa Ġlimited Ġset Ġof Ġroots Ġby Ġfitting Ġthem Ġinto Ġso - called Ġstem - tem plates Ġ( producing Ġstems ) Ġand Ġmay Ġaccept Ġa Ġvariety Ġof Ġprefix es Ġand Ġsuffix es Ġsuch Ġas Ġpre pos itions , Ġdetermin ers , Ġand Ġpronouns Ġ( producing Ġwords ). ĠWord Ġstems Ġspecify Ġthe Ġlex ical Ġselection Ġand Ġare Ġtypically Ġunaffected Ġby Ġthe Ġattached Ġaff ix es . ĠWe Ġused Ġ4 Ġfeature Ġtypes , Ġnamely : </s> CHAR : Ġthe Ġcharacters . </s> SE G : Ġthe Ġposition Ġof Ġthe Ġcharacter Ġin Ġa Ġword Ġsegment . ĠFor Ġexample , Ġgiven Ġthe Ġword ĠâĢ ľ w Al kt Ab âĢ Ŀ Ġ( ÙĪ Ø§ÙĦ Ù ĥ Øª Ø§Ø ¨ > Ġand Ġthe Ġbook / writers ), Ġwhich Ġis Ġcomposed Ġof Ġ3 Ġsegments ĠâĢ ľ w + Al + kt Ab âĢ Ŀ Ġ( ÙĪ + Ø§ÙĦ + Ù ĥ Øª Ø§Ø ¨ > ). ĠLetters Ġwere Ġmarked Ġas ĠâĢ ľ B âĢ Ŀ Ġif Ġthey Ġbegin Ġa Ġsegment , ĠâĢ ľ M âĢ Ŀ Ġif Ġthey Ġare Ġin Ġthe Ġmiddle Ġof Ġa Ġsegment , ĠâĢ ľ E âĢ Ŀ Ġif Ġthey Ġend Ġa Ġsegment , Ġand ĠâĢ ľ S âĢ Ŀ Ġif Ġthey Ġare Ġsingle Ġletter Ġsegments . ĠSo Ġfor ĠâĢ ľ w + Al + kt Ab âĢ Ŀ , Ġthe Ġcorresponding Ġcharacter Ġpositions Ġare ĠâĢ ľ S + BE + BM ME âĢ Ŀ . ĠWe Ġused ĠFar asa Ġto Ġperform Ġsegment ation , Ġwhich Ġhas Ġa Ġreported Ġsegment ation Ġaccuracy Ġof Ġ99 % Ġon Ġthe ĠWiki News Ġdataset ĠB IB REF 36 . </s> PR IOR : Ġdi ac rit ics Ġseen Ġin Ġthe Ġtraining Ġset Ġper Ġsegment . ĠSince Ġwe Ġused Ġa Ġcharacter Ġlevel Ġmodel , Ġthis Ġfeature Ġinformed Ġthe Ġmodel Ġwith Ġword Ġlevel Ġinformation . ĠFor Ġexample , Ġthe Ġword ĠâĢ ľ kt Ab âĢ Ŀ Ġ( Ù ĥ Øª Ø§Ø ¨ >) Ġwas Ġobserved Ġto Ġhave Ġtwo Ġdi ac rit ized Ġforms Ġin Ġthe Ġtraining Ġset , Ġnamely ĠâĢ ľ k ita Ab âĢ Ŀ Ġ( Ù ĥ ÙĲ Øª Ùİ Ø§Ø ¨ > ĠâĢĵ Ġbook ) Ġand ĠâĢ ľ k ut $ \ sim Ġ$ a Ab âĢ Ŀ Ġ( Ù ĥ Ù ı Øª Ù ĳ Ùİ Ø§Ø ¨ > ĠâĢĵ Ġwriters ). ĠThe Ġfirst Ġletter Ġin Ġthe Ġword Ġ( âĢ ľ k âĢ Ŀ ) Ġaccepted Ġthe Ġdi ac rit ics ĠâĢ ľ i âĢ Ŀ Ġand ĠâĢ ľ u âĢ Ŀ . ĠThus Ġgiven Ġa Ġbinary Ġvector Ġrepresenting Ġwhether Ġa Ġcharacter Ġis Ġallowed Ġto Ġassume Ġany Ġof Ġthe Ġeight Ġprimitive ĠArabic Ġdi ac ritic Ġmarks Ġ( a , Ġi , Ġu , Ġo , ĠK , ĠN , ĠF , Ġand Ġ$\ sim Ġ$ Ġin Ġorder ), Ġthe Ġfirst Ġletter Ġwould Ġbe Ġgiven Ġthe Ġfollowing Ġvector ĠâĢ ľ 01 100 000 âĢ Ŀ . ĠIf Ġa Ġword Ġsegment Ġwas Ġnever Ġobserved Ġduring Ġtraining , Ġthe Ġvector Ġfor Ġall Ġletters Ġtherein Ġwould Ġbe Ġset Ġto Ġ111 11 111 . ĠThis Ġfeature Ġborrow s Ġinformation Ġfrom ĠH MM Ġmodels , Ġwhich Ġhave Ġbeen Ġfairly Ġsuccessful Ġin Ġdi ac rit izing Ġword Ġcores . </s> C ASE : Ġwhether Ġthe Ġletter Ġexpects Ġa Ġcore Ġword Ġdi ac ritic Ġor Ġa Ġcase Ġending . ĠCase Ġendings Ġare Ġplaced Ġon Ġonly Ġone Ġletter Ġin Ġa Ġword , Ġwhich Ġmay Ġor Ġmay Ġnot Ġbe Ġthe Ġlast Ġletter Ġin Ġthe Ġword . ĠThis Ġis Ġa Ġbinary Ġfeature . </s> Our ĠDi ac rit izer Ġ: :: ĠCore ĠWord ĠDi ac rit ization Ġ: :: ĠD NN ĠModel . </s> Using Ġa ĠD NN Ġmodel , Ġparticularly Ġwith Ġa Ġbi L ST M ĠB IB REF 37 , Ġis Ġadvantageous Ġbecause Ġthe Ġmodel Ġautomatically Ġexplores Ġthe Ġspace Ġof Ġfeature Ġcombinations Ġand Ġis Ġable Ġto Ġcapture Ġdistant Ġdependencies . ĠA Ġnumber Ġof Ġstudies Ġhave Ġexplored Ġvarious Ġbi L ST M Ġarchitectures ĠB IB REF 17 , ĠB IB REF 7 , ĠB IB REF 32 Ġincluding Ġstacked Ġbi L ST Ms Ġconfirming Ġtheir Ġeffectiveness . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 14 , Ġwe Ġemployed Ġa Ġcharacter - based Ġbi L ST M Ġmodel Ġwith Ġassociated Ġfeatures Ġfor Ġeach Ġcharacter . ĠEvery Ġinput Ġcharacter Ġhad Ġan Ġassociated Ġlist Ġof Ġ$ m $ Ġfeatures , Ġand Ġwe Ġtrained Ġrandomly Ġinitialized Ġembed d ings Ġof Ġsize Ġ50 Ġfor Ġeach Ġfeature . ĠThen , Ġwe Ġconc aten ated Ġthe Ġfeature Ġembed d ings Ġvectors Ġcreating Ġan Ġ$ m \ times Ġ50 $ Ġvector Ġfor Ġeach Ġcharacter , Ġwhich Ġwas Ġfed Ġinto Ġthe Ġbi L ST M Ġlayer Ġof Ġlength Ġ100 . ĠThe Ġoutput Ġof Ġthe Ġbi L ST M Ġlayer Ġwas Ġfed Ġdirectly Ġinto Ġa Ġdense Ġlayer Ġof Ġsize Ġ100 . ĠWe Ġused Ġearly Ġstopping Ġwith Ġpatience Ġof Ġ5 Ġepoch s , Ġa Ġlearning Ġrate Ġof Ġ0 . 001 , Ġa Ġbatch Ġsize Ġof Ġ256 , Ġand Ġan ĠAdam ax Ġoptim izer . ĠThe Ġinput Ġwas Ġthe Ġcharacter Ġsequence Ġin Ġa Ġsentence Ġwith Ġwords Ġbeing Ġseparated Ġby Ġword Ġboundary Ġmarkers Ġ( WB ), Ġand Ġwe Ġset Ġthe Ġmaximum Ġsentence Ġlength Ġto Ġ1 , 250 Ġcharacters . </s> Our ĠDi ac rit izer Ġ: :: ĠCase ĠEnding ĠDi ac rit ization Ġ: :: ĠFeatures . </s> Table ĠT AB REF 17 Ġlists Ġthe Ġfeatures Ġthat Ġwe Ġused Ġfor ĠCE Ġrecovery . ĠWe Ġused ĠFar asa Ġto Ġperform Ġsegment ation Ġand ĠPOS Ġtagging Ġand Ġto Ġdetermine Ġstem - tem plates ĠB IB REF 31 . ĠFar asa Ġhas Ġa Ġreported ĠPOS Ġaccuracy Ġof Ġ96 % Ġon Ġthe ĠWiki News Ġdataset ĠB IB REF 31 . ĠThough Ġthe ĠFar asa Ġdi ac rit izer Ġutilizes Ġa Ġcombination Ġof Ġsome Ġthe Ġfeatures Ġpresented Ġherein , Ġnamely Ġsegment ation , ĠPOS Ġtagging , Ġand Ġstem Ġtemplates , ĠFar asa 's ĠS VM - ranking Ġapproach Ġrequires Ġexplicit Ġspecification Ġof Ġfeature Ġcombinations Ġ( ex . Ġ$ Pro b ( CE \ Vert Ġcurrent \ _ word , Ġprev \ _ word , Ġprev \ _ CE ) $ ). ĠManual Ġexploration Ġof Ġthe Ġfeature Ġspace Ġis Ġundesirable , Ġand Ġideally Ġwe Ġwould Ġwant Ġour Ġlearning Ġalgorithm Ġto Ġdo Ġso Ġautomatically . ĠThe Ġflexibility Ġof Ġthe ĠD NN Ġmodel Ġallowed Ġus Ġto Ġinclude Ġmany Ġmore Ġsurface Ġlevel Ġfeatures Ġsuch Ġas Ġaff ix es , Ġleading Ġand Ġtrailing Ġcharacters Ġin Ġwords Ġand Ġstems , Ġand Ġthe Ġpresence Ġof Ġwords Ġin Ġlarge Ġg az ette ers Ġof Ġnamed Ġentities . ĠAs Ġwe Ġshow Ġlater , Ġthese Ġadditional Ġfeatures Ġsignificantly Ġlowered ĠCE ER . </s> Our ĠDi ac rit izer Ġ: :: ĠCase ĠEnding ĠDi ac rit ization Ġ: :: ĠD NN ĠModel </s> Figure ĠFIG REF 19 Ġshows Ġthe Ġarchitecture Ġof Ġour ĠD NN Ġalgorithm . ĠEvery Ġinput Ġword Ġhad Ġan Ġassociated Ġlist Ġof Ġ$ n $ Ġfeatures , Ġand Ġwe Ġtrained Ġrandomly Ġinitialized Ġembed d ings Ġof Ġsize Ġ100 Ġfor Ġeach Ġfeature . ĠThen , Ġwe Ġconc aten ated Ġthe Ġfeature Ġembed d ings Ġvectors Ġcreating Ġan Ġ$ n \ times Ġ100 $ Ġvector Ġfor Ġeach Ġword . ĠWe Ġfed Ġthese Ġvectors Ġinto Ġa Ġbi L ST M Ġlayer Ġof Ġ100 Ġdimensions Ġafter Ġapplying Ġa Ġdrop out Ġof Ġ75 %, Ġwhere Ġdrop out Ġbehaves Ġlike Ġa Ġregular l izer Ġto Ġavoid Ġover fitting ĠB IB REF 40 . ĠWe Ġconducted Ġside Ġexperiments Ġwith Ġlower Ġdrop out Ġrates , Ġbut Ġthe Ġhigher Ġdrop out Ġrate Ġworked Ġbest . ĠThe Ġoutput Ġof Ġthe Ġbi L ST M Ġlayer Ġwas Ġfed Ġinto Ġa Ġ100 Ġdimensional Ġdense Ġlayer Ġwith Ġ15 % Ġdrop out Ġand Ġsoft max Ġactivation . ĠWe Ġconducted Ġside Ġexperiments Ġwhere Ġwe Ġadded Ġadditional Ġbi L ST M Ġlayers Ġand Ġreplaced Ġsoft max Ġwith Ġa Ġconditional Ġrandom Ġfield Ġlayer , Ġbut Ġwe Ġdid Ġnot Ġobserve Ġimprovements . ĠThus , Ġwe Ġopted Ġfor Ġa Ġsimpler Ġmodel . ĠWe Ġused Ġa Ġvalidation Ġset Ġto Ġdetermine Ġoptimal Ġparameters Ġsuch Ġas Ġdrop out Ġrate . ĠAgain , Ġwe Ġused Ġthe ĠâĢ ľ Adam ax âĢ Ŀ Ġoptim izer Ġwith Ġcateg orical Ġcross Ġentropy Ġloss Ġand Ġa Ġlearning Ġrate Ġof Ġ0 . 001 . ĠWe Ġalso Ġapplied Ġearly Ġstopping Ġwith Ġpatience Ġof Ġup Ġto Ġ5 Ġconsecutive Ġepoch s Ġwithout Ġimprovement . </s> Exper iments Ġand ĠResults Ġ: :: ĠCore ĠWord Ġ: :: ĠExperimental ĠSetup </s> For Ġall Ġthe Ġexperiments Ġconducted Ġherein , Ġwe Ġused Ġthe ĠKer as Ġtool kit ĠB IB REF 41 Ġwith Ġa ĠT ensor Flow Ġbackend ĠB IB REF 42 . ĠWe Ġused Ġthe Ġentirety Ġof Ġthe Ġtraining Ġset Ġas Ġinput , Ġand Ġwe Ġinstructed ĠKer as Ġto Ġuse Ġ5 % Ġof Ġthe Ġdata Ġfor Ġtuning Ġ( valid ation ). ĠWe Ġincluded Ġthe ĠCASE Ġfeature , Ġwhich Ġspecifies Ġwhether Ġthe Ġletter Ġaccepts Ġa Ġnormal Ġdi ac ritic Ġor Ġcase Ġending , Ġin Ġall Ġour Ġsetups . ĠWe Ġconducted Ġmultiple Ġexperiment Ġusing Ġdifferent Ġfeatures , Ġnamely : </s> CHAR : ĠThis Ġis Ġour Ġbaseline Ġsetup Ġwhere Ġwe Ġonly Ġused Ġthe Ġcharacters Ġas Ġfeatures . </s> CHAR + SE G : ĠThis Ġtakes Ġthe Ġcharacters Ġand Ġtheir Ġsegment ation Ġinformation Ġas Ġfeatures . </s> CHAR + PR IOR : ĠThis Ġtakes Ġthe Ġcharacters Ġand Ġtheir Ġthe Ġobserved Ġdi ac rit ized Ġforms Ġin Ġthe Ġtraining Ġset . </s> All : ĠThis Ġsetup Ġincludes Ġall Ġthe Ġfeatures . </s> We Ġalso Ġoptionally Ġemployed Ġpost Ġcorrection . ĠFor Ġwords Ġthat Ġwere Ġseen Ġin Ġtraining , Ġif Ġthe Ġmodel Ġproduced Ġa Ġdi ac rit ized Ġform Ġthat Ġwas Ġnot Ġseen Ġin Ġthe Ġtraining Ġdata , Ġwe Ġassumed Ġit Ġwas Ġan Ġerror Ġand Ġreplaced Ġit Ġwith Ġthe Ġmost Ġfrequently Ġobserved Ġdi ac rit ized Ġform Ġ( using Ġa Ġun ig ram Ġlanguage Ġmodel ). ĠWe Ġreport Ġtwo Ġerror Ġrates , Ġnamely ĠW ER Ġ( at Ġword Ġlevel ) Ġand ĠD ER Ġ( at Ġcharacter Ġlevel ). ĠWe Ġused Ġrelaxed Ġscoring Ġwhere Ġwe Ġassumed Ġan Ġempty Ġcase Ġto Ġbe Ġequivalent Ġto Ġsu kun , Ġand Ġwe Ġremoved Ġdefault Ġdi ac rit ics ĠâĢĵ Ġfat Ha Ġfollowed Ġby Ġale f , Ġk as ra Ġfollowed Ġby Ġya , Ġand Ġdam ma Ġfollowed Ġby Ġwa . ĠUsing Ġsuch Ġscoring Ġwould Ġallow Ġto Ġcompare Ġto Ġother Ġsystems Ġin Ġthe Ġliterature Ġthat Ġmay Ġuse Ġdifferent Ġdi ac rit ization Ġconventions . </s> Exper iments Ġand ĠResults Ġ: :: ĠCore ĠWord Ġ: :: ĠResults Ġand ĠError Ġanalysis </s> For Ġtesting , Ġwe Ġused Ġthe Ġaforementioned ĠWiki News Ġdataset Ġto Ġtest Ġthe ĠM SA Ġdi ac rit izer Ġand Ġthe Ġheld - out Ġ5 , 000 Ġsentences Ġfor ĠCA . ĠTable ĠT AB REF 27 Ġshows ĠW ER Ġand ĠD ER Ġresults Ġusing Ġdifferent Ġfeatures Ġwith Ġand Ġwithout Ġpost Ġcorrection . </s> Exper iments Ġand ĠResults Ġ: :: ĠCore ĠWord Ġ: :: ĠResults Ġand ĠError Ġanalysis Ġ: :: ĠM SA ĠResults : </s> For ĠM SA , Ġthough Ġthe ĠCHAR + PR IOR Ġfeature Ġled Ġto Ġworse Ġresults Ġthan Ġusing ĠCHAR Ġalone , Ġthe Ġresults Ġshow Ġthat Ġcombining Ġall Ġthe Ġfeatures Ġachieved Ġthe Ġbest Ġresults . ĠMoreover , Ġpost Ġcorrection Ġimproved Ġresults Ġoverall . ĠWe Ġcompare Ġour Ġresults Ġto Ġfive Ġother Ġsystems , Ġnamely ĠFar asa ĠB IB REF 31 , ĠMAD AMI RA ĠB IB REF 29 , ĠR DI Ġ( R ash wan Ġet Ġal ., Ġ2015 ), ĠMIT Ġ( Bel ink ow Ġand ĠGlass , Ġ2015 ), Ġand ĠMicrosoft ĠATK S ĠB IB REF 28 . ĠTable ĠT AB REF 34 Ġcompares Ġour Ġsystem Ġwith Ġothers Ġin Ġthe Ġaforementioned Ġsystems . ĠAs Ġthe Ġresults Ġshow , Ġour Ġresults Ġbeat Ġthe Ġcurrent Ġstate - of - the - art . </s> For Ġerror Ġanalysis , Ġwe Ġanalyzed Ġall Ġthe Ġerrors Ġ( 5 27 Ġerrors ). ĠThe Ġerrors Ġtypes Ġalong Ġwith Ġexamples Ġof Ġeach Ġare Ġshown Ġin ĠTable ĠT AB REF 29 . ĠThe Ġmost Ġprominent Ġerror Ġtype Ġarises Ġfrom Ġthe Ġselection Ġof Ġa Ġvalid Ġdi ac rit ized Ġform Ġthat Ġdoes Ġnot Ġmatch Ġthe Ġcontext Ġ( 40 . 8 %). ĠPerhaps , Ġincluding ĠPOS Ġtags Ġas Ġa Ġfeature Ġor Ġaugment ing Ġthe ĠPR IOR Ġfeature Ġwith ĠPOS Ġtag Ġinformation Ġand Ġa Ġbig ram Ġlanguage Ġmodel Ġmay Ġreduce Ġthe Ġerror Ġrate Ġfurther . ĠThe Ġsecond Ġmost Ġcommon Ġerror Ġis Ġdue Ġto Ġtransl iter ated Ġforeign Ġwords Ġincluding Ġforeign Ġnamed Ġentities Ġ( 23 . 5 %). ĠSuch Ġwords Ġwere Ġnot Ġobserved Ġduring Ġtraining . ĠFurther , ĠArabic ĠNamed Ġentities Ġaccount Ġfor Ġ10 . 6 % Ġof Ġthe Ġerrors , Ġwhere Ġthey Ġwere Ġeither Ġnot Ġseen Ġin Ġtraining Ġor Ġthey Ġshare Ġidentical Ġnon - d iac rit ized Ġforms Ġwith Ġother Ġwords . ĠPerhaps , Ġbuilding Ġlarger Ġg az ette ers Ġof Ġdi ac rit ized Ġnamed Ġentities Ġmay Ġresolve ĠNE Ġrelated Ġerrors . ĠIn Ġ10 . 8 % Ġof Ġthe Ġcases , Ġthe Ġdi ac rit izer Ġproduced Ġin Ġcompletely Ġincorrect Ġdi ac rit ized Ġforms . ĠIn Ġsome Ġthe Ġcases Ġ( 9 . 1 %), Ġthough Ġthe Ġdi ac rit izer Ġproduced Ġa Ġform Ġthat Ġis Ġdifferent Ġfrom Ġthe Ġreference , Ġboth Ġforms Ġwere Ġin Ġfact Ġcorrect . ĠMost Ġof Ġthese Ġcases Ġwere Ġdue Ġto Ġvariations Ġin Ġdi ac rit ization Ġconventions Ġ( ex . ĠâĢ ľ bare Ġale f âĢ Ŀ Ġ( A ) Ġat Ġstart Ġof Ġa Ġword Ġreceiving Ġa Ġdi ac ritic Ġor Ġnot ). ĠOther Ġcases Ġinclude Ġforeign Ġwords Ġand Ġsome Ġwords Ġwhere Ġboth Ġdi ac rit ized Ġforms Ġare Ġequally Ġvalid . </s> Exper iments Ġand ĠResults Ġ: :: ĠCore ĠWord Ġ: :: ĠResults Ġand ĠError Ġanalysis Ġ: :: ĠCA ĠResults : </s> For ĠCA Ġresults , Ġthe ĠCHAR + SE G Ġand ĠCHAR + PR IOR Ġperformed Ġbetter Ġthan Ġusing Ġcharacters Ġalone Ġwith ĠCHAR + PR IOR Ġperforming Ġbetter Ġthan ĠCHAR + SE G . ĠAs Ġin Ġthe Ġcase Ġwith ĠM SA , Ġcombining Ġall Ġthe Ġfeatures Ġled Ġto Ġthe Ġbest Ġresults . ĠPost Ġcorrection Ġhad Ġa Ġsignificantly Ġlarger Ġpositive Ġimpact Ġon Ġresults Ġcompared Ġto Ġwhat Ġwe Ġobserved Ġfor ĠM SA . ĠThis Ġmay Ġindicate Ġthat Ġwe Ġneed Ġa Ġlarger Ġtraining Ġset . ĠThe Ġbest ĠW ER Ġthat Ġwe Ġachieved Ġfor ĠCW Ġdi ac rit ics Ġwith Ġpost Ġcorrections Ġis Ġ2 . 2 %. ĠSince Ġwe Ġdid Ġnot Ġhave Ġaccess Ġto Ġany Ġpublicly Ġavailable Ġsystem Ġthat Ġis Ġtuned Ġfor ĠCA , Ġwe Ġcompared Ġour Ġbest Ġsystem Ġto Ġusing Ġour Ġbest ĠM SA Ġsystem Ġto Ġdi ac rit ize Ġthe ĠCA Ġtest Ġset , Ġand Ġthe ĠM SA Ġdi ac rit izer Ġproduced Ġsignificantly Ġlower Ġresults Ġwith Ġa ĠW ER Ġof Ġ8 . 5 % Ġ( see ĠTable ĠT AB REF 34 ). ĠThis Ġhighlights Ġthe Ġlarge Ġdifference Ġbetween ĠM SA Ġand ĠCA Ġand Ġthe Ġneed Ġfor Ġsystems Ġthat Ġare Ġspecifically Ġtuned Ġfor Ġboth . </s> We Ġrandomly Ġselected Ġand Ġanalyzed Ġ500 Ġerrors Ġ( 5 . 2 % Ġof Ġthe Ġerrors ). ĠThe Ġerrors Ġtypes Ġalong Ġwith Ġexamples Ġof Ġeach Ġare Ġshown Ġin ĠTable ĠT AB REF 33 . ĠThe Ġtwo Ġmost Ġcommon Ġerrors Ġinvolve Ġthe Ġsystem Ġproducing Ġcompletely Ġcorrect Ġdi ac rit ized Ġforms Ġ( 38 . 8 %) Ġor Ġcorrect Ġforms Ġthat Ġdon 't Ġmatch Ġthe Ġcontext Ġ( 31 . 4 %). ĠThe Ġrelatively Ġhigher Ġpercentage Ġof Ġcompletely Ġincorrect Ġguesses , Ġcompared Ġto ĠM SA , Ġmay Ġpoint Ġto Ġthe Ġhigher Ġlex ical Ġdiversity Ġof Ġclassical ĠArabic . ĠAs Ġfor ĠM SA , Ġwe Ġsuspect Ġthat Ġadding Ġadditional ĠPOS Ġinformation Ġand Ġemploying Ġa Ġword Ġbig ram Ġto Ġconst rain Ġthe ĠPR IOR Ġfeature Ġmay Ġhelp Ġreduce Ġselection Ġerrors . ĠAnother Ġprominent Ġerror Ġis Ġrelated Ġto Ġthe Ġdi ac rit ics Ġthat Ġappear Ġon Ġattached Ġsuffix es , Ġparticularly Ġpronouns , Ġwhich Ġdepend Ġon Ġthe Ġchoice Ġof Ġcase Ġending Ġ( 13 . 2 %). ĠErrors Ġdue Ġto Ġnamed Ġentities Ġare Ġslightly Ġfewer Ġthan Ġthose Ġseen Ġfor ĠM SA Ġ( 8 . 8 %). ĠA Ġnoticeable Ġnumber Ġof Ġmism atches Ġbetween Ġthe Ġguess Ġand Ġthe Ġreference Ġare Ġdue Ġto Ġpartial Ġdi ac rit ization Ġof Ġthe Ġreference Ġ( 4 . 4 %). ĠWe Ġplan Ġto Ġconduct Ġan Ġextra Ġround Ġof Ġchecks Ġon Ġthe Ġtest Ġset . </s> Exper iments Ġand ĠResults Ġ: :: ĠCase ĠEnding Ġ: :: ĠExperimental ĠSetup </s> We Ġconducted Ġmultiple Ġexperiments Ġto Ġdetermine Ġthe Ġrelative Ġeffect Ġof Ġthe Ġdifferent Ġfeatures Ġas Ġfollows : </s> word : ĠThis Ġis Ġour Ġbaseline Ġsetup , Ġwhich Ġuses Ġword Ġsurface Ġforms Ġonly . </s> word - surface : ĠThis Ġsetup Ġuses Ġthe Ġword Ġsurface Ġforms , Ġstems , Ġprefix es , Ġand Ġsuffix es Ġ( including Ġnoun Ġsuffix es ). ĠThis Ġsim ulates Ġthe Ġcase Ġwhen Ġno ĠPOS Ġtagging Ġinformation Ġis Ġavailable . </s> word - POS : ĠThis Ġincludes Ġthe Ġword Ġsurface Ġform Ġand ĠPOS Ġinformation , Ġincluding Ġgender Ġand Ġnumber Ġof Ġstems , Ġprefix es , Ġand Ġsuffix es . </s> word - morph : ĠThis Ġincludes Ġwords Ġand Ġtheir Ġstem Ġtemplates Ġto Ġcapture Ġmorph ological Ġpatterns . </s> word - surface - POS - morph : ĠThis Ġsetup Ġuses Ġall Ġthe Ġfeatures Ġ( surface , ĠPOS , Ġand Ġmorph ological ). </s> all - misc : ĠThis Ġuses Ġall Ġfeatures Ġplus Ġword Ġand Ġstem Ġleading Ġand Ġtrailing Ġcharacter Ġun ig rams Ġand Ġbig rams Ġin Ġaddition Ġto Ġsu kun Ġwords Ġand Ġnamed Ġentities . </s> For Ġtesting ĠM SA , Ġwe Ġused Ġthe Ġaforementioned ĠWiki News Ġdataset . ĠAgain , Ġwe Ġcompared Ġour Ġresults Ġto Ġfive Ġother Ġsystems , Ġnamely ĠFar asa ĠB IB REF 31 , ĠMAD AMI RA ĠB IB REF 29 , ĠR DI Ġ( R ash wan Ġet Ġal ., Ġ2015 ), ĠMIT Ġ( Bel ink ow Ġand ĠGlass , Ġ2015 ), Ġand ĠMicrosoft ĠATK S ĠB IB REF 28 . ĠFor ĠCA Ġtesting , Ġwe Ġused Ġthe Ġ5 , 000 Ġsentences Ġthat Ġwe Ġset Ġaside . ĠAgain , Ġwe Ġcompared Ġto Ġour Ġbest ĠM SA Ġsystem . </s> Exper iments Ġand ĠResults Ġ: :: ĠCase ĠEnding Ġ: :: ĠResults Ġand ĠError ĠAnalysis </s> Table ĠT AB REF 45 Ġlists Ġthe Ġresults Ġof Ġour Ġsetups Ġcompared Ġto Ġother Ġsystems . </s> Exper iments Ġand ĠResults Ġ: :: ĠCase ĠEnding Ġ: :: ĠResults Ġand ĠError ĠAnalysis Ġ: :: ĠM SA ĠResults : </s> As Ġthe Ġresults Ġshow , Ġour Ġbaseline ĠD NN Ġsystem Ġoutper forms Ġall Ġstate - of - the - art Ġsystems . ĠFurther , Ġadding Ġmore Ġfeatures Ġyielded Ġbetter Ġresults Ġoverall . ĠSurface - level Ġfeatures Ġresulted Ġin Ġthe Ġmost Ġgain , Ġfollowed Ġby ĠPOS Ġtags , Ġand Ġlast ly Ġstem Ġtemplates . ĠFurther , Ġadding Ġhead Ġand Ġtail Ġcharacters Ġalong Ġwith Ġa Ġlist Ġof Ġsu kun Ġwords Ġand Ġnamed Ġentities Ġled Ġto Ġfurther Ġimprovement . ĠOur Ġproposed Ġfeature - rich Ġsystem Ġhas Ġa ĠCE ER Ġthat Ġis Ġapproximately Ġ61 % Ġlower Ġthan Ġany Ġof Ġthe Ġstate - of - the - art Ġsystems . </s> Figure ĠFIG REF 49 Ġshows ĠCE Ġdistribution Ġand Ġprediction Ġaccuracy . ĠFor Ġthe Ġfour Ġbasic Ġmarkers Ġk as ra , Ġfat Ha , Ġdam ma Ġand Ġsu kun , Ġwhich Ġappear Ġ27 %, Ġ14 %, Ġ9 % Ġand Ġ10 % Ġrespectively , Ġthe Ġsystem Ġhas ĠCE ER Ġof Ġ$\ sim Ġ$ 1 % Ġfor Ġeach . ĠDetect ing Ġthe Ġvirtual ĠCE Ġmark Ġis Ġa Ġfairly Ġeasy Ġtask . ĠAll Ġother ĠCE Ġmarkers Ġrepresent Ġ13 % Ġwith Ġalmost Ġnegligible Ġerrors . ĠTable ĠT AB REF 30 Ġlists Ġa Ġthorough Ġbreakdown Ġof Ġall Ġerrors Ġaccounting Ġfor Ġat Ġ1 % Ġof Ġthe Ġerrors Ġalong Ġwith Ġthe Ġmost Ġcommon Ġreasons Ġof Ġthe Ġerrors Ġand Ġexamples Ġillustrating Ġthese Ġreasons . ĠFor Ġexample , Ġthe Ġmost Ġcommon Ġerror Ġtype Ġinvolves Ġguessing Ġa Ġfat Ha Ġ( a ) Ġinstead Ġof Ġdam ma Ġ( u ) Ġor Ġvice Ġversa Ġ( 19 . 3 %). ĠThe Ġmost Ġcommon Ġreasons Ġfor Ġthis Ġerror Ġtype , Ġbased Ġon Ġinspecting Ġthe Ġerrors , Ġwere Ġdue Ġto : ĠPOS Ġerrors Ġ( ex . Ġa Ġword Ġis Ġtagged Ġas Ġa Ġverb Ġinstead Ġof Ġa Ġnoun ); Ġand Ġa Ġnoun Ġis Ġtreated Ġas Ġa Ġsubject Ġinstead Ġof Ġan Ġobject Ġor Ġvice Ġversa . ĠThe Ġtable Ġdetails Ġthe Ġrest Ġof Ġthe Ġerror Ġtypes . ĠOverall , Ġsome Ġof Ġthe Ġerrors Ġare Ġpotentially Ġfix able Ġusing Ġbetter ĠPOS Ġtagging , Ġimproved Ġdetection Ġof Ġnon - Arab ized Ġforeign Ġnames , Ġand Ġdetection Ġof Ġind e cl in ability . ĠHowever , Ġsome Ġerrors Ġare Ġmore Ġdifficult Ġand Ġrequire Ġgreater Ġunderstanding Ġof Ġsemantics Ġsuch Ġas Ġimproper Ġattachment , Ġincorrect Ġid afa , Ġand Ġconfusion Ġbetween Ġsubject Ġand Ġobject . ĠPerhaps , Ġsuch Ġsemantic Ġerrors Ġcan Ġbe Ġresolved Ġusing Ġparsing . </s> Exper iments Ġand ĠResults Ġ: :: ĠCase ĠEnding Ġ: :: ĠResults Ġand ĠError ĠAnalysis Ġ: :: ĠCA ĠResults : </s> The Ġresults Ġshow Ġthat Ġthe ĠPOS Ġtagging Ġfeatures Ġled Ġto Ġthe Ġmost Ġimprovements Ġfollowed Ġby Ġthe Ġsurface Ġfeatures . ĠComb ining Ġall Ġfeatures Ġled Ġto Ġthe Ġbest Ġresults Ġwith ĠW ER Ġof Ġ2 . 5 %. ĠAs Ġwe Ġsaw Ġfor ĠCW Ġdi ac rit ics , Ġusing Ġour Ġbest ĠM SA Ġsystem Ġto Ġdi ac rit ize ĠCA Ġled Ġto Ġsignificantly Ġlower Ġresults Ġwith ĠCE ER Ġof Ġ8 . 9 %. </s> Figure ĠFIG REF 50 Ġshows ĠCE Ġdistribution Ġand Ġprediction Ġaccuracy . ĠFor Ġthe Ġfour Ġbasic Ġmarkers Ġfat Ha , Ġk as ra , Ġdam ma Ġand Ġsu kun , Ġwhich Ġappear Ġ18 %, Ġ14 %, Ġ13 % Ġand Ġ8 % Ġrespectively , Ġthe Ġsystem Ġhas ĠCE ER Ġ$\ sim Ġ$ 0 . 5 % Ġfor Ġeach . ĠAgain , Ġdetecting Ġthe Ġvirtual ĠCE Ġmark Ġwas Ġa Ġfairly Ġeasy Ġtask . ĠAll Ġother ĠCE Ġmarkers Ġrepresenting Ġ20 % Ġhave Ġnegligible Ġerrors . </s> Table ĠT AB REF 31 Ġlists Ġall Ġthe Ġerror Ġtypes , Ġwhich Ġaccount Ġfor Ġat Ġleast Ġ1 % Ġof Ġthe Ġerrors , Ġalong Ġwith Ġtheir Ġmost Ġcommon Ġcauses Ġand Ġexplanatory Ġexamples . ĠThe Ġerror Ġtypes Ġare Ġsimilar Ġto Ġthose Ġobserved Ġfor ĠM SA . ĠSome Ġerrors Ġare Ġmore Ġsynt actic Ġand Ġmorph ological Ġin Ġnature Ġand Ġcan Ġbe Ġaddressed Ġusing Ġbetter ĠPOS Ġtagging Ġand Ġidentification Ġof Ġind e cl in ability , Ġparticularly Ġas Ġthey Ġrelate Ġto Ġnamed Ġentities Ġand Ġnoun s Ġwith Ġfeminine Ġmarkers . ĠOther Ġerrors Ġsuch Ġas Ġincorrect Ġattachment , Ġincorrect Ġid afa , Ġfalse Ġsubject , Ġand Ġconfusion Ġbetween Ġsubject Ġand Ġobject Ġcan Ġperhaps Ġbenefit Ġfrom Ġthe Ġuse Ġof Ġparsing . ĠAs Ġwith Ġthe Ġcore - word Ġerrors Ġfor ĠCA , Ġthe Ġreference Ġhas Ġsome Ġerrors Ġ( ex . Ġ{ a , i , o } Ġ$\ Right arrow Ġ$ Ġ# ), Ġand Ġextra Ġrounds Ġof Ġreviews Ġof Ġthe Ġreference Ġare Ġin Ġorder . </s> Exper iments Ġand ĠResults Ġ: :: ĠFull ĠDi ac rit ization ĠResults </s> Table ĠT AB REF 48 Ġcompares Ġthe Ġfull Ġword Ġdi ac rit ization Ġ( CW + CE ) Ġof Ġour Ġbest Ġsetup Ġto Ġother Ġsystems Ġin Ġthe Ġliterature . ĠAs Ġthe Ġresults Ġshow Ġfor ĠM SA , Ġour Ġoverall Ġdi ac rit ization ĠW ER Ġis Ġ6 . 0 % Ġwhile Ġthe Ġstate Ġof Ġthe Ġart Ġsystem Ġhas Ġa ĠW ER Ġof Ġ12 . 2 %. ĠAs Ġfor ĠCA , Ġour Ġbest Ġsystem Ġproduced Ġan Ġerror Ġrate Ġof Ġ4 . 3 %, Ġwhich Ġis Ġsignificantly Ġbetter Ġthan Ġusing Ġour Ġbest ĠM SA Ġsystem Ġto Ġdi ac rit ize ĠCA . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġfeature - rich ĠD NN Ġapproach Ġfor ĠM SA ĠCW Ġand ĠCE Ġrecovery Ġthat Ġproduces Ġa Ġword Ġlevel Ġerror Ġfor ĠM SA Ġof Ġ6 . 0 %, Ġwhich Ġis Ġmore Ġthan Ġ50 % Ġlower Ġthan Ġstate - of - the - art Ġsystems Ġ( 6 . 0 % Ġcompared Ġto Ġ12 . 2 %) Ġand Ġword Ġerror Ġrate Ġof Ġ4 . 3 % Ġfor ĠCA . ĠSpecifically , Ġwe Ġused Ġbi L ST M - based Ġmodel Ġwith Ġa Ġvariety Ġof Ġsurface , Ġmorph ological , Ġand Ġsynt actic Ġfeatures . ĠRel iable ĠN LP Ġtools Ġmay Ġbe Ġrequired Ġto Ġgenerate Ġsome Ġof Ġthese Ġfeatures , Ġand Ġsuch Ġtools Ġmay Ġnot Ġbe Ġreadily Ġavailable Ġfor Ġother Ġlanguage Ġvarieties , Ġsuch Ġas Ġdialect al ĠArabic . ĠHowever , Ġwe Ġshowed Ġthe Ġefficacy Ġof Ġdifferent Ġvarieties Ġof Ġfeatures , Ġsuch Ġas Ġsurface Ġlevel - features , Ġand Ġthey Ġcan Ġhelp Ġimprove Ġdi ac rit ization Ġindividually . ĠFurther , Ġthough Ġsome Ġerrors Ġmay Ġbe Ġovercome Ġusing Ġimproved ĠN LP Ġtools Ġ( ex . Ġbetter ĠPOS Ġtagging ), Ġsemantic Ġerrors , Ġsuch Ġincorrect Ġatt ch ment , Ġare Ġmore Ġdifficult Ġto Ġfix . ĠPerhaps , Ġusing Ġdependency Ġparsing Ġmay Ġhelp Ġovercome Ġsome Ġsemantic Ġerrors . ĠAs Ġfor Ġfeature Ġengineering , Ġthe Ġbroad Ġcategories Ġof Ġfeatures , Ġsuch Ġas Ġsurface , Ġsynt actic , Ġand Ġmorph ological Ġfeatures , Ġmay Ġlikely Ġcarry - over Ġto Ġother Ġlanguages , Ġlanguage - specific Ġfeature Ġengineering Ġmay Ġbe Ġrequire Ġto Ġhandle Ġthe Ġspecificity Ġof Ġeach Ġlanguage . ĠLastly , Ġsince Ġmultiple Ġdi ac rit ization Ġconventions Ġmay Ġexist , Ġas Ġin Ġthe Ġcase Ġof ĠArabic , Ġadopting Ġone Ġconvention Ġconsistently Ġis Ġimportant Ġfor Ġtraining Ġa Ġgood Ġsystem Ġand Ġfor Ġproperly Ġtesting Ġit . ĠThough Ġwe Ġhave Ġmostly Ġachieved Ġthis Ġfor ĠM SA , Ġthe ĠCA Ġdataset Ġrequires Ġmore Ġchecks Ġto Ġinsure Ġgreater Ġconsistency . </s> For Ġfuture Ġwork , Ġwe Ġwant Ġto Ġexplore Ġthe Ġeffectiveness Ġof Ġaugment ing Ġour ĠCW Ġmodel Ġwith ĠPOS Ġtagging Ġinformation Ġand Ġa Ġbig ram Ġlanguage Ġmodel . ĠFurther , Ġwe Ġplan Ġto Ġcreate Ġa Ġmulti Ġreference Ġdi ac rit ization Ġtest Ġset Ġto Ġhandle Ġwords Ġthat Ġhave Ġmultiple Ġvalid Ġdi ac rit ized Ġforms . ĠFor ĠCE , Ġwe Ġwant Ġto Ġexamine Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġfeatures Ġfor ĠArabic Ġparsing . ĠWe Ġplan Ġto Ġexplore : Ġcharacter - level Ġconv olution al Ġneural Ġnetworks Ġthat Ġmay Ġcapture Ġsub - word Ġmorph ological Ġfeatures ; Ġpre - trained Ġembed d ings ; Ġand Ġattention Ġmechanisms Ġto Ġfocus Ġon Ġsalient Ġfeatures . ĠWe Ġalso Ġplan Ġto Ġexplore Ġjoint Ġmodeling Ġfor Ġboth Ġcore Ġword Ġand Ġcase Ġending Ġdi ac rit ics .
Introduction Modern Standard Arabic (MSA) and Classical Arabic (CA) have two types of vowels, namely long vowels, which are explicitly written, and short vowels, aka diacritics, which are typically omitted in writing but are reintroduced by readers to properly pronounce words. Since diacritics disambiguate the sense of the words in context and their syntactic roles in sentences, automatic diacritic recovery is essential for applications such as text-to-speech and educational tools for language learners, who may not know how to properly verbalize words. Diacritics have two types, namely: core-word (CW) diacritics, which are internal to words and specify lexical selection; and case-endings (CE), which appear on the last letter of word stems, typically specifying their syntactic role. For example, the word “ktb” (كتب>) can have multiple diacritized forms such as “katab” (كَتَب> – meaning “he wrote”) “kutub” (كُتُب> – “books”). While “katab” can only assume one CE, namely “fatHa” (“a”), “kutub” can accept the CEs: “damma” (“u”) (nominal – ex. subject), “a” (accusative – ex. object), “kasra” (“i”) (genitive – ex. PP predicate), or their nunations. There are 14 diacritic combinations. When used as CEs, they typically convey specific syntactic information, namely: fatHa “a” for accusative nouns, past verbs and subjunctive present verbs; kasra “i” for genitive nouns; damma “u” for nominative nouns and indicative present verbs; sukun “o” for jussive present verbs and imperative verbs. FatHa, kasra and damma can be preceded by shadda “$\sim $” for gemination (consonant doubling) and/or converted to nunation forms following some grammar rules. In addition, according to Arabic orthography and phonology, some words take a virtual (null) “#” marker when they end with certain characters (ex: long vowels). This applies also to all non-Arabic words (ex: punctuation, digits, Latin words, etc.). Generally, function words, adverbs and foreign named entities (NEs) have set CEs (sukun, fatHa or virtual). Similar to other Semitic languages, Arabic allows flexible Verb-Subject-Object as well as Verb-Object-Subject constructs BIBREF1. Such flexibility creates inherent ambiguity, which is resolved by diacritics as in “r$>$Y Emr Ely” (رأى عمر علي> Omar saw Ali/Ali saw Omar). In the absence of diacritics it is not clear who saw whom. Similarly, in the sub-sentence “kAn Alm&tmr AltAsE” (كان المؤتمر التاسع>), if the last word, is a predicate of the verb “kAn”, then the sentence would mean “this conference was the ninth” and would receive a fatHa (a) as a case ending. Conversely, if it was an adjective to the “conference”, then the sentence would mean “the ninth conference was ...” and would receive a damma (u) as a case ending. Thus, a consideration of context is required for proper disambiguation. Due to the inter-word dependence of CEs, they are typically harder to predict compared to core-word diacritics BIBREF2, BIBREF3, BIBREF4, BIBREF5, with CEER of state-of-the-art systems being in double digits compared to nearly 3% for word-cores. Since recovering CEs is akin to shallow parsing BIBREF6 and requires morphological and syntactic processing, it is a difficult problem in Arabic NLP. In this paper, we focus on recovering both CW diacritics and CEs. We employ two separate Deep Neural Network (DNN) architectures for recovering both kinds of diacritic types. We use character-level and word-level bidirectional Long-Short Term Memory (biLSTM) based recurrent neural models for CW diacritic and CE recovery respectively. We train models for both Modern Standard Arabic (MSA) and Classical Arabic (CA). For CW diacritics, the model is informed using word segmentation information and a unigram language model. We also employ a unigram language model to perform post correction on the model output. We achieve word error rates for CW diacritics of 2.9% and 2.2% for MSA and CA. The MSA word error rate is 6% lower than the best results in the literature (the RDI diacritizer BIBREF7). The CE model is trained with a rich set of surface, morphological, and syntactic features. The proposed features would aid the biLSTM model in capturing syntactic dependencies indicated by Part-Of-Speech (POS) tags, gender and number features, morphological patterns, and affixes. We show that our model achieves a case ending error rate (CEER) of 3.7% for MSA and 2.5% for CA. For MSA, this CEER is more than 60% lower than other state-of-the-art systems such as Farasa and the RDI diacritizer, which are trained on the same dataset and achieve CEERs of 10.7% and 14.4% respectively. The contributions of this paper are as follows: We employ a character-level RNN model that is informed using word morphological information and a word unigram language model to recover CW diacritics. Our model beats the best state-of-the-art system by 6% for MSA. We introduce a new feature rich RNN-based CE recovery model that achieves errors rates that are 60% lower than the current state-of-the-art for MSA. We explore the effect of different features, which may potentially be exploited for Arabic parsing. We show the effectiveness of our approach for both MSA and CA. Background Automatic diacritics restoration has been investigated for many different language such as European languages (e.g. Romanian BIBREF8, BIBREF9, French BIBREF10, and Croatian BIBREF11), African languages (e.g. Yorba BIBREF12), Southeast Asian languages (e.g. Vietnamese BIBREF13), Semitic language (e.g. Arabic and Hebrew BIBREF14), and many others BIBREF15. For many languages, diacritic (or accent restoration) is limited to a handful of letters. However, for Semitic languages, diacritic recovery extends to most letters. Many general approaches have been explored for this problem including linguistically motivated rule-based approaches, machine learning approaches, such as Hidden Markov Models (HMM) BIBREF14 and Conditional Random Fields (CRF) BIBREF16, and lately deep learning approaches such as Arabic BIBREF17, BIBREF18, BIBREF19, Slovak BIBREF20, and Yorba BIBREF12. Aside from rule-based approaches BIBREF21, different methods were used to recover diacritics in Arabic text. Using a hidden Markov model (HMM) BIBREF14, BIBREF22 with an input character sequence, the model attempts to find the best state sequence given previous observations. BIBREF14 reported a 14% word error rate (WER) while BIBREF22 achieved a 4.1% diacritic error rate (DER) on the Quran (CA). BIBREF23 combined both morphological, acoustic, and contextual features to build a diacritizer trained on FBIS and LDC CallHome ECA collections. They reported a 9% (DER) without CE, and 28% DER with CE. BIBREF24 employed a cascade of a finite state transducers. The cascade stacked a word language model (LM), a charachter LM, and a morphological model. The model achieved an accuracy of 7.33% WER without CE and and 23.61% WER with CE. BIBREF25 employed a maximum entropy model for sequence classification. The system was trained on the LDC’s Arabic Treebank (ATB) and evaluated on a 600 articles from An-Nahar Newspaper (340K words) and achieved 5.5% DER and 18% WER on words without CE. BIBREF26 used a hybrid approach that utilizes the output of Alkhalil morphological Analyzer BIBREF27 to generate all possible out of context diacritizations of a word. Then, an HMM guesses the correct diacritized form. Similarly, Microsoft Arabic Toolkit Services (ATKS) diacritizer BIBREF28 uses a rule-based morphological analyzer that produces possible analyses and an HMM in conjunction with rules to guess the most likely analysis. They report WER of 11.4% and 4.4% with and without CE. MADAMIRA BIBREF29 uses a combinations of morpho-syntactic features to rank a list of potential analyses provided by the Buckwalter Arabic Morphological Analyzer (BAMA) BIBREF30. An SVM trained on ATB selects the most probable analysis, including the diacritized form. MADAMIRA achieves 19.0% and 6.7% WER with and without CE respectively BIBREF31. Farasa BIBREF31 uses an HMM to guess CW diacritics and an SVM-rank based model trained on morphological and syntactic features to guess CEs. Farasa achieves WER of 12.8% and 3.3% with and without CEs. More recent work employed different neural architectures to model the diacritization problem. BIBREF17 used a biLSTM recurrent neural network trained on the same dataset as BIBREF25. They explored one, two and three BiLSTM layers with 250 nodes in each layers, achieving WER of 9.1% including CE on ATB. Similar architectures were used but achieved lower results BIBREF7, BIBREF32. BIBREF33 provide a comprehensive survey on Arabic diacritization. A more recent survey by BIBREF34 concluded that reported results are often incomparable due to the usage of different test sets. They concluded that a large unigram LM for CW diacritic recovery is competitive with many of the systems in the literature, which prompted us to utilize a unigram language model for post correction. As mentioned earlier, two conclusions can be drawn, namely: restoring CEs is more challenging than CW diacritic restoration; and combining multiple features typically improves CE restoration. In this paper, we expand upon the work in the literature by introducing feature-rich DNN models for restoring both CW and CE diacritics. We compare our models to multiple systems on the same test set. We achieve results that reduce diacritization error rates by more than half compared to the best SOTA systems. We further conduct an ablation study to determine the relative effect of the different features. As for Arabic, it is a Semitic language with derivational morphology. Arabic nouns, adjectives, adverbs, and verbs are typically derived from a closed set of 10,000 roots of length 3, 4, or rarely 5. Arabic nouns and verbs are derived from roots by applying templates to the roots to generate stems. Such templates may carry information that indicate morphological features of words such POS tag, gender, and number. For example, given a 3-letter root with 3 consonants CCC, a valid template may be CwACC , where the infix “wA” (وا>) is inserted, this template typically indicates an Arabic broken, or irregular, plural template for a noun of template CACC or CACCp if masculine or feminine respectively. Further, stems may accept prefixes and/or suffixes to form words. Prefixes include coordinating conjunctions, determiner, and prepositions, and suffixes include attached pronouns and gender and number markers. Our Diacritizer ::: Training and Test Corpora For MSA, we acquired the diacritized corpus that was used to train the RDI BIBREF7 diacritizer and the Farasa diacritizer BIBREF31. The corpus contains 9.7M tokens with approximately 194K unique surface forms (excluding numbers and punctuation marks). The corpus covers multiple genres such as politics and sports and is a mix of MSA and CA. This corpus is considerably larger than the Arabic Treebank BIBREF35 and is more consistent in its diacritization. For testing, we used the freely available WikiNews test set BIBREF31, which is composed of 70 MSA WikiNews articles (18,300 tokens) and evenly covers a variety of genres including politics, economics, health, science and technology, sports, arts and culture. For CA, we obtained a large collection of fully diacritized classical texts (2.7M tokens) from a book publisher, and we held-out a small subset of 5,000 sentences (approximately 400k words) for testing. Then, we used the remaining sentences to train the CA models. Our Diacritizer ::: Core Word Diacritization ::: Features. Arabic words are typically derived from a limited set of roots by fitting them into so-called stem-templates (producing stems) and may accept a variety of prefixes and suffixes such as prepositions, determiners, and pronouns (producing words). Word stems specify the lexical selection and are typically unaffected by the attached affixes. We used 4 feature types, namely: CHAR: the characters. SEG: the position of the character in a word segment. For example, given the word “wAlktAb” (والكتاب> and the book/writers), which is composed of 3 segments “w+Al+ktAb” (و+ال+كتاب>). Letters were marked as “B” if they begin a segment, “M” if they are in the middle of a segment, “E” if they end a segment, and “S” if they are single letter segments. So for “w+Al+ktAb”, the corresponding character positions are “S+BE+BMME”. We used Farasa to perform segmentation, which has a reported segmentation accuracy of 99% on the WikiNews dataset BIBREF36. PRIOR: diacritics seen in the training set per segment. Since we used a character level model, this feature informed the model with word level information. For example, the word “ktAb” (كتاب>) was observed to have two diacritized forms in the training set, namely “kitaAb” (كِتَاب> – book) and “kut$\sim $aAb” (كُتَّاب> – writers). The first letter in the word (“k”) accepted the diacritics “i” and “u”. Thus given a binary vector representing whether a character is allowed to assume any of the eight primitive Arabic diacritic marks (a, i, u, o, K, N, F, and $\sim $ in order), the first letter would be given the following vector “01100000”. If a word segment was never observed during training, the vector for all letters therein would be set to 11111111. This feature borrows information from HMM models, which have been fairly successful in diacritizing word cores. CASE: whether the letter expects a core word diacritic or a case ending. Case endings are placed on only one letter in a word, which may or may not be the last letter in the word. This is a binary feature. Our Diacritizer ::: Core Word Diacritization ::: DNN Model. Using a DNN model, particularly with a biLSTM BIBREF37, is advantageous because the model automatically explores the space of feature combinations and is able to capture distant dependencies. A number of studies have explored various biLSTM architectures BIBREF17, BIBREF7, BIBREF32 including stacked biLSTMs confirming their effectiveness. As shown in Figure FIGREF14, we employed a character-based biLSTM model with associated features for each character. Every input character had an associated list of $m$ features, and we trained randomly initialized embeddings of size 50 for each feature. Then, we concatenated the feature embeddings vectors creating an $m\times 50$ vector for each character, which was fed into the biLSTM layer of length 100. The output of the biLSTM layer was fed directly into a dense layer of size 100. We used early stopping with patience of 5 epochs, a learning rate of 0.001, a batch size of 256, and an Adamax optimizer. The input was the character sequence in a sentence with words being separated by word boundary markers (WB), and we set the maximum sentence length to 1,250 characters. Our Diacritizer ::: Case Ending Diacritization ::: Features. Table TABREF17 lists the features that we used for CE recovery. We used Farasa to perform segmentation and POS tagging and to determine stem-templates BIBREF31. Farasa has a reported POS accuracy of 96% on the WikiNews dataset BIBREF31. Though the Farasa diacritizer utilizes a combination of some the features presented herein, namely segmentation, POS tagging, and stem templates, Farasa's SVM-ranking approach requires explicit specification of feature combinations (ex. $Prob(CE\Vert current\_word, prev\_word, prev\_CE)$). Manual exploration of the feature space is undesirable, and ideally we would want our learning algorithm to do so automatically. The flexibility of the DNN model allowed us to include many more surface level features such as affixes, leading and trailing characters in words and stems, and the presence of words in large gazetteers of named entities. As we show later, these additional features significantly lowered CEER. Our Diacritizer ::: Case Ending Diacritization ::: DNN Model Figure FIGREF19 shows the architecture of our DNN algorithm. Every input word had an associated list of $n$ features, and we trained randomly initialized embeddings of size 100 for each feature. Then, we concatenated the feature embeddings vectors creating an $n\times 100$ vector for each word. We fed these vectors into a biLSTM layer of 100 dimensions after applying a dropout of 75%, where dropout behaves like a regularlizer to avoid overfitting BIBREF40. We conducted side experiments with lower dropout rates, but the higher dropout rate worked best. The output of the biLSTM layer was fed into a 100 dimensional dense layer with 15% dropout and softmax activation. We conducted side experiments where we added additional biLSTM layers and replaced softmax with a conditional random field layer, but we did not observe improvements. Thus, we opted for a simpler model. We used a validation set to determine optimal parameters such as dropout rate. Again, we used the “Adamax” optimizer with categorical cross entropy loss and a learning rate of 0.001. We also applied early stopping with patience of up to 5 consecutive epochs without improvement. Experiments and Results ::: Core Word ::: Experimental Setup For all the experiments conducted herein, we used the Keras toolkit BIBREF41 with a TensorFlow backend BIBREF42. We used the entirety of the training set as input, and we instructed Keras to use 5% of the data for tuning (validation). We included the CASE feature, which specifies whether the letter accepts a normal diacritic or case ending, in all our setups. We conducted multiple experiment using different features, namely: CHAR: This is our baseline setup where we only used the characters as features. CHAR+SEG: This takes the characters and their segmentation information as features. CHAR+PRIOR: This takes the characters and their the observed diacritized forms in the training set. All: This setup includes all the features. We also optionally employed post correction. For words that were seen in training, if the model produced a diacritized form that was not seen in the training data, we assumed it was an error and replaced it with the most frequently observed diacritized form (using a unigram language model). We report two error rates, namely WER (at word level) and DER (at character level). We used relaxed scoring where we assumed an empty case to be equivalent to sukun, and we removed default diacritics – fatHa followed by alef, kasra followed by ya, and damma followed by wa. Using such scoring would allow to compare to other systems in the literature that may use different diacritization conventions. Experiments and Results ::: Core Word ::: Results and Error analysis For testing, we used the aforementioned WikiNews dataset to test the MSA diacritizer and the held-out 5,000 sentences for CA. Table TABREF27 shows WER and DER results using different features with and without post correction. Experiments and Results ::: Core Word ::: Results and Error analysis ::: MSA Results: For MSA, though the CHAR+PRIOR feature led to worse results than using CHAR alone, the results show that combining all the features achieved the best results. Moreover, post correction improved results overall. We compare our results to five other systems, namely Farasa BIBREF31, MADAMIRA BIBREF29, RDI (Rashwan et al., 2015), MIT (Belinkow and Glass, 2015), and Microsoft ATKS BIBREF28. Table TABREF34 compares our system with others in the aforementioned systems. As the results show, our results beat the current state-of-the-art. For error analysis, we analyzed all the errors (527 errors). The errors types along with examples of each are shown in Table TABREF29. The most prominent error type arises from the selection of a valid diacritized form that does not match the context (40.8%). Perhaps, including POS tags as a feature or augmenting the PRIOR feature with POS tag information and a bigram language model may reduce the error rate further. The second most common error is due to transliterated foreign words including foreign named entities (23.5%). Such words were not observed during training. Further, Arabic Named entities account for 10.6% of the errors, where they were either not seen in training or they share identical non-diacritized forms with other words. Perhaps, building larger gazetteers of diacritized named entities may resolve NE related errors. In 10.8% of the cases, the diacritizer produced in completely incorrect diacritized forms. In some the cases (9.1%), though the diacritizer produced a form that is different from the reference, both forms were in fact correct. Most of these cases were due to variations in diacritization conventions (ex. “bare alef” (A) at start of a word receiving a diacritic or not). Other cases include foreign words and some words where both diacritized forms are equally valid. Experiments and Results ::: Core Word ::: Results and Error analysis ::: CA Results: For CA results, the CHAR+SEG and CHAR+PRIOR performed better than using characters alone with CHAR+PRIOR performing better than CHAR+SEG. As in the case with MSA, combining all the features led to the best results. Post correction had a significantly larger positive impact on results compared to what we observed for MSA. This may indicate that we need a larger training set. The best WER that we achieved for CW diacritics with post corrections is 2.2%. Since we did not have access to any publicly available system that is tuned for CA, we compared our best system to using our best MSA system to diacritize the CA test set, and the MSA diacritizer produced significantly lower results with a WER of 8.5% (see Table TABREF34). This highlights the large difference between MSA and CA and the need for systems that are specifically tuned for both. We randomly selected and analyzed 500 errors (5.2% of the errors). The errors types along with examples of each are shown in Table TABREF33. The two most common errors involve the system producing completely correct diacritized forms (38.8%) or correct forms that don't match the context (31.4%). The relatively higher percentage of completely incorrect guesses, compared to MSA, may point to the higher lexical diversity of classical Arabic. As for MSA, we suspect that adding additional POS information and employing a word bigram to constrain the PRIOR feature may help reduce selection errors. Another prominent error is related to the diacritics that appear on attached suffixes, particularly pronouns, which depend on the choice of case ending (13.2%). Errors due to named entities are slightly fewer than those seen for MSA (8.8%). A noticeable number of mismatches between the guess and the reference are due to partial diacritization of the reference (4.4%). We plan to conduct an extra round of checks on the test set. Experiments and Results ::: Case Ending ::: Experimental Setup We conducted multiple experiments to determine the relative effect of the different features as follows: word: This is our baseline setup, which uses word surface forms only. word-surface: This setup uses the word surface forms, stems, prefixes, and suffixes (including noun suffixes). This simulates the case when no POS tagging information is available. word-POS: This includes the word surface form and POS information, including gender and number of stems, prefixes, and suffixes. word-morph: This includes words and their stem templates to capture morphological patterns. word-surface-POS-morph: This setup uses all the features (surface, POS, and morphological). all-misc: This uses all features plus word and stem leading and trailing character unigrams and bigrams in addition to sukun words and named entities. For testing MSA, we used the aforementioned WikiNews dataset. Again, we compared our results to five other systems, namely Farasa BIBREF31, MADAMIRA BIBREF29, RDI (Rashwan et al., 2015), MIT (Belinkow and Glass, 2015), and Microsoft ATKS BIBREF28. For CA testing, we used the 5,000 sentences that we set aside. Again, we compared to our best MSA system. Experiments and Results ::: Case Ending ::: Results and Error Analysis Table TABREF45 lists the results of our setups compared to other systems. Experiments and Results ::: Case Ending ::: Results and Error Analysis ::: MSA Results: As the results show, our baseline DNN system outperforms all state-of-the-art systems. Further, adding more features yielded better results overall. Surface-level features resulted in the most gain, followed by POS tags, and lastly stem templates. Further, adding head and tail characters along with a list of sukun words and named entities led to further improvement. Our proposed feature-rich system has a CEER that is approximately 61% lower than any of the state-of-the-art systems. Figure FIGREF49 shows CE distribution and prediction accuracy. For the four basic markers kasra, fatHa, damma and sukun, which appear 27%, 14%, 9% and 10% respectively, the system has CEER of $\sim $1% for each. Detecting the virtual CE mark is a fairly easy task. All other CE markers represent 13% with almost negligible errors. Table TABREF30 lists a thorough breakdown of all errors accounting for at 1% of the errors along with the most common reasons of the errors and examples illustrating these reasons. For example, the most common error type involves guessing a fatHa (a) instead of damma (u) or vice versa (19.3%). The most common reasons for this error type, based on inspecting the errors, were due to: POS errors (ex. a word is tagged as a verb instead of a noun); and a noun is treated as a subject instead of an object or vice versa. The table details the rest of the error types. Overall, some of the errors are potentially fixable using better POS tagging, improved detection of non-Arabized foreign names, and detection of indeclinability. However, some errors are more difficult and require greater understanding of semantics such as improper attachment, incorrect idafa, and confusion between subject and object. Perhaps, such semantic errors can be resolved using parsing. Experiments and Results ::: Case Ending ::: Results and Error Analysis ::: CA Results: The results show that the POS tagging features led to the most improvements followed by the surface features. Combining all features led to the best results with WER of 2.5%. As we saw for CW diacritics, using our best MSA system to diacritize CA led to significantly lower results with CEER of 8.9%. Figure FIGREF50 shows CE distribution and prediction accuracy. For the four basic markers fatHa, kasra, damma and sukun, which appear 18%, 14%, 13% and 8% respectively, the system has CEER $\sim $0.5% for each. Again, detecting the virtual CE mark was a fairly easy task. All other CE markers representing 20% have negligible errors. Table TABREF31 lists all the error types, which account for at least 1% of the errors, along with their most common causes and explanatory examples. The error types are similar to those observed for MSA. Some errors are more syntactic and morphological in nature and can be addressed using better POS tagging and identification of indeclinability, particularly as they relate to named entities and nouns with feminine markers. Other errors such as incorrect attachment, incorrect idafa, false subject, and confusion between subject and object can perhaps benefit from the use of parsing. As with the core-word errors for CA, the reference has some errors (ex. {a,i,o} $\Rightarrow $ #), and extra rounds of reviews of the reference are in order. Experiments and Results ::: Full Diacritization Results Table TABREF48 compares the full word diacritization (CW+CE) of our best setup to other systems in the literature. As the results show for MSA, our overall diacritization WER is 6.0% while the state of the art system has a WER of 12.2%. As for CA, our best system produced an error rate of 4.3%, which is significantly better than using our best MSA system to diacritize CA. Conclusion and Future Work In this paper, we presented a feature-rich DNN approach for MSA CW and CE recovery that produces a word level error for MSA of 6.0%, which is more than 50% lower than state-of-the-art systems (6.0% compared to 12.2%) and word error rate of 4.3% for CA. Specifically, we used biLSTM-based model with a variety of surface, morphological, and syntactic features. Reliable NLP tools may be required to generate some of these features, and such tools may not be readily available for other language varieties, such as dialectal Arabic. However, we showed the efficacy of different varieties of features, such as surface level-features, and they can help improve diacritization individually. Further, though some errors may be overcome using improved NLP tools (ex. better POS tagging), semantic errors, such incorrect attchment, are more difficult to fix. Perhaps, using dependency parsing may help overcome some semantic errors. As for feature engineering, the broad categories of features, such as surface, syntactic, and morphological features, may likely carry-over to other languages, language-specific feature engineering may be require to handle the specificity of each language. Lastly, since multiple diacritization conventions may exist, as in the case of Arabic, adopting one convention consistently is important for training a good system and for properly testing it. Though we have mostly achieved this for MSA, the CA dataset requires more checks to insure greater consistency. For future work, we want to explore the effectiveness of augmenting our CW model with POS tagging information and a bigram language model. Further, we plan to create a multi reference diacritization test set to handle words that have multiple valid diacritized forms. For CE, we want to examine the effectiveness of the proposed features for Arabic parsing. We plan to explore: character-level convolutional neural networks that may capture sub-word morphological features; pre-trained embeddings; and attention mechanisms to focus on salient features. We also plan to explore joint modeling for both core word and case ending diacritics.
what linguistics features are used?
76ed74788e3eb3321e646c48ae8bf6cdfe46dca1
what Ġlingu istics Ġfeatures Ġare Ġused ?
POS , Ġgender / number Ġand Ġstem ĠPOS
<s> what Ġlingu istics Ġfeatures Ġare Ġused ? </s> Introduction </s> Modern ĠStandard ĠArabic Ġ( M SA ) Ġand ĠClassical ĠArabic Ġ( CA ) Ġhave Ġtwo Ġtypes Ġof Ġvow els , Ġnamely Ġlong Ġvow els , Ġwhich Ġare Ġexplicitly Ġwritten , Ġand Ġshort Ġvow els , Ġaka Ġdi ac rit ics , Ġwhich Ġare Ġtypically Ġomitted Ġin Ġwriting Ġbut Ġare Ġreintrodu ced Ġby Ġreaders Ġto Ġproperly Ġpronounce Ġwords . ĠSince Ġdi ac rit ics Ġdis amb ig uate Ġthe Ġsense Ġof Ġthe Ġwords Ġin Ġcontext Ġand Ġtheir Ġsynt actic Ġroles Ġin Ġsentences , Ġautomatic Ġdi ac ritic Ġrecovery Ġis Ġessential Ġfor Ġapplications Ġsuch Ġas Ġtext - to - speech Ġand Ġeducational Ġtools Ġfor Ġlanguage Ġlearners , Ġwho Ġmay Ġnot Ġknow Ġhow Ġto Ġproperly Ġverbal ize Ġwords . ĠDi ac rit ics Ġhave Ġtwo Ġtypes , Ġnamely : Ġcore - word Ġ( CW ) Ġdi ac rit ics , Ġwhich Ġare Ġinternal Ġto Ġwords Ġand Ġspecify Ġlex ical Ġselection ; Ġand Ġcase - end ings Ġ( CE ), Ġwhich Ġappear Ġon Ġthe Ġlast Ġletter Ġof Ġword Ġstems , Ġtypically Ġspecifying Ġtheir Ġsynt actic Ġrole . ĠFor Ġexample , Ġthe Ġword ĠâĢ ľ kt b âĢ Ŀ Ġ( Ù ĥ Øª Ø¨ >) Ġcan Ġhave Ġmultiple Ġdi ac rit ized Ġforms Ġsuch Ġas ĠâĢ ľ kat ab âĢ Ŀ Ġ( Ù ĥ Ùİ Øª Ùİ Ø¨ > ĠâĢĵ Ġmeaning ĠâĢ ľ he Ġwrote âĢ Ŀ ) ĠâĢ ľ k ut ub âĢ Ŀ Ġ( Ù ĥ Ù ı Øª Ù ı Ø¨ > ĠâĢĵ ĠâĢ ľ books âĢ Ŀ ). ĠWhile ĠâĢ ľ kat ab âĢ Ŀ Ġcan Ġonly Ġassume Ġone ĠCE , Ġnamely ĠâĢ ľ fat Ha âĢ Ŀ Ġ( âĢ ľ a âĢ Ŀ ), ĠâĢ ľ k ut ub âĢ Ŀ Ġcan Ġaccept Ġthe ĠCE s : ĠâĢ ľ dam ma âĢ Ŀ Ġ( âĢ ľ u âĢ Ŀ ) Ġ( nom inal ĠâĢĵ Ġex . Ġsubject ), ĠâĢ ľ a âĢ Ŀ Ġ( acc us ative ĠâĢĵ Ġex . Ġobject ), ĠâĢ ľ kas ra âĢ Ŀ Ġ( âĢ ľ i âĢ Ŀ ) Ġ( gen itive ĠâĢĵ Ġex . ĠPP Ġpredicate ), Ġor Ġtheir Ġnun ations . ĠThere Ġare Ġ14 Ġdi ac ritic Ġcombinations . ĠWhen Ġused Ġas ĠCE s , Ġthey Ġtypically Ġconvey Ġspecific Ġsynt actic Ġinformation , Ġnamely : Ġfat Ha ĠâĢ ľ a âĢ Ŀ Ġfor Ġaccus ative Ġnoun s , Ġpast Ġverbs Ġand Ġsub jun ctive Ġpresent Ġverbs ; Ġk as ra ĠâĢ ľ i âĢ Ŀ Ġfor Ġgen itive Ġnoun s ; Ġdam ma ĠâĢ ľ u âĢ Ŀ Ġfor Ġnom in ative Ġnoun s Ġand Ġindicative Ġpresent Ġverbs ; Ġsu kun ĠâĢ ľ o âĢ Ŀ Ġfor Ġj uss ive Ġpresent Ġverbs Ġand Ġimperative Ġverbs . ĠFat Ha , Ġk as ra Ġand Ġdam ma Ġcan Ġbe Ġpreceded Ġby Ġsh add a ĠâĢ ľ $ \ sim Ġ$ âĢ Ŀ Ġfor Ġgem ination Ġ( cons on ant Ġdoubling ) Ġand / or Ġconverted Ġto Ġnun ation Ġforms Ġfollowing Ġsome Ġgrammar Ġrules . ĠIn Ġaddition , Ġaccording Ġto ĠArabic Ġorth ography Ġand Ġphon ology , Ġsome Ġwords Ġtake Ġa Ġvirtual Ġ( null ) ĠâĢ ľ # âĢ Ŀ Ġmarker Ġwhen Ġthey Ġend Ġwith Ġcertain Ġcharacters Ġ( ex : Ġlong Ġvow els ). ĠThis Ġapplies Ġalso Ġto Ġall Ġnon - Arab ic Ġwords Ġ( ex : Ġpunct uation , Ġdigits , ĠLatin Ġwords , Ġetc .). ĠGenerally , Ġfunction Ġwords , Ġad verbs Ġand Ġforeign Ġnamed Ġentities Ġ( NE s ) Ġhave Ġset ĠCE s Ġ( su kun , Ġfat Ha Ġor Ġvirtual ). ĠSimilar Ġto Ġother ĠSem itic Ġlanguages , ĠArabic Ġallows Ġflexible ĠVerb - Subject - Object Ġas Ġwell Ġas ĠVerb - Object - Subject Ġconstructs ĠB IB REF 1 . ĠSuch Ġflexibility Ġcreates Ġinherent Ġambiguity , Ġwhich Ġis Ġresolved Ġby Ġdi ac rit ics Ġas Ġin ĠâĢ ľ r $ > $ Y ĠEm r ĠEly âĢ Ŀ Ġ( Ø± Ø £ Ù ī ĠØ ¹ Ùħ Ø± ĠØ ¹ ÙĦ ÙĬ > ĠOmar Ġsaw ĠAli / Ali Ġsaw ĠOmar ). ĠIn Ġthe Ġabsence Ġof Ġdi ac rit ics Ġit Ġis Ġnot Ġclear Ġwho Ġsaw Ġwhom . ĠSimilarly , Ġin Ġthe Ġsub - sent ence ĠâĢ ľ k An ĠAlm & tm r ĠAlt As E âĢ Ŀ Ġ( Ù ĥ Ø§ ÙĨ ĠØ§ÙĦ Ùħ Ø ¤ Øª Ùħ Ø± ĠØ§ÙĦ Øª Ø§Ø ³ Ø¹ > ), Ġif Ġthe Ġlast Ġword , Ġis Ġa Ġpredicate Ġof Ġthe Ġverb ĠâĢ ľ k An âĢ Ŀ , Ġthen Ġthe Ġsentence Ġwould Ġmean ĠâĢ ľ this Ġconference Ġwas Ġthe Ġninth âĢ Ŀ Ġand Ġwould Ġreceive Ġa Ġfat Ha Ġ( a ) Ġas Ġa Ġcase Ġending . ĠConversely , Ġif Ġit Ġwas Ġan Ġadjective Ġto Ġthe ĠâĢ ľ conference âĢ Ŀ , Ġthen Ġthe Ġsentence Ġwould Ġmean ĠâĢ ľ the Ġninth Ġconference Ġwas Ġ... âĢ Ŀ Ġand Ġwould Ġreceive Ġa Ġdam ma Ġ( u ) Ġas Ġa Ġcase Ġending . ĠThus , Ġa Ġconsideration Ġof Ġcontext Ġis Ġrequired Ġfor Ġproper Ġdis amb ig uation . ĠDue Ġto Ġthe Ġinter - word Ġdependence Ġof ĠCE s , Ġthey Ġare Ġtypically Ġharder Ġto Ġpredict Ġcompared Ġto Ġcore - word Ġdi ac rit ics ĠB IB REF 2 , ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , Ġwith ĠCE ER Ġof Ġstate - of - the - art Ġsystems Ġbeing Ġin Ġdouble Ġdigits Ġcompared Ġto Ġnearly Ġ3 % Ġfor Ġword - c ores . ĠSince Ġrecovering ĠCE s Ġis Ġakin Ġto Ġshallow Ġparsing ĠB IB REF 6 Ġand Ġrequires Ġmorph ological Ġand Ġsynt actic Ġprocessing , Ġit Ġis Ġa Ġdifficult Ġproblem Ġin ĠArabic ĠN LP . ĠIn Ġthis Ġpaper , Ġwe Ġfocus Ġon Ġrecovering Ġboth ĠCW Ġdi ac rit ics Ġand ĠCE s . ĠWe Ġemploy Ġtwo Ġseparate ĠDeep ĠNeural ĠNetwork Ġ( D NN ) Ġarchitectures Ġfor Ġrecovering Ġboth Ġkinds Ġof Ġdi ac ritic Ġtypes . ĠWe Ġuse Ġcharacter - level Ġand Ġword - level Ġbid irection al ĠLong - Short ĠTerm ĠMemory Ġ( bi L ST M ) Ġbased Ġrecurrent Ġneural Ġmodels Ġfor ĠCW Ġdi ac ritic Ġand ĠCE Ġrecovery Ġrespectively . ĠWe Ġtrain Ġmodels Ġfor Ġboth ĠModern ĠStandard ĠArabic Ġ( M SA ) Ġand ĠClassical ĠArabic Ġ( CA ). ĠFor ĠCW Ġdi ac rit ics , Ġthe Ġmodel Ġis Ġinformed Ġusing Ġword Ġsegment ation Ġinformation Ġand Ġa Ġun ig ram Ġlanguage Ġmodel . ĠWe Ġalso Ġemploy Ġa Ġun ig ram Ġlanguage Ġmodel Ġto Ġperform Ġpost Ġcorrection Ġon Ġthe Ġmodel Ġoutput . ĠWe Ġachieve Ġword Ġerror Ġrates Ġfor ĠCW Ġdi ac rit ics Ġof Ġ2 . 9 % Ġand Ġ2 . 2 % Ġfor ĠM SA Ġand ĠCA . ĠThe ĠM SA Ġword Ġerror Ġrate Ġis Ġ6 % Ġlower Ġthan Ġthe Ġbest Ġresults Ġin Ġthe Ġliterature Ġ( the ĠR DI Ġdi ac rit izer ĠB IB REF 7 ). ĠThe ĠCE Ġmodel Ġis Ġtrained Ġwith Ġa Ġrich Ġset Ġof Ġsurface , Ġmorph ological , Ġand Ġsynt actic Ġfeatures . ĠThe Ġproposed Ġfeatures Ġwould Ġaid Ġthe Ġbi L ST M Ġmodel Ġin Ġcapturing Ġsynt actic Ġdependencies Ġindicated Ġby ĠPart - Of - Spe ech Ġ( POS ) Ġtags , Ġgender Ġand Ġnumber Ġfeatures , Ġmorph ological Ġpatterns , Ġand Ġaff ix es . ĠWe Ġshow Ġthat Ġour Ġmodel Ġachieves Ġa Ġcase Ġending Ġerror Ġrate Ġ( CE ER ) Ġof Ġ3 . 7 % Ġfor ĠM SA Ġand Ġ2 . 5 % Ġfor ĠCA . ĠFor ĠM SA , Ġthis ĠCE ER Ġis Ġmore Ġthan Ġ60 % Ġlower Ġthan Ġother Ġstate - of - the - art Ġsystems Ġsuch Ġas ĠFar asa Ġand Ġthe ĠR DI Ġdi ac rit izer , Ġwhich Ġare Ġtrained Ġon Ġthe Ġsame Ġdataset Ġand Ġachieve ĠCE ER s Ġof Ġ10 . 7 % Ġand Ġ14 . 4 % Ġrespectively . ĠThe Ġcontributions Ġof Ġthis Ġpaper Ġare Ġas Ġfollows : </s> We Ġemploy Ġa Ġcharacter - level ĠR NN Ġmodel Ġthat Ġis Ġinformed Ġusing Ġword Ġmorph ological Ġinformation Ġand Ġa Ġword Ġun ig ram Ġlanguage Ġmodel Ġto Ġrecover ĠCW Ġdi ac rit ics . ĠOur Ġmodel Ġbeats Ġthe Ġbest Ġstate - of - the - art Ġsystem Ġby Ġ6 % Ġfor ĠM SA . </s> We Ġintroduce Ġa Ġnew Ġfeature Ġrich ĠR NN - based ĠCE Ġrecovery Ġmodel Ġthat Ġachieves Ġerrors Ġrates Ġthat Ġare Ġ60 % Ġlower Ġthan Ġthe Ġcurrent Ġstate - of - the - art Ġfor ĠM SA . </s> We Ġexplore Ġthe Ġeffect Ġof Ġdifferent Ġfeatures , Ġwhich Ġmay Ġpotentially Ġbe Ġexploited Ġfor ĠArabic Ġparsing . </s> We Ġshow Ġthe Ġeffectiveness Ġof Ġour Ġapproach Ġfor Ġboth ĠM SA Ġand ĠCA . </s> Background </s> Aut omatic Ġdi ac rit ics Ġrestoration Ġhas Ġbeen Ġinvestigated Ġfor Ġmany Ġdifferent Ġlanguage Ġsuch Ġas ĠEuropean Ġlanguages Ġ( e . g . ĠRomanian ĠB IB REF 8 , ĠB IB REF 9 , ĠFrench ĠB IB REF 10 , Ġand ĠCroatian ĠB IB REF 11 ), ĠAfrican Ġlanguages Ġ( e . g . ĠYor ba ĠB IB REF 12 ), ĠSoutheast ĠAsian Ġlanguages Ġ( e . g . ĠVietnamese ĠB IB REF 13 ), ĠSem itic Ġlanguage Ġ( e . g . ĠArabic Ġand ĠHebrew ĠB IB REF 14 ), Ġand Ġmany Ġothers ĠB IB REF 15 . ĠFor Ġmany Ġlanguages , Ġdi ac ritic Ġ( or Ġaccent Ġrestoration ) Ġis Ġlimited Ġto Ġa Ġhandful Ġof Ġletters . ĠHowever , Ġfor ĠSem itic Ġlanguages , Ġdi ac ritic Ġrecovery Ġextends Ġto Ġmost Ġletters . ĠMany Ġgeneral Ġapproaches Ġhave Ġbeen Ġexplored Ġfor Ġthis Ġproblem Ġincluding Ġlingu istically Ġmotivated Ġrule - based Ġapproaches , Ġmachine Ġlearning Ġapproaches , Ġsuch Ġas ĠHidden ĠMark ov ĠModels Ġ( H MM ) ĠB IB REF 14 Ġand ĠCond itional ĠRandom ĠFields Ġ( CR F ) ĠB IB REF 16 , Ġand Ġlately Ġdeep Ġlearning Ġapproaches Ġsuch Ġas ĠArabic ĠB IB REF 17 , ĠB IB REF 18 , ĠB IB REF 19 , ĠSlov ak ĠB IB REF 20 , Ġand ĠYor ba ĠB IB REF 12 . ĠAside Ġfrom Ġrule - based Ġapproaches ĠB IB REF 21 , Ġdifferent Ġmethods Ġwere Ġused Ġto Ġrecover Ġdi ac rit ics Ġin ĠArabic Ġtext . ĠUsing Ġa Ġhidden ĠMark ov Ġmodel Ġ( H MM ) ĠB IB REF 14 , ĠB IB REF 22 Ġwith Ġan Ġinput Ġcharacter Ġsequence , Ġthe Ġmodel Ġattempts Ġto Ġfind Ġthe Ġbest Ġstate Ġsequence Ġgiven Ġprevious Ġobservations . ĠB IB REF 14 Ġreported Ġa Ġ14 % Ġword Ġerror Ġrate Ġ( WER ) Ġwhile ĠB IB REF 22 Ġachieved Ġa Ġ4 . 1 % Ġdi ac ritic Ġerror Ġrate Ġ( DER ) Ġon Ġthe ĠQuran Ġ( CA ). ĠB IB REF 23 Ġcombined Ġboth Ġmorph ological , Ġacoustic , Ġand Ġcontextual Ġfeatures Ġto Ġbuild Ġa Ġdi ac rit izer Ġtrained Ġon ĠFB IS Ġand ĠL DC ĠCall Home ĠE CA Ġcollections . ĠThey Ġreported Ġa Ġ9 % Ġ( DER ) Ġwithout ĠCE , Ġand Ġ28 % ĠD ER Ġwith ĠCE . ĠB IB REF 24 Ġemployed Ġa Ġcascade Ġof Ġa Ġfinite Ġstate Ġtrans ducers . ĠThe Ġcascade Ġstacked Ġa Ġword Ġlanguage Ġmodel Ġ( LM ), Ġa Ġchar ach ter ĠLM , Ġand Ġa Ġmorph ological Ġmodel . ĠThe Ġmodel Ġachieved Ġan Ġaccuracy Ġof Ġ7 . 33 % ĠW ER Ġwithout ĠCE Ġand Ġand Ġ23 . 61 % ĠW ER Ġwith ĠCE . ĠB IB REF 25 Ġemployed Ġa Ġmaximum Ġentropy Ġmodel Ġfor Ġsequence Ġclassification . ĠThe Ġsystem Ġwas Ġtrained Ġon Ġthe ĠL DC âĢ Ļ s ĠArabic ĠTree bank Ġ( AT B ) Ġand Ġevaluated Ġon Ġa Ġ600 Ġarticles Ġfrom ĠAn - N ahar ĠNewspaper Ġ( 340 K Ġwords ) Ġand Ġachieved Ġ5 . 5 % ĠD ER Ġand Ġ18 % ĠW ER Ġon Ġwords Ġwithout ĠCE . </s> BI B REF 26 Ġused Ġa Ġhybrid Ġapproach Ġthat Ġutilizes Ġthe Ġoutput Ġof ĠAl k hal il Ġmorph ological ĠAnaly zer ĠB IB REF 27 Ġto Ġgenerate Ġall Ġpossible Ġout Ġof Ġcontext Ġdi ac rit izations Ġof Ġa Ġword . ĠThen , Ġan ĠH MM Ġguesses Ġthe Ġcorrect Ġdi ac rit ized Ġform . ĠSimilarly , ĠMicrosoft ĠArabic ĠTool kit ĠServices Ġ( AT KS ) Ġdi ac rit izer ĠB IB REF 28 Ġuses Ġa Ġrule - based Ġmorph ological Ġanaly zer Ġthat Ġproduces Ġpossible Ġanalyses Ġand Ġan ĠH MM Ġin Ġconjunction Ġwith Ġrules Ġto Ġguess Ġthe Ġmost Ġlikely Ġanalysis . ĠThey Ġreport ĠW ER Ġof Ġ11 . 4 % Ġand Ġ4 . 4 % Ġwith Ġand Ġwithout ĠCE . ĠMAD AMI RA ĠB IB REF 29 Ġuses Ġa Ġcombinations Ġof Ġmorph o - sy nt actic Ġfeatures Ġto Ġrank Ġa Ġlist Ġof Ġpotential Ġanalyses Ġprovided Ġby Ġthe ĠBuck wal ter ĠArabic ĠMorph ological ĠAnaly zer Ġ( B AMA ) ĠB IB REF 30 . ĠAn ĠS VM Ġtrained Ġon ĠAT B Ġselects Ġthe Ġmost Ġprobable Ġanalysis , Ġincluding Ġthe Ġdi ac rit ized Ġform . ĠMAD AMI RA Ġachieves Ġ19 . 0 % Ġand Ġ6 . 7 % ĠW ER Ġwith Ġand Ġwithout ĠCE Ġrespectively ĠB IB REF 31 . ĠFar asa ĠB IB REF 31 Ġuses Ġan ĠH MM Ġto Ġguess ĠCW Ġdi ac rit ics Ġand Ġan ĠS VM - rank Ġbased Ġmodel Ġtrained Ġon Ġmorph ological Ġand Ġsynt actic Ġfeatures Ġto Ġguess ĠCE s . ĠFar asa Ġachieves ĠW ER Ġof Ġ12 . 8 % Ġand Ġ3 . 3 % Ġwith Ġand Ġwithout ĠCE s . </s> More Ġrecent Ġwork Ġemployed Ġdifferent Ġneural Ġarchitectures Ġto Ġmodel Ġthe Ġdi ac rit ization Ġproblem . ĠB IB REF 17 Ġused Ġa Ġbi L ST M Ġrecurrent Ġneural Ġnetwork Ġtrained Ġon Ġthe Ġsame Ġdataset Ġas ĠB IB REF 25 . ĠThey Ġexplored Ġone , Ġtwo Ġand Ġthree ĠBi L ST M Ġlayers Ġwith Ġ250 Ġnodes Ġin Ġeach Ġlayers , Ġachieving ĠW ER Ġof Ġ9 . 1 % Ġincluding ĠCE Ġon ĠAT B . ĠSimilar Ġarchitectures Ġwere Ġused Ġbut Ġachieved Ġlower Ġresults ĠB IB REF 7 , ĠB IB REF 32 . ĠB IB REF 33 Ġprovide Ġa Ġcomprehensive Ġsurvey Ġon ĠArabic Ġdi ac rit ization . ĠA Ġmore Ġrecent Ġsurvey Ġby ĠB IB REF 34 Ġconcluded Ġthat Ġreported Ġresults Ġare Ġoften Ġincom parable Ġdue Ġto Ġthe Ġusage Ġof Ġdifferent Ġtest Ġsets . ĠThey Ġconcluded Ġthat Ġa Ġlarge Ġun ig ram ĠLM Ġfor ĠCW Ġdi ac ritic Ġrecovery Ġis Ġcompetitive Ġwith Ġmany Ġof Ġthe Ġsystems Ġin Ġthe Ġliterature , Ġwhich Ġprompted Ġus Ġto Ġutilize Ġa Ġun ig ram Ġlanguage Ġmodel Ġfor Ġpost Ġcorrection . ĠAs Ġmentioned Ġearlier , Ġtwo Ġconclusions Ġcan Ġbe Ġdrawn , Ġnamely : Ġrestoring ĠCE s Ġis Ġmore Ġchallenging Ġthan ĠCW Ġdi ac ritic Ġrestoration ; Ġand Ġcombining Ġmultiple Ġfeatures Ġtypically Ġimproves ĠCE Ġrestoration . </s> In Ġthis Ġpaper , Ġwe Ġexpand Ġupon Ġthe Ġwork Ġin Ġthe Ġliterature Ġby Ġintroducing Ġfeature - rich ĠD NN Ġmodels Ġfor Ġrestoring Ġboth ĠCW Ġand ĠCE Ġdi ac rit ics . ĠWe Ġcompare Ġour Ġmodels Ġto Ġmultiple Ġsystems Ġon Ġthe Ġsame Ġtest Ġset . ĠWe Ġachieve Ġresults Ġthat Ġreduce Ġdi ac rit ization Ġerror Ġrates Ġby Ġmore Ġthan Ġhalf Ġcompared Ġto Ġthe Ġbest ĠS OTA Ġsystems . ĠWe Ġfurther Ġconduct Ġan Ġab lation Ġstudy Ġto Ġdetermine Ġthe Ġrelative Ġeffect Ġof Ġthe Ġdifferent Ġfeatures . ĠAs Ġfor ĠArabic , Ġit Ġis Ġa ĠSem itic Ġlanguage Ġwith Ġderiv ational Ġmorphology . ĠArabic Ġnoun s , Ġadject ives , Ġad verbs , Ġand Ġverbs Ġare Ġtypically Ġderived Ġfrom Ġa Ġclosed Ġset Ġof Ġ10 , 000 Ġroots Ġof Ġlength Ġ3 , Ġ4 , Ġor Ġrarely Ġ5 . ĠArabic Ġnoun s Ġand Ġverbs Ġare Ġderived Ġfrom Ġroots Ġby Ġapplying Ġtemplates Ġto Ġthe Ġroots Ġto Ġgenerate Ġstems . ĠSuch Ġtemplates Ġmay Ġcarry Ġinformation Ġthat Ġindicate Ġmorph ological Ġfeatures Ġof Ġwords Ġsuch ĠPOS Ġtag , Ġgender , Ġand Ġnumber . ĠFor Ġexample , Ġgiven Ġa Ġ3 - letter Ġroot Ġwith Ġ3 Ġconson ants ĠC CC , Ġa Ġvalid Ġtemplate Ġmay Ġbe ĠC w ACC Ġ, Ġwhere Ġthe Ġinf ix ĠâĢ ľ w A âĢ Ŀ Ġ( ÙĪ Ø§ >) Ġis Ġinserted , Ġthis Ġtemplate Ġtypically Ġindicates Ġan ĠArabic Ġbroken , Ġor Ġirregular , Ġplural Ġtemplate Ġfor Ġa Ġnoun Ġof Ġtemplate ĠC ACC Ġor ĠC ACC p Ġif Ġmasculine Ġor Ġfeminine Ġrespectively . ĠFurther , Ġstems Ġmay Ġaccept Ġprefix es Ġand / or Ġsuffix es Ġto Ġform Ġwords . ĠPre fixes Ġinclude Ġcoordinating Ġconj unctions , Ġdetermin er , Ġand Ġpre pos itions , Ġand Ġsuffix es Ġinclude Ġattached Ġpronouns Ġand Ġgender Ġand Ġnumber Ġmarkers . </s> Our ĠDi ac rit izer Ġ: :: ĠTraining Ġand ĠTest ĠCorpor a </s> For ĠM SA , Ġwe Ġacquired Ġthe Ġdi ac rit ized Ġcorpus Ġthat Ġwas Ġused Ġto Ġtrain Ġthe ĠR DI ĠB IB REF 7 Ġdi ac rit izer Ġand Ġthe ĠFar asa Ġdi ac rit izer ĠB IB REF 31 . ĠThe Ġcorpus Ġcontains Ġ9 . 7 M Ġtokens Ġwith Ġapproximately Ġ194 K Ġunique Ġsurface Ġforms Ġ( excluding Ġnumbers Ġand Ġpunct uation Ġmarks ). ĠThe Ġcorpus Ġcovers Ġmultiple Ġgenres Ġsuch Ġas Ġpolitics Ġand Ġsports Ġand Ġis Ġa Ġmix Ġof ĠM SA Ġand ĠCA . ĠThis Ġcorpus Ġis Ġconsiderably Ġlarger Ġthan Ġthe ĠArabic ĠTree bank ĠB IB REF 35 Ġand Ġis Ġmore Ġconsistent Ġin Ġits Ġdi ac rit ization . ĠFor Ġtesting , Ġwe Ġused Ġthe Ġfreely Ġavailable ĠWiki News Ġtest Ġset ĠB IB REF 31 , Ġwhich Ġis Ġcomposed Ġof Ġ70 ĠM SA ĠWiki News Ġarticles Ġ( 18 , 300 Ġtokens ) Ġand Ġevenly Ġcovers Ġa Ġvariety Ġof Ġgenres Ġincluding Ġpolitics , Ġeconomics , Ġhealth , Ġscience Ġand Ġtechnology , Ġsports , Ġarts Ġand Ġculture . </s> For ĠCA , Ġwe Ġobtained Ġa Ġlarge Ġcollection Ġof Ġfully Ġdi ac rit ized Ġclassical Ġtexts Ġ( 2 . 7 M Ġtokens ) Ġfrom Ġa Ġbook Ġpublisher , Ġand Ġwe Ġheld - out Ġa Ġsmall Ġsubset Ġof Ġ5 , 000 Ġsentences Ġ( approximately Ġ400 k Ġwords ) Ġfor Ġtesting . ĠThen , Ġwe Ġused Ġthe Ġremaining Ġsentences Ġto Ġtrain Ġthe ĠCA Ġmodels . </s> Our ĠDi ac rit izer Ġ: :: ĠCore ĠWord ĠDi ac rit ization Ġ: :: ĠFeatures . </s> Arab ic Ġwords Ġare Ġtypically Ġderived Ġfrom Ġa Ġlimited Ġset Ġof Ġroots Ġby Ġfitting Ġthem Ġinto Ġso - called Ġstem - tem plates Ġ( producing Ġstems ) Ġand Ġmay Ġaccept Ġa Ġvariety Ġof Ġprefix es Ġand Ġsuffix es Ġsuch Ġas Ġpre pos itions , Ġdetermin ers , Ġand Ġpronouns Ġ( producing Ġwords ). ĠWord Ġstems Ġspecify Ġthe Ġlex ical Ġselection Ġand Ġare Ġtypically Ġunaffected Ġby Ġthe Ġattached Ġaff ix es . ĠWe Ġused Ġ4 Ġfeature Ġtypes , Ġnamely : </s> CHAR : Ġthe Ġcharacters . </s> SE G : Ġthe Ġposition Ġof Ġthe Ġcharacter Ġin Ġa Ġword Ġsegment . ĠFor Ġexample , Ġgiven Ġthe Ġword ĠâĢ ľ w Al kt Ab âĢ Ŀ Ġ( ÙĪ Ø§ÙĦ Ù ĥ Øª Ø§Ø ¨ > Ġand Ġthe Ġbook / writers ), Ġwhich Ġis Ġcomposed Ġof Ġ3 Ġsegments ĠâĢ ľ w + Al + kt Ab âĢ Ŀ Ġ( ÙĪ + Ø§ÙĦ + Ù ĥ Øª Ø§Ø ¨ > ). ĠLetters Ġwere Ġmarked Ġas ĠâĢ ľ B âĢ Ŀ Ġif Ġthey Ġbegin Ġa Ġsegment , ĠâĢ ľ M âĢ Ŀ Ġif Ġthey Ġare Ġin Ġthe Ġmiddle Ġof Ġa Ġsegment , ĠâĢ ľ E âĢ Ŀ Ġif Ġthey Ġend Ġa Ġsegment , Ġand ĠâĢ ľ S âĢ Ŀ Ġif Ġthey Ġare Ġsingle Ġletter Ġsegments . ĠSo Ġfor ĠâĢ ľ w + Al + kt Ab âĢ Ŀ , Ġthe Ġcorresponding Ġcharacter Ġpositions Ġare ĠâĢ ľ S + BE + BM ME âĢ Ŀ . ĠWe Ġused ĠFar asa Ġto Ġperform Ġsegment ation , Ġwhich Ġhas Ġa Ġreported Ġsegment ation Ġaccuracy Ġof Ġ99 % Ġon Ġthe ĠWiki News Ġdataset ĠB IB REF 36 . </s> PR IOR : Ġdi ac rit ics Ġseen Ġin Ġthe Ġtraining Ġset Ġper Ġsegment . ĠSince Ġwe Ġused Ġa Ġcharacter Ġlevel Ġmodel , Ġthis Ġfeature Ġinformed Ġthe Ġmodel Ġwith Ġword Ġlevel Ġinformation . ĠFor Ġexample , Ġthe Ġword ĠâĢ ľ kt Ab âĢ Ŀ Ġ( Ù ĥ Øª Ø§Ø ¨ >) Ġwas Ġobserved Ġto Ġhave Ġtwo Ġdi ac rit ized Ġforms Ġin Ġthe Ġtraining Ġset , Ġnamely ĠâĢ ľ k ita Ab âĢ Ŀ Ġ( Ù ĥ ÙĲ Øª Ùİ Ø§Ø ¨ > ĠâĢĵ Ġbook ) Ġand ĠâĢ ľ k ut $ \ sim Ġ$ a Ab âĢ Ŀ Ġ( Ù ĥ Ù ı Øª Ù ĳ Ùİ Ø§Ø ¨ > ĠâĢĵ Ġwriters ). ĠThe Ġfirst Ġletter Ġin Ġthe Ġword Ġ( âĢ ľ k âĢ Ŀ ) Ġaccepted Ġthe Ġdi ac rit ics ĠâĢ ľ i âĢ Ŀ Ġand ĠâĢ ľ u âĢ Ŀ . ĠThus Ġgiven Ġa Ġbinary Ġvector Ġrepresenting Ġwhether Ġa Ġcharacter Ġis Ġallowed Ġto Ġassume Ġany Ġof Ġthe Ġeight Ġprimitive ĠArabic Ġdi ac ritic Ġmarks Ġ( a , Ġi , Ġu , Ġo , ĠK , ĠN , ĠF , Ġand Ġ$\ sim Ġ$ Ġin Ġorder ), Ġthe Ġfirst Ġletter Ġwould Ġbe Ġgiven Ġthe Ġfollowing Ġvector ĠâĢ ľ 01 100 000 âĢ Ŀ . ĠIf Ġa Ġword Ġsegment Ġwas Ġnever Ġobserved Ġduring Ġtraining , Ġthe Ġvector Ġfor Ġall Ġletters Ġtherein Ġwould Ġbe Ġset Ġto Ġ111 11 111 . ĠThis Ġfeature Ġborrow s Ġinformation Ġfrom ĠH MM Ġmodels , Ġwhich Ġhave Ġbeen Ġfairly Ġsuccessful Ġin Ġdi ac rit izing Ġword Ġcores . </s> C ASE : Ġwhether Ġthe Ġletter Ġexpects Ġa Ġcore Ġword Ġdi ac ritic Ġor Ġa Ġcase Ġending . ĠCase Ġendings Ġare Ġplaced Ġon Ġonly Ġone Ġletter Ġin Ġa Ġword , Ġwhich Ġmay Ġor Ġmay Ġnot Ġbe Ġthe Ġlast Ġletter Ġin Ġthe Ġword . ĠThis Ġis Ġa Ġbinary Ġfeature . </s> Our ĠDi ac rit izer Ġ: :: ĠCore ĠWord ĠDi ac rit ization Ġ: :: ĠD NN ĠModel . </s> Using Ġa ĠD NN Ġmodel , Ġparticularly Ġwith Ġa Ġbi L ST M ĠB IB REF 37 , Ġis Ġadvantageous Ġbecause Ġthe Ġmodel Ġautomatically Ġexplores Ġthe Ġspace Ġof Ġfeature Ġcombinations Ġand Ġis Ġable Ġto Ġcapture Ġdistant Ġdependencies . ĠA Ġnumber Ġof Ġstudies Ġhave Ġexplored Ġvarious Ġbi L ST M Ġarchitectures ĠB IB REF 17 , ĠB IB REF 7 , ĠB IB REF 32 Ġincluding Ġstacked Ġbi L ST Ms Ġconfirming Ġtheir Ġeffectiveness . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 14 , Ġwe Ġemployed Ġa Ġcharacter - based Ġbi L ST M Ġmodel Ġwith Ġassociated Ġfeatures Ġfor Ġeach Ġcharacter . ĠEvery Ġinput Ġcharacter Ġhad Ġan Ġassociated Ġlist Ġof Ġ$ m $ Ġfeatures , Ġand Ġwe Ġtrained Ġrandomly Ġinitialized Ġembed d ings Ġof Ġsize Ġ50 Ġfor Ġeach Ġfeature . ĠThen , Ġwe Ġconc aten ated Ġthe Ġfeature Ġembed d ings Ġvectors Ġcreating Ġan Ġ$ m \ times Ġ50 $ Ġvector Ġfor Ġeach Ġcharacter , Ġwhich Ġwas Ġfed Ġinto Ġthe Ġbi L ST M Ġlayer Ġof Ġlength Ġ100 . ĠThe Ġoutput Ġof Ġthe Ġbi L ST M Ġlayer Ġwas Ġfed Ġdirectly Ġinto Ġa Ġdense Ġlayer Ġof Ġsize Ġ100 . ĠWe Ġused Ġearly Ġstopping Ġwith Ġpatience Ġof Ġ5 Ġepoch s , Ġa Ġlearning Ġrate Ġof Ġ0 . 001 , Ġa Ġbatch Ġsize Ġof Ġ256 , Ġand Ġan ĠAdam ax Ġoptim izer . ĠThe Ġinput Ġwas Ġthe Ġcharacter Ġsequence Ġin Ġa Ġsentence Ġwith Ġwords Ġbeing Ġseparated Ġby Ġword Ġboundary Ġmarkers Ġ( WB ), Ġand Ġwe Ġset Ġthe Ġmaximum Ġsentence Ġlength Ġto Ġ1 , 250 Ġcharacters . </s> Our ĠDi ac rit izer Ġ: :: ĠCase ĠEnding ĠDi ac rit ization Ġ: :: ĠFeatures . </s> Table ĠT AB REF 17 Ġlists Ġthe Ġfeatures Ġthat Ġwe Ġused Ġfor ĠCE Ġrecovery . ĠWe Ġused ĠFar asa Ġto Ġperform Ġsegment ation Ġand ĠPOS Ġtagging Ġand Ġto Ġdetermine Ġstem - tem plates ĠB IB REF 31 . ĠFar asa Ġhas Ġa Ġreported ĠPOS Ġaccuracy Ġof Ġ96 % Ġon Ġthe ĠWiki News Ġdataset ĠB IB REF 31 . ĠThough Ġthe ĠFar asa Ġdi ac rit izer Ġutilizes Ġa Ġcombination Ġof Ġsome Ġthe Ġfeatures Ġpresented Ġherein , Ġnamely Ġsegment ation , ĠPOS Ġtagging , Ġand Ġstem Ġtemplates , ĠFar asa 's ĠS VM - ranking Ġapproach Ġrequires Ġexplicit Ġspecification Ġof Ġfeature Ġcombinations Ġ( ex . Ġ$ Pro b ( CE \ Vert Ġcurrent \ _ word , Ġprev \ _ word , Ġprev \ _ CE ) $ ). ĠManual Ġexploration Ġof Ġthe Ġfeature Ġspace Ġis Ġundesirable , Ġand Ġideally Ġwe Ġwould Ġwant Ġour Ġlearning Ġalgorithm Ġto Ġdo Ġso Ġautomatically . ĠThe Ġflexibility Ġof Ġthe ĠD NN Ġmodel Ġallowed Ġus Ġto Ġinclude Ġmany Ġmore Ġsurface Ġlevel Ġfeatures Ġsuch Ġas Ġaff ix es , Ġleading Ġand Ġtrailing Ġcharacters Ġin Ġwords Ġand Ġstems , Ġand Ġthe Ġpresence Ġof Ġwords Ġin Ġlarge Ġg az ette ers Ġof Ġnamed Ġentities . ĠAs Ġwe Ġshow Ġlater , Ġthese Ġadditional Ġfeatures Ġsignificantly Ġlowered ĠCE ER . </s> Our ĠDi ac rit izer Ġ: :: ĠCase ĠEnding ĠDi ac rit ization Ġ: :: ĠD NN ĠModel </s> Figure ĠFIG REF 19 Ġshows Ġthe Ġarchitecture Ġof Ġour ĠD NN Ġalgorithm . ĠEvery Ġinput Ġword Ġhad Ġan Ġassociated Ġlist Ġof Ġ$ n $ Ġfeatures , Ġand Ġwe Ġtrained Ġrandomly Ġinitialized Ġembed d ings Ġof Ġsize Ġ100 Ġfor Ġeach Ġfeature . ĠThen , Ġwe Ġconc aten ated Ġthe Ġfeature Ġembed d ings Ġvectors Ġcreating Ġan Ġ$ n \ times Ġ100 $ Ġvector Ġfor Ġeach Ġword . ĠWe Ġfed Ġthese Ġvectors Ġinto Ġa Ġbi L ST M Ġlayer Ġof Ġ100 Ġdimensions Ġafter Ġapplying Ġa Ġdrop out Ġof Ġ75 %, Ġwhere Ġdrop out Ġbehaves Ġlike Ġa Ġregular l izer Ġto Ġavoid Ġover fitting ĠB IB REF 40 . ĠWe Ġconducted Ġside Ġexperiments Ġwith Ġlower Ġdrop out Ġrates , Ġbut Ġthe Ġhigher Ġdrop out Ġrate Ġworked Ġbest . ĠThe Ġoutput Ġof Ġthe Ġbi L ST M Ġlayer Ġwas Ġfed Ġinto Ġa Ġ100 Ġdimensional Ġdense Ġlayer Ġwith Ġ15 % Ġdrop out Ġand Ġsoft max Ġactivation . ĠWe Ġconducted Ġside Ġexperiments Ġwhere Ġwe Ġadded Ġadditional Ġbi L ST M Ġlayers Ġand Ġreplaced Ġsoft max Ġwith Ġa Ġconditional Ġrandom Ġfield Ġlayer , Ġbut Ġwe Ġdid Ġnot Ġobserve Ġimprovements . ĠThus , Ġwe Ġopted Ġfor Ġa Ġsimpler Ġmodel . ĠWe Ġused Ġa Ġvalidation Ġset Ġto Ġdetermine Ġoptimal Ġparameters Ġsuch Ġas Ġdrop out Ġrate . ĠAgain , Ġwe Ġused Ġthe ĠâĢ ľ Adam ax âĢ Ŀ Ġoptim izer Ġwith Ġcateg orical Ġcross Ġentropy Ġloss Ġand Ġa Ġlearning Ġrate Ġof Ġ0 . 001 . ĠWe Ġalso Ġapplied Ġearly Ġstopping Ġwith Ġpatience Ġof Ġup Ġto Ġ5 Ġconsecutive Ġepoch s Ġwithout Ġimprovement . </s> Exper iments Ġand ĠResults Ġ: :: ĠCore ĠWord Ġ: :: ĠExperimental ĠSetup </s> For Ġall Ġthe Ġexperiments Ġconducted Ġherein , Ġwe Ġused Ġthe ĠKer as Ġtool kit ĠB IB REF 41 Ġwith Ġa ĠT ensor Flow Ġbackend ĠB IB REF 42 . ĠWe Ġused Ġthe Ġentirety Ġof Ġthe Ġtraining Ġset Ġas Ġinput , Ġand Ġwe Ġinstructed ĠKer as Ġto Ġuse Ġ5 % Ġof Ġthe Ġdata Ġfor Ġtuning Ġ( valid ation ). ĠWe Ġincluded Ġthe ĠCASE Ġfeature , Ġwhich Ġspecifies Ġwhether Ġthe Ġletter Ġaccepts Ġa Ġnormal Ġdi ac ritic Ġor Ġcase Ġending , Ġin Ġall Ġour Ġsetups . ĠWe Ġconducted Ġmultiple Ġexperiment Ġusing Ġdifferent Ġfeatures , Ġnamely : </s> CHAR : ĠThis Ġis Ġour Ġbaseline Ġsetup Ġwhere Ġwe Ġonly Ġused Ġthe Ġcharacters Ġas Ġfeatures . </s> CHAR + SE G : ĠThis Ġtakes Ġthe Ġcharacters Ġand Ġtheir Ġsegment ation Ġinformation Ġas Ġfeatures . </s> CHAR + PR IOR : ĠThis Ġtakes Ġthe Ġcharacters Ġand Ġtheir Ġthe Ġobserved Ġdi ac rit ized Ġforms Ġin Ġthe Ġtraining Ġset . </s> All : ĠThis Ġsetup Ġincludes Ġall Ġthe Ġfeatures . </s> We Ġalso Ġoptionally Ġemployed Ġpost Ġcorrection . ĠFor Ġwords Ġthat Ġwere Ġseen Ġin Ġtraining , Ġif Ġthe Ġmodel Ġproduced Ġa Ġdi ac rit ized Ġform Ġthat Ġwas Ġnot Ġseen Ġin Ġthe Ġtraining Ġdata , Ġwe Ġassumed Ġit Ġwas Ġan Ġerror Ġand Ġreplaced Ġit Ġwith Ġthe Ġmost Ġfrequently Ġobserved Ġdi ac rit ized Ġform Ġ( using Ġa Ġun ig ram Ġlanguage Ġmodel ). ĠWe Ġreport Ġtwo Ġerror Ġrates , Ġnamely ĠW ER Ġ( at Ġword Ġlevel ) Ġand ĠD ER Ġ( at Ġcharacter Ġlevel ). ĠWe Ġused Ġrelaxed Ġscoring Ġwhere Ġwe Ġassumed Ġan Ġempty Ġcase Ġto Ġbe Ġequivalent Ġto Ġsu kun , Ġand Ġwe Ġremoved Ġdefault Ġdi ac rit ics ĠâĢĵ Ġfat Ha Ġfollowed Ġby Ġale f , Ġk as ra Ġfollowed Ġby Ġya , Ġand Ġdam ma Ġfollowed Ġby Ġwa . ĠUsing Ġsuch Ġscoring Ġwould Ġallow Ġto Ġcompare Ġto Ġother Ġsystems Ġin Ġthe Ġliterature Ġthat Ġmay Ġuse Ġdifferent Ġdi ac rit ization Ġconventions . </s> Exper iments Ġand ĠResults Ġ: :: ĠCore ĠWord Ġ: :: ĠResults Ġand ĠError Ġanalysis </s> For Ġtesting , Ġwe Ġused Ġthe Ġaforementioned ĠWiki News Ġdataset Ġto Ġtest Ġthe ĠM SA Ġdi ac rit izer Ġand Ġthe Ġheld - out Ġ5 , 000 Ġsentences Ġfor ĠCA . ĠTable ĠT AB REF 27 Ġshows ĠW ER Ġand ĠD ER Ġresults Ġusing Ġdifferent Ġfeatures Ġwith Ġand Ġwithout Ġpost Ġcorrection . </s> Exper iments Ġand ĠResults Ġ: :: ĠCore ĠWord Ġ: :: ĠResults Ġand ĠError Ġanalysis Ġ: :: ĠM SA ĠResults : </s> For ĠM SA , Ġthough Ġthe ĠCHAR + PR IOR Ġfeature Ġled Ġto Ġworse Ġresults Ġthan Ġusing ĠCHAR Ġalone , Ġthe Ġresults Ġshow Ġthat Ġcombining Ġall Ġthe Ġfeatures Ġachieved Ġthe Ġbest Ġresults . ĠMoreover , Ġpost Ġcorrection Ġimproved Ġresults Ġoverall . ĠWe Ġcompare Ġour Ġresults Ġto Ġfive Ġother Ġsystems , Ġnamely ĠFar asa ĠB IB REF 31 , ĠMAD AMI RA ĠB IB REF 29 , ĠR DI Ġ( R ash wan Ġet Ġal ., Ġ2015 ), ĠMIT Ġ( Bel ink ow Ġand ĠGlass , Ġ2015 ), Ġand ĠMicrosoft ĠATK S ĠB IB REF 28 . ĠTable ĠT AB REF 34 Ġcompares Ġour Ġsystem Ġwith Ġothers Ġin Ġthe Ġaforementioned Ġsystems . ĠAs Ġthe Ġresults Ġshow , Ġour Ġresults Ġbeat Ġthe Ġcurrent Ġstate - of - the - art . </s> For Ġerror Ġanalysis , Ġwe Ġanalyzed Ġall Ġthe Ġerrors Ġ( 5 27 Ġerrors ). ĠThe Ġerrors Ġtypes Ġalong Ġwith Ġexamples Ġof Ġeach Ġare Ġshown Ġin ĠTable ĠT AB REF 29 . ĠThe Ġmost Ġprominent Ġerror Ġtype Ġarises Ġfrom Ġthe Ġselection Ġof Ġa Ġvalid Ġdi ac rit ized Ġform Ġthat Ġdoes Ġnot Ġmatch Ġthe Ġcontext Ġ( 40 . 8 %). ĠPerhaps , Ġincluding ĠPOS Ġtags Ġas Ġa Ġfeature Ġor Ġaugment ing Ġthe ĠPR IOR Ġfeature Ġwith ĠPOS Ġtag Ġinformation Ġand Ġa Ġbig ram Ġlanguage Ġmodel Ġmay Ġreduce Ġthe Ġerror Ġrate Ġfurther . ĠThe Ġsecond Ġmost Ġcommon Ġerror Ġis Ġdue Ġto Ġtransl iter ated Ġforeign Ġwords Ġincluding Ġforeign Ġnamed Ġentities Ġ( 23 . 5 %). ĠSuch Ġwords Ġwere Ġnot Ġobserved Ġduring Ġtraining . ĠFurther , ĠArabic ĠNamed Ġentities Ġaccount Ġfor Ġ10 . 6 % Ġof Ġthe Ġerrors , Ġwhere Ġthey Ġwere Ġeither Ġnot Ġseen Ġin Ġtraining Ġor Ġthey Ġshare Ġidentical Ġnon - d iac rit ized Ġforms Ġwith Ġother Ġwords . ĠPerhaps , Ġbuilding Ġlarger Ġg az ette ers Ġof Ġdi ac rit ized Ġnamed Ġentities Ġmay Ġresolve ĠNE Ġrelated Ġerrors . ĠIn Ġ10 . 8 % Ġof Ġthe Ġcases , Ġthe Ġdi ac rit izer Ġproduced Ġin Ġcompletely Ġincorrect Ġdi ac rit ized Ġforms . ĠIn Ġsome Ġthe Ġcases Ġ( 9 . 1 %), Ġthough Ġthe Ġdi ac rit izer Ġproduced Ġa Ġform Ġthat Ġis Ġdifferent Ġfrom Ġthe Ġreference , Ġboth Ġforms Ġwere Ġin Ġfact Ġcorrect . ĠMost Ġof Ġthese Ġcases Ġwere Ġdue Ġto Ġvariations Ġin Ġdi ac rit ization Ġconventions Ġ( ex . ĠâĢ ľ bare Ġale f âĢ Ŀ Ġ( A ) Ġat Ġstart Ġof Ġa Ġword Ġreceiving Ġa Ġdi ac ritic Ġor Ġnot ). ĠOther Ġcases Ġinclude Ġforeign Ġwords Ġand Ġsome Ġwords Ġwhere Ġboth Ġdi ac rit ized Ġforms Ġare Ġequally Ġvalid . </s> Exper iments Ġand ĠResults Ġ: :: ĠCore ĠWord Ġ: :: ĠResults Ġand ĠError Ġanalysis Ġ: :: ĠCA ĠResults : </s> For ĠCA Ġresults , Ġthe ĠCHAR + SE G Ġand ĠCHAR + PR IOR Ġperformed Ġbetter Ġthan Ġusing Ġcharacters Ġalone Ġwith ĠCHAR + PR IOR Ġperforming Ġbetter Ġthan ĠCHAR + SE G . ĠAs Ġin Ġthe Ġcase Ġwith ĠM SA , Ġcombining Ġall Ġthe Ġfeatures Ġled Ġto Ġthe Ġbest Ġresults . ĠPost Ġcorrection Ġhad Ġa Ġsignificantly Ġlarger Ġpositive Ġimpact Ġon Ġresults Ġcompared Ġto Ġwhat Ġwe Ġobserved Ġfor ĠM SA . ĠThis Ġmay Ġindicate Ġthat Ġwe Ġneed Ġa Ġlarger Ġtraining Ġset . ĠThe Ġbest ĠW ER Ġthat Ġwe Ġachieved Ġfor ĠCW Ġdi ac rit ics Ġwith Ġpost Ġcorrections Ġis Ġ2 . 2 %. ĠSince Ġwe Ġdid Ġnot Ġhave Ġaccess Ġto Ġany Ġpublicly Ġavailable Ġsystem Ġthat Ġis Ġtuned Ġfor ĠCA , Ġwe Ġcompared Ġour Ġbest Ġsystem Ġto Ġusing Ġour Ġbest ĠM SA Ġsystem Ġto Ġdi ac rit ize Ġthe ĠCA Ġtest Ġset , Ġand Ġthe ĠM SA Ġdi ac rit izer Ġproduced Ġsignificantly Ġlower Ġresults Ġwith Ġa ĠW ER Ġof Ġ8 . 5 % Ġ( see ĠTable ĠT AB REF 34 ). ĠThis Ġhighlights Ġthe Ġlarge Ġdifference Ġbetween ĠM SA Ġand ĠCA Ġand Ġthe Ġneed Ġfor Ġsystems Ġthat Ġare Ġspecifically Ġtuned Ġfor Ġboth . </s> We Ġrandomly Ġselected Ġand Ġanalyzed Ġ500 Ġerrors Ġ( 5 . 2 % Ġof Ġthe Ġerrors ). ĠThe Ġerrors Ġtypes Ġalong Ġwith Ġexamples Ġof Ġeach Ġare Ġshown Ġin ĠTable ĠT AB REF 33 . ĠThe Ġtwo Ġmost Ġcommon Ġerrors Ġinvolve Ġthe Ġsystem Ġproducing Ġcompletely Ġcorrect Ġdi ac rit ized Ġforms Ġ( 38 . 8 %) Ġor Ġcorrect Ġforms Ġthat Ġdon 't Ġmatch Ġthe Ġcontext Ġ( 31 . 4 %). ĠThe Ġrelatively Ġhigher Ġpercentage Ġof Ġcompletely Ġincorrect Ġguesses , Ġcompared Ġto ĠM SA , Ġmay Ġpoint Ġto Ġthe Ġhigher Ġlex ical Ġdiversity Ġof Ġclassical ĠArabic . ĠAs Ġfor ĠM SA , Ġwe Ġsuspect Ġthat Ġadding Ġadditional ĠPOS Ġinformation Ġand Ġemploying Ġa Ġword Ġbig ram Ġto Ġconst rain Ġthe ĠPR IOR Ġfeature Ġmay Ġhelp Ġreduce Ġselection Ġerrors . ĠAnother Ġprominent Ġerror Ġis Ġrelated Ġto Ġthe Ġdi ac rit ics Ġthat Ġappear Ġon Ġattached Ġsuffix es , Ġparticularly Ġpronouns , Ġwhich Ġdepend Ġon Ġthe Ġchoice Ġof Ġcase Ġending Ġ( 13 . 2 %). ĠErrors Ġdue Ġto Ġnamed Ġentities Ġare Ġslightly Ġfewer Ġthan Ġthose Ġseen Ġfor ĠM SA Ġ( 8 . 8 %). ĠA Ġnoticeable Ġnumber Ġof Ġmism atches Ġbetween Ġthe Ġguess Ġand Ġthe Ġreference Ġare Ġdue Ġto Ġpartial Ġdi ac rit ization Ġof Ġthe Ġreference Ġ( 4 . 4 %). ĠWe Ġplan Ġto Ġconduct Ġan Ġextra Ġround Ġof Ġchecks Ġon Ġthe Ġtest Ġset . </s> Exper iments Ġand ĠResults Ġ: :: ĠCase ĠEnding Ġ: :: ĠExperimental ĠSetup </s> We Ġconducted Ġmultiple Ġexperiments Ġto Ġdetermine Ġthe Ġrelative Ġeffect Ġof Ġthe Ġdifferent Ġfeatures Ġas Ġfollows : </s> word : ĠThis Ġis Ġour Ġbaseline Ġsetup , Ġwhich Ġuses Ġword Ġsurface Ġforms Ġonly . </s> word - surface : ĠThis Ġsetup Ġuses Ġthe Ġword Ġsurface Ġforms , Ġstems , Ġprefix es , Ġand Ġsuffix es Ġ( including Ġnoun Ġsuffix es ). ĠThis Ġsim ulates Ġthe Ġcase Ġwhen Ġno ĠPOS Ġtagging Ġinformation Ġis Ġavailable . </s> word - POS : ĠThis Ġincludes Ġthe Ġword Ġsurface Ġform Ġand ĠPOS Ġinformation , Ġincluding Ġgender Ġand Ġnumber Ġof Ġstems , Ġprefix es , Ġand Ġsuffix es . </s> word - morph : ĠThis Ġincludes Ġwords Ġand Ġtheir Ġstem Ġtemplates Ġto Ġcapture Ġmorph ological Ġpatterns . </s> word - surface - POS - morph : ĠThis Ġsetup Ġuses Ġall Ġthe Ġfeatures Ġ( surface , ĠPOS , Ġand Ġmorph ological ). </s> all - misc : ĠThis Ġuses Ġall Ġfeatures Ġplus Ġword Ġand Ġstem Ġleading Ġand Ġtrailing Ġcharacter Ġun ig rams Ġand Ġbig rams Ġin Ġaddition Ġto Ġsu kun Ġwords Ġand Ġnamed Ġentities . </s> For Ġtesting ĠM SA , Ġwe Ġused Ġthe Ġaforementioned ĠWiki News Ġdataset . ĠAgain , Ġwe Ġcompared Ġour Ġresults Ġto Ġfive Ġother Ġsystems , Ġnamely ĠFar asa ĠB IB REF 31 , ĠMAD AMI RA ĠB IB REF 29 , ĠR DI Ġ( R ash wan Ġet Ġal ., Ġ2015 ), ĠMIT Ġ( Bel ink ow Ġand ĠGlass , Ġ2015 ), Ġand ĠMicrosoft ĠATK S ĠB IB REF 28 . ĠFor ĠCA Ġtesting , Ġwe Ġused Ġthe Ġ5 , 000 Ġsentences Ġthat Ġwe Ġset Ġaside . ĠAgain , Ġwe Ġcompared Ġto Ġour Ġbest ĠM SA Ġsystem . </s> Exper iments Ġand ĠResults Ġ: :: ĠCase ĠEnding Ġ: :: ĠResults Ġand ĠError ĠAnalysis </s> Table ĠT AB REF 45 Ġlists Ġthe Ġresults Ġof Ġour Ġsetups Ġcompared Ġto Ġother Ġsystems . </s> Exper iments Ġand ĠResults Ġ: :: ĠCase ĠEnding Ġ: :: ĠResults Ġand ĠError ĠAnalysis Ġ: :: ĠM SA ĠResults : </s> As Ġthe Ġresults Ġshow , Ġour Ġbaseline ĠD NN Ġsystem Ġoutper forms Ġall Ġstate - of - the - art Ġsystems . ĠFurther , Ġadding Ġmore Ġfeatures Ġyielded Ġbetter Ġresults Ġoverall . ĠSurface - level Ġfeatures Ġresulted Ġin Ġthe Ġmost Ġgain , Ġfollowed Ġby ĠPOS Ġtags , Ġand Ġlast ly Ġstem Ġtemplates . ĠFurther , Ġadding Ġhead Ġand Ġtail Ġcharacters Ġalong Ġwith Ġa Ġlist Ġof Ġsu kun Ġwords Ġand Ġnamed Ġentities Ġled Ġto Ġfurther Ġimprovement . ĠOur Ġproposed Ġfeature - rich Ġsystem Ġhas Ġa ĠCE ER Ġthat Ġis Ġapproximately Ġ61 % Ġlower Ġthan Ġany Ġof Ġthe Ġstate - of - the - art Ġsystems . </s> Figure ĠFIG REF 49 Ġshows ĠCE Ġdistribution Ġand Ġprediction Ġaccuracy . ĠFor Ġthe Ġfour Ġbasic Ġmarkers Ġk as ra , Ġfat Ha , Ġdam ma Ġand Ġsu kun , Ġwhich Ġappear Ġ27 %, Ġ14 %, Ġ9 % Ġand Ġ10 % Ġrespectively , Ġthe Ġsystem Ġhas ĠCE ER Ġof Ġ$\ sim Ġ$ 1 % Ġfor Ġeach . ĠDetect ing Ġthe Ġvirtual ĠCE Ġmark Ġis Ġa Ġfairly Ġeasy Ġtask . ĠAll Ġother ĠCE Ġmarkers Ġrepresent Ġ13 % Ġwith Ġalmost Ġnegligible Ġerrors . ĠTable ĠT AB REF 30 Ġlists Ġa Ġthorough Ġbreakdown Ġof Ġall Ġerrors Ġaccounting Ġfor Ġat Ġ1 % Ġof Ġthe Ġerrors Ġalong Ġwith Ġthe Ġmost Ġcommon Ġreasons Ġof Ġthe Ġerrors Ġand Ġexamples Ġillustrating Ġthese Ġreasons . ĠFor Ġexample , Ġthe Ġmost Ġcommon Ġerror Ġtype Ġinvolves Ġguessing Ġa Ġfat Ha Ġ( a ) Ġinstead Ġof Ġdam ma Ġ( u ) Ġor Ġvice Ġversa Ġ( 19 . 3 %). ĠThe Ġmost Ġcommon Ġreasons Ġfor Ġthis Ġerror Ġtype , Ġbased Ġon Ġinspecting Ġthe Ġerrors , Ġwere Ġdue Ġto : ĠPOS Ġerrors Ġ( ex . Ġa Ġword Ġis Ġtagged Ġas Ġa Ġverb Ġinstead Ġof Ġa Ġnoun ); Ġand Ġa Ġnoun Ġis Ġtreated Ġas Ġa Ġsubject Ġinstead Ġof Ġan Ġobject Ġor Ġvice Ġversa . ĠThe Ġtable Ġdetails Ġthe Ġrest Ġof Ġthe Ġerror Ġtypes . ĠOverall , Ġsome Ġof Ġthe Ġerrors Ġare Ġpotentially Ġfix able Ġusing Ġbetter ĠPOS Ġtagging , Ġimproved Ġdetection Ġof Ġnon - Arab ized Ġforeign Ġnames , Ġand Ġdetection Ġof Ġind e cl in ability . ĠHowever , Ġsome Ġerrors Ġare Ġmore Ġdifficult Ġand Ġrequire Ġgreater Ġunderstanding Ġof Ġsemantics Ġsuch Ġas Ġimproper Ġattachment , Ġincorrect Ġid afa , Ġand Ġconfusion Ġbetween Ġsubject Ġand Ġobject . ĠPerhaps , Ġsuch Ġsemantic Ġerrors Ġcan Ġbe Ġresolved Ġusing Ġparsing . </s> Exper iments Ġand ĠResults Ġ: :: ĠCase ĠEnding Ġ: :: ĠResults Ġand ĠError ĠAnalysis Ġ: :: ĠCA ĠResults : </s> The Ġresults Ġshow Ġthat Ġthe ĠPOS Ġtagging Ġfeatures Ġled Ġto Ġthe Ġmost Ġimprovements Ġfollowed Ġby Ġthe Ġsurface Ġfeatures . ĠComb ining Ġall Ġfeatures Ġled Ġto Ġthe Ġbest Ġresults Ġwith ĠW ER Ġof Ġ2 . 5 %. ĠAs Ġwe Ġsaw Ġfor ĠCW Ġdi ac rit ics , Ġusing Ġour Ġbest ĠM SA Ġsystem Ġto Ġdi ac rit ize ĠCA Ġled Ġto Ġsignificantly Ġlower Ġresults Ġwith ĠCE ER Ġof Ġ8 . 9 %. </s> Figure ĠFIG REF 50 Ġshows ĠCE Ġdistribution Ġand Ġprediction Ġaccuracy . ĠFor Ġthe Ġfour Ġbasic Ġmarkers Ġfat Ha , Ġk as ra , Ġdam ma Ġand Ġsu kun , Ġwhich Ġappear Ġ18 %, Ġ14 %, Ġ13 % Ġand Ġ8 % Ġrespectively , Ġthe Ġsystem Ġhas ĠCE ER Ġ$\ sim Ġ$ 0 . 5 % Ġfor Ġeach . ĠAgain , Ġdetecting Ġthe Ġvirtual ĠCE Ġmark Ġwas Ġa Ġfairly Ġeasy Ġtask . ĠAll Ġother ĠCE Ġmarkers Ġrepresenting Ġ20 % Ġhave Ġnegligible Ġerrors . </s> Table ĠT AB REF 31 Ġlists Ġall Ġthe Ġerror Ġtypes , Ġwhich Ġaccount Ġfor Ġat Ġleast Ġ1 % Ġof Ġthe Ġerrors , Ġalong Ġwith Ġtheir Ġmost Ġcommon Ġcauses Ġand Ġexplanatory Ġexamples . ĠThe Ġerror Ġtypes Ġare Ġsimilar Ġto Ġthose Ġobserved Ġfor ĠM SA . ĠSome Ġerrors Ġare Ġmore Ġsynt actic Ġand Ġmorph ological Ġin Ġnature Ġand Ġcan Ġbe Ġaddressed Ġusing Ġbetter ĠPOS Ġtagging Ġand Ġidentification Ġof Ġind e cl in ability , Ġparticularly Ġas Ġthey Ġrelate Ġto Ġnamed Ġentities Ġand Ġnoun s Ġwith Ġfeminine Ġmarkers . ĠOther Ġerrors Ġsuch Ġas Ġincorrect Ġattachment , Ġincorrect Ġid afa , Ġfalse Ġsubject , Ġand Ġconfusion Ġbetween Ġsubject Ġand Ġobject Ġcan Ġperhaps Ġbenefit Ġfrom Ġthe Ġuse Ġof Ġparsing . ĠAs Ġwith Ġthe Ġcore - word Ġerrors Ġfor ĠCA , Ġthe Ġreference Ġhas Ġsome Ġerrors Ġ( ex . Ġ{ a , i , o } Ġ$\ Right arrow Ġ$ Ġ# ), Ġand Ġextra Ġrounds Ġof Ġreviews Ġof Ġthe Ġreference Ġare Ġin Ġorder . </s> Exper iments Ġand ĠResults Ġ: :: ĠFull ĠDi ac rit ization ĠResults </s> Table ĠT AB REF 48 Ġcompares Ġthe Ġfull Ġword Ġdi ac rit ization Ġ( CW + CE ) Ġof Ġour Ġbest Ġsetup Ġto Ġother Ġsystems Ġin Ġthe Ġliterature . ĠAs Ġthe Ġresults Ġshow Ġfor ĠM SA , Ġour Ġoverall Ġdi ac rit ization ĠW ER Ġis Ġ6 . 0 % Ġwhile Ġthe Ġstate Ġof Ġthe Ġart Ġsystem Ġhas Ġa ĠW ER Ġof Ġ12 . 2 %. ĠAs Ġfor ĠCA , Ġour Ġbest Ġsystem Ġproduced Ġan Ġerror Ġrate Ġof Ġ4 . 3 %, Ġwhich Ġis Ġsignificantly Ġbetter Ġthan Ġusing Ġour Ġbest ĠM SA Ġsystem Ġto Ġdi ac rit ize ĠCA . </s> Conclusion Ġand ĠFuture ĠWork </s> In Ġthis Ġpaper , Ġwe Ġpresented Ġa Ġfeature - rich ĠD NN Ġapproach Ġfor ĠM SA ĠCW Ġand ĠCE Ġrecovery Ġthat Ġproduces Ġa Ġword Ġlevel Ġerror Ġfor ĠM SA Ġof Ġ6 . 0 %, Ġwhich Ġis Ġmore Ġthan Ġ50 % Ġlower Ġthan Ġstate - of - the - art Ġsystems Ġ( 6 . 0 % Ġcompared Ġto Ġ12 . 2 %) Ġand Ġword Ġerror Ġrate Ġof Ġ4 . 3 % Ġfor ĠCA . ĠSpecifically , Ġwe Ġused Ġbi L ST M - based Ġmodel Ġwith Ġa Ġvariety Ġof Ġsurface , Ġmorph ological , Ġand Ġsynt actic Ġfeatures . ĠRel iable ĠN LP Ġtools Ġmay Ġbe Ġrequired Ġto Ġgenerate Ġsome Ġof Ġthese Ġfeatures , Ġand Ġsuch Ġtools Ġmay Ġnot Ġbe Ġreadily Ġavailable Ġfor Ġother Ġlanguage Ġvarieties , Ġsuch Ġas Ġdialect al ĠArabic . ĠHowever , Ġwe Ġshowed Ġthe Ġefficacy Ġof Ġdifferent Ġvarieties Ġof Ġfeatures , Ġsuch Ġas Ġsurface Ġlevel - features , Ġand Ġthey Ġcan Ġhelp Ġimprove Ġdi ac rit ization Ġindividually . ĠFurther , Ġthough Ġsome Ġerrors Ġmay Ġbe Ġovercome Ġusing Ġimproved ĠN LP Ġtools Ġ( ex . Ġbetter ĠPOS Ġtagging ), Ġsemantic Ġerrors , Ġsuch Ġincorrect Ġatt ch ment , Ġare Ġmore Ġdifficult Ġto Ġfix . ĠPerhaps , Ġusing Ġdependency Ġparsing Ġmay Ġhelp Ġovercome Ġsome Ġsemantic Ġerrors . ĠAs Ġfor Ġfeature Ġengineering , Ġthe Ġbroad Ġcategories Ġof Ġfeatures , Ġsuch Ġas Ġsurface , Ġsynt actic , Ġand Ġmorph ological Ġfeatures , Ġmay Ġlikely Ġcarry - over Ġto Ġother Ġlanguages , Ġlanguage - specific Ġfeature Ġengineering Ġmay Ġbe Ġrequire Ġto Ġhandle Ġthe Ġspecificity Ġof Ġeach Ġlanguage . ĠLastly , Ġsince Ġmultiple Ġdi ac rit ization Ġconventions Ġmay Ġexist , Ġas Ġin Ġthe Ġcase Ġof ĠArabic , Ġadopting Ġone Ġconvention Ġconsistently Ġis Ġimportant Ġfor Ġtraining Ġa Ġgood Ġsystem Ġand Ġfor Ġproperly Ġtesting Ġit . ĠThough Ġwe Ġhave Ġmostly Ġachieved Ġthis Ġfor ĠM SA , Ġthe ĠCA Ġdataset Ġrequires Ġmore Ġchecks Ġto Ġinsure Ġgreater Ġconsistency . </s> For Ġfuture Ġwork , Ġwe Ġwant Ġto Ġexplore Ġthe Ġeffectiveness Ġof Ġaugment ing Ġour ĠCW Ġmodel Ġwith ĠPOS Ġtagging Ġinformation Ġand Ġa Ġbig ram Ġlanguage Ġmodel . ĠFurther , Ġwe Ġplan Ġto Ġcreate Ġa Ġmulti Ġreference Ġdi ac rit ization Ġtest Ġset Ġto Ġhandle Ġwords Ġthat Ġhave Ġmultiple Ġvalid Ġdi ac rit ized Ġforms . ĠFor ĠCE , Ġwe Ġwant Ġto Ġexamine Ġthe Ġeffectiveness Ġof Ġthe Ġproposed Ġfeatures Ġfor ĠArabic Ġparsing . ĠWe Ġplan Ġto Ġexplore : Ġcharacter - level Ġconv olution al Ġneural Ġnetworks Ġthat Ġmay Ġcapture Ġsub - word Ġmorph ological Ġfeatures ; Ġpre - trained Ġembed d ings ; Ġand Ġattention Ġmechanisms Ġto Ġfocus Ġon Ġsalient Ġfeatures . ĠWe Ġalso Ġplan Ġto Ġexplore Ġjoint Ġmodeling Ġfor Ġboth Ġcore Ġword Ġand Ġcase Ġending Ġdi ac rit ics .