50 human annotators ranked a random sample of 100 translations by Adequacy, Fluency and overall ranking on a 5-point scale.
ABSTRACTIVE
adequacy, precision and ranking values
EXTRACTIVE
To ensure that the increase in BLEU score correlated to actual increase in performance of translation, human evaluation metrics like adequacy, precision and ranking values (between RNNSearch and RNNMorph outputs) were estimated in Table TABREF30 . A group of 50 native people who were well-versed in both English and Tamil languages acted as annotators for the evaluation. A collection of samples of about 100 sentences were taken from the test set results for comparison. This set included a randomized selection of the translation results to ensure the objectivity of evaluation. Fluency and adequacy results for the RNNMorph results are tabulated. Adequacy rating was calculated on a 5-point scale of how much of the meaning is conveyed by the translation (All, Most, Much, Little, None). The fluency rating was calculated based on grammatical correctness on a 5-point scale of (Flawless, Good, Non-native, Disfluent, Incomprehensive). For the comparison process, the RNNMorph and the RNNSearch + Word2Vec models’ sentence level translations were individually ranked between each other, permitting the two translations to have ties in the ranking. The intra-annotator values were computed for these metrics and the scores are shown in Table TABREF32 BIBREF12 , BIBREF13 .
To ensure that the increase in BLEU score correlated to actual increase in performance of translation, human evaluation metrics like adequacy, precision and ranking values (between RNNSearch and RNNMorph outputs) were estimated in Table TABREF30 . A group of 50 native people who were well-versed in both English and Tamil languages acted as annotators for the evaluation. A collection of samples of about 100 sentences were taken from the test set results for comparison. This set included a randomized selection of the translation results to ensure the objectivity of evaluation. Fluency and adequacy results for the RNNMorph results are tabulated. Adequacy rating was calculated on a 5-point scale of how much of the meaning is conveyed by the translation (All, Most, Much, Little, None). The fluency rating was calculated based on grammatical correctness on a 5-point scale of (Flawless, Good, Non-native, Disfluent, Incomprehensive). For the comparison process, the RNNMorph and the RNNSearch + Word2Vec models’ sentence level translations were individually ranked between each other, permitting the two translations to have ties in the ranking. The intra-annotator values were computed for these metrics and the scores are shown in Table TABREF32 BIBREF12 , BIBREF13 .
1612.02482
Introduction </s> The Ġuse Ġof ĠR NN s Ġin Ġthe Ġfield Ġof ĠStatistical ĠMachine ĠTranslation Ġ( SM T ) Ġhas Ġrevolution ised Ġthe Ġapproaches Ġto Ġautomated Ġtranslation . ĠAs Ġopposed Ġto Ġtraditional Ġshallow ĠSM T Ġmodels , Ġwhich Ġrequire Ġa Ġlot Ġof Ġmemory Ġto Ġrun , Ġthese Ġneural Ġtranslation Ġmodels Ġrequire Ġonly Ġa Ġsmall Ġfraction Ġof Ġmemory Ġused , Ġabout Ġ5 % ĠB IB REF 0 Ġ. ĠAlso , Ġneural Ġtranslation Ġmodels Ġare Ġoptimized Ġsuch Ġthat Ġevery Ġmodule Ġis Ġtrained Ġto Ġjointly Ġimprove Ġtranslation Ġquality . ĠWith Ġthat Ġbeing Ġsaid , Ġone Ġof Ġthe Ġmain Ġdowns ides Ġof Ġneural Ġtranslation Ġmodels Ġis Ġthe Ġheavy Ġcorpus Ġrequirement Ġin Ġorder Ġto Ġensure Ġlearning Ġof Ġdeeper Ġcontexts . ĠThis Ġis Ġwhere Ġthe Ġapplication Ġof Ġthese Ġenc oder Ġdec oder Ġarchitectures Ġin Ġtranslation Ġto Ġand / or Ġfrom Ġmorph ologically Ġrich Ġlanguages Ġtakes Ġa Ġsevere Ġhit . </s> For Ġany Ġlanguage Ġpair , Ġthe Ġefficiency Ġof Ġan ĠMT Ġsystem Ġdepends Ġon Ġtwo Ġmajor Ġfactors : Ġthe Ġavailability Ġand Ġsize Ġof Ġparallel Ġcorpus Ġused Ġfor Ġtraining Ġand Ġthe Ġsynt actic Ġdivergence Ġbetween Ġthe Ġtwo Ġlanguages Ġi . e Ġmorph ological Ġrichness , Ġword Ġorder Ġdifferences , Ġgram matical Ġstructure Ġetc . ĠB IB REF 0 Ġ. ĠThe Ġmain Ġdifferences Ġbetween Ġthe Ġlanguages Ġstem Ġfrom Ġthe Ġfact Ġthat Ġlanguages Ġsimilar Ġto ĠEnglish Ġare Ġpredominantly Ġf usional Ġlanguages Ġwhereas Ġmany Ġof Ġthe Ġmorph ologically Ġrich Ġlanguages Ġare Ġagg l ut in ative Ġin Ġnature . ĠThe Ġnature Ġof Ġmorph ologically Ġrich Ġlanguages Ġbeing Ġstruct urally Ġand Ġsem antically Ġdiscord ant Ġfrom Ġlanguages Ġlike ĠEnglish Ġadds Ġto Ġthe Ġdifficulty Ġof ĠSM T Ġinvolving Ġsuch Ġlanguages . </s> In Ġmorph ologically Ġrich Ġlanguages , Ġany Ġsuffix Ġcan Ġbe Ġadded Ġto Ġany Ġverb Ġor Ġnoun Ġto Ġsimply Ġmean Ġone Ġspecific Ġthing Ġabout Ġthat Ġparticular Ġword Ġthat Ġthe Ġsuffix Ġcommonly Ġrepresents Ġ( ag gl ut ination ). ĠThis Ġmeans Ġthat Ġthere Ġexists Ġa Ġlot Ġof Ġinf lection al Ġforms Ġof Ġthe Ġsame Ġnoun Ġand Ġverb Ġbase Ġwords , Ġconve ying Ġsimilar Ġnotions . ĠFor Ġexample , Ġin ĠTamil , Ġthere Ġare Ġat Ġleast Ġ30 , 000 Ġinf lection al Ġforms Ġof Ġany Ġgiven Ġverb Ġand Ġabout Ġ5 , 000 Ġforms Ġof Ġinf lection al Ġforms Ġfor Ġany Ġnoun . ĠThe Ġmerged Ġwords Ġcarry Ġinformation Ġabout Ġpart Ġof Ġspeech Ġ( POS ) Ġtags , Ġtense , Ġplurality Ġand Ġso Ġforth Ġthat Ġare Ġimportant Ġfor Ġanalyzing Ġtext Ġfor ĠMachine ĠTranslation Ġ( MT ). ĠNot Ġonly Ġare Ġthese Ġhidden Ġmeanings Ġnot Ġcaptured , Ġthe Ġcorresponding Ġroot Ġwords Ġare Ġtrained Ġas Ġdifferent Ġunits , Ġthereby Ġincreasing Ġthe Ġcomplexity Ġof Ġdeveloping Ġsuch ĠMT Ġsystems ĠB IB REF 1 Ġ. </s> To Ġadd Ġto Ġthe Ġcomplexities Ġof Ġbeing Ġa Ġmorph ologically Ġrich Ġlanguage , Ġthere Ġare Ġseveral Ġfactors Ġunique Ġto ĠTamil Ġthat Ġmake Ġtranslation Ġvery Ġdifficult . ĠThe Ġavailability Ġof Ġparallel Ġcorpus Ġfor ĠTamil Ġis Ġvery Ġscarce . ĠMost Ġof Ġthe Ġother Ġmodels Ġin Ġthe Ġfield Ġof ĠEnglish âĢĵ Tam il ĠMT Ġhave Ġmade Ġuse Ġof Ġtheir Ġown Ġtranslation Ġcorpor a Ġthat Ġwere Ġmanually Ġcreated Ġfor Ġthe Ġpurposes Ġof Ġresearch . ĠMost Ġof Ġthese Ġcorpor a Ġare Ġnot Ġavailable Ġonline Ġfor Ġuse . </s> Another Ġissue Ġspecific Ġto ĠTamil Ġis Ġthe Ġaddition Ġof Ġsuffix Ġcharacters Ġincluded Ġto Ġthe Ġwords Ġin Ġthe Ġlanguage Ġfor Ġsmooth ness Ġin Ġpronunciation . ĠThese Ġcharacters Ġare Ġof Ġso Ġmany Ġdifferent Ġtypes ; Ġthere Ġis Ġa Ġunique Ġsuffix Ġfor Ġeach Ġand Ġevery Ġconson ant Ġin Ġthe Ġlanguage . ĠThese Ġsuffix es Ġdegrade Ġperformance Ġof ĠMT Ġbecause Ġthe Ġsame Ġwords Ġwith Ġdifferent Ġsuch Ġpronoun ciation - based Ġsuffix es Ġwill Ġbe Ġtaken Ġas Ġdifferent Ġwords Ġin Ġtraining . </s> Also Ġto Ġtake Ġinto Ġconsideration Ġis Ġthe Ġexistence Ġof Ġtwo Ġdifferent Ġforms Ġof Ġthe Ġlanguage Ġbeing Ġused . ĠTrad itionally Ġdefined ĠTamil Ġand Ġits Ġpron unci ations Ġaren 't Ġac oust ically Ġpleasing Ġto Ġuse . ĠThere 's Ġno Ġlinguistic Ġflow Ġbetween Ġsyll ables Ġand Ġits Ġusage Ġin Ġverbal Ġcommunication Ġis Ġtime Ġconsuming . ĠTherefore , Ġthere Ġexists Ġtwo Ġforms Ġof Ġthe Ġlanguage , Ġthe Ġwritten Ġform , Ġrigid Ġin Ġstructure Ġand Ġsyntax , Ġand Ġthe Ġspoken Ġform , Ġin Ġwhich Ġthe Ġflow Ġand Ġpace Ġof Ġthe Ġlanguage Ġis Ġgiven Ġpriority Ġover Ġsyntax Ġand Ġcorrectness Ġof Ġspelling . ĠThis Ġdivide Ġleads Ġto Ġthe Ġcorpus Ġhaving Ġ2 Ġdifferent Ġversions Ġof Ġthe Ġlanguage Ġthat Ġincrease Ġthe Ġvocabulary Ġeven Ġwith Ġthe Ġsame Ġwords . ĠThis Ġcan Ġbe Ġevidently Ġseen Ġin Ġthe Ġcorpus Ġbetween Ġthe Ġsentences Ġused Ġin Ġthe ĠBible , Ġwhich Ġis Ġin Ġtraditional ĠTamil Ġand Ġsentences Ġfrom Ġmovie Ġsubtitles , Ġbeing Ġin Ġspoken ĠTamil Ġformat . </s> To Ġaccount Ġfor Ġsuch Ġdifficulties , Ġa Ġtrade - off Ġbetween Ġdomain Ġspecificity Ġand Ġsize Ġof Ġthe Ġcorpus Ġis Ġintegral Ġin Ġbuilding Ġan ĠEnglish âĢĵ Tam il Ġneural ĠMT Ġsystem . </s> Corp us </s> The Ġcorpus Ġselected Ġfor Ġthis Ġexperiment Ġwas Ġa Ġcombination Ġof Ġdifferent Ġcorpor a Ġfrom Ġvarious Ġdomains . ĠThe Ġmajor Ġpart Ġof Ġthe Ġcorpus Ġwas Ġmade Ġup Ġby Ġthe ĠEn Tam Ġv 2 Ġcorpus ĠB IB REF 2 Ġ. ĠThis Ġcorpus Ġcontained Ġsentences Ġtaken Ġfrom Ġparallel Ġnews Ġarticles , ĠEnglish Ġand ĠTamil Ġbible Ġcorpus Ġand Ġmovie Ġsubtitles . ĠIt Ġalso Ġcomprised Ġof Ġa Ġtourism Ġcorpus Ġthat Ġwas Ġobtained Ġfrom ĠTD IL Ġ( Technology ĠDevelopment Ġfor ĠIndian ĠLanguages ) Ġand Ġa Ġcorpus Ġcreated Ġfrom ĠTamil Ġnovels Ġand Ġshort Ġstories Ġfrom ĠAU - K BC , ĠAnna Ġuniversity . ĠThe Ġcomplete Ġcorpus Ġconsisted Ġof Ġ197 , 792 Ġsentences . ĠFig . ĠFIG REF 20 Ġshows Ġthe Ġskinny Ġshift Ġand Ġheat map Ġrepresentations Ġof Ġthe Ġrelativity Ġbetween Ġthe Ġsentences Ġin Ġterms Ġof Ġtheir Ġsentence Ġlengths . </s> An Ġextra Ġmon oling ual ĠTamil Ġcorpus , Ġcoll ated Ġfrom Ġvarious Ġonline Ġsources Ġwas Ġused Ġfor Ġthe Ġword 2 vec Ġembed ding Ġof Ġthe ĠTamil Ġtarget Ġlanguage Ġto Ġenhance Ġthe Ġrichness Ġof Ġcontext Ġof Ġthe Ġword Ġvectors . ĠIt Ġwas Ġalso Ġused Ġto Ġcreate Ġthe Ġlanguage Ġmodel Ġfor Ġthe Ġphrase - based ĠSM T Ġmodel . ĠThis Ġcorpus Ġcontained Ġ5 67 , 772 Ġsentences Ġand Ġwas Ġself - col lected Ġby Ġcombining Ġhundreds Ġof Ġancient ĠTamil Ġscriptures , Ġnovels Ġand Ġpoems Ġby Ġaccessing Ġthe Ġwebsites Ġof Ġpopular Ġonline Ġebook Ġlibraries Ġin ĠPython Ġusing Ġthe Ġur ll ib Ġpackage . ĠSince Ġthe Ġsources Ġhad ĠTamil Ġtext Ġin Ġdifferent Ġenc od ings , Ġthe Ġencoding Ġscheme Ġwas Ġstandardized Ġto Ġbe ĠUTF - 8 Ġfor Ġthe Ġentirety Ġof Ġthe Ġmon oling ual Ġand Ġparallel Ġcorpor a Ġusing Ġthe Ġch ard et Ġpackage . ĠThe Ġcorpor a Ġwere Ġcleaned Ġfor Ġany Ġstray Ġspecial Ġcharacters , Ġunnecessary Ġhtml Ġtags Ġand Ġwebsite ĠURLs . </s> Word 2 V ec </s> The Ġword Ġembed d ings Ġof Ġthe Ġsource Ġand Ġtarget Ġlanguage Ġsentences Ġare Ġused Ġas Ġinitial Ġvectors Ġof Ġthe Ġmodel Ġto Ġimprove Ġcontextual ization . ĠThe Ġskip Ġgram Ġmodel Ġof Ġthe Ġword 2 vec Ġalgorithm Ġoptim izes Ġthe Ġvectors Ġby Ġaccounting Ġfor Ġthe Ġaverage Ġlog Ġprobability Ġof Ġcontext Ġwords Ġgiven Ġa Ġsource Ġword . ĠDIS PLAY FORM 0 </s> where Ġk Ġis Ġthe Ġcontext Ġwindow Ġtaken Ġfor Ġthe Ġvector ization , ĠIN LINE FORM 0 Ġrefers Ġto Ġthe ĠIN LINE FORM 1 Ġword Ġof Ġthe Ġcorpus Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġsize Ġof Ġthe Ġtraining Ġcorpus Ġin Ġterms Ġof Ġthe Ġnumber Ġof Ġwords . ĠHere , Ġthe Ġprob ab ily ĠIN LINE FORM 3 Ġis Ġcomputed Ġas Ġa Ġhierarchical Ġsoft max Ġof Ġthe Ġproduct Ġof Ġthe Ġtrans pose Ġof Ġthe Ġoutput Ġvector Ġof ĠIN LINE FORM 4 Ġand Ġthe Ġinput Ġvector Ġof ĠIN LINE FORM 5 Ġfor Ġeach Ġand Ġevery Ġpair Ġover Ġthe Ġentire Ġvocabulary . ĠThe Ġprocesses Ġof Ġnegative Ġsampling Ġand Ġsubs am pling Ġof Ġfrequent Ġwords Ġthat Ġwere Ġused Ġin Ġthe Ġoriginal Ġmodel Ġaren 't Ġused Ġin Ġthis Ġexperiment ĠB IB REF 3 Ġ. </s> For Ġthe Ġprocess Ġof Ġcreating Ġsem antically Ġmeaningful Ġword Ġembed d ings , Ġa Ġmon oling ual Ġcorpus Ġof Ġ5 69 , 772 ĠTamil Ġsentences Ġwas Ġused . ĠThis Ġgave Ġthe Ġvectors Ġmore Ġcontextual Ġrichness Ġdue Ġto Ġthe Ġincreased Ġsize Ġof Ġthe Ġcorpus Ġas Ġopposed Ġto Ġusing Ġjust Ġthe Ġbilingual Ġcorpus ' Ġtarget Ġside Ġsentences ĠB IB REF 3 Ġ. </s> In Ġthe Ġexperiment , Ġthe Ġword 2 vec Ġmodel Ġwas Ġtrained Ġusing Ġa Ġvector Ġsize Ġof Ġ100 Ġto Ġensure Ġthat Ġthe Ġbulk Ġof Ġthe Ġlimited Ġmemory Ġof Ġthe ĠGPU Ġwill Ġbe Ġused Ġfor Ġthe Ġneural Ġattention Ġtranslation Ġmodel . ĠIt Ġhas Ġbeen Ġshown Ġthat Ġany Ġsize Ġover Ġthat Ġof Ġ150 Ġused Ġfor Ġword Ġvector ization Ġgives Ġsimilar Ġresults Ġand Ġthat Ġa Ġsize Ġof Ġ100 Ġperforms Ġclose Ġto Ġthe Ġmodel Ġwith Ġ150 - sized Ġword Ġvectors ĠB IB REF 7 Ġ. ĠA Ġstandard Ġsize Ġof Ġ5 Ġwas Ġused Ġas Ġwindow Ġsize Ġand Ġthe Ġmodel Ġwas Ġtrained Ġover Ġ7 Ġworker Ġthreads Ġsimultaneously . ĠA Ġbatch Ġsize Ġof Ġ50 Ġwords Ġwas Ġused Ġfor Ġtraining . ĠThe Ġnegative Ġsampling Ġwas Ġset Ġat Ġ1 Ġas Ġit Ġis Ġthe Ġnature Ġof Ġmorph ologically Ġrich Ġlanguages Ġto Ġhave Ġa Ġlot Ġof Ġimportant Ġwords Ġthat Ġdon 't Ġoccur Ġmore Ġthan Ġonce Ġin Ġthe Ġcorpus . ĠThe Ġg ens im Ġword 2 vec Ġtool kit Ġwas Ġused Ġto Ġimplement Ġthis Ġword Ġembed ding Ġprocess ĠB IB REF 8 Ġ. </s> Ne ural ĠTranslation ĠModel </s> The Ġmodel Ġused Ġfor Ġtranslation Ġis Ġthe Ġone Ġimplemented Ġby ĠBah dan au Ġet Ġal . ĠBah dan au 2014 . ĠA Ġbid irection al ĠL ST M Ġenc oder Ġfirst Ġtakes Ġthe Ġsource Ġsentence Ġand Ġenc odes Ġit Ġinto Ġa Ġcontext Ġvector Ġwhich Ġacts Ġas Ġinput Ġfor Ġthe Ġdec oder . ĠThe Ġdec oder Ġis Ġattention - based Ġwhere Ġthe Ġhidden Ġstates Ġof Ġthe Ġdec oder Ġget Ġas Ġinput Ġthe Ġweighted Ġsum Ġof Ġall Ġthe Ġhidden Ġlayer Ġoutputs Ġof Ġthe Ġenc oder Ġalong with Ġthe Ġoutput Ġof Ġthe Ġprevious Ġhidden Ġlayer Ġand Ġthe Ġpreviously Ġdec oded Ġword . ĠThis Ġprovides Ġa Ġcontextual Ġreference Ġinto Ġthe Ġsource Ġlanguage Ġsentence ĠB IB REF 4 Ġ. </s> Ne ural ĠMachine ĠTranslation Ġmodels Ġdirectly Ġcompute Ġthe Ġprobability Ġof Ġthe Ġtarget Ġlanguage Ġsentence Ġgiven Ġthe Ġsource Ġlanguage Ġsentence , Ġword Ġby Ġword Ġfor Ġevery Ġtime Ġstep . ĠThe Ġmodel Ġwith Ġa Ġbasic Ġdec oder Ġwithout Ġthe Ġattention Ġmodule Ġcomp utes Ġthe Ġlog Ġprobability Ġof Ġtarget Ġsentence Ġgiven Ġsource Ġsentence Ġas Ġthe Ġsum Ġof Ġlog Ġprobabilities Ġof Ġevery Ġword Ġgiven Ġevery Ġword Ġbefore Ġthat . ĠThe Ġattention - based Ġmodel , Ġon Ġthe Ġother Ġhand , Ġcalculates : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġwords Ġin Ġthe Ġtarget Ġsentence , ĠIN LINE FORM 1 Ġis Ġthe Ġtarget Ġsentence , ĠIN LINE FORM 2 Ġis Ġthe Ġsource Ġsentence , ĠIN LINE FORM 3 Ġis Ġthe Ġfixed Ġlength Ġoutput Ġvector Ġof Ġthe Ġenc oder Ġand ĠIN LINE FORM 4 Ġis Ġthe Ġweighted Ġsum Ġof Ġall Ġthe Ġhidden Ġlayer Ġoutputs Ġof Ġthe Ġenc oder Ġat Ġevery Ġtime Ġstep . ĠBoth Ġthe Ġenc oder 's Ġoutput Ġcontext Ġvector Ġand Ġthe Ġweighted Ġsum Ġ( known Ġas Ġattention Ġvector ) Ġhelp Ġto Ġimprove Ġthe Ġquality Ġof Ġtranslation Ġby Ġenabling Ġselective Ġsource Ġsentence Ġlookup . </s> The Ġdec oder ĠL ST M Ġcomp utes : ĠDIS PLAY FORM 0 </s> where Ġthe Ġprobability Ġis Ġcomputed Ġas Ġa Ġfunction Ġof Ġthe Ġdec oder 's Ġoutput Ġin Ġthe Ġprevious Ġtime Ġstep ĠIN LINE FORM 0 Ġ, Ġthe Ġhidden Ġlayer Ġvector Ġof Ġthe Ġdec oder Ġin Ġthe Ġcurrent Ġtim est ep ĠIN LINE FORM 1 Ġand Ġthe Ġcontext Ġvector Ġfrom Ġthe Ġattention Ġmechanism ĠIN LINE FORM 2 Ġ. ĠThe Ġcontext Ġvector ĠIN LINE FORM 3 Ġfor Ġtime Ġstep ĠIN LINE FORM 4 Ġis Ġcomputed Ġas Ġa Ġweighted Ġsum Ġof Ġthe Ġoutput Ġof Ġthe Ġentire Ġsentence Ġusing Ġa Ġweight Ġparameter ĠIN LINE FORM 5 Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġtokens Ġin Ġthe Ġsource Ġsentence , ĠIN LINE FORM 1 Ġrefers Ġto Ġthe Ġvalue Ġof Ġthe Ġhidden Ġlayer Ġof Ġthe Ġenc oder Ġat Ġtime Ġstep ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġalignment Ġparameter . ĠThis Ġparameter Ġis Ġcalculated Ġby Ġmeans Ġof Ġa Ġfeed Ġforward Ġneural Ġnetwork Ġto Ġensure Ġthat Ġthe Ġalignment Ġmodel Ġis Ġfree Ġfrom Ġthe Ġdifficulties Ġof Ġcontextual ization Ġof Ġlong Ġsentences Ġinto Ġa Ġsingle Ġvector . ĠThe Ġfeed Ġforward Ġnetwork Ġis Ġtrained Ġalong Ġwith Ġthe Ġneural Ġtranslation Ġmodel Ġto Ġjointly Ġimprove Ġthe Ġperformance Ġof Ġthe Ġtranslation . ĠMat hematically , ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġsoft max Ġoutput Ġof Ġthe Ġresult Ġof Ġthe Ġfeed forward Ġnetwork , ĠIN LINE FORM 1 Ġis Ġthe Ġhidden Ġstate Ġvalue Ġof Ġthe Ġdec oder Ġat Ġtim est ep ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġenc oder 's Ġhidden Ġlayer Ġannotation Ġat Ġtim est ep ĠIN LINE FORM 4 Ġ. ĠA Ġconc aten ation Ġof Ġthe Ġforward Ġand Ġthe Ġreverse Ġhidden Ġlayer Ġparameters Ġof Ġthe Ġenc oder Ġis Ġused Ġat Ġeach Ġstep Ġto Ġcompute Ġthe Ġweights ĠIN LINE FORM 5 Ġfor Ġthe Ġattention Ġmechanism . ĠThis Ġis Ġdone Ġto Ġenable Ġan Ġoverall Ġcontext Ġof Ġthe Ġsentence , Ġas Ġopposed Ġto Ġa Ġcontext Ġof Ġonly Ġall Ġthe Ġprevious Ġwords Ġof Ġthe Ġsentence Ġfor Ġevery Ġword Ġin Ġconsideration . ĠFig . ĠFIG REF 12 Ġis Ġthe Ġgeneral Ġarchitecture Ġof Ġthe Ġneural Ġtranslation Ġmodel Ġwithout Ġthe ĠBid irection al ĠL ST M Ġenc oder . </s> A Ġglobal Ġattention Ġmechanism Ġis Ġpreferred Ġover Ġlocal Ġattention Ġbecause Ġthe Ġdifferences Ġin Ġthe Ġstructures Ġof Ġthe Ġlanguages Ġcannot Ġbe Ġmapped Ġefficiently Ġto Ġenable Ġlookup Ġinto Ġthe Ġright Ġparts Ġof Ġthe Ġsource Ġsentence . ĠUsing Ġlocal Ġattention Ġmechanism Ġwith Ġa Ġmon ot onic Ġcontext Ġlookup , Ġwhere Ġthe Ġregion Ġaround ĠIN LINE FORM 0 Ġsource Ġword Ġis Ġlooked Ġup Ġfor Ġthe Ġprediction Ġof Ġthe ĠIN LINE FORM 1 Ġtarget Ġword , Ġis Ġimpractical Ġbecause Ġof Ġthe Ġstructural Ġdiscord ance Ġbetween Ġthe ĠEnglish Ġand ĠTamil Ġsentences Ġ( see ĠF igs . ĠFIG REF 37 Ġand ĠFIG REF 44 Ġ). ĠThe Ġuse Ġof Ġga ussian Ġand Ġother Ġsuch Ġdistributions Ġto Ġfacilitate Ġlocal Ġattention Ġwould Ġalso Ġbe Ġinefficient Ġbecause Ġthe Ġexistence Ġof Ġvarious Ġforms Ġof Ġtranslations Ġfor Ġthe Ġsame Ġsource Ġsentence Ġinvolving Ġmorph ological Ġand Ġstructural Ġvariations Ġthat Ġdon 't Ġstay Ġuniform Ġthrough Ġthe Ġentire Ġcorpus ĠB IB REF 5 Ġ. </s> The ĠNo ĠPe ep holes Ġ( NP ) Ġvariant Ġof Ġthe ĠL ST M Ġcell , Ġformulated Ġin ĠGre ff Ġet Ġal . Ġgre ff 2015 l st m Ġis Ġused Ġin Ġthis Ġexperiment Ġas Ġit Ġproved Ġto Ġgive Ġthe Ġbest Ġresults Ġamongst Ġall Ġthe Ġvariants Ġof Ġan ĠL ST M Ġcell . ĠIt Ġis Ġspecified Ġby Ġmeans Ġof Ġa Ġg ated Ġmechanism Ġdesigned Ġto Ġensure Ġthat Ġthe Ġvanishing Ġgradient Ġproblem Ġis Ġprevented . ĠL ST M Ġmaintains Ġits Ġhidden Ġlayer Ġin Ġtwo Ġcomponents , Ġthe Ġcell Ġvector ĠIN LINE FORM 0 Ġand Ġthe Ġactual Ġhidden Ġlayer Ġoutput Ġvector ĠIN LINE FORM 1 Ġ. ĠThe Ġcell Ġvector Ġis Ġensured Ġto Ġnever Ġreach Ġzero Ġby Ġmeans Ġof Ġa Ġweighted Ġsum Ġof Ġthe Ġprevious Ġlayer 's Ġcell Ġvector ĠIN LINE FORM 2 Ġregulated Ġby Ġthe Ġforget Ġgate ĠIN LINE FORM 3 Ġand Ġan Ġactivation Ġof Ġthe Ġweighted Ġsum Ġof Ġthe Ġinput ĠIN LINE FORM 4 Ġin Ġthe Ġcurrent Ġtim est ep ĠIN LINE FORM 5 Ġand Ġthe Ġprevious Ġtim est ep 's Ġhidden Ġlayer Ġoutput Ġvector ĠIN LINE FORM 6 Ġ. ĠThe Ġcombination Ġis Ġsimilarly Ġregulated Ġby Ġthe Ġinput Ġgate ĠIN LINE FORM 7 Ġ. ĠThe Ġhidden Ġlayer Ġoutput Ġis Ġdetermined Ġas Ġan Ġactivation Ġof Ġthe Ġcell Ġgate , Ġregulated Ġby Ġthe Ġoutput Ġgate ĠIN LINE FORM 8 Ġ. ĠThe Ġinter play Ġbetween Ġthese Ġtwo Ġvectors Ġ( ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġ) Ġat Ġevery Ġtim est ep Ġensures Ġthat Ġthe Ġproblem Ġof Ġvanishing Ġgrad ients Ġdoesn 't Ġoccur . ĠThe Ġthree Ġgates Ġare Ġalso Ġformed Ġas Ġa Ġs igm oid Ġof Ġthe Ġweighted Ġsum Ġof Ġthe Ġprevious Ġhidden Ġlayer Ġoutput ĠIN LINE FORM 11 Ġand Ġthe Ġinput Ġin Ġthe Ġcurrent Ġtim est ep ĠIN LINE FORM 12 Ġ. ĠThe Ġoutput Ġgenerated Ġout Ġof Ġthe ĠL ST M 's Ġhidden Ġlayer Ġis Ġspecified Ġas Ġa Ġweighted Ġsoft max Ġover Ġthe Ġhidden Ġlayer Ġoutput ĠIN LINE FORM 13 Ġ. ĠThe Ġlearn able Ġparameters Ġof Ġan ĠL ST M Ġcell Ġare Ġall Ġthe Ġweights ĠIN LINE FORM 14 Ġand Ġthe Ġbiases ĠIN LINE FORM 15 Ġ. ĠDIS PLAY FORM 0 </s> The ĠL ST M Ġspecified Ġby Ġequations Ġ7 Ġthrough Ġ11 Ġis Ġthe Ġone Ġused Ġfor Ġthe Ġdec oder Ġof Ġthe Ġmodel . ĠThe Ġenc oder Ġuses Ġa Ġbid irection al ĠR NN ĠL ST M Ġcell Ġin Ġwhich Ġthere Ġare Ġtwo Ġhidden Ġlayer Ġcomponents ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġthat Ġcontribute Ġto Ġthe Ġoutput ĠIN LINE FORM 2 Ġof Ġeach Ġtime Ġstep ĠIN LINE FORM 3 Ġ. ĠBoth Ġthe Ġcomponents Ġhave Ġtheir Ġown Ġsets Ġof ĠL ST M Ġequations Ġin Ġsuch Ġa Ġway Ġthat ĠIN LINE FORM 4 Ġfor Ġevery Ġtim est ep Ġis Ġcomputed Ġfrom Ġthe Ġfirst Ġtim est ep Ġtill Ġthe ĠIN LINE FORM 5 Ġtoken Ġis Ġreached Ġand ĠIN LINE FORM 6 Ġis Ġcomputed Ġfrom Ġthe ĠIN LINE FORM 7 Ġtim est ep Ġbackwards Ġuntil Ġthe Ġfirst Ġtoken Ġis Ġreached . ĠAll Ġthe Ġfive Ġvectors Ġof Ġthe Ġtwo Ġcomponents Ġare Ġall Ġexactly Ġthe Ġsame Ġas Ġthe ĠL ST M Ġequations Ġspecified Ġwith Ġone Ġvariation Ġin Ġthe Ġcomputation Ġof Ġthe Ġresult . ĠDIS PLAY FORM 0 </s> M orph ological ĠSe gment ation </s> The Ġmorph ological Ġsegment ation Ġused Ġis Ġa Ġsemi - super vised Ġextension Ġto Ġthe Ġgener ative Ġprob abil istic Ġmodel Ġof Ġmaximizing Ġthe Ġprobability Ġof Ġa ĠIN LINE FORM 0 Ġprefix , root , post fix ĠIN LINE FORM 1 Ġrecursive Ġsplit Ġup Ġof Ġwords Ġbased Ġon Ġan Ġexhaustive Ġcombination Ġof Ġall Ġpossible Ġmorp hem es . ĠThe Ġdetails Ġof Ġthis Ġmodel Ġare Ġspecified Ġand Ġextensively Ġstudied Ġin ĠKoh onen Ġet Ġal . Ġk oh onen 2010 se mi . ĠThe Ġmodel Ġparameters ĠIN LINE FORM 2 Ġinclude Ġthe Ġmorph Ġtype Ġcount , Ġmorph Ġtoken Ġcount Ġof Ġtraining Ġdata , Ġthe Ġmorph Ġstrings Ġand Ġtheir Ġcounts . ĠThe Ġmodel Ġis Ġtrained Ġby Ġmaximizing Ġthe ĠMaximum ĠA ĠPoster ior i Ġ( MAP ) Ġprobability Ġusing ĠBay es ' Ġrule : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrefers Ġto Ġevery Ġword Ġin Ġthe Ġtraining Ġlex icon . ĠThe Ġprior ĠIN LINE FORM 1 Ġis Ġestimated Ġusing Ġthe ĠMinimum ĠDescription ĠLength ( MD L ) Ġprinciple . ĠThe Ġlikelihood ĠIN LINE FORM 2 Ġis Ġestimated Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrefers Ġto Ġthe Ġintermediate Ġanalyses Ġand ĠIN LINE FORM 1 Ġrefers Ġto Ġthe ĠIN LINE FORM 2 Ġmor phe me Ġof Ġword ĠIN LINE FORM 3 Ġ. </s> An Ġextension Ġto Ġthe ĠV iter bi Ġalgorithm Ġis Ġused Ġfor Ġthe Ġdecoding Ġstep Ġbased Ġon Ġexhaustive Ġmapping Ġof Ġmorp hem es . ĠTo Ġaccount Ġfor Ġover - se gment ation Ġand Ġunder - se gment ation Ġissues Ġassociated Ġwith Ġun super vised Ġmorph ological Ġsegment ation , Ġextra Ġparameters Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġ( ĠIN LINE FORM 1 Ġ) Ġare Ġused Ġwith Ġthe Ġcost Ġfunction ĠIN LINE FORM 2 ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġlikelihood Ġof Ġthe Ġcost Ġfunction , ĠIN LINE FORM 1 Ġdescribes Ġthe Ġlikelihood Ġof Ġcontribution Ġof Ġthe Ġannot ated Ġdataset Ġto Ġthe Ġcost Ġfunction Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġlikelihood Ġof Ġthe Ġlabeled Ġdata . ĠA Ġdecrease Ġin Ġthe Ġvalue Ġof ĠIN LINE FORM 3 Ġwill Ġcause Ġsmaller Ġsegments Ġand Ġvice Ġversa . ĠIN LINE FORM 4 Ġtakes Ġcare Ġof Ġsize Ġdiscrepancies Ġdue Ġto Ġreduced Ġavailability Ġof Ġannot ated Ġcorpus Ġas Ġcompared Ġto Ġthe Ġtraining Ġcorpus ĠB IB REF 2 Ġ, ĠB IB REF 6 Ġ. </s> The ĠPython Ġextension Ġto Ġthe Ġmorph ological Ġsegment ation Ġtool Ġmor f essor Ġ2 . 0 Ġwas Ġused Ġfor Ġthis Ġexperiment Ġto Ġperform Ġthe Ġsegment ation . ĠThe Ġannotation Ġdata Ġfor ĠTamil Ġlanguage Ġcoll ated Ġand Ġreleased Ġby ĠAn oop ĠK unch uk k utan Ġin Ġthe ĠInd ic ĠN LP ĠLibrary Ġwas Ġused Ġas Ġthe Ġsemi - super vised Ġinput Ġto Ġthe Ġmodel ĠB IB REF 9 Ġ, ĠB IB REF 6 Ġ. </s> Exper iment </s> The Ġcomplexities Ġof Ġneural Ġmachine Ġtranslation Ġof Ġmorph ologically Ġrich Ġlanguages Ġwere Ġstudied Ġwith Ġrespect Ġto ĠEnglish Ġto ĠTamil Ġmachine Ġtranslation Ġusing Ġthe ĠR NN ĠL ST M ĠBi - direction al Ġenc oder Ġattention Ġdec oder Ġarchitecture . ĠTo Ġcompare Ġwith Ġa Ġbaseline Ġsystem , Ġa Ġphrase Ġbased ĠSM T Ġsystem Ġwas Ġimplemented Ġusing Ġthe Ġsame Ġcorpus . ĠThe ĠFact ored ĠSM T Ġmodel Ġwith Ġsource - side Ġpre processing Ġby ĠKumar Ġet Ġal . Ġk umar 2014 impro ving Ġwas Ġused Ġas Ġa Ġreference Ġfor Ġthe Ġtranslation Ġbetween Ġthese Ġlanguage Ġpairs . ĠAlso , Ġan Ġadditional Ġ5 69 , 772 Ġmon oling ual ĠTamil Ġsentences Ġwere Ġused Ġfor Ġthe Ġlanguage Ġmodel Ġof Ġthe ĠSM T Ġsystem . ĠThe Ġmodel Ġused Ġcould Ġbe Ġsplit Ġup Ġinto Ġvarious Ġmodules Ġas Ġexpanded Ġin ĠFig . ĠFIG REF 17 Ġ. </s> B uck eting </s> The Ġinput Ġsource Ġand Ġtarget Ġlanguage Ġsentences Ġused Ġfor Ġtraining Ġwere Ġtaken Ġand Ġdivided Ġinto Ġbucket ed Ġpairs Ġof Ġsentences Ġof Ġa Ġfixed Ġnumber Ġof Ġsizes . ĠThis Ġrelationship Ġwas Ġdetermined Ġby Ġexamining Ġthe Ġdistribution Ġof Ġwords Ġin Ġthe Ġcorpus Ġprimarily Ġto Ġminimize Ġthe Ġnumber Ġof ĠP AD Ġtokens Ġin Ġthe Ġsentence . ĠThe Ġheat Ġmap Ġof Ġthe Ġnumber Ġof Ġwords Ġin Ġthe ĠEnglish âĢĵ Tam il Ġsentence Ġpairs Ġof Ġthe Ġcorpus Ġrevealed Ġthat Ġthe Ġdistribution Ġis Ġcentered Ġaround Ġthe Ġ10 âĢĵ 20 Ġwords Ġregion . ĠTherefore , Ġmore Ġbuckets Ġin Ġthat Ġregion Ġwere Ġapplied Ġas Ġthere Ġwould Ġbe Ġenough Ġnumber Ġof Ġexamples Ġin Ġeach Ġof Ġthese Ġbucket Ġpairs Ġfor Ġthe Ġmodel Ġto Ġlearn Ġabout Ġthe Ġsentences Ġin Ġeach Ġand Ġevery Ġbucket . ĠThe Ġexact Ġscheme Ġused Ġfor Ġthe ĠR NN Search Ġmodels Ġis Ġspecified Ġby ĠFig . ĠFIG REF 21 Ġ. ĠThe Ġbuck eting Ġscheme Ġfor Ġthe ĠR NN M orph Ġmodel , Ġinvolving Ġmorph s Ġinstead Ġof Ġwords , Ġwas Ġa Ġsimple Ġshifted Ġscheme Ġof Ġthe Ġone Ġused Ġin ĠFig . ĠFIG REF 21 Ġ, Ġwhere Ġevery Ġtarget Ġsentence Ġbucket Ġcount Ġwas Ġincreased Ġuniformly Ġby Ġ5 . </s> Model ĠDetails </s> Due Ġto Ġvarious Ġcomputational Ġconstraints Ġand Ġlack Ġof Ġavailability Ġof Ġcomprehensive Ġcorpor a , Ġthe Ġvoc ab ular ies Ġfor ĠEnglish Ġand ĠTamil Ġlanguages Ġfor Ġthe ĠR NN Search Ġmodel Ġwere Ġrestricted Ġto Ġ60 , 000 Ġout Ġof Ġ67 , 768 Ġand Ġ150 , 000 Ġout Ġof Ġ340 , 325 Ġrespectively . ĠThe Ġvocabulary Ġof Ġthe Ġlanguages Ġfor Ġthe ĠR NN M orph Ġdidn 't Ġhave Ġto Ġbe Ġrestricted Ġand Ġthe Ġactual Ġnumber Ġof Ġwords Ġin Ġthe Ġcorpus Ġi . e . Ġ67 , 768 Ġwords Ġfor ĠEnglish Ġand Ġ41 , 9 06 Ġwords Ġfor ĠTamil Ġcould Ġbe Ġaccommod ated Ġinto Ġthe Ġtraining . ĠWords Ġnot Ġin Ġthe Ġvocabulary Ġfrom Ġthe Ġtest Ġset Ġinput Ġand Ġoutput Ġwere Ġreplaced Ġwith Ġthe Ġuniversal ĠIN LINE FORM 0 ĠUN K ĠIN LINE FORM 1 Ġtoken , Ġsymbol izing Ġan Ġunknown Ġword . ĠThe ĠL ST M Ġhidden Ġlayer Ġsize , Ġthe Ġtraining Ġbatch Ġsize , Ġand Ġthe Ġvocabulary Ġsizes Ġof Ġthe Ġlanguages , Ġtogether , Ġacted Ġas Ġa Ġbottleneck . ĠThe Ġmodel Ġwas Ġrun Ġon Ġa Ġ2 GB ĠNVIDIA ĠGeForce ĠGT Ġ650 M Ġcard Ġwith Ġ384 Ġcores Ġand Ġthe Ġmemory Ġallot ment Ġwas Ġconstrained Ġto Ġthe Ġlimits Ġof Ġthe ĠGPU . ĠTherefore , Ġafter Ġrepeated Ġexperimentation , Ġit Ġwas Ġdetermined Ġthat Ġwith Ġa Ġbatch Ġsize Ġof Ġ16 , Ġthe Ġmaximum Ġhidden Ġlayer Ġsize Ġpossible Ġwas Ġ500 , Ġwhich Ġwas Ġthe Ġsize Ġused . ĠAttempt s Ġto Ġreduce Ġthe Ġbatch Ġsize Ġresulted Ġin Ġpoor Ġconvergence , Ġand Ġso Ġthe Ġparameters Ġwere Ġset Ġto Ġcenter Ġaround Ġthe Ġbatch Ġsize Ġof Ġ16 . ĠThe Ġmodels Ġused Ġwere Ġof Ġ4 Ġlayers Ġof ĠL ST M Ġhidden Ġunits Ġin Ġthe Ġbid irection al Ġenc oder Ġand Ġattention Ġdec oder . </s> The Ġmodel Ġused Ġa ĠSt och astic ĠGrad ient ĠDes cent Ġ( SG D ) Ġoptimization Ġalgorithm Ġwith Ġa Ġsampled Ġsoft max Ġloss Ġof Ġ512 Ġper Ġsample Ġto Ġhandle Ġlarge Ġvocabulary Ġsize Ġof Ġthe Ġtarget Ġlanguage ĠB IB REF 10 Ġ. ĠThe Ġmodel Ġwas Ġtrained Ġwith Ġa Ġlearning Ġrate Ġ1 . 0 Ġand Ġa Ġdecay Ġof Ġrate Ġ0 . 5 Ġenforced Ġmanually . ĠGrad ient Ġclipping Ġbased Ġon Ġthe Ġglobal Ġnorm Ġof Ġ5 . 0 Ġwas Ġcarried Ġout Ġto Ġprevent Ġgrad ients Ġexploding Ġand Ġgoing Ġto Ġunre cover able Ġvalues Ġtending Ġtowards Ġinfinity . ĠThe Ġmodel Ġdescribed Ġis Ġthe Ġone Ġused Ġin Ġthe ĠT ensor flow ĠB IB REF 11 Ġseq 2 seq Ġlibrary . </s> Results Ġand ĠDiscussion </s> The ĠB LE U Ġmetric Ġparameters Ġ( modified Ġ1 - gram , Ġ2 - gram , Ġ3 - gram Ġand Ġ4 - gram Ġprecision Ġvalues ) Ġand Ġhuman Ġevaluation Ġmetrics Ġof Ġadequ acy , Ġflu ency Ġand Ġrelative Ġranking Ġvalues Ġwere Ġused Ġto Ġevaluate Ġthe Ġperformance Ġof Ġthe Ġmodels . </s> BLE U ĠEvaluation </s> The ĠB LE U Ġscores Ġobtained Ġusing Ġthe Ġvarious Ġmodels Ġused Ġin Ġthe Ġexperiment Ġare Ġtab ulated Ġin ĠTable ĠT AB REF 25 Ġ. </s> The ĠB LE U Ġmetric Ġcomp utes Ġthe ĠB LE U Ġun ig ram , Ġbig ram , Ġtrig ram Ġand ĠB LE U - 4 Ġmodified Ġprecision Ġvalues , Ġeach Ġmicro - aver aged Ġover Ġthe Ġtest Ġset Ġsentences ĠB IB REF 7 Ġ. ĠIt Ġwas Ġobserved , Ġas Ġexpected , Ġthat Ġthe Ġperformance Ġof Ġthe Ġphrase - based ĠSM T Ġmodel Ġwas Ġinferior Ġto Ġthat Ġof Ġthe ĠR NN Search Ġmodel . ĠThe Ġbaseline ĠR NN Search Ġsystem Ġwas Ġfurther Ġrefined Ġby Ġusing Ġword 2 vec Ġvectors Ġto Ġembed Ġsemantic Ġunderstanding , Ġas Ġobserved Ġwith Ġthe Ġslight Ġincrease Ġin Ġthe ĠB LE U Ġscores . ĠFig . ĠFIG REF 26 Ġplots Ġthe ĠB LE U Ġscores Ġas Ġa Ġline Ġgraph Ġfor Ġvisualization Ġof Ġthe Ġimprovement Ġin Ġperformance . ĠAlso , Ġthe Ġ4 - gram ĠB LE U Ġscores Ġfor Ġthe Ġvarious Ġmodels Ġwere Ġplotted Ġas Ġa Ġbar Ġgraph Ġin ĠFig . ĠFIG REF 26 </s> Due Ġto Ġthe Ġagg l ut in ative Ġand Ġmorph ologically Ġrich Ġnature Ġof Ġthe Ġtarget Ġlanguage Ġi . e . ĠTamil , Ġthe Ġuse Ġof Ġmorph ological Ġsegment ation Ġto Ġsplit Ġthe Ġwords Ġinto Ġmorp hem es Ġfurther Ġimproved Ġthe ĠB LE U Ġprecision Ġvalues Ġin Ġthe ĠR NN M orph Ġmodel . ĠOne Ġof Ġthe Ġreasons Ġfor Ġthe Ġlarge Ġextent Ġof Ġincrease Ġin Ġthe ĠB LE U Ġscore Ġcould Ġbe Ġattributed Ġto Ġthe Ġoverall Ġincrease Ġin Ġthe Ġnumber Ġof Ġword Ġunits Ġper Ġsentence . ĠSince Ġthe ĠB LE U Ġscore Ġcomp utes Ġmicro - average Ġprecision Ġscores , Ġan Ġincrease Ġin Ġboth Ġthe Ġnumer ator Ġand Ġdenomin ator Ġof Ġthe Ġprecision Ġscores Ġis Ġapparent Ġwith Ġan Ġincrease Ġin Ġthe Ġnumber Ġof Ġtokens Ġdue Ġto Ġmorph ological Ġsegment ation Ġof Ġthe Ġtarget Ġlanguage . ĠThus , Ġthe Ġnumeric Ġextent Ġof Ġthe Ġincrease Ġof Ġaccuracy Ġmight Ġnot Ġefficiently Ġdescribe Ġthe Ġimprovement Ġin Ġperformance Ġof Ġthe Ġtranslation . </s> Human ĠEvaluation </s> To Ġensure Ġthat Ġthe Ġincrease Ġin ĠB LE U Ġscore Ġcorrelated Ġto Ġactual Ġincrease Ġin Ġperformance Ġof Ġtranslation , Ġhuman Ġevaluation Ġmetrics Ġlike Ġadequ acy , Ġprecision Ġand Ġranking Ġvalues Ġ( between ĠR NN Search Ġand ĠR NN M orph Ġoutputs ) Ġwere Ġestimated Ġin ĠTable ĠT AB REF 30 Ġ. ĠA Ġgroup Ġof Ġ50 Ġnative Ġpeople Ġwho Ġwere Ġwell - vers ed Ġin Ġboth ĠEnglish Ġand ĠTamil Ġlanguages Ġacted Ġas Ġannot ators Ġfor Ġthe Ġevaluation . ĠA Ġcollection Ġof Ġsamples Ġof Ġabout Ġ100 Ġsentences Ġwere Ġtaken Ġfrom Ġthe Ġtest Ġset Ġresults Ġfor Ġcomparison . ĠThis Ġset Ġincluded Ġa Ġrandomized Ġselection Ġof Ġthe Ġtranslation Ġresults Ġto Ġensure Ġthe Ġobject ivity Ġof Ġevaluation . ĠFlu ency Ġand Ġadequ acy Ġresults Ġfor Ġthe ĠR NN M orph Ġresults Ġare Ġtab ulated . ĠAd equ acy Ġrating Ġwas Ġcalculated Ġon Ġa Ġ5 - point Ġscale Ġof Ġhow Ġmuch Ġof Ġthe Ġmeaning Ġis Ġconveyed Ġby Ġthe Ġtranslation Ġ( All , ĠMost , ĠMuch , ĠLittle , ĠNone ). ĠThe Ġflu ency Ġrating Ġwas Ġcalculated Ġbased Ġon Ġgram matical Ġcorrectness Ġon Ġa Ġ5 - point Ġscale Ġof Ġ( F law less , ĠGood , ĠNon - native , ĠDis f luent , ĠIn com pre hens ive ). ĠFor Ġthe Ġcomparison Ġprocess , Ġthe ĠR NN M orph Ġand Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġmodels âĢ Ļ Ġsentence Ġlevel Ġtranslations Ġwere Ġindividually Ġranked Ġbetween Ġeach Ġother , Ġpermitting Ġthe Ġtwo Ġtranslations Ġto Ġhave Ġties Ġin Ġthe Ġranking . ĠThe Ġintra - annot ator Ġvalues Ġwere Ġcomputed Ġfor Ġthese Ġmetrics Ġand Ġthe Ġscores Ġare Ġshown Ġin ĠTable ĠT AB REF 32 ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> The Ġhuman Ġevaluation ĠKappa Ġco - efficient Ġresults Ġare Ġcalculated Ġwith Ġrespect Ġto : ĠDIS PLAY FORM 0 </s> It Ġwas Ġobserved Ġthat Ġthe Ġranking ĠKappa Ġco - efficient Ġfor Ġintra - annot ator Ġranking Ġof Ġthe ĠR NN M orph Ġmodel Ġwas Ġat Ġ0 . 573 , Ġhigher Ġthat Ġthe Ġ0 . 410 Ġof Ġthe ĠR NN Search + Word 2 V ec Ġmodel , Ġimplying Ġthat Ġthe Ġannot ators Ġfound Ġthe ĠR NN M orph Ġmodel Ġto Ġproduce Ġbetter Ġresults Ġwhen Ġcompared Ġto Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġmodel . </s> Model ĠParameters </s> The Ġlearning Ġrate Ġdecay Ġthrough Ġthe Ġtraining Ġprocess Ġof Ġthe ĠR NN M orph Ġmodel Ġis Ġshowcased Ġin Ġthe Ġgraph Ġin ĠFig . ĠFIG REF 34 Ġ. ĠThis Ġprocess Ġwas Ġdone Ġmanually Ġwhere Ġthe Ġlearning Ġrate Ġwas Ġdec ayed Ġafter Ġthe Ġend Ġof Ġspecific Ġepoch s Ġbased Ġon Ġan Ġobserved Ġstagnation Ġin Ġperplex ity . The ĠR NN M orph Ġmodel Ġachieved Ġsaturation Ġof Ġperplex ities Ġmuch Ġearlier Ġthrough Ġthe Ġepoch s Ġthan Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġmodel . ĠThis Ġconform s Ġto Ġthe Ġexpected Ġoutcome Ġas Ġthe Ġmorph ological Ġsegment ation Ġhas Ġreduced Ġthe Ġvocabulary Ġsize Ġof Ġthe Ġtarget Ġlanguage Ġfrom Ġ340 , 325 Ġwords Ġto Ġa Ġmere Ġ41 , 9 06 Ġmorph s . </s> The Ġerror Ġfunction Ġused Ġwas Ġthe Ġsampled ĠSoft Max Ġloss Ġto Ġensure Ġa Ġlarge Ġtarget Ġvocabulary Ġcould Ġbe Ġaccommod ated ĠB IB REF 10 Ġ. ĠA Ġzoom ed Ġins et Ġgraph Ġ( Fig . ĠFIG REF 35 Ġ) Ġhas Ġbeen Ġused Ġto Ġvisualize Ġthe Ġvalues Ġof Ġthe Ġerror Ġfunction Ġfor Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġand ĠR NN M orph Ġmodels Ġwith Ġ4 Ġhidden Ġlayers . ĠIt Ġcan Ġbe Ġseen Ġthat Ġthe ĠR NN M orph Ġmodel Ġis Ġconsistently Ġbetter Ġin Ġterms Ġof Ġthe Ġperplex ity Ġvalues Ġthrough Ġthe Ġtime Ġsteps . </s> Att ention ĠV ect ors </s> In Ġorder Ġto Ġfurther Ġdemonstrate Ġthe Ġquality Ġof Ġthe ĠR NN M orph Ġmodel , Ġthe Ġattention Ġvectors Ġof Ġboth Ġthe ĠR NN Search Ġwith ĠWord 2 V ec Ġembed ding Ġand ĠR NN M orph Ġmodels Ġare Ġcompared Ġfor Ġseveral Ġgood Ġtranslations Ġin ĠF igs . ĠFIG REF 37 Ġand ĠFIG REF 44 Ġ. ĠIt Ġis Ġobserved Ġthat Ġthe Ġreduction Ġin Ġvocabulary Ġsize Ġhas Ġimproved Ġthe Ġsource Ġsentence Ġlookup Ġby Ġquite Ġan Ġextent . ĠEach Ġcell Ġin Ġthe Ġheat map Ġdisplays Ġthe Ġmagnitude Ġof Ġthe Ġattention Ġlayer Ġweight ĠIN LINE FORM 0 Ġfor Ġthe ĠIN LINE FORM 1 ĠTamil Ġword Ġand Ġthe ĠIN LINE FORM 2 ĠEnglish Ġword Ġin Ġthe Ġrespective Ġsentences . ĠThe Ġintensity Ġof Ġblack Ġcorresponds Ġto Ġthe Ġmagnitude Ġof Ġthe Ġcell ĠIN LINE FORM 3 Ġ. ĠAlso , Ġthe Ġattention Ġvectors Ġof Ġthe ĠR NN Search Ġmodel Ġwith ĠWord 2 V ec Ġembed d ings Ġtend Ġto Ġattend Ġto ĠIN LINE FORM 4 ĠE OS ĠIN LINE FORM 5 Ġtoken Ġin Ġthe Ġmiddle Ġof Ġthe Ġsentence Ġleading Ġto Ġincomplete Ġtranslations . ĠThis Ġcould Ġbe Ġdue Ġto Ġthe Ġfact Ġthat Ġonly Ġ44 % Ġof Ġthe ĠTamil Ġvocabulary Ġand Ġ74 % Ġof Ġthe ĠEnglish Ġvocabulary Ġis Ġtaken Ġfor Ġtraining Ġin Ġthis Ġmodel , Ġas Ġopposed Ġto Ġ100 % Ġof ĠEnglish Ġand ĠTamil Ġwords Ġin Ġthe ĠR NN M orph Ġmodel . </s> Target Ġvocabulary Ġsize </s> A Ġvery Ġlarge Ġtarget Ġvocabulary Ġis Ġan Ġinadvert ent Ġconsequence Ġof Ġthe Ġmorph ological Ġrichness Ġof Ġthe ĠTamil Ġlanguage . ĠThis Ġcreates Ġa Ġpotential Ġrestriction Ġon Ġthe Ġaccuracy Ġof Ġthe Ġmodel Ġas Ġmany Ġinf lection al Ġforms Ġof Ġthe Ġsame Ġword Ġare Ġtrained Ġas Ġindependent Ġunits . ĠOne Ġof Ġthe Ġadvantages Ġof Ġmorph ological Ġsegment ation Ġof ĠTamil Ġtext Ġis Ġthat Ġthe Ġtarget Ġvocabulary Ġsize Ġdecreased Ġfrom Ġ340 , 325 Ġto Ġa Ġmere Ġ41 , 9 06 . ĠThis Ġreduction Ġhelps Ġimprove Ġthe Ġperformance Ġof Ġthe Ġtranslation Ġas Ġthe Ġoccurrence Ġof Ġunknown Ġtokens Ġwas Ġreduced Ġcompared Ġto Ġthe ĠR NN Search Ġmodel . ĠThis Ġmorph ologically Ġsegment ed Ġvocabulary Ġis Ġdivided Ġinto Ġa Ġcollection Ġof Ġmorph ological Ġroots Ġand Ġinf lections Ġas Ġindividual Ġunits . </s> Rep et itions </s> Some Ġof Ġthe Ġtranslations Ġof Ġthe ĠR NN M orph Ġmodel Ġhave Ġrepet itions Ġof Ġthe Ġsame Ġphrases Ġ( Fig . ĠFIG REF 53 Ġ), Ġwhereas Ġsuch Ġrepet itions Ġoccur Ġmuch Ġless Ġfrequently Ġin Ġthe ĠR NN Search Ġpredictions . ĠSuch Ġtranslations Ġwould Ġmake Ġfor Ġgood Ġresults Ġif Ġthe Ġrepet itions Ġweren 't Ġpresent Ġand Ġall Ġparts Ġof Ġthe Ġsentence Ġoccur Ġjust Ġonce . ĠThese Ġrepet itions Ġmight Ġbe Ġdue Ġto Ġthe Ġincrease Ġin Ġthe Ġgeneral Ġsequence Ġlength Ġof Ġthe Ġtarget Ġsentences Ġbecause Ġof Ġthe Ġmorph ological Ġsegment ation . ĠWhile Ġit Ġis Ġtrue Ġthe Ġtarget Ġvocabulary Ġsize Ġhas Ġdecreased Ġdue Ġto Ġmorph ological Ġsegment ation , Ġthe ĠR NN M orph Ġhas Ġmore Ġinput Ġunits Ġ( morph s ) Ġper Ġsentence , Ġwhich Ġmakes Ġit Ġmore Ġdemanding Ġof Ġthe ĠL ST M 's Ġmemory Ġunits Ġand Ġthe Ġfeed Ġforward Ġnetwork Ġof Ġthe Ġattention Ġmodel . ĠAdditionally , Ġthis Ġbehavior Ġcould Ġalso Ġbe Ġattributed Ġto Ġthe Ġerrors Ġin Ġthe Ġsemi - super vised Ġmorph ological Ġsegment ation Ġdue Ġto Ġthe Ġcomplexities Ġof Ġthe ĠTamil Ġlanguage Ġand Ġthe Ġextent Ġof Ġthe Ġcorpus . </s> Model ĠOutput s </s> The Ġtranslation Ġoutputs Ġof Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġand ĠMorph 2 V ec Ġmodels Ġfor Ġthe Ġsame Ġinput Ġsentences Ġfrom Ġthe Ġtest Ġset Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġusing Ġa Ġmorph ological Ġsegment ation Ġtool Ġand Ġhow Ġthe Ġmorp hem es Ġhave Ġchanged Ġthe Ġsentence Ġto Ġbe Ġmore Ġgram matically Ġsound . ĠIt Ġis Ġalso Ġobserved Ġ( from ĠFig . ĠFIG REF 55 Ġ) Ġthat Ġmost Ġof Ġthe Ġtranslation Ġsentences Ġof Ġthe ĠMorph 2 V ec Ġmodel Ġhave Ġno ĠIN LINE FORM 0 ĠUN K ĠIN LINE FORM 1 Ġtokens . ĠThey Ġexist Ġin Ġthe Ġpredictions Ġmostly Ġonly Ġdue Ġto Ġa Ġword Ġin Ġthe ĠEnglish Ġtest Ġsentence Ġnot Ġpresent Ġin Ġthe Ġsource Ġvocabulary . </s> Related ĠWork </s> Prof essors ĠCN ĠKrish nan , ĠSob ha Ġet Ġal Ġdeveloped Ġa Ġmachine - a ided - translation Ġ( MAT ) Ġsystem Ġsimilar Ġto Ġthe ĠAn usa ak ara ĠEnglish ĠHindi ĠMT Ġsystem , Ġusing Ġa Ġsmall Ġcorpus Ġand Ġvery Ġfew Ġtransfer Ġrules , Ġavailable Ġat ĠAU - K BC Ġwebsite ĠB IB REF 14 Ġ. ĠBal aj ap ally Ġet Ġal . Ġbal aj ap ally 2006 mult ilingual Ġdeveloped Ġan Ġexample Ġbased Ġmachine Ġtranslation Ġ( E BM T ) Ġsystem Ġwith Ġ700 000 Ġsentences Ġfor ĠEnglish Ġto ĠIN LINE FORM 0 ĠTamil , ĠK ann ada , ĠHindi ĠIN LINE FORM 1 Ġtransl iter ated Ġtext ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠRen gan athan Ġre ng an athan 2002 inter active Ġdeveloped Ġa Ġrule Ġbased ĠMT Ġsystem Ġfor ĠEnglish Ġand ĠTamil Ġusing Ġgrammar Ġrules Ġfor Ġthe Ġlanguage Ġpair . ĠVet ri vel Ġet Ġal . Ġvet ri vel 2010 english Ġused ĠHM Ms Ġto Ġalign Ġand Ġtranslate ĠEnglish Ġand ĠTamil Ġparallel Ġsentences Ġto Ġbuild Ġan ĠSM T Ġsystem . ĠIrvine Ġet Ġal . Ġir vine 2013 comb ining Ġtried Ġto Ġcombine Ġparallel Ġand Ġsimilar Ġcorpor a Ġto Ġimprove Ġthe Ġperformance Ġof ĠEnglish Ġto ĠTamil ĠSM T Ġamongst Ġother Ġlanguages . ĠKas th uri Ġet Ġal . Ġk as th uri 2014 rule Ġused Ġa Ġrule Ġbased ĠMT Ġsystem Ġusing Ġtransfer Ġlex icon Ġand Ġmorph ological Ġanalysis Ġtools . ĠAng lab har athi Ġwas Ġdeveloped Ġat ĠI IT ĠKan pur , Ġa Ġsystem Ġtranslating ĠEnglish Ġto Ġa Ġcollection Ġof ĠIndian Ġlanguages Ġincluding ĠTamil Ġusing ĠCF G Ġlike Ġstructures Ġto Ġcreate Ġa Ġpseudo Ġtarget Ġto Ġconvert Ġto ĠIndian Ġlanguages ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠA Ġvariety Ġof Ġhybrid Ġapproaches Ġhave Ġalso Ġbeen Ġused Ġfor ĠEnglish âĢĵ Tam il ĠMT Ġin Ġcombinations Ġof Ġrule Ġbased Ġ( transfer Ġmethods ), Ġinter ling ua Ġrepresentations ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. ĠThe Ġuse Ġof ĠStatistical ĠMachine ĠTranslation Ġtook Ġover Ġthe ĠEnglish âĢĵ Tam il ĠMT Ġsystem Ġresearch Ġbecause Ġof Ġits Ġdesirable Ġproperties Ġof Ġlanguage Ġindependence , Ġbetter Ġgeneral ization Ġfeatures Ġand Ġa Ġreduced Ġrequirement Ġof Ġlinguistic Ġexpertise ĠB IB REF 1 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ. ĠVarious Ġenhancement Ġtechniques Ġexternal Ġto Ġthe ĠMT Ġsystem Ġhave Ġalso Ġbeen Ġproposed Ġto Ġimprove Ġthe Ġperformance Ġof Ġtranslation Ġusing Ġmorph ological Ġpre Ġand Ġpost Ġprocessing Ġtechniques ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ. </s> The Ġuse Ġof ĠR NN ĠEnc oder ĠDecoder Ġmodels Ġin Ġmachine Ġtranslation Ġhas Ġshown Ġgood Ġresults Ġin Ġlanguages Ġwith Ġsimilar Ġgram matical Ġstructure . ĠDeep ĠMT Ġsystems Ġhave Ġbeen Ġperforming Ġbetter Ġthan Ġthe Ġother Ġshallow ĠSM T Ġmodels Ġrecently , Ġwith Ġthe Ġavailability Ġof Ġcomputational Ġresources Ġand Ġhardware Ġmaking Ġit Ġfeasible Ġto Ġtrain Ġsuch Ġmodels . ĠThe Ġfirst Ġof Ġthese Ġmodels Ġcame Ġin Ġ2014 , Ġwith ĠCho Ġet Ġal ĠSecond One By Cho . ĠThe Ġmodel Ġused Ġwas Ġthe ĠR NN ĠL ST M Ġenc oder Ġdec oder Ġmodel Ġwith Ġthe Ġcontext Ġvector Ġoutput Ġof Ġthe Ġenc oder Ġ( run Ġfor Ġevery Ġword Ġin Ġthe Ġsentence ) Ġis Ġfed Ġto Ġevery Ġdec oder Ġunit Ġalong Ġwith Ġthe Ġprevious Ġword Ġoutput Ġuntil ĠIN LINE FORM 0 ĠE OS ĠIN LINE FORM 1 Ġis Ġreached . ĠThis Ġmodel Ġwas Ġused Ġto Ġscore Ġtranslation Ġresults Ġof Ġanother ĠMT Ġsystem . ĠS uts ke ver Ġet Ġal . Ġs uts ke ver 2014 sequence Ġcreated Ġa Ġsimilar Ġenc oder Ġdec oder Ġmodel Ġwith Ġthe Ġdec oder Ġgetting Ġthe Ġcontext Ġvector Ġonly Ġfor Ġthe Ġfirst Ġword Ġof Ġthe Ġtarget Ġlanguage Ġsentence . ĠAfter Ġthat , Ġonly Ġthe Ġdec oded Ġtarget Ġoutputs Ġact Ġas Ġinputs Ġto Ġthe Ġvarious Ġtime Ġsteps Ġof Ġthe Ġdec oder . ĠOne Ġmajor Ġdrawback Ġof Ġthese Ġmodels Ġis Ġthe Ġsize Ġof Ġthe Ġcontext Ġvector Ġof Ġthe Ġenc oder Ġbeing Ġstatic Ġin Ġnature . ĠThe Ġsame Ġsized Ġvector Ġwas Ġexpected Ġto Ġto Ġrepresent Ġsentences Ġof Ġarbitrary Ġlength , Ġwhich Ġwas Ġimpractical Ġwhen Ġit Ġcame Ġto Ġvery Ġlong Ġsentences . </s> The Ġnext Ġbreakthrough Ġcame Ġfrom ĠBah dan au Ġet Ġal . ĠBah dan au 2014 Ġwhere Ġvariable Ġlength Ġword Ġvectors Ġwere Ġused Ġand Ġinstead Ġof Ġjust Ġthe Ġcontext Ġvector , Ġa Ġweighted Ġsum Ġof Ġthe Ġinputs Ġis Ġgiven Ġfor Ġthe Ġdec oder . ĠThis Ġenabled Ġselective Ġlookup Ġto Ġthe Ġsource Ġsentence Ġduring Ġdecoding Ġand Ġis Ġknown Ġas Ġthe Ġattention Ġmechanism ĠB IB REF 27 Ġ. ĠThe Ġattention Ġmechanism Ġwas Ġfurther Ġanalysed Ġby ĠLu ong Ġet Ġal . Ġl u ong 2015 effective Ġwhere Ġthey Ġmade Ġa Ġdistinction Ġbetween Ġglobal Ġand Ġlocal Ġattention Ġby Ġmeans Ġof ĠA ER Ġscores Ġof Ġthe Ġattention Ġvectors . ĠA ĠGa ussian Ġdistribution Ġand Ġa Ġmon ot onic Ġlookup Ġwere Ġused Ġto Ġfacilitate Ġthe Ġcorresponding Ġlocal Ġsource Ġsentence Ġlook - up . </s> Conclusion </s> Thus , Ġit Ġis Ġseen Ġthat Ġthe Ġuse Ġof Ġmorph ological Ġsegment ation Ġon Ġa Ġmorph ologically Ġrich Ġlanguage Ġbefore Ġtranslation Ġhelps Ġwith Ġthe Ġperformance Ġof Ġthe Ġtranslation Ġin Ġmultiple Ġways . ĠThus , Ġmachine Ġtranslation Ġinvolving Ġmorph ologically Ġrich Ġlanguages Ġshould Ġideally Ġbe Ġcarried Ġout Ġonly Ġafter Ġmorph ological Ġsegment ation . ĠIf Ġthe Ġtranslation Ġhas Ġto Ġbe Ġcarried Ġout Ġbetween Ġtwo Ġmorph ologically Ġrich Ġlanguages , Ġthen Ġboth Ġthe Ġlanguages ' Ġsentences Ġshould Ġbe Ġindividually Ġsegment ed Ġbased Ġon Ġmorphology . ĠThis Ġis Ġbecause Ġwhile Ġit Ġis Ġtrue Ġthat Ġthey Ġare Ġboth Ġmorph ologically Ġrich Ġlanguages , Ġthe Ġschemes Ġthat Ġthe Ġlanguages Ġuse Ġfor Ġthe Ġprocess Ġof Ġagg l ut ination Ġmight Ġbe Ġdifferent , Ġin Ġwhich Ġcase Ġa Ġmapping Ġbetween Ġthe Ġunits Ġwould Ġbe Ġdifficult Ġwithout Ġthe Ġsegment ation . </s> One Ġdrawback Ġof Ġmorph ological Ġsegment ation Ġis Ġthe Ġincrease Ġin Ġcomplexity Ġof Ġthe Ġmodel Ġdue Ġto Ġan Ġincrease Ġin Ġthe Ġaverage Ġsentence Ġlengths . ĠThis Ġcannot Ġbe Ġavoided Ġas Ġit Ġis Ġessential Ġto Ġenable Ġa Ġcorrespondence Ġbetween Ġthe Ġsentences Ġof Ġthe Ġtwo Ġlanguages Ġwhen Ġone Ġof Ġthem Ġis Ġa Ġsimple Ġf usional Ġlanguage . ĠEven Ġwith Ġthe Ġincrease Ġin Ġthe Ġaverage Ġsentence Ġlength , Ġthe Ġattention Ġmodels Ġthat Ġhave Ġbeen Ġdeveloped Ġto Ġensure Ġcorrectness Ġof Ġtranslation Ġof Ġlong Ġsequences Ġcan Ġbe Ġput Ġto Ġgood Ġuse Ġwhen Ġinvolving Ġmorph ologically Ġrich Ġlanguages . ĠAnother Ġpoint Ġto Ġnote Ġhere Ġis Ġthat Ġmorph ologically Ġrich Ġlanguages Ġlike ĠTamil Ġgenerally Ġhave Ġlesser Ġnumber Ġof Ġwords Ġper Ġsentence Ġthan Ġlanguages Ġlike ĠEnglish Ġdue Ġto Ġthe Ġinherent Ġproperty Ġof Ġagg l ut ination . </s> Future ĠWork </s> The Ġmodel Ġimplemented Ġin Ġthis Ġpaper Ġonly Ġincludes Ġsource - side Ġmorph ological Ġsegment ation Ġand Ġdoes Ġnot Ġinclude Ġa Ġtarget Ġside Ġmorph ological Ġagg l ut ination Ġto Ġgive Ġback Ġthe Ġoutput Ġin Ġwords Ġrather Ġthan Ġmorp hem es . ĠIn Ġorder Ġto Ġimplement Ġan Ġend - to - end Ġtranslation Ġsystem Ġfor Ġmorph ologically Ġrich Ġlanguages , Ġa Ġmorph ological Ġgenerator Ġis Ġessential Ġbecause Ġthe Ġoutput Ġunits Ġof Ġthe Ġtranslation Ġcannot Ġbe Ġmorp hem es . </s> The Ġsame Ġmodel Ġimplemented Ġcan Ġbe Ġfurther Ġenhanced Ġby Ġmeans Ġof Ġa Ġbetter Ġcorpus Ġthat Ġcan Ġgeneral ize Ġover Ġmore Ġthan Ġjust Ġdomain Ġspecific Ġsource Ġsentences . ĠAlso , Ġthe Ġuse Ġof Ġa Ġbetter ĠGPU Ġwould Ġresult Ġin Ġa Ġbetter Ġallocation Ġof Ġthe Ġhidden Ġlayer Ġsizes Ġand Ġthe Ġbatch Ġsizes Ġthereby Ġpossibly Ġincreasing Ġthe Ġscope Ġand Ġaccuracy Ġof Ġlearning Ġof Ġthe Ġtranslation Ġmodel . </s> Although Ġnot Ġdirectly Ġrelated Ġto ĠMachine ĠTranslation , Ġthe Ġnovel Ġenc oder âĢĵ Ġdec oder Ġarchitecture Ġproposed Ġin Ġby ĠRock t as chel Ġet Ġal . Ġrock t as chel 2015 reason ing Ġfor ĠNatural ĠLanguage ĠIn ference Ġ( N LI ) Ġcan Ġbe Ġused Ġfor Ġthe Ġsame . ĠTheir Ġmodel Ġf uses Ġinf erences Ġfrom Ġeach Ġand Ġevery Ġindividual Ġword , Ġsummar izing Ġinformation Ġat Ġeach Ġstep , Ġthereby Ġlinking Ġthe Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġwith Ġthat Ġof Ġthe Ġdec oder Ġby Ġmeans Ġof Ġa Ġweighted Ġsum , Ġtrained Ġfor Ġoptimization . </s> Acknowled gements </s> I Ġwould Ġlike Ġto Ġthank ĠDr . ĠM . ĠAn and ĠKumar , ĠAssistant ĠProfessor , ĠAm rit a ĠVish wa ĠV idy ape et ham Ġfor Ġhis Ġcontinuous Ġsupport Ġand Ġguidance . ĠI Ġwould Ġalso Ġlike Ġto Ġthank ĠDr . ĠAr vind an , ĠProfessor , ĠSS N ĠCollege ĠOf ĠEngineering Ġfor Ġhis Ġinputs Ġand Ġsuggestions .
Introduction The use of RNNs in the field of Statistical Machine Translation (SMT) has revolutionised the approaches to automated translation. As opposed to traditional shallow SMT models, which require a lot of memory to run, these neural translation models require only a small fraction of memory used, about 5% BIBREF0 . Also, neural translation models are optimized such that every module is trained to jointly improve translation quality. With that being said, one of the main downsides of neural translation models is the heavy corpus requirement in order to ensure learning of deeper contexts. This is where the application of these encoder decoder architectures in translation to and/or from morphologically rich languages takes a severe hit. For any language pair, the efficiency of an MT system depends on two major factors: the availability and size of parallel corpus used for training and the syntactic divergence between the two languages i.e morphological richness, word order differences, grammatical structure etc. BIBREF0 . The main differences between the languages stem from the fact that languages similar to English are predominantly fusional languages whereas many of the morphologically rich languages are agglutinative in nature. The nature of morphologically rich languages being structurally and semantically discordant from languages like English adds to the difficulty of SMT involving such languages. In morphologically rich languages, any suffix can be added to any verb or noun to simply mean one specific thing about that particular word that the suffix commonly represents (agglutination). This means that there exists a lot of inflectional forms of the same noun and verb base words, conveying similar notions. For example, in Tamil, there are at least 30,000 inflectional forms of any given verb and about 5,000 forms of inflectional forms for any noun. The merged words carry information about part of speech (POS) tags, tense, plurality and so forth that are important for analyzing text for Machine Translation (MT). Not only are these hidden meanings not captured, the corresponding root words are trained as different units, thereby increasing the complexity of developing such MT systems BIBREF1 . To add to the complexities of being a morphologically rich language, there are several factors unique to Tamil that make translation very difficult. The availability of parallel corpus for Tamil is very scarce. Most of the other models in the field of English–Tamil MT have made use of their own translation corpora that were manually created for the purposes of research. Most of these corpora are not available online for use. Another issue specific to Tamil is the addition of suffix characters included to the words in the language for smoothness in pronunciation. These characters are of so many different types; there is a unique suffix for each and every consonant in the language. These suffixes degrade performance of MT because the same words with different such pronounciation-based suffixes will be taken as different words in training. Also to take into consideration is the existence of two different forms of the language being used. Traditionally defined Tamil and its pronunciations aren't acoustically pleasing to use. There's no linguistic flow between syllables and its usage in verbal communication is time consuming. Therefore, there exists two forms of the language, the written form, rigid in structure and syntax, and the spoken form, in which the flow and pace of the language is given priority over syntax and correctness of spelling. This divide leads to the corpus having 2 different versions of the language that increase the vocabulary even with the same words. This can be evidently seen in the corpus between the sentences used in the Bible, which is in traditional Tamil and sentences from movie subtitles, being in spoken Tamil format. To account for such difficulties, a trade-off between domain specificity and size of the corpus is integral in building an English–Tamil neural MT system. Corpus The corpus selected for this experiment was a combination of different corpora from various domains. The major part of the corpus was made up by the EnTam v2 corpus BIBREF2 . This corpus contained sentences taken from parallel news articles, English and Tamil bible corpus and movie subtitles. It also comprised of a tourism corpus that was obtained from TDIL (Technology Development for Indian Languages) and a corpus created from Tamil novels and short stories from AU-KBC, Anna university. The complete corpus consisted of 197,792 sentences. Fig. FIGREF20 shows the skinny shift and heatmap representations of the relativity between the sentences in terms of their sentence lengths. An extra monolingual Tamil corpus, collated from various online sources was used for the word2vec embedding of the Tamil target language to enhance the richness of context of the word vectors. It was also used to create the language model for the phrase-based SMT model. This corpus contained 567,772 sentences and was self-collected by combining hundreds of ancient Tamil scriptures, novels and poems by accessing the websites of popular online ebook libraries in Python using the urllib package. Since the sources had Tamil text in different encodings, the encoding scheme was standardized to be UTF-8 for the entirety of the monolingual and parallel corpora using the chardet package. The corpora were cleaned for any stray special characters, unnecessary html tags and website URLs. Word2Vec The word embeddings of the source and target language sentences are used as initial vectors of the model to improve contextualization. The skip gram model of the word2vec algorithm optimizes the vectors by accounting for the average log probability of context words given a source word. DISPLAYFORM0 where k is the context window taken for the vectorization, INLINEFORM0 refers to the INLINEFORM1 word of the corpus and INLINEFORM2 is the size of the training corpus in terms of the number of words. Here, the probabily INLINEFORM3 is computed as a hierarchical softmax of the product of the transpose of the output vector of INLINEFORM4 and the input vector of INLINEFORM5 for each and every pair over the entire vocabulary. The processes of negative sampling and subsampling of frequent words that were used in the original model aren't used in this experiment BIBREF3 . For the process of creating semantically meaningful word embeddings, a monolingual corpus of 569,772 Tamil sentences was used. This gave the vectors more contextual richness due to the increased size of the corpus as opposed to using just the bilingual corpus' target side sentences BIBREF3 . In the experiment, the word2vec model was trained using a vector size of 100 to ensure that the bulk of the limited memory of the GPU will be used for the neural attention translation model. It has been shown that any size over that of 150 used for word vectorization gives similar results and that a size of 100 performs close to the model with 150-sized word vectors BIBREF7 . A standard size of 5 was used as window size and the model was trained over 7 worker threads simultaneously. A batch size of 50 words was used for training. The negative sampling was set at 1 as it is the nature of morphologically rich languages to have a lot of important words that don't occur more than once in the corpus. The gensim word2vec toolkit was used to implement this word embedding process BIBREF8 . Neural Translation Model The model used for translation is the one implemented by Bahdanau et al. Bahdanau2014. A bidirectional LSTM encoder first takes the source sentence and encodes it into a context vector which acts as input for the decoder. The decoder is attention-based where the hidden states of the decoder get as input the weighted sum of all the hidden layer outputs of the encoder alongwith the output of the previous hidden layer and the previously decoded word. This provides a contextual reference into the source language sentence BIBREF4 . Neural Machine Translation models directly compute the probability of the target language sentence given the source language sentence, word by word for every time step. The model with a basic decoder without the attention module computes the log probability of target sentence given source sentence as the sum of log probabilities of every word given every word before that. The attention-based model, on the other hand, calculates: DISPLAYFORM0 where INLINEFORM0 is the number of words in the target sentence, INLINEFORM1 is the target sentence, INLINEFORM2 is the source sentence, INLINEFORM3 is the fixed length output vector of the encoder and INLINEFORM4 is the weighted sum of all the hidden layer outputs of the encoder at every time step. Both the encoder's output context vector and the weighted sum (known as attention vector) help to improve the quality of translation by enabling selective source sentence lookup. The decoder LSTM computes: DISPLAYFORM0 where the probability is computed as a function of the decoder's output in the previous time step INLINEFORM0 , the hidden layer vector of the decoder in the current timestep INLINEFORM1 and the context vector from the attention mechanism INLINEFORM2 . The context vector INLINEFORM3 for time step INLINEFORM4 is computed as a weighted sum of the output of the entire sentence using a weight parameter INLINEFORM5 : DISPLAYFORM0 where INLINEFORM0 is the number of tokens in the source sentence, INLINEFORM1 refers to the value of the hidden layer of the encoder at time step INLINEFORM2 , and INLINEFORM3 is the alignment parameter. This parameter is calculated by means of a feed forward neural network to ensure that the alignment model is free from the difficulties of contextualization of long sentences into a single vector. The feed forward network is trained along with the neural translation model to jointly improve the performance of the translation. Mathematically, DISPLAYFORM0 DISPLAYFORM1 where INLINEFORM0 is the softmax output of the result of the feedforward network, INLINEFORM1 is the hidden state value of the decoder at timestep INLINEFORM2 and INLINEFORM3 is the encoder's hidden layer annotation at timestep INLINEFORM4 . A concatenation of the forward and the reverse hidden layer parameters of the encoder is used at each step to compute the weights INLINEFORM5 for the attention mechanism. This is done to enable an overall context of the sentence, as opposed to a context of only all the previous words of the sentence for every word in consideration. Fig. FIGREF12 is the general architecture of the neural translation model without the Bidirectional LSTM encoder. A global attention mechanism is preferred over local attention because the differences in the structures of the languages cannot be mapped efficiently to enable lookup into the right parts of the source sentence. Using local attention mechanism with a monotonic context lookup, where the region around INLINEFORM0 source word is looked up for the prediction of the INLINEFORM1 target word, is impractical because of the structural discordance between the English and Tamil sentences (see Figs. FIGREF37 and FIGREF44 ). The use of gaussian and other such distributions to facilitate local attention would also be inefficient because the existence of various forms of translations for the same source sentence involving morphological and structural variations that don't stay uniform through the entire corpus BIBREF5 . The No Peepholes (NP) variant of the LSTM cell, formulated in Greff et al. greff2015lstm is used in this experiment as it proved to give the best results amongst all the variants of an LSTM cell. It is specified by means of a gated mechanism designed to ensure that the vanishing gradient problem is prevented. LSTM maintains its hidden layer in two components, the cell vector INLINEFORM0 and the actual hidden layer output vector INLINEFORM1 . The cell vector is ensured to never reach zero by means of a weighted sum of the previous layer's cell vector INLINEFORM2 regulated by the forget gate INLINEFORM3 and an activation of the weighted sum of the input INLINEFORM4 in the current timestep INLINEFORM5 and the previous timestep's hidden layer output vector INLINEFORM6 . The combination is similarly regulated by the input gate INLINEFORM7 . The hidden layer output is determined as an activation of the cell gate, regulated by the output gate INLINEFORM8 . The interplay between these two vectors ( INLINEFORM9 and INLINEFORM10 ) at every timestep ensures that the problem of vanishing gradients doesn't occur. The three gates are also formed as a sigmoid of the weighted sum of the previous hidden layer output INLINEFORM11 and the input in the current timestep INLINEFORM12 . The output generated out of the LSTM's hidden layer is specified as a weighted softmax over the hidden layer output INLINEFORM13 . The learnable parameters of an LSTM cell are all the weights INLINEFORM14 and the biases INLINEFORM15 . DISPLAYFORM0 The LSTM specified by equations 7 through 11 is the one used for the decoder of the model. The encoder uses a bidirectional RNN LSTM cell in which there are two hidden layer components INLINEFORM0 and INLINEFORM1 that contribute to the output INLINEFORM2 of each time step INLINEFORM3 . Both the components have their own sets of LSTM equations in such a way that INLINEFORM4 for every timestep is computed from the first timestep till the INLINEFORM5 token is reached and INLINEFORM6 is computed from the INLINEFORM7 timestep backwards until the first token is reached. All the five vectors of the two components are all exactly the same as the LSTM equations specified with one variation in the computation of the result. DISPLAYFORM0 Morphological Segmentation The morphological segmentation used is a semi-supervised extension to the generative probabilistic model of maximizing the probability of a INLINEFORM0 prefix,root,postfix INLINEFORM1 recursive split up of words based on an exhaustive combination of all possible morphemes. The details of this model are specified and extensively studied in Kohonen et al. kohonen2010semi. The model parameters INLINEFORM2 include the morph type count, morph token count of training data, the morph strings and their counts. The model is trained by maximizing the Maximum A Posteriori (MAP) probability using Bayes' rule: DISPLAYFORM0 where INLINEFORM0 refers to every word in the training lexicon. The prior INLINEFORM1 is estimated using the Minimum Description Length(MDL) principle. The likelihood INLINEFORM2 is estimated as: DISPLAYFORM0 where INLINEFORM0 refers to the intermediate analyses and INLINEFORM1 refers to the INLINEFORM2 morpheme of word INLINEFORM3 . An extension to the Viterbi algorithm is used for the decoding step based on exhaustive mapping of morphemes. To account for over-segmentation and under-segmentation issues associated with unsupervised morphological segmentation, extra parameters ( INLINEFORM0 ) and ( INLINEFORM1 ) are used with the cost function INLINEFORM2 DISPLAYFORM0 where INLINEFORM0 is the likelihood of the cost function, INLINEFORM1 describes the likelihood of contribution of the annotated dataset to the cost function and INLINEFORM2 is the likelihood of the labeled data. A decrease in the value of INLINEFORM3 will cause smaller segments and vice versa. INLINEFORM4 takes care of size discrepancies due to reduced availability of annotated corpus as compared to the training corpus BIBREF2 , BIBREF6 . The Python extension to the morphological segmentation tool morfessor 2.0 was used for this experiment to perform the segmentation. The annotation data for Tamil language collated and released by Anoop Kunchukkutan in the Indic NLP Library was used as the semi-supervised input to the model BIBREF9 , BIBREF6 . Experiment The complexities of neural machine translation of morphologically rich languages were studied with respect to English to Tamil machine translation using the RNN LSTM Bi-directional encoder attention decoder architecture. To compare with a baseline system, a phrase based SMT system was implemented using the same corpus. The Factored SMT model with source-side preprocessing by Kumar et al. kumar2014improving was used as a reference for the translation between these language pairs. Also, an additional 569,772 monolingual Tamil sentences were used for the language model of the SMT system. The model used could be split up into various modules as expanded in Fig. FIGREF17 . Bucketing The input source and target language sentences used for training were taken and divided into bucketed pairs of sentences of a fixed number of sizes. This relationship was determined by examining the distribution of words in the corpus primarily to minimize the number of PAD tokens in the sentence. The heat map of the number of words in the English–Tamil sentence pairs of the corpus revealed that the distribution is centered around the 10–20 words region. Therefore, more buckets in that region were applied as there would be enough number of examples in each of these bucket pairs for the model to learn about the sentences in each and every bucket. The exact scheme used for the RNNSearch models is specified by Fig. FIGREF21 . The bucketing scheme for the RNNMorph model, involving morphs instead of words, was a simple shifted scheme of the one used in Fig. FIGREF21 , where every target sentence bucket count was increased uniformly by 5. Model Details Due to various computational constraints and lack of availability of comprehensive corpora, the vocabularies for English and Tamil languages for the RNNSearch model were restricted to 60,000 out of 67,768 and 150,000 out of 340,325 respectively. The vocabulary of the languages for the RNNMorph didn't have to be restricted and the actual number of words in the corpus i.e. 67,768 words for English and 41,906 words for Tamil could be accommodated into the training. Words not in the vocabulary from the test set input and output were replaced with the universal INLINEFORM0 UNK INLINEFORM1 token, symbolizing an unknown word. The LSTM hidden layer size, the training batch size, and the vocabulary sizes of the languages, together, acted as a bottleneck. The model was run on a 2GB NVIDIA GeForce GT 650M card with 384 cores and the memory allotment was constrained to the limits of the GPU. Therefore, after repeated experimentation, it was determined that with a batch size of 16, the maximum hidden layer size possible was 500, which was the size used. Attempts to reduce the batch size resulted in poor convergence, and so the parameters were set to center around the batch size of 16. The models used were of 4 layers of LSTM hidden units in the bidirectional encoder and attention decoder. The model used a Stochastic Gradient Descent (SGD) optimization algorithm with a sampled softmax loss of 512 per sample to handle large vocabulary size of the target language BIBREF10 . The model was trained with a learning rate 1.0 and a decay of rate 0.5 enforced manually. Gradient clipping based on the global norm of 5.0 was carried out to prevent gradients exploding and going to unrecoverable values tending towards infinity. The model described is the one used in the Tensorflow BIBREF11 seq2seq library. Results and Discussion The BLEU metric parameters (modified 1-gram, 2-gram, 3-gram and 4-gram precision values) and human evaluation metrics of adequacy, fluency and relative ranking values were used to evaluate the performance of the models. BLEU Evaluation The BLEU scores obtained using the various models used in the experiment are tabulated in Table TABREF25 . The BLEU metric computes the BLEU unigram, bigram, trigram and BLEU-4 modified precision values, each micro-averaged over the test set sentences BIBREF7 . It was observed, as expected, that the performance of the phrase-based SMT model was inferior to that of the RNNSearch model. The baseline RNNSearch system was further refined by using word2vec vectors to embed semantic understanding, as observed with the slight increase in the BLEU scores. Fig. FIGREF26 plots the BLEU scores as a line graph for visualization of the improvement in performance. Also, the 4-gram BLEU scores for the various models were plotted as a bar graph in Fig. FIGREF26 Due to the agglutinative and morphologically rich nature of the target language i.e. Tamil, the use of morphological segmentation to split the words into morphemes further improved the BLEU precision values in the RNNMorph model. One of the reasons for the large extent of increase in the BLEU score could be attributed to the overall increase in the number of word units per sentence. Since the BLEU score computes micro-average precision scores, an increase in both the numerator and denominator of the precision scores is apparent with an increase in the number of tokens due to morphological segmentation of the target language. Thus, the numeric extent of the increase of accuracy might not efficiently describe the improvement in performance of the translation. Human Evaluation To ensure that the increase in BLEU score correlated to actual increase in performance of translation, human evaluation metrics like adequacy, precision and ranking values (between RNNSearch and RNNMorph outputs) were estimated in Table TABREF30 . A group of 50 native people who were well-versed in both English and Tamil languages acted as annotators for the evaluation. A collection of samples of about 100 sentences were taken from the test set results for comparison. This set included a randomized selection of the translation results to ensure the objectivity of evaluation. Fluency and adequacy results for the RNNMorph results are tabulated. Adequacy rating was calculated on a 5-point scale of how much of the meaning is conveyed by the translation (All, Most, Much, Little, None). The fluency rating was calculated based on grammatical correctness on a 5-point scale of (Flawless, Good, Non-native, Disfluent, Incomprehensive). For the comparison process, the RNNMorph and the RNNSearch + Word2Vec models’ sentence level translations were individually ranked between each other, permitting the two translations to have ties in the ranking. The intra-annotator values were computed for these metrics and the scores are shown in Table TABREF32 BIBREF12 , BIBREF13 . The human evaluation Kappa co-efficient results are calculated with respect to: DISPLAYFORM0 It was observed that the ranking Kappa co-efficient for intra-annotator ranking of the RNNMorph model was at 0.573, higher that the 0.410 of the RNNSearch+Word2Vec model, implying that the annotators found the RNNMorph model to produce better results when compared to the RNNSearch + Word2Vec model. Model Parameters The learning rate decay through the training process of the RNNMorph model is showcased in the graph in Fig. FIGREF34 . This process was done manually where the learning rate was decayed after the end of specific epochs based on an observed stagnation in perplexity.The RNNMorph model achieved saturation of perplexities much earlier through the epochs than the RNNSearch + Word2Vec model. This conforms to the expected outcome as the morphological segmentation has reduced the vocabulary size of the target language from 340,325 words to a mere 41,906 morphs. The error function used was the sampled SoftMax loss to ensure a large target vocabulary could be accommodated BIBREF10 . A zoomed inset graph (Fig. FIGREF35 ) has been used to visualize the values of the error function for the RNNSearch + Word2Vec and RNNMorph models with 4 hidden layers. It can be seen that the RNNMorph model is consistently better in terms of the perplexity values through the time steps. Attention Vectors In order to further demonstrate the quality of the RNNMorph model, the attention vectors of both the RNNSearch with Word2Vec embedding and RNNMorph models are compared for several good translations in Figs. FIGREF37 and FIGREF44 . It is observed that the reduction in vocabulary size has improved the source sentence lookup by quite an extent. Each cell in the heatmap displays the magnitude of the attention layer weight INLINEFORM0 for the INLINEFORM1 Tamil word and the INLINEFORM2 English word in the respective sentences. The intensity of black corresponds to the magnitude of the cell INLINEFORM3 . Also, the attention vectors of the RNNSearch model with Word2Vec embeddings tend to attend to INLINEFORM4 EOS INLINEFORM5 token in the middle of the sentence leading to incomplete translations. This could be due to the fact that only 44% of the Tamil vocabulary and 74% of the English vocabulary is taken for training in this model, as opposed to 100% of English and Tamil words in the RNNMorph model. Target vocabulary size A very large target vocabulary is an inadvertent consequence of the morphological richness of the Tamil language. This creates a potential restriction on the accuracy of the model as many inflectional forms of the same word are trained as independent units. One of the advantages of morphological segmentation of Tamil text is that the target vocabulary size decreased from 340,325 to a mere 41,906. This reduction helps improve the performance of the translation as the occurrence of unknown tokens was reduced compared to the RNNSearch model. This morphologically segmented vocabulary is divided into a collection of morphological roots and inflections as individual units. Repetitions Some of the translations of the RNNMorph model have repetitions of the same phrases (Fig. FIGREF53 ), whereas such repetitions occur much less frequently in the RNNSearch predictions. Such translations would make for good results if the repetitions weren't present and all parts of the sentence occur just once. These repetitions might be due to the increase in the general sequence length of the target sentences because of the morphological segmentation. While it is true the target vocabulary size has decreased due to morphological segmentation, the RNNMorph has more input units (morphs) per sentence, which makes it more demanding of the LSTM's memory units and the feed forward network of the attention model. Additionally, this behavior could also be attributed to the errors in the semi-supervised morphological segmentation due to the complexities of the Tamil language and the extent of the corpus. Model Outputs The translation outputs of the RNNSearch + Word2Vec and Morph2Vec models for the same input sentences from the test set demonstrate the effectiveness of using a morphological segmentation tool and how the morphemes have changed the sentence to be more grammatically sound. It is also observed (from Fig. FIGREF55 ) that most of the translation sentences of the Morph2Vec model have no INLINEFORM0 UNK INLINEFORM1 tokens. They exist in the predictions mostly only due to a word in the English test sentence not present in the source vocabulary. Related Work Professors CN Krishnan, Sobha et al developed a machine-aided-translation (MAT) system similar to the Anusaakara English Hindi MT system, using a small corpus and very few transfer rules, available at AU-KBC website BIBREF14 . Balajapally et al. balajapally2006multilingual developed an example based machine translation (EBMT) system with 700000 sentences for English to INLINEFORM0 Tamil, Kannada, Hindi INLINEFORM1 transliterated text BIBREF15 , BIBREF16 . Renganathan renganathan2002interactive developed a rule based MT system for English and Tamil using grammar rules for the language pair. Vetrivel et al. vetrivel2010english used HMMs to align and translate English and Tamil parallel sentences to build an SMT system. Irvine et al. irvine2013combining tried to combine parallel and similar corpora to improve the performance of English to Tamil SMT amongst other languages. Kasthuri et al. kasthuri2014rule used a rule based MT system using transfer lexicon and morphological analysis tools. Anglabharathi was developed at IIT Kanpur, a system translating English to a collection of Indian languages including Tamil using CFG like structures to create a pseudo target to convert to Indian languages BIBREF17 , BIBREF18 . A variety of hybrid approaches have also been used for English–Tamil MT in combinations of rule based (transfer methods), interlingua representations BIBREF19 , BIBREF20 , BIBREF21 . The use of Statistical Machine Translation took over the English–Tamil MT system research because of its desirable properties of language independence, better generalization features and a reduced requirement of linguistic expertise BIBREF1 , BIBREF22 , BIBREF23 . Various enhancement techniques external to the MT system have also been proposed to improve the performance of translation using morphological pre and post processing techniques BIBREF24 , BIBREF25 , BIBREF26 . The use of RNN Encoder Decoder models in machine translation has shown good results in languages with similar grammatical structure. Deep MT systems have been performing better than the other shallow SMT models recently, with the availability of computational resources and hardware making it feasible to train such models. The first of these models came in 2014, with Cho et al SecondOneByCho. The model used was the RNN LSTM encoder decoder model with the context vector output of the encoder (run for every word in the sentence) is fed to every decoder unit along with the previous word output until INLINEFORM0 EOS INLINEFORM1 is reached. This model was used to score translation results of another MT system. Sutskever et al. sutskever2014sequence created a similar encoder decoder model with the decoder getting the context vector only for the first word of the target language sentence. After that, only the decoded target outputs act as inputs to the various time steps of the decoder. One major drawback of these models is the size of the context vector of the encoder being static in nature. The same sized vector was expected to to represent sentences of arbitrary length, which was impractical when it came to very long sentences. The next breakthrough came from Bahdanau et al. Bahdanau2014 where variable length word vectors were used and instead of just the context vector, a weighted sum of the inputs is given for the decoder. This enabled selective lookup to the source sentence during decoding and is known as the attention mechanism BIBREF27 . The attention mechanism was further analysed by Luong et al. luong2015effective where they made a distinction between global and local attention by means of AER scores of the attention vectors. A Gaussian distribution and a monotonic lookup were used to facilitate the corresponding local source sentence look-up. Conclusion Thus, it is seen that the use of morphological segmentation on a morphologically rich language before translation helps with the performance of the translation in multiple ways. Thus, machine translation involving morphologically rich languages should ideally be carried out only after morphological segmentation. If the translation has to be carried out between two morphologically rich languages, then both the languages' sentences should be individually segmented based on morphology. This is because while it is true that they are both morphologically rich languages, the schemes that the languages use for the process of agglutination might be different, in which case a mapping between the units would be difficult without the segmentation. One drawback of morphological segmentation is the increase in complexity of the model due to an increase in the average sentence lengths. This cannot be avoided as it is essential to enable a correspondence between the sentences of the two languages when one of them is a simple fusional language. Even with the increase in the average sentence length, the attention models that have been developed to ensure correctness of translation of long sequences can be put to good use when involving morphologically rich languages. Another point to note here is that morphologically rich languages like Tamil generally have lesser number of words per sentence than languages like English due to the inherent property of agglutination. Future Work The model implemented in this paper only includes source-side morphological segmentation and does not include a target side morphological agglutination to give back the output in words rather than morphemes. In order to implement an end-to-end translation system for morphologically rich languages, a morphological generator is essential because the output units of the translation cannot be morphemes. The same model implemented can be further enhanced by means of a better corpus that can generalize over more than just domain specific source sentences. Also, the use of a better GPU would result in a better allocation of the hidden layer sizes and the batch sizes thereby possibly increasing the scope and accuracy of learning of the translation model. Although not directly related to Machine Translation, the novel encoder– decoder architecture proposed in by Rocktaschel et al. rocktaschel2015reasoning for Natural Language Inference (NLI) can be used for the same. Their model fuses inferences from each and every individual word, summarizing information at each step, thereby linking the hidden state of the encoder with that of the decoder by means of a weighted sum, trained for optimization. Acknowledgements I would like to thank Dr. M. Anand Kumar, Assistant Professor, Amrita Vishwa Vidyapeetham for his continuous support and guidance. I would also like to thank Dr. Arvindan, Professor, SSN College Of Engineering for his inputs and suggestions.
How were the human judgements assembled?
0ee73909ac638903da4a0e5565c8571fc794ab96
How Ġwere Ġthe Ġhuman Ġjud gements Ġassembled ?
50 Ġhuman Ġannot ators Ġranked Ġa Ġrandom Ġsample Ġof Ġ100 Ġtranslations Ġby ĠAd equ acy , ĠFlu ency Ġand Ġoverall Ġranking Ġon Ġa Ġ5 - point Ġscale .
<s> How Ġwere Ġthe Ġhuman Ġjud gements Ġassembled ? </s> Introduction </s> The Ġuse Ġof ĠR NN s Ġin Ġthe Ġfield Ġof ĠStatistical ĠMachine ĠTranslation Ġ( SM T ) Ġhas Ġrevolution ised Ġthe Ġapproaches Ġto Ġautomated Ġtranslation . ĠAs Ġopposed Ġto Ġtraditional Ġshallow ĠSM T Ġmodels , Ġwhich Ġrequire Ġa Ġlot Ġof Ġmemory Ġto Ġrun , Ġthese Ġneural Ġtranslation Ġmodels Ġrequire Ġonly Ġa Ġsmall Ġfraction Ġof Ġmemory Ġused , Ġabout Ġ5 % ĠB IB REF 0 Ġ. ĠAlso , Ġneural Ġtranslation Ġmodels Ġare Ġoptimized Ġsuch Ġthat Ġevery Ġmodule Ġis Ġtrained Ġto Ġjointly Ġimprove Ġtranslation Ġquality . ĠWith Ġthat Ġbeing Ġsaid , Ġone Ġof Ġthe Ġmain Ġdowns ides Ġof Ġneural Ġtranslation Ġmodels Ġis Ġthe Ġheavy Ġcorpus Ġrequirement Ġin Ġorder Ġto Ġensure Ġlearning Ġof Ġdeeper Ġcontexts . ĠThis Ġis Ġwhere Ġthe Ġapplication Ġof Ġthese Ġenc oder Ġdec oder Ġarchitectures Ġin Ġtranslation Ġto Ġand / or Ġfrom Ġmorph ologically Ġrich Ġlanguages Ġtakes Ġa Ġsevere Ġhit . </s> For Ġany Ġlanguage Ġpair , Ġthe Ġefficiency Ġof Ġan ĠMT Ġsystem Ġdepends Ġon Ġtwo Ġmajor Ġfactors : Ġthe Ġavailability Ġand Ġsize Ġof Ġparallel Ġcorpus Ġused Ġfor Ġtraining Ġand Ġthe Ġsynt actic Ġdivergence Ġbetween Ġthe Ġtwo Ġlanguages Ġi . e Ġmorph ological Ġrichness , Ġword Ġorder Ġdifferences , Ġgram matical Ġstructure Ġetc . ĠB IB REF 0 Ġ. ĠThe Ġmain Ġdifferences Ġbetween Ġthe Ġlanguages Ġstem Ġfrom Ġthe Ġfact Ġthat Ġlanguages Ġsimilar Ġto ĠEnglish Ġare Ġpredominantly Ġf usional Ġlanguages Ġwhereas Ġmany Ġof Ġthe Ġmorph ologically Ġrich Ġlanguages Ġare Ġagg l ut in ative Ġin Ġnature . ĠThe Ġnature Ġof Ġmorph ologically Ġrich Ġlanguages Ġbeing Ġstruct urally Ġand Ġsem antically Ġdiscord ant Ġfrom Ġlanguages Ġlike ĠEnglish Ġadds Ġto Ġthe Ġdifficulty Ġof ĠSM T Ġinvolving Ġsuch Ġlanguages . </s> In Ġmorph ologically Ġrich Ġlanguages , Ġany Ġsuffix Ġcan Ġbe Ġadded Ġto Ġany Ġverb Ġor Ġnoun Ġto Ġsimply Ġmean Ġone Ġspecific Ġthing Ġabout Ġthat Ġparticular Ġword Ġthat Ġthe Ġsuffix Ġcommonly Ġrepresents Ġ( ag gl ut ination ). ĠThis Ġmeans Ġthat Ġthere Ġexists Ġa Ġlot Ġof Ġinf lection al Ġforms Ġof Ġthe Ġsame Ġnoun Ġand Ġverb Ġbase Ġwords , Ġconve ying Ġsimilar Ġnotions . ĠFor Ġexample , Ġin ĠTamil , Ġthere Ġare Ġat Ġleast Ġ30 , 000 Ġinf lection al Ġforms Ġof Ġany Ġgiven Ġverb Ġand Ġabout Ġ5 , 000 Ġforms Ġof Ġinf lection al Ġforms Ġfor Ġany Ġnoun . ĠThe Ġmerged Ġwords Ġcarry Ġinformation Ġabout Ġpart Ġof Ġspeech Ġ( POS ) Ġtags , Ġtense , Ġplurality Ġand Ġso Ġforth Ġthat Ġare Ġimportant Ġfor Ġanalyzing Ġtext Ġfor ĠMachine ĠTranslation Ġ( MT ). ĠNot Ġonly Ġare Ġthese Ġhidden Ġmeanings Ġnot Ġcaptured , Ġthe Ġcorresponding Ġroot Ġwords Ġare Ġtrained Ġas Ġdifferent Ġunits , Ġthereby Ġincreasing Ġthe Ġcomplexity Ġof Ġdeveloping Ġsuch ĠMT Ġsystems ĠB IB REF 1 Ġ. </s> To Ġadd Ġto Ġthe Ġcomplexities Ġof Ġbeing Ġa Ġmorph ologically Ġrich Ġlanguage , Ġthere Ġare Ġseveral Ġfactors Ġunique Ġto ĠTamil Ġthat Ġmake Ġtranslation Ġvery Ġdifficult . ĠThe Ġavailability Ġof Ġparallel Ġcorpus Ġfor ĠTamil Ġis Ġvery Ġscarce . ĠMost Ġof Ġthe Ġother Ġmodels Ġin Ġthe Ġfield Ġof ĠEnglish âĢĵ Tam il ĠMT Ġhave Ġmade Ġuse Ġof Ġtheir Ġown Ġtranslation Ġcorpor a Ġthat Ġwere Ġmanually Ġcreated Ġfor Ġthe Ġpurposes Ġof Ġresearch . ĠMost Ġof Ġthese Ġcorpor a Ġare Ġnot Ġavailable Ġonline Ġfor Ġuse . </s> Another Ġissue Ġspecific Ġto ĠTamil Ġis Ġthe Ġaddition Ġof Ġsuffix Ġcharacters Ġincluded Ġto Ġthe Ġwords Ġin Ġthe Ġlanguage Ġfor Ġsmooth ness Ġin Ġpronunciation . ĠThese Ġcharacters Ġare Ġof Ġso Ġmany Ġdifferent Ġtypes ; Ġthere Ġis Ġa Ġunique Ġsuffix Ġfor Ġeach Ġand Ġevery Ġconson ant Ġin Ġthe Ġlanguage . ĠThese Ġsuffix es Ġdegrade Ġperformance Ġof ĠMT Ġbecause Ġthe Ġsame Ġwords Ġwith Ġdifferent Ġsuch Ġpronoun ciation - based Ġsuffix es Ġwill Ġbe Ġtaken Ġas Ġdifferent Ġwords Ġin Ġtraining . </s> Also Ġto Ġtake Ġinto Ġconsideration Ġis Ġthe Ġexistence Ġof Ġtwo Ġdifferent Ġforms Ġof Ġthe Ġlanguage Ġbeing Ġused . ĠTrad itionally Ġdefined ĠTamil Ġand Ġits Ġpron unci ations Ġaren 't Ġac oust ically Ġpleasing Ġto Ġuse . ĠThere 's Ġno Ġlinguistic Ġflow Ġbetween Ġsyll ables Ġand Ġits Ġusage Ġin Ġverbal Ġcommunication Ġis Ġtime Ġconsuming . ĠTherefore , Ġthere Ġexists Ġtwo Ġforms Ġof Ġthe Ġlanguage , Ġthe Ġwritten Ġform , Ġrigid Ġin Ġstructure Ġand Ġsyntax , Ġand Ġthe Ġspoken Ġform , Ġin Ġwhich Ġthe Ġflow Ġand Ġpace Ġof Ġthe Ġlanguage Ġis Ġgiven Ġpriority Ġover Ġsyntax Ġand Ġcorrectness Ġof Ġspelling . ĠThis Ġdivide Ġleads Ġto Ġthe Ġcorpus Ġhaving Ġ2 Ġdifferent Ġversions Ġof Ġthe Ġlanguage Ġthat Ġincrease Ġthe Ġvocabulary Ġeven Ġwith Ġthe Ġsame Ġwords . ĠThis Ġcan Ġbe Ġevidently Ġseen Ġin Ġthe Ġcorpus Ġbetween Ġthe Ġsentences Ġused Ġin Ġthe ĠBible , Ġwhich Ġis Ġin Ġtraditional ĠTamil Ġand Ġsentences Ġfrom Ġmovie Ġsubtitles , Ġbeing Ġin Ġspoken ĠTamil Ġformat . </s> To Ġaccount Ġfor Ġsuch Ġdifficulties , Ġa Ġtrade - off Ġbetween Ġdomain Ġspecificity Ġand Ġsize Ġof Ġthe Ġcorpus Ġis Ġintegral Ġin Ġbuilding Ġan ĠEnglish âĢĵ Tam il Ġneural ĠMT Ġsystem . </s> Corp us </s> The Ġcorpus Ġselected Ġfor Ġthis Ġexperiment Ġwas Ġa Ġcombination Ġof Ġdifferent Ġcorpor a Ġfrom Ġvarious Ġdomains . ĠThe Ġmajor Ġpart Ġof Ġthe Ġcorpus Ġwas Ġmade Ġup Ġby Ġthe ĠEn Tam Ġv 2 Ġcorpus ĠB IB REF 2 Ġ. ĠThis Ġcorpus Ġcontained Ġsentences Ġtaken Ġfrom Ġparallel Ġnews Ġarticles , ĠEnglish Ġand ĠTamil Ġbible Ġcorpus Ġand Ġmovie Ġsubtitles . ĠIt Ġalso Ġcomprised Ġof Ġa Ġtourism Ġcorpus Ġthat Ġwas Ġobtained Ġfrom ĠTD IL Ġ( Technology ĠDevelopment Ġfor ĠIndian ĠLanguages ) Ġand Ġa Ġcorpus Ġcreated Ġfrom ĠTamil Ġnovels Ġand Ġshort Ġstories Ġfrom ĠAU - K BC , ĠAnna Ġuniversity . ĠThe Ġcomplete Ġcorpus Ġconsisted Ġof Ġ197 , 792 Ġsentences . ĠFig . ĠFIG REF 20 Ġshows Ġthe Ġskinny Ġshift Ġand Ġheat map Ġrepresentations Ġof Ġthe Ġrelativity Ġbetween Ġthe Ġsentences Ġin Ġterms Ġof Ġtheir Ġsentence Ġlengths . </s> An Ġextra Ġmon oling ual ĠTamil Ġcorpus , Ġcoll ated Ġfrom Ġvarious Ġonline Ġsources Ġwas Ġused Ġfor Ġthe Ġword 2 vec Ġembed ding Ġof Ġthe ĠTamil Ġtarget Ġlanguage Ġto Ġenhance Ġthe Ġrichness Ġof Ġcontext Ġof Ġthe Ġword Ġvectors . ĠIt Ġwas Ġalso Ġused Ġto Ġcreate Ġthe Ġlanguage Ġmodel Ġfor Ġthe Ġphrase - based ĠSM T Ġmodel . ĠThis Ġcorpus Ġcontained Ġ5 67 , 772 Ġsentences Ġand Ġwas Ġself - col lected Ġby Ġcombining Ġhundreds Ġof Ġancient ĠTamil Ġscriptures , Ġnovels Ġand Ġpoems Ġby Ġaccessing Ġthe Ġwebsites Ġof Ġpopular Ġonline Ġebook Ġlibraries Ġin ĠPython Ġusing Ġthe Ġur ll ib Ġpackage . ĠSince Ġthe Ġsources Ġhad ĠTamil Ġtext Ġin Ġdifferent Ġenc od ings , Ġthe Ġencoding Ġscheme Ġwas Ġstandardized Ġto Ġbe ĠUTF - 8 Ġfor Ġthe Ġentirety Ġof Ġthe Ġmon oling ual Ġand Ġparallel Ġcorpor a Ġusing Ġthe Ġch ard et Ġpackage . ĠThe Ġcorpor a Ġwere Ġcleaned Ġfor Ġany Ġstray Ġspecial Ġcharacters , Ġunnecessary Ġhtml Ġtags Ġand Ġwebsite ĠURLs . </s> Word 2 V ec </s> The Ġword Ġembed d ings Ġof Ġthe Ġsource Ġand Ġtarget Ġlanguage Ġsentences Ġare Ġused Ġas Ġinitial Ġvectors Ġof Ġthe Ġmodel Ġto Ġimprove Ġcontextual ization . ĠThe Ġskip Ġgram Ġmodel Ġof Ġthe Ġword 2 vec Ġalgorithm Ġoptim izes Ġthe Ġvectors Ġby Ġaccounting Ġfor Ġthe Ġaverage Ġlog Ġprobability Ġof Ġcontext Ġwords Ġgiven Ġa Ġsource Ġword . ĠDIS PLAY FORM 0 </s> where Ġk Ġis Ġthe Ġcontext Ġwindow Ġtaken Ġfor Ġthe Ġvector ization , ĠIN LINE FORM 0 Ġrefers Ġto Ġthe ĠIN LINE FORM 1 Ġword Ġof Ġthe Ġcorpus Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġsize Ġof Ġthe Ġtraining Ġcorpus Ġin Ġterms Ġof Ġthe Ġnumber Ġof Ġwords . ĠHere , Ġthe Ġprob ab ily ĠIN LINE FORM 3 Ġis Ġcomputed Ġas Ġa Ġhierarchical Ġsoft max Ġof Ġthe Ġproduct Ġof Ġthe Ġtrans pose Ġof Ġthe Ġoutput Ġvector Ġof ĠIN LINE FORM 4 Ġand Ġthe Ġinput Ġvector Ġof ĠIN LINE FORM 5 Ġfor Ġeach Ġand Ġevery Ġpair Ġover Ġthe Ġentire Ġvocabulary . ĠThe Ġprocesses Ġof Ġnegative Ġsampling Ġand Ġsubs am pling Ġof Ġfrequent Ġwords Ġthat Ġwere Ġused Ġin Ġthe Ġoriginal Ġmodel Ġaren 't Ġused Ġin Ġthis Ġexperiment ĠB IB REF 3 Ġ. </s> For Ġthe Ġprocess Ġof Ġcreating Ġsem antically Ġmeaningful Ġword Ġembed d ings , Ġa Ġmon oling ual Ġcorpus Ġof Ġ5 69 , 772 ĠTamil Ġsentences Ġwas Ġused . ĠThis Ġgave Ġthe Ġvectors Ġmore Ġcontextual Ġrichness Ġdue Ġto Ġthe Ġincreased Ġsize Ġof Ġthe Ġcorpus Ġas Ġopposed Ġto Ġusing Ġjust Ġthe Ġbilingual Ġcorpus ' Ġtarget Ġside Ġsentences ĠB IB REF 3 Ġ. </s> In Ġthe Ġexperiment , Ġthe Ġword 2 vec Ġmodel Ġwas Ġtrained Ġusing Ġa Ġvector Ġsize Ġof Ġ100 Ġto Ġensure Ġthat Ġthe Ġbulk Ġof Ġthe Ġlimited Ġmemory Ġof Ġthe ĠGPU Ġwill Ġbe Ġused Ġfor Ġthe Ġneural Ġattention Ġtranslation Ġmodel . ĠIt Ġhas Ġbeen Ġshown Ġthat Ġany Ġsize Ġover Ġthat Ġof Ġ150 Ġused Ġfor Ġword Ġvector ization Ġgives Ġsimilar Ġresults Ġand Ġthat Ġa Ġsize Ġof Ġ100 Ġperforms Ġclose Ġto Ġthe Ġmodel Ġwith Ġ150 - sized Ġword Ġvectors ĠB IB REF 7 Ġ. ĠA Ġstandard Ġsize Ġof Ġ5 Ġwas Ġused Ġas Ġwindow Ġsize Ġand Ġthe Ġmodel Ġwas Ġtrained Ġover Ġ7 Ġworker Ġthreads Ġsimultaneously . ĠA Ġbatch Ġsize Ġof Ġ50 Ġwords Ġwas Ġused Ġfor Ġtraining . ĠThe Ġnegative Ġsampling Ġwas Ġset Ġat Ġ1 Ġas Ġit Ġis Ġthe Ġnature Ġof Ġmorph ologically Ġrich Ġlanguages Ġto Ġhave Ġa Ġlot Ġof Ġimportant Ġwords Ġthat Ġdon 't Ġoccur Ġmore Ġthan Ġonce Ġin Ġthe Ġcorpus . ĠThe Ġg ens im Ġword 2 vec Ġtool kit Ġwas Ġused Ġto Ġimplement Ġthis Ġword Ġembed ding Ġprocess ĠB IB REF 8 Ġ. </s> Ne ural ĠTranslation ĠModel </s> The Ġmodel Ġused Ġfor Ġtranslation Ġis Ġthe Ġone Ġimplemented Ġby ĠBah dan au Ġet Ġal . ĠBah dan au 2014 . ĠA Ġbid irection al ĠL ST M Ġenc oder Ġfirst Ġtakes Ġthe Ġsource Ġsentence Ġand Ġenc odes Ġit Ġinto Ġa Ġcontext Ġvector Ġwhich Ġacts Ġas Ġinput Ġfor Ġthe Ġdec oder . ĠThe Ġdec oder Ġis Ġattention - based Ġwhere Ġthe Ġhidden Ġstates Ġof Ġthe Ġdec oder Ġget Ġas Ġinput Ġthe Ġweighted Ġsum Ġof Ġall Ġthe Ġhidden Ġlayer Ġoutputs Ġof Ġthe Ġenc oder Ġalong with Ġthe Ġoutput Ġof Ġthe Ġprevious Ġhidden Ġlayer Ġand Ġthe Ġpreviously Ġdec oded Ġword . ĠThis Ġprovides Ġa Ġcontextual Ġreference Ġinto Ġthe Ġsource Ġlanguage Ġsentence ĠB IB REF 4 Ġ. </s> Ne ural ĠMachine ĠTranslation Ġmodels Ġdirectly Ġcompute Ġthe Ġprobability Ġof Ġthe Ġtarget Ġlanguage Ġsentence Ġgiven Ġthe Ġsource Ġlanguage Ġsentence , Ġword Ġby Ġword Ġfor Ġevery Ġtime Ġstep . ĠThe Ġmodel Ġwith Ġa Ġbasic Ġdec oder Ġwithout Ġthe Ġattention Ġmodule Ġcomp utes Ġthe Ġlog Ġprobability Ġof Ġtarget Ġsentence Ġgiven Ġsource Ġsentence Ġas Ġthe Ġsum Ġof Ġlog Ġprobabilities Ġof Ġevery Ġword Ġgiven Ġevery Ġword Ġbefore Ġthat . ĠThe Ġattention - based Ġmodel , Ġon Ġthe Ġother Ġhand , Ġcalculates : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġwords Ġin Ġthe Ġtarget Ġsentence , ĠIN LINE FORM 1 Ġis Ġthe Ġtarget Ġsentence , ĠIN LINE FORM 2 Ġis Ġthe Ġsource Ġsentence , ĠIN LINE FORM 3 Ġis Ġthe Ġfixed Ġlength Ġoutput Ġvector Ġof Ġthe Ġenc oder Ġand ĠIN LINE FORM 4 Ġis Ġthe Ġweighted Ġsum Ġof Ġall Ġthe Ġhidden Ġlayer Ġoutputs Ġof Ġthe Ġenc oder Ġat Ġevery Ġtime Ġstep . ĠBoth Ġthe Ġenc oder 's Ġoutput Ġcontext Ġvector Ġand Ġthe Ġweighted Ġsum Ġ( known Ġas Ġattention Ġvector ) Ġhelp Ġto Ġimprove Ġthe Ġquality Ġof Ġtranslation Ġby Ġenabling Ġselective Ġsource Ġsentence Ġlookup . </s> The Ġdec oder ĠL ST M Ġcomp utes : ĠDIS PLAY FORM 0 </s> where Ġthe Ġprobability Ġis Ġcomputed Ġas Ġa Ġfunction Ġof Ġthe Ġdec oder 's Ġoutput Ġin Ġthe Ġprevious Ġtime Ġstep ĠIN LINE FORM 0 Ġ, Ġthe Ġhidden Ġlayer Ġvector Ġof Ġthe Ġdec oder Ġin Ġthe Ġcurrent Ġtim est ep ĠIN LINE FORM 1 Ġand Ġthe Ġcontext Ġvector Ġfrom Ġthe Ġattention Ġmechanism ĠIN LINE FORM 2 Ġ. ĠThe Ġcontext Ġvector ĠIN LINE FORM 3 Ġfor Ġtime Ġstep ĠIN LINE FORM 4 Ġis Ġcomputed Ġas Ġa Ġweighted Ġsum Ġof Ġthe Ġoutput Ġof Ġthe Ġentire Ġsentence Ġusing Ġa Ġweight Ġparameter ĠIN LINE FORM 5 Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġnumber Ġof Ġtokens Ġin Ġthe Ġsource Ġsentence , ĠIN LINE FORM 1 Ġrefers Ġto Ġthe Ġvalue Ġof Ġthe Ġhidden Ġlayer Ġof Ġthe Ġenc oder Ġat Ġtime Ġstep ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġalignment Ġparameter . ĠThis Ġparameter Ġis Ġcalculated Ġby Ġmeans Ġof Ġa Ġfeed Ġforward Ġneural Ġnetwork Ġto Ġensure Ġthat Ġthe Ġalignment Ġmodel Ġis Ġfree Ġfrom Ġthe Ġdifficulties Ġof Ġcontextual ization Ġof Ġlong Ġsentences Ġinto Ġa Ġsingle Ġvector . ĠThe Ġfeed Ġforward Ġnetwork Ġis Ġtrained Ġalong Ġwith Ġthe Ġneural Ġtranslation Ġmodel Ġto Ġjointly Ġimprove Ġthe Ġperformance Ġof Ġthe Ġtranslation . ĠMat hematically , ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġsoft max Ġoutput Ġof Ġthe Ġresult Ġof Ġthe Ġfeed forward Ġnetwork , ĠIN LINE FORM 1 Ġis Ġthe Ġhidden Ġstate Ġvalue Ġof Ġthe Ġdec oder Ġat Ġtim est ep ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġenc oder 's Ġhidden Ġlayer Ġannotation Ġat Ġtim est ep ĠIN LINE FORM 4 Ġ. ĠA Ġconc aten ation Ġof Ġthe Ġforward Ġand Ġthe Ġreverse Ġhidden Ġlayer Ġparameters Ġof Ġthe Ġenc oder Ġis Ġused Ġat Ġeach Ġstep Ġto Ġcompute Ġthe Ġweights ĠIN LINE FORM 5 Ġfor Ġthe Ġattention Ġmechanism . ĠThis Ġis Ġdone Ġto Ġenable Ġan Ġoverall Ġcontext Ġof Ġthe Ġsentence , Ġas Ġopposed Ġto Ġa Ġcontext Ġof Ġonly Ġall Ġthe Ġprevious Ġwords Ġof Ġthe Ġsentence Ġfor Ġevery Ġword Ġin Ġconsideration . ĠFig . ĠFIG REF 12 Ġis Ġthe Ġgeneral Ġarchitecture Ġof Ġthe Ġneural Ġtranslation Ġmodel Ġwithout Ġthe ĠBid irection al ĠL ST M Ġenc oder . </s> A Ġglobal Ġattention Ġmechanism Ġis Ġpreferred Ġover Ġlocal Ġattention Ġbecause Ġthe Ġdifferences Ġin Ġthe Ġstructures Ġof Ġthe Ġlanguages Ġcannot Ġbe Ġmapped Ġefficiently Ġto Ġenable Ġlookup Ġinto Ġthe Ġright Ġparts Ġof Ġthe Ġsource Ġsentence . ĠUsing Ġlocal Ġattention Ġmechanism Ġwith Ġa Ġmon ot onic Ġcontext Ġlookup , Ġwhere Ġthe Ġregion Ġaround ĠIN LINE FORM 0 Ġsource Ġword Ġis Ġlooked Ġup Ġfor Ġthe Ġprediction Ġof Ġthe ĠIN LINE FORM 1 Ġtarget Ġword , Ġis Ġimpractical Ġbecause Ġof Ġthe Ġstructural Ġdiscord ance Ġbetween Ġthe ĠEnglish Ġand ĠTamil Ġsentences Ġ( see ĠF igs . ĠFIG REF 37 Ġand ĠFIG REF 44 Ġ). ĠThe Ġuse Ġof Ġga ussian Ġand Ġother Ġsuch Ġdistributions Ġto Ġfacilitate Ġlocal Ġattention Ġwould Ġalso Ġbe Ġinefficient Ġbecause Ġthe Ġexistence Ġof Ġvarious Ġforms Ġof Ġtranslations Ġfor Ġthe Ġsame Ġsource Ġsentence Ġinvolving Ġmorph ological Ġand Ġstructural Ġvariations Ġthat Ġdon 't Ġstay Ġuniform Ġthrough Ġthe Ġentire Ġcorpus ĠB IB REF 5 Ġ. </s> The ĠNo ĠPe ep holes Ġ( NP ) Ġvariant Ġof Ġthe ĠL ST M Ġcell , Ġformulated Ġin ĠGre ff Ġet Ġal . Ġgre ff 2015 l st m Ġis Ġused Ġin Ġthis Ġexperiment Ġas Ġit Ġproved Ġto Ġgive Ġthe Ġbest Ġresults Ġamongst Ġall Ġthe Ġvariants Ġof Ġan ĠL ST M Ġcell . ĠIt Ġis Ġspecified Ġby Ġmeans Ġof Ġa Ġg ated Ġmechanism Ġdesigned Ġto Ġensure Ġthat Ġthe Ġvanishing Ġgradient Ġproblem Ġis Ġprevented . ĠL ST M Ġmaintains Ġits Ġhidden Ġlayer Ġin Ġtwo Ġcomponents , Ġthe Ġcell Ġvector ĠIN LINE FORM 0 Ġand Ġthe Ġactual Ġhidden Ġlayer Ġoutput Ġvector ĠIN LINE FORM 1 Ġ. ĠThe Ġcell Ġvector Ġis Ġensured Ġto Ġnever Ġreach Ġzero Ġby Ġmeans Ġof Ġa Ġweighted Ġsum Ġof Ġthe Ġprevious Ġlayer 's Ġcell Ġvector ĠIN LINE FORM 2 Ġregulated Ġby Ġthe Ġforget Ġgate ĠIN LINE FORM 3 Ġand Ġan Ġactivation Ġof Ġthe Ġweighted Ġsum Ġof Ġthe Ġinput ĠIN LINE FORM 4 Ġin Ġthe Ġcurrent Ġtim est ep ĠIN LINE FORM 5 Ġand Ġthe Ġprevious Ġtim est ep 's Ġhidden Ġlayer Ġoutput Ġvector ĠIN LINE FORM 6 Ġ. ĠThe Ġcombination Ġis Ġsimilarly Ġregulated Ġby Ġthe Ġinput Ġgate ĠIN LINE FORM 7 Ġ. ĠThe Ġhidden Ġlayer Ġoutput Ġis Ġdetermined Ġas Ġan Ġactivation Ġof Ġthe Ġcell Ġgate , Ġregulated Ġby Ġthe Ġoutput Ġgate ĠIN LINE FORM 8 Ġ. ĠThe Ġinter play Ġbetween Ġthese Ġtwo Ġvectors Ġ( ĠIN LINE FORM 9 Ġand ĠIN LINE FORM 10 Ġ) Ġat Ġevery Ġtim est ep Ġensures Ġthat Ġthe Ġproblem Ġof Ġvanishing Ġgrad ients Ġdoesn 't Ġoccur . ĠThe Ġthree Ġgates Ġare Ġalso Ġformed Ġas Ġa Ġs igm oid Ġof Ġthe Ġweighted Ġsum Ġof Ġthe Ġprevious Ġhidden Ġlayer Ġoutput ĠIN LINE FORM 11 Ġand Ġthe Ġinput Ġin Ġthe Ġcurrent Ġtim est ep ĠIN LINE FORM 12 Ġ. ĠThe Ġoutput Ġgenerated Ġout Ġof Ġthe ĠL ST M 's Ġhidden Ġlayer Ġis Ġspecified Ġas Ġa Ġweighted Ġsoft max Ġover Ġthe Ġhidden Ġlayer Ġoutput ĠIN LINE FORM 13 Ġ. ĠThe Ġlearn able Ġparameters Ġof Ġan ĠL ST M Ġcell Ġare Ġall Ġthe Ġweights ĠIN LINE FORM 14 Ġand Ġthe Ġbiases ĠIN LINE FORM 15 Ġ. ĠDIS PLAY FORM 0 </s> The ĠL ST M Ġspecified Ġby Ġequations Ġ7 Ġthrough Ġ11 Ġis Ġthe Ġone Ġused Ġfor Ġthe Ġdec oder Ġof Ġthe Ġmodel . ĠThe Ġenc oder Ġuses Ġa Ġbid irection al ĠR NN ĠL ST M Ġcell Ġin Ġwhich Ġthere Ġare Ġtwo Ġhidden Ġlayer Ġcomponents ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġthat Ġcontribute Ġto Ġthe Ġoutput ĠIN LINE FORM 2 Ġof Ġeach Ġtime Ġstep ĠIN LINE FORM 3 Ġ. ĠBoth Ġthe Ġcomponents Ġhave Ġtheir Ġown Ġsets Ġof ĠL ST M Ġequations Ġin Ġsuch Ġa Ġway Ġthat ĠIN LINE FORM 4 Ġfor Ġevery Ġtim est ep Ġis Ġcomputed Ġfrom Ġthe Ġfirst Ġtim est ep Ġtill Ġthe ĠIN LINE FORM 5 Ġtoken Ġis Ġreached Ġand ĠIN LINE FORM 6 Ġis Ġcomputed Ġfrom Ġthe ĠIN LINE FORM 7 Ġtim est ep Ġbackwards Ġuntil Ġthe Ġfirst Ġtoken Ġis Ġreached . ĠAll Ġthe Ġfive Ġvectors Ġof Ġthe Ġtwo Ġcomponents Ġare Ġall Ġexactly Ġthe Ġsame Ġas Ġthe ĠL ST M Ġequations Ġspecified Ġwith Ġone Ġvariation Ġin Ġthe Ġcomputation Ġof Ġthe Ġresult . ĠDIS PLAY FORM 0 </s> M orph ological ĠSe gment ation </s> The Ġmorph ological Ġsegment ation Ġused Ġis Ġa Ġsemi - super vised Ġextension Ġto Ġthe Ġgener ative Ġprob abil istic Ġmodel Ġof Ġmaximizing Ġthe Ġprobability Ġof Ġa ĠIN LINE FORM 0 Ġprefix , root , post fix ĠIN LINE FORM 1 Ġrecursive Ġsplit Ġup Ġof Ġwords Ġbased Ġon Ġan Ġexhaustive Ġcombination Ġof Ġall Ġpossible Ġmorp hem es . ĠThe Ġdetails Ġof Ġthis Ġmodel Ġare Ġspecified Ġand Ġextensively Ġstudied Ġin ĠKoh onen Ġet Ġal . Ġk oh onen 2010 se mi . ĠThe Ġmodel Ġparameters ĠIN LINE FORM 2 Ġinclude Ġthe Ġmorph Ġtype Ġcount , Ġmorph Ġtoken Ġcount Ġof Ġtraining Ġdata , Ġthe Ġmorph Ġstrings Ġand Ġtheir Ġcounts . ĠThe Ġmodel Ġis Ġtrained Ġby Ġmaximizing Ġthe ĠMaximum ĠA ĠPoster ior i Ġ( MAP ) Ġprobability Ġusing ĠBay es ' Ġrule : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrefers Ġto Ġevery Ġword Ġin Ġthe Ġtraining Ġlex icon . ĠThe Ġprior ĠIN LINE FORM 1 Ġis Ġestimated Ġusing Ġthe ĠMinimum ĠDescription ĠLength ( MD L ) Ġprinciple . ĠThe Ġlikelihood ĠIN LINE FORM 2 Ġis Ġestimated Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrefers Ġto Ġthe Ġintermediate Ġanalyses Ġand ĠIN LINE FORM 1 Ġrefers Ġto Ġthe ĠIN LINE FORM 2 Ġmor phe me Ġof Ġword ĠIN LINE FORM 3 Ġ. </s> An Ġextension Ġto Ġthe ĠV iter bi Ġalgorithm Ġis Ġused Ġfor Ġthe Ġdecoding Ġstep Ġbased Ġon Ġexhaustive Ġmapping Ġof Ġmorp hem es . ĠTo Ġaccount Ġfor Ġover - se gment ation Ġand Ġunder - se gment ation Ġissues Ġassociated Ġwith Ġun super vised Ġmorph ological Ġsegment ation , Ġextra Ġparameters Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġ( ĠIN LINE FORM 1 Ġ) Ġare Ġused Ġwith Ġthe Ġcost Ġfunction ĠIN LINE FORM 2 ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġlikelihood Ġof Ġthe Ġcost Ġfunction , ĠIN LINE FORM 1 Ġdescribes Ġthe Ġlikelihood Ġof Ġcontribution Ġof Ġthe Ġannot ated Ġdataset Ġto Ġthe Ġcost Ġfunction Ġand ĠIN LINE FORM 2 Ġis Ġthe Ġlikelihood Ġof Ġthe Ġlabeled Ġdata . ĠA Ġdecrease Ġin Ġthe Ġvalue Ġof ĠIN LINE FORM 3 Ġwill Ġcause Ġsmaller Ġsegments Ġand Ġvice Ġversa . ĠIN LINE FORM 4 Ġtakes Ġcare Ġof Ġsize Ġdiscrepancies Ġdue Ġto Ġreduced Ġavailability Ġof Ġannot ated Ġcorpus Ġas Ġcompared Ġto Ġthe Ġtraining Ġcorpus ĠB IB REF 2 Ġ, ĠB IB REF 6 Ġ. </s> The ĠPython Ġextension Ġto Ġthe Ġmorph ological Ġsegment ation Ġtool Ġmor f essor Ġ2 . 0 Ġwas Ġused Ġfor Ġthis Ġexperiment Ġto Ġperform Ġthe Ġsegment ation . ĠThe Ġannotation Ġdata Ġfor ĠTamil Ġlanguage Ġcoll ated Ġand Ġreleased Ġby ĠAn oop ĠK unch uk k utan Ġin Ġthe ĠInd ic ĠN LP ĠLibrary Ġwas Ġused Ġas Ġthe Ġsemi - super vised Ġinput Ġto Ġthe Ġmodel ĠB IB REF 9 Ġ, ĠB IB REF 6 Ġ. </s> Exper iment </s> The Ġcomplexities Ġof Ġneural Ġmachine Ġtranslation Ġof Ġmorph ologically Ġrich Ġlanguages Ġwere Ġstudied Ġwith Ġrespect Ġto ĠEnglish Ġto ĠTamil Ġmachine Ġtranslation Ġusing Ġthe ĠR NN ĠL ST M ĠBi - direction al Ġenc oder Ġattention Ġdec oder Ġarchitecture . ĠTo Ġcompare Ġwith Ġa Ġbaseline Ġsystem , Ġa Ġphrase Ġbased ĠSM T Ġsystem Ġwas Ġimplemented Ġusing Ġthe Ġsame Ġcorpus . ĠThe ĠFact ored ĠSM T Ġmodel Ġwith Ġsource - side Ġpre processing Ġby ĠKumar Ġet Ġal . Ġk umar 2014 impro ving Ġwas Ġused Ġas Ġa Ġreference Ġfor Ġthe Ġtranslation Ġbetween Ġthese Ġlanguage Ġpairs . ĠAlso , Ġan Ġadditional Ġ5 69 , 772 Ġmon oling ual ĠTamil Ġsentences Ġwere Ġused Ġfor Ġthe Ġlanguage Ġmodel Ġof Ġthe ĠSM T Ġsystem . ĠThe Ġmodel Ġused Ġcould Ġbe Ġsplit Ġup Ġinto Ġvarious Ġmodules Ġas Ġexpanded Ġin ĠFig . ĠFIG REF 17 Ġ. </s> B uck eting </s> The Ġinput Ġsource Ġand Ġtarget Ġlanguage Ġsentences Ġused Ġfor Ġtraining Ġwere Ġtaken Ġand Ġdivided Ġinto Ġbucket ed Ġpairs Ġof Ġsentences Ġof Ġa Ġfixed Ġnumber Ġof Ġsizes . ĠThis Ġrelationship Ġwas Ġdetermined Ġby Ġexamining Ġthe Ġdistribution Ġof Ġwords Ġin Ġthe Ġcorpus Ġprimarily Ġto Ġminimize Ġthe Ġnumber Ġof ĠP AD Ġtokens Ġin Ġthe Ġsentence . ĠThe Ġheat Ġmap Ġof Ġthe Ġnumber Ġof Ġwords Ġin Ġthe ĠEnglish âĢĵ Tam il Ġsentence Ġpairs Ġof Ġthe Ġcorpus Ġrevealed Ġthat Ġthe Ġdistribution Ġis Ġcentered Ġaround Ġthe Ġ10 âĢĵ 20 Ġwords Ġregion . ĠTherefore , Ġmore Ġbuckets Ġin Ġthat Ġregion Ġwere Ġapplied Ġas Ġthere Ġwould Ġbe Ġenough Ġnumber Ġof Ġexamples Ġin Ġeach Ġof Ġthese Ġbucket Ġpairs Ġfor Ġthe Ġmodel Ġto Ġlearn Ġabout Ġthe Ġsentences Ġin Ġeach Ġand Ġevery Ġbucket . ĠThe Ġexact Ġscheme Ġused Ġfor Ġthe ĠR NN Search Ġmodels Ġis Ġspecified Ġby ĠFig . ĠFIG REF 21 Ġ. ĠThe Ġbuck eting Ġscheme Ġfor Ġthe ĠR NN M orph Ġmodel , Ġinvolving Ġmorph s Ġinstead Ġof Ġwords , Ġwas Ġa Ġsimple Ġshifted Ġscheme Ġof Ġthe Ġone Ġused Ġin ĠFig . ĠFIG REF 21 Ġ, Ġwhere Ġevery Ġtarget Ġsentence Ġbucket Ġcount Ġwas Ġincreased Ġuniformly Ġby Ġ5 . </s> Model ĠDetails </s> Due Ġto Ġvarious Ġcomputational Ġconstraints Ġand Ġlack Ġof Ġavailability Ġof Ġcomprehensive Ġcorpor a , Ġthe Ġvoc ab ular ies Ġfor ĠEnglish Ġand ĠTamil Ġlanguages Ġfor Ġthe ĠR NN Search Ġmodel Ġwere Ġrestricted Ġto Ġ60 , 000 Ġout Ġof Ġ67 , 768 Ġand Ġ150 , 000 Ġout Ġof Ġ340 , 325 Ġrespectively . ĠThe Ġvocabulary Ġof Ġthe Ġlanguages Ġfor Ġthe ĠR NN M orph Ġdidn 't Ġhave Ġto Ġbe Ġrestricted Ġand Ġthe Ġactual Ġnumber Ġof Ġwords Ġin Ġthe Ġcorpus Ġi . e . Ġ67 , 768 Ġwords Ġfor ĠEnglish Ġand Ġ41 , 9 06 Ġwords Ġfor ĠTamil Ġcould Ġbe Ġaccommod ated Ġinto Ġthe Ġtraining . ĠWords Ġnot Ġin Ġthe Ġvocabulary Ġfrom Ġthe Ġtest Ġset Ġinput Ġand Ġoutput Ġwere Ġreplaced Ġwith Ġthe Ġuniversal ĠIN LINE FORM 0 ĠUN K ĠIN LINE FORM 1 Ġtoken , Ġsymbol izing Ġan Ġunknown Ġword . ĠThe ĠL ST M Ġhidden Ġlayer Ġsize , Ġthe Ġtraining Ġbatch Ġsize , Ġand Ġthe Ġvocabulary Ġsizes Ġof Ġthe Ġlanguages , Ġtogether , Ġacted Ġas Ġa Ġbottleneck . ĠThe Ġmodel Ġwas Ġrun Ġon Ġa Ġ2 GB ĠNVIDIA ĠGeForce ĠGT Ġ650 M Ġcard Ġwith Ġ384 Ġcores Ġand Ġthe Ġmemory Ġallot ment Ġwas Ġconstrained Ġto Ġthe Ġlimits Ġof Ġthe ĠGPU . ĠTherefore , Ġafter Ġrepeated Ġexperimentation , Ġit Ġwas Ġdetermined Ġthat Ġwith Ġa Ġbatch Ġsize Ġof Ġ16 , Ġthe Ġmaximum Ġhidden Ġlayer Ġsize Ġpossible Ġwas Ġ500 , Ġwhich Ġwas Ġthe Ġsize Ġused . ĠAttempt s Ġto Ġreduce Ġthe Ġbatch Ġsize Ġresulted Ġin Ġpoor Ġconvergence , Ġand Ġso Ġthe Ġparameters Ġwere Ġset Ġto Ġcenter Ġaround Ġthe Ġbatch Ġsize Ġof Ġ16 . ĠThe Ġmodels Ġused Ġwere Ġof Ġ4 Ġlayers Ġof ĠL ST M Ġhidden Ġunits Ġin Ġthe Ġbid irection al Ġenc oder Ġand Ġattention Ġdec oder . </s> The Ġmodel Ġused Ġa ĠSt och astic ĠGrad ient ĠDes cent Ġ( SG D ) Ġoptimization Ġalgorithm Ġwith Ġa Ġsampled Ġsoft max Ġloss Ġof Ġ512 Ġper Ġsample Ġto Ġhandle Ġlarge Ġvocabulary Ġsize Ġof Ġthe Ġtarget Ġlanguage ĠB IB REF 10 Ġ. ĠThe Ġmodel Ġwas Ġtrained Ġwith Ġa Ġlearning Ġrate Ġ1 . 0 Ġand Ġa Ġdecay Ġof Ġrate Ġ0 . 5 Ġenforced Ġmanually . ĠGrad ient Ġclipping Ġbased Ġon Ġthe Ġglobal Ġnorm Ġof Ġ5 . 0 Ġwas Ġcarried Ġout Ġto Ġprevent Ġgrad ients Ġexploding Ġand Ġgoing Ġto Ġunre cover able Ġvalues Ġtending Ġtowards Ġinfinity . ĠThe Ġmodel Ġdescribed Ġis Ġthe Ġone Ġused Ġin Ġthe ĠT ensor flow ĠB IB REF 11 Ġseq 2 seq Ġlibrary . </s> Results Ġand ĠDiscussion </s> The ĠB LE U Ġmetric Ġparameters Ġ( modified Ġ1 - gram , Ġ2 - gram , Ġ3 - gram Ġand Ġ4 - gram Ġprecision Ġvalues ) Ġand Ġhuman Ġevaluation Ġmetrics Ġof Ġadequ acy , Ġflu ency Ġand Ġrelative Ġranking Ġvalues Ġwere Ġused Ġto Ġevaluate Ġthe Ġperformance Ġof Ġthe Ġmodels . </s> BLE U ĠEvaluation </s> The ĠB LE U Ġscores Ġobtained Ġusing Ġthe Ġvarious Ġmodels Ġused Ġin Ġthe Ġexperiment Ġare Ġtab ulated Ġin ĠTable ĠT AB REF 25 Ġ. </s> The ĠB LE U Ġmetric Ġcomp utes Ġthe ĠB LE U Ġun ig ram , Ġbig ram , Ġtrig ram Ġand ĠB LE U - 4 Ġmodified Ġprecision Ġvalues , Ġeach Ġmicro - aver aged Ġover Ġthe Ġtest Ġset Ġsentences ĠB IB REF 7 Ġ. ĠIt Ġwas Ġobserved , Ġas Ġexpected , Ġthat Ġthe Ġperformance Ġof Ġthe Ġphrase - based ĠSM T Ġmodel Ġwas Ġinferior Ġto Ġthat Ġof Ġthe ĠR NN Search Ġmodel . ĠThe Ġbaseline ĠR NN Search Ġsystem Ġwas Ġfurther Ġrefined Ġby Ġusing Ġword 2 vec Ġvectors Ġto Ġembed Ġsemantic Ġunderstanding , Ġas Ġobserved Ġwith Ġthe Ġslight Ġincrease Ġin Ġthe ĠB LE U Ġscores . ĠFig . ĠFIG REF 26 Ġplots Ġthe ĠB LE U Ġscores Ġas Ġa Ġline Ġgraph Ġfor Ġvisualization Ġof Ġthe Ġimprovement Ġin Ġperformance . ĠAlso , Ġthe Ġ4 - gram ĠB LE U Ġscores Ġfor Ġthe Ġvarious Ġmodels Ġwere Ġplotted Ġas Ġa Ġbar Ġgraph Ġin ĠFig . ĠFIG REF 26 </s> Due Ġto Ġthe Ġagg l ut in ative Ġand Ġmorph ologically Ġrich Ġnature Ġof Ġthe Ġtarget Ġlanguage Ġi . e . ĠTamil , Ġthe Ġuse Ġof Ġmorph ological Ġsegment ation Ġto Ġsplit Ġthe Ġwords Ġinto Ġmorp hem es Ġfurther Ġimproved Ġthe ĠB LE U Ġprecision Ġvalues Ġin Ġthe ĠR NN M orph Ġmodel . ĠOne Ġof Ġthe Ġreasons Ġfor Ġthe Ġlarge Ġextent Ġof Ġincrease Ġin Ġthe ĠB LE U Ġscore Ġcould Ġbe Ġattributed Ġto Ġthe Ġoverall Ġincrease Ġin Ġthe Ġnumber Ġof Ġword Ġunits Ġper Ġsentence . ĠSince Ġthe ĠB LE U Ġscore Ġcomp utes Ġmicro - average Ġprecision Ġscores , Ġan Ġincrease Ġin Ġboth Ġthe Ġnumer ator Ġand Ġdenomin ator Ġof Ġthe Ġprecision Ġscores Ġis Ġapparent Ġwith Ġan Ġincrease Ġin Ġthe Ġnumber Ġof Ġtokens Ġdue Ġto Ġmorph ological Ġsegment ation Ġof Ġthe Ġtarget Ġlanguage . ĠThus , Ġthe Ġnumeric Ġextent Ġof Ġthe Ġincrease Ġof Ġaccuracy Ġmight Ġnot Ġefficiently Ġdescribe Ġthe Ġimprovement Ġin Ġperformance Ġof Ġthe Ġtranslation . </s> Human ĠEvaluation </s> To Ġensure Ġthat Ġthe Ġincrease Ġin ĠB LE U Ġscore Ġcorrelated Ġto Ġactual Ġincrease Ġin Ġperformance Ġof Ġtranslation , Ġhuman Ġevaluation Ġmetrics Ġlike Ġadequ acy , Ġprecision Ġand Ġranking Ġvalues Ġ( between ĠR NN Search Ġand ĠR NN M orph Ġoutputs ) Ġwere Ġestimated Ġin ĠTable ĠT AB REF 30 Ġ. ĠA Ġgroup Ġof Ġ50 Ġnative Ġpeople Ġwho Ġwere Ġwell - vers ed Ġin Ġboth ĠEnglish Ġand ĠTamil Ġlanguages Ġacted Ġas Ġannot ators Ġfor Ġthe Ġevaluation . ĠA Ġcollection Ġof Ġsamples Ġof Ġabout Ġ100 Ġsentences Ġwere Ġtaken Ġfrom Ġthe Ġtest Ġset Ġresults Ġfor Ġcomparison . ĠThis Ġset Ġincluded Ġa Ġrandomized Ġselection Ġof Ġthe Ġtranslation Ġresults Ġto Ġensure Ġthe Ġobject ivity Ġof Ġevaluation . ĠFlu ency Ġand Ġadequ acy Ġresults Ġfor Ġthe ĠR NN M orph Ġresults Ġare Ġtab ulated . ĠAd equ acy Ġrating Ġwas Ġcalculated Ġon Ġa Ġ5 - point Ġscale Ġof Ġhow Ġmuch Ġof Ġthe Ġmeaning Ġis Ġconveyed Ġby Ġthe Ġtranslation Ġ( All , ĠMost , ĠMuch , ĠLittle , ĠNone ). ĠThe Ġflu ency Ġrating Ġwas Ġcalculated Ġbased Ġon Ġgram matical Ġcorrectness Ġon Ġa Ġ5 - point Ġscale Ġof Ġ( F law less , ĠGood , ĠNon - native , ĠDis f luent , ĠIn com pre hens ive ). ĠFor Ġthe Ġcomparison Ġprocess , Ġthe ĠR NN M orph Ġand Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġmodels âĢ Ļ Ġsentence Ġlevel Ġtranslations Ġwere Ġindividually Ġranked Ġbetween Ġeach Ġother , Ġpermitting Ġthe Ġtwo Ġtranslations Ġto Ġhave Ġties Ġin Ġthe Ġranking . ĠThe Ġintra - annot ator Ġvalues Ġwere Ġcomputed Ġfor Ġthese Ġmetrics Ġand Ġthe Ġscores Ġare Ġshown Ġin ĠTable ĠT AB REF 32 ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> The Ġhuman Ġevaluation ĠKappa Ġco - efficient Ġresults Ġare Ġcalculated Ġwith Ġrespect Ġto : ĠDIS PLAY FORM 0 </s> It Ġwas Ġobserved Ġthat Ġthe Ġranking ĠKappa Ġco - efficient Ġfor Ġintra - annot ator Ġranking Ġof Ġthe ĠR NN M orph Ġmodel Ġwas Ġat Ġ0 . 573 , Ġhigher Ġthat Ġthe Ġ0 . 410 Ġof Ġthe ĠR NN Search + Word 2 V ec Ġmodel , Ġimplying Ġthat Ġthe Ġannot ators Ġfound Ġthe ĠR NN M orph Ġmodel Ġto Ġproduce Ġbetter Ġresults Ġwhen Ġcompared Ġto Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġmodel . </s> Model ĠParameters </s> The Ġlearning Ġrate Ġdecay Ġthrough Ġthe Ġtraining Ġprocess Ġof Ġthe ĠR NN M orph Ġmodel Ġis Ġshowcased Ġin Ġthe Ġgraph Ġin ĠFig . ĠFIG REF 34 Ġ. ĠThis Ġprocess Ġwas Ġdone Ġmanually Ġwhere Ġthe Ġlearning Ġrate Ġwas Ġdec ayed Ġafter Ġthe Ġend Ġof Ġspecific Ġepoch s Ġbased Ġon Ġan Ġobserved Ġstagnation Ġin Ġperplex ity . The ĠR NN M orph Ġmodel Ġachieved Ġsaturation Ġof Ġperplex ities Ġmuch Ġearlier Ġthrough Ġthe Ġepoch s Ġthan Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġmodel . ĠThis Ġconform s Ġto Ġthe Ġexpected Ġoutcome Ġas Ġthe Ġmorph ological Ġsegment ation Ġhas Ġreduced Ġthe Ġvocabulary Ġsize Ġof Ġthe Ġtarget Ġlanguage Ġfrom Ġ340 , 325 Ġwords Ġto Ġa Ġmere Ġ41 , 9 06 Ġmorph s . </s> The Ġerror Ġfunction Ġused Ġwas Ġthe Ġsampled ĠSoft Max Ġloss Ġto Ġensure Ġa Ġlarge Ġtarget Ġvocabulary Ġcould Ġbe Ġaccommod ated ĠB IB REF 10 Ġ. ĠA Ġzoom ed Ġins et Ġgraph Ġ( Fig . ĠFIG REF 35 Ġ) Ġhas Ġbeen Ġused Ġto Ġvisualize Ġthe Ġvalues Ġof Ġthe Ġerror Ġfunction Ġfor Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġand ĠR NN M orph Ġmodels Ġwith Ġ4 Ġhidden Ġlayers . ĠIt Ġcan Ġbe Ġseen Ġthat Ġthe ĠR NN M orph Ġmodel Ġis Ġconsistently Ġbetter Ġin Ġterms Ġof Ġthe Ġperplex ity Ġvalues Ġthrough Ġthe Ġtime Ġsteps . </s> Att ention ĠV ect ors </s> In Ġorder Ġto Ġfurther Ġdemonstrate Ġthe Ġquality Ġof Ġthe ĠR NN M orph Ġmodel , Ġthe Ġattention Ġvectors Ġof Ġboth Ġthe ĠR NN Search Ġwith ĠWord 2 V ec Ġembed ding Ġand ĠR NN M orph Ġmodels Ġare Ġcompared Ġfor Ġseveral Ġgood Ġtranslations Ġin ĠF igs . ĠFIG REF 37 Ġand ĠFIG REF 44 Ġ. ĠIt Ġis Ġobserved Ġthat Ġthe Ġreduction Ġin Ġvocabulary Ġsize Ġhas Ġimproved Ġthe Ġsource Ġsentence Ġlookup Ġby Ġquite Ġan Ġextent . ĠEach Ġcell Ġin Ġthe Ġheat map Ġdisplays Ġthe Ġmagnitude Ġof Ġthe Ġattention Ġlayer Ġweight ĠIN LINE FORM 0 Ġfor Ġthe ĠIN LINE FORM 1 ĠTamil Ġword Ġand Ġthe ĠIN LINE FORM 2 ĠEnglish Ġword Ġin Ġthe Ġrespective Ġsentences . ĠThe Ġintensity Ġof Ġblack Ġcorresponds Ġto Ġthe Ġmagnitude Ġof Ġthe Ġcell ĠIN LINE FORM 3 Ġ. ĠAlso , Ġthe Ġattention Ġvectors Ġof Ġthe ĠR NN Search Ġmodel Ġwith ĠWord 2 V ec Ġembed d ings Ġtend Ġto Ġattend Ġto ĠIN LINE FORM 4 ĠE OS ĠIN LINE FORM 5 Ġtoken Ġin Ġthe Ġmiddle Ġof Ġthe Ġsentence Ġleading Ġto Ġincomplete Ġtranslations . ĠThis Ġcould Ġbe Ġdue Ġto Ġthe Ġfact Ġthat Ġonly Ġ44 % Ġof Ġthe ĠTamil Ġvocabulary Ġand Ġ74 % Ġof Ġthe ĠEnglish Ġvocabulary Ġis Ġtaken Ġfor Ġtraining Ġin Ġthis Ġmodel , Ġas Ġopposed Ġto Ġ100 % Ġof ĠEnglish Ġand ĠTamil Ġwords Ġin Ġthe ĠR NN M orph Ġmodel . </s> Target Ġvocabulary Ġsize </s> A Ġvery Ġlarge Ġtarget Ġvocabulary Ġis Ġan Ġinadvert ent Ġconsequence Ġof Ġthe Ġmorph ological Ġrichness Ġof Ġthe ĠTamil Ġlanguage . ĠThis Ġcreates Ġa Ġpotential Ġrestriction Ġon Ġthe Ġaccuracy Ġof Ġthe Ġmodel Ġas Ġmany Ġinf lection al Ġforms Ġof Ġthe Ġsame Ġword Ġare Ġtrained Ġas Ġindependent Ġunits . ĠOne Ġof Ġthe Ġadvantages Ġof Ġmorph ological Ġsegment ation Ġof ĠTamil Ġtext Ġis Ġthat Ġthe Ġtarget Ġvocabulary Ġsize Ġdecreased Ġfrom Ġ340 , 325 Ġto Ġa Ġmere Ġ41 , 9 06 . ĠThis Ġreduction Ġhelps Ġimprove Ġthe Ġperformance Ġof Ġthe Ġtranslation Ġas Ġthe Ġoccurrence Ġof Ġunknown Ġtokens Ġwas Ġreduced Ġcompared Ġto Ġthe ĠR NN Search Ġmodel . ĠThis Ġmorph ologically Ġsegment ed Ġvocabulary Ġis Ġdivided Ġinto Ġa Ġcollection Ġof Ġmorph ological Ġroots Ġand Ġinf lections Ġas Ġindividual Ġunits . </s> Rep et itions </s> Some Ġof Ġthe Ġtranslations Ġof Ġthe ĠR NN M orph Ġmodel Ġhave Ġrepet itions Ġof Ġthe Ġsame Ġphrases Ġ( Fig . ĠFIG REF 53 Ġ), Ġwhereas Ġsuch Ġrepet itions Ġoccur Ġmuch Ġless Ġfrequently Ġin Ġthe ĠR NN Search Ġpredictions . ĠSuch Ġtranslations Ġwould Ġmake Ġfor Ġgood Ġresults Ġif Ġthe Ġrepet itions Ġweren 't Ġpresent Ġand Ġall Ġparts Ġof Ġthe Ġsentence Ġoccur Ġjust Ġonce . ĠThese Ġrepet itions Ġmight Ġbe Ġdue Ġto Ġthe Ġincrease Ġin Ġthe Ġgeneral Ġsequence Ġlength Ġof Ġthe Ġtarget Ġsentences Ġbecause Ġof Ġthe Ġmorph ological Ġsegment ation . ĠWhile Ġit Ġis Ġtrue Ġthe Ġtarget Ġvocabulary Ġsize Ġhas Ġdecreased Ġdue Ġto Ġmorph ological Ġsegment ation , Ġthe ĠR NN M orph Ġhas Ġmore Ġinput Ġunits Ġ( morph s ) Ġper Ġsentence , Ġwhich Ġmakes Ġit Ġmore Ġdemanding Ġof Ġthe ĠL ST M 's Ġmemory Ġunits Ġand Ġthe Ġfeed Ġforward Ġnetwork Ġof Ġthe Ġattention Ġmodel . ĠAdditionally , Ġthis Ġbehavior Ġcould Ġalso Ġbe Ġattributed Ġto Ġthe Ġerrors Ġin Ġthe Ġsemi - super vised Ġmorph ological Ġsegment ation Ġdue Ġto Ġthe Ġcomplexities Ġof Ġthe ĠTamil Ġlanguage Ġand Ġthe Ġextent Ġof Ġthe Ġcorpus . </s> Model ĠOutput s </s> The Ġtranslation Ġoutputs Ġof Ġthe ĠR NN Search Ġ+ ĠWord 2 V ec Ġand ĠMorph 2 V ec Ġmodels Ġfor Ġthe Ġsame Ġinput Ġsentences Ġfrom Ġthe Ġtest Ġset Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġusing Ġa Ġmorph ological Ġsegment ation Ġtool Ġand Ġhow Ġthe Ġmorp hem es Ġhave Ġchanged Ġthe Ġsentence Ġto Ġbe Ġmore Ġgram matically Ġsound . ĠIt Ġis Ġalso Ġobserved Ġ( from ĠFig . ĠFIG REF 55 Ġ) Ġthat Ġmost Ġof Ġthe Ġtranslation Ġsentences Ġof Ġthe ĠMorph 2 V ec Ġmodel Ġhave Ġno ĠIN LINE FORM 0 ĠUN K ĠIN LINE FORM 1 Ġtokens . ĠThey Ġexist Ġin Ġthe Ġpredictions Ġmostly Ġonly Ġdue Ġto Ġa Ġword Ġin Ġthe ĠEnglish Ġtest Ġsentence Ġnot Ġpresent Ġin Ġthe Ġsource Ġvocabulary . </s> Related ĠWork </s> Prof essors ĠCN ĠKrish nan , ĠSob ha Ġet Ġal Ġdeveloped Ġa Ġmachine - a ided - translation Ġ( MAT ) Ġsystem Ġsimilar Ġto Ġthe ĠAn usa ak ara ĠEnglish ĠHindi ĠMT Ġsystem , Ġusing Ġa Ġsmall Ġcorpus Ġand Ġvery Ġfew Ġtransfer Ġrules , Ġavailable Ġat ĠAU - K BC Ġwebsite ĠB IB REF 14 Ġ. ĠBal aj ap ally Ġet Ġal . Ġbal aj ap ally 2006 mult ilingual Ġdeveloped Ġan Ġexample Ġbased Ġmachine Ġtranslation Ġ( E BM T ) Ġsystem Ġwith Ġ700 000 Ġsentences Ġfor ĠEnglish Ġto ĠIN LINE FORM 0 ĠTamil , ĠK ann ada , ĠHindi ĠIN LINE FORM 1 Ġtransl iter ated Ġtext ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠRen gan athan Ġre ng an athan 2002 inter active Ġdeveloped Ġa Ġrule Ġbased ĠMT Ġsystem Ġfor ĠEnglish Ġand ĠTamil Ġusing Ġgrammar Ġrules Ġfor Ġthe Ġlanguage Ġpair . ĠVet ri vel Ġet Ġal . Ġvet ri vel 2010 english Ġused ĠHM Ms Ġto Ġalign Ġand Ġtranslate ĠEnglish Ġand ĠTamil Ġparallel Ġsentences Ġto Ġbuild Ġan ĠSM T Ġsystem . ĠIrvine Ġet Ġal . Ġir vine 2013 comb ining Ġtried Ġto Ġcombine Ġparallel Ġand Ġsimilar Ġcorpor a Ġto Ġimprove Ġthe Ġperformance Ġof ĠEnglish Ġto ĠTamil ĠSM T Ġamongst Ġother Ġlanguages . ĠKas th uri Ġet Ġal . Ġk as th uri 2014 rule Ġused Ġa Ġrule Ġbased ĠMT Ġsystem Ġusing Ġtransfer Ġlex icon Ġand Ġmorph ological Ġanalysis Ġtools . ĠAng lab har athi Ġwas Ġdeveloped Ġat ĠI IT ĠKan pur , Ġa Ġsystem Ġtranslating ĠEnglish Ġto Ġa Ġcollection Ġof ĠIndian Ġlanguages Ġincluding ĠTamil Ġusing ĠCF G Ġlike Ġstructures Ġto Ġcreate Ġa Ġpseudo Ġtarget Ġto Ġconvert Ġto ĠIndian Ġlanguages ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠA Ġvariety Ġof Ġhybrid Ġapproaches Ġhave Ġalso Ġbeen Ġused Ġfor ĠEnglish âĢĵ Tam il ĠMT Ġin Ġcombinations Ġof Ġrule Ġbased Ġ( transfer Ġmethods ), Ġinter ling ua Ġrepresentations ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. ĠThe Ġuse Ġof ĠStatistical ĠMachine ĠTranslation Ġtook Ġover Ġthe ĠEnglish âĢĵ Tam il ĠMT Ġsystem Ġresearch Ġbecause Ġof Ġits Ġdesirable Ġproperties Ġof Ġlanguage Ġindependence , Ġbetter Ġgeneral ization Ġfeatures Ġand Ġa Ġreduced Ġrequirement Ġof Ġlinguistic Ġexpertise ĠB IB REF 1 Ġ, ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ. ĠVarious Ġenhancement Ġtechniques Ġexternal Ġto Ġthe ĠMT Ġsystem Ġhave Ġalso Ġbeen Ġproposed Ġto Ġimprove Ġthe Ġperformance Ġof Ġtranslation Ġusing Ġmorph ological Ġpre Ġand Ġpost Ġprocessing Ġtechniques ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ, ĠB IB REF 26 Ġ. </s> The Ġuse Ġof ĠR NN ĠEnc oder ĠDecoder Ġmodels Ġin Ġmachine Ġtranslation Ġhas Ġshown Ġgood Ġresults Ġin Ġlanguages Ġwith Ġsimilar Ġgram matical Ġstructure . ĠDeep ĠMT Ġsystems Ġhave Ġbeen Ġperforming Ġbetter Ġthan Ġthe Ġother Ġshallow ĠSM T Ġmodels Ġrecently , Ġwith Ġthe Ġavailability Ġof Ġcomputational Ġresources Ġand Ġhardware Ġmaking Ġit Ġfeasible Ġto Ġtrain Ġsuch Ġmodels . ĠThe Ġfirst Ġof Ġthese Ġmodels Ġcame Ġin Ġ2014 , Ġwith ĠCho Ġet Ġal ĠSecond One By Cho . ĠThe Ġmodel Ġused Ġwas Ġthe ĠR NN ĠL ST M Ġenc oder Ġdec oder Ġmodel Ġwith Ġthe Ġcontext Ġvector Ġoutput Ġof Ġthe Ġenc oder Ġ( run Ġfor Ġevery Ġword Ġin Ġthe Ġsentence ) Ġis Ġfed Ġto Ġevery Ġdec oder Ġunit Ġalong Ġwith Ġthe Ġprevious Ġword Ġoutput Ġuntil ĠIN LINE FORM 0 ĠE OS ĠIN LINE FORM 1 Ġis Ġreached . ĠThis Ġmodel Ġwas Ġused Ġto Ġscore Ġtranslation Ġresults Ġof Ġanother ĠMT Ġsystem . ĠS uts ke ver Ġet Ġal . Ġs uts ke ver 2014 sequence Ġcreated Ġa Ġsimilar Ġenc oder Ġdec oder Ġmodel Ġwith Ġthe Ġdec oder Ġgetting Ġthe Ġcontext Ġvector Ġonly Ġfor Ġthe Ġfirst Ġword Ġof Ġthe Ġtarget Ġlanguage Ġsentence . ĠAfter Ġthat , Ġonly Ġthe Ġdec oded Ġtarget Ġoutputs Ġact Ġas Ġinputs Ġto Ġthe Ġvarious Ġtime Ġsteps Ġof Ġthe Ġdec oder . ĠOne Ġmajor Ġdrawback Ġof Ġthese Ġmodels Ġis Ġthe Ġsize Ġof Ġthe Ġcontext Ġvector Ġof Ġthe Ġenc oder Ġbeing Ġstatic Ġin Ġnature . ĠThe Ġsame Ġsized Ġvector Ġwas Ġexpected Ġto Ġto Ġrepresent Ġsentences Ġof Ġarbitrary Ġlength , Ġwhich Ġwas Ġimpractical Ġwhen Ġit Ġcame Ġto Ġvery Ġlong Ġsentences . </s> The Ġnext Ġbreakthrough Ġcame Ġfrom ĠBah dan au Ġet Ġal . ĠBah dan au 2014 Ġwhere Ġvariable Ġlength Ġword Ġvectors Ġwere Ġused Ġand Ġinstead Ġof Ġjust Ġthe Ġcontext Ġvector , Ġa Ġweighted Ġsum Ġof Ġthe Ġinputs Ġis Ġgiven Ġfor Ġthe Ġdec oder . ĠThis Ġenabled Ġselective Ġlookup Ġto Ġthe Ġsource Ġsentence Ġduring Ġdecoding Ġand Ġis Ġknown Ġas Ġthe Ġattention Ġmechanism ĠB IB REF 27 Ġ. ĠThe Ġattention Ġmechanism Ġwas Ġfurther Ġanalysed Ġby ĠLu ong Ġet Ġal . Ġl u ong 2015 effective Ġwhere Ġthey Ġmade Ġa Ġdistinction Ġbetween Ġglobal Ġand Ġlocal Ġattention Ġby Ġmeans Ġof ĠA ER Ġscores Ġof Ġthe Ġattention Ġvectors . ĠA ĠGa ussian Ġdistribution Ġand Ġa Ġmon ot onic Ġlookup Ġwere Ġused Ġto Ġfacilitate Ġthe Ġcorresponding Ġlocal Ġsource Ġsentence Ġlook - up . </s> Conclusion </s> Thus , Ġit Ġis Ġseen Ġthat Ġthe Ġuse Ġof Ġmorph ological Ġsegment ation Ġon Ġa Ġmorph ologically Ġrich Ġlanguage Ġbefore Ġtranslation Ġhelps Ġwith Ġthe Ġperformance Ġof Ġthe Ġtranslation Ġin Ġmultiple Ġways . ĠThus , Ġmachine Ġtranslation Ġinvolving Ġmorph ologically Ġrich Ġlanguages Ġshould Ġideally Ġbe Ġcarried Ġout Ġonly Ġafter Ġmorph ological Ġsegment ation . ĠIf Ġthe Ġtranslation Ġhas Ġto Ġbe Ġcarried Ġout Ġbetween Ġtwo Ġmorph ologically Ġrich Ġlanguages , Ġthen Ġboth Ġthe Ġlanguages ' Ġsentences Ġshould Ġbe Ġindividually Ġsegment ed Ġbased Ġon Ġmorphology . ĠThis Ġis Ġbecause Ġwhile Ġit Ġis Ġtrue Ġthat Ġthey Ġare Ġboth Ġmorph ologically Ġrich Ġlanguages , Ġthe Ġschemes Ġthat Ġthe Ġlanguages Ġuse Ġfor Ġthe Ġprocess Ġof Ġagg l ut ination Ġmight Ġbe Ġdifferent , Ġin Ġwhich Ġcase Ġa Ġmapping Ġbetween Ġthe Ġunits Ġwould Ġbe Ġdifficult Ġwithout Ġthe Ġsegment ation . </s> One Ġdrawback Ġof Ġmorph ological Ġsegment ation Ġis Ġthe Ġincrease Ġin Ġcomplexity Ġof Ġthe Ġmodel Ġdue Ġto Ġan Ġincrease Ġin Ġthe Ġaverage Ġsentence Ġlengths . ĠThis Ġcannot Ġbe Ġavoided Ġas Ġit Ġis Ġessential Ġto Ġenable Ġa Ġcorrespondence Ġbetween Ġthe Ġsentences Ġof Ġthe Ġtwo Ġlanguages Ġwhen Ġone Ġof Ġthem Ġis Ġa Ġsimple Ġf usional Ġlanguage . ĠEven Ġwith Ġthe Ġincrease Ġin Ġthe Ġaverage Ġsentence Ġlength , Ġthe Ġattention Ġmodels Ġthat Ġhave Ġbeen Ġdeveloped Ġto Ġensure Ġcorrectness Ġof Ġtranslation Ġof Ġlong Ġsequences Ġcan Ġbe Ġput Ġto Ġgood Ġuse Ġwhen Ġinvolving Ġmorph ologically Ġrich Ġlanguages . ĠAnother Ġpoint Ġto Ġnote Ġhere Ġis Ġthat Ġmorph ologically Ġrich Ġlanguages Ġlike ĠTamil Ġgenerally Ġhave Ġlesser Ġnumber Ġof Ġwords Ġper Ġsentence Ġthan Ġlanguages Ġlike ĠEnglish Ġdue Ġto Ġthe Ġinherent Ġproperty Ġof Ġagg l ut ination . </s> Future ĠWork </s> The Ġmodel Ġimplemented Ġin Ġthis Ġpaper Ġonly Ġincludes Ġsource - side Ġmorph ological Ġsegment ation Ġand Ġdoes Ġnot Ġinclude Ġa Ġtarget Ġside Ġmorph ological Ġagg l ut ination Ġto Ġgive Ġback Ġthe Ġoutput Ġin Ġwords Ġrather Ġthan Ġmorp hem es . ĠIn Ġorder Ġto Ġimplement Ġan Ġend - to - end Ġtranslation Ġsystem Ġfor Ġmorph ologically Ġrich Ġlanguages , Ġa Ġmorph ological Ġgenerator Ġis Ġessential Ġbecause Ġthe Ġoutput Ġunits Ġof Ġthe Ġtranslation Ġcannot Ġbe Ġmorp hem es . </s> The Ġsame Ġmodel Ġimplemented Ġcan Ġbe Ġfurther Ġenhanced Ġby Ġmeans Ġof Ġa Ġbetter Ġcorpus Ġthat Ġcan Ġgeneral ize Ġover Ġmore Ġthan Ġjust Ġdomain Ġspecific Ġsource Ġsentences . ĠAlso , Ġthe Ġuse Ġof Ġa Ġbetter ĠGPU Ġwould Ġresult Ġin Ġa Ġbetter Ġallocation Ġof Ġthe Ġhidden Ġlayer Ġsizes Ġand Ġthe Ġbatch Ġsizes Ġthereby Ġpossibly Ġincreasing Ġthe Ġscope Ġand Ġaccuracy Ġof Ġlearning Ġof Ġthe Ġtranslation Ġmodel . </s> Although Ġnot Ġdirectly Ġrelated Ġto ĠMachine ĠTranslation , Ġthe Ġnovel Ġenc oder âĢĵ Ġdec oder Ġarchitecture Ġproposed Ġin Ġby ĠRock t as chel Ġet Ġal . Ġrock t as chel 2015 reason ing Ġfor ĠNatural ĠLanguage ĠIn ference Ġ( N LI ) Ġcan Ġbe Ġused Ġfor Ġthe Ġsame . ĠTheir Ġmodel Ġf uses Ġinf erences Ġfrom Ġeach Ġand Ġevery Ġindividual Ġword , Ġsummar izing Ġinformation Ġat Ġeach Ġstep , Ġthereby Ġlinking Ġthe Ġhidden Ġstate Ġof Ġthe Ġenc oder Ġwith Ġthat Ġof Ġthe Ġdec oder Ġby Ġmeans Ġof Ġa Ġweighted Ġsum , Ġtrained Ġfor Ġoptimization . </s> Acknowled gements </s> I Ġwould Ġlike Ġto Ġthank ĠDr . ĠM . ĠAn and ĠKumar , ĠAssistant ĠProfessor , ĠAm rit a ĠVish wa ĠV idy ape et ham Ġfor Ġhis Ġcontinuous Ġsupport Ġand Ġguidance . ĠI Ġwould Ġalso Ġlike Ġto Ġthank ĠDr . ĠAr vind an , ĠProfessor , ĠSS N ĠCollege ĠOf ĠEngineering Ġfor Ġhis Ġinputs Ġand Ġsuggestions .