For the WikiLarge dataset, the improvement over baseline NMT is 2.11 BLEU, 1.7 FKGL and 1.07 SARI.
For the WikiSmall dataset, the improvement over baseline NMT is  8.37 BLEU.
ABSTRACTIVE
6.37 BLEU
EXTRACTIVE
Results. Table 1 shows the results of all models on WikiLarge dataset. We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI. It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2.11 BLEU, 1.7 FKGL and 1.07 SARI. We also substantially outperform Dress, who previously reported SOTA result. The results of our human evaluation using Simplicity are also presented in Table 1. NMT on synthetic data is significantly better than PBMT-R, Dress, and SBMT-SARI on Simplicity. It indicates that our method with simplified data is effective at creating simpler output. Results on WikiSmall dataset are shown in Table 2. We see substantial improvements (6.37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences. Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI. Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models. In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data.
Results on WikiSmall dataset are shown in Table 2. We see substantial improvements (6.37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences. Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI. Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models. In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data.
1810.04428
Introduction </s> Text Ġsimpl ification Ġaims Ġto Ġreduce Ġthe Ġlex ical Ġand Ġstructural Ġcomplexity Ġof Ġa Ġtext , Ġwhile Ġstill Ġretaining Ġthe Ġsemantic Ġmeaning , Ġwhich Ġcan Ġhelp Ġchildren , Ġnon - native Ġspeakers , Ġand Ġpeople Ġwith Ġcognitive Ġdisabilities , Ġto Ġunderstand Ġtext Ġbetter . ĠOne Ġof Ġthe Ġmethods Ġof Ġautomatic Ġtext Ġsimpl ification Ġcan Ġbe Ġgenerally Ġdivided Ġinto Ġthree Ġcategories : Ġlex ical Ġsimpl ification Ġ( LS ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġrule - based ĠB IB REF 2 Ġ, Ġand Ġmachine Ġtranslation Ġ( MT ) ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠLS Ġis Ġmainly Ġused Ġto Ġsimplify Ġtext Ġby Ġsubstit uting Ġinf requent Ġand Ġdifficult Ġwords Ġwith Ġfrequent Ġand Ġeasier Ġwords . ĠHowever , Ġthere Ġare Ġseveral Ġchallenges Ġfor Ġthe ĠLS Ġapproach : Ġa Ġgreat Ġnumber Ġof Ġtransformation Ġrules Ġare Ġrequired Ġfor Ġreasonable Ġcoverage Ġand Ġshould Ġbe Ġapplied Ġbased Ġon Ġthe Ġspecific Ġcontext ; Ġthird , Ġthe Ġsyntax Ġand Ġsemantic Ġmeaning Ġof Ġthe Ġsentence Ġis Ġhard Ġto Ġretain . ĠRule - based Ġapproaches Ġuse Ġhand - crafted Ġrules Ġfor Ġlex ical Ġand Ġsynt actic Ġsimpl ification , Ġfor Ġexample , Ġsubstit uting Ġdifficult Ġwords Ġin Ġa Ġpred efined Ġvocabulary . ĠHowever , Ġsuch Ġapproaches Ġneed Ġa Ġlot Ġof Ġhuman - inv olve ment Ġto Ġmanually Ġdefine Ġthese Ġrules , Ġand Ġit Ġis Ġimpossible Ġto Ġgive Ġall Ġpossible Ġsimpl ification Ġrules . ĠMT - based Ġapproach Ġhas Ġattracted Ġgreat Ġattention Ġin Ġthe Ġlast Ġseveral Ġyears , Ġwhich Ġaddresses Ġtext Ġsimpl ification Ġas Ġa Ġmon oling ual Ġmachine Ġtranslation Ġproblem Ġtranslating Ġfrom Ġ' ordinary ' Ġand Ġ' sim pl ified ' Ġsentences . </s> In Ġrecent Ġyears , Ġneural ĠMachine ĠTranslation Ġ( N MT ) Ġis Ġa Ġnewly - prop osed Ġdeep Ġlearning Ġapproach Ġand Ġachieves Ġvery Ġimpressive Ġresults ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠUnlike Ġthe Ġtraditional Ġphr ased - based Ġmachine Ġtranslation Ġsystem Ġwhich Ġoperates Ġon Ġsmall Ġcomponents Ġseparately , ĠN MT Ġsystem Ġis Ġbeing Ġtrained Ġend - to - end , Ġwithout Ġthe Ġneed Ġto Ġhave Ġexternal Ġdec od ers , Ġlanguage Ġmodels Ġor Ġphrase Ġtables . ĠTherefore , Ġthe Ġexisting Ġarchitectures Ġin ĠN MT Ġare Ġused Ġfor Ġtext Ġsimpl ification ĠB IB REF 8 Ġ, ĠB IB REF 4 Ġ. ĠHowever , Ġmost Ġrecent Ġwork Ġusing ĠN MT Ġis Ġlimited Ġto Ġthe Ġtraining Ġdata Ġthat Ġare Ġscarce Ġand Ġexpensive Ġto Ġbuild . ĠLanguage Ġmodels Ġtrained Ġon Ġsimplified Ġcorpor a Ġhave Ġplayed Ġa Ġcentral Ġrole Ġin Ġstatistical Ġtext Ġsimpl ification ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠOne Ġmain Ġreason Ġis Ġthe Ġamount Ġof Ġavailable Ġsimplified Ġcorpor a Ġtypically Ġfar Ġexceeds Ġthe Ġamount Ġof Ġparallel Ġdata . ĠThe Ġperformance Ġof Ġmodels Ġcan Ġbe Ġtypically Ġimproved Ġwhen Ġtrained Ġon Ġmore Ġdata . ĠTherefore , Ġwe Ġexpect Ġsimplified Ġcorpor a Ġto Ġbe Ġespecially Ġhelpful Ġfor ĠN MT Ġmodels . </s> In Ġcontrast Ġto Ġprevious Ġwork , Ġwhich Ġuses Ġthe Ġexisting ĠN MT Ġmodels , Ġwe Ġexplore Ġstrategy Ġto Ġinclude Ġsimplified Ġtraining Ġcorpor a Ġin Ġthe Ġtraining Ġprocess Ġwithout Ġchanging Ġthe Ġneural Ġnetwork Ġarchitecture . ĠWe Ġfirst Ġpropose Ġto Ġpair Ġsimplified Ġtraining Ġsentences Ġwith Ġsynthetic Ġordinary Ġsentences Ġduring Ġtraining , Ġand Ġtreat Ġthis Ġsynthetic Ġdata Ġas Ġadditional Ġtraining Ġdata . ĠWe Ġobtain Ġsynthetic Ġordinary Ġsentences Ġthrough Ġback - translation , Ġi . e . Ġan Ġautomatic Ġtranslation Ġof Ġthe Ġsimplified Ġsentence Ġinto Ġthe Ġordinary Ġsentence ĠB IB REF 11 Ġ. ĠThen , Ġwe Ġmix Ġthe Ġsynthetic Ġdata Ġinto Ġthe Ġoriginal Ġ( sim pl ified - ordinary ) Ġdata Ġto Ġtrain ĠN MT Ġmodel . ĠExperimental Ġresults Ġon Ġtwo Ġpublicly Ġavailable Ġdatasets Ġshow Ġthat Ġwe Ġcan Ġimprove Ġthe Ġtext Ġsimpl ification Ġquality Ġof ĠN MT Ġmodels Ġby Ġmixing Ġsimplified Ġsentences Ġinto Ġthe Ġtraining Ġset Ġover ĠN MT Ġmodel Ġonly Ġusing Ġthe Ġoriginal Ġtraining Ġdata . </s> Related ĠWork </s> Aut omatic ĠTS Ġis Ġa Ġcomplicated Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġtask , Ġwhich Ġconsists Ġof Ġlex ical Ġand Ġsynt actic Ġsimpl ification Ġlevels ĠB IB REF 12 Ġ. ĠIt Ġhas Ġattracted Ġmuch Ġattention Ġrecently Ġas Ġit Ġcould Ġmake Ġtexts Ġmore Ġaccessible Ġto Ġwider Ġaudiences , Ġand Ġused Ġas Ġa Ġpre - processing Ġstep , Ġimprove Ġperformances Ġof Ġvarious ĠN LP Ġtasks Ġand Ġsystems ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠUsually , Ġhand - crafted , Ġsupervised , Ġand Ġun super vised Ġmethods Ġbased Ġon Ġresources Ġlike ĠEnglish ĠWikipedia Ġand ĠSimple ĠEnglish ĠWikipedia Ġ( EW - SE W ) ĠB IB REF 10 Ġare Ġutilized Ġfor Ġextracting Ġsimpl ification Ġrules . ĠIt Ġis Ġvery Ġeasy Ġto Ġmix Ġup Ġthe Ġautomatic ĠTS Ġtask Ġand Ġthe Ġautomatic Ġsummar ization Ġtask ĠB IB REF 3 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 6 Ġ. ĠTS Ġis Ġdifferent Ġfrom Ġtext Ġsummar ization Ġas Ġthe Ġfocus Ġof Ġtext Ġsummar ization Ġis Ġto Ġreduce Ġthe Ġlength Ġand Ġredundant Ġcontent . </s> At Ġthe Ġlex ical Ġlevel , Ġlex ical Ġsimpl ification Ġsystems Ġoften Ġsubstitute Ġdifficult Ġwords Ġusing Ġmore Ġcommon Ġwords , Ġwhich Ġonly Ġrequire Ġa Ġlarge Ġcorpus Ġof Ġregular Ġtext Ġto Ġobtain Ġword Ġembed d ings Ġto Ġget Ġwords Ġsimilar Ġto Ġthe Ġcomplex Ġword ĠB IB REF 1 Ġ, ĠB IB REF 9 Ġ. ĠBir an Ġet Ġal . ĠB IB REF 0 Ġadopted Ġan Ġun super vised Ġmethod Ġfor Ġlearning Ġpairs Ġof Ġcomplex Ġand Ġsimpler Ġsyn onyms Ġfrom Ġa Ġcorpus Ġconsisting Ġof ĠWikipedia Ġand ĠSimple ĠWikipedia . ĠAt Ġthe Ġsentence Ġlevel , Ġa Ġsentence Ġsimpl ification Ġmodel Ġwas Ġproposed Ġby Ġtree Ġtransformation Ġbased Ġon Ġstatistical Ġmachine Ġtranslation Ġ( SM T ) ĠB IB REF 3 Ġ. ĠWoods end Ġand ĠLap ata ĠB IB REF 17 Ġpresented Ġa Ġdata - driven Ġmodel Ġbased Ġon Ġa Ġquasi - syn chron ous Ġgrammar , Ġa Ġformal ism Ġthat Ġcan Ġnaturally Ġcapture Ġstructural Ġmism atches Ġand Ġcomplex Ġrewrite Ġoperations . ĠW ub ben Ġet Ġal . ĠB IB REF 18 Ġproposed Ġa Ġphrase - based Ġmachine Ġtranslation Ġ( P BM T ) Ġmodel Ġthat Ġis Ġtrained Ġon Ġordinary - sim pl ified Ġsentence Ġpairs . ĠXu Ġet Ġal . ĠB IB REF 19 Ġproposed Ġa Ġsyntax - based Ġmachine Ġtranslation Ġmodel Ġusing Ġsimpl ification - specific Ġobjective Ġfunctions Ġand Ġfeatures Ġto Ġencourage Ġsimpler Ġoutput . </s> Compared Ġwith ĠSM T , Ġneural Ġmachine Ġtranslation Ġ( N MT ) Ġhas Ġshown Ġto Ġproduce Ġstate - of - the - art Ġresults ĠB IB REF 5 Ġ, ĠB IB REF 7 Ġ. ĠThe Ġcentral Ġapproach Ġof ĠN MT Ġis Ġan Ġenc oder - dec oder Ġarchitecture Ġimplemented Ġby Ġrecurrent Ġneural Ġnetworks , Ġwhich Ġcan Ġrepresent Ġthe Ġinput Ġsequence Ġas Ġa Ġvector , Ġand Ġthen Ġdecode Ġthat Ġvector Ġinto Ġan Ġoutput Ġsequence . ĠTherefore , ĠN MT Ġmodels Ġwere Ġused Ġfor Ġtext Ġsimpl ification Ġtask , Ġand Ġachieved Ġgood Ġresults ĠB IB REF 8 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 20 Ġ. ĠThe Ġmain Ġlimitation Ġof Ġthe Ġaforementioned ĠN MT Ġmodels Ġfor Ġtext Ġsimpl ification Ġdepended Ġon Ġthe Ġparallel Ġordinary - sim pl ified Ġsentence Ġpairs . ĠBecause Ġordinary - sim pl ified Ġsentence Ġpairs Ġare Ġexpensive Ġand Ġtime - consuming Ġto Ġbuild , Ġthe Ġavailable Ġlargest Ġdata Ġis ĠEW - SE W Ġthat Ġonly Ġhave Ġ296 , 402 Ġsentence Ġpairs . ĠThe Ġdataset Ġis Ġins uff iciency Ġfor ĠN MT Ġmodel Ġif Ġwe Ġwant Ġto ĠN MT Ġmodel Ġcan Ġobtain Ġthe Ġbest Ġparameters . ĠConsidering Ġsimplified Ġdata Ġplays Ġan Ġimportant Ġrole Ġin Ġboosting Ġflu ency Ġfor Ġphrase - based Ġtext Ġsimpl ification , Ġand Ġwe Ġinvestigate Ġthe Ġuse Ġof Ġsimplified Ġdata Ġfor Ġtext Ġsimpl ification . ĠWe Ġare Ġthe Ġfirst Ġto Ġshow Ġthat Ġwe Ġcan Ġeffectively Ġadapt Ġneural Ġtranslation Ġmodels Ġfor Ġtext Ġsimpl if iation Ġwith Ġsimplified Ġcorpor a . </s> Sim pl ified ĠCorpor a </s> We Ġcollected Ġa Ġsimplified Ġdataset Ġfrom ĠSimple ĠEnglish ĠWikipedia Ġthat Ġare Ġfreely Ġavailable , Ġwhich Ġhas Ġbeen Ġpreviously Ġused Ġfor Ġmany Ġtext Ġsimpl ification Ġmethods ĠB IB REF 0 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 3 Ġ. ĠThe Ġsimple ĠEnglish ĠWikipedia Ġis Ġpretty Ġeasy Ġto Ġunderstand Ġthan Ġnormal ĠEnglish ĠWikipedia . ĠWe Ġdownloaded Ġall Ġarticles Ġfrom ĠSimple ĠEnglish ĠWikipedia . ĠFor Ġthese Ġarticles , Ġwe Ġremoved Ġstub s , Ġnavigation Ġpages Ġand Ġany Ġarticle Ġthat Ġconsisted Ġof Ġa Ġsingle Ġsentence . ĠWe Ġthen Ġsplit Ġthem Ġinto Ġsentences Ġwith Ġthe ĠStanford ĠCor N LP ĠB IB REF 21 Ġ, Ġand Ġdeleted Ġthese Ġsentences Ġwhose Ġnumber Ġof Ġwords Ġare Ġsmaller Ġthan Ġ10 Ġor Ġlarge Ġthan Ġ40 . ĠAfter Ġremoving Ġrepeated Ġsentences , Ġwe Ġchose Ġ600 K Ġsentences Ġas Ġthe Ġsimplified Ġdata Ġwith Ġ11 . 6 M Ġwords , Ġand Ġthe Ġsize Ġof Ġvocabulary Ġis Ġ82 K . </s> Text ĠSimpl ification Ġusing ĠNeural ĠMachine ĠTranslation </s> Our Ġwork Ġis Ġbuilt Ġon Ġattention - based ĠN MT ĠB IB REF 5 Ġas Ġan Ġenc oder - dec oder Ġnetwork Ġwith Ġrecurrent Ġneural Ġnetworks Ġ( R NN ), Ġwhich Ġsimultaneously Ġconducts Ġdynamic Ġalignment Ġand Ġgeneration Ġof Ġthe Ġtarget Ġsimplified Ġsentence . </s> The Ġenc oder Ġuses Ġa Ġbid irection al ĠR NN Ġthat Ġconsists Ġof Ġforward Ġand Ġbackward ĠR NN . ĠGiven Ġa Ġsource Ġsentence ĠIN LINE FORM 0 Ġ, Ġthe Ġforward ĠR NN Ġand Ġbackward ĠR NN Ġcalculate Ġforward Ġhidden Ġstates ĠIN LINE FORM 1 Ġand Ġbackward Ġhidden Ġstates ĠIN LINE FORM 2 Ġ, Ġrespectively . ĠThe Ġannotation Ġvector ĠIN LINE FORM 3 Ġis Ġobtained Ġby Ġconc aten ating ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ. </s> The Ġdec oder Ġis Ġa ĠR NN Ġthat Ġpredicts Ġa Ġtarget Ġsimpl ific ated Ġsentence Ġwith ĠG ated ĠRec urrent ĠUnit Ġ( GR U ) ĠB IB REF 22 Ġ. ĠGiven Ġthe Ġpreviously Ġgenerated Ġtarget Ġ( sim pl ified ) Ġsentence ĠIN LINE FORM 0 Ġ, Ġthe Ġprobability Ġof Ġnext Ġtarget Ġword ĠIN LINE FORM 1 Ġis ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnon - linear Ġfunction , ĠIN LINE FORM 1 Ġis Ġthe Ġembed ding Ġof ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġis Ġa Ġdecoding Ġstate Ġfor Ġtime Ġstep ĠIN LINE FORM 4 Ġ. </s> State ĠIN LINE FORM 0 Ġis Ġcalculated Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġactivation Ġfunction ĠGR U . </s> The ĠIN LINE FORM 0 Ġis Ġthe Ġcontext Ġvector Ġcomputed Ġas Ġa Ġweighted Ġannotation ĠIN LINE FORM 1 Ġ, Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> where Ġthe Ġweight ĠIN LINE FORM 0 Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġweight Ġmat rices . ĠThe Ġtraining Ġobjective Ġis Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġtraining Ġdata . ĠBeam Ġsearch Ġis Ġemployed Ġfor Ġdecoding . </s> Sy nt hetic ĠSimpl ified ĠSent ences </s> We Ġtrain Ġan Ġauxiliary Ġsystem Ġusing ĠN MT Ġmodel Ġfrom Ġthe Ġsimplified Ġsentence Ġto Ġthe Ġordinary Ġsentence , Ġwhich Ġis Ġfirst Ġtrained Ġon Ġthe Ġavailable Ġparallel Ġdata . ĠFor Ġleveraging Ġsimplified Ġsentences Ġto Ġimprove Ġthe Ġquality Ġof ĠN MT Ġmodel Ġfor Ġtext Ġsimpl ification , Ġwe Ġpropose Ġto Ġadapt Ġthe Ġback - translation Ġapproach Ġproposed Ġby ĠS enn rich Ġet Ġal . ĠB IB REF 11 Ġto Ġour Ġscenario . ĠMore Ġconcrete ly , ĠGiven Ġone Ġsentence Ġin Ġsimplified Ġsentences , Ġwe Ġuse Ġthe Ġsimplified - ordinary Ġsystem Ġin Ġtranslate Ġmode Ġwith Ġgreedy Ġdecoding Ġto Ġtranslate Ġit Ġto Ġthe Ġordinary Ġsentences , Ġwhich Ġis Ġden oted Ġas Ġback - translation . ĠThis Ġway , Ġwe Ġobtain Ġa Ġsynthetic Ġparallel Ġsimplified - ordinary Ġsentences . ĠBoth Ġthe Ġsynthetic Ġsentences Ġand Ġthe Ġavailable Ġparallel Ġdata Ġare Ġused Ġas Ġtraining Ġdata Ġfor Ġthe Ġoriginal ĠN MT Ġsystem . </s> E val uation </s> We Ġevaluate Ġthe Ġperformance Ġof Ġtext Ġsimpl ification Ġusing Ġneural Ġmachine Ġtranslation Ġon Ġavailable Ġparallel Ġsentences Ġand Ġadditional Ġsimplified Ġsentences . </s> Dat as et . ĠWe Ġuse Ġtwo Ġsimpl ification Ġdatasets Ġ( Wiki Small Ġand ĠWiki Large ). ĠWiki Small Ġconsists Ġof Ġordinary Ġand Ġsimplified Ġsentences Ġfrom Ġthe Ġordinary Ġand Ġsimple ĠEnglish ĠWik iped ias , Ġwhich Ġhas Ġbeen Ġused Ġas Ġbenchmark Ġfor Ġevaluating Ġtext Ġsimpl ification ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 8 Ġ. ĠThe Ġtraining Ġset Ġhas Ġ89 , 04 2 Ġsentence Ġpairs , Ġand Ġthe Ġtest Ġset Ġhas Ġ100 Ġpairs . ĠWiki Large Ġis Ġalso Ġfrom ĠWikipedia Ġcorpus Ġwhose Ġtraining Ġset Ġcontains Ġ296 , 402 Ġsentence Ġpairs ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠWiki Large Ġincludes Ġ8 Ġ( reference ) Ġsimpl ifications Ġfor Ġ2 , 359 Ġsentences Ġsplit Ġinto Ġ2 , 000 Ġfor Ġdevelopment Ġand Ġ359 Ġfor Ġtesting . </s> Met rics . ĠThree Ġmetrics Ġin Ġtext Ġsimpl ification Ġare Ġchosen Ġin Ġthis Ġpaper . ĠB LE U ĠB IB REF 5 Ġis Ġone Ġtraditional Ġmachine Ġtranslation Ġmetric Ġto Ġassess Ġthe Ġdegree Ġto Ġwhich Ġtranslated Ġsimpl ifications Ġdiffered Ġfrom Ġreference Ġsimpl ifications . ĠF K GL Ġmeasures Ġthe Ġread ability Ġof Ġthe Ġoutput ĠB IB REF 23 Ġ. ĠA Ġsmall ĠF K GL Ġrepresents Ġsimpler Ġoutput . ĠS ARI Ġis Ġa Ġrecent Ġtext - sim pl ification Ġmetric Ġby Ġcomparing Ġthe Ġoutput Ġagainst Ġthe Ġsource Ġand Ġreference Ġsimpl ifications ĠB IB REF 20 Ġ. </s> We Ġevaluate Ġthe Ġoutput Ġof Ġall Ġsystems Ġusing Ġhuman Ġevaluation . ĠThe Ġmetric Ġis Ġden oted Ġas ĠSimpl icity ĠB IB REF 8 Ġ. ĠThe Ġthree Ġnon - native Ġfluent ĠEnglish Ġspeakers Ġare Ġshown Ġreference Ġsentences Ġand Ġoutput Ġsentences . ĠThey Ġare Ġasked Ġwhether Ġthe Ġoutput Ġsentence Ġis Ġmuch Ġsimpler Ġ(+ 2 ), Ġsomewhat Ġsimpler Ġ(+ 1 ), Ġequally Ġ( 0 ), Ġsomewhat Ġmore Ġdifficult Ġ(- 1 ), Ġand Ġmuch Ġmore Ġdifficult Ġ(- 2 ) Ġthan Ġthe Ġreference Ġsentence . </s> Methods . ĠWe Ġuse ĠOpen N MT ĠB IB REF 24 Ġas Ġthe Ġimplementation Ġof Ġthe ĠN MT Ġsystem Ġfor Ġall Ġexperiments ĠB IB REF 5 Ġ. ĠWe Ġgenerally Ġfollow Ġthe Ġdefault Ġsettings Ġand Ġtraining Ġprocedure Ġdescribed Ġby ĠKlein Ġet Ġal .( 2017 ). ĠWe Ġreplace Ġout - of - voc abulary Ġwords Ġwith Ġa Ġspecial ĠUN K Ġsymbol . ĠAt Ġprediction Ġtime , Ġwe Ġreplace ĠUN K Ġwords Ġwith Ġthe Ġhighest Ġprobability Ġscore Ġfrom Ġthe Ġattention Ġlayer . ĠOpen N MT Ġsystem Ġused Ġon Ġparallel Ġdata Ġis Ġthe Ġbaseline Ġsystem . ĠTo Ġobtain Ġa Ġsynthetic Ġparallel Ġtraining Ġset , Ġwe Ġback - trans late Ġa Ġrandom Ġsample Ġof Ġ100 K Ġsentences Ġfrom Ġthe Ġcollected Ġsimplified Ġcorpor a . ĠOpen N MT Ġused Ġon Ġparallel Ġdata Ġand Ġsynthetic Ġdata Ġis Ġour Ġmodel . ĠThe Ġbenchmarks Ġare Ġrun Ġon Ġa ĠIntel ( R ) ĠCore ( TM ) Ġi 7 - 59 30 K ĠCPU @ 3 . 50 GHz , Ġ32 GB ĠMem , Ġtrained Ġon Ġ1 ĠGPU ĠGeForce ĠGTX Ġ1080 Ġ( P ascal ) Ġwith ĠCU DA Ġv . Ġ8 . 0 . </s> We Ġchoose Ġthree Ġstatistical Ġtext Ġsimpl ification Ġsystems . ĠP BM T - R Ġis Ġa Ġphrase - based Ġmethod Ġwith Ġa Ġre ranking Ġpost - processing Ġstep ĠB IB REF 18 Ġ. ĠHybrid Ġperforms Ġsentence Ġsplitting Ġand Ġdeletion Ġoperations Ġbased Ġon Ġdiscourse Ġrepresentation Ġstructures , Ġand Ġthen Ġsimpl ifies Ġsentences Ġwith ĠP BM T - R ĠB IB REF 25 Ġ. ĠS BM T - S ARI ĠB IB REF 19 Ġis Ġsyntax - based Ġtranslation Ġmodel Ġusing ĠP PD B Ġparaph rase Ġdatabase ĠB IB REF 26 Ġand Ġmod ifies Ġtuning Ġfunction Ġ( using ĠS ARI ). ĠWe Ġchoose Ġtwo Ġneural Ġtext Ġsimpl ification Ġsystems . ĠN MT Ġis Ġa Ġbasic Ġattention - based Ġenc oder - dec oder Ġmodel Ġwhich Ġuses ĠOpen N MT Ġframework Ġto Ġtrain Ġwith Ġtwo ĠL ST M Ġlayers , Ġhidden Ġstates Ġof Ġsize Ġ500 Ġand Ġ500 Ġhidden Ġunits , ĠSG D Ġoptim izer , Ġand Ġa Ġdrop out Ġrate Ġof Ġ0 . 3 ĠB IB REF 8 Ġ. ĠDress Ġis Ġan Ġenc oder - dec oder Ġmodel Ġcoupled Ġwith Ġa Ġdeep Ġreinforcement Ġlearning Ġframework , Ġand Ġthe Ġparameters Ġare Ġchosen Ġaccording Ġto Ġthe Ġoriginal Ġpaper ĠB IB REF 20 Ġ. ĠFor Ġthe Ġexperiments Ġwith Ġsynthetic Ġparallel Ġdata , Ġwe Ġback - trans late Ġa Ġrandom Ġsample Ġof Ġ60 Ġ000 Ġsentences Ġfrom Ġthe Ġcollected Ġsimplified Ġsentences Ġinto Ġordinary Ġsentences . ĠOur Ġmodel Ġis Ġtrained Ġon Ġsynthetic Ġdata Ġand Ġthe Ġavailable Ġparallel Ġdata , Ġden oted Ġas ĠN MT + sy nt hetic . </s> Results . ĠTable Ġ1 Ġshows Ġthe Ġresults Ġof Ġall Ġmodels Ġon ĠWiki Large Ġdataset . ĠWe Ġcan Ġsee Ġthat Ġour Ġmethod Ġ( N MT + sy nt hetic ) Ġcan Ġobtain Ġhigher ĠB LE U , Ġlower ĠF K GL Ġand Ġhigh ĠS ARI Ġcompared Ġwith Ġother Ġmodels , Ġexcept ĠDress Ġon ĠF K GL Ġand ĠS BM T - S ARI Ġon ĠS ARI . ĠIt Ġverified Ġthat Ġincluding Ġsynthetic Ġdata Ġduring Ġtraining Ġis Ġvery Ġeffective , Ġand Ġyields Ġan Ġimprovement Ġover Ġour Ġbaseline ĠNM F Ġby Ġ2 . 11 ĠB LE U , Ġ1 . 7 ĠF K GL Ġand Ġ1 . 07 ĠS ARI . ĠWe Ġalso Ġsubstantially Ġoutper form ĠDress , Ġwho Ġpreviously Ġreported ĠS OTA Ġresult . ĠThe Ġresults Ġof Ġour Ġhuman Ġevaluation Ġusing ĠSimpl icity Ġare Ġalso Ġpresented Ġin ĠTable Ġ1 . ĠN MT Ġon Ġsynthetic Ġdata Ġis Ġsignificantly Ġbetter Ġthan ĠP BM T - R , ĠDress , Ġand ĠS BM T - S ARI Ġon ĠSimpl icity . ĠIt Ġindicates Ġthat Ġour Ġmethod Ġwith Ġsimplified Ġdata Ġis Ġeffective Ġat Ġcreating Ġsimpler Ġoutput . </s> Results Ġon ĠWiki Small Ġdataset Ġare Ġshown Ġin ĠTable Ġ2 . ĠWe Ġsee Ġsubstantial Ġimprovements Ġ( 6 . 37 ĠB LE U ) Ġthan ĠN MT Ġfrom Ġadding Ġsimplified Ġtraining Ġdata Ġwith Ġsynthetic Ġordinary Ġsentences . ĠCompared Ġwith Ġstatistical Ġmachine Ġtranslation Ġmodels Ġ( P BM T - R , ĠHybrid , ĠS BM T - S ARI ), Ġour Ġmethod Ġ( N MT + sy nt hetic ) Ġstill Ġhave Ġbetter Ġresults , Ġbut Ġslightly Ġworse ĠF K GL Ġand ĠS ARI . ĠSimilar Ġto Ġthe Ġresults Ġin ĠWiki Large , Ġthe Ġresults Ġof Ġour Ġhuman Ġevaluation Ġusing ĠSimpl icity Ġoutper forms Ġthe Ġother Ġmodels . ĠIn Ġconclusion , ĠOur Ġmethod Ġproduces Ġbetter Ġresults Ġcomparing Ġwith Ġthe Ġbas elines , Ġwhich Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġadding Ġsimplified Ġtraining Ġdata . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġone Ġsimple Ġmethod Ġto Ġuse Ġsimplified Ġcorpor a Ġduring Ġtraining Ġof ĠN MT Ġsystems , Ġwith Ġno Ġchanges Ġto Ġthe Ġnetwork Ġarchitecture . ĠIn Ġthe Ġexperiments Ġon Ġtwo Ġdatasets , Ġwe Ġachieve Ġsubstantial Ġgains Ġin Ġall Ġtasks , Ġand Ġnew ĠS OTA Ġresults , Ġvia Ġback - translation Ġof Ġsimplified Ġsentences Ġinto Ġthe Ġordinary Ġsentences , Ġand Ġtreating Ġthis Ġsynthetic Ġdata Ġas Ġadditional Ġtraining Ġdata . ĠBecause Ġwe Ġdo Ġnot Ġchange Ġthe Ġneural Ġnetwork Ġarchitecture Ġto Ġintegrate Ġsimplified Ġcorpor a , Ġour Ġmethod Ġcan Ġbe Ġeasily Ġapplied Ġto Ġother ĠNeural ĠText ĠSimpl ification Ġ( N TS ) Ġsystems . ĠWe Ġexpect Ġthat Ġthe Ġeffectiveness Ġof Ġour Ġmethod Ġnot Ġonly Ġvaries Ġwith Ġthe Ġquality Ġof Ġthe ĠNTS Ġsystem Ġused Ġfor Ġback - translation , Ġbut Ġalso Ġdepends Ġon Ġthe Ġamount Ġof Ġavailable Ġparallel Ġand Ġsimplified Ġcorpor a . ĠIn Ġthe Ġpaper , Ġwe Ġhave Ġonly Ġutilized Ġdata Ġfrom ĠWikipedia Ġfor Ġsimplified Ġsentences . ĠIn Ġthe Ġfuture , Ġmany Ġother Ġtext Ġsources Ġare Ġavailable Ġand Ġthe Ġimpact Ġof Ġnot Ġonly Ġsize , Ġbut Ġalso Ġof Ġdomain Ġshould Ġbe Ġinvestigated .
Introduction Text simplification aims to reduce the lexical and structural complexity of a text, while still retaining the semantic meaning, which can help children, non-native speakers, and people with cognitive disabilities, to understand text better. One of the methods of automatic text simplification can be generally divided into three categories: lexical simplification (LS) BIBREF0 , BIBREF1 , rule-based BIBREF2 , and machine translation (MT) BIBREF3 , BIBREF4 . LS is mainly used to simplify text by substituting infrequent and difficult words with frequent and easier words. However, there are several challenges for the LS approach: a great number of transformation rules are required for reasonable coverage and should be applied based on the specific context; third, the syntax and semantic meaning of the sentence is hard to retain. Rule-based approaches use hand-crafted rules for lexical and syntactic simplification, for example, substituting difficult words in a predefined vocabulary. However, such approaches need a lot of human-involvement to manually define these rules, and it is impossible to give all possible simplification rules. MT-based approach has attracted great attention in the last several years, which addresses text simplification as a monolingual machine translation problem translating from 'ordinary' and 'simplified' sentences. In recent years, neural Machine Translation (NMT) is a newly-proposed deep learning approach and achieves very impressive results BIBREF5 , BIBREF6 , BIBREF7 . Unlike the traditional phrased-based machine translation system which operates on small components separately, NMT system is being trained end-to-end, without the need to have external decoders, language models or phrase tables. Therefore, the existing architectures in NMT are used for text simplification BIBREF8 , BIBREF4 . However, most recent work using NMT is limited to the training data that are scarce and expensive to build. Language models trained on simplified corpora have played a central role in statistical text simplification BIBREF9 , BIBREF10 . One main reason is the amount of available simplified corpora typically far exceeds the amount of parallel data. The performance of models can be typically improved when trained on more data. Therefore, we expect simplified corpora to be especially helpful for NMT models. In contrast to previous work, which uses the existing NMT models, we explore strategy to include simplified training corpora in the training process without changing the neural network architecture. We first propose to pair simplified training sentences with synthetic ordinary sentences during training, and treat this synthetic data as additional training data. We obtain synthetic ordinary sentences through back-translation, i.e. an automatic translation of the simplified sentence into the ordinary sentence BIBREF11 . Then, we mix the synthetic data into the original (simplified-ordinary) data to train NMT model. Experimental results on two publicly available datasets show that we can improve the text simplification quality of NMT models by mixing simplified sentences into the training set over NMT model only using the original training data. Related Work Automatic TS is a complicated natural language processing (NLP) task, which consists of lexical and syntactic simplification levels BIBREF12 . It has attracted much attention recently as it could make texts more accessible to wider audiences, and used as a pre-processing step, improve performances of various NLP tasks and systems BIBREF13 , BIBREF14 , BIBREF15 . Usually, hand-crafted, supervised, and unsupervised methods based on resources like English Wikipedia and Simple English Wikipedia (EW-SEW) BIBREF10 are utilized for extracting simplification rules. It is very easy to mix up the automatic TS task and the automatic summarization task BIBREF3 , BIBREF16 , BIBREF6 . TS is different from text summarization as the focus of text summarization is to reduce the length and redundant content. At the lexical level, lexical simplification systems often substitute difficult words using more common words, which only require a large corpus of regular text to obtain word embeddings to get words similar to the complex word BIBREF1 , BIBREF9 . Biran et al. BIBREF0 adopted an unsupervised method for learning pairs of complex and simpler synonyms from a corpus consisting of Wikipedia and Simple Wikipedia. At the sentence level, a sentence simplification model was proposed by tree transformation based on statistical machine translation (SMT) BIBREF3 . Woodsend and Lapata BIBREF17 presented a data-driven model based on a quasi-synchronous grammar, a formalism that can naturally capture structural mismatches and complex rewrite operations. Wubben et al. BIBREF18 proposed a phrase-based machine translation (PBMT) model that is trained on ordinary-simplified sentence pairs. Xu et al. BIBREF19 proposed a syntax-based machine translation model using simplification-specific objective functions and features to encourage simpler output. Compared with SMT, neural machine translation (NMT) has shown to produce state-of-the-art results BIBREF5 , BIBREF7 . The central approach of NMT is an encoder-decoder architecture implemented by recurrent neural networks, which can represent the input sequence as a vector, and then decode that vector into an output sequence. Therefore, NMT models were used for text simplification task, and achieved good results BIBREF8 , BIBREF4 , BIBREF20 . The main limitation of the aforementioned NMT models for text simplification depended on the parallel ordinary-simplified sentence pairs. Because ordinary-simplified sentence pairs are expensive and time-consuming to build, the available largest data is EW-SEW that only have 296,402 sentence pairs. The dataset is insufficiency for NMT model if we want to NMT model can obtain the best parameters. Considering simplified data plays an important role in boosting fluency for phrase-based text simplification, and we investigate the use of simplified data for text simplification. We are the first to show that we can effectively adapt neural translation models for text simplifiation with simplified corpora. Simplified Corpora We collected a simplified dataset from Simple English Wikipedia that are freely available, which has been previously used for many text simplification methods BIBREF0 , BIBREF10 , BIBREF3 . The simple English Wikipedia is pretty easy to understand than normal English Wikipedia. We downloaded all articles from Simple English Wikipedia. For these articles, we removed stubs, navigation pages and any article that consisted of a single sentence. We then split them into sentences with the Stanford CorNLP BIBREF21 , and deleted these sentences whose number of words are smaller than 10 or large than 40. After removing repeated sentences, we chose 600K sentences as the simplified data with 11.6M words, and the size of vocabulary is 82K. Text Simplification using Neural Machine Translation Our work is built on attention-based NMT BIBREF5 as an encoder-decoder network with recurrent neural networks (RNN), which simultaneously conducts dynamic alignment and generation of the target simplified sentence. The encoder uses a bidirectional RNN that consists of forward and backward RNN. Given a source sentence INLINEFORM0 , the forward RNN and backward RNN calculate forward hidden states INLINEFORM1 and backward hidden states INLINEFORM2 , respectively. The annotation vector INLINEFORM3 is obtained by concatenating INLINEFORM4 and INLINEFORM5 . The decoder is a RNN that predicts a target simplificated sentence with Gated Recurrent Unit (GRU) BIBREF22 . Given the previously generated target (simplified) sentence INLINEFORM0 , the probability of next target word INLINEFORM1 is DISPLAYFORM0 where INLINEFORM0 is a non-linear function, INLINEFORM1 is the embedding of INLINEFORM2 , and INLINEFORM3 is a decoding state for time step INLINEFORM4 . State INLINEFORM0 is calculated by DISPLAYFORM0 where INLINEFORM0 is the activation function GRU. The INLINEFORM0 is the context vector computed as a weighted annotation INLINEFORM1 , computed by DISPLAYFORM0 where the weight INLINEFORM0 is computed by DISPLAYFORM0 DISPLAYFORM1 where INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are weight matrices. The training objective is to maximize the likelihood of the training data. Beam search is employed for decoding. Synthetic Simplified Sentences We train an auxiliary system using NMT model from the simplified sentence to the ordinary sentence, which is first trained on the available parallel data. For leveraging simplified sentences to improve the quality of NMT model for text simplification, we propose to adapt the back-translation approach proposed by Sennrich et al. BIBREF11 to our scenario. More concretely, Given one sentence in simplified sentences, we use the simplified-ordinary system in translate mode with greedy decoding to translate it to the ordinary sentences, which is denoted as back-translation. This way, we obtain a synthetic parallel simplified-ordinary sentences. Both the synthetic sentences and the available parallel data are used as training data for the original NMT system. Evaluation We evaluate the performance of text simplification using neural machine translation on available parallel sentences and additional simplified sentences. Dataset. We use two simplification datasets (WikiSmall and WikiLarge). WikiSmall consists of ordinary and simplified sentences from the ordinary and simple English Wikipedias, which has been used as benchmark for evaluating text simplification BIBREF17 , BIBREF18 , BIBREF8 . The training set has 89,042 sentence pairs, and the test set has 100 pairs. WikiLarge is also from Wikipedia corpus whose training set contains 296,402 sentence pairs BIBREF19 , BIBREF20 . WikiLarge includes 8 (reference) simplifications for 2,359 sentences split into 2,000 for development and 359 for testing. Metrics. Three metrics in text simplification are chosen in this paper. BLEU BIBREF5 is one traditional machine translation metric to assess the degree to which translated simplifications differed from reference simplifications. FKGL measures the readability of the output BIBREF23 . A small FKGL represents simpler output. SARI is a recent text-simplification metric by comparing the output against the source and reference simplifications BIBREF20 . We evaluate the output of all systems using human evaluation. The metric is denoted as Simplicity BIBREF8 . The three non-native fluent English speakers are shown reference sentences and output sentences. They are asked whether the output sentence is much simpler (+2), somewhat simpler (+1), equally (0), somewhat more difficult (-1), and much more difficult (-2) than the reference sentence. Methods. We use OpenNMT BIBREF24 as the implementation of the NMT system for all experiments BIBREF5 . We generally follow the default settings and training procedure described by Klein et al.(2017). We replace out-of-vocabulary words with a special UNK symbol. At prediction time, we replace UNK words with the highest probability score from the attention layer. OpenNMT system used on parallel data is the baseline system. To obtain a synthetic parallel training set, we back-translate a random sample of 100K sentences from the collected simplified corpora. OpenNMT used on parallel data and synthetic data is our model. The benchmarks are run on a Intel(R) Core(TM) i7-5930K CPU@3.50GHz, 32GB Mem, trained on 1 GPU GeForce GTX 1080 (Pascal) with CUDA v. 8.0. We choose three statistical text simplification systems. PBMT-R is a phrase-based method with a reranking post-processing step BIBREF18 . Hybrid performs sentence splitting and deletion operations based on discourse representation structures, and then simplifies sentences with PBMT-R BIBREF25 . SBMT-SARI BIBREF19 is syntax-based translation model using PPDB paraphrase database BIBREF26 and modifies tuning function (using SARI). We choose two neural text simplification systems. NMT is a basic attention-based encoder-decoder model which uses OpenNMT framework to train with two LSTM layers, hidden states of size 500 and 500 hidden units, SGD optimizer, and a dropout rate of 0.3 BIBREF8 . Dress is an encoder-decoder model coupled with a deep reinforcement learning framework, and the parameters are chosen according to the original paper BIBREF20 . For the experiments with synthetic parallel data, we back-translate a random sample of 60 000 sentences from the collected simplified sentences into ordinary sentences. Our model is trained on synthetic data and the available parallel data, denoted as NMT+synthetic. Results. Table 1 shows the results of all models on WikiLarge dataset. We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI. It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2.11 BLEU, 1.7 FKGL and 1.07 SARI. We also substantially outperform Dress, who previously reported SOTA result. The results of our human evaluation using Simplicity are also presented in Table 1. NMT on synthetic data is significantly better than PBMT-R, Dress, and SBMT-SARI on Simplicity. It indicates that our method with simplified data is effective at creating simpler output. Results on WikiSmall dataset are shown in Table 2. We see substantial improvements (6.37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences. Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI. Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models. In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data. Conclusion In this paper, we propose one simple method to use simplified corpora during training of NMT systems, with no changes to the network architecture. In the experiments on two datasets, we achieve substantial gains in all tasks, and new SOTA results, via back-translation of simplified sentences into the ordinary sentences, and treating this synthetic data as additional training data. Because we do not change the neural network architecture to integrate simplified corpora, our method can be easily applied to other Neural Text Simplification (NTS) systems. We expect that the effectiveness of our method not only varies with the quality of the NTS system used for back-translation, but also depends on the amount of available parallel and simplified corpora. In the paper, we have only utilized data from Wikipedia for simplified sentences. In the future, many other text sources are available and the impact of not only size, but also of domain should be investigated.
by how much did their model improve?
4625cfba3083346a96e573af5464bc26c34ec943
by Ġhow Ġmuch Ġdid Ġtheir Ġmodel Ġimprove ?
For Ġthe ĠWiki Large Ġdataset , Ġthe Ġimprovement Ġover Ġbaseline ĠN MT Ġis Ġ2 . 11 ĠB LE U , Ġ1 . 7 ĠF K GL Ġand Ġ1 . 07 ĠS ARI . Ċ For Ġthe ĠWiki Small Ġdataset , Ġthe Ġimprovement Ġover Ġbaseline ĠN MT Ġis Ġ Ġ8 . 37 ĠB LE U .
<s> by Ġhow Ġmuch Ġdid Ġtheir Ġmodel Ġimprove ? </s> Introduction </s> Text Ġsimpl ification Ġaims Ġto Ġreduce Ġthe Ġlex ical Ġand Ġstructural Ġcomplexity Ġof Ġa Ġtext , Ġwhile Ġstill Ġretaining Ġthe Ġsemantic Ġmeaning , Ġwhich Ġcan Ġhelp Ġchildren , Ġnon - native Ġspeakers , Ġand Ġpeople Ġwith Ġcognitive Ġdisabilities , Ġto Ġunderstand Ġtext Ġbetter . ĠOne Ġof Ġthe Ġmethods Ġof Ġautomatic Ġtext Ġsimpl ification Ġcan Ġbe Ġgenerally Ġdivided Ġinto Ġthree Ġcategories : Ġlex ical Ġsimpl ification Ġ( LS ) ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġrule - based ĠB IB REF 2 Ġ, Ġand Ġmachine Ġtranslation Ġ( MT ) ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠLS Ġis Ġmainly Ġused Ġto Ġsimplify Ġtext Ġby Ġsubstit uting Ġinf requent Ġand Ġdifficult Ġwords Ġwith Ġfrequent Ġand Ġeasier Ġwords . ĠHowever , Ġthere Ġare Ġseveral Ġchallenges Ġfor Ġthe ĠLS Ġapproach : Ġa Ġgreat Ġnumber Ġof Ġtransformation Ġrules Ġare Ġrequired Ġfor Ġreasonable Ġcoverage Ġand Ġshould Ġbe Ġapplied Ġbased Ġon Ġthe Ġspecific Ġcontext ; Ġthird , Ġthe Ġsyntax Ġand Ġsemantic Ġmeaning Ġof Ġthe Ġsentence Ġis Ġhard Ġto Ġretain . ĠRule - based Ġapproaches Ġuse Ġhand - crafted Ġrules Ġfor Ġlex ical Ġand Ġsynt actic Ġsimpl ification , Ġfor Ġexample , Ġsubstit uting Ġdifficult Ġwords Ġin Ġa Ġpred efined Ġvocabulary . ĠHowever , Ġsuch Ġapproaches Ġneed Ġa Ġlot Ġof Ġhuman - inv olve ment Ġto Ġmanually Ġdefine Ġthese Ġrules , Ġand Ġit Ġis Ġimpossible Ġto Ġgive Ġall Ġpossible Ġsimpl ification Ġrules . ĠMT - based Ġapproach Ġhas Ġattracted Ġgreat Ġattention Ġin Ġthe Ġlast Ġseveral Ġyears , Ġwhich Ġaddresses Ġtext Ġsimpl ification Ġas Ġa Ġmon oling ual Ġmachine Ġtranslation Ġproblem Ġtranslating Ġfrom Ġ' ordinary ' Ġand Ġ' sim pl ified ' Ġsentences . </s> In Ġrecent Ġyears , Ġneural ĠMachine ĠTranslation Ġ( N MT ) Ġis Ġa Ġnewly - prop osed Ġdeep Ġlearning Ġapproach Ġand Ġachieves Ġvery Ġimpressive Ġresults ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠUnlike Ġthe Ġtraditional Ġphr ased - based Ġmachine Ġtranslation Ġsystem Ġwhich Ġoperates Ġon Ġsmall Ġcomponents Ġseparately , ĠN MT Ġsystem Ġis Ġbeing Ġtrained Ġend - to - end , Ġwithout Ġthe Ġneed Ġto Ġhave Ġexternal Ġdec od ers , Ġlanguage Ġmodels Ġor Ġphrase Ġtables . ĠTherefore , Ġthe Ġexisting Ġarchitectures Ġin ĠN MT Ġare Ġused Ġfor Ġtext Ġsimpl ification ĠB IB REF 8 Ġ, ĠB IB REF 4 Ġ. ĠHowever , Ġmost Ġrecent Ġwork Ġusing ĠN MT Ġis Ġlimited Ġto Ġthe Ġtraining Ġdata Ġthat Ġare Ġscarce Ġand Ġexpensive Ġto Ġbuild . ĠLanguage Ġmodels Ġtrained Ġon Ġsimplified Ġcorpor a Ġhave Ġplayed Ġa Ġcentral Ġrole Ġin Ġstatistical Ġtext Ġsimpl ification ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. ĠOne Ġmain Ġreason Ġis Ġthe Ġamount Ġof Ġavailable Ġsimplified Ġcorpor a Ġtypically Ġfar Ġexceeds Ġthe Ġamount Ġof Ġparallel Ġdata . ĠThe Ġperformance Ġof Ġmodels Ġcan Ġbe Ġtypically Ġimproved Ġwhen Ġtrained Ġon Ġmore Ġdata . ĠTherefore , Ġwe Ġexpect Ġsimplified Ġcorpor a Ġto Ġbe Ġespecially Ġhelpful Ġfor ĠN MT Ġmodels . </s> In Ġcontrast Ġto Ġprevious Ġwork , Ġwhich Ġuses Ġthe Ġexisting ĠN MT Ġmodels , Ġwe Ġexplore Ġstrategy Ġto Ġinclude Ġsimplified Ġtraining Ġcorpor a Ġin Ġthe Ġtraining Ġprocess Ġwithout Ġchanging Ġthe Ġneural Ġnetwork Ġarchitecture . ĠWe Ġfirst Ġpropose Ġto Ġpair Ġsimplified Ġtraining Ġsentences Ġwith Ġsynthetic Ġordinary Ġsentences Ġduring Ġtraining , Ġand Ġtreat Ġthis Ġsynthetic Ġdata Ġas Ġadditional Ġtraining Ġdata . ĠWe Ġobtain Ġsynthetic Ġordinary Ġsentences Ġthrough Ġback - translation , Ġi . e . Ġan Ġautomatic Ġtranslation Ġof Ġthe Ġsimplified Ġsentence Ġinto Ġthe Ġordinary Ġsentence ĠB IB REF 11 Ġ. ĠThen , Ġwe Ġmix Ġthe Ġsynthetic Ġdata Ġinto Ġthe Ġoriginal Ġ( sim pl ified - ordinary ) Ġdata Ġto Ġtrain ĠN MT Ġmodel . ĠExperimental Ġresults Ġon Ġtwo Ġpublicly Ġavailable Ġdatasets Ġshow Ġthat Ġwe Ġcan Ġimprove Ġthe Ġtext Ġsimpl ification Ġquality Ġof ĠN MT Ġmodels Ġby Ġmixing Ġsimplified Ġsentences Ġinto Ġthe Ġtraining Ġset Ġover ĠN MT Ġmodel Ġonly Ġusing Ġthe Ġoriginal Ġtraining Ġdata . </s> Related ĠWork </s> Aut omatic ĠTS Ġis Ġa Ġcomplicated Ġnatural Ġlanguage Ġprocessing Ġ( N LP ) Ġtask , Ġwhich Ġconsists Ġof Ġlex ical Ġand Ġsynt actic Ġsimpl ification Ġlevels ĠB IB REF 12 Ġ. ĠIt Ġhas Ġattracted Ġmuch Ġattention Ġrecently Ġas Ġit Ġcould Ġmake Ġtexts Ġmore Ġaccessible Ġto Ġwider Ġaudiences , Ġand Ġused Ġas Ġa Ġpre - processing Ġstep , Ġimprove Ġperformances Ġof Ġvarious ĠN LP Ġtasks Ġand Ġsystems ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠUsually , Ġhand - crafted , Ġsupervised , Ġand Ġun super vised Ġmethods Ġbased Ġon Ġresources Ġlike ĠEnglish ĠWikipedia Ġand ĠSimple ĠEnglish ĠWikipedia Ġ( EW - SE W ) ĠB IB REF 10 Ġare Ġutilized Ġfor Ġextracting Ġsimpl ification Ġrules . ĠIt Ġis Ġvery Ġeasy Ġto Ġmix Ġup Ġthe Ġautomatic ĠTS Ġtask Ġand Ġthe Ġautomatic Ġsummar ization Ġtask ĠB IB REF 3 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 6 Ġ. ĠTS Ġis Ġdifferent Ġfrom Ġtext Ġsummar ization Ġas Ġthe Ġfocus Ġof Ġtext Ġsummar ization Ġis Ġto Ġreduce Ġthe Ġlength Ġand Ġredundant Ġcontent . </s> At Ġthe Ġlex ical Ġlevel , Ġlex ical Ġsimpl ification Ġsystems Ġoften Ġsubstitute Ġdifficult Ġwords Ġusing Ġmore Ġcommon Ġwords , Ġwhich Ġonly Ġrequire Ġa Ġlarge Ġcorpus Ġof Ġregular Ġtext Ġto Ġobtain Ġword Ġembed d ings Ġto Ġget Ġwords Ġsimilar Ġto Ġthe Ġcomplex Ġword ĠB IB REF 1 Ġ, ĠB IB REF 9 Ġ. ĠBir an Ġet Ġal . ĠB IB REF 0 Ġadopted Ġan Ġun super vised Ġmethod Ġfor Ġlearning Ġpairs Ġof Ġcomplex Ġand Ġsimpler Ġsyn onyms Ġfrom Ġa Ġcorpus Ġconsisting Ġof ĠWikipedia Ġand ĠSimple ĠWikipedia . ĠAt Ġthe Ġsentence Ġlevel , Ġa Ġsentence Ġsimpl ification Ġmodel Ġwas Ġproposed Ġby Ġtree Ġtransformation Ġbased Ġon Ġstatistical Ġmachine Ġtranslation Ġ( SM T ) ĠB IB REF 3 Ġ. ĠWoods end Ġand ĠLap ata ĠB IB REF 17 Ġpresented Ġa Ġdata - driven Ġmodel Ġbased Ġon Ġa Ġquasi - syn chron ous Ġgrammar , Ġa Ġformal ism Ġthat Ġcan Ġnaturally Ġcapture Ġstructural Ġmism atches Ġand Ġcomplex Ġrewrite Ġoperations . ĠW ub ben Ġet Ġal . ĠB IB REF 18 Ġproposed Ġa Ġphrase - based Ġmachine Ġtranslation Ġ( P BM T ) Ġmodel Ġthat Ġis Ġtrained Ġon Ġordinary - sim pl ified Ġsentence Ġpairs . ĠXu Ġet Ġal . ĠB IB REF 19 Ġproposed Ġa Ġsyntax - based Ġmachine Ġtranslation Ġmodel Ġusing Ġsimpl ification - specific Ġobjective Ġfunctions Ġand Ġfeatures Ġto Ġencourage Ġsimpler Ġoutput . </s> Compared Ġwith ĠSM T , Ġneural Ġmachine Ġtranslation Ġ( N MT ) Ġhas Ġshown Ġto Ġproduce Ġstate - of - the - art Ġresults ĠB IB REF 5 Ġ, ĠB IB REF 7 Ġ. ĠThe Ġcentral Ġapproach Ġof ĠN MT Ġis Ġan Ġenc oder - dec oder Ġarchitecture Ġimplemented Ġby Ġrecurrent Ġneural Ġnetworks , Ġwhich Ġcan Ġrepresent Ġthe Ġinput Ġsequence Ġas Ġa Ġvector , Ġand Ġthen Ġdecode Ġthat Ġvector Ġinto Ġan Ġoutput Ġsequence . ĠTherefore , ĠN MT Ġmodels Ġwere Ġused Ġfor Ġtext Ġsimpl ification Ġtask , Ġand Ġachieved Ġgood Ġresults ĠB IB REF 8 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 20 Ġ. ĠThe Ġmain Ġlimitation Ġof Ġthe Ġaforementioned ĠN MT Ġmodels Ġfor Ġtext Ġsimpl ification Ġdepended Ġon Ġthe Ġparallel Ġordinary - sim pl ified Ġsentence Ġpairs . ĠBecause Ġordinary - sim pl ified Ġsentence Ġpairs Ġare Ġexpensive Ġand Ġtime - consuming Ġto Ġbuild , Ġthe Ġavailable Ġlargest Ġdata Ġis ĠEW - SE W Ġthat Ġonly Ġhave Ġ296 , 402 Ġsentence Ġpairs . ĠThe Ġdataset Ġis Ġins uff iciency Ġfor ĠN MT Ġmodel Ġif Ġwe Ġwant Ġto ĠN MT Ġmodel Ġcan Ġobtain Ġthe Ġbest Ġparameters . ĠConsidering Ġsimplified Ġdata Ġplays Ġan Ġimportant Ġrole Ġin Ġboosting Ġflu ency Ġfor Ġphrase - based Ġtext Ġsimpl ification , Ġand Ġwe Ġinvestigate Ġthe Ġuse Ġof Ġsimplified Ġdata Ġfor Ġtext Ġsimpl ification . ĠWe Ġare Ġthe Ġfirst Ġto Ġshow Ġthat Ġwe Ġcan Ġeffectively Ġadapt Ġneural Ġtranslation Ġmodels Ġfor Ġtext Ġsimpl if iation Ġwith Ġsimplified Ġcorpor a . </s> Sim pl ified ĠCorpor a </s> We Ġcollected Ġa Ġsimplified Ġdataset Ġfrom ĠSimple ĠEnglish ĠWikipedia Ġthat Ġare Ġfreely Ġavailable , Ġwhich Ġhas Ġbeen Ġpreviously Ġused Ġfor Ġmany Ġtext Ġsimpl ification Ġmethods ĠB IB REF 0 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 3 Ġ. ĠThe Ġsimple ĠEnglish ĠWikipedia Ġis Ġpretty Ġeasy Ġto Ġunderstand Ġthan Ġnormal ĠEnglish ĠWikipedia . ĠWe Ġdownloaded Ġall Ġarticles Ġfrom ĠSimple ĠEnglish ĠWikipedia . ĠFor Ġthese Ġarticles , Ġwe Ġremoved Ġstub s , Ġnavigation Ġpages Ġand Ġany Ġarticle Ġthat Ġconsisted Ġof Ġa Ġsingle Ġsentence . ĠWe Ġthen Ġsplit Ġthem Ġinto Ġsentences Ġwith Ġthe ĠStanford ĠCor N LP ĠB IB REF 21 Ġ, Ġand Ġdeleted Ġthese Ġsentences Ġwhose Ġnumber Ġof Ġwords Ġare Ġsmaller Ġthan Ġ10 Ġor Ġlarge Ġthan Ġ40 . ĠAfter Ġremoving Ġrepeated Ġsentences , Ġwe Ġchose Ġ600 K Ġsentences Ġas Ġthe Ġsimplified Ġdata Ġwith Ġ11 . 6 M Ġwords , Ġand Ġthe Ġsize Ġof Ġvocabulary Ġis Ġ82 K . </s> Text ĠSimpl ification Ġusing ĠNeural ĠMachine ĠTranslation </s> Our Ġwork Ġis Ġbuilt Ġon Ġattention - based ĠN MT ĠB IB REF 5 Ġas Ġan Ġenc oder - dec oder Ġnetwork Ġwith Ġrecurrent Ġneural Ġnetworks Ġ( R NN ), Ġwhich Ġsimultaneously Ġconducts Ġdynamic Ġalignment Ġand Ġgeneration Ġof Ġthe Ġtarget Ġsimplified Ġsentence . </s> The Ġenc oder Ġuses Ġa Ġbid irection al ĠR NN Ġthat Ġconsists Ġof Ġforward Ġand Ġbackward ĠR NN . ĠGiven Ġa Ġsource Ġsentence ĠIN LINE FORM 0 Ġ, Ġthe Ġforward ĠR NN Ġand Ġbackward ĠR NN Ġcalculate Ġforward Ġhidden Ġstates ĠIN LINE FORM 1 Ġand Ġbackward Ġhidden Ġstates ĠIN LINE FORM 2 Ġ, Ġrespectively . ĠThe Ġannotation Ġvector ĠIN LINE FORM 3 Ġis Ġobtained Ġby Ġconc aten ating ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġ. </s> The Ġdec oder Ġis Ġa ĠR NN Ġthat Ġpredicts Ġa Ġtarget Ġsimpl ific ated Ġsentence Ġwith ĠG ated ĠRec urrent ĠUnit Ġ( GR U ) ĠB IB REF 22 Ġ. ĠGiven Ġthe Ġpreviously Ġgenerated Ġtarget Ġ( sim pl ified ) Ġsentence ĠIN LINE FORM 0 Ġ, Ġthe Ġprobability Ġof Ġnext Ġtarget Ġword ĠIN LINE FORM 1 Ġis ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġnon - linear Ġfunction , ĠIN LINE FORM 1 Ġis Ġthe Ġembed ding Ġof ĠIN LINE FORM 2 Ġ, Ġand ĠIN LINE FORM 3 Ġis Ġa Ġdecoding Ġstate Ġfor Ġtime Ġstep ĠIN LINE FORM 4 Ġ. </s> State ĠIN LINE FORM 0 Ġis Ġcalculated Ġby ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġactivation Ġfunction ĠGR U . </s> The ĠIN LINE FORM 0 Ġis Ġthe Ġcontext Ġvector Ġcomputed Ġas Ġa Ġweighted Ġannotation ĠIN LINE FORM 1 Ġ, Ġcomputed Ġby ĠDIS PLAY FORM 0 </s> where Ġthe Ġweight ĠIN LINE FORM 0 Ġis Ġcomputed Ġby ĠDIS PLAY FORM 0 ĠDIS PLAY FORM 1 </s> where ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġweight Ġmat rices . ĠThe Ġtraining Ġobjective Ġis Ġto Ġmaximize Ġthe Ġlikelihood Ġof Ġthe Ġtraining Ġdata . ĠBeam Ġsearch Ġis Ġemployed Ġfor Ġdecoding . </s> Sy nt hetic ĠSimpl ified ĠSent ences </s> We Ġtrain Ġan Ġauxiliary Ġsystem Ġusing ĠN MT Ġmodel Ġfrom Ġthe Ġsimplified Ġsentence Ġto Ġthe Ġordinary Ġsentence , Ġwhich Ġis Ġfirst Ġtrained Ġon Ġthe Ġavailable Ġparallel Ġdata . ĠFor Ġleveraging Ġsimplified Ġsentences Ġto Ġimprove Ġthe Ġquality Ġof ĠN MT Ġmodel Ġfor Ġtext Ġsimpl ification , Ġwe Ġpropose Ġto Ġadapt Ġthe Ġback - translation Ġapproach Ġproposed Ġby ĠS enn rich Ġet Ġal . ĠB IB REF 11 Ġto Ġour Ġscenario . ĠMore Ġconcrete ly , ĠGiven Ġone Ġsentence Ġin Ġsimplified Ġsentences , Ġwe Ġuse Ġthe Ġsimplified - ordinary Ġsystem Ġin Ġtranslate Ġmode Ġwith Ġgreedy Ġdecoding Ġto Ġtranslate Ġit Ġto Ġthe Ġordinary Ġsentences , Ġwhich Ġis Ġden oted Ġas Ġback - translation . ĠThis Ġway , Ġwe Ġobtain Ġa Ġsynthetic Ġparallel Ġsimplified - ordinary Ġsentences . ĠBoth Ġthe Ġsynthetic Ġsentences Ġand Ġthe Ġavailable Ġparallel Ġdata Ġare Ġused Ġas Ġtraining Ġdata Ġfor Ġthe Ġoriginal ĠN MT Ġsystem . </s> E val uation </s> We Ġevaluate Ġthe Ġperformance Ġof Ġtext Ġsimpl ification Ġusing Ġneural Ġmachine Ġtranslation Ġon Ġavailable Ġparallel Ġsentences Ġand Ġadditional Ġsimplified Ġsentences . </s> Dat as et . ĠWe Ġuse Ġtwo Ġsimpl ification Ġdatasets Ġ( Wiki Small Ġand ĠWiki Large ). ĠWiki Small Ġconsists Ġof Ġordinary Ġand Ġsimplified Ġsentences Ġfrom Ġthe Ġordinary Ġand Ġsimple ĠEnglish ĠWik iped ias , Ġwhich Ġhas Ġbeen Ġused Ġas Ġbenchmark Ġfor Ġevaluating Ġtext Ġsimpl ification ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 8 Ġ. ĠThe Ġtraining Ġset Ġhas Ġ89 , 04 2 Ġsentence Ġpairs , Ġand Ġthe Ġtest Ġset Ġhas Ġ100 Ġpairs . ĠWiki Large Ġis Ġalso Ġfrom ĠWikipedia Ġcorpus Ġwhose Ġtraining Ġset Ġcontains Ġ296 , 402 Ġsentence Ġpairs ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ. ĠWiki Large Ġincludes Ġ8 Ġ( reference ) Ġsimpl ifications Ġfor Ġ2 , 359 Ġsentences Ġsplit Ġinto Ġ2 , 000 Ġfor Ġdevelopment Ġand Ġ359 Ġfor Ġtesting . </s> Met rics . ĠThree Ġmetrics Ġin Ġtext Ġsimpl ification Ġare Ġchosen Ġin Ġthis Ġpaper . ĠB LE U ĠB IB REF 5 Ġis Ġone Ġtraditional Ġmachine Ġtranslation Ġmetric Ġto Ġassess Ġthe Ġdegree Ġto Ġwhich Ġtranslated Ġsimpl ifications Ġdiffered Ġfrom Ġreference Ġsimpl ifications . ĠF K GL Ġmeasures Ġthe Ġread ability Ġof Ġthe Ġoutput ĠB IB REF 23 Ġ. ĠA Ġsmall ĠF K GL Ġrepresents Ġsimpler Ġoutput . ĠS ARI Ġis Ġa Ġrecent Ġtext - sim pl ification Ġmetric Ġby Ġcomparing Ġthe Ġoutput Ġagainst Ġthe Ġsource Ġand Ġreference Ġsimpl ifications ĠB IB REF 20 Ġ. </s> We Ġevaluate Ġthe Ġoutput Ġof Ġall Ġsystems Ġusing Ġhuman Ġevaluation . ĠThe Ġmetric Ġis Ġden oted Ġas ĠSimpl icity ĠB IB REF 8 Ġ. ĠThe Ġthree Ġnon - native Ġfluent ĠEnglish Ġspeakers Ġare Ġshown Ġreference Ġsentences Ġand Ġoutput Ġsentences . ĠThey Ġare Ġasked Ġwhether Ġthe Ġoutput Ġsentence Ġis Ġmuch Ġsimpler Ġ(+ 2 ), Ġsomewhat Ġsimpler Ġ(+ 1 ), Ġequally Ġ( 0 ), Ġsomewhat Ġmore Ġdifficult Ġ(- 1 ), Ġand Ġmuch Ġmore Ġdifficult Ġ(- 2 ) Ġthan Ġthe Ġreference Ġsentence . </s> Methods . ĠWe Ġuse ĠOpen N MT ĠB IB REF 24 Ġas Ġthe Ġimplementation Ġof Ġthe ĠN MT Ġsystem Ġfor Ġall Ġexperiments ĠB IB REF 5 Ġ. ĠWe Ġgenerally Ġfollow Ġthe Ġdefault Ġsettings Ġand Ġtraining Ġprocedure Ġdescribed Ġby ĠKlein Ġet Ġal .( 2017 ). ĠWe Ġreplace Ġout - of - voc abulary Ġwords Ġwith Ġa Ġspecial ĠUN K Ġsymbol . ĠAt Ġprediction Ġtime , Ġwe Ġreplace ĠUN K Ġwords Ġwith Ġthe Ġhighest Ġprobability Ġscore Ġfrom Ġthe Ġattention Ġlayer . ĠOpen N MT Ġsystem Ġused Ġon Ġparallel Ġdata Ġis Ġthe Ġbaseline Ġsystem . ĠTo Ġobtain Ġa Ġsynthetic Ġparallel Ġtraining Ġset , Ġwe Ġback - trans late Ġa Ġrandom Ġsample Ġof Ġ100 K Ġsentences Ġfrom Ġthe Ġcollected Ġsimplified Ġcorpor a . ĠOpen N MT Ġused Ġon Ġparallel Ġdata Ġand Ġsynthetic Ġdata Ġis Ġour Ġmodel . ĠThe Ġbenchmarks Ġare Ġrun Ġon Ġa ĠIntel ( R ) ĠCore ( TM ) Ġi 7 - 59 30 K ĠCPU @ 3 . 50 GHz , Ġ32 GB ĠMem , Ġtrained Ġon Ġ1 ĠGPU ĠGeForce ĠGTX Ġ1080 Ġ( P ascal ) Ġwith ĠCU DA Ġv . Ġ8 . 0 . </s> We Ġchoose Ġthree Ġstatistical Ġtext Ġsimpl ification Ġsystems . ĠP BM T - R Ġis Ġa Ġphrase - based Ġmethod Ġwith Ġa Ġre ranking Ġpost - processing Ġstep ĠB IB REF 18 Ġ. ĠHybrid Ġperforms Ġsentence Ġsplitting Ġand Ġdeletion Ġoperations Ġbased Ġon Ġdiscourse Ġrepresentation Ġstructures , Ġand Ġthen Ġsimpl ifies Ġsentences Ġwith ĠP BM T - R ĠB IB REF 25 Ġ. ĠS BM T - S ARI ĠB IB REF 19 Ġis Ġsyntax - based Ġtranslation Ġmodel Ġusing ĠP PD B Ġparaph rase Ġdatabase ĠB IB REF 26 Ġand Ġmod ifies Ġtuning Ġfunction Ġ( using ĠS ARI ). ĠWe Ġchoose Ġtwo Ġneural Ġtext Ġsimpl ification Ġsystems . ĠN MT Ġis Ġa Ġbasic Ġattention - based Ġenc oder - dec oder Ġmodel Ġwhich Ġuses ĠOpen N MT Ġframework Ġto Ġtrain Ġwith Ġtwo ĠL ST M Ġlayers , Ġhidden Ġstates Ġof Ġsize Ġ500 Ġand Ġ500 Ġhidden Ġunits , ĠSG D Ġoptim izer , Ġand Ġa Ġdrop out Ġrate Ġof Ġ0 . 3 ĠB IB REF 8 Ġ. ĠDress Ġis Ġan Ġenc oder - dec oder Ġmodel Ġcoupled Ġwith Ġa Ġdeep Ġreinforcement Ġlearning Ġframework , Ġand Ġthe Ġparameters Ġare Ġchosen Ġaccording Ġto Ġthe Ġoriginal Ġpaper ĠB IB REF 20 Ġ. ĠFor Ġthe Ġexperiments Ġwith Ġsynthetic Ġparallel Ġdata , Ġwe Ġback - trans late Ġa Ġrandom Ġsample Ġof Ġ60 Ġ000 Ġsentences Ġfrom Ġthe Ġcollected Ġsimplified Ġsentences Ġinto Ġordinary Ġsentences . ĠOur Ġmodel Ġis Ġtrained Ġon Ġsynthetic Ġdata Ġand Ġthe Ġavailable Ġparallel Ġdata , Ġden oted Ġas ĠN MT + sy nt hetic . </s> Results . ĠTable Ġ1 Ġshows Ġthe Ġresults Ġof Ġall Ġmodels Ġon ĠWiki Large Ġdataset . ĠWe Ġcan Ġsee Ġthat Ġour Ġmethod Ġ( N MT + sy nt hetic ) Ġcan Ġobtain Ġhigher ĠB LE U , Ġlower ĠF K GL Ġand Ġhigh ĠS ARI Ġcompared Ġwith Ġother Ġmodels , Ġexcept ĠDress Ġon ĠF K GL Ġand ĠS BM T - S ARI Ġon ĠS ARI . ĠIt Ġverified Ġthat Ġincluding Ġsynthetic Ġdata Ġduring Ġtraining Ġis Ġvery Ġeffective , Ġand Ġyields Ġan Ġimprovement Ġover Ġour Ġbaseline ĠNM F Ġby Ġ2 . 11 ĠB LE U , Ġ1 . 7 ĠF K GL Ġand Ġ1 . 07 ĠS ARI . ĠWe Ġalso Ġsubstantially Ġoutper form ĠDress , Ġwho Ġpreviously Ġreported ĠS OTA Ġresult . ĠThe Ġresults Ġof Ġour Ġhuman Ġevaluation Ġusing ĠSimpl icity Ġare Ġalso Ġpresented Ġin ĠTable Ġ1 . ĠN MT Ġon Ġsynthetic Ġdata Ġis Ġsignificantly Ġbetter Ġthan ĠP BM T - R , ĠDress , Ġand ĠS BM T - S ARI Ġon ĠSimpl icity . ĠIt Ġindicates Ġthat Ġour Ġmethod Ġwith Ġsimplified Ġdata Ġis Ġeffective Ġat Ġcreating Ġsimpler Ġoutput . </s> Results Ġon ĠWiki Small Ġdataset Ġare Ġshown Ġin ĠTable Ġ2 . ĠWe Ġsee Ġsubstantial Ġimprovements Ġ( 6 . 37 ĠB LE U ) Ġthan ĠN MT Ġfrom Ġadding Ġsimplified Ġtraining Ġdata Ġwith Ġsynthetic Ġordinary Ġsentences . ĠCompared Ġwith Ġstatistical Ġmachine Ġtranslation Ġmodels Ġ( P BM T - R , ĠHybrid , ĠS BM T - S ARI ), Ġour Ġmethod Ġ( N MT + sy nt hetic ) Ġstill Ġhave Ġbetter Ġresults , Ġbut Ġslightly Ġworse ĠF K GL Ġand ĠS ARI . ĠSimilar Ġto Ġthe Ġresults Ġin ĠWiki Large , Ġthe Ġresults Ġof Ġour Ġhuman Ġevaluation Ġusing ĠSimpl icity Ġoutper forms Ġthe Ġother Ġmodels . ĠIn Ġconclusion , ĠOur Ġmethod Ġproduces Ġbetter Ġresults Ġcomparing Ġwith Ġthe Ġbas elines , Ġwhich Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġadding Ġsimplified Ġtraining Ġdata . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġpropose Ġone Ġsimple Ġmethod Ġto Ġuse Ġsimplified Ġcorpor a Ġduring Ġtraining Ġof ĠN MT Ġsystems , Ġwith Ġno Ġchanges Ġto Ġthe Ġnetwork Ġarchitecture . ĠIn Ġthe Ġexperiments Ġon Ġtwo Ġdatasets , Ġwe Ġachieve Ġsubstantial Ġgains Ġin Ġall Ġtasks , Ġand Ġnew ĠS OTA Ġresults , Ġvia Ġback - translation Ġof Ġsimplified Ġsentences Ġinto Ġthe Ġordinary Ġsentences , Ġand Ġtreating Ġthis Ġsynthetic Ġdata Ġas Ġadditional Ġtraining Ġdata . ĠBecause Ġwe Ġdo Ġnot Ġchange Ġthe Ġneural Ġnetwork Ġarchitecture Ġto Ġintegrate Ġsimplified Ġcorpor a , Ġour Ġmethod Ġcan Ġbe Ġeasily Ġapplied Ġto Ġother ĠNeural ĠText ĠSimpl ification Ġ( N TS ) Ġsystems . ĠWe Ġexpect Ġthat Ġthe Ġeffectiveness Ġof Ġour Ġmethod Ġnot Ġonly Ġvaries Ġwith Ġthe Ġquality Ġof Ġthe ĠNTS Ġsystem Ġused Ġfor Ġback - translation , Ġbut Ġalso Ġdepends Ġon Ġthe Ġamount Ġof Ġavailable Ġparallel Ġand Ġsimplified Ġcorpor a . ĠIn Ġthe Ġpaper , Ġwe Ġhave Ġonly Ġutilized Ġdata Ġfrom ĠWikipedia Ġfor Ġsimplified Ġsentences . ĠIn Ġthe Ġfuture , Ġmany Ġother Ġtext Ġsources Ġare Ġavailable Ġand Ġthe Ġimpact Ġof Ġnot Ġonly Ġsize , Ġbut Ġalso Ġof Ġdomain Ġshould Ġbe Ġinvestigated .