(VVD shop_VV0 II, VVD shopping_VVG II)
EXTRACTIVE
patterns for generating all types of errors
EXTRACTIVE
For example, the original sentence `We went shop on Saturday' and the corrected version `We went shopping on Saturday' would produce the following pattern: (VVD shop_VV0 II, VVD shopping_VVG II) After collecting statistics from the background corpus, errors can be inserted into error-free text. The learned patterns are now reversed, looking for the correct side of the tuple in the input sentence. We only use patterns with frequency INLINEFORM0 , which yields a total of 35,625 patterns from our training data. For each input sentence, we first decide how many errors will be generated (using probabilities from the background corpus) and attempt to create them by sampling from the collection of applicable patterns. This process is repeated until all the required errors have been generated or the sentence is exhausted. During generation, we try to balance the distribution of error types as well as keeping the same proportion of incorrect and correct sentences as in the background corpus BIBREF10 . The required POS tags were generated with RASP BIBREF11 , using the CLAWS2 tagset.
We also describe a method for AEG using patterns over words and part-of-speech (POS) tags, extracting known incorrect sequences from a corpus of annotated corrections. This approach is based on the best method identified by Felice2014a, using error type distributions; while they covered only 5 error types, we relax this restriction and learn patterns for generating all types of errors. The original and corrected sentences in the corpus are aligned and used to identify short transformation patterns in the form of (incorrect phrase, correct phrase). The length of each pattern is the affected phrase, plus up to one token of context on both sides. If a word form changes between the incorrect and correct text, it is fully saved in the pattern, otherwise the POS tags are used for matching.
1707.05236
Introduction </s> Writing Ġerrors Ġcan Ġoccur Ġin Ġmany Ġdifferent Ġforms ĠâĢĵ Ġfrom Ġrelatively Ġsimple Ġpunct uation Ġand Ġdetermin er Ġerrors , Ġto Ġmistakes Ġincluding Ġword Ġtense Ġand Ġform , Ġincorrect Ġcoll ocations Ġand Ġerroneous Ġidi oms . ĠAutom atically Ġidentifying Ġall Ġof Ġthese Ġerrors Ġis Ġa Ġchallenging Ġtask , Ġespecially Ġas Ġthe Ġamount Ġof Ġavailable Ġannot ated Ġdata Ġis Ġvery Ġlimited . ĠRei 2016 Ġshowed Ġthat Ġwhile Ġsome Ġerror Ġdetection Ġalgorithms Ġperform Ġbetter Ġthan Ġothers , Ġit Ġis Ġadditional Ġtraining Ġdata Ġthat Ġhas Ġthe Ġbiggest Ġimpact Ġon Ġimproving Ġperformance . </s> Being Ġable Ġto Ġgenerate Ġrealistic Ġartificial Ġdata Ġwould Ġallow Ġfor Ġany Ġgram matically Ġcorrect Ġtext Ġto Ġbe Ġtransformed Ġinto Ġannot ated Ġexamples Ġcontaining Ġwriting Ġerrors , Ġproducing Ġlarge Ġamounts Ġof Ġadditional Ġtraining Ġexamples . ĠSuper vised Ġerror Ġgeneration Ġsystems Ġwould Ġalso Ġprovide Ġan Ġefficient Ġmethod Ġfor Ġanonym ising Ġthe Ġsource Ġcorpus ĠâĢĵ Ġerror Ġstatistics Ġfrom Ġa Ġprivate Ġcorpus Ġcan Ġbe Ġaggreg ated Ġand Ġapplied Ġto Ġa Ġdifferent Ġtarget Ġtext , Ġobsc uring Ġsensitive Ġinformation Ġin Ġthe Ġoriginal Ġexamination Ġscripts . ĠHowever , Ġthe Ġtask Ġof Ġcreating Ġincorrect Ġdata Ġis Ġsomewhat Ġmore Ġdifficult Ġthan Ġmight Ġinitially Ġappear ĠâĢĵ Ġnaive Ġmethods Ġfor Ġerror Ġgeneration Ġcan Ġcreate Ġdata Ġthat Ġdoes Ġnot Ġresemble Ġnatural Ġerrors , Ġthereby Ġmaking Ġdownstream Ġsystems Ġlearn Ġmisleading Ġor Ġunin form ative Ġpatterns . </s> Previous Ġwork Ġon Ġartificial Ġerror Ġgeneration Ġ( A EG ) Ġhas Ġfocused Ġon Ġspecific Ġerror Ġtypes , Ġsuch Ġas Ġpre pos itions Ġand Ġdetermin ers ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġor Ġnoun Ġnumber Ġerrors ĠB IB REF 2 Ġ. ĠFel ice 2014 a Ġinvestigated Ġthe Ġuse Ġof Ġlinguistic Ġinformation Ġwhen Ġgenerating Ġartificial Ġdata Ġfor Ġerror Ġcorrection , Ġbut Ġalso Ġrestricting Ġthe Ġapproach Ġto Ġonly Ġfive Ġerror Ġtypes . ĠThere Ġhas Ġbeen Ġvery Ġlimited Ġresearch Ġon Ġgenerating Ġartificial Ġdata Ġfor Ġall Ġtypes , Ġwhich Ġis Ġimportant Ġfor Ġgeneral - purpose Ġerror Ġdetection Ġsystems . ĠFor Ġexample , Ġthe Ġerror Ġtypes Ġinvestigated Ġby ĠFel ice 2014 a Ġcover Ġonly Ġ35 . 74 % Ġof Ġall Ġerrors Ġpresent Ġin Ġthe ĠCo N LL Ġ2014 Ġtraining Ġdataset , Ġproviding Ġno Ġadditional Ġinformation Ġfor Ġthe Ġmajority Ġof Ġerrors . </s> In Ġthis Ġpaper , Ġwe Ġinvestigate Ġtwo Ġsupervised Ġapproaches Ġfor Ġgenerating Ġall Ġtypes Ġof Ġartificial Ġerrors . ĠWe Ġpropose Ġa Ġframework Ġfor Ġgenerating Ġerrors Ġbased Ġon Ġstatistical Ġmachine Ġtranslation Ġ( SM T ), Ġtraining Ġa Ġmodel Ġto Ġtranslate Ġfrom Ġcorrect Ġinto Ġincorrect Ġsentences . ĠIn Ġaddition , Ġwe Ġdescribe Ġa Ġmethod Ġfor Ġlearning Ġerror Ġpatterns Ġfrom Ġan Ġannot ated Ġcorpus Ġand Ġtransplant ing Ġthem Ġinto Ġerror - free Ġtext . ĠWe Ġevaluate Ġthe Ġeffect Ġof Ġintroducing Ġartificial Ġdata Ġon Ġtwo Ġerror Ġdetection Ġbenchmarks . ĠOur Ġresults Ġshow Ġthat Ġeach Ġmethod Ġprovides Ġsignificant Ġimprovements Ġover Ġusing Ġonly Ġthe Ġavailable Ġtraining Ġset , Ġand Ġa Ġcombination Ġof Ġboth Ġgives Ġan Ġabsolute Ġimprovement Ġof Ġ4 . 3 % Ġin ĠIN LINE FORM 0 Ġ, Ġwithout Ġrequiring Ġany Ġadditional Ġannot ated Ġdata . </s> Error ĠGeneration ĠMethods </s> We Ġinvestigate Ġtwo Ġalternative Ġmethods Ġfor ĠA EG . ĠThe Ġmodels Ġreceive Ġgram matically Ġcorrect Ġtext Ġas Ġinput Ġand Ġmodify Ġcertain Ġtokens Ġto Ġproduce Ġincorrect Ġsequences . ĠThe Ġalternative Ġversions Ġof Ġeach Ġsentence Ġare Ġaligned Ġusing ĠLe ven sh tein Ġdistance , Ġallowing Ġus Ġto Ġidentify Ġspecific Ġwords Ġthat Ġneed Ġto Ġbe Ġmarked Ġas Ġerrors . ĠWhile Ġthese Ġalign ments Ġare Ġnot Ġalways Ġperfect , Ġwe Ġfound Ġthem Ġto Ġbe Ġsufficient Ġfor Ġpractical Ġpurposes , Ġsince Ġalternative Ġalign ments Ġof Ġsimilar Ġsentences Ġoften Ġresult Ġin Ġthe Ġsame Ġbinary Ġlabeling . ĠFuture Ġwork Ġcould Ġexplore Ġmore Ġadvanced Ġalignment Ġmethods , Ġsuch Ġas Ġproposed Ġby Ġfel ice - b ry ant - br isco e . </s> In ĠSection ĠSEC REF 4 Ġ, Ġthis Ġautomatically Ġlabeled Ġdata Ġis Ġthen Ġused Ġfor Ġtraining Ġerror Ġdetection Ġmodels . </s> Machine ĠTranslation </s> We Ġtreat ĠA EG Ġas Ġa Ġtranslation Ġtask ĠâĢĵ Ġgiven Ġa Ġcorrect Ġsentence Ġas Ġinput , Ġthe Ġsystem Ġwould Ġlearn Ġto Ġtranslate Ġit Ġto Ġcontain Ġlikely Ġerrors , Ġbased Ġon Ġa Ġtraining Ġcorpus Ġof Ġparallel Ġdata . ĠEx isting ĠSM T Ġapproaches Ġare Ġalready Ġoptim ised Ġfor Ġidentifying Ġcontext Ġpatterns Ġthat Ġcorrespond Ġto Ġspecific Ġoutput Ġsequences , Ġwhich Ġis Ġalso Ġrequired Ġfor Ġgenerating Ġhuman - like Ġerrors . ĠThe Ġreverse Ġof Ġthis Ġidea , Ġtranslating Ġfrom Ġincorrect Ġto Ġcorrect Ġsentences , Ġhas Ġbeen Ġshown Ġto Ġwork Ġwell Ġfor Ġerror Ġcorrection Ġtasks ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġand Ġround - trip Ġtranslation Ġhas Ġalso Ġbeen Ġshown Ġto Ġbe Ġpromising Ġfor Ġcorrecting Ġgram matical Ġerrors ĠB IB REF 4 Ġ. </s> Following Ġprevious Ġwork ĠB IB REF 2 Ġ, ĠB IB REF 5 Ġ, Ġwe Ġbuild Ġa Ġphrase - based ĠSM T Ġerror Ġgeneration Ġsystem . ĠDuring Ġtraining , Ġerror - correct ed Ġsentences Ġin Ġthe Ġtraining Ġdata Ġare Ġtreated Ġas Ġthe Ġsource , Ġand Ġthe Ġoriginal Ġsentences Ġwritten Ġby Ġlanguage Ġlearners Ġas Ġthe Ġtarget . ĠP ial ign ĠB IB REF 6 Ġis Ġused Ġto Ġcreate Ġa Ġphrase Ġtranslation Ġtable Ġdirectly Ġfrom Ġmodel Ġprobabilities . ĠIn Ġaddition Ġto Ġdefault Ġfeatures , Ġwe Ġadd Ġcharacter - level ĠLe ven sh tein Ġdistance Ġto Ġeach Ġmapping Ġin Ġthe Ġphrase Ġtable , Ġas Ġproposed Ġby ĠFel ice : 2014 - Co N LL . ĠDec oding Ġis Ġperformed Ġusing ĠMoses ĠB IB REF 7 Ġand Ġthe Ġlanguage Ġmodel Ġused Ġduring Ġdecoding Ġis Ġbuilt Ġfrom Ġthe Ġoriginal Ġerroneous Ġsentences Ġin Ġthe Ġlear ner Ġcorpus . ĠThe ĠIR ST LM ĠTool kit ĠB IB REF 8 Ġis Ġused Ġfor Ġbuilding Ġa Ġ5 - gram Ġlanguage Ġmodel Ġwith Ġmodified ĠK nes er - N ey Ġsmoot hing ĠB IB REF 9 Ġ. </s> Pattern ĠExt raction </s> We Ġalso Ġdescribe Ġa Ġmethod Ġfor ĠA EG Ġusing Ġpatterns Ġover Ġwords Ġand Ġpart - of - speech Ġ( POS ) Ġtags , Ġextracting Ġknown Ġincorrect Ġsequences Ġfrom Ġa Ġcorpus Ġof Ġannot ated Ġcorrections . ĠThis Ġapproach Ġis Ġbased Ġon Ġthe Ġbest Ġmethod Ġidentified Ġby ĠFel ice 2014 a , Ġusing Ġerror Ġtype Ġdistributions ; Ġwhile Ġthey Ġcovered Ġonly Ġ5 Ġerror Ġtypes , Ġwe Ġrelax Ġthis Ġrestriction Ġand Ġlearn Ġpatterns Ġfor Ġgenerating Ġall Ġtypes Ġof Ġerrors . </s> The Ġoriginal Ġand Ġcorrected Ġsentences Ġin Ġthe Ġcorpus Ġare Ġaligned Ġand Ġused Ġto Ġidentify Ġshort Ġtransformation Ġpatterns Ġin Ġthe Ġform Ġof Ġ( inc orrect Ġphrase , Ġcorrect Ġphrase ). ĠThe Ġlength Ġof Ġeach Ġpattern Ġis Ġthe Ġaffected Ġphrase , Ġplus Ġup Ġto Ġone Ġtoken Ġof Ġcontext Ġon Ġboth Ġsides . ĠIf Ġa Ġword Ġform Ġchanges Ġbetween Ġthe Ġincorrect Ġand Ġcorrect Ġtext , Ġit Ġis Ġfully Ġsaved Ġin Ġthe Ġpattern , Ġotherwise Ġthe ĠPOS Ġtags Ġare Ġused Ġfor Ġmatching . </s> For Ġexample , Ġthe Ġoriginal Ġsentence Ġ` We Ġwent Ġshop Ġon ĠSaturday ' Ġand Ġthe Ġcorrected Ġversion Ġ` We Ġwent Ġshopping Ġon ĠSaturday ' Ġwould Ġproduce Ġthe Ġfollowing Ġpattern : </s> ( V VD Ġshop _ V V 0 ĠII , ĠV VD Ġshopping _ V VG ĠII ) </s> After Ġcollecting Ġstatistics Ġfrom Ġthe Ġbackground Ġcorpus , Ġerrors Ġcan Ġbe Ġinserted Ġinto Ġerror - free Ġtext . ĠThe Ġlearned Ġpatterns Ġare Ġnow Ġreversed , Ġlooking Ġfor Ġthe Ġcorrect Ġside Ġof Ġthe Ġtuple Ġin Ġthe Ġinput Ġsentence . ĠWe Ġonly Ġuse Ġpatterns Ġwith Ġfrequency ĠIN LINE FORM 0 Ġ, Ġwhich Ġyields Ġa Ġtotal Ġof Ġ35 , 625 Ġpatterns Ġfrom Ġour Ġtraining Ġdata . ĠFor Ġeach Ġinput Ġsentence , Ġwe Ġfirst Ġdecide Ġhow Ġmany Ġerrors Ġwill Ġbe Ġgenerated Ġ( using Ġprobabilities Ġfrom Ġthe Ġbackground Ġcorpus ) Ġand Ġattempt Ġto Ġcreate Ġthem Ġby Ġsampling Ġfrom Ġthe Ġcollection Ġof Ġapplicable Ġpatterns . ĠThis Ġprocess Ġis Ġrepeated Ġuntil Ġall Ġthe Ġrequired Ġerrors Ġhave Ġbeen Ġgenerated Ġor Ġthe Ġsentence Ġis Ġexhausted . ĠDuring Ġgeneration , Ġwe Ġtry Ġto Ġbalance Ġthe Ġdistribution Ġof Ġerror Ġtypes Ġas Ġwell Ġas Ġkeeping Ġthe Ġsame Ġproportion Ġof Ġincorrect Ġand Ġcorrect Ġsentences Ġas Ġin Ġthe Ġbackground Ġcorpus ĠB IB REF 10 Ġ. ĠThe Ġrequired ĠPOS Ġtags Ġwere Ġgenerated Ġwith ĠR AS P ĠB IB REF 11 Ġ, Ġusing Ġthe ĠCL AW S 2 Ġtag set . </s> Error ĠDetection ĠModel </s> We Ġconstruct Ġa Ġneural Ġsequence Ġlabeling Ġmodel Ġfor Ġerror Ġdetection , Ġfollowing Ġthe Ġprevious Ġwork ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠThe Ġmodel Ġreceives Ġa Ġsequence Ġof Ġtokens Ġas Ġinput Ġand Ġoutputs Ġa Ġprediction Ġfor Ġeach Ġposition , Ġindicating Ġwhether Ġthe Ġtoken Ġis Ġcorrect Ġor Ġincorrect Ġin Ġthe Ġcurrent Ġcontext . ĠThe Ġtokens Ġare Ġfirst Ġmapped Ġto Ġa Ġdistributed Ġvector Ġspace , Ġresulting Ġin Ġa Ġsequence Ġof Ġword Ġembed d ings . ĠNext , Ġthe Ġembed d ings Ġare Ġgiven Ġas Ġinput Ġto Ġa Ġbid irection al ĠL ST M ĠB IB REF 14 Ġ, Ġin Ġorder Ġto Ġcreate Ġcontext - dependent Ġrepresentations Ġfor Ġevery Ġtoken . ĠThe Ġhidden Ġstates Ġfrom Ġforward - Ġand Ġbackward - L ST Ms Ġare Ġconc aten ated Ġfor Ġeach Ġword Ġposition , Ġresulting Ġin Ġrepresentations Ġthat Ġare Ġconditioned Ġon Ġthe Ġwhole Ġsequence . ĠThis Ġconc aten ated Ġvector Ġis Ġthen Ġpassed Ġthrough Ġan Ġadditional Ġfeed forward Ġlayer , Ġand Ġa Ġsoft max Ġover Ġthe Ġtwo Ġpossible Ġlabels Ġ( correct Ġand Ġincorrect ) Ġis Ġused Ġto Ġoutput Ġa Ġprobability Ġdistribution Ġfor Ġeach Ġtoken . ĠThe Ġmodel Ġis Ġoptim ised Ġby Ġminim ising Ġcateg orical Ġcross - ent ropy Ġwith Ġrespect Ġto Ġthe Ġcorrect Ġlabels . ĠWe Ġuse ĠAda Delta ĠB IB REF 15 Ġfor Ġcalculating Ġan Ġadaptive Ġlearning Ġrate Ġduring Ġtraining , Ġwhich Ġaccounts Ġfor Ġa Ġhigher Ġbaseline Ġperformance Ġcompared Ġto Ġprevious Ġresults . </s> E val uation </s> We Ġtrained Ġour Ġerror Ġgeneration Ġmodels Ġon Ġthe Ġpublic ĠF CE Ġtraining Ġset ĠB IB REF 16 Ġand Ġused Ġthem Ġto Ġgenerate Ġadditional Ġartificial Ġtraining Ġdata . ĠGram matically Ġcorrect Ġtext Ġis Ġneeded Ġas Ġthe Ġstarting Ġpoint Ġfor Ġinserting Ġartificial Ġerrors , Ġand Ġwe Ġused Ġtwo Ġdifferent Ġsources : Ġ1 ) Ġthe Ġcorrected Ġversion Ġof Ġthe Ġsame ĠF CE Ġtraining Ġset Ġon Ġwhich Ġthe Ġsystem Ġis Ġtrained Ġ( 450 K Ġtokens ), Ġand Ġ2 ) Ġexample Ġsentences Ġextracted Ġfrom Ġthe ĠEnglish ĠVoc abulary ĠProfile Ġ( 270 K Ġtokens ) .. ĠWhile Ġthere Ġare Ġother Ġtext Ġcorpor a Ġthat Ġcould Ġbe Ġused Ġ( e . g ., ĠWikipedia Ġand Ġnews Ġarticles ), Ġour Ġdevelopment Ġexperiments Ġshowed Ġthat Ġkeeping Ġthe Ġwriting Ġstyle Ġand Ġvocabulary Ġclose Ġto Ġthe Ġtarget Ġdomain Ġgives Ġbetter Ġresults Ġcompared Ġto Ġsimply Ġincluding Ġmore Ġdata . </s> We Ġevaluated Ġour Ġdetection Ġmodels Ġon Ġthree Ġbenchmarks : Ġthe ĠF CE Ġtest Ġdata Ġ( 41 K Ġtokens ) Ġand Ġthe Ġtwo Ġalternative Ġannotations Ġof Ġthe ĠCo N LL Ġ2014 ĠShared ĠTask Ġdataset Ġ( 30 K Ġtokens ) ĠB IB REF 3 Ġ. ĠEach Ġartificial Ġerror Ġgeneration Ġsystem Ġwas Ġused Ġto Ġgenerate Ġ3 Ġdifferent Ġversions Ġof Ġthe Ġartificial Ġdata , Ġwhich Ġwere Ġthen Ġcombined Ġwith Ġthe Ġoriginal Ġannot ated Ġdataset Ġand Ġused Ġfor Ġtraining Ġan Ġerror Ġdetection Ġsystem . ĠTable ĠT AB REF 1 Ġcontains Ġexample Ġsentences Ġfrom Ġthe Ġerror Ġgeneration Ġsystems , Ġhighlighting Ġeach Ġof Ġthe Ġedits Ġthat Ġare Ġmarked Ġas Ġerrors . </s> The Ġerror Ġdetection Ġresults Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 4 Ġ. ĠWe Ġuse ĠIN LINE FORM 0 Ġas Ġthe Ġmain Ġevaluation Ġmeasure , Ġwhich Ġwas Ġestablished Ġas Ġthe Ġpreferred Ġmeasure Ġfor Ġerror Ġcorrection Ġand Ġdetection Ġby Ġthe ĠCo N LL - 14 Ġshared Ġtask ĠB IB REF 3 Ġ. ĠIN LINE FORM 1 Ġcalculates Ġa Ġweighted Ġharmonic Ġmean Ġof Ġprecision Ġand Ġrecall , Ġwhich Ġassigns Ġtwice Ġas Ġmuch Ġimportance Ġto Ġprecision ĠâĢĵ Ġthis Ġis Ġmotivated Ġby Ġpractical Ġapplications , Ġwhere Ġaccurate Ġpredictions Ġfrom Ġan Ġerror Ġdetection Ġsystem Ġare Ġmore Ġimportant Ġcompared Ġto Ġcoverage . ĠFor Ġcomparison , Ġwe Ġalso Ġreport Ġthe Ġperformance Ġof Ġthe Ġerror Ġdetection Ġsystem Ġby ĠRei 2016 , Ġtrained Ġusing Ġthe Ġsame ĠF CE Ġdataset . </s> The Ġresults Ġshow Ġthat Ġerror Ġdetection Ġperformance Ġis Ġsubstantially Ġimproved Ġby Ġmaking Ġuse Ġof Ġartificially Ġgenerated Ġdata , Ġcreated Ġby Ġany Ġof Ġthe Ġdescribed Ġmethods . ĠWhen Ġcomparing Ġthe Ġerror Ġgeneration Ġsystem Ġby ĠFel ice 2014 a Ġ( FY 14 ) Ġwith Ġour Ġpattern - based Ġ( P AT ) Ġand Ġmachine Ġtranslation Ġ( MT ) Ġapproaches , Ġwe Ġsee Ġthat Ġthe Ġlatter Ġmethods Ġcovering Ġall Ġerror Ġtypes Ġconsistently Ġimprove Ġperformance . ĠWhile Ġthe Ġadded Ġerror Ġtypes Ġtend Ġto Ġbe Ġless Ġfrequent Ġand Ġmore Ġcomplicated Ġto Ġcapture , Ġthe Ġadded Ġcoverage Ġis Ġindeed Ġbeneficial Ġfor Ġerror Ġdetection . ĠComb ining Ġthe Ġpattern - based Ġapproach Ġwith Ġthe Ġmachine Ġtranslation Ġsystem Ġ( Ann + P AT + MT ) Ġgave Ġthe Ġbest Ġoverall Ġperformance Ġon Ġall Ġdatasets . ĠThe Ġtwo Ġframeworks Ġlearn Ġto Ġgenerate Ġdifferent Ġtypes Ġof Ġerrors , Ġand Ġtaking Ġadvantage Ġof Ġboth Ġleads Ġto Ġsubstantial Ġimprovements Ġin Ġerror Ġdetection . </s> We Ġused Ġthe ĠApp rox imate ĠRandom isation ĠTest ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġto Ġcalculate Ġstatistical Ġsignificance Ġand Ġfound Ġthat Ġthe Ġimprovement Ġfor Ġeach Ġof Ġthe Ġsystems Ġusing Ġartificial Ġdata Ġwas Ġsignificant Ġover Ġusing Ġonly Ġmanual Ġannotation . ĠIn Ġaddition , Ġthe Ġfinal Ġcombination Ġsystem Ġis Ġalso Ġsignificantly Ġbetter Ġcompared Ġto Ġthe ĠFel ice 2014 a Ġsystem , Ġon Ġall Ġthree Ġdatasets . ĠWhile ĠRei 2016 Ġalso Ġreport Ġseparate Ġexperiments Ġthat Ġachieve Ġeven Ġhigher Ġperformance , Ġthese Ġmodels Ġwere Ġtrained Ġon Ġa Ġconsiderably Ġlarger Ġproprietary Ġcorpus . ĠIn Ġthis Ġpaper Ġwe Ġcompare Ġerror Ġdetection Ġframeworks Ġtrained Ġon Ġthe Ġsame Ġpublicly Ġavailable ĠF CE Ġdataset , Ġthereby Ġremoving Ġthe Ġconfounding Ġfactor Ġof Ġdataset Ġsize Ġand Ġonly Ġfocusing Ġon Ġthe Ġmodel Ġarchitectures . </s> The Ġerror Ġgeneration Ġmethods Ġcan Ġgenerate Ġalternative Ġversions Ġof Ġthe Ġsame Ġinput Ġtext ĠâĢĵ Ġthe Ġpattern - based Ġmethod Ġrandomly Ġsamples Ġthe Ġerror Ġlocations , Ġand Ġthe ĠSM T Ġsystem Ġcan Ġprovide Ġan Ġn - best Ġlist Ġof Ġalternative Ġtranslations . ĠTherefore , Ġwe Ġalso Ġinvestigated Ġthe Ġcombination Ġof Ġmultiple Ġerror - generated Ġversions Ġof Ġthe Ġinput Ġfiles Ġwhen Ġtraining Ġerror Ġdetection Ġmodels . ĠFigure ĠFIG REF 6 Ġshows Ġthe ĠIN LINE FORM 0 Ġscore Ġon Ġthe Ġdevelopment Ġset , Ġas Ġthe Ġtraining Ġdata Ġis Ġincreased Ġby Ġusing Ġmore Ġtranslations Ġfrom Ġthe Ġn - best Ġlist Ġof Ġthe ĠSM T Ġsystem . ĠThese Ġresults Ġreveal Ġthat Ġallowing Ġthe Ġmodel Ġto Ġsee Ġmultiple Ġalternative Ġversions Ġof Ġthe Ġsame Ġfile Ġgives Ġa Ġdistinct Ġimprovement ĠâĢĵ Ġshowing Ġthe Ġmodel Ġboth Ġcorrect Ġand Ġincorrect Ġvariations Ġof Ġthe Ġsame Ġsentences Ġlikely Ġassists Ġin Ġlearning Ġa Ġdiscrim in ative Ġmodel . </s> Related ĠWork </s> Our Ġwork Ġbuilds Ġon Ġprior Ġresearch Ġinto ĠA EG . ĠBrock ett 2006 Ġconstructed Ġregular Ġexpressions Ġfor Ġtransforming Ġcorrect Ġsentences Ġto Ġcontain Ġnoun Ġnumber Ġerrors . ĠRoz ov s kaya 2010 a Ġlearned Ġconfusion Ġsets Ġfrom Ġan Ġannot ated Ġcorpus Ġin Ġorder Ġto Ġgenerate Ġpre position Ġerrors . ĠFoster 2009 Ġdevised Ġa Ġtool Ġfor Ġgenerating Ġerrors Ġfor Ġdifferent Ġtypes Ġusing Ġpatterns Ġprovided Ġby Ġthe Ġuser Ġor Ġcollected Ġautomatically Ġfrom Ġan Ġannot ated Ġcorpus . ĠHowever , Ġtheir Ġmethod Ġuses Ġa Ġlimited Ġnumber Ġof Ġedit Ġoperations Ġand Ġis Ġthus Ġunable Ġto Ġgenerate Ġcomplex Ġerrors . ĠCah ill 2013 Ġcompared Ġdifferent Ġtraining Ġmethod ologies Ġand Ġshowed Ġthat Ġartificial Ġerrors Ġhelped Ġcorrect Ġpre pos itions . ĠFel ice 2014 a Ġlearned Ġerror Ġtype Ġdistributions Ġfor Ġgenerating Ġfive Ġtypes Ġof Ġerrors , Ġand Ġthe Ġsystem Ġin ĠSection ĠSEC REF 3 Ġis Ġan Ġextension Ġof Ġthis Ġmodel . ĠWhile Ġprevious Ġwork Ġfocused Ġon Ġgenerating Ġa Ġspecific Ġsubset Ġof Ġerror Ġtypes , Ġwe Ġexplored Ġtwo Ġholistic Ġapproaches Ġto ĠA EG Ġand Ġshowed Ġthat Ġthey Ġare Ġable Ġto Ġsignificantly Ġimprove Ġerror Ġdetection Ġperformance . </s> Conclusion </s> This Ġpaper Ġinvestigated Ġtwo ĠA EG Ġmethods , Ġin Ġorder Ġto Ġcreate Ġadditional Ġtraining Ġdata Ġfor Ġerror Ġdetection . ĠFirst , Ġwe Ġexplored Ġa Ġmethod Ġusing Ġtextual Ġpatterns Ġlearned Ġfrom Ġan Ġannot ated Ġcorpus , Ġwhich Ġare Ġused Ġfor Ġinserting Ġerrors Ġinto Ġcorrect Ġinput Ġtext . ĠIn Ġaddition , Ġwe Ġproposed Ġform ulating Ġerror Ġgeneration Ġas Ġan ĠMT Ġframework , Ġlearning Ġto Ġtranslate Ġfrom Ġgram matically Ġcorrect Ġto Ġincorrect Ġsentences . </s> The Ġaddition Ġof Ġartificial Ġdata Ġto Ġthe Ġtraining Ġprocess Ġwas Ġevaluated Ġon Ġthree Ġerror Ġdetection Ġannotations , Ġusing Ġthe ĠF CE Ġand ĠCo N LL Ġ2014 Ġdatasets . ĠMaking Ġuse Ġof Ġartificial Ġdata Ġprovided Ġimprovements Ġfor Ġall Ġdata Ġgeneration Ġmethods . ĠBy Ġrelaxing Ġthe Ġtype Ġrestrictions Ġand Ġgenerating Ġall Ġtypes Ġof Ġerrors , Ġour Ġpattern - based Ġmethod Ġconsistently Ġoutper formed Ġthe Ġsystem Ġby ĠFel ice 2014 a . ĠThe Ġcombination Ġof Ġthe Ġpattern - based Ġmethod Ġwith Ġthe Ġmachine Ġtranslation Ġapproach Ġgave Ġfurther Ġsubstantial Ġimprovements Ġand Ġthe Ġbest Ġperformance Ġon Ġall Ġdatasets .
Introduction Writing errors can occur in many different forms – from relatively simple punctuation and determiner errors, to mistakes including word tense and form, incorrect collocations and erroneous idioms. Automatically identifying all of these errors is a challenging task, especially as the amount of available annotated data is very limited. Rei2016 showed that while some error detection algorithms perform better than others, it is additional training data that has the biggest impact on improving performance. Being able to generate realistic artificial data would allow for any grammatically correct text to be transformed into annotated examples containing writing errors, producing large amounts of additional training examples. Supervised error generation systems would also provide an efficient method for anonymising the source corpus – error statistics from a private corpus can be aggregated and applied to a different target text, obscuring sensitive information in the original examination scripts. However, the task of creating incorrect data is somewhat more difficult than might initially appear – naive methods for error generation can create data that does not resemble natural errors, thereby making downstream systems learn misleading or uninformative patterns. Previous work on artificial error generation (AEG) has focused on specific error types, such as prepositions and determiners BIBREF0 , BIBREF1 , or noun number errors BIBREF2 . Felice2014a investigated the use of linguistic information when generating artificial data for error correction, but also restricting the approach to only five error types. There has been very limited research on generating artificial data for all types, which is important for general-purpose error detection systems. For example, the error types investigated by Felice2014a cover only 35.74% of all errors present in the CoNLL 2014 training dataset, providing no additional information for the majority of errors. In this paper, we investigate two supervised approaches for generating all types of artificial errors. We propose a framework for generating errors based on statistical machine translation (SMT), training a model to translate from correct into incorrect sentences. In addition, we describe a method for learning error patterns from an annotated corpus and transplanting them into error-free text. We evaluate the effect of introducing artificial data on two error detection benchmarks. Our results show that each method provides significant improvements over using only the available training set, and a combination of both gives an absolute improvement of 4.3% in INLINEFORM0 , without requiring any additional annotated data. Error Generation Methods We investigate two alternative methods for AEG. The models receive grammatically correct text as input and modify certain tokens to produce incorrect sequences. The alternative versions of each sentence are aligned using Levenshtein distance, allowing us to identify specific words that need to be marked as errors. While these alignments are not always perfect, we found them to be sufficient for practical purposes, since alternative alignments of similar sentences often result in the same binary labeling. Future work could explore more advanced alignment methods, such as proposed by felice-bryant-briscoe. In Section SECREF4 , this automatically labeled data is then used for training error detection models. Machine Translation We treat AEG as a translation task – given a correct sentence as input, the system would learn to translate it to contain likely errors, based on a training corpus of parallel data. Existing SMT approaches are already optimised for identifying context patterns that correspond to specific output sequences, which is also required for generating human-like errors. The reverse of this idea, translating from incorrect to correct sentences, has been shown to work well for error correction tasks BIBREF2 , BIBREF3 , and round-trip translation has also been shown to be promising for correcting grammatical errors BIBREF4 . Following previous work BIBREF2 , BIBREF5 , we build a phrase-based SMT error generation system. During training, error-corrected sentences in the training data are treated as the source, and the original sentences written by language learners as the target. Pialign BIBREF6 is used to create a phrase translation table directly from model probabilities. In addition to default features, we add character-level Levenshtein distance to each mapping in the phrase table, as proposed by Felice:2014-CoNLL. Decoding is performed using Moses BIBREF7 and the language model used during decoding is built from the original erroneous sentences in the learner corpus. The IRSTLM Toolkit BIBREF8 is used for building a 5-gram language model with modified Kneser-Ney smoothing BIBREF9 . Pattern Extraction We also describe a method for AEG using patterns over words and part-of-speech (POS) tags, extracting known incorrect sequences from a corpus of annotated corrections. This approach is based on the best method identified by Felice2014a, using error type distributions; while they covered only 5 error types, we relax this restriction and learn patterns for generating all types of errors. The original and corrected sentences in the corpus are aligned and used to identify short transformation patterns in the form of (incorrect phrase, correct phrase). The length of each pattern is the affected phrase, plus up to one token of context on both sides. If a word form changes between the incorrect and correct text, it is fully saved in the pattern, otherwise the POS tags are used for matching. For example, the original sentence `We went shop on Saturday' and the corrected version `We went shopping on Saturday' would produce the following pattern: (VVD shop_VV0 II, VVD shopping_VVG II) After collecting statistics from the background corpus, errors can be inserted into error-free text. The learned patterns are now reversed, looking for the correct side of the tuple in the input sentence. We only use patterns with frequency INLINEFORM0 , which yields a total of 35,625 patterns from our training data. For each input sentence, we first decide how many errors will be generated (using probabilities from the background corpus) and attempt to create them by sampling from the collection of applicable patterns. This process is repeated until all the required errors have been generated or the sentence is exhausted. During generation, we try to balance the distribution of error types as well as keeping the same proportion of incorrect and correct sentences as in the background corpus BIBREF10 . The required POS tags were generated with RASP BIBREF11 , using the CLAWS2 tagset. Error Detection Model We construct a neural sequence labeling model for error detection, following the previous work BIBREF12 , BIBREF13 . The model receives a sequence of tokens as input and outputs a prediction for each position, indicating whether the token is correct or incorrect in the current context. The tokens are first mapped to a distributed vector space, resulting in a sequence of word embeddings. Next, the embeddings are given as input to a bidirectional LSTM BIBREF14 , in order to create context-dependent representations for every token. The hidden states from forward- and backward-LSTMs are concatenated for each word position, resulting in representations that are conditioned on the whole sequence. This concatenated vector is then passed through an additional feedforward layer, and a softmax over the two possible labels (correct and incorrect) is used to output a probability distribution for each token. The model is optimised by minimising categorical cross-entropy with respect to the correct labels. We use AdaDelta BIBREF15 for calculating an adaptive learning rate during training, which accounts for a higher baseline performance compared to previous results. Evaluation We trained our error generation models on the public FCE training set BIBREF16 and used them to generate additional artificial training data. Grammatically correct text is needed as the starting point for inserting artificial errors, and we used two different sources: 1) the corrected version of the same FCE training set on which the system is trained (450K tokens), and 2) example sentences extracted from the English Vocabulary Profile (270K tokens).. While there are other text corpora that could be used (e.g., Wikipedia and news articles), our development experiments showed that keeping the writing style and vocabulary close to the target domain gives better results compared to simply including more data. We evaluated our detection models on three benchmarks: the FCE test data (41K tokens) and the two alternative annotations of the CoNLL 2014 Shared Task dataset (30K tokens) BIBREF3 . Each artificial error generation system was used to generate 3 different versions of the artificial data, which were then combined with the original annotated dataset and used for training an error detection system. Table TABREF1 contains example sentences from the error generation systems, highlighting each of the edits that are marked as errors. The error detection results can be seen in Table TABREF4 . We use INLINEFORM0 as the main evaluation measure, which was established as the preferred measure for error correction and detection by the CoNLL-14 shared task BIBREF3 . INLINEFORM1 calculates a weighted harmonic mean of precision and recall, which assigns twice as much importance to precision – this is motivated by practical applications, where accurate predictions from an error detection system are more important compared to coverage. For comparison, we also report the performance of the error detection system by Rei2016, trained using the same FCE dataset. The results show that error detection performance is substantially improved by making use of artificially generated data, created by any of the described methods. When comparing the error generation system by Felice2014a (FY14) with our pattern-based (PAT) and machine translation (MT) approaches, we see that the latter methods covering all error types consistently improve performance. While the added error types tend to be less frequent and more complicated to capture, the added coverage is indeed beneficial for error detection. Combining the pattern-based approach with the machine translation system (Ann+PAT+MT) gave the best overall performance on all datasets. The two frameworks learn to generate different types of errors, and taking advantage of both leads to substantial improvements in error detection. We used the Approximate Randomisation Test BIBREF17 , BIBREF18 to calculate statistical significance and found that the improvement for each of the systems using artificial data was significant over using only manual annotation. In addition, the final combination system is also significantly better compared to the Felice2014a system, on all three datasets. While Rei2016 also report separate experiments that achieve even higher performance, these models were trained on a considerably larger proprietary corpus. In this paper we compare error detection frameworks trained on the same publicly available FCE dataset, thereby removing the confounding factor of dataset size and only focusing on the model architectures. The error generation methods can generate alternative versions of the same input text – the pattern-based method randomly samples the error locations, and the SMT system can provide an n-best list of alternative translations. Therefore, we also investigated the combination of multiple error-generated versions of the input files when training error detection models. Figure FIGREF6 shows the INLINEFORM0 score on the development set, as the training data is increased by using more translations from the n-best list of the SMT system. These results reveal that allowing the model to see multiple alternative versions of the same file gives a distinct improvement – showing the model both correct and incorrect variations of the same sentences likely assists in learning a discriminative model. Related Work Our work builds on prior research into AEG. Brockett2006 constructed regular expressions for transforming correct sentences to contain noun number errors. Rozovskaya2010a learned confusion sets from an annotated corpus in order to generate preposition errors. Foster2009 devised a tool for generating errors for different types using patterns provided by the user or collected automatically from an annotated corpus. However, their method uses a limited number of edit operations and is thus unable to generate complex errors. Cahill2013 compared different training methodologies and showed that artificial errors helped correct prepositions. Felice2014a learned error type distributions for generating five types of errors, and the system in Section SECREF3 is an extension of this model. While previous work focused on generating a specific subset of error types, we explored two holistic approaches to AEG and showed that they are able to significantly improve error detection performance. Conclusion This paper investigated two AEG methods, in order to create additional training data for error detection. First, we explored a method using textual patterns learned from an annotated corpus, which are used for inserting errors into correct input text. In addition, we proposed formulating error generation as an MT framework, learning to translate from grammatically correct to incorrect sentences. The addition of artificial data to the training process was evaluated on three error detection annotations, using the FCE and CoNLL 2014 datasets. Making use of artificial data provided improvements for all data generation methods. By relaxing the type restrictions and generating all types of errors, our pattern-based method consistently outperformed the system by Felice2014a. The combination of the pattern-based method with the machine translation approach gave further substantial improvements and the best performance on all datasets.
What textual patterns are extracted?
9a9d225f9ac35ed35ea02f554f6056af3b42471d
What Ġtextual Ġpatterns Ġare Ġextracted ?
( V VD Ġshop _ V V 0 ĠII , ĠV VD Ġshopping _ V VG ĠII )
<s> What Ġtextual Ġpatterns Ġare Ġextracted ? </s> Introduction </s> Writing Ġerrors Ġcan Ġoccur Ġin Ġmany Ġdifferent Ġforms ĠâĢĵ Ġfrom Ġrelatively Ġsimple Ġpunct uation Ġand Ġdetermin er Ġerrors , Ġto Ġmistakes Ġincluding Ġword Ġtense Ġand Ġform , Ġincorrect Ġcoll ocations Ġand Ġerroneous Ġidi oms . ĠAutom atically Ġidentifying Ġall Ġof Ġthese Ġerrors Ġis Ġa Ġchallenging Ġtask , Ġespecially Ġas Ġthe Ġamount Ġof Ġavailable Ġannot ated Ġdata Ġis Ġvery Ġlimited . ĠRei 2016 Ġshowed Ġthat Ġwhile Ġsome Ġerror Ġdetection Ġalgorithms Ġperform Ġbetter Ġthan Ġothers , Ġit Ġis Ġadditional Ġtraining Ġdata Ġthat Ġhas Ġthe Ġbiggest Ġimpact Ġon Ġimproving Ġperformance . </s> Being Ġable Ġto Ġgenerate Ġrealistic Ġartificial Ġdata Ġwould Ġallow Ġfor Ġany Ġgram matically Ġcorrect Ġtext Ġto Ġbe Ġtransformed Ġinto Ġannot ated Ġexamples Ġcontaining Ġwriting Ġerrors , Ġproducing Ġlarge Ġamounts Ġof Ġadditional Ġtraining Ġexamples . ĠSuper vised Ġerror Ġgeneration Ġsystems Ġwould Ġalso Ġprovide Ġan Ġefficient Ġmethod Ġfor Ġanonym ising Ġthe Ġsource Ġcorpus ĠâĢĵ Ġerror Ġstatistics Ġfrom Ġa Ġprivate Ġcorpus Ġcan Ġbe Ġaggreg ated Ġand Ġapplied Ġto Ġa Ġdifferent Ġtarget Ġtext , Ġobsc uring Ġsensitive Ġinformation Ġin Ġthe Ġoriginal Ġexamination Ġscripts . ĠHowever , Ġthe Ġtask Ġof Ġcreating Ġincorrect Ġdata Ġis Ġsomewhat Ġmore Ġdifficult Ġthan Ġmight Ġinitially Ġappear ĠâĢĵ Ġnaive Ġmethods Ġfor Ġerror Ġgeneration Ġcan Ġcreate Ġdata Ġthat Ġdoes Ġnot Ġresemble Ġnatural Ġerrors , Ġthereby Ġmaking Ġdownstream Ġsystems Ġlearn Ġmisleading Ġor Ġunin form ative Ġpatterns . </s> Previous Ġwork Ġon Ġartificial Ġerror Ġgeneration Ġ( A EG ) Ġhas Ġfocused Ġon Ġspecific Ġerror Ġtypes , Ġsuch Ġas Ġpre pos itions Ġand Ġdetermin ers ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, Ġor Ġnoun Ġnumber Ġerrors ĠB IB REF 2 Ġ. ĠFel ice 2014 a Ġinvestigated Ġthe Ġuse Ġof Ġlinguistic Ġinformation Ġwhen Ġgenerating Ġartificial Ġdata Ġfor Ġerror Ġcorrection , Ġbut Ġalso Ġrestricting Ġthe Ġapproach Ġto Ġonly Ġfive Ġerror Ġtypes . ĠThere Ġhas Ġbeen Ġvery Ġlimited Ġresearch Ġon Ġgenerating Ġartificial Ġdata Ġfor Ġall Ġtypes , Ġwhich Ġis Ġimportant Ġfor Ġgeneral - purpose Ġerror Ġdetection Ġsystems . ĠFor Ġexample , Ġthe Ġerror Ġtypes Ġinvestigated Ġby ĠFel ice 2014 a Ġcover Ġonly Ġ35 . 74 % Ġof Ġall Ġerrors Ġpresent Ġin Ġthe ĠCo N LL Ġ2014 Ġtraining Ġdataset , Ġproviding Ġno Ġadditional Ġinformation Ġfor Ġthe Ġmajority Ġof Ġerrors . </s> In Ġthis Ġpaper , Ġwe Ġinvestigate Ġtwo Ġsupervised Ġapproaches Ġfor Ġgenerating Ġall Ġtypes Ġof Ġartificial Ġerrors . ĠWe Ġpropose Ġa Ġframework Ġfor Ġgenerating Ġerrors Ġbased Ġon Ġstatistical Ġmachine Ġtranslation Ġ( SM T ), Ġtraining Ġa Ġmodel Ġto Ġtranslate Ġfrom Ġcorrect Ġinto Ġincorrect Ġsentences . ĠIn Ġaddition , Ġwe Ġdescribe Ġa Ġmethod Ġfor Ġlearning Ġerror Ġpatterns Ġfrom Ġan Ġannot ated Ġcorpus Ġand Ġtransplant ing Ġthem Ġinto Ġerror - free Ġtext . ĠWe Ġevaluate Ġthe Ġeffect Ġof Ġintroducing Ġartificial Ġdata Ġon Ġtwo Ġerror Ġdetection Ġbenchmarks . ĠOur Ġresults Ġshow Ġthat Ġeach Ġmethod Ġprovides Ġsignificant Ġimprovements Ġover Ġusing Ġonly Ġthe Ġavailable Ġtraining Ġset , Ġand Ġa Ġcombination Ġof Ġboth Ġgives Ġan Ġabsolute Ġimprovement Ġof Ġ4 . 3 % Ġin ĠIN LINE FORM 0 Ġ, Ġwithout Ġrequiring Ġany Ġadditional Ġannot ated Ġdata . </s> Error ĠGeneration ĠMethods </s> We Ġinvestigate Ġtwo Ġalternative Ġmethods Ġfor ĠA EG . ĠThe Ġmodels Ġreceive Ġgram matically Ġcorrect Ġtext Ġas Ġinput Ġand Ġmodify Ġcertain Ġtokens Ġto Ġproduce Ġincorrect Ġsequences . ĠThe Ġalternative Ġversions Ġof Ġeach Ġsentence Ġare Ġaligned Ġusing ĠLe ven sh tein Ġdistance , Ġallowing Ġus Ġto Ġidentify Ġspecific Ġwords Ġthat Ġneed Ġto Ġbe Ġmarked Ġas Ġerrors . ĠWhile Ġthese Ġalign ments Ġare Ġnot Ġalways Ġperfect , Ġwe Ġfound Ġthem Ġto Ġbe Ġsufficient Ġfor Ġpractical Ġpurposes , Ġsince Ġalternative Ġalign ments Ġof Ġsimilar Ġsentences Ġoften Ġresult Ġin Ġthe Ġsame Ġbinary Ġlabeling . ĠFuture Ġwork Ġcould Ġexplore Ġmore Ġadvanced Ġalignment Ġmethods , Ġsuch Ġas Ġproposed Ġby Ġfel ice - b ry ant - br isco e . </s> In ĠSection ĠSEC REF 4 Ġ, Ġthis Ġautomatically Ġlabeled Ġdata Ġis Ġthen Ġused Ġfor Ġtraining Ġerror Ġdetection Ġmodels . </s> Machine ĠTranslation </s> We Ġtreat ĠA EG Ġas Ġa Ġtranslation Ġtask ĠâĢĵ Ġgiven Ġa Ġcorrect Ġsentence Ġas Ġinput , Ġthe Ġsystem Ġwould Ġlearn Ġto Ġtranslate Ġit Ġto Ġcontain Ġlikely Ġerrors , Ġbased Ġon Ġa Ġtraining Ġcorpus Ġof Ġparallel Ġdata . ĠEx isting ĠSM T Ġapproaches Ġare Ġalready Ġoptim ised Ġfor Ġidentifying Ġcontext Ġpatterns Ġthat Ġcorrespond Ġto Ġspecific Ġoutput Ġsequences , Ġwhich Ġis Ġalso Ġrequired Ġfor Ġgenerating Ġhuman - like Ġerrors . ĠThe Ġreverse Ġof Ġthis Ġidea , Ġtranslating Ġfrom Ġincorrect Ġto Ġcorrect Ġsentences , Ġhas Ġbeen Ġshown Ġto Ġwork Ġwell Ġfor Ġerror Ġcorrection Ġtasks ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, Ġand Ġround - trip Ġtranslation Ġhas Ġalso Ġbeen Ġshown Ġto Ġbe Ġpromising Ġfor Ġcorrecting Ġgram matical Ġerrors ĠB IB REF 4 Ġ. </s> Following Ġprevious Ġwork ĠB IB REF 2 Ġ, ĠB IB REF 5 Ġ, Ġwe Ġbuild Ġa Ġphrase - based ĠSM T Ġerror Ġgeneration Ġsystem . ĠDuring Ġtraining , Ġerror - correct ed Ġsentences Ġin Ġthe Ġtraining Ġdata Ġare Ġtreated Ġas Ġthe Ġsource , Ġand Ġthe Ġoriginal Ġsentences Ġwritten Ġby Ġlanguage Ġlearners Ġas Ġthe Ġtarget . ĠP ial ign ĠB IB REF 6 Ġis Ġused Ġto Ġcreate Ġa Ġphrase Ġtranslation Ġtable Ġdirectly Ġfrom Ġmodel Ġprobabilities . ĠIn Ġaddition Ġto Ġdefault Ġfeatures , Ġwe Ġadd Ġcharacter - level ĠLe ven sh tein Ġdistance Ġto Ġeach Ġmapping Ġin Ġthe Ġphrase Ġtable , Ġas Ġproposed Ġby ĠFel ice : 2014 - Co N LL . ĠDec oding Ġis Ġperformed Ġusing ĠMoses ĠB IB REF 7 Ġand Ġthe Ġlanguage Ġmodel Ġused Ġduring Ġdecoding Ġis Ġbuilt Ġfrom Ġthe Ġoriginal Ġerroneous Ġsentences Ġin Ġthe Ġlear ner Ġcorpus . ĠThe ĠIR ST LM ĠTool kit ĠB IB REF 8 Ġis Ġused Ġfor Ġbuilding Ġa Ġ5 - gram Ġlanguage Ġmodel Ġwith Ġmodified ĠK nes er - N ey Ġsmoot hing ĠB IB REF 9 Ġ. </s> Pattern ĠExt raction </s> We Ġalso Ġdescribe Ġa Ġmethod Ġfor ĠA EG Ġusing Ġpatterns Ġover Ġwords Ġand Ġpart - of - speech Ġ( POS ) Ġtags , Ġextracting Ġknown Ġincorrect Ġsequences Ġfrom Ġa Ġcorpus Ġof Ġannot ated Ġcorrections . ĠThis Ġapproach Ġis Ġbased Ġon Ġthe Ġbest Ġmethod Ġidentified Ġby ĠFel ice 2014 a , Ġusing Ġerror Ġtype Ġdistributions ; Ġwhile Ġthey Ġcovered Ġonly Ġ5 Ġerror Ġtypes , Ġwe Ġrelax Ġthis Ġrestriction Ġand Ġlearn Ġpatterns Ġfor Ġgenerating Ġall Ġtypes Ġof Ġerrors . </s> The Ġoriginal Ġand Ġcorrected Ġsentences Ġin Ġthe Ġcorpus Ġare Ġaligned Ġand Ġused Ġto Ġidentify Ġshort Ġtransformation Ġpatterns Ġin Ġthe Ġform Ġof Ġ( inc orrect Ġphrase , Ġcorrect Ġphrase ). ĠThe Ġlength Ġof Ġeach Ġpattern Ġis Ġthe Ġaffected Ġphrase , Ġplus Ġup Ġto Ġone Ġtoken Ġof Ġcontext Ġon Ġboth Ġsides . ĠIf Ġa Ġword Ġform Ġchanges Ġbetween Ġthe Ġincorrect Ġand Ġcorrect Ġtext , Ġit Ġis Ġfully Ġsaved Ġin Ġthe Ġpattern , Ġotherwise Ġthe ĠPOS Ġtags Ġare Ġused Ġfor Ġmatching . </s> For Ġexample , Ġthe Ġoriginal Ġsentence Ġ` We Ġwent Ġshop Ġon ĠSaturday ' Ġand Ġthe Ġcorrected Ġversion Ġ` We Ġwent Ġshopping Ġon ĠSaturday ' Ġwould Ġproduce Ġthe Ġfollowing Ġpattern : </s> ( V VD Ġshop _ V V 0 ĠII , ĠV VD Ġshopping _ V VG ĠII ) </s> After Ġcollecting Ġstatistics Ġfrom Ġthe Ġbackground Ġcorpus , Ġerrors Ġcan Ġbe Ġinserted Ġinto Ġerror - free Ġtext . ĠThe Ġlearned Ġpatterns Ġare Ġnow Ġreversed , Ġlooking Ġfor Ġthe Ġcorrect Ġside Ġof Ġthe Ġtuple Ġin Ġthe Ġinput Ġsentence . ĠWe Ġonly Ġuse Ġpatterns Ġwith Ġfrequency ĠIN LINE FORM 0 Ġ, Ġwhich Ġyields Ġa Ġtotal Ġof Ġ35 , 625 Ġpatterns Ġfrom Ġour Ġtraining Ġdata . ĠFor Ġeach Ġinput Ġsentence , Ġwe Ġfirst Ġdecide Ġhow Ġmany Ġerrors Ġwill Ġbe Ġgenerated Ġ( using Ġprobabilities Ġfrom Ġthe Ġbackground Ġcorpus ) Ġand Ġattempt Ġto Ġcreate Ġthem Ġby Ġsampling Ġfrom Ġthe Ġcollection Ġof Ġapplicable Ġpatterns . ĠThis Ġprocess Ġis Ġrepeated Ġuntil Ġall Ġthe Ġrequired Ġerrors Ġhave Ġbeen Ġgenerated Ġor Ġthe Ġsentence Ġis Ġexhausted . ĠDuring Ġgeneration , Ġwe Ġtry Ġto Ġbalance Ġthe Ġdistribution Ġof Ġerror Ġtypes Ġas Ġwell Ġas Ġkeeping Ġthe Ġsame Ġproportion Ġof Ġincorrect Ġand Ġcorrect Ġsentences Ġas Ġin Ġthe Ġbackground Ġcorpus ĠB IB REF 10 Ġ. ĠThe Ġrequired ĠPOS Ġtags Ġwere Ġgenerated Ġwith ĠR AS P ĠB IB REF 11 Ġ, Ġusing Ġthe ĠCL AW S 2 Ġtag set . </s> Error ĠDetection ĠModel </s> We Ġconstruct Ġa Ġneural Ġsequence Ġlabeling Ġmodel Ġfor Ġerror Ġdetection , Ġfollowing Ġthe Ġprevious Ġwork ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. ĠThe Ġmodel Ġreceives Ġa Ġsequence Ġof Ġtokens Ġas Ġinput Ġand Ġoutputs Ġa Ġprediction Ġfor Ġeach Ġposition , Ġindicating Ġwhether Ġthe Ġtoken Ġis Ġcorrect Ġor Ġincorrect Ġin Ġthe Ġcurrent Ġcontext . ĠThe Ġtokens Ġare Ġfirst Ġmapped Ġto Ġa Ġdistributed Ġvector Ġspace , Ġresulting Ġin Ġa Ġsequence Ġof Ġword Ġembed d ings . ĠNext , Ġthe Ġembed d ings Ġare Ġgiven Ġas Ġinput Ġto Ġa Ġbid irection al ĠL ST M ĠB IB REF 14 Ġ, Ġin Ġorder Ġto Ġcreate Ġcontext - dependent Ġrepresentations Ġfor Ġevery Ġtoken . ĠThe Ġhidden Ġstates Ġfrom Ġforward - Ġand Ġbackward - L ST Ms Ġare Ġconc aten ated Ġfor Ġeach Ġword Ġposition , Ġresulting Ġin Ġrepresentations Ġthat Ġare Ġconditioned Ġon Ġthe Ġwhole Ġsequence . ĠThis Ġconc aten ated Ġvector Ġis Ġthen Ġpassed Ġthrough Ġan Ġadditional Ġfeed forward Ġlayer , Ġand Ġa Ġsoft max Ġover Ġthe Ġtwo Ġpossible Ġlabels Ġ( correct Ġand Ġincorrect ) Ġis Ġused Ġto Ġoutput Ġa Ġprobability Ġdistribution Ġfor Ġeach Ġtoken . ĠThe Ġmodel Ġis Ġoptim ised Ġby Ġminim ising Ġcateg orical Ġcross - ent ropy Ġwith Ġrespect Ġto Ġthe Ġcorrect Ġlabels . ĠWe Ġuse ĠAda Delta ĠB IB REF 15 Ġfor Ġcalculating Ġan Ġadaptive Ġlearning Ġrate Ġduring Ġtraining , Ġwhich Ġaccounts Ġfor Ġa Ġhigher Ġbaseline Ġperformance Ġcompared Ġto Ġprevious Ġresults . </s> E val uation </s> We Ġtrained Ġour Ġerror Ġgeneration Ġmodels Ġon Ġthe Ġpublic ĠF CE Ġtraining Ġset ĠB IB REF 16 Ġand Ġused Ġthem Ġto Ġgenerate Ġadditional Ġartificial Ġtraining Ġdata . ĠGram matically Ġcorrect Ġtext Ġis Ġneeded Ġas Ġthe Ġstarting Ġpoint Ġfor Ġinserting Ġartificial Ġerrors , Ġand Ġwe Ġused Ġtwo Ġdifferent Ġsources : Ġ1 ) Ġthe Ġcorrected Ġversion Ġof Ġthe Ġsame ĠF CE Ġtraining Ġset Ġon Ġwhich Ġthe Ġsystem Ġis Ġtrained Ġ( 450 K Ġtokens ), Ġand Ġ2 ) Ġexample Ġsentences Ġextracted Ġfrom Ġthe ĠEnglish ĠVoc abulary ĠProfile Ġ( 270 K Ġtokens ) .. ĠWhile Ġthere Ġare Ġother Ġtext Ġcorpor a Ġthat Ġcould Ġbe Ġused Ġ( e . g ., ĠWikipedia Ġand Ġnews Ġarticles ), Ġour Ġdevelopment Ġexperiments Ġshowed Ġthat Ġkeeping Ġthe Ġwriting Ġstyle Ġand Ġvocabulary Ġclose Ġto Ġthe Ġtarget Ġdomain Ġgives Ġbetter Ġresults Ġcompared Ġto Ġsimply Ġincluding Ġmore Ġdata . </s> We Ġevaluated Ġour Ġdetection Ġmodels Ġon Ġthree Ġbenchmarks : Ġthe ĠF CE Ġtest Ġdata Ġ( 41 K Ġtokens ) Ġand Ġthe Ġtwo Ġalternative Ġannotations Ġof Ġthe ĠCo N LL Ġ2014 ĠShared ĠTask Ġdataset Ġ( 30 K Ġtokens ) ĠB IB REF 3 Ġ. ĠEach Ġartificial Ġerror Ġgeneration Ġsystem Ġwas Ġused Ġto Ġgenerate Ġ3 Ġdifferent Ġversions Ġof Ġthe Ġartificial Ġdata , Ġwhich Ġwere Ġthen Ġcombined Ġwith Ġthe Ġoriginal Ġannot ated Ġdataset Ġand Ġused Ġfor Ġtraining Ġan Ġerror Ġdetection Ġsystem . ĠTable ĠT AB REF 1 Ġcontains Ġexample Ġsentences Ġfrom Ġthe Ġerror Ġgeneration Ġsystems , Ġhighlighting Ġeach Ġof Ġthe Ġedits Ġthat Ġare Ġmarked Ġas Ġerrors . </s> The Ġerror Ġdetection Ġresults Ġcan Ġbe Ġseen Ġin ĠTable ĠT AB REF 4 Ġ. ĠWe Ġuse ĠIN LINE FORM 0 Ġas Ġthe Ġmain Ġevaluation Ġmeasure , Ġwhich Ġwas Ġestablished Ġas Ġthe Ġpreferred Ġmeasure Ġfor Ġerror Ġcorrection Ġand Ġdetection Ġby Ġthe ĠCo N LL - 14 Ġshared Ġtask ĠB IB REF 3 Ġ. ĠIN LINE FORM 1 Ġcalculates Ġa Ġweighted Ġharmonic Ġmean Ġof Ġprecision Ġand Ġrecall , Ġwhich Ġassigns Ġtwice Ġas Ġmuch Ġimportance Ġto Ġprecision ĠâĢĵ Ġthis Ġis Ġmotivated Ġby Ġpractical Ġapplications , Ġwhere Ġaccurate Ġpredictions Ġfrom Ġan Ġerror Ġdetection Ġsystem Ġare Ġmore Ġimportant Ġcompared Ġto Ġcoverage . ĠFor Ġcomparison , Ġwe Ġalso Ġreport Ġthe Ġperformance Ġof Ġthe Ġerror Ġdetection Ġsystem Ġby ĠRei 2016 , Ġtrained Ġusing Ġthe Ġsame ĠF CE Ġdataset . </s> The Ġresults Ġshow Ġthat Ġerror Ġdetection Ġperformance Ġis Ġsubstantially Ġimproved Ġby Ġmaking Ġuse Ġof Ġartificially Ġgenerated Ġdata , Ġcreated Ġby Ġany Ġof Ġthe Ġdescribed Ġmethods . ĠWhen Ġcomparing Ġthe Ġerror Ġgeneration Ġsystem Ġby ĠFel ice 2014 a Ġ( FY 14 ) Ġwith Ġour Ġpattern - based Ġ( P AT ) Ġand Ġmachine Ġtranslation Ġ( MT ) Ġapproaches , Ġwe Ġsee Ġthat Ġthe Ġlatter Ġmethods Ġcovering Ġall Ġerror Ġtypes Ġconsistently Ġimprove Ġperformance . ĠWhile Ġthe Ġadded Ġerror Ġtypes Ġtend Ġto Ġbe Ġless Ġfrequent Ġand Ġmore Ġcomplicated Ġto Ġcapture , Ġthe Ġadded Ġcoverage Ġis Ġindeed Ġbeneficial Ġfor Ġerror Ġdetection . ĠComb ining Ġthe Ġpattern - based Ġapproach Ġwith Ġthe Ġmachine Ġtranslation Ġsystem Ġ( Ann + P AT + MT ) Ġgave Ġthe Ġbest Ġoverall Ġperformance Ġon Ġall Ġdatasets . ĠThe Ġtwo Ġframeworks Ġlearn Ġto Ġgenerate Ġdifferent Ġtypes Ġof Ġerrors , Ġand Ġtaking Ġadvantage Ġof Ġboth Ġleads Ġto Ġsubstantial Ġimprovements Ġin Ġerror Ġdetection . </s> We Ġused Ġthe ĠApp rox imate ĠRandom isation ĠTest ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġto Ġcalculate Ġstatistical Ġsignificance Ġand Ġfound Ġthat Ġthe Ġimprovement Ġfor Ġeach Ġof Ġthe Ġsystems Ġusing Ġartificial Ġdata Ġwas Ġsignificant Ġover Ġusing Ġonly Ġmanual Ġannotation . ĠIn Ġaddition , Ġthe Ġfinal Ġcombination Ġsystem Ġis Ġalso Ġsignificantly Ġbetter Ġcompared Ġto Ġthe ĠFel ice 2014 a Ġsystem , Ġon Ġall Ġthree Ġdatasets . ĠWhile ĠRei 2016 Ġalso Ġreport Ġseparate Ġexperiments Ġthat Ġachieve Ġeven Ġhigher Ġperformance , Ġthese Ġmodels Ġwere Ġtrained Ġon Ġa Ġconsiderably Ġlarger Ġproprietary Ġcorpus . ĠIn Ġthis Ġpaper Ġwe Ġcompare Ġerror Ġdetection Ġframeworks Ġtrained Ġon Ġthe Ġsame Ġpublicly Ġavailable ĠF CE Ġdataset , Ġthereby Ġremoving Ġthe Ġconfounding Ġfactor Ġof Ġdataset Ġsize Ġand Ġonly Ġfocusing Ġon Ġthe Ġmodel Ġarchitectures . </s> The Ġerror Ġgeneration Ġmethods Ġcan Ġgenerate Ġalternative Ġversions Ġof Ġthe Ġsame Ġinput Ġtext ĠâĢĵ Ġthe Ġpattern - based Ġmethod Ġrandomly Ġsamples Ġthe Ġerror Ġlocations , Ġand Ġthe ĠSM T Ġsystem Ġcan Ġprovide Ġan Ġn - best Ġlist Ġof Ġalternative Ġtranslations . ĠTherefore , Ġwe Ġalso Ġinvestigated Ġthe Ġcombination Ġof Ġmultiple Ġerror - generated Ġversions Ġof Ġthe Ġinput Ġfiles Ġwhen Ġtraining Ġerror Ġdetection Ġmodels . ĠFigure ĠFIG REF 6 Ġshows Ġthe ĠIN LINE FORM 0 Ġscore Ġon Ġthe Ġdevelopment Ġset , Ġas Ġthe Ġtraining Ġdata Ġis Ġincreased Ġby Ġusing Ġmore Ġtranslations Ġfrom Ġthe Ġn - best Ġlist Ġof Ġthe ĠSM T Ġsystem . ĠThese Ġresults Ġreveal Ġthat Ġallowing Ġthe Ġmodel Ġto Ġsee Ġmultiple Ġalternative Ġversions Ġof Ġthe Ġsame Ġfile Ġgives Ġa Ġdistinct Ġimprovement ĠâĢĵ Ġshowing Ġthe Ġmodel Ġboth Ġcorrect Ġand Ġincorrect Ġvariations Ġof Ġthe Ġsame Ġsentences Ġlikely Ġassists Ġin Ġlearning Ġa Ġdiscrim in ative Ġmodel . </s> Related ĠWork </s> Our Ġwork Ġbuilds Ġon Ġprior Ġresearch Ġinto ĠA EG . ĠBrock ett 2006 Ġconstructed Ġregular Ġexpressions Ġfor Ġtransforming Ġcorrect Ġsentences Ġto Ġcontain Ġnoun Ġnumber Ġerrors . ĠRoz ov s kaya 2010 a Ġlearned Ġconfusion Ġsets Ġfrom Ġan Ġannot ated Ġcorpus Ġin Ġorder Ġto Ġgenerate Ġpre position Ġerrors . ĠFoster 2009 Ġdevised Ġa Ġtool Ġfor Ġgenerating Ġerrors Ġfor Ġdifferent Ġtypes Ġusing Ġpatterns Ġprovided Ġby Ġthe Ġuser Ġor Ġcollected Ġautomatically Ġfrom Ġan Ġannot ated Ġcorpus . ĠHowever , Ġtheir Ġmethod Ġuses Ġa Ġlimited Ġnumber Ġof Ġedit Ġoperations Ġand Ġis Ġthus Ġunable Ġto Ġgenerate Ġcomplex Ġerrors . ĠCah ill 2013 Ġcompared Ġdifferent Ġtraining Ġmethod ologies Ġand Ġshowed Ġthat Ġartificial Ġerrors Ġhelped Ġcorrect Ġpre pos itions . ĠFel ice 2014 a Ġlearned Ġerror Ġtype Ġdistributions Ġfor Ġgenerating Ġfive Ġtypes Ġof Ġerrors , Ġand Ġthe Ġsystem Ġin ĠSection ĠSEC REF 3 Ġis Ġan Ġextension Ġof Ġthis Ġmodel . ĠWhile Ġprevious Ġwork Ġfocused Ġon Ġgenerating Ġa Ġspecific Ġsubset Ġof Ġerror Ġtypes , Ġwe Ġexplored Ġtwo Ġholistic Ġapproaches Ġto ĠA EG Ġand Ġshowed Ġthat Ġthey Ġare Ġable Ġto Ġsignificantly Ġimprove Ġerror Ġdetection Ġperformance . </s> Conclusion </s> This Ġpaper Ġinvestigated Ġtwo ĠA EG Ġmethods , Ġin Ġorder Ġto Ġcreate Ġadditional Ġtraining Ġdata Ġfor Ġerror Ġdetection . ĠFirst , Ġwe Ġexplored Ġa Ġmethod Ġusing Ġtextual Ġpatterns Ġlearned Ġfrom Ġan Ġannot ated Ġcorpus , Ġwhich Ġare Ġused Ġfor Ġinserting Ġerrors Ġinto Ġcorrect Ġinput Ġtext . ĠIn Ġaddition , Ġwe Ġproposed Ġform ulating Ġerror Ġgeneration Ġas Ġan ĠMT Ġframework , Ġlearning Ġto Ġtranslate Ġfrom Ġgram matically Ġcorrect Ġto Ġincorrect Ġsentences . </s> The Ġaddition Ġof Ġartificial Ġdata Ġto Ġthe Ġtraining Ġprocess Ġwas Ġevaluated Ġon Ġthree Ġerror Ġdetection Ġannotations , Ġusing Ġthe ĠF CE Ġand ĠCo N LL Ġ2014 Ġdatasets . ĠMaking Ġuse Ġof Ġartificial Ġdata Ġprovided Ġimprovements Ġfor Ġall Ġdata Ġgeneration Ġmethods . ĠBy Ġrelaxing Ġthe Ġtype Ġrestrictions Ġand Ġgenerating Ġall Ġtypes Ġof Ġerrors , Ġour Ġpattern - based Ġmethod Ġconsistently Ġoutper formed Ġthe Ġsystem Ġby ĠFel ice 2014 a . ĠThe Ġcombination Ġof Ġthe Ġpattern - based Ġmethod Ġwith Ġthe Ġmachine Ġtranslation Ġapproach Ġgave Ġfurther Ġsubstantial Ġimprovements Ġand Ġthe Ġbest Ġperformance Ġon Ġall Ġdatasets .