pointer networks with coverage mechanism (PG-net)
EXTRACTIVE
 pointer networks with coverage mechanism (PG-net)BIBREF0
EXTRACTIVE
To overcome the size issue of the student reflection dataset, we first explore the effect of incorporating domain transfer into a recent abstractive summarization model: pointer networks with coverage mechanism (PG-net)BIBREF0. To experiment with domain transfer, the model was pretrained using the CNN/DM dataset, then fine tuned using the student reflection dataset (see the Experiments section). A second approach we explore to overcome the lack of reflection data is data synthesis. We first propose a template model for synthesizing new data, then investigate the performance impact of using this data when training the summarization model. The proposed model makes use of the nature of datasets such as ours, where the reference summaries tend to be close in structure: humans try to find the major points that students raise, then present the points in a way that marks their relative importance (recall the CS example in Table TABREF4). Our third explored approach is to combine domain transfer with data synthesis.
To overcome the size issue of the student reflection dataset, we first explore the effect of incorporating domain transfer into a recent abstractive summarization model: pointer networks with coverage mechanism (PG-net)BIBREF0. To experiment with domain transfer, the model was pretrained using the CNN/DM dataset, then fine tuned using the student reflection dataset (see the Experiments section). A second approach we explore to overcome the lack of reflection data is data synthesis. We first propose a template model for synthesizing new data, then investigate the performance impact of using this data when training the summarization model. The proposed model makes use of the nature of datasets such as ours, where the reference summaries tend to be close in structure: humans try to find the major points that students raise, then present the points in a way that marks their relative importance (recall the CS example in Table TABREF4). Our third explored approach is to combine domain transfer with data synthesis.
2002.03407
Introduction </s> Recently , Ġwith Ġthe Ġemergence Ġof Ġneural Ġseq 2 seq Ġmodels , Ġabstract ive Ġsummar ization Ġmethods Ġhave Ġseen Ġgreat Ġperformance Ġstrides ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 . ĠHowever , Ġcomplex Ġneural Ġsummar ization Ġmodels Ġwith Ġthousands Ġof Ġparameters Ġusually Ġrequire Ġa Ġlarge Ġamount Ġof Ġtraining Ġdata . ĠIn Ġfact , Ġmuch Ġof Ġthe Ġneural Ġsummar ization Ġwork Ġhas Ġbeen Ġtrained Ġand Ġtested Ġin Ġnews Ġdomains Ġwhere Ġnumerous Ġlarge Ġdatasets Ġexist . ĠFor Ġexample , Ġthe ĠCNN / Daily Mail Ġ( CNN / DM ) ĠB IB REF 3 , ĠB IB REF 4 Ġand ĠNew ĠYork ĠTimes Ġ( NY T ) Ġdatasets Ġare Ġin Ġthe Ġmagnitude Ġof Ġ300 k Ġand Ġ700 k Ġdocuments , Ġrespectively . ĠIn Ġcontrast , Ġin Ġother Ġdomains Ġsuch Ġas Ġstudent Ġreflections , Ġsummar ization Ġdatasets Ġare Ġonly Ġin Ġthe Ġmagnitude Ġof Ġtens Ġor Ġhundreds Ġof Ġdocuments Ġ( e . g ., ĠB IB REF 5 ). ĠWe Ġhypothes ize Ġthat Ġtraining Ġcomplex Ġneural Ġabstract ive Ġsummar ization Ġmodels Ġin Ġsuch Ġdomains Ġwill Ġnot Ġyield Ġgood Ġperforming Ġmodels , Ġand Ġwe Ġwill Ġindeed Ġlater Ġshow Ġthat Ġthis Ġis Ġthe Ġcase Ġfor Ġstudent Ġreflections . </s> To Ġimprove Ġperformance Ġin Ġlow Ġresource Ġdomains , Ġwe Ġexplore Ġthree Ġdirections . ĠFirst , Ġwe Ġexplore Ġdomain Ġtransfer Ġfor Ġabstract ive Ġsummar ization . ĠWhile Ġdomain Ġtransfer Ġis Ġnot Ġnew , Ġcompared Ġto Ġprior Ġsummar ization Ġstudies ĠB IB REF 6 , ĠB IB REF 7 , Ġour Ġtraining Ġ( news ) Ġand Ġtuning Ġ( student Ġreflection ) Ġdomains Ġare Ġquite Ġdiss imilar , Ġand Ġthe Ġin - domain Ġdata Ġis Ġsmall . ĠSecond , Ġwe Ġpropose Ġa Ġtemplate - based Ġsynthesis Ġmethod Ġto Ġcreate Ġsynthes ized Ġsumm aries , Ġthen Ġexplore Ġthe Ġeffect Ġof Ġenrich ing Ġtraining Ġdata Ġfor Ġabstract ive Ġsummar ization Ġusing Ġthe Ġproposed Ġmodel Ġcompared Ġto Ġa Ġsynthesis Ġbaseline . ĠLastly , Ġwe Ġcombine Ġboth Ġdirections . ĠEval uations Ġof Ġneural Ġabstract ive Ġsummar ization Ġmethod Ġacross Ġfour Ġstudent Ġreflection Ġcorpor a Ġshow Ġthe Ġutility Ġof Ġall Ġthree Ġmethods . </s> Related ĠWork </s> Abstract ive ĠSum mar ization . ĠAbstract ive Ġsummar ization Ġaims Ġto Ġgenerate Ġcoherent Ġsumm aries Ġwith Ġhigh Ġread ability , Ġand Ġhas Ġseen Ġincreasing Ġinterest Ġand Ġimproved Ġperformance Ġdue Ġto Ġthe Ġemergence Ġof Ġseq 2 seq Ġmodels ĠB IB REF 8 Ġand Ġattention Ġmechanisms ĠB IB REF 9 . ĠFor Ġexample , ĠB IB REF 0 , ĠB IB REF 2 , Ġand ĠB IB REF 1 Ġin Ġaddition Ġto Ġusing Ġenc oder - dec oder Ġmodel Ġwith Ġattention , Ġthey Ġused Ġpointer Ġnetworks Ġto Ġsolve Ġthe Ġout Ġof Ġvocabulary Ġissue , Ġwhile ĠB IB REF 0 Ġused Ġcoverage Ġmechanism Ġto Ġsolve Ġthe Ġproblem Ġof Ġword Ġrepetition . ĠIn Ġaddition , ĠB IB REF 2 Ġand ĠB IB REF 10 Ġused Ġreinforcement Ġlearning Ġin Ġan Ġend - to - end Ġsetting . </s> To Ġour Ġknowledge , Ġtraining Ġsuch Ġneural Ġabstract ive Ġsummar ization Ġmodels Ġin Ġlow Ġresource Ġdomains Ġusing Ġdomain Ġtransfer Ġhas Ġnot Ġbeen Ġthoroughly Ġexplored Ġon Ġdomains Ġdifferent Ġthan Ġnews . ĠFor Ġexample , ĠB IB REF 4 Ġreported Ġthe Ġresults Ġof Ġtraining Ġon ĠCNN / DM Ġdata Ġwhile Ġevaluating Ġon ĠD UC Ġdata Ġwithout Ġany Ġtuning . ĠNote Ġthat Ġthese Ġtwo Ġdatasets Ġare Ġboth Ġin Ġthe Ġnews Ġdomain , Ġand Ġboth Ġconsist Ġof Ġwell Ġwritten , Ġstructured Ġdocuments . ĠThe Ġdomain Ġtransfer Ġexperiments Ġof ĠB IB REF 1 Ġsimilarly Ġused Ġtwo Ġdifferent Ġnews Ġsummar ization Ġdatasets Ġ( CNN / DM Ġand ĠNYT ). ĠOur Ġwork Ġdiffers Ġin Ġseveral Ġways Ġfrom Ġthese Ġtwo Ġprior Ġdomain Ġtransfer Ġefforts . ĠFirst , Ġour Ġexperiments Ġinvolve Ġtwo Ġentirely Ġdifferent Ġdomains : Ġnews Ġand Ġstudent Ġreflections . ĠUnlike Ġnews , Ġstudent Ġreflection Ġdocuments Ġlack Ġglobal Ġstructure , Ġare Ġrepetitive , Ġand Ġcontain Ġmany Ġsentence Ġfragments Ġand Ġgram matical Ġmistakes . ĠSecond , Ġthe Ġprior Ġapproaches Ġeither Ġtrained Ġa Ġpart Ġof Ġthe Ġmodel Ġusing ĠNYT Ġdata Ġwhile Ġretaining Ġthe Ġother Ġpart Ġof Ġthe Ġmodel Ġtrained Ġonly Ġon ĠCNN / DM Ġdata ĠB IB REF 1 , Ġor Ġdidn 't Ġperform Ġany Ġtuning Ġat Ġall ĠB IB REF 4 . ĠIn Ġcontrast , Ġwe Ġdo Ġthe Ġtraining Ġin Ġtwo Ġconsecutive Ġphases , Ġpret raining Ġand Ġfine Ġtuning . ĠFinally , ĠB IB REF 1 Ġreported Ġthat Ġwhile Ġtraining Ġwith Ġdomain Ġtransfer Ġoutper formed Ġtraining Ġonly Ġon Ġout - of - domain Ġdata , Ġit Ġwas Ġnot Ġable Ġto Ġbeat Ġtraining Ġonly Ġon Ġin - domain Ġdata . ĠThis Ġis Ġlikely Ġbecause Ġtheir Ġin Ġand Ġout - of - domain Ġdata Ġsizes Ġare Ġcomparable , Ġunlike Ġin Ġour Ġcase Ġof Ġscarce Ġin - domain Ġdata . </s> In Ġa Ġdifferent Ġapproach Ġto Ġabstract ive Ġsummar ization , ĠB IB REF 11 Ġdeveloped Ġa Ġsoft Ġtemplate Ġbased Ġneural Ġmethod Ġconsisting Ġof Ġan Ġend - to - end Ġdeep Ġmodel Ġfor Ġtemplate Ġretrieval , Ġre ranking Ġand Ġsummary Ġrewriting . ĠWhile Ġwe Ġalso Ġdevelop Ġa Ġtemplate Ġbased Ġmodel , Ġour Ġwork Ġdiffers Ġin Ġboth Ġmodel Ġstructure Ġand Ġpurpose . </s> Data ĠSynt hesis . ĠData Ġsynthesis Ġfor Ġtext Ġsummar ization Ġis Ġund ere x pl ored , Ġwith Ġmost Ġprior Ġwork Ġfocusing Ġon Ġmachine Ġtranslation , Ġand Ġtext Ġnormal ization . ĠB IB REF 12 Ġproposed Ġdoing Ġdata Ġaug mentation Ġthrough Ġword Ġreplacement , Ġusing ĠWord Net ĠB IB REF 13 Ġand Ġvector Ġspace Ġsimilarity , Ġrespectively . ĠWe Ġwill Ġuse Ġa ĠWord Net Ġreplacement Ġmethod Ġas Ġa Ġbaseline Ġsynthesis Ġmethod Ġin Ġthe Ġexperiments Ġdescribed Ġbelow . ĠIn Ġcontrast , ĠB IB REF 14 Ġsynthes ized / au gment ed Ġdata Ġthrough Ġback - translation Ġand Ġword Ġreplacement Ġusing Ġlanguage Ġmodels . ĠB IB REF 15 Ġis Ġanother Ġrecent Ġwork Ġthat Ġwas Ġdone Ġin Ġparallel Ġand Ġis Ġvery Ġclose Ġto Ġours . ĠHowever , Ġin Ġaddition Ġto Ġthe Ġdifference Ġin Ġboth Ġour Ġand Ġtheir Ġmodel , Ġwe Ġthink Ġit Ġmight Ġbe Ġinf eas ible Ġto Ġback Ġgenerate Ġstudent Ġreflections Ġfrom Ġa Ġhuman Ġsummary , Ġespecially Ġan Ġabstract ive Ġone . </s> Ref lection ĠSum mar ization ĠDat as et </s> Student Ġreflections Ġare Ġcomments Ġprovided Ġby Ġstudents Ġin Ġresponse Ġto Ġa Ġset Ġof Ġinstructor Ġprompts . ĠThe Ġprompts Ġare Ġdirected Ġtowards Ġgathering Ġstudents ' Ġfeedback Ġon Ġcourse Ġmaterial . ĠStudent Ġreflections Ġare Ġcollected Ġdirectly Ġfollowing Ġeach Ġof Ġa Ġset Ġof Ġclassroom Ġlectures Ġover Ġa Ġsemester . ĠIn Ġthis Ġpaper , Ġthe Ġset Ġof Ġreflections Ġfor Ġeach Ġprompt Ġin Ġeach Ġlecture Ġis Ġconsidered Ġa Ġstudent Ġreflection Ġdocument . ĠThe Ġobjective Ġof Ġour Ġwork Ġis Ġto Ġprovide Ġa Ġcomprehensive Ġand Ġmeaningful Ġabstract ive Ġsummary Ġof Ġeach Ġstudent Ġreflection Ġdocument . ĠOur Ġdataset Ġconsists Ġof Ġdocuments Ġand Ġsumm aries Ġfrom Ġfour Ġcourse Ġinstant iations : ĠENG R Ġ( Introduction Ġto ĠMaterials ĠScience Ġand ĠEngineering ), ĠStat 2015 Ġand ĠStat 2016 Ġ( Statistics Ġfor ĠIndustrial ĠEngineers , Ġtaught Ġin Ġ2015 Ġand Ġ2016 , Ġrespectively ), Ġand ĠCS Ġ( Data ĠStruct ures Ġin ĠComputer ĠScience ). ĠAll Ġreflections Ġwere Ġcollected Ġin Ġresponse Ġto Ġtwo Ġped agog ically - mot ivated Ġprompts ĠB IB REF 16 : ĠâĢ ľ Point Ġof ĠInterest Ġ( PO I ): ĠDesc ribe Ġwhat Ġyou Ġfound Ġmost Ġinteresting Ġin Ġtoday 's Ġclass âĢ Ŀ Ġand ĠâĢ ľ M udd iest ĠPoint Ġ( MP ): ĠDesc ribe Ġwhat Ġwas Ġconfusing Ġor Ġneeded Ġmore Ġdetail . âĢ Ŀ </s> For Ġeach Ġreflection Ġdocument , Ġat Ġleast Ġone Ġhuman Ġ( either Ġa ĠTA Ġor Ġdomain Ġexpert ) Ġcreated Ġsumm aries . ĠTable ĠT AB REF 4 Ġshows Ġexample Ġreference Ġsummary Ġproduced Ġby Ġone Ġannot ator Ġfor Ġthe ĠCS Ġcourse . ĠTable ĠT AB REF 5 Ġsummarizes Ġthe Ġdataset Ġin Ġterms Ġof Ġnumber Ġof Ġlectures , Ġnumber Ġof Ġprompts Ġper Ġlecture , Ġaverage Ġnumber Ġof Ġreflections Ġper Ġprompt , Ġand Ġnumber Ġof Ġabstract ive Ġreference Ġsumm aries Ġfor Ġeach Ġset Ġof Ġreflections . </s> Expl ored ĠAppro aches Ġfor ĠLimited ĠResources </s> To Ġovercome Ġthe Ġsize Ġissue Ġof Ġthe Ġstudent Ġreflection Ġdataset , Ġwe Ġfirst Ġexplore Ġthe Ġeffect Ġof Ġincorporating Ġdomain Ġtransfer Ġinto Ġa Ġrecent Ġabstract ive Ġsummar ization Ġmodel : Ġpointer Ġnetworks Ġwith Ġcoverage Ġmechanism Ġ( PG - net ) BI B REF 0 . ĠTo Ġexperiment Ġwith Ġdomain Ġtransfer , Ġthe Ġmodel Ġwas Ġpret rained Ġusing Ġthe ĠCNN / DM Ġdataset , Ġthen Ġfine Ġtuned Ġusing Ġthe Ġstudent Ġreflection Ġdataset Ġ( see Ġthe ĠExper iments Ġsection ). ĠA Ġsecond Ġapproach Ġwe Ġexplore Ġto Ġovercome Ġthe Ġlack Ġof Ġreflection Ġdata Ġis Ġdata Ġsynthesis . ĠWe Ġfirst Ġpropose Ġa Ġtemplate Ġmodel Ġfor Ġsynthes izing Ġnew Ġdata , Ġthen Ġinvestigate Ġthe Ġperformance Ġimpact Ġof Ġusing Ġthis Ġdata Ġwhen Ġtraining Ġthe Ġsummar ization Ġmodel . ĠThe Ġproposed Ġmodel Ġmakes Ġuse Ġof Ġthe Ġnature Ġof Ġdatasets Ġsuch Ġas Ġours , Ġwhere Ġthe Ġreference Ġsumm aries Ġtend Ġto Ġbe Ġclose Ġin Ġstructure : Ġhumans Ġtry Ġto Ġfind Ġthe Ġmajor Ġpoints Ġthat Ġstudents Ġraise , Ġthen Ġpresent Ġthe Ġpoints Ġin Ġa Ġway Ġthat Ġmarks Ġtheir Ġrelative Ġimportance Ġ( rec all Ġthe ĠCS Ġexample Ġin ĠTable ĠT AB REF 4 ). ĠOur Ġthird Ġexplored Ġapproach Ġis Ġto Ġcombine Ġdomain Ġtransfer Ġwith Ġdata Ġsynthesis . </s> Prop osed ĠTemplate - Based ĠSynt hesis ĠModel </s> Our Ġmotivation Ġfor Ġusing Ġtemplates Ġfor Ġdata Ġsynthesis Ġis Ġthat Ġseq 2 seq Ġsynthesis Ġmodels Ġ( as Ġdiscussed Ġin Ġrelated Ġwork ) Ġtend Ġto Ġgenerate Ġirrelevant Ġand Ġrepeated Ġwords ĠB IB REF 17 , Ġwhile Ġtemplates Ġcan Ġproduce Ġmore Ġcoherent Ġand Ġconcise Ġoutput . ĠAlso , Ġextracting Ġtemplates Ġcan Ġbe Ġdone Ġeither Ġmanually Ġor Ġautomatically Ġtypically Ġby Ġtraining Ġa Ġfew Ġparameters Ġor Ġeven Ġdoing Ġno Ġtraining , Ġthen Ġexternal Ġinformation Ġin Ġthe Ġform Ġof Ġkeywords Ġor Ġsnippets Ġcan Ġbe Ġpopulated Ġinto Ġthe Ġtemplates Ġwith Ġthe Ġhelp Ġof Ġmore Ġsophisticated Ġmodels . ĠAccordingly , Ġusing Ġtemplates Ġcan Ġbe Ġvery Ġtempting Ġfor Ġdomains Ġwith Ġlimited Ġresources Ġsuch Ġas Ġours . </s> Model ĠStructure . ĠThe Ġmodel Ġconsists Ġof Ġ4 Ġmodules : </s> 1 . ĠTemplate Ġextraction : ĠTo Ġconvert Ġhuman Ġsumm aries Ġinto Ġtemplates , Ġwe Ġremove Ġkeywords Ġin Ġthe Ġsummary Ġto Ġleave Ġonly Ġnon - key words . ĠWe Ġuse ĠRapid ĠAutomatic ĠKey word ĠExt raction Ġ( RA KE ) ĠB IB REF 18 Ġto Ġidentify Ġkeywords . </s> 2 . ĠTemplate Ġclust ering : ĠUpon Ġconverting Ġhuman Ġsumm aries Ġinto Ġtemplates , Ġwe Ġcluster Ġthem Ġinto Ġ$ N $ Ġclusters Ġwith Ġthe Ġgoal Ġof Ġusing Ġany Ġtemplate Ġfrom Ġthe Ġsame Ġcluster Ġinterchange ably . ĠA Ġtemplate Ġis Ġfirst Ġconverted Ġinto Ġembed d ings Ġusing Ġa Ġpret rained ĠB ERT Ġmodel ĠB IB REF 19 , Ġwhere Ġtemplate Ġembed ding Ġis Ġconstructed Ġby Ġaverage Ġpool ing Ġword Ġembed d ings . ĠTem plates Ġare Ġthen Ġclustered Ġusing Ġk - med oid . </s> 3 . ĠSummary Ġrewriting : ĠAn Ġenc oder - att ention - dec oder Ġwith Ġpointer Ġnetwork Ġis Ġtrained Ġto Ġperform Ġthe Ġrewriting Ġtask . ĠThe Ġmodel Ġis Ġtrained Ġto Ġinject Ġkeywords Ġinto Ġa Ġtemplate Ġand Ġperform Ġrewriting Ġinto Ġa Ġcoherent Ġparagraph . ĠThe Ġproduced Ġre writ es Ġare Ġconsidered Ġas Ġcandidate Ġsumm aries . </s> 4 . ĠSummary Ġselection : ĠAfter Ġproducing Ġcandidate Ġsumm aries , Ġwe Ġneed Ġto Ġpick Ġthe Ġbest Ġones . ĠWe Ġargue Ġthat Ġthe Ġbest Ġcandidates Ġare Ġthose Ġthat Ġare Ġcoherent Ġand Ġalso Ġconvey Ġthe Ġsame Ġmeaning Ġas Ġthe Ġoriginal Ġhuman Ġsummary . ĠWe Ġthus Ġuse Ġa Ġhybrid Ġmetric Ġto Ġscore Ġcandidates , Ġwhere Ġthe Ġmetric Ġis Ġa Ġweighted Ġsum Ġof Ġtwo Ġscores Ġand Ġis Ġcalculated Ġusing ĠEqu ations Ġ1 , Ġ2 , Ġand Ġ3 . ĠE q . 1 Ġmeasures Ġco he rency Ġusing Ġa Ġlanguage Ġmodel Ġ( LM ), ĠE q . 2 Ġmeasures Ġhow Ġclose Ġa Ġcandidate Ġis Ġto Ġa Ġhuman Ġsummary Ġusing ĠR OU GE Ġscores , Ġwhile ĠE q . 3 Ġpicks Ġthe Ġhighest Ġscored Ġ$ N $ Ġcandidates Ġas Ġthe Ġfinal Ġsynthetic Ġset . </s> CS Ġand ĠHS Ġare Ġa Ġcandidate Ġand Ġhuman Ġsummary . Ġ$ P ( w ) $ Ġis Ġthe Ġprobability Ġof Ġword Ġ$ w $ Ġusing Ġa Ġlanguage Ġmodel . Ġ$\ alpha Ġ, Ġ\ beta Ġ$ Ġare Ġweight ing Ġparameters . ĠIn Ġthis Ġwork Ġwe Ġuse Ġ$\ alpha Ġ= \ beta Ġ= 1 $ Ġfor Ġall Ġexperiments . Ġ$ R _{ i } ( CS , HS ) $ Ġis ĠR OU GE - i Ġscore Ġbetween ĠCS Ġand ĠHS Ġfor Ġi = 1 , Ġ2 , Ġand Ġ$ l $. </s> Model ĠTraining . ĠBefore Ġusing Ġthe Ġsynthesis Ġmodel , Ġsome Ġof Ġthe Ġconstructing Ġmodules Ġ( rew rit ing Ġmodule , Ġscoring ĠLM ) Ġneed Ġtraining . ĠTo Ġtrain Ġthe Ġrewriting Ġmodel , Ġwe Ġuse Ġanother Ġdataset Ġconsisting Ġof Ġa Ġset Ġof Ġsamples , Ġwhere Ġeach Ġsample Ġcan Ġbe Ġa Ġtext Ġsnippet Ġ( sent ence , Ġparagraph , Ġetc .). ĠFor Ġeach Ġsample , Ġkeywords Ġare Ġextracted Ġusing ĠRA KE , Ġthen Ġremoved . ĠThe Ġkeywords Ġplus Ġthe Ġsample Ġwith Ġno Ġkeywords Ġare Ġthen Ġpassed Ġto Ġthe Ġrewriting Ġmodel . ĠThe Ġtraining Ġobjective Ġof Ġthis Ġmodel Ġis Ġto Ġreconstruct Ġthe Ġoriginal Ġsample , Ġwhich Ġcan Ġbe Ġseen Ġas Ġtrying Ġto Ġinject Ġextracted Ġkeywords Ġback Ġinto Ġa Ġtemplate . ĠModel ĠUsage . ĠTo Ġuse Ġthe Ġsynthesis Ġmodel Ġto Ġgenerate Ġnew Ġsamples , Ġthe Ġset Ġof Ġhuman Ġsumm aries Ġare Ġfed Ġto Ġthe Ġmodel , Ġpassing Ġthrough Ġthe Ġsub - modules Ġin Ġthe Ġfollowing Ġorder : </s> 1 . ĠHuman Ġsumm aries Ġfirst Ġpass Ġthrough Ġthe Ġtemplate Ġextraction Ġmodule , Ġconverting Ġeach Ġsummary Ġ$ s _ i $ Ġinto Ġtemplate Ġ$ t _ i $ Ġand Ġthe Ġcorresponding Ġkeywords Ġ$ kw _ i $. </s> 2 . ĠTem plates Ġare Ġthen Ġpassed Ġto Ġthe Ġclust ering Ġmodule , Ġproducing Ġa Ġset Ġof Ġclusters . ĠEach Ġcluster Ġ$ C $ Ġcontains Ġa Ġnumber Ġof Ġsimilar Ġtemplates . </s> 3 . ĠFor Ġeach Ġtemplate Ġ$ t _ i $ Ġand Ġcorresponding Ġkeywords Ġ$ kw _ i $ Ġfrom Ġstep Ġ1 , Ġfind Ġthe Ġcluster Ġ$ C _ i $ Ġthat Ġcontains Ġthe Ġtemplate Ġ$ t _ i $, Ġthen Ġpass Ġthe Ġset Ġof Ġtemplates Ġwithin Ġthat Ġclusters Ġ$\ l brace Ġt _ j \ r brace Ġ\ for all Ġ{ j }, $ Ġif Ġ$ t _ j Ġ\ in ĠC _ i $ Ġalongside Ġthe Ġkeywords Ġ$ kw _ i $ Ġto Ġthe Ġsummary Ġrewriting Ġmodule . ĠThis Ġwill Ġproduce Ġa Ġset Ġof Ġcandidate Ġsumm aries . </s> 4 . ĠThe Ġsummary Ġselection Ġmodule Ġscores Ġand Ġselects Ġthe Ġhighest Ġ$ N $ Ġcandidates Ġas Ġthe Ġsynthetic Ġsumm aries . </s> Exper iments </s> Our Ġexperimental Ġdesigns Ġaddress Ġthe Ġfollowing Ġhypotheses : </s> Hyp ot hesis Ġ1 Ġ( H 1 ) Ġ: ĠTraining Ġcomplex Ġabstract ive Ġmodels Ġwith Ġlimited Ġin - domain Ġor Ġlarge Ġquantities Ġof Ġout - of - domain Ġdata Ġwon 't Ġbe Ġenough Ġto Ġoutper form Ġextract ive Ġbas elines . </s> Hyp ot hesis Ġ2 Ġ( H 2 ) Ġ: ĠDomain Ġtransfer Ġhelps Ġabstract ive Ġmodels Ġeven Ġif Ġin - domain Ġand Ġout - of - domain Ġdata Ġare Ġvery Ġdifferent Ġand Ġthe Ġamount Ġof Ġin - domain Ġdata Ġis Ġvery Ġsmall . </s> Hyp ot hesis Ġ3 Ġ( H 3 ) Ġ: ĠEn rich ing Ġabstract ive Ġtraining Ġdata Ġwith Ġsynthetic Ġdata Ġhelps Ġovercome Ġin - domain Ġdata Ġscarcity . </s> Hyp ot hesis Ġ4 Ġ( H 4 ) Ġ: ĠThe Ġproposed Ġtemplate - based Ġsynthesis Ġmodel Ġoutper forms Ġa Ġsimple Ġword Ġreplacement Ġmodel . </s> Hyp ot hesis Ġ5 Ġ( H 5 ) Ġ: ĠComb ining Ġdomain Ġtransfer Ġwith Ġdata Ġsynthesis Ġoutper forms Ġusing Ġeach Ġapproach Ġon Ġits Ġown . </s> Hyp ot hesis Ġ6 Ġ( H 6 ) Ġ: ĠThe Ġsynthesis Ġmodel Ġcan Ġbe Ġextended Ġto Ġperform Ġreflection Ġsummar ization Ġdirectly . </s> Ext ractive ĠBas elines Ġ( for Ġtesting ĠH 1 ). ĠWhile ĠB IB REF 0 Ġused ĠLead - 3 Ġas Ġan Ġextract ive Ġbaseline , Ġin Ġour Ġdata Ġsentence Ġorder Ġdoesn 't Ġmatter Ġas Ġreflections Ġare Ġindependent . ĠWe Ġthus Ġuse Ġa Ġsimilar Ġin Ġconcept Ġbaseline : Ġrandomly Ġselect ĠN Ġreflections . ĠSince Ġthe Ġbaseline Ġis Ġrandom Ġwe Ġreport Ġthe Ġaverage Ġresult Ġof Ġ100 Ġruns . ĠFollowing ĠB IB REF 5 , Ġwe Ġcompare Ġresults Ġto ĠME AD ĠB IB REF 20 Ġand Ġto ĠB IB REF 5 's Ġextract ive Ġphrase - based Ġmodel . ĠSince Ġthese Ġmodels Ġextracted Ġ5 Ġphrases Ġas Ġextract ive Ġsummary , Ġwe Ġuse ĠN = 5 Ġfor Ġour Ġthree Ġextract ive Ġbas elines . ĠAdditionally Ġwe Ġcompare Ġto Ġrunning Ġonly Ġthe Ġextract ive Ġpart Ġof ĠFast - RL . </s> Domain ĠTransfer Ġ( for Ġtesting ĠH 2 , ĠH 5 ). </s> To Ġobserve Ġthe Ġimpact Ġof Ġusing Ġout - of - domain Ġ( news ) Ġdata Ġfor Ġpret raining Ġto Ġcompensate Ġfor Ġlow Ġresource Ġin - domain Ġ( ref lection ) Ġdata , Ġwe Ġtrain Ġ3 Ġvariants Ġof ĠPG - net : Ġmodel Ġtraining Ġon ĠCNN / DM ; Ġmodel Ġtraining Ġon Ġreflections ; Ġand Ġmodel Ġtraining Ġon ĠCNN / DM Ġthen Ġtuning Ġusing Ġreflections . ĠTable ĠT AB REF 11 Ġshows Ġexample Ġsumm aries Ġgenerated Ġby Ġthe Ġthree Ġvariants Ġof ĠPG - net Ġfor Ġa ĠCS Ġdocument . ĠFor Ġall Ġexperiments Ġwhere Ġreflections Ġare Ġused Ġfor Ġtraining / tun ing , Ġwe Ġtrain Ġusing Ġa Ġleave Ġone Ġcourse Ġout Ġapproach Ġ( i . e , Ġin Ġeach Ġfold , Ġthree Ġcourses Ġare Ġused Ġfor Ġtraining Ġand Ġthe Ġremaining Ġcourse Ġfor Ġtesting ). ĠIf Ġthe Ġexperiment Ġinvolves Ġtuning Ġa Ġcombined Ġdictionary Ġof ĠCNN / DM Ġand Ġreflections Ġis Ġused Ġto Ġavoid Ġdomain Ġmismatch . ĠTo Ġtune Ġmodel Ġparameters , Ġthe Ġbest Ġnumber Ġof Ġsteps Ġfor Ġtraining , Ġthe Ġlearning Ġrate , Ġetc ., Ġa Ġrandomly Ġselected Ġ50 % Ġof Ġthe Ġtraining Ġdata Ġis Ġused Ġfor Ġvalidation . ĠWe Ġchoose Ġthe Ġparameters Ġthat Ġmaximize ĠR OU GE Ġscores Ġover Ġthis Ġvalidation Ġset . </s> To Ġimplement ĠPG - net Ġwe Ġuse ĠOpen N MT ĠB IB REF 21 Ġwith Ġthe Ġoriginal Ġset Ġof Ġparameters . ĠThe Ġout - of - domain Ġmodel Ġis Ġtrained Ġfor Ġ100 k Ġsteps Ġusing Ġthe ĠCNN / DM Ġdataset . ĠFollowing Ġbase Ġmodel Ġtraining , Ġwe Ġtune Ġthe Ġmodel Ġby Ġtraining Ġit Ġusing Ġstudent Ġreflections . ĠThe Ġtuning Ġis Ġdone Ġby Ġlowering Ġthe ĠLR Ġfrom Ġ0 . 15 Ġto Ġ0 . 1 Ġand Ġtraining Ġthe Ġmodel Ġfor Ġadditional Ġ500 Ġsteps . ĠThe Ġin - domain Ġmodel Ġis Ġtrained Ġonly Ġusing Ġreflections . ĠWe Ġuse Ġthe Ġsame Ġmodel Ġarchitecture Ġas Ġabove Ġand Ġtrain Ġthe Ġmodel Ġfor Ġ20 k Ġsteps Ġusing Ġad ag rad Ġand ĠLR Ġof Ġ0 . 15 . </s> Sy nt hesis ĠBas eline Ġ( for Ġtesting ĠH 3 , ĠH 4 ). ĠFollowing ĠB IB REF 12 , Ġwe Ġdeveloped Ġa Ġdata Ġsynthesis Ġbaseline Ġusing Ġword Ġreplacement Ġvia ĠWord Net . ĠThe Ġbaseline Ġiter ates Ġover Ġall Ġwords Ġin Ġa Ġsummary . ĠIf Ġword Ġ$ X $ Ġhas Ġ$ N $ Ġsyn onyms Ġin ĠWord Net , Ġthe Ġmodel Ġcreates Ġ$ N $ Ġnew Ġversions Ġof Ġthe Ġsummary Ġand Ġcorresponding Ġreflections Ġby Ġreplacing Ġthe Ġword Ġ$ X $ Ġwith Ġeach Ġof Ġthe Ġ$ N $ Ġsyn onyms . </s> Template ĠSynt hesis ĠModel Ġ( for Ġtesting ĠH 4 , ĠH 5 ). ĠTo Ġsynthes ize Ġsumm aries , Ġwe Ġuse Ġthe Ġsame Ġleave Ġone Ġcourse Ġout Ġapproach . ĠFor Ġeach Ġcourse , Ġwe Ġuse Ġthe Ġdata Ġfrom Ġthe Ġother Ġthree Ġcourses Ġto Ġtrain Ġthe Ġrewriting Ġmodule Ġand Ġtune Ġthe Ġscoring Ġlanguage Ġmodel . ĠWe Ġcan Ġalso Ġuse Ġthe Ġsumm aries Ġfrom ĠCNN / DM Ġdata Ġas Ġadditional Ġsamples Ġto Ġfurther Ġtrain Ġthe Ġrewriting Ġmodule . ĠWe Ġthen Ġstart Ġsynthes izing Ġdata Ġusing Ġthat Ġtraining Ġdata Ġas Ġinput . ĠFirst Ġtemplates Ġare Ġconstructed . ĠThe Ġtemplates Ġare Ġthen Ġclustered Ġinto Ġ8 Ġclusters . ĠWe Ġdecided Ġto Ġuse Ġ8 Ġto Ġavoid Ġclust ering Ġtemplates Ġfrom ĠPO I Ġwith ĠMP , Ġas Ġthe Ġtemplates Ġfrom Ġboth Ġprompts Ġwould Ġcontain Ġvery Ġdifferent Ġsupporting Ġwords . ĠWe Ġalso Ġwanted Ġto Ġavoid Ġa Ġhigh Ġlevel Ġof Ġdiss imilar ity Ġwithin Ġeach Ġcluster , Ġand Ġallow Ġsome Ġdiversity . ĠFollowing Ġthe Ġclust ering , Ġthe Ġrewriting Ġmodel Ġproduces Ġcandidate Ġsumm aries Ġfor Ġeach Ġhuman Ġsummary . ĠThe Ġrewriting Ġmodel Ġis Ġanother ĠPG - net Ġwith Ġthe Ġsame Ġexact Ġparameters . ĠAfter Ġproducing Ġthe Ġcandidate Ġsumm aries , Ġa Ġlanguage Ġmodel Ġis Ġused Ġto Ġscore Ġthem . ĠThe Ġlanguage Ġmodel Ġis Ġa Ġsingle Ġlayer ĠL ST M Ġlanguage Ġmodel Ġtrained Ġon Ġ36 K Ġsentences Ġfrom ĠWikipedia Ġand Ġfine Ġtuned Ġusing Ġstudent Ġreflections . ĠIn Ġthis Ġwork Ġwe Ġdecided Ġto Ġpick Ġonly Ġthe Ġhighest Ġ3 Ġscored Ġcandidate Ġsumm aries Ġas Ġsynthetic Ġdata , Ġto Ġavoid Ġadding Ġill - formed Ġsumm aries Ġto Ġthe Ġtraining Ġdata . ĠSince Ġwe Ġare Ġadding Ġ$ N $ Ġsynthetic Ġsumm aries Ġfor Ġeach Ġset Ġof Ġreflections , Ġthat Ġmeans Ġwe Ġare Ġessentially Ġdupl icating Ġthe Ġsize Ġof Ġour Ġoriginal Ġreflection Ġtraining Ġdata Ġby Ġ$ N $, Ġwhich Ġis Ġ3 Ġin Ġour Ġcase . ĠTable ĠT AB REF 11 Ġshows Ġa Ġhuman Ġsummary , Ġthe Ġkeywords Ġextracted , Ġthen Ġthe Ġoutput Ġof Ġinjecting Ġkeywords Ġin Ġa Ġdifferent Ġtemplate Ġusing Ġrewriting . </s> Template - based ĠSum mar ization Ġ( for Ġtesting ĠH 6 ). ĠWhile Ġthe Ġproposed Ġtemplate - based Ġmodel Ġwas Ġintended Ġfor Ġdata Ġsynthesis , Ġwith Ġminor Ġmodification Ġit Ġcan Ġbe Ġadapted Ġfor Ġsummar ization Ġitself . ĠBecause Ġthe Ġmodifications Ġintroduce Ġfew Ġparameters , Ġthe Ġmodel Ġis Ġsuitable Ġfor Ġsmall Ġdatasets . ĠRecall Ġthat Ġfor Ġdata Ġsynthesis , Ġthe Ġinput Ġto Ġthe Ġtemplate Ġmethod Ġis Ġa Ġsummary . ĠSince Ġfor Ġsummar ization Ġthe Ġinput Ġinstead Ġis Ġa Ġset Ġof Ġreflections , Ġwe Ġperform Ġkeyword Ġextraction Ġover Ġthe Ġset Ġof Ġreflections . ĠWe Ġthen Ġadd Ġan Ġextra Ġlog istic Ġregression Ġclass ifier Ġthat Ġuses Ġthe Ġset Ġof Ġreflections Ġas Ġinput Ġand Ġpredicts Ġa Ġcluster Ġof Ġtemplates Ġconstructed Ġfrom Ġother Ġcourses . ĠUsing Ġthe Ġkeywords Ġand Ġthe Ġpredicted Ġcluster Ġof Ġtemplates , Ġwe Ġuse Ġthe Ġsame Ġrewriting Ġmodel Ġto Ġproduce Ġcandidate Ġsumm aries . ĠThe Ġlast Ġstep Ġin Ġthe Ġpipeline Ġis Ġscoring . ĠIn Ġdata Ġsynthesis , Ġa Ġreference Ġsummary Ġis Ġused Ġfor Ġscoring ; Ġhowever , Ġin Ġsummar ization Ġwe Ġdon 't Ġhave Ġsuch Ġa Ġreference . ĠTo Ġscore Ġthe Ġcandidate Ġsumm aries , Ġthe Ġmodel Ġonly Ġuses Ġthe Ġlanguage Ġmodel Ġand Ġproduces Ġthe Ġcandidate Ġwith Ġthe Ġhighest Ġscore . </s> Results </s> R OU GE ĠEvaluation ĠResults . </s> Table ĠT AB REF 13 Ġpresents Ġsummar ization Ġperformance Ġresults Ġfor Ġthe Ġ4 Ġextract ive Ġbas elines , Ġfor Ġthe Ġoriginal Ġand Ġproposed Ġvariants Ġof ĠPG - net , Ġand Ġfinally Ġfor Ġtemplate - sum mar ization . ĠFollowing ĠB IB REF 0 , Ġperformance Ġis Ġevaluated Ġusing ĠR OU GE Ġ( 1 , Ġ2 , Ġand Ġ$ L $ ) ĠB IB REF 22 Ġon ĠF 1 . ĠThe Ġmotivation Ġfor Ġusing Ġdomain Ġtransfer Ġand Ġdata Ġsynthesis Ġis Ġour Ġhypothesis Ġ( H 1 ). ĠTable ĠT AB REF 13 Ġsupports Ġthis Ġhypothesis . ĠAll ĠR OU GE Ġscores Ġfor ĠPG - net Ġthat Ġoutper form Ġall Ġextract ive Ġbas elines Ġ( in Ġital ics ) Ġinvolve Ġtuning Ġand / or Ġuse Ġof Ġsynthes ised Ġdata , Ġexcept Ġfor Ġone ĠR - 1 Ġ( row Ġ18 ). </s> As Ġfor Ġour Ġsecond Ġhypothesis Ġ( H 2 ), Ġtable ĠT AB REF 13 Ġshows Ġthat Ġit Ġis Ġa Ġvalid Ġone . ĠFor ĠPG - net , Ġcomparing Ġthe ĠCNN / DM Ġout - of - domain Ġand ĠStudent ĠRef lection Ġin - domain Ġresults Ġin Ġrows Ġ( 5 Ġand Ġ6 ) Ġand Ġ( 17 Ġand Ġ18 ) Ġwith Ġtheir Ġcorresponding Ġtuned Ġresults Ġin Ġrows Ġ9 Ġand Ġ21 , Ġwe Ġsee Ġthat Ġfine Ġtuning Ġimproves ĠR - 1 , ĠR - 2 , Ġand ĠR -$ L $ Ġfor Ġall Ġcourses Ġ( rows Ġ5 , Ġ6 , Ġ9 Ġand Ġ17 , Ġ18 , Ġ21 ). ĠQual itatively , Ġthe Ġexamples Ġpresented Ġin ĠTable ĠT AB REF 11 Ġclearly Ġshow Ġthat Ġtuning Ġyields Ġa Ġmore Ġcoherent Ġand Ġrelevant Ġsummary . ĠOver Ġall Ġcourses , Ġthe Ġtuned Ġversion Ġof ĠPG - net Ġconsistently Ġoutper forms Ġthe Ġbest Ġbaseline Ġresult Ġfor Ġeach Ġmetric Ġ( rows Ġ9 Ġvs . Ġ1 , Ġ2 , Ġ3 , Ġ4 Ġand Ġ21 Ġvs . Ġ13 , Ġ14 , Ġ15 , Ġ16 ) Ġexcept Ġfor ĠR - 2 Ġin ĠStat 2016 . </s> To Ġvalidate Ġour Ġnext Ġset Ġof Ġhypothes ises Ġ( H 3 , ĠH 4 . ĠH 5 ), Ġwe Ġuse Ġthe Ġsynthes ized Ġdata Ġin Ġtwo Ġsettings : Ġeither Ġusing Ġit Ġfor Ġtraining Ġ( rows Ġ7 , Ġ8 Ġand Ġ19 , Ġ20 ) Ġor Ġtuning Ġ( rows Ġ10 , Ġ11 Ġand Ġ22 , Ġ23 ). ĠTable ĠT AB REF 13 Ġsupports ĠH 4 Ġby Ġshowing Ġthat Ġthe Ġproposed Ġsynthesis Ġmodel Ġoutper forms Ġthe ĠWord Net Ġbaseline Ġin Ġtraining Ġ( rows Ġ7 , Ġ8 Ġand Ġ19 , Ġ20 ) Ġexcept ĠStat 2016 , Ġand Ġtuning Ġ( 10 , Ġ11 Ġand Ġ22 , Ġ23 ) Ġover Ġall Ġcourses . ĠIt Ġalso Ġshows Ġthat Ġwhile Ġadding Ġsynthetic Ġdata Ġfrom Ġthe Ġbaseline Ġis Ġnot Ġalways Ġhelpful , Ġadding Ġsynthetic Ġdata Ġfrom Ġthe Ġtemplate Ġmodel Ġhelps Ġto Ġimprove Ġboth Ġthe Ġtraining Ġand Ġthe Ġtuning Ġprocess . ĠIn Ġboth ĠCS Ġand ĠENG R Ġcourses , Ġtuning Ġwith Ġsynthetic Ġdata Ġenhances Ġall ĠR OU GE Ġscores Ġcompared Ġto Ġtuning Ġwith Ġonly Ġthe Ġoriginal Ġdata . Ġ( rows Ġ9 Ġand Ġ11 ). ĠAs Ġfor ĠStat 2015 , ĠR - 1 Ġand ĠR -$ L $ Ġimproved , Ġwhile ĠR - 2 Ġdecreased . ĠFor ĠStat 2016 , ĠR - 2 Ġand ĠR -$ L $ Ġimproved , Ġand ĠR - 1 Ġdecreased Ġ( rows Ġ21 Ġand Ġ23 ). ĠTraining Ġwith Ġboth Ġstudent Ġreflection Ġdata Ġand Ġsynthetic Ġdata Ġcompared Ġto Ġtraining Ġwith Ġonly Ġstudent Ġreflection Ġdata Ġyields Ġsimilar Ġimprovements , Ġsupporting ĠH 3 Ġ( rows Ġ6 , Ġ8 Ġand Ġ18 , Ġ20 ). ĠWhile Ġthe Ġincrease Ġin ĠR OU GE Ġscores Ġis Ġsmall , Ġour Ġresults Ġshow Ġthat Ġenrich ing Ġtraining Ġdata Ġwith Ġsynthetic Ġdata Ġcan Ġbenefit Ġboth Ġthe Ġtraining Ġand Ġtuning Ġof Ġother Ġmodels . ĠIn Ġgeneral , Ġthe Ġbest Ġresults Ġare Ġobtained Ġwhen Ġusing Ġdata Ġsynthesis Ġfor Ġboth Ġtraining Ġand Ġtuning Ġ( rows Ġ11 Ġand Ġ23 ), Ġsupporting ĠH 5 . </s> Finally , Ġwhile Ġthe Ġgoal Ġof Ġour Ġtemplate Ġmodel Ġwas Ġto Ġsynthes ize Ġdata , Ġusing Ġit Ġfor Ġsummar ization Ġis Ġsurprisingly Ġcompetitive , Ġsupporting ĠH 6 . ĠWe Ġbelieve Ġthat Ġtraining Ġthe Ġmodel Ġwith Ġlittle Ġdata Ġis Ġdo able Ġdue Ġto Ġthe Ġsmall Ġnumber Ġof Ġparameters Ġ( log istic Ġregression Ġclass ifier Ġonly ). ĠWhile Ġrows Ġ12 Ġand Ġ24 Ġare Ġnever Ġthe Ġbest Ġresults , Ġthey Ġare Ġclose Ġto Ġthe Ġbest Ġinvolving Ġtuning . ĠThis Ġencourages Ġus Ġto Ġenhance Ġour Ġtemplate Ġmodel Ġand Ġexplore Ġtemplates Ġnot Ġso Ġtailored Ġto Ġour Ġdata . </s> Human ĠEvaluation ĠResults . ĠWhile Ġautomated Ġevaluation Ġmetrics Ġlike ĠR OU GE Ġmeasure Ġlex ical Ġsimilarity Ġbetween Ġmachine Ġand Ġhuman Ġsumm aries , Ġhumans Ġcan Ġbetter Ġmeasure Ġhow Ġcoherent Ġand Ġreadable Ġa Ġsummary Ġis . ĠOur Ġevaluation Ġstudy Ġinvestigates Ġwhether Ġtuning Ġthe ĠPG - net Ġmodel Ġincreases Ġsummary Ġco herence , Ġby Ġasking Ġeval u ators Ġto Ġselect Ġwhich Ġof Ġthree Ġsumm aries Ġfor Ġthe Ġsame Ġdocument Ġthey Ġlike Ġmost : Ġthe ĠPG - net Ġmodel Ġtrained Ġon ĠCNN / DM ; Ġthe Ġmodel Ġtrained Ġon Ġstudent Ġreflections ; Ġand Ġfinally Ġthe Ġmodel Ġtrained Ġon ĠCNN / DM Ġand Ġtuned Ġon Ġstudent Ġreflections . Ġ20 Ġeval u ators Ġwere Ġrecruited Ġfrom Ġour Ġinstitution Ġand Ġasked Ġto Ġeach Ġperform Ġ20 Ġannotations . ĠSum m aries Ġare Ġpresented Ġto Ġeval u ators Ġin Ġrandom Ġorder . ĠEval u ators Ġare Ġthen Ġasked Ġto Ġselect Ġthe Ġsummary Ġthey Ġfeel Ġto Ġbe Ġmost Ġreadable Ġand Ġcoherent . ĠUnlike ĠR OU GE , Ġwhich Ġmeasures Ġthe Ġcoverage Ġof Ġa Ġgenerated Ġsummary Ġrelative Ġto Ġa Ġreference Ġsummary , Ġour Ġeval u ators Ġdon 't Ġread Ġthe Ġreflections Ġor Ġreference Ġsummary . ĠThey Ġchoose Ġthe Ġsummary Ġthat Ġis Ġmost Ġcoherent Ġand Ġreadable , Ġregardless Ġof Ġthe Ġsource Ġof Ġthe Ġsummary . ĠFor Ġboth Ġcourses , Ġthe Ġmajority Ġof Ġselected Ġsumm aries Ġwere Ġproduced Ġby Ġthe Ġtuned Ġmodel Ġ( 49 % Ġfor ĠCS Ġand Ġ41 % Ġfor ĠStat 2015 ), Ġcompared Ġto Ġ( 31 % Ġfor ĠCS Ġand Ġ30 . 9 % Ġfor ĠStat 2015 ) Ġfor ĠCNN / DM Ġmodel , Ġand Ġ( 19 . 7 % Ġfor ĠCS Ġand Ġ28 . 5 % Ġfor ĠStat 2015 ) Ġfor Ġstudent Ġreflections Ġmodel . ĠThese Ġresults Ġagain Ġsuggest Ġthat Ġdomain Ġtransfer Ġcan Ġremedy Ġthe Ġsize Ġof Ġin - domain Ġdata Ġand Ġimprove Ġperformance . </s> Con clusions Ġand ĠFuture ĠWork </s> We Ġexplored Ġimproving Ġthe Ġperformance Ġof Ġneural Ġabstract ive Ġsummar izers Ġwhen Ġapplied Ġto Ġthe Ġlow Ġresource Ġdomain Ġof Ġstudent Ġreflections Ġusing Ġthree Ġapproaches : Ġdomain Ġtransfer , Ġdata Ġsynthesis Ġand Ġthe Ġcombination Ġof Ġboth . ĠFor Ġdomain Ġtransfer , Ġstate Ġof Ġthe Ġart Ġabstract ive Ġsummar ization Ġmodel Ġwas Ġpret rained Ġusing Ġout - of - domain Ġdata Ġ( CNN / DM ), Ġthen Ġtuned Ġusing Ġin - domain Ġdata Ġ( student Ġreflections ). ĠThe Ġprocess Ġof Ġtuning Ġimproved ĠR OU GE Ġscores Ġon Ġthe Ġstudent Ġreflection Ġdata , Ġand Ġat Ġthe Ġsame Ġtime Ġproduced Ġmore Ġreadable Ġsumm aries . ĠTo Ġincorporate Ġsynthetic Ġdata , Ġwe Ġproposed Ġa Ġnew Ġtemplate Ġbased Ġsynthesis Ġmodel Ġto Ġsynthes ize Ġnew Ġsumm aries . ĠWe Ġshowed Ġthat Ġenrich ing Ġthe Ġtraining Ġdata Ġwith Ġthis Ġsynthes ized Ġdata Ġcan Ġfurther Ġincrease Ġthe Ġbenefits Ġof Ġusing Ġdomain Ġtransfer Ġ/ Ġtuning Ġto Ġincrease ĠR OU GE Ġscores . ĠWe Ġadditionally Ġshowed Ġthat Ġthe Ġproposed Ġsynthesis Ġmodel Ġoutper formed Ġa Ġword Ġreplacement Ġsynthesis Ġbaseline . ĠFuture Ġplans Ġinclude Ġtrying Ġdomain Ġadaptation , Ġenhancing Ġthe Ġsynthes ising Ġprocess Ġby Ġusing Ġother Ġmodels , Ġfurther Ġexploring Ġtemplate - based Ġmethods , Ġand Ġextending Ġthe Ġanalysis Ġof Ġthe Ġsynthesis Ġmodel Ġto Ġcover Ġother Ġtypes Ġof Ġdata Ġlike Ġreviews Ġand Ġopinions . </s> Acknowled gments </s> The Ġresearch Ġreported Ġhere Ġwas Ġsupported , Ġin Ġwhole Ġor Ġin Ġpart , Ġby Ġthe Ġinstitute Ġof ĠEducation ĠSciences , ĠU . S . ĠDepartment Ġof ĠEducation , Ġthrough ĠGrant ĠR 305 A 180 477 Ġto Ġthe ĠUniversity Ġof ĠPittsburgh . ĠThe Ġopin ons Ġexpressed Ġare Ġthose Ġof Ġthe Ġauthors Ġand Ġdo Ġnot Ġrepresent Ġthe Ġviews Ġof Ġthe Ġinstitute Ġor Ġthe ĠU . S . ĠDepartment Ġof ĠEducation
Introduction Recently, with the emergence of neural seq2seq models, abstractive summarization methods have seen great performance strides BIBREF0, BIBREF1, BIBREF2. However, complex neural summarization models with thousands of parameters usually require a large amount of training data. In fact, much of the neural summarization work has been trained and tested in news domains where numerous large datasets exist. For example, the CNN/DailyMail (CNN/DM) BIBREF3, BIBREF4 and New York Times (NYT) datasets are in the magnitude of 300k and 700k documents, respectively. In contrast, in other domains such as student reflections, summarization datasets are only in the magnitude of tens or hundreds of documents (e.g., BIBREF5). We hypothesize that training complex neural abstractive summarization models in such domains will not yield good performing models, and we will indeed later show that this is the case for student reflections. To improve performance in low resource domains, we explore three directions. First, we explore domain transfer for abstractive summarization. While domain transfer is not new, compared to prior summarization studies BIBREF6, BIBREF7, our training (news) and tuning (student reflection) domains are quite dissimilar, and the in-domain data is small. Second, we propose a template-based synthesis method to create synthesized summaries, then explore the effect of enriching training data for abstractive summarization using the proposed model compared to a synthesis baseline. Lastly, we combine both directions. Evaluations of neural abstractive summarization method across four student reflection corpora show the utility of all three methods. Related Work Abstractive Summarization. Abstractive summarization aims to generate coherent summaries with high readability, and has seen increasing interest and improved performance due to the emergence of seq2seq models BIBREF8 and attention mechanisms BIBREF9. For example, BIBREF0, BIBREF2, and BIBREF1 in addition to using encoder-decoder model with attention, they used pointer networks to solve the out of vocabulary issue, while BIBREF0 used coverage mechanism to solve the problem of word repetition. In addition, BIBREF2 and BIBREF10 used reinforcement learning in an end-to-end setting. To our knowledge, training such neural abstractive summarization models in low resource domains using domain transfer has not been thoroughly explored on domains different than news. For example, BIBREF4 reported the results of training on CNN/DM data while evaluating on DUC data without any tuning. Note that these two datasets are both in the news domain, and both consist of well written, structured documents. The domain transfer experiments of BIBREF1 similarly used two different news summarization datasets (CNN/DM and NYT). Our work differs in several ways from these two prior domain transfer efforts. First, our experiments involve two entirely different domains: news and student reflections. Unlike news, student reflection documents lack global structure, are repetitive, and contain many sentence fragments and grammatical mistakes. Second, the prior approaches either trained a part of the model using NYT data while retaining the other part of the model trained only on CNN/DM data BIBREF1, or didn't perform any tuning at all BIBREF4. In contrast, we do the training in two consecutive phases, pretraining and fine tuning. Finally, BIBREF1 reported that while training with domain transfer outperformed training only on out-of-domain data, it was not able to beat training only on in-domain data. This is likely because their in and out-of-domain data sizes are comparable, unlike in our case of scarce in-domain data. In a different approach to abstractive summarization, BIBREF11 developed a soft template based neural method consisting of an end-to-end deep model for template retrieval, reranking and summary rewriting. While we also develop a template based model, our work differs in both model structure and purpose. Data Synthesis. Data synthesis for text summarization is underexplored, with most prior work focusing on machine translation, and text normalization. BIBREF12 proposed doing data augmentation through word replacement, using WordNet BIBREF13 and vector space similarity, respectively. We will use a WordNet replacement method as a baseline synthesis method in the experiments described below. In contrast, BIBREF14 synthesized/augmented data through back-translation and word replacement using language models. BIBREF15 is another recent work that was done in parallel and is very close to ours. However, in addition to the difference in both our and their model, we think it might be infeasible to back generate student reflections from a human summary, especially an abstractive one. Reflection Summarization Dataset Student reflections are comments provided by students in response to a set of instructor prompts. The prompts are directed towards gathering students' feedback on course material. Student reflections are collected directly following each of a set of classroom lectures over a semester. In this paper, the set of reflections for each prompt in each lecture is considered a student reflection document. The objective of our work is to provide a comprehensive and meaningful abstractive summary of each student reflection document. Our dataset consists of documents and summaries from four course instantiations: ENGR (Introduction to Materials Science and Engineering), Stat2015 and Stat2016 (Statistics for Industrial Engineers, taught in 2015 and 2016, respectively), and CS (Data Structures in Computer Science). All reflections were collected in response to two pedagogically-motivated prompts BIBREF16: “Point of Interest (POI): Describe what you found most interesting in today's class” and “Muddiest Point (MP): Describe what was confusing or needed more detail.” For each reflection document, at least one human (either a TA or domain expert) created summaries. Table TABREF4 shows example reference summary produced by one annotator for the CS course. Table TABREF5 summarizes the dataset in terms of number of lectures, number of prompts per lecture, average number of reflections per prompt, and number of abstractive reference summaries for each set of reflections. Explored Approaches for Limited Resources To overcome the size issue of the student reflection dataset, we first explore the effect of incorporating domain transfer into a recent abstractive summarization model: pointer networks with coverage mechanism (PG-net)BIBREF0. To experiment with domain transfer, the model was pretrained using the CNN/DM dataset, then fine tuned using the student reflection dataset (see the Experiments section). A second approach we explore to overcome the lack of reflection data is data synthesis. We first propose a template model for synthesizing new data, then investigate the performance impact of using this data when training the summarization model. The proposed model makes use of the nature of datasets such as ours, where the reference summaries tend to be close in structure: humans try to find the major points that students raise, then present the points in a way that marks their relative importance (recall the CS example in Table TABREF4). Our third explored approach is to combine domain transfer with data synthesis. Proposed Template-Based Synthesis Model Our motivation for using templates for data synthesis is that seq2seq synthesis models (as discussed in related work) tend to generate irrelevant and repeated words BIBREF17, while templates can produce more coherent and concise output. Also, extracting templates can be done either manually or automatically typically by training a few parameters or even doing no training, then external information in the form of keywords or snippets can be populated into the templates with the help of more sophisticated models. Accordingly, using templates can be very tempting for domains with limited resources such as ours. Model Structure. The model consists of 4 modules: 1. Template extraction: To convert human summaries into templates, we remove keywords in the summary to leave only non-keywords. We use Rapid Automatic Keyword Extraction (RAKE) BIBREF18 to identify keywords. 2. Template clustering: Upon converting human summaries into templates, we cluster them into $N$ clusters with the goal of using any template from the same cluster interchangeably. A template is first converted into embeddings using a pretrained BERT model BIBREF19, where template embedding is constructed by average pooling word embeddings. Templates are then clustered using k-medoid. 3. Summary rewriting: An encoder-attention-decoder with pointer network is trained to perform the rewriting task. The model is trained to inject keywords into a template and perform rewriting into a coherent paragraph. The produced rewrites are considered as candidate summaries. 4. Summary selection: After producing candidate summaries, we need to pick the best ones. We argue that the best candidates are those that are coherent and also convey the same meaning as the original human summary. We thus use a hybrid metric to score candidates, where the metric is a weighted sum of two scores and is calculated using Equations 1, 2, and 3. Eq.1 measures coherency using a language model (LM), Eq.2 measures how close a candidate is to a human summary using ROUGE scores, while Eq.3 picks the highest scored $N$ candidates as the final synthetic set. CS and HS are a candidate and human summary. $P(w)$ is the probability of word $w$ using a language model. $\alpha , \beta $ are weighting parameters. In this work we use $\alpha =\beta =1$ for all experiments. $R_{i}(CS,HS)$ is ROUGE-i score between CS and HS for i=1, 2, and $l$. Model Training. Before using the synthesis model, some of the constructing modules (rewriting module, scoring LM) need training. To train the rewriting model, we use another dataset consisting of a set of samples, where each sample can be a text snippet (sentence, paragraph, etc.). For each sample, keywords are extracted using RAKE, then removed. The keywords plus the sample with no keywords are then passed to the rewriting model. The training objective of this model is to reconstruct the original sample, which can be seen as trying to inject extracted keywords back into a template. Model Usage. To use the synthesis model to generate new samples, the set of human summaries are fed to the model, passing through the sub-modules in the following order: 1. Human summaries first pass through the template extraction module, converting each summary $s_i$ into template $t_i$ and the corresponding keywords $kw_i$. 2. Templates are then passed to the clustering module, producing a set of clusters. Each cluster $C$ contains a number of similar templates. 3. For each template $t_i$ and corresponding keywords $kw_i$ from step 1, find the cluster $C_i$ that contains the template $t_i$, then pass the set of templates within that clusters $\lbrace t_j\rbrace \forall {j},$ if $t_j \in C_i$ alongside the keywords $kw_i$ to the summary rewriting module. This will produce a set of candidate summaries. 4. The summary selection module scores and selects the highest $N$ candidates as the synthetic summaries. Experiments Our experimental designs address the following hypotheses: Hypothesis 1 (H1) : Training complex abstractive models with limited in-domain or large quantities of out-of-domain data won't be enough to outperform extractive baselines. Hypothesis 2 (H2) : Domain transfer helps abstractive models even if in-domain and out-of-domain data are very different and the amount of in-domain data is very small. Hypothesis 3 (H3) : Enriching abstractive training data with synthetic data helps overcome in-domain data scarcity. Hypothesis 4 (H4) : The proposed template-based synthesis model outperforms a simple word replacement model. Hypothesis 5 (H5) : Combining domain transfer with data synthesis outperforms using each approach on its own. Hypothesis 6 (H6) : The synthesis model can be extended to perform reflection summarization directly. Extractive Baselines (for testing H1). While BIBREF0 used Lead-3 as an extractive baseline, in our data sentence order doesn't matter as reflections are independent. We thus use a similar in concept baseline: randomly select N reflections. Since the baseline is random we report the average result of 100 runs. Following BIBREF5, we compare results to MEAD BIBREF20 and to BIBREF5's extractive phrase-based model. Since these models extracted 5 phrases as extractive summary, we use N=5 for our three extractive baselines. Additionally we compare to running only the extractive part of Fast-RL. Domain Transfer (for testing H2, H5). To observe the impact of using out-of-domain (news) data for pretraining to compensate for low resource in-domain (reflection) data, we train 3 variants of PG-net: model training on CNN/DM; model training on reflections; and model training on CNN/DM then tuning using reflections. Table TABREF11 shows example summaries generated by the three variants of PG-net for a CS document. For all experiments where reflections are used for training/tuning, we train using a leave one course out approach (i.e, in each fold, three courses are used for training and the remaining course for testing). If the experiment involves tuning a combined dictionary of CNN/DM and reflections is used to avoid domain mismatch. To tune model parameters, the best number of steps for training, the learning rate, etc., a randomly selected 50% of the training data is used for validation. We choose the parameters that maximize ROUGE scores over this validation set. To implement PG-net we use OpenNMT BIBREF21 with the original set of parameters. The out-of-domain model is trained for 100k steps using the CNN/DM dataset. Following base model training, we tune the model by training it using student reflections. The tuning is done by lowering the LR from 0.15 to 0.1 and training the model for additional 500 steps. The in-domain model is trained only using reflections. We use the same model architecture as above and train the model for 20k steps using adagrad and LR of 0.15. Synthesis Baseline (for testing H3, H4). Following BIBREF12, we developed a data synthesis baseline using word replacement via WordNet. The baseline iterates over all words in a summary. If word $X$ has $N$ synonyms in WordNet, the model creates $N$ new versions of the summary and corresponding reflections by replacing the word $X$ with each of the $N$ synonyms. Template Synthesis Model (for testing H4, H5). To synthesize summaries, we use the same leave one course out approach. For each course, we use the data from the other three courses to train the rewriting module and tune the scoring language model. We can also use the summaries from CNN/DM data as additional samples to further train the rewriting module. We then start synthesizing data using that training data as input. First templates are constructed. The templates are then clustered into 8 clusters. We decided to use 8 to avoid clustering templates from POI with MP, as the templates from both prompts would contain very different supporting words. We also wanted to avoid a high level of dissimilarity within each cluster, and allow some diversity. Following the clustering, the rewriting model produces candidate summaries for each human summary. The rewriting model is another PG-net with the same exact parameters. After producing the candidate summaries, a language model is used to score them. The language model is a single layer LSTM language model trained on 36K sentences from Wikipedia and fine tuned using student reflections. In this work we decided to pick only the highest 3 scored candidate summaries as synthetic data, to avoid adding ill-formed summaries to the training data. Since we are adding $N$ synthetic summaries for each set of reflections, that means we are essentially duplicating the size of our original reflection training data by $N$, which is 3 in our case. Table TABREF11 shows a human summary, the keywords extracted, then the output of injecting keywords in a different template using rewriting. Template-based Summarization (for testing H6). While the proposed template-based model was intended for data synthesis, with minor modification it can be adapted for summarization itself. Because the modifications introduce few parameters, the model is suitable for small datasets. Recall that for data synthesis, the input to the template method is a summary. Since for summarization the input instead is a set of reflections, we perform keyword extraction over the set of reflections. We then add an extra logistic regression classifier that uses the set of reflections as input and predicts a cluster of templates constructed from other courses. Using the keywords and the predicted cluster of templates, we use the same rewriting model to produce candidate summaries. The last step in the pipeline is scoring. In data synthesis, a reference summary is used for scoring; however, in summarization we don't have such a reference. To score the candidate summaries, the model only uses the language model and produces the candidate with the highest score. Results ROUGE Evaluation Results. Table TABREF13 presents summarization performance results for the 4 extractive baselines, for the original and proposed variants of PG-net, and finally for template-summarization. Following BIBREF0, performance is evaluated using ROUGE (1, 2, and $L$) BIBREF22 on F1. The motivation for using domain transfer and data synthesis is our hypothesis (H1). Table TABREF13 supports this hypothesis. All ROUGE scores for PG-net that outperform all extractive baselines (in italics) involve tuning and/or use of synthesised data, except for one R-1 (row 18). As for our second hypothesis (H2), table TABREF13 shows that it is a valid one. For PG-net, comparing the CNN/DM out-of-domain and Student Reflection in-domain results in rows (5 and 6) and (17 and 18) with their corresponding tuned results in rows 9 and 21, we see that fine tuning improves R-1, R-2, and R-$L$ for all courses (rows 5, 6, 9 and 17, 18, 21). Qualitatively, the examples presented in Table TABREF11 clearly show that tuning yields a more coherent and relevant summary. Over all courses, the tuned version of PG-net consistently outperforms the best baseline result for each metric (rows 9 vs. 1, 2, 3, 4 and 21 vs. 13, 14, 15, 16) except for R-2 in Stat2016. To validate our next set of hypothesises (H3, H4. H5), we use the synthesized data in two settings: either using it for training (rows 7, 8 and 19, 20) or tuning (rows 10, 11 and 22, 23). Table TABREF13 supports H4 by showing that the proposed synthesis model outperforms the WordNet baseline in training (rows 7, 8 and 19, 20) except Stat2016, and tuning (10, 11 and 22, 23) over all courses. It also shows that while adding synthetic data from the baseline is not always helpful, adding synthetic data from the template model helps to improve both the training and the tuning process. In both CS and ENGR courses, tuning with synthetic data enhances all ROUGE scores compared to tuning with only the original data. (rows 9 and 11). As for Stat2015, R-1 and R-$L$ improved, while R-2 decreased. For Stat2016, R-2 and R-$L$ improved, and R-1 decreased (rows 21 and 23). Training with both student reflection data and synthetic data compared to training with only student reflection data yields similar improvements, supporting H3 (rows 6, 8 and 18, 20). While the increase in ROUGE scores is small, our results show that enriching training data with synthetic data can benefit both the training and tuning of other models. In general, the best results are obtained when using data synthesis for both training and tuning (rows 11 and 23), supporting H5. Finally, while the goal of our template model was to synthesize data, using it for summarization is surprisingly competitive, supporting H6. We believe that training the model with little data is doable due to the small number of parameters (logistic regression classifier only). While rows 12 and 24 are never the best results, they are close to the best involving tuning. This encourages us to enhance our template model and explore templates not so tailored to our data. Human Evaluation Results. While automated evaluation metrics like ROUGE measure lexical similarity between machine and human summaries, humans can better measure how coherent and readable a summary is. Our evaluation study investigates whether tuning the PG-net model increases summary coherence, by asking evaluators to select which of three summaries for the same document they like most: the PG-net model trained on CNN/DM; the model trained on student reflections; and finally the model trained on CNN/DM and tuned on student reflections. 20 evaluators were recruited from our institution and asked to each perform 20 annotations. Summaries are presented to evaluators in random order. Evaluators are then asked to select the summary they feel to be most readable and coherent. Unlike ROUGE, which measures the coverage of a generated summary relative to a reference summary, our evaluators don't read the reflections or reference summary. They choose the summary that is most coherent and readable, regardless of the source of the summary. For both courses, the majority of selected summaries were produced by the tuned model (49% for CS and 41% for Stat2015), compared to (31% for CS and 30.9% for Stat2015) for CNN/DM model, and (19.7% for CS and 28.5% for Stat2015) for student reflections model. These results again suggest that domain transfer can remedy the size of in-domain data and improve performance. Conclusions and Future Work We explored improving the performance of neural abstractive summarizers when applied to the low resource domain of student reflections using three approaches: domain transfer, data synthesis and the combination of both. For domain transfer, state of the art abstractive summarization model was pretrained using out-of-domain data (CNN/DM), then tuned using in-domain data (student reflections). The process of tuning improved ROUGE scores on the student reflection data, and at the same time produced more readable summaries. To incorporate synthetic data, we proposed a new template based synthesis model to synthesize new summaries. We showed that enriching the training data with this synthesized data can further increase the benefits of using domain transfer / tuning to increase ROUGE scores. We additionally showed that the proposed synthesis model outperformed a word replacement synthesis baseline. Future plans include trying domain adaptation, enhancing the synthesising process by using other models, further exploring template-based methods, and extending the analysis of the synthesis model to cover other types of data like reviews and opinions. Acknowledgments The research reported here was supported, in whole or in part, by the institute of Education Sciences, U.S. Department of Education, through Grant R305A180477 to the University of Pittsburgh. The opinons expressed are those of the authors and do not represent the views of the institute or the U.S. Department of Education
What is the recent abstractive summarization method in this paper?
c21b87c97d1afac85ece2450ee76d01c946de668
What Ġis Ġthe Ġrecent Ġabstract ive Ġsummar ization Ġmethod Ġin Ġthis Ġpaper ?
pointer Ġnetworks Ġwith Ġcoverage Ġmechanism Ġ( PG - net )
<s> What Ġis Ġthe Ġrecent Ġabstract ive Ġsummar ization Ġmethod Ġin Ġthis Ġpaper ? </s> Introduction </s> Recently , Ġwith Ġthe Ġemergence Ġof Ġneural Ġseq 2 seq Ġmodels , Ġabstract ive Ġsummar ization Ġmethods Ġhave Ġseen Ġgreat Ġperformance Ġstrides ĠB IB REF 0 , ĠB IB REF 1 , ĠB IB REF 2 . ĠHowever , Ġcomplex Ġneural Ġsummar ization Ġmodels Ġwith Ġthousands Ġof Ġparameters Ġusually Ġrequire Ġa Ġlarge Ġamount Ġof Ġtraining Ġdata . ĠIn Ġfact , Ġmuch Ġof Ġthe Ġneural Ġsummar ization Ġwork Ġhas Ġbeen Ġtrained Ġand Ġtested Ġin Ġnews Ġdomains Ġwhere Ġnumerous Ġlarge Ġdatasets Ġexist . ĠFor Ġexample , Ġthe ĠCNN / Daily Mail Ġ( CNN / DM ) ĠB IB REF 3 , ĠB IB REF 4 Ġand ĠNew ĠYork ĠTimes Ġ( NY T ) Ġdatasets Ġare Ġin Ġthe Ġmagnitude Ġof Ġ300 k Ġand Ġ700 k Ġdocuments , Ġrespectively . ĠIn Ġcontrast , Ġin Ġother Ġdomains Ġsuch Ġas Ġstudent Ġreflections , Ġsummar ization Ġdatasets Ġare Ġonly Ġin Ġthe Ġmagnitude Ġof Ġtens Ġor Ġhundreds Ġof Ġdocuments Ġ( e . g ., ĠB IB REF 5 ). ĠWe Ġhypothes ize Ġthat Ġtraining Ġcomplex Ġneural Ġabstract ive Ġsummar ization Ġmodels Ġin Ġsuch Ġdomains Ġwill Ġnot Ġyield Ġgood Ġperforming Ġmodels , Ġand Ġwe Ġwill Ġindeed Ġlater Ġshow Ġthat Ġthis Ġis Ġthe Ġcase Ġfor Ġstudent Ġreflections . </s> To Ġimprove Ġperformance Ġin Ġlow Ġresource Ġdomains , Ġwe Ġexplore Ġthree Ġdirections . ĠFirst , Ġwe Ġexplore Ġdomain Ġtransfer Ġfor Ġabstract ive Ġsummar ization . ĠWhile Ġdomain Ġtransfer Ġis Ġnot Ġnew , Ġcompared Ġto Ġprior Ġsummar ization Ġstudies ĠB IB REF 6 , ĠB IB REF 7 , Ġour Ġtraining Ġ( news ) Ġand Ġtuning Ġ( student Ġreflection ) Ġdomains Ġare Ġquite Ġdiss imilar , Ġand Ġthe Ġin - domain Ġdata Ġis Ġsmall . ĠSecond , Ġwe Ġpropose Ġa Ġtemplate - based Ġsynthesis Ġmethod Ġto Ġcreate Ġsynthes ized Ġsumm aries , Ġthen Ġexplore Ġthe Ġeffect Ġof Ġenrich ing Ġtraining Ġdata Ġfor Ġabstract ive Ġsummar ization Ġusing Ġthe Ġproposed Ġmodel Ġcompared Ġto Ġa Ġsynthesis Ġbaseline . ĠLastly , Ġwe Ġcombine Ġboth Ġdirections . ĠEval uations Ġof Ġneural Ġabstract ive Ġsummar ization Ġmethod Ġacross Ġfour Ġstudent Ġreflection Ġcorpor a Ġshow Ġthe Ġutility Ġof Ġall Ġthree Ġmethods . </s> Related ĠWork </s> Abstract ive ĠSum mar ization . ĠAbstract ive Ġsummar ization Ġaims Ġto Ġgenerate Ġcoherent Ġsumm aries Ġwith Ġhigh Ġread ability , Ġand Ġhas Ġseen Ġincreasing Ġinterest Ġand Ġimproved Ġperformance Ġdue Ġto Ġthe Ġemergence Ġof Ġseq 2 seq Ġmodels ĠB IB REF 8 Ġand Ġattention Ġmechanisms ĠB IB REF 9 . ĠFor Ġexample , ĠB IB REF 0 , ĠB IB REF 2 , Ġand ĠB IB REF 1 Ġin Ġaddition Ġto Ġusing Ġenc oder - dec oder Ġmodel Ġwith Ġattention , Ġthey Ġused Ġpointer Ġnetworks Ġto Ġsolve Ġthe Ġout Ġof Ġvocabulary Ġissue , Ġwhile ĠB IB REF 0 Ġused Ġcoverage Ġmechanism Ġto Ġsolve Ġthe Ġproblem Ġof Ġword Ġrepetition . ĠIn Ġaddition , ĠB IB REF 2 Ġand ĠB IB REF 10 Ġused Ġreinforcement Ġlearning Ġin Ġan Ġend - to - end Ġsetting . </s> To Ġour Ġknowledge , Ġtraining Ġsuch Ġneural Ġabstract ive Ġsummar ization Ġmodels Ġin Ġlow Ġresource Ġdomains Ġusing Ġdomain Ġtransfer Ġhas Ġnot Ġbeen Ġthoroughly Ġexplored Ġon Ġdomains Ġdifferent Ġthan Ġnews . ĠFor Ġexample , ĠB IB REF 4 Ġreported Ġthe Ġresults Ġof Ġtraining Ġon ĠCNN / DM Ġdata Ġwhile Ġevaluating Ġon ĠD UC Ġdata Ġwithout Ġany Ġtuning . ĠNote Ġthat Ġthese Ġtwo Ġdatasets Ġare Ġboth Ġin Ġthe Ġnews Ġdomain , Ġand Ġboth Ġconsist Ġof Ġwell Ġwritten , Ġstructured Ġdocuments . ĠThe Ġdomain Ġtransfer Ġexperiments Ġof ĠB IB REF 1 Ġsimilarly Ġused Ġtwo Ġdifferent Ġnews Ġsummar ization Ġdatasets Ġ( CNN / DM Ġand ĠNYT ). ĠOur Ġwork Ġdiffers Ġin Ġseveral Ġways Ġfrom Ġthese Ġtwo Ġprior Ġdomain Ġtransfer Ġefforts . ĠFirst , Ġour Ġexperiments Ġinvolve Ġtwo Ġentirely Ġdifferent Ġdomains : Ġnews Ġand Ġstudent Ġreflections . ĠUnlike Ġnews , Ġstudent Ġreflection Ġdocuments Ġlack Ġglobal Ġstructure , Ġare Ġrepetitive , Ġand Ġcontain Ġmany Ġsentence Ġfragments Ġand Ġgram matical Ġmistakes . ĠSecond , Ġthe Ġprior Ġapproaches Ġeither Ġtrained Ġa Ġpart Ġof Ġthe Ġmodel Ġusing ĠNYT Ġdata Ġwhile Ġretaining Ġthe Ġother Ġpart Ġof Ġthe Ġmodel Ġtrained Ġonly Ġon ĠCNN / DM Ġdata ĠB IB REF 1 , Ġor Ġdidn 't Ġperform Ġany Ġtuning Ġat Ġall ĠB IB REF 4 . ĠIn Ġcontrast , Ġwe Ġdo Ġthe Ġtraining Ġin Ġtwo Ġconsecutive Ġphases , Ġpret raining Ġand Ġfine Ġtuning . ĠFinally , ĠB IB REF 1 Ġreported Ġthat Ġwhile Ġtraining Ġwith Ġdomain Ġtransfer Ġoutper formed Ġtraining Ġonly Ġon Ġout - of - domain Ġdata , Ġit Ġwas Ġnot Ġable Ġto Ġbeat Ġtraining Ġonly Ġon Ġin - domain Ġdata . ĠThis Ġis Ġlikely Ġbecause Ġtheir Ġin Ġand Ġout - of - domain Ġdata Ġsizes Ġare Ġcomparable , Ġunlike Ġin Ġour Ġcase Ġof Ġscarce Ġin - domain Ġdata . </s> In Ġa Ġdifferent Ġapproach Ġto Ġabstract ive Ġsummar ization , ĠB IB REF 11 Ġdeveloped Ġa Ġsoft Ġtemplate Ġbased Ġneural Ġmethod Ġconsisting Ġof Ġan Ġend - to - end Ġdeep Ġmodel Ġfor Ġtemplate Ġretrieval , Ġre ranking Ġand Ġsummary Ġrewriting . ĠWhile Ġwe Ġalso Ġdevelop Ġa Ġtemplate Ġbased Ġmodel , Ġour Ġwork Ġdiffers Ġin Ġboth Ġmodel Ġstructure Ġand Ġpurpose . </s> Data ĠSynt hesis . ĠData Ġsynthesis Ġfor Ġtext Ġsummar ization Ġis Ġund ere x pl ored , Ġwith Ġmost Ġprior Ġwork Ġfocusing Ġon Ġmachine Ġtranslation , Ġand Ġtext Ġnormal ization . ĠB IB REF 12 Ġproposed Ġdoing Ġdata Ġaug mentation Ġthrough Ġword Ġreplacement , Ġusing ĠWord Net ĠB IB REF 13 Ġand Ġvector Ġspace Ġsimilarity , Ġrespectively . ĠWe Ġwill Ġuse Ġa ĠWord Net Ġreplacement Ġmethod Ġas Ġa Ġbaseline Ġsynthesis Ġmethod Ġin Ġthe Ġexperiments Ġdescribed Ġbelow . ĠIn Ġcontrast , ĠB IB REF 14 Ġsynthes ized / au gment ed Ġdata Ġthrough Ġback - translation Ġand Ġword Ġreplacement Ġusing Ġlanguage Ġmodels . ĠB IB REF 15 Ġis Ġanother Ġrecent Ġwork Ġthat Ġwas Ġdone Ġin Ġparallel Ġand Ġis Ġvery Ġclose Ġto Ġours . ĠHowever , Ġin Ġaddition Ġto Ġthe Ġdifference Ġin Ġboth Ġour Ġand Ġtheir Ġmodel , Ġwe Ġthink Ġit Ġmight Ġbe Ġinf eas ible Ġto Ġback Ġgenerate Ġstudent Ġreflections Ġfrom Ġa Ġhuman Ġsummary , Ġespecially Ġan Ġabstract ive Ġone . </s> Ref lection ĠSum mar ization ĠDat as et </s> Student Ġreflections Ġare Ġcomments Ġprovided Ġby Ġstudents Ġin Ġresponse Ġto Ġa Ġset Ġof Ġinstructor Ġprompts . ĠThe Ġprompts Ġare Ġdirected Ġtowards Ġgathering Ġstudents ' Ġfeedback Ġon Ġcourse Ġmaterial . ĠStudent Ġreflections Ġare Ġcollected Ġdirectly Ġfollowing Ġeach Ġof Ġa Ġset Ġof Ġclassroom Ġlectures Ġover Ġa Ġsemester . ĠIn Ġthis Ġpaper , Ġthe Ġset Ġof Ġreflections Ġfor Ġeach Ġprompt Ġin Ġeach Ġlecture Ġis Ġconsidered Ġa Ġstudent Ġreflection Ġdocument . ĠThe Ġobjective Ġof Ġour Ġwork Ġis Ġto Ġprovide Ġa Ġcomprehensive Ġand Ġmeaningful Ġabstract ive Ġsummary Ġof Ġeach Ġstudent Ġreflection Ġdocument . ĠOur Ġdataset Ġconsists Ġof Ġdocuments Ġand Ġsumm aries Ġfrom Ġfour Ġcourse Ġinstant iations : ĠENG R Ġ( Introduction Ġto ĠMaterials ĠScience Ġand ĠEngineering ), ĠStat 2015 Ġand ĠStat 2016 Ġ( Statistics Ġfor ĠIndustrial ĠEngineers , Ġtaught Ġin Ġ2015 Ġand Ġ2016 , Ġrespectively ), Ġand ĠCS Ġ( Data ĠStruct ures Ġin ĠComputer ĠScience ). ĠAll Ġreflections Ġwere Ġcollected Ġin Ġresponse Ġto Ġtwo Ġped agog ically - mot ivated Ġprompts ĠB IB REF 16 : ĠâĢ ľ Point Ġof ĠInterest Ġ( PO I ): ĠDesc ribe Ġwhat Ġyou Ġfound Ġmost Ġinteresting Ġin Ġtoday 's Ġclass âĢ Ŀ Ġand ĠâĢ ľ M udd iest ĠPoint Ġ( MP ): ĠDesc ribe Ġwhat Ġwas Ġconfusing Ġor Ġneeded Ġmore Ġdetail . âĢ Ŀ </s> For Ġeach Ġreflection Ġdocument , Ġat Ġleast Ġone Ġhuman Ġ( either Ġa ĠTA Ġor Ġdomain Ġexpert ) Ġcreated Ġsumm aries . ĠTable ĠT AB REF 4 Ġshows Ġexample Ġreference Ġsummary Ġproduced Ġby Ġone Ġannot ator Ġfor Ġthe ĠCS Ġcourse . ĠTable ĠT AB REF 5 Ġsummarizes Ġthe Ġdataset Ġin Ġterms Ġof Ġnumber Ġof Ġlectures , Ġnumber Ġof Ġprompts Ġper Ġlecture , Ġaverage Ġnumber Ġof Ġreflections Ġper Ġprompt , Ġand Ġnumber Ġof Ġabstract ive Ġreference Ġsumm aries Ġfor Ġeach Ġset Ġof Ġreflections . </s> Expl ored ĠAppro aches Ġfor ĠLimited ĠResources </s> To Ġovercome Ġthe Ġsize Ġissue Ġof Ġthe Ġstudent Ġreflection Ġdataset , Ġwe Ġfirst Ġexplore Ġthe Ġeffect Ġof Ġincorporating Ġdomain Ġtransfer Ġinto Ġa Ġrecent Ġabstract ive Ġsummar ization Ġmodel : Ġpointer Ġnetworks Ġwith Ġcoverage Ġmechanism Ġ( PG - net ) BI B REF 0 . ĠTo Ġexperiment Ġwith Ġdomain Ġtransfer , Ġthe Ġmodel Ġwas Ġpret rained Ġusing Ġthe ĠCNN / DM Ġdataset , Ġthen Ġfine Ġtuned Ġusing Ġthe Ġstudent Ġreflection Ġdataset Ġ( see Ġthe ĠExper iments Ġsection ). ĠA Ġsecond Ġapproach Ġwe Ġexplore Ġto Ġovercome Ġthe Ġlack Ġof Ġreflection Ġdata Ġis Ġdata Ġsynthesis . ĠWe Ġfirst Ġpropose Ġa Ġtemplate Ġmodel Ġfor Ġsynthes izing Ġnew Ġdata , Ġthen Ġinvestigate Ġthe Ġperformance Ġimpact Ġof Ġusing Ġthis Ġdata Ġwhen Ġtraining Ġthe Ġsummar ization Ġmodel . ĠThe Ġproposed Ġmodel Ġmakes Ġuse Ġof Ġthe Ġnature Ġof Ġdatasets Ġsuch Ġas Ġours , Ġwhere Ġthe Ġreference Ġsumm aries Ġtend Ġto Ġbe Ġclose Ġin Ġstructure : Ġhumans Ġtry Ġto Ġfind Ġthe Ġmajor Ġpoints Ġthat Ġstudents Ġraise , Ġthen Ġpresent Ġthe Ġpoints Ġin Ġa Ġway Ġthat Ġmarks Ġtheir Ġrelative Ġimportance Ġ( rec all Ġthe ĠCS Ġexample Ġin ĠTable ĠT AB REF 4 ). ĠOur Ġthird Ġexplored Ġapproach Ġis Ġto Ġcombine Ġdomain Ġtransfer Ġwith Ġdata Ġsynthesis . </s> Prop osed ĠTemplate - Based ĠSynt hesis ĠModel </s> Our Ġmotivation Ġfor Ġusing Ġtemplates Ġfor Ġdata Ġsynthesis Ġis Ġthat Ġseq 2 seq Ġsynthesis Ġmodels Ġ( as Ġdiscussed Ġin Ġrelated Ġwork ) Ġtend Ġto Ġgenerate Ġirrelevant Ġand Ġrepeated Ġwords ĠB IB REF 17 , Ġwhile Ġtemplates Ġcan Ġproduce Ġmore Ġcoherent Ġand Ġconcise Ġoutput . ĠAlso , Ġextracting Ġtemplates Ġcan Ġbe Ġdone Ġeither Ġmanually Ġor Ġautomatically Ġtypically Ġby Ġtraining Ġa Ġfew Ġparameters Ġor Ġeven Ġdoing Ġno Ġtraining , Ġthen Ġexternal Ġinformation Ġin Ġthe Ġform Ġof Ġkeywords Ġor Ġsnippets Ġcan Ġbe Ġpopulated Ġinto Ġthe Ġtemplates Ġwith Ġthe Ġhelp Ġof Ġmore Ġsophisticated Ġmodels . ĠAccordingly , Ġusing Ġtemplates Ġcan Ġbe Ġvery Ġtempting Ġfor Ġdomains Ġwith Ġlimited Ġresources Ġsuch Ġas Ġours . </s> Model ĠStructure . ĠThe Ġmodel Ġconsists Ġof Ġ4 Ġmodules : </s> 1 . ĠTemplate Ġextraction : ĠTo Ġconvert Ġhuman Ġsumm aries Ġinto Ġtemplates , Ġwe Ġremove Ġkeywords Ġin Ġthe Ġsummary Ġto Ġleave Ġonly Ġnon - key words . ĠWe Ġuse ĠRapid ĠAutomatic ĠKey word ĠExt raction Ġ( RA KE ) ĠB IB REF 18 Ġto Ġidentify Ġkeywords . </s> 2 . ĠTemplate Ġclust ering : ĠUpon Ġconverting Ġhuman Ġsumm aries Ġinto Ġtemplates , Ġwe Ġcluster Ġthem Ġinto Ġ$ N $ Ġclusters Ġwith Ġthe Ġgoal Ġof Ġusing Ġany Ġtemplate Ġfrom Ġthe Ġsame Ġcluster Ġinterchange ably . ĠA Ġtemplate Ġis Ġfirst Ġconverted Ġinto Ġembed d ings Ġusing Ġa Ġpret rained ĠB ERT Ġmodel ĠB IB REF 19 , Ġwhere Ġtemplate Ġembed ding Ġis Ġconstructed Ġby Ġaverage Ġpool ing Ġword Ġembed d ings . ĠTem plates Ġare Ġthen Ġclustered Ġusing Ġk - med oid . </s> 3 . ĠSummary Ġrewriting : ĠAn Ġenc oder - att ention - dec oder Ġwith Ġpointer Ġnetwork Ġis Ġtrained Ġto Ġperform Ġthe Ġrewriting Ġtask . ĠThe Ġmodel Ġis Ġtrained Ġto Ġinject Ġkeywords Ġinto Ġa Ġtemplate Ġand Ġperform Ġrewriting Ġinto Ġa Ġcoherent Ġparagraph . ĠThe Ġproduced Ġre writ es Ġare Ġconsidered Ġas Ġcandidate Ġsumm aries . </s> 4 . ĠSummary Ġselection : ĠAfter Ġproducing Ġcandidate Ġsumm aries , Ġwe Ġneed Ġto Ġpick Ġthe Ġbest Ġones . ĠWe Ġargue Ġthat Ġthe Ġbest Ġcandidates Ġare Ġthose Ġthat Ġare Ġcoherent Ġand Ġalso Ġconvey Ġthe Ġsame Ġmeaning Ġas Ġthe Ġoriginal Ġhuman Ġsummary . ĠWe Ġthus Ġuse Ġa Ġhybrid Ġmetric Ġto Ġscore Ġcandidates , Ġwhere Ġthe Ġmetric Ġis Ġa Ġweighted Ġsum Ġof Ġtwo Ġscores Ġand Ġis Ġcalculated Ġusing ĠEqu ations Ġ1 , Ġ2 , Ġand Ġ3 . ĠE q . 1 Ġmeasures Ġco he rency Ġusing Ġa Ġlanguage Ġmodel Ġ( LM ), ĠE q . 2 Ġmeasures Ġhow Ġclose Ġa Ġcandidate Ġis Ġto Ġa Ġhuman Ġsummary Ġusing ĠR OU GE Ġscores , Ġwhile ĠE q . 3 Ġpicks Ġthe Ġhighest Ġscored Ġ$ N $ Ġcandidates Ġas Ġthe Ġfinal Ġsynthetic Ġset . </s> CS Ġand ĠHS Ġare Ġa Ġcandidate Ġand Ġhuman Ġsummary . Ġ$ P ( w ) $ Ġis Ġthe Ġprobability Ġof Ġword Ġ$ w $ Ġusing Ġa Ġlanguage Ġmodel . Ġ$\ alpha Ġ, Ġ\ beta Ġ$ Ġare Ġweight ing Ġparameters . ĠIn Ġthis Ġwork Ġwe Ġuse Ġ$\ alpha Ġ= \ beta Ġ= 1 $ Ġfor Ġall Ġexperiments . Ġ$ R _{ i } ( CS , HS ) $ Ġis ĠR OU GE - i Ġscore Ġbetween ĠCS Ġand ĠHS Ġfor Ġi = 1 , Ġ2 , Ġand Ġ$ l $. </s> Model ĠTraining . ĠBefore Ġusing Ġthe Ġsynthesis Ġmodel , Ġsome Ġof Ġthe Ġconstructing Ġmodules Ġ( rew rit ing Ġmodule , Ġscoring ĠLM ) Ġneed Ġtraining . ĠTo Ġtrain Ġthe Ġrewriting Ġmodel , Ġwe Ġuse Ġanother Ġdataset Ġconsisting Ġof Ġa Ġset Ġof Ġsamples , Ġwhere Ġeach Ġsample Ġcan Ġbe Ġa Ġtext Ġsnippet Ġ( sent ence , Ġparagraph , Ġetc .). ĠFor Ġeach Ġsample , Ġkeywords Ġare Ġextracted Ġusing ĠRA KE , Ġthen Ġremoved . ĠThe Ġkeywords Ġplus Ġthe Ġsample Ġwith Ġno Ġkeywords Ġare Ġthen Ġpassed Ġto Ġthe Ġrewriting Ġmodel . ĠThe Ġtraining Ġobjective Ġof Ġthis Ġmodel Ġis Ġto Ġreconstruct Ġthe Ġoriginal Ġsample , Ġwhich Ġcan Ġbe Ġseen Ġas Ġtrying Ġto Ġinject Ġextracted Ġkeywords Ġback Ġinto Ġa Ġtemplate . ĠModel ĠUsage . ĠTo Ġuse Ġthe Ġsynthesis Ġmodel Ġto Ġgenerate Ġnew Ġsamples , Ġthe Ġset Ġof Ġhuman Ġsumm aries Ġare Ġfed Ġto Ġthe Ġmodel , Ġpassing Ġthrough Ġthe Ġsub - modules Ġin Ġthe Ġfollowing Ġorder : </s> 1 . ĠHuman Ġsumm aries Ġfirst Ġpass Ġthrough Ġthe Ġtemplate Ġextraction Ġmodule , Ġconverting Ġeach Ġsummary Ġ$ s _ i $ Ġinto Ġtemplate Ġ$ t _ i $ Ġand Ġthe Ġcorresponding Ġkeywords Ġ$ kw _ i $. </s> 2 . ĠTem plates Ġare Ġthen Ġpassed Ġto Ġthe Ġclust ering Ġmodule , Ġproducing Ġa Ġset Ġof Ġclusters . ĠEach Ġcluster Ġ$ C $ Ġcontains Ġa Ġnumber Ġof Ġsimilar Ġtemplates . </s> 3 . ĠFor Ġeach Ġtemplate Ġ$ t _ i $ Ġand Ġcorresponding Ġkeywords Ġ$ kw _ i $ Ġfrom Ġstep Ġ1 , Ġfind Ġthe Ġcluster Ġ$ C _ i $ Ġthat Ġcontains Ġthe Ġtemplate Ġ$ t _ i $, Ġthen Ġpass Ġthe Ġset Ġof Ġtemplates Ġwithin Ġthat Ġclusters Ġ$\ l brace Ġt _ j \ r brace Ġ\ for all Ġ{ j }, $ Ġif Ġ$ t _ j Ġ\ in ĠC _ i $ Ġalongside Ġthe Ġkeywords Ġ$ kw _ i $ Ġto Ġthe Ġsummary Ġrewriting Ġmodule . ĠThis Ġwill Ġproduce Ġa Ġset Ġof Ġcandidate Ġsumm aries . </s> 4 . ĠThe Ġsummary Ġselection Ġmodule Ġscores Ġand Ġselects Ġthe Ġhighest Ġ$ N $ Ġcandidates Ġas Ġthe Ġsynthetic Ġsumm aries . </s> Exper iments </s> Our Ġexperimental Ġdesigns Ġaddress Ġthe Ġfollowing Ġhypotheses : </s> Hyp ot hesis Ġ1 Ġ( H 1 ) Ġ: ĠTraining Ġcomplex Ġabstract ive Ġmodels Ġwith Ġlimited Ġin - domain Ġor Ġlarge Ġquantities Ġof Ġout - of - domain Ġdata Ġwon 't Ġbe Ġenough Ġto Ġoutper form Ġextract ive Ġbas elines . </s> Hyp ot hesis Ġ2 Ġ( H 2 ) Ġ: ĠDomain Ġtransfer Ġhelps Ġabstract ive Ġmodels Ġeven Ġif Ġin - domain Ġand Ġout - of - domain Ġdata Ġare Ġvery Ġdifferent Ġand Ġthe Ġamount Ġof Ġin - domain Ġdata Ġis Ġvery Ġsmall . </s> Hyp ot hesis Ġ3 Ġ( H 3 ) Ġ: ĠEn rich ing Ġabstract ive Ġtraining Ġdata Ġwith Ġsynthetic Ġdata Ġhelps Ġovercome Ġin - domain Ġdata Ġscarcity . </s> Hyp ot hesis Ġ4 Ġ( H 4 ) Ġ: ĠThe Ġproposed Ġtemplate - based Ġsynthesis Ġmodel Ġoutper forms Ġa Ġsimple Ġword Ġreplacement Ġmodel . </s> Hyp ot hesis Ġ5 Ġ( H 5 ) Ġ: ĠComb ining Ġdomain Ġtransfer Ġwith Ġdata Ġsynthesis Ġoutper forms Ġusing Ġeach Ġapproach Ġon Ġits Ġown . </s> Hyp ot hesis Ġ6 Ġ( H 6 ) Ġ: ĠThe Ġsynthesis Ġmodel Ġcan Ġbe Ġextended Ġto Ġperform Ġreflection Ġsummar ization Ġdirectly . </s> Ext ractive ĠBas elines Ġ( for Ġtesting ĠH 1 ). ĠWhile ĠB IB REF 0 Ġused ĠLead - 3 Ġas Ġan Ġextract ive Ġbaseline , Ġin Ġour Ġdata Ġsentence Ġorder Ġdoesn 't Ġmatter Ġas Ġreflections Ġare Ġindependent . ĠWe Ġthus Ġuse Ġa Ġsimilar Ġin Ġconcept Ġbaseline : Ġrandomly Ġselect ĠN Ġreflections . ĠSince Ġthe Ġbaseline Ġis Ġrandom Ġwe Ġreport Ġthe Ġaverage Ġresult Ġof Ġ100 Ġruns . ĠFollowing ĠB IB REF 5 , Ġwe Ġcompare Ġresults Ġto ĠME AD ĠB IB REF 20 Ġand Ġto ĠB IB REF 5 's Ġextract ive Ġphrase - based Ġmodel . ĠSince Ġthese Ġmodels Ġextracted Ġ5 Ġphrases Ġas Ġextract ive Ġsummary , Ġwe Ġuse ĠN = 5 Ġfor Ġour Ġthree Ġextract ive Ġbas elines . ĠAdditionally Ġwe Ġcompare Ġto Ġrunning Ġonly Ġthe Ġextract ive Ġpart Ġof ĠFast - RL . </s> Domain ĠTransfer Ġ( for Ġtesting ĠH 2 , ĠH 5 ). </s> To Ġobserve Ġthe Ġimpact Ġof Ġusing Ġout - of - domain Ġ( news ) Ġdata Ġfor Ġpret raining Ġto Ġcompensate Ġfor Ġlow Ġresource Ġin - domain Ġ( ref lection ) Ġdata , Ġwe Ġtrain Ġ3 Ġvariants Ġof ĠPG - net : Ġmodel Ġtraining Ġon ĠCNN / DM ; Ġmodel Ġtraining Ġon Ġreflections ; Ġand Ġmodel Ġtraining Ġon ĠCNN / DM Ġthen Ġtuning Ġusing Ġreflections . ĠTable ĠT AB REF 11 Ġshows Ġexample Ġsumm aries Ġgenerated Ġby Ġthe Ġthree Ġvariants Ġof ĠPG - net Ġfor Ġa ĠCS Ġdocument . ĠFor Ġall Ġexperiments Ġwhere Ġreflections Ġare Ġused Ġfor Ġtraining / tun ing , Ġwe Ġtrain Ġusing Ġa Ġleave Ġone Ġcourse Ġout Ġapproach Ġ( i . e , Ġin Ġeach Ġfold , Ġthree Ġcourses Ġare Ġused Ġfor Ġtraining Ġand Ġthe Ġremaining Ġcourse Ġfor Ġtesting ). ĠIf Ġthe Ġexperiment Ġinvolves Ġtuning Ġa Ġcombined Ġdictionary Ġof ĠCNN / DM Ġand Ġreflections Ġis Ġused Ġto Ġavoid Ġdomain Ġmismatch . ĠTo Ġtune Ġmodel Ġparameters , Ġthe Ġbest Ġnumber Ġof Ġsteps Ġfor Ġtraining , Ġthe Ġlearning Ġrate , Ġetc ., Ġa Ġrandomly Ġselected Ġ50 % Ġof Ġthe Ġtraining Ġdata Ġis Ġused Ġfor Ġvalidation . ĠWe Ġchoose Ġthe Ġparameters Ġthat Ġmaximize ĠR OU GE Ġscores Ġover Ġthis Ġvalidation Ġset . </s> To Ġimplement ĠPG - net Ġwe Ġuse ĠOpen N MT ĠB IB REF 21 Ġwith Ġthe Ġoriginal Ġset Ġof Ġparameters . ĠThe Ġout - of - domain Ġmodel Ġis Ġtrained Ġfor Ġ100 k Ġsteps Ġusing Ġthe ĠCNN / DM Ġdataset . ĠFollowing Ġbase Ġmodel Ġtraining , Ġwe Ġtune Ġthe Ġmodel Ġby Ġtraining Ġit Ġusing Ġstudent Ġreflections . ĠThe Ġtuning Ġis Ġdone Ġby Ġlowering Ġthe ĠLR Ġfrom Ġ0 . 15 Ġto Ġ0 . 1 Ġand Ġtraining Ġthe Ġmodel Ġfor Ġadditional Ġ500 Ġsteps . ĠThe Ġin - domain Ġmodel Ġis Ġtrained Ġonly Ġusing Ġreflections . ĠWe Ġuse Ġthe Ġsame Ġmodel Ġarchitecture Ġas Ġabove Ġand Ġtrain Ġthe Ġmodel Ġfor Ġ20 k Ġsteps Ġusing Ġad ag rad Ġand ĠLR Ġof Ġ0 . 15 . </s> Sy nt hesis ĠBas eline Ġ( for Ġtesting ĠH 3 , ĠH 4 ). ĠFollowing ĠB IB REF 12 , Ġwe Ġdeveloped Ġa Ġdata Ġsynthesis Ġbaseline Ġusing Ġword Ġreplacement Ġvia ĠWord Net . ĠThe Ġbaseline Ġiter ates Ġover Ġall Ġwords Ġin Ġa Ġsummary . ĠIf Ġword Ġ$ X $ Ġhas Ġ$ N $ Ġsyn onyms Ġin ĠWord Net , Ġthe Ġmodel Ġcreates Ġ$ N $ Ġnew Ġversions Ġof Ġthe Ġsummary Ġand Ġcorresponding Ġreflections Ġby Ġreplacing Ġthe Ġword Ġ$ X $ Ġwith Ġeach Ġof Ġthe Ġ$ N $ Ġsyn onyms . </s> Template ĠSynt hesis ĠModel Ġ( for Ġtesting ĠH 4 , ĠH 5 ). ĠTo Ġsynthes ize Ġsumm aries , Ġwe Ġuse Ġthe Ġsame Ġleave Ġone Ġcourse Ġout Ġapproach . ĠFor Ġeach Ġcourse , Ġwe Ġuse Ġthe Ġdata Ġfrom Ġthe Ġother Ġthree Ġcourses Ġto Ġtrain Ġthe Ġrewriting Ġmodule Ġand Ġtune Ġthe Ġscoring Ġlanguage Ġmodel . ĠWe Ġcan Ġalso Ġuse Ġthe Ġsumm aries Ġfrom ĠCNN / DM Ġdata Ġas Ġadditional Ġsamples Ġto Ġfurther Ġtrain Ġthe Ġrewriting Ġmodule . ĠWe Ġthen Ġstart Ġsynthes izing Ġdata Ġusing Ġthat Ġtraining Ġdata Ġas Ġinput . ĠFirst Ġtemplates Ġare Ġconstructed . ĠThe Ġtemplates Ġare Ġthen Ġclustered Ġinto Ġ8 Ġclusters . ĠWe Ġdecided Ġto Ġuse Ġ8 Ġto Ġavoid Ġclust ering Ġtemplates Ġfrom ĠPO I Ġwith ĠMP , Ġas Ġthe Ġtemplates Ġfrom Ġboth Ġprompts Ġwould Ġcontain Ġvery Ġdifferent Ġsupporting Ġwords . ĠWe Ġalso Ġwanted Ġto Ġavoid Ġa Ġhigh Ġlevel Ġof Ġdiss imilar ity Ġwithin Ġeach Ġcluster , Ġand Ġallow Ġsome Ġdiversity . ĠFollowing Ġthe Ġclust ering , Ġthe Ġrewriting Ġmodel Ġproduces Ġcandidate Ġsumm aries Ġfor Ġeach Ġhuman Ġsummary . ĠThe Ġrewriting Ġmodel Ġis Ġanother ĠPG - net Ġwith Ġthe Ġsame Ġexact Ġparameters . ĠAfter Ġproducing Ġthe Ġcandidate Ġsumm aries , Ġa Ġlanguage Ġmodel Ġis Ġused Ġto Ġscore Ġthem . ĠThe Ġlanguage Ġmodel Ġis Ġa Ġsingle Ġlayer ĠL ST M Ġlanguage Ġmodel Ġtrained Ġon Ġ36 K Ġsentences Ġfrom ĠWikipedia Ġand Ġfine Ġtuned Ġusing Ġstudent Ġreflections . ĠIn Ġthis Ġwork Ġwe Ġdecided Ġto Ġpick Ġonly Ġthe Ġhighest Ġ3 Ġscored Ġcandidate Ġsumm aries Ġas Ġsynthetic Ġdata , Ġto Ġavoid Ġadding Ġill - formed Ġsumm aries Ġto Ġthe Ġtraining Ġdata . ĠSince Ġwe Ġare Ġadding Ġ$ N $ Ġsynthetic Ġsumm aries Ġfor Ġeach Ġset Ġof Ġreflections , Ġthat Ġmeans Ġwe Ġare Ġessentially Ġdupl icating Ġthe Ġsize Ġof Ġour Ġoriginal Ġreflection Ġtraining Ġdata Ġby Ġ$ N $, Ġwhich Ġis Ġ3 Ġin Ġour Ġcase . ĠTable ĠT AB REF 11 Ġshows Ġa Ġhuman Ġsummary , Ġthe Ġkeywords Ġextracted , Ġthen Ġthe Ġoutput Ġof Ġinjecting Ġkeywords Ġin Ġa Ġdifferent Ġtemplate Ġusing Ġrewriting . </s> Template - based ĠSum mar ization Ġ( for Ġtesting ĠH 6 ). ĠWhile Ġthe Ġproposed Ġtemplate - based Ġmodel Ġwas Ġintended Ġfor Ġdata Ġsynthesis , Ġwith Ġminor Ġmodification Ġit Ġcan Ġbe Ġadapted Ġfor Ġsummar ization Ġitself . ĠBecause Ġthe Ġmodifications Ġintroduce Ġfew Ġparameters , Ġthe Ġmodel Ġis Ġsuitable Ġfor Ġsmall Ġdatasets . ĠRecall Ġthat Ġfor Ġdata Ġsynthesis , Ġthe Ġinput Ġto Ġthe Ġtemplate Ġmethod Ġis Ġa Ġsummary . ĠSince Ġfor Ġsummar ization Ġthe Ġinput Ġinstead Ġis Ġa Ġset Ġof Ġreflections , Ġwe Ġperform Ġkeyword Ġextraction Ġover Ġthe Ġset Ġof Ġreflections . ĠWe Ġthen Ġadd Ġan Ġextra Ġlog istic Ġregression Ġclass ifier Ġthat Ġuses Ġthe Ġset Ġof Ġreflections Ġas Ġinput Ġand Ġpredicts Ġa Ġcluster Ġof Ġtemplates Ġconstructed Ġfrom Ġother Ġcourses . ĠUsing Ġthe Ġkeywords Ġand Ġthe Ġpredicted Ġcluster Ġof Ġtemplates , Ġwe Ġuse Ġthe Ġsame Ġrewriting Ġmodel Ġto Ġproduce Ġcandidate Ġsumm aries . ĠThe Ġlast Ġstep Ġin Ġthe Ġpipeline Ġis Ġscoring . ĠIn Ġdata Ġsynthesis , Ġa Ġreference Ġsummary Ġis Ġused Ġfor Ġscoring ; Ġhowever , Ġin Ġsummar ization Ġwe Ġdon 't Ġhave Ġsuch Ġa Ġreference . ĠTo Ġscore Ġthe Ġcandidate Ġsumm aries , Ġthe Ġmodel Ġonly Ġuses Ġthe Ġlanguage Ġmodel Ġand Ġproduces Ġthe Ġcandidate Ġwith Ġthe Ġhighest Ġscore . </s> Results </s> R OU GE ĠEvaluation ĠResults . </s> Table ĠT AB REF 13 Ġpresents Ġsummar ization Ġperformance Ġresults Ġfor Ġthe Ġ4 Ġextract ive Ġbas elines , Ġfor Ġthe Ġoriginal Ġand Ġproposed Ġvariants Ġof ĠPG - net , Ġand Ġfinally Ġfor Ġtemplate - sum mar ization . ĠFollowing ĠB IB REF 0 , Ġperformance Ġis Ġevaluated Ġusing ĠR OU GE Ġ( 1 , Ġ2 , Ġand Ġ$ L $ ) ĠB IB REF 22 Ġon ĠF 1 . ĠThe Ġmotivation Ġfor Ġusing Ġdomain Ġtransfer Ġand Ġdata Ġsynthesis Ġis Ġour Ġhypothesis Ġ( H 1 ). ĠTable ĠT AB REF 13 Ġsupports Ġthis Ġhypothesis . ĠAll ĠR OU GE Ġscores Ġfor ĠPG - net Ġthat Ġoutper form Ġall Ġextract ive Ġbas elines Ġ( in Ġital ics ) Ġinvolve Ġtuning Ġand / or Ġuse Ġof Ġsynthes ised Ġdata , Ġexcept Ġfor Ġone ĠR - 1 Ġ( row Ġ18 ). </s> As Ġfor Ġour Ġsecond Ġhypothesis Ġ( H 2 ), Ġtable ĠT AB REF 13 Ġshows Ġthat Ġit Ġis Ġa Ġvalid Ġone . ĠFor ĠPG - net , Ġcomparing Ġthe ĠCNN / DM Ġout - of - domain Ġand ĠStudent ĠRef lection Ġin - domain Ġresults Ġin Ġrows Ġ( 5 Ġand Ġ6 ) Ġand Ġ( 17 Ġand Ġ18 ) Ġwith Ġtheir Ġcorresponding Ġtuned Ġresults Ġin Ġrows Ġ9 Ġand Ġ21 , Ġwe Ġsee Ġthat Ġfine Ġtuning Ġimproves ĠR - 1 , ĠR - 2 , Ġand ĠR -$ L $ Ġfor Ġall Ġcourses Ġ( rows Ġ5 , Ġ6 , Ġ9 Ġand Ġ17 , Ġ18 , Ġ21 ). ĠQual itatively , Ġthe Ġexamples Ġpresented Ġin ĠTable ĠT AB REF 11 Ġclearly Ġshow Ġthat Ġtuning Ġyields Ġa Ġmore Ġcoherent Ġand Ġrelevant Ġsummary . ĠOver Ġall Ġcourses , Ġthe Ġtuned Ġversion Ġof ĠPG - net Ġconsistently Ġoutper forms Ġthe Ġbest Ġbaseline Ġresult Ġfor Ġeach Ġmetric Ġ( rows Ġ9 Ġvs . Ġ1 , Ġ2 , Ġ3 , Ġ4 Ġand Ġ21 Ġvs . Ġ13 , Ġ14 , Ġ15 , Ġ16 ) Ġexcept Ġfor ĠR - 2 Ġin ĠStat 2016 . </s> To Ġvalidate Ġour Ġnext Ġset Ġof Ġhypothes ises Ġ( H 3 , ĠH 4 . ĠH 5 ), Ġwe Ġuse Ġthe Ġsynthes ized Ġdata Ġin Ġtwo Ġsettings : Ġeither Ġusing Ġit Ġfor Ġtraining Ġ( rows Ġ7 , Ġ8 Ġand Ġ19 , Ġ20 ) Ġor Ġtuning Ġ( rows Ġ10 , Ġ11 Ġand Ġ22 , Ġ23 ). ĠTable ĠT AB REF 13 Ġsupports ĠH 4 Ġby Ġshowing Ġthat Ġthe Ġproposed Ġsynthesis Ġmodel Ġoutper forms Ġthe ĠWord Net Ġbaseline Ġin Ġtraining Ġ( rows Ġ7 , Ġ8 Ġand Ġ19 , Ġ20 ) Ġexcept ĠStat 2016 , Ġand Ġtuning Ġ( 10 , Ġ11 Ġand Ġ22 , Ġ23 ) Ġover Ġall Ġcourses . ĠIt Ġalso Ġshows Ġthat Ġwhile Ġadding Ġsynthetic Ġdata Ġfrom Ġthe Ġbaseline Ġis Ġnot Ġalways Ġhelpful , Ġadding Ġsynthetic Ġdata Ġfrom Ġthe Ġtemplate Ġmodel Ġhelps Ġto Ġimprove Ġboth Ġthe Ġtraining Ġand Ġthe Ġtuning Ġprocess . ĠIn Ġboth ĠCS Ġand ĠENG R Ġcourses , Ġtuning Ġwith Ġsynthetic Ġdata Ġenhances Ġall ĠR OU GE Ġscores Ġcompared Ġto Ġtuning Ġwith Ġonly Ġthe Ġoriginal Ġdata . Ġ( rows Ġ9 Ġand Ġ11 ). ĠAs Ġfor ĠStat 2015 , ĠR - 1 Ġand ĠR -$ L $ Ġimproved , Ġwhile ĠR - 2 Ġdecreased . ĠFor ĠStat 2016 , ĠR - 2 Ġand ĠR -$ L $ Ġimproved , Ġand ĠR - 1 Ġdecreased Ġ( rows Ġ21 Ġand Ġ23 ). ĠTraining Ġwith Ġboth Ġstudent Ġreflection Ġdata Ġand Ġsynthetic Ġdata Ġcompared Ġto Ġtraining Ġwith Ġonly Ġstudent Ġreflection Ġdata Ġyields Ġsimilar Ġimprovements , Ġsupporting ĠH 3 Ġ( rows Ġ6 , Ġ8 Ġand Ġ18 , Ġ20 ). ĠWhile Ġthe Ġincrease Ġin ĠR OU GE Ġscores Ġis Ġsmall , Ġour Ġresults Ġshow Ġthat Ġenrich ing Ġtraining Ġdata Ġwith Ġsynthetic Ġdata Ġcan Ġbenefit Ġboth Ġthe Ġtraining Ġand Ġtuning Ġof Ġother Ġmodels . ĠIn Ġgeneral , Ġthe Ġbest Ġresults Ġare Ġobtained Ġwhen Ġusing Ġdata Ġsynthesis Ġfor Ġboth Ġtraining Ġand Ġtuning Ġ( rows Ġ11 Ġand Ġ23 ), Ġsupporting ĠH 5 . </s> Finally , Ġwhile Ġthe Ġgoal Ġof Ġour Ġtemplate Ġmodel Ġwas Ġto Ġsynthes ize Ġdata , Ġusing Ġit Ġfor Ġsummar ization Ġis Ġsurprisingly Ġcompetitive , Ġsupporting ĠH 6 . ĠWe Ġbelieve Ġthat Ġtraining Ġthe Ġmodel Ġwith Ġlittle Ġdata Ġis Ġdo able Ġdue Ġto Ġthe Ġsmall Ġnumber Ġof Ġparameters Ġ( log istic Ġregression Ġclass ifier Ġonly ). ĠWhile Ġrows Ġ12 Ġand Ġ24 Ġare Ġnever Ġthe Ġbest Ġresults , Ġthey Ġare Ġclose Ġto Ġthe Ġbest Ġinvolving Ġtuning . ĠThis Ġencourages Ġus Ġto Ġenhance Ġour Ġtemplate Ġmodel Ġand Ġexplore Ġtemplates Ġnot Ġso Ġtailored Ġto Ġour Ġdata . </s> Human ĠEvaluation ĠResults . ĠWhile Ġautomated Ġevaluation Ġmetrics Ġlike ĠR OU GE Ġmeasure Ġlex ical Ġsimilarity Ġbetween Ġmachine Ġand Ġhuman Ġsumm aries , Ġhumans Ġcan Ġbetter Ġmeasure Ġhow Ġcoherent Ġand Ġreadable Ġa Ġsummary Ġis . ĠOur Ġevaluation Ġstudy Ġinvestigates Ġwhether Ġtuning Ġthe ĠPG - net Ġmodel Ġincreases Ġsummary Ġco herence , Ġby Ġasking Ġeval u ators Ġto Ġselect Ġwhich Ġof Ġthree Ġsumm aries Ġfor Ġthe Ġsame Ġdocument Ġthey Ġlike Ġmost : Ġthe ĠPG - net Ġmodel Ġtrained Ġon ĠCNN / DM ; Ġthe Ġmodel Ġtrained Ġon Ġstudent Ġreflections ; Ġand Ġfinally Ġthe Ġmodel Ġtrained Ġon ĠCNN / DM Ġand Ġtuned Ġon Ġstudent Ġreflections . Ġ20 Ġeval u ators Ġwere Ġrecruited Ġfrom Ġour Ġinstitution Ġand Ġasked Ġto Ġeach Ġperform Ġ20 Ġannotations . ĠSum m aries Ġare Ġpresented Ġto Ġeval u ators Ġin Ġrandom Ġorder . ĠEval u ators Ġare Ġthen Ġasked Ġto Ġselect Ġthe Ġsummary Ġthey Ġfeel Ġto Ġbe Ġmost Ġreadable Ġand Ġcoherent . ĠUnlike ĠR OU GE , Ġwhich Ġmeasures Ġthe Ġcoverage Ġof Ġa Ġgenerated Ġsummary Ġrelative Ġto Ġa Ġreference Ġsummary , Ġour Ġeval u ators Ġdon 't Ġread Ġthe Ġreflections Ġor Ġreference Ġsummary . ĠThey Ġchoose Ġthe Ġsummary Ġthat Ġis Ġmost Ġcoherent Ġand Ġreadable , Ġregardless Ġof Ġthe Ġsource Ġof Ġthe Ġsummary . ĠFor Ġboth Ġcourses , Ġthe Ġmajority Ġof Ġselected Ġsumm aries Ġwere Ġproduced Ġby Ġthe Ġtuned Ġmodel Ġ( 49 % Ġfor ĠCS Ġand Ġ41 % Ġfor ĠStat 2015 ), Ġcompared Ġto Ġ( 31 % Ġfor ĠCS Ġand Ġ30 . 9 % Ġfor ĠStat 2015 ) Ġfor ĠCNN / DM Ġmodel , Ġand Ġ( 19 . 7 % Ġfor ĠCS Ġand Ġ28 . 5 % Ġfor ĠStat 2015 ) Ġfor Ġstudent Ġreflections Ġmodel . ĠThese Ġresults Ġagain Ġsuggest Ġthat Ġdomain Ġtransfer Ġcan Ġremedy Ġthe Ġsize Ġof Ġin - domain Ġdata Ġand Ġimprove Ġperformance . </s> Con clusions Ġand ĠFuture ĠWork </s> We Ġexplored Ġimproving Ġthe Ġperformance Ġof Ġneural Ġabstract ive Ġsummar izers Ġwhen Ġapplied Ġto Ġthe Ġlow Ġresource Ġdomain Ġof Ġstudent Ġreflections Ġusing Ġthree Ġapproaches : Ġdomain Ġtransfer , Ġdata Ġsynthesis Ġand Ġthe Ġcombination Ġof Ġboth . ĠFor Ġdomain Ġtransfer , Ġstate Ġof Ġthe Ġart Ġabstract ive Ġsummar ization Ġmodel Ġwas Ġpret rained Ġusing Ġout - of - domain Ġdata Ġ( CNN / DM ), Ġthen Ġtuned Ġusing Ġin - domain Ġdata Ġ( student Ġreflections ). ĠThe Ġprocess Ġof Ġtuning Ġimproved ĠR OU GE Ġscores Ġon Ġthe Ġstudent Ġreflection Ġdata , Ġand Ġat Ġthe Ġsame Ġtime Ġproduced Ġmore Ġreadable Ġsumm aries . ĠTo Ġincorporate Ġsynthetic Ġdata , Ġwe Ġproposed Ġa Ġnew Ġtemplate Ġbased Ġsynthesis Ġmodel Ġto Ġsynthes ize Ġnew Ġsumm aries . ĠWe Ġshowed Ġthat Ġenrich ing Ġthe Ġtraining Ġdata Ġwith Ġthis Ġsynthes ized Ġdata Ġcan Ġfurther Ġincrease Ġthe Ġbenefits Ġof Ġusing Ġdomain Ġtransfer Ġ/ Ġtuning Ġto Ġincrease ĠR OU GE Ġscores . ĠWe Ġadditionally Ġshowed Ġthat Ġthe Ġproposed Ġsynthesis Ġmodel Ġoutper formed Ġa Ġword Ġreplacement Ġsynthesis Ġbaseline . ĠFuture Ġplans Ġinclude Ġtrying Ġdomain Ġadaptation , Ġenhancing Ġthe Ġsynthes ising Ġprocess Ġby Ġusing Ġother Ġmodels , Ġfurther Ġexploring Ġtemplate - based Ġmethods , Ġand Ġextending Ġthe Ġanalysis Ġof Ġthe Ġsynthesis Ġmodel Ġto Ġcover Ġother Ġtypes Ġof Ġdata Ġlike Ġreviews Ġand Ġopinions . </s> Acknowled gments </s> The Ġresearch Ġreported Ġhere Ġwas Ġsupported , Ġin Ġwhole Ġor Ġin Ġpart , Ġby Ġthe Ġinstitute Ġof ĠEducation ĠSciences , ĠU . S . ĠDepartment Ġof ĠEducation , Ġthrough ĠGrant ĠR 305 A 180 477 Ġto Ġthe ĠUniversity Ġof ĠPittsburgh . ĠThe Ġopin ons Ġexpressed Ġare Ġthose Ġof Ġthe Ġauthors Ġand Ġdo Ġnot Ġrepresent Ġthe Ġviews Ġof Ġthe Ġinstitute Ġor Ġthe ĠU . S . ĠDepartment Ġof ĠEducation