Low data: SST-5, TREC, IMDB around 1-2 accuracy points better than baseline
Imbalanced labels: the improvement over the base model increases as the data gets more imbalanced, ranging from around 6 accuracy points on 100:1000 to over 20 accuracy points on 20:1000
ABSTRACTIVE
FLOAT SELECTED: Table 1: Accuracy of Data Manipulation on Text Classification. All results are averaged over 15 runs ± one standard deviation. The numbers in parentheses next to the dataset names indicate the size of the datasets. For example, (40+2) denotes 40 training instances and 2 validation instances per class. Table TABREF26 shows the manipulation results on text classification. For data augmentation, our approach significantly improves over the base model on all the three datasets. Besides, compared to both the conventional synonym substitution and the approach that keeps the augmentation network fixed, our adaptive method that fine-tunes the augmentation network jointly with model training achieves superior results. Indeed, the heuristic-based synonym approach can sometimes harm the model performance (e.g., SST-5 and IMDB), as also observed in previous work BIBREF19, BIBREF18. This can be because the heuristic rules do not fit the task or datasets well. In contrast, learning-based augmentation has the advantage of adaptively generating useful samples to improve model training. Table TABREF29 shows the classification results on SST-2 with varying imbalance ratios. We can see our data weighting performs best across all settings. In particular, the improvement over the base model increases as the data gets more imbalanced, ranging from around 6 accuracy points on 100:1000 to over 20 accuracy points on 20:1000. Our method is again consistently better than BIBREF4, validating that the parametric treatment is beneficial. The proportion-based data weighting provides only limited improvement, showing the advantage of adaptive data weighting. The base model trained on the joint training-validation data for fixed steps fails to perform well, partly due to the lack of a proper mechanism for selecting steps.
1910.12795
Introduction </s> The Ġperformance Ġof Ġmachines Ġoften Ġcru cially Ġdepend Ġon Ġthe Ġamount Ġand Ġquality Ġof Ġthe Ġdata Ġused Ġfor Ġtraining . ĠIt Ġhas Ġbecome Ġincreasingly Ġubiquitous Ġto Ġmanipulate Ġdata Ġto Ġimprove Ġlearning , Ġespecially Ġin Ġlow Ġdata Ġregime Ġor Ġin Ġpresence Ġof Ġlow - quality Ġdatasets Ġ( e . g ., Ġim balanced Ġlabels ). ĠFor Ġexample , Ġdata Ġaug mentation Ġapplies Ġlabel - pres erving Ġtransformations Ġon Ġoriginal Ġdata Ġpoints Ġto Ġexpand Ġthe Ġdata Ġsize ; Ġdata Ġweight ing Ġassigns Ġan Ġimportance Ġweight Ġto Ġeach Ġinstance Ġto Ġadapt Ġits Ġeffect Ġon Ġlearning ; Ġand Ġdata Ġsynthesis Ġgenerates Ġentire Ġartificial Ġexamples . ĠDifferent Ġtypes Ġof Ġmanipulation Ġcan Ġbe Ġsuitable Ġfor Ġdifferent Ġapplication Ġsettings . </s> Common Ġdata Ġmanipulation Ġmethods Ġare Ġusually Ġdesigned Ġmanually , Ġe . g ., Ġaugment ing Ġby Ġflipping Ġan Ġimage Ġor Ġreplacing Ġa Ġword Ġwith Ġsyn onyms , Ġand Ġweight ing Ġwith Ġinverse Ġclass Ġfrequency Ġor Ġloss Ġvalues ĠB IB REF 0 , ĠB IB REF 1 . ĠRecent Ġwork Ġhas Ġstudied Ġautomated Ġapproaches , Ġsuch Ġas Ġlearning Ġthe Ġcomposition Ġof Ġaug mentation Ġoperators Ġwith Ġreinforcement Ġlearning ĠB IB REF 2 , ĠB IB REF 3 , Ġder iving Ġsample Ġweights Ġadapt ively Ġfrom Ġa Ġvalidation Ġset Ġvia Ġmeta Ġlearning ĠB IB REF 4 , Ġor Ġlearning Ġa Ġweight ing Ġnetwork Ġby Ġinducing Ġa Ġcurriculum ĠB IB REF 5 . ĠThese Ġlearning - based Ġapproaches Ġhave Ġallev iated Ġthe Ġengineering Ġburden Ġand Ġproduced Ġimpressive Ġresults . ĠHowever , Ġthe Ġalgorithms Ġare Ġusually Ġdesigned Ġspecifically Ġfor Ġcertain Ġtypes Ġof Ġmanipulation Ġ( e . g ., Ġeither Ġaug mentation Ġor Ġweight ing ) Ġand Ġthus Ġhave Ġlimited Ġapplication Ġscope Ġin Ġpractice . </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġnew Ġapproach Ġthat Ġenables Ġlearning Ġfor Ġdifferent Ġmanipulation Ġschemes Ġwith Ġthe Ġsame Ġsingle Ġalgorithm . ĠOur Ġapproach Ġdraws Ġinspiration Ġfrom Ġthe Ġrecent Ġwork ĠB IB REF 6 Ġthat Ġshows Ġequival ence Ġbetween Ġthe Ġdata Ġin Ġsupervised Ġlearning Ġand Ġthe Ġreward Ġfunction Ġin Ġreinforcement Ġlearning . ĠWe Ġthus Ġadapt Ġan Ġoff - the - she lf Ġreward Ġlearning Ġalgorithm ĠB IB REF 7 Ġto Ġthe Ġsupervised Ġsetting Ġfor Ġautomated Ġdata Ġmanipulation . ĠThe Ġmarriage Ġof Ġthe Ġtwo Ġparad ig ms Ġresults Ġin Ġa Ġsimple Ġyet Ġgeneral Ġalgorithm , Ġwhere Ġvarious Ġmanipulation Ġschemes Ġare Ġreduced Ġto Ġdifferent Ġparameter ization Ġof Ġthe Ġdata Ġreward . ĠFree Ġparameters Ġof Ġmanipulation Ġare Ġlearned Ġjointly Ġwith Ġthe Ġtarget Ġmodel Ġthrough Ġefficient Ġgradient Ġdescent Ġon Ġvalidation Ġexamples . ĠWe Ġdemonstrate Ġinstant iations Ġof Ġthe Ġapproach Ġfor Ġautomatically Ġfine - tun ing Ġan Ġaug mentation Ġnetwork Ġand Ġlearning Ġdata Ġweights , Ġrespectively . </s> We Ġconduct Ġextensive Ġexperiments Ġon Ġtext Ġand Ġimage Ġclassification Ġin Ġchallenging Ġsituations Ġof Ġvery Ġlimited Ġdata Ġand Ġim balanced Ġlabels . ĠBoth Ġaug mentation Ġand Ġweight ing Ġby Ġour Ġapproach Ġsignificantly Ġimprove Ġover Ġstrong Ġbase Ġmodels , Ġeven Ġthough Ġthe Ġmodels Ġare Ġinitialized Ġwith Ġlarge - scale Ġpret rained Ġnetworks Ġsuch Ġas ĠB ERT ĠB IB REF 8 Ġfor Ġtext Ġand ĠRes Net ĠB IB REF 9 Ġfor Ġimages . ĠOur Ġapproach , Ġbesides Ġits Ġgener ality , Ġalso Ġoutper forms Ġa Ġvariety Ġof Ġdedicated Ġrule - Ġand Ġlearning - based Ġmethods Ġfor Ġeither Ġaug mentation Ġor Ġweight ing , Ġrespectively . ĠLastly , Ġwe Ġobserve Ġthat Ġthe Ġtwo Ġtypes Ġof Ġmanipulation Ġtend Ġto Ġexcel Ġin Ġdifferent Ġcontexts : Ġaug mentation Ġshows Ġsuperiority Ġover Ġweight ing Ġwith Ġa Ġsmall Ġamount Ġof Ġdata Ġavailable , Ġwhile Ġweight ing Ġis Ġbetter Ġat Ġaddressing Ġclass Ġimbalance Ġproblems . </s> The Ġway Ġwe Ġderive Ġthe Ġmanipulation Ġalgorithm Ġrepresents Ġa Ġgeneral Ġmeans Ġof Ġproblem Ġsolving Ġthrough Ġalgorithm Ġextrap olation Ġbetween Ġlearning Ġparad ig ms , Ġwhich Ġwe Ġdiscuss Ġmore Ġin Ġsection ĠSEC REF 6 . </s> Related ĠWork </s> Rich Ġtypes Ġof Ġdata Ġmanipulation Ġhave Ġbeen Ġincreasingly Ġused Ġin Ġmodern Ġmachine Ġlearning Ġpipelines . ĠPrevious Ġwork Ġeach Ġhas Ġtypically Ġfocused Ġon Ġa Ġparticular Ġmanipulation Ġtype . ĠData Ġaug mentation Ġthat Ġpert ur bs Ġexamples Ġwithout Ġchanging Ġthe Ġlabels Ġis Ġwidely Ġused Ġespecially Ġin Ġvision ĠB IB REF 10 , ĠB IB REF 11 Ġand Ġspeech ĠB IB REF 12 , ĠB IB REF 13 Ġdomains . ĠCommon Ġhe uristic - based Ġmethods Ġon Ġimages Ġinclude Ġcro pping , Ġmirror ing , Ġrotation ĠB IB REF 11 , Ġand Ġso Ġforth . ĠRecent Ġwork Ġhas Ġdeveloped Ġautomated Ġaug mentation Ġapproaches ĠB IB REF 3 , ĠB IB REF 2 , ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 . ĠB IB REF 17 Ġadditionally Ġuse Ġlarge - scale Ġunl abel ed Ġdata . ĠB IB REF 3 , ĠB IB REF 2 Ġlearn Ġto Ġinduce Ġthe Ġcomposition Ġof Ġdata Ġtransformation Ġoperators . ĠInstead Ġof Ġtreating Ġdata Ġaug mentation Ġas Ġa Ġpolicy Ġin Ġreinforcement Ġlearning ĠB IB REF 3 , Ġwe Ġformulate Ġmanipulation Ġas Ġa Ġreward Ġfunction Ġand Ġuse Ġefficient Ġsto ch astic Ġgradient Ġdescent Ġto Ġlearn Ġthe Ġmanipulation Ġparameters . ĠText Ġdata Ġaug mentation Ġhas Ġalso Ġachieved Ġimpressive Ġsuccess , Ġsuch Ġas Ġcontextual Ġaug mentation ĠB IB REF 18 , ĠB IB REF 19 , Ġback - translation ĠB IB REF 20 , Ġand Ġmanual Ġapproaches ĠB IB REF 21 , ĠB IB REF 22 . ĠIn Ġaddition Ġto Ġpert ur bing Ġthe Ġinput Ġtext Ġas Ġin Ġclassification Ġtasks , Ġtext Ġgeneration Ġproblems Ġexpose Ġopportunities Ġto Ġadding Ġnoise Ġalso Ġin Ġthe Ġoutput Ġtext , Ġsuch Ġas ĠB IB REF 23 , ĠB IB REF 24 . ĠRecent Ġwork ĠB IB REF 6 Ġshows Ġoutput Ġn osing Ġin Ġsequence Ġgeneration Ġcan Ġbe Ġtreated Ġas Ġan Ġintermediate Ġapproach Ġin Ġbetween Ġsupervised Ġlearning Ġand Ġreinforcement Ġlearning , Ġand Ġdeveloped Ġa Ġnew Ġsequence Ġlearning Ġalgorithm Ġthat Ġinterpol ates Ġbetween Ġthe Ġspectrum Ġof Ġexisting Ġalgorithms . ĠWe Ġinstant iate Ġour Ġapproach Ġfor Ġtext Ġcontextual Ġaug mentation Ġas Ġin ĠB IB REF 18 , ĠB IB REF 19 , Ġbut Ġenhance Ġthe Ġprevious Ġwork Ġby Ġadditionally Ġfine - tun ing Ġthe Ġaug mentation Ġnetwork Ġjointly Ġwith Ġthe Ġtarget Ġmodel . </s> Data Ġweight ing Ġhas Ġbeen Ġused Ġin Ġvarious Ġalgorithms , Ġsuch Ġas ĠAda Boost ĠB IB REF 0 , Ġself - paced Ġlearning ĠB IB REF 25 , Ġhard - example Ġmining ĠB IB REF 26 , Ġand Ġothers ĠB IB REF 27 , ĠB IB REF 28 . ĠThese Ġalgorithms Ġlargely Ġdefine Ġsample Ġweights Ġbased Ġon Ġtraining Ġloss . ĠRecent Ġwork ĠB IB REF 5 , ĠB IB REF 29 Ġlearns Ġa Ġseparate Ġnetwork Ġto Ġpredict Ġsample Ġweights . ĠOf Ġparticular Ġrelevance Ġto Ġour Ġwork Ġis ĠB IB REF 4 Ġwhich Ġinduces Ġsample Ġweights Ġusing Ġa Ġvalidation Ġset . ĠThe Ġdata Ġweight ing Ġmechanism Ġinstant iated Ġby Ġour Ġframework Ġhas Ġa Ġkey Ġdifference Ġin Ġthat Ġsamples Ġweights Ġare Ġtreated Ġas Ġparameters Ġthat Ġare Ġupdated Ġiter atively , Ġinstead Ġof Ġre - est imated Ġfrom Ġscratch Ġat Ġeach Ġstep . ĠWe Ġshow Ġimproved Ġperformance Ġof Ġour Ġapproach . ĠBesides , Ġour Ġdata Ġmanipulation Ġapproach Ġis Ġderived Ġbased Ġon Ġa Ġdifferent Ġperspective Ġof Ġreward Ġlearning , Ġinstead Ġof Ġmeta - learning Ġas Ġin ĠB IB REF 4 . </s> Another Ġpopular Ġtype Ġof Ġdata Ġmanipulation Ġinvolves Ġdata Ġsynthesis , Ġwhich Ġcreates Ġentire Ġartificial Ġsamples Ġfrom Ġscratch . ĠG AN - based Ġapproaches Ġhave Ġachieved Ġimpressive Ġresults Ġfor Ġsynthes izing Ġconditional Ġimage Ġdata ĠB IB REF 30 , ĠB IB REF 31 . ĠIn Ġthe Ġtext Ġdomain , Ġcontro ll able Ġtext Ġgeneration ĠB IB REF 32 Ġpresents Ġa Ġway Ġof Ġco - training Ġthe Ġdata Ġgenerator Ġand Ġclass ifier Ġin Ġa Ġcycl ic Ġmanner Ġwithin Ġa Ġjoint ĠVA E ĠB IB REF 33 Ġand Ġwake - sleep ĠB IB REF 34 Ġframework . ĠIt Ġis Ġinteresting Ġto Ġexplore Ġthe Ġinstant iation Ġof Ġthe Ġpresent Ġapproach Ġfor Ġadaptive Ġdata Ġsynthesis Ġin Ġthe Ġfuture . </s> Background </s> We Ġfirst Ġpresent Ġthe Ġrelevant Ġwork Ġupon Ġwhich Ġour Ġautomated Ġdata Ġmanipulation Ġis Ġbuilt . ĠThis Ġsection Ġalso Ġestablishes Ġthe Ġnot ations Ġused Ġthroughout Ġthe Ġpaper . </s> Let Ġ$\ math bf Ġ{ x } $ Ġdenote Ġthe Ġinput Ġand Ġ$ y $ Ġthe Ġoutput . ĠFor Ġexample , Ġin Ġtext Ġclassification , Ġ$\ math bf Ġ{ x } $ Ġcan Ġbe Ġa Ġsentence Ġand Ġ$ y $ Ġis Ġthe Ġsentence Ġlabel . ĠDen ote Ġthe Ġmodel Ġof Ġinterest Ġas Ġ$ p _ \ the ta Ġ( y | \ math bf Ġ{ x }) $, Ġwhere Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġis Ġthe Ġmodel Ġparameters Ġto Ġbe Ġlearned . ĠIn Ġsupervised Ġsetting , Ġgiven Ġa Ġset Ġof Ġtraining Ġexamples Ġ$\ math cal Ġ{ D } = \ l brace Ġ( \ math bf Ġ{ x } ^ *, Ġy ^ * )\ r brace Ġ$ , Ġwe Ġlearn Ġthe Ġmodel Ġby Ġmaximizing Ġthe Ġdata Ġlog - like lihood . </s> Background Ġ: :: ĠEqu ival ence Ġbetween ĠData Ġand ĠReward </s> The Ġrecent Ġwork ĠB IB REF 6 Ġintroduced Ġa Ġun ifying Ġperspective Ġof Ġreform ulating Ġmaximum Ġlikelihood Ġsupervised Ġlearning Ġas Ġa Ġspecial Ġinstance Ġof Ġa Ġpolicy Ġoptimization Ġframework . ĠIn Ġthis Ġperspective , Ġdata Ġexamples Ġproviding Ġsupervision Ġsignals Ġare Ġequivalent Ġto Ġa Ġspecialized Ġreward Ġfunction . ĠSince Ġthe Ġoriginal Ġframework ĠB IB REF 6 Ġwas Ġderived Ġfor Ġsequence Ġgeneration Ġproblems , Ġhere Ġwe Ġpresent Ġa Ġslightly Ġadapted Ġformulation Ġfor Ġour Ġcontext Ġof Ġdata Ġmanipulation . </s> To Ġconnect Ġthe Ġmaximum Ġlikelihood Ġsupervised Ġlearning Ġwith Ġpolicy Ġoptimization , Ġconsider Ġthe Ġmodel Ġ$ p _ \ the ta Ġ( y | \ math bf Ġ{ x }) $ Ġas Ġa Ġpolicy Ġthat Ġtakes ĠâĢ ľ action âĢ Ŀ Ġ$ y $ Ġgiven Ġthe ĠâĢ ľ state âĢ Ŀ Ġ$\ math bf Ġ{ x } $. ĠLet Ġ$ R (\ math bf Ġ{ x }, Ġy Ġ| \ math cal Ġ{ D } )\ in Ġ\ math bb Ġ{ R } $ Ġdenote Ġa Ġreward Ġfunction , Ġand Ġ$ p (\ math bf Ġ{ x }) $ Ġbe Ġthe Ġempirical Ġdata Ġdistribution Ġwhich Ġis Ġknown Ġgiven Ġ$\ math cal Ġ{ D } $. ĠFurther Ġassume Ġa Ġvari ational Ġdistribution Ġ$ q (\ math bf Ġ{ x }, Ġy ) $ Ġthat Ġfactor izes Ġas Ġ$ q (\ math bf Ġ{ x }, y )= p (\ math bf Ġ{ x }) q ( y | \ math bf Ġ{ x }) $. ĠA Ġvari ational Ġpolicy Ġoptimization Ġobjective Ġis Ġthen Ġwritten Ġas : </s> where Ġ$\ text { K L } (\ cd ot Ġ\ Vert Ġ\ cd ot Ġ) $ Ġis Ġthe ĠK ull back âĢĵ Le ib ler Ġdivergence ; Ġ$\ text { H } (\ cd ot Ġ) $ Ġis Ġthe ĠShannon Ġentropy ; Ġand Ġ$\ alpha Ġ, \ beta Ġ> 0 $ Ġare Ġbalancing Ġweights . ĠThe Ġobjective Ġis Ġin Ġthe Ġsame Ġform Ġwith Ġthe ĠRL - as - in ference Ġformal ism Ġof Ġpolicy Ġoptimization ĠB IB REF 35 , ĠB IB REF 36 , ĠB IB REF 37 . ĠInt uitive ly , Ġthe Ġobjective Ġmaxim izes Ġthe Ġexpected Ġreward Ġunder Ġ$ q $, Ġand Ġen forces Ġthe Ġmodel Ġ$ p _ \ the ta Ġ$ Ġto Ġstay Ġclose Ġto Ġ$ q $, Ġwith Ġa Ġmaximum Ġentropy Ġregular ization Ġover Ġ$ q $. ĠThe Ġproblem Ġis Ġsolved Ġwith Ġan ĠEM Ġprocedure Ġthat Ġoptim izes Ġ$ q $ Ġand Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġalternating ly : </s> where Ġ$ Z $ Ġis Ġthe Ġnormal ization Ġterm . ĠWith Ġthe Ġestablished Ġframework , Ġit Ġis Ġeasy Ġto Ġshow Ġthat Ġthe Ġabove Ġoptimization Ġprocedure Ġreduces Ġto Ġmaximum Ġlikelihood Ġlearning Ġby Ġtaking Ġ$\ alpha Ġ\ right arrow Ġ0 , Ġ\ beta Ġ= 1 $, Ġand Ġthe Ġreward Ġfunction : </s> That Ġis , Ġa Ġsample Ġ$( \ math bf Ġ{ x }, Ġy ) $ Ġreceives Ġa Ġunit Ġreward Ġonly Ġwhen Ġit Ġmatches Ġa Ġtraining Ġexample Ġin Ġthe Ġdataset , Ġwhile Ġthe Ġreward Ġis Ġnegative Ġinfinite Ġin Ġall Ġother Ġcases . ĠTo Ġmake Ġthe Ġequival ence Ġto Ġmaximum Ġlikelihood Ġlearning Ġclearer , Ġnote Ġthat Ġthe Ġabove ĠM - step Ġnow Ġreduces Ġto </s> where Ġthe Ġjoint Ġdistribution Ġ$ p (\ math bf Ġ{ x } )\ exp Ġ\ l brace ĠR _ \ d elta Ġ\ r brace Ġ/ Z $ Ġequals Ġthe Ġempirical Ġdata Ġdistribution , Ġwhich Ġmeans Ġthe ĠM - step Ġis Ġin Ġfact Ġmaximizing Ġthe Ġdata Ġlog - like lihood Ġof Ġthe Ġmodel Ġ$ p _ \ the ta Ġ$ . </s> Background Ġ: :: ĠGrad ient - based ĠReward ĠLearning </s> There Ġis Ġa Ġrich Ġline Ġof Ġresearch Ġon Ġlearning Ġthe Ġreward Ġin Ġreinforcement Ġlearning . ĠOf Ġparticular Ġinterest Ġto Ġthis Ġwork Ġis ĠB IB REF 7 Ġwhich Ġlearns Ġa Ġparam etric Ġintrinsic Ġreward Ġthat Ġadditive ly Ġtransforms Ġthe Ġoriginal Ġtask Ġreward Ġ( a . k . a Ġextr ins ic Ġreward ) Ġto Ġimprove Ġthe Ġpolicy Ġoptimization . ĠFor Ġconsistency Ġof Ġnot ations Ġwith Ġabove , Ġformally , Ġlet Ġ$ p _ \ the ta Ġ( y | \ math bf Ġ{ x }) $ Ġbe Ġa Ġpolicy Ġwhere Ġ$ y $ Ġis Ġan Ġaction Ġand Ġ$\ math bf Ġ{ x } $ Ġis Ġa Ġstate . ĠLet Ġ$ R _ \ phi Ġ^ { in } $ Ġbe Ġthe Ġintrinsic Ġreward Ġwith Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $. ĠIn Ġeach Ġiteration , Ġthe Ġpolicy Ġparameter Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġis Ġupdated Ġto Ġmaximize Ġthe Ġjoint Ġrewards , Ġthrough : </s> where Ġ$\ math cal Ġ{ L } ^{ ex + in } $ Ġis Ġthe Ġexpectation Ġof Ġthe Ġsum Ġof Ġextr ins ic Ġand Ġintrinsic Ġrewards ; Ġand Ġ$\ gam ma Ġ$ Ġis Ġthe Ġstep Ġsize . ĠThe Ġequation Ġshows Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġdepends Ġon Ġ$\ math bf Ġ{\ phi Ġ} $, Ġthus Ġwe Ġcan Ġwrite Ġas Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} = \ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} (\ math bf Ġ{\ phi Ġ}) $. </s> The Ġnext Ġstep Ġis Ġto Ġoptimize Ġthe Ġintrinsic Ġreward Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $. ĠRecall Ġthat Ġthe Ġultimate Ġmeasure Ġof Ġthe Ġperformance Ġof Ġa Ġpolicy Ġis Ġthe Ġvalue Ġof Ġextr ins ic Ġreward Ġit Ġachieves . ĠTherefore , Ġa Ġgood Ġintrinsic Ġreward Ġis Ġsupposed Ġto , Ġwhen Ġthe Ġpolicy Ġis Ġtrained Ġwith Ġit , Ġincrease Ġthe Ġeventual Ġextr ins ic Ġreward . ĠThe Ġupdate Ġto Ġ$\ math bf Ġ{\ phi Ġ} $ Ġis Ġthen Ġwritten Ġas : </s> That Ġis , Ġwe Ġwant Ġthe Ġexpected Ġextr ins ic Ġreward Ġ$\ math cal Ġ{ L } ^{ ex } (\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ}) $ Ġof Ġthe Ġnew Ġpolicy Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġto Ġbe Ġmaxim ized . ĠSince Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġis Ġa Ġfunction Ġof Ġ$\ math bf Ġ{\ phi Ġ} $, Ġwe Ġcan Ġdirectly Ġback prop agate Ġthe Ġgradient Ġthrough Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġto Ġ$\ math bf Ġ{\ phi Ġ} $. </s> Learning ĠData ĠManip ulation Ġ: :: ĠMethod Ġ: :: ĠParam eter izing ĠData ĠManip ulation </s> We Ġnow Ġdevelop Ġour Ġapproach Ġof Ġlearning Ġdata Ġmanipulation , Ġthrough Ġa Ġnovel Ġmarriage Ġof Ġsupervised Ġlearning Ġand Ġthe Ġabove Ġreward Ġlearning . ĠSpecifically , Ġfrom Ġthe Ġpolicy Ġoptimization Ġperspective , Ġdue Ġto Ġthe Ġ$\ d elta Ġ$ - function Ġreward Ġ( E q . DIS PLAY _ FORM 4 ), Ġthe Ġstandard Ġmaximum Ġlikelihood Ġlearning Ġis Ġrestricted Ġto Ġuse Ġonly Ġthe Ġexact Ġtraining Ġexamples Ġ$\ math cal Ġ{ D } $ Ġin Ġa Ġuniform Ġway . ĠA Ġnatural Ġidea Ġof Ġenabling Ġdata Ġmanipulation Ġis Ġto Ġrelax Ġthe Ġstrong Ġrestrictions Ġof Ġthe Ġ$\ d elta Ġ$ - function Ġreward Ġand Ġinstead Ġuse Ġa Ġrelaxed Ġreward Ġ$ R _ \ phi Ġ( \ math bf Ġ{ x }, Ġy Ġ| Ġ\ math cal Ġ{ D }) $ Ġwith Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $. ĠThe Ġrelaxed Ġreward Ġcan Ġbe Ġparameter ized Ġin Ġvarious Ġways , Ġresulting Ġin Ġdifferent Ġtypes Ġof Ġmanipulation . ĠFor Ġexample , Ġwhen Ġa Ġsample Ġ$( \ math bf Ġ{ x }, Ġy ) $ Ġmatches Ġa Ġdata Ġinstance , Ġinstead Ġof Ġreturning Ġconstant Ġ1 Ġby Ġ$ R _ \ d elta Ġ$ , Ġthe Ġnew Ġ$ R _ \ phi Ġ$ Ġcan Ġreturn Ġvarying Ġreward Ġvalues Ġdepending Ġon Ġthe Ġmatched Ġinstance , Ġresulting Ġin Ġa Ġdata Ġweight ing Ġscheme . ĠAlternatively , Ġ$ R _ \ phi Ġ$ Ġcan Ġreturn Ġa Ġvalid Ġreward Ġeven Ġwhen Ġ$\ math bf Ġ{ x } $ Ġmatches Ġa Ġdata Ġexample Ġonly Ġin Ġpart , Ġor Ġ$( \ math bf Ġ{ x }, Ġy ) $ Ġis Ġan Ġentire Ġnew Ġsample Ġnot Ġin Ġ$\ math cal Ġ{ D } $, Ġwhich Ġin Ġeffect Ġmakes Ġdata Ġaug mentation Ġand Ġdata Ġsynthesis , Ġrespectively , Ġin Ġwhich Ġcases Ġ$\ math bf Ġ{\ phi Ġ} $ Ġis Ġeither Ġa Ġdata Ġtransformer Ġor Ġa Ġgenerator . ĠIn Ġthe Ġnext Ġsection , Ġwe Ġdemonstrate Ġtwo Ġparticular Ġparameter izations Ġfor Ġdata Ġaug mentation Ġand Ġweight ing , Ġrespectively . </s> We Ġthus Ġhave Ġshown Ġthat Ġthe Ġdiverse Ġtypes Ġof Ġmanipulation Ġall Ġboil Ġdown Ġto Ġa Ġparameter ized Ġdata Ġreward Ġ$ R _ \ phi Ġ$ . ĠSuch Ġan Ġconcise , Ġuniform Ġformulation Ġof Ġdata Ġmanipulation Ġhas Ġthe Ġadvantage Ġthat , Ġonce Ġwe Ġdevise Ġa Ġmethod Ġof Ġlearning Ġthe Ġmanipulation Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $, Ġthe Ġresulting Ġalgorithm Ġcan Ġdirectly Ġbe Ġapplied Ġto Ġautomate Ġany Ġmanipulation Ġtype . ĠWe Ġpresent Ġa Ġlearning Ġalgorithm Ġnext . </s> Learning ĠData ĠManip ulation Ġ: :: ĠMethod Ġ: :: ĠLearning ĠManip ulation ĠParameters </s> To Ġlearn Ġthe Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $ Ġin Ġthe Ġmanipulation Ġreward Ġ$ R _ \ phi Ġ( \ math bf Ġ{ x }, Ġy Ġ| Ġ\ math cal Ġ{ D }) $, Ġwe Ġcould Ġin Ġprinciple Ġadopt Ġany Ġoff - the - she lf Ġreward Ġlearning Ġalgorithm Ġin Ġthe Ġliterature . ĠIn Ġthis Ġwork , Ġwe Ġdraw Ġinspiration Ġfrom Ġthe Ġabove Ġgradient - based Ġreward Ġlearning Ġ( section ĠSEC REF 3 ) Ġdue Ġto Ġits Ġsimplicity Ġand Ġefficiency . ĠBrief ly , Ġthe Ġobjective Ġof Ġ$\ math bf Ġ{\ phi Ġ} $ Ġis Ġto Ġmaximize Ġthe Ġultimate Ġmeasure Ġof Ġthe Ġperformance Ġof Ġmodel Ġ$ p _ \ the ta Ġ( \ math bf Ġ{ y } | \ math bf Ġ{ x }) $, Ġwhich , Ġin Ġthe Ġcontext Ġof Ġsupervised Ġlearning , Ġis Ġthe Ġmodel Ġperformance Ġon Ġa Ġheld - out Ġvalidation Ġset . </s> The Ġalgorithm Ġoptim izes Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġand Ġ$\ math bf Ġ{\ phi Ġ} $ Ġalternating ly , Ġcorresponding Ġto ĠE q .( DIS PLAY _ FORM 7 ) Ġand ĠE q .( DIS PLAY _ FORM 8 ), Ġrespectively . ĠMore Ġconcrete ly , Ġin Ġeach Ġiteration , Ġwe Ġfirst Ġupdate Ġthe Ġmodel Ġparameters Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġin Ġanalogue Ġto ĠE q .( DIS PLAY _ FORM 7 ) Ġwhich Ġoptim izes Ġintrinsic Ġreward - en riched Ġobjective . ĠHere , Ġwe Ġoptimize Ġthe Ġlog - like lihood Ġof Ġthe Ġtraining Ġset Ġenriched Ġwith Ġdata Ġmanipulation . ĠThat Ġis , Ġwe Ġreplace Ġ$ R _ \ d elta Ġ$ Ġwith Ġ$ R _ \ phi Ġ$ Ġin ĠE q .( DIS PLAY _ FORM 5 ), Ġand Ġobtain Ġthe Ġaugmented ĠM - step : </s> By Ġnoticing Ġthat Ġthe Ġnew Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġdepends Ġon Ġ$\ math bf Ġ{\ phi Ġ} $, Ġwe Ġcan Ġwrite Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġas Ġa Ġfunction Ġof Ġ$\ math bf Ġ{\ phi Ġ} $, Ġnamely , Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} = \ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} (\ math bf Ġ{\ phi Ġ}) $. ĠThe Ġpractical Ġimplementation Ġof Ġthe Ġabove Ġupdate Ġdepends Ġon Ġthe Ġactual Ġparameter ization Ġof Ġmanipulation Ġ$ R _ \ phi Ġ$ , Ġwhich Ġwe Ġdiscuss Ġin Ġmore Ġdetails Ġin Ġthe Ġnext Ġsection . </s> The Ġnext Ġstep Ġis Ġto Ġoptimize Ġ$\ math bf Ġ{\ phi Ġ} $ Ġin Ġterms Ġof Ġthe Ġmodel Ġvalidation Ġperformance , Ġin Ġanalogue Ġto ĠE q .( DIS PLAY _ FORM 8 ). ĠForm ally , Ġlet Ġ$\ math cal Ġ{ D } ^ v $ Ġbe Ġthe Ġvalidation Ġset Ġof Ġdata Ġexamples . ĠThe Ġupdate Ġis Ġthen : </s> where , Ġsince Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġis Ġa Ġfunction Ġof Ġ$\ math bf Ġ{\ phi Ġ} $, Ġthe Ġgradient Ġis Ġback prop ag ated Ġto Ġ$\ math bf Ġ{\ phi Ġ} $ Ġthrough Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} (\ math bf Ġ{\ phi Ġ}) $. ĠTaking Ġdata Ġweight ing Ġfor Ġexample Ġwhere Ġ$\ math bf Ġ{\ phi Ġ} $ Ġis Ġthe Ġtraining Ġsample Ġweights Ġ( more Ġdetails Ġin Ġsection ĠSEC REF 15 ), Ġthe Ġupdate Ġis Ġto Ġoptimize Ġthe Ġweights Ġof Ġtraining Ġsamples Ġso Ġthat Ġthe Ġmodel Ġperforms Ġbest Ġon Ġthe Ġvalidation Ġset . </s> The Ġresulting Ġalgorithm Ġis Ġsummarized Ġin ĠAl gorithm ĠFIG REF 11 . ĠFigure ĠFIG REF 11 Ġillustrates Ġthe Ġcomputation Ġflow . ĠLearning Ġthe Ġmanipulation Ġparameters Ġeffectively Ġuses Ġa Ġheld - out Ġvalidation Ġset . ĠWe Ġshow Ġin Ġour Ġexperiments Ġthat Ġa Ġvery Ġsmall Ġset Ġof Ġvalidation Ġexamples Ġ( e . g ., Ġ2 Ġlabels Ġper Ġclass ) Ġis Ġenough Ġto Ġsignificantly Ġimprove Ġthe Ġmodel Ġperformance Ġin Ġlow Ġdata Ġregime . </s> It Ġis Ġworth Ġnoting Ġthat Ġsome Ġprevious Ġwork Ġhas Ġalso Ġlever aged Ġvalidation Ġexamples , Ġsuch Ġas Ġlearning Ġdata Ġaug mentation Ġwith Ġpolicy Ġgradient ĠB IB REF 3 Ġor Ġinducing Ġdata Ġweights Ġwith Ġmeta - learning ĠB IB REF 4 . ĠOur Ġapproach Ġis Ġinspired Ġfrom Ġa Ġdistinct Ġparadigm Ġof Ġ( int r ins ic ) Ġreward Ġlearning . ĠIn Ġcontrast Ġto ĠB IB REF 3 Ġthat Ġtreats Ġdata Ġaug mentation Ġas Ġa Ġpolicy , Ġwe Ġinstead Ġformulate Ġmanipulation Ġas Ġa Ġreward Ġfunction Ġand Ġenable Ġefficient Ġsto ch astic Ġgradient Ġupdates . ĠOur Ġapproach Ġis Ġalso Ġmore Ġbroadly Ġapplicable Ġto Ġdiverse Ġdata Ġmanipulation Ġtypes Ġthan ĠB IB REF 4 , ĠB IB REF 3 . </s> Learning ĠData ĠManip ulation Ġ: :: ĠInst anti ations : ĠAug mentation Ġ& ĠWeight ing </s> As Ġa Ġcase Ġstudy , Ġwe Ġshow Ġtwo Ġparameter izations Ġof Ġ$ R _ \ phi Ġ$ Ġwhich Ġinstant iate Ġdistinct Ġdata Ġmanipulation Ġschemes . ĠThe Ġfirst Ġexample Ġlearns Ġaug mentation Ġfor Ġtext Ġdata , Ġa Ġdomain Ġthat Ġhas Ġbeen Ġless Ġstudied Ġin Ġthe Ġliterature Ġcompared Ġto Ġvision Ġand Ġspeech ĠB IB REF 18 , ĠB IB REF 38 . ĠThe Ġsecond Ġinstance Ġfocuses Ġon Ġautomated Ġdata Ġweight ing , Ġwhich Ġis Ġapplicable Ġto Ġany Ġdata Ġdomains . </s> Learning ĠData ĠManip ulation Ġ: :: ĠInst anti ations : ĠAug mentation Ġ& ĠWeight ing Ġ: :: ĠFine - tun ing ĠText ĠAug mentation </s> The Ġrecent Ġwork ĠB IB REF 18 , ĠB IB REF 19 Ġdeveloped Ġa Ġnovel Ġcontextual Ġaug mentation Ġapproach Ġfor Ġtext Ġdata , Ġin Ġwhich Ġa Ġpowerful Ġpret rained Ġlanguage Ġmodel Ġ( LM ), Ġsuch Ġas ĠB ERT ĠB IB REF 8 , Ġis Ġused Ġto Ġgenerate Ġsubstit utions Ġof Ġwords Ġin Ġa Ġsentence . ĠSpecifically , Ġgiven Ġan Ġobserved Ġsentence Ġ$\ math bf Ġ{ x } ^ * $, Ġthe Ġmethod Ġfirst Ġrandomly Ġmasks Ġout Ġa Ġfew Ġwords . ĠThe Ġmasked Ġsentence Ġis Ġthen Ġfed Ġto ĠB ERT Ġwhich Ġfills Ġthe Ġmasked Ġpositions Ġwith Ġnew Ġwords . ĠTo Ġpreserve Ġthe Ġoriginal Ġsentence Ġclass , Ġthe ĠB ERT ĠLM Ġis Ġretro fitted Ġas Ġa Ġlabel - cond itional Ġmodel , Ġand Ġtrained Ġon Ġthe Ġtask Ġtraining Ġexamples . ĠThe Ġresulting Ġmodel Ġis Ġthen Ġfixed Ġand Ġused Ġto Ġaugment Ġdata Ġduring Ġthe Ġtraining Ġof Ġtarget Ġmodel . ĠWe Ġdenote Ġthe Ġaug mentation Ġdistribution Ġas Ġ$ g _{ \ phi Ġ_ 0 } (\ math bf Ġ{ x } | \ math bf Ġ{ x } ^ *, Ġ\ math bf Ġ{ y } ^ *) $, Ġwhere Ġ$\ math bf Ġ{\ phi Ġ} _ 0 $ Ġis Ġthe Ġfixed ĠB ERT ĠLM Ġparameters . </s> The Ġabove Ġprocess Ġhas Ġtwo Ġdrawbacks . ĠFirst , Ġthe ĠLM Ġis Ġfixed Ġafter Ġfitting Ġto Ġthe Ġtask Ġdata . ĠIn Ġthe Ġsubsequent Ġphase Ġof Ġtraining Ġthe Ġtarget Ġmodel , Ġthe ĠLM Ġaug ments Ġdata Ġwithout Ġknowing Ġthe Ġstate Ġof Ġthe Ġtarget Ġmodel , Ġwhich Ġcan Ġlead Ġto Ġsub - opt imal Ġresults . ĠSecond , Ġin Ġthe Ġcases Ġwhere Ġthe Ġtask Ġdataset Ġis Ġsmall , Ġthe ĠLM Ġcan Ġbe Ġinsufficient ly Ġtrained Ġfor Ġpreserving Ġthe Ġlabels Ġfaithfully , Ġresulting Ġin Ġnoisy Ġaugmented Ġsamples . </s> To Ġaddress Ġthe Ġdifficulties , Ġit Ġis Ġbeneficial Ġto Ġapply Ġthe Ġproposed Ġlearning Ġdata Ġmanipulation Ġalgorithm Ġto Ġadditionally Ġfine - t une Ġthe ĠLM Ġjointly Ġwith Ġtarget Ġmodel Ġtraining . ĠAs Ġdiscussed Ġin Ġsection ĠSEC REF 4 , Ġthis Ġreduces Ġto Ġproperly Ġparameter izing Ġthe Ġdata Ġreward Ġfunction : </s> That Ġis , Ġa Ġsample Ġ$( \ math bf Ġ{ x }, Ġy ) $ Ġreceives Ġa Ġunit Ġreward Ġwhen Ġ$ y $ Ġis Ġthe Ġtrue Ġlabel Ġand Ġ$\ math bf Ġ{ x } $ Ġis Ġthe Ġaugmented Ġsample Ġby Ġthe ĠLM Ġ( instead Ġof Ġthe Ġexact Ġoriginal Ġdata Ġ$\ math bf Ġ{ x } ^ * $ ). ĠPlug ging Ġthe Ġreward Ġinto ĠE q .( DIS PLAY _ FORM 13 ), Ġwe Ġobtain Ġthe Ġdata - au gment ed Ġupdate Ġfor Ġthe Ġmodel Ġparameters : </s> That Ġis , Ġwe Ġpick Ġan Ġexample Ġfrom Ġthe Ġtraining Ġset , Ġand Ġuse Ġthe ĠLM Ġto Ġcreate Ġaugmented Ġsamples , Ġwhich Ġare Ġthen Ġused Ġto Ġupdate Ġthe Ġtarget Ġmodel . ĠRegarding Ġthe Ġupdate Ġof Ġaug mentation Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $ Ġ( E q . DIS PLAY _ FORM 14 ), Ġsince Ġtext Ġsamples Ġare Ġdiscrete , Ġto Ġenable Ġefficient Ġgradient Ġpropagation Ġthrough Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġto Ġ$\ math bf Ġ{\ phi Ġ} $, Ġwe Ġuse Ġa Ġg umb el - soft max Ġapproximation ĠB IB REF 39 Ġto Ġ$\ math bf Ġ{ x } $ Ġwhen Ġsampling Ġsubstitution Ġwords Ġfrom Ġthe ĠLM . </s> Learning ĠData ĠManip ulation Ġ: :: ĠInst anti ations : ĠAug mentation Ġ& ĠWeight ing Ġ: :: ĠLearning ĠData ĠWe ights </s> We Ġnow Ġdemonstrate Ġthe Ġinstant iation Ġof Ġdata Ġweight ing . ĠWe Ġaim Ġto Ġassign Ġan Ġimportance Ġweight Ġto Ġeach Ġtraining Ġexample Ġto Ġadapt Ġits Ġeffect Ġon Ġmodel Ġtraining . ĠWe Ġautomate Ġthe Ġprocess Ġby Ġlearning Ġthe Ġdata Ġweights . ĠThis Ġis Ġachieved Ġby Ġparameter izing Ġ$ R _ \ phi Ġ$ Ġas : </s> where Ġ$\ phi Ġ_ i \ in Ġ\ math bb Ġ{ R } $ Ġis Ġthe Ġweight Ġassociated Ġwith Ġthe Ġ$ i $ th Ġexample . ĠPlug ging Ġ$ R ^{ w } _ \ phi Ġ$ Ġinto ĠE q .( DIS PLAY _ FORM 13 ), Ġwe Ġobtain Ġthe Ġweighted Ġupdate Ġfor Ġthe Ġmodel Ġ$\ math bf Ġ{\ the ta Ġ} $ : </s> In Ġpractice , Ġwhen Ġmin ib atch Ġsto ch astic Ġoptimization Ġis Ġused , Ġwe Ġapproximate Ġthe Ġweighted Ġsampling Ġby Ġtaking Ġthe Ġsoft max Ġover Ġthe Ġweights Ġof Ġonly Ġthe Ġmin ib atch Ġexamples . ĠThe Ġdata Ġweights Ġ$\ math bf Ġ{\ phi Ġ} $ Ġare Ġupdated Ġwith ĠE q .( DIS PLAY _ FORM 14 ). ĠIt Ġis Ġworth Ġnoting Ġthat Ġthe Ġprevious Ġwork ĠB IB REF 4 Ġsimilarly Ġderives Ġdata Ġweights Ġbased Ġon Ġtheir Ġgradient Ġdirections Ġon Ġa Ġvalidation Ġset . ĠOur Ġalgorithm Ġdiffers Ġin Ġthat Ġthe Ġdata Ġweights Ġare Ġparameters Ġmaintained Ġand Ġupdated Ġthroughout Ġthe Ġtraining , Ġinstead Ġof Ġre - est imated Ġfrom Ġscratch Ġin Ġeach Ġiteration . ĠExper iments Ġshow Ġthe Ġparam etric Ġtreatment Ġachieves Ġsuperior Ġperformance Ġin Ġvarious Ġsettings . ĠThere Ġare Ġalternative Ġparameter izations Ġof Ġ$ R _ \ phi Ġ$ Ġother Ġthan ĠE q .( DIS PLAY _ FORM 20 ). ĠFor Ġexample , Ġreplacing Ġ$\ phi Ġ_ i $ Ġin ĠE q .( DIS PLAY _ FORM 20 ) Ġwith Ġ$\ log Ġ\ phi Ġ_ i $ Ġin Ġeffect Ġchanges Ġthe Ġsoft max Ġnormal ization Ġin ĠE q .( DIS PLAY _ FORM 21 ) Ġto Ġlinear Ġnormal ization , Ġwhich Ġis Ġused Ġin ĠB IB REF 4 . </s> Exper iments </s> We Ġempir ically Ġvalidate Ġthe Ġproposed Ġdata Ġmanipulation Ġapproach Ġthrough Ġextensive Ġexperiments Ġon Ġlearning Ġaug mentation Ġand Ġweight ing . ĠWe Ġstudy Ġboth Ġtext Ġand Ġimage Ġclassification , Ġin Ġtwo Ġdifficult Ġsettings Ġof Ġlow Ġdata Ġregime Ġand Ġim balanced Ġlabels . </s> Exper iments Ġ: :: ĠExperimental ĠSetup </s> Base ĠModels . ĠWe Ġchoose Ġstrong Ġpret rained Ġnetworks Ġas Ġour Ġbase Ġmodels Ġfor Ġboth Ġtext Ġand Ġimage Ġclassification . ĠSpecifically , Ġon Ġtext Ġdata , Ġwe Ġuse Ġthe ĠB ERT Ġ( base , Ġunc ased ) Ġmodel ĠB IB REF 8 ; Ġwhile Ġon Ġimage Ġdata , Ġwe Ġuse ĠRes Net - 34 ĠB IB REF 9 Ġpret rained Ġon ĠImage Net . ĠWe Ġshow Ġthat , Ġeven Ġwith Ġthe Ġlarge - scale Ġpret raining , Ġdata Ġmanipulation Ġcan Ġstill Ġbe Ġvery Ġhelpful Ġto Ġboost Ġthe Ġmodel Ġperformance Ġon Ġdownstream Ġtasks . ĠSince Ġour Ġapproach Ġuses Ġvalidation Ġsets Ġfor Ġmanipulation Ġparameter Ġlearning , Ġfor Ġa Ġfair Ġcomparison Ġwith Ġthe Ġbase Ġmodel , Ġwe Ġtrain Ġthe Ġbase Ġmodel Ġin Ġtwo Ġways . ĠThe Ġfirst Ġis Ġto Ġtrain Ġthe Ġmodel Ġon Ġthe Ġtraining Ġsets Ġas Ġusual Ġand Ġselect Ġthe Ġbest Ġstep Ġusing Ġthe Ġvalidation Ġsets ; Ġthe Ġsecond Ġis Ġto Ġtrain Ġon Ġthe Ġmerged Ġtraining Ġand Ġvalidation Ġsets Ġfor Ġa Ġfixed Ġnumber Ġof Ġsteps . ĠThe Ġstep Ġnumber Ġis Ġset Ġto Ġthe Ġaverage Ġnumber Ġof Ġsteps Ġselected Ġin Ġthe Ġfirst Ġmethod . ĠWe Ġreport Ġthe Ġresults Ġof Ġboth Ġmethods . </s> Compar ison ĠMethods . ĠWe Ġcompare Ġour Ġapproach Ġwith Ġa Ġvariety Ġof Ġprevious Ġmethods Ġthat Ġwere Ġdesigned Ġfor Ġspecific Ġmanipulation Ġschemes : Ġ( 1 ) ĠFor Ġtext Ġdata Ġaug mentation , Ġwe Ġcompare Ġwith Ġthe Ġlatest Ġmodel - based Ġaug mentation ĠB IB REF 19 Ġwhich Ġuses Ġa Ġfixed Ġconditional ĠB ERT Ġlanguage Ġmodel Ġfor Ġword Ġsubstitution Ġ( section ĠSEC REF 15 ). ĠAs Ġwith Ġbase Ġmodels , Ġwe Ġalso Ġtried Ġfitting Ġthe Ġaugment atin Ġmodel Ġto Ġboth Ġthe Ġtraining Ġdata Ġand Ġthe Ġjoint Ġtraining - valid ation Ġdata , Ġand Ġdid Ġnot Ġobserve Ġsignificant Ġdifference . ĠFollowing ĠB IB REF 19 , Ġwe Ġalso Ġstudy Ġa Ġconventional Ġapproach Ġthat Ġreplaces Ġwords Ġwith Ġtheir Ġsyn onyms Ġusing ĠWord Net ĠB IB REF 40 . Ġ( 2 ) ĠFor Ġdata Ġweight ing , Ġwe Ġcompare Ġwith Ġthe Ġstate - of - the - art Ġapproach ĠB IB REF 4 Ġthat Ġdynamically Ġre - est imates Ġsample Ġweights Ġin Ġeach Ġiteration Ġbased Ġon Ġthe Ġvalidation Ġset Ġgradient Ġdirections . ĠWe Ġfollow ĠB IB REF 4 Ġand Ġalso Ġevaluate Ġthe Ġcommonly - used Ġproportion Ġmethod Ġthat Ġweights Ġdata Ġby Ġinverse Ġclass Ġfrequency . </s> Training . ĠFor Ġboth Ġthe ĠB ERT Ġclass ifier Ġand Ġthe Ġaug mentation Ġmodel Ġ( which Ġis Ġalso Ġbased Ġon ĠB ERT ), Ġwe Ġuse ĠAdam Ġoptimization Ġwith Ġan Ġinitial Ġlearning Ġrate Ġof Ġ4 e - 5 . ĠFor ĠRes N ets , Ġwe Ġuse ĠSG D Ġoptimization Ġwith Ġa Ġlearning Ġrate Ġof Ġ1 e - 3 . ĠFor Ġtext Ġdata Ġaug mentation , Ġwe Ġaugment Ġeach Ġmin ib atch Ġby Ġgenerating Ġtwo Ġor Ġthree Ġsamples Ġfor Ġeach Ġdata Ġpoints Ġ( each Ġwith Ġ1 , Ġ2 Ġor Ġ3 Ġsubstit utions ), Ġand Ġuse Ġboth Ġthe Ġsamples Ġand Ġthe Ġoriginal Ġdata Ġto Ġtrain Ġthe Ġmodel . ĠFor Ġdata Ġweight ing , Ġto Ġavoid Ġexploding Ġvalue , Ġwe Ġupdate Ġthe Ġweight Ġof Ġeach Ġdata Ġpoint Ġin Ġa Ġmin ib atch Ġby Ġdecaying Ġthe Ġprevious Ġweight Ġvalue Ġwith Ġa Ġfactor Ġof Ġ0 . 1 Ġand Ġthen Ġadding Ġthe Ġgradient . ĠAll Ġexperiments Ġwere Ġimplemented Ġwith ĠPy Tor ch Ġ( py tor ch . org ) Ġand Ġwere Ġperformed Ġon Ġa ĠLinux Ġmachine Ġwith Ġ4 ĠGTX Ġ1080 Ti ĠGPUs Ġand Ġ64 GB ĠRAM . ĠAll Ġreported Ġresults Ġare Ġaveraged Ġover Ġ15 Ġruns Ġ$\ pm Ġ$ Ġone Ġstandard Ġdeviation . </s> Exper iments Ġ: :: ĠLow ĠData ĠReg ime </s> We Ġstudy Ġthe Ġproblem Ġwhere Ġonly Ġvery Ġfew Ġlabeled Ġexamples Ġfor Ġeach Ġclass Ġare Ġavailable . ĠBoth Ġof Ġour Ġaug mentation Ġand Ġweight ing Ġboost Ġbase Ġmodel Ġperformance , Ġand Ġare Ġsuperior Ġto Ġrespective Ġcomparison Ġmethods . ĠWe Ġalso Ġobserve Ġthat Ġaug mentation Ġperforms Ġbetter Ġthan Ġweight ing Ġin Ġthe Ġlow - data Ġsetting . </s> Exper iments Ġ: :: ĠLow ĠData ĠReg ime Ġ: :: ĠSetup </s> For Ġtext Ġclassification , Ġwe Ġuse Ġthe Ġpopular Ġbenchmark Ġdatasets , Ġincluding ĠS ST - 5 Ġfor Ġ5 - class Ġsentence Ġsentiment ĠB IB REF 41 , ĠIM DB Ġfor Ġbinary Ġmovie Ġreview Ġsentiment ĠB IB REF 42 , Ġand ĠT REC Ġfor Ġ6 - class Ġquestion Ġtypes ĠB IB REF 43 . ĠWe Ġsubs ample Ġa Ġsmall Ġtraining Ġset Ġon Ġeach Ġtask Ġby Ġrandomly Ġpicking Ġ40 Ġinstances Ġfor Ġeach Ġclass . ĠWe Ġfurther Ġcreate Ġsmall Ġvalidation Ġsets , Ġi . e ., Ġ2 Ġinstances Ġper Ġclass Ġfor ĠS ST - 5 , Ġand Ġ5 Ġinstances Ġper Ġclass Ġfor ĠIM DB Ġand ĠT REC , Ġrespectively . ĠThe Ġreason Ġwe Ġuse Ġslightly Ġmore Ġvalidation Ġexamples Ġon ĠIM DB Ġand ĠT REC Ġis Ġthat Ġthe Ġmodel Ġcan Ġeasily Ġachieve Ġ100 % Ġvalidation Ġaccuracy Ġif Ġthe Ġvalidation Ġsets Ġare Ġtoo Ġsmall . ĠThus , Ġthe ĠS ST - 5 Ġtask Ġhas Ġ210 Ġlabeled Ġexamples Ġin Ġtotal , Ġwhile ĠIM DB Ġhas Ġ90 Ġlabels Ġand ĠT REC Ġhas Ġ270 . ĠSuch Ġextremely Ġsmall Ġdatasets Ġpose Ġsignificant Ġchallenges Ġfor Ġlearning Ġdeep Ġneural Ġnetworks . ĠSince Ġthe Ġmanipulation Ġparameters Ġare Ġtrained Ġusing Ġthe Ġsmall Ġvalidation Ġsets , Ġto Ġavoid Ġpossible Ġover fitting Ġwe Ġrestrict Ġthe Ġtraining Ġto Ġsmall Ġnumber Ġ( e . g ., Ġ5 Ġor Ġ10 ) Ġof Ġepoch s . ĠFor Ġimage Ġclassification , Ġwe Ġsimilarly Ġcreate Ġa Ġsmall Ġsubset Ġof Ġthe ĠC IF AR 10 Ġdata , Ġwhich Ġincludes Ġ40 Ġinstances Ġper Ġclass Ġfor Ġtraining , Ġand Ġ2 Ġinstances Ġper Ġclass Ġfor Ġvalidation . </s> Exper iments Ġ: :: ĠLow ĠData ĠReg ime Ġ: :: ĠResults </s> Table ĠT AB REF 26 Ġshows Ġthe Ġmanipulation Ġresults Ġon Ġtext Ġclassification . ĠFor Ġdata Ġaug mentation , Ġour Ġapproach Ġsignificantly Ġimproves Ġover Ġthe Ġbase Ġmodel Ġon Ġall Ġthe Ġthree Ġdatasets . ĠBesides , Ġcompared Ġto Ġboth Ġthe Ġconventional Ġsyn onym Ġsubstitution Ġand Ġthe Ġapproach Ġthat Ġkeeps Ġthe Ġaug mentation Ġnetwork Ġfixed , Ġour Ġadaptive Ġmethod Ġthat Ġfine - t unes Ġthe Ġaug mentation Ġnetwork Ġjointly Ġwith Ġmodel Ġtraining Ġachieves Ġsuperior Ġresults . ĠIndeed , Ġthe Ġhe uristic - based Ġsyn onym Ġapproach Ġcan Ġsometimes Ġharm Ġthe Ġmodel Ġperformance Ġ( e . g ., ĠS ST - 5 Ġand ĠIM DB ), Ġas Ġalso Ġobserved Ġin Ġprevious Ġwork ĠB IB REF 19 , ĠB IB REF 18 . ĠThis Ġcan Ġbe Ġbecause Ġthe Ġhe uristic Ġrules Ġdo Ġnot Ġfit Ġthe Ġtask Ġor Ġdatasets Ġwell . ĠIn Ġcontrast , Ġlearning - based Ġaug mentation Ġhas Ġthe Ġadvantage Ġof Ġadapt ively Ġgenerating Ġuseful Ġsamples Ġto Ġimprove Ġmodel Ġtraining . </s> Table ĠT AB REF 26 Ġalso Ġshows Ġthe Ġdata Ġweight ing Ġresults . ĠOur Ġweight Ġlearning Ġconsistently Ġimproves Ġover Ġthe Ġbase Ġmodel Ġand Ġthe Ġlatest Ġweight ing Ġmethod ĠB IB REF 4 . ĠIn Ġparticular , Ġinstead Ġof Ġre - est imating Ġsample Ġweights Ġfrom Ġscratch Ġin Ġeach Ġiteration ĠB IB REF 4 , Ġour Ġapproach Ġtreats Ġthe Ġweights Ġas Ġmanipulation Ġparameters Ġmaintained Ġthroughout Ġthe Ġtraining . ĠWe Ġspeculate Ġthat Ġthe Ġparam etric Ġtreatment Ġcan Ġadapt Ġweights Ġmore Ġsmoothly Ġand Ġprovide Ġhistorical Ġinformation , Ġwhich Ġis Ġbeneficial Ġin Ġthe Ġsmall - data Ġcontext . </s> It Ġis Ġinteresting Ġto Ġsee Ġfrom ĠTable ĠT AB REF 26 Ġthat Ġour Ġaug mentation Ġmethod Ġconsistently Ġoutper forms Ġthe Ġweight ing Ġmethod , Ġshowing Ġthat Ġdata Ġaug mentation Ġcan Ġbe Ġa Ġmore Ġsuitable Ġtechnique Ġthan Ġdata Ġweight ing Ġfor Ġmanipulating Ġsmall - size Ġdata . ĠOur Ġapproach Ġprovides Ġthe Ġgener ality Ġto Ġinstant iate Ġdiverse Ġmanipulation Ġtypes Ġand Ġlearn Ġwith Ġthe Ġsame Ġsingle Ġprocedure . </s> To Ġinvestigate Ġthe Ġaug mentation Ġmodel Ġand Ġhow Ġthe Ġfine - tun ing Ġaffects Ġthe Ġaug mentation Ġresults , Ġwe Ġshow Ġin ĠFigure ĠT AB REF 27 Ġthe Ġtop - 5 Ġmost Ġprobable Ġword Ġsubstit utions Ġpredicted Ġby Ġthe Ġaug mentation Ġmodel Ġfor Ġtwo Ġmasked Ġtokens , Ġrespectively . ĠComp aring Ġthe Ġresults Ġof Ġepoch Ġ1 Ġand Ġepoch Ġ3 , Ġwe Ġcan Ġsee Ġthe Ġaug mentation Ġmodel Ġevolves Ġand Ġdynamically Ġadjusts Ġthe Ġaug mentation Ġbehavior Ġas Ġthe Ġtraining Ġproceeds . ĠThrough Ġfine - tun ing , Ġthe Ġmodel Ġseems Ġto Ġmake Ġsubstit utions Ġthat Ġare Ġmore Ġcoherent Ġwith Ġthe Ġconditioning Ġlabel Ġand Ġrelevant Ġto Ġthe Ġoriginal Ġwords Ġ( e . g ., Ġreplacing Ġthe Ġword ĠâĢ ľ stri king âĢ Ŀ Ġwith ĠâĢ ľ b land âĢ Ŀ Ġin Ġepoch Ġ1 Ġv . s . ĠâĢ ľ ch arming âĢ Ŀ Ġin Ġepoch Ġ3 ). </s> Table ĠT AB REF 27 Ġshows Ġthe Ġdata Ġweight ing Ġresults Ġon Ġimage Ġclassification . ĠWe Ġevaluate Ġtwo Ġsettings Ġwith Ġthe ĠRes Net - 34 Ġbase Ġmodel Ġbeing Ġinitialized Ġrandomly Ġor Ġwith Ġpret rained Ġweights , Ġrespectively . ĠOur Ġdata Ġweight ing Ġconsistently Ġimproves Ġover Ġthe Ġbase Ġmodel Ġand ĠB IB REF 4 Ġregardless Ġof Ġthe Ġinitialization . </s> Exper iments Ġ: :: ĠIm balanced ĠLab els </s> We Ġnext Ġstudy Ġa Ġdifferent Ġproblem Ġsetting Ġwhere Ġthe Ġtraining Ġdata Ġof Ġdifferent Ġclasses Ġare Ġim balanced . ĠWe Ġshow Ġthe Ġdata Ġweight ing Ġapproach Ġgreatly Ġimproves Ġthe Ġclassification Ġperformance . ĠIt Ġis Ġalso Ġobserved Ġthat , Ġthe ĠLM Ġdata Ġaug mentation Ġapproach , Ġwhich Ġperforms Ġwell Ġin Ġthe Ġlow - data Ġsetting , Ġfails Ġon Ġthe Ġclass - im balance Ġproblems . </s> Exper iments Ġ: :: ĠIm balanced ĠLab els Ġ: :: ĠSetup </s> Though Ġthe Ġmethods Ġare Ġbroadly Ġapplicable Ġto Ġmulti - way Ġclassification Ġproblems , Ġhere Ġwe Ġonly Ġstudy Ġbinary Ġclassification Ġtasks Ġfor Ġsimplicity . ĠFor Ġtext Ġclassification , Ġwe Ġuse Ġthe ĠS ST - 2 Ġsentiment Ġanalysis Ġbenchmark ĠB IB REF 41 ; Ġwhile Ġfor Ġimage , Ġwe Ġselect Ġclass Ġ1 Ġand Ġ2 Ġfrom ĠC IF AR 10 Ġfor Ġbinary Ġclassification . ĠWe Ġuse Ġthe Ġsame Ġprocessing Ġon Ġboth Ġdatasets Ġto Ġbuild Ġthe Ġclass - im balance Ġsetting . ĠSpecifically , Ġwe Ġrandomly Ġselect Ġ1 , 000 Ġtraining Ġinstances Ġof Ġclass Ġ2 , Ġand Ġvary Ġthe Ġnumber Ġof Ġclass - 1 Ġinstances Ġin Ġ$\ l brace Ġ20 , Ġ50 , Ġ100 \ r brace Ġ$ . ĠFor Ġeach Ġdataset , Ġwe Ġuse Ġ10 Ġvalidation Ġexamples Ġin Ġeach Ġclass . ĠTr ained Ġmodels Ġare Ġevaluated Ġon Ġthe Ġfull Ġbinary - class Ġtest Ġset . </s> Exper iments Ġ: :: ĠIm balanced ĠLab els Ġ: :: ĠResults </s> Table ĠT AB REF 29 Ġshows Ġthe Ġclassification Ġresults Ġon ĠS ST - 2 Ġwith Ġvarying Ġimbalance Ġratios . ĠWe Ġcan Ġsee Ġour Ġdata Ġweight ing Ġperforms Ġbest Ġacross Ġall Ġsettings . ĠIn Ġparticular , Ġthe Ġimprovement Ġover Ġthe Ġbase Ġmodel Ġincreases Ġas Ġthe Ġdata Ġgets Ġmore Ġim balanced , Ġranging Ġfrom Ġaround Ġ6 Ġaccuracy Ġpoints Ġon Ġ100 : 1000 Ġto Ġover Ġ20 Ġaccuracy Ġpoints Ġon Ġ20 : 1000 . ĠOur Ġmethod Ġis Ġagain Ġconsistently Ġbetter Ġthan ĠB IB REF 4 , Ġvalid ating Ġthat Ġthe Ġparam etric Ġtreatment Ġis Ġbeneficial . ĠThe Ġproportion - based Ġdata Ġweight ing Ġprovides Ġonly Ġlimited Ġimprovement , Ġshowing Ġthe Ġadvantage Ġof Ġadaptive Ġdata Ġweight ing . ĠThe Ġbase Ġmodel Ġtrained Ġon Ġthe Ġjoint Ġtraining - valid ation Ġdata Ġfor Ġfixed Ġsteps Ġfails Ġto Ġperform Ġwell , Ġpartly Ġdue Ġto Ġthe Ġlack Ġof Ġa Ġproper Ġmechanism Ġfor Ġselecting Ġsteps . </s> Table ĠT AB REF 30 Ġshows Ġthe Ġresults Ġon Ġim balanced ĠC IF AR 10 Ġclassification . ĠSimilarly , Ġour Ġmethod Ġoutper forms Ġother Ġcomparison Ġapproaches . ĠIn Ġcontrast , Ġthe Ġfixed Ġproportion - based Ġmethod Ġsometimes Ġharms Ġthe Ġperformance Ġas Ġin Ġthe Ġ50 : 1000 Ġand Ġ100 : 1000 Ġsettings . </s> We Ġalso Ġtested Ġthe Ġtext Ġaug mentation ĠLM Ġon Ġthe ĠS ST - 2 Ġim balanced Ġdata . ĠInterestingly , Ġthe Ġaug mentation Ġtends Ġto Ġhinder Ġmodel Ġtraining Ġand Ġyields Ġaccuracy Ġof Ġaround Ġ50 % Ġ( random Ġguess ). ĠThis Ġis Ġbecause Ġthe Ġaug mentation ĠLM Ġis Ġfirst Ġfit Ġto Ġthe Ġim balanced Ġdata , Ġwhich Ġmakes Ġlabel Ġpreservation Ġinaccurate Ġand Ġintroduces Ġlots Ġof Ġnoise Ġduring Ġaug mentation . ĠThough Ġa Ġmore Ġcarefully Ġdesigned Ġaug mentation Ġmechanism Ġcan Ġpotentially Ġhelp Ġwith Ġim balanced Ġclassification Ġ( e . g ., Ġaugment ing Ġonly Ġthe Ġrare Ġclasses ), Ġthe Ġabove Ġobservation Ġfurther Ġshows Ġthat Ġthe Ġvarying Ġdata Ġmanipulation Ġschemes Ġhave Ġdifferent Ġapplicable Ġsc opes . ĠOur Ġapproach Ġis Ġthus Ġfavorable Ġas Ġthe Ġsingle Ġalgorithm Ġcan Ġbe Ġinstant iated Ġto Ġlearn Ġdifferent Ġschemes . </s> Disc ussions : ĠAl gorithm ĠExt rap olation Ġbetween ĠLearning ĠParad ig ms </s> Con clusions . ĠWe Ġhave Ġdeveloped Ġa Ġnew Ġmethod Ġof Ġlearning Ġdifferent Ġdata Ġmanipulation Ġschemes Ġwith Ġthe Ġsame Ġsingle Ġalgorithm . ĠDifferent Ġmanipulation Ġschemes Ġreduce Ġto Ġjust Ġdifferent Ġparameter ization Ġof Ġthe Ġdata Ġreward Ġfunction . ĠThe Ġmanipulation Ġparameters Ġare Ġtrained Ġjointly Ġwith Ġthe Ġtarget Ġmodel Ġparameters . ĠWe Ġinstant iate Ġthe Ġalgorithm Ġfor Ġdata Ġaug mentation Ġand Ġweight ing , Ġand Ġshow Ġimproved Ġperformance Ġover Ġstrong Ġbase Ġmodels Ġand Ġprevious Ġmanipulation Ġmethods . ĠWe Ġare Ġexcited Ġto Ġexplore Ġmore Ġtypes Ġof Ġmanip ulations Ġsuch Ġas Ġdata Ġsynthesis , Ġand Ġin Ġparticular Ġstudy Ġthe Ġcombination Ġof Ġdifferent Ġmanipulation Ġschemes . </s> The Ġproposed Ġmethod Ġbuilds Ġupon Ġthe Ġconnections Ġbetween Ġsupervised Ġlearning Ġand Ġreinforcement Ġlearning Ġ( RL ) ĠB IB REF 6 Ġthrough Ġwhich Ġwe Ġextrap olate Ġan Ġoff - the - she lf Ġreward Ġlearning Ġalgorithm Ġin Ġthe ĠRL Ġliterature Ġto Ġthe Ġsupervised Ġsetting . ĠThe Ġway Ġwe Ġobtained Ġthe Ġmanipulation Ġalgorithm Ġrepresents Ġa Ġgeneral Ġmeans Ġof Ġinnov ating Ġproblem Ġsolutions Ġbased Ġon Ġun ifying Ġformal isms Ġof Ġdifferent Ġlearning Ġparad ig ms . ĠSpecifically , Ġa Ġun ifying Ġformal ism Ġnot Ġonly Ġoffers Ġnew Ġunderstand ings Ġof Ġthe Ġseemingly Ġdistinct Ġparad ig ms , Ġbut Ġalso Ġallows Ġus Ġto Ġsystematically Ġapply Ġsolutions Ġto Ġproblems Ġin Ġone Ġparadigm Ġto Ġsimilar Ġproblems Ġin Ġanother . ĠPrevious Ġwork Ġalong Ġthis Ġline Ġhas Ġmade Ġfruitful Ġresults Ġin Ġother Ġdomains . ĠFor Ġexample , Ġan Ġextended Ġformulation Ġof ĠB IB REF 6 Ġthat Ġconnects ĠRL Ġand Ġposterior Ġregular ization Ġ( PR ) ĠB IB REF 44 , ĠB IB REF 45 Ġhas Ġenabled Ġto Ġsimilarly Ġexport Ġa Ġreward Ġlearning Ġalgorithm Ġto Ġthe Ġcontext Ġof ĠPR Ġfor Ġlearning Ġstructured Ġknowledge ĠB IB REF 46 . ĠBy Ġestablishing Ġa Ġuniform Ġab stration Ġof ĠG AN s ĠB IB REF 47 Ġand ĠVA Es ĠB IB REF 33 , ĠB IB REF 48 Ġexchange Ġtechniques Ġbetween Ġthe Ġtwo Ġfamilies Ġand Ġget Ġimproved Ġgener ative Ġmodeling . ĠOther Ġwork Ġin Ġthe Ġsimilar Ġspirit Ġincludes ĠB IB REF 49 , ĠB IB REF 50 , ĠB IB REF 51 . </s> By Ġextrap ol ating Ġalgorithms Ġbetween Ġparad ig ms , Ġone Ġcan Ġgo Ġbeyond Ġcrafting Ġnew Ġalgorithms Ġfrom Ġscratch Ġas Ġin Ġmost Ġexisting Ġstudies , Ġwhich Ġoften Ġrequires Ġdeep Ġexpertise Ġand Ġyields Ġunique Ġsolutions Ġin Ġa Ġdedicated Ġcontext . ĠInstead , Ġinnovation Ġbecomes Ġeasier Ġby Ġimporting Ġrich Ġideas Ġfrom Ġother Ġparad ig ms , Ġand Ġis Ġrepeat able Ġas Ġa Ġnew Ġalgorithm Ġcan Ġbe Ġmethod ically Ġextrap olated Ġto Ġmultiple Ġdifferent Ġcontexts .
Introduction The performance of machines often crucially depend on the amount and quality of the data used for training. It has become increasingly ubiquitous to manipulate data to improve learning, especially in low data regime or in presence of low-quality datasets (e.g., imbalanced labels). For example, data augmentation applies label-preserving transformations on original data points to expand the data size; data weighting assigns an importance weight to each instance to adapt its effect on learning; and data synthesis generates entire artificial examples. Different types of manipulation can be suitable for different application settings. Common data manipulation methods are usually designed manually, e.g., augmenting by flipping an image or replacing a word with synonyms, and weighting with inverse class frequency or loss values BIBREF0, BIBREF1. Recent work has studied automated approaches, such as learning the composition of augmentation operators with reinforcement learning BIBREF2, BIBREF3, deriving sample weights adaptively from a validation set via meta learning BIBREF4, or learning a weighting network by inducing a curriculum BIBREF5. These learning-based approaches have alleviated the engineering burden and produced impressive results. However, the algorithms are usually designed specifically for certain types of manipulation (e.g., either augmentation or weighting) and thus have limited application scope in practice. In this work, we propose a new approach that enables learning for different manipulation schemes with the same single algorithm. Our approach draws inspiration from the recent work BIBREF6 that shows equivalence between the data in supervised learning and the reward function in reinforcement learning. We thus adapt an off-the-shelf reward learning algorithm BIBREF7 to the supervised setting for automated data manipulation. The marriage of the two paradigms results in a simple yet general algorithm, where various manipulation schemes are reduced to different parameterization of the data reward. Free parameters of manipulation are learned jointly with the target model through efficient gradient descent on validation examples. We demonstrate instantiations of the approach for automatically fine-tuning an augmentation network and learning data weights, respectively. We conduct extensive experiments on text and image classification in challenging situations of very limited data and imbalanced labels. Both augmentation and weighting by our approach significantly improve over strong base models, even though the models are initialized with large-scale pretrained networks such as BERT BIBREF8 for text and ResNet BIBREF9 for images. Our approach, besides its generality, also outperforms a variety of dedicated rule- and learning-based methods for either augmentation or weighting, respectively. Lastly, we observe that the two types of manipulation tend to excel in different contexts: augmentation shows superiority over weighting with a small amount of data available, while weighting is better at addressing class imbalance problems. The way we derive the manipulation algorithm represents a general means of problem solving through algorithm extrapolation between learning paradigms, which we discuss more in section SECREF6. Related Work Rich types of data manipulation have been increasingly used in modern machine learning pipelines. Previous work each has typically focused on a particular manipulation type. Data augmentation that perturbs examples without changing the labels is widely used especially in vision BIBREF10, BIBREF11 and speech BIBREF12, BIBREF13 domains. Common heuristic-based methods on images include cropping, mirroring, rotation BIBREF11, and so forth. Recent work has developed automated augmentation approaches BIBREF3, BIBREF2, BIBREF14, BIBREF15, BIBREF16. BIBREF17 additionally use large-scale unlabeled data. BIBREF3, BIBREF2 learn to induce the composition of data transformation operators. Instead of treating data augmentation as a policy in reinforcement learning BIBREF3, we formulate manipulation as a reward function and use efficient stochastic gradient descent to learn the manipulation parameters. Text data augmentation has also achieved impressive success, such as contextual augmentation BIBREF18, BIBREF19, back-translation BIBREF20, and manual approaches BIBREF21, BIBREF22. In addition to perturbing the input text as in classification tasks, text generation problems expose opportunities to adding noise also in the output text, such as BIBREF23, BIBREF24. Recent work BIBREF6 shows output nosing in sequence generation can be treated as an intermediate approach in between supervised learning and reinforcement learning, and developed a new sequence learning algorithm that interpolates between the spectrum of existing algorithms. We instantiate our approach for text contextual augmentation as in BIBREF18, BIBREF19, but enhance the previous work by additionally fine-tuning the augmentation network jointly with the target model. Data weighting has been used in various algorithms, such as AdaBoost BIBREF0, self-paced learning BIBREF25, hard-example mining BIBREF26, and others BIBREF27, BIBREF28. These algorithms largely define sample weights based on training loss. Recent work BIBREF5, BIBREF29 learns a separate network to predict sample weights. Of particular relevance to our work is BIBREF4 which induces sample weights using a validation set. The data weighting mechanism instantiated by our framework has a key difference in that samples weights are treated as parameters that are updated iteratively, instead of re-estimated from scratch at each step. We show improved performance of our approach. Besides, our data manipulation approach is derived based on a different perspective of reward learning, instead of meta-learning as in BIBREF4. Another popular type of data manipulation involves data synthesis, which creates entire artificial samples from scratch. GAN-based approaches have achieved impressive results for synthesizing conditional image data BIBREF30, BIBREF31. In the text domain, controllable text generation BIBREF32 presents a way of co-training the data generator and classifier in a cyclic manner within a joint VAE BIBREF33 and wake-sleep BIBREF34 framework. It is interesting to explore the instantiation of the present approach for adaptive data synthesis in the future. Background We first present the relevant work upon which our automated data manipulation is built. This section also establishes the notations used throughout the paper. Let $\mathbf {x}$ denote the input and $y$ the output. For example, in text classification, $\mathbf {x}$ can be a sentence and $y$ is the sentence label. Denote the model of interest as $p_\theta (y|\mathbf {x})$, where $\mathbf {\theta }$ is the model parameters to be learned. In supervised setting, given a set of training examples $\mathcal {D}=\lbrace (\mathbf {x}^*, y^*)\rbrace $, we learn the model by maximizing the data log-likelihood. Background ::: Equivalence between Data and Reward The recent work BIBREF6 introduced a unifying perspective of reformulating maximum likelihood supervised learning as a special instance of a policy optimization framework. In this perspective, data examples providing supervision signals are equivalent to a specialized reward function. Since the original framework BIBREF6 was derived for sequence generation problems, here we present a slightly adapted formulation for our context of data manipulation. To connect the maximum likelihood supervised learning with policy optimization, consider the model $p_\theta (y|\mathbf {x})$ as a policy that takes “action” $y$ given the “state” $\mathbf {x}$. Let $R(\mathbf {x}, y |\mathcal {D})\in \mathbb {R}$ denote a reward function, and $p(\mathbf {x})$ be the empirical data distribution which is known given $\mathcal {D}$. Further assume a variational distribution $q(\mathbf {x}, y)$ that factorizes as $q(\mathbf {x},y)=p(\mathbf {x})q(y|\mathbf {x})$. A variational policy optimization objective is then written as: where $\text{KL}(\cdot \Vert \cdot )$ is the Kullback–Leibler divergence; $\text{H}(\cdot )$ is the Shannon entropy; and $\alpha ,\beta >0$ are balancing weights. The objective is in the same form with the RL-as-inference formalism of policy optimization BIBREF35, BIBREF36, BIBREF37. Intuitively, the objective maximizes the expected reward under $q$, and enforces the model $p_\theta $ to stay close to $q$, with a maximum entropy regularization over $q$. The problem is solved with an EM procedure that optimizes $q$ and $\mathbf {\theta }$ alternatingly: where $Z$ is the normalization term. With the established framework, it is easy to show that the above optimization procedure reduces to maximum likelihood learning by taking $\alpha \rightarrow 0, \beta =1$, and the reward function: That is, a sample $(\mathbf {x}, y)$ receives a unit reward only when it matches a training example in the dataset, while the reward is negative infinite in all other cases. To make the equivalence to maximum likelihood learning clearer, note that the above M-step now reduces to where the joint distribution $p(\mathbf {x})\exp \lbrace R_\delta \rbrace /Z$ equals the empirical data distribution, which means the M-step is in fact maximizing the data log-likelihood of the model $p_\theta $. Background ::: Gradient-based Reward Learning There is a rich line of research on learning the reward in reinforcement learning. Of particular interest to this work is BIBREF7 which learns a parametric intrinsic reward that additively transforms the original task reward (a.k.a extrinsic reward) to improve the policy optimization. For consistency of notations with above, formally, let $p_\theta (y|\mathbf {x})$ be a policy where $y$ is an action and $\mathbf {x}$ is a state. Let $R_\phi ^{in}$ be the intrinsic reward with parameters $\mathbf {\phi }$. In each iteration, the policy parameter $\mathbf {\theta }$ is updated to maximize the joint rewards, through: where $\mathcal {L}^{ex+in}$ is the expectation of the sum of extrinsic and intrinsic rewards; and $\gamma $ is the step size. The equation shows $\mathbf {\theta }^{\prime }$ depends on $\mathbf {\phi }$, thus we can write as $\mathbf {\theta }^{\prime }=\mathbf {\theta }^{\prime }(\mathbf {\phi })$. The next step is to optimize the intrinsic reward parameters $\mathbf {\phi }$. Recall that the ultimate measure of the performance of a policy is the value of extrinsic reward it achieves. Therefore, a good intrinsic reward is supposed to, when the policy is trained with it, increase the eventual extrinsic reward. The update to $\mathbf {\phi }$ is then written as: That is, we want the expected extrinsic reward $\mathcal {L}^{ex}(\mathbf {\theta }^{\prime })$ of the new policy $\mathbf {\theta }^{\prime }$ to be maximized. Since $\mathbf {\theta }^{\prime }$ is a function of $\mathbf {\phi }$, we can directly backpropagate the gradient through $\mathbf {\theta }^{\prime }$ to $\mathbf {\phi }$. Learning Data Manipulation ::: Method ::: Parameterizing Data Manipulation We now develop our approach of learning data manipulation, through a novel marriage of supervised learning and the above reward learning. Specifically, from the policy optimization perspective, due to the $\delta $-function reward (Eq.DISPLAY_FORM4), the standard maximum likelihood learning is restricted to use only the exact training examples $\mathcal {D}$ in a uniform way. A natural idea of enabling data manipulation is to relax the strong restrictions of the $\delta $-function reward and instead use a relaxed reward $R_\phi (\mathbf {x}, y | \mathcal {D})$ with parameters $\mathbf {\phi }$. The relaxed reward can be parameterized in various ways, resulting in different types of manipulation. For example, when a sample $(\mathbf {x}, y)$ matches a data instance, instead of returning constant 1 by $R_\delta $, the new $R_\phi $ can return varying reward values depending on the matched instance, resulting in a data weighting scheme. Alternatively, $R_\phi $ can return a valid reward even when $\mathbf {x}$ matches a data example only in part, or $(\mathbf {x}, y)$ is an entire new sample not in $\mathcal {D}$, which in effect makes data augmentation and data synthesis, respectively, in which cases $\mathbf {\phi }$ is either a data transformer or a generator. In the next section, we demonstrate two particular parameterizations for data augmentation and weighting, respectively. We thus have shown that the diverse types of manipulation all boil down to a parameterized data reward $R_\phi $. Such an concise, uniform formulation of data manipulation has the advantage that, once we devise a method of learning the manipulation parameters $\mathbf {\phi }$, the resulting algorithm can directly be applied to automate any manipulation type. We present a learning algorithm next. Learning Data Manipulation ::: Method ::: Learning Manipulation Parameters To learn the parameters $\mathbf {\phi }$ in the manipulation reward $R_\phi (\mathbf {x}, y | \mathcal {D})$, we could in principle adopt any off-the-shelf reward learning algorithm in the literature. In this work, we draw inspiration from the above gradient-based reward learning (section SECREF3) due to its simplicity and efficiency. Briefly, the objective of $\mathbf {\phi }$ is to maximize the ultimate measure of the performance of model $p_\theta (\mathbf {y}|\mathbf {x})$, which, in the context of supervised learning, is the model performance on a held-out validation set. The algorithm optimizes $\mathbf {\theta }$ and $\mathbf {\phi }$ alternatingly, corresponding to Eq.(DISPLAY_FORM7) and Eq.(DISPLAY_FORM8), respectively. More concretely, in each iteration, we first update the model parameters $\mathbf {\theta }$ in analogue to Eq.(DISPLAY_FORM7) which optimizes intrinsic reward-enriched objective. Here, we optimize the log-likelihood of the training set enriched with data manipulation. That is, we replace $R_\delta $ with $R_\phi $ in Eq.(DISPLAY_FORM5), and obtain the augmented M-step: By noticing that the new $\mathbf {\theta }^{\prime }$ depends on $\mathbf {\phi }$, we can write $\mathbf {\theta }^{\prime }$ as a function of $\mathbf {\phi }$, namely, $\mathbf {\theta }^{\prime }=\mathbf {\theta }^{\prime }(\mathbf {\phi })$. The practical implementation of the above update depends on the actual parameterization of manipulation $R_\phi $, which we discuss in more details in the next section. The next step is to optimize $\mathbf {\phi }$ in terms of the model validation performance, in analogue to Eq.(DISPLAY_FORM8). Formally, let $\mathcal {D}^v$ be the validation set of data examples. The update is then: where, since $\mathbf {\theta }^{\prime }$ is a function of $\mathbf {\phi }$, the gradient is backpropagated to $\mathbf {\phi }$ through $\mathbf {\theta }^{\prime }(\mathbf {\phi })$. Taking data weighting for example where $\mathbf {\phi }$ is the training sample weights (more details in section SECREF15), the update is to optimize the weights of training samples so that the model performs best on the validation set. The resulting algorithm is summarized in Algorithm FIGREF11. Figure FIGREF11 illustrates the computation flow. Learning the manipulation parameters effectively uses a held-out validation set. We show in our experiments that a very small set of validation examples (e.g., 2 labels per class) is enough to significantly improve the model performance in low data regime. It is worth noting that some previous work has also leveraged validation examples, such as learning data augmentation with policy gradient BIBREF3 or inducing data weights with meta-learning BIBREF4. Our approach is inspired from a distinct paradigm of (intrinsic) reward learning. In contrast to BIBREF3 that treats data augmentation as a policy, we instead formulate manipulation as a reward function and enable efficient stochastic gradient updates. Our approach is also more broadly applicable to diverse data manipulation types than BIBREF4, BIBREF3. Learning Data Manipulation ::: Instantiations: Augmentation & Weighting As a case study, we show two parameterizations of $R_\phi $ which instantiate distinct data manipulation schemes. The first example learns augmentation for text data, a domain that has been less studied in the literature compared to vision and speech BIBREF18, BIBREF38. The second instance focuses on automated data weighting, which is applicable to any data domains. Learning Data Manipulation ::: Instantiations: Augmentation & Weighting ::: Fine-tuning Text Augmentation The recent work BIBREF18, BIBREF19 developed a novel contextual augmentation approach for text data, in which a powerful pretrained language model (LM), such as BERT BIBREF8, is used to generate substitutions of words in a sentence. Specifically, given an observed sentence $\mathbf {x}^*$, the method first randomly masks out a few words. The masked sentence is then fed to BERT which fills the masked positions with new words. To preserve the original sentence class, the BERT LM is retrofitted as a label-conditional model, and trained on the task training examples. The resulting model is then fixed and used to augment data during the training of target model. We denote the augmentation distribution as $g_{\phi _0}(\mathbf {x}|\mathbf {x}^*, \mathbf {y}^*)$, where $\mathbf {\phi }_0$ is the fixed BERT LM parameters. The above process has two drawbacks. First, the LM is fixed after fitting to the task data. In the subsequent phase of training the target model, the LM augments data without knowing the state of the target model, which can lead to sub-optimal results. Second, in the cases where the task dataset is small, the LM can be insufficiently trained for preserving the labels faithfully, resulting in noisy augmented samples. To address the difficulties, it is beneficial to apply the proposed learning data manipulation algorithm to additionally fine-tune the LM jointly with target model training. As discussed in section SECREF4, this reduces to properly parameterizing the data reward function: That is, a sample $(\mathbf {x}, y)$ receives a unit reward when $y$ is the true label and $\mathbf {x}$ is the augmented sample by the LM (instead of the exact original data $\mathbf {x}^*$). Plugging the reward into Eq.(DISPLAY_FORM13), we obtain the data-augmented update for the model parameters: That is, we pick an example from the training set, and use the LM to create augmented samples, which are then used to update the target model. Regarding the update of augmentation parameters $\mathbf {\phi }$ (Eq.DISPLAY_FORM14), since text samples are discrete, to enable efficient gradient propagation through $\mathbf {\theta }^{\prime }$ to $\mathbf {\phi }$, we use a gumbel-softmax approximation BIBREF39 to $\mathbf {x}$ when sampling substitution words from the LM. Learning Data Manipulation ::: Instantiations: Augmentation & Weighting ::: Learning Data Weights We now demonstrate the instantiation of data weighting. We aim to assign an importance weight to each training example to adapt its effect on model training. We automate the process by learning the data weights. This is achieved by parameterizing $R_\phi $ as: where $\phi _i\in \mathbb {R}$ is the weight associated with the $i$th example. Plugging $R^{w}_\phi $ into Eq.(DISPLAY_FORM13), we obtain the weighted update for the model $\mathbf {\theta }$: In practice, when minibatch stochastic optimization is used, we approximate the weighted sampling by taking the softmax over the weights of only the minibatch examples. The data weights $\mathbf {\phi }$ are updated with Eq.(DISPLAY_FORM14). It is worth noting that the previous work BIBREF4 similarly derives data weights based on their gradient directions on a validation set. Our algorithm differs in that the data weights are parameters maintained and updated throughout the training, instead of re-estimated from scratch in each iteration. Experiments show the parametric treatment achieves superior performance in various settings. There are alternative parameterizations of $R_\phi $ other than Eq.(DISPLAY_FORM20). For example, replacing $\phi _i$ in Eq.(DISPLAY_FORM20) with $\log \phi _i$ in effect changes the softmax normalization in Eq.(DISPLAY_FORM21) to linear normalization, which is used in BIBREF4. Experiments We empirically validate the proposed data manipulation approach through extensive experiments on learning augmentation and weighting. We study both text and image classification, in two difficult settings of low data regime and imbalanced labels. Experiments ::: Experimental Setup Base Models. We choose strong pretrained networks as our base models for both text and image classification. Specifically, on text data, we use the BERT (base, uncased) model BIBREF8; while on image data, we use ResNet-34 BIBREF9 pretrained on ImageNet. We show that, even with the large-scale pretraining, data manipulation can still be very helpful to boost the model performance on downstream tasks. Since our approach uses validation sets for manipulation parameter learning, for a fair comparison with the base model, we train the base model in two ways. The first is to train the model on the training sets as usual and select the best step using the validation sets; the second is to train on the merged training and validation sets for a fixed number of steps. The step number is set to the average number of steps selected in the first method. We report the results of both methods. Comparison Methods. We compare our approach with a variety of previous methods that were designed for specific manipulation schemes: (1) For text data augmentation, we compare with the latest model-based augmentation BIBREF19 which uses a fixed conditional BERT language model for word substitution (section SECREF15). As with base models, we also tried fitting the augmentatin model to both the training data and the joint training-validation data, and did not observe significant difference. Following BIBREF19, we also study a conventional approach that replaces words with their synonyms using WordNet BIBREF40. (2) For data weighting, we compare with the state-of-the-art approach BIBREF4 that dynamically re-estimates sample weights in each iteration based on the validation set gradient directions. We follow BIBREF4 and also evaluate the commonly-used proportion method that weights data by inverse class frequency. Training. For both the BERT classifier and the augmentation model (which is also based on BERT), we use Adam optimization with an initial learning rate of 4e-5. For ResNets, we use SGD optimization with a learning rate of 1e-3. For text data augmentation, we augment each minibatch by generating two or three samples for each data points (each with 1, 2 or 3 substitutions), and use both the samples and the original data to train the model. For data weighting, to avoid exploding value, we update the weight of each data point in a minibatch by decaying the previous weight value with a factor of 0.1 and then adding the gradient. All experiments were implemented with PyTorch (pytorch.org) and were performed on a Linux machine with 4 GTX 1080Ti GPUs and 64GB RAM. All reported results are averaged over 15 runs $\pm $ one standard deviation. Experiments ::: Low Data Regime We study the problem where only very few labeled examples for each class are available. Both of our augmentation and weighting boost base model performance, and are superior to respective comparison methods. We also observe that augmentation performs better than weighting in the low-data setting. Experiments ::: Low Data Regime ::: Setup For text classification, we use the popular benchmark datasets, including SST-5 for 5-class sentence sentiment BIBREF41, IMDB for binary movie review sentiment BIBREF42, and TREC for 6-class question types BIBREF43. We subsample a small training set on each task by randomly picking 40 instances for each class. We further create small validation sets, i.e., 2 instances per class for SST-5, and 5 instances per class for IMDB and TREC, respectively. The reason we use slightly more validation examples on IMDB and TREC is that the model can easily achieve 100% validation accuracy if the validation sets are too small. Thus, the SST-5 task has 210 labeled examples in total, while IMDB has 90 labels and TREC has 270. Such extremely small datasets pose significant challenges for learning deep neural networks. Since the manipulation parameters are trained using the small validation sets, to avoid possible overfitting we restrict the training to small number (e.g., 5 or 10) of epochs. For image classification, we similarly create a small subset of the CIFAR10 data, which includes 40 instances per class for training, and 2 instances per class for validation. Experiments ::: Low Data Regime ::: Results Table TABREF26 shows the manipulation results on text classification. For data augmentation, our approach significantly improves over the base model on all the three datasets. Besides, compared to both the conventional synonym substitution and the approach that keeps the augmentation network fixed, our adaptive method that fine-tunes the augmentation network jointly with model training achieves superior results. Indeed, the heuristic-based synonym approach can sometimes harm the model performance (e.g., SST-5 and IMDB), as also observed in previous work BIBREF19, BIBREF18. This can be because the heuristic rules do not fit the task or datasets well. In contrast, learning-based augmentation has the advantage of adaptively generating useful samples to improve model training. Table TABREF26 also shows the data weighting results. Our weight learning consistently improves over the base model and the latest weighting method BIBREF4. In particular, instead of re-estimating sample weights from scratch in each iteration BIBREF4, our approach treats the weights as manipulation parameters maintained throughout the training. We speculate that the parametric treatment can adapt weights more smoothly and provide historical information, which is beneficial in the small-data context. It is interesting to see from Table TABREF26 that our augmentation method consistently outperforms the weighting method, showing that data augmentation can be a more suitable technique than data weighting for manipulating small-size data. Our approach provides the generality to instantiate diverse manipulation types and learn with the same single procedure. To investigate the augmentation model and how the fine-tuning affects the augmentation results, we show in Figure TABREF27 the top-5 most probable word substitutions predicted by the augmentation model for two masked tokens, respectively. Comparing the results of epoch 1 and epoch 3, we can see the augmentation model evolves and dynamically adjusts the augmentation behavior as the training proceeds. Through fine-tuning, the model seems to make substitutions that are more coherent with the conditioning label and relevant to the original words (e.g., replacing the word “striking” with “bland” in epoch 1 v.s. “charming” in epoch 3). Table TABREF27 shows the data weighting results on image classification. We evaluate two settings with the ResNet-34 base model being initialized randomly or with pretrained weights, respectively. Our data weighting consistently improves over the base model and BIBREF4 regardless of the initialization. Experiments ::: Imbalanced Labels We next study a different problem setting where the training data of different classes are imbalanced. We show the data weighting approach greatly improves the classification performance. It is also observed that, the LM data augmentation approach, which performs well in the low-data setting, fails on the class-imbalance problems. Experiments ::: Imbalanced Labels ::: Setup Though the methods are broadly applicable to multi-way classification problems, here we only study binary classification tasks for simplicity. For text classification, we use the SST-2 sentiment analysis benchmark BIBREF41; while for image, we select class 1 and 2 from CIFAR10 for binary classification. We use the same processing on both datasets to build the class-imbalance setting. Specifically, we randomly select 1,000 training instances of class 2, and vary the number of class-1 instances in $\lbrace 20, 50, 100\rbrace $. For each dataset, we use 10 validation examples in each class. Trained models are evaluated on the full binary-class test set. Experiments ::: Imbalanced Labels ::: Results Table TABREF29 shows the classification results on SST-2 with varying imbalance ratios. We can see our data weighting performs best across all settings. In particular, the improvement over the base model increases as the data gets more imbalanced, ranging from around 6 accuracy points on 100:1000 to over 20 accuracy points on 20:1000. Our method is again consistently better than BIBREF4, validating that the parametric treatment is beneficial. The proportion-based data weighting provides only limited improvement, showing the advantage of adaptive data weighting. The base model trained on the joint training-validation data for fixed steps fails to perform well, partly due to the lack of a proper mechanism for selecting steps. Table TABREF30 shows the results on imbalanced CIFAR10 classification. Similarly, our method outperforms other comparison approaches. In contrast, the fixed proportion-based method sometimes harms the performance as in the 50:1000 and 100:1000 settings. We also tested the text augmentation LM on the SST-2 imbalanced data. Interestingly, the augmentation tends to hinder model training and yields accuracy of around 50% (random guess). This is because the augmentation LM is first fit to the imbalanced data, which makes label preservation inaccurate and introduces lots of noise during augmentation. Though a more carefully designed augmentation mechanism can potentially help with imbalanced classification (e.g., augmenting only the rare classes), the above observation further shows that the varying data manipulation schemes have different applicable scopes. Our approach is thus favorable as the single algorithm can be instantiated to learn different schemes. Discussions: Algorithm Extrapolation between Learning Paradigms Conclusions. We have developed a new method of learning different data manipulation schemes with the same single algorithm. Different manipulation schemes reduce to just different parameterization of the data reward function. The manipulation parameters are trained jointly with the target model parameters. We instantiate the algorithm for data augmentation and weighting, and show improved performance over strong base models and previous manipulation methods. We are excited to explore more types of manipulations such as data synthesis, and in particular study the combination of different manipulation schemes. The proposed method builds upon the connections between supervised learning and reinforcement learning (RL) BIBREF6 through which we extrapolate an off-the-shelf reward learning algorithm in the RL literature to the supervised setting. The way we obtained the manipulation algorithm represents a general means of innovating problem solutions based on unifying formalisms of different learning paradigms. Specifically, a unifying formalism not only offers new understandings of the seemingly distinct paradigms, but also allows us to systematically apply solutions to problems in one paradigm to similar problems in another. Previous work along this line has made fruitful results in other domains. For example, an extended formulation of BIBREF6 that connects RL and posterior regularization (PR) BIBREF44, BIBREF45 has enabled to similarly export a reward learning algorithm to the context of PR for learning structured knowledge BIBREF46. By establishing a uniform abstration of GANs BIBREF47 and VAEs BIBREF33, BIBREF48 exchange techniques between the two families and get improved generative modeling. Other work in the similar spirit includes BIBREF49, BIBREF50, BIBREF51. By extrapolating algorithms between paradigms, one can go beyond crafting new algorithms from scratch as in most existing studies, which often requires deep expertise and yields unique solutions in a dedicated context. Instead, innovation becomes easier by importing rich ideas from other paradigms, and is repeatable as a new algorithm can be methodically extrapolated to multiple different contexts.
How much is classification performance improved in experiments for low data regime and class-imbalance problems?
3415762847ed13acc3c90de60e3ef42612bc49af
How Ġmuch Ġis Ġclassification Ġperformance Ġimproved Ġin Ġexperiments Ġfor Ġlow Ġdata Ġregime Ġand Ġclass - im balance Ġproblems ?
Low Ġdata : ĠS ST - 5 , ĠT REC , ĠIM DB Ġaround Ġ1 - 2 Ġaccuracy Ġpoints Ġbetter Ġthan Ġbaseline Ċ I mb al anced Ġlabels : Ġthe Ġimprovement Ġover Ġthe Ġbase Ġmodel Ġincreases Ġas Ġthe Ġdata Ġgets Ġmore Ġim balanced , Ġranging Ġfrom Ġaround Ġ6 Ġaccuracy Ġpoints Ġon Ġ100 : 1000 Ġto Ġover Ġ20 Ġaccuracy Ġpoints Ġon Ġ20 : 1000
<s> How Ġmuch Ġis Ġclassification Ġperformance Ġimproved Ġin Ġexperiments Ġfor Ġlow Ġdata Ġregime Ġand Ġclass - im balance Ġproblems ? </s> Introduction </s> The Ġperformance Ġof Ġmachines Ġoften Ġcru cially Ġdepend Ġon Ġthe Ġamount Ġand Ġquality Ġof Ġthe Ġdata Ġused Ġfor Ġtraining . ĠIt Ġhas Ġbecome Ġincreasingly Ġubiquitous Ġto Ġmanipulate Ġdata Ġto Ġimprove Ġlearning , Ġespecially Ġin Ġlow Ġdata Ġregime Ġor Ġin Ġpresence Ġof Ġlow - quality Ġdatasets Ġ( e . g ., Ġim balanced Ġlabels ). ĠFor Ġexample , Ġdata Ġaug mentation Ġapplies Ġlabel - pres erving Ġtransformations Ġon Ġoriginal Ġdata Ġpoints Ġto Ġexpand Ġthe Ġdata Ġsize ; Ġdata Ġweight ing Ġassigns Ġan Ġimportance Ġweight Ġto Ġeach Ġinstance Ġto Ġadapt Ġits Ġeffect Ġon Ġlearning ; Ġand Ġdata Ġsynthesis Ġgenerates Ġentire Ġartificial Ġexamples . ĠDifferent Ġtypes Ġof Ġmanipulation Ġcan Ġbe Ġsuitable Ġfor Ġdifferent Ġapplication Ġsettings . </s> Common Ġdata Ġmanipulation Ġmethods Ġare Ġusually Ġdesigned Ġmanually , Ġe . g ., Ġaugment ing Ġby Ġflipping Ġan Ġimage Ġor Ġreplacing Ġa Ġword Ġwith Ġsyn onyms , Ġand Ġweight ing Ġwith Ġinverse Ġclass Ġfrequency Ġor Ġloss Ġvalues ĠB IB REF 0 , ĠB IB REF 1 . ĠRecent Ġwork Ġhas Ġstudied Ġautomated Ġapproaches , Ġsuch Ġas Ġlearning Ġthe Ġcomposition Ġof Ġaug mentation Ġoperators Ġwith Ġreinforcement Ġlearning ĠB IB REF 2 , ĠB IB REF 3 , Ġder iving Ġsample Ġweights Ġadapt ively Ġfrom Ġa Ġvalidation Ġset Ġvia Ġmeta Ġlearning ĠB IB REF 4 , Ġor Ġlearning Ġa Ġweight ing Ġnetwork Ġby Ġinducing Ġa Ġcurriculum ĠB IB REF 5 . ĠThese Ġlearning - based Ġapproaches Ġhave Ġallev iated Ġthe Ġengineering Ġburden Ġand Ġproduced Ġimpressive Ġresults . ĠHowever , Ġthe Ġalgorithms Ġare Ġusually Ġdesigned Ġspecifically Ġfor Ġcertain Ġtypes Ġof Ġmanipulation Ġ( e . g ., Ġeither Ġaug mentation Ġor Ġweight ing ) Ġand Ġthus Ġhave Ġlimited Ġapplication Ġscope Ġin Ġpractice . </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġnew Ġapproach Ġthat Ġenables Ġlearning Ġfor Ġdifferent Ġmanipulation Ġschemes Ġwith Ġthe Ġsame Ġsingle Ġalgorithm . ĠOur Ġapproach Ġdraws Ġinspiration Ġfrom Ġthe Ġrecent Ġwork ĠB IB REF 6 Ġthat Ġshows Ġequival ence Ġbetween Ġthe Ġdata Ġin Ġsupervised Ġlearning Ġand Ġthe Ġreward Ġfunction Ġin Ġreinforcement Ġlearning . ĠWe Ġthus Ġadapt Ġan Ġoff - the - she lf Ġreward Ġlearning Ġalgorithm ĠB IB REF 7 Ġto Ġthe Ġsupervised Ġsetting Ġfor Ġautomated Ġdata Ġmanipulation . ĠThe Ġmarriage Ġof Ġthe Ġtwo Ġparad ig ms Ġresults Ġin Ġa Ġsimple Ġyet Ġgeneral Ġalgorithm , Ġwhere Ġvarious Ġmanipulation Ġschemes Ġare Ġreduced Ġto Ġdifferent Ġparameter ization Ġof Ġthe Ġdata Ġreward . ĠFree Ġparameters Ġof Ġmanipulation Ġare Ġlearned Ġjointly Ġwith Ġthe Ġtarget Ġmodel Ġthrough Ġefficient Ġgradient Ġdescent Ġon Ġvalidation Ġexamples . ĠWe Ġdemonstrate Ġinstant iations Ġof Ġthe Ġapproach Ġfor Ġautomatically Ġfine - tun ing Ġan Ġaug mentation Ġnetwork Ġand Ġlearning Ġdata Ġweights , Ġrespectively . </s> We Ġconduct Ġextensive Ġexperiments Ġon Ġtext Ġand Ġimage Ġclassification Ġin Ġchallenging Ġsituations Ġof Ġvery Ġlimited Ġdata Ġand Ġim balanced Ġlabels . ĠBoth Ġaug mentation Ġand Ġweight ing Ġby Ġour Ġapproach Ġsignificantly Ġimprove Ġover Ġstrong Ġbase Ġmodels , Ġeven Ġthough Ġthe Ġmodels Ġare Ġinitialized Ġwith Ġlarge - scale Ġpret rained Ġnetworks Ġsuch Ġas ĠB ERT ĠB IB REF 8 Ġfor Ġtext Ġand ĠRes Net ĠB IB REF 9 Ġfor Ġimages . ĠOur Ġapproach , Ġbesides Ġits Ġgener ality , Ġalso Ġoutper forms Ġa Ġvariety Ġof Ġdedicated Ġrule - Ġand Ġlearning - based Ġmethods Ġfor Ġeither Ġaug mentation Ġor Ġweight ing , Ġrespectively . ĠLastly , Ġwe Ġobserve Ġthat Ġthe Ġtwo Ġtypes Ġof Ġmanipulation Ġtend Ġto Ġexcel Ġin Ġdifferent Ġcontexts : Ġaug mentation Ġshows Ġsuperiority Ġover Ġweight ing Ġwith Ġa Ġsmall Ġamount Ġof Ġdata Ġavailable , Ġwhile Ġweight ing Ġis Ġbetter Ġat Ġaddressing Ġclass Ġimbalance Ġproblems . </s> The Ġway Ġwe Ġderive Ġthe Ġmanipulation Ġalgorithm Ġrepresents Ġa Ġgeneral Ġmeans Ġof Ġproblem Ġsolving Ġthrough Ġalgorithm Ġextrap olation Ġbetween Ġlearning Ġparad ig ms , Ġwhich Ġwe Ġdiscuss Ġmore Ġin Ġsection ĠSEC REF 6 . </s> Related ĠWork </s> Rich Ġtypes Ġof Ġdata Ġmanipulation Ġhave Ġbeen Ġincreasingly Ġused Ġin Ġmodern Ġmachine Ġlearning Ġpipelines . ĠPrevious Ġwork Ġeach Ġhas Ġtypically Ġfocused Ġon Ġa Ġparticular Ġmanipulation Ġtype . ĠData Ġaug mentation Ġthat Ġpert ur bs Ġexamples Ġwithout Ġchanging Ġthe Ġlabels Ġis Ġwidely Ġused Ġespecially Ġin Ġvision ĠB IB REF 10 , ĠB IB REF 11 Ġand Ġspeech ĠB IB REF 12 , ĠB IB REF 13 Ġdomains . ĠCommon Ġhe uristic - based Ġmethods Ġon Ġimages Ġinclude Ġcro pping , Ġmirror ing , Ġrotation ĠB IB REF 11 , Ġand Ġso Ġforth . ĠRecent Ġwork Ġhas Ġdeveloped Ġautomated Ġaug mentation Ġapproaches ĠB IB REF 3 , ĠB IB REF 2 , ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 . ĠB IB REF 17 Ġadditionally Ġuse Ġlarge - scale Ġunl abel ed Ġdata . ĠB IB REF 3 , ĠB IB REF 2 Ġlearn Ġto Ġinduce Ġthe Ġcomposition Ġof Ġdata Ġtransformation Ġoperators . ĠInstead Ġof Ġtreating Ġdata Ġaug mentation Ġas Ġa Ġpolicy Ġin Ġreinforcement Ġlearning ĠB IB REF 3 , Ġwe Ġformulate Ġmanipulation Ġas Ġa Ġreward Ġfunction Ġand Ġuse Ġefficient Ġsto ch astic Ġgradient Ġdescent Ġto Ġlearn Ġthe Ġmanipulation Ġparameters . ĠText Ġdata Ġaug mentation Ġhas Ġalso Ġachieved Ġimpressive Ġsuccess , Ġsuch Ġas Ġcontextual Ġaug mentation ĠB IB REF 18 , ĠB IB REF 19 , Ġback - translation ĠB IB REF 20 , Ġand Ġmanual Ġapproaches ĠB IB REF 21 , ĠB IB REF 22 . ĠIn Ġaddition Ġto Ġpert ur bing Ġthe Ġinput Ġtext Ġas Ġin Ġclassification Ġtasks , Ġtext Ġgeneration Ġproblems Ġexpose Ġopportunities Ġto Ġadding Ġnoise Ġalso Ġin Ġthe Ġoutput Ġtext , Ġsuch Ġas ĠB IB REF 23 , ĠB IB REF 24 . ĠRecent Ġwork ĠB IB REF 6 Ġshows Ġoutput Ġn osing Ġin Ġsequence Ġgeneration Ġcan Ġbe Ġtreated Ġas Ġan Ġintermediate Ġapproach Ġin Ġbetween Ġsupervised Ġlearning Ġand Ġreinforcement Ġlearning , Ġand Ġdeveloped Ġa Ġnew Ġsequence Ġlearning Ġalgorithm Ġthat Ġinterpol ates Ġbetween Ġthe Ġspectrum Ġof Ġexisting Ġalgorithms . ĠWe Ġinstant iate Ġour Ġapproach Ġfor Ġtext Ġcontextual Ġaug mentation Ġas Ġin ĠB IB REF 18 , ĠB IB REF 19 , Ġbut Ġenhance Ġthe Ġprevious Ġwork Ġby Ġadditionally Ġfine - tun ing Ġthe Ġaug mentation Ġnetwork Ġjointly Ġwith Ġthe Ġtarget Ġmodel . </s> Data Ġweight ing Ġhas Ġbeen Ġused Ġin Ġvarious Ġalgorithms , Ġsuch Ġas ĠAda Boost ĠB IB REF 0 , Ġself - paced Ġlearning ĠB IB REF 25 , Ġhard - example Ġmining ĠB IB REF 26 , Ġand Ġothers ĠB IB REF 27 , ĠB IB REF 28 . ĠThese Ġalgorithms Ġlargely Ġdefine Ġsample Ġweights Ġbased Ġon Ġtraining Ġloss . ĠRecent Ġwork ĠB IB REF 5 , ĠB IB REF 29 Ġlearns Ġa Ġseparate Ġnetwork Ġto Ġpredict Ġsample Ġweights . ĠOf Ġparticular Ġrelevance Ġto Ġour Ġwork Ġis ĠB IB REF 4 Ġwhich Ġinduces Ġsample Ġweights Ġusing Ġa Ġvalidation Ġset . ĠThe Ġdata Ġweight ing Ġmechanism Ġinstant iated Ġby Ġour Ġframework Ġhas Ġa Ġkey Ġdifference Ġin Ġthat Ġsamples Ġweights Ġare Ġtreated Ġas Ġparameters Ġthat Ġare Ġupdated Ġiter atively , Ġinstead Ġof Ġre - est imated Ġfrom Ġscratch Ġat Ġeach Ġstep . ĠWe Ġshow Ġimproved Ġperformance Ġof Ġour Ġapproach . ĠBesides , Ġour Ġdata Ġmanipulation Ġapproach Ġis Ġderived Ġbased Ġon Ġa Ġdifferent Ġperspective Ġof Ġreward Ġlearning , Ġinstead Ġof Ġmeta - learning Ġas Ġin ĠB IB REF 4 . </s> Another Ġpopular Ġtype Ġof Ġdata Ġmanipulation Ġinvolves Ġdata Ġsynthesis , Ġwhich Ġcreates Ġentire Ġartificial Ġsamples Ġfrom Ġscratch . ĠG AN - based Ġapproaches Ġhave Ġachieved Ġimpressive Ġresults Ġfor Ġsynthes izing Ġconditional Ġimage Ġdata ĠB IB REF 30 , ĠB IB REF 31 . ĠIn Ġthe Ġtext Ġdomain , Ġcontro ll able Ġtext Ġgeneration ĠB IB REF 32 Ġpresents Ġa Ġway Ġof Ġco - training Ġthe Ġdata Ġgenerator Ġand Ġclass ifier Ġin Ġa Ġcycl ic Ġmanner Ġwithin Ġa Ġjoint ĠVA E ĠB IB REF 33 Ġand Ġwake - sleep ĠB IB REF 34 Ġframework . ĠIt Ġis Ġinteresting Ġto Ġexplore Ġthe Ġinstant iation Ġof Ġthe Ġpresent Ġapproach Ġfor Ġadaptive Ġdata Ġsynthesis Ġin Ġthe Ġfuture . </s> Background </s> We Ġfirst Ġpresent Ġthe Ġrelevant Ġwork Ġupon Ġwhich Ġour Ġautomated Ġdata Ġmanipulation Ġis Ġbuilt . ĠThis Ġsection Ġalso Ġestablishes Ġthe Ġnot ations Ġused Ġthroughout Ġthe Ġpaper . </s> Let Ġ$\ math bf Ġ{ x } $ Ġdenote Ġthe Ġinput Ġand Ġ$ y $ Ġthe Ġoutput . ĠFor Ġexample , Ġin Ġtext Ġclassification , Ġ$\ math bf Ġ{ x } $ Ġcan Ġbe Ġa Ġsentence Ġand Ġ$ y $ Ġis Ġthe Ġsentence Ġlabel . ĠDen ote Ġthe Ġmodel Ġof Ġinterest Ġas Ġ$ p _ \ the ta Ġ( y | \ math bf Ġ{ x }) $, Ġwhere Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġis Ġthe Ġmodel Ġparameters Ġto Ġbe Ġlearned . ĠIn Ġsupervised Ġsetting , Ġgiven Ġa Ġset Ġof Ġtraining Ġexamples Ġ$\ math cal Ġ{ D } = \ l brace Ġ( \ math bf Ġ{ x } ^ *, Ġy ^ * )\ r brace Ġ$ , Ġwe Ġlearn Ġthe Ġmodel Ġby Ġmaximizing Ġthe Ġdata Ġlog - like lihood . </s> Background Ġ: :: ĠEqu ival ence Ġbetween ĠData Ġand ĠReward </s> The Ġrecent Ġwork ĠB IB REF 6 Ġintroduced Ġa Ġun ifying Ġperspective Ġof Ġreform ulating Ġmaximum Ġlikelihood Ġsupervised Ġlearning Ġas Ġa Ġspecial Ġinstance Ġof Ġa Ġpolicy Ġoptimization Ġframework . ĠIn Ġthis Ġperspective , Ġdata Ġexamples Ġproviding Ġsupervision Ġsignals Ġare Ġequivalent Ġto Ġa Ġspecialized Ġreward Ġfunction . ĠSince Ġthe Ġoriginal Ġframework ĠB IB REF 6 Ġwas Ġderived Ġfor Ġsequence Ġgeneration Ġproblems , Ġhere Ġwe Ġpresent Ġa Ġslightly Ġadapted Ġformulation Ġfor Ġour Ġcontext Ġof Ġdata Ġmanipulation . </s> To Ġconnect Ġthe Ġmaximum Ġlikelihood Ġsupervised Ġlearning Ġwith Ġpolicy Ġoptimization , Ġconsider Ġthe Ġmodel Ġ$ p _ \ the ta Ġ( y | \ math bf Ġ{ x }) $ Ġas Ġa Ġpolicy Ġthat Ġtakes ĠâĢ ľ action âĢ Ŀ Ġ$ y $ Ġgiven Ġthe ĠâĢ ľ state âĢ Ŀ Ġ$\ math bf Ġ{ x } $. ĠLet Ġ$ R (\ math bf Ġ{ x }, Ġy Ġ| \ math cal Ġ{ D } )\ in Ġ\ math bb Ġ{ R } $ Ġdenote Ġa Ġreward Ġfunction , Ġand Ġ$ p (\ math bf Ġ{ x }) $ Ġbe Ġthe Ġempirical Ġdata Ġdistribution Ġwhich Ġis Ġknown Ġgiven Ġ$\ math cal Ġ{ D } $. ĠFurther Ġassume Ġa Ġvari ational Ġdistribution Ġ$ q (\ math bf Ġ{ x }, Ġy ) $ Ġthat Ġfactor izes Ġas Ġ$ q (\ math bf Ġ{ x }, y )= p (\ math bf Ġ{ x }) q ( y | \ math bf Ġ{ x }) $. ĠA Ġvari ational Ġpolicy Ġoptimization Ġobjective Ġis Ġthen Ġwritten Ġas : </s> where Ġ$\ text { K L } (\ cd ot Ġ\ Vert Ġ\ cd ot Ġ) $ Ġis Ġthe ĠK ull back âĢĵ Le ib ler Ġdivergence ; Ġ$\ text { H } (\ cd ot Ġ) $ Ġis Ġthe ĠShannon Ġentropy ; Ġand Ġ$\ alpha Ġ, \ beta Ġ> 0 $ Ġare Ġbalancing Ġweights . ĠThe Ġobjective Ġis Ġin Ġthe Ġsame Ġform Ġwith Ġthe ĠRL - as - in ference Ġformal ism Ġof Ġpolicy Ġoptimization ĠB IB REF 35 , ĠB IB REF 36 , ĠB IB REF 37 . ĠInt uitive ly , Ġthe Ġobjective Ġmaxim izes Ġthe Ġexpected Ġreward Ġunder Ġ$ q $, Ġand Ġen forces Ġthe Ġmodel Ġ$ p _ \ the ta Ġ$ Ġto Ġstay Ġclose Ġto Ġ$ q $, Ġwith Ġa Ġmaximum Ġentropy Ġregular ization Ġover Ġ$ q $. ĠThe Ġproblem Ġis Ġsolved Ġwith Ġan ĠEM Ġprocedure Ġthat Ġoptim izes Ġ$ q $ Ġand Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġalternating ly : </s> where Ġ$ Z $ Ġis Ġthe Ġnormal ization Ġterm . ĠWith Ġthe Ġestablished Ġframework , Ġit Ġis Ġeasy Ġto Ġshow Ġthat Ġthe Ġabove Ġoptimization Ġprocedure Ġreduces Ġto Ġmaximum Ġlikelihood Ġlearning Ġby Ġtaking Ġ$\ alpha Ġ\ right arrow Ġ0 , Ġ\ beta Ġ= 1 $, Ġand Ġthe Ġreward Ġfunction : </s> That Ġis , Ġa Ġsample Ġ$( \ math bf Ġ{ x }, Ġy ) $ Ġreceives Ġa Ġunit Ġreward Ġonly Ġwhen Ġit Ġmatches Ġa Ġtraining Ġexample Ġin Ġthe Ġdataset , Ġwhile Ġthe Ġreward Ġis Ġnegative Ġinfinite Ġin Ġall Ġother Ġcases . ĠTo Ġmake Ġthe Ġequival ence Ġto Ġmaximum Ġlikelihood Ġlearning Ġclearer , Ġnote Ġthat Ġthe Ġabove ĠM - step Ġnow Ġreduces Ġto </s> where Ġthe Ġjoint Ġdistribution Ġ$ p (\ math bf Ġ{ x } )\ exp Ġ\ l brace ĠR _ \ d elta Ġ\ r brace Ġ/ Z $ Ġequals Ġthe Ġempirical Ġdata Ġdistribution , Ġwhich Ġmeans Ġthe ĠM - step Ġis Ġin Ġfact Ġmaximizing Ġthe Ġdata Ġlog - like lihood Ġof Ġthe Ġmodel Ġ$ p _ \ the ta Ġ$ . </s> Background Ġ: :: ĠGrad ient - based ĠReward ĠLearning </s> There Ġis Ġa Ġrich Ġline Ġof Ġresearch Ġon Ġlearning Ġthe Ġreward Ġin Ġreinforcement Ġlearning . ĠOf Ġparticular Ġinterest Ġto Ġthis Ġwork Ġis ĠB IB REF 7 Ġwhich Ġlearns Ġa Ġparam etric Ġintrinsic Ġreward Ġthat Ġadditive ly Ġtransforms Ġthe Ġoriginal Ġtask Ġreward Ġ( a . k . a Ġextr ins ic Ġreward ) Ġto Ġimprove Ġthe Ġpolicy Ġoptimization . ĠFor Ġconsistency Ġof Ġnot ations Ġwith Ġabove , Ġformally , Ġlet Ġ$ p _ \ the ta Ġ( y | \ math bf Ġ{ x }) $ Ġbe Ġa Ġpolicy Ġwhere Ġ$ y $ Ġis Ġan Ġaction Ġand Ġ$\ math bf Ġ{ x } $ Ġis Ġa Ġstate . ĠLet Ġ$ R _ \ phi Ġ^ { in } $ Ġbe Ġthe Ġintrinsic Ġreward Ġwith Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $. ĠIn Ġeach Ġiteration , Ġthe Ġpolicy Ġparameter Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġis Ġupdated Ġto Ġmaximize Ġthe Ġjoint Ġrewards , Ġthrough : </s> where Ġ$\ math cal Ġ{ L } ^{ ex + in } $ Ġis Ġthe Ġexpectation Ġof Ġthe Ġsum Ġof Ġextr ins ic Ġand Ġintrinsic Ġrewards ; Ġand Ġ$\ gam ma Ġ$ Ġis Ġthe Ġstep Ġsize . ĠThe Ġequation Ġshows Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġdepends Ġon Ġ$\ math bf Ġ{\ phi Ġ} $, Ġthus Ġwe Ġcan Ġwrite Ġas Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} = \ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} (\ math bf Ġ{\ phi Ġ}) $. </s> The Ġnext Ġstep Ġis Ġto Ġoptimize Ġthe Ġintrinsic Ġreward Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $. ĠRecall Ġthat Ġthe Ġultimate Ġmeasure Ġof Ġthe Ġperformance Ġof Ġa Ġpolicy Ġis Ġthe Ġvalue Ġof Ġextr ins ic Ġreward Ġit Ġachieves . ĠTherefore , Ġa Ġgood Ġintrinsic Ġreward Ġis Ġsupposed Ġto , Ġwhen Ġthe Ġpolicy Ġis Ġtrained Ġwith Ġit , Ġincrease Ġthe Ġeventual Ġextr ins ic Ġreward . ĠThe Ġupdate Ġto Ġ$\ math bf Ġ{\ phi Ġ} $ Ġis Ġthen Ġwritten Ġas : </s> That Ġis , Ġwe Ġwant Ġthe Ġexpected Ġextr ins ic Ġreward Ġ$\ math cal Ġ{ L } ^{ ex } (\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ}) $ Ġof Ġthe Ġnew Ġpolicy Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġto Ġbe Ġmaxim ized . ĠSince Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġis Ġa Ġfunction Ġof Ġ$\ math bf Ġ{\ phi Ġ} $, Ġwe Ġcan Ġdirectly Ġback prop agate Ġthe Ġgradient Ġthrough Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġto Ġ$\ math bf Ġ{\ phi Ġ} $. </s> Learning ĠData ĠManip ulation Ġ: :: ĠMethod Ġ: :: ĠParam eter izing ĠData ĠManip ulation </s> We Ġnow Ġdevelop Ġour Ġapproach Ġof Ġlearning Ġdata Ġmanipulation , Ġthrough Ġa Ġnovel Ġmarriage Ġof Ġsupervised Ġlearning Ġand Ġthe Ġabove Ġreward Ġlearning . ĠSpecifically , Ġfrom Ġthe Ġpolicy Ġoptimization Ġperspective , Ġdue Ġto Ġthe Ġ$\ d elta Ġ$ - function Ġreward Ġ( E q . DIS PLAY _ FORM 4 ), Ġthe Ġstandard Ġmaximum Ġlikelihood Ġlearning Ġis Ġrestricted Ġto Ġuse Ġonly Ġthe Ġexact Ġtraining Ġexamples Ġ$\ math cal Ġ{ D } $ Ġin Ġa Ġuniform Ġway . ĠA Ġnatural Ġidea Ġof Ġenabling Ġdata Ġmanipulation Ġis Ġto Ġrelax Ġthe Ġstrong Ġrestrictions Ġof Ġthe Ġ$\ d elta Ġ$ - function Ġreward Ġand Ġinstead Ġuse Ġa Ġrelaxed Ġreward Ġ$ R _ \ phi Ġ( \ math bf Ġ{ x }, Ġy Ġ| Ġ\ math cal Ġ{ D }) $ Ġwith Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $. ĠThe Ġrelaxed Ġreward Ġcan Ġbe Ġparameter ized Ġin Ġvarious Ġways , Ġresulting Ġin Ġdifferent Ġtypes Ġof Ġmanipulation . ĠFor Ġexample , Ġwhen Ġa Ġsample Ġ$( \ math bf Ġ{ x }, Ġy ) $ Ġmatches Ġa Ġdata Ġinstance , Ġinstead Ġof Ġreturning Ġconstant Ġ1 Ġby Ġ$ R _ \ d elta Ġ$ , Ġthe Ġnew Ġ$ R _ \ phi Ġ$ Ġcan Ġreturn Ġvarying Ġreward Ġvalues Ġdepending Ġon Ġthe Ġmatched Ġinstance , Ġresulting Ġin Ġa Ġdata Ġweight ing Ġscheme . ĠAlternatively , Ġ$ R _ \ phi Ġ$ Ġcan Ġreturn Ġa Ġvalid Ġreward Ġeven Ġwhen Ġ$\ math bf Ġ{ x } $ Ġmatches Ġa Ġdata Ġexample Ġonly Ġin Ġpart , Ġor Ġ$( \ math bf Ġ{ x }, Ġy ) $ Ġis Ġan Ġentire Ġnew Ġsample Ġnot Ġin Ġ$\ math cal Ġ{ D } $, Ġwhich Ġin Ġeffect Ġmakes Ġdata Ġaug mentation Ġand Ġdata Ġsynthesis , Ġrespectively , Ġin Ġwhich Ġcases Ġ$\ math bf Ġ{\ phi Ġ} $ Ġis Ġeither Ġa Ġdata Ġtransformer Ġor Ġa Ġgenerator . ĠIn Ġthe Ġnext Ġsection , Ġwe Ġdemonstrate Ġtwo Ġparticular Ġparameter izations Ġfor Ġdata Ġaug mentation Ġand Ġweight ing , Ġrespectively . </s> We Ġthus Ġhave Ġshown Ġthat Ġthe Ġdiverse Ġtypes Ġof Ġmanipulation Ġall Ġboil Ġdown Ġto Ġa Ġparameter ized Ġdata Ġreward Ġ$ R _ \ phi Ġ$ . ĠSuch Ġan Ġconcise , Ġuniform Ġformulation Ġof Ġdata Ġmanipulation Ġhas Ġthe Ġadvantage Ġthat , Ġonce Ġwe Ġdevise Ġa Ġmethod Ġof Ġlearning Ġthe Ġmanipulation Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $, Ġthe Ġresulting Ġalgorithm Ġcan Ġdirectly Ġbe Ġapplied Ġto Ġautomate Ġany Ġmanipulation Ġtype . ĠWe Ġpresent Ġa Ġlearning Ġalgorithm Ġnext . </s> Learning ĠData ĠManip ulation Ġ: :: ĠMethod Ġ: :: ĠLearning ĠManip ulation ĠParameters </s> To Ġlearn Ġthe Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $ Ġin Ġthe Ġmanipulation Ġreward Ġ$ R _ \ phi Ġ( \ math bf Ġ{ x }, Ġy Ġ| Ġ\ math cal Ġ{ D }) $, Ġwe Ġcould Ġin Ġprinciple Ġadopt Ġany Ġoff - the - she lf Ġreward Ġlearning Ġalgorithm Ġin Ġthe Ġliterature . ĠIn Ġthis Ġwork , Ġwe Ġdraw Ġinspiration Ġfrom Ġthe Ġabove Ġgradient - based Ġreward Ġlearning Ġ( section ĠSEC REF 3 ) Ġdue Ġto Ġits Ġsimplicity Ġand Ġefficiency . ĠBrief ly , Ġthe Ġobjective Ġof Ġ$\ math bf Ġ{\ phi Ġ} $ Ġis Ġto Ġmaximize Ġthe Ġultimate Ġmeasure Ġof Ġthe Ġperformance Ġof Ġmodel Ġ$ p _ \ the ta Ġ( \ math bf Ġ{ y } | \ math bf Ġ{ x }) $, Ġwhich , Ġin Ġthe Ġcontext Ġof Ġsupervised Ġlearning , Ġis Ġthe Ġmodel Ġperformance Ġon Ġa Ġheld - out Ġvalidation Ġset . </s> The Ġalgorithm Ġoptim izes Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġand Ġ$\ math bf Ġ{\ phi Ġ} $ Ġalternating ly , Ġcorresponding Ġto ĠE q .( DIS PLAY _ FORM 7 ) Ġand ĠE q .( DIS PLAY _ FORM 8 ), Ġrespectively . ĠMore Ġconcrete ly , Ġin Ġeach Ġiteration , Ġwe Ġfirst Ġupdate Ġthe Ġmodel Ġparameters Ġ$\ math bf Ġ{\ the ta Ġ} $ Ġin Ġanalogue Ġto ĠE q .( DIS PLAY _ FORM 7 ) Ġwhich Ġoptim izes Ġintrinsic Ġreward - en riched Ġobjective . ĠHere , Ġwe Ġoptimize Ġthe Ġlog - like lihood Ġof Ġthe Ġtraining Ġset Ġenriched Ġwith Ġdata Ġmanipulation . ĠThat Ġis , Ġwe Ġreplace Ġ$ R _ \ d elta Ġ$ Ġwith Ġ$ R _ \ phi Ġ$ Ġin ĠE q .( DIS PLAY _ FORM 5 ), Ġand Ġobtain Ġthe Ġaugmented ĠM - step : </s> By Ġnoticing Ġthat Ġthe Ġnew Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġdepends Ġon Ġ$\ math bf Ġ{\ phi Ġ} $, Ġwe Ġcan Ġwrite Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġas Ġa Ġfunction Ġof Ġ$\ math bf Ġ{\ phi Ġ} $, Ġnamely , Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} = \ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} (\ math bf Ġ{\ phi Ġ}) $. ĠThe Ġpractical Ġimplementation Ġof Ġthe Ġabove Ġupdate Ġdepends Ġon Ġthe Ġactual Ġparameter ization Ġof Ġmanipulation Ġ$ R _ \ phi Ġ$ , Ġwhich Ġwe Ġdiscuss Ġin Ġmore Ġdetails Ġin Ġthe Ġnext Ġsection . </s> The Ġnext Ġstep Ġis Ġto Ġoptimize Ġ$\ math bf Ġ{\ phi Ġ} $ Ġin Ġterms Ġof Ġthe Ġmodel Ġvalidation Ġperformance , Ġin Ġanalogue Ġto ĠE q .( DIS PLAY _ FORM 8 ). ĠForm ally , Ġlet Ġ$\ math cal Ġ{ D } ^ v $ Ġbe Ġthe Ġvalidation Ġset Ġof Ġdata Ġexamples . ĠThe Ġupdate Ġis Ġthen : </s> where , Ġsince Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġis Ġa Ġfunction Ġof Ġ$\ math bf Ġ{\ phi Ġ} $, Ġthe Ġgradient Ġis Ġback prop ag ated Ġto Ġ$\ math bf Ġ{\ phi Ġ} $ Ġthrough Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} (\ math bf Ġ{\ phi Ġ}) $. ĠTaking Ġdata Ġweight ing Ġfor Ġexample Ġwhere Ġ$\ math bf Ġ{\ phi Ġ} $ Ġis Ġthe Ġtraining Ġsample Ġweights Ġ( more Ġdetails Ġin Ġsection ĠSEC REF 15 ), Ġthe Ġupdate Ġis Ġto Ġoptimize Ġthe Ġweights Ġof Ġtraining Ġsamples Ġso Ġthat Ġthe Ġmodel Ġperforms Ġbest Ġon Ġthe Ġvalidation Ġset . </s> The Ġresulting Ġalgorithm Ġis Ġsummarized Ġin ĠAl gorithm ĠFIG REF 11 . ĠFigure ĠFIG REF 11 Ġillustrates Ġthe Ġcomputation Ġflow . ĠLearning Ġthe Ġmanipulation Ġparameters Ġeffectively Ġuses Ġa Ġheld - out Ġvalidation Ġset . ĠWe Ġshow Ġin Ġour Ġexperiments Ġthat Ġa Ġvery Ġsmall Ġset Ġof Ġvalidation Ġexamples Ġ( e . g ., Ġ2 Ġlabels Ġper Ġclass ) Ġis Ġenough Ġto Ġsignificantly Ġimprove Ġthe Ġmodel Ġperformance Ġin Ġlow Ġdata Ġregime . </s> It Ġis Ġworth Ġnoting Ġthat Ġsome Ġprevious Ġwork Ġhas Ġalso Ġlever aged Ġvalidation Ġexamples , Ġsuch Ġas Ġlearning Ġdata Ġaug mentation Ġwith Ġpolicy Ġgradient ĠB IB REF 3 Ġor Ġinducing Ġdata Ġweights Ġwith Ġmeta - learning ĠB IB REF 4 . ĠOur Ġapproach Ġis Ġinspired Ġfrom Ġa Ġdistinct Ġparadigm Ġof Ġ( int r ins ic ) Ġreward Ġlearning . ĠIn Ġcontrast Ġto ĠB IB REF 3 Ġthat Ġtreats Ġdata Ġaug mentation Ġas Ġa Ġpolicy , Ġwe Ġinstead Ġformulate Ġmanipulation Ġas Ġa Ġreward Ġfunction Ġand Ġenable Ġefficient Ġsto ch astic Ġgradient Ġupdates . ĠOur Ġapproach Ġis Ġalso Ġmore Ġbroadly Ġapplicable Ġto Ġdiverse Ġdata Ġmanipulation Ġtypes Ġthan ĠB IB REF 4 , ĠB IB REF 3 . </s> Learning ĠData ĠManip ulation Ġ: :: ĠInst anti ations : ĠAug mentation Ġ& ĠWeight ing </s> As Ġa Ġcase Ġstudy , Ġwe Ġshow Ġtwo Ġparameter izations Ġof Ġ$ R _ \ phi Ġ$ Ġwhich Ġinstant iate Ġdistinct Ġdata Ġmanipulation Ġschemes . ĠThe Ġfirst Ġexample Ġlearns Ġaug mentation Ġfor Ġtext Ġdata , Ġa Ġdomain Ġthat Ġhas Ġbeen Ġless Ġstudied Ġin Ġthe Ġliterature Ġcompared Ġto Ġvision Ġand Ġspeech ĠB IB REF 18 , ĠB IB REF 38 . ĠThe Ġsecond Ġinstance Ġfocuses Ġon Ġautomated Ġdata Ġweight ing , Ġwhich Ġis Ġapplicable Ġto Ġany Ġdata Ġdomains . </s> Learning ĠData ĠManip ulation Ġ: :: ĠInst anti ations : ĠAug mentation Ġ& ĠWeight ing Ġ: :: ĠFine - tun ing ĠText ĠAug mentation </s> The Ġrecent Ġwork ĠB IB REF 18 , ĠB IB REF 19 Ġdeveloped Ġa Ġnovel Ġcontextual Ġaug mentation Ġapproach Ġfor Ġtext Ġdata , Ġin Ġwhich Ġa Ġpowerful Ġpret rained Ġlanguage Ġmodel Ġ( LM ), Ġsuch Ġas ĠB ERT ĠB IB REF 8 , Ġis Ġused Ġto Ġgenerate Ġsubstit utions Ġof Ġwords Ġin Ġa Ġsentence . ĠSpecifically , Ġgiven Ġan Ġobserved Ġsentence Ġ$\ math bf Ġ{ x } ^ * $, Ġthe Ġmethod Ġfirst Ġrandomly Ġmasks Ġout Ġa Ġfew Ġwords . ĠThe Ġmasked Ġsentence Ġis Ġthen Ġfed Ġto ĠB ERT Ġwhich Ġfills Ġthe Ġmasked Ġpositions Ġwith Ġnew Ġwords . ĠTo Ġpreserve Ġthe Ġoriginal Ġsentence Ġclass , Ġthe ĠB ERT ĠLM Ġis Ġretro fitted Ġas Ġa Ġlabel - cond itional Ġmodel , Ġand Ġtrained Ġon Ġthe Ġtask Ġtraining Ġexamples . ĠThe Ġresulting Ġmodel Ġis Ġthen Ġfixed Ġand Ġused Ġto Ġaugment Ġdata Ġduring Ġthe Ġtraining Ġof Ġtarget Ġmodel . ĠWe Ġdenote Ġthe Ġaug mentation Ġdistribution Ġas Ġ$ g _{ \ phi Ġ_ 0 } (\ math bf Ġ{ x } | \ math bf Ġ{ x } ^ *, Ġ\ math bf Ġ{ y } ^ *) $, Ġwhere Ġ$\ math bf Ġ{\ phi Ġ} _ 0 $ Ġis Ġthe Ġfixed ĠB ERT ĠLM Ġparameters . </s> The Ġabove Ġprocess Ġhas Ġtwo Ġdrawbacks . ĠFirst , Ġthe ĠLM Ġis Ġfixed Ġafter Ġfitting Ġto Ġthe Ġtask Ġdata . ĠIn Ġthe Ġsubsequent Ġphase Ġof Ġtraining Ġthe Ġtarget Ġmodel , Ġthe ĠLM Ġaug ments Ġdata Ġwithout Ġknowing Ġthe Ġstate Ġof Ġthe Ġtarget Ġmodel , Ġwhich Ġcan Ġlead Ġto Ġsub - opt imal Ġresults . ĠSecond , Ġin Ġthe Ġcases Ġwhere Ġthe Ġtask Ġdataset Ġis Ġsmall , Ġthe ĠLM Ġcan Ġbe Ġinsufficient ly Ġtrained Ġfor Ġpreserving Ġthe Ġlabels Ġfaithfully , Ġresulting Ġin Ġnoisy Ġaugmented Ġsamples . </s> To Ġaddress Ġthe Ġdifficulties , Ġit Ġis Ġbeneficial Ġto Ġapply Ġthe Ġproposed Ġlearning Ġdata Ġmanipulation Ġalgorithm Ġto Ġadditionally Ġfine - t une Ġthe ĠLM Ġjointly Ġwith Ġtarget Ġmodel Ġtraining . ĠAs Ġdiscussed Ġin Ġsection ĠSEC REF 4 , Ġthis Ġreduces Ġto Ġproperly Ġparameter izing Ġthe Ġdata Ġreward Ġfunction : </s> That Ġis , Ġa Ġsample Ġ$( \ math bf Ġ{ x }, Ġy ) $ Ġreceives Ġa Ġunit Ġreward Ġwhen Ġ$ y $ Ġis Ġthe Ġtrue Ġlabel Ġand Ġ$\ math bf Ġ{ x } $ Ġis Ġthe Ġaugmented Ġsample Ġby Ġthe ĠLM Ġ( instead Ġof Ġthe Ġexact Ġoriginal Ġdata Ġ$\ math bf Ġ{ x } ^ * $ ). ĠPlug ging Ġthe Ġreward Ġinto ĠE q .( DIS PLAY _ FORM 13 ), Ġwe Ġobtain Ġthe Ġdata - au gment ed Ġupdate Ġfor Ġthe Ġmodel Ġparameters : </s> That Ġis , Ġwe Ġpick Ġan Ġexample Ġfrom Ġthe Ġtraining Ġset , Ġand Ġuse Ġthe ĠLM Ġto Ġcreate Ġaugmented Ġsamples , Ġwhich Ġare Ġthen Ġused Ġto Ġupdate Ġthe Ġtarget Ġmodel . ĠRegarding Ġthe Ġupdate Ġof Ġaug mentation Ġparameters Ġ$\ math bf Ġ{\ phi Ġ} $ Ġ( E q . DIS PLAY _ FORM 14 ), Ġsince Ġtext Ġsamples Ġare Ġdiscrete , Ġto Ġenable Ġefficient Ġgradient Ġpropagation Ġthrough Ġ$\ math bf Ġ{\ the ta Ġ} ^ {\ prime Ġ} $ Ġto Ġ$\ math bf Ġ{\ phi Ġ} $, Ġwe Ġuse Ġa Ġg umb el - soft max Ġapproximation ĠB IB REF 39 Ġto Ġ$\ math bf Ġ{ x } $ Ġwhen Ġsampling Ġsubstitution Ġwords Ġfrom Ġthe ĠLM . </s> Learning ĠData ĠManip ulation Ġ: :: ĠInst anti ations : ĠAug mentation Ġ& ĠWeight ing Ġ: :: ĠLearning ĠData ĠWe ights </s> We Ġnow Ġdemonstrate Ġthe Ġinstant iation Ġof Ġdata Ġweight ing . ĠWe Ġaim Ġto Ġassign Ġan Ġimportance Ġweight Ġto Ġeach Ġtraining Ġexample Ġto Ġadapt Ġits Ġeffect Ġon Ġmodel Ġtraining . ĠWe Ġautomate Ġthe Ġprocess Ġby Ġlearning Ġthe Ġdata Ġweights . ĠThis Ġis Ġachieved Ġby Ġparameter izing Ġ$ R _ \ phi Ġ$ Ġas : </s> where Ġ$\ phi Ġ_ i \ in Ġ\ math bb Ġ{ R } $ Ġis Ġthe Ġweight Ġassociated Ġwith Ġthe Ġ$ i $ th Ġexample . ĠPlug ging Ġ$ R ^{ w } _ \ phi Ġ$ Ġinto ĠE q .( DIS PLAY _ FORM 13 ), Ġwe Ġobtain Ġthe Ġweighted Ġupdate Ġfor Ġthe Ġmodel Ġ$\ math bf Ġ{\ the ta Ġ} $ : </s> In Ġpractice , Ġwhen Ġmin ib atch Ġsto ch astic Ġoptimization Ġis Ġused , Ġwe Ġapproximate Ġthe Ġweighted Ġsampling Ġby Ġtaking Ġthe Ġsoft max Ġover Ġthe Ġweights Ġof Ġonly Ġthe Ġmin ib atch Ġexamples . ĠThe Ġdata Ġweights Ġ$\ math bf Ġ{\ phi Ġ} $ Ġare Ġupdated Ġwith ĠE q .( DIS PLAY _ FORM 14 ). ĠIt Ġis Ġworth Ġnoting Ġthat Ġthe Ġprevious Ġwork ĠB IB REF 4 Ġsimilarly Ġderives Ġdata Ġweights Ġbased Ġon Ġtheir Ġgradient Ġdirections Ġon Ġa Ġvalidation Ġset . ĠOur Ġalgorithm Ġdiffers Ġin Ġthat Ġthe Ġdata Ġweights Ġare Ġparameters Ġmaintained Ġand Ġupdated Ġthroughout Ġthe Ġtraining , Ġinstead Ġof Ġre - est imated Ġfrom Ġscratch Ġin Ġeach Ġiteration . ĠExper iments Ġshow Ġthe Ġparam etric Ġtreatment Ġachieves Ġsuperior Ġperformance Ġin Ġvarious Ġsettings . ĠThere Ġare Ġalternative Ġparameter izations Ġof Ġ$ R _ \ phi Ġ$ Ġother Ġthan ĠE q .( DIS PLAY _ FORM 20 ). ĠFor Ġexample , Ġreplacing Ġ$\ phi Ġ_ i $ Ġin ĠE q .( DIS PLAY _ FORM 20 ) Ġwith Ġ$\ log Ġ\ phi Ġ_ i $ Ġin Ġeffect Ġchanges Ġthe Ġsoft max Ġnormal ization Ġin ĠE q .( DIS PLAY _ FORM 21 ) Ġto Ġlinear Ġnormal ization , Ġwhich Ġis Ġused Ġin ĠB IB REF 4 . </s> Exper iments </s> We Ġempir ically Ġvalidate Ġthe Ġproposed Ġdata Ġmanipulation Ġapproach Ġthrough Ġextensive Ġexperiments Ġon Ġlearning Ġaug mentation Ġand Ġweight ing . ĠWe Ġstudy Ġboth Ġtext Ġand Ġimage Ġclassification , Ġin Ġtwo Ġdifficult Ġsettings Ġof Ġlow Ġdata Ġregime Ġand Ġim balanced Ġlabels . </s> Exper iments Ġ: :: ĠExperimental ĠSetup </s> Base ĠModels . ĠWe Ġchoose Ġstrong Ġpret rained Ġnetworks Ġas Ġour Ġbase Ġmodels Ġfor Ġboth Ġtext Ġand Ġimage Ġclassification . ĠSpecifically , Ġon Ġtext Ġdata , Ġwe Ġuse Ġthe ĠB ERT Ġ( base , Ġunc ased ) Ġmodel ĠB IB REF 8 ; Ġwhile Ġon Ġimage Ġdata , Ġwe Ġuse ĠRes Net - 34 ĠB IB REF 9 Ġpret rained Ġon ĠImage Net . ĠWe Ġshow Ġthat , Ġeven Ġwith Ġthe Ġlarge - scale Ġpret raining , Ġdata Ġmanipulation Ġcan Ġstill Ġbe Ġvery Ġhelpful Ġto Ġboost Ġthe Ġmodel Ġperformance Ġon Ġdownstream Ġtasks . ĠSince Ġour Ġapproach Ġuses Ġvalidation Ġsets Ġfor Ġmanipulation Ġparameter Ġlearning , Ġfor Ġa Ġfair Ġcomparison Ġwith Ġthe Ġbase Ġmodel , Ġwe Ġtrain Ġthe Ġbase Ġmodel Ġin Ġtwo Ġways . ĠThe Ġfirst Ġis Ġto Ġtrain Ġthe Ġmodel Ġon Ġthe Ġtraining Ġsets Ġas Ġusual Ġand Ġselect Ġthe Ġbest Ġstep Ġusing Ġthe Ġvalidation Ġsets ; Ġthe Ġsecond Ġis Ġto Ġtrain Ġon Ġthe Ġmerged Ġtraining Ġand Ġvalidation Ġsets Ġfor Ġa Ġfixed Ġnumber Ġof Ġsteps . ĠThe Ġstep Ġnumber Ġis Ġset Ġto Ġthe Ġaverage Ġnumber Ġof Ġsteps Ġselected Ġin Ġthe Ġfirst Ġmethod . ĠWe Ġreport Ġthe Ġresults Ġof Ġboth Ġmethods . </s> Compar ison ĠMethods . ĠWe Ġcompare Ġour Ġapproach Ġwith Ġa Ġvariety Ġof Ġprevious Ġmethods Ġthat Ġwere Ġdesigned Ġfor Ġspecific Ġmanipulation Ġschemes : Ġ( 1 ) ĠFor Ġtext Ġdata Ġaug mentation , Ġwe Ġcompare Ġwith Ġthe Ġlatest Ġmodel - based Ġaug mentation ĠB IB REF 19 Ġwhich Ġuses Ġa Ġfixed Ġconditional ĠB ERT Ġlanguage Ġmodel Ġfor Ġword Ġsubstitution Ġ( section ĠSEC REF 15 ). ĠAs Ġwith Ġbase Ġmodels , Ġwe Ġalso Ġtried Ġfitting Ġthe Ġaugment atin Ġmodel Ġto Ġboth Ġthe Ġtraining Ġdata Ġand Ġthe Ġjoint Ġtraining - valid ation Ġdata , Ġand Ġdid Ġnot Ġobserve Ġsignificant Ġdifference . ĠFollowing ĠB IB REF 19 , Ġwe Ġalso Ġstudy Ġa Ġconventional Ġapproach Ġthat Ġreplaces Ġwords Ġwith Ġtheir Ġsyn onyms Ġusing ĠWord Net ĠB IB REF 40 . Ġ( 2 ) ĠFor Ġdata Ġweight ing , Ġwe Ġcompare Ġwith Ġthe Ġstate - of - the - art Ġapproach ĠB IB REF 4 Ġthat Ġdynamically Ġre - est imates Ġsample Ġweights Ġin Ġeach Ġiteration Ġbased Ġon Ġthe Ġvalidation Ġset Ġgradient Ġdirections . ĠWe Ġfollow ĠB IB REF 4 Ġand Ġalso Ġevaluate Ġthe Ġcommonly - used Ġproportion Ġmethod Ġthat Ġweights Ġdata Ġby Ġinverse Ġclass Ġfrequency . </s> Training . ĠFor Ġboth Ġthe ĠB ERT Ġclass ifier Ġand Ġthe Ġaug mentation Ġmodel Ġ( which Ġis Ġalso Ġbased Ġon ĠB ERT ), Ġwe Ġuse ĠAdam Ġoptimization Ġwith Ġan Ġinitial Ġlearning Ġrate Ġof Ġ4 e - 5 . ĠFor ĠRes N ets , Ġwe Ġuse ĠSG D Ġoptimization Ġwith Ġa Ġlearning Ġrate Ġof Ġ1 e - 3 . ĠFor Ġtext Ġdata Ġaug mentation , Ġwe Ġaugment Ġeach Ġmin ib atch Ġby Ġgenerating Ġtwo Ġor Ġthree Ġsamples Ġfor Ġeach Ġdata Ġpoints Ġ( each Ġwith Ġ1 , Ġ2 Ġor Ġ3 Ġsubstit utions ), Ġand Ġuse Ġboth Ġthe Ġsamples Ġand Ġthe Ġoriginal Ġdata Ġto Ġtrain Ġthe Ġmodel . ĠFor Ġdata Ġweight ing , Ġto Ġavoid Ġexploding Ġvalue , Ġwe Ġupdate Ġthe Ġweight Ġof Ġeach Ġdata Ġpoint Ġin Ġa Ġmin ib atch Ġby Ġdecaying Ġthe Ġprevious Ġweight Ġvalue Ġwith Ġa Ġfactor Ġof Ġ0 . 1 Ġand Ġthen Ġadding Ġthe Ġgradient . ĠAll Ġexperiments Ġwere Ġimplemented Ġwith ĠPy Tor ch Ġ( py tor ch . org ) Ġand Ġwere Ġperformed Ġon Ġa ĠLinux Ġmachine Ġwith Ġ4 ĠGTX Ġ1080 Ti ĠGPUs Ġand Ġ64 GB ĠRAM . ĠAll Ġreported Ġresults Ġare Ġaveraged Ġover Ġ15 Ġruns Ġ$\ pm Ġ$ Ġone Ġstandard Ġdeviation . </s> Exper iments Ġ: :: ĠLow ĠData ĠReg ime </s> We Ġstudy Ġthe Ġproblem Ġwhere Ġonly Ġvery Ġfew Ġlabeled Ġexamples Ġfor Ġeach Ġclass Ġare Ġavailable . ĠBoth Ġof Ġour Ġaug mentation Ġand Ġweight ing Ġboost Ġbase Ġmodel Ġperformance , Ġand Ġare Ġsuperior Ġto Ġrespective Ġcomparison Ġmethods . ĠWe Ġalso Ġobserve Ġthat Ġaug mentation Ġperforms Ġbetter Ġthan Ġweight ing Ġin Ġthe Ġlow - data Ġsetting . </s> Exper iments Ġ: :: ĠLow ĠData ĠReg ime Ġ: :: ĠSetup </s> For Ġtext Ġclassification , Ġwe Ġuse Ġthe Ġpopular Ġbenchmark Ġdatasets , Ġincluding ĠS ST - 5 Ġfor Ġ5 - class Ġsentence Ġsentiment ĠB IB REF 41 , ĠIM DB Ġfor Ġbinary Ġmovie Ġreview Ġsentiment ĠB IB REF 42 , Ġand ĠT REC Ġfor Ġ6 - class Ġquestion Ġtypes ĠB IB REF 43 . ĠWe Ġsubs ample Ġa Ġsmall Ġtraining Ġset Ġon Ġeach Ġtask Ġby Ġrandomly Ġpicking Ġ40 Ġinstances Ġfor Ġeach Ġclass . ĠWe Ġfurther Ġcreate Ġsmall Ġvalidation Ġsets , Ġi . e ., Ġ2 Ġinstances Ġper Ġclass Ġfor ĠS ST - 5 , Ġand Ġ5 Ġinstances Ġper Ġclass Ġfor ĠIM DB Ġand ĠT REC , Ġrespectively . ĠThe Ġreason Ġwe Ġuse Ġslightly Ġmore Ġvalidation Ġexamples Ġon ĠIM DB Ġand ĠT REC Ġis Ġthat Ġthe Ġmodel Ġcan Ġeasily Ġachieve Ġ100 % Ġvalidation Ġaccuracy Ġif Ġthe Ġvalidation Ġsets Ġare Ġtoo Ġsmall . ĠThus , Ġthe ĠS ST - 5 Ġtask Ġhas Ġ210 Ġlabeled Ġexamples Ġin Ġtotal , Ġwhile ĠIM DB Ġhas Ġ90 Ġlabels Ġand ĠT REC Ġhas Ġ270 . ĠSuch Ġextremely Ġsmall Ġdatasets Ġpose Ġsignificant Ġchallenges Ġfor Ġlearning Ġdeep Ġneural Ġnetworks . ĠSince Ġthe Ġmanipulation Ġparameters Ġare Ġtrained Ġusing Ġthe Ġsmall Ġvalidation Ġsets , Ġto Ġavoid Ġpossible Ġover fitting Ġwe Ġrestrict Ġthe Ġtraining Ġto Ġsmall Ġnumber Ġ( e . g ., Ġ5 Ġor Ġ10 ) Ġof Ġepoch s . ĠFor Ġimage Ġclassification , Ġwe Ġsimilarly Ġcreate Ġa Ġsmall Ġsubset Ġof Ġthe ĠC IF AR 10 Ġdata , Ġwhich Ġincludes Ġ40 Ġinstances Ġper Ġclass Ġfor Ġtraining , Ġand Ġ2 Ġinstances Ġper Ġclass Ġfor Ġvalidation . </s> Exper iments Ġ: :: ĠLow ĠData ĠReg ime Ġ: :: ĠResults </s> Table ĠT AB REF 26 Ġshows Ġthe Ġmanipulation Ġresults Ġon Ġtext Ġclassification . ĠFor Ġdata Ġaug mentation , Ġour Ġapproach Ġsignificantly Ġimproves Ġover Ġthe Ġbase Ġmodel Ġon Ġall Ġthe Ġthree Ġdatasets . ĠBesides , Ġcompared Ġto Ġboth Ġthe Ġconventional Ġsyn onym Ġsubstitution Ġand Ġthe Ġapproach Ġthat Ġkeeps Ġthe Ġaug mentation Ġnetwork Ġfixed , Ġour Ġadaptive Ġmethod Ġthat Ġfine - t unes Ġthe Ġaug mentation Ġnetwork Ġjointly Ġwith Ġmodel Ġtraining Ġachieves Ġsuperior Ġresults . ĠIndeed , Ġthe Ġhe uristic - based Ġsyn onym Ġapproach Ġcan Ġsometimes Ġharm Ġthe Ġmodel Ġperformance Ġ( e . g ., ĠS ST - 5 Ġand ĠIM DB ), Ġas Ġalso Ġobserved Ġin Ġprevious Ġwork ĠB IB REF 19 , ĠB IB REF 18 . ĠThis Ġcan Ġbe Ġbecause Ġthe Ġhe uristic Ġrules Ġdo Ġnot Ġfit Ġthe Ġtask Ġor Ġdatasets Ġwell . ĠIn Ġcontrast , Ġlearning - based Ġaug mentation Ġhas Ġthe Ġadvantage Ġof Ġadapt ively Ġgenerating Ġuseful Ġsamples Ġto Ġimprove Ġmodel Ġtraining . </s> Table ĠT AB REF 26 Ġalso Ġshows Ġthe Ġdata Ġweight ing Ġresults . ĠOur Ġweight Ġlearning Ġconsistently Ġimproves Ġover Ġthe Ġbase Ġmodel Ġand Ġthe Ġlatest Ġweight ing Ġmethod ĠB IB REF 4 . ĠIn Ġparticular , Ġinstead Ġof Ġre - est imating Ġsample Ġweights Ġfrom Ġscratch Ġin Ġeach Ġiteration ĠB IB REF 4 , Ġour Ġapproach Ġtreats Ġthe Ġweights Ġas Ġmanipulation Ġparameters Ġmaintained Ġthroughout Ġthe Ġtraining . ĠWe Ġspeculate Ġthat Ġthe Ġparam etric Ġtreatment Ġcan Ġadapt Ġweights Ġmore Ġsmoothly Ġand Ġprovide Ġhistorical Ġinformation , Ġwhich Ġis Ġbeneficial Ġin Ġthe Ġsmall - data Ġcontext . </s> It Ġis Ġinteresting Ġto Ġsee Ġfrom ĠTable ĠT AB REF 26 Ġthat Ġour Ġaug mentation Ġmethod Ġconsistently Ġoutper forms Ġthe Ġweight ing Ġmethod , Ġshowing Ġthat Ġdata Ġaug mentation Ġcan Ġbe Ġa Ġmore Ġsuitable Ġtechnique Ġthan Ġdata Ġweight ing Ġfor Ġmanipulating Ġsmall - size Ġdata . ĠOur Ġapproach Ġprovides Ġthe Ġgener ality Ġto Ġinstant iate Ġdiverse Ġmanipulation Ġtypes Ġand Ġlearn Ġwith Ġthe Ġsame Ġsingle Ġprocedure . </s> To Ġinvestigate Ġthe Ġaug mentation Ġmodel Ġand Ġhow Ġthe Ġfine - tun ing Ġaffects Ġthe Ġaug mentation Ġresults , Ġwe Ġshow Ġin ĠFigure ĠT AB REF 27 Ġthe Ġtop - 5 Ġmost Ġprobable Ġword Ġsubstit utions Ġpredicted Ġby Ġthe Ġaug mentation Ġmodel Ġfor Ġtwo Ġmasked Ġtokens , Ġrespectively . ĠComp aring Ġthe Ġresults Ġof Ġepoch Ġ1 Ġand Ġepoch Ġ3 , Ġwe Ġcan Ġsee Ġthe Ġaug mentation Ġmodel Ġevolves Ġand Ġdynamically Ġadjusts Ġthe Ġaug mentation Ġbehavior Ġas Ġthe Ġtraining Ġproceeds . ĠThrough Ġfine - tun ing , Ġthe Ġmodel Ġseems Ġto Ġmake Ġsubstit utions Ġthat Ġare Ġmore Ġcoherent Ġwith Ġthe Ġconditioning Ġlabel Ġand Ġrelevant Ġto Ġthe Ġoriginal Ġwords Ġ( e . g ., Ġreplacing Ġthe Ġword ĠâĢ ľ stri king âĢ Ŀ Ġwith ĠâĢ ľ b land âĢ Ŀ Ġin Ġepoch Ġ1 Ġv . s . ĠâĢ ľ ch arming âĢ Ŀ Ġin Ġepoch Ġ3 ). </s> Table ĠT AB REF 27 Ġshows Ġthe Ġdata Ġweight ing Ġresults Ġon Ġimage Ġclassification . ĠWe Ġevaluate Ġtwo Ġsettings Ġwith Ġthe ĠRes Net - 34 Ġbase Ġmodel Ġbeing Ġinitialized Ġrandomly Ġor Ġwith Ġpret rained Ġweights , Ġrespectively . ĠOur Ġdata Ġweight ing Ġconsistently Ġimproves Ġover Ġthe Ġbase Ġmodel Ġand ĠB IB REF 4 Ġregardless Ġof Ġthe Ġinitialization . </s> Exper iments Ġ: :: ĠIm balanced ĠLab els </s> We Ġnext Ġstudy Ġa Ġdifferent Ġproblem Ġsetting Ġwhere Ġthe Ġtraining Ġdata Ġof Ġdifferent Ġclasses Ġare Ġim balanced . ĠWe Ġshow Ġthe Ġdata Ġweight ing Ġapproach Ġgreatly Ġimproves Ġthe Ġclassification Ġperformance . ĠIt Ġis Ġalso Ġobserved Ġthat , Ġthe ĠLM Ġdata Ġaug mentation Ġapproach , Ġwhich Ġperforms Ġwell Ġin Ġthe Ġlow - data Ġsetting , Ġfails Ġon Ġthe Ġclass - im balance Ġproblems . </s> Exper iments Ġ: :: ĠIm balanced ĠLab els Ġ: :: ĠSetup </s> Though Ġthe Ġmethods Ġare Ġbroadly Ġapplicable Ġto Ġmulti - way Ġclassification Ġproblems , Ġhere Ġwe Ġonly Ġstudy Ġbinary Ġclassification Ġtasks Ġfor Ġsimplicity . ĠFor Ġtext Ġclassification , Ġwe Ġuse Ġthe ĠS ST - 2 Ġsentiment Ġanalysis Ġbenchmark ĠB IB REF 41 ; Ġwhile Ġfor Ġimage , Ġwe Ġselect Ġclass Ġ1 Ġand Ġ2 Ġfrom ĠC IF AR 10 Ġfor Ġbinary Ġclassification . ĠWe Ġuse Ġthe Ġsame Ġprocessing Ġon Ġboth Ġdatasets Ġto Ġbuild Ġthe Ġclass - im balance Ġsetting . ĠSpecifically , Ġwe Ġrandomly Ġselect Ġ1 , 000 Ġtraining Ġinstances Ġof Ġclass Ġ2 , Ġand Ġvary Ġthe Ġnumber Ġof Ġclass - 1 Ġinstances Ġin Ġ$\ l brace Ġ20 , Ġ50 , Ġ100 \ r brace Ġ$ . ĠFor Ġeach Ġdataset , Ġwe Ġuse Ġ10 Ġvalidation Ġexamples Ġin Ġeach Ġclass . ĠTr ained Ġmodels Ġare Ġevaluated Ġon Ġthe Ġfull Ġbinary - class Ġtest Ġset . </s> Exper iments Ġ: :: ĠIm balanced ĠLab els Ġ: :: ĠResults </s> Table ĠT AB REF 29 Ġshows Ġthe Ġclassification Ġresults Ġon ĠS ST - 2 Ġwith Ġvarying Ġimbalance Ġratios . ĠWe Ġcan Ġsee Ġour Ġdata Ġweight ing Ġperforms Ġbest Ġacross Ġall Ġsettings . ĠIn Ġparticular , Ġthe Ġimprovement Ġover Ġthe Ġbase Ġmodel Ġincreases Ġas Ġthe Ġdata Ġgets Ġmore Ġim balanced , Ġranging Ġfrom Ġaround Ġ6 Ġaccuracy Ġpoints Ġon Ġ100 : 1000 Ġto Ġover Ġ20 Ġaccuracy Ġpoints Ġon Ġ20 : 1000 . ĠOur Ġmethod Ġis Ġagain Ġconsistently Ġbetter Ġthan ĠB IB REF 4 , Ġvalid ating Ġthat Ġthe Ġparam etric Ġtreatment Ġis Ġbeneficial . ĠThe Ġproportion - based Ġdata Ġweight ing Ġprovides Ġonly Ġlimited Ġimprovement , Ġshowing Ġthe Ġadvantage Ġof Ġadaptive Ġdata Ġweight ing . ĠThe Ġbase Ġmodel Ġtrained Ġon Ġthe Ġjoint Ġtraining - valid ation Ġdata Ġfor Ġfixed Ġsteps Ġfails Ġto Ġperform Ġwell , Ġpartly Ġdue Ġto Ġthe Ġlack Ġof Ġa Ġproper Ġmechanism Ġfor Ġselecting Ġsteps . </s> Table ĠT AB REF 30 Ġshows Ġthe Ġresults Ġon Ġim balanced ĠC IF AR 10 Ġclassification . ĠSimilarly , Ġour Ġmethod Ġoutper forms Ġother Ġcomparison Ġapproaches . ĠIn Ġcontrast , Ġthe Ġfixed Ġproportion - based Ġmethod Ġsometimes Ġharms Ġthe Ġperformance Ġas Ġin Ġthe Ġ50 : 1000 Ġand Ġ100 : 1000 Ġsettings . </s> We Ġalso Ġtested Ġthe Ġtext Ġaug mentation ĠLM Ġon Ġthe ĠS ST - 2 Ġim balanced Ġdata . ĠInterestingly , Ġthe Ġaug mentation Ġtends Ġto Ġhinder Ġmodel Ġtraining Ġand Ġyields Ġaccuracy Ġof Ġaround Ġ50 % Ġ( random Ġguess ). ĠThis Ġis Ġbecause Ġthe Ġaug mentation ĠLM Ġis Ġfirst Ġfit Ġto Ġthe Ġim balanced Ġdata , Ġwhich Ġmakes Ġlabel Ġpreservation Ġinaccurate Ġand Ġintroduces Ġlots Ġof Ġnoise Ġduring Ġaug mentation . ĠThough Ġa Ġmore Ġcarefully Ġdesigned Ġaug mentation Ġmechanism Ġcan Ġpotentially Ġhelp Ġwith Ġim balanced Ġclassification Ġ( e . g ., Ġaugment ing Ġonly Ġthe Ġrare Ġclasses ), Ġthe Ġabove Ġobservation Ġfurther Ġshows Ġthat Ġthe Ġvarying Ġdata Ġmanipulation Ġschemes Ġhave Ġdifferent Ġapplicable Ġsc opes . ĠOur Ġapproach Ġis Ġthus Ġfavorable Ġas Ġthe Ġsingle Ġalgorithm Ġcan Ġbe Ġinstant iated Ġto Ġlearn Ġdifferent Ġschemes . </s> Disc ussions : ĠAl gorithm ĠExt rap olation Ġbetween ĠLearning ĠParad ig ms </s> Con clusions . ĠWe Ġhave Ġdeveloped Ġa Ġnew Ġmethod Ġof Ġlearning Ġdifferent Ġdata Ġmanipulation Ġschemes Ġwith Ġthe Ġsame Ġsingle Ġalgorithm . ĠDifferent Ġmanipulation Ġschemes Ġreduce Ġto Ġjust Ġdifferent Ġparameter ization Ġof Ġthe Ġdata Ġreward Ġfunction . ĠThe Ġmanipulation Ġparameters Ġare Ġtrained Ġjointly Ġwith Ġthe Ġtarget Ġmodel Ġparameters . ĠWe Ġinstant iate Ġthe Ġalgorithm Ġfor Ġdata Ġaug mentation Ġand Ġweight ing , Ġand Ġshow Ġimproved Ġperformance Ġover Ġstrong Ġbase Ġmodels Ġand Ġprevious Ġmanipulation Ġmethods . ĠWe Ġare Ġexcited Ġto Ġexplore Ġmore Ġtypes Ġof Ġmanip ulations Ġsuch Ġas Ġdata Ġsynthesis , Ġand Ġin Ġparticular Ġstudy Ġthe Ġcombination Ġof Ġdifferent Ġmanipulation Ġschemes . </s> The Ġproposed Ġmethod Ġbuilds Ġupon Ġthe Ġconnections Ġbetween Ġsupervised Ġlearning Ġand Ġreinforcement Ġlearning Ġ( RL ) ĠB IB REF 6 Ġthrough Ġwhich Ġwe Ġextrap olate Ġan Ġoff - the - she lf Ġreward Ġlearning Ġalgorithm Ġin Ġthe ĠRL Ġliterature Ġto Ġthe Ġsupervised Ġsetting . ĠThe Ġway Ġwe Ġobtained Ġthe Ġmanipulation Ġalgorithm Ġrepresents Ġa Ġgeneral Ġmeans Ġof Ġinnov ating Ġproblem Ġsolutions Ġbased Ġon Ġun ifying Ġformal isms Ġof Ġdifferent Ġlearning Ġparad ig ms . ĠSpecifically , Ġa Ġun ifying Ġformal ism Ġnot Ġonly Ġoffers Ġnew Ġunderstand ings Ġof Ġthe Ġseemingly Ġdistinct Ġparad ig ms , Ġbut Ġalso Ġallows Ġus Ġto Ġsystematically Ġapply Ġsolutions Ġto Ġproblems Ġin Ġone Ġparadigm Ġto Ġsimilar Ġproblems Ġin Ġanother . ĠPrevious Ġwork Ġalong Ġthis Ġline Ġhas Ġmade Ġfruitful Ġresults Ġin Ġother Ġdomains . ĠFor Ġexample , Ġan Ġextended Ġformulation Ġof ĠB IB REF 6 Ġthat Ġconnects ĠRL Ġand Ġposterior Ġregular ization Ġ( PR ) ĠB IB REF 44 , ĠB IB REF 45 Ġhas Ġenabled Ġto Ġsimilarly Ġexport Ġa Ġreward Ġlearning Ġalgorithm Ġto Ġthe Ġcontext Ġof ĠPR Ġfor Ġlearning Ġstructured Ġknowledge ĠB IB REF 46 . ĠBy Ġestablishing Ġa Ġuniform Ġab stration Ġof ĠG AN s ĠB IB REF 47 Ġand ĠVA Es ĠB IB REF 33 , ĠB IB REF 48 Ġexchange Ġtechniques Ġbetween Ġthe Ġtwo Ġfamilies Ġand Ġget Ġimproved Ġgener ative Ġmodeling . ĠOther Ġwork Ġin Ġthe Ġsimilar Ġspirit Ġincludes ĠB IB REF 49 , ĠB IB REF 50 , ĠB IB REF 51 . </s> By Ġextrap ol ating Ġalgorithms Ġbetween Ġparad ig ms , Ġone Ġcan Ġgo Ġbeyond Ġcrafting Ġnew Ġalgorithms Ġfrom Ġscratch Ġas Ġin Ġmost Ġexisting Ġstudies , Ġwhich Ġoften Ġrequires Ġdeep Ġexpertise Ġand Ġyields Ġunique Ġsolutions Ġin Ġa Ġdedicated Ġcontext . ĠInstead , Ġinnovation Ġbecomes Ġeasier Ġby Ġimporting Ġrich Ġideas Ġfrom Ġother Ġparad ig ms , Ġand Ġis Ġrepeat able Ġas Ġa Ġnew Ġalgorithm Ġcan Ġbe Ġmethod ically Ġextrap olated Ġto Ġmultiple Ġdifferent Ġcontexts .