 Penn Treebank, WikiText2
EXTRACTIVE
Penn Treebank (PTB) , WikiText2 (WT-2)
EXTRACTIVE
Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20 . For both datasets, we follow the same training, validation, and test splits as in BIBREF0 .
Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20 . For both datasets, we follow the same training, validation, and test splits as in BIBREF0 .
1808.09029
Introduction </s> Long Ġshort Ġterm Ġmemory Ġ( L ST M ) Ġunits ĠB IB REF 1 Ġare Ġpopular Ġfor Ġmany Ġsequence Ġmodeling Ġtasks Ġand Ġare Ġused Ġextensively Ġin Ġlanguage Ġmodeling . ĠA Ġkey Ġto Ġtheir Ġsuccess Ġis Ġtheir Ġarticulated Ġg ating Ġstructure , Ġwhich Ġallows Ġfor Ġmore Ġcontrol Ġover Ġthe Ġinformation Ġpassed Ġalong Ġthe Ġrec urrence . ĠHowever , Ġdespite Ġthe Ġsophistication Ġof Ġthe Ġg ating Ġmechanisms Ġemployed Ġin ĠL ST Ms Ġand Ġsimilar Ġrecurrent Ġunits , Ġthe Ġinput Ġand Ġcontext Ġvectors Ġare Ġtreated Ġwith Ġsimple Ġlinear Ġtransformations Ġprior Ġto Ġg ating . ĠNon - linear Ġtransformations Ġsuch Ġas Ġconv olutions ĠB IB REF 2 Ġhave Ġbeen Ġused , Ġbut Ġthese Ġhave Ġnot Ġachieved Ġthe Ġperformance Ġof Ġwell Ġregular ized ĠL ST Ms Ġfor Ġlanguage Ġmodeling ĠB IB REF 3 Ġ. </s> A Ġnatural Ġway Ġto Ġimprove Ġthe Ġexpress iveness Ġof Ġlinear Ġtransformations Ġis Ġto Ġincrease Ġthe Ġnumber Ġof Ġdimensions Ġof Ġthe Ġinput Ġand Ġcontext Ġvectors , Ġbut Ġthis Ġcomes Ġwith Ġa Ġsignificant Ġincrease Ġin Ġthe Ġnumber Ġof Ġparameters Ġwhich Ġmay Ġlimit Ġgeneral iz ability . ĠAn Ġexample Ġis Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ, Ġwhere ĠL ST Ms Ġperformance Ġdecreases Ġwith Ġthe Ġincrease Ġin Ġdimensions Ġof Ġthe Ġinput Ġand Ġcontext Ġvectors . ĠMoreover , Ġthe Ġsemantics Ġof Ġthe Ġinput Ġand Ġcontext Ġvectors Ġare Ġdifferent , Ġsuggesting Ġthat Ġeach Ġmay Ġbenefit Ġfrom Ġspecialized Ġtreatment . </s> Gu ided Ġby Ġthese Ġinsights , Ġwe Ġintroduce Ġa Ġnew Ġrecurrent Ġunit , Ġthe ĠPy ram idal ĠRec urrent ĠUnit Ġ( PR U ), Ġwhich Ġis Ġbased Ġon Ġthe ĠL ST M Ġg ating Ġstructure . ĠFigure ĠFIG REF 2 Ġprovides Ġan Ġoverview Ġof Ġthe ĠPR U . ĠAt Ġthe Ġheart Ġof Ġthe ĠPR U Ġis Ġthe Ġpy ram idal Ġtransformation Ġ( PT ), Ġwhich Ġuses Ġsubs am pling Ġto Ġeffect Ġmultiple Ġviews Ġof Ġthe Ġinput Ġvector . ĠThe Ġsubs am pled Ġrepresentations Ġare Ġcombined Ġin Ġa Ġpy ram idal Ġfusion Ġstructure , Ġresulting Ġin Ġricher Ġinteractions Ġbetween Ġthe Ġindividual Ġdimensions Ġof Ġthe Ġinput Ġvector Ġthan Ġis Ġpossible Ġwith Ġa Ġlinear Ġtransformation . ĠContext Ġvectors , Ġwhich Ġhave Ġalready Ġundergone Ġthis Ġtransformation Ġin Ġthe Ġprevious Ġcell , Ġare Ġmodified Ġwith Ġa Ġgrouped Ġlinear Ġtransformation Ġ( GL T ) Ġwhich Ġallows Ġthe Ġnetwork Ġto Ġlearn Ġlatent Ġrepresentations Ġin Ġhigh Ġdimensional Ġspace Ġwith Ġfewer Ġparameters Ġand Ġbetter Ġgeneral iz ability Ġ( see ĠFigure ĠFIG REF 1 Ġ). </s> We Ġshow Ġthat ĠPR Us Ġcan Ġbetter Ġmodel Ġcontextual Ġinformation Ġand Ġdemonstrate Ġperformance Ġgains Ġon Ġthe Ġtask Ġof Ġlanguage Ġmodeling . ĠThe ĠPR U Ġimproves Ġthe Ġperplex ity Ġof Ġthe Ġcurrent Ġstate - of - the - art Ġlanguage Ġmodel ĠB IB REF 0 Ġby Ġup Ġto Ġ1 . 3 Ġpoints , Ġreaching Ġperplex ities Ġof Ġ56 . 56 Ġand Ġ64 . 53 Ġon Ġthe ĠPenn ĠTree bank Ġand ĠWiki Text 2 Ġdatasets Ġwhile Ġlearning Ġ15 - 20 % Ġfewer Ġparameters . ĠRepl acing Ġan ĠL ST M Ġwith Ġa ĠPR U Ġresults Ġin Ġimprovements Ġin Ġperplex ity Ġacross Ġa Ġvariety Ġof Ġexperimental Ġsettings . ĠWe Ġprovide Ġdetailed Ġabl ations Ġwhich Ġmotivate Ġthe Ġdesign Ġof Ġthe ĠPR U Ġarchitecture , Ġas Ġwell Ġas Ġdetailed Ġanalysis Ġof Ġthe Ġeffect Ġof Ġthe ĠPR U Ġon Ġother Ġcomponents Ġof Ġthe Ġlanguage Ġmodel . </s> Related Ġwork </s> Multiple Ġmethods , Ġincluding Ġa Ġvariety Ġof Ġg ating Ġstructures Ġand Ġtransformations , Ġhave Ġbeen Ġproposed Ġto Ġimprove Ġthe Ġperformance Ġof Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ). ĠWe Ġfirst Ġdescribe Ġthese Ġapproaches Ġand Ġthen Ġprovide Ġan Ġoverview Ġof Ġrecent Ġwork Ġin Ġlanguage Ġmodeling . </s> Py ram idal ĠRec urrent ĠUnits </s> We Ġintroduce ĠPy ram idal ĠRec urrent ĠUnits Ġ( PR Us ), Ġa Ġnew ĠR NN Ġarchitecture Ġwhich Ġimproves Ġmodeling Ġof Ġcontext Ġby Ġallowing Ġfor Ġhigher Ġdimensional Ġvector Ġrepresentations Ġwhile Ġlearning Ġfewer Ġparameters . ĠFigure ĠFIG REF 2 Ġprovides Ġan Ġoverview Ġof ĠPR U . ĠWe Ġfirst Ġelaborate Ġon Ġthe Ġdetails Ġof Ġthe Ġpy ram idal Ġtransformation Ġand Ġthe Ġgrouped Ġlinear Ġtransformation . ĠWe Ġthen Ġdescribe Ġour Ġrecurrent Ġunit , ĠPR U . </s> Py ram idal Ġtransformation Ġfor Ġinput </s> The Ġbasic Ġtransformation Ġin Ġmany Ġrecurrent Ġunits Ġis Ġa Ġlinear Ġtransformation ĠIN LINE FORM 0 Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġare Ġlearned Ġweights Ġthat Ġlin early Ġmap ĠIN LINE FORM 1 Ġto ĠIN LINE FORM 2 Ġ. ĠTo Ġsimplify Ġnotation , Ġwe Ġomit Ġthe Ġbiases . </s> Mot ivated Ġby Ġsuccessful Ġapplications Ġof Ġsub - sam pling Ġin Ġcomputer Ġvision Ġ( e . g ., ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 24 Ġ), Ġwe Ġsubs ample Ġinput Ġvector ĠIN LINE FORM 0 Ġinto ĠIN LINE FORM 1 Ġpy ram idal Ġlevels Ġto Ġachieve Ġrepresentation Ġof Ġthe Ġinput Ġvector Ġat Ġmultiple Ġscales . ĠThis Ġsub - sam pling Ġoperation Ġproduces ĠIN LINE FORM 2 Ġvectors , Ġrepresented Ġas ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġsampling Ġrate Ġand ĠIN LINE FORM 5 Ġ. ĠWe Ġlearn Ġscale - specific Ġtransformations ĠIN LINE FORM 6 Ġfor Ġeach ĠIN LINE FORM 7 Ġ. ĠThe Ġtransformed Ġsubs amples Ġare Ġconc aten ated Ġto Ġproduce Ġthe Ġpy ram idal Ġanalog Ġto ĠIN LINE FORM 8 Ġ, Ġhere Ġden oted Ġas ĠIN LINE FORM 9 Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġindicates Ġconc aten ation . ĠWe Ġnote Ġthat Ġpy ram idal Ġtransformation Ġwith ĠIN LINE FORM 1 Ġis Ġthe Ġsame Ġas Ġthe Ġlinear Ġtransformation . </s> To Ġimprove Ġgradient Ġflow Ġinside Ġthe Ġrecurrent Ġunit , Ġwe Ġcombine Ġthe Ġinput Ġand Ġoutput Ġusing Ġan Ġelement - wise Ġsum Ġ( when Ġdimension Ġmatches ) Ġto Ġproduce Ġresidual Ġanalog Ġof Ġpy ram idal Ġtransformation , Ġas Ġshown Ġin ĠFigure ĠFIG REF 2 ĠB IB REF 25 Ġ. </s> We Ġsub - sample Ġthe Ġinput Ġvector ĠIN LINE FORM 0 Ġinto ĠIN LINE FORM 1 Ġpy ram idal Ġlevels Ġusing Ġthe Ġkernel - based Ġapproach ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠLet Ġus Ġassume Ġthat Ġwe Ġhave Ġa Ġkernel ĠIN LINE FORM 2 Ġwith ĠIN LINE FORM 3 Ġelements . ĠThen , Ġthe Ġinput Ġvector ĠIN LINE FORM 4 Ġcan Ġbe Ġsub - sam pled Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġstride Ġand ĠIN LINE FORM 1 Ġ. </s> The Ġnumber Ġof Ġparameters Ġlearned Ġby Ġthe Ġlinear Ġtransformation Ġand Ġthe Ġpy ram idal Ġtransformation Ġwith ĠIN LINE FORM 0 Ġpy ram idal Ġlevels Ġto Ġmap ĠIN LINE FORM 1 Ġto ĠIN LINE FORM 2 Ġare ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġrespectively . ĠThus , Ġpy ram idal Ġtransformation Ġreduces Ġthe Ġparameters Ġof Ġa Ġlinear Ġtransformation Ġby Ġa Ġfactor Ġof ĠIN LINE FORM 5 Ġ. ĠFor Ġexample , Ġthe Ġpy ram idal Ġtransformation Ġ( with ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ) Ġlearns ĠIN LINE FORM 8 Ġfewer Ġparameters Ġthan Ġthe Ġlinear Ġtransformation . </s> Group ed Ġlinear Ġtransformation Ġfor Ġcontext </s> Many ĠR NN Ġarchitectures Ġapply Ġlinear Ġtransformations Ġto Ġboth Ġthe Ġinput Ġand Ġcontext Ġvector . ĠHowever , Ġthis Ġmay Ġnot Ġbe Ġideal Ġdue Ġto Ġthe Ġdiffering Ġsemantics Ġof Ġeach Ġvector . ĠIn Ġmany ĠN LP Ġapplications Ġincluding Ġlanguage Ġmodeling , Ġthe Ġinput Ġvector Ġis Ġa Ġdense Ġword Ġembed ding Ġwhich Ġis Ġshared Ġacross Ġall Ġcontexts Ġfor Ġa Ġgiven Ġword Ġin Ġa Ġdataset . ĠIn Ġcontrast , Ġthe Ġcontext Ġvector Ġis Ġhighly Ġcontextual ized Ġby Ġthe Ġcurrent Ġsequence . ĠThe Ġdifferences Ġbetween Ġthe Ġinput Ġand Ġcontext Ġvector Ġmotivate Ġtheir Ġseparate Ġtreatment Ġin Ġthe ĠPR U Ġarchitecture . </s> The Ġweights Ġlearned Ġusing Ġthe Ġlinear Ġtransformation Ġ( E q . ĠEQ REF 9 Ġ) Ġare Ġreused Ġover Ġmultiple Ġtime Ġsteps , Ġwhich Ġmakes Ġthem Ġprone Ġto Ġover - fitting ĠB IB REF 26 Ġ. ĠTo Ġcombat Ġover - fitting , Ġvarious Ġmethods , Ġsuch Ġas Ġvari ational Ġdrop out ĠB IB REF 26 Ġand Ġweight Ġdrop out ĠB IB REF 0 Ġ, Ġhave Ġbeen Ġproposed Ġto Ġregular ize Ġthese Ġrecurrent Ġconnections . ĠTo Ġfurther Ġimprove Ġgeneral ization Ġabilities Ġwhile Ġsimultaneously Ġenabling Ġthe Ġrecurrent Ġunit Ġto Ġlearn Ġrepresentations Ġat Ġvery Ġhigh Ġdimensional Ġspace , Ġwe Ġpropose Ġto Ġuse Ġgrouped Ġlinear Ġtransformation Ġ( GL T ) Ġinstead Ġof Ġstandard Ġlinear Ġtransformation Ġfor Ġrecurrent Ġconnections ĠB IB REF 27 Ġ. ĠWhile Ġpy ram idal Ġand Ġlinear Ġtransformations Ġcan Ġbe Ġapplied Ġto Ġtransform Ġcontext Ġvectors , Ġour Ġexperimental Ġresults Ġin ĠSection ĠSEC REF 39 Ġsuggests Ġthat ĠGL Ts Ġare Ġmore Ġeffective . </s> The Ġlinear Ġtransformation ĠIN LINE FORM 0 Ġmaps ĠIN LINE FORM 1 Ġlin early Ġto ĠIN LINE FORM 2 Ġ. ĠGroup ed Ġlinear Ġtransformations Ġbreak Ġthe Ġlinear Ġinteractions Ġby Ġfact oring Ġthe Ġlinear Ġtransformation Ġinto Ġtwo Ġsteps . ĠFirst , Ġa ĠGL T Ġsplits Ġthe Ġinput Ġvector ĠIN LINE FORM 3 Ġinto ĠIN LINE FORM 4 Ġsmaller Ġgroups Ġsuch Ġthat ĠIN LINE FORM 5 Ġ. ĠSecond , Ġa Ġlinear Ġtransformation ĠIN LINE FORM 6 Ġis Ġapplied Ġto Ġmap ĠIN LINE FORM 7 Ġlin early Ġto ĠIN LINE FORM 8 Ġ, Ġfor Ġeach ĠIN LINE FORM 9 Ġ. ĠThe ĠIN LINE FORM 10 Ġresultant Ġoutput Ġvectors ĠIN LINE FORM 11 Ġare Ġconc aten ated Ġto Ġproduce Ġthe Ġfinal Ġoutput Ġvector ĠIN LINE FORM 12 Ġ. ĠDIS PLAY FORM 0 </s> GL Ts Ġlearn Ġrepresentations Ġat Ġlow Ġdimension ality . ĠTherefore , Ġa ĠGL T Ġrequires ĠIN LINE FORM 0 Ġfewer Ġparameters Ġthan Ġthe Ġlinear Ġtransformation . ĠWe Ġnote Ġthat ĠGL Ts Ġare Ġsubset Ġof Ġlinear Ġtransformations . ĠIn Ġa Ġlinear Ġtransformation , Ġeach Ġneuron Ġreceives Ġan Ġinput Ġfrom Ġeach Ġelement Ġin Ġthe Ġinput Ġvector Ġwhile Ġin Ġa ĠGL T , Ġeach Ġneuron Ġreceives Ġan Ġinput Ġfrom Ġa Ġsubset Ġof Ġthe Ġinput Ġvector . ĠTherefore , ĠGL T Ġis Ġthe Ġsame Ġas Ġa Ġlinear Ġtransformation Ġwhen ĠIN LINE FORM 1 Ġ. </s> Py ram idal ĠRec urrent ĠUnit </s> We Ġextend Ġthe Ġbasic Ġg ating Ġarchitecture Ġof ĠL ST M Ġwith Ġthe Ġpy ram idal Ġand Ġgrouped Ġlinear Ġtransformations Ġoutlined Ġabove Ġto Ġproduce Ġthe ĠPy ram idal ĠRec urrent ĠUnit Ġ( PR U ), Ġwhose Ġimproved Ġsequence Ġmodeling Ġcapacity Ġis Ġevidenced Ġin ĠSection ĠSEC REF 4 Ġ. </s> At Ġtime ĠIN LINE FORM 0 Ġ, Ġthe ĠPR U Ġcombines Ġthe Ġinput Ġvector ĠIN LINE FORM 1 Ġand Ġthe Ġprevious Ġcontext Ġvector Ġ( or Ġprevious Ġhidden Ġstate Ġvector ) ĠIN LINE FORM 2 Ġusing Ġthe Ġfollowing Ġtransformation Ġfunction Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġindexes Ġthe Ġvarious Ġgates Ġin Ġthe ĠL ST M Ġmodel , Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġrepresents Ġthe Ġpy ram idal Ġand Ġgrouped Ġlinear Ġtransformations Ġdefined Ġin ĠE q ns . ĠEQ REF 10 Ġand ĠEQ REF 15 Ġ, Ġrespectively . </s> We Ġwill Ġnow Ġincorporate ĠIN LINE FORM 0 Ġinto ĠL ST M Ġg ating Ġarchitecture Ġto Ġproduce ĠPR U . ĠAt Ġtime ĠIN LINE FORM 1 Ġ, Ġa ĠPR U Ġcell Ġtakes ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġas Ġinputs Ġto Ġproduce Ġforget ĠIN LINE FORM 5 Ġ, Ġinput ĠIN LINE FORM 6 Ġ, Ġoutput ĠIN LINE FORM 7 Ġ, Ġand Ġcontent ĠIN LINE FORM 8 Ġgate Ġsignals . ĠThe Ġinputs Ġare Ġcombined Ġwith Ġthese Ġgate Ġsignals Ġto Ġproduce Ġcontext Ġvector ĠIN LINE FORM 9 Ġand Ġcell Ġstate ĠIN LINE FORM 10 Ġ. ĠMat hematically , Ġthe ĠPR U Ġwith Ġthe ĠL ST M Ġg ating Ġarchitecture Ġcan Ġbe Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġelement - wise Ġmultiplication Ġoperation , Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġthe Ġs igm oid Ġand Ġhyper b olic Ġtang ent Ġactivation Ġfunctions . ĠWe Ġnote Ġthat ĠL ST M Ġis Ġa Ġspecial Ġcase Ġof ĠPR U Ġwhen ĠIN LINE FORM 3 Ġ= ĠIN LINE FORM 4 Ġ= 1 . </s> Exper iments </s> To Ġshowcase Ġthe Ġeffectiveness Ġof Ġthe ĠPR U , Ġwe Ġevaluate Ġthe Ġperformance Ġon Ġtwo Ġstandard Ġdatasets Ġfor Ġword - level Ġlanguage Ġmodeling Ġand Ġcompare Ġwith Ġstate - of - the - art Ġmethods . ĠAdditionally , Ġwe Ġprovide Ġa Ġdetailed Ġexamination Ġof Ġthe ĠPR U Ġand Ġits Ġbehavior Ġon Ġthe Ġlanguage Ġmodeling Ġtasks . </s> Set - up </s> Following Ġrecent Ġworks , Ġwe Ġcompare Ġon Ġtwo Ġwidely Ġused Ġdatasets , Ġthe ĠPenn ĠTree bank Ġ( PT B ) ĠB IB REF 28 Ġas Ġprepared Ġby ĠB IB REF 29 Ġand ĠWiki Text 2 Ġ( WT - 2 ) ĠB IB REF 20 Ġ. ĠFor Ġboth Ġdatasets , Ġwe Ġfollow Ġthe Ġsame Ġtraining , Ġvalidation , Ġand Ġtest Ġsplits Ġas Ġin ĠB IB REF 0 Ġ. </s> We Ġextend Ġthe Ġlanguage Ġmodel , ĠAW D - L ST M ĠB IB REF 0 Ġ, Ġby Ġreplacing ĠL ST M Ġlayers Ġwith ĠPR U . ĠOur Ġmodel Ġuses Ġ3 - l ayers Ġof ĠPR U Ġwith Ġan Ġembed ding Ġsize Ġof Ġ400 . ĠThe Ġnumber Ġof Ġparameters Ġlearned Ġby Ġstate - of - the - art Ġmethods Ġvary Ġfrom Ġ18 M Ġto Ġ66 M Ġwith Ġmajority Ġof Ġthe Ġmethods Ġlearning Ġabout Ġ22 M Ġto Ġ24 M Ġparameters Ġon Ġthe ĠPT B Ġdataset . ĠFor Ġa Ġfair Ġcomparison Ġwith Ġstate - of - the - art Ġmethods , Ġwe Ġfix Ġthe Ġmodel Ġsize Ġto Ġ19 M Ġand Ġvary Ġthe Ġvalue Ġof ĠIN LINE FORM 0 Ġand Ġhidden Ġlayer Ġsizes Ġso Ġthat Ġtotal Ġnumber Ġof Ġlearned Ġparameters Ġis Ġsimilar Ġacross Ġdifferent Ġconfigurations . ĠWe Ġuse Ġ1000 , Ġ1200 , Ġand Ġ1400 Ġas Ġhidden Ġlayer Ġsizes Ġfor Ġvalues Ġof ĠIN LINE FORM 1 Ġ= 1 , 2 , Ġand Ġ4 , Ġrespectively . ĠWe Ġuse Ġthe Ġsame Ġsettings Ġfor Ġthe ĠWT - 2 Ġdataset . ĠWe Ġset Ġthe Ġnumber Ġof Ġpy ram idal Ġlevels ĠIN LINE FORM 2 Ġto Ġtwo Ġin Ġour Ġexperiments Ġand Ġuse Ġaverage Ġpool ing Ġfor Ġsub - sam pling . ĠThese Ġvalues Ġare Ġselected Ġbased Ġon Ġour Ġab lation Ġexperiments Ġon Ġthe Ġvalidation Ġset Ġ( Section ĠSEC REF 39 Ġ). ĠWe Ġmeasure Ġthe Ġperformance Ġof Ġour Ġmodels Ġin Ġterms Ġof Ġword - level Ġperplex ity . ĠWe Ġfollow Ġthe Ġsame Ġtraining Ġstrategy Ġas Ġin ĠB IB REF 0 Ġ. </s> To Ġunderstand Ġthe Ġeffect Ġof Ġregular ization Ġmethods Ġon Ġthe Ġperformance Ġof ĠPR Us , Ġwe Ġperform Ġexperiments Ġunder Ġtwo Ġdifferent Ġsettings : Ġ( 1 ) ĠStandard Ġdrop out : ĠWe Ġuse Ġa Ġstandard Ġdrop out ĠB IB REF 12 Ġwith Ġprobability Ġof Ġ0 . 5 Ġafter Ġembed ding Ġlayer , Ġthe Ġoutput Ġbetween ĠL ST M Ġlayers , Ġand Ġthe Ġoutput Ġof Ġfinal ĠL ST M Ġlayer . Ġ( 2 ) ĠAdvanced Ġdrop out : ĠWe Ġuse Ġthe Ġsame Ġdrop out Ġtechniques Ġwith Ġthe Ġsame Ġdrop out Ġvalues Ġas Ġin ĠB IB REF 0 Ġ. ĠWe Ġcall Ġthis Ġmodel Ġas ĠAW D - PR U . </s> Results </s> Table ĠT AB REF 23 Ġcompares Ġthe Ġperformance Ġof Ġthe ĠPR U Ġwith Ġstate - of - the - art Ġmethods . ĠWe Ġcan Ġsee Ġthat Ġthe ĠPR U Ġachieves Ġthe Ġbest Ġperformance Ġwith Ġfewer Ġparameters . </s> PR Us Ġachieve Ġeither Ġthe Ġsame Ġor Ġbetter Ġperformance Ġthan ĠL ST Ms . ĠIn Ġparticular , Ġthe Ġperformance Ġof ĠPR Us Ġimproves Ġwith Ġthe Ġincreasing Ġvalue Ġof ĠIN LINE FORM 0 Ġ. ĠAt ĠIN LINE FORM 1 Ġ, ĠPR Us Ġoutper form ĠL ST Ms Ġby Ġabout Ġ4 Ġpoints Ġon Ġthe ĠPT B Ġdataset Ġand Ġby Ġabout Ġ3 Ġpoints Ġon Ġthe ĠWT - 2 Ġdataset . ĠThis Ġis Ġexplained Ġin Ġpart Ġby Ġthe Ġregular ization Ġeffect Ġof Ġthe Ġgrouped Ġlinear Ġtransformation Ġ( Figure ĠFIG REF 1 Ġ). ĠWith Ġgrouped Ġlinear Ġand Ġpy ram idal Ġtransformations , ĠPR Us Ġlearn Ġrich Ġrepresentations Ġat Ġvery Ġhigh Ġdimensional Ġspace Ġwhile Ġlearning Ġfewer Ġparameters . ĠOn Ġthe Ġother Ġhand , ĠL ST Ms Ġover fit Ġto Ġthe Ġtraining Ġdata Ġat Ġsuch Ġhigh Ġdimensions Ġand Ġlearn ĠIN LINE FORM 2 Ġto ĠIN LINE FORM 3 Ġmore Ġparameters Ġthan ĠPR Us . </s> With Ġthe Ġadvanced Ġdrop outs , Ġthe Ġperformance Ġof ĠPR Us Ġimproves Ġby Ġabout Ġ4 Ġpoints Ġon Ġthe ĠPT B Ġdataset Ġand Ġ7 Ġpoints Ġon Ġthe ĠWT - 2 Ġdataset . ĠThis Ġfurther Ġimproves Ġwith Ġfin et uning Ġon Ġthe ĠPT B Ġ( about Ġ2 Ġpoints ) Ġand ĠWT - 2 Ġ( about Ġ1 Ġpoint ) Ġdatasets . </s> For Ġsimilar Ġnumber Ġof Ġparameters , Ġthe ĠPR U Ġwith Ġstandard Ġdrop out Ġoutper forms Ġmost Ġof Ġthe Ġstate - of - the - art Ġmethods Ġby Ġlarge Ġmargin Ġon Ġthe ĠPT B Ġdataset Ġ( e . g . ĠR AN ĠB IB REF 7 Ġby Ġ16 Ġpoints Ġwith Ġ4 M Ġless Ġparameters , ĠQR NN ĠB IB REF 33 Ġby Ġ16 Ġpoints Ġwith Ġ1 M Ġmore Ġparameters , Ġand ĠNAS ĠB IB REF 31 Ġby Ġ1 . 58 Ġpoints Ġwith Ġ6 M Ġless Ġparameters ). ĠWith Ġadvanced Ġdrop outs , Ġthe ĠPR U Ġdelivers Ġthe Ġbest Ġperformance . ĠOn Ġboth Ġdatasets , Ġthe ĠPR U Ġimproves Ġthe Ġperplex ity Ġby Ġabout Ġ1 Ġpoint Ġwhile Ġlearning Ġ15 - 20 % Ġfewer Ġparameters . </s> PR U Ġis Ġa Ġdrop - in Ġreplacement Ġfor ĠL ST M , Ġtherefore , Ġit Ġcan Ġimprove Ġlanguage Ġmodels Ġwith Ġmodern Ġinference Ġtechniques Ġsuch Ġas Ġdynamic Ġevaluation ĠB IB REF 21 Ġ. ĠWhen Ġwe Ġevaluate ĠPR U - based Ġlanguage Ġmodels Ġ( only Ġwith Ġstandard Ġdrop out ) Ġwith Ġdynamic Ġevaluation Ġon Ġthe ĠPT B Ġtest Ġset , Ġthe Ġperplex ity Ġof ĠPR U Ġ( ĠIN LINE FORM 0 Ġ) Ġimproves Ġfrom Ġ62 . 42 Ġto Ġ55 . 23 Ġwhile Ġthe Ġperplex ity Ġof Ġan ĠL ST M Ġ( ĠIN LINE FORM 1 Ġ) Ġwith Ġsimilar Ġsettings Ġimproves Ġfrom Ġ66 . 29 Ġto Ġ58 . 79 ; Ġsuggesting Ġthat Ġmodern Ġinference Ġtechniques Ġare Ġequally Ġapplicable Ġto ĠPR U - based Ġlanguage Ġmodels . </s> Analysis </s> It Ġis Ġshown Ġabove Ġthat Ġthe ĠPR U Ġcan Ġlearn Ġrepresentations Ġat Ġhigher Ġdimension ality Ġwith Ġmore Ġgeneral ization Ġpower , Ġresulting Ġin Ġperformance Ġgains Ġfor Ġlanguage Ġmodeling . ĠA Ġcloser Ġanalysis Ġof Ġthe Ġimpact Ġof Ġthe ĠPR U Ġin Ġa Ġlanguage Ġmodeling Ġsystem Ġreveals Ġseveral Ġfactors Ġthat Ġhelp Ġexplain Ġhow Ġthe ĠPR U Ġachieves Ġthese Ġgains . </s> As Ġexempl ified Ġin ĠTable ĠT AB REF 34 Ġ, Ġthe ĠPR U Ġtends Ġtoward Ġmore Ġconfident Ġdecisions , Ġplacing Ġmore Ġof Ġthe Ġprobability Ġmass Ġon Ġthe Ġtop Ġnext - word Ġprediction Ġthan Ġthe ĠL ST M . ĠTo Ġquantify Ġthis Ġeffect , Ġwe Ġcalculate Ġthe Ġentropy Ġof Ġthe Ġnext - token Ġdistribution Ġfor Ġboth Ġthe ĠPR U Ġand Ġthe ĠL ST M Ġusing Ġ3 687 Ġcontexts Ġfrom Ġthe ĠPT B Ġvalidation Ġset . ĠFigure ĠFIG REF 32 Ġshows Ġa Ġhist ogram Ġof Ġthe Ġent rop ies Ġof Ġthe Ġdistribution , Ġwhere Ġbins Ġof Ġsize Ġ0 . 23 Ġare Ġused Ġto Ġeffect Ġcategories . ĠWe Ġsee Ġthat Ġthe ĠPR U Ġmore Ġoften Ġproduces Ġlower Ġentropy Ġdistributions Ġcorresponding Ġto Ġhigher Ġconf idences Ġfor Ġnext - token Ġchoices . ĠThis Ġis Ġevidenced Ġby Ġthe Ġmass Ġof Ġthe Ġred ĠPR U Ġcurve Ġlying Ġin Ġthe Ġlower Ġentropy Ġranges Ġcompared Ġto Ġthe Ġblue ĠL ST M 's Ġcurve . ĠThe ĠPR U Ġcan Ġproduce Ġconfident Ġdecisions Ġin Ġpart Ġbecause Ġmore Ġinformation Ġis Ġencoded Ġin Ġthe Ġhigher Ġdimensional Ġcontext Ġvectors . </s> The ĠPR U Ġhas Ġthe Ġability Ġto Ġmodel Ġindividual Ġwords Ġat Ġdifferent Ġresolutions Ġthrough Ġthe Ġpy ram idal Ġtransform ; Ġwhich Ġprovides Ġmultiple Ġpaths Ġfor Ġthe Ġgradient Ġto Ġthe Ġembed ding Ġlayer Ġ( similar Ġto Ġmulti - task Ġlearning ) Ġand Ġimproves Ġthe Ġflow Ġof Ġinformation . ĠWhen Ġconsidering Ġthe Ġembed d ings Ġby Ġpart Ġof Ġspeech , Ġwe Ġfind Ġthat Ġthe Ġpyramid Ġlevel Ġ1 Ġembed d ings Ġexhibit Ġhigher Ġvariance Ġthan Ġthe ĠL ST M Ġacross Ġall ĠPOS Ġcategories Ġ( Figure ĠFIG REF 33 Ġ), Ġand Ġthat Ġpyramid Ġlevel Ġ2 Ġembed d ings Ġshow Ġextremely Ġlow Ġvariance . ĠWe Ġhypothes ize Ġthat Ġthe ĠL ST M Ġmust Ġencode Ġboth Ġcoarse Ġgroup Ġsimilarities Ġand Ġindividual Ġword Ġdifferences Ġinto Ġthe Ġsame Ġvector Ġspace , Ġreducing Ġthe Ġspace Ġbetween Ġindividual Ġwords Ġof Ġthe Ġsame Ġcategory . ĠThe ĠPR U Ġcan Ġrely Ġon Ġthe Ġsubs am pled Ġembed d ings Ġto Ġaccount Ġfor Ġcoarse - gr ained Ġgroup Ġsimilarities , Ġallowing Ġfor Ġfiner Ġindividual Ġword Ġdistinctions Ġin Ġthe Ġembed ding Ġlayer . ĠThis Ġhypothesis Ġis Ġstrengthened Ġby Ġthe Ġentropy Ġresults Ġdescribed Ġabove : Ġa Ġmodel Ġwhich Ġcan Ġmake Ġfiner Ġdistinctions Ġbetween Ġindividual Ġwords Ġcan Ġmore Ġconfidently Ġassign Ġprobability Ġmass . ĠA Ġmodel Ġthat Ġcannot Ġmake Ġthese Ġdistinctions , Ġsuch Ġas Ġthe ĠL ST M , Ġmust Ġspread Ġits Ġprobability Ġmass Ġacross Ġa Ġlarger Ġclass Ġof Ġsimilar Ġwords . </s> Sal iency Ġanalysis Ġusing Ġgrad ients Ġhelp Ġidentify Ġrelevant Ġwords Ġin Ġa Ġtest Ġsequence Ġthat Ġcontribute Ġto Ġthe Ġprediction ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ. ĠThese Ġapproaches Ġcompute Ġthe Ġrelevance Ġas Ġthe Ġsquared Ġnorm Ġof Ġthe Ġgrad ients Ġobtained Ġthrough Ġback - prop ag ation . ĠTable ĠT AB REF 34 Ġvisual izes Ġthe Ġheat maps Ġfor Ġdifferent Ġsequences . ĠPR Us , Ġin Ġgeneral , Ġgive Ġmore Ġrelevance Ġto Ġcontextual Ġwords Ġthan ĠL ST Ms , Ġsuch Ġas Ġsoutheast Ġ( sample Ġ1 ), Ġcost Ġ( sample Ġ2 ), Ġface Ġ( sample Ġ4 ), Ġand Ġintroduced Ġ( sample Ġ5 ), Ġwhich Ġhelp Ġin Ġmaking Ġmore Ġconfident Ġdecisions . ĠFurthermore , Ġwhen Ġgrad ients Ġduring Ġback - prop ag ation Ġare Ġvisual ized ĠB IB REF 37 Ġ( Table ĠT AB REF 34 Ġ), Ġwe Ġfind Ġthat ĠPR Us Ġhave Ġbetter Ġgradient Ġcoverage Ġthan ĠL ST Ms , Ġsuggesting ĠPR Us Ġuse Ġmore Ġfeatures Ġthan ĠL ST Ms Ġthat Ġcontributes Ġto Ġthe Ġdecision . ĠThis Ġalso Ġsuggests Ġthat ĠPR Us Ġupdate Ġmore Ġparameters Ġat Ġeach Ġiteration Ġwhich Ġresults Ġin Ġfaster Ġtraining . ĠLanguage Ġmodel Ġin ĠB IB REF 0 Ġtakes Ġ500 Ġand Ġ750 Ġepoch s Ġto Ġconverge Ġwith ĠPR U Ġand ĠL ST M Ġas Ġa Ġrecurrent Ġunit , Ġrespectively . </s> A bl ation Ġstudies </s> In Ġthis Ġsection , Ġwe Ġprovide Ġa Ġsystematic Ġanalysis Ġof Ġour Ġdesign Ġchoices . ĠOur Ġtraining Ġmethodology Ġis Ġthe Ġsame Ġas Ġdescribed Ġin ĠSection ĠSEC REF 19 Ġwith Ġthe Ġstandard Ġdrop outs . ĠFor Ġa Ġthorough Ġunderstanding Ġof Ġour Ġdesign Ġchoices , Ġwe Ġuse Ġa Ġlanguage Ġmodel Ġwith Ġa Ġsingle Ġlayer Ġof ĠPR U Ġand Ġfix Ġthe Ġsize Ġof Ġembed ding Ġand Ġhidden Ġlayers Ġto Ġ600 . ĠThe Ġword - level Ġperplex ities Ġare Ġreported Ġon Ġthe Ġvalidation Ġsets Ġof Ġthe ĠPT B Ġand Ġthe ĠWT - 2 Ġdatasets . </s> The Ġtwo Ġhyper - param eters Ġthat Ġcontrol Ġthe Ġtrade - off Ġbetween Ġperformance Ġand Ġnumber Ġof Ġparameters Ġin ĠPR Us Ġare Ġthe Ġnumber Ġof Ġpy ram idal Ġlevels ĠIN LINE FORM 0 Ġand Ġgroups ĠIN LINE FORM 1 Ġ. ĠFigure ĠFIG REF 35 Ġprovides Ġa Ġtrade - off Ġbetween Ġperplex ity Ġand Ġrecurrent Ġunit Ġ( R U ) Ġparameters . </s> Variable ĠIN LINE FORM 0 Ġand Ġfixed ĠIN LINE FORM 1 Ġ: ĠWhen Ġwe Ġincrease Ġthe Ġnumber Ġof Ġpy ram idal Ġlevels ĠIN LINE FORM 2 Ġat Ġa Ġfixed Ġvalue Ġof ĠIN LINE FORM 3 Ġ, Ġthe Ġperformance Ġof Ġthe ĠPR U Ġdrops Ġby Ġabout Ġ1 Ġto Ġ4 Ġpoints Ġwhile Ġreducing Ġthe Ġtotal Ġnumber Ġof Ġrecurrent Ġunit Ġparameters Ġby Ġup Ġto Ġ15 %. ĠWe Ġnote Ġthat Ġthe ĠPR U Ġwith ĠIN LINE FORM 4 Ġat ĠIN LINE FORM 5 Ġdelivers Ġsimilar Ġperformance Ġas Ġthe ĠL ST M Ġwhile Ġlearning Ġabout Ġ15 % Ġfewer Ġrecurrent Ġunit Ġparameters . </s> Fixed ĠIN LINE FORM 0 Ġand Ġvariable ĠIN LINE FORM 1 Ġ: ĠWhen Ġwe Ġvary Ġthe Ġvalue Ġof ĠIN LINE FORM 2 Ġat Ġfixed Ġnumber Ġof Ġpy ram idal Ġlevels ĠIN LINE FORM 3 Ġ, Ġthe Ġtotal Ġnumber Ġof Ġrecurrent Ġunit Ġparameters Ġdecreases Ġsignificantly Ġwith Ġa Ġminimal Ġimpact Ġon Ġthe Ġperplex ity . ĠFor Ġexample , ĠPR Us Ġwith ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġlearns Ġ77 % Ġfewer Ġrecurrent Ġunit Ġparameters Ġwhile Ġits Ġperplex ity Ġ( lower Ġis Ġbetter ) Ġincreases Ġby Ġabout Ġ12 % Ġin Ġcomparison Ġto ĠL ST Ms . ĠMoreover , Ġthe Ġdecrease Ġin Ġnumber Ġof Ġparameters Ġat Ġhigher Ġvalue Ġof ĠIN LINE FORM 6 Ġenables ĠPR Us Ġto Ġlearn Ġthe Ġrepresentations Ġin Ġhigh Ġdimensional Ġspace Ġwith Ġbetter Ġgeneral iz ability Ġ( Table ĠT AB REF 23 Ġ). </s> Table ĠT AB REF 43 Ġshows Ġthe Ġimpact Ġof Ġdifferent Ġtransformations Ġof Ġthe Ġinput Ġvector ĠIN LINE FORM 0 Ġand Ġthe Ġcontext Ġvector ĠIN LINE FORM 1 Ġ. ĠWe Ġmake Ġfollowing Ġobservations : Ġ( 1 ) ĠUsing Ġthe Ġpy ram idal Ġtransformation Ġfor Ġthe Ġinput Ġvectors Ġimproves Ġthe Ġperplex ity Ġby Ġabout Ġ1 Ġpoint Ġon Ġboth Ġthe ĠPT B Ġand ĠWT - 2 Ġdatasets Ġwhile Ġreducing Ġthe Ġnumber Ġof Ġrecurrent Ġunit Ġparameters Ġby Ġabout Ġ14 % Ġ( see ĠR 1 Ġand ĠR 4 ). ĠWe Ġnote Ġthat Ġthe Ġperformance Ġof Ġthe ĠPR U Ġdrops Ġby Ġup Ġto Ġ1 Ġpoint Ġwhen Ġresidual Ġconnections Ġare Ġnot Ġused Ġ( R 4 Ġand ĠR 6 ). Ġ( 2 ) ĠUsing Ġthe Ġgrouped Ġlinear Ġtransformation Ġfor Ġcontext Ġvectors Ġreduces Ġthe Ġtotal Ġnumber Ġof Ġrecurrent Ġunit Ġparameters Ġby Ġabout Ġ75 % Ġwhile Ġthe Ġperformance Ġdrops Ġby Ġabout Ġ11 % Ġ( see ĠR 3 Ġand ĠR 4 ). ĠWhen Ġwe Ġuse Ġthe Ġpy ram idal Ġtransformation Ġinstead Ġof Ġthe Ġlinear Ġtransformation , Ġthe Ġperformance Ġdrops Ġby Ġup Ġto Ġ2 % Ġwhile Ġthere Ġis Ġno Ġsignificant Ġdrop Ġin Ġthe Ġnumber Ġof Ġparameters Ġ( R 4 Ġand ĠR 5 ). </s> We Ġset Ġsub - sam pling Ġkernel ĠIN LINE FORM 0 Ġ( E q . ĠEQ REF 12 Ġ) Ġwith Ġstride ĠIN LINE FORM 1 Ġand Ġsize Ġof Ġ3 Ġ( ĠIN LINE FORM 2 Ġ) Ġin Ġfour Ġdifferent Ġways : Ġ( 1 ) ĠSkip : ĠWe Ġskip Ġevery Ġother Ġelement Ġin Ġthe Ġinput Ġvector . Ġ( 2 ) ĠConv olution : ĠWe Ġinitialize Ġthe Ġelements Ġof ĠIN LINE FORM 3 Ġrandomly Ġfrom Ġnormal Ġdistribution Ġand Ġlearn Ġthem Ġduring Ġtraining Ġthe Ġmodel . ĠWe Ġlimit Ġthe Ġoutput Ġvalues Ġbetween Ġ- 1 Ġand Ġ1 Ġusing ĠIN LINE FORM 4 Ġactivation Ġfunction Ġto Ġmake Ġtraining Ġstable . Ġ( 3 ) ĠAvg . Ġpool : ĠWe Ġinitialize Ġthe Ġelements Ġof ĠIN LINE FORM 5 Ġto ĠIN LINE FORM 6 Ġ. Ġ( 4 ) ĠMax Ġpool : ĠWe Ġselect Ġthe Ġmaximum Ġvalue Ġin Ġthe Ġkernel Ġwindow ĠIN LINE FORM 7 Ġ. </s> Table ĠT AB REF 45 Ġcompares Ġthe Ġperformance Ġof Ġthe ĠPR U Ġwith Ġdifferent Ġsampling Ġmethods . ĠAverage Ġpool ing Ġperforms Ġthe Ġbest Ġwhile Ġskipping Ġgive Ġcomparable Ġperformance . ĠBoth Ġof Ġthese Ġmethods Ġenable Ġthe Ġnetwork Ġto Ġlearn Ġricher Ġword Ġrepresentations Ġwhile Ġrepresenting Ġthe Ġinput Ġvector Ġin Ġdifferent Ġforms , Ġthus Ġdelivering Ġhigher Ġperformance . ĠSurprisingly , Ġa Ġconv olution - based Ġsub - sam pling Ġmethod Ġdoes Ġnot Ġperform Ġas Ġwell Ġas Ġthe Ġaveraging Ġmethod . ĠThe ĠIN LINE FORM 0 Ġfunction Ġused Ġafter Ġconv olution Ġlimits Ġthe Ġrange Ġof Ġoutput Ġvalues Ġwhich Ġare Ġfurther Ġlimited Ġby Ġthe ĠL ST M Ġg ating Ġstructure , Ġthereby Ġimp eding Ġin Ġthe Ġflow Ġof Ġinformation Ġinside Ġthe Ġcell . ĠMax Ġpool ing Ġforces Ġthe Ġnetwork Ġto Ġlearn Ġrepresentations Ġfrom Ġhigh Ġmagnitude Ġelements , Ġthus Ġdistinguishing Ġfeatures Ġbetween Ġelements Ġvan ishes , Ġresulting Ġin Ġpoor Ġperformance . </s> Conclusion </s> We Ġintroduce Ġthe ĠPy ram idal ĠRec urrent ĠUnit , Ġwhich Ġbetter Ġmodel Ġcontextual Ġinformation Ġby Ġadmitting Ġhigher Ġdimensional Ġrepresentations Ġwith Ġgood Ġgeneral iz ability . ĠWhen Ġapplied Ġto Ġthe Ġtask Ġof Ġlanguage Ġmodeling , ĠPR Us Ġimprove Ġperplex ity Ġacross Ġseveral Ġsettings , Ġincluding Ġrecent Ġstate - of - the - art Ġsystems . ĠOur Ġanalysis Ġshows Ġthat Ġthe ĠPR U Ġimproves Ġthe Ġflow Ġof Ġgradient Ġand Ġexpand Ġthe Ġword Ġembed ding Ġsub space , Ġresulting Ġin Ġmore Ġconfident Ġdecisions . ĠHere Ġwe Ġhave Ġshown Ġimprovements Ġfor Ġlanguage Ġmodeling . ĠIn Ġfuture , Ġwe Ġplan Ġto Ġstudy Ġthe Ġperformance Ġof ĠPR Us Ġon Ġdifferent Ġtasks , Ġincluding Ġmachine Ġtranslation Ġand Ġquestion Ġanswering . ĠIn Ġaddition , Ġwe Ġwill Ġstudy Ġthe Ġperformance Ġof Ġthe ĠPR U Ġon Ġlanguage Ġmodeling Ġwith Ġmore Ġrecent Ġinference Ġtechniques , Ġsuch Ġas Ġdynamic Ġevaluation Ġand Ġmixture Ġof Ġsoft max . </s> Acknowled gments </s> This Ġresearch Ġwas Ġsupported Ġby ĠNS F Ġ( I IS Ġ16 16 112 , ĠIII Ġ17 03 166 ), ĠAllen ĠDist inguished ĠInvestigator ĠAward , Ġand Ġgifts Ġfrom ĠAllen ĠInstitute Ġfor ĠAI , ĠGoogle , ĠAmazon , Ġand ĠBloomberg . ĠWe Ġare Ġgrateful Ġto ĠAaron ĠJa ech , ĠHannah ĠRash kin , ĠMand ar ĠJosh i , ĠAn ir udd ha ĠK emb h avi , Ġand Ġanonymous Ġreviewers Ġfor Ġtheir Ġhelpful Ġcomments .
Introduction Long short term memory (LSTM) units BIBREF1 are popular for many sequence modeling tasks and are used extensively in language modeling. A key to their success is their articulated gating structure, which allows for more control over the information passed along the recurrence. However, despite the sophistication of the gating mechanisms employed in LSTMs and similar recurrent units, the input and context vectors are treated with simple linear transformations prior to gating. Non-linear transformations such as convolutions BIBREF2 have been used, but these have not achieved the performance of well regularized LSTMs for language modeling BIBREF3 . A natural way to improve the expressiveness of linear transformations is to increase the number of dimensions of the input and context vectors, but this comes with a significant increase in the number of parameters which may limit generalizability. An example is shown in Figure FIGREF1 , where LSTMs performance decreases with the increase in dimensions of the input and context vectors. Moreover, the semantics of the input and context vectors are different, suggesting that each may benefit from specialized treatment. Guided by these insights, we introduce a new recurrent unit, the Pyramidal Recurrent Unit (PRU), which is based on the LSTM gating structure. Figure FIGREF2 provides an overview of the PRU. At the heart of the PRU is the pyramidal transformation (PT), which uses subsampling to effect multiple views of the input vector. The subsampled representations are combined in a pyramidal fusion structure, resulting in richer interactions between the individual dimensions of the input vector than is possible with a linear transformation. Context vectors, which have already undergone this transformation in the previous cell, are modified with a grouped linear transformation (GLT) which allows the network to learn latent representations in high dimensional space with fewer parameters and better generalizability (see Figure FIGREF1 ). We show that PRUs can better model contextual information and demonstrate performance gains on the task of language modeling. The PRU improves the perplexity of the current state-of-the-art language model BIBREF0 by up to 1.3 points, reaching perplexities of 56.56 and 64.53 on the Penn Treebank and WikiText2 datasets while learning 15-20% fewer parameters. Replacing an LSTM with a PRU results in improvements in perplexity across a variety of experimental settings. We provide detailed ablations which motivate the design of the PRU architecture, as well as detailed analysis of the effect of the PRU on other components of the language model. Related work Multiple methods, including a variety of gating structures and transformations, have been proposed to improve the performance of recurrent neural networks (RNNs). We first describe these approaches and then provide an overview of recent work in language modeling. Pyramidal Recurrent Units We introduce Pyramidal Recurrent Units (PRUs), a new RNN architecture which improves modeling of context by allowing for higher dimensional vector representations while learning fewer parameters. Figure FIGREF2 provides an overview of PRU. We first elaborate on the details of the pyramidal transformation and the grouped linear transformation. We then describe our recurrent unit, PRU. Pyramidal transformation for input The basic transformation in many recurrent units is a linear transformation INLINEFORM0 defined as: DISPLAYFORM0 where INLINEFORM0 are learned weights that linearly map INLINEFORM1 to INLINEFORM2 . To simplify notation, we omit the biases. Motivated by successful applications of sub-sampling in computer vision (e.g., BIBREF22 , BIBREF23 , BIBREF9 , BIBREF24 ), we subsample input vector INLINEFORM0 into INLINEFORM1 pyramidal levels to achieve representation of the input vector at multiple scales. This sub-sampling operation produces INLINEFORM2 vectors, represented as INLINEFORM3 , where INLINEFORM4 is the sampling rate and INLINEFORM5 . We learn scale-specific transformations INLINEFORM6 for each INLINEFORM7 . The transformed subsamples are concatenated to produce the pyramidal analog to INLINEFORM8 , here denoted as INLINEFORM9 : DISPLAYFORM0 where INLINEFORM0 indicates concatenation. We note that pyramidal transformation with INLINEFORM1 is the same as the linear transformation. To improve gradient flow inside the recurrent unit, we combine the input and output using an element-wise sum (when dimension matches) to produce residual analog of pyramidal transformation, as shown in Figure FIGREF2 BIBREF25 . We sub-sample the input vector INLINEFORM0 into INLINEFORM1 pyramidal levels using the kernel-based approach BIBREF8 , BIBREF9 . Let us assume that we have a kernel INLINEFORM2 with INLINEFORM3 elements. Then, the input vector INLINEFORM4 can be sub-sampled as: DISPLAYFORM0 where INLINEFORM0 represents the stride and INLINEFORM1 . The number of parameters learned by the linear transformation and the pyramidal transformation with INLINEFORM0 pyramidal levels to map INLINEFORM1 to INLINEFORM2 are INLINEFORM3 and INLINEFORM4 respectively. Thus, pyramidal transformation reduces the parameters of a linear transformation by a factor of INLINEFORM5 . For example, the pyramidal transformation (with INLINEFORM6 and INLINEFORM7 ) learns INLINEFORM8 fewer parameters than the linear transformation. Grouped linear transformation for context Many RNN architectures apply linear transformations to both the input and context vector. However, this may not be ideal due to the differing semantics of each vector. In many NLP applications including language modeling, the input vector is a dense word embedding which is shared across all contexts for a given word in a dataset. In contrast, the context vector is highly contextualized by the current sequence. The differences between the input and context vector motivate their separate treatment in the PRU architecture. The weights learned using the linear transformation (Eq. EQREF9 ) are reused over multiple time steps, which makes them prone to over-fitting BIBREF26 . To combat over-fitting, various methods, such as variational dropout BIBREF26 and weight dropout BIBREF0 , have been proposed to regularize these recurrent connections. To further improve generalization abilities while simultaneously enabling the recurrent unit to learn representations at very high dimensional space, we propose to use grouped linear transformation (GLT) instead of standard linear transformation for recurrent connections BIBREF27 . While pyramidal and linear transformations can be applied to transform context vectors, our experimental results in Section SECREF39 suggests that GLTs are more effective. The linear transformation INLINEFORM0 maps INLINEFORM1 linearly to INLINEFORM2 . Grouped linear transformations break the linear interactions by factoring the linear transformation into two steps. First, a GLT splits the input vector INLINEFORM3 into INLINEFORM4 smaller groups such that INLINEFORM5 . Second, a linear transformation INLINEFORM6 is applied to map INLINEFORM7 linearly to INLINEFORM8 , for each INLINEFORM9 . The INLINEFORM10 resultant output vectors INLINEFORM11 are concatenated to produce the final output vector INLINEFORM12 . DISPLAYFORM0 GLTs learn representations at low dimensionality. Therefore, a GLT requires INLINEFORM0 fewer parameters than the linear transformation. We note that GLTs are subset of linear transformations. In a linear transformation, each neuron receives an input from each element in the input vector while in a GLT, each neuron receives an input from a subset of the input vector. Therefore, GLT is the same as a linear transformation when INLINEFORM1 . Pyramidal Recurrent Unit We extend the basic gating architecture of LSTM with the pyramidal and grouped linear transformations outlined above to produce the Pyramidal Recurrent Unit (PRU), whose improved sequence modeling capacity is evidenced in Section SECREF4 . At time INLINEFORM0 , the PRU combines the input vector INLINEFORM1 and the previous context vector (or previous hidden state vector) INLINEFORM2 using the following transformation function as: DISPLAYFORM0 where INLINEFORM0 indexes the various gates in the LSTM model, and INLINEFORM1 and INLINEFORM2 represents the pyramidal and grouped linear transformations defined in Eqns. EQREF10 and EQREF15 , respectively. We will now incorporate INLINEFORM0 into LSTM gating architecture to produce PRU. At time INLINEFORM1 , a PRU cell takes INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 as inputs to produce forget INLINEFORM5 , input INLINEFORM6 , output INLINEFORM7 , and content INLINEFORM8 gate signals. The inputs are combined with these gate signals to produce context vector INLINEFORM9 and cell state INLINEFORM10 . Mathematically, the PRU with the LSTM gating architecture can be defined as: DISPLAYFORM0 where INLINEFORM0 represents the element-wise multiplication operation, and INLINEFORM1 and INLINEFORM2 are the sigmoid and hyperbolic tangent activation functions. We note that LSTM is a special case of PRU when INLINEFORM3 = INLINEFORM4 =1. Experiments To showcase the effectiveness of the PRU, we evaluate the performance on two standard datasets for word-level language modeling and compare with state-of-the-art methods. Additionally, we provide a detailed examination of the PRU and its behavior on the language modeling tasks. Set-up Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20 . For both datasets, we follow the same training, validation, and test splits as in BIBREF0 . We extend the language model, AWD-LSTM BIBREF0 , by replacing LSTM layers with PRU. Our model uses 3-layers of PRU with an embedding size of 400. The number of parameters learned by state-of-the-art methods vary from 18M to 66M with majority of the methods learning about 22M to 24M parameters on the PTB dataset. For a fair comparison with state-of-the-art methods, we fix the model size to 19M and vary the value of INLINEFORM0 and hidden layer sizes so that total number of learned parameters is similar across different configurations. We use 1000, 1200, and 1400 as hidden layer sizes for values of INLINEFORM1 =1,2, and 4, respectively. We use the same settings for the WT-2 dataset. We set the number of pyramidal levels INLINEFORM2 to two in our experiments and use average pooling for sub-sampling. These values are selected based on our ablation experiments on the validation set (Section SECREF39 ). We measure the performance of our models in terms of word-level perplexity. We follow the same training strategy as in BIBREF0 . To understand the effect of regularization methods on the performance of PRUs, we perform experiments under two different settings: (1) Standard dropout: We use a standard dropout BIBREF12 with probability of 0.5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer. (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0 . We call this model as AWD-PRU. Results Table TABREF23 compares the performance of the PRU with state-of-the-art methods. We can see that the PRU achieves the best performance with fewer parameters. PRUs achieve either the same or better performance than LSTMs. In particular, the performance of PRUs improves with the increasing value of INLINEFORM0 . At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset. This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 ). With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters. On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs. With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset. This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets. For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e.g. RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1.58 points with 6M less parameters). With advanced dropouts, the PRU delivers the best performance. On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters. PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21 . When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62.42 to 55.23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66.29 to 58.79; suggesting that modern inference techniques are equally applicable to PRU-based language models. Analysis It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling. A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains. As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM. To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set. Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0.23 are used to effect categories. We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices. This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve. The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors. The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information. When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance. We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category. The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer. This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass. A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words. Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36 . These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation. Table TABREF34 visualizes the heatmaps for different sequences. PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions. Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision. This also suggests that PRUs update more parameters at each iteration which results in faster training. Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively. Ablation studies In this section, we provide a systematic analysis of our design choices. Our training methodology is the same as described in Section SECREF19 with the standard dropouts. For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600. The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets. The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1 . Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters. Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%. We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters. Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity. For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs. Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ). Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1 . We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4). We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6). (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4). When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5). We set sub-sampling kernel INLINEFORM0 (Eq. EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector. (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model. We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable. (3) Avg. pool: We initialize the elements of INLINEFORM5 to INLINEFORM6 . (4) Max pool: We select the maximum value in the kernel window INLINEFORM7 . Table TABREF45 compares the performance of the PRU with different sampling methods. Average pooling performs the best while skipping give comparable performance. Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance. Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method. The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell. Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance. Conclusion We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability. When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems. Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions. Here we have shown improvements for language modeling. In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering. In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax. Acknowledgments This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg. We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments.
what data did they use?
7bd6a6ec230e1efb27d691762cc0674237dc7967
what Ġdata Ġdid Ġthey Ġuse ?
ĠPenn ĠTree bank , ĠWiki Text 2
<s> what Ġdata Ġdid Ġthey Ġuse ? </s> Introduction </s> Long Ġshort Ġterm Ġmemory Ġ( L ST M ) Ġunits ĠB IB REF 1 Ġare Ġpopular Ġfor Ġmany Ġsequence Ġmodeling Ġtasks Ġand Ġare Ġused Ġextensively Ġin Ġlanguage Ġmodeling . ĠA Ġkey Ġto Ġtheir Ġsuccess Ġis Ġtheir Ġarticulated Ġg ating Ġstructure , Ġwhich Ġallows Ġfor Ġmore Ġcontrol Ġover Ġthe Ġinformation Ġpassed Ġalong Ġthe Ġrec urrence . ĠHowever , Ġdespite Ġthe Ġsophistication Ġof Ġthe Ġg ating Ġmechanisms Ġemployed Ġin ĠL ST Ms Ġand Ġsimilar Ġrecurrent Ġunits , Ġthe Ġinput Ġand Ġcontext Ġvectors Ġare Ġtreated Ġwith Ġsimple Ġlinear Ġtransformations Ġprior Ġto Ġg ating . ĠNon - linear Ġtransformations Ġsuch Ġas Ġconv olutions ĠB IB REF 2 Ġhave Ġbeen Ġused , Ġbut Ġthese Ġhave Ġnot Ġachieved Ġthe Ġperformance Ġof Ġwell Ġregular ized ĠL ST Ms Ġfor Ġlanguage Ġmodeling ĠB IB REF 3 Ġ. </s> A Ġnatural Ġway Ġto Ġimprove Ġthe Ġexpress iveness Ġof Ġlinear Ġtransformations Ġis Ġto Ġincrease Ġthe Ġnumber Ġof Ġdimensions Ġof Ġthe Ġinput Ġand Ġcontext Ġvectors , Ġbut Ġthis Ġcomes Ġwith Ġa Ġsignificant Ġincrease Ġin Ġthe Ġnumber Ġof Ġparameters Ġwhich Ġmay Ġlimit Ġgeneral iz ability . ĠAn Ġexample Ġis Ġshown Ġin ĠFigure ĠFIG REF 1 Ġ, Ġwhere ĠL ST Ms Ġperformance Ġdecreases Ġwith Ġthe Ġincrease Ġin Ġdimensions Ġof Ġthe Ġinput Ġand Ġcontext Ġvectors . ĠMoreover , Ġthe Ġsemantics Ġof Ġthe Ġinput Ġand Ġcontext Ġvectors Ġare Ġdifferent , Ġsuggesting Ġthat Ġeach Ġmay Ġbenefit Ġfrom Ġspecialized Ġtreatment . </s> Gu ided Ġby Ġthese Ġinsights , Ġwe Ġintroduce Ġa Ġnew Ġrecurrent Ġunit , Ġthe ĠPy ram idal ĠRec urrent ĠUnit Ġ( PR U ), Ġwhich Ġis Ġbased Ġon Ġthe ĠL ST M Ġg ating Ġstructure . ĠFigure ĠFIG REF 2 Ġprovides Ġan Ġoverview Ġof Ġthe ĠPR U . ĠAt Ġthe Ġheart Ġof Ġthe ĠPR U Ġis Ġthe Ġpy ram idal Ġtransformation Ġ( PT ), Ġwhich Ġuses Ġsubs am pling Ġto Ġeffect Ġmultiple Ġviews Ġof Ġthe Ġinput Ġvector . ĠThe Ġsubs am pled Ġrepresentations Ġare Ġcombined Ġin Ġa Ġpy ram idal Ġfusion Ġstructure , Ġresulting Ġin Ġricher Ġinteractions Ġbetween Ġthe Ġindividual Ġdimensions Ġof Ġthe Ġinput Ġvector Ġthan Ġis Ġpossible Ġwith Ġa Ġlinear Ġtransformation . ĠContext Ġvectors , Ġwhich Ġhave Ġalready Ġundergone Ġthis Ġtransformation Ġin Ġthe Ġprevious Ġcell , Ġare Ġmodified Ġwith Ġa Ġgrouped Ġlinear Ġtransformation Ġ( GL T ) Ġwhich Ġallows Ġthe Ġnetwork Ġto Ġlearn Ġlatent Ġrepresentations Ġin Ġhigh Ġdimensional Ġspace Ġwith Ġfewer Ġparameters Ġand Ġbetter Ġgeneral iz ability Ġ( see ĠFigure ĠFIG REF 1 Ġ). </s> We Ġshow Ġthat ĠPR Us Ġcan Ġbetter Ġmodel Ġcontextual Ġinformation Ġand Ġdemonstrate Ġperformance Ġgains Ġon Ġthe Ġtask Ġof Ġlanguage Ġmodeling . ĠThe ĠPR U Ġimproves Ġthe Ġperplex ity Ġof Ġthe Ġcurrent Ġstate - of - the - art Ġlanguage Ġmodel ĠB IB REF 0 Ġby Ġup Ġto Ġ1 . 3 Ġpoints , Ġreaching Ġperplex ities Ġof Ġ56 . 56 Ġand Ġ64 . 53 Ġon Ġthe ĠPenn ĠTree bank Ġand ĠWiki Text 2 Ġdatasets Ġwhile Ġlearning Ġ15 - 20 % Ġfewer Ġparameters . ĠRepl acing Ġan ĠL ST M Ġwith Ġa ĠPR U Ġresults Ġin Ġimprovements Ġin Ġperplex ity Ġacross Ġa Ġvariety Ġof Ġexperimental Ġsettings . ĠWe Ġprovide Ġdetailed Ġabl ations Ġwhich Ġmotivate Ġthe Ġdesign Ġof Ġthe ĠPR U Ġarchitecture , Ġas Ġwell Ġas Ġdetailed Ġanalysis Ġof Ġthe Ġeffect Ġof Ġthe ĠPR U Ġon Ġother Ġcomponents Ġof Ġthe Ġlanguage Ġmodel . </s> Related Ġwork </s> Multiple Ġmethods , Ġincluding Ġa Ġvariety Ġof Ġg ating Ġstructures Ġand Ġtransformations , Ġhave Ġbeen Ġproposed Ġto Ġimprove Ġthe Ġperformance Ġof Ġrecurrent Ġneural Ġnetworks Ġ( R NN s ). ĠWe Ġfirst Ġdescribe Ġthese Ġapproaches Ġand Ġthen Ġprovide Ġan Ġoverview Ġof Ġrecent Ġwork Ġin Ġlanguage Ġmodeling . </s> Py ram idal ĠRec urrent ĠUnits </s> We Ġintroduce ĠPy ram idal ĠRec urrent ĠUnits Ġ( PR Us ), Ġa Ġnew ĠR NN Ġarchitecture Ġwhich Ġimproves Ġmodeling Ġof Ġcontext Ġby Ġallowing Ġfor Ġhigher Ġdimensional Ġvector Ġrepresentations Ġwhile Ġlearning Ġfewer Ġparameters . ĠFigure ĠFIG REF 2 Ġprovides Ġan Ġoverview Ġof ĠPR U . ĠWe Ġfirst Ġelaborate Ġon Ġthe Ġdetails Ġof Ġthe Ġpy ram idal Ġtransformation Ġand Ġthe Ġgrouped Ġlinear Ġtransformation . ĠWe Ġthen Ġdescribe Ġour Ġrecurrent Ġunit , ĠPR U . </s> Py ram idal Ġtransformation Ġfor Ġinput </s> The Ġbasic Ġtransformation Ġin Ġmany Ġrecurrent Ġunits Ġis Ġa Ġlinear Ġtransformation ĠIN LINE FORM 0 Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġare Ġlearned Ġweights Ġthat Ġlin early Ġmap ĠIN LINE FORM 1 Ġto ĠIN LINE FORM 2 Ġ. ĠTo Ġsimplify Ġnotation , Ġwe Ġomit Ġthe Ġbiases . </s> Mot ivated Ġby Ġsuccessful Ġapplications Ġof Ġsub - sam pling Ġin Ġcomputer Ġvision Ġ( e . g ., ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 24 Ġ), Ġwe Ġsubs ample Ġinput Ġvector ĠIN LINE FORM 0 Ġinto ĠIN LINE FORM 1 Ġpy ram idal Ġlevels Ġto Ġachieve Ġrepresentation Ġof Ġthe Ġinput Ġvector Ġat Ġmultiple Ġscales . ĠThis Ġsub - sam pling Ġoperation Ġproduces ĠIN LINE FORM 2 Ġvectors , Ġrepresented Ġas ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġsampling Ġrate Ġand ĠIN LINE FORM 5 Ġ. ĠWe Ġlearn Ġscale - specific Ġtransformations ĠIN LINE FORM 6 Ġfor Ġeach ĠIN LINE FORM 7 Ġ. ĠThe Ġtransformed Ġsubs amples Ġare Ġconc aten ated Ġto Ġproduce Ġthe Ġpy ram idal Ġanalog Ġto ĠIN LINE FORM 8 Ġ, Ġhere Ġden oted Ġas ĠIN LINE FORM 9 Ġ: ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġindicates Ġconc aten ation . ĠWe Ġnote Ġthat Ġpy ram idal Ġtransformation Ġwith ĠIN LINE FORM 1 Ġis Ġthe Ġsame Ġas Ġthe Ġlinear Ġtransformation . </s> To Ġimprove Ġgradient Ġflow Ġinside Ġthe Ġrecurrent Ġunit , Ġwe Ġcombine Ġthe Ġinput Ġand Ġoutput Ġusing Ġan Ġelement - wise Ġsum Ġ( when Ġdimension Ġmatches ) Ġto Ġproduce Ġresidual Ġanalog Ġof Ġpy ram idal Ġtransformation , Ġas Ġshown Ġin ĠFigure ĠFIG REF 2 ĠB IB REF 25 Ġ. </s> We Ġsub - sample Ġthe Ġinput Ġvector ĠIN LINE FORM 0 Ġinto ĠIN LINE FORM 1 Ġpy ram idal Ġlevels Ġusing Ġthe Ġkernel - based Ġapproach ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠLet Ġus Ġassume Ġthat Ġwe Ġhave Ġa Ġkernel ĠIN LINE FORM 2 Ġwith ĠIN LINE FORM 3 Ġelements . ĠThen , Ġthe Ġinput Ġvector ĠIN LINE FORM 4 Ġcan Ġbe Ġsub - sam pled Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġstride Ġand ĠIN LINE FORM 1 Ġ. </s> The Ġnumber Ġof Ġparameters Ġlearned Ġby Ġthe Ġlinear Ġtransformation Ġand Ġthe Ġpy ram idal Ġtransformation Ġwith ĠIN LINE FORM 0 Ġpy ram idal Ġlevels Ġto Ġmap ĠIN LINE FORM 1 Ġto ĠIN LINE FORM 2 Ġare ĠIN LINE FORM 3 Ġand ĠIN LINE FORM 4 Ġrespectively . ĠThus , Ġpy ram idal Ġtransformation Ġreduces Ġthe Ġparameters Ġof Ġa Ġlinear Ġtransformation Ġby Ġa Ġfactor Ġof ĠIN LINE FORM 5 Ġ. ĠFor Ġexample , Ġthe Ġpy ram idal Ġtransformation Ġ( with ĠIN LINE FORM 6 Ġand ĠIN LINE FORM 7 Ġ) Ġlearns ĠIN LINE FORM 8 Ġfewer Ġparameters Ġthan Ġthe Ġlinear Ġtransformation . </s> Group ed Ġlinear Ġtransformation Ġfor Ġcontext </s> Many ĠR NN Ġarchitectures Ġapply Ġlinear Ġtransformations Ġto Ġboth Ġthe Ġinput Ġand Ġcontext Ġvector . ĠHowever , Ġthis Ġmay Ġnot Ġbe Ġideal Ġdue Ġto Ġthe Ġdiffering Ġsemantics Ġof Ġeach Ġvector . ĠIn Ġmany ĠN LP Ġapplications Ġincluding Ġlanguage Ġmodeling , Ġthe Ġinput Ġvector Ġis Ġa Ġdense Ġword Ġembed ding Ġwhich Ġis Ġshared Ġacross Ġall Ġcontexts Ġfor Ġa Ġgiven Ġword Ġin Ġa Ġdataset . ĠIn Ġcontrast , Ġthe Ġcontext Ġvector Ġis Ġhighly Ġcontextual ized Ġby Ġthe Ġcurrent Ġsequence . ĠThe Ġdifferences Ġbetween Ġthe Ġinput Ġand Ġcontext Ġvector Ġmotivate Ġtheir Ġseparate Ġtreatment Ġin Ġthe ĠPR U Ġarchitecture . </s> The Ġweights Ġlearned Ġusing Ġthe Ġlinear Ġtransformation Ġ( E q . ĠEQ REF 9 Ġ) Ġare Ġreused Ġover Ġmultiple Ġtime Ġsteps , Ġwhich Ġmakes Ġthem Ġprone Ġto Ġover - fitting ĠB IB REF 26 Ġ. ĠTo Ġcombat Ġover - fitting , Ġvarious Ġmethods , Ġsuch Ġas Ġvari ational Ġdrop out ĠB IB REF 26 Ġand Ġweight Ġdrop out ĠB IB REF 0 Ġ, Ġhave Ġbeen Ġproposed Ġto Ġregular ize Ġthese Ġrecurrent Ġconnections . ĠTo Ġfurther Ġimprove Ġgeneral ization Ġabilities Ġwhile Ġsimultaneously Ġenabling Ġthe Ġrecurrent Ġunit Ġto Ġlearn Ġrepresentations Ġat Ġvery Ġhigh Ġdimensional Ġspace , Ġwe Ġpropose Ġto Ġuse Ġgrouped Ġlinear Ġtransformation Ġ( GL T ) Ġinstead Ġof Ġstandard Ġlinear Ġtransformation Ġfor Ġrecurrent Ġconnections ĠB IB REF 27 Ġ. ĠWhile Ġpy ram idal Ġand Ġlinear Ġtransformations Ġcan Ġbe Ġapplied Ġto Ġtransform Ġcontext Ġvectors , Ġour Ġexperimental Ġresults Ġin ĠSection ĠSEC REF 39 Ġsuggests Ġthat ĠGL Ts Ġare Ġmore Ġeffective . </s> The Ġlinear Ġtransformation ĠIN LINE FORM 0 Ġmaps ĠIN LINE FORM 1 Ġlin early Ġto ĠIN LINE FORM 2 Ġ. ĠGroup ed Ġlinear Ġtransformations Ġbreak Ġthe Ġlinear Ġinteractions Ġby Ġfact oring Ġthe Ġlinear Ġtransformation Ġinto Ġtwo Ġsteps . ĠFirst , Ġa ĠGL T Ġsplits Ġthe Ġinput Ġvector ĠIN LINE FORM 3 Ġinto ĠIN LINE FORM 4 Ġsmaller Ġgroups Ġsuch Ġthat ĠIN LINE FORM 5 Ġ. ĠSecond , Ġa Ġlinear Ġtransformation ĠIN LINE FORM 6 Ġis Ġapplied Ġto Ġmap ĠIN LINE FORM 7 Ġlin early Ġto ĠIN LINE FORM 8 Ġ, Ġfor Ġeach ĠIN LINE FORM 9 Ġ. ĠThe ĠIN LINE FORM 10 Ġresultant Ġoutput Ġvectors ĠIN LINE FORM 11 Ġare Ġconc aten ated Ġto Ġproduce Ġthe Ġfinal Ġoutput Ġvector ĠIN LINE FORM 12 Ġ. ĠDIS PLAY FORM 0 </s> GL Ts Ġlearn Ġrepresentations Ġat Ġlow Ġdimension ality . ĠTherefore , Ġa ĠGL T Ġrequires ĠIN LINE FORM 0 Ġfewer Ġparameters Ġthan Ġthe Ġlinear Ġtransformation . ĠWe Ġnote Ġthat ĠGL Ts Ġare Ġsubset Ġof Ġlinear Ġtransformations . ĠIn Ġa Ġlinear Ġtransformation , Ġeach Ġneuron Ġreceives Ġan Ġinput Ġfrom Ġeach Ġelement Ġin Ġthe Ġinput Ġvector Ġwhile Ġin Ġa ĠGL T , Ġeach Ġneuron Ġreceives Ġan Ġinput Ġfrom Ġa Ġsubset Ġof Ġthe Ġinput Ġvector . ĠTherefore , ĠGL T Ġis Ġthe Ġsame Ġas Ġa Ġlinear Ġtransformation Ġwhen ĠIN LINE FORM 1 Ġ. </s> Py ram idal ĠRec urrent ĠUnit </s> We Ġextend Ġthe Ġbasic Ġg ating Ġarchitecture Ġof ĠL ST M Ġwith Ġthe Ġpy ram idal Ġand Ġgrouped Ġlinear Ġtransformations Ġoutlined Ġabove Ġto Ġproduce Ġthe ĠPy ram idal ĠRec urrent ĠUnit Ġ( PR U ), Ġwhose Ġimproved Ġsequence Ġmodeling Ġcapacity Ġis Ġevidenced Ġin ĠSection ĠSEC REF 4 Ġ. </s> At Ġtime ĠIN LINE FORM 0 Ġ, Ġthe ĠPR U Ġcombines Ġthe Ġinput Ġvector ĠIN LINE FORM 1 Ġand Ġthe Ġprevious Ġcontext Ġvector Ġ( or Ġprevious Ġhidden Ġstate Ġvector ) ĠIN LINE FORM 2 Ġusing Ġthe Ġfollowing Ġtransformation Ġfunction Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġindexes Ġthe Ġvarious Ġgates Ġin Ġthe ĠL ST M Ġmodel , Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġrepresents Ġthe Ġpy ram idal Ġand Ġgrouped Ġlinear Ġtransformations Ġdefined Ġin ĠE q ns . ĠEQ REF 10 Ġand ĠEQ REF 15 Ġ, Ġrespectively . </s> We Ġwill Ġnow Ġincorporate ĠIN LINE FORM 0 Ġinto ĠL ST M Ġg ating Ġarchitecture Ġto Ġproduce ĠPR U . ĠAt Ġtime ĠIN LINE FORM 1 Ġ, Ġa ĠPR U Ġcell Ġtakes ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġas Ġinputs Ġto Ġproduce Ġforget ĠIN LINE FORM 5 Ġ, Ġinput ĠIN LINE FORM 6 Ġ, Ġoutput ĠIN LINE FORM 7 Ġ, Ġand Ġcontent ĠIN LINE FORM 8 Ġgate Ġsignals . ĠThe Ġinputs Ġare Ġcombined Ġwith Ġthese Ġgate Ġsignals Ġto Ġproduce Ġcontext Ġvector ĠIN LINE FORM 9 Ġand Ġcell Ġstate ĠIN LINE FORM 10 Ġ. ĠMat hematically , Ġthe ĠPR U Ġwith Ġthe ĠL ST M Ġg ating Ġarchitecture Ġcan Ġbe Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġelement - wise Ġmultiplication Ġoperation , Ġand ĠIN LINE FORM 1 Ġand ĠIN LINE FORM 2 Ġare Ġthe Ġs igm oid Ġand Ġhyper b olic Ġtang ent Ġactivation Ġfunctions . ĠWe Ġnote Ġthat ĠL ST M Ġis Ġa Ġspecial Ġcase Ġof ĠPR U Ġwhen ĠIN LINE FORM 3 Ġ= ĠIN LINE FORM 4 Ġ= 1 . </s> Exper iments </s> To Ġshowcase Ġthe Ġeffectiveness Ġof Ġthe ĠPR U , Ġwe Ġevaluate Ġthe Ġperformance Ġon Ġtwo Ġstandard Ġdatasets Ġfor Ġword - level Ġlanguage Ġmodeling Ġand Ġcompare Ġwith Ġstate - of - the - art Ġmethods . ĠAdditionally , Ġwe Ġprovide Ġa Ġdetailed Ġexamination Ġof Ġthe ĠPR U Ġand Ġits Ġbehavior Ġon Ġthe Ġlanguage Ġmodeling Ġtasks . </s> Set - up </s> Following Ġrecent Ġworks , Ġwe Ġcompare Ġon Ġtwo Ġwidely Ġused Ġdatasets , Ġthe ĠPenn ĠTree bank Ġ( PT B ) ĠB IB REF 28 Ġas Ġprepared Ġby ĠB IB REF 29 Ġand ĠWiki Text 2 Ġ( WT - 2 ) ĠB IB REF 20 Ġ. ĠFor Ġboth Ġdatasets , Ġwe Ġfollow Ġthe Ġsame Ġtraining , Ġvalidation , Ġand Ġtest Ġsplits Ġas Ġin ĠB IB REF 0 Ġ. </s> We Ġextend Ġthe Ġlanguage Ġmodel , ĠAW D - L ST M ĠB IB REF 0 Ġ, Ġby Ġreplacing ĠL ST M Ġlayers Ġwith ĠPR U . ĠOur Ġmodel Ġuses Ġ3 - l ayers Ġof ĠPR U Ġwith Ġan Ġembed ding Ġsize Ġof Ġ400 . ĠThe Ġnumber Ġof Ġparameters Ġlearned Ġby Ġstate - of - the - art Ġmethods Ġvary Ġfrom Ġ18 M Ġto Ġ66 M Ġwith Ġmajority Ġof Ġthe Ġmethods Ġlearning Ġabout Ġ22 M Ġto Ġ24 M Ġparameters Ġon Ġthe ĠPT B Ġdataset . ĠFor Ġa Ġfair Ġcomparison Ġwith Ġstate - of - the - art Ġmethods , Ġwe Ġfix Ġthe Ġmodel Ġsize Ġto Ġ19 M Ġand Ġvary Ġthe Ġvalue Ġof ĠIN LINE FORM 0 Ġand Ġhidden Ġlayer Ġsizes Ġso Ġthat Ġtotal Ġnumber Ġof Ġlearned Ġparameters Ġis Ġsimilar Ġacross Ġdifferent Ġconfigurations . ĠWe Ġuse Ġ1000 , Ġ1200 , Ġand Ġ1400 Ġas Ġhidden Ġlayer Ġsizes Ġfor Ġvalues Ġof ĠIN LINE FORM 1 Ġ= 1 , 2 , Ġand Ġ4 , Ġrespectively . ĠWe Ġuse Ġthe Ġsame Ġsettings Ġfor Ġthe ĠWT - 2 Ġdataset . ĠWe Ġset Ġthe Ġnumber Ġof Ġpy ram idal Ġlevels ĠIN LINE FORM 2 Ġto Ġtwo Ġin Ġour Ġexperiments Ġand Ġuse Ġaverage Ġpool ing Ġfor Ġsub - sam pling . ĠThese Ġvalues Ġare Ġselected Ġbased Ġon Ġour Ġab lation Ġexperiments Ġon Ġthe Ġvalidation Ġset Ġ( Section ĠSEC REF 39 Ġ). ĠWe Ġmeasure Ġthe Ġperformance Ġof Ġour Ġmodels Ġin Ġterms Ġof Ġword - level Ġperplex ity . ĠWe Ġfollow Ġthe Ġsame Ġtraining Ġstrategy Ġas Ġin ĠB IB REF 0 Ġ. </s> To Ġunderstand Ġthe Ġeffect Ġof Ġregular ization Ġmethods Ġon Ġthe Ġperformance Ġof ĠPR Us , Ġwe Ġperform Ġexperiments Ġunder Ġtwo Ġdifferent Ġsettings : Ġ( 1 ) ĠStandard Ġdrop out : ĠWe Ġuse Ġa Ġstandard Ġdrop out ĠB IB REF 12 Ġwith Ġprobability Ġof Ġ0 . 5 Ġafter Ġembed ding Ġlayer , Ġthe Ġoutput Ġbetween ĠL ST M Ġlayers , Ġand Ġthe Ġoutput Ġof Ġfinal ĠL ST M Ġlayer . Ġ( 2 ) ĠAdvanced Ġdrop out : ĠWe Ġuse Ġthe Ġsame Ġdrop out Ġtechniques Ġwith Ġthe Ġsame Ġdrop out Ġvalues Ġas Ġin ĠB IB REF 0 Ġ. ĠWe Ġcall Ġthis Ġmodel Ġas ĠAW D - PR U . </s> Results </s> Table ĠT AB REF 23 Ġcompares Ġthe Ġperformance Ġof Ġthe ĠPR U Ġwith Ġstate - of - the - art Ġmethods . ĠWe Ġcan Ġsee Ġthat Ġthe ĠPR U Ġachieves Ġthe Ġbest Ġperformance Ġwith Ġfewer Ġparameters . </s> PR Us Ġachieve Ġeither Ġthe Ġsame Ġor Ġbetter Ġperformance Ġthan ĠL ST Ms . ĠIn Ġparticular , Ġthe Ġperformance Ġof ĠPR Us Ġimproves Ġwith Ġthe Ġincreasing Ġvalue Ġof ĠIN LINE FORM 0 Ġ. ĠAt ĠIN LINE FORM 1 Ġ, ĠPR Us Ġoutper form ĠL ST Ms Ġby Ġabout Ġ4 Ġpoints Ġon Ġthe ĠPT B Ġdataset Ġand Ġby Ġabout Ġ3 Ġpoints Ġon Ġthe ĠWT - 2 Ġdataset . ĠThis Ġis Ġexplained Ġin Ġpart Ġby Ġthe Ġregular ization Ġeffect Ġof Ġthe Ġgrouped Ġlinear Ġtransformation Ġ( Figure ĠFIG REF 1 Ġ). ĠWith Ġgrouped Ġlinear Ġand Ġpy ram idal Ġtransformations , ĠPR Us Ġlearn Ġrich Ġrepresentations Ġat Ġvery Ġhigh Ġdimensional Ġspace Ġwhile Ġlearning Ġfewer Ġparameters . ĠOn Ġthe Ġother Ġhand , ĠL ST Ms Ġover fit Ġto Ġthe Ġtraining Ġdata Ġat Ġsuch Ġhigh Ġdimensions Ġand Ġlearn ĠIN LINE FORM 2 Ġto ĠIN LINE FORM 3 Ġmore Ġparameters Ġthan ĠPR Us . </s> With Ġthe Ġadvanced Ġdrop outs , Ġthe Ġperformance Ġof ĠPR Us Ġimproves Ġby Ġabout Ġ4 Ġpoints Ġon Ġthe ĠPT B Ġdataset Ġand Ġ7 Ġpoints Ġon Ġthe ĠWT - 2 Ġdataset . ĠThis Ġfurther Ġimproves Ġwith Ġfin et uning Ġon Ġthe ĠPT B Ġ( about Ġ2 Ġpoints ) Ġand ĠWT - 2 Ġ( about Ġ1 Ġpoint ) Ġdatasets . </s> For Ġsimilar Ġnumber Ġof Ġparameters , Ġthe ĠPR U Ġwith Ġstandard Ġdrop out Ġoutper forms Ġmost Ġof Ġthe Ġstate - of - the - art Ġmethods Ġby Ġlarge Ġmargin Ġon Ġthe ĠPT B Ġdataset Ġ( e . g . ĠR AN ĠB IB REF 7 Ġby Ġ16 Ġpoints Ġwith Ġ4 M Ġless Ġparameters , ĠQR NN ĠB IB REF 33 Ġby Ġ16 Ġpoints Ġwith Ġ1 M Ġmore Ġparameters , Ġand ĠNAS ĠB IB REF 31 Ġby Ġ1 . 58 Ġpoints Ġwith Ġ6 M Ġless Ġparameters ). ĠWith Ġadvanced Ġdrop outs , Ġthe ĠPR U Ġdelivers Ġthe Ġbest Ġperformance . ĠOn Ġboth Ġdatasets , Ġthe ĠPR U Ġimproves Ġthe Ġperplex ity Ġby Ġabout Ġ1 Ġpoint Ġwhile Ġlearning Ġ15 - 20 % Ġfewer Ġparameters . </s> PR U Ġis Ġa Ġdrop - in Ġreplacement Ġfor ĠL ST M , Ġtherefore , Ġit Ġcan Ġimprove Ġlanguage Ġmodels Ġwith Ġmodern Ġinference Ġtechniques Ġsuch Ġas Ġdynamic Ġevaluation ĠB IB REF 21 Ġ. ĠWhen Ġwe Ġevaluate ĠPR U - based Ġlanguage Ġmodels Ġ( only Ġwith Ġstandard Ġdrop out ) Ġwith Ġdynamic Ġevaluation Ġon Ġthe ĠPT B Ġtest Ġset , Ġthe Ġperplex ity Ġof ĠPR U Ġ( ĠIN LINE FORM 0 Ġ) Ġimproves Ġfrom Ġ62 . 42 Ġto Ġ55 . 23 Ġwhile Ġthe Ġperplex ity Ġof Ġan ĠL ST M Ġ( ĠIN LINE FORM 1 Ġ) Ġwith Ġsimilar Ġsettings Ġimproves Ġfrom Ġ66 . 29 Ġto Ġ58 . 79 ; Ġsuggesting Ġthat Ġmodern Ġinference Ġtechniques Ġare Ġequally Ġapplicable Ġto ĠPR U - based Ġlanguage Ġmodels . </s> Analysis </s> It Ġis Ġshown Ġabove Ġthat Ġthe ĠPR U Ġcan Ġlearn Ġrepresentations Ġat Ġhigher Ġdimension ality Ġwith Ġmore Ġgeneral ization Ġpower , Ġresulting Ġin Ġperformance Ġgains Ġfor Ġlanguage Ġmodeling . ĠA Ġcloser Ġanalysis Ġof Ġthe Ġimpact Ġof Ġthe ĠPR U Ġin Ġa Ġlanguage Ġmodeling Ġsystem Ġreveals Ġseveral Ġfactors Ġthat Ġhelp Ġexplain Ġhow Ġthe ĠPR U Ġachieves Ġthese Ġgains . </s> As Ġexempl ified Ġin ĠTable ĠT AB REF 34 Ġ, Ġthe ĠPR U Ġtends Ġtoward Ġmore Ġconfident Ġdecisions , Ġplacing Ġmore Ġof Ġthe Ġprobability Ġmass Ġon Ġthe Ġtop Ġnext - word Ġprediction Ġthan Ġthe ĠL ST M . ĠTo Ġquantify Ġthis Ġeffect , Ġwe Ġcalculate Ġthe Ġentropy Ġof Ġthe Ġnext - token Ġdistribution Ġfor Ġboth Ġthe ĠPR U Ġand Ġthe ĠL ST M Ġusing Ġ3 687 Ġcontexts Ġfrom Ġthe ĠPT B Ġvalidation Ġset . ĠFigure ĠFIG REF 32 Ġshows Ġa Ġhist ogram Ġof Ġthe Ġent rop ies Ġof Ġthe Ġdistribution , Ġwhere Ġbins Ġof Ġsize Ġ0 . 23 Ġare Ġused Ġto Ġeffect Ġcategories . ĠWe Ġsee Ġthat Ġthe ĠPR U Ġmore Ġoften Ġproduces Ġlower Ġentropy Ġdistributions Ġcorresponding Ġto Ġhigher Ġconf idences Ġfor Ġnext - token Ġchoices . ĠThis Ġis Ġevidenced Ġby Ġthe Ġmass Ġof Ġthe Ġred ĠPR U Ġcurve Ġlying Ġin Ġthe Ġlower Ġentropy Ġranges Ġcompared Ġto Ġthe Ġblue ĠL ST M 's Ġcurve . ĠThe ĠPR U Ġcan Ġproduce Ġconfident Ġdecisions Ġin Ġpart Ġbecause Ġmore Ġinformation Ġis Ġencoded Ġin Ġthe Ġhigher Ġdimensional Ġcontext Ġvectors . </s> The ĠPR U Ġhas Ġthe Ġability Ġto Ġmodel Ġindividual Ġwords Ġat Ġdifferent Ġresolutions Ġthrough Ġthe Ġpy ram idal Ġtransform ; Ġwhich Ġprovides Ġmultiple Ġpaths Ġfor Ġthe Ġgradient Ġto Ġthe Ġembed ding Ġlayer Ġ( similar Ġto Ġmulti - task Ġlearning ) Ġand Ġimproves Ġthe Ġflow Ġof Ġinformation . ĠWhen Ġconsidering Ġthe Ġembed d ings Ġby Ġpart Ġof Ġspeech , Ġwe Ġfind Ġthat Ġthe Ġpyramid Ġlevel Ġ1 Ġembed d ings Ġexhibit Ġhigher Ġvariance Ġthan Ġthe ĠL ST M Ġacross Ġall ĠPOS Ġcategories Ġ( Figure ĠFIG REF 33 Ġ), Ġand Ġthat Ġpyramid Ġlevel Ġ2 Ġembed d ings Ġshow Ġextremely Ġlow Ġvariance . ĠWe Ġhypothes ize Ġthat Ġthe ĠL ST M Ġmust Ġencode Ġboth Ġcoarse Ġgroup Ġsimilarities Ġand Ġindividual Ġword Ġdifferences Ġinto Ġthe Ġsame Ġvector Ġspace , Ġreducing Ġthe Ġspace Ġbetween Ġindividual Ġwords Ġof Ġthe Ġsame Ġcategory . ĠThe ĠPR U Ġcan Ġrely Ġon Ġthe Ġsubs am pled Ġembed d ings Ġto Ġaccount Ġfor Ġcoarse - gr ained Ġgroup Ġsimilarities , Ġallowing Ġfor Ġfiner Ġindividual Ġword Ġdistinctions Ġin Ġthe Ġembed ding Ġlayer . ĠThis Ġhypothesis Ġis Ġstrengthened Ġby Ġthe Ġentropy Ġresults Ġdescribed Ġabove : Ġa Ġmodel Ġwhich Ġcan Ġmake Ġfiner Ġdistinctions Ġbetween Ġindividual Ġwords Ġcan Ġmore Ġconfidently Ġassign Ġprobability Ġmass . ĠA Ġmodel Ġthat Ġcannot Ġmake Ġthese Ġdistinctions , Ġsuch Ġas Ġthe ĠL ST M , Ġmust Ġspread Ġits Ġprobability Ġmass Ġacross Ġa Ġlarger Ġclass Ġof Ġsimilar Ġwords . </s> Sal iency Ġanalysis Ġusing Ġgrad ients Ġhelp Ġidentify Ġrelevant Ġwords Ġin Ġa Ġtest Ġsequence Ġthat Ġcontribute Ġto Ġthe Ġprediction ĠB IB REF 34 Ġ, ĠB IB REF 35 Ġ, ĠB IB REF 36 Ġ. ĠThese Ġapproaches Ġcompute Ġthe Ġrelevance Ġas Ġthe Ġsquared Ġnorm Ġof Ġthe Ġgrad ients Ġobtained Ġthrough Ġback - prop ag ation . ĠTable ĠT AB REF 34 Ġvisual izes Ġthe Ġheat maps Ġfor Ġdifferent Ġsequences . ĠPR Us , Ġin Ġgeneral , Ġgive Ġmore Ġrelevance Ġto Ġcontextual Ġwords Ġthan ĠL ST Ms , Ġsuch Ġas Ġsoutheast Ġ( sample Ġ1 ), Ġcost Ġ( sample Ġ2 ), Ġface Ġ( sample Ġ4 ), Ġand Ġintroduced Ġ( sample Ġ5 ), Ġwhich Ġhelp Ġin Ġmaking Ġmore Ġconfident Ġdecisions . ĠFurthermore , Ġwhen Ġgrad ients Ġduring Ġback - prop ag ation Ġare Ġvisual ized ĠB IB REF 37 Ġ( Table ĠT AB REF 34 Ġ), Ġwe Ġfind Ġthat ĠPR Us Ġhave Ġbetter Ġgradient Ġcoverage Ġthan ĠL ST Ms , Ġsuggesting ĠPR Us Ġuse Ġmore Ġfeatures Ġthan ĠL ST Ms Ġthat Ġcontributes Ġto Ġthe Ġdecision . ĠThis Ġalso Ġsuggests Ġthat ĠPR Us Ġupdate Ġmore Ġparameters Ġat Ġeach Ġiteration Ġwhich Ġresults Ġin Ġfaster Ġtraining . ĠLanguage Ġmodel Ġin ĠB IB REF 0 Ġtakes Ġ500 Ġand Ġ750 Ġepoch s Ġto Ġconverge Ġwith ĠPR U Ġand ĠL ST M Ġas Ġa Ġrecurrent Ġunit , Ġrespectively . </s> A bl ation Ġstudies </s> In Ġthis Ġsection , Ġwe Ġprovide Ġa Ġsystematic Ġanalysis Ġof Ġour Ġdesign Ġchoices . ĠOur Ġtraining Ġmethodology Ġis Ġthe Ġsame Ġas Ġdescribed Ġin ĠSection ĠSEC REF 19 Ġwith Ġthe Ġstandard Ġdrop outs . ĠFor Ġa Ġthorough Ġunderstanding Ġof Ġour Ġdesign Ġchoices , Ġwe Ġuse Ġa Ġlanguage Ġmodel Ġwith Ġa Ġsingle Ġlayer Ġof ĠPR U Ġand Ġfix Ġthe Ġsize Ġof Ġembed ding Ġand Ġhidden Ġlayers Ġto Ġ600 . ĠThe Ġword - level Ġperplex ities Ġare Ġreported Ġon Ġthe Ġvalidation Ġsets Ġof Ġthe ĠPT B Ġand Ġthe ĠWT - 2 Ġdatasets . </s> The Ġtwo Ġhyper - param eters Ġthat Ġcontrol Ġthe Ġtrade - off Ġbetween Ġperformance Ġand Ġnumber Ġof Ġparameters Ġin ĠPR Us Ġare Ġthe Ġnumber Ġof Ġpy ram idal Ġlevels ĠIN LINE FORM 0 Ġand Ġgroups ĠIN LINE FORM 1 Ġ. ĠFigure ĠFIG REF 35 Ġprovides Ġa Ġtrade - off Ġbetween Ġperplex ity Ġand Ġrecurrent Ġunit Ġ( R U ) Ġparameters . </s> Variable ĠIN LINE FORM 0 Ġand Ġfixed ĠIN LINE FORM 1 Ġ: ĠWhen Ġwe Ġincrease Ġthe Ġnumber Ġof Ġpy ram idal Ġlevels ĠIN LINE FORM 2 Ġat Ġa Ġfixed Ġvalue Ġof ĠIN LINE FORM 3 Ġ, Ġthe Ġperformance Ġof Ġthe ĠPR U Ġdrops Ġby Ġabout Ġ1 Ġto Ġ4 Ġpoints Ġwhile Ġreducing Ġthe Ġtotal Ġnumber Ġof Ġrecurrent Ġunit Ġparameters Ġby Ġup Ġto Ġ15 %. ĠWe Ġnote Ġthat Ġthe ĠPR U Ġwith ĠIN LINE FORM 4 Ġat ĠIN LINE FORM 5 Ġdelivers Ġsimilar Ġperformance Ġas Ġthe ĠL ST M Ġwhile Ġlearning Ġabout Ġ15 % Ġfewer Ġrecurrent Ġunit Ġparameters . </s> Fixed ĠIN LINE FORM 0 Ġand Ġvariable ĠIN LINE FORM 1 Ġ: ĠWhen Ġwe Ġvary Ġthe Ġvalue Ġof ĠIN LINE FORM 2 Ġat Ġfixed Ġnumber Ġof Ġpy ram idal Ġlevels ĠIN LINE FORM 3 Ġ, Ġthe Ġtotal Ġnumber Ġof Ġrecurrent Ġunit Ġparameters Ġdecreases Ġsignificantly Ġwith Ġa Ġminimal Ġimpact Ġon Ġthe Ġperplex ity . ĠFor Ġexample , ĠPR Us Ġwith ĠIN LINE FORM 4 Ġand ĠIN LINE FORM 5 Ġlearns Ġ77 % Ġfewer Ġrecurrent Ġunit Ġparameters Ġwhile Ġits Ġperplex ity Ġ( lower Ġis Ġbetter ) Ġincreases Ġby Ġabout Ġ12 % Ġin Ġcomparison Ġto ĠL ST Ms . ĠMoreover , Ġthe Ġdecrease Ġin Ġnumber Ġof Ġparameters Ġat Ġhigher Ġvalue Ġof ĠIN LINE FORM 6 Ġenables ĠPR Us Ġto Ġlearn Ġthe Ġrepresentations Ġin Ġhigh Ġdimensional Ġspace Ġwith Ġbetter Ġgeneral iz ability Ġ( Table ĠT AB REF 23 Ġ). </s> Table ĠT AB REF 43 Ġshows Ġthe Ġimpact Ġof Ġdifferent Ġtransformations Ġof Ġthe Ġinput Ġvector ĠIN LINE FORM 0 Ġand Ġthe Ġcontext Ġvector ĠIN LINE FORM 1 Ġ. ĠWe Ġmake Ġfollowing Ġobservations : Ġ( 1 ) ĠUsing Ġthe Ġpy ram idal Ġtransformation Ġfor Ġthe Ġinput Ġvectors Ġimproves Ġthe Ġperplex ity Ġby Ġabout Ġ1 Ġpoint Ġon Ġboth Ġthe ĠPT B Ġand ĠWT - 2 Ġdatasets Ġwhile Ġreducing Ġthe Ġnumber Ġof Ġrecurrent Ġunit Ġparameters Ġby Ġabout Ġ14 % Ġ( see ĠR 1 Ġand ĠR 4 ). ĠWe Ġnote Ġthat Ġthe Ġperformance Ġof Ġthe ĠPR U Ġdrops Ġby Ġup Ġto Ġ1 Ġpoint Ġwhen Ġresidual Ġconnections Ġare Ġnot Ġused Ġ( R 4 Ġand ĠR 6 ). Ġ( 2 ) ĠUsing Ġthe Ġgrouped Ġlinear Ġtransformation Ġfor Ġcontext Ġvectors Ġreduces Ġthe Ġtotal Ġnumber Ġof Ġrecurrent Ġunit Ġparameters Ġby Ġabout Ġ75 % Ġwhile Ġthe Ġperformance Ġdrops Ġby Ġabout Ġ11 % Ġ( see ĠR 3 Ġand ĠR 4 ). ĠWhen Ġwe Ġuse Ġthe Ġpy ram idal Ġtransformation Ġinstead Ġof Ġthe Ġlinear Ġtransformation , Ġthe Ġperformance Ġdrops Ġby Ġup Ġto Ġ2 % Ġwhile Ġthere Ġis Ġno Ġsignificant Ġdrop Ġin Ġthe Ġnumber Ġof Ġparameters Ġ( R 4 Ġand ĠR 5 ). </s> We Ġset Ġsub - sam pling Ġkernel ĠIN LINE FORM 0 Ġ( E q . ĠEQ REF 12 Ġ) Ġwith Ġstride ĠIN LINE FORM 1 Ġand Ġsize Ġof Ġ3 Ġ( ĠIN LINE FORM 2 Ġ) Ġin Ġfour Ġdifferent Ġways : Ġ( 1 ) ĠSkip : ĠWe Ġskip Ġevery Ġother Ġelement Ġin Ġthe Ġinput Ġvector . Ġ( 2 ) ĠConv olution : ĠWe Ġinitialize Ġthe Ġelements Ġof ĠIN LINE FORM 3 Ġrandomly Ġfrom Ġnormal Ġdistribution Ġand Ġlearn Ġthem Ġduring Ġtraining Ġthe Ġmodel . ĠWe Ġlimit Ġthe Ġoutput Ġvalues Ġbetween Ġ- 1 Ġand Ġ1 Ġusing ĠIN LINE FORM 4 Ġactivation Ġfunction Ġto Ġmake Ġtraining Ġstable . Ġ( 3 ) ĠAvg . Ġpool : ĠWe Ġinitialize Ġthe Ġelements Ġof ĠIN LINE FORM 5 Ġto ĠIN LINE FORM 6 Ġ. Ġ( 4 ) ĠMax Ġpool : ĠWe Ġselect Ġthe Ġmaximum Ġvalue Ġin Ġthe Ġkernel Ġwindow ĠIN LINE FORM 7 Ġ. </s> Table ĠT AB REF 45 Ġcompares Ġthe Ġperformance Ġof Ġthe ĠPR U Ġwith Ġdifferent Ġsampling Ġmethods . ĠAverage Ġpool ing Ġperforms Ġthe Ġbest Ġwhile Ġskipping Ġgive Ġcomparable Ġperformance . ĠBoth Ġof Ġthese Ġmethods Ġenable Ġthe Ġnetwork Ġto Ġlearn Ġricher Ġword Ġrepresentations Ġwhile Ġrepresenting Ġthe Ġinput Ġvector Ġin Ġdifferent Ġforms , Ġthus Ġdelivering Ġhigher Ġperformance . ĠSurprisingly , Ġa Ġconv olution - based Ġsub - sam pling Ġmethod Ġdoes Ġnot Ġperform Ġas Ġwell Ġas Ġthe Ġaveraging Ġmethod . ĠThe ĠIN LINE FORM 0 Ġfunction Ġused Ġafter Ġconv olution Ġlimits Ġthe Ġrange Ġof Ġoutput Ġvalues Ġwhich Ġare Ġfurther Ġlimited Ġby Ġthe ĠL ST M Ġg ating Ġstructure , Ġthereby Ġimp eding Ġin Ġthe Ġflow Ġof Ġinformation Ġinside Ġthe Ġcell . ĠMax Ġpool ing Ġforces Ġthe Ġnetwork Ġto Ġlearn Ġrepresentations Ġfrom Ġhigh Ġmagnitude Ġelements , Ġthus Ġdistinguishing Ġfeatures Ġbetween Ġelements Ġvan ishes , Ġresulting Ġin Ġpoor Ġperformance . </s> Conclusion </s> We Ġintroduce Ġthe ĠPy ram idal ĠRec urrent ĠUnit , Ġwhich Ġbetter Ġmodel Ġcontextual Ġinformation Ġby Ġadmitting Ġhigher Ġdimensional Ġrepresentations Ġwith Ġgood Ġgeneral iz ability . ĠWhen Ġapplied Ġto Ġthe Ġtask Ġof Ġlanguage Ġmodeling , ĠPR Us Ġimprove Ġperplex ity Ġacross Ġseveral Ġsettings , Ġincluding Ġrecent Ġstate - of - the - art Ġsystems . ĠOur Ġanalysis Ġshows Ġthat Ġthe ĠPR U Ġimproves Ġthe Ġflow Ġof Ġgradient Ġand Ġexpand Ġthe Ġword Ġembed ding Ġsub space , Ġresulting Ġin Ġmore Ġconfident Ġdecisions . ĠHere Ġwe Ġhave Ġshown Ġimprovements Ġfor Ġlanguage Ġmodeling . ĠIn Ġfuture , Ġwe Ġplan Ġto Ġstudy Ġthe Ġperformance Ġof ĠPR Us Ġon Ġdifferent Ġtasks , Ġincluding Ġmachine Ġtranslation Ġand Ġquestion Ġanswering . ĠIn Ġaddition , Ġwe Ġwill Ġstudy Ġthe Ġperformance Ġof Ġthe ĠPR U Ġon Ġlanguage Ġmodeling Ġwith Ġmore Ġrecent Ġinference Ġtechniques , Ġsuch Ġas Ġdynamic Ġevaluation Ġand Ġmixture Ġof Ġsoft max . </s> Acknowled gments </s> This Ġresearch Ġwas Ġsupported Ġby ĠNS F Ġ( I IS Ġ16 16 112 , ĠIII Ġ17 03 166 ), ĠAllen ĠDist inguished ĠInvestigator ĠAward , Ġand Ġgifts Ġfrom ĠAllen ĠInstitute Ġfor ĠAI , ĠGoogle , ĠAmazon , Ġand ĠBloomberg . ĠWe Ġare Ġgrateful Ġto ĠAaron ĠJa ech , ĠHannah ĠRash kin , ĠMand ar ĠJosh i , ĠAn ir udd ha ĠK emb h avi , Ġand Ġanonymous Ġreviewers Ġfor Ġtheir Ġhelpful Ġcomments .