Answer with content missing: (Data section) Chinese with version 5.1 of the Chinese Penn Treebank (CTB)
ABSTRACTIVE
Experimental Setup
1906.10225
Introduction </s> G ram mar Ġinduction Ġis Ġthe Ġtask Ġof Ġinducing Ġhierarchical Ġsynt actic Ġstructure Ġfrom Ġdata . ĠStatistical Ġapproaches Ġto Ġgrammar Ġinduction Ġrequire Ġspecifying Ġa Ġprob abil istic Ġgrammar Ġ( e . g . Ġformal ism , Ġnumber Ġand Ġshape Ġof Ġrules ), Ġand Ġfitting Ġits Ġparameters Ġthrough Ġoptimization . ĠEarly Ġwork Ġfound Ġthat Ġit Ġwas Ġdifficult Ġto Ġinduce Ġprob abil istic Ġcontext - free Ġgram m ars Ġ( PC FG ) Ġfrom Ġnatural Ġlanguage Ġdata Ġthrough Ġdirect Ġmethods , Ġsuch Ġas Ġoptimizing Ġthe Ġlog Ġlikelihood Ġwith Ġthe ĠEM Ġalgorithm ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠWhile Ġthe Ġreasons Ġfor Ġthe Ġfailure Ġare Ġmanifold Ġand Ġnot Ġcompletely Ġunderstood , Ġtwo Ġmajor Ġpotential Ġcauses Ġare Ġthe Ġill - beh aved Ġoptimization Ġlandscape Ġand Ġthe Ġoverly Ġstrict Ġindependence Ġassumptions Ġof ĠPC FG s . ĠMore Ġsuccessful Ġapproaches Ġto Ġgrammar Ġinduction Ġhave Ġthus Ġresorted Ġto Ġcarefully - crafted Ġauxiliary Ġobjectives ĠB IB REF 2 Ġ, Ġpri ors Ġor Ġnon - param etric Ġmodels ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġand Ġmanually - engine ered Ġfeatures ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġto Ġencourage Ġthe Ġdesired Ġstructures Ġto Ġemerge . </s> We Ġrevisit Ġthese Ġaforementioned Ġissues Ġin Ġlight Ġof Ġadvances Ġin Ġmodel Ġparameter ization Ġand Ġinference . ĠFirst , Ġcontrary Ġto Ġcommon Ġwisdom , Ġwe Ġfind Ġthat Ġparameter izing Ġa ĠPC FG 's Ġrule Ġprobabilities Ġwith Ġneural Ġnetworks Ġover Ġdistributed Ġrepresentations Ġmakes Ġit Ġpossible Ġto Ġinduce Ġlingu istically Ġmeaningful Ġgram m ars Ġby Ġsimply Ġoptimizing Ġlog Ġlikelihood . ĠWhile Ġthe Ġoptimization Ġproblem Ġremains Ġnon - con ve x , Ġrecent Ġwork Ġsuggests Ġthat Ġthere Ġare Ġoptimization Ġbenefits Ġafforded Ġby Ġover - param eter ized Ġmodels ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, Ġand Ġwe Ġindeed Ġfind Ġthat Ġthis Ġneural ĠPC FG Ġis Ġsignificantly Ġeasier Ġto Ġoptimize Ġthan Ġthe Ġtraditional ĠPC FG . ĠSecond , Ġthis Ġfact ored Ġparameter ization Ġmakes Ġit Ġstraightforward Ġto Ġincorporate Ġside Ġinformation Ġinto Ġrule Ġprobabilities Ġthrough Ġa Ġsentence - level Ġcontinuous Ġlatent Ġvector , Ġwhich Ġeffectively Ġallows Ġdifferent Ġcontexts Ġin Ġa Ġderiv ation Ġto Ġcoordinate . ĠIn Ġthis Ġcompound ĠPC FG âĢĶ contin uous Ġmixture Ġof ĠPC FG s âĢĶ the Ġcontext - free Ġassumptions Ġhold Ġconditioned Ġon Ġthe Ġlatent Ġvector Ġbut Ġnot Ġuncond itionally , Ġthereby Ġobtaining Ġlonger - range Ġdependencies Ġwithin Ġa Ġtree - based Ġgener ative Ġprocess . </s> To Ġutilize Ġthis Ġapproach , Ġwe Ġneed Ġto Ġefficiently Ġoptimize Ġthe Ġlog Ġmarginal Ġlikelihood Ġof Ġobserved Ġsentences . ĠWhile Ġcompound ĠPC FG s Ġbreak Ġefficient Ġinference , Ġif Ġthe Ġlatent Ġvector Ġis Ġknown Ġthe Ġdistribution Ġover Ġtrees Ġreduces Ġto Ġa Ġstandard ĠPC FG . ĠThis Ġproperty Ġallows Ġus Ġto Ġperform Ġgrammar Ġinduction Ġusing Ġa Ġcollapsed Ġapproach Ġwhere Ġthe Ġlatent Ġtrees Ġare Ġmarginalized Ġout Ġexactly Ġwith Ġdynamic Ġprogramming . ĠTo Ġhandle Ġthe Ġlatent Ġvector , Ġwe Ġemploy Ġstandard Ġam ort ized Ġinference Ġusing Ġrep aram eter ized Ġsamples Ġfrom Ġa Ġvari ational Ġposterior Ġapprox imated Ġfrom Ġan Ġinference Ġnetwork ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> On Ġstandard Ġbenchmarks Ġfor ĠEnglish Ġand ĠChinese , Ġthe Ġproposed Ġapproach Ġis Ġfound Ġto Ġperform Ġfavorably Ġagainst Ġrecent Ġneural Ġnetwork - based Ġapproaches Ġto Ġgrammar Ġinduction ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. </s> Pro b abil istic ĠContext - Free ĠGram m ars </s> We Ġconsider Ġcontext - free Ġgram m ars Ġ( CF G ) Ġconsisting Ġof Ġa Ġ5 - t uple ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġdistinguished Ġstart Ġsymbol , ĠIN LINE FORM 2 Ġis Ġa Ġfinite Ġset Ġof Ġnon termin als , ĠIN LINE FORM 3 Ġis Ġa Ġfinite Ġset Ġof Ġpre termin als , ĠIN LINE FORM 6 Ġis Ġa Ġfinite Ġset Ġof Ġterminal Ġsymbols , Ġand ĠIN LINE FORM 7 Ġis Ġa Ġfinite Ġset Ġof Ġrules Ġof Ġthe Ġform , </s> IN LINE FORM 0 </s> A Ġprob abil istic Ġcontext - free Ġgrammar Ġ( PC FG ) Ġconsists Ġof Ġa Ġgrammar ĠIN LINE FORM 0 Ġand Ġrule Ġprobabilities ĠIN LINE FORM 1 Ġsuch Ġthat ĠIN LINE FORM 2 Ġis Ġthe Ġprobability Ġof Ġthe Ġrule ĠIN LINE FORM 3 Ġ. ĠLet ting ĠIN LINE FORM 4 Ġbe Ġthe Ġset Ġof Ġall Ġparse Ġtrees Ġof ĠIN LINE FORM 5 Ġ, Ġa ĠPC FG Ġdefines Ġa Ġprobability Ġdistribution Ġover ĠIN LINE FORM 6 Ġvia ĠIN LINE FORM 7 Ġwhere ĠIN LINE FORM 8 Ġis Ġthe Ġset Ġof Ġrules Ġused Ġin Ġthe Ġderiv ation Ġof ĠIN LINE FORM 9 Ġ. ĠIt Ġalso Ġdefines Ġa Ġdistribution Ġover Ġstring Ġof Ġterminals ĠIN LINE FORM 10 Ġvia </s> IN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġ, Ġi . e . Ġthe Ġset Ġof Ġtrees ĠIN LINE FORM 1 Ġsuch Ġthat ĠIN LINE FORM 2 Ġ' s Ġleaves Ġare ĠIN LINE FORM 3 Ġ. ĠWe Ġwill Ġslightly Ġabuse Ġnotation Ġand Ġuse </s> IN LINE FORM 0 </s> to Ġdenote Ġthe Ġposterior Ġdistribution Ġover Ġthe Ġunob served Ġlatent Ġtrees Ġgiven Ġthe Ġobserved Ġsentence ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġindicator Ġfunction . </s> Comp ound ĠPC FG s </s> A Ġcompound Ġprobability Ġdistribution ĠB IB REF 19 Ġis Ġa Ġdistribution Ġwhose Ġparameters Ġare Ġthemselves Ġrandom Ġvariables . ĠThese Ġdistributions Ġgeneral ize Ġmixture Ġmodels Ġto Ġthe Ġcontinuous Ġcase , Ġfor Ġexample Ġin Ġfactor Ġanalysis Ġwhich Ġassumes Ġthe Ġfollowing Ġgener ative Ġprocess , </s> IN LINE FORM 0 </s> Comp ound Ġdistributions Ġprovide Ġthe Ġability Ġto Ġmodel Ġrich Ġgener ative Ġprocesses , Ġbut Ġmarginal izing Ġover Ġthe Ġlatent Ġparameter Ġcan Ġbe Ġcomput ationally Ġint ract able Ġunless Ġconj ug acy Ġcan Ġbe Ġexploited . </s> In Ġthis Ġwork , Ġwe Ġstudy Ġcompound Ġprob abil istic Ġcontext - free Ġgram m ars Ġwhose Ġdistribution Ġover Ġtrees Ġarises Ġfrom Ġthe Ġfollowing Ġgener ative Ġprocess : Ġwe Ġfirst Ġobtain Ġrule Ġprobabilities Ġvia </s> IN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġprior Ġwith Ġparameters ĠIN LINE FORM 1 Ġ( sp herical ĠGa ussian Ġin Ġthis Ġpaper ), Ġand ĠIN LINE FORM 2 Ġis Ġa Ġneural Ġnetwork Ġthat Ġconc aten ates Ġthe Ġinput Ġsymbol Ġembed d ings Ġwith ĠIN LINE FORM 3 Ġand Ġoutputs Ġthe Ġsentence - level Ġrule Ġprobabilities ĠIN LINE FORM 4 Ġ, </s> IN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġvector Ġconc aten ation . ĠThen Ġa Ġtree / sent ence Ġis Ġsampled Ġfrom Ġa ĠPC FG Ġwith Ġrule Ġprobabilities Ġgiven Ġby ĠIN LINE FORM 1 Ġ, </s> IN LINE FORM 0 </s> This Ġcan Ġbe Ġviewed Ġas Ġa Ġcontinuous Ġmixture Ġof ĠPC FG s , Ġor Ġalternatively , Ġa ĠBay esian ĠPC FG Ġwith Ġa Ġprior Ġon Ġsentence - level Ġrule Ġprobabilities Ġparameter ized Ġby ĠIN LINE FORM 0 Ġ. ĠImport antly , Ġunder Ġthis Ġgener ative Ġmodel Ġthe Ġcontext - free Ġassumptions Ġhold Ġconditioned Ġon ĠIN LINE FORM 3 Ġ, Ġbut Ġthey Ġdo Ġnot Ġhold Ġuncond itionally . ĠThis Ġis Ġshown Ġin ĠFigure ĠFIG REF 3 Ġ( right ) Ġwhere Ġthere Ġis Ġa Ġdependence Ġpath Ġthrough ĠIN LINE FORM 4 Ġif Ġit Ġis Ġnot Ġconditioned Ġupon . ĠComp ound ĠPC FG s Ġgive Ġrise Ġto Ġa Ġmarginal Ġdistribution Ġover Ġparse Ġtrees ĠIN LINE FORM 5 Ġvia </s> IN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġ. ĠThe Ġsubscript Ġin ĠIN LINE FORM 1 Ġdenotes Ġthe Ġfact Ġthat Ġthe Ġrule Ġprobabilities Ġdepend Ġon ĠIN LINE FORM 2 Ġ. ĠComp ound ĠPC FG s Ġare Ġclearly Ġmore Ġexpressive Ġthan ĠPC FG s Ġas Ġeach Ġsentence Ġhas Ġits Ġown Ġset Ġof Ġrule Ġprobabilities . ĠHowever , Ġit Ġstill Ġassumes Ġa Ġtree - based Ġgener ative Ġprocess , Ġmaking Ġit Ġpossible Ġto Ġlearn Ġlatent Ġtree Ġstructures . </s> Our Ġmotivation Ġfor Ġthe Ġcompound ĠPC FG Ġis Ġbased Ġon Ġthe Ġobservation Ġthat Ġfor Ġgrammar Ġinduction , Ġcontext - free Ġassumptions Ġare Ġgenerally Ġmade Ġnot Ġbecause Ġthey Ġrepresent Ġan Ġadequate Ġmodel Ġof Ġnatural Ġlanguage , Ġbut Ġbecause Ġthey Ġallow Ġfor Ġtract able Ġtraining . ĠWe Ġcan Ġin Ġprinciple Ġmodel Ġricher Ġdependencies Ġthrough Ġvertical / hor izontal ĠMark ov ization ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġand Ġlex ical ization ĠB IB REF 23 Ġ. ĠHowever Ġsuch Ġdependencies Ġcomplicate Ġtraining Ġdue Ġto Ġthe Ġrapid Ġincrease Ġin Ġthe Ġnumber Ġof Ġrules . ĠUnder Ġthis Ġview , Ġwe Ġcan Ġinterpret Ġthe Ġcompound ĠPC FG Ġas Ġa Ġrestricted Ġversion Ġof Ġsome Ġlex ical ized , Ġhigher - order ĠPC FG Ġwhere Ġa Ġchild Ġcan Ġdepend Ġon Ġstructural Ġand Ġlex ical Ġcontext Ġthrough Ġa Ġshared Ġlatent Ġvector . ĠWe Ġhypothes ize Ġthat Ġthis Ġdependence Ġamong Ġsiblings Ġis Ġespecially Ġuseful Ġin Ġgrammar Ġinduction Ġfrom Ġwords , Ġwhere Ġ( for Ġexample ) Ġif Ġwe Ġknow Ġthat Ġwatched Ġis Ġused Ġas Ġa Ġverb Ġthen Ġthe Ġnoun Ġphrase Ġis Ġlikely Ġto Ġbe Ġa Ġmovie . </s> In Ġcontrast Ġto Ġthe Ġusual ĠBay esian Ġtreatment Ġof ĠPC FG s Ġwhich Ġplaces Ġpri ors Ġon Ġglobal Ġrule Ġprobabilities ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 6 Ġ, Ġthe Ġcompound ĠPC FG Ġassumes Ġa Ġprior Ġon Ġlocal , Ġsentence - level Ġrule Ġprobabilities . ĠIt Ġis Ġtherefore Ġclosely Ġrelated Ġto Ġthe ĠBay esian Ġgram m ars Ġstudied Ġby ĠB IB REF 25 Ġand ĠB IB REF 26 Ġ, Ġwho Ġalso Ġsample Ġlocal Ġrule Ġprobabilities Ġfrom Ġa Ġlog istic Ġnormal Ġprior Ġfor Ġtraining Ġdependency Ġmodels Ġwith Ġval ence Ġ( DM V ) ĠB IB REF 27 Ġ. </s> Exper imental ĠSetup </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 23 Ġshows Ġthe Ġunl abel ed ĠIN LINE FORM 0 Ġscores Ġfor Ġour Ġmodels Ġand Ġvarious Ġbas elines . ĠAll Ġmodels Ġsound ly Ġoutper form Ġright Ġbranching Ġbas elines , Ġand Ġwe Ġfind Ġthat Ġthe Ġneural ĠPC FG / comp ound ĠPC FG Ġare Ġstrong Ġmodels Ġfor Ġgrammar Ġinduction . ĠIn Ġparticular Ġthe Ġcompound ĠPC FG Ġoutper forms Ġother Ġmodels Ġby Ġan Ġappreci able Ġmargin Ġon Ġboth ĠEnglish Ġand ĠChinese . ĠWe Ġagain Ġnote Ġthat Ġwe Ġwere Ġunable Ġto Ġinduce Ġmeaningful Ġgram m ars Ġthrough Ġa Ġtraditional ĠPC FG Ġwith Ġthe Ġscal ar Ġparameter ization Ġdespite Ġa Ġthorough Ġhyper param eter Ġsearch . ĠSee Ġlab : full Ġfor Ġthe Ġfull Ġresults Ġ( including Ġcorpus - level ĠIN LINE FORM 1 Ġ) Ġbroken Ġdown Ġby Ġsentence Ġlength . </s> Table ĠT AB REF 27 Ġanaly zes Ġthe Ġlearned Ġtree Ġstructures . ĠWe Ġcompare Ġsimilarity Ġas Ġmeasured Ġby ĠIN LINE FORM 0 Ġagainst Ġgold , Ġleft , Ġright , Ġand ĠâĢ ľ self " Ġtrees Ġ( top ), Ġwhere Ġself ĠIN LINE FORM 1 Ġscore Ġis Ġcalculated Ġby Ġaveraging Ġover Ġall Ġ6 Ġpairs Ġobtained Ġfrom Ġ4 Ġdifferent Ġruns . ĠWe Ġfind Ġthat ĠPR PN Ġis Ġparticularly Ġconsistent Ġacross Ġmultiple Ġruns . ĠWe Ġalso Ġobserve Ġthat Ġdifferent Ġmodels Ġare Ġbetter Ġat Ġidentifying Ġdifferent Ġconstituent Ġlabels , Ġas Ġmeasured Ġby Ġlabel Ġrecall Ġ( Table ĠT AB REF 27 Ġ, Ġbottom ). ĠWhile Ġleft Ġas Ġfuture Ġwork , Ġthis Ġnaturally Ġsuggests Ġan Ġensemble Ġapproach Ġwherein Ġthe Ġempirical Ġprobabilities Ġof Ġconstituents Ġ( ob tained Ġby Ġaveraging Ġthe Ġpredicted Ġbinary Ġconstituent Ġlabels Ġfrom Ġthe Ġdifferent Ġmodels ) Ġare Ġused Ġeither Ġto Ġsuperv ise Ġanother Ġmodel Ġor Ġdirectly Ġas Ġpotential s Ġin Ġa ĠCR F Ġconstituency Ġparser . ĠFinally , Ġall Ġmodels Ġseemed Ġto Ġhave Ġsome Ġdifficulty Ġin Ġidentifying ĠSB AR / VP Ġconstituents Ġwhich Ġtypically Ġspan Ġmore Ġwords Ġthan ĠNP Ġconstituents . </s> Related ĠWork </s> G ram mar Ġinduction Ġhas Ġa Ġlong Ġand Ġrich Ġhistory Ġin Ġnatural Ġlanguage Ġprocessing . ĠEarly Ġwork Ġon Ġgrammar Ġinduction Ġwith Ġpure Ġun super vised Ġlearning Ġwas Ġmostly Ġnegative ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 74 Ġ, Ġthough ĠB IB REF 75 Ġreported Ġsome Ġsuccess Ġon Ġpartially Ġbracket ed Ġdata . ĠB IB REF 76 Ġand ĠB IB REF 2 Ġwere Ġsome Ġof Ġthe Ġfirst Ġsuccessful Ġstatistical Ġapproaches Ġto Ġgrammar Ġinduction . ĠIn Ġparticular , Ġthe Ġconstituent - context Ġmodel Ġ( CC M ) Ġof ĠB IB REF 2 Ġ, Ġwhich Ġexplicitly Ġmodels Ġboth Ġconstituents Ġand Ġdist itu ents , Ġwas Ġthe Ġbasis Ġfor Ġmuch Ġsubsequent Ġwork ĠB IB REF 27 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠOther Ġworks Ġhave Ġexplored Ġimposing Ġinduct ive Ġbiases Ġthrough ĠBay esian Ġpri ors ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġmodified Ġobjectives ĠB IB REF 42 Ġ, Ġand Ġadditional Ġconstraints Ġon Ġrec ursion Ġdepth ĠB IB REF 77 Ġ, ĠB IB REF 48 Ġ. </s> While Ġthe Ġframework Ġof Ġspecifying Ġthe Ġstructure Ġof Ġa Ġgrammar Ġand Ġlearning Ġthe Ġparameters Ġis Ġcommon , Ġother Ġmethods Ġexist . ĠB IB REF 43 Ġconsider Ġa Ġnon param etric - style Ġapproach Ġto Ġun super vised Ġparsing Ġby Ġusing Ġrandom Ġsubs ets Ġof Ġtraining Ġsubt rees Ġto Ġparse Ġnew Ġsentences . ĠB IB REF 46 Ġutilize Ġan Ġincremental Ġalgorithm Ġto Ġun super vised Ġparsing Ġwhich Ġmakes Ġlocal Ġdecisions Ġto Ġcreate Ġconstituents Ġbased Ġon Ġa Ġcomplex Ġset Ġof Ġhe ur istics . ĠB IB REF 47 Ġinduce Ġparse Ġtrees Ġthrough Ġcasc aded Ġapplications Ġof Ġfinite Ġstate Ġmodels . </s> More Ġrecently , Ġneural Ġnetwork - based Ġapproaches Ġto Ġgrammar Ġinduction Ġhave Ġshown Ġpromising Ġresults Ġon Ġinducing Ġparse Ġtrees Ġdirectly Ġfrom Ġwords . ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġlearn Ġtree Ġstructures Ġthrough Ġsoft Ġg ating Ġlayers Ġwithin Ġneural Ġlanguage Ġmodels , Ġwhile ĠB IB REF 16 Ġcombine Ġrecursive Ġaut oen cod ers Ġwith Ġthe Ġinside - outside Ġalgorithm . ĠB IB REF 17 Ġtrain Ġun super vised Ġrecurrent Ġneural Ġnetwork Ġgram m ars Ġwith Ġa Ġstructured Ġinference Ġnetwork Ġto Ġinduce Ġlatent Ġtrees , Ġand ĠB IB REF 78 Ġutilize Ġimage Ġcapt ions Ġto Ġidentify Ġand Ġground Ġconstituents . </s> Our Ġwork Ġis Ġalso Ġrelated Ġto Ġlatent Ġvariable ĠPC FG s ĠB IB REF 79 Ġ, ĠB IB REF 80 Ġ, ĠB IB REF 81 Ġ, Ġwhich Ġextend ĠPC FG s Ġto Ġthe Ġlatent Ġvariable Ġsetting Ġby Ġsplitting Ġnon termin al Ġsymbols Ġinto Ġlatent Ġsub sy mb ols . ĠIn Ġparticular , Ġlatent Ġvector Ġgram m ars ĠB IB REF 82 Ġand Ġcompos itional Ġvector Ġgram m ars ĠB IB REF 83 Ġalso Ġemploy Ġcontinuous Ġvectors Ġwithin Ġtheir Ġgram m ars . ĠHowever Ġthese Ġapproaches Ġhave Ġbeen Ġemployed Ġfor Ġlearning Ġsupervised Ġpars ers Ġon Ġannot ated Ġtree banks , Ġin Ġcontrast Ġto Ġthe Ġun super vised Ġsetting Ġof Ġthe Ġcurrent Ġwork . </s> Conclusion </s> This Ġwork Ġexplores Ġgrammar Ġinduction Ġwith Ġcompound ĠPC FG s , Ġwhich Ġmod ulate Ġrule Ġprobabilities Ġwith Ġper - sent ence Ġcontinuous Ġlatent Ġvectors . ĠThe Ġlatent Ġvector Ġinduces Ġmarginal Ġdependencies Ġbeyond Ġthe Ġtraditional Ġfirst - order Ġcontext - free Ġassumptions Ġwithin Ġa Ġtree - based Ġgener ative Ġprocess , Ġleading Ġto Ġimproved Ġperformance . ĠThe Ġcollapsed Ġam ort ized Ġvari ational Ġinference Ġapproach Ġis Ġgeneral Ġand Ġcan Ġbe Ġused Ġfor Ġgener ative Ġmodels Ġwhich Ġadmit Ġtract able Ġinference Ġthrough Ġpartial Ġconditioning . ĠLearning Ġdeep Ġgener ative Ġmodels Ġwhich Ġexhibit Ġsuch Ġconditional ĠMark ov Ġproperties Ġis Ġan Ġinteresting Ġdirection Ġfor Ġfuture Ġwork . </s> Acknowled gments </s> We Ġthank ĠPhil ĠBl uns om Ġfor Ġinitial Ġdiscussions Ġwhich Ġseeded Ġmany Ġof Ġthe Ġcore Ġideas Ġin Ġthe Ġpresent Ġwork . ĠWe Ġalso Ġthank ĠYon atan ĠBel ink ov Ġand ĠShay ĠCohen Ġfor Ġhelpful Ġfeedback , Ġand ĠAndrew ĠDro z d ov Ġfor Ġproviding Ġthe Ġparsed Ġdataset Ġfrom Ġtheir ĠDI OR A Ġmodel . ĠY K Ġis Ġsupported Ġby Ġa ĠGoogle ĠFellowship . ĠAM R Ġacknowledges Ġthe Ġsupport Ġof ĠNS F Ġ170 48 34 , Ġ18 456 64 , ĠAWS , Ġand ĠOracle . </s> Model ĠParam eter ization </s> We Ġassociate Ġan Ġinput Ġembed ding ĠIN LINE FORM 0 Ġfor Ġeach Ġsymbol ĠIN LINE FORM 1 Ġon Ġthe Ġleft Ġside Ġof Ġa Ġrule Ġ( i . e . ĠIN LINE FORM 2 Ġ) Ġand Ġrun Ġa Ġneural Ġnetwork Ġover ĠIN LINE FORM 3 Ġto Ġobtain Ġthe Ġrule Ġprobabilities . ĠCon crete ly , Ġeach Ġrule Ġtype ĠIN LINE FORM 4 Ġis Ġparameter ized Ġas Ġfollows , ĠIN LINE FORM 5 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġproduct Ġspace ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare ĠML Ps Ġwith Ġtwo Ġresidual Ġlayers , ĠIN LINE FORM 3 </s> The Ġbias Ġterms Ġfor Ġthe Ġabove Ġexpressions Ġ( including Ġfor Ġthe Ġrule Ġprobabilities ) Ġare Ġomitted Ġfor Ġnot ational Ġbre vity . ĠIn ĠFigure ĠFIG REF 3 Ġwe Ġuse Ġthe Ġfollowing Ġto Ġrefer Ġto Ġrule Ġprobabilities Ġof Ġdifferent Ġrule Ġtypes , ĠIN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġset Ġof Ġrules Ġwith ĠIN LINE FORM 1 Ġon Ġthe Ġleft Ġhand Ġside . </s> The Ġcompound ĠPC FG Ġrule Ġprobabilities ĠIN LINE FORM 0 Ġgiven Ġa Ġlatent Ġvector ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 </s> Again Ġthe Ġbias Ġterms Ġare Ġomitted Ġfor Ġbre vity , Ġand ĠIN LINE FORM 0 Ġare Ġas Ġbefore Ġwhere Ġthe Ġfirst Ġlayer 's Ġinput Ġdimensions Ġare Ġappropriately Ġchanged Ġto Ġaccount Ġfor Ġconc aten ation Ġwith ĠIN LINE FORM 1 Ġ. </s> Corp us / Sent ence ĠF Ġ1 ĠF _ 1 Ġby ĠSent ence ĠLength </s> For Ġcomple teness Ġwe Ġshow Ġthe Ġcorpus - level Ġand Ġsentence - level ĠIN LINE FORM 0 Ġbroken Ġdown Ġby Ġsentence Ġlength Ġin ĠTable ĠT AB REF 44 Ġ, Ġaveraged Ġacross Ġ4 Ġdifferent Ġruns Ġof Ġeach Ġmodel . </s> Exper iments Ġwith ĠR NN Gs </s> For Ġexperiments Ġon Ġsuperv ising ĠR NN Gs Ġwith Ġinduced Ġtrees , Ġwe Ġuse Ġthe Ġparameter ization Ġand Ġhyper param eters Ġfrom ĠB IB REF 17 Ġ, Ġwhich Ġuses Ġa Ġ2 - layer Ġ650 - dimensional Ġstack ĠL ST M Ġ( with Ġdrop out Ġof Ġ0 . 5 ) Ġand Ġa Ġ650 - dimensional Ġtree ĠL ST M ĠB IB REF 88 Ġ, ĠB IB REF 90 Ġas Ġthe Ġcomposition Ġfunction . </s> Con crete ly , Ġthe Ġgener ative Ġstory Ġis Ġas Ġfollows : Ġfirst , Ġthe Ġstack Ġrepresentation Ġis Ġused Ġto Ġpredict Ġthe Ġnext Ġaction Ġ( shift Ġor Ġreduce ) Ġvia Ġan Ġaff ine Ġtransformation Ġfollowed Ġby Ġa Ġs igm oid . ĠIf Ġshift Ġis Ġchosen , Ġwe Ġobtain Ġa Ġdistribution Ġover Ġthe Ġvocabulary Ġvia Ġanother Ġaff ine Ġtransformation Ġover Ġthe Ġstack Ġrepresentation Ġfollowed Ġby Ġa Ġsoft max . ĠThen Ġwe Ġsample Ġthe Ġnext Ġword Ġfrom Ġthis Ġdistribution Ġand Ġshift Ġthe Ġgenerated Ġword Ġonto Ġthe Ġstack Ġusing Ġthe Ġstack ĠL ST M . ĠIf Ġreduce Ġis Ġchosen , Ġwe Ġpop Ġthe Ġlast Ġtwo Ġelements Ġoff Ġthe Ġstack Ġand Ġuse Ġthe Ġtree ĠL ST M Ġto Ġobtain Ġa Ġnew Ġrepresentation . ĠThis Ġnew Ġrepresentation Ġis Ġshifted Ġonto Ġthe Ġstack Ġvia Ġthe Ġstack ĠL ST M . ĠNote Ġthat Ġthis ĠR NN G Ġparameter ization Ġis Ġslightly Ġdifferent Ġthan Ġthe Ġoriginal Ġfrom ĠB IB REF 53 Ġ, Ġwhich Ġdoes Ġnot Ġignore Ġconstituent Ġlabels Ġand Ġutilizes Ġa Ġbid irection al ĠL ST M Ġas Ġthe Ġcomposition Ġfunction Ġinstead Ġof Ġa Ġtree ĠL ST M . ĠAs Ġour ĠR NN G Ġparameter ization Ġonly Ġworks Ġwith Ġbinary Ġtrees , Ġwe Ġbin ar ize Ġthe Ġgold Ġtrees Ġwith Ġright Ġbin ar ization Ġfor Ġthe ĠR NN G Ġtrained Ġon Ġgold Ġtrees Ġ( t rees Ġfrom Ġthe Ġun super vised Ġmethods Ġexplored Ġin Ġthis Ġpaper Ġare Ġalready Ġbinary ). ĠThe ĠR NN G Ġalso Ġtrains Ġa Ġdiscrim in ative Ġparser Ġalongside Ġthe Ġgener ative Ġmodel Ġfor Ġevaluation Ġwith Ġimportance Ġsampling . ĠWe Ġuse Ġa ĠCR F Ġparser Ġwhose Ġspan Ġscore Ġparameter ization Ġis Ġsimilar Ġsimilar Ġto Ġrecent Ġworks ĠB IB REF 89 Ġ, ĠB IB REF 87 Ġ, ĠB IB REF 85 Ġ: Ġposition Ġembed d ings Ġare Ġadded Ġto Ġword Ġembed d ings , Ġand Ġa Ġbid irection al ĠL ST M Ġwith Ġ256 Ġhidden Ġdimensions Ġis Ġrun Ġover Ġthe Ġinput Ġrepresentations Ġto Ġobtain Ġthe Ġforward Ġand Ġbackward Ġhidden Ġstates . ĠThe Ġscore ĠIN LINE FORM 0 Ġfor Ġa Ġconstituent Ġspanning Ġthe ĠIN LINE FORM 1 Ġ- th Ġand ĠIN LINE FORM 2 Ġ- th Ġword Ġis Ġgiven Ġby , </s> IN LINE FORM 0 </s> where Ġthe ĠML P Ġhas Ġa Ġsingle Ġhidden Ġlayer Ġwith ĠIN LINE FORM 0 Ġnon linear ity Ġfollowed Ġby Ġlayer Ġnormal ization ĠB IB REF 84 Ġ. </s> For Ġexperiments Ġon Ġfine - tun ing Ġthe ĠR NN G Ġwith Ġthe Ġun super vised ĠR NN G , Ġwe Ġtake Ġthe Ġdiscrim in ative Ġparser Ġ( which Ġis Ġalso Ġpret rained Ġalongside Ġthe ĠR NN G Ġon Ġinduced Ġtrees ) Ġto Ġbe Ġthe Ġstructured Ġinference Ġnetwork Ġfor Ġoptimizing Ġthe Ġevidence Ġlower Ġbound . ĠWe Ġrefer Ġthe Ġreader Ġto ĠB IB REF 17 Ġand Ġtheir Ġopen Ġsource Ġimplementation Ġfor Ġadditional Ġdetails . ĠWe Ġalso Ġobserve Ġthat Ġas Ġnoted Ġby ĠB IB REF 17 Ġ, Ġa ĠUR NN G Ġtrained Ġfrom Ġscratch Ġon Ġthis Ġversion Ġof ĠPT B Ġwithout Ġpunct uation Ġfailed Ġto Ġoutper form Ġa Ġright - br anch ing Ġbaseline . </s> The ĠL ST M Ġlanguage Ġmodel Ġbaseline Ġis Ġthe Ġsame Ġsize Ġas Ġthe Ġstack ĠL ST M Ġ( i . e . Ġ2 Ġlayers , Ġ650 Ġhidden Ġunits , Ġdrop out Ġof Ġ0 . 5 ), Ġand Ġis Ġtherefore Ġequivalent Ġto Ġan ĠR NN G Ġwith Ġcompletely Ġright Ġbranching Ġtrees . ĠThe ĠPR PN / ON Ġbas elines Ġfor Ġperplex ity / sy nt actic Ġevaluation Ġin ĠTable ĠT AB REF 30 Ġalso Ġhave Ġ2 Ġlayers Ġwith Ġ650 Ġhidden Ġunits Ġand Ġ0 . 5 Ġdrop out . ĠTherefore Ġall Ġmodels Ġconsidered Ġin ĠTable ĠT AB REF 30 Ġhave Ġroughly Ġthe Ġsame Ġcapacity . ĠFor Ġall Ġmodels Ġwe Ġshare Ġinput / output Ġword Ġembed d ings ĠB IB REF 86 Ġ. ĠPer plex ity Ġestimation Ġfor Ġthe ĠR NN Gs Ġand Ġthe Ġcompound ĠPC FG Ġuses Ġ1000 Ġimportance - weight ed Ġsamples . </s> For Ġgram matical ity Ġjudgment , Ġwe Ġmodify Ġthe Ġpublicly Ġavailable Ġdataset Ġfrom ĠB IB REF 56 Ġto Ġonly Ġkeep Ġsentence Ġpairs Ġthat Ġdid Ġnot Ġhave Ġany Ġunknown Ġwords Ġwith Ġrespect Ġto Ġour ĠPT B Ġvocabulary Ġof Ġ10 K Ġwords . ĠThis Ġresults Ġin Ġ33 K Ġsentence Ġpairs Ġfor Ġevaluation . </s> Non termin al / Pre termin al ĠAl ign ments </s> Figure ĠFIG REF 50 Ġshows Ġthe Ġpart - of - speech Ġalign ments Ġand ĠTable ĠT AB REF 46 Ġshows Ġthe Ġnon termin al Ġlabel Ġalign ments Ġfor Ġthe Ġcompound ĠPC FG / ne ural ĠPC FG . </s> Sub tree ĠAnalysis </s> Table ĠT AB REF 53 Ġlists Ġmore Ġexamples Ġof Ġconstituents Ġwithin Ġeach Ġsubt ree Ġas Ġthe Ġtop Ġprincip ical Ġcomponent Ġis Ġvaried . ĠDue Ġto Ġdata Ġsp arsity , Ġthe Ġsubt ree Ġanalysis Ġis Ġperformed Ġon Ġthe Ġfull Ġdataset . ĠSee Ġsection ĠUID 36 Ġfor Ġmore Ġdetails .
Introduction Grammar induction is the task of inducing hierarchical syntactic structure from data. Statistical approaches to grammar induction require specifying a probabilistic grammar (e.g. formalism, number and shape of rules), and fitting its parameters through optimization. Early work found that it was difficult to induce probabilistic context-free grammars (PCFG) from natural language data through direct methods, such as optimizing the log likelihood with the EM algorithm BIBREF0 , BIBREF1 . While the reasons for the failure are manifold and not completely understood, two major potential causes are the ill-behaved optimization landscape and the overly strict independence assumptions of PCFGs. More successful approaches to grammar induction have thus resorted to carefully-crafted auxiliary objectives BIBREF2 , priors or non-parametric models BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , and manually-engineered features BIBREF7 , BIBREF8 to encourage the desired structures to emerge. We revisit these aforementioned issues in light of advances in model parameterization and inference. First, contrary to common wisdom, we find that parameterizing a PCFG's rule probabilities with neural networks over distributed representations makes it possible to induce linguistically meaningful grammars by simply optimizing log likelihood. While the optimization problem remains non-convex, recent work suggests that there are optimization benefits afforded by over-parameterized models BIBREF9 , BIBREF10 , BIBREF11 , and we indeed find that this neural PCFG is significantly easier to optimize than the traditional PCFG. Second, this factored parameterization makes it straightforward to incorporate side information into rule probabilities through a sentence-level continuous latent vector, which effectively allows different contexts in a derivation to coordinate. In this compound PCFG—continuous mixture of PCFGs—the context-free assumptions hold conditioned on the latent vector but not unconditionally, thereby obtaining longer-range dependencies within a tree-based generative process. To utilize this approach, we need to efficiently optimize the log marginal likelihood of observed sentences. While compound PCFGs break efficient inference, if the latent vector is known the distribution over trees reduces to a standard PCFG. This property allows us to perform grammar induction using a collapsed approach where the latent trees are marginalized out exactly with dynamic programming. To handle the latent vector, we employ standard amortized inference using reparameterized samples from a variational posterior approximated from an inference network BIBREF12 , BIBREF13 . On standard benchmarks for English and Chinese, the proposed approach is found to perform favorably against recent neural network-based approaches to grammar induction BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 . Probabilistic Context-Free Grammars We consider context-free grammars (CFG) consisting of a 5-tuple INLINEFORM0 where INLINEFORM1 is the distinguished start symbol, INLINEFORM2 is a finite set of nonterminals, INLINEFORM3 is a finite set of preterminals, INLINEFORM6 is a finite set of terminal symbols, and INLINEFORM7 is a finite set of rules of the form, INLINEFORM0 A probabilistic context-free grammar (PCFG) consists of a grammar INLINEFORM0 and rule probabilities INLINEFORM1 such that INLINEFORM2 is the probability of the rule INLINEFORM3 . Letting INLINEFORM4 be the set of all parse trees of INLINEFORM5 , a PCFG defines a probability distribution over INLINEFORM6 via INLINEFORM7 where INLINEFORM8 is the set of rules used in the derivation of INLINEFORM9 . It also defines a distribution over string of terminals INLINEFORM10 via INLINEFORM0 where INLINEFORM0 , i.e. the set of trees INLINEFORM1 such that INLINEFORM2 's leaves are INLINEFORM3 . We will slightly abuse notation and use INLINEFORM0 to denote the posterior distribution over the unobserved latent trees given the observed sentence INLINEFORM0 , where INLINEFORM1 is the indicator function. Compound PCFGs A compound probability distribution BIBREF19 is a distribution whose parameters are themselves random variables. These distributions generalize mixture models to the continuous case, for example in factor analysis which assumes the following generative process, INLINEFORM0 Compound distributions provide the ability to model rich generative processes, but marginalizing over the latent parameter can be computationally intractable unless conjugacy can be exploited. In this work, we study compound probabilistic context-free grammars whose distribution over trees arises from the following generative process: we first obtain rule probabilities via INLINEFORM0 where INLINEFORM0 is a prior with parameters INLINEFORM1 (spherical Gaussian in this paper), and INLINEFORM2 is a neural network that concatenates the input symbol embeddings with INLINEFORM3 and outputs the sentence-level rule probabilities INLINEFORM4 , INLINEFORM0 where INLINEFORM0 denotes vector concatenation. Then a tree/sentence is sampled from a PCFG with rule probabilities given by INLINEFORM1 , INLINEFORM0 This can be viewed as a continuous mixture of PCFGs, or alternatively, a Bayesian PCFG with a prior on sentence-level rule probabilities parameterized by INLINEFORM0 . Importantly, under this generative model the context-free assumptions hold conditioned on INLINEFORM3 , but they do not hold unconditionally. This is shown in Figure FIGREF3 (right) where there is a dependence path through INLINEFORM4 if it is not conditioned upon. Compound PCFGs give rise to a marginal distribution over parse trees INLINEFORM5 via INLINEFORM0 where INLINEFORM0 . The subscript in INLINEFORM1 denotes the fact that the rule probabilities depend on INLINEFORM2 . Compound PCFGs are clearly more expressive than PCFGs as each sentence has its own set of rule probabilities. However, it still assumes a tree-based generative process, making it possible to learn latent tree structures. Our motivation for the compound PCFG is based on the observation that for grammar induction, context-free assumptions are generally made not because they represent an adequate model of natural language, but because they allow for tractable training. We can in principle model richer dependencies through vertical/horizontal Markovization BIBREF21 , BIBREF22 and lexicalization BIBREF23 . However such dependencies complicate training due to the rapid increase in the number of rules. Under this view, we can interpret the compound PCFG as a restricted version of some lexicalized, higher-order PCFG where a child can depend on structural and lexical context through a shared latent vector. We hypothesize that this dependence among siblings is especially useful in grammar induction from words, where (for example) if we know that watched is used as a verb then the noun phrase is likely to be a movie. In contrast to the usual Bayesian treatment of PCFGs which places priors on global rule probabilities BIBREF3 , BIBREF4 , BIBREF6 , the compound PCFG assumes a prior on local, sentence-level rule probabilities. It is therefore closely related to the Bayesian grammars studied by BIBREF25 and BIBREF26 , who also sample local rule probabilities from a logistic normal prior for training dependency models with valence (DMV) BIBREF27 . Experimental Setup Results and Discussion Table TABREF23 shows the unlabeled INLINEFORM0 scores for our models and various baselines. All models soundly outperform right branching baselines, and we find that the neural PCFG/compound PCFG are strong models for grammar induction. In particular the compound PCFG outperforms other models by an appreciable margin on both English and Chinese. We again note that we were unable to induce meaningful grammars through a traditional PCFG with the scalar parameterization despite a thorough hyperparameter search. See lab:full for the full results (including corpus-level INLINEFORM1 ) broken down by sentence length. Table TABREF27 analyzes the learned tree structures. We compare similarity as measured by INLINEFORM0 against gold, left, right, and “self" trees (top), where self INLINEFORM1 score is calculated by averaging over all 6 pairs obtained from 4 different runs. We find that PRPN is particularly consistent across multiple runs. We also observe that different models are better at identifying different constituent labels, as measured by label recall (Table TABREF27 , bottom). While left as future work, this naturally suggests an ensemble approach wherein the empirical probabilities of constituents (obtained by averaging the predicted binary constituent labels from the different models) are used either to supervise another model or directly as potentials in a CRF constituency parser. Finally, all models seemed to have some difficulty in identifying SBAR/VP constituents which typically span more words than NP constituents. Related Work Grammar induction has a long and rich history in natural language processing. Early work on grammar induction with pure unsupervised learning was mostly negative BIBREF0 , BIBREF1 , BIBREF74 , though BIBREF75 reported some success on partially bracketed data. BIBREF76 and BIBREF2 were some of the first successful statistical approaches to grammar induction. In particular, the constituent-context model (CCM) of BIBREF2 , which explicitly models both constituents and distituents, was the basis for much subsequent work BIBREF27 , BIBREF7 , BIBREF8 . Other works have explored imposing inductive biases through Bayesian priors BIBREF4 , BIBREF5 , BIBREF6 , modified objectives BIBREF42 , and additional constraints on recursion depth BIBREF77 , BIBREF48 . While the framework of specifying the structure of a grammar and learning the parameters is common, other methods exist. BIBREF43 consider a nonparametric-style approach to unsupervised parsing by using random subsets of training subtrees to parse new sentences. BIBREF46 utilize an incremental algorithm to unsupervised parsing which makes local decisions to create constituents based on a complex set of heuristics. BIBREF47 induce parse trees through cascaded applications of finite state models. More recently, neural network-based approaches to grammar induction have shown promising results on inducing parse trees directly from words. BIBREF14 , BIBREF15 learn tree structures through soft gating layers within neural language models, while BIBREF16 combine recursive autoencoders with the inside-outside algorithm. BIBREF17 train unsupervised recurrent neural network grammars with a structured inference network to induce latent trees, and BIBREF78 utilize image captions to identify and ground constituents. Our work is also related to latent variable PCFGs BIBREF79 , BIBREF80 , BIBREF81 , which extend PCFGs to the latent variable setting by splitting nonterminal symbols into latent subsymbols. In particular, latent vector grammars BIBREF82 and compositional vector grammars BIBREF83 also employ continuous vectors within their grammars. However these approaches have been employed for learning supervised parsers on annotated treebanks, in contrast to the unsupervised setting of the current work. Conclusion This work explores grammar induction with compound PCFGs, which modulate rule probabilities with per-sentence continuous latent vectors. The latent vector induces marginal dependencies beyond the traditional first-order context-free assumptions within a tree-based generative process, leading to improved performance. The collapsed amortized variational inference approach is general and can be used for generative models which admit tractable inference through partial conditioning. Learning deep generative models which exhibit such conditional Markov properties is an interesting direction for future work. Acknowledgments We thank Phil Blunsom for initial discussions which seeded many of the core ideas in the present work. We also thank Yonatan Belinkov and Shay Cohen for helpful feedback, and Andrew Drozdov for providing the parsed dataset from their DIORA model. YK is supported by a Google Fellowship. AMR acknowledges the support of NSF 1704834, 1845664, AWS, and Oracle. Model Parameterization We associate an input embedding INLINEFORM0 for each symbol INLINEFORM1 on the left side of a rule (i.e. INLINEFORM2 ) and run a neural network over INLINEFORM3 to obtain the rule probabilities. Concretely, each rule type INLINEFORM4 is parameterized as follows, INLINEFORM5 where INLINEFORM0 is the product space INLINEFORM1 , and INLINEFORM2 are MLPs with two residual layers, INLINEFORM3 The bias terms for the above expressions (including for the rule probabilities) are omitted for notational brevity. In Figure FIGREF3 we use the following to refer to rule probabilities of different rule types, INLINEFORM0 where INLINEFORM0 denotes the set of rules with INLINEFORM1 on the left hand side. The compound PCFG rule probabilities INLINEFORM0 given a latent vector INLINEFORM1 , INLINEFORM2 Again the bias terms are omitted for brevity, and INLINEFORM0 are as before where the first layer's input dimensions are appropriately changed to account for concatenation with INLINEFORM1 . Corpus/Sentence F 1 F_1 by Sentence Length For completeness we show the corpus-level and sentence-level INLINEFORM0 broken down by sentence length in Table TABREF44 , averaged across 4 different runs of each model. Experiments with RNNGs For experiments on supervising RNNGs with induced trees, we use the parameterization and hyperparameters from BIBREF17 , which uses a 2-layer 650-dimensional stack LSTM (with dropout of 0.5) and a 650-dimensional tree LSTM BIBREF88 , BIBREF90 as the composition function. Concretely, the generative story is as follows: first, the stack representation is used to predict the next action (shift or reduce) via an affine transformation followed by a sigmoid. If shift is chosen, we obtain a distribution over the vocabulary via another affine transformation over the stack representation followed by a softmax. Then we sample the next word from this distribution and shift the generated word onto the stack using the stack LSTM. If reduce is chosen, we pop the last two elements off the stack and use the tree LSTM to obtain a new representation. This new representation is shifted onto the stack via the stack LSTM. Note that this RNNG parameterization is slightly different than the original from BIBREF53 , which does not ignore constituent labels and utilizes a bidirectional LSTM as the composition function instead of a tree LSTM. As our RNNG parameterization only works with binary trees, we binarize the gold trees with right binarization for the RNNG trained on gold trees (trees from the unsupervised methods explored in this paper are already binary). The RNNG also trains a discriminative parser alongside the generative model for evaluation with importance sampling. We use a CRF parser whose span score parameterization is similar similar to recent works BIBREF89 , BIBREF87 , BIBREF85 : position embeddings are added to word embeddings, and a bidirectional LSTM with 256 hidden dimensions is run over the input representations to obtain the forward and backward hidden states. The score INLINEFORM0 for a constituent spanning the INLINEFORM1 -th and INLINEFORM2 -th word is given by, INLINEFORM0 where the MLP has a single hidden layer with INLINEFORM0 nonlinearity followed by layer normalization BIBREF84 . For experiments on fine-tuning the RNNG with the unsupervised RNNG, we take the discriminative parser (which is also pretrained alongside the RNNG on induced trees) to be the structured inference network for optimizing the evidence lower bound. We refer the reader to BIBREF17 and their open source implementation for additional details. We also observe that as noted by BIBREF17 , a URNNG trained from scratch on this version of PTB without punctuation failed to outperform a right-branching baseline. The LSTM language model baseline is the same size as the stack LSTM (i.e. 2 layers, 650 hidden units, dropout of 0.5), and is therefore equivalent to an RNNG with completely right branching trees. The PRPN/ON baselines for perplexity/syntactic evaluation in Table TABREF30 also have 2 layers with 650 hidden units and 0.5 dropout. Therefore all models considered in Table TABREF30 have roughly the same capacity. For all models we share input/output word embeddings BIBREF86 . Perplexity estimation for the RNNGs and the compound PCFG uses 1000 importance-weighted samples. For grammaticality judgment, we modify the publicly available dataset from BIBREF56 to only keep sentence pairs that did not have any unknown words with respect to our PTB vocabulary of 10K words. This results in 33K sentence pairs for evaluation. Nonterminal/Preterminal Alignments Figure FIGREF50 shows the part-of-speech alignments and Table TABREF46 shows the nonterminal label alignments for the compound PCFG/neural PCFG. Subtree Analysis Table TABREF53 lists more examples of constituents within each subtree as the top principical component is varied. Due to data sparsity, the subtree analysis is performed on the full dataset. See section UID36 for more details.
which chinese datasets were used?
867290103f762e1ddfa6f2ea30dd0a327f595182
which Ġch inese Ġdatasets Ġwere Ġused ?
Answer Ġwith Ġcontent Ġmissing : Ġ( Data Ġsection ) ĠChinese Ġwith Ġversion Ġ5 . 1 Ġof Ġthe ĠChinese ĠPenn ĠTree bank Ġ( CT B )
<s> which Ġch inese Ġdatasets Ġwere Ġused ? </s> Introduction </s> G ram mar Ġinduction Ġis Ġthe Ġtask Ġof Ġinducing Ġhierarchical Ġsynt actic Ġstructure Ġfrom Ġdata . ĠStatistical Ġapproaches Ġto Ġgrammar Ġinduction Ġrequire Ġspecifying Ġa Ġprob abil istic Ġgrammar Ġ( e . g . Ġformal ism , Ġnumber Ġand Ġshape Ġof Ġrules ), Ġand Ġfitting Ġits Ġparameters Ġthrough Ġoptimization . ĠEarly Ġwork Ġfound Ġthat Ġit Ġwas Ġdifficult Ġto Ġinduce Ġprob abil istic Ġcontext - free Ġgram m ars Ġ( PC FG ) Ġfrom Ġnatural Ġlanguage Ġdata Ġthrough Ġdirect Ġmethods , Ġsuch Ġas Ġoptimizing Ġthe Ġlog Ġlikelihood Ġwith Ġthe ĠEM Ġalgorithm ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠWhile Ġthe Ġreasons Ġfor Ġthe Ġfailure Ġare Ġmanifold Ġand Ġnot Ġcompletely Ġunderstood , Ġtwo Ġmajor Ġpotential Ġcauses Ġare Ġthe Ġill - beh aved Ġoptimization Ġlandscape Ġand Ġthe Ġoverly Ġstrict Ġindependence Ġassumptions Ġof ĠPC FG s . ĠMore Ġsuccessful Ġapproaches Ġto Ġgrammar Ġinduction Ġhave Ġthus Ġresorted Ġto Ġcarefully - crafted Ġauxiliary Ġobjectives ĠB IB REF 2 Ġ, Ġpri ors Ġor Ġnon - param etric Ġmodels ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġand Ġmanually - engine ered Ġfeatures ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġto Ġencourage Ġthe Ġdesired Ġstructures Ġto Ġemerge . </s> We Ġrevisit Ġthese Ġaforementioned Ġissues Ġin Ġlight Ġof Ġadvances Ġin Ġmodel Ġparameter ization Ġand Ġinference . ĠFirst , Ġcontrary Ġto Ġcommon Ġwisdom , Ġwe Ġfind Ġthat Ġparameter izing Ġa ĠPC FG 's Ġrule Ġprobabilities Ġwith Ġneural Ġnetworks Ġover Ġdistributed Ġrepresentations Ġmakes Ġit Ġpossible Ġto Ġinduce Ġlingu istically Ġmeaningful Ġgram m ars Ġby Ġsimply Ġoptimizing Ġlog Ġlikelihood . ĠWhile Ġthe Ġoptimization Ġproblem Ġremains Ġnon - con ve x , Ġrecent Ġwork Ġsuggests Ġthat Ġthere Ġare Ġoptimization Ġbenefits Ġafforded Ġby Ġover - param eter ized Ġmodels ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, Ġand Ġwe Ġindeed Ġfind Ġthat Ġthis Ġneural ĠPC FG Ġis Ġsignificantly Ġeasier Ġto Ġoptimize Ġthan Ġthe Ġtraditional ĠPC FG . ĠSecond , Ġthis Ġfact ored Ġparameter ization Ġmakes Ġit Ġstraightforward Ġto Ġincorporate Ġside Ġinformation Ġinto Ġrule Ġprobabilities Ġthrough Ġa Ġsentence - level Ġcontinuous Ġlatent Ġvector , Ġwhich Ġeffectively Ġallows Ġdifferent Ġcontexts Ġin Ġa Ġderiv ation Ġto Ġcoordinate . ĠIn Ġthis Ġcompound ĠPC FG âĢĶ contin uous Ġmixture Ġof ĠPC FG s âĢĶ the Ġcontext - free Ġassumptions Ġhold Ġconditioned Ġon Ġthe Ġlatent Ġvector Ġbut Ġnot Ġuncond itionally , Ġthereby Ġobtaining Ġlonger - range Ġdependencies Ġwithin Ġa Ġtree - based Ġgener ative Ġprocess . </s> To Ġutilize Ġthis Ġapproach , Ġwe Ġneed Ġto Ġefficiently Ġoptimize Ġthe Ġlog Ġmarginal Ġlikelihood Ġof Ġobserved Ġsentences . ĠWhile Ġcompound ĠPC FG s Ġbreak Ġefficient Ġinference , Ġif Ġthe Ġlatent Ġvector Ġis Ġknown Ġthe Ġdistribution Ġover Ġtrees Ġreduces Ġto Ġa Ġstandard ĠPC FG . ĠThis Ġproperty Ġallows Ġus Ġto Ġperform Ġgrammar Ġinduction Ġusing Ġa Ġcollapsed Ġapproach Ġwhere Ġthe Ġlatent Ġtrees Ġare Ġmarginalized Ġout Ġexactly Ġwith Ġdynamic Ġprogramming . ĠTo Ġhandle Ġthe Ġlatent Ġvector , Ġwe Ġemploy Ġstandard Ġam ort ized Ġinference Ġusing Ġrep aram eter ized Ġsamples Ġfrom Ġa Ġvari ational Ġposterior Ġapprox imated Ġfrom Ġan Ġinference Ġnetwork ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> On Ġstandard Ġbenchmarks Ġfor ĠEnglish Ġand ĠChinese , Ġthe Ġproposed Ġapproach Ġis Ġfound Ġto Ġperform Ġfavorably Ġagainst Ġrecent Ġneural Ġnetwork - based Ġapproaches Ġto Ġgrammar Ġinduction ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ. </s> Pro b abil istic ĠContext - Free ĠGram m ars </s> We Ġconsider Ġcontext - free Ġgram m ars Ġ( CF G ) Ġconsisting Ġof Ġa Ġ5 - t uple ĠIN LINE FORM 0 Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġdistinguished Ġstart Ġsymbol , ĠIN LINE FORM 2 Ġis Ġa Ġfinite Ġset Ġof Ġnon termin als , ĠIN LINE FORM 3 Ġis Ġa Ġfinite Ġset Ġof Ġpre termin als , ĠIN LINE FORM 6 Ġis Ġa Ġfinite Ġset Ġof Ġterminal Ġsymbols , Ġand ĠIN LINE FORM 7 Ġis Ġa Ġfinite Ġset Ġof Ġrules Ġof Ġthe Ġform , </s> IN LINE FORM 0 </s> A Ġprob abil istic Ġcontext - free Ġgrammar Ġ( PC FG ) Ġconsists Ġof Ġa Ġgrammar ĠIN LINE FORM 0 Ġand Ġrule Ġprobabilities ĠIN LINE FORM 1 Ġsuch Ġthat ĠIN LINE FORM 2 Ġis Ġthe Ġprobability Ġof Ġthe Ġrule ĠIN LINE FORM 3 Ġ. ĠLet ting ĠIN LINE FORM 4 Ġbe Ġthe Ġset Ġof Ġall Ġparse Ġtrees Ġof ĠIN LINE FORM 5 Ġ, Ġa ĠPC FG Ġdefines Ġa Ġprobability Ġdistribution Ġover ĠIN LINE FORM 6 Ġvia ĠIN LINE FORM 7 Ġwhere ĠIN LINE FORM 8 Ġis Ġthe Ġset Ġof Ġrules Ġused Ġin Ġthe Ġderiv ation Ġof ĠIN LINE FORM 9 Ġ. ĠIt Ġalso Ġdefines Ġa Ġdistribution Ġover Ġstring Ġof Ġterminals ĠIN LINE FORM 10 Ġvia </s> IN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġ, Ġi . e . Ġthe Ġset Ġof Ġtrees ĠIN LINE FORM 1 Ġsuch Ġthat ĠIN LINE FORM 2 Ġ' s Ġleaves Ġare ĠIN LINE FORM 3 Ġ. ĠWe Ġwill Ġslightly Ġabuse Ġnotation Ġand Ġuse </s> IN LINE FORM 0 </s> to Ġdenote Ġthe Ġposterior Ġdistribution Ġover Ġthe Ġunob served Ġlatent Ġtrees Ġgiven Ġthe Ġobserved Ġsentence ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġindicator Ġfunction . </s> Comp ound ĠPC FG s </s> A Ġcompound Ġprobability Ġdistribution ĠB IB REF 19 Ġis Ġa Ġdistribution Ġwhose Ġparameters Ġare Ġthemselves Ġrandom Ġvariables . ĠThese Ġdistributions Ġgeneral ize Ġmixture Ġmodels Ġto Ġthe Ġcontinuous Ġcase , Ġfor Ġexample Ġin Ġfactor Ġanalysis Ġwhich Ġassumes Ġthe Ġfollowing Ġgener ative Ġprocess , </s> IN LINE FORM 0 </s> Comp ound Ġdistributions Ġprovide Ġthe Ġability Ġto Ġmodel Ġrich Ġgener ative Ġprocesses , Ġbut Ġmarginal izing Ġover Ġthe Ġlatent Ġparameter Ġcan Ġbe Ġcomput ationally Ġint ract able Ġunless Ġconj ug acy Ġcan Ġbe Ġexploited . </s> In Ġthis Ġwork , Ġwe Ġstudy Ġcompound Ġprob abil istic Ġcontext - free Ġgram m ars Ġwhose Ġdistribution Ġover Ġtrees Ġarises Ġfrom Ġthe Ġfollowing Ġgener ative Ġprocess : Ġwe Ġfirst Ġobtain Ġrule Ġprobabilities Ġvia </s> IN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġprior Ġwith Ġparameters ĠIN LINE FORM 1 Ġ( sp herical ĠGa ussian Ġin Ġthis Ġpaper ), Ġand ĠIN LINE FORM 2 Ġis Ġa Ġneural Ġnetwork Ġthat Ġconc aten ates Ġthe Ġinput Ġsymbol Ġembed d ings Ġwith ĠIN LINE FORM 3 Ġand Ġoutputs Ġthe Ġsentence - level Ġrule Ġprobabilities ĠIN LINE FORM 4 Ġ, </s> IN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġvector Ġconc aten ation . ĠThen Ġa Ġtree / sent ence Ġis Ġsampled Ġfrom Ġa ĠPC FG Ġwith Ġrule Ġprobabilities Ġgiven Ġby ĠIN LINE FORM 1 Ġ, </s> IN LINE FORM 0 </s> This Ġcan Ġbe Ġviewed Ġas Ġa Ġcontinuous Ġmixture Ġof ĠPC FG s , Ġor Ġalternatively , Ġa ĠBay esian ĠPC FG Ġwith Ġa Ġprior Ġon Ġsentence - level Ġrule Ġprobabilities Ġparameter ized Ġby ĠIN LINE FORM 0 Ġ. ĠImport antly , Ġunder Ġthis Ġgener ative Ġmodel Ġthe Ġcontext - free Ġassumptions Ġhold Ġconditioned Ġon ĠIN LINE FORM 3 Ġ, Ġbut Ġthey Ġdo Ġnot Ġhold Ġuncond itionally . ĠThis Ġis Ġshown Ġin ĠFigure ĠFIG REF 3 Ġ( right ) Ġwhere Ġthere Ġis Ġa Ġdependence Ġpath Ġthrough ĠIN LINE FORM 4 Ġif Ġit Ġis Ġnot Ġconditioned Ġupon . ĠComp ound ĠPC FG s Ġgive Ġrise Ġto Ġa Ġmarginal Ġdistribution Ġover Ġparse Ġtrees ĠIN LINE FORM 5 Ġvia </s> IN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġ. ĠThe Ġsubscript Ġin ĠIN LINE FORM 1 Ġdenotes Ġthe Ġfact Ġthat Ġthe Ġrule Ġprobabilities Ġdepend Ġon ĠIN LINE FORM 2 Ġ. ĠComp ound ĠPC FG s Ġare Ġclearly Ġmore Ġexpressive Ġthan ĠPC FG s Ġas Ġeach Ġsentence Ġhas Ġits Ġown Ġset Ġof Ġrule Ġprobabilities . ĠHowever , Ġit Ġstill Ġassumes Ġa Ġtree - based Ġgener ative Ġprocess , Ġmaking Ġit Ġpossible Ġto Ġlearn Ġlatent Ġtree Ġstructures . </s> Our Ġmotivation Ġfor Ġthe Ġcompound ĠPC FG Ġis Ġbased Ġon Ġthe Ġobservation Ġthat Ġfor Ġgrammar Ġinduction , Ġcontext - free Ġassumptions Ġare Ġgenerally Ġmade Ġnot Ġbecause Ġthey Ġrepresent Ġan Ġadequate Ġmodel Ġof Ġnatural Ġlanguage , Ġbut Ġbecause Ġthey Ġallow Ġfor Ġtract able Ġtraining . ĠWe Ġcan Ġin Ġprinciple Ġmodel Ġricher Ġdependencies Ġthrough Ġvertical / hor izontal ĠMark ov ization ĠB IB REF 21 Ġ, ĠB IB REF 22 Ġand Ġlex ical ization ĠB IB REF 23 Ġ. ĠHowever Ġsuch Ġdependencies Ġcomplicate Ġtraining Ġdue Ġto Ġthe Ġrapid Ġincrease Ġin Ġthe Ġnumber Ġof Ġrules . ĠUnder Ġthis Ġview , Ġwe Ġcan Ġinterpret Ġthe Ġcompound ĠPC FG Ġas Ġa Ġrestricted Ġversion Ġof Ġsome Ġlex ical ized , Ġhigher - order ĠPC FG Ġwhere Ġa Ġchild Ġcan Ġdepend Ġon Ġstructural Ġand Ġlex ical Ġcontext Ġthrough Ġa Ġshared Ġlatent Ġvector . ĠWe Ġhypothes ize Ġthat Ġthis Ġdependence Ġamong Ġsiblings Ġis Ġespecially Ġuseful Ġin Ġgrammar Ġinduction Ġfrom Ġwords , Ġwhere Ġ( for Ġexample ) Ġif Ġwe Ġknow Ġthat Ġwatched Ġis Ġused Ġas Ġa Ġverb Ġthen Ġthe Ġnoun Ġphrase Ġis Ġlikely Ġto Ġbe Ġa Ġmovie . </s> In Ġcontrast Ġto Ġthe Ġusual ĠBay esian Ġtreatment Ġof ĠPC FG s Ġwhich Ġplaces Ġpri ors Ġon Ġglobal Ġrule Ġprobabilities ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 6 Ġ, Ġthe Ġcompound ĠPC FG Ġassumes Ġa Ġprior Ġon Ġlocal , Ġsentence - level Ġrule Ġprobabilities . ĠIt Ġis Ġtherefore Ġclosely Ġrelated Ġto Ġthe ĠBay esian Ġgram m ars Ġstudied Ġby ĠB IB REF 25 Ġand ĠB IB REF 26 Ġ, Ġwho Ġalso Ġsample Ġlocal Ġrule Ġprobabilities Ġfrom Ġa Ġlog istic Ġnormal Ġprior Ġfor Ġtraining Ġdependency Ġmodels Ġwith Ġval ence Ġ( DM V ) ĠB IB REF 27 Ġ. </s> Exper imental ĠSetup </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 23 Ġshows Ġthe Ġunl abel ed ĠIN LINE FORM 0 Ġscores Ġfor Ġour Ġmodels Ġand Ġvarious Ġbas elines . ĠAll Ġmodels Ġsound ly Ġoutper form Ġright Ġbranching Ġbas elines , Ġand Ġwe Ġfind Ġthat Ġthe Ġneural ĠPC FG / comp ound ĠPC FG Ġare Ġstrong Ġmodels Ġfor Ġgrammar Ġinduction . ĠIn Ġparticular Ġthe Ġcompound ĠPC FG Ġoutper forms Ġother Ġmodels Ġby Ġan Ġappreci able Ġmargin Ġon Ġboth ĠEnglish Ġand ĠChinese . ĠWe Ġagain Ġnote Ġthat Ġwe Ġwere Ġunable Ġto Ġinduce Ġmeaningful Ġgram m ars Ġthrough Ġa Ġtraditional ĠPC FG Ġwith Ġthe Ġscal ar Ġparameter ization Ġdespite Ġa Ġthorough Ġhyper param eter Ġsearch . ĠSee Ġlab : full Ġfor Ġthe Ġfull Ġresults Ġ( including Ġcorpus - level ĠIN LINE FORM 1 Ġ) Ġbroken Ġdown Ġby Ġsentence Ġlength . </s> Table ĠT AB REF 27 Ġanaly zes Ġthe Ġlearned Ġtree Ġstructures . ĠWe Ġcompare Ġsimilarity Ġas Ġmeasured Ġby ĠIN LINE FORM 0 Ġagainst Ġgold , Ġleft , Ġright , Ġand ĠâĢ ľ self " Ġtrees Ġ( top ), Ġwhere Ġself ĠIN LINE FORM 1 Ġscore Ġis Ġcalculated Ġby Ġaveraging Ġover Ġall Ġ6 Ġpairs Ġobtained Ġfrom Ġ4 Ġdifferent Ġruns . ĠWe Ġfind Ġthat ĠPR PN Ġis Ġparticularly Ġconsistent Ġacross Ġmultiple Ġruns . ĠWe Ġalso Ġobserve Ġthat Ġdifferent Ġmodels Ġare Ġbetter Ġat Ġidentifying Ġdifferent Ġconstituent Ġlabels , Ġas Ġmeasured Ġby Ġlabel Ġrecall Ġ( Table ĠT AB REF 27 Ġ, Ġbottom ). ĠWhile Ġleft Ġas Ġfuture Ġwork , Ġthis Ġnaturally Ġsuggests Ġan Ġensemble Ġapproach Ġwherein Ġthe Ġempirical Ġprobabilities Ġof Ġconstituents Ġ( ob tained Ġby Ġaveraging Ġthe Ġpredicted Ġbinary Ġconstituent Ġlabels Ġfrom Ġthe Ġdifferent Ġmodels ) Ġare Ġused Ġeither Ġto Ġsuperv ise Ġanother Ġmodel Ġor Ġdirectly Ġas Ġpotential s Ġin Ġa ĠCR F Ġconstituency Ġparser . ĠFinally , Ġall Ġmodels Ġseemed Ġto Ġhave Ġsome Ġdifficulty Ġin Ġidentifying ĠSB AR / VP Ġconstituents Ġwhich Ġtypically Ġspan Ġmore Ġwords Ġthan ĠNP Ġconstituents . </s> Related ĠWork </s> G ram mar Ġinduction Ġhas Ġa Ġlong Ġand Ġrich Ġhistory Ġin Ġnatural Ġlanguage Ġprocessing . ĠEarly Ġwork Ġon Ġgrammar Ġinduction Ġwith Ġpure Ġun super vised Ġlearning Ġwas Ġmostly Ġnegative ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 74 Ġ, Ġthough ĠB IB REF 75 Ġreported Ġsome Ġsuccess Ġon Ġpartially Ġbracket ed Ġdata . ĠB IB REF 76 Ġand ĠB IB REF 2 Ġwere Ġsome Ġof Ġthe Ġfirst Ġsuccessful Ġstatistical Ġapproaches Ġto Ġgrammar Ġinduction . ĠIn Ġparticular , Ġthe Ġconstituent - context Ġmodel Ġ( CC M ) Ġof ĠB IB REF 2 Ġ, Ġwhich Ġexplicitly Ġmodels Ġboth Ġconstituents Ġand Ġdist itu ents , Ġwas Ġthe Ġbasis Ġfor Ġmuch Ġsubsequent Ġwork ĠB IB REF 27 Ġ, ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠOther Ġworks Ġhave Ġexplored Ġimposing Ġinduct ive Ġbiases Ġthrough ĠBay esian Ġpri ors ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, Ġmodified Ġobjectives ĠB IB REF 42 Ġ, Ġand Ġadditional Ġconstraints Ġon Ġrec ursion Ġdepth ĠB IB REF 77 Ġ, ĠB IB REF 48 Ġ. </s> While Ġthe Ġframework Ġof Ġspecifying Ġthe Ġstructure Ġof Ġa Ġgrammar Ġand Ġlearning Ġthe Ġparameters Ġis Ġcommon , Ġother Ġmethods Ġexist . ĠB IB REF 43 Ġconsider Ġa Ġnon param etric - style Ġapproach Ġto Ġun super vised Ġparsing Ġby Ġusing Ġrandom Ġsubs ets Ġof Ġtraining Ġsubt rees Ġto Ġparse Ġnew Ġsentences . ĠB IB REF 46 Ġutilize Ġan Ġincremental Ġalgorithm Ġto Ġun super vised Ġparsing Ġwhich Ġmakes Ġlocal Ġdecisions Ġto Ġcreate Ġconstituents Ġbased Ġon Ġa Ġcomplex Ġset Ġof Ġhe ur istics . ĠB IB REF 47 Ġinduce Ġparse Ġtrees Ġthrough Ġcasc aded Ġapplications Ġof Ġfinite Ġstate Ġmodels . </s> More Ġrecently , Ġneural Ġnetwork - based Ġapproaches Ġto Ġgrammar Ġinduction Ġhave Ġshown Ġpromising Ġresults Ġon Ġinducing Ġparse Ġtrees Ġdirectly Ġfrom Ġwords . ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġlearn Ġtree Ġstructures Ġthrough Ġsoft Ġg ating Ġlayers Ġwithin Ġneural Ġlanguage Ġmodels , Ġwhile ĠB IB REF 16 Ġcombine Ġrecursive Ġaut oen cod ers Ġwith Ġthe Ġinside - outside Ġalgorithm . ĠB IB REF 17 Ġtrain Ġun super vised Ġrecurrent Ġneural Ġnetwork Ġgram m ars Ġwith Ġa Ġstructured Ġinference Ġnetwork Ġto Ġinduce Ġlatent Ġtrees , Ġand ĠB IB REF 78 Ġutilize Ġimage Ġcapt ions Ġto Ġidentify Ġand Ġground Ġconstituents . </s> Our Ġwork Ġis Ġalso Ġrelated Ġto Ġlatent Ġvariable ĠPC FG s ĠB IB REF 79 Ġ, ĠB IB REF 80 Ġ, ĠB IB REF 81 Ġ, Ġwhich Ġextend ĠPC FG s Ġto Ġthe Ġlatent Ġvariable Ġsetting Ġby Ġsplitting Ġnon termin al Ġsymbols Ġinto Ġlatent Ġsub sy mb ols . ĠIn Ġparticular , Ġlatent Ġvector Ġgram m ars ĠB IB REF 82 Ġand Ġcompos itional Ġvector Ġgram m ars ĠB IB REF 83 Ġalso Ġemploy Ġcontinuous Ġvectors Ġwithin Ġtheir Ġgram m ars . ĠHowever Ġthese Ġapproaches Ġhave Ġbeen Ġemployed Ġfor Ġlearning Ġsupervised Ġpars ers Ġon Ġannot ated Ġtree banks , Ġin Ġcontrast Ġto Ġthe Ġun super vised Ġsetting Ġof Ġthe Ġcurrent Ġwork . </s> Conclusion </s> This Ġwork Ġexplores Ġgrammar Ġinduction Ġwith Ġcompound ĠPC FG s , Ġwhich Ġmod ulate Ġrule Ġprobabilities Ġwith Ġper - sent ence Ġcontinuous Ġlatent Ġvectors . ĠThe Ġlatent Ġvector Ġinduces Ġmarginal Ġdependencies Ġbeyond Ġthe Ġtraditional Ġfirst - order Ġcontext - free Ġassumptions Ġwithin Ġa Ġtree - based Ġgener ative Ġprocess , Ġleading Ġto Ġimproved Ġperformance . ĠThe Ġcollapsed Ġam ort ized Ġvari ational Ġinference Ġapproach Ġis Ġgeneral Ġand Ġcan Ġbe Ġused Ġfor Ġgener ative Ġmodels Ġwhich Ġadmit Ġtract able Ġinference Ġthrough Ġpartial Ġconditioning . ĠLearning Ġdeep Ġgener ative Ġmodels Ġwhich Ġexhibit Ġsuch Ġconditional ĠMark ov Ġproperties Ġis Ġan Ġinteresting Ġdirection Ġfor Ġfuture Ġwork . </s> Acknowled gments </s> We Ġthank ĠPhil ĠBl uns om Ġfor Ġinitial Ġdiscussions Ġwhich Ġseeded Ġmany Ġof Ġthe Ġcore Ġideas Ġin Ġthe Ġpresent Ġwork . ĠWe Ġalso Ġthank ĠYon atan ĠBel ink ov Ġand ĠShay ĠCohen Ġfor Ġhelpful Ġfeedback , Ġand ĠAndrew ĠDro z d ov Ġfor Ġproviding Ġthe Ġparsed Ġdataset Ġfrom Ġtheir ĠDI OR A Ġmodel . ĠY K Ġis Ġsupported Ġby Ġa ĠGoogle ĠFellowship . ĠAM R Ġacknowledges Ġthe Ġsupport Ġof ĠNS F Ġ170 48 34 , Ġ18 456 64 , ĠAWS , Ġand ĠOracle . </s> Model ĠParam eter ization </s> We Ġassociate Ġan Ġinput Ġembed ding ĠIN LINE FORM 0 Ġfor Ġeach Ġsymbol ĠIN LINE FORM 1 Ġon Ġthe Ġleft Ġside Ġof Ġa Ġrule Ġ( i . e . ĠIN LINE FORM 2 Ġ) Ġand Ġrun Ġa Ġneural Ġnetwork Ġover ĠIN LINE FORM 3 Ġto Ġobtain Ġthe Ġrule Ġprobabilities . ĠCon crete ly , Ġeach Ġrule Ġtype ĠIN LINE FORM 4 Ġis Ġparameter ized Ġas Ġfollows , ĠIN LINE FORM 5 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġproduct Ġspace ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġare ĠML Ps Ġwith Ġtwo Ġresidual Ġlayers , ĠIN LINE FORM 3 </s> The Ġbias Ġterms Ġfor Ġthe Ġabove Ġexpressions Ġ( including Ġfor Ġthe Ġrule Ġprobabilities ) Ġare Ġomitted Ġfor Ġnot ational Ġbre vity . ĠIn ĠFigure ĠFIG REF 3 Ġwe Ġuse Ġthe Ġfollowing Ġto Ġrefer Ġto Ġrule Ġprobabilities Ġof Ġdifferent Ġrule Ġtypes , ĠIN LINE FORM 0 </s> where ĠIN LINE FORM 0 Ġdenotes Ġthe Ġset Ġof Ġrules Ġwith ĠIN LINE FORM 1 Ġon Ġthe Ġleft Ġhand Ġside . </s> The Ġcompound ĠPC FG Ġrule Ġprobabilities ĠIN LINE FORM 0 Ġgiven Ġa Ġlatent Ġvector ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 </s> Again Ġthe Ġbias Ġterms Ġare Ġomitted Ġfor Ġbre vity , Ġand ĠIN LINE FORM 0 Ġare Ġas Ġbefore Ġwhere Ġthe Ġfirst Ġlayer 's Ġinput Ġdimensions Ġare Ġappropriately Ġchanged Ġto Ġaccount Ġfor Ġconc aten ation Ġwith ĠIN LINE FORM 1 Ġ. </s> Corp us / Sent ence ĠF Ġ1 ĠF _ 1 Ġby ĠSent ence ĠLength </s> For Ġcomple teness Ġwe Ġshow Ġthe Ġcorpus - level Ġand Ġsentence - level ĠIN LINE FORM 0 Ġbroken Ġdown Ġby Ġsentence Ġlength Ġin ĠTable ĠT AB REF 44 Ġ, Ġaveraged Ġacross Ġ4 Ġdifferent Ġruns Ġof Ġeach Ġmodel . </s> Exper iments Ġwith ĠR NN Gs </s> For Ġexperiments Ġon Ġsuperv ising ĠR NN Gs Ġwith Ġinduced Ġtrees , Ġwe Ġuse Ġthe Ġparameter ization Ġand Ġhyper param eters Ġfrom ĠB IB REF 17 Ġ, Ġwhich Ġuses Ġa Ġ2 - layer Ġ650 - dimensional Ġstack ĠL ST M Ġ( with Ġdrop out Ġof Ġ0 . 5 ) Ġand Ġa Ġ650 - dimensional Ġtree ĠL ST M ĠB IB REF 88 Ġ, ĠB IB REF 90 Ġas Ġthe Ġcomposition Ġfunction . </s> Con crete ly , Ġthe Ġgener ative Ġstory Ġis Ġas Ġfollows : Ġfirst , Ġthe Ġstack Ġrepresentation Ġis Ġused Ġto Ġpredict Ġthe Ġnext Ġaction Ġ( shift Ġor Ġreduce ) Ġvia Ġan Ġaff ine Ġtransformation Ġfollowed Ġby Ġa Ġs igm oid . ĠIf Ġshift Ġis Ġchosen , Ġwe Ġobtain Ġa Ġdistribution Ġover Ġthe Ġvocabulary Ġvia Ġanother Ġaff ine Ġtransformation Ġover Ġthe Ġstack Ġrepresentation Ġfollowed Ġby Ġa Ġsoft max . ĠThen Ġwe Ġsample Ġthe Ġnext Ġword Ġfrom Ġthis Ġdistribution Ġand Ġshift Ġthe Ġgenerated Ġword Ġonto Ġthe Ġstack Ġusing Ġthe Ġstack ĠL ST M . ĠIf Ġreduce Ġis Ġchosen , Ġwe Ġpop Ġthe Ġlast Ġtwo Ġelements Ġoff Ġthe Ġstack Ġand Ġuse Ġthe Ġtree ĠL ST M Ġto Ġobtain Ġa Ġnew Ġrepresentation . ĠThis Ġnew Ġrepresentation Ġis Ġshifted Ġonto Ġthe Ġstack Ġvia Ġthe Ġstack ĠL ST M . ĠNote Ġthat Ġthis ĠR NN G Ġparameter ization Ġis Ġslightly Ġdifferent Ġthan Ġthe Ġoriginal Ġfrom ĠB IB REF 53 Ġ, Ġwhich Ġdoes Ġnot Ġignore Ġconstituent Ġlabels Ġand Ġutilizes Ġa Ġbid irection al ĠL ST M Ġas Ġthe Ġcomposition Ġfunction Ġinstead Ġof Ġa Ġtree ĠL ST M . ĠAs Ġour ĠR NN G Ġparameter ization Ġonly Ġworks Ġwith Ġbinary Ġtrees , Ġwe Ġbin ar ize Ġthe Ġgold Ġtrees Ġwith Ġright Ġbin ar ization Ġfor Ġthe ĠR NN G Ġtrained Ġon Ġgold Ġtrees Ġ( t rees Ġfrom Ġthe Ġun super vised Ġmethods Ġexplored Ġin Ġthis Ġpaper Ġare Ġalready Ġbinary ). ĠThe ĠR NN G Ġalso Ġtrains Ġa Ġdiscrim in ative Ġparser Ġalongside Ġthe Ġgener ative Ġmodel Ġfor Ġevaluation Ġwith Ġimportance Ġsampling . ĠWe Ġuse Ġa ĠCR F Ġparser Ġwhose Ġspan Ġscore Ġparameter ization Ġis Ġsimilar Ġsimilar Ġto Ġrecent Ġworks ĠB IB REF 89 Ġ, ĠB IB REF 87 Ġ, ĠB IB REF 85 Ġ: Ġposition Ġembed d ings Ġare Ġadded Ġto Ġword Ġembed d ings , Ġand Ġa Ġbid irection al ĠL ST M Ġwith Ġ256 Ġhidden Ġdimensions Ġis Ġrun Ġover Ġthe Ġinput Ġrepresentations Ġto Ġobtain Ġthe Ġforward Ġand Ġbackward Ġhidden Ġstates . ĠThe Ġscore ĠIN LINE FORM 0 Ġfor Ġa Ġconstituent Ġspanning Ġthe ĠIN LINE FORM 1 Ġ- th Ġand ĠIN LINE FORM 2 Ġ- th Ġword Ġis Ġgiven Ġby , </s> IN LINE FORM 0 </s> where Ġthe ĠML P Ġhas Ġa Ġsingle Ġhidden Ġlayer Ġwith ĠIN LINE FORM 0 Ġnon linear ity Ġfollowed Ġby Ġlayer Ġnormal ization ĠB IB REF 84 Ġ. </s> For Ġexperiments Ġon Ġfine - tun ing Ġthe ĠR NN G Ġwith Ġthe Ġun super vised ĠR NN G , Ġwe Ġtake Ġthe Ġdiscrim in ative Ġparser Ġ( which Ġis Ġalso Ġpret rained Ġalongside Ġthe ĠR NN G Ġon Ġinduced Ġtrees ) Ġto Ġbe Ġthe Ġstructured Ġinference Ġnetwork Ġfor Ġoptimizing Ġthe Ġevidence Ġlower Ġbound . ĠWe Ġrefer Ġthe Ġreader Ġto ĠB IB REF 17 Ġand Ġtheir Ġopen Ġsource Ġimplementation Ġfor Ġadditional Ġdetails . ĠWe Ġalso Ġobserve Ġthat Ġas Ġnoted Ġby ĠB IB REF 17 Ġ, Ġa ĠUR NN G Ġtrained Ġfrom Ġscratch Ġon Ġthis Ġversion Ġof ĠPT B Ġwithout Ġpunct uation Ġfailed Ġto Ġoutper form Ġa Ġright - br anch ing Ġbaseline . </s> The ĠL ST M Ġlanguage Ġmodel Ġbaseline Ġis Ġthe Ġsame Ġsize Ġas Ġthe Ġstack ĠL ST M Ġ( i . e . Ġ2 Ġlayers , Ġ650 Ġhidden Ġunits , Ġdrop out Ġof Ġ0 . 5 ), Ġand Ġis Ġtherefore Ġequivalent Ġto Ġan ĠR NN G Ġwith Ġcompletely Ġright Ġbranching Ġtrees . ĠThe ĠPR PN / ON Ġbas elines Ġfor Ġperplex ity / sy nt actic Ġevaluation Ġin ĠTable ĠT AB REF 30 Ġalso Ġhave Ġ2 Ġlayers Ġwith Ġ650 Ġhidden Ġunits Ġand Ġ0 . 5 Ġdrop out . ĠTherefore Ġall Ġmodels Ġconsidered Ġin ĠTable ĠT AB REF 30 Ġhave Ġroughly Ġthe Ġsame Ġcapacity . ĠFor Ġall Ġmodels Ġwe Ġshare Ġinput / output Ġword Ġembed d ings ĠB IB REF 86 Ġ. ĠPer plex ity Ġestimation Ġfor Ġthe ĠR NN Gs Ġand Ġthe Ġcompound ĠPC FG Ġuses Ġ1000 Ġimportance - weight ed Ġsamples . </s> For Ġgram matical ity Ġjudgment , Ġwe Ġmodify Ġthe Ġpublicly Ġavailable Ġdataset Ġfrom ĠB IB REF 56 Ġto Ġonly Ġkeep Ġsentence Ġpairs Ġthat Ġdid Ġnot Ġhave Ġany Ġunknown Ġwords Ġwith Ġrespect Ġto Ġour ĠPT B Ġvocabulary Ġof Ġ10 K Ġwords . ĠThis Ġresults Ġin Ġ33 K Ġsentence Ġpairs Ġfor Ġevaluation . </s> Non termin al / Pre termin al ĠAl ign ments </s> Figure ĠFIG REF 50 Ġshows Ġthe Ġpart - of - speech Ġalign ments Ġand ĠTable ĠT AB REF 46 Ġshows Ġthe Ġnon termin al Ġlabel Ġalign ments Ġfor Ġthe Ġcompound ĠPC FG / ne ural ĠPC FG . </s> Sub tree ĠAnalysis </s> Table ĠT AB REF 53 Ġlists Ġmore Ġexamples Ġof Ġconstituents Ġwithin Ġeach Ġsubt ree Ġas Ġthe Ġtop Ġprincip ical Ġcomponent Ġis Ġvaried . ĠDue Ġto Ġdata Ġsp arsity , Ġthe Ġsubt ree Ġanalysis Ġis Ġperformed Ġon Ġthe Ġfull Ġdataset . ĠSee Ġsection ĠUID 36 Ġfor Ġmore Ġdetails .