The neural projector must be invertible.
ABSTRACTIVE
we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists
EXTRACTIVE
In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge. Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists. Invertible transformations have been explored before in independent components analysis BIBREF14 , gaussianization BIBREF15 , and deep density models BIBREF16 , BIBREF17 , BIBREF18 , for unstructured data. Here, we generalize this style of approach to structured learning, and augment it with discrete latent variables ( INLINEFORM2 ). Under the invertibility condition, we derive a learning algorithm and give another view of our approach revealed by the objective function. Then, we present the architecture of a neural projector we use in experiments: a volume-preserving invertible neural network proposed by BIBREF16 for independent components estimation.
In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge. Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists. Invertible transformations have been explored before in independent components analysis BIBREF14 , gaussianization BIBREF15 , and deep density models BIBREF16 , BIBREF17 , BIBREF18 , for unstructured data. Here, we generalize this style of approach to structured learning, and augment it with discrete latent variables ( INLINEFORM2 ). Under the invertibility condition, we derive a learning algorithm and give another view of our approach revealed by the objective function. Then, we present the architecture of a neural projector we use in experiments: a volume-preserving invertible neural network proposed by BIBREF16 for independent components estimation.
1808.09111
Introduction </s> Data Ġannotation Ġis Ġa Ġmajor Ġbottleneck Ġfor Ġthe Ġapplication Ġof Ġsupervised Ġlearning Ġapproaches Ġto Ġmany Ġproblems . ĠAs Ġa Ġresult , Ġun super vised Ġmethods Ġthat Ġlearn Ġdirectly Ġfrom Ġunl abel ed Ġdata Ġare Ġincreasingly Ġimportant . ĠFor Ġtasks Ġrelated Ġto Ġun super vised Ġsynt actic Ġanalysis , Ġdiscrete Ġgener ative Ġmodels Ġhave Ġdominated Ġin Ġrecent Ġyears ĠâĢĵ Ġfor Ġexample , Ġfor Ġboth Ġpart - of - speech Ġ( POS ) Ġinduction ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġand Ġun super vised Ġdependency Ġparsing ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠWhile Ġsimilar Ġmodels Ġhave Ġhad Ġsuccess Ġon Ġa Ġrange Ġof Ġun super vised Ġtasks , Ġthey Ġhave Ġmostly Ġignored Ġthe Ġapparent Ġutility Ġof Ġcontinuous Ġword Ġrepresentations Ġevident Ġfrom Ġsupervised ĠN LP Ġapplications ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġfocus Ġon Ġleveraging Ġand Ġexplicitly Ġrepresenting Ġcontinuous Ġword Ġembed d ings Ġwithin Ġun super vised Ġmodels Ġof Ġsynt actic Ġstructure . </s> Pre - trained Ġword Ġembed d ings Ġfrom Ġmassive Ġunl abel ed Ġcorpor a Ġoffer Ġa Ġcompact Ġway Ġof Ġinjecting Ġa Ġprior Ġnotion Ġof Ġword Ġsimilarity Ġinto Ġmodels Ġthat Ġwould Ġotherwise Ġtreat Ġwords Ġas Ġdiscrete , Ġisolated Ġcategories . ĠHowever , Ġthe Ġspecific Ġproperties Ġof Ġlanguage Ġcaptured Ġby Ġany Ġparticular Ġembed ding Ġscheme Ġcan Ġbe Ġdifficult Ġto Ġcontrol , Ġand , Ġfurther , Ġmay Ġnot Ġbe Ġideally Ġsuited Ġto Ġthe Ġtask Ġat Ġhand . ĠFor Ġexample , Ġpre - trained Ġskip - gram Ġembed d ings ĠB IB REF 7 Ġwith Ġsmall Ġcontext Ġwindow Ġsize Ġare Ġfound Ġto Ġcapture Ġthe Ġsynt actic Ġproperties Ġof Ġlanguage Ġwell ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠHowever , Ġif Ġour Ġgoal Ġis Ġto Ġseparate Ġsynt actic Ġcategories , Ġthis Ġembed ding Ġspace Ġis Ġnot Ġideal ĠâĢĵ ĠPOS Ġcategories Ġcorrespond Ġto Ġoverlapping Ġinter sp ersed Ġregions Ġin Ġthe Ġembed ding Ġspace , Ġevident Ġin ĠFigure ĠSEC REF 4 Ġ. </s> In Ġour Ġapproach , Ġwe Ġpropose Ġto Ġlearn Ġa Ġnew Ġlatent Ġembed ding Ġspace Ġas Ġa Ġprojection Ġof Ġpre - trained Ġembed d ings Ġ( dep icted Ġin ĠFigure ĠSEC REF 5 Ġ), Ġwhile Ġjointly Ġlearning Ġlatent Ġsynt actic Ġstructure ĠâĢĵ Ġfor Ġexample , ĠPOS Ġcategories Ġor Ġsynt actic Ġdependencies . ĠTo Ġthis Ġend , Ġwe Ġintroduce Ġa Ġnew Ġgener ative Ġmodel Ġ( shown Ġin ĠFigure ĠFIG REF 6 Ġ) Ġthat Ġfirst Ġgenerates Ġa Ġlatent Ġsynt actic Ġrepresentation Ġ( e . g . Ġa Ġdependency Ġparse ) Ġfrom Ġa Ġdiscrete Ġstructured Ġprior Ġ( which Ġwe Ġalso Ġcall Ġthe ĠâĢ ľ sy ntax Ġmodel âĢ Ŀ ), Ġthen , Ġconditioned Ġon Ġthis Ġrepresentation , Ġgenerates Ġa Ġsequence Ġof Ġlatent Ġembed ding Ġrandom Ġvariables Ġcorresponding Ġto Ġeach Ġword , Ġand Ġfinally Ġproduces Ġthe Ġobserved Ġ( pre - trained ) Ġword Ġembed d ings Ġby Ġprojecting Ġthese Ġlatent Ġvectors Ġthrough Ġa Ġparameter ized Ġnon - linear Ġfunction . ĠThe Ġlatent Ġembed d ings Ġcan Ġbe Ġjointly Ġlearned Ġwith Ġthe Ġstructured Ġsyntax Ġmodel Ġin Ġa Ġcompletely Ġun super vised Ġfashion . </s> By Ġchoosing Ġan Ġin vert ible Ġneural Ġnetwork Ġas Ġour Ġnon - linear Ġprojector , Ġand Ġthen Ġparameter izing Ġour Ġmodel Ġin Ġterms Ġof Ġthe Ġprojection 's Ġinverse , Ġwe Ġare Ġable Ġto Ġderive Ġtract able Ġexact Ġinference Ġand Ġmarginal Ġlikelihood Ġcomputation Ġprocedures Ġso Ġlong Ġas Ġinference Ġis Ġtract able Ġin Ġthe Ġunderlying Ġsyntax Ġmodel . ĠIn Ġsec : learn - with - inv Ġwe Ġshow Ġthat Ġthis Ġderiv ation Ġcorresponds Ġto Ġan Ġalternate Ġview Ġof Ġour Ġapproach Ġwhereby Ġwe Ġjointly Ġlearn Ġa Ġmapping Ġof Ġobserved Ġword Ġembed d ings Ġto Ġa Ġnew Ġembed ding Ġspace Ġthat Ġis Ġmore Ġsuitable Ġfor Ġthe Ġsyntax Ġmodel , Ġbut Ġinclude Ġan Ġadditional ĠJacob ian Ġregular ization Ġterm Ġto Ġprevent Ġinformation Ġloss . </s> Recent Ġwork Ġhas Ġsought Ġto Ġtake Ġadvantage Ġof Ġword Ġembed d ings Ġin Ġun super vised Ġgener ative Ġmodels Ġwith Ġalternate Ġapproaches ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠB IB REF 9 Ġbuild Ġan ĠH MM Ġwith ĠGa ussian Ġemissions Ġon Ġobserved Ġword Ġembed d ings , Ġbut Ġthey Ġdo Ġnot Ġattempt Ġto Ġlearn Ġnew Ġembed d ings . ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, Ġand ĠB IB REF 12 Ġextend ĠH MM Ġor Ġdependency Ġmodel Ġwith Ġval ence Ġ( DM V ) ĠB IB REF 2 Ġwith Ġmult in om ials Ġthat Ġuse Ġword Ġ( or Ġtag ) Ġembed d ings Ġin Ġtheir Ġparameter ization . ĠHowever , Ġthey Ġdo Ġnot Ġrepresent Ġthe Ġembed d ings Ġas Ġlatent Ġvariables . </s> In Ġexperiments , Ġwe Ġinstant iate Ġour Ġapproach Ġusing Ġboth Ġa ĠMark ov - struct ured Ġsyntax Ġmodel Ġand Ġa Ġtree - struct ured Ġsyntax Ġmodel ĠâĢĵ Ġspecifically , Ġthe ĠDMV . ĠWe Ġevaluate Ġon Ġtwo Ġtasks : Ġpart - of - speech Ġ( POS ) Ġinduction Ġand Ġun super vised Ġdependency Ġparsing Ġwithout Ġgold ĠPOS Ġtags . ĠExperimental Ġresults Ġon Ġthe ĠPenn ĠTree bank ĠB IB REF 13 Ġdemonstrate Ġthat Ġour Ġapproach Ġimproves Ġthe Ġbasic ĠH MM Ġand ĠDMV Ġby Ġa Ġlarge Ġmargin , Ġleading Ġto Ġthe Ġstate - of - the - art Ġresults Ġon ĠPOS Ġinduction , Ġand Ġstate - of - the - art Ġresults Ġon Ġun super vised Ġdependency Ġparsing Ġin Ġthe Ġdifficult Ġtraining Ġscenario Ġwhere Ġneither Ġgold ĠPOS Ġannotation Ġnor Ġpunct uation - based Ġconstraints Ġare Ġavailable . </s> Model </s> As Ġan Ġillust rative Ġexample , Ġwe Ġfirst Ġpresent Ġa Ġbaseline Ġmodel Ġfor ĠMark ov Ġsynt actic Ġstructure Ġ( POS Ġinduction ) Ġthat Ġtreats Ġa Ġsequence Ġof Ġpre - trained Ġword Ġembed d ings Ġas Ġobservations . ĠThen , Ġwe Ġpropose Ġour Ġnovel Ġapproach , Ġagain Ġusing ĠMark ov Ġstructure , Ġthat Ġintroduces Ġlatent Ġword Ġembed ding Ġvariables Ġand Ġa Ġneural Ġprojector . ĠLastly , Ġwe Ġextend Ġour Ġapproach Ġto Ġmore Ġgeneral Ġsynt actic Ġstructures . </s> Example : ĠGa ussian ĠH MM </s> We Ġstart Ġby Ġdescribing Ġthe ĠGa ussian Ġhidden ĠMark ov Ġmodel Ġintroduced Ġby ĠB IB REF 9 Ġ, Ġwhich Ġis Ġa Ġlocally Ġnormalized Ġmodel Ġwith Ġmult in omial Ġtransitions Ġand ĠGa ussian Ġemissions . ĠGiven Ġa Ġsentence Ġof Ġlength ĠIN LINE FORM 0 Ġ, Ġwe Ġdenote Ġthe Ġlatent ĠPOS Ġtags Ġas ĠIN LINE FORM 1 Ġ, Ġobserved Ġ( pre - trained ) Ġword Ġembed d ings Ġas ĠIN LINE FORM 2 Ġ, Ġtransition Ġparameters Ġas ĠIN LINE FORM 3 Ġ, Ġand ĠGa ussian Ġemission Ġparameters Ġas ĠIN LINE FORM 4 Ġ. ĠThe Ġjoint Ġdistribution Ġof Ġdata Ġand Ġlatent Ġvariables Ġfactors Ġas : </s> DIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġmult in omial Ġtransition Ġprobability Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġmult ivariate ĠGa ussian Ġemission Ġprobability . </s> While Ġthe Ġobserved Ġword Ġembed d ings Ġdo Ġinform Ġthis Ġmodel Ġwith Ġa Ġnotion Ġof Ġword Ġsimilarity ĠâĢĵ Ġlacking Ġin Ġthe Ġbasic Ġmult in omial ĠH MM ĠâĢĵ Ġthe ĠGa ussian Ġemissions Ġmay Ġnot Ġbe Ġsufficiently Ġflexible Ġto Ġseparate Ġsome Ġsynt actic Ġcategories Ġin Ġthe Ġcomplex Ġpre - trained Ġembed ding Ġspace ĠâĢĵ Ġfor Ġexample Ġthe Ġskip - gram Ġembed ding Ġspace Ġas Ġvisual ized Ġin ĠFigure ĠSEC REF 4 Ġwhere Ġdifferent ĠPOS Ġcategories Ġoverlap . ĠNext Ġwe Ġintroduce Ġa Ġnew Ġapproach Ġthat Ġadds Ġflexibility Ġto Ġthe Ġemission Ġdistribution Ġby Ġincorporating Ġnew Ġlatent Ġembed ding Ġvariables . </s> Mark ov ĠStructure Ġwith ĠNeural ĠProject or </s> To Ġflex ibly Ġmodel Ġobserved Ġembed d ings Ġand Ġyield Ġa Ġnew Ġrepresentation Ġspace Ġthat Ġis Ġmore Ġsuitable Ġfor Ġthe Ġsyntax Ġmodel , Ġwe Ġpropose Ġto Ġcascade Ġa Ġneural Ġnetwork Ġas Ġa Ġprojection Ġfunction , Ġdeter minist ically Ġtransforming Ġthe Ġsimple Ġspace Ġdefined Ġby Ġthe ĠGa ussian ĠH MM Ġto Ġthe Ġobserved Ġembed ding Ġspace . ĠWe Ġdenote Ġthe Ġlatent Ġembed ding Ġof Ġthe ĠIN LINE FORM 0 Ġword Ġin Ġa Ġsentence Ġas ĠIN LINE FORM 1 Ġ, Ġand Ġthe Ġneural Ġprojection Ġfunction Ġas ĠIN LINE FORM 2 Ġ, Ġparameter ized Ġby ĠIN LINE FORM 3 Ġ. ĠIn Ġthe Ġcase Ġof Ġsequential ĠMark ov Ġstructure , Ġour Ġnew Ġmodel Ġcorresponds Ġto Ġthe Ġfollowing Ġgener ative Ġprocess : </s> For Ġeach Ġtime Ġstep ĠIN LINE FORM 0 Ġ, </s> [ no item se p , Ġleft margin = * ] </s> Draw Ġthe Ġlatent Ġstate ĠIN LINE FORM 0 </s> Draw Ġthe Ġlatent Ġembed ding ĠIN LINE FORM 0 </s> D eter minist ically Ġproduce Ġembed ding </s> IN LINE FORM 0 </s> The Ġgraphical Ġmodel Ġis Ġdepicted Ġin ĠFigure ĠFIG REF 6 Ġ. ĠThe Ġdeter ministic Ġprojection Ġcan Ġalso Ġbe Ġviewed Ġas Ġsampling Ġeach Ġobservation Ġfrom Ġa Ġpoint Ġmass Ġat ĠIN LINE FORM 0 Ġ. ĠThe Ġjoint Ġdistribution Ġof Ġour Ġmodel Ġis : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġconditional ĠGa ussian Ġdistribution , Ġand ĠIN LINE FORM 1 Ġis Ġthe ĠDir ac Ġdelta Ġfunction Ġcentered Ġat ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> General ĠStructure Ġwith ĠNeural ĠProject or </s> Our Ġapproach Ġcan Ġbe Ġapplied Ġto Ġa Ġbroad Ġfamily Ġof Ġstructured Ġsyntax Ġmodels . ĠWe Ġdenote Ġlatent Ġembed ding Ġvariables Ġas ĠIN LINE FORM 0 Ġ, Ġdiscrete Ġlatent Ġvariables Ġin Ġthe Ġsyntax Ġmodel Ġas ĠIN LINE FORM 1 Ġ( ĠIN LINE FORM 2 Ġ), Ġwhere ĠIN LINE FORM 3 Ġare Ġconditioned Ġto Ġgenerate ĠIN LINE FORM 4 Ġ. ĠThe Ġjoint Ġprobability Ġof Ġour Ġmodel Ġfactors Ġas : </s> DIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġprobability Ġof Ġthe Ġsyntax Ġmodel , Ġand Ġcan Ġencode Ġany Ġsynt actic Ġstructure ĠâĢĵ Ġthough , Ġits Ġfactor ization Ġstructure Ġwill Ġdetermine Ġwhether Ġinference Ġis Ġtract able Ġin Ġour Ġfull Ġmodel . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 6 Ġ, Ġwe Ġfocus Ġon Ġtwo Ġsyntax Ġmodels Ġfor Ġsynt actic Ġanalysis Ġin Ġthis Ġpaper . ĠThe Ġfirst Ġis ĠMark ov - struct ured , Ġwhich Ġwe Ġuse Ġfor ĠPOS Ġinduction , Ġand Ġthe Ġsecond Ġis ĠDMV - struct ured , Ġwhich Ġwe Ġuse Ġto Ġlearn Ġdependency Ġpars es Ġwithout Ġsupervision . </s> The Ġmarginal Ġdata Ġlikelihood Ġof Ġour Ġmodel Ġis : ĠDIS PLAY FORM 0 </s> While Ġthe Ġdiscrete Ġvariables ĠIN LINE FORM 0 Ġcan Ġbe Ġmarginalized Ġout Ġwith Ġdynamic Ġprogram Ġin Ġmany Ġcases , Ġit Ġis Ġgenerally Ġint ract able Ġto Ġmarginal ize Ġout Ġthe Ġlatent Ġcontinuous Ġvariables , ĠIN LINE FORM 1 Ġ, Ġfor Ġan Ġarbitrary Ġprojection ĠIN LINE FORM 2 Ġin ĠE q . Ġ( ĠEQ REF 17 Ġ), Ġwhich Ġmeans Ġinference Ġand Ġlearning Ġmay Ġbe Ġdifficult . ĠIn Ġsec : opt , Ġwe Ġaddress Ġthis Ġissue Ġby Ġconst raining ĠIN LINE FORM 3 Ġto Ġbe Ġin vert ible , Ġand Ġshow Ġthat Ġthis Ġconstraint Ġenables Ġtract able Ġexact Ġinference Ġand Ġmarginal Ġlikelihood Ġcomputation . </s> Learning Ġ& ĠIn ference </s> In Ġthis Ġsection , Ġwe Ġintroduce Ġan Ġin vert ibility Ġcondition Ġfor Ġour Ġneural Ġprojector Ġto Ġtackle Ġthe Ġoptimization Ġchallenge . ĠSpecifically , Ġwe Ġconst rain Ġour Ġneural Ġprojector Ġwith Ġtwo Ġrequirements : Ġ( 1 ) ĠIN LINE FORM 0 Ġand Ġ( 2 ) ĠIN LINE FORM 1 Ġexists . ĠIn vert ible Ġtransformations Ġhave Ġbeen Ġexplored Ġbefore Ġin Ġindependent Ġcomponents Ġanalysis ĠB IB REF 14 Ġ, Ġga ussian ization ĠB IB REF 15 Ġ, Ġand Ġdeep Ġdensity Ġmodels ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġfor Ġun struct ured Ġdata . ĠHere , Ġwe Ġgeneral ize Ġthis Ġstyle Ġof Ġapproach Ġto Ġstructured Ġlearning , Ġand Ġaugment Ġit Ġwith Ġdiscrete Ġlatent Ġvariables Ġ( ĠIN LINE FORM 2 Ġ). ĠUnder Ġthe Ġin vert ibility Ġcondition , Ġwe Ġderive Ġa Ġlearning Ġalgorithm Ġand Ġgive Ġanother Ġview Ġof Ġour Ġapproach Ġrevealed Ġby Ġthe Ġobjective Ġfunction . ĠThen , Ġwe Ġpresent Ġthe Ġarchitecture Ġof Ġa Ġneural Ġprojector Ġwe Ġuse Ġin Ġexperiments : Ġa Ġvolume - pres erving Ġin vert ible Ġneural Ġnetwork Ġproposed Ġby ĠB IB REF 16 Ġfor Ġindependent Ġcomponents Ġestimation . </s> Learning Ġwith ĠIn vert ibility </s> For Ġease Ġof Ġexposition , Ġwe Ġexplain Ġthe Ġlearning Ġalgorithm Ġin Ġterms Ġof ĠMark ov Ġstructure Ġwithout Ġloss Ġof Ġgener ality . ĠAs Ġshown Ġin ĠE q . Ġ( ĠEQ REF 17 Ġ), Ġthe Ġoptimization Ġchallenge Ġin Ġour Ġapproach Ġcomes Ġfrom Ġthe Ġint ract ability Ġof Ġthe Ġmarginalized Ġemission Ġfactor ĠIN LINE FORM 0 Ġ. ĠIf Ġwe Ġcan Ġmarginal ize Ġout ĠIN LINE FORM 1 Ġand Ġcompute ĠIN LINE FORM 2 Ġ, Ġthen Ġthe Ġposterior Ġand Ġmarginal Ġlikelihood Ġof Ġour ĠMark ov - struct ured Ġmodel Ġcan Ġbe Ġcomputed Ġwith Ġthe Ġforward - back ward Ġalgorithm . ĠWe Ġcan Ġapply ĠE q . Ġ( ĠEQ REF 14 Ġ) Ġand Ġobtain Ġ: ĠIN LINE FORM 3 </s> By Ġusing Ġthe Ġchange Ġof Ġvariable Ġrule Ġto Ġthe Ġintegration , Ġwhich Ġallows Ġthe Ġintegration Ġvariable ĠIN LINE FORM 0 Ġto Ġbe Ġreplaced Ġby ĠIN LINE FORM 1 Ġ, Ġthe Ġmarginal Ġemission Ġfactor Ġcan Ġbe Ġcomputed Ġin Ġclosed - form Ġwhen Ġthe Ġin vert ibility Ġcondition Ġis Ġsatisfied : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġconditional ĠGa ussian Ġdistribution , ĠIN LINE FORM 1 Ġis Ġthe ĠJacob ian Ġmatrix Ġof Ġfunction ĠIN LINE FORM 2 Ġat ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġrepresents Ġthe Ġabsolute Ġvalue Ġof Ġits Ġdetermin ant . ĠThis ĠJacob ian Ġterm Ġis Ġnon zero Ġand Ġdifferent iable Ġif Ġand Ġonly Ġif ĠIN LINE FORM 5 Ġexists . </s> E q . Ġ( ĠEQ REF 19 Ġ) Ġshows Ġthat Ġwe Ġcan Ġdirectly Ġcalculate Ġthe Ġmarginal Ġemission Ġdistribution ĠIN LINE FORM 0 Ġ. ĠDen ote Ġthe Ġmarginal Ġdata Ġlikelihood Ġof ĠGa ussian ĠH MM Ġas ĠIN LINE FORM 1 Ġ, Ġthen Ġthe Ġlog Ġmarginal Ġdata Ġlikelihood Ġof Ġour Ġmodel Ġcan Ġbe Ġdirectly Ġwritten Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġnew Ġsequence Ġof Ġembed d ings Ġafter Ġapplying ĠIN LINE FORM 1 Ġto Ġeach ĠIN LINE FORM 2 Ġ. ĠE q . Ġ( ĠEQ REF 20 Ġ) Ġshows Ġthat Ġthe Ġtraining Ġobjective Ġof Ġour Ġmodel Ġis Ġsimply Ġthe ĠGa ussian ĠH MM Ġlog Ġlikelihood Ġwith Ġan Ġadditional ĠJacob ian Ġregular ization Ġterm . ĠFrom Ġthis Ġview , Ġour Ġapproach Ġcan Ġbe Ġseen Ġas Ġequivalent Ġto Ġreverse ly Ġprojecting Ġthe Ġdata Ġthrough ĠIN LINE FORM 3 Ġto Ġanother Ġmanifold ĠIN LINE FORM 4 Ġthat Ġis Ġdirectly Ġmodeled Ġby Ġthe ĠGa ussian ĠH MM , Ġwith Ġa Ġregular ization Ġterm . ĠInt uitive ly , Ġwe Ġoptimize Ġthe Ġreverse Ġprojection ĠIN LINE FORM 5 Ġto Ġmodify Ġthe ĠIN LINE FORM 6 Ġspace , Ġmaking Ġit Ġmore Ġappropriate Ġfor Ġthe Ġsyntax Ġmodel . ĠThe ĠJacob ian Ġregular ization Ġterm Ġaccounts Ġfor Ġthe Ġvolume Ġexpansion Ġor Ġcontraction Ġbehavior Ġof Ġthe Ġprojection . ĠMaxim izing Ġit Ġcan Ġbe Ġthought Ġof Ġas Ġpreventing Ġinformation Ġloss . ĠIn Ġthe Ġextreme Ġcase , Ġthe ĠJacob ian Ġdetermin ant Ġis Ġequal Ġto Ġzero , Ġwhich Ġmeans Ġthe Ġprojection Ġis Ġnon - in vert ible Ġand Ġthus Ġinformation Ġis Ġbeing Ġlost Ġthrough Ġthe Ġprojection . ĠSuch ĠâĢ ľ information Ġpreserving âĢ Ŀ Ġregular ization Ġis Ġcrucial Ġduring Ġoptimization , Ġotherwise Ġthe Ġtrivial Ġsolution Ġof Ġalways Ġprojecting Ġdata Ġto Ġthe Ġsame Ġsingle Ġpoint Ġto Ġmaximize Ġlikelihood Ġis Ġviable . </s> More Ġgenerally , Ġfor Ġan Ġarbitrary Ġsyntax Ġmodel Ġthe Ġdata Ġlikelihood Ġof Ġour Ġapproach Ġis : ĠDIS PLAY FORM 0 </s> If Ġthe Ġsyntax Ġmodel Ġitself Ġallows Ġfor Ġtract able Ġinference Ġand Ġmarginal Ġlikelihood Ġcomputation , Ġthe Ġsame Ġdynamic Ġprogram Ġcan Ġbe Ġused Ġto Ġmarginal ize Ġout ĠIN LINE FORM 0 Ġ. ĠTherefore , Ġour Ġjoint Ġmodel Ġinher its Ġthe Ġtract ability Ġof Ġthe Ġunderlying Ġsyntax Ġmodel . </s> In vert ible ĠVolume - Pres erving ĠNeural ĠNet </s> For Ġthe Ġprojection Ġwe Ġcan Ġuse Ġan Ġarbitrary Ġin vert ible Ġfunction , Ġand Ġgiven Ġthe Ġrepresent ational Ġpower Ġof Ġneural Ġnetworks Ġthey Ġseem Ġa Ġnatural Ġchoice . ĠHowever , Ġcalculating Ġthe Ġinverse Ġand ĠJacob ian Ġof Ġan Ġarbitrary Ġneural Ġnetwork Ġcan Ġbe Ġdifficult , Ġas Ġit Ġrequires Ġthat Ġall Ġcomponent Ġfunctions Ġbe Ġin vert ible Ġand Ġalso Ġrequires Ġstorage Ġof Ġlarge ĠJacob ian Ġmat rices , Ġwhich Ġis Ġmemory Ġintensive . ĠTo Ġaddress Ġthis Ġissue , Ġseveral Ġrecent Ġpapers Ġpropose Ġspecially Ġdesigned Ġin vert ible Ġnetworks Ġthat Ġare Ġeasily Ġtrain able Ġyet Ġstill Ġpowerful ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 19 Ġ. ĠInspired Ġby Ġthese Ġworks , Ġwe Ġuse Ġthe Ġin vert ible Ġtransformation Ġproposed Ġby ĠB IB REF 16 Ġ, Ġwhich Ġconsists Ġof Ġa Ġseries Ġof ĠâĢ ľ c ou pling Ġlayers âĢ Ŀ . ĠThis Ġarchitecture Ġis Ġspecially Ġdesigned Ġto Ġguarantee Ġa Ġunit ĠJacob ian Ġdetermin ant Ġ( and Ġthus Ġthe Ġin vert ibility Ġproperty ). </s> From ĠE q . Ġ( ĠEQ REF 22 Ġ) Ġwe Ġknow Ġthat Ġonly ĠIN LINE FORM 0 Ġis Ġrequired Ġfor Ġaccompl ishing Ġlearning Ġand Ġinference ; Ġwe Ġnever Ġneed Ġto Ġexplicitly Ġconstruct ĠIN LINE FORM 1 Ġ. ĠThus , Ġwe Ġdirectly Ġdefine Ġthe Ġarchitecture Ġof ĠIN LINE FORM 2 Ġ. ĠAs Ġshown Ġin ĠFigure ĠFIG REF 24 Ġ, Ġthe Ġnon linear Ġtransformation Ġfrom Ġthe Ġobserved Ġembed ding ĠIN LINE FORM 3 Ġto ĠIN LINE FORM 4 Ġrepresents Ġthe Ġfirst Ġcoupling Ġlayer . ĠThe Ġinput Ġin Ġthis Ġlayer Ġis Ġpartition ed Ġinto Ġleft Ġand Ġright Ġhalves Ġof Ġdimensions , ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ, Ġrespectively . ĠA Ġsingle Ġcoupling Ġlayer Ġis Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġcoupling Ġfunction Ġand Ġcan Ġbe Ġany Ġnon linear Ġform . ĠThis Ġtransformation Ġsatisfies ĠIN LINE FORM 1 Ġ, Ġand ĠB IB REF 16 Ġshow Ġthat Ġits ĠJacob ian Ġmatrix Ġis Ġtriangular Ġwith Ġall Ġones Ġon Ġthe Ġmain Ġdiagonal . ĠThus Ġthe ĠJacob ian Ġdetermin ant Ġis Ġalways Ġequal Ġto Ġone Ġ( i . e . Ġvolume - pres erving ) Ġand Ġthe Ġin vert ibility Ġcondition Ġis Ġnaturally Ġsatisfied . </s> To Ġbe Ġsufficiently Ġexpressive , Ġwe Ġcompose Ġmultiple Ġcoupling Ġlayers Ġas Ġsuggested Ġin ĠB IB REF 16 Ġ. ĠSpecifically , Ġwe Ġexchange Ġthe Ġrole Ġof Ġleft Ġand Ġright Ġhalf Ġvectors Ġat Ġeach Ġlayer Ġas Ġshown Ġin ĠFigure ĠFIG REF 24 Ġ. ĠFor Ġinstance , Ġfrom ĠIN LINE FORM 0 Ġto ĠIN LINE FORM 1 Ġthe Ġleft Ġsubset ĠIN LINE FORM 2 Ġis Ġunchanged , Ġwhile Ġfrom ĠIN LINE FORM 3 Ġto ĠIN LINE FORM 4 Ġthe Ġright Ġsubset ĠIN LINE FORM 5 Ġremains Ġthe Ġsame . ĠAlso Ġnote Ġthat Ġcomposing Ġmultiple Ġcoupling Ġlayers Ġdoes Ġnot Ġchange Ġthe Ġvolume - pres erving Ġand Ġin vert ibility Ġproperties . ĠSuch Ġa Ġsequence Ġof Ġin vert ible Ġtransformations Ġfrom Ġthe Ġdata Ġspace ĠIN LINE FORM 6 Ġto ĠIN LINE FORM 7 Ġis Ġalso Ġcalled Ġnormal izing Ġflow ĠB IB REF 20 Ġ. </s> Exper iments </s> In Ġthis Ġsection , Ġwe Ġfirst Ġdescribe Ġour Ġdatasets Ġand Ġexperimental Ġsetup . ĠWe Ġthen Ġinstant iate Ġour Ġapproach Ġwith ĠMark ov Ġand ĠDMV - struct ured Ġsyntax Ġmodels , Ġand Ġreport Ġresults Ġon ĠPOS Ġtagging Ġand Ġdependency Ġgrammar Ġinduction Ġrespectively . ĠLastly , Ġwe Ġanalyze Ġthe Ġlearned Ġlatent Ġembed d ings . </s> Data </s> For Ġboth ĠPOS Ġtagging Ġand Ġdependency Ġparsing , Ġwe Ġrun Ġexperiments Ġon Ġthe ĠWall ĠStreet ĠJournal Ġ( WS J ) Ġportion Ġof Ġthe ĠPenn ĠTree bank . ĠTo Ġcreate Ġthe Ġobserved Ġdata Ġembed d ings , Ġwe Ġtrain Ġskip - gram Ġword Ġembed d ings ĠB IB REF 7 Ġthat Ġare Ġfound Ġto Ġcapture Ġsynt actic Ġproperties Ġwell Ġwhen Ġtrained Ġwith Ġsmall Ġcontext Ġwindow ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠFollowing ĠB IB REF 9 Ġ, Ġthe Ġdimension ality ĠIN LINE FORM 0 Ġis Ġset Ġto Ġ100 , Ġand Ġthe Ġtraining Ġcontext Ġwindow Ġsize Ġis Ġset Ġto Ġ1 Ġto Ġencode Ġmore Ġsynt actic Ġinformation . ĠThe Ġskip - gram Ġembed d ings Ġare Ġtrained Ġon Ġthe Ġone Ġbillion Ġword Ġlanguage Ġmodeling Ġbenchmark Ġdataset ĠB IB REF 21 Ġin Ġaddition Ġto Ġthe ĠWS J Ġcorpus . </s> General ĠExperimental ĠSetup </s> For Ġthe Ġneural Ġprojector , Ġwe Ġemploy Ġrect ified Ġnetworks Ġas Ġcoupling Ġfunction ĠIN LINE FORM 0 Ġfollowing ĠB IB REF 16 Ġ. ĠWe Ġuse Ġa Ġrect ified Ġnetwork Ġwith Ġan Ġinput Ġlayer , Ġone Ġhidden Ġlayer , Ġand Ġlinear Ġoutput Ġunits , Ġthe Ġnumber Ġof Ġhidden Ġunits Ġis Ġset Ġto Ġthe Ġsame Ġas Ġthe Ġnumber Ġof Ġinput Ġunits . ĠThe Ġnumber Ġof Ġcoupling Ġlayers Ġare Ġvaried Ġas Ġ4 , Ġ8 , Ġ16 Ġfor Ġboth Ġtasks . ĠWe Ġoptimize Ġmarginal Ġdata Ġlikelihood Ġdirectly Ġusing ĠAdam ĠB IB REF 22 Ġ. ĠFor Ġboth Ġtasks Ġin Ġthe Ġfully Ġun super vised Ġsetting , Ġwe Ġdo Ġnot Ġtune Ġthe Ġhyper - param eters Ġusing Ġsupervised Ġdata . </s> Un super vised ĠPOS Ġtagging </s> For Ġun super vised ĠPOS Ġtagging , Ġwe Ġuse Ġa ĠMark ov - struct ured Ġsyntax Ġmodel Ġin Ġour Ġapproach , Ġwhich Ġis Ġa Ġpopular Ġstructure Ġfor Ġun super vised Ġtagging Ġtasks ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. </s> Following Ġexisting Ġliterature , Ġwe Ġtrain Ġand Ġtest Ġon Ġthe Ġentire ĠWS J Ġcorpus Ġ( 49 208 Ġsentences , Ġ1 M Ġtokens ). ĠWe Ġuse Ġ45 Ġtag Ġclusters , Ġthe Ġnumber Ġof ĠPOS Ġtags Ġthat Ġappear Ġin ĠWS J Ġcorpus . ĠWe Ġtrain Ġthe Ġdiscrete ĠH MM Ġand Ġthe ĠGa ussian ĠH MM ĠB IB REF 9 Ġas Ġbas elines . ĠFor Ġthe ĠGa ussian ĠH MM , Ġmean Ġvectors Ġof ĠGa ussian Ġemissions Ġare Ġinitialized Ġwith Ġthe Ġempirical Ġmean Ġof Ġall Ġword Ġvectors Ġwith Ġan Ġadditive Ġnoise . ĠWe Ġassume Ġdiagonal Ġcovari ance Ġmatrix Ġfor ĠIN LINE FORM 0 Ġand Ġinitialize Ġit Ġwith Ġthe Ġempirical Ġvariance Ġof Ġthe Ġword Ġvectors . ĠFollowing ĠB IB REF 9 Ġ, Ġthe Ġcovari ance Ġmatrix Ġis Ġfixed Ġduring Ġtraining . ĠThe Ġmult in omial Ġprobabilities Ġare Ġinitialized Ġas ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġ. ĠFor Ġour Ġapproach , Ġwe Ġinitialize Ġthe Ġsyntax Ġmodel Ġand ĠGa ussian Ġparameters Ġwith Ġthe Ġpre - trained ĠGa ussian ĠH MM . ĠThe Ġweights Ġof Ġlayers Ġin Ġthe Ġrect ified Ġnetwork Ġare Ġinitialized Ġfrom Ġa Ġuniform Ġdistribution Ġwith Ġmean Ġzero Ġand Ġa Ġstandard Ġdeviation Ġof ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġinput Ġdimension . ĠWe Ġevaluate Ġthe Ġperformance Ġof ĠPOS Ġtagging Ġwith Ġboth ĠMany - to - One Ġ( M - 1 ) Ġaccuracy ĠB IB REF 23 Ġand ĠV - Measure Ġ( VM ) ĠB IB REF 24 Ġ. ĠGiven Ġa Ġmodel Ġwe Ġfound Ġthat Ġthe Ġtagging Ġperformance Ġis Ġwell - cor related Ġwith Ġthe Ġtraining Ġdata Ġlikelihood , Ġthus Ġwe Ġuse Ġtraining Ġdata Ġlikelihood Ġas Ġa Ġun super vised Ġcriterion Ġto Ġselect Ġthe Ġtrained Ġmodel Ġover Ġ10 Ġrandom Ġrest arts Ġafter Ġtraining Ġ50 Ġepoch s . ĠWe Ġrepeat Ġthis Ġprocess Ġ5 Ġtimes Ġand Ġreport Ġthe Ġmean Ġand Ġstandard Ġdeviation Ġof Ġperformance . </s> We Ġcompare Ġour Ġapproach Ġwith Ġbasic ĠH MM , ĠGa ussian ĠH MM , Ġand Ġseveral Ġstate - of - the - art Ġsystems , Ġincluding Ġsophisticated ĠH MM Ġvariants Ġand Ġclust ering Ġtechniques Ġwith Ġhand - engine ered Ġfeatures . ĠThe Ġresults Ġare Ġpresented Ġin ĠTable ĠT AB REF 32 Ġ. ĠThrough Ġthe Ġintroduced Ġlatent Ġembed d ings Ġand Ġadditional Ġneural Ġprojection , Ġour Ġapproach Ġimproves Ġover Ġthe ĠGa ussian ĠH MM Ġby Ġ5 . 4 Ġpoints Ġin ĠM - 1 Ġand Ġ5 . 6 Ġpoints Ġin ĠVM . ĠNeural ĠH MM Ġ( NH MM ) ĠB IB REF 10 Ġis Ġa Ġbaseline Ġthat Ġalso Ġlearns Ġword Ġrepresentation Ġjointly . ĠBoth Ġtheir Ġbasic Ġmodel Ġand Ġextended ĠConv Ġversion Ġdoes Ġnot Ġoutper form Ġthe ĠGa ussian ĠH MM . ĠTheir Ġbest Ġmodel Ġincorporates Ġanother ĠL ST M Ġto Ġmodel Ġlong Ġdistance Ġdependency Ġand Ġbreaks Ġthe ĠMark ov Ġassumption , Ġyet Ġour Ġapproach Ġstill Ġachieves Ġsubstantial Ġimprovement Ġover Ġit Ġwithout Ġconsidering Ġmore Ġcontext Ġinformation . ĠMoreover , Ġour Ġmethod Ġoutper forms Ġthe Ġbest Ġpublished Ġresult Ġthat Ġbenefits Ġfrom Ġhand - engine ered Ġfeatures ĠB IB REF 27 Ġby Ġ2 . 0 Ġpoints Ġon ĠVM . </s> We Ġfound Ġthat Ġmost Ġtagging Ġerrors Ġhappen Ġin Ġnoun Ġsub c ategories . ĠTherefore , Ġwe Ġdo Ġthe Ġone - to - one Ġmapping Ġbetween Ġgold ĠPOS Ġtags Ġand Ġinduced Ġclusters Ġand Ġplot Ġthe Ġnormalized Ġconfusion Ġmatrix Ġof Ġnoun Ġsub c ategories Ġin ĠFigure ĠFIG REF 35 Ġ. ĠThe ĠGa ussian ĠH MM Ġfails Ġto Ġidentify ĠâĢ ľ NN âĢ Ŀ Ġand ĠâĢ ľ NN S âĢ Ŀ Ġcorrectly Ġfor Ġmost Ġcases , Ġand Ġit Ġoften Ġrecognizes ĠâĢ ľ NN PS âĢ Ŀ Ġas ĠâĢ ľ NN P âĢ Ŀ . ĠIn Ġcontrast , Ġour Ġapproach Ġcorrect s Ġthese Ġerrors Ġwell . </s> Un super vised ĠDepend ency ĠPars ing Ġwithout Ġgold ĠPOS Ġtags </s> For Ġthe Ġtask Ġof Ġun super vised Ġdependency Ġparse Ġinduction , Ġwe Ġemploy Ġthe ĠDepend ency ĠModel Ġwith ĠVal ence Ġ( DM V ) ĠB IB REF 2 Ġas Ġthe Ġsyntax Ġmodel Ġin Ġour Ġapproach . ĠDMV Ġis Ġa Ġgener ative Ġmodel Ġthat Ġdefines Ġa Ġprobability Ġdistribution Ġover Ġdependency Ġparse Ġtrees Ġand Ġsynt actic Ġcategories , Ġgenerating Ġtokens Ġand Ġdependencies Ġin Ġa Ġhead - out ward Ġfashion . ĠWhile , Ġtraditionally , ĠDMV Ġis Ġtrained Ġusing Ġgold ĠPOS Ġtags Ġas Ġobserved Ġsynt actic Ġcategories , Ġin Ġour Ġapproach , Ġwe Ġtreat Ġeach Ġtag Ġas Ġa Ġlatent Ġvariable , Ġas Ġdescribed Ġin Ġsec : general - ne ural . </s> Most Ġexisting Ġapproaches Ġto Ġthis Ġtask Ġare Ġnot Ġfully Ġun super vised Ġsince Ġthey Ġrely Ġon Ġgold ĠPOS Ġtags Ġfollowing Ġthe Ġoriginal Ġexperimental Ġsetup Ġfor ĠDMV . ĠThis Ġis Ġpartially Ġbecause Ġautomatically Ġparsing Ġfrom Ġwords Ġis Ġdifficult Ġeven Ġwhen Ġusing Ġun super vised Ġsynt actic Ġcategories ĠB IB REF 29 Ġ. ĠHowever , Ġinducing Ġdependencies Ġfrom Ġwords Ġalone Ġrepresents Ġa Ġmore Ġrealistic Ġexperimental Ġcondition Ġsince Ġgold ĠPOS Ġtags Ġare Ġoften Ġunavailable Ġin Ġpractice . ĠPrevious Ġwork Ġthat Ġhas Ġtrained Ġfrom Ġwords Ġalone Ġoften Ġrequires Ġadditional Ġlinguistic Ġconstraints Ġ( like Ġsentence Ġinternal Ġboundaries ) ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, Ġacoustic Ġcues ĠB IB REF 33 Ġ, Ġadditional Ġtraining Ġdata ĠB IB REF 4 Ġ, Ġor Ġannot ated Ġdata Ġfrom Ġrelated Ġlanguages ĠB IB REF 34 Ġ. ĠOur Ġapproach Ġis Ġnaturally Ġdesigned Ġto Ġtrain Ġon Ġword Ġembed d ings Ġdirectly , Ġthus Ġwe Ġattempt Ġto Ġinduce Ġdependencies Ġwithout Ġusing Ġgold ĠPOS Ġtags Ġor Ġother Ġextra Ġlinguistic Ġinformation . </s> Like Ġprevious Ġwork Ġwe Ġuse Ġsections Ġ02 - 21 Ġof ĠWS J Ġcorpus Ġas Ġtraining Ġdata Ġand Ġevaluate Ġon Ġsection Ġ23 , Ġwe Ġremove Ġpunct uations Ġand Ġtrain Ġthe Ġmodels Ġon Ġsentences Ġof Ġlength ĠIN LINE FORM 0 Ġ, ĠâĢ ľ head - per col ation âĢ Ŀ Ġrules ĠB IB REF 39 Ġare Ġapplied Ġto Ġobtain Ġgold Ġdependencies Ġfor Ġevaluation . ĠWe Ġtrain Ġbasic ĠDMV , Ġextended ĠDMV Ġ( E - DM V ) ĠB IB REF 35 Ġand ĠGa ussian ĠDMV Ġ( which Ġtreats ĠPOS Ġtag Ġas Ġunknown Ġlatent Ġvariables Ġand Ġgenerates Ġobserved Ġword Ġembed d ings Ġdirectly Ġconditioned Ġon Ġthem Ġfollowing ĠGa ussian Ġdistribution ) Ġas Ġbas elines . ĠBasic ĠDMV Ġand ĠE - DM V Ġare Ġtrained Ġwith ĠV iter bi ĠEM ĠB IB REF 40 Ġon Ġun super vised ĠPOS Ġtags Ġinduced Ġfrom Ġour ĠMark ov - struct ured Ġmodel Ġdescribed Ġin Ġsec : pos . ĠMult in omial Ġparameters Ġof Ġthe Ġsyntax Ġmodel Ġin Ġboth ĠGa ussian ĠDMV Ġand Ġour Ġmodel Ġare Ġinitialized Ġwith Ġthe Ġpre - trained ĠDMV Ġbaseline . ĠOther Ġparameters Ġare Ġinitialized Ġin Ġthe Ġsame Ġway Ġas Ġin Ġthe ĠPOS Ġtagging Ġexperiment . ĠThe Ġdirected Ġdependency Ġaccuracy Ġ( D DA ) Ġis Ġused Ġfor Ġevaluation Ġand Ġwe Ġreport Ġaccuracy Ġon Ġsentences Ġof Ġlength ĠIN LINE FORM 1 Ġand Ġall Ġlengths . ĠWe Ġtrain Ġthe Ġparser Ġuntil Ġtraining Ġdata Ġlikelihood Ġconver ges , Ġand Ġreport Ġthe Ġmean Ġand Ġstandard Ġdeviation Ġover Ġ20 Ġrandom Ġrest arts . </s> Our Ġmodel Ġdirectly Ġobserves Ġword Ġembed d ings Ġand Ġdoes Ġnot Ġrequire Ġgold ĠPOS Ġtags Ġduring Ġtraining . ĠThus , Ġresults Ġfrom Ġrelated Ġwork Ġtrained Ġon Ġgold Ġtags Ġare Ġnot Ġdirectly Ġcomparable . ĠHowever , Ġto Ġmeasure Ġhow Ġthese Ġsystems Ġmight Ġperform Ġwithout Ġgold Ġtags , Ġwe Ġrun Ġthree Ġrecent Ġstate - of - the - art Ġsystems Ġin Ġour Ġexperimental Ġsetting : ĠUR - A ĠE - DM V ĠB IB REF 36 Ġ, ĠNeural ĠE - DM V ĠB IB REF 11 Ġ, Ġand ĠCR F ĠAut oen c oder Ġ( CR FA E ) ĠB IB REF 37 Ġ. ĠWe Ġuse Ġun super vised ĠPOS Ġtags Ġ( induced Ġfrom Ġour ĠMark ov - struct ured Ġmodel ) Ġin Ġplace Ġof Ġgold Ġtags . ĠWe Ġalso Ġtrain Ġbasic ĠDMV Ġon Ġgold Ġtags Ġand Ġinclude Ġseveral Ġstate - of - the - art Ġresults Ġon Ġgold Ġtags Ġas Ġreference Ġpoints . </s> As Ġshown Ġin ĠTable ĠT AB REF 39 Ġ, Ġour Ġapproach Ġis Ġable Ġto Ġimprove Ġover Ġthe ĠGa ussian ĠDMV Ġby Ġ4 . 8 Ġpoints Ġon Ġlength ĠIN LINE FORM 0 Ġand Ġ4 . 8 Ġpoints Ġon Ġall Ġlengths , Ġwhich Ġsuggests Ġthe Ġadditional Ġlatent Ġembed ding Ġlayer Ġand Ġneural Ġprojector Ġare Ġhelpful . ĠThe Ġproposed Ġapproach Ġyields , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġstate - of - the - art Ġperformance Ġwithout Ġgold ĠPOS Ġannotation Ġand Ġwithout Ġsentence - internal Ġboundary Ġinformation . ĠDMV , ĠUR - A ĠE - DM V , ĠNeural ĠE - DM V , Ġand ĠCR FA E Ġsuffer Ġa Ġlarge Ġdecrease Ġin Ġperformance Ġwhen Ġtrained Ġon Ġun super vised Ġtags ĠâĢĵ Ġan Ġeffect Ġalso Ġseen Ġin Ġprevious Ġwork ĠB IB REF 29 Ġ, ĠB IB REF 34 Ġ. ĠSince Ġour Ġapproach Ġinduces Ġlatent ĠPOS Ġtags Ġjointly Ġwith Ġdependency Ġtrees , Ġit Ġmay Ġbe Ġable Ġto Ġlearn ĠPOS Ġclusters Ġthat Ġare Ġmore Ġam enable Ġto Ġgrammar Ġinduction Ġthan Ġthe Ġun super vised Ġtags . ĠWe Ġobserve Ġthat ĠCR FA E Ġunder per forms Ġits Ġgold - tag Ġcounterpart Ġsubstantially . ĠThis Ġmay Ġlargely Ġbe Ġa Ġresult Ġof Ġthe Ġmodel 's Ġreliance Ġon Ġprior Ġlinguistic Ġrules Ġthat Ġbecome Ġunavailable Ġwhen Ġgold ĠPOS Ġtag Ġtypes Ġare Ġunknown . ĠMany Ġextensions Ġto ĠDMV Ġcan Ġbe Ġconsidered Ġorth og onal Ġto Ġour Ġapproach ĠâĢĵ Ġthey Ġessentially Ġfocus Ġon Ġimproving Ġthe Ġsyntax Ġmodel . ĠIt Ġis Ġpossible Ġthat Ġincorporating Ġthese Ġmore Ġsophisticated Ġsyntax Ġmodels Ġinto Ġour Ġapproach Ġmay Ġlead Ġto Ġfurther Ġimprovements . </s> S ensitivity ĠAnalysis </s> In Ġthe Ġabove Ġexperiments Ġwe Ġinitialize Ġthe Ġstructured Ġsyntax Ġcomponents Ġwith Ġthe Ġpre - trained ĠGa ussian Ġor Ġdiscrete Ġbaseline , Ġwhich Ġis Ġshown Ġas Ġa Ġuseful Ġtechnique Ġto Ġhelp Ġtrain Ġour Ġdeep Ġmodels . ĠWe Ġfurther Ġstudy Ġthe Ġresults Ġwith Ġfully Ġrandom Ġinitialization . ĠIn Ġthe ĠPOS Ġtagging Ġexperiment , Ġwe Ġreport Ġthe Ġresults Ġin ĠTable ĠT AB REF 48 Ġ. ĠWhile Ġthe Ġperformance Ġwith Ġ4 Ġlayers Ġis Ġcomparable Ġto Ġthe Ġpre - trained ĠGa ussian Ġinitialization , Ġdeeper Ġprojections Ġ( 8 Ġor Ġ16 Ġlayers ) Ġresult Ġin Ġa Ġdramatic Ġdrop Ġin Ġperformance . ĠThis Ġsuggests Ġthat Ġthe Ġstructured Ġsyntax Ġmodel Ġwith Ġvery Ġdeep Ġprojections Ġis Ġdifficult Ġto Ġtrain Ġfrom Ġscratch , Ġand Ġa Ġsimpler Ġprojection Ġmight Ġbe Ġa Ġgood Ġcompromise Ġin Ġthe Ġrandom Ġinitialization Ġsetting . </s> Different Ġfrom Ġthe ĠMark ov Ġprior Ġin ĠPOS Ġtagging Ġexperiments , Ġour Ġparsing Ġmodel Ġseems Ġto Ġbe Ġquite Ġsensitive Ġto Ġthe Ġinitialization . ĠFor Ġexample , Ġdirected Ġaccuracy Ġof Ġour Ġapproach Ġon Ġsentences Ġof Ġlength ĠIN LINE FORM 0 Ġis Ġbelow Ġ40 . 0 Ġwith Ġrandom Ġinitialization . ĠThis Ġis Ġconsistent Ġwith Ġprevious Ġwork Ġthat Ġhas Ġnoted Ġthe Ġimportance Ġof Ġcareful Ġinitialization Ġfor ĠDMV - based Ġmodels Ġsuch Ġas Ġthe Ġcommonly Ġused Ġharmonic Ġinitial izer ĠB IB REF 2 Ġ. ĠHowever , Ġit Ġis Ġnot Ġstraightforward Ġto Ġapply Ġthe Ġharmonic Ġinitial izer Ġfor ĠDMV Ġdirectly Ġin Ġour Ġmodel Ġwithout Ġusing Ġsome Ġkind Ġof Ġpre - training Ġsince Ġwe Ġdo Ġnot Ġobserve Ġgold ĠPOS . </s> We Ġinvestigate Ġthe Ġeffect Ġof Ġthe Ġchoice Ġof Ġpre - trained Ġembed ding Ġon Ġperformance Ġwhile Ġusing Ġour Ġapproach . ĠTo Ġthis Ġend , Ġwe Ġadditionally Ġinclude Ġresults Ġusing Ġfast Text Ġembed d ings ĠB IB REF 41 ĠâĢĵ Ġwhich , Ġin Ġcontrast Ġwith Ġskip - gram Ġembed d ings , Ġinclude Ġcharacter - level Ġinformation . ĠWe Ġset Ġthe Ġcontext Ġwindows Ġsize Ġto Ġ1 Ġand Ġthe Ġdimension Ġsize Ġto Ġ100 Ġas Ġin Ġthe Ġskip - gram Ġtraining , Ġwhile Ġkeeping Ġother Ġparameters Ġset Ġto Ġtheir Ġdefaults . ĠThese Ġresults Ġare Ġsummarized Ġin ĠTable ĠT AB REF 50 Ġand ĠTable ĠT AB REF 51 Ġ. ĠWhile Ġfast Text Ġembed d ings Ġlead Ġto Ġreduced Ġperformance Ġwith Ġour Ġmodel , Ġour Ġapproach Ġstill Ġyields Ġan Ġimprovement Ġover Ġthe ĠGa ussian Ġbaseline Ġwith Ġthe Ġnew Ġobserved Ġembed d ings Ġspace . </s> Qual itative ĠAnalysis Ġof ĠEmb edd ings </s> We Ġperform Ġqualitative Ġanalysis Ġto Ġunderstand Ġhow Ġthe Ġlatent Ġembed d ings Ġhelp Ġinduce Ġsynt actic Ġstructures . ĠFirst Ġwe Ġfilter Ġout Ġlow - frequency Ġwords Ġand Ġpunct uations Ġin ĠWS J , Ġand Ġvisualize Ġthe Ġrest Ġwords Ġ( 10 k ) Ġwith Ġt - S NE ĠB IB REF 42 Ġunder Ġdifferent Ġembed d ings . ĠWe Ġassign Ġeach Ġword Ġwith Ġits Ġmost Ġlikely Ġgold ĠPOS Ġtags Ġin ĠWS J Ġand Ġcolor Ġthem Ġaccording Ġto Ġthe Ġgold ĠPOS Ġtags . </s> For Ġour ĠMark ov - struct ured Ġmodel , Ġwe Ġhave Ġdisplayed Ġthe Ġembed ding Ġspace Ġin ĠFigure ĠSEC REF 5 Ġ, Ġwhere Ġthe Ġgold ĠPOS Ġclusters Ġare Ġwell - formed . ĠFurther , Ġwe Ġpresent Ġfive Ġexample Ġtarget Ġwords Ġand Ġtheir Ġfive Ġnearest Ġneighbors Ġin Ġterms Ġof Ġcos ine Ġsimilarity . ĠAs Ġshown Ġin ĠTable ĠT AB REF 53 Ġ, Ġthe Ġskip - gram Ġembed ding Ġcaptures Ġboth Ġsemantic Ġand Ġsynt actic Ġaspects Ġto Ġsome Ġdegree , Ġyet Ġour Ġembed d ings Ġare Ġable Ġto Ġfocus Ġespecially Ġon Ġthe Ġsynt actic Ġaspects Ġof Ġwords , Ġin Ġan Ġun super vised Ġfashion Ġwithout Ġusing Ġany Ġextra Ġmorph ological Ġinformation . </s> In ĠFigure ĠFIG REF 54 Ġwe Ġdepict Ġthe Ġlearned Ġlatent Ġembed d ings Ġwith Ġthe ĠDMV - struct ured Ġsyntax Ġmodel . ĠUnlike Ġthe ĠMark ov Ġstructure , Ġthe ĠDMV Ġstructure Ġmaps Ġa Ġlarge Ġsubset Ġof Ġsingular Ġand Ġplural Ġnoun s Ġto Ġthe Ġsame Ġoverlapping Ġregion . ĠHowever , Ġtwo Ġclusters Ġof Ġsingular Ġand Ġplural Ġnoun s Ġare Ġactually Ġseparated . ĠWe Ġinspect Ġthe Ġtwo Ġclusters Ġand Ġthe Ġoverlapping Ġregion Ġin ĠFigure ĠFIG REF 54 Ġ, Ġit Ġturns Ġout Ġthat Ġthe Ġnoun s Ġin Ġthe Ġseparated Ġclusters Ġare Ġwords Ġthat Ġcan Ġappear Ġas Ġsubjects Ġand , Ġtherefore , Ġfor Ġwhich Ġverb Ġagreement Ġis Ġimportant Ġto Ġmodel . ĠIn Ġcontrast , Ġthe Ġnoun s Ġin Ġthe Ġoverlapping Ġregion Ġare Ġtypically Ġobjects . ĠThis Ġdemonstrates Ġthat Ġthe Ġlatent Ġembed d ings Ġare Ġfocusing Ġon Ġaspects Ġof Ġlanguage Ġthat Ġare Ġspecifically Ġimportant Ġfor Ġmodeling Ġdependency Ġwithout Ġever Ġhaving Ġseen Ġexamples Ġof Ġdependency Ġpars es . ĠSome Ġprevious Ġwork Ġhas Ġdeliberately Ġcreated Ġembed d ings Ġto Ġcapture Ġdifferent Ġnotions Ġof Ġsimilarity ĠB IB REF 43 Ġ, ĠB IB REF 44 Ġ, Ġwhile Ġthey Ġuse Ġextra Ġmorphology Ġor Ġdependency Ġannotations Ġto Ġguide Ġthe Ġembed ding Ġlearning , Ġour Ġapproach Ġprovides Ġa Ġpotential Ġalternative Ġto Ġcreate Ġnew Ġembed d ings Ġthat Ġare Ġguided Ġby Ġstructured Ġsyntax Ġmodel , Ġonly Ġusing Ġunl abel ed Ġtext Ġcorpor a . </s> Related ĠWork </s> Our Ġapproach Ġis Ġrelated Ġto Ġflow - based Ġgener ative Ġmodels , Ġwhich Ġare Ġfirst Ġdescribed Ġin ĠN ICE ĠB IB REF 16 Ġand Ġhave Ġrecently Ġreceived Ġmore Ġattention ĠB IB REF 17 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 18 Ġ. ĠThis Ġrelevant Ġwork Ġmostly Ġadop ts Ġsimple Ġ( e . g . ĠGa ussian ) Ġand Ġfixed Ġpri ors Ġand Ġdoes Ġnot Ġattempt Ġto Ġlearn Ġinterpret able Ġlatent Ġstructures . ĠAnother Ġrelated Ġgener ative Ġmodel Ġclass Ġis Ġvari ational Ġauto - enc od ers Ġ( VA Es ) ĠB IB REF 45 Ġthat Ġoptimize Ġa Ġlower Ġbound Ġon Ġthe Ġmarginal Ġdata Ġlikelihood , Ġand Ġcan Ġbe Ġextended Ġto Ġlearn Ġlatent Ġstructures ĠB IB REF 46 Ġ, ĠB IB REF 47 Ġ. ĠAgainst Ġthe Ġflow - based Ġmodels , ĠVA Es Ġremove Ġthe Ġin vert ibility Ġconstraint Ġbut Ġsacrifice Ġthe Ġmerits Ġof Ġexact Ġinference Ġand Ġexact Ġlog Ġlikelihood Ġcomputation , Ġwhich Ġpotentially Ġresults Ġin Ġoptimization Ġchallenges ĠB IB REF 48 Ġ. ĠOur Ġapproach Ġcan Ġalso Ġbe Ġviewed Ġin Ġconnection Ġwith Ġgener ative Ġadvers arial Ġnetworks Ġ( GAN s ) ĠB IB REF 49 Ġthat Ġis Ġa Ġlikelihood - free Ġframework Ġto Ġlearn Ġimplicit Ġgener ative Ġmodels . ĠHowever , Ġit Ġis Ġnon - t riv ial Ġfor Ġa Ġgradient - based Ġmethod Ġlike ĠG AN s Ġto Ġpropagate Ġgrad ients Ġthrough Ġdiscrete Ġstructures . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġdefine Ġa Ġnovel Ġgener ative Ġapproach Ġto Ġleverage Ġcontinuous Ġword Ġrepresentations Ġfor Ġun super vised Ġlearning Ġof Ġsynt actic Ġstructure . ĠExper iments Ġon Ġboth ĠPOS Ġinduction Ġand Ġun super vised Ġdependency Ġparsing Ġtasks Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġproposed Ġapproach . ĠFuture Ġwork Ġmight Ġexplore Ġmore Ġsophisticated Ġin vert ible Ġprojections , Ġor Ġrecurrent Ġprojections Ġthat Ġjointly Ġtransform Ġthe Ġentire Ġinput Ġsequence .
Introduction Data annotation is a major bottleneck for the application of supervised learning approaches to many problems. As a result, unsupervised methods that learn directly from unlabeled data are increasingly important. For tasks related to unsupervised syntactic analysis, discrete generative models have dominated in recent years – for example, for both part-of-speech (POS) induction BIBREF0 , BIBREF1 and unsupervised dependency parsing BIBREF2 , BIBREF3 , BIBREF4 . While similar models have had success on a range of unsupervised tasks, they have mostly ignored the apparent utility of continuous word representations evident from supervised NLP applications BIBREF5 , BIBREF6 . In this work, we focus on leveraging and explicitly representing continuous word embeddings within unsupervised models of syntactic structure. Pre-trained word embeddings from massive unlabeled corpora offer a compact way of injecting a prior notion of word similarity into models that would otherwise treat words as discrete, isolated categories. However, the specific properties of language captured by any particular embedding scheme can be difficult to control, and, further, may not be ideally suited to the task at hand. For example, pre-trained skip-gram embeddings BIBREF7 with small context window size are found to capture the syntactic properties of language well BIBREF8 , BIBREF9 . However, if our goal is to separate syntactic categories, this embedding space is not ideal – POS categories correspond to overlapping interspersed regions in the embedding space, evident in Figure SECREF4 . In our approach, we propose to learn a new latent embedding space as a projection of pre-trained embeddings (depicted in Figure SECREF5 ), while jointly learning latent syntactic structure – for example, POS categories or syntactic dependencies. To this end, we introduce a new generative model (shown in Figure FIGREF6 ) that first generates a latent syntactic representation (e.g. a dependency parse) from a discrete structured prior (which we also call the “syntax model”), then, conditioned on this representation, generates a sequence of latent embedding random variables corresponding to each word, and finally produces the observed (pre-trained) word embeddings by projecting these latent vectors through a parameterized non-linear function. The latent embeddings can be jointly learned with the structured syntax model in a completely unsupervised fashion. By choosing an invertible neural network as our non-linear projector, and then parameterizing our model in terms of the projection's inverse, we are able to derive tractable exact inference and marginal likelihood computation procedures so long as inference is tractable in the underlying syntax model. In sec:learn-with-inv we show that this derivation corresponds to an alternate view of our approach whereby we jointly learn a mapping of observed word embeddings to a new embedding space that is more suitable for the syntax model, but include an additional Jacobian regularization term to prevent information loss. Recent work has sought to take advantage of word embeddings in unsupervised generative models with alternate approaches BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . BIBREF9 build an HMM with Gaussian emissions on observed word embeddings, but they do not attempt to learn new embeddings. BIBREF10 , BIBREF11 , and BIBREF12 extend HMM or dependency model with valence (DMV) BIBREF2 with multinomials that use word (or tag) embeddings in their parameterization. However, they do not represent the embeddings as latent variables. In experiments, we instantiate our approach using both a Markov-structured syntax model and a tree-structured syntax model – specifically, the DMV. We evaluate on two tasks: part-of-speech (POS) induction and unsupervised dependency parsing without gold POS tags. Experimental results on the Penn Treebank BIBREF13 demonstrate that our approach improves the basic HMM and DMV by a large margin, leading to the state-of-the-art results on POS induction, and state-of-the-art results on unsupervised dependency parsing in the difficult training scenario where neither gold POS annotation nor punctuation-based constraints are available. Model As an illustrative example, we first present a baseline model for Markov syntactic structure (POS induction) that treats a sequence of pre-trained word embeddings as observations. Then, we propose our novel approach, again using Markov structure, that introduces latent word embedding variables and a neural projector. Lastly, we extend our approach to more general syntactic structures. Example: Gaussian HMM We start by describing the Gaussian hidden Markov model introduced by BIBREF9 , which is a locally normalized model with multinomial transitions and Gaussian emissions. Given a sentence of length INLINEFORM0 , we denote the latent POS tags as INLINEFORM1 , observed (pre-trained) word embeddings as INLINEFORM2 , transition parameters as INLINEFORM3 , and Gaussian emission parameters as INLINEFORM4 . The joint distribution of data and latent variables factors as: DISPLAYFORM0 where INLINEFORM0 is the multinomial transition probability and INLINEFORM1 is the multivariate Gaussian emission probability. While the observed word embeddings do inform this model with a notion of word similarity – lacking in the basic multinomial HMM – the Gaussian emissions may not be sufficiently flexible to separate some syntactic categories in the complex pre-trained embedding space – for example the skip-gram embedding space as visualized in Figure SECREF4 where different POS categories overlap. Next we introduce a new approach that adds flexibility to the emission distribution by incorporating new latent embedding variables. Markov Structure with Neural Projector To flexibly model observed embeddings and yield a new representation space that is more suitable for the syntax model, we propose to cascade a neural network as a projection function, deterministically transforming the simple space defined by the Gaussian HMM to the observed embedding space. We denote the latent embedding of the INLINEFORM0 word in a sentence as INLINEFORM1 , and the neural projection function as INLINEFORM2 , parameterized by INLINEFORM3 . In the case of sequential Markov structure, our new model corresponds to the following generative process: For each time step INLINEFORM0 , [noitemsep, leftmargin=*] Draw the latent state INLINEFORM0 Draw the latent embedding INLINEFORM0 Deterministically produce embedding INLINEFORM0 The graphical model is depicted in Figure FIGREF6 . The deterministic projection can also be viewed as sampling each observation from a point mass at INLINEFORM0 . The joint distribution of our model is: DISPLAYFORM0 where INLINEFORM0 is a conditional Gaussian distribution, and INLINEFORM1 is the Dirac delta function centered at INLINEFORM2 : DISPLAYFORM0 General Structure with Neural Projector Our approach can be applied to a broad family of structured syntax models. We denote latent embedding variables as INLINEFORM0 , discrete latent variables in the syntax model as INLINEFORM1 ( INLINEFORM2 ), where INLINEFORM3 are conditioned to generate INLINEFORM4 . The joint probability of our model factors as: DISPLAYFORM0 where INLINEFORM0 represents the probability of the syntax model, and can encode any syntactic structure – though, its factorization structure will determine whether inference is tractable in our full model. As shown in Figure FIGREF6 , we focus on two syntax models for syntactic analysis in this paper. The first is Markov-structured, which we use for POS induction, and the second is DMV-structured, which we use to learn dependency parses without supervision. The marginal data likelihood of our model is: DISPLAYFORM0 While the discrete variables INLINEFORM0 can be marginalized out with dynamic program in many cases, it is generally intractable to marginalize out the latent continuous variables, INLINEFORM1 , for an arbitrary projection INLINEFORM2 in Eq. ( EQREF17 ), which means inference and learning may be difficult. In sec:opt, we address this issue by constraining INLINEFORM3 to be invertible, and show that this constraint enables tractable exact inference and marginal likelihood computation. Learning & Inference In this section, we introduce an invertibility condition for our neural projector to tackle the optimization challenge. Specifically, we constrain our neural projector with two requirements: (1) INLINEFORM0 and (2) INLINEFORM1 exists. Invertible transformations have been explored before in independent components analysis BIBREF14 , gaussianization BIBREF15 , and deep density models BIBREF16 , BIBREF17 , BIBREF18 , for unstructured data. Here, we generalize this style of approach to structured learning, and augment it with discrete latent variables ( INLINEFORM2 ). Under the invertibility condition, we derive a learning algorithm and give another view of our approach revealed by the objective function. Then, we present the architecture of a neural projector we use in experiments: a volume-preserving invertible neural network proposed by BIBREF16 for independent components estimation. Learning with Invertibility For ease of exposition, we explain the learning algorithm in terms of Markov structure without loss of generality. As shown in Eq. ( EQREF17 ), the optimization challenge in our approach comes from the intractability of the marginalized emission factor INLINEFORM0 . If we can marginalize out INLINEFORM1 and compute INLINEFORM2 , then the posterior and marginal likelihood of our Markov-structured model can be computed with the forward-backward algorithm. We can apply Eq. ( EQREF14 ) and obtain : INLINEFORM3 By using the change of variable rule to the integration, which allows the integration variable INLINEFORM0 to be replaced by INLINEFORM1 , the marginal emission factor can be computed in closed-form when the invertibility condition is satisfied: DISPLAYFORM0 where INLINEFORM0 is a conditional Gaussian distribution, INLINEFORM1 is the Jacobian matrix of function INLINEFORM2 at INLINEFORM3 , and INLINEFORM4 represents the absolute value of its determinant. This Jacobian term is nonzero and differentiable if and only if INLINEFORM5 exists. Eq. ( EQREF19 ) shows that we can directly calculate the marginal emission distribution INLINEFORM0 . Denote the marginal data likelihood of Gaussian HMM as INLINEFORM1 , then the log marginal data likelihood of our model can be directly written as: DISPLAYFORM0 where INLINEFORM0 represents the new sequence of embeddings after applying INLINEFORM1 to each INLINEFORM2 . Eq. ( EQREF20 ) shows that the training objective of our model is simply the Gaussian HMM log likelihood with an additional Jacobian regularization term. From this view, our approach can be seen as equivalent to reversely projecting the data through INLINEFORM3 to another manifold INLINEFORM4 that is directly modeled by the Gaussian HMM, with a regularization term. Intuitively, we optimize the reverse projection INLINEFORM5 to modify the INLINEFORM6 space, making it more appropriate for the syntax model. The Jacobian regularization term accounts for the volume expansion or contraction behavior of the projection. Maximizing it can be thought of as preventing information loss. In the extreme case, the Jacobian determinant is equal to zero, which means the projection is non-invertible and thus information is being lost through the projection. Such “information preserving” regularization is crucial during optimization, otherwise the trivial solution of always projecting data to the same single point to maximize likelihood is viable. More generally, for an arbitrary syntax model the data likelihood of our approach is: DISPLAYFORM0 If the syntax model itself allows for tractable inference and marginal likelihood computation, the same dynamic program can be used to marginalize out INLINEFORM0 . Therefore, our joint model inherits the tractability of the underlying syntax model. Invertible Volume-Preserving Neural Net For the projection we can use an arbitrary invertible function, and given the representational power of neural networks they seem a natural choice. However, calculating the inverse and Jacobian of an arbitrary neural network can be difficult, as it requires that all component functions be invertible and also requires storage of large Jacobian matrices, which is memory intensive. To address this issue, several recent papers propose specially designed invertible networks that are easily trainable yet still powerful BIBREF16 , BIBREF17 , BIBREF19 . Inspired by these works, we use the invertible transformation proposed by BIBREF16 , which consists of a series of “coupling layers”. This architecture is specially designed to guarantee a unit Jacobian determinant (and thus the invertibility property). From Eq. ( EQREF22 ) we know that only INLINEFORM0 is required for accomplishing learning and inference; we never need to explicitly construct INLINEFORM1 . Thus, we directly define the architecture of INLINEFORM2 . As shown in Figure FIGREF24 , the nonlinear transformation from the observed embedding INLINEFORM3 to INLINEFORM4 represents the first coupling layer. The input in this layer is partitioned into left and right halves of dimensions, INLINEFORM5 and INLINEFORM6 , respectively. A single coupling layer is defined as: DISPLAYFORM0 where INLINEFORM0 is the coupling function and can be any nonlinear form. This transformation satisfies INLINEFORM1 , and BIBREF16 show that its Jacobian matrix is triangular with all ones on the main diagonal. Thus the Jacobian determinant is always equal to one (i.e. volume-preserving) and the invertibility condition is naturally satisfied. To be sufficiently expressive, we compose multiple coupling layers as suggested in BIBREF16 . Specifically, we exchange the role of left and right half vectors at each layer as shown in Figure FIGREF24 . For instance, from INLINEFORM0 to INLINEFORM1 the left subset INLINEFORM2 is unchanged, while from INLINEFORM3 to INLINEFORM4 the right subset INLINEFORM5 remains the same. Also note that composing multiple coupling layers does not change the volume-preserving and invertibility properties. Such a sequence of invertible transformations from the data space INLINEFORM6 to INLINEFORM7 is also called normalizing flow BIBREF20 . Experiments In this section, we first describe our datasets and experimental setup. We then instantiate our approach with Markov and DMV-structured syntax models, and report results on POS tagging and dependency grammar induction respectively. Lastly, we analyze the learned latent embeddings. Data For both POS tagging and dependency parsing, we run experiments on the Wall Street Journal (WSJ) portion of the Penn Treebank. To create the observed data embeddings, we train skip-gram word embeddings BIBREF7 that are found to capture syntactic properties well when trained with small context window BIBREF8 , BIBREF9 . Following BIBREF9 , the dimensionality INLINEFORM0 is set to 100, and the training context window size is set to 1 to encode more syntactic information. The skip-gram embeddings are trained on the one billion word language modeling benchmark dataset BIBREF21 in addition to the WSJ corpus. General Experimental Setup For the neural projector, we employ rectified networks as coupling function INLINEFORM0 following BIBREF16 . We use a rectified network with an input layer, one hidden layer, and linear output units, the number of hidden units is set to the same as the number of input units. The number of coupling layers are varied as 4, 8, 16 for both tasks. We optimize marginal data likelihood directly using Adam BIBREF22 . For both tasks in the fully unsupervised setting, we do not tune the hyper-parameters using supervised data. Unsupervised POS tagging For unsupervised POS tagging, we use a Markov-structured syntax model in our approach, which is a popular structure for unsupervised tagging tasks BIBREF9 , BIBREF10 . Following existing literature, we train and test on the entire WSJ corpus (49208 sentences, 1M tokens). We use 45 tag clusters, the number of POS tags that appear in WSJ corpus. We train the discrete HMM and the Gaussian HMM BIBREF9 as baselines. For the Gaussian HMM, mean vectors of Gaussian emissions are initialized with the empirical mean of all word vectors with an additive noise. We assume diagonal covariance matrix for INLINEFORM0 and initialize it with the empirical variance of the word vectors. Following BIBREF9 , the covariance matrix is fixed during training. The multinomial probabilities are initialized as INLINEFORM1 , where INLINEFORM2 . For our approach, we initialize the syntax model and Gaussian parameters with the pre-trained Gaussian HMM. The weights of layers in the rectified network are initialized from a uniform distribution with mean zero and a standard deviation of INLINEFORM3 , where INLINEFORM4 is the input dimension. We evaluate the performance of POS tagging with both Many-to-One (M-1) accuracy BIBREF23 and V-Measure (VM) BIBREF24 . Given a model we found that the tagging performance is well-correlated with the training data likelihood, thus we use training data likelihood as a unsupervised criterion to select the trained model over 10 random restarts after training 50 epochs. We repeat this process 5 times and report the mean and standard deviation of performance. We compare our approach with basic HMM, Gaussian HMM, and several state-of-the-art systems, including sophisticated HMM variants and clustering techniques with hand-engineered features. The results are presented in Table TABREF32 . Through the introduced latent embeddings and additional neural projection, our approach improves over the Gaussian HMM by 5.4 points in M-1 and 5.6 points in VM. Neural HMM (NHMM) BIBREF10 is a baseline that also learns word representation jointly. Both their basic model and extended Conv version does not outperform the Gaussian HMM. Their best model incorporates another LSTM to model long distance dependency and breaks the Markov assumption, yet our approach still achieves substantial improvement over it without considering more context information. Moreover, our method outperforms the best published result that benefits from hand-engineered features BIBREF27 by 2.0 points on VM. We found that most tagging errors happen in noun subcategories. Therefore, we do the one-to-one mapping between gold POS tags and induced clusters and plot the normalized confusion matrix of noun subcategories in Figure FIGREF35 . The Gaussian HMM fails to identify “NN” and “NNS” correctly for most cases, and it often recognizes “NNPS” as “NNP”. In contrast, our approach corrects these errors well. Unsupervised Dependency Parsing without gold POS tags For the task of unsupervised dependency parse induction, we employ the Dependency Model with Valence (DMV) BIBREF2 as the syntax model in our approach. DMV is a generative model that defines a probability distribution over dependency parse trees and syntactic categories, generating tokens and dependencies in a head-outward fashion. While, traditionally, DMV is trained using gold POS tags as observed syntactic categories, in our approach, we treat each tag as a latent variable, as described in sec:general-neural. Most existing approaches to this task are not fully unsupervised since they rely on gold POS tags following the original experimental setup for DMV. This is partially because automatically parsing from words is difficult even when using unsupervised syntactic categories BIBREF29 . However, inducing dependencies from words alone represents a more realistic experimental condition since gold POS tags are often unavailable in practice. Previous work that has trained from words alone often requires additional linguistic constraints (like sentence internal boundaries) BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , acoustic cues BIBREF33 , additional training data BIBREF4 , or annotated data from related languages BIBREF34 . Our approach is naturally designed to train on word embeddings directly, thus we attempt to induce dependencies without using gold POS tags or other extra linguistic information. Like previous work we use sections 02-21 of WSJ corpus as training data and evaluate on section 23, we remove punctuations and train the models on sentences of length INLINEFORM0 , “head-percolation” rules BIBREF39 are applied to obtain gold dependencies for evaluation. We train basic DMV, extended DMV (E-DMV) BIBREF35 and Gaussian DMV (which treats POS tag as unknown latent variables and generates observed word embeddings directly conditioned on them following Gaussian distribution) as baselines. Basic DMV and E-DMV are trained with Viterbi EM BIBREF40 on unsupervised POS tags induced from our Markov-structured model described in sec:pos. Multinomial parameters of the syntax model in both Gaussian DMV and our model are initialized with the pre-trained DMV baseline. Other parameters are initialized in the same way as in the POS tagging experiment. The directed dependency accuracy (DDA) is used for evaluation and we report accuracy on sentences of length INLINEFORM1 and all lengths. We train the parser until training data likelihood converges, and report the mean and standard deviation over 20 random restarts. Our model directly observes word embeddings and does not require gold POS tags during training. Thus, results from related work trained on gold tags are not directly comparable. However, to measure how these systems might perform without gold tags, we run three recent state-of-the-art systems in our experimental setting: UR-A E-DMV BIBREF36 , Neural E-DMV BIBREF11 , and CRF Autoencoder (CRFAE) BIBREF37 . We use unsupervised POS tags (induced from our Markov-structured model) in place of gold tags. We also train basic DMV on gold tags and include several state-of-the-art results on gold tags as reference points. As shown in Table TABREF39 , our approach is able to improve over the Gaussian DMV by 4.8 points on length INLINEFORM0 and 4.8 points on all lengths, which suggests the additional latent embedding layer and neural projector are helpful. The proposed approach yields, to the best of our knowledge, state-of-the-art performance without gold POS annotation and without sentence-internal boundary information. DMV, UR-A E-DMV, Neural E-DMV, and CRFAE suffer a large decrease in performance when trained on unsupervised tags – an effect also seen in previous work BIBREF29 , BIBREF34 . Since our approach induces latent POS tags jointly with dependency trees, it may be able to learn POS clusters that are more amenable to grammar induction than the unsupervised tags. We observe that CRFAE underperforms its gold-tag counterpart substantially. This may largely be a result of the model's reliance on prior linguistic rules that become unavailable when gold POS tag types are unknown. Many extensions to DMV can be considered orthogonal to our approach – they essentially focus on improving the syntax model. It is possible that incorporating these more sophisticated syntax models into our approach may lead to further improvements. Sensitivity Analysis In the above experiments we initialize the structured syntax components with the pre-trained Gaussian or discrete baseline, which is shown as a useful technique to help train our deep models. We further study the results with fully random initialization. In the POS tagging experiment, we report the results in Table TABREF48 . While the performance with 4 layers is comparable to the pre-trained Gaussian initialization, deeper projections (8 or 16 layers) result in a dramatic drop in performance. This suggests that the structured syntax model with very deep projections is difficult to train from scratch, and a simpler projection might be a good compromise in the random initialization setting. Different from the Markov prior in POS tagging experiments, our parsing model seems to be quite sensitive to the initialization. For example, directed accuracy of our approach on sentences of length INLINEFORM0 is below 40.0 with random initialization. This is consistent with previous work that has noted the importance of careful initialization for DMV-based models such as the commonly used harmonic initializer BIBREF2 . However, it is not straightforward to apply the harmonic initializer for DMV directly in our model without using some kind of pre-training since we do not observe gold POS. We investigate the effect of the choice of pre-trained embedding on performance while using our approach. To this end, we additionally include results using fastText embeddings BIBREF41 – which, in contrast with skip-gram embeddings, include character-level information. We set the context windows size to 1 and the dimension size to 100 as in the skip-gram training, while keeping other parameters set to their defaults. These results are summarized in Table TABREF50 and Table TABREF51 . While fastText embeddings lead to reduced performance with our model, our approach still yields an improvement over the Gaussian baseline with the new observed embeddings space. Qualitative Analysis of Embeddings We perform qualitative analysis to understand how the latent embeddings help induce syntactic structures. First we filter out low-frequency words and punctuations in WSJ, and visualize the rest words (10k) with t-SNE BIBREF42 under different embeddings. We assign each word with its most likely gold POS tags in WSJ and color them according to the gold POS tags. For our Markov-structured model, we have displayed the embedding space in Figure SECREF5 , where the gold POS clusters are well-formed. Further, we present five example target words and their five nearest neighbors in terms of cosine similarity. As shown in Table TABREF53 , the skip-gram embedding captures both semantic and syntactic aspects to some degree, yet our embeddings are able to focus especially on the syntactic aspects of words, in an unsupervised fashion without using any extra morphological information. In Figure FIGREF54 we depict the learned latent embeddings with the DMV-structured syntax model. Unlike the Markov structure, the DMV structure maps a large subset of singular and plural nouns to the same overlapping region. However, two clusters of singular and plural nouns are actually separated. We inspect the two clusters and the overlapping region in Figure FIGREF54 , it turns out that the nouns in the separated clusters are words that can appear as subjects and, therefore, for which verb agreement is important to model. In contrast, the nouns in the overlapping region are typically objects. This demonstrates that the latent embeddings are focusing on aspects of language that are specifically important for modeling dependency without ever having seen examples of dependency parses. Some previous work has deliberately created embeddings to capture different notions of similarity BIBREF43 , BIBREF44 , while they use extra morphology or dependency annotations to guide the embedding learning, our approach provides a potential alternative to create new embeddings that are guided by structured syntax model, only using unlabeled text corpora. Related Work Our approach is related to flow-based generative models, which are first described in NICE BIBREF16 and have recently received more attention BIBREF17 , BIBREF19 , BIBREF18 . This relevant work mostly adopts simple (e.g. Gaussian) and fixed priors and does not attempt to learn interpretable latent structures. Another related generative model class is variational auto-encoders (VAEs) BIBREF45 that optimize a lower bound on the marginal data likelihood, and can be extended to learn latent structures BIBREF46 , BIBREF47 . Against the flow-based models, VAEs remove the invertibility constraint but sacrifice the merits of exact inference and exact log likelihood computation, which potentially results in optimization challenges BIBREF48 . Our approach can also be viewed in connection with generative adversarial networks (GANs) BIBREF49 that is a likelihood-free framework to learn implicit generative models. However, it is non-trivial for a gradient-based method like GANs to propagate gradients through discrete structures. Conclusion In this work, we define a novel generative approach to leverage continuous word representations for unsupervised learning of syntactic structure. Experiments on both POS induction and unsupervised dependency parsing tasks demonstrate the effectiveness of our proposed approach. Future work might explore more sophisticated invertible projections, or recurrent projections that jointly transform the entire input sequence.
What is the invertibility condition?
887c6727e9f25ade61b4853a869fe712fe0b703d
What Ġis Ġthe Ġin vert ibility Ġcondition ?
The Ġneural Ġprojector Ġmust Ġbe Ġin vert ible .
<s> What Ġis Ġthe Ġin vert ibility Ġcondition ? </s> Introduction </s> Data Ġannotation Ġis Ġa Ġmajor Ġbottleneck Ġfor Ġthe Ġapplication Ġof Ġsupervised Ġlearning Ġapproaches Ġto Ġmany Ġproblems . ĠAs Ġa Ġresult , Ġun super vised Ġmethods Ġthat Ġlearn Ġdirectly Ġfrom Ġunl abel ed Ġdata Ġare Ġincreasingly Ġimportant . ĠFor Ġtasks Ġrelated Ġto Ġun super vised Ġsynt actic Ġanalysis , Ġdiscrete Ġgener ative Ġmodels Ġhave Ġdominated Ġin Ġrecent Ġyears ĠâĢĵ Ġfor Ġexample , Ġfor Ġboth Ġpart - of - speech Ġ( POS ) Ġinduction ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġand Ġun super vised Ġdependency Ġparsing ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠWhile Ġsimilar Ġmodels Ġhave Ġhad Ġsuccess Ġon Ġa Ġrange Ġof Ġun super vised Ġtasks , Ġthey Ġhave Ġmostly Ġignored Ġthe Ġapparent Ġutility Ġof Ġcontinuous Ġword Ġrepresentations Ġevident Ġfrom Ġsupervised ĠN LP Ġapplications ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġfocus Ġon Ġleveraging Ġand Ġexplicitly Ġrepresenting Ġcontinuous Ġword Ġembed d ings Ġwithin Ġun super vised Ġmodels Ġof Ġsynt actic Ġstructure . </s> Pre - trained Ġword Ġembed d ings Ġfrom Ġmassive Ġunl abel ed Ġcorpor a Ġoffer Ġa Ġcompact Ġway Ġof Ġinjecting Ġa Ġprior Ġnotion Ġof Ġword Ġsimilarity Ġinto Ġmodels Ġthat Ġwould Ġotherwise Ġtreat Ġwords Ġas Ġdiscrete , Ġisolated Ġcategories . ĠHowever , Ġthe Ġspecific Ġproperties Ġof Ġlanguage Ġcaptured Ġby Ġany Ġparticular Ġembed ding Ġscheme Ġcan Ġbe Ġdifficult Ġto Ġcontrol , Ġand , Ġfurther , Ġmay Ġnot Ġbe Ġideally Ġsuited Ġto Ġthe Ġtask Ġat Ġhand . ĠFor Ġexample , Ġpre - trained Ġskip - gram Ġembed d ings ĠB IB REF 7 Ġwith Ġsmall Ġcontext Ġwindow Ġsize Ġare Ġfound Ġto Ġcapture Ġthe Ġsynt actic Ġproperties Ġof Ġlanguage Ġwell ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠHowever , Ġif Ġour Ġgoal Ġis Ġto Ġseparate Ġsynt actic Ġcategories , Ġthis Ġembed ding Ġspace Ġis Ġnot Ġideal ĠâĢĵ ĠPOS Ġcategories Ġcorrespond Ġto Ġoverlapping Ġinter sp ersed Ġregions Ġin Ġthe Ġembed ding Ġspace , Ġevident Ġin ĠFigure ĠSEC REF 4 Ġ. </s> In Ġour Ġapproach , Ġwe Ġpropose Ġto Ġlearn Ġa Ġnew Ġlatent Ġembed ding Ġspace Ġas Ġa Ġprojection Ġof Ġpre - trained Ġembed d ings Ġ( dep icted Ġin ĠFigure ĠSEC REF 5 Ġ), Ġwhile Ġjointly Ġlearning Ġlatent Ġsynt actic Ġstructure ĠâĢĵ Ġfor Ġexample , ĠPOS Ġcategories Ġor Ġsynt actic Ġdependencies . ĠTo Ġthis Ġend , Ġwe Ġintroduce Ġa Ġnew Ġgener ative Ġmodel Ġ( shown Ġin ĠFigure ĠFIG REF 6 Ġ) Ġthat Ġfirst Ġgenerates Ġa Ġlatent Ġsynt actic Ġrepresentation Ġ( e . g . Ġa Ġdependency Ġparse ) Ġfrom Ġa Ġdiscrete Ġstructured Ġprior Ġ( which Ġwe Ġalso Ġcall Ġthe ĠâĢ ľ sy ntax Ġmodel âĢ Ŀ ), Ġthen , Ġconditioned Ġon Ġthis Ġrepresentation , Ġgenerates Ġa Ġsequence Ġof Ġlatent Ġembed ding Ġrandom Ġvariables Ġcorresponding Ġto Ġeach Ġword , Ġand Ġfinally Ġproduces Ġthe Ġobserved Ġ( pre - trained ) Ġword Ġembed d ings Ġby Ġprojecting Ġthese Ġlatent Ġvectors Ġthrough Ġa Ġparameter ized Ġnon - linear Ġfunction . ĠThe Ġlatent Ġembed d ings Ġcan Ġbe Ġjointly Ġlearned Ġwith Ġthe Ġstructured Ġsyntax Ġmodel Ġin Ġa Ġcompletely Ġun super vised Ġfashion . </s> By Ġchoosing Ġan Ġin vert ible Ġneural Ġnetwork Ġas Ġour Ġnon - linear Ġprojector , Ġand Ġthen Ġparameter izing Ġour Ġmodel Ġin Ġterms Ġof Ġthe Ġprojection 's Ġinverse , Ġwe Ġare Ġable Ġto Ġderive Ġtract able Ġexact Ġinference Ġand Ġmarginal Ġlikelihood Ġcomputation Ġprocedures Ġso Ġlong Ġas Ġinference Ġis Ġtract able Ġin Ġthe Ġunderlying Ġsyntax Ġmodel . ĠIn Ġsec : learn - with - inv Ġwe Ġshow Ġthat Ġthis Ġderiv ation Ġcorresponds Ġto Ġan Ġalternate Ġview Ġof Ġour Ġapproach Ġwhereby Ġwe Ġjointly Ġlearn Ġa Ġmapping Ġof Ġobserved Ġword Ġembed d ings Ġto Ġa Ġnew Ġembed ding Ġspace Ġthat Ġis Ġmore Ġsuitable Ġfor Ġthe Ġsyntax Ġmodel , Ġbut Ġinclude Ġan Ġadditional ĠJacob ian Ġregular ization Ġterm Ġto Ġprevent Ġinformation Ġloss . </s> Recent Ġwork Ġhas Ġsought Ġto Ġtake Ġadvantage Ġof Ġword Ġembed d ings Ġin Ġun super vised Ġgener ative Ġmodels Ġwith Ġalternate Ġapproaches ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ. ĠB IB REF 9 Ġbuild Ġan ĠH MM Ġwith ĠGa ussian Ġemissions Ġon Ġobserved Ġword Ġembed d ings , Ġbut Ġthey Ġdo Ġnot Ġattempt Ġto Ġlearn Ġnew Ġembed d ings . ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, Ġand ĠB IB REF 12 Ġextend ĠH MM Ġor Ġdependency Ġmodel Ġwith Ġval ence Ġ( DM V ) ĠB IB REF 2 Ġwith Ġmult in om ials Ġthat Ġuse Ġword Ġ( or Ġtag ) Ġembed d ings Ġin Ġtheir Ġparameter ization . ĠHowever , Ġthey Ġdo Ġnot Ġrepresent Ġthe Ġembed d ings Ġas Ġlatent Ġvariables . </s> In Ġexperiments , Ġwe Ġinstant iate Ġour Ġapproach Ġusing Ġboth Ġa ĠMark ov - struct ured Ġsyntax Ġmodel Ġand Ġa Ġtree - struct ured Ġsyntax Ġmodel ĠâĢĵ Ġspecifically , Ġthe ĠDMV . ĠWe Ġevaluate Ġon Ġtwo Ġtasks : Ġpart - of - speech Ġ( POS ) Ġinduction Ġand Ġun super vised Ġdependency Ġparsing Ġwithout Ġgold ĠPOS Ġtags . ĠExperimental Ġresults Ġon Ġthe ĠPenn ĠTree bank ĠB IB REF 13 Ġdemonstrate Ġthat Ġour Ġapproach Ġimproves Ġthe Ġbasic ĠH MM Ġand ĠDMV Ġby Ġa Ġlarge Ġmargin , Ġleading Ġto Ġthe Ġstate - of - the - art Ġresults Ġon ĠPOS Ġinduction , Ġand Ġstate - of - the - art Ġresults Ġon Ġun super vised Ġdependency Ġparsing Ġin Ġthe Ġdifficult Ġtraining Ġscenario Ġwhere Ġneither Ġgold ĠPOS Ġannotation Ġnor Ġpunct uation - based Ġconstraints Ġare Ġavailable . </s> Model </s> As Ġan Ġillust rative Ġexample , Ġwe Ġfirst Ġpresent Ġa Ġbaseline Ġmodel Ġfor ĠMark ov Ġsynt actic Ġstructure Ġ( POS Ġinduction ) Ġthat Ġtreats Ġa Ġsequence Ġof Ġpre - trained Ġword Ġembed d ings Ġas Ġobservations . ĠThen , Ġwe Ġpropose Ġour Ġnovel Ġapproach , Ġagain Ġusing ĠMark ov Ġstructure , Ġthat Ġintroduces Ġlatent Ġword Ġembed ding Ġvariables Ġand Ġa Ġneural Ġprojector . ĠLastly , Ġwe Ġextend Ġour Ġapproach Ġto Ġmore Ġgeneral Ġsynt actic Ġstructures . </s> Example : ĠGa ussian ĠH MM </s> We Ġstart Ġby Ġdescribing Ġthe ĠGa ussian Ġhidden ĠMark ov Ġmodel Ġintroduced Ġby ĠB IB REF 9 Ġ, Ġwhich Ġis Ġa Ġlocally Ġnormalized Ġmodel Ġwith Ġmult in omial Ġtransitions Ġand ĠGa ussian Ġemissions . ĠGiven Ġa Ġsentence Ġof Ġlength ĠIN LINE FORM 0 Ġ, Ġwe Ġdenote Ġthe Ġlatent ĠPOS Ġtags Ġas ĠIN LINE FORM 1 Ġ, Ġobserved Ġ( pre - trained ) Ġword Ġembed d ings Ġas ĠIN LINE FORM 2 Ġ, Ġtransition Ġparameters Ġas ĠIN LINE FORM 3 Ġ, Ġand ĠGa ussian Ġemission Ġparameters Ġas ĠIN LINE FORM 4 Ġ. ĠThe Ġjoint Ġdistribution Ġof Ġdata Ġand Ġlatent Ġvariables Ġfactors Ġas : </s> DIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġmult in omial Ġtransition Ġprobability Ġand ĠIN LINE FORM 1 Ġis Ġthe Ġmult ivariate ĠGa ussian Ġemission Ġprobability . </s> While Ġthe Ġobserved Ġword Ġembed d ings Ġdo Ġinform Ġthis Ġmodel Ġwith Ġa Ġnotion Ġof Ġword Ġsimilarity ĠâĢĵ Ġlacking Ġin Ġthe Ġbasic Ġmult in omial ĠH MM ĠâĢĵ Ġthe ĠGa ussian Ġemissions Ġmay Ġnot Ġbe Ġsufficiently Ġflexible Ġto Ġseparate Ġsome Ġsynt actic Ġcategories Ġin Ġthe Ġcomplex Ġpre - trained Ġembed ding Ġspace ĠâĢĵ Ġfor Ġexample Ġthe Ġskip - gram Ġembed ding Ġspace Ġas Ġvisual ized Ġin ĠFigure ĠSEC REF 4 Ġwhere Ġdifferent ĠPOS Ġcategories Ġoverlap . ĠNext Ġwe Ġintroduce Ġa Ġnew Ġapproach Ġthat Ġadds Ġflexibility Ġto Ġthe Ġemission Ġdistribution Ġby Ġincorporating Ġnew Ġlatent Ġembed ding Ġvariables . </s> Mark ov ĠStructure Ġwith ĠNeural ĠProject or </s> To Ġflex ibly Ġmodel Ġobserved Ġembed d ings Ġand Ġyield Ġa Ġnew Ġrepresentation Ġspace Ġthat Ġis Ġmore Ġsuitable Ġfor Ġthe Ġsyntax Ġmodel , Ġwe Ġpropose Ġto Ġcascade Ġa Ġneural Ġnetwork Ġas Ġa Ġprojection Ġfunction , Ġdeter minist ically Ġtransforming Ġthe Ġsimple Ġspace Ġdefined Ġby Ġthe ĠGa ussian ĠH MM Ġto Ġthe Ġobserved Ġembed ding Ġspace . ĠWe Ġdenote Ġthe Ġlatent Ġembed ding Ġof Ġthe ĠIN LINE FORM 0 Ġword Ġin Ġa Ġsentence Ġas ĠIN LINE FORM 1 Ġ, Ġand Ġthe Ġneural Ġprojection Ġfunction Ġas ĠIN LINE FORM 2 Ġ, Ġparameter ized Ġby ĠIN LINE FORM 3 Ġ. ĠIn Ġthe Ġcase Ġof Ġsequential ĠMark ov Ġstructure , Ġour Ġnew Ġmodel Ġcorresponds Ġto Ġthe Ġfollowing Ġgener ative Ġprocess : </s> For Ġeach Ġtime Ġstep ĠIN LINE FORM 0 Ġ, </s> [ no item se p , Ġleft margin = * ] </s> Draw Ġthe Ġlatent Ġstate ĠIN LINE FORM 0 </s> Draw Ġthe Ġlatent Ġembed ding ĠIN LINE FORM 0 </s> D eter minist ically Ġproduce Ġembed ding </s> IN LINE FORM 0 </s> The Ġgraphical Ġmodel Ġis Ġdepicted Ġin ĠFigure ĠFIG REF 6 Ġ. ĠThe Ġdeter ministic Ġprojection Ġcan Ġalso Ġbe Ġviewed Ġas Ġsampling Ġeach Ġobservation Ġfrom Ġa Ġpoint Ġmass Ġat ĠIN LINE FORM 0 Ġ. ĠThe Ġjoint Ġdistribution Ġof Ġour Ġmodel Ġis : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġconditional ĠGa ussian Ġdistribution , Ġand ĠIN LINE FORM 1 Ġis Ġthe ĠDir ac Ġdelta Ġfunction Ġcentered Ġat ĠIN LINE FORM 2 Ġ: ĠDIS PLAY FORM 0 </s> General ĠStructure Ġwith ĠNeural ĠProject or </s> Our Ġapproach Ġcan Ġbe Ġapplied Ġto Ġa Ġbroad Ġfamily Ġof Ġstructured Ġsyntax Ġmodels . ĠWe Ġdenote Ġlatent Ġembed ding Ġvariables Ġas ĠIN LINE FORM 0 Ġ, Ġdiscrete Ġlatent Ġvariables Ġin Ġthe Ġsyntax Ġmodel Ġas ĠIN LINE FORM 1 Ġ( ĠIN LINE FORM 2 Ġ), Ġwhere ĠIN LINE FORM 3 Ġare Ġconditioned Ġto Ġgenerate ĠIN LINE FORM 4 Ġ. ĠThe Ġjoint Ġprobability Ġof Ġour Ġmodel Ġfactors Ġas : </s> DIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġprobability Ġof Ġthe Ġsyntax Ġmodel , Ġand Ġcan Ġencode Ġany Ġsynt actic Ġstructure ĠâĢĵ Ġthough , Ġits Ġfactor ization Ġstructure Ġwill Ġdetermine Ġwhether Ġinference Ġis Ġtract able Ġin Ġour Ġfull Ġmodel . ĠAs Ġshown Ġin ĠFigure ĠFIG REF 6 Ġ, Ġwe Ġfocus Ġon Ġtwo Ġsyntax Ġmodels Ġfor Ġsynt actic Ġanalysis Ġin Ġthis Ġpaper . ĠThe Ġfirst Ġis ĠMark ov - struct ured , Ġwhich Ġwe Ġuse Ġfor ĠPOS Ġinduction , Ġand Ġthe Ġsecond Ġis ĠDMV - struct ured , Ġwhich Ġwe Ġuse Ġto Ġlearn Ġdependency Ġpars es Ġwithout Ġsupervision . </s> The Ġmarginal Ġdata Ġlikelihood Ġof Ġour Ġmodel Ġis : ĠDIS PLAY FORM 0 </s> While Ġthe Ġdiscrete Ġvariables ĠIN LINE FORM 0 Ġcan Ġbe Ġmarginalized Ġout Ġwith Ġdynamic Ġprogram Ġin Ġmany Ġcases , Ġit Ġis Ġgenerally Ġint ract able Ġto Ġmarginal ize Ġout Ġthe Ġlatent Ġcontinuous Ġvariables , ĠIN LINE FORM 1 Ġ, Ġfor Ġan Ġarbitrary Ġprojection ĠIN LINE FORM 2 Ġin ĠE q . Ġ( ĠEQ REF 17 Ġ), Ġwhich Ġmeans Ġinference Ġand Ġlearning Ġmay Ġbe Ġdifficult . ĠIn Ġsec : opt , Ġwe Ġaddress Ġthis Ġissue Ġby Ġconst raining ĠIN LINE FORM 3 Ġto Ġbe Ġin vert ible , Ġand Ġshow Ġthat Ġthis Ġconstraint Ġenables Ġtract able Ġexact Ġinference Ġand Ġmarginal Ġlikelihood Ġcomputation . </s> Learning Ġ& ĠIn ference </s> In Ġthis Ġsection , Ġwe Ġintroduce Ġan Ġin vert ibility Ġcondition Ġfor Ġour Ġneural Ġprojector Ġto Ġtackle Ġthe Ġoptimization Ġchallenge . ĠSpecifically , Ġwe Ġconst rain Ġour Ġneural Ġprojector Ġwith Ġtwo Ġrequirements : Ġ( 1 ) ĠIN LINE FORM 0 Ġand Ġ( 2 ) ĠIN LINE FORM 1 Ġexists . ĠIn vert ible Ġtransformations Ġhave Ġbeen Ġexplored Ġbefore Ġin Ġindependent Ġcomponents Ġanalysis ĠB IB REF 14 Ġ, Ġga ussian ization ĠB IB REF 15 Ġ, Ġand Ġdeep Ġdensity Ġmodels ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġfor Ġun struct ured Ġdata . ĠHere , Ġwe Ġgeneral ize Ġthis Ġstyle Ġof Ġapproach Ġto Ġstructured Ġlearning , Ġand Ġaugment Ġit Ġwith Ġdiscrete Ġlatent Ġvariables Ġ( ĠIN LINE FORM 2 Ġ). ĠUnder Ġthe Ġin vert ibility Ġcondition , Ġwe Ġderive Ġa Ġlearning Ġalgorithm Ġand Ġgive Ġanother Ġview Ġof Ġour Ġapproach Ġrevealed Ġby Ġthe Ġobjective Ġfunction . ĠThen , Ġwe Ġpresent Ġthe Ġarchitecture Ġof Ġa Ġneural Ġprojector Ġwe Ġuse Ġin Ġexperiments : Ġa Ġvolume - pres erving Ġin vert ible Ġneural Ġnetwork Ġproposed Ġby ĠB IB REF 16 Ġfor Ġindependent Ġcomponents Ġestimation . </s> Learning Ġwith ĠIn vert ibility </s> For Ġease Ġof Ġexposition , Ġwe Ġexplain Ġthe Ġlearning Ġalgorithm Ġin Ġterms Ġof ĠMark ov Ġstructure Ġwithout Ġloss Ġof Ġgener ality . ĠAs Ġshown Ġin ĠE q . Ġ( ĠEQ REF 17 Ġ), Ġthe Ġoptimization Ġchallenge Ġin Ġour Ġapproach Ġcomes Ġfrom Ġthe Ġint ract ability Ġof Ġthe Ġmarginalized Ġemission Ġfactor ĠIN LINE FORM 0 Ġ. ĠIf Ġwe Ġcan Ġmarginal ize Ġout ĠIN LINE FORM 1 Ġand Ġcompute ĠIN LINE FORM 2 Ġ, Ġthen Ġthe Ġposterior Ġand Ġmarginal Ġlikelihood Ġof Ġour ĠMark ov - struct ured Ġmodel Ġcan Ġbe Ġcomputed Ġwith Ġthe Ġforward - back ward Ġalgorithm . ĠWe Ġcan Ġapply ĠE q . Ġ( ĠEQ REF 14 Ġ) Ġand Ġobtain Ġ: ĠIN LINE FORM 3 </s> By Ġusing Ġthe Ġchange Ġof Ġvariable Ġrule Ġto Ġthe Ġintegration , Ġwhich Ġallows Ġthe Ġintegration Ġvariable ĠIN LINE FORM 0 Ġto Ġbe Ġreplaced Ġby ĠIN LINE FORM 1 Ġ, Ġthe Ġmarginal Ġemission Ġfactor Ġcan Ġbe Ġcomputed Ġin Ġclosed - form Ġwhen Ġthe Ġin vert ibility Ġcondition Ġis Ġsatisfied : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġa Ġconditional ĠGa ussian Ġdistribution , ĠIN LINE FORM 1 Ġis Ġthe ĠJacob ian Ġmatrix Ġof Ġfunction ĠIN LINE FORM 2 Ġat ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġrepresents Ġthe Ġabsolute Ġvalue Ġof Ġits Ġdetermin ant . ĠThis ĠJacob ian Ġterm Ġis Ġnon zero Ġand Ġdifferent iable Ġif Ġand Ġonly Ġif ĠIN LINE FORM 5 Ġexists . </s> E q . Ġ( ĠEQ REF 19 Ġ) Ġshows Ġthat Ġwe Ġcan Ġdirectly Ġcalculate Ġthe Ġmarginal Ġemission Ġdistribution ĠIN LINE FORM 0 Ġ. ĠDen ote Ġthe Ġmarginal Ġdata Ġlikelihood Ġof ĠGa ussian ĠH MM Ġas ĠIN LINE FORM 1 Ġ, Ġthen Ġthe Ġlog Ġmarginal Ġdata Ġlikelihood Ġof Ġour Ġmodel Ġcan Ġbe Ġdirectly Ġwritten Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġrepresents Ġthe Ġnew Ġsequence Ġof Ġembed d ings Ġafter Ġapplying ĠIN LINE FORM 1 Ġto Ġeach ĠIN LINE FORM 2 Ġ. ĠE q . Ġ( ĠEQ REF 20 Ġ) Ġshows Ġthat Ġthe Ġtraining Ġobjective Ġof Ġour Ġmodel Ġis Ġsimply Ġthe ĠGa ussian ĠH MM Ġlog Ġlikelihood Ġwith Ġan Ġadditional ĠJacob ian Ġregular ization Ġterm . ĠFrom Ġthis Ġview , Ġour Ġapproach Ġcan Ġbe Ġseen Ġas Ġequivalent Ġto Ġreverse ly Ġprojecting Ġthe Ġdata Ġthrough ĠIN LINE FORM 3 Ġto Ġanother Ġmanifold ĠIN LINE FORM 4 Ġthat Ġis Ġdirectly Ġmodeled Ġby Ġthe ĠGa ussian ĠH MM , Ġwith Ġa Ġregular ization Ġterm . ĠInt uitive ly , Ġwe Ġoptimize Ġthe Ġreverse Ġprojection ĠIN LINE FORM 5 Ġto Ġmodify Ġthe ĠIN LINE FORM 6 Ġspace , Ġmaking Ġit Ġmore Ġappropriate Ġfor Ġthe Ġsyntax Ġmodel . ĠThe ĠJacob ian Ġregular ization Ġterm Ġaccounts Ġfor Ġthe Ġvolume Ġexpansion Ġor Ġcontraction Ġbehavior Ġof Ġthe Ġprojection . ĠMaxim izing Ġit Ġcan Ġbe Ġthought Ġof Ġas Ġpreventing Ġinformation Ġloss . ĠIn Ġthe Ġextreme Ġcase , Ġthe ĠJacob ian Ġdetermin ant Ġis Ġequal Ġto Ġzero , Ġwhich Ġmeans Ġthe Ġprojection Ġis Ġnon - in vert ible Ġand Ġthus Ġinformation Ġis Ġbeing Ġlost Ġthrough Ġthe Ġprojection . ĠSuch ĠâĢ ľ information Ġpreserving âĢ Ŀ Ġregular ization Ġis Ġcrucial Ġduring Ġoptimization , Ġotherwise Ġthe Ġtrivial Ġsolution Ġof Ġalways Ġprojecting Ġdata Ġto Ġthe Ġsame Ġsingle Ġpoint Ġto Ġmaximize Ġlikelihood Ġis Ġviable . </s> More Ġgenerally , Ġfor Ġan Ġarbitrary Ġsyntax Ġmodel Ġthe Ġdata Ġlikelihood Ġof Ġour Ġapproach Ġis : ĠDIS PLAY FORM 0 </s> If Ġthe Ġsyntax Ġmodel Ġitself Ġallows Ġfor Ġtract able Ġinference Ġand Ġmarginal Ġlikelihood Ġcomputation , Ġthe Ġsame Ġdynamic Ġprogram Ġcan Ġbe Ġused Ġto Ġmarginal ize Ġout ĠIN LINE FORM 0 Ġ. ĠTherefore , Ġour Ġjoint Ġmodel Ġinher its Ġthe Ġtract ability Ġof Ġthe Ġunderlying Ġsyntax Ġmodel . </s> In vert ible ĠVolume - Pres erving ĠNeural ĠNet </s> For Ġthe Ġprojection Ġwe Ġcan Ġuse Ġan Ġarbitrary Ġin vert ible Ġfunction , Ġand Ġgiven Ġthe Ġrepresent ational Ġpower Ġof Ġneural Ġnetworks Ġthey Ġseem Ġa Ġnatural Ġchoice . ĠHowever , Ġcalculating Ġthe Ġinverse Ġand ĠJacob ian Ġof Ġan Ġarbitrary Ġneural Ġnetwork Ġcan Ġbe Ġdifficult , Ġas Ġit Ġrequires Ġthat Ġall Ġcomponent Ġfunctions Ġbe Ġin vert ible Ġand Ġalso Ġrequires Ġstorage Ġof Ġlarge ĠJacob ian Ġmat rices , Ġwhich Ġis Ġmemory Ġintensive . ĠTo Ġaddress Ġthis Ġissue , Ġseveral Ġrecent Ġpapers Ġpropose Ġspecially Ġdesigned Ġin vert ible Ġnetworks Ġthat Ġare Ġeasily Ġtrain able Ġyet Ġstill Ġpowerful ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 19 Ġ. ĠInspired Ġby Ġthese Ġworks , Ġwe Ġuse Ġthe Ġin vert ible Ġtransformation Ġproposed Ġby ĠB IB REF 16 Ġ, Ġwhich Ġconsists Ġof Ġa Ġseries Ġof ĠâĢ ľ c ou pling Ġlayers âĢ Ŀ . ĠThis Ġarchitecture Ġis Ġspecially Ġdesigned Ġto Ġguarantee Ġa Ġunit ĠJacob ian Ġdetermin ant Ġ( and Ġthus Ġthe Ġin vert ibility Ġproperty ). </s> From ĠE q . Ġ( ĠEQ REF 22 Ġ) Ġwe Ġknow Ġthat Ġonly ĠIN LINE FORM 0 Ġis Ġrequired Ġfor Ġaccompl ishing Ġlearning Ġand Ġinference ; Ġwe Ġnever Ġneed Ġto Ġexplicitly Ġconstruct ĠIN LINE FORM 1 Ġ. ĠThus , Ġwe Ġdirectly Ġdefine Ġthe Ġarchitecture Ġof ĠIN LINE FORM 2 Ġ. ĠAs Ġshown Ġin ĠFigure ĠFIG REF 24 Ġ, Ġthe Ġnon linear Ġtransformation Ġfrom Ġthe Ġobserved Ġembed ding ĠIN LINE FORM 3 Ġto ĠIN LINE FORM 4 Ġrepresents Ġthe Ġfirst Ġcoupling Ġlayer . ĠThe Ġinput Ġin Ġthis Ġlayer Ġis Ġpartition ed Ġinto Ġleft Ġand Ġright Ġhalves Ġof Ġdimensions , ĠIN LINE FORM 5 Ġand ĠIN LINE FORM 6 Ġ, Ġrespectively . ĠA Ġsingle Ġcoupling Ġlayer Ġis Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġcoupling Ġfunction Ġand Ġcan Ġbe Ġany Ġnon linear Ġform . ĠThis Ġtransformation Ġsatisfies ĠIN LINE FORM 1 Ġ, Ġand ĠB IB REF 16 Ġshow Ġthat Ġits ĠJacob ian Ġmatrix Ġis Ġtriangular Ġwith Ġall Ġones Ġon Ġthe Ġmain Ġdiagonal . ĠThus Ġthe ĠJacob ian Ġdetermin ant Ġis Ġalways Ġequal Ġto Ġone Ġ( i . e . Ġvolume - pres erving ) Ġand Ġthe Ġin vert ibility Ġcondition Ġis Ġnaturally Ġsatisfied . </s> To Ġbe Ġsufficiently Ġexpressive , Ġwe Ġcompose Ġmultiple Ġcoupling Ġlayers Ġas Ġsuggested Ġin ĠB IB REF 16 Ġ. ĠSpecifically , Ġwe Ġexchange Ġthe Ġrole Ġof Ġleft Ġand Ġright Ġhalf Ġvectors Ġat Ġeach Ġlayer Ġas Ġshown Ġin ĠFigure ĠFIG REF 24 Ġ. ĠFor Ġinstance , Ġfrom ĠIN LINE FORM 0 Ġto ĠIN LINE FORM 1 Ġthe Ġleft Ġsubset ĠIN LINE FORM 2 Ġis Ġunchanged , Ġwhile Ġfrom ĠIN LINE FORM 3 Ġto ĠIN LINE FORM 4 Ġthe Ġright Ġsubset ĠIN LINE FORM 5 Ġremains Ġthe Ġsame . ĠAlso Ġnote Ġthat Ġcomposing Ġmultiple Ġcoupling Ġlayers Ġdoes Ġnot Ġchange Ġthe Ġvolume - pres erving Ġand Ġin vert ibility Ġproperties . ĠSuch Ġa Ġsequence Ġof Ġin vert ible Ġtransformations Ġfrom Ġthe Ġdata Ġspace ĠIN LINE FORM 6 Ġto ĠIN LINE FORM 7 Ġis Ġalso Ġcalled Ġnormal izing Ġflow ĠB IB REF 20 Ġ. </s> Exper iments </s> In Ġthis Ġsection , Ġwe Ġfirst Ġdescribe Ġour Ġdatasets Ġand Ġexperimental Ġsetup . ĠWe Ġthen Ġinstant iate Ġour Ġapproach Ġwith ĠMark ov Ġand ĠDMV - struct ured Ġsyntax Ġmodels , Ġand Ġreport Ġresults Ġon ĠPOS Ġtagging Ġand Ġdependency Ġgrammar Ġinduction Ġrespectively . ĠLastly , Ġwe Ġanalyze Ġthe Ġlearned Ġlatent Ġembed d ings . </s> Data </s> For Ġboth ĠPOS Ġtagging Ġand Ġdependency Ġparsing , Ġwe Ġrun Ġexperiments Ġon Ġthe ĠWall ĠStreet ĠJournal Ġ( WS J ) Ġportion Ġof Ġthe ĠPenn ĠTree bank . ĠTo Ġcreate Ġthe Ġobserved Ġdata Ġembed d ings , Ġwe Ġtrain Ġskip - gram Ġword Ġembed d ings ĠB IB REF 7 Ġthat Ġare Ġfound Ġto Ġcapture Ġsynt actic Ġproperties Ġwell Ġwhen Ġtrained Ġwith Ġsmall Ġcontext Ġwindow ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ. ĠFollowing ĠB IB REF 9 Ġ, Ġthe Ġdimension ality ĠIN LINE FORM 0 Ġis Ġset Ġto Ġ100 , Ġand Ġthe Ġtraining Ġcontext Ġwindow Ġsize Ġis Ġset Ġto Ġ1 Ġto Ġencode Ġmore Ġsynt actic Ġinformation . ĠThe Ġskip - gram Ġembed d ings Ġare Ġtrained Ġon Ġthe Ġone Ġbillion Ġword Ġlanguage Ġmodeling Ġbenchmark Ġdataset ĠB IB REF 21 Ġin Ġaddition Ġto Ġthe ĠWS J Ġcorpus . </s> General ĠExperimental ĠSetup </s> For Ġthe Ġneural Ġprojector , Ġwe Ġemploy Ġrect ified Ġnetworks Ġas Ġcoupling Ġfunction ĠIN LINE FORM 0 Ġfollowing ĠB IB REF 16 Ġ. ĠWe Ġuse Ġa Ġrect ified Ġnetwork Ġwith Ġan Ġinput Ġlayer , Ġone Ġhidden Ġlayer , Ġand Ġlinear Ġoutput Ġunits , Ġthe Ġnumber Ġof Ġhidden Ġunits Ġis Ġset Ġto Ġthe Ġsame Ġas Ġthe Ġnumber Ġof Ġinput Ġunits . ĠThe Ġnumber Ġof Ġcoupling Ġlayers Ġare Ġvaried Ġas Ġ4 , Ġ8 , Ġ16 Ġfor Ġboth Ġtasks . ĠWe Ġoptimize Ġmarginal Ġdata Ġlikelihood Ġdirectly Ġusing ĠAdam ĠB IB REF 22 Ġ. ĠFor Ġboth Ġtasks Ġin Ġthe Ġfully Ġun super vised Ġsetting , Ġwe Ġdo Ġnot Ġtune Ġthe Ġhyper - param eters Ġusing Ġsupervised Ġdata . </s> Un super vised ĠPOS Ġtagging </s> For Ġun super vised ĠPOS Ġtagging , Ġwe Ġuse Ġa ĠMark ov - struct ured Ġsyntax Ġmodel Ġin Ġour Ġapproach , Ġwhich Ġis Ġa Ġpopular Ġstructure Ġfor Ġun super vised Ġtagging Ġtasks ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ. </s> Following Ġexisting Ġliterature , Ġwe Ġtrain Ġand Ġtest Ġon Ġthe Ġentire ĠWS J Ġcorpus Ġ( 49 208 Ġsentences , Ġ1 M Ġtokens ). ĠWe Ġuse Ġ45 Ġtag Ġclusters , Ġthe Ġnumber Ġof ĠPOS Ġtags Ġthat Ġappear Ġin ĠWS J Ġcorpus . ĠWe Ġtrain Ġthe Ġdiscrete ĠH MM Ġand Ġthe ĠGa ussian ĠH MM ĠB IB REF 9 Ġas Ġbas elines . ĠFor Ġthe ĠGa ussian ĠH MM , Ġmean Ġvectors Ġof ĠGa ussian Ġemissions Ġare Ġinitialized Ġwith Ġthe Ġempirical Ġmean Ġof Ġall Ġword Ġvectors Ġwith Ġan Ġadditive Ġnoise . ĠWe Ġassume Ġdiagonal Ġcovari ance Ġmatrix Ġfor ĠIN LINE FORM 0 Ġand Ġinitialize Ġit Ġwith Ġthe Ġempirical Ġvariance Ġof Ġthe Ġword Ġvectors . ĠFollowing ĠB IB REF 9 Ġ, Ġthe Ġcovari ance Ġmatrix Ġis Ġfixed Ġduring Ġtraining . ĠThe Ġmult in omial Ġprobabilities Ġare Ġinitialized Ġas ĠIN LINE FORM 1 Ġ, Ġwhere ĠIN LINE FORM 2 Ġ. ĠFor Ġour Ġapproach , Ġwe Ġinitialize Ġthe Ġsyntax Ġmodel Ġand ĠGa ussian Ġparameters Ġwith Ġthe Ġpre - trained ĠGa ussian ĠH MM . ĠThe Ġweights Ġof Ġlayers Ġin Ġthe Ġrect ified Ġnetwork Ġare Ġinitialized Ġfrom Ġa Ġuniform Ġdistribution Ġwith Ġmean Ġzero Ġand Ġa Ġstandard Ġdeviation Ġof ĠIN LINE FORM 3 Ġ, Ġwhere ĠIN LINE FORM 4 Ġis Ġthe Ġinput Ġdimension . ĠWe Ġevaluate Ġthe Ġperformance Ġof ĠPOS Ġtagging Ġwith Ġboth ĠMany - to - One Ġ( M - 1 ) Ġaccuracy ĠB IB REF 23 Ġand ĠV - Measure Ġ( VM ) ĠB IB REF 24 Ġ. ĠGiven Ġa Ġmodel Ġwe Ġfound Ġthat Ġthe Ġtagging Ġperformance Ġis Ġwell - cor related Ġwith Ġthe Ġtraining Ġdata Ġlikelihood , Ġthus Ġwe Ġuse Ġtraining Ġdata Ġlikelihood Ġas Ġa Ġun super vised Ġcriterion Ġto Ġselect Ġthe Ġtrained Ġmodel Ġover Ġ10 Ġrandom Ġrest arts Ġafter Ġtraining Ġ50 Ġepoch s . ĠWe Ġrepeat Ġthis Ġprocess Ġ5 Ġtimes Ġand Ġreport Ġthe Ġmean Ġand Ġstandard Ġdeviation Ġof Ġperformance . </s> We Ġcompare Ġour Ġapproach Ġwith Ġbasic ĠH MM , ĠGa ussian ĠH MM , Ġand Ġseveral Ġstate - of - the - art Ġsystems , Ġincluding Ġsophisticated ĠH MM Ġvariants Ġand Ġclust ering Ġtechniques Ġwith Ġhand - engine ered Ġfeatures . ĠThe Ġresults Ġare Ġpresented Ġin ĠTable ĠT AB REF 32 Ġ. ĠThrough Ġthe Ġintroduced Ġlatent Ġembed d ings Ġand Ġadditional Ġneural Ġprojection , Ġour Ġapproach Ġimproves Ġover Ġthe ĠGa ussian ĠH MM Ġby Ġ5 . 4 Ġpoints Ġin ĠM - 1 Ġand Ġ5 . 6 Ġpoints Ġin ĠVM . ĠNeural ĠH MM Ġ( NH MM ) ĠB IB REF 10 Ġis Ġa Ġbaseline Ġthat Ġalso Ġlearns Ġword Ġrepresentation Ġjointly . ĠBoth Ġtheir Ġbasic Ġmodel Ġand Ġextended ĠConv Ġversion Ġdoes Ġnot Ġoutper form Ġthe ĠGa ussian ĠH MM . ĠTheir Ġbest Ġmodel Ġincorporates Ġanother ĠL ST M Ġto Ġmodel Ġlong Ġdistance Ġdependency Ġand Ġbreaks Ġthe ĠMark ov Ġassumption , Ġyet Ġour Ġapproach Ġstill Ġachieves Ġsubstantial Ġimprovement Ġover Ġit Ġwithout Ġconsidering Ġmore Ġcontext Ġinformation . ĠMoreover , Ġour Ġmethod Ġoutper forms Ġthe Ġbest Ġpublished Ġresult Ġthat Ġbenefits Ġfrom Ġhand - engine ered Ġfeatures ĠB IB REF 27 Ġby Ġ2 . 0 Ġpoints Ġon ĠVM . </s> We Ġfound Ġthat Ġmost Ġtagging Ġerrors Ġhappen Ġin Ġnoun Ġsub c ategories . ĠTherefore , Ġwe Ġdo Ġthe Ġone - to - one Ġmapping Ġbetween Ġgold ĠPOS Ġtags Ġand Ġinduced Ġclusters Ġand Ġplot Ġthe Ġnormalized Ġconfusion Ġmatrix Ġof Ġnoun Ġsub c ategories Ġin ĠFigure ĠFIG REF 35 Ġ. ĠThe ĠGa ussian ĠH MM Ġfails Ġto Ġidentify ĠâĢ ľ NN âĢ Ŀ Ġand ĠâĢ ľ NN S âĢ Ŀ Ġcorrectly Ġfor Ġmost Ġcases , Ġand Ġit Ġoften Ġrecognizes ĠâĢ ľ NN PS âĢ Ŀ Ġas ĠâĢ ľ NN P âĢ Ŀ . ĠIn Ġcontrast , Ġour Ġapproach Ġcorrect s Ġthese Ġerrors Ġwell . </s> Un super vised ĠDepend ency ĠPars ing Ġwithout Ġgold ĠPOS Ġtags </s> For Ġthe Ġtask Ġof Ġun super vised Ġdependency Ġparse Ġinduction , Ġwe Ġemploy Ġthe ĠDepend ency ĠModel Ġwith ĠVal ence Ġ( DM V ) ĠB IB REF 2 Ġas Ġthe Ġsyntax Ġmodel Ġin Ġour Ġapproach . ĠDMV Ġis Ġa Ġgener ative Ġmodel Ġthat Ġdefines Ġa Ġprobability Ġdistribution Ġover Ġdependency Ġparse Ġtrees Ġand Ġsynt actic Ġcategories , Ġgenerating Ġtokens Ġand Ġdependencies Ġin Ġa Ġhead - out ward Ġfashion . ĠWhile , Ġtraditionally , ĠDMV Ġis Ġtrained Ġusing Ġgold ĠPOS Ġtags Ġas Ġobserved Ġsynt actic Ġcategories , Ġin Ġour Ġapproach , Ġwe Ġtreat Ġeach Ġtag Ġas Ġa Ġlatent Ġvariable , Ġas Ġdescribed Ġin Ġsec : general - ne ural . </s> Most Ġexisting Ġapproaches Ġto Ġthis Ġtask Ġare Ġnot Ġfully Ġun super vised Ġsince Ġthey Ġrely Ġon Ġgold ĠPOS Ġtags Ġfollowing Ġthe Ġoriginal Ġexperimental Ġsetup Ġfor ĠDMV . ĠThis Ġis Ġpartially Ġbecause Ġautomatically Ġparsing Ġfrom Ġwords Ġis Ġdifficult Ġeven Ġwhen Ġusing Ġun super vised Ġsynt actic Ġcategories ĠB IB REF 29 Ġ. ĠHowever , Ġinducing Ġdependencies Ġfrom Ġwords Ġalone Ġrepresents Ġa Ġmore Ġrealistic Ġexperimental Ġcondition Ġsince Ġgold ĠPOS Ġtags Ġare Ġoften Ġunavailable Ġin Ġpractice . ĠPrevious Ġwork Ġthat Ġhas Ġtrained Ġfrom Ġwords Ġalone Ġoften Ġrequires Ġadditional Ġlinguistic Ġconstraints Ġ( like Ġsentence Ġinternal Ġboundaries ) ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ, ĠB IB REF 32 Ġ, Ġacoustic Ġcues ĠB IB REF 33 Ġ, Ġadditional Ġtraining Ġdata ĠB IB REF 4 Ġ, Ġor Ġannot ated Ġdata Ġfrom Ġrelated Ġlanguages ĠB IB REF 34 Ġ. ĠOur Ġapproach Ġis Ġnaturally Ġdesigned Ġto Ġtrain Ġon Ġword Ġembed d ings Ġdirectly , Ġthus Ġwe Ġattempt Ġto Ġinduce Ġdependencies Ġwithout Ġusing Ġgold ĠPOS Ġtags Ġor Ġother Ġextra Ġlinguistic Ġinformation . </s> Like Ġprevious Ġwork Ġwe Ġuse Ġsections Ġ02 - 21 Ġof ĠWS J Ġcorpus Ġas Ġtraining Ġdata Ġand Ġevaluate Ġon Ġsection Ġ23 , Ġwe Ġremove Ġpunct uations Ġand Ġtrain Ġthe Ġmodels Ġon Ġsentences Ġof Ġlength ĠIN LINE FORM 0 Ġ, ĠâĢ ľ head - per col ation âĢ Ŀ Ġrules ĠB IB REF 39 Ġare Ġapplied Ġto Ġobtain Ġgold Ġdependencies Ġfor Ġevaluation . ĠWe Ġtrain Ġbasic ĠDMV , Ġextended ĠDMV Ġ( E - DM V ) ĠB IB REF 35 Ġand ĠGa ussian ĠDMV Ġ( which Ġtreats ĠPOS Ġtag Ġas Ġunknown Ġlatent Ġvariables Ġand Ġgenerates Ġobserved Ġword Ġembed d ings Ġdirectly Ġconditioned Ġon Ġthem Ġfollowing ĠGa ussian Ġdistribution ) Ġas Ġbas elines . ĠBasic ĠDMV Ġand ĠE - DM V Ġare Ġtrained Ġwith ĠV iter bi ĠEM ĠB IB REF 40 Ġon Ġun super vised ĠPOS Ġtags Ġinduced Ġfrom Ġour ĠMark ov - struct ured Ġmodel Ġdescribed Ġin Ġsec : pos . ĠMult in omial Ġparameters Ġof Ġthe Ġsyntax Ġmodel Ġin Ġboth ĠGa ussian ĠDMV Ġand Ġour Ġmodel Ġare Ġinitialized Ġwith Ġthe Ġpre - trained ĠDMV Ġbaseline . ĠOther Ġparameters Ġare Ġinitialized Ġin Ġthe Ġsame Ġway Ġas Ġin Ġthe ĠPOS Ġtagging Ġexperiment . ĠThe Ġdirected Ġdependency Ġaccuracy Ġ( D DA ) Ġis Ġused Ġfor Ġevaluation Ġand Ġwe Ġreport Ġaccuracy Ġon Ġsentences Ġof Ġlength ĠIN LINE FORM 1 Ġand Ġall Ġlengths . ĠWe Ġtrain Ġthe Ġparser Ġuntil Ġtraining Ġdata Ġlikelihood Ġconver ges , Ġand Ġreport Ġthe Ġmean Ġand Ġstandard Ġdeviation Ġover Ġ20 Ġrandom Ġrest arts . </s> Our Ġmodel Ġdirectly Ġobserves Ġword Ġembed d ings Ġand Ġdoes Ġnot Ġrequire Ġgold ĠPOS Ġtags Ġduring Ġtraining . ĠThus , Ġresults Ġfrom Ġrelated Ġwork Ġtrained Ġon Ġgold Ġtags Ġare Ġnot Ġdirectly Ġcomparable . ĠHowever , Ġto Ġmeasure Ġhow Ġthese Ġsystems Ġmight Ġperform Ġwithout Ġgold Ġtags , Ġwe Ġrun Ġthree Ġrecent Ġstate - of - the - art Ġsystems Ġin Ġour Ġexperimental Ġsetting : ĠUR - A ĠE - DM V ĠB IB REF 36 Ġ, ĠNeural ĠE - DM V ĠB IB REF 11 Ġ, Ġand ĠCR F ĠAut oen c oder Ġ( CR FA E ) ĠB IB REF 37 Ġ. ĠWe Ġuse Ġun super vised ĠPOS Ġtags Ġ( induced Ġfrom Ġour ĠMark ov - struct ured Ġmodel ) Ġin Ġplace Ġof Ġgold Ġtags . ĠWe Ġalso Ġtrain Ġbasic ĠDMV Ġon Ġgold Ġtags Ġand Ġinclude Ġseveral Ġstate - of - the - art Ġresults Ġon Ġgold Ġtags Ġas Ġreference Ġpoints . </s> As Ġshown Ġin ĠTable ĠT AB REF 39 Ġ, Ġour Ġapproach Ġis Ġable Ġto Ġimprove Ġover Ġthe ĠGa ussian ĠDMV Ġby Ġ4 . 8 Ġpoints Ġon Ġlength ĠIN LINE FORM 0 Ġand Ġ4 . 8 Ġpoints Ġon Ġall Ġlengths , Ġwhich Ġsuggests Ġthe Ġadditional Ġlatent Ġembed ding Ġlayer Ġand Ġneural Ġprojector Ġare Ġhelpful . ĠThe Ġproposed Ġapproach Ġyields , Ġto Ġthe Ġbest Ġof Ġour Ġknowledge , Ġstate - of - the - art Ġperformance Ġwithout Ġgold ĠPOS Ġannotation Ġand Ġwithout Ġsentence - internal Ġboundary Ġinformation . ĠDMV , ĠUR - A ĠE - DM V , ĠNeural ĠE - DM V , Ġand ĠCR FA E Ġsuffer Ġa Ġlarge Ġdecrease Ġin Ġperformance Ġwhen Ġtrained Ġon Ġun super vised Ġtags ĠâĢĵ Ġan Ġeffect Ġalso Ġseen Ġin Ġprevious Ġwork ĠB IB REF 29 Ġ, ĠB IB REF 34 Ġ. ĠSince Ġour Ġapproach Ġinduces Ġlatent ĠPOS Ġtags Ġjointly Ġwith Ġdependency Ġtrees , Ġit Ġmay Ġbe Ġable Ġto Ġlearn ĠPOS Ġclusters Ġthat Ġare Ġmore Ġam enable Ġto Ġgrammar Ġinduction Ġthan Ġthe Ġun super vised Ġtags . ĠWe Ġobserve Ġthat ĠCR FA E Ġunder per forms Ġits Ġgold - tag Ġcounterpart Ġsubstantially . ĠThis Ġmay Ġlargely Ġbe Ġa Ġresult Ġof Ġthe Ġmodel 's Ġreliance Ġon Ġprior Ġlinguistic Ġrules Ġthat Ġbecome Ġunavailable Ġwhen Ġgold ĠPOS Ġtag Ġtypes Ġare Ġunknown . ĠMany Ġextensions Ġto ĠDMV Ġcan Ġbe Ġconsidered Ġorth og onal Ġto Ġour Ġapproach ĠâĢĵ Ġthey Ġessentially Ġfocus Ġon Ġimproving Ġthe Ġsyntax Ġmodel . ĠIt Ġis Ġpossible Ġthat Ġincorporating Ġthese Ġmore Ġsophisticated Ġsyntax Ġmodels Ġinto Ġour Ġapproach Ġmay Ġlead Ġto Ġfurther Ġimprovements . </s> S ensitivity ĠAnalysis </s> In Ġthe Ġabove Ġexperiments Ġwe Ġinitialize Ġthe Ġstructured Ġsyntax Ġcomponents Ġwith Ġthe Ġpre - trained ĠGa ussian Ġor Ġdiscrete Ġbaseline , Ġwhich Ġis Ġshown Ġas Ġa Ġuseful Ġtechnique Ġto Ġhelp Ġtrain Ġour Ġdeep Ġmodels . ĠWe Ġfurther Ġstudy Ġthe Ġresults Ġwith Ġfully Ġrandom Ġinitialization . ĠIn Ġthe ĠPOS Ġtagging Ġexperiment , Ġwe Ġreport Ġthe Ġresults Ġin ĠTable ĠT AB REF 48 Ġ. ĠWhile Ġthe Ġperformance Ġwith Ġ4 Ġlayers Ġis Ġcomparable Ġto Ġthe Ġpre - trained ĠGa ussian Ġinitialization , Ġdeeper Ġprojections Ġ( 8 Ġor Ġ16 Ġlayers ) Ġresult Ġin Ġa Ġdramatic Ġdrop Ġin Ġperformance . ĠThis Ġsuggests Ġthat Ġthe Ġstructured Ġsyntax Ġmodel Ġwith Ġvery Ġdeep Ġprojections Ġis Ġdifficult Ġto Ġtrain Ġfrom Ġscratch , Ġand Ġa Ġsimpler Ġprojection Ġmight Ġbe Ġa Ġgood Ġcompromise Ġin Ġthe Ġrandom Ġinitialization Ġsetting . </s> Different Ġfrom Ġthe ĠMark ov Ġprior Ġin ĠPOS Ġtagging Ġexperiments , Ġour Ġparsing Ġmodel Ġseems Ġto Ġbe Ġquite Ġsensitive Ġto Ġthe Ġinitialization . ĠFor Ġexample , Ġdirected Ġaccuracy Ġof Ġour Ġapproach Ġon Ġsentences Ġof Ġlength ĠIN LINE FORM 0 Ġis Ġbelow Ġ40 . 0 Ġwith Ġrandom Ġinitialization . ĠThis Ġis Ġconsistent Ġwith Ġprevious Ġwork Ġthat Ġhas Ġnoted Ġthe Ġimportance Ġof Ġcareful Ġinitialization Ġfor ĠDMV - based Ġmodels Ġsuch Ġas Ġthe Ġcommonly Ġused Ġharmonic Ġinitial izer ĠB IB REF 2 Ġ. ĠHowever , Ġit Ġis Ġnot Ġstraightforward Ġto Ġapply Ġthe Ġharmonic Ġinitial izer Ġfor ĠDMV Ġdirectly Ġin Ġour Ġmodel Ġwithout Ġusing Ġsome Ġkind Ġof Ġpre - training Ġsince Ġwe Ġdo Ġnot Ġobserve Ġgold ĠPOS . </s> We Ġinvestigate Ġthe Ġeffect Ġof Ġthe Ġchoice Ġof Ġpre - trained Ġembed ding Ġon Ġperformance Ġwhile Ġusing Ġour Ġapproach . ĠTo Ġthis Ġend , Ġwe Ġadditionally Ġinclude Ġresults Ġusing Ġfast Text Ġembed d ings ĠB IB REF 41 ĠâĢĵ Ġwhich , Ġin Ġcontrast Ġwith Ġskip - gram Ġembed d ings , Ġinclude Ġcharacter - level Ġinformation . ĠWe Ġset Ġthe Ġcontext Ġwindows Ġsize Ġto Ġ1 Ġand Ġthe Ġdimension Ġsize Ġto Ġ100 Ġas Ġin Ġthe Ġskip - gram Ġtraining , Ġwhile Ġkeeping Ġother Ġparameters Ġset Ġto Ġtheir Ġdefaults . ĠThese Ġresults Ġare Ġsummarized Ġin ĠTable ĠT AB REF 50 Ġand ĠTable ĠT AB REF 51 Ġ. ĠWhile Ġfast Text Ġembed d ings Ġlead Ġto Ġreduced Ġperformance Ġwith Ġour Ġmodel , Ġour Ġapproach Ġstill Ġyields Ġan Ġimprovement Ġover Ġthe ĠGa ussian Ġbaseline Ġwith Ġthe Ġnew Ġobserved Ġembed d ings Ġspace . </s> Qual itative ĠAnalysis Ġof ĠEmb edd ings </s> We Ġperform Ġqualitative Ġanalysis Ġto Ġunderstand Ġhow Ġthe Ġlatent Ġembed d ings Ġhelp Ġinduce Ġsynt actic Ġstructures . ĠFirst Ġwe Ġfilter Ġout Ġlow - frequency Ġwords Ġand Ġpunct uations Ġin ĠWS J , Ġand Ġvisualize Ġthe Ġrest Ġwords Ġ( 10 k ) Ġwith Ġt - S NE ĠB IB REF 42 Ġunder Ġdifferent Ġembed d ings . ĠWe Ġassign Ġeach Ġword Ġwith Ġits Ġmost Ġlikely Ġgold ĠPOS Ġtags Ġin ĠWS J Ġand Ġcolor Ġthem Ġaccording Ġto Ġthe Ġgold ĠPOS Ġtags . </s> For Ġour ĠMark ov - struct ured Ġmodel , Ġwe Ġhave Ġdisplayed Ġthe Ġembed ding Ġspace Ġin ĠFigure ĠSEC REF 5 Ġ, Ġwhere Ġthe Ġgold ĠPOS Ġclusters Ġare Ġwell - formed . ĠFurther , Ġwe Ġpresent Ġfive Ġexample Ġtarget Ġwords Ġand Ġtheir Ġfive Ġnearest Ġneighbors Ġin Ġterms Ġof Ġcos ine Ġsimilarity . ĠAs Ġshown Ġin ĠTable ĠT AB REF 53 Ġ, Ġthe Ġskip - gram Ġembed ding Ġcaptures Ġboth Ġsemantic Ġand Ġsynt actic Ġaspects Ġto Ġsome Ġdegree , Ġyet Ġour Ġembed d ings Ġare Ġable Ġto Ġfocus Ġespecially Ġon Ġthe Ġsynt actic Ġaspects Ġof Ġwords , Ġin Ġan Ġun super vised Ġfashion Ġwithout Ġusing Ġany Ġextra Ġmorph ological Ġinformation . </s> In ĠFigure ĠFIG REF 54 Ġwe Ġdepict Ġthe Ġlearned Ġlatent Ġembed d ings Ġwith Ġthe ĠDMV - struct ured Ġsyntax Ġmodel . ĠUnlike Ġthe ĠMark ov Ġstructure , Ġthe ĠDMV Ġstructure Ġmaps Ġa Ġlarge Ġsubset Ġof Ġsingular Ġand Ġplural Ġnoun s Ġto Ġthe Ġsame Ġoverlapping Ġregion . ĠHowever , Ġtwo Ġclusters Ġof Ġsingular Ġand Ġplural Ġnoun s Ġare Ġactually Ġseparated . ĠWe Ġinspect Ġthe Ġtwo Ġclusters Ġand Ġthe Ġoverlapping Ġregion Ġin ĠFigure ĠFIG REF 54 Ġ, Ġit Ġturns Ġout Ġthat Ġthe Ġnoun s Ġin Ġthe Ġseparated Ġclusters Ġare Ġwords Ġthat Ġcan Ġappear Ġas Ġsubjects Ġand , Ġtherefore , Ġfor Ġwhich Ġverb Ġagreement Ġis Ġimportant Ġto Ġmodel . ĠIn Ġcontrast , Ġthe Ġnoun s Ġin Ġthe Ġoverlapping Ġregion Ġare Ġtypically Ġobjects . ĠThis Ġdemonstrates Ġthat Ġthe Ġlatent Ġembed d ings Ġare Ġfocusing Ġon Ġaspects Ġof Ġlanguage Ġthat Ġare Ġspecifically Ġimportant Ġfor Ġmodeling Ġdependency Ġwithout Ġever Ġhaving Ġseen Ġexamples Ġof Ġdependency Ġpars es . ĠSome Ġprevious Ġwork Ġhas Ġdeliberately Ġcreated Ġembed d ings Ġto Ġcapture Ġdifferent Ġnotions Ġof Ġsimilarity ĠB IB REF 43 Ġ, ĠB IB REF 44 Ġ, Ġwhile Ġthey Ġuse Ġextra Ġmorphology Ġor Ġdependency Ġannotations Ġto Ġguide Ġthe Ġembed ding Ġlearning , Ġour Ġapproach Ġprovides Ġa Ġpotential Ġalternative Ġto Ġcreate Ġnew Ġembed d ings Ġthat Ġare Ġguided Ġby Ġstructured Ġsyntax Ġmodel , Ġonly Ġusing Ġunl abel ed Ġtext Ġcorpor a . </s> Related ĠWork </s> Our Ġapproach Ġis Ġrelated Ġto Ġflow - based Ġgener ative Ġmodels , Ġwhich Ġare Ġfirst Ġdescribed Ġin ĠN ICE ĠB IB REF 16 Ġand Ġhave Ġrecently Ġreceived Ġmore Ġattention ĠB IB REF 17 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 18 Ġ. ĠThis Ġrelevant Ġwork Ġmostly Ġadop ts Ġsimple Ġ( e . g . ĠGa ussian ) Ġand Ġfixed Ġpri ors Ġand Ġdoes Ġnot Ġattempt Ġto Ġlearn Ġinterpret able Ġlatent Ġstructures . ĠAnother Ġrelated Ġgener ative Ġmodel Ġclass Ġis Ġvari ational Ġauto - enc od ers Ġ( VA Es ) ĠB IB REF 45 Ġthat Ġoptimize Ġa Ġlower Ġbound Ġon Ġthe Ġmarginal Ġdata Ġlikelihood , Ġand Ġcan Ġbe Ġextended Ġto Ġlearn Ġlatent Ġstructures ĠB IB REF 46 Ġ, ĠB IB REF 47 Ġ. ĠAgainst Ġthe Ġflow - based Ġmodels , ĠVA Es Ġremove Ġthe Ġin vert ibility Ġconstraint Ġbut Ġsacrifice Ġthe Ġmerits Ġof Ġexact Ġinference Ġand Ġexact Ġlog Ġlikelihood Ġcomputation , Ġwhich Ġpotentially Ġresults Ġin Ġoptimization Ġchallenges ĠB IB REF 48 Ġ. ĠOur Ġapproach Ġcan Ġalso Ġbe Ġviewed Ġin Ġconnection Ġwith Ġgener ative Ġadvers arial Ġnetworks Ġ( GAN s ) ĠB IB REF 49 Ġthat Ġis Ġa Ġlikelihood - free Ġframework Ġto Ġlearn Ġimplicit Ġgener ative Ġmodels . ĠHowever , Ġit Ġis Ġnon - t riv ial Ġfor Ġa Ġgradient - based Ġmethod Ġlike ĠG AN s Ġto Ġpropagate Ġgrad ients Ġthrough Ġdiscrete Ġstructures . </s> Conclusion </s> In Ġthis Ġwork , Ġwe Ġdefine Ġa Ġnovel Ġgener ative Ġapproach Ġto Ġleverage Ġcontinuous Ġword Ġrepresentations Ġfor Ġun super vised Ġlearning Ġof Ġsynt actic Ġstructure . ĠExper iments Ġon Ġboth ĠPOS Ġinduction Ġand Ġun super vised Ġdependency Ġparsing Ġtasks Ġdemonstrate Ġthe Ġeffectiveness Ġof Ġour Ġproposed Ġapproach . ĠFuture Ġwork Ġmight Ġexplore Ġmore Ġsophisticated Ġin vert ible Ġprojections , Ġor Ġrecurrent Ġprojections Ġthat Ġjointly Ġtransform Ġthe Ġentire Ġinput Ġsequence .