On each dataset, we compare four approaches based on sefe with two efe BIBREF10 baselines. All are fit using sgd BIBREF34 . In particular, we compare the following methods:
EXTRACTIVE
Calculate test log-likelihood on the three considered datasets
ABSTRACTIVE
In this section, we describe the experimental study. We fit the sefe model on three datasets and compare it against the efe BIBREF10 . Our quantitative results show that sharing the context vectors provides better results, and that amortization and hierarchical structure give further improvements. Data. We apply the sefe on three datasets: ArXiv papers, U.S. Senate speeches, and purchases on supermarket grocery shopping data. We describe these datasets below, and we provide a summary of the datasets in Table TABREF17 . ArXiv papers: This dataset contains the abstracts of papers published on the ArXiv under the 19 different tags between April 2007 and June 2015. We treat each tag as a group and fit sefe with the goal of uncovering which words have the strongest shift in usage. We split the abstracts into training, validation, and test sets, with proportions of INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , respectively. Senate speeches: This dataset contains U.S. Senate speeches from 1994 to mid 2009. In contrast to the ArXiv collection, it is a transcript of spoken language. We group the data into state of origin of the speaker and his or her party affiliation. Only affiliations with the Republican and Democratic Party are considered. As a result, there are 83 groups (Republicans from Alabama, Democrats from Alabama, Republicans from Arkansas, etc.). Some of the state/party combinations are not available in the data, as some of the 50 states have only had Senators with the same party affiliation. We split the speeches into training ( INLINEFORM0 ), validation ( INLINEFORM1 ), and testing ( INLINEFORM2 ). Grocery shopping data: This dataset contains the purchases of INLINEFORM0 customers. The data covers a period of 97 weeks. After removing low-frequency items, the data contains INLINEFORM1 unique items at the 1.10upc (Universal Product Code) level. We split the data into a training, test, and validation sets, with proportions of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 , respectively. The training data contains INLINEFORM5 shopping trips and INLINEFORM6 purchases in total. For the text corpora, we fix the vocabulary to the 15k most frequent terms and remove all words that are not in the vocabulary. Following BIBREF2 , we additionally remove each word with probability INLINEFORM0 , where INLINEFORM1 is the word frequency. This downsamples especially the frequent words and speeds up training. (Sizes reported in Table TABREF17 are the number of words remaining after preprocessing.) Models. Our goal is to fit the sefe model on these datasets. For the text data, we use the Bernoulli distribution as the conditional exponential family, while for the shopping data we use the Poisson distribution, which is more appropriate for count data. On each dataset, we compare four approaches based on sefe with two efe BIBREF10 baselines. All are fit using sgd BIBREF34 . In particular, we compare the following methods:
Our contributions are thus as follows. We introduce the sefe model, extending efe to grouped data. We present two techniques to share statistical strength among the embedding vectors, one based on hierarchical modeling and one based on amortization. We carry out a thorough experimental study on two text databases, ArXiv papers by section and U.S. Congressional speeches by home state and political party. Using Poisson embeddings, we study market basket data from a large grocery store, grouped by season. On all three data sets, sefe outperforms efe in terms of held-out log-likelihood. Qualitatively, we demonstrate how sefe discovers which words are used most differently across U.S. states and political parties, and show how word usage changes in different ArXiv disciplines.
1709.10367
Introduction </s> Word Ġembed d ings ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġare Ġun super vised Ġlearning Ġmethods Ġfor Ġcapturing Ġlatent Ġsemantic Ġstructure Ġin Ġlanguage . ĠWord Ġembed ding Ġmethods Ġanalyze Ġtext Ġdata Ġto Ġlearn Ġdistributed Ġrepresentations Ġof Ġthe Ġvocabulary Ġthat Ġcapture Ġits Ġco - occ urrence Ġstatistics . ĠThese Ġrepresentations Ġare Ġuseful Ġfor Ġreasoning Ġabout Ġword Ġusage Ġand Ġmeaning ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠWord Ġembed d ings Ġhave Ġalso Ġbeen Ġextended Ġto Ġdata Ġbeyond Ġtext ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, Ġsuch Ġas Ġitems Ġin Ġa Ġgrocery Ġstore Ġor Ġneurons Ġin Ġthe Ġbrain . Ġe fe Ġis Ġa Ġprob abil istic Ġperspective Ġon Ġembed d ings Ġthat Ġencompasses Ġmany Ġexisting Ġmethods Ġand Ġopens Ġthe Ġdoor Ġto Ġbringing Ġexpressive Ġprob abil istic Ġmodeling ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġto Ġthe Ġproblem Ġof Ġlearning Ġdistributed Ġrepresentations . </s> We Ġdevelop Ġse fe , Ġan Ġextension Ġof Ġe fe Ġfor Ġstudying Ġhow Ġembed d ings Ġcan Ġvary Ġacross Ġgroups Ġof Ġrelated Ġdata . ĠWe Ġwill Ġstudy Ġseveral Ġexamples : Ġin ĠU . S . ĠCongressional Ġspeeches , Ġword Ġusage Ġcan Ġvary Ġacross Ġstates Ġor Ġparty Ġaffili ations ; Ġin Ġscientific Ġliterature , Ġthe Ġusage Ġpatterns Ġof Ġtechnical Ġterms Ġcan Ġvary Ġacross Ġfields ; Ġin Ġsupermarket Ġshopping Ġdata , Ġco - p urchase Ġpatterns Ġof Ġitems Ġcan Ġvary Ġacross Ġseasons Ġof Ġthe Ġyear . ĠWe Ġwill Ġsee Ġthat Ġse fe Ġdiscovers Ġa Ġper - group Ġembed ding Ġrepresentation Ġof Ġobjects . ĠWhile Ġthe ĠnaÃ¯ve Ġapproach Ġof Ġfitting Ġan Ġindividual Ġembed ding Ġmodel Ġfor Ġeach Ġgroup Ġwould Ġtypically Ġsuffer Ġfrom Ġlack Ġof Ġdata âĢĶ especially Ġin Ġgroups Ġfor Ġwhich Ġfewer Ġobservations Ġare Ġavailable âĢĶ we Ġdevelop Ġtwo Ġmethods Ġthat Ġcan Ġshare Ġinformation Ġacross Ġgroups . </s> Figure ĠFIG REF 1 Ġillustrates Ġthe Ġkind Ġof Ġvariation Ġthat Ġwe Ġcan Ġcapture . ĠWe Ġfit Ġan Ġse fe Ġto ĠAr X iv Ġabstract s Ġgrouped Ġinto Ġdifferent Ġsections , Ġsuch Ġas Ġcomputer Ġscience Ġ( cs ), Ġquantitative Ġfinance Ġ( q - fin ), Ġand Ġnon linear Ġsciences Ġ( n lin ). Ġse fe Ġresults Ġin Ġa Ġper - section Ġembed ding Ġof Ġeach Ġterm Ġin Ġthe Ġvocabulary . ĠUsing Ġthe Ġfitted Ġembed d ings , Ġwe Ġillustrate Ġsimilar Ġwords Ġto Ġthe Ġword Ġ1 . 10 intelligence . ĠWe Ġcan Ġsee Ġthat Ġhow Ġ1 . 10 intelligence Ġis Ġused Ġvaries Ġby Ġfield : Ġin Ġcomputer Ġscience Ġthe Ġmost Ġsimilar Ġwords Ġinclude Ġ1 . 10 art ificial Ġand Ġ1 . 10 ai ; Ġin Ġfinance , Ġsimilar Ġwords Ġinclude Ġ1 . 10 abilities Ġand Ġ1 . 10 conscious ness . </s> In Ġmore Ġdetail , Ġembed ding Ġmethods Ġposit Ġtwo Ġrepresentation Ġvectors Ġfor Ġeach Ġterm Ġin Ġthe Ġvocabulary ; Ġan Ġembed ding Ġvector Ġand Ġa Ġcontext Ġvector . Ġ( We Ġuse Ġthe Ġlanguage Ġof Ġtext Ġfor Ġconc ret eness ; Ġas Ġwe Ġmentioned , Ġe fe Ġextend Ġto Ġother Ġtypes Ġof Ġdata .) ĠThe Ġidea Ġis Ġthat Ġthe Ġconditional Ġprobability Ġof Ġeach Ġobserved Ġword Ġdepends Ġon Ġthe Ġinteraction Ġbetween Ġthe Ġembed ding Ġvector Ġand Ġthe Ġcontext Ġvectors Ġof Ġthe Ġsurrounding Ġwords . ĠIn Ġse fe , Ġwe Ġposit Ġa Ġseparate Ġset Ġof Ġembed ding Ġvectors Ġfor Ġeach Ġgroup Ġbut Ġa Ġshared Ġset Ġof Ġcontext Ġvectors ; Ġthis Ġensures Ġthat Ġthe Ġembed ding Ġvectors Ġare Ġin Ġthe Ġsame Ġspace . </s> We Ġpropose Ġtwo Ġmethods Ġto Ġshare Ġstatistical Ġstrength Ġamong Ġthe Ġembed ding Ġvectors . ĠThe Ġfirst Ġapproach Ġis Ġbased Ġon Ġhierarchical Ġmodeling ĠB IB REF 13 Ġ, Ġwhich Ġassumes Ġthat Ġthe Ġgroup - specific Ġembed ding Ġrepresentations Ġare Ġtied Ġthrough Ġa Ġglobal Ġembed ding . ĠThe Ġsecond Ġapproach Ġis Ġbased Ġon Ġam ort ization ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, Ġwhich Ġconsiders Ġthat Ġthe Ġindividual Ġembed d ings Ġare Ġthe Ġoutput Ġof Ġa Ġdeter ministic Ġfunction Ġof Ġa Ġglobal Ġembed ding Ġrepresentation . ĠWe Ġuse Ġsto ch astic Ġoptimization Ġto Ġfit Ġlarge Ġdata Ġsets . </s> Our Ġwork Ġrelates Ġclosely Ġto Ġtwo Ġthreads Ġof Ġresearch Ġin Ġthe Ġembed ding Ġliterature . ĠOne Ġis Ġembed ding Ġmethods Ġthat Ġstudy Ġhow Ġlanguage Ġevolves Ġover Ġtime ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. ĠTime Ġcan Ġbe Ġthought Ġof Ġas Ġa Ġtype Ġof ĠâĢ ľ group âĢ Ŀ , Ġthough Ġwith Ġevolutionary Ġstructure Ġthat Ġwe Ġdo Ġnot Ġconsider . ĠThe Ġsecond Ġthread Ġis Ġmult ilingual Ġembed d ings ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ; Ġour Ġapproach Ġis Ġdifferent Ġin Ġthat Ġmost Ġwords Ġappear Ġin Ġall Ġgroups Ġand Ġwe Ġare Ġinterested Ġin Ġthe Ġvariations Ġof Ġthe Ġembed d ings Ġacross Ġthose Ġgroups . </s> Our Ġcontributions Ġare Ġthus Ġas Ġfollows . ĠWe Ġintroduce Ġthe Ġse fe Ġmodel , Ġextending Ġe fe Ġto Ġgrouped Ġdata . ĠWe Ġpresent Ġtwo Ġtechniques Ġto Ġshare Ġstatistical Ġstrength Ġamong Ġthe Ġembed ding Ġvectors , Ġone Ġbased Ġon Ġhierarchical Ġmodeling Ġand Ġone Ġbased Ġon Ġam ort ization . ĠWe Ġcarry Ġout Ġa Ġthorough Ġexperimental Ġstudy Ġon Ġtwo Ġtext Ġdatabases , ĠAr X iv Ġpapers Ġby Ġsection Ġand ĠU . S . ĠCongressional Ġspeeches Ġby Ġhome Ġstate Ġand Ġpolitical Ġparty . ĠUsing ĠPo isson Ġembed d ings , Ġwe Ġstudy Ġmarket Ġbasket Ġdata Ġfrom Ġa Ġlarge Ġgrocery Ġstore , Ġgrouped Ġby Ġseason . ĠOn Ġall Ġthree Ġdata Ġsets , Ġse fe Ġoutper forms Ġe fe Ġin Ġterms Ġof Ġheld - out Ġlog - like lihood . ĠQual itatively , Ġwe Ġdemonstrate Ġhow Ġse fe Ġdiscovers Ġwhich Ġwords Ġare Ġused Ġmost Ġdifferently Ġacross ĠU . S . Ġstates Ġand Ġpolitical Ġparties , Ġand Ġshow Ġhow Ġword Ġusage Ġchanges Ġin Ġdifferent ĠAr X iv Ġdisciplines . </s> Model ĠDescription </s> In Ġthis Ġsection , Ġwe Ġdevelop Ġse fe , Ġa Ġmodel Ġthat Ġbuilds Ġon Ġe fe ĠB IB REF 10 Ġto Ġcapture Ġsemantic Ġvariations Ġacross Ġgroups Ġof Ġdata . ĠIn Ġembed ding Ġmodels , Ġwe Ġrepresent Ġeach Ġobject Ġ( e . g ., Ġa Ġword Ġin Ġtext , Ġor Ġan Ġitem Ġin Ġshopping Ġdata ) Ġusing Ġtwo Ġsets Ġof Ġvectors , Ġan Ġembed ding Ġvector Ġand Ġa Ġcontext Ġvector . ĠIn Ġthis Ġpaper , Ġwe Ġare Ġinterested Ġin Ġhow Ġthe Ġembed d ings Ġvary Ġacross Ġgroups Ġof Ġdata , Ġand Ġfor Ġeach Ġobject Ġwe Ġwant Ġto Ġlearn Ġa Ġseparate Ġembed ding Ġvector Ġfor Ġeach Ġgroup . ĠHaving Ġa Ġseparate Ġembed ding Ġfor Ġeach Ġgroup Ġallows Ġus Ġto Ġstudy Ġhow Ġthe Ġusage Ġof Ġa Ġword Ġlike Ġ1 . 10 intelligence Ġvaries Ġacross Ġcategories Ġof Ġthe ĠAr X iv , Ġor Ġwhich Ġwords Ġare Ġused Ġmost Ġdifferently Ġby ĠU . S . ĠSenators Ġdepending Ġon Ġwhich Ġstate Ġthey Ġare Ġfrom Ġand Ġwhether Ġthey Ġare ĠDemocrats Ġor ĠRepublicans . </s> The Ġse fe Ġmodel Ġextends Ġe fe Ġto Ġgrouped Ġdata , Ġby Ġhaving Ġthe Ġembed ding Ġvectors Ġbe Ġspecific Ġfor Ġeach Ġgroup , Ġwhile Ġsharing Ġthe Ġcontext Ġvectors Ġacross Ġall Ġgroups . ĠWe Ġreview Ġthe Ġe fe Ġmodel Ġin ĠSection ĠSEC REF 4 Ġ. ĠWe Ġthen Ġformal ize Ġthe Ġidea Ġof Ġsharing Ġthe Ġcontext Ġvectors Ġin ĠSection ĠSEC REF 8 Ġ, Ġwhere Ġwe Ġpresent Ġtwo Ġapproaches Ġto Ġbuild Ġa Ġhierarchical Ġstructure Ġover Ġthe Ġgroup - specific Ġembed d ings . </s> Background : ĠExp onential ĠFamily ĠEmb edd ings </s> In Ġexponential Ġfamily Ġembed d ings , Ġwe Ġhave Ġa Ġcollection Ġof Ġobjects , Ġand Ġour Ġgoal Ġis Ġto Ġlearn Ġa Ġvector Ġrepresentation Ġof Ġthese Ġobjects Ġbased Ġon Ġtheir Ġco - occ urrence Ġpatterns . </s> Let Ġus Ġconsider Ġa Ġdataset Ġrepresented Ġas Ġa Ġ( typically Ġsparse ) Ġmatrix ĠIN LINE FORM 0 Ġ, Ġwhere Ġcolumns Ġare Ġdat ap oint s Ġand Ġrows Ġare Ġobjects . ĠFor Ġexample , Ġin Ġtext , Ġeach Ġcolumn Ġcorresponds Ġto Ġa Ġlocation Ġin Ġthe Ġtext , Ġand Ġeach Ġentry ĠIN LINE FORM 1 Ġis Ġa Ġbinary Ġvariable Ġthat Ġindicates Ġwhether Ġword ĠIN LINE FORM 2 Ġappears Ġat Ġlocation ĠIN LINE FORM 3 Ġ. </s> In Ġe fe , Ġwe Ġrepresent Ġeach Ġobject ĠIN LINE FORM 0 Ġwith Ġtwo Ġsets Ġof Ġvectors , Ġembed d ings Ġvectors ĠIN LINE FORM 1 Ġand Ġcontext Ġvectors ĠIN LINE FORM 2 Ġ, Ġand Ġwe Ġposit Ġa Ġprobability Ġdistribution Ġof Ġdata Ġentries ĠIN LINE FORM 3 Ġin Ġwhich Ġthese Ġvectors Ġinteract . ĠThe Ġdefinition Ġof Ġthe Ġe fe Ġmodel Ġrequires Ġthree Ġingredients : Ġa Ġcontext , Ġa Ġconditional Ġexponential Ġfamily , Ġand Ġa Ġparameter Ġsharing Ġstructure . ĠWe Ġnext Ġdescribe Ġthese Ġthree Ġcomponents . </s> Exp onential Ġfamily Ġembed d ings Ġlearn Ġthe Ġvector Ġrepresentation Ġof Ġobjects Ġbased Ġon Ġthe Ġconditional Ġprobability Ġof Ġeach Ġobservation , Ġconditioned Ġon Ġthe Ġobservations Ġin Ġits Ġcontext . ĠThe Ġcontext ĠIN LINE FORM 0 Ġgives Ġthe Ġindices Ġof Ġthe Ġobservations Ġthat Ġappear Ġin Ġthe Ġconditional Ġprobability Ġdistribution Ġof ĠIN LINE FORM 1 Ġ. ĠThe Ġdefinition Ġof Ġthe Ġcontext Ġvaries Ġacross Ġapplications . ĠIn Ġtext , Ġit Ġcorresponds Ġto Ġthe Ġset Ġof Ġwords Ġin Ġa Ġfixed - size Ġwindow Ġcentered Ġat Ġlocation ĠIN LINE FORM 2 Ġ. </s> Given Ġthe Ġcontext ĠIN LINE FORM 0 Ġand Ġthe Ġcorresponding Ġobservations ĠIN LINE FORM 1 Ġindexed Ġby ĠIN LINE FORM 2 Ġ, Ġthe Ġdistribution Ġfor ĠIN LINE FORM 3 Ġis Ġin Ġthe Ġexponential Ġfamily , ĠDIS PLAY FORM 0 </s> with Ġsufficient Ġstatistics ĠIN LINE FORM 0 Ġand Ġnatural Ġparameter ĠIN LINE FORM 1 Ġ. ĠThe Ġparameter Ġvectors Ġinteract Ġin Ġthe Ġconditional Ġprobability Ġdistributions Ġof Ġeach Ġobservation ĠIN LINE FORM 2 Ġas Ġfollows . ĠThe Ġembed ding Ġvectors ĠIN LINE FORM 3 Ġand Ġthe Ġcontext Ġvectors ĠIN LINE FORM 4 Ġare Ġcombined Ġto Ġform Ġthe Ġnatural Ġparameter , ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġlink Ġfunction . ĠExp onential Ġfamily Ġembed d ings Ġcan Ġbe Ġunderstood Ġas Ġa Ġbank Ġof Ġgl m . ĠThe Ġcontext Ġvectors Ġare Ġcombined Ġto Ġgive Ġthe Ġcovari ates , Ġand Ġthe ĠâĢ ľ reg ression Ġcoefficients âĢ Ŀ Ġare Ġthe Ġembed ding Ġvectors . ĠIn ĠE q . ĠEQ REF 6 Ġ, Ġthe Ġlink Ġfunction ĠIN LINE FORM 1 Ġplays Ġthe Ġsame Ġrole Ġas Ġin Ġgl m Ġand Ġis Ġa Ġmodeling Ġchoice . ĠWe Ġuse Ġthe Ġidentity Ġlink Ġfunction . </s> The Ġthird Ġingredient Ġof Ġthe Ġe fe Ġmodel Ġis Ġthe Ġparameter Ġsharing Ġstructure , Ġwhich Ġindicates Ġhow Ġthe Ġembed ding Ġvectors Ġare Ġshared Ġacross Ġobservations . ĠIn Ġthe Ġstandard Ġe fe Ġmodel , Ġwe Ġuse ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġfor Ġall Ġcolumns Ġof ĠIN LINE FORM 2 Ġ. ĠThat Ġis , Ġeach Ġunique Ġobject ĠIN LINE FORM 3 Ġhas Ġa Ġshared Ġrepresentation Ġacross Ġall Ġinstances . </s> The Ġobjective Ġfunction . ĠIn Ġe fe , Ġwe Ġmaximize Ġthe Ġobjective Ġfunction , Ġwhich Ġis Ġgiven Ġby Ġthe Ġsum Ġof Ġthe Ġlog - cond itional Ġlikelihood s Ġin ĠE q . ĠEQ REF 5 Ġ. ĠIn Ġaddition , Ġwe Ġadd Ġan ĠIN LINE FORM 0 Ġ- regular ization Ġterm Ġ( we Ġuse Ġthe Ġnotation Ġof Ġthe Ġlog ĠGa ussian Ġpdf ) Ġover Ġthe Ġembed ding Ġand Ġcontext Ġvectors , Ġyielding ĠDIS PLAY FORM 0 </s> Note Ġthat Ġmaximizing Ġthe Ġregular ized Ġconditional Ġlikelihood Ġis Ġnot Ġequivalent Ġto Ġmaximum Ġa Ġposterior i . ĠRather , Ġit Ġis Ġsimilar Ġto Ġmaxim ization Ġof Ġthe Ġpseudo - like lihood Ġin Ġcondition ally Ġspecified Ġmodels ĠB IB REF 26 Ġ, ĠB IB REF 10 Ġ. </s> Struct ured ĠExp onential ĠFamily ĠEmb edd ings </s> Here , Ġwe Ġdescribe Ġthe Ġse fe Ġmodel Ġfor Ġgrouped Ġdata . ĠIn Ġtext , Ġsome Ġexamples Ġof Ġgrouped Ġdata Ġare ĠCongressional Ġspeeches Ġgrouped Ġinto Ġpolitical Ġparties Ġor Ġscientific Ġdocuments Ġgrouped Ġby Ġdiscipline . ĠOur Ġgoal Ġis Ġto Ġlearn Ġgroup - specific Ġembed d ings Ġfrom Ġdata Ġpartition ed Ġinto ĠIN LINE FORM 0 Ġgroups , Ġi . e ., Ġeach Ġinstance ĠIN LINE FORM 1 Ġis Ġassociated Ġwith Ġa Ġgroup ĠIN LINE FORM 2 Ġ. ĠThe Ġse fe Ġmodel Ġextends Ġe fe Ġto Ġlearn Ġa Ġseparate Ġset Ġof Ġembed ding Ġvectors Ġfor Ġeach Ġgroup . </s> To Ġbuild Ġthe Ġse fe Ġmodel , Ġwe Ġimpose Ġa Ġparticular Ġparameter Ġsharing Ġstructure Ġover Ġthe Ġset Ġof Ġembed ding Ġand Ġcontext Ġvectors . ĠWe Ġposit Ġa Ġstructured Ġmodel Ġin Ġwhich Ġthe Ġcontext Ġvectors Ġare Ġshared Ġacross Ġgroups , Ġi . e ., ĠIN LINE FORM 0 Ġ( as Ġin Ġthe Ġstandard Ġe fe Ġmodel ), Ġbut Ġthe Ġembed ding Ġvectors Ġare Ġonly Ġshared Ġat Ġthe Ġgroup Ġlevel , Ġi . e ., Ġfor Ġan Ġobservation ĠIN LINE FORM 1 Ġbelonging Ġto Ġgroup ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ. ĠHere , ĠIN LINE FORM 4 Ġdenotes Ġthe Ġembed ding Ġvector Ġcorresponding Ġto Ġgroup ĠIN LINE FORM 5 Ġ. ĠWe Ġshow Ġa Ġgraphical Ġrepresentation Ġof Ġthe Ġse fe Ġin ĠFigure ĠFIG REF 1 Ġ. </s> Sh aring Ġthe Ġcontext Ġvectors ĠIN LINE FORM 0 Ġhas Ġtwo Ġadvantages . ĠFirst , Ġthe Ġshared Ġstructure Ġreduces Ġthe Ġnumber Ġof Ġparameters , Ġwhile Ġthe Ġresulting Ġse fe Ġmodel Ġis Ġstill Ġflexible Ġto Ġcapture Ġhow Ġdifferently Ġwords Ġare Ġused Ġacross Ġdifferent Ġgroups , Ġas ĠIN LINE FORM 1 Ġis Ġallowed Ġto Ġvary . ĠSecond , Ġit Ġhas Ġthe Ġimportant Ġeffect Ġof Ġun iting Ġall Ġembed ding Ġparameters Ġin Ġthe Ġsame Ġspace , Ġas Ġthe Ġgroup - specific Ġvectors ĠIN LINE FORM 4 Ġneed Ġto Ġagree Ġwith Ġthe Ġcomponents Ġof ĠIN LINE FORM 5 Ġ. ĠWhile Ġone Ġcould Ġlearn Ġa Ġseparate Ġembed ding Ġmodel Ġfor Ġeach Ġgroup , Ġas Ġhas Ġbeen Ġdone Ġfor Ġtext Ġgrouped Ġinto Ġtime Ġslices ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġthis Ġapproach Ġwould Ġrequire Ġad - h oc Ġpost processing Ġsteps Ġto Ġalign Ġthe Ġembed d ings . </s> When Ġthere Ġare ĠIN LINE FORM 0 Ġgroups , Ġthe Ġse fe Ġmodel Ġhas ĠIN LINE FORM 1 Ġtimes Ġas Ġmany Ġembed ding Ġvectors Ġthan Ġthe Ġstandard Ġembed ding Ġmodel . ĠThis Ġmay Ġcomplicate Ġinf erences Ġabout Ġthe Ġgroup - specific Ġvectors , Ġespecially Ġfor Ġgroups Ġwith Ġless Ġdata . ĠAdditionally , Ġan Ġobject ĠIN LINE FORM 2 Ġmay Ġappear Ġwith Ġvery Ġlow Ġfrequency Ġin Ġa Ġparticular Ġgroup . ĠThus , Ġthe ĠnaÃ¯ve Ġapproach Ġfor Ġbuilding Ġthe Ġse fe Ġmodel Ġwithout Ġadditional Ġstructure Ġmay Ġbe Ġdetrimental Ġfor Ġthe Ġquality Ġof Ġthe Ġembed d ings , Ġespecially Ġfor Ġsmall - sized Ġgroups . ĠTo Ġaddress Ġthis Ġproblem , Ġwe Ġpropose Ġtwo Ġdifferent Ġmethods Ġto Ġtie Ġthe Ġindividual ĠIN LINE FORM 3 Ġtogether , Ġsharing Ġstatistical Ġstrength Ġamong Ġthem . ĠThe Ġfirst Ġapproach Ġconsists Ġin Ġa Ġhierarchical Ġembed ding Ġstructure . ĠThe Ġsecond Ġapproach Ġis Ġbased Ġon Ġam ort ization . ĠIn Ġboth Ġmethods , Ġwe Ġintroduce Ġa Ġset Ġof Ġglobal Ġembed ding Ġvectors ĠIN LINE FORM 4 Ġ, Ġand Ġimpose Ġa Ġparticular Ġstructure Ġto Ġgenerate ĠIN LINE FORM 5 Ġfrom ĠIN LINE FORM 6 Ġ. </s> H ier arch ical Ġembed ding Ġstructure . ĠHere , Ġwe Ġimpose Ġa Ġhierarchical Ġstructure Ġthat Ġallows Ġsharing Ġstatistical Ġstrength Ġamong Ġthe Ġper - group Ġvariables . ĠFor Ġthat , Ġwe Ġassume Ġthat Ġeach ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġa Ġfixed Ġhyper param eter . ĠThus , Ġwe Ġreplace Ġthe Ġe fe Ġobjective Ġfunction Ġin ĠE q . ĠEQ REF 7 Ġwith ĠDIS PLAY FORM 0 </s> where Ġthe ĠIN LINE FORM 0 Ġ- regular ization Ġterm Ġnow Ġapplies Ġonly Ġon ĠIN LINE FORM 1 Ġand Ġthe Ġglobal Ġvectors ĠIN LINE FORM 2 Ġ. </s> F itting Ġthe Ġhierarchical Ġmodel Ġinvolves Ġmaximizing ĠE q . ĠEQ REF 11 Ġwith Ġrespect Ġto ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ. ĠWe Ġnote Ġthat Ġwe Ġhave Ġnot Ġreduced Ġthe Ġnumber Ġof Ġparameters Ġto Ġbe Ġinferred ; Ġrather , Ġwe Ġtie Ġthem Ġtogether Ġthrough Ġa Ġcommon Ġprior Ġdistribution . ĠWe Ġuse Ġsto ch astic Ġgradient Ġascent Ġto Ġmaximize ĠE q . ĠEQ REF 11 Ġ. </s> Am ort ization . ĠThe Ġidea Ġof Ġam ort ization Ġhas Ġbeen Ġapplied Ġin Ġthe Ġliterature Ġto Ġdevelop Ġam ort ized Ġinference Ġalgorithms ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠThe Ġmain Ġinsight Ġbehind Ġam ort ization Ġis Ġto Ġreuse Ġinf erences Ġabout Ġpast Ġexperiences Ġwhen Ġpresented Ġwith Ġa Ġnew Ġtask , Ġleveraging Ġthe Ġaccumulated Ġknowledge Ġto Ġquickly Ġsolve Ġthe Ġnew Ġproblem . ĠHere , Ġwe Ġuse Ġam ort ization Ġto Ġcontrol Ġthe Ġnumber Ġof Ġparameters Ġof Ġthe Ġse fe Ġmodel . ĠIn Ġparticular , Ġwe Ġset Ġthe Ġper - group Ġembed d ings ĠIN LINE FORM 0 Ġto Ġbe Ġthe Ġoutput Ġof Ġa Ġdeter ministic Ġfunction Ġof Ġthe Ġglobal Ġembed ding Ġvectors , ĠIN LINE FORM 1 Ġ. ĠWe Ġuse Ġa Ġdifferent Ġfunction ĠIN LINE FORM 2 Ġfor Ġeach Ġgroup ĠIN LINE FORM 3 Ġ, Ġand Ġwe Ġparameter ize Ġthem Ġusing Ġneural Ġnetworks , Ġsimilarly Ġto Ġother Ġworks Ġon Ġam ort ized Ġinference ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ. ĠUnlike Ġstandard Ġuses Ġof Ġam ort ized Ġinference , Ġin Ġse fe Ġthe Ġinput Ġto Ġthe Ġfunctions ĠIN LINE FORM 4 Ġis Ġunob served Ġand Ġmust Ġbe Ġestimated Ġtogether Ġwith Ġthe Ġparameters Ġof Ġthe Ġfunctions ĠIN LINE FORM 5 Ġ. </s> Depending Ġon Ġthe Ġarchitecture Ġof Ġthe Ġneural Ġnetworks , Ġthe Ġam ort ization Ġcan Ġsignificantly Ġreduce Ġthe Ġnumber Ġof Ġparameters Ġin Ġthe Ġmodel Ġ( as Ġcompared Ġto Ġthe Ġnon - am ort ized Ġmodel ), Ġwhile Ġstill Ġhaving Ġthe Ġflexibility Ġto Ġmodel Ġdifferent Ġembed ding Ġvectors Ġfor Ġeach Ġgroup . ĠThe Ġnumber Ġof Ġparameters Ġin Ġthe Ġse fe Ġmodel Ġis ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġnumber Ġof Ġgroups , ĠIN LINE FORM 2 Ġis Ġthe Ġdimension ality Ġof Ġthe Ġembed ding Ġvectors , Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġnumber Ġof Ġobjects Ġ( e . g ., Ġthe Ġvocabulary Ġsize ). ĠWith Ġam ort ization , Ġwe Ġreduce Ġthe Ġnumber Ġof Ġparameters Ġto ĠIN LINE FORM 4 Ġ, Ġwhere ĠIN LINE FORM 5 Ġis Ġthe Ġnumber Ġof Ġparameters Ġof Ġthe Ġneural Ġnetwork . ĠSince Ġtypically ĠIN LINE FORM 6 Ġ, Ġthis Ġcorresponds Ġto Ġa Ġsignificant Ġreduction Ġin Ġthe Ġnumber Ġof Ġparameters , Ġeven Ġwhen ĠIN LINE FORM 7 Ġscales Ġlin early Ġwith ĠIN LINE FORM 8 Ġ. </s> In Ġthe Ġam ort ized Ġse fe Ġmodel , Ġwe Ġneed Ġto Ġintroduce Ġa Ġnew Ġset Ġof Ġparameters ĠIN LINE FORM 0 Ġfor Ġeach Ġgroup ĠIN LINE FORM 1 Ġ, Ġcorresponding Ġto Ġthe Ġneural Ġnetwork Ġparameters . ĠGiven Ġthese , Ġthe Ġgroup - specific Ġembed ding Ġvectors ĠIN LINE FORM 2 Ġare Ġobtained Ġas ĠDIS PLAY FORM 0 </s> We Ġcompare Ġtwo Ġarchitectures Ġfor Ġthe Ġfunction ĠIN LINE FORM 0 Ġ: Ġfully Ġconnected Ġfeed - forward Ġneural Ġnetworks Ġand Ġresidual Ġnetworks ĠB IB REF 32 Ġ. ĠFor Ġboth , Ġwe Ġconsider Ġone Ġhidden Ġlayer Ġwith ĠIN LINE FORM 1 Ġunits . ĠHence , Ġthe Ġnetwork Ġparameters ĠIN LINE FORM 2 Ġare Ġtwo Ġweight Ġmat rices , ĠDIS PLAY FORM 0 </s> i . e ., ĠIN LINE FORM 0 Ġparameters . ĠThe Ġneural Ġnetwork Ġtakes Ġas Ġinput Ġthe Ġglobal Ġembed ding Ġvector ĠIN LINE FORM 1 Ġ, Ġand Ġit Ġoutputs Ġthe Ġgroup - specific Ġembed ding Ġvectors ĠIN LINE FORM 2 Ġ. ĠThe Ġmathematical Ġexpression Ġfor ĠIN LINE FORM 3 Ġfor Ġa Ġfeed - forward Ġneural Ġnetwork Ġand Ġa Ġresidual Ġnetwork Ġis Ġrespectively Ġgiven Ġby ĠDIS PLAY FORM 0 </s> where Ġwe Ġhave Ġconsidered Ġthe Ġhyper b olic Ġtang ent Ġnon linear ity . ĠThe Ġmain Ġdifference Ġbetween Ġboth Ġnetwork Ġarchitectures Ġis Ġthat Ġthe Ġresidual Ġnetwork Ġfocuses Ġon Ġmodeling Ġhow Ġthe Ġgroup - specific Ġembed ding Ġvectors ĠIN LINE FORM 0 Ġdiffer Ġfrom Ġthe Ġglobal Ġvectors ĠIN LINE FORM 1 Ġ. ĠThat Ġis , Ġif Ġall Ġweights Ġwere Ġset Ġto Ġ0 , Ġthe Ġfeed - forward Ġnetwork Ġwould Ġoutput Ġ0 , Ġwhile Ġthe Ġresidual Ġnetwork Ġwould Ġoutput Ġthe Ġglobal Ġvector ĠIN LINE FORM 2 Ġfor Ġall Ġgroups . </s> The Ġobjective Ġfunction Ġunder Ġam ort ization Ġis Ġgiven Ġby ĠDIS PLAY FORM 0 </s> We Ġmaximize Ġthis Ġobjective Ġwith Ġrespect Ġto ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġusing Ġsto ch astic Ġgradient Ġascent . ĠWe Ġimplement Ġthe Ġhierarchical Ġand Ġam ort ized Ġse fe Ġmodels Ġin ĠT ensor Flow ĠB IB REF 33 Ġ, Ġwhich Ġallows Ġus Ġto Ġleverage Ġautomatic Ġdifferentiation . </s> Example : Ġstructured ĠBern ou lli Ġembed d ings Ġfor Ġgrouped Ġtext Ġdata . ĠHere , Ġwe Ġconsider Ġa Ġset Ġof Ġdocuments Ġbroken Ġdown Ġinto Ġgroups , Ġsuch Ġas Ġpolitical Ġaffili ations Ġor Ġscientific Ġdisciplines . ĠWe Ġcan Ġrepresent Ġthe Ġdata Ġas Ġa Ġbinary Ġmatrix ĠIN LINE FORM 0 Ġand Ġa Ġset Ġof Ġgroup Ġindicators ĠIN LINE FORM 1 Ġ. ĠSince Ġonly Ġone Ġword Ġcan Ġappear Ġin Ġa Ġcertain Ġposition ĠIN LINE FORM 2 Ġ, Ġthe Ġmatrix ĠIN LINE FORM 3 Ġcontains Ġone Ġnon - zero Ġelement Ġper Ġcolumn . ĠIn Ġembed ding Ġmodels , Ġwe Ġignore Ġthis Ġone - hot Ġconstraint Ġfor Ġcomputational Ġefficiency , Ġand Ġconsider Ġthat Ġthe Ġobservations Ġare Ġgenerated Ġfollowing Ġa Ġset Ġof Ġconditional ĠBern ou lli Ġdistributions ĠB IB REF 2 Ġ, ĠB IB REF 10 Ġ. ĠGiven Ġthat Ġmost Ġof Ġthe Ġentries Ġin ĠIN LINE FORM 4 Ġare Ġzero , Ġembed ding Ġmodels Ġtypically Ġdown we igh Ġthe Ġcontribution Ġof Ġthe Ġz eros Ġto Ġthe Ġobjective Ġfunction . ĠB IB REF 2 Ġuse Ġnegative Ġsampling , Ġwhich Ġconsists Ġin Ġrandomly Ġchoosing Ġa Ġsubset Ġof Ġthe Ġzero Ġobservations . ĠThis Ġcorresponds Ġto Ġa Ġbiased Ġestimate Ġof Ġthe Ġgradient Ġin Ġa ĠBern ou lli Ġexponential Ġfamily Ġembed ding Ġmodel ĠB IB REF 10 Ġ. </s> The Ġcontext ĠIN LINE FORM 0 Ġis Ġgiven Ġat Ġeach Ġposition ĠIN LINE FORM 1 Ġby Ġthe Ġset Ġof Ġsurrounding Ġwords Ġin Ġthe Ġdocument , Ġaccording Ġto Ġa Ġfixed - size Ġwindow . </s> Example : Ġstructured ĠPo isson Ġembed d ings Ġfor Ġgrouped Ġshopping Ġdata . Ġe fe Ġand Ġse fe Ġextend Ġto Ġapplications Ġbeyond Ġtext Ġand Ġwe Ġuse Ġse fe Ġto Ġmodel Ġsupermarket Ġpurchases Ġbroken Ġdown Ġby Ġmonth . ĠFor Ġeach Ġmarket Ġbasket ĠIN LINE FORM 0 Ġ, Ġwe Ġhave Ġaccess Ġto Ġthe Ġmonth ĠIN LINE FORM 1 Ġin Ġwhich Ġthat Ġshopping Ġtrip Ġhappened . ĠNow , Ġthe Ġrows Ġof Ġthe Ġdata Ġmatrix ĠIN LINE FORM 2 Ġindex Ġitems , Ġwhile Ġcolumns Ġindex Ġshopping Ġtrips . ĠEach Ġelement ĠIN LINE FORM 3 Ġdenotes Ġthe Ġnumber Ġof Ġunits Ġof Ġitem ĠIN LINE FORM 4 Ġpurchased Ġat Ġtrip ĠIN LINE FORM 5 Ġ. ĠUnlike Ġtext , Ġeach Ġcolumn Ġof ĠIN LINE FORM 6 Ġmay Ġcontain Ġmore Ġthan Ġone Ġnon - zero Ġelement . ĠThe Ġcontext ĠIN LINE FORM 7 Ġcorresponds Ġto Ġthe Ġset Ġof Ġitems Ġpurchased Ġin Ġtrip ĠIN LINE FORM 8 Ġ, Ġexcluding ĠIN LINE FORM 9 Ġ. </s> In Ġthis Ġcase , Ġwe Ġuse Ġthe ĠPo isson Ġconditional Ġdistribution , Ġwhich Ġis Ġmore Ġappropriate Ġfor Ġcount Ġdata . ĠIn ĠPo isson Ġse fe , Ġwe Ġalso Ġdown we igh Ġthe Ġcontribution Ġof Ġthe Ġz eros Ġin Ġthe Ġobjective Ġfunction , Ġwhich Ġprovides Ġbetter Ġresults Ġbecause Ġit Ġallows Ġthe Ġinference Ġto Ġfocus Ġon Ġthe Ġpositive Ġsignal Ġof Ġthe Ġactual Ġpurchases ĠB IB REF 10 Ġ, ĠB IB REF 2 Ġ. </s> E mp ir ical ĠStudy </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġexperimental Ġstudy . ĠWe Ġfit Ġthe Ġse fe Ġmodel Ġon Ġthree Ġdatasets Ġand Ġcompare Ġit Ġagainst Ġthe Ġe fe ĠB IB REF 10 Ġ. ĠOur Ġquantitative Ġresults Ġshow Ġthat Ġsharing Ġthe Ġcontext Ġvectors Ġprovides Ġbetter Ġresults , Ġand Ġthat Ġam ort ization Ġand Ġhierarchical Ġstructure Ġgive Ġfurther Ġimprovements . </s> Data . ĠWe Ġapply Ġthe Ġse fe Ġon Ġthree Ġdatasets : ĠAr X iv Ġpapers , ĠU . S . ĠSenate Ġspeeches , Ġand Ġpurchases Ġon Ġsupermarket Ġgrocery Ġshopping Ġdata . ĠWe Ġdescribe Ġthese Ġdatasets Ġbelow , Ġand Ġwe Ġprovide Ġa Ġsummary Ġof Ġthe Ġdatasets Ġin ĠTable ĠT AB REF 17 Ġ. </s> Ar X iv Ġpapers : ĠThis Ġdataset Ġcontains Ġthe Ġabstract s Ġof Ġpapers Ġpublished Ġon Ġthe ĠAr X iv Ġunder Ġthe Ġ19 Ġdifferent Ġtags Ġbetween ĠApril Ġ2007 Ġand ĠJune Ġ2015 . ĠWe Ġtreat Ġeach Ġtag Ġas Ġa Ġgroup Ġand Ġfit Ġse fe Ġwith Ġthe Ġgoal Ġof Ġuncover ing Ġwhich Ġwords Ġhave Ġthe Ġstrongest Ġshift Ġin Ġusage . ĠWe Ġsplit Ġthe Ġabstract s Ġinto Ġtraining , Ġvalidation , Ġand Ġtest Ġsets , Ġwith Ġproportions Ġof ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ, Ġrespectively . </s> Senate Ġspeeches : ĠThis Ġdataset Ġcontains ĠU . S . ĠSenate Ġspeeches Ġfrom Ġ1994 Ġto Ġmid Ġ2009 . ĠIn Ġcontrast Ġto Ġthe ĠAr X iv Ġcollection , Ġit Ġis Ġa Ġtranscript Ġof Ġspoken Ġlanguage . ĠWe Ġgroup Ġthe Ġdata Ġinto Ġstate Ġof Ġorigin Ġof Ġthe Ġspeaker Ġand Ġhis Ġor Ġher Ġparty Ġaffiliation . ĠOnly Ġaffili ations Ġwith Ġthe ĠRepublican Ġand ĠDemocratic ĠParty Ġare Ġconsidered . ĠAs Ġa Ġresult , Ġthere Ġare Ġ83 Ġgroups Ġ( Republicans Ġfrom ĠAlabama , ĠDemocrats Ġfrom ĠAlabama , ĠRepublicans Ġfrom ĠArkansas , Ġetc .). ĠSome Ġof Ġthe Ġstate / party Ġcombinations Ġare Ġnot Ġavailable Ġin Ġthe Ġdata , Ġas Ġsome Ġof Ġthe Ġ50 Ġstates Ġhave Ġonly Ġhad ĠSenators Ġwith Ġthe Ġsame Ġparty Ġaffiliation . ĠWe Ġsplit Ġthe Ġspeeches Ġinto Ġtraining Ġ( ĠIN LINE FORM 0 Ġ), Ġvalidation Ġ( ĠIN LINE FORM 1 Ġ), Ġand Ġtesting Ġ( ĠIN LINE FORM 2 Ġ). </s> G roc ery Ġshopping Ġdata : ĠThis Ġdataset Ġcontains Ġthe Ġpurchases Ġof ĠIN LINE FORM 0 Ġcustomers . ĠThe Ġdata Ġcovers Ġa Ġperiod Ġof Ġ97 Ġweeks . ĠAfter Ġremoving Ġlow - frequency Ġitems , Ġthe Ġdata Ġcontains ĠIN LINE FORM 1 Ġunique Ġitems Ġat Ġthe Ġ1 . 10 up c Ġ( Universal ĠProduct ĠCode ) Ġlevel . ĠWe Ġsplit Ġthe Ġdata Ġinto Ġa Ġtraining , Ġtest , Ġand Ġvalidation Ġsets , Ġwith Ġproportions Ġof ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġ, Ġrespectively . ĠThe Ġtraining Ġdata Ġcontains ĠIN LINE FORM 5 Ġshopping Ġtrips Ġand ĠIN LINE FORM 6 Ġpurchases Ġin Ġtotal . </s> For Ġthe Ġtext Ġcorpor a , Ġwe Ġfix Ġthe Ġvocabulary Ġto Ġthe Ġ15 k Ġmost Ġfrequent Ġterms Ġand Ġremove Ġall Ġwords Ġthat Ġare Ġnot Ġin Ġthe Ġvocabulary . ĠFollowing ĠB IB REF 2 Ġ, Ġwe Ġadditionally Ġremove Ġeach Ġword Ġwith Ġprobability ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġword Ġfrequency . ĠThis Ġdowns amples Ġespecially Ġthe Ġfrequent Ġwords Ġand Ġspeeds Ġup Ġtraining . Ġ( S izes Ġreported Ġin ĠTable ĠT AB REF 17 Ġare Ġthe Ġnumber Ġof Ġwords Ġremaining Ġafter Ġpre processing .) </s> Mod els . ĠOur Ġgoal Ġis Ġto Ġfit Ġthe Ġse fe Ġmodel Ġon Ġthese Ġdatasets . ĠFor Ġthe Ġtext Ġdata , Ġwe Ġuse Ġthe ĠBern ou lli Ġdistribution Ġas Ġthe Ġconditional Ġexponential Ġfamily , Ġwhile Ġfor Ġthe Ġshopping Ġdata Ġwe Ġuse Ġthe ĠPo isson Ġdistribution , Ġwhich Ġis Ġmore Ġappropriate Ġfor Ġcount Ġdata . </s> On Ġeach Ġdataset , Ġwe Ġcompare Ġfour Ġapproaches Ġbased Ġon Ġse fe Ġwith Ġtwo Ġe fe ĠB IB REF 10 Ġbas elines . ĠAll Ġare Ġfit Ġusing Ġs gd ĠB IB REF 34 Ġ. ĠIn Ġparticular , Ġwe Ġcompare Ġthe Ġfollowing Ġmethods :
Introduction Word embeddings BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 are unsupervised learning methods for capturing latent semantic structure in language. Word embedding methods analyze text data to learn distributed representations of the vocabulary that capture its co-occurrence statistics. These representations are useful for reasoning about word usage and meaning BIBREF7 , BIBREF8 . Word embeddings have also been extended to data beyond text BIBREF9 , BIBREF10 , such as items in a grocery store or neurons in the brain. efe is a probabilistic perspective on embeddings that encompasses many existing methods and opens the door to bringing expressive probabilistic modeling BIBREF11 , BIBREF12 to the problem of learning distributed representations. We develop sefe, an extension of efe for studying how embeddings can vary across groups of related data. We will study several examples: in U.S. Congressional speeches, word usage can vary across states or party affiliations; in scientific literature, the usage patterns of technical terms can vary across fields; in supermarket shopping data, co-purchase patterns of items can vary across seasons of the year. We will see that sefe discovers a per-group embedding representation of objects. While the naïve approach of fitting an individual embedding model for each group would typically suffer from lack of data—especially in groups for which fewer observations are available—we develop two methods that can share information across groups. Figure FIGREF1 illustrates the kind of variation that we can capture. We fit an sefe to ArXiv abstracts grouped into different sections, such as computer science (cs), quantitative finance (q-fin), and nonlinear sciences (nlin). sefe results in a per-section embedding of each term in the vocabulary. Using the fitted embeddings, we illustrate similar words to the word 1.10intelligence. We can see that how 1.10intelligence is used varies by field: in computer science the most similar words include 1.10artificial and 1.10ai; in finance, similar words include 1.10abilities and 1.10consciousness. In more detail, embedding methods posit two representation vectors for each term in the vocabulary; an embedding vector and a context vector. (We use the language of text for concreteness; as we mentioned, efe extend to other types of data.) The idea is that the conditional probability of each observed word depends on the interaction between the embedding vector and the context vectors of the surrounding words. In sefe, we posit a separate set of embedding vectors for each group but a shared set of context vectors; this ensures that the embedding vectors are in the same space. We propose two methods to share statistical strength among the embedding vectors. The first approach is based on hierarchical modeling BIBREF13 , which assumes that the group-specific embedding representations are tied through a global embedding. The second approach is based on amortization BIBREF14 , BIBREF15 , which considers that the individual embeddings are the output of a deterministic function of a global embedding representation. We use stochastic optimization to fit large data sets. Our work relates closely to two threads of research in the embedding literature. One is embedding methods that study how language evolves over time BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 . Time can be thought of as a type of “group”, though with evolutionary structure that we do not consider. The second thread is multilingual embeddings BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 ; our approach is different in that most words appear in all groups and we are interested in the variations of the embeddings across those groups. Our contributions are thus as follows. We introduce the sefe model, extending efe to grouped data. We present two techniques to share statistical strength among the embedding vectors, one based on hierarchical modeling and one based on amortization. We carry out a thorough experimental study on two text databases, ArXiv papers by section and U.S. Congressional speeches by home state and political party. Using Poisson embeddings, we study market basket data from a large grocery store, grouped by season. On all three data sets, sefe outperforms efe in terms of held-out log-likelihood. Qualitatively, we demonstrate how sefe discovers which words are used most differently across U.S. states and political parties, and show how word usage changes in different ArXiv disciplines. Model Description In this section, we develop sefe, a model that builds on efe BIBREF10 to capture semantic variations across groups of data. In embedding models, we represent each object (e.g., a word in text, or an item in shopping data) using two sets of vectors, an embedding vector and a context vector. In this paper, we are interested in how the embeddings vary across groups of data, and for each object we want to learn a separate embedding vector for each group. Having a separate embedding for each group allows us to study how the usage of a word like 1.10intelligence varies across categories of the ArXiv, or which words are used most differently by U.S. Senators depending on which state they are from and whether they are Democrats or Republicans. The sefe model extends efe to grouped data, by having the embedding vectors be specific for each group, while sharing the context vectors across all groups. We review the efe model in Section SECREF4 . We then formalize the idea of sharing the context vectors in Section SECREF8 , where we present two approaches to build a hierarchical structure over the group-specific embeddings. Background: Exponential Family Embeddings In exponential family embeddings, we have a collection of objects, and our goal is to learn a vector representation of these objects based on their co-occurrence patterns. Let us consider a dataset represented as a (typically sparse) matrix INLINEFORM0 , where columns are datapoints and rows are objects. For example, in text, each column corresponds to a location in the text, and each entry INLINEFORM1 is a binary variable that indicates whether word INLINEFORM2 appears at location INLINEFORM3 . In efe, we represent each object INLINEFORM0 with two sets of vectors, embeddings vectors INLINEFORM1 and context vectors INLINEFORM2 , and we posit a probability distribution of data entries INLINEFORM3 in which these vectors interact. The definition of the efe model requires three ingredients: a context, a conditional exponential family, and a parameter sharing structure. We next describe these three components. Exponential family embeddings learn the vector representation of objects based on the conditional probability of each observation, conditioned on the observations in its context. The context INLINEFORM0 gives the indices of the observations that appear in the conditional probability distribution of INLINEFORM1 . The definition of the context varies across applications. In text, it corresponds to the set of words in a fixed-size window centered at location INLINEFORM2 . Given the context INLINEFORM0 and the corresponding observations INLINEFORM1 indexed by INLINEFORM2 , the distribution for INLINEFORM3 is in the exponential family, DISPLAYFORM0 with sufficient statistics INLINEFORM0 and natural parameter INLINEFORM1 . The parameter vectors interact in the conditional probability distributions of each observation INLINEFORM2 as follows. The embedding vectors INLINEFORM3 and the context vectors INLINEFORM4 are combined to form the natural parameter, DISPLAYFORM0 where INLINEFORM0 is the link function. Exponential family embeddings can be understood as a bank of glm. The context vectors are combined to give the covariates, and the “regression coefficients” are the embedding vectors. In Eq. EQREF6 , the link function INLINEFORM1 plays the same role as in glm and is a modeling choice. We use the identity link function. The third ingredient of the efe model is the parameter sharing structure, which indicates how the embedding vectors are shared across observations. In the standard efe model, we use INLINEFORM0 and INLINEFORM1 for all columns of INLINEFORM2 . That is, each unique object INLINEFORM3 has a shared representation across all instances. The objective function. In efe, we maximize the objective function, which is given by the sum of the log-conditional likelihoods in Eq. EQREF5 . In addition, we add an INLINEFORM0 -regularization term (we use the notation of the log Gaussian pdf) over the embedding and context vectors, yielding DISPLAYFORM0 Note that maximizing the regularized conditional likelihood is not equivalent to maximum a posteriori. Rather, it is similar to maximization of the pseudo-likelihood in conditionally specified models BIBREF26 , BIBREF10 . Structured Exponential Family Embeddings Here, we describe the sefe model for grouped data. In text, some examples of grouped data are Congressional speeches grouped into political parties or scientific documents grouped by discipline. Our goal is to learn group-specific embeddings from data partitioned into INLINEFORM0 groups, i.e., each instance INLINEFORM1 is associated with a group INLINEFORM2 . The sefe model extends efe to learn a separate set of embedding vectors for each group. To build the sefe model, we impose a particular parameter sharing structure over the set of embedding and context vectors. We posit a structured model in which the context vectors are shared across groups, i.e., INLINEFORM0 (as in the standard efe model), but the embedding vectors are only shared at the group level, i.e., for an observation INLINEFORM1 belonging to group INLINEFORM2 , INLINEFORM3 . Here, INLINEFORM4 denotes the embedding vector corresponding to group INLINEFORM5 . We show a graphical representation of the sefe in Figure FIGREF1 . Sharing the context vectors INLINEFORM0 has two advantages. First, the shared structure reduces the number of parameters, while the resulting sefe model is still flexible to capture how differently words are used across different groups, as INLINEFORM1 is allowed to vary. Second, it has the important effect of uniting all embedding parameters in the same space, as the group-specific vectors INLINEFORM4 need to agree with the components of INLINEFORM5 . While one could learn a separate embedding model for each group, as has been done for text grouped into time slices BIBREF16 , BIBREF17 , BIBREF18 , this approach would require ad-hoc postprocessing steps to align the embeddings. When there are INLINEFORM0 groups, the sefe model has INLINEFORM1 times as many embedding vectors than the standard embedding model. This may complicate inferences about the group-specific vectors, especially for groups with less data. Additionally, an object INLINEFORM2 may appear with very low frequency in a particular group. Thus, the naïve approach for building the sefe model without additional structure may be detrimental for the quality of the embeddings, especially for small-sized groups. To address this problem, we propose two different methods to tie the individual INLINEFORM3 together, sharing statistical strength among them. The first approach consists in a hierarchical embedding structure. The second approach is based on amortization. In both methods, we introduce a set of global embedding vectors INLINEFORM4 , and impose a particular structure to generate INLINEFORM5 from INLINEFORM6 . Hierarchical embedding structure. Here, we impose a hierarchical structure that allows sharing statistical strength among the per-group variables. For that, we assume that each INLINEFORM0 , where INLINEFORM1 is a fixed hyperparameter. Thus, we replace the efe objective function in Eq. EQREF7 with DISPLAYFORM0 where the INLINEFORM0 -regularization term now applies only on INLINEFORM1 and the global vectors INLINEFORM2 . Fitting the hierarchical model involves maximizing Eq. EQREF11 with respect to INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 . We note that we have not reduced the number of parameters to be inferred; rather, we tie them together through a common prior distribution. We use stochastic gradient ascent to maximize Eq. EQREF11 . Amortization. The idea of amortization has been applied in the literature to develop amortized inference algorithms BIBREF14 , BIBREF15 . The main insight behind amortization is to reuse inferences about past experiences when presented with a new task, leveraging the accumulated knowledge to quickly solve the new problem. Here, we use amortization to control the number of parameters of the sefe model. In particular, we set the per-group embeddings INLINEFORM0 to be the output of a deterministic function of the global embedding vectors, INLINEFORM1 . We use a different function INLINEFORM2 for each group INLINEFORM3 , and we parameterize them using neural networks, similarly to other works on amortized inference BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 . Unlike standard uses of amortized inference, in sefe the input to the functions INLINEFORM4 is unobserved and must be estimated together with the parameters of the functions INLINEFORM5 . Depending on the architecture of the neural networks, the amortization can significantly reduce the number of parameters in the model (as compared to the non-amortized model), while still having the flexibility to model different embedding vectors for each group. The number of parameters in the sefe model is INLINEFORM0 , where INLINEFORM1 is the number of groups, INLINEFORM2 is the dimensionality of the embedding vectors, and INLINEFORM3 is the number of objects (e.g., the vocabulary size). With amortization, we reduce the number of parameters to INLINEFORM4 , where INLINEFORM5 is the number of parameters of the neural network. Since typically INLINEFORM6 , this corresponds to a significant reduction in the number of parameters, even when INLINEFORM7 scales linearly with INLINEFORM8 . In the amortized sefe model, we need to introduce a new set of parameters INLINEFORM0 for each group INLINEFORM1 , corresponding to the neural network parameters. Given these, the group-specific embedding vectors INLINEFORM2 are obtained as DISPLAYFORM0 We compare two architectures for the function INLINEFORM0 : fully connected feed-forward neural networks and residual networks BIBREF32 . For both, we consider one hidden layer with INLINEFORM1 units. Hence, the network parameters INLINEFORM2 are two weight matrices, DISPLAYFORM0 i.e., INLINEFORM0 parameters. The neural network takes as input the global embedding vector INLINEFORM1 , and it outputs the group-specific embedding vectors INLINEFORM2 . The mathematical expression for INLINEFORM3 for a feed-forward neural network and a residual network is respectively given by DISPLAYFORM0 where we have considered the hyperbolic tangent nonlinearity. The main difference between both network architectures is that the residual network focuses on modeling how the group-specific embedding vectors INLINEFORM0 differ from the global vectors INLINEFORM1 . That is, if all weights were set to 0, the feed-forward network would output 0, while the residual network would output the global vector INLINEFORM2 for all groups. The objective function under amortization is given by DISPLAYFORM0 We maximize this objective with respect to INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 using stochastic gradient ascent. We implement the hierarchical and amortized sefe models in TensorFlow BIBREF33 , which allows us to leverage automatic differentiation. Example: structured Bernoulli embeddings for grouped text data. Here, we consider a set of documents broken down into groups, such as political affiliations or scientific disciplines. We can represent the data as a binary matrix INLINEFORM0 and a set of group indicators INLINEFORM1 . Since only one word can appear in a certain position INLINEFORM2 , the matrix INLINEFORM3 contains one non-zero element per column. In embedding models, we ignore this one-hot constraint for computational efficiency, and consider that the observations are generated following a set of conditional Bernoulli distributions BIBREF2 , BIBREF10 . Given that most of the entries in INLINEFORM4 are zero, embedding models typically downweigh the contribution of the zeros to the objective function. BIBREF2 use negative sampling, which consists in randomly choosing a subset of the zero observations. This corresponds to a biased estimate of the gradient in a Bernoulli exponential family embedding model BIBREF10 . The context INLINEFORM0 is given at each position INLINEFORM1 by the set of surrounding words in the document, according to a fixed-size window. Example: structured Poisson embeddings for grouped shopping data. efe and sefe extend to applications beyond text and we use sefe to model supermarket purchases broken down by month. For each market basket INLINEFORM0 , we have access to the month INLINEFORM1 in which that shopping trip happened. Now, the rows of the data matrix INLINEFORM2 index items, while columns index shopping trips. Each element INLINEFORM3 denotes the number of units of item INLINEFORM4 purchased at trip INLINEFORM5 . Unlike text, each column of INLINEFORM6 may contain more than one non-zero element. The context INLINEFORM7 corresponds to the set of items purchased in trip INLINEFORM8 , excluding INLINEFORM9 . In this case, we use the Poisson conditional distribution, which is more appropriate for count data. In Poisson sefe, we also downweigh the contribution of the zeros in the objective function, which provides better results because it allows the inference to focus on the positive signal of the actual purchases BIBREF10 , BIBREF2 . Empirical Study In this section, we describe the experimental study. We fit the sefe model on three datasets and compare it against the efe BIBREF10 . Our quantitative results show that sharing the context vectors provides better results, and that amortization and hierarchical structure give further improvements. Data. We apply the sefe on three datasets: ArXiv papers, U.S. Senate speeches, and purchases on supermarket grocery shopping data. We describe these datasets below, and we provide a summary of the datasets in Table TABREF17 . ArXiv papers: This dataset contains the abstracts of papers published on the ArXiv under the 19 different tags between April 2007 and June 2015. We treat each tag as a group and fit sefe with the goal of uncovering which words have the strongest shift in usage. We split the abstracts into training, validation, and test sets, with proportions of INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , respectively. Senate speeches: This dataset contains U.S. Senate speeches from 1994 to mid 2009. In contrast to the ArXiv collection, it is a transcript of spoken language. We group the data into state of origin of the speaker and his or her party affiliation. Only affiliations with the Republican and Democratic Party are considered. As a result, there are 83 groups (Republicans from Alabama, Democrats from Alabama, Republicans from Arkansas, etc.). Some of the state/party combinations are not available in the data, as some of the 50 states have only had Senators with the same party affiliation. We split the speeches into training ( INLINEFORM0 ), validation ( INLINEFORM1 ), and testing ( INLINEFORM2 ). Grocery shopping data: This dataset contains the purchases of INLINEFORM0 customers. The data covers a period of 97 weeks. After removing low-frequency items, the data contains INLINEFORM1 unique items at the 1.10upc (Universal Product Code) level. We split the data into a training, test, and validation sets, with proportions of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 , respectively. The training data contains INLINEFORM5 shopping trips and INLINEFORM6 purchases in total. For the text corpora, we fix the vocabulary to the 15k most frequent terms and remove all words that are not in the vocabulary. Following BIBREF2 , we additionally remove each word with probability INLINEFORM0 , where INLINEFORM1 is the word frequency. This downsamples especially the frequent words and speeds up training. (Sizes reported in Table TABREF17 are the number of words remaining after preprocessing.) Models. Our goal is to fit the sefe model on these datasets. For the text data, we use the Bernoulli distribution as the conditional exponential family, while for the shopping data we use the Poisson distribution, which is more appropriate for count data. On each dataset, we compare four approaches based on sefe with two efe BIBREF10 baselines. All are fit using sgd BIBREF34 . In particular, we compare the following methods:
What experiments are used to demonstrate the benefits of this approach?
777217e025132ddc173cf33747ee590628a8f62f
What Ġexperiments Ġare Ġused Ġto Ġdemonstrate Ġthe Ġbenefits Ġof Ġthis Ġapproach ?
On Ġeach Ġdataset , Ġwe Ġcompare Ġfour Ġapproaches Ġbased Ġon Ġse fe Ġwith Ġtwo Ġe fe ĠB IB REF 10 Ġbas elines . ĠAll Ġare Ġfit Ġusing Ġs gd ĠB IB REF 34 Ġ. ĠIn Ġparticular , Ġwe Ġcompare Ġthe Ġfollowing Ġmethods :
<s> What Ġexperiments Ġare Ġused Ġto Ġdemonstrate Ġthe Ġbenefits Ġof Ġthis Ġapproach ? </s> Introduction </s> Word Ġembed d ings ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġare Ġun super vised Ġlearning Ġmethods Ġfor Ġcapturing Ġlatent Ġsemantic Ġstructure Ġin Ġlanguage . ĠWord Ġembed ding Ġmethods Ġanalyze Ġtext Ġdata Ġto Ġlearn Ġdistributed Ġrepresentations Ġof Ġthe Ġvocabulary Ġthat Ġcapture Ġits Ġco - occ urrence Ġstatistics . ĠThese Ġrepresentations Ġare Ġuseful Ġfor Ġreasoning Ġabout Ġword Ġusage Ġand Ġmeaning ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ. ĠWord Ġembed d ings Ġhave Ġalso Ġbeen Ġextended Ġto Ġdata Ġbeyond Ġtext ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, Ġsuch Ġas Ġitems Ġin Ġa Ġgrocery Ġstore Ġor Ġneurons Ġin Ġthe Ġbrain . Ġe fe Ġis Ġa Ġprob abil istic Ġperspective Ġon Ġembed d ings Ġthat Ġencompasses Ġmany Ġexisting Ġmethods Ġand Ġopens Ġthe Ġdoor Ġto Ġbringing Ġexpressive Ġprob abil istic Ġmodeling ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġto Ġthe Ġproblem Ġof Ġlearning Ġdistributed Ġrepresentations . </s> We Ġdevelop Ġse fe , Ġan Ġextension Ġof Ġe fe Ġfor Ġstudying Ġhow Ġembed d ings Ġcan Ġvary Ġacross Ġgroups Ġof Ġrelated Ġdata . ĠWe Ġwill Ġstudy Ġseveral Ġexamples : Ġin ĠU . S . ĠCongressional Ġspeeches , Ġword Ġusage Ġcan Ġvary Ġacross Ġstates Ġor Ġparty Ġaffili ations ; Ġin Ġscientific Ġliterature , Ġthe Ġusage Ġpatterns Ġof Ġtechnical Ġterms Ġcan Ġvary Ġacross Ġfields ; Ġin Ġsupermarket Ġshopping Ġdata , Ġco - p urchase Ġpatterns Ġof Ġitems Ġcan Ġvary Ġacross Ġseasons Ġof Ġthe Ġyear . ĠWe Ġwill Ġsee Ġthat Ġse fe Ġdiscovers Ġa Ġper - group Ġembed ding Ġrepresentation Ġof Ġobjects . ĠWhile Ġthe ĠnaÃ¯ve Ġapproach Ġof Ġfitting Ġan Ġindividual Ġembed ding Ġmodel Ġfor Ġeach Ġgroup Ġwould Ġtypically Ġsuffer Ġfrom Ġlack Ġof Ġdata âĢĶ especially Ġin Ġgroups Ġfor Ġwhich Ġfewer Ġobservations Ġare Ġavailable âĢĶ we Ġdevelop Ġtwo Ġmethods Ġthat Ġcan Ġshare Ġinformation Ġacross Ġgroups . </s> Figure ĠFIG REF 1 Ġillustrates Ġthe Ġkind Ġof Ġvariation Ġthat Ġwe Ġcan Ġcapture . ĠWe Ġfit Ġan Ġse fe Ġto ĠAr X iv Ġabstract s Ġgrouped Ġinto Ġdifferent Ġsections , Ġsuch Ġas Ġcomputer Ġscience Ġ( cs ), Ġquantitative Ġfinance Ġ( q - fin ), Ġand Ġnon linear Ġsciences Ġ( n lin ). Ġse fe Ġresults Ġin Ġa Ġper - section Ġembed ding Ġof Ġeach Ġterm Ġin Ġthe Ġvocabulary . ĠUsing Ġthe Ġfitted Ġembed d ings , Ġwe Ġillustrate Ġsimilar Ġwords Ġto Ġthe Ġword Ġ1 . 10 intelligence . ĠWe Ġcan Ġsee Ġthat Ġhow Ġ1 . 10 intelligence Ġis Ġused Ġvaries Ġby Ġfield : Ġin Ġcomputer Ġscience Ġthe Ġmost Ġsimilar Ġwords Ġinclude Ġ1 . 10 art ificial Ġand Ġ1 . 10 ai ; Ġin Ġfinance , Ġsimilar Ġwords Ġinclude Ġ1 . 10 abilities Ġand Ġ1 . 10 conscious ness . </s> In Ġmore Ġdetail , Ġembed ding Ġmethods Ġposit Ġtwo Ġrepresentation Ġvectors Ġfor Ġeach Ġterm Ġin Ġthe Ġvocabulary ; Ġan Ġembed ding Ġvector Ġand Ġa Ġcontext Ġvector . Ġ( We Ġuse Ġthe Ġlanguage Ġof Ġtext Ġfor Ġconc ret eness ; Ġas Ġwe Ġmentioned , Ġe fe Ġextend Ġto Ġother Ġtypes Ġof Ġdata .) ĠThe Ġidea Ġis Ġthat Ġthe Ġconditional Ġprobability Ġof Ġeach Ġobserved Ġword Ġdepends Ġon Ġthe Ġinteraction Ġbetween Ġthe Ġembed ding Ġvector Ġand Ġthe Ġcontext Ġvectors Ġof Ġthe Ġsurrounding Ġwords . ĠIn Ġse fe , Ġwe Ġposit Ġa Ġseparate Ġset Ġof Ġembed ding Ġvectors Ġfor Ġeach Ġgroup Ġbut Ġa Ġshared Ġset Ġof Ġcontext Ġvectors ; Ġthis Ġensures Ġthat Ġthe Ġembed ding Ġvectors Ġare Ġin Ġthe Ġsame Ġspace . </s> We Ġpropose Ġtwo Ġmethods Ġto Ġshare Ġstatistical Ġstrength Ġamong Ġthe Ġembed ding Ġvectors . ĠThe Ġfirst Ġapproach Ġis Ġbased Ġon Ġhierarchical Ġmodeling ĠB IB REF 13 Ġ, Ġwhich Ġassumes Ġthat Ġthe Ġgroup - specific Ġembed ding Ġrepresentations Ġare Ġtied Ġthrough Ġa Ġglobal Ġembed ding . ĠThe Ġsecond Ġapproach Ġis Ġbased Ġon Ġam ort ization ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, Ġwhich Ġconsiders Ġthat Ġthe Ġindividual Ġembed d ings Ġare Ġthe Ġoutput Ġof Ġa Ġdeter ministic Ġfunction Ġof Ġa Ġglobal Ġembed ding Ġrepresentation . ĠWe Ġuse Ġsto ch astic Ġoptimization Ġto Ġfit Ġlarge Ġdata Ġsets . </s> Our Ġwork Ġrelates Ġclosely Ġto Ġtwo Ġthreads Ġof Ġresearch Ġin Ġthe Ġembed ding Ġliterature . ĠOne Ġis Ġembed ding Ġmethods Ġthat Ġstudy Ġhow Ġlanguage Ġevolves Ġover Ġtime ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, ĠB IB REF 19 Ġ, ĠB IB REF 20 Ġ, ĠB IB REF 21 Ġ. ĠTime Ġcan Ġbe Ġthought Ġof Ġas Ġa Ġtype Ġof ĠâĢ ľ group âĢ Ŀ , Ġthough Ġwith Ġevolutionary Ġstructure Ġthat Ġwe Ġdo Ġnot Ġconsider . ĠThe Ġsecond Ġthread Ġis Ġmult ilingual Ġembed d ings ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġ, ĠB IB REF 24 Ġ, ĠB IB REF 25 Ġ; Ġour Ġapproach Ġis Ġdifferent Ġin Ġthat Ġmost Ġwords Ġappear Ġin Ġall Ġgroups Ġand Ġwe Ġare Ġinterested Ġin Ġthe Ġvariations Ġof Ġthe Ġembed d ings Ġacross Ġthose Ġgroups . </s> Our Ġcontributions Ġare Ġthus Ġas Ġfollows . ĠWe Ġintroduce Ġthe Ġse fe Ġmodel , Ġextending Ġe fe Ġto Ġgrouped Ġdata . ĠWe Ġpresent Ġtwo Ġtechniques Ġto Ġshare Ġstatistical Ġstrength Ġamong Ġthe Ġembed ding Ġvectors , Ġone Ġbased Ġon Ġhierarchical Ġmodeling Ġand Ġone Ġbased Ġon Ġam ort ization . ĠWe Ġcarry Ġout Ġa Ġthorough Ġexperimental Ġstudy Ġon Ġtwo Ġtext Ġdatabases , ĠAr X iv Ġpapers Ġby Ġsection Ġand ĠU . S . ĠCongressional Ġspeeches Ġby Ġhome Ġstate Ġand Ġpolitical Ġparty . ĠUsing ĠPo isson Ġembed d ings , Ġwe Ġstudy Ġmarket Ġbasket Ġdata Ġfrom Ġa Ġlarge Ġgrocery Ġstore , Ġgrouped Ġby Ġseason . ĠOn Ġall Ġthree Ġdata Ġsets , Ġse fe Ġoutper forms Ġe fe Ġin Ġterms Ġof Ġheld - out Ġlog - like lihood . ĠQual itatively , Ġwe Ġdemonstrate Ġhow Ġse fe Ġdiscovers Ġwhich Ġwords Ġare Ġused Ġmost Ġdifferently Ġacross ĠU . S . Ġstates Ġand Ġpolitical Ġparties , Ġand Ġshow Ġhow Ġword Ġusage Ġchanges Ġin Ġdifferent ĠAr X iv Ġdisciplines . </s> Model ĠDescription </s> In Ġthis Ġsection , Ġwe Ġdevelop Ġse fe , Ġa Ġmodel Ġthat Ġbuilds Ġon Ġe fe ĠB IB REF 10 Ġto Ġcapture Ġsemantic Ġvariations Ġacross Ġgroups Ġof Ġdata . ĠIn Ġembed ding Ġmodels , Ġwe Ġrepresent Ġeach Ġobject Ġ( e . g ., Ġa Ġword Ġin Ġtext , Ġor Ġan Ġitem Ġin Ġshopping Ġdata ) Ġusing Ġtwo Ġsets Ġof Ġvectors , Ġan Ġembed ding Ġvector Ġand Ġa Ġcontext Ġvector . ĠIn Ġthis Ġpaper , Ġwe Ġare Ġinterested Ġin Ġhow Ġthe Ġembed d ings Ġvary Ġacross Ġgroups Ġof Ġdata , Ġand Ġfor Ġeach Ġobject Ġwe Ġwant Ġto Ġlearn Ġa Ġseparate Ġembed ding Ġvector Ġfor Ġeach Ġgroup . ĠHaving Ġa Ġseparate Ġembed ding Ġfor Ġeach Ġgroup Ġallows Ġus Ġto Ġstudy Ġhow Ġthe Ġusage Ġof Ġa Ġword Ġlike Ġ1 . 10 intelligence Ġvaries Ġacross Ġcategories Ġof Ġthe ĠAr X iv , Ġor Ġwhich Ġwords Ġare Ġused Ġmost Ġdifferently Ġby ĠU . S . ĠSenators Ġdepending Ġon Ġwhich Ġstate Ġthey Ġare Ġfrom Ġand Ġwhether Ġthey Ġare ĠDemocrats Ġor ĠRepublicans . </s> The Ġse fe Ġmodel Ġextends Ġe fe Ġto Ġgrouped Ġdata , Ġby Ġhaving Ġthe Ġembed ding Ġvectors Ġbe Ġspecific Ġfor Ġeach Ġgroup , Ġwhile Ġsharing Ġthe Ġcontext Ġvectors Ġacross Ġall Ġgroups . ĠWe Ġreview Ġthe Ġe fe Ġmodel Ġin ĠSection ĠSEC REF 4 Ġ. ĠWe Ġthen Ġformal ize Ġthe Ġidea Ġof Ġsharing Ġthe Ġcontext Ġvectors Ġin ĠSection ĠSEC REF 8 Ġ, Ġwhere Ġwe Ġpresent Ġtwo Ġapproaches Ġto Ġbuild Ġa Ġhierarchical Ġstructure Ġover Ġthe Ġgroup - specific Ġembed d ings . </s> Background : ĠExp onential ĠFamily ĠEmb edd ings </s> In Ġexponential Ġfamily Ġembed d ings , Ġwe Ġhave Ġa Ġcollection Ġof Ġobjects , Ġand Ġour Ġgoal Ġis Ġto Ġlearn Ġa Ġvector Ġrepresentation Ġof Ġthese Ġobjects Ġbased Ġon Ġtheir Ġco - occ urrence Ġpatterns . </s> Let Ġus Ġconsider Ġa Ġdataset Ġrepresented Ġas Ġa Ġ( typically Ġsparse ) Ġmatrix ĠIN LINE FORM 0 Ġ, Ġwhere Ġcolumns Ġare Ġdat ap oint s Ġand Ġrows Ġare Ġobjects . ĠFor Ġexample , Ġin Ġtext , Ġeach Ġcolumn Ġcorresponds Ġto Ġa Ġlocation Ġin Ġthe Ġtext , Ġand Ġeach Ġentry ĠIN LINE FORM 1 Ġis Ġa Ġbinary Ġvariable Ġthat Ġindicates Ġwhether Ġword ĠIN LINE FORM 2 Ġappears Ġat Ġlocation ĠIN LINE FORM 3 Ġ. </s> In Ġe fe , Ġwe Ġrepresent Ġeach Ġobject ĠIN LINE FORM 0 Ġwith Ġtwo Ġsets Ġof Ġvectors , Ġembed d ings Ġvectors ĠIN LINE FORM 1 Ġand Ġcontext Ġvectors ĠIN LINE FORM 2 Ġ, Ġand Ġwe Ġposit Ġa Ġprobability Ġdistribution Ġof Ġdata Ġentries ĠIN LINE FORM 3 Ġin Ġwhich Ġthese Ġvectors Ġinteract . ĠThe Ġdefinition Ġof Ġthe Ġe fe Ġmodel Ġrequires Ġthree Ġingredients : Ġa Ġcontext , Ġa Ġconditional Ġexponential Ġfamily , Ġand Ġa Ġparameter Ġsharing Ġstructure . ĠWe Ġnext Ġdescribe Ġthese Ġthree Ġcomponents . </s> Exp onential Ġfamily Ġembed d ings Ġlearn Ġthe Ġvector Ġrepresentation Ġof Ġobjects Ġbased Ġon Ġthe Ġconditional Ġprobability Ġof Ġeach Ġobservation , Ġconditioned Ġon Ġthe Ġobservations Ġin Ġits Ġcontext . ĠThe Ġcontext ĠIN LINE FORM 0 Ġgives Ġthe Ġindices Ġof Ġthe Ġobservations Ġthat Ġappear Ġin Ġthe Ġconditional Ġprobability Ġdistribution Ġof ĠIN LINE FORM 1 Ġ. ĠThe Ġdefinition Ġof Ġthe Ġcontext Ġvaries Ġacross Ġapplications . ĠIn Ġtext , Ġit Ġcorresponds Ġto Ġthe Ġset Ġof Ġwords Ġin Ġa Ġfixed - size Ġwindow Ġcentered Ġat Ġlocation ĠIN LINE FORM 2 Ġ. </s> Given Ġthe Ġcontext ĠIN LINE FORM 0 Ġand Ġthe Ġcorresponding Ġobservations ĠIN LINE FORM 1 Ġindexed Ġby ĠIN LINE FORM 2 Ġ, Ġthe Ġdistribution Ġfor ĠIN LINE FORM 3 Ġis Ġin Ġthe Ġexponential Ġfamily , ĠDIS PLAY FORM 0 </s> with Ġsufficient Ġstatistics ĠIN LINE FORM 0 Ġand Ġnatural Ġparameter ĠIN LINE FORM 1 Ġ. ĠThe Ġparameter Ġvectors Ġinteract Ġin Ġthe Ġconditional Ġprobability Ġdistributions Ġof Ġeach Ġobservation ĠIN LINE FORM 2 Ġas Ġfollows . ĠThe Ġembed ding Ġvectors ĠIN LINE FORM 3 Ġand Ġthe Ġcontext Ġvectors ĠIN LINE FORM 4 Ġare Ġcombined Ġto Ġform Ġthe Ġnatural Ġparameter , ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġlink Ġfunction . ĠExp onential Ġfamily Ġembed d ings Ġcan Ġbe Ġunderstood Ġas Ġa Ġbank Ġof Ġgl m . ĠThe Ġcontext Ġvectors Ġare Ġcombined Ġto Ġgive Ġthe Ġcovari ates , Ġand Ġthe ĠâĢ ľ reg ression Ġcoefficients âĢ Ŀ Ġare Ġthe Ġembed ding Ġvectors . ĠIn ĠE q . ĠEQ REF 6 Ġ, Ġthe Ġlink Ġfunction ĠIN LINE FORM 1 Ġplays Ġthe Ġsame Ġrole Ġas Ġin Ġgl m Ġand Ġis Ġa Ġmodeling Ġchoice . ĠWe Ġuse Ġthe Ġidentity Ġlink Ġfunction . </s> The Ġthird Ġingredient Ġof Ġthe Ġe fe Ġmodel Ġis Ġthe Ġparameter Ġsharing Ġstructure , Ġwhich Ġindicates Ġhow Ġthe Ġembed ding Ġvectors Ġare Ġshared Ġacross Ġobservations . ĠIn Ġthe Ġstandard Ġe fe Ġmodel , Ġwe Ġuse ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġfor Ġall Ġcolumns Ġof ĠIN LINE FORM 2 Ġ. ĠThat Ġis , Ġeach Ġunique Ġobject ĠIN LINE FORM 3 Ġhas Ġa Ġshared Ġrepresentation Ġacross Ġall Ġinstances . </s> The Ġobjective Ġfunction . ĠIn Ġe fe , Ġwe Ġmaximize Ġthe Ġobjective Ġfunction , Ġwhich Ġis Ġgiven Ġby Ġthe Ġsum Ġof Ġthe Ġlog - cond itional Ġlikelihood s Ġin ĠE q . ĠEQ REF 5 Ġ. ĠIn Ġaddition , Ġwe Ġadd Ġan ĠIN LINE FORM 0 Ġ- regular ization Ġterm Ġ( we Ġuse Ġthe Ġnotation Ġof Ġthe Ġlog ĠGa ussian Ġpdf ) Ġover Ġthe Ġembed ding Ġand Ġcontext Ġvectors , Ġyielding ĠDIS PLAY FORM 0 </s> Note Ġthat Ġmaximizing Ġthe Ġregular ized Ġconditional Ġlikelihood Ġis Ġnot Ġequivalent Ġto Ġmaximum Ġa Ġposterior i . ĠRather , Ġit Ġis Ġsimilar Ġto Ġmaxim ization Ġof Ġthe Ġpseudo - like lihood Ġin Ġcondition ally Ġspecified Ġmodels ĠB IB REF 26 Ġ, ĠB IB REF 10 Ġ. </s> Struct ured ĠExp onential ĠFamily ĠEmb edd ings </s> Here , Ġwe Ġdescribe Ġthe Ġse fe Ġmodel Ġfor Ġgrouped Ġdata . ĠIn Ġtext , Ġsome Ġexamples Ġof Ġgrouped Ġdata Ġare ĠCongressional Ġspeeches Ġgrouped Ġinto Ġpolitical Ġparties Ġor Ġscientific Ġdocuments Ġgrouped Ġby Ġdiscipline . ĠOur Ġgoal Ġis Ġto Ġlearn Ġgroup - specific Ġembed d ings Ġfrom Ġdata Ġpartition ed Ġinto ĠIN LINE FORM 0 Ġgroups , Ġi . e ., Ġeach Ġinstance ĠIN LINE FORM 1 Ġis Ġassociated Ġwith Ġa Ġgroup ĠIN LINE FORM 2 Ġ. ĠThe Ġse fe Ġmodel Ġextends Ġe fe Ġto Ġlearn Ġa Ġseparate Ġset Ġof Ġembed ding Ġvectors Ġfor Ġeach Ġgroup . </s> To Ġbuild Ġthe Ġse fe Ġmodel , Ġwe Ġimpose Ġa Ġparticular Ġparameter Ġsharing Ġstructure Ġover Ġthe Ġset Ġof Ġembed ding Ġand Ġcontext Ġvectors . ĠWe Ġposit Ġa Ġstructured Ġmodel Ġin Ġwhich Ġthe Ġcontext Ġvectors Ġare Ġshared Ġacross Ġgroups , Ġi . e ., ĠIN LINE FORM 0 Ġ( as Ġin Ġthe Ġstandard Ġe fe Ġmodel ), Ġbut Ġthe Ġembed ding Ġvectors Ġare Ġonly Ġshared Ġat Ġthe Ġgroup Ġlevel , Ġi . e ., Ġfor Ġan Ġobservation ĠIN LINE FORM 1 Ġbelonging Ġto Ġgroup ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ. ĠHere , ĠIN LINE FORM 4 Ġdenotes Ġthe Ġembed ding Ġvector Ġcorresponding Ġto Ġgroup ĠIN LINE FORM 5 Ġ. ĠWe Ġshow Ġa Ġgraphical Ġrepresentation Ġof Ġthe Ġse fe Ġin ĠFigure ĠFIG REF 1 Ġ. </s> Sh aring Ġthe Ġcontext Ġvectors ĠIN LINE FORM 0 Ġhas Ġtwo Ġadvantages . ĠFirst , Ġthe Ġshared Ġstructure Ġreduces Ġthe Ġnumber Ġof Ġparameters , Ġwhile Ġthe Ġresulting Ġse fe Ġmodel Ġis Ġstill Ġflexible Ġto Ġcapture Ġhow Ġdifferently Ġwords Ġare Ġused Ġacross Ġdifferent Ġgroups , Ġas ĠIN LINE FORM 1 Ġis Ġallowed Ġto Ġvary . ĠSecond , Ġit Ġhas Ġthe Ġimportant Ġeffect Ġof Ġun iting Ġall Ġembed ding Ġparameters Ġin Ġthe Ġsame Ġspace , Ġas Ġthe Ġgroup - specific Ġvectors ĠIN LINE FORM 4 Ġneed Ġto Ġagree Ġwith Ġthe Ġcomponents Ġof ĠIN LINE FORM 5 Ġ. ĠWhile Ġone Ġcould Ġlearn Ġa Ġseparate Ġembed ding Ġmodel Ġfor Ġeach Ġgroup , Ġas Ġhas Ġbeen Ġdone Ġfor Ġtext Ġgrouped Ġinto Ġtime Ġslices ĠB IB REF 16 Ġ, ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġthis Ġapproach Ġwould Ġrequire Ġad - h oc Ġpost processing Ġsteps Ġto Ġalign Ġthe Ġembed d ings . </s> When Ġthere Ġare ĠIN LINE FORM 0 Ġgroups , Ġthe Ġse fe Ġmodel Ġhas ĠIN LINE FORM 1 Ġtimes Ġas Ġmany Ġembed ding Ġvectors Ġthan Ġthe Ġstandard Ġembed ding Ġmodel . ĠThis Ġmay Ġcomplicate Ġinf erences Ġabout Ġthe Ġgroup - specific Ġvectors , Ġespecially Ġfor Ġgroups Ġwith Ġless Ġdata . ĠAdditionally , Ġan Ġobject ĠIN LINE FORM 2 Ġmay Ġappear Ġwith Ġvery Ġlow Ġfrequency Ġin Ġa Ġparticular Ġgroup . ĠThus , Ġthe ĠnaÃ¯ve Ġapproach Ġfor Ġbuilding Ġthe Ġse fe Ġmodel Ġwithout Ġadditional Ġstructure Ġmay Ġbe Ġdetrimental Ġfor Ġthe Ġquality Ġof Ġthe Ġembed d ings , Ġespecially Ġfor Ġsmall - sized Ġgroups . ĠTo Ġaddress Ġthis Ġproblem , Ġwe Ġpropose Ġtwo Ġdifferent Ġmethods Ġto Ġtie Ġthe Ġindividual ĠIN LINE FORM 3 Ġtogether , Ġsharing Ġstatistical Ġstrength Ġamong Ġthem . ĠThe Ġfirst Ġapproach Ġconsists Ġin Ġa Ġhierarchical Ġembed ding Ġstructure . ĠThe Ġsecond Ġapproach Ġis Ġbased Ġon Ġam ort ization . ĠIn Ġboth Ġmethods , Ġwe Ġintroduce Ġa Ġset Ġof Ġglobal Ġembed ding Ġvectors ĠIN LINE FORM 4 Ġ, Ġand Ġimpose Ġa Ġparticular Ġstructure Ġto Ġgenerate ĠIN LINE FORM 5 Ġfrom ĠIN LINE FORM 6 Ġ. </s> H ier arch ical Ġembed ding Ġstructure . ĠHere , Ġwe Ġimpose Ġa Ġhierarchical Ġstructure Ġthat Ġallows Ġsharing Ġstatistical Ġstrength Ġamong Ġthe Ġper - group Ġvariables . ĠFor Ġthat , Ġwe Ġassume Ġthat Ġeach ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġa Ġfixed Ġhyper param eter . ĠThus , Ġwe Ġreplace Ġthe Ġe fe Ġobjective Ġfunction Ġin ĠE q . ĠEQ REF 7 Ġwith ĠDIS PLAY FORM 0 </s> where Ġthe ĠIN LINE FORM 0 Ġ- regular ization Ġterm Ġnow Ġapplies Ġonly Ġon ĠIN LINE FORM 1 Ġand Ġthe Ġglobal Ġvectors ĠIN LINE FORM 2 Ġ. </s> F itting Ġthe Ġhierarchical Ġmodel Ġinvolves Ġmaximizing ĠE q . ĠEQ REF 11 Ġwith Ġrespect Ġto ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ. ĠWe Ġnote Ġthat Ġwe Ġhave Ġnot Ġreduced Ġthe Ġnumber Ġof Ġparameters Ġto Ġbe Ġinferred ; Ġrather , Ġwe Ġtie Ġthem Ġtogether Ġthrough Ġa Ġcommon Ġprior Ġdistribution . ĠWe Ġuse Ġsto ch astic Ġgradient Ġascent Ġto Ġmaximize ĠE q . ĠEQ REF 11 Ġ. </s> Am ort ization . ĠThe Ġidea Ġof Ġam ort ization Ġhas Ġbeen Ġapplied Ġin Ġthe Ġliterature Ġto Ġdevelop Ġam ort ized Ġinference Ġalgorithms ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠThe Ġmain Ġinsight Ġbehind Ġam ort ization Ġis Ġto Ġreuse Ġinf erences Ġabout Ġpast Ġexperiences Ġwhen Ġpresented Ġwith Ġa Ġnew Ġtask , Ġleveraging Ġthe Ġaccumulated Ġknowledge Ġto Ġquickly Ġsolve Ġthe Ġnew Ġproblem . ĠHere , Ġwe Ġuse Ġam ort ization Ġto Ġcontrol Ġthe Ġnumber Ġof Ġparameters Ġof Ġthe Ġse fe Ġmodel . ĠIn Ġparticular , Ġwe Ġset Ġthe Ġper - group Ġembed d ings ĠIN LINE FORM 0 Ġto Ġbe Ġthe Ġoutput Ġof Ġa Ġdeter ministic Ġfunction Ġof Ġthe Ġglobal Ġembed ding Ġvectors , ĠIN LINE FORM 1 Ġ. ĠWe Ġuse Ġa Ġdifferent Ġfunction ĠIN LINE FORM 2 Ġfor Ġeach Ġgroup ĠIN LINE FORM 3 Ġ, Ġand Ġwe Ġparameter ize Ġthem Ġusing Ġneural Ġnetworks , Ġsimilarly Ġto Ġother Ġworks Ġon Ġam ort ized Ġinference ĠB IB REF 28 Ġ, ĠB IB REF 29 Ġ, ĠB IB REF 30 Ġ, ĠB IB REF 31 Ġ. ĠUnlike Ġstandard Ġuses Ġof Ġam ort ized Ġinference , Ġin Ġse fe Ġthe Ġinput Ġto Ġthe Ġfunctions ĠIN LINE FORM 4 Ġis Ġunob served Ġand Ġmust Ġbe Ġestimated Ġtogether Ġwith Ġthe Ġparameters Ġof Ġthe Ġfunctions ĠIN LINE FORM 5 Ġ. </s> Depending Ġon Ġthe Ġarchitecture Ġof Ġthe Ġneural Ġnetworks , Ġthe Ġam ort ization Ġcan Ġsignificantly Ġreduce Ġthe Ġnumber Ġof Ġparameters Ġin Ġthe Ġmodel Ġ( as Ġcompared Ġto Ġthe Ġnon - am ort ized Ġmodel ), Ġwhile Ġstill Ġhaving Ġthe Ġflexibility Ġto Ġmodel Ġdifferent Ġembed ding Ġvectors Ġfor Ġeach Ġgroup . ĠThe Ġnumber Ġof Ġparameters Ġin Ġthe Ġse fe Ġmodel Ġis ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġnumber Ġof Ġgroups , ĠIN LINE FORM 2 Ġis Ġthe Ġdimension ality Ġof Ġthe Ġembed ding Ġvectors , Ġand ĠIN LINE FORM 3 Ġis Ġthe Ġnumber Ġof Ġobjects Ġ( e . g ., Ġthe Ġvocabulary Ġsize ). ĠWith Ġam ort ization , Ġwe Ġreduce Ġthe Ġnumber Ġof Ġparameters Ġto ĠIN LINE FORM 4 Ġ, Ġwhere ĠIN LINE FORM 5 Ġis Ġthe Ġnumber Ġof Ġparameters Ġof Ġthe Ġneural Ġnetwork . ĠSince Ġtypically ĠIN LINE FORM 6 Ġ, Ġthis Ġcorresponds Ġto Ġa Ġsignificant Ġreduction Ġin Ġthe Ġnumber Ġof Ġparameters , Ġeven Ġwhen ĠIN LINE FORM 7 Ġscales Ġlin early Ġwith ĠIN LINE FORM 8 Ġ. </s> In Ġthe Ġam ort ized Ġse fe Ġmodel , Ġwe Ġneed Ġto Ġintroduce Ġa Ġnew Ġset Ġof Ġparameters ĠIN LINE FORM 0 Ġfor Ġeach Ġgroup ĠIN LINE FORM 1 Ġ, Ġcorresponding Ġto Ġthe Ġneural Ġnetwork Ġparameters . ĠGiven Ġthese , Ġthe Ġgroup - specific Ġembed ding Ġvectors ĠIN LINE FORM 2 Ġare Ġobtained Ġas ĠDIS PLAY FORM 0 </s> We Ġcompare Ġtwo Ġarchitectures Ġfor Ġthe Ġfunction ĠIN LINE FORM 0 Ġ: Ġfully Ġconnected Ġfeed - forward Ġneural Ġnetworks Ġand Ġresidual Ġnetworks ĠB IB REF 32 Ġ. ĠFor Ġboth , Ġwe Ġconsider Ġone Ġhidden Ġlayer Ġwith ĠIN LINE FORM 1 Ġunits . ĠHence , Ġthe Ġnetwork Ġparameters ĠIN LINE FORM 2 Ġare Ġtwo Ġweight Ġmat rices , ĠDIS PLAY FORM 0 </s> i . e ., ĠIN LINE FORM 0 Ġparameters . ĠThe Ġneural Ġnetwork Ġtakes Ġas Ġinput Ġthe Ġglobal Ġembed ding Ġvector ĠIN LINE FORM 1 Ġ, Ġand Ġit Ġoutputs Ġthe Ġgroup - specific Ġembed ding Ġvectors ĠIN LINE FORM 2 Ġ. ĠThe Ġmathematical Ġexpression Ġfor ĠIN LINE FORM 3 Ġfor Ġa Ġfeed - forward Ġneural Ġnetwork Ġand Ġa Ġresidual Ġnetwork Ġis Ġrespectively Ġgiven Ġby ĠDIS PLAY FORM 0 </s> where Ġwe Ġhave Ġconsidered Ġthe Ġhyper b olic Ġtang ent Ġnon linear ity . ĠThe Ġmain Ġdifference Ġbetween Ġboth Ġnetwork Ġarchitectures Ġis Ġthat Ġthe Ġresidual Ġnetwork Ġfocuses Ġon Ġmodeling Ġhow Ġthe Ġgroup - specific Ġembed ding Ġvectors ĠIN LINE FORM 0 Ġdiffer Ġfrom Ġthe Ġglobal Ġvectors ĠIN LINE FORM 1 Ġ. ĠThat Ġis , Ġif Ġall Ġweights Ġwere Ġset Ġto Ġ0 , Ġthe Ġfeed - forward Ġnetwork Ġwould Ġoutput Ġ0 , Ġwhile Ġthe Ġresidual Ġnetwork Ġwould Ġoutput Ġthe Ġglobal Ġvector ĠIN LINE FORM 2 Ġfor Ġall Ġgroups . </s> The Ġobjective Ġfunction Ġunder Ġam ort ization Ġis Ġgiven Ġby ĠDIS PLAY FORM 0 </s> We Ġmaximize Ġthis Ġobjective Ġwith Ġrespect Ġto ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġusing Ġsto ch astic Ġgradient Ġascent . ĠWe Ġimplement Ġthe Ġhierarchical Ġand Ġam ort ized Ġse fe Ġmodels Ġin ĠT ensor Flow ĠB IB REF 33 Ġ, Ġwhich Ġallows Ġus Ġto Ġleverage Ġautomatic Ġdifferentiation . </s> Example : Ġstructured ĠBern ou lli Ġembed d ings Ġfor Ġgrouped Ġtext Ġdata . ĠHere , Ġwe Ġconsider Ġa Ġset Ġof Ġdocuments Ġbroken Ġdown Ġinto Ġgroups , Ġsuch Ġas Ġpolitical Ġaffili ations Ġor Ġscientific Ġdisciplines . ĠWe Ġcan Ġrepresent Ġthe Ġdata Ġas Ġa Ġbinary Ġmatrix ĠIN LINE FORM 0 Ġand Ġa Ġset Ġof Ġgroup Ġindicators ĠIN LINE FORM 1 Ġ. ĠSince Ġonly Ġone Ġword Ġcan Ġappear Ġin Ġa Ġcertain Ġposition ĠIN LINE FORM 2 Ġ, Ġthe Ġmatrix ĠIN LINE FORM 3 Ġcontains Ġone Ġnon - zero Ġelement Ġper Ġcolumn . ĠIn Ġembed ding Ġmodels , Ġwe Ġignore Ġthis Ġone - hot Ġconstraint Ġfor Ġcomputational Ġefficiency , Ġand Ġconsider Ġthat Ġthe Ġobservations Ġare Ġgenerated Ġfollowing Ġa Ġset Ġof Ġconditional ĠBern ou lli Ġdistributions ĠB IB REF 2 Ġ, ĠB IB REF 10 Ġ. ĠGiven Ġthat Ġmost Ġof Ġthe Ġentries Ġin ĠIN LINE FORM 4 Ġare Ġzero , Ġembed ding Ġmodels Ġtypically Ġdown we igh Ġthe Ġcontribution Ġof Ġthe Ġz eros Ġto Ġthe Ġobjective Ġfunction . ĠB IB REF 2 Ġuse Ġnegative Ġsampling , Ġwhich Ġconsists Ġin Ġrandomly Ġchoosing Ġa Ġsubset Ġof Ġthe Ġzero Ġobservations . ĠThis Ġcorresponds Ġto Ġa Ġbiased Ġestimate Ġof Ġthe Ġgradient Ġin Ġa ĠBern ou lli Ġexponential Ġfamily Ġembed ding Ġmodel ĠB IB REF 10 Ġ. </s> The Ġcontext ĠIN LINE FORM 0 Ġis Ġgiven Ġat Ġeach Ġposition ĠIN LINE FORM 1 Ġby Ġthe Ġset Ġof Ġsurrounding Ġwords Ġin Ġthe Ġdocument , Ġaccording Ġto Ġa Ġfixed - size Ġwindow . </s> Example : Ġstructured ĠPo isson Ġembed d ings Ġfor Ġgrouped Ġshopping Ġdata . Ġe fe Ġand Ġse fe Ġextend Ġto Ġapplications Ġbeyond Ġtext Ġand Ġwe Ġuse Ġse fe Ġto Ġmodel Ġsupermarket Ġpurchases Ġbroken Ġdown Ġby Ġmonth . ĠFor Ġeach Ġmarket Ġbasket ĠIN LINE FORM 0 Ġ, Ġwe Ġhave Ġaccess Ġto Ġthe Ġmonth ĠIN LINE FORM 1 Ġin Ġwhich Ġthat Ġshopping Ġtrip Ġhappened . ĠNow , Ġthe Ġrows Ġof Ġthe Ġdata Ġmatrix ĠIN LINE FORM 2 Ġindex Ġitems , Ġwhile Ġcolumns Ġindex Ġshopping Ġtrips . ĠEach Ġelement ĠIN LINE FORM 3 Ġdenotes Ġthe Ġnumber Ġof Ġunits Ġof Ġitem ĠIN LINE FORM 4 Ġpurchased Ġat Ġtrip ĠIN LINE FORM 5 Ġ. ĠUnlike Ġtext , Ġeach Ġcolumn Ġof ĠIN LINE FORM 6 Ġmay Ġcontain Ġmore Ġthan Ġone Ġnon - zero Ġelement . ĠThe Ġcontext ĠIN LINE FORM 7 Ġcorresponds Ġto Ġthe Ġset Ġof Ġitems Ġpurchased Ġin Ġtrip ĠIN LINE FORM 8 Ġ, Ġexcluding ĠIN LINE FORM 9 Ġ. </s> In Ġthis Ġcase , Ġwe Ġuse Ġthe ĠPo isson Ġconditional Ġdistribution , Ġwhich Ġis Ġmore Ġappropriate Ġfor Ġcount Ġdata . ĠIn ĠPo isson Ġse fe , Ġwe Ġalso Ġdown we igh Ġthe Ġcontribution Ġof Ġthe Ġz eros Ġin Ġthe Ġobjective Ġfunction , Ġwhich Ġprovides Ġbetter Ġresults Ġbecause Ġit Ġallows Ġthe Ġinference Ġto Ġfocus Ġon Ġthe Ġpositive Ġsignal Ġof Ġthe Ġactual Ġpurchases ĠB IB REF 10 Ġ, ĠB IB REF 2 Ġ. </s> E mp ir ical ĠStudy </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġexperimental Ġstudy . ĠWe Ġfit Ġthe Ġse fe Ġmodel Ġon Ġthree Ġdatasets Ġand Ġcompare Ġit Ġagainst Ġthe Ġe fe ĠB IB REF 10 Ġ. ĠOur Ġquantitative Ġresults Ġshow Ġthat Ġsharing Ġthe Ġcontext Ġvectors Ġprovides Ġbetter Ġresults , Ġand Ġthat Ġam ort ization Ġand Ġhierarchical Ġstructure Ġgive Ġfurther Ġimprovements . </s> Data . ĠWe Ġapply Ġthe Ġse fe Ġon Ġthree Ġdatasets : ĠAr X iv Ġpapers , ĠU . S . ĠSenate Ġspeeches , Ġand Ġpurchases Ġon Ġsupermarket Ġgrocery Ġshopping Ġdata . ĠWe Ġdescribe Ġthese Ġdatasets Ġbelow , Ġand Ġwe Ġprovide Ġa Ġsummary Ġof Ġthe Ġdatasets Ġin ĠTable ĠT AB REF 17 Ġ. </s> Ar X iv Ġpapers : ĠThis Ġdataset Ġcontains Ġthe Ġabstract s Ġof Ġpapers Ġpublished Ġon Ġthe ĠAr X iv Ġunder Ġthe Ġ19 Ġdifferent Ġtags Ġbetween ĠApril Ġ2007 Ġand ĠJune Ġ2015 . ĠWe Ġtreat Ġeach Ġtag Ġas Ġa Ġgroup Ġand Ġfit Ġse fe Ġwith Ġthe Ġgoal Ġof Ġuncover ing Ġwhich Ġwords Ġhave Ġthe Ġstrongest Ġshift Ġin Ġusage . ĠWe Ġsplit Ġthe Ġabstract s Ġinto Ġtraining , Ġvalidation , Ġand Ġtest Ġsets , Ġwith Ġproportions Ġof ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, Ġand ĠIN LINE FORM 2 Ġ, Ġrespectively . </s> Senate Ġspeeches : ĠThis Ġdataset Ġcontains ĠU . S . ĠSenate Ġspeeches Ġfrom Ġ1994 Ġto Ġmid Ġ2009 . ĠIn Ġcontrast Ġto Ġthe ĠAr X iv Ġcollection , Ġit Ġis Ġa Ġtranscript Ġof Ġspoken Ġlanguage . ĠWe Ġgroup Ġthe Ġdata Ġinto Ġstate Ġof Ġorigin Ġof Ġthe Ġspeaker Ġand Ġhis Ġor Ġher Ġparty Ġaffiliation . ĠOnly Ġaffili ations Ġwith Ġthe ĠRepublican Ġand ĠDemocratic ĠParty Ġare Ġconsidered . ĠAs Ġa Ġresult , Ġthere Ġare Ġ83 Ġgroups Ġ( Republicans Ġfrom ĠAlabama , ĠDemocrats Ġfrom ĠAlabama , ĠRepublicans Ġfrom ĠArkansas , Ġetc .). ĠSome Ġof Ġthe Ġstate / party Ġcombinations Ġare Ġnot Ġavailable Ġin Ġthe Ġdata , Ġas Ġsome Ġof Ġthe Ġ50 Ġstates Ġhave Ġonly Ġhad ĠSenators Ġwith Ġthe Ġsame Ġparty Ġaffiliation . ĠWe Ġsplit Ġthe Ġspeeches Ġinto Ġtraining Ġ( ĠIN LINE FORM 0 Ġ), Ġvalidation Ġ( ĠIN LINE FORM 1 Ġ), Ġand Ġtesting Ġ( ĠIN LINE FORM 2 Ġ). </s> G roc ery Ġshopping Ġdata : ĠThis Ġdataset Ġcontains Ġthe Ġpurchases Ġof ĠIN LINE FORM 0 Ġcustomers . ĠThe Ġdata Ġcovers Ġa Ġperiod Ġof Ġ97 Ġweeks . ĠAfter Ġremoving Ġlow - frequency Ġitems , Ġthe Ġdata Ġcontains ĠIN LINE FORM 1 Ġunique Ġitems Ġat Ġthe Ġ1 . 10 up c Ġ( Universal ĠProduct ĠCode ) Ġlevel . ĠWe Ġsplit Ġthe Ġdata Ġinto Ġa Ġtraining , Ġtest , Ġand Ġvalidation Ġsets , Ġwith Ġproportions Ġof ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ, Ġand ĠIN LINE FORM 4 Ġ, Ġrespectively . ĠThe Ġtraining Ġdata Ġcontains ĠIN LINE FORM 5 Ġshopping Ġtrips Ġand ĠIN LINE FORM 6 Ġpurchases Ġin Ġtotal . </s> For Ġthe Ġtext Ġcorpor a , Ġwe Ġfix Ġthe Ġvocabulary Ġto Ġthe Ġ15 k Ġmost Ġfrequent Ġterms Ġand Ġremove Ġall Ġwords Ġthat Ġare Ġnot Ġin Ġthe Ġvocabulary . ĠFollowing ĠB IB REF 2 Ġ, Ġwe Ġadditionally Ġremove Ġeach Ġword Ġwith Ġprobability ĠIN LINE FORM 0 Ġ, Ġwhere ĠIN LINE FORM 1 Ġis Ġthe Ġword Ġfrequency . ĠThis Ġdowns amples Ġespecially Ġthe Ġfrequent Ġwords Ġand Ġspeeds Ġup Ġtraining . Ġ( S izes Ġreported Ġin ĠTable ĠT AB REF 17 Ġare Ġthe Ġnumber Ġof Ġwords Ġremaining Ġafter Ġpre processing .) </s> Mod els . ĠOur Ġgoal Ġis Ġto Ġfit Ġthe Ġse fe Ġmodel Ġon Ġthese Ġdatasets . ĠFor Ġthe Ġtext Ġdata , Ġwe Ġuse Ġthe ĠBern ou lli Ġdistribution Ġas Ġthe Ġconditional Ġexponential Ġfamily , Ġwhile Ġfor Ġthe Ġshopping Ġdata Ġwe Ġuse Ġthe ĠPo isson Ġdistribution , Ġwhich Ġis Ġmore Ġappropriate Ġfor Ġcount Ġdata . </s> On Ġeach Ġdataset , Ġwe Ġcompare Ġfour Ġapproaches Ġbased Ġon Ġse fe Ġwith Ġtwo Ġe fe ĠB IB REF 10 Ġbas elines . ĠAll Ġare Ġfit Ġusing Ġs gd ĠB IB REF 34 Ġ. ĠIn Ġparticular , Ġwe Ġcompare Ġthe Ġfollowing Ġmethods :