By using Euclidean distance computed between the context vector representations of the equations
ABSTRACTIVE
Similar words were ranked by computing Cosine distance between the embedding vector ( INLINEFORM0 ) representation of the query equation and the context vector representation of the words ( INLINEFORM1 ). Similar equations were discovered using Euclidean distance computed between the context vector representations of the equations ( INLINEFORM2 ). We give additional example results in Appendix B.
EXTRACTIVE
In addition to words, EqEmb models can capture the semantic similarity between equations in the collection. We performed qualitative analysis of the model performance using all discovered equations across the 4 collection. Table TABREF24 shows the query equation used in the previous analysis and its 5 most similar equations discovered using EqEmb-U. For qualitative comparisons across the other embedding models, in Appendix A we provide results over the same query using CBOW, PV-DM, GloVe and EqEmb. In Appendix A reader should notice the difference in performance between EqEmb-U and EqEmb compared to existing embedding models which fail to discover semantically similar equations. tab:irexample1,tab:nlpexample2 show two additional example equation and its 5 most similar equations and words discovered using the EqEmb model. Similar words were ranked by computing Cosine distance between the embedding vector ( INLINEFORM0 ) representation of the query equation and the context vector representation of the words ( INLINEFORM1 ). Similar equations were discovered using Euclidean distance computed between the context vector representations of the equations ( INLINEFORM2 ). We give additional example results in Appendix B.
In addition to words, EqEmb models can capture the semantic similarity between equations in the collection. We performed qualitative analysis of the model performance using all discovered equations across the 4 collection. Table TABREF24 shows the query equation used in the previous analysis and its 5 most similar equations discovered using EqEmb-U. For qualitative comparisons across the other embedding models, in Appendix A we provide results over the same query using CBOW, PV-DM, GloVe and EqEmb. In Appendix A reader should notice the difference in performance between EqEmb-U and EqEmb compared to existing embedding models which fail to discover semantically similar equations. tab:irexample1,tab:nlpexample2 show two additional example equation and its 5 most similar equations and words discovered using the EqEmb model. Similar words were ranked by computing Cosine distance between the embedding vector ( INLINEFORM0 ) representation of the query equation and the context vector representation of the words ( INLINEFORM1 ). Similar equations were discovered using Euclidean distance computed between the context vector representations of the equations ( INLINEFORM2 ). We give additional example results in Appendix B.
1803.09123
Introduction </s> Equ ations Ġare Ġan Ġimportant Ġpart Ġof Ġscientific Ġarticles , Ġbut Ġmany Ġexisting Ġmachine Ġlearning Ġmethods Ġdo Ġnot Ġeasily Ġhandle Ġthem . ĠThey Ġare Ġchallenging Ġto Ġwork Ġwith Ġbecause Ġeach Ġis Ġunique Ġor Ġnearly Ġunique ; Ġmost Ġequations Ġoccur Ġonly Ġonce . ĠAn Ġautomatic Ġunderstanding Ġof Ġequations , Ġhowever , Ġwould Ġsignificantly Ġbenefit Ġmethods Ġfor Ġanalyzing Ġscientific Ġliterature . ĠUseful Ġrepresentations Ġof Ġequations Ġcan Ġhelp Ġdraw Ġconnections Ġbetween Ġarticles , Ġimprove Ġretrieval Ġof Ġscientific Ġtexts , Ġand Ġhelp Ġcreate Ġtools Ġfor Ġexploring Ġand Ġnavigating Ġscientific Ġliterature . </s> In Ġthis Ġpaper Ġwe Ġpropose Ġequation Ġembed d ings Ġ( E q Emb ), Ġan Ġun super vised Ġapproach Ġfor Ġlearning Ġdistributed Ġrepresentations Ġof Ġequations . ĠThe Ġidea Ġis Ġto Ġtreat Ġthe Ġequation Ġas Ġa Ġ" sing leton Ġword ," Ġone Ġthat Ġappears Ġonce Ġbut Ġthat Ġappears Ġin Ġthe Ġcontext Ġof Ġother Ġwords . ĠThe Ġsurrounding Ġtext Ġof Ġthe Ġequation âĢĶ and Ġin Ġparticular , Ġthe Ġdistributed Ġrepresentations Ġof Ġthat Ġtext âĢĶ prov ides Ġthe Ġdata Ġwe Ġneed Ġto Ġdevelop Ġa Ġuseful Ġrepresentation Ġof Ġthe Ġequation . </s> Figure ĠFIG REF 1 Ġillustrates Ġour Ġapproach . ĠOn Ġthe Ġleft Ġis Ġan Ġarticle Ġsnippet ĠB IB REF 0 Ġ. ĠHigh light ed Ġin Ġorange Ġis Ġan Ġequation ; Ġin Ġthis Ġexample Ġit Ġrepresents Ġa Ġneural Ġnetwork Ġlayer . ĠWe Ġnote Ġthat Ġthis Ġparticular Ġequation Ġ( in Ġthis Ġform Ġand Ġwith Ġthis Ġnotation ) Ġonly Ġoccurs Ġonce Ġin Ġthe Ġcollection Ġof Ġarticles Ġ( from Ġar X iv ). ĠThe Ġrepresentations Ġof Ġthe Ġsurrounding Ġtext , Ġhowever , Ġprovide Ġa Ġmeaningful Ġcontext Ġfor Ġthe Ġequation . ĠThose Ġwords Ġallow Ġus Ġto Ġlearn Ġits Ġembed ding , Ġspecifically Ġas Ġa Ġ" word " Ġwhich Ġappears Ġin Ġthe Ġcontext Ġof Ġits Ġsurroundings . ĠThe Ġresulting Ġrepresentation , Ġwhen Ġcompared Ġto Ġother Ġequations ' Ġrepresentations Ġand Ġword Ġrepresentations , Ġhelps Ġfind Ġboth Ġrelated Ġequations Ġand Ġrelated Ġwords . ĠThese Ġare Ġillustrated Ġon Ġthe Ġright . </s> E q Em bs Ġbuild Ġon Ġexponential Ġfamily Ġembed d ings ĠB IB REF 1 Ġto Ġinclude Ġequations Ġas Ġsingle ton Ġobservations Ġand Ġto Ġmodel Ġequation Ġelements Ġsuch Ġas Ġvariables , Ġsymbols Ġand Ġoperators . ĠExp onential Ġfamily Ġembed d ings , Ġlike Ġall Ġembed ding Ġmethods , Ġdefine Ġa Ġcontext Ġof Ġeach Ġword . ĠIn Ġour Ġinitial ĠE q Emb , Ġthe Ġcontext Ġfor Ġthe Ġwords Ġis Ġa Ġsmall Ġwindow , Ġsuch Ġas Ġfour Ġor Ġeight Ġwords , Ġbut Ġthe Ġcontext Ġof Ġan Ġequation Ġis Ġa Ġlarger Ġwindow , Ġsuch Ġas Ġsixteen Ġwords . ĠUsing Ġthese Ġtwo Ġtypes Ġof Ġcontexts Ġtogether Ġfinds Ġmeaningful Ġrepresentations Ġof Ġwords Ġand Ġequations . ĠIn Ġthe Ġnext ĠE q Emb , Ġwhich Ġbuilds Ġon Ġthe Ġfirst , Ġwe Ġconsider Ġequations Ġto Ġbe Ġsentences Ġconsisting Ġof Ġequation Ġunits , Ġi . e ., Ġvariables , Ġsymbols , Ġand Ġoperators . ĠEqu ation Ġunits Ġhelp Ġmodel Ġequations Ġacross Ġtwo Ġtypes Ġof Ġcontext âĢĶ over Ġthe Ġsurrounding Ġunits Ġand Ġover Ġthe Ġsurrounding Ġwords . </s> We Ġstudied ĠE q Em bs Ġon Ġfour Ġcollections Ġof Ġscientific Ġarticles Ġfrom Ġthe Ġar X iv , Ġcovering Ġfour Ġcomputer Ġscience Ġdomains : Ġnatural Ġlanguage Ġprocessing Ġ( N LP ), Ġinformation Ġretrieval Ġ( IR ), Ġartificial Ġintelligence Ġ( AI ) Ġand Ġmachine Ġlearning Ġ( ML ). ĠWe Ġfound Ġthat ĠE q Em bs Ġprovide Ġmore Ġefficient Ġmodeling Ġthan Ġexisting Ġword Ġembed ding Ġmethods . ĠWe Ġfurther Ġcarried Ġout Ġan Ġexplor atory Ġanalysis Ġof Ġa Ġlarge Ġset Ġof ĠIN LINE FORM 0 Ġ87 k Ġequations . ĠWe Ġfound Ġthat ĠE q Em bs Ġprovide Ġbetter Ġmodels Ġwhen Ġcompared Ġto Ġexisting Ġword Ġembed ding Ġapproaches . ĠE q Em bs Ġalso Ġprovide Ġcoherent Ġsemantic Ġrepresentations Ġof Ġequations Ġand Ġcan Ġcapture Ġsemantic Ġsimilarity Ġto Ġother Ġequations Ġand Ġto Ġwords . </s> Related ĠWork </s> Word Ġembed d ings Ġwere Ġfirst Ġintroduced Ġin ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġand Ġthere Ġhave Ġbeen Ġmany Ġvariants ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠCommon Ġfor Ġall Ġof Ġthem Ġis Ġthe Ġidea Ġthat Ġwords Ġcan Ġbe Ġrepresented Ġby Ġlatent Ġfeature Ġvectors . ĠThese Ġfeature Ġvectors Ġare Ġoptimized Ġto Ġmaximize Ġthe Ġconditional Ġprobability Ġof Ġthe Ġdataset . ĠRecently ĠB IB REF 1 Ġextended Ġthe Ġidea Ġof Ġword Ġembed d ings Ġto Ġother Ġtypes Ġof Ġdata . ĠE q Emb Ġexpand Ġthe Ġidea Ġof Ġword Ġembed d ings Ġto Ġa Ġnew Ġtype Ġof Ġdata Ġpoints ĠâĢĵ Ġequations . </s> There Ġhave Ġbeen Ġdifferent Ġproposed Ġapproaches Ġfor Ġrepresenting Ġmathematical Ġequations . ĠB IB REF 8 Ġintroduced Ġthe Ġsymbol Ġlayout Ġtree , Ġa Ġrepresentation Ġthat Ġenc odes Ġthe Ġspatial Ġrelationship Ġof Ġvariables Ġand Ġoperators Ġfor Ġthe Ġpurpose Ġof Ġindex ing Ġand Ġretrieving Ġmathematical Ġequations . ĠOur Ġwork Ġalso Ġfalls Ġinto Ġthe Ġframework Ġof Ġmathematical Ġlanguage Ġprocessing Ġ( ML P ) ĠB IB REF 9 Ġwhose Ġfirst Ġstep Ġis Ġconverting Ġmathematical Ġsolutions Ġinto Ġa Ġseries Ġof Ġnumerical Ġfeatures . </s> Equ ation ĠEmb edd ings ĠModels </s> E q Emb Ġare Ġbased Ġon Ġword Ġembed d ings ĠB IB REF 5 Ġor Ġspecifically ĠBern ou lli Ġembed d ings Ġ( b - em bs ) ĠB IB REF 1 Ġ. ĠWord Ġembed d ings Ġmodels Ġthe Ġprobability Ġof Ġa Ġword ĠIN LINE FORM 0 Ġgiven Ġits Ġcontext ĠIN LINE FORM 1 Ġas Ġa Ġconditional Ġdistribution ĠIN LINE FORM 2 Ġwhere Ġthe Ġcontext Ġis Ġdefined Ġas Ġthe Ġset Ġof Ġwords ĠIN LINE FORM 3 Ġin Ġa Ġwindow Ġof Ġsize ĠIN LINE FORM 4 Ġthat Ġsurrounds Ġit . ĠIn Ġword Ġembed d ings Ġeach Ġword Ġis Ġassigned Ġto Ġtwo Ġtypes Ġof Ġlatent Ġfeature Ġvectors , Ġthe Ġembed ding Ġ( ĠIN LINE FORM 5 Ġ) Ġand Ġcontext Ġ( ĠIN LINE FORM 6 Ġ) Ġvectors , Ġboth Ġof Ġwhich Ġare ĠIN LINE FORM 7 Ġdimensional . </s> B - emb Ġis Ġan Ġexponential Ġfamily Ġembed ding Ġmodel Ġwhere Ġthe Ġconditional Ġdistribution Ġis Ġa ĠBern ou lli : ĠDIS PLAY FORM 0 </s> The Ġparameter ĠIN LINE FORM 0 Ġis Ġdefined Ġusing Ġthe Ġword Ġembed ding ĠIN LINE FORM 1 Ġand Ġthe Ġword Ġcontext ĠIN LINE FORM 2 Ġvectors : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġlog istic Ġfunction . </s> Equ ation ĠEmb edd ings </s> Given Ġa Ġdataset Ġof Ġwords Ġand Ġequations Ġthe Ġgoal Ġof Ġthe ĠE q Emb Ġmodels Ġis Ġto Ġderive Ġa Ġsemantic Ġrepresentation Ġof Ġeach Ġequation . ĠE q Emb Ġmodel Ġequations Ġin Ġthe Ġcontext Ġof Ġwords . ĠE q Emb Ġis Ġbased Ġon Ġthe Ġidea Ġthat Ġa Ġgood Ġsemantic Ġrepresentation Ġof Ġequations Ġcould Ġbe Ġdiscovered Ġby Ġexpanding Ġthe Ġoriginal Ġword Ġcontext Ġto Ġinclude Ġany Ġequations Ġthat Ġappear Ġin Ġa Ġpossibly Ġlarger Ġwindow Ġaround Ġit . </s> We Ġassign Ġembed d ings Ġto Ġwords Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ) Ġand Ġequations Ġ( ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ). ĠThe Ġobjective Ġfunction Ġcontains Ġcondition als Ġover Ġthe Ġobserved Ġwords Ġand Ġequations : ĠDIS PLAY FORM 0 </s> This Ġis Ġa Ġsum Ġof Ġtwo Ġsets Ġof Ġconditional Ġdistributions , Ġthe Ġfirst Ġover Ġobserved Ġwords Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġthe Ġsecond Ġover Ġobserved Ġequations Ġ( ĠIN LINE FORM 1 Ġ). ĠIn Ġword Ġembed ding Ġmodels , ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġare Ġreferred Ġto Ġas Ġembed ding Ġand Ġcontext Ġvectors . ĠHere Ġwe Ġuse Ġa Ġdifferent Ġterminology : Ġthe Ġinteraction ĠIN LINE FORM 4 Ġand Ġfeature Ġvector ĠIN LINE FORM 5 Ġ. </s> In Ġword Ġembed d ings , Ġthe Ġcontext Ġof Ġthe Ġword ĠIN LINE FORM 0 Ġis Ġdefined Ġto Ġindex Ġthe Ġsurrounding Ġwords Ġin Ġa Ġsmall Ġwindow Ġaround Ġit . ĠHere Ġthe Ġcontext Ġof Ġthe Ġword ĠIN LINE FORM 1 Ġwill Ġbe Ġthe Ġoriginal Ġcontext Ġ( ĠIN LINE FORM 2 Ġ) Ġand Ġany Ġequations Ġ( ĠIN LINE FORM 3 Ġ) Ġthat Ġare Ġin Ġa Ġpossibly Ġlarger Ġwindow Ġaround Ġit . ĠThis Ġis Ġreferred Ġto Ġas Ġthe Ġword - equ ation Ġcontext Ġwindow . </s> Both Ġcondition als Ġare ĠBern ou lli Ġdistributions . ĠThe Ġfirst Ġconditional Ġis Ġdefined Ġover Ġthe Ġwords Ġin Ġthe Ġcollection . ĠIt Ġhas Ġthe Ġfollowing Ġparameter : ĠDIS PLAY FORM 0 </s> The Ġword Ġcontext Ġfunction Ġis : ĠDIS PLAY FORM 0 </s> This Ġfunction Ġencompasses Ġthe Ġwords Ġin Ġthe Ġoriginal Ġword Ġcontext Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġany Ġequations Ġ( ĠIN LINE FORM 1 Ġ) Ġthat Ġappear Ġin Ġa Ġpossibly Ġlarger Ġwindow Ġ( ĠIN LINE FORM 2 Ġ) Ġaround Ġit . </s> The Ġsecond Ġterm Ġin Ġthe Ġobjective Ġcorresponds Ġto Ġthe Ġsum Ġof Ġthe Ġlog Ġconditional Ġprobabilities Ġof Ġeach Ġequation . ĠIts Ġparameter Ġis : ĠDIS PLAY FORM 0 </s> Similar Ġto Ġword Ġembed d ings , Ġequation Ġcontext ĠIN LINE FORM 0 Ġcontains Ġwords Ġthat Ġare Ġin Ġa Ġcontext Ġwindow Ġaround Ġthe Ġequation : ĠDIS PLAY FORM 0 </s> The Ġequation Ġcontext Ġcan Ġhave Ġa Ġlarger Ġwindow Ġthan Ġthe Ġword Ġcontext . ĠEqu ation Ġfeature Ġvectors Ġ( ĠIN LINE FORM 0 Ġ) Ġare Ġonly Ġassociated Ġwith Ġthe Ġfirst Ġterm Ġof Ġthe Ġobjective Ġfunction . ĠThis Ġfunction Ġcontains Ġthe Ġwords Ġwhere Ġthe Ġequation Ġappears Ġin Ġtheir Ġlarger Ġcontext ĠIN LINE FORM 1 Ġ. </s> The Ġleft Ġside Ġof ĠFigure ĠFIG REF 1 Ġshows Ġan Ġexample Ġequation Ġin Ġa Ġscientific Ġarticle . ĠWith Ġa Ġword Ġcontext Ġof Ġsize ĠIN LINE FORM 0 Ġwe Ġmodel Ġthe Ġwords Ġin Ġthe Ġarticle Ġwhile Ġignoring Ġequations . ĠFor Ġexample Ġwhen Ġmodeling Ġthe Ġword Ġ" embed ding " Ġ( high light ed Ġin Ġgreen ) Ġwith Ġcontext Ġwindow Ġsize Ġof Ġ4 Ġ( i . e . ĠIN LINE FORM 1 Ġ), Ġthe Ġcontext Ġcontains Ġthe Ġwords Ġthat Ġappear Ġtwo Ġwords Ġbefore Ġ(" current " Ġand Ġ" word ") Ġand Ġafter Ġ(" rec urrent " Ġand Ġ" version ") Ġthis Ġword . ĠWith Ġa Ġword - equ ation Ġcontext Ġwindow Ġof Ġsize ĠIN LINE FORM 2 Ġ= 16 , Ġthe Ġterm Ġfor Ġthe Ġword Ġ" embed ding " Ġwould Ġhave Ġthe Ġfeature Ġvector Ġof Ġthe Ġequation Ġas Ġone Ġof Ġits Ġcomponents . </s> Equ ation ĠUnit ĠEmb edd ings </s> Building Ġon Ġour Ġprevious Ġmethod , Ġwe Ġdefine Ġa Ġnew Ġmodel Ġwhich Ġwe Ġcall Ġequation Ġunit Ġembed d ings Ġ( E q Emb - U ). ĠE q Emb - U Ġmodel Ġequations Ġby Ġtreating Ġthem Ġas Ġsentences Ġwhere Ġthe Ġwords Ġare Ġthe Ġequation Ġvariables , Ġsymbols Ġand Ġoperators Ġwhich Ġwe Ġrefer Ġto Ġas Ġunits . ĠThe Ġfirst Ġstep Ġin Ġrepresenting Ġequations Ġusing Ġequation Ġunits Ġis Ġto Ġtoken ize Ġthem . ĠWe Ġuse Ġthe Ġapproach Ġoutlined Ġin ĠB IB REF 8 Ġwhich Ġrepresents Ġequations Ġinto Ġa Ġsyntax Ġlayout Ġtree Ġ( SL T ), Ġa Ġsequence Ġof ĠSL T Ġtu ples Ġeach Ġof Ġwhich Ġcontains Ġthe Ġspatial Ġrelationship Ġinformation Ġbetween Ġtwo Ġequation Ġsymbols Ġfound Ġwithin Ġa Ġparticular Ġwindow Ġof Ġequation Ġsymbols . ĠFigure ĠFIG REF 11 Ġshows Ġexample ĠSL T Ġrepresentations Ġof Ġthree Ġequations . </s> Each Ġequation ĠIN LINE FORM 0 Ġis Ġa Ġsequence Ġof Ġequation Ġunits ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġsimilar Ġto Ġa Ġsentence Ġwhere Ġthe Ġwords Ġare Ġthe Ġequation Ġunits . ĠFor Ġeach Ġequation Ġunit ĠIN LINE FORM 3 Ġwe Ġassign Ġinteraction ĠIN LINE FORM 4 Ġand Ġfeature ĠIN LINE FORM 5 Ġvectors . </s> We Ġassume Ġthat Ġthe Ġcontext Ġof Ġthe Ġword ĠIN LINE FORM 0 Ġwill Ġbe Ġthe Ġoriginal Ġcontext Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġthe Ġequation Ġunits Ġ( ĠIN LINE FORM 2 Ġ) Ġof Ġany Ġequations Ġthat Ġare Ġin Ġthe Ġword - equ ation Ġcontext Ġwindow . ĠIn Ġaddition Ġfor Ġeach Ġequation Ġunit Ġwe Ġdefine Ġits Ġunit Ġcontext ĠIN LINE FORM 3 Ġto Ġbe Ġthe Ġset Ġof Ġsurrounding Ġequation Ġunits Ġin Ġa Ġsmall Ġwindow ĠIN LINE FORM 4 Ġaround Ġit : ĠDIS PLAY FORM 0 </s> The Ġobjective Ġis Ġover Ġtwo Ġcondition als , Ġone Ġfor Ġeach Ġcontext Ġtype : ĠDIS PLAY FORM 0 </s> The Ġtwo Ġparameters Ġare : ĠDIS PLAY FORM 0 </s> We Ġdefine Ġequation - level Ġrepresentations Ġby Ġaveraging Ġthe Ġrepresentations Ġof Ġtheir Ġconstituent Ġunits : ĠDIS PLAY FORM 0 </s> Com put ation </s> We Ġuse Ġsto ch astic Ġgradient Ġdescent Ġwith ĠAd ag rad ĠB IB REF 10 Ġto Ġfit Ġthe Ġembed ding Ġand Ġcontext Ġvectors . ĠFollowing ĠB IB REF 1 Ġ, Ġwe Ġreduce Ġthe Ġcomputational Ġcomplexity Ġby Ġsplitting Ġthe Ġgradient Ġinto Ġtwo Ġterms . ĠThe Ġfirst Ġterm Ġcontains Ġthe Ġnon - zero Ġentries Ġ( ĠIN LINE FORM 0 Ġ); Ġthe Ġsecond Ġterm Ġcontains Ġthe Ġzero Ġentries Ġ( ĠIN LINE FORM 1 Ġ). ĠWe Ġcompute Ġthe Ġexact Ġgradient Ġfor Ġthe Ġnon - zero Ġpoints ; ĠWe Ġsubs ample Ġfor Ġthe Ġzero Ġdata Ġpoints . ĠThis Ġis Ġsimilar Ġto Ġnegative Ġsampling ĠB IB REF 5 Ġ, Ġwhich Ġalso Ġdown - weights Ġthe Ġcontributions Ġof Ġthe Ġzero Ġpoints . ĠUnlike ĠB IB REF 1 Ġwhich Ġuses ĠIN LINE FORM 2 Ġregular ization Ġto Ġprotect Ġagainst Ġover fitting Ġwhen Ġfitting Ġthe Ġembed ding Ġvectors Ġwe Ġuse Ġearly Ġstopping Ġbased Ġon Ġvalidation Ġaccuracy , Ġfor Ġthe Ġsame Ġeffect . </s> E mp ir ical ĠStudy </s> We Ġstudied Ġthe Ġperformance Ġof ĠE q Emb Ġon Ġarticles Ġfrom Ġthe Ġar X iv . ĠE q Emb Ġmodels Ġprovide Ġbetter Ġfits Ġthan Ġexisting Ġembed ding Ġapproaches , Ġand Ġinfer Ġmeaningful Ġsemantic Ġrelationships Ġbetween Ġequations Ġand Ġwords Ġin Ġthe Ġcollection . </s> We Ġpresent Ġa Ġcomparison Ġof Ġthe Ġproposed Ġmodels Ġto Ġexisting Ġword Ġembed d ings Ġapproaches . ĠThese Ġare : Ġthe ĠBern ou lli Ġembed d ings Ġ( b - emb ) ĠB IB REF 1 Ġ, Ġcontinuous Ġbag - of - words Ġ( CB OW ) ĠB IB REF 5 Ġ, ĠDist ributed ĠMemory Ġversion Ġof ĠPar agraph ĠVector Ġ( P V - DM ) ĠB IB REF 11 Ġand Ġthe ĠGlobal ĠV ect ors Ġ( G lo Ve ) ĠB IB REF 6 Ġmodel . </s> Dat as ets </s> Our Ġdatasets Ġare Ġscientific Ġarticles Ġthat Ġwere Ġpublished Ġon Ġar X iv . ĠThe Ġsets Ġcontain Ġarticles Ġ( in ĠLa TeX Ġformat ) Ġfrom Ġfour Ġcomputer Ġscience Ġdomains : ĠN LP , ĠIR , ĠAI , Ġand ĠML . ĠThey Ġwere Ġcreated Ġby Ġfiltering Ġar X iv Ġarticles Ġbased Ġon Ġtheir Ġprimary Ġand Ġsecondary Ġcategories . ĠWe Ġused Ġthe Ġfollowing Ġcategories Ġfor Ġthe Ġfour Ġcollections : Ġcs . cl Ġfor ĠN LP ; Ġcs . ir Ġfor ĠIR ; Ġcs . ai Ġfor ĠAI Ġand Ġstat . ml , Ġstat . co , Ġstat . me Ġor Ġcs . l g Ġfor ĠML . </s> Table ĠT AB REF 22 Ġshows Ġthe Ġnumber Ġof Ġdocuments Ġalong Ġwith Ġthe Ġnumber Ġof Ġunique Ġwords , Ġequations Ġand Ġequation Ġunits Ġfor Ġeach Ġcollection . ĠThe Ġequations Ġare Ġdisplay Ġequations Ġthat Ġwere Ġenumer ated Ġin Ġthe ĠLa TeX Ġversion Ġof Ġthe Ġarticles . ĠUnlike Ġinline Ġequations , Ġwhich Ġin Ġmany Ġinstances Ġrepresent Ġvariables Ġwith Ġgeneral Ġmeaning Ġ( e . g . ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġetc .) Ġand Ġeven Ġnumerical Ġvalues , Ġdisplay Ġequations Ġtypically Ġrepresent Ġmathematical Ġconcepts Ġwith Ġmore Ġspecific Ġsemantics . ĠFor Ġthe Ġempirical Ġstudy Ġwe Ġused Ġa Ġrandom Ġsubset Ġof Ġ2 k Ġsing let ons Ġfrom Ġthe Ġtotal Ġcollection , Ġalong Ġwith Ġall Ġequations Ġthat Ġoccur Ġmore Ġthan Ġonce . ĠFor Ġthe Ġqualitative Ġanalysis , Ġwe Ġused Ġall Ġequations . </s> We Ġextracted Ġwords Ġby Ġtoken izing Ġarticles Ġusing Ġthe ĠNL T K Ġpackage ĠB IB REF 12 Ġand Ġrestricted Ġthe Ġvocabulary Ġto Ġnoun Ġphrases Ġand Ġadject ives . ĠThe Ġvocabulary Ġwas Ġselected Ġby : </s> rem oving Ġcommon Ġstop words </s> t reating Ġthe Ġtop Ġ25 Ġmost Ġfrequent Ġwords Ġas Ġstop Ġwords Ġand Ġremoving Ġthem </s> including Ġwords Ġwhose Ġterm Ġfrequency Ġis Ġgreater Ġthan Ġor Ġequal Ġto Ġ10 Ġand Ġwhose Ġcharacter Ġlength Ġis Ġgreater Ġthan Ġor Ġequal Ġto Ġ4 </s> including Ġthe Ġtop Ġ50 Ġmost Ġfrequent Ġabbrevi ations Ġwhose Ġcharacter Ġlength Ġis Ġ3 Ġ( an Ġexception Ġto Ġour Ġprevious Ġrule ) </s> When Ġtoken izing Ġequations , Ġwe Ġfirst Ġcreate Ġan Ġeffective Ġvocabulary Ġof Ġequation Ġunits . ĠWe Ġconvert Ġequations Ġinto ĠSL T Ġformat Ġand Ġcollect Ġcollection Ġwide Ġfrequency Ġstatistics Ġover Ġthe Ġequation Ġunits . ĠThe Ġvocabulary Ġcontains Ġall Ġequation Ġunits Ġwhose Ġfrequency Ġcount Ġis Ġgreater Ġthan ĠIN LINE FORM 0 Ġ. </s> Exper imental ĠSetup </s> We Ġanalyzed ĠE q Emb Ġmodels Ġperformance Ġusing Ġa Ġheld Ġout Ġset Ġof Ġwords Ġthat Ġwe Ġgenerate Ġfor Ġeach Ġequation Ġin Ġour Ġcollections . ĠHeld Ġout Ġsets Ġare Ġconstructed Ġusing Ġthe Ġfollowing Ġprocedure : ĠWe Ġtraverse Ġover Ġthe Ġcollections Ġand Ġfor Ġevery Ġdiscovered Ġequation Ġwe Ġrandomly Ġsample Ġwords Ġfrom Ġits Ġcontext Ġset . ĠThe Ġheld Ġout Ġset Ġcontains Ġthe Ġsampled Ġwords Ġand Ġtheir Ġcontext Ġwindow Ġwhich Ġalso Ġincludes Ġthe Ġequation . ĠFor Ġeach Ġheld Ġout Ġword Ġwe Ġalso Ġgenerate Ġa Ġset Ġof Ġnegative Ġsamples Ġfor Ġthe Ġgiven Ġword Ġcontext . ĠWe Ġperform Ġthe Ġsame Ġprocedure Ġto Ġform Ġa Ġvalidation Ġset . ĠFor Ġeach Ġof Ġthe ĠIN LINE FORM 0 Ġequations Ġin Ġa Ġcollection , Ġtwo Ġheld Ġout Ġwords ĠIN LINE FORM 1 Ġare Ġsampled . ĠFor Ġa Ġcontext Ġwindow Ġof Ġsize Ġ4 Ġthe Ġsampled Ġword Ġcontext Ġis Ġdefined Ġas ĠIN LINE FORM 2 Ġ. </s> During Ġtraining Ġwe Ġcompute Ġthe Ġpredictive Ġlog - like lihood Ġon Ġthe Ġvalidation Ġset Ġof Ġwords Ġusing Ġthe Ġfitted Ġmodel Ġafter Ġeach Ġiteration Ġover Ġthe Ġcollection . ĠA Ġfitted Ġmodel Ġis Ġa Ġcollection Ġof Ġinteraction Ġand Ġfeature Ġvectors Ġfor Ġeach Ġequation Ġand Ġword . ĠGiven Ġa Ġfitted Ġmodel , Ġthe Ġlog Ġprobability Ġof Ġa Ġheld Ġout Ġword Ġis Ġcomputed Ġusing Ġthe Ġfollowing Ġformula : ĠDIS PLAY FORM 0 </s> which Ġis Ġthe Ġsoft max Ġfunction Ġcomputed Ġover Ġa Ġset Ġof Ġnegative Ġsamples ĠIN LINE FORM 0 Ġand Ġthe Ġheld Ġout Ġword . ĠIn Ġparticular , Ġwe Ġran Ġthe Ġmodel Ġ20 Ġtimes Ġacross Ġthe Ġcollection . ĠAfter Ġeach Ġcollection Ġiteration ĠIN LINE FORM 1 Ġwe Ġobserve Ġwhether Ġthe Ġpredictive Ġlog - like lihood Ġcontinues Ġto Ġimprove Ġcompared Ġto Ġthe Ġprevious Ġ( ĠIN LINE FORM 2 Ġ) Ġiteration . ĠWe Ġstop Ġat Ġthe ĠIN LINE FORM 3 Ġ- th Ġiteration Ġwhen Ġthat Ġis Ġno Ġlonger Ġthe Ġcase . </s> When Ġmodeling Ġequations Ġusing ĠE q Emb Ġwe Ġperform Ġtwo Ġpasses Ġover Ġthe Ġcollection . ĠIn Ġthe Ġfirst Ġpass Ġwe Ġonly Ġmodel Ġwords Ġwhile Ġignoring Ġequations . ĠIn Ġthe Ġsecond Ġpass Ġwe Ġonly Ġmodel Ġequations Ġwhile Ġholding Ġfixed Ġthe Ġinteraction Ġand Ġfeature Ġvectors Ġof Ġall Ġwords . ĠIn Ġcontext Ġof ĠE q Emb Ġwe Ġtreat Ġequations Ġas Ġsingle ton Ġwords Ġand Ġthe Ġbroader Ġquestion Ġthat Ġwe Ġare Ġtrying Ġto Ġanswer Ġis Ġwhether Ġwe Ġcan Ġlearn Ġsomething Ġabout Ġthe Ġmeaning Ġof Ġthe Ġsingle ton Ġwords Ġgiven Ġthe Ġfixed Ġword Ġinteraction Ġand Ġfeature Ġvectors . </s> In Ġour Ġanalysis Ġwe Ġevaluated Ġthe Ġperformance Ġof Ġthe ĠE q Emb Ġmodels Ġacross Ġdifferent Ġsizes Ġfor Ġthe Ġword Ġcontext Ġ( W ), Ġword - equ ation Ġcontext Ġ( E ) Ġand Ġembed ding Ġvector Ġsize Ġ( K ). ĠModel Ġperformance Ġwas Ġcompared Ġwith Ġ4 Ġexisting Ġembed ding Ġmodels : Ġb - emb , ĠCB OW , ĠGl o Ve Ġand ĠPV - DM . ĠWe Ġused Ġthe Ġg ens im ĠB IB REF 13 Ġimplementation Ġof Ġthe ĠCB OW Ġand ĠPV - DM Ġmodels . ĠWhen Ġmodeling Ġequations Ġusing Ġthe Ġfirst Ġ3 Ġembed ding Ġmodels Ġwe Ġtreat Ġequations Ġas Ġregular Ġwords Ġin Ġthe Ġcollection . ĠIn Ġcase Ġof Ġthe ĠPV - DM Ġmodel Ġwe Ġparse Ġthe Ġarticle Ġso Ġthat Ġequations Ġand Ġtheir Ġsurrounding Ġcontext Ġof Ġlength Ġequivalent Ġto Ġthe Ġword - equ ation Ġcontext Ġwindow Ġare Ġlabeled Ġas Ġa Ġseparate Ġparagraph . ĠWe Ġalso Ġassign Ġparagraph Ġlabels Ġto Ġthe Ġarticle Ġtext Ġoccurring Ġbetween Ġequation Ġparagraphs . </s> Results </s> Table ĠT AB REF 23 Ġshows Ġthe Ġperformance Ġcomparison Ġresults Ġacross Ġthe Ġdifferent Ġembed d ings Ġmodels . ĠFor Ġeach Ġmodel , Ġperformance Ġresults Ġare Ġshown Ġon Ġ4 Ġlatent Ġdimension Ġvalues Ġ( K = 25 , Ġ50 , Ġ75 Ġand Ġ100 ). ĠFor Ġeach Ġdimension Ġwe Ġran Ġexperiments Ġby Ġvarying Ġthe Ġcontext Ġwindow Ġsize Ġfor Ġwords Ġ( Word ĠContext = 4 , Ġ8 Ġand Ġ16 ). ĠIn Ġaddition Ġfor Ġthe ĠE q Emb , ĠE q Emb - U Ġand ĠPV - DM Ġmodels Ġwe Ġalso Ġvaried Ġthe Ġword - equ ation Ġwindow Ġsize Ġ( E = 8 Ġand Ġ16 ). ĠCompar isons Ġacross Ġmodels Ġare Ġperformed Ġusing Ġthe Ġpseudo Ġlog - like lihood Ġmeasure ĠB IB REF 14 Ġ. ĠFor Ġa Ġgiven Ġheld - out Ġword ĠIN LINE FORM 0 Ġand Ġa Ġset Ġof Ġnegative Ġsamples ĠIN LINE FORM 1 Ġthe Ġpseudo Ġlog - like lihood Ġis Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> We Ġtreat Ġthis Ġa Ġdownstream Ġtask . ĠFor Ġeach Ġmodel Ġtype Ġand Ġlatent Ġdimension Ġconfiguration , Ġwe Ġuse Ġthe Ġvalidation Ġset Ġto Ġselect Ġthe Ġbest Ġmodel Ġconfiguration Ġ( i . e . Ġcombination Ġof Ġcontext Ġwindow Ġsizes ). ĠWe Ġreport Ġvalues Ġon Ġboth Ġdatasets . </s> Across Ġall Ġcollections ĠE q Emb Ġoutper form Ġprevious Ġembed ding Ġmodels Ġand ĠE q Emb - U Ġfurther Ġimproves Ġperformance . </s> Word ĠRepresent ation Ġof ĠEqu ations </s> E q Emb Ġhelp Ġobtain Ġword Ġdescriptions Ġof Ġthe Ġequations . ĠTable ĠT AB REF 25 Ġshows Ġexample Ġequation Ġand Ġthe Ġ5 Ġmost Ġsimilar Ġwords Ġobtained Ġusing Ġ4 Ġdifferent Ġembed ding Ġapproaches Ġwhich Ġinclude ĠCB OW , ĠPV - DM , ĠGl o Ve Ġand ĠE q Emb . ĠFor Ġthe Ġquery Ġequation Ġwe Ġobtain Ġmost Ġsimilar Ġwords Ġby Ġcomputing ĠCos ine Ġdistance Ġbetween Ġthe Ġembed ding Ġvector Ġ( ĠIN LINE FORM 0 Ġ) Ġrepresentation Ġof Ġthe Ġquery Ġequation Ġand Ġthe Ġcontext Ġvector Ġrepresentation Ġof Ġthe Ġwords Ġ( ĠIN LINE FORM 1 Ġ). </s> With Ġthe Ġembed ding Ġrepresentation Ġof Ġwords Ġand Ġequations Ġwe Ġcould Ġalso Ġperform Ġequation Ġsearch Ġusing Ġwords Ġas Ġqueries . ĠFor Ġa Ġset Ġof Ġquery Ġwords Ġwe Ġgenerate Ġits Ġembed ding Ġrepresentation Ġby Ġtaking Ġthe Ġaverage Ġof Ġthe Ġembed ding Ġrepresentation Ġof Ġeach Ġword Ġand Ġcompute ĠCos ine Ġdistance Ġacross Ġall Ġthe Ġequations Ġembed d ings . ĠTable ĠT AB REF 25 Ġshows Ġan Ġexample Ġquery , Ġwhich Ġconsists Ġof Ġthree Ġwords , Ġand Ġits Ġ5 Ġnearest Ġequations Ġdiscovered Ġusing ĠE q Emb . ĠFor Ġa Ġgiven Ġword Ġquery , ĠE q Emb Ġare Ġable Ġto Ġretrieve Ġquery Ġrelevant Ġequations . </s> Discover ing ĠSem antically ĠSimilar ĠEqu ations </s> In Ġaddition Ġto Ġwords , ĠE q Emb Ġmodels Ġcan Ġcapture Ġthe Ġsemantic Ġsimilarity Ġbetween Ġequations Ġin Ġthe Ġcollection . ĠWe Ġperformed Ġqualitative Ġanalysis Ġof Ġthe Ġmodel Ġperformance Ġusing Ġall Ġdiscovered Ġequations Ġacross Ġthe Ġ4 Ġcollection . ĠTable ĠT AB REF 24 Ġshows Ġthe Ġquery Ġequation Ġused Ġin Ġthe Ġprevious Ġanalysis Ġand Ġits Ġ5 Ġmost Ġsimilar Ġequations Ġdiscovered Ġusing ĠE q Emb - U . ĠFor Ġqualitative Ġcomparisons Ġacross Ġthe Ġother Ġembed ding Ġmodels , Ġin ĠAppendix ĠA Ġwe Ġprovide Ġresults Ġover Ġthe Ġsame Ġquery Ġusing ĠCB OW , ĠPV - DM , ĠGl o Ve Ġand ĠE q Emb . ĠIn ĠAppendix ĠA Ġreader Ġshould Ġnotice Ġthe Ġdifference Ġin Ġperformance Ġbetween ĠE q Emb - U Ġand ĠE q Emb Ġcompared Ġto Ġexisting Ġembed ding Ġmodels Ġwhich Ġfail Ġto Ġdiscover Ġsem antically Ġsimilar Ġequations . Ġtab : ire x ample 1 , tab : nl pex ample 2 Ġshow Ġtwo Ġadditional Ġexample Ġequation Ġand Ġits Ġ5 Ġmost Ġsimilar Ġequations Ġand Ġwords Ġdiscovered Ġusing Ġthe ĠE q Emb Ġmodel . ĠSimilar Ġwords Ġwere Ġranked Ġby Ġcomputing ĠCos ine Ġdistance Ġbetween Ġthe Ġembed ding Ġvector Ġ( ĠIN LINE FORM 0 Ġ) Ġrepresentation Ġof Ġthe Ġquery Ġequation Ġand Ġthe Ġcontext Ġvector Ġrepresentation Ġof Ġthe Ġwords Ġ( ĠIN LINE FORM 1 Ġ). ĠSimilar Ġequations Ġwere Ġdiscovered Ġusing ĠEucl ide an Ġdistance Ġcomputed Ġbetween Ġthe Ġcontext Ġvector Ġrepresentations Ġof Ġthe Ġequations Ġ( ĠIN LINE FORM 2 Ġ). ĠWe Ġgive Ġadditional Ġexample Ġresults Ġin ĠAppendix ĠB . </s> Conclusion </s> We Ġpresented Ġun super vised Ġapproaches Ġfor Ġsemantic Ġrepresentations Ġof Ġmathematical Ġequations Ġusing Ġtheir Ġsurrounding Ġwords . ĠAcross Ġ4 Ġdifferent Ġcollections Ġwe Ġshowed Ġthat Ġout Ġmethods Ġoffer Ġmore Ġeffective Ġmodeling Ġcompared Ġto Ġexisting Ġembed ding Ġmodels . ĠWe Ġalso Ġdemonstrate Ġthat Ġthey Ġcan Ġcapture Ġthe Ġsemantic Ġsimilarity Ġbetween Ġequations Ġand Ġthe Ġwords Ġin Ġthe Ġcollection . ĠIn Ġthe Ġfuture Ġwe Ġplan Ġto Ġexplore Ġhow ĠE q Emb Ġcould Ġbe Ġexpend Ġto Ġrepresent Ġother Ġobjects Ġsuch Ġas Ġimages , Ġcapt ions Ġand Ġinline Ġfigures .
Introduction Equations are an important part of scientific articles, but many existing machine learning methods do not easily handle them. They are challenging to work with because each is unique or nearly unique; most equations occur only once. An automatic understanding of equations, however, would significantly benefit methods for analyzing scientific literature. Useful representations of equations can help draw connections between articles, improve retrieval of scientific texts, and help create tools for exploring and navigating scientific literature. In this paper we propose equation embeddings (EqEmb), an unsupervised approach for learning distributed representations of equations. The idea is to treat the equation as a "singleton word," one that appears once but that appears in the context of other words. The surrounding text of the equation—and in particular, the distributed representations of that text—provides the data we need to develop a useful representation of the equation. Figure FIGREF1 illustrates our approach. On the left is an article snippet BIBREF0 . Highlighted in orange is an equation; in this example it represents a neural network layer. We note that this particular equation (in this form and with this notation) only occurs once in the collection of articles (from arXiv). The representations of the surrounding text, however, provide a meaningful context for the equation. Those words allow us to learn its embedding, specifically as a "word" which appears in the context of its surroundings. The resulting representation, when compared to other equations' representations and word representations, helps find both related equations and related words. These are illustrated on the right. EqEmbs build on exponential family embeddings BIBREF1 to include equations as singleton observations and to model equation elements such as variables, symbols and operators. Exponential family embeddings, like all embedding methods, define a context of each word. In our initial EqEmb, the context for the words is a small window, such as four or eight words, but the context of an equation is a larger window, such as sixteen words. Using these two types of contexts together finds meaningful representations of words and equations. In the next EqEmb, which builds on the first, we consider equations to be sentences consisting of equation units, i.e., variables, symbols, and operators. Equation units help model equations across two types of context—over the surrounding units and over the surrounding words. We studied EqEmbs on four collections of scientific articles from the arXiv, covering four computer science domains: natural language processing (NLP), information retrieval (IR), artificial intelligence (AI) and machine learning (ML). We found that EqEmbs provide more efficient modeling than existing word embedding methods. We further carried out an exploratory analysis of a large set of INLINEFORM0 87k equations. We found that EqEmbs provide better models when compared to existing word embedding approaches. EqEmbs also provide coherent semantic representations of equations and can capture semantic similarity to other equations and to words. Related Work Word embeddings were first introduced in BIBREF2 , BIBREF3 and there have been many variants BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . Common for all of them is the idea that words can be represented by latent feature vectors. These feature vectors are optimized to maximize the conditional probability of the dataset. Recently BIBREF1 extended the idea of word embeddings to other types of data. EqEmb expand the idea of word embeddings to a new type of data points – equations. There have been different proposed approaches for representing mathematical equations. BIBREF8 introduced the symbol layout tree, a representation that encodes the spatial relationship of variables and operators for the purpose of indexing and retrieving mathematical equations. Our work also falls into the framework of mathematical language processing (MLP) BIBREF9 whose first step is converting mathematical solutions into a series of numerical features. Equation Embeddings Models EqEmb are based on word embeddings BIBREF5 or specifically Bernoulli embeddings (b-embs) BIBREF1 . Word embeddings models the probability of a word INLINEFORM0 given its context INLINEFORM1 as a conditional distribution INLINEFORM2 where the context is defined as the set of words INLINEFORM3 in a window of size INLINEFORM4 that surrounds it. In word embeddings each word is assigned to two types of latent feature vectors, the embedding ( INLINEFORM5 ) and context ( INLINEFORM6 ) vectors, both of which are INLINEFORM7 dimensional. B-emb is an exponential family embedding model where the conditional distribution is a Bernoulli: DISPLAYFORM0 The parameter INLINEFORM0 is defined using the word embedding INLINEFORM1 and the word context INLINEFORM2 vectors: DISPLAYFORM0 where INLINEFORM0 is the logistic function. Equation Embeddings Given a dataset of words and equations the goal of the EqEmb models is to derive a semantic representation of each equation. EqEmb model equations in the context of words. EqEmb is based on the idea that a good semantic representation of equations could be discovered by expanding the original word context to include any equations that appear in a possibly larger window around it. We assign embeddings to words ( INLINEFORM0 , INLINEFORM1 ) and equations ( INLINEFORM2 , INLINEFORM3 ). The objective function contains conditionals over the observed words and equations: DISPLAYFORM0 This is a sum of two sets of conditional distributions, the first over observed words ( INLINEFORM0 ) and the second over observed equations ( INLINEFORM1 ). In word embedding models, INLINEFORM2 and INLINEFORM3 are referred to as embedding and context vectors. Here we use a different terminology: the interaction INLINEFORM4 and feature vector INLINEFORM5 . In word embeddings, the context of the word INLINEFORM0 is defined to index the surrounding words in a small window around it. Here the context of the word INLINEFORM1 will be the original context ( INLINEFORM2 ) and any equations ( INLINEFORM3 ) that are in a possibly larger window around it. This is referred to as the word-equation context window. Both conditionals are Bernoulli distributions. The first conditional is defined over the words in the collection. It has the following parameter: DISPLAYFORM0 The word context function is: DISPLAYFORM0 This function encompasses the words in the original word context ( INLINEFORM0 ) and any equations ( INLINEFORM1 ) that appear in a possibly larger window ( INLINEFORM2 ) around it. The second term in the objective corresponds to the sum of the log conditional probabilities of each equation. Its parameter is: DISPLAYFORM0 Similar to word embeddings, equation context INLINEFORM0 contains words that are in a context window around the equation: DISPLAYFORM0 The equation context can have a larger window than the word context. Equation feature vectors ( INLINEFORM0 ) are only associated with the first term of the objective function. This function contains the words where the equation appears in their larger context INLINEFORM1 . The left side of Figure FIGREF1 shows an example equation in a scientific article. With a word context of size INLINEFORM0 we model the words in the article while ignoring equations. For example when modeling the word "embedding" (highlighted in green) with context window size of 4 (i.e. INLINEFORM1 ), the context contains the words that appear two words before ("current" and "word") and after ("recurrent" and "version") this word. With a word-equation context window of size INLINEFORM2 =16, the term for the word "embedding" would have the feature vector of the equation as one of its components. Equation Unit Embeddings Building on our previous method, we define a new model which we call equation unit embeddings (EqEmb-U). EqEmb-U model equations by treating them as sentences where the words are the equation variables, symbols and operators which we refer to as units. The first step in representing equations using equation units is to tokenize them. We use the approach outlined in BIBREF8 which represents equations into a syntax layout tree (SLT), a sequence of SLT tuples each of which contains the spatial relationship information between two equation symbols found within a particular window of equation symbols. Figure FIGREF11 shows example SLT representations of three equations. Each equation INLINEFORM0 is a sequence of equation units INLINEFORM1 , INLINEFORM2 similar to a sentence where the words are the equation units. For each equation unit INLINEFORM3 we assign interaction INLINEFORM4 and feature INLINEFORM5 vectors. We assume that the context of the word INLINEFORM0 will be the original context ( INLINEFORM1 ) and the equation units ( INLINEFORM2 ) of any equations that are in the word-equation context window. In addition for each equation unit we define its unit context INLINEFORM3 to be the set of surrounding equation units in a small window INLINEFORM4 around it: DISPLAYFORM0 The objective is over two conditionals, one for each context type: DISPLAYFORM0 The two parameters are: DISPLAYFORM0 We define equation-level representations by averaging the representations of their constituent units: DISPLAYFORM0 Computation We use stochastic gradient descent with Adagrad BIBREF10 to fit the embedding and context vectors. Following BIBREF1 , we reduce the computational complexity by splitting the gradient into two terms. The first term contains the non-zero entries ( INLINEFORM0 ); the second term contains the zero entries ( INLINEFORM1 ). We compute the exact gradient for the non-zero points; We subsample for the zero data points. This is similar to negative sampling BIBREF5 , which also down-weights the contributions of the zero points. Unlike BIBREF1 which uses INLINEFORM2 regularization to protect against overfitting when fitting the embedding vectors we use early stopping based on validation accuracy, for the same effect. Empirical Study We studied the performance of EqEmb on articles from the arXiv. EqEmb models provide better fits than existing embedding approaches, and infer meaningful semantic relationships between equations and words in the collection. We present a comparison of the proposed models to existing word embeddings approaches. These are: the Bernoulli embeddings (b-emb) BIBREF1 , continuous bag-of-words (CBOW) BIBREF5 , Distributed Memory version of Paragraph Vector (PV-DM) BIBREF11 and the Global Vectors (GloVe) BIBREF6 model. Datasets Our datasets are scientific articles that were published on arXiv. The sets contain articles (in LaTeX format) from four computer science domains: NLP, IR, AI, and ML. They were created by filtering arXiv articles based on their primary and secondary categories. We used the following categories for the four collections: cs.cl for NLP; cs.ir for IR; cs.ai for AI and stat.ml, stat.co, stat.me or cs.lg for ML. Table TABREF22 shows the number of documents along with the number of unique words, equations and equation units for each collection. The equations are display equations that were enumerated in the LaTeX version of the articles. Unlike inline equations, which in many instances represent variables with general meaning (e.g. INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , etc.) and even numerical values, display equations typically represent mathematical concepts with more specific semantics. For the empirical study we used a random subset of 2k singletons from the total collection, along with all equations that occur more than once. For the qualitative analysis, we used all equations. We extracted words by tokenizing articles using the NLTK package BIBREF12 and restricted the vocabulary to noun phrases and adjectives. The vocabulary was selected by: removing common stopwords treating the top 25 most frequent words as stop words and removing them including words whose term frequency is greater than or equal to 10 and whose character length is greater than or equal to 4 including the top 50 most frequent abbreviations whose character length is 3 (an exception to our previous rule) When tokenizing equations, we first create an effective vocabulary of equation units. We convert equations into SLT format and collect collection wide frequency statistics over the equation units. The vocabulary contains all equation units whose frequency count is greater than INLINEFORM0 . Experimental Setup We analyzed EqEmb models performance using a held out set of words that we generate for each equation in our collections. Held out sets are constructed using the following procedure: We traverse over the collections and for every discovered equation we randomly sample words from its context set. The held out set contains the sampled words and their context window which also includes the equation. For each held out word we also generate a set of negative samples for the given word context. We perform the same procedure to form a validation set. For each of the INLINEFORM0 equations in a collection, two held out words INLINEFORM1 are sampled. For a context window of size 4 the sampled word context is defined as INLINEFORM2 . During training we compute the predictive log-likelihood on the validation set of words using the fitted model after each iteration over the collection. A fitted model is a collection of interaction and feature vectors for each equation and word. Given a fitted model, the log probability of a held out word is computed using the following formula: DISPLAYFORM0 which is the softmax function computed over a set of negative samples INLINEFORM0 and the held out word. In particular, we ran the model 20 times across the collection. After each collection iteration INLINEFORM1 we observe whether the predictive log-likelihood continues to improve compared to the previous ( INLINEFORM2 ) iteration. We stop at the INLINEFORM3 -th iteration when that is no longer the case. When modeling equations using EqEmb we perform two passes over the collection. In the first pass we only model words while ignoring equations. In the second pass we only model equations while holding fixed the interaction and feature vectors of all words. In context of EqEmb we treat equations as singleton words and the broader question that we are trying to answer is whether we can learn something about the meaning of the singleton words given the fixed word interaction and feature vectors. In our analysis we evaluated the performance of the EqEmb models across different sizes for the word context (W), word-equation context (E) and embedding vector size (K). Model performance was compared with 4 existing embedding models: b-emb, CBOW, GloVe and PV-DM. We used the gensim BIBREF13 implementation of the CBOW and PV-DM models. When modeling equations using the first 3 embedding models we treat equations as regular words in the collection. In case of the PV-DM model we parse the article so that equations and their surrounding context of length equivalent to the word-equation context window are labeled as a separate paragraph. We also assign paragraph labels to the article text occurring between equation paragraphs. Results Table TABREF23 shows the performance comparison results across the different embeddings models. For each model, performance results are shown on 4 latent dimension values (K=25, 50, 75 and 100). For each dimension we ran experiments by varying the context window size for words (Word Context=4, 8 and 16). In addition for the EqEmb, EqEmb-U and PV-DM models we also varied the word-equation window size (E=8 and 16). Comparisons across models are performed using the pseudo log-likelihood measure BIBREF14 . For a given held-out word INLINEFORM0 and a set of negative samples INLINEFORM1 the pseudo log-likelihood is defined as: DISPLAYFORM0 We treat this a downstream task. For each model type and latent dimension configuration, we use the validation set to select the best model configuration (i.e. combination of context window sizes). We report values on both datasets. Across all collections EqEmb outperform previous embedding models and EqEmb-U further improves performance. Word Representation of Equations EqEmb help obtain word descriptions of the equations. Table TABREF25 shows example equation and the 5 most similar words obtained using 4 different embedding approaches which include CBOW, PV-DM, GloVe and EqEmb. For the query equation we obtain most similar words by computing Cosine distance between the embedding vector ( INLINEFORM0 ) representation of the query equation and the context vector representation of the words ( INLINEFORM1 ). With the embedding representation of words and equations we could also perform equation search using words as queries. For a set of query words we generate its embedding representation by taking the average of the embedding representation of each word and compute Cosine distance across all the equations embeddings. Table TABREF25 shows an example query, which consists of three words, and its 5 nearest equations discovered using EqEmb. For a given word query, EqEmb are able to retrieve query relevant equations. Discovering Semantically Similar Equations In addition to words, EqEmb models can capture the semantic similarity between equations in the collection. We performed qualitative analysis of the model performance using all discovered equations across the 4 collection. Table TABREF24 shows the query equation used in the previous analysis and its 5 most similar equations discovered using EqEmb-U. For qualitative comparisons across the other embedding models, in Appendix A we provide results over the same query using CBOW, PV-DM, GloVe and EqEmb. In Appendix A reader should notice the difference in performance between EqEmb-U and EqEmb compared to existing embedding models which fail to discover semantically similar equations. tab:irexample1,tab:nlpexample2 show two additional example equation and its 5 most similar equations and words discovered using the EqEmb model. Similar words were ranked by computing Cosine distance between the embedding vector ( INLINEFORM0 ) representation of the query equation and the context vector representation of the words ( INLINEFORM1 ). Similar equations were discovered using Euclidean distance computed between the context vector representations of the equations ( INLINEFORM2 ). We give additional example results in Appendix B. Conclusion We presented unsupervised approaches for semantic representations of mathematical equations using their surrounding words. Across 4 different collections we showed that out methods offer more effective modeling compared to existing embedding models. We also demonstrate that they can capture the semantic similarity between equations and the words in the collection. In the future we plan to explore how EqEmb could be expend to represent other objects such as images, captions and inline figures.
How do they define similar equations?
8dd8e5599fc56562f2acbc16dd8544689cddd938
How Ġdo Ġthey Ġdefine Ġsimilar Ġequations ?
By Ġusing ĠEucl ide an Ġdistance Ġcomputed Ġbetween Ġthe Ġcontext Ġvector Ġrepresentations Ġof Ġthe Ġequations
<s> How Ġdo Ġthey Ġdefine Ġsimilar Ġequations ? </s> Introduction </s> Equ ations Ġare Ġan Ġimportant Ġpart Ġof Ġscientific Ġarticles , Ġbut Ġmany Ġexisting Ġmachine Ġlearning Ġmethods Ġdo Ġnot Ġeasily Ġhandle Ġthem . ĠThey Ġare Ġchallenging Ġto Ġwork Ġwith Ġbecause Ġeach Ġis Ġunique Ġor Ġnearly Ġunique ; Ġmost Ġequations Ġoccur Ġonly Ġonce . ĠAn Ġautomatic Ġunderstanding Ġof Ġequations , Ġhowever , Ġwould Ġsignificantly Ġbenefit Ġmethods Ġfor Ġanalyzing Ġscientific Ġliterature . ĠUseful Ġrepresentations Ġof Ġequations Ġcan Ġhelp Ġdraw Ġconnections Ġbetween Ġarticles , Ġimprove Ġretrieval Ġof Ġscientific Ġtexts , Ġand Ġhelp Ġcreate Ġtools Ġfor Ġexploring Ġand Ġnavigating Ġscientific Ġliterature . </s> In Ġthis Ġpaper Ġwe Ġpropose Ġequation Ġembed d ings Ġ( E q Emb ), Ġan Ġun super vised Ġapproach Ġfor Ġlearning Ġdistributed Ġrepresentations Ġof Ġequations . ĠThe Ġidea Ġis Ġto Ġtreat Ġthe Ġequation Ġas Ġa Ġ" sing leton Ġword ," Ġone Ġthat Ġappears Ġonce Ġbut Ġthat Ġappears Ġin Ġthe Ġcontext Ġof Ġother Ġwords . ĠThe Ġsurrounding Ġtext Ġof Ġthe Ġequation âĢĶ and Ġin Ġparticular , Ġthe Ġdistributed Ġrepresentations Ġof Ġthat Ġtext âĢĶ prov ides Ġthe Ġdata Ġwe Ġneed Ġto Ġdevelop Ġa Ġuseful Ġrepresentation Ġof Ġthe Ġequation . </s> Figure ĠFIG REF 1 Ġillustrates Ġour Ġapproach . ĠOn Ġthe Ġleft Ġis Ġan Ġarticle Ġsnippet ĠB IB REF 0 Ġ. ĠHigh light ed Ġin Ġorange Ġis Ġan Ġequation ; Ġin Ġthis Ġexample Ġit Ġrepresents Ġa Ġneural Ġnetwork Ġlayer . ĠWe Ġnote Ġthat Ġthis Ġparticular Ġequation Ġ( in Ġthis Ġform Ġand Ġwith Ġthis Ġnotation ) Ġonly Ġoccurs Ġonce Ġin Ġthe Ġcollection Ġof Ġarticles Ġ( from Ġar X iv ). ĠThe Ġrepresentations Ġof Ġthe Ġsurrounding Ġtext , Ġhowever , Ġprovide Ġa Ġmeaningful Ġcontext Ġfor Ġthe Ġequation . ĠThose Ġwords Ġallow Ġus Ġto Ġlearn Ġits Ġembed ding , Ġspecifically Ġas Ġa Ġ" word " Ġwhich Ġappears Ġin Ġthe Ġcontext Ġof Ġits Ġsurroundings . ĠThe Ġresulting Ġrepresentation , Ġwhen Ġcompared Ġto Ġother Ġequations ' Ġrepresentations Ġand Ġword Ġrepresentations , Ġhelps Ġfind Ġboth Ġrelated Ġequations Ġand Ġrelated Ġwords . ĠThese Ġare Ġillustrated Ġon Ġthe Ġright . </s> E q Em bs Ġbuild Ġon Ġexponential Ġfamily Ġembed d ings ĠB IB REF 1 Ġto Ġinclude Ġequations Ġas Ġsingle ton Ġobservations Ġand Ġto Ġmodel Ġequation Ġelements Ġsuch Ġas Ġvariables , Ġsymbols Ġand Ġoperators . ĠExp onential Ġfamily Ġembed d ings , Ġlike Ġall Ġembed ding Ġmethods , Ġdefine Ġa Ġcontext Ġof Ġeach Ġword . ĠIn Ġour Ġinitial ĠE q Emb , Ġthe Ġcontext Ġfor Ġthe Ġwords Ġis Ġa Ġsmall Ġwindow , Ġsuch Ġas Ġfour Ġor Ġeight Ġwords , Ġbut Ġthe Ġcontext Ġof Ġan Ġequation Ġis Ġa Ġlarger Ġwindow , Ġsuch Ġas Ġsixteen Ġwords . ĠUsing Ġthese Ġtwo Ġtypes Ġof Ġcontexts Ġtogether Ġfinds Ġmeaningful Ġrepresentations Ġof Ġwords Ġand Ġequations . ĠIn Ġthe Ġnext ĠE q Emb , Ġwhich Ġbuilds Ġon Ġthe Ġfirst , Ġwe Ġconsider Ġequations Ġto Ġbe Ġsentences Ġconsisting Ġof Ġequation Ġunits , Ġi . e ., Ġvariables , Ġsymbols , Ġand Ġoperators . ĠEqu ation Ġunits Ġhelp Ġmodel Ġequations Ġacross Ġtwo Ġtypes Ġof Ġcontext âĢĶ over Ġthe Ġsurrounding Ġunits Ġand Ġover Ġthe Ġsurrounding Ġwords . </s> We Ġstudied ĠE q Em bs Ġon Ġfour Ġcollections Ġof Ġscientific Ġarticles Ġfrom Ġthe Ġar X iv , Ġcovering Ġfour Ġcomputer Ġscience Ġdomains : Ġnatural Ġlanguage Ġprocessing Ġ( N LP ), Ġinformation Ġretrieval Ġ( IR ), Ġartificial Ġintelligence Ġ( AI ) Ġand Ġmachine Ġlearning Ġ( ML ). ĠWe Ġfound Ġthat ĠE q Em bs Ġprovide Ġmore Ġefficient Ġmodeling Ġthan Ġexisting Ġword Ġembed ding Ġmethods . ĠWe Ġfurther Ġcarried Ġout Ġan Ġexplor atory Ġanalysis Ġof Ġa Ġlarge Ġset Ġof ĠIN LINE FORM 0 Ġ87 k Ġequations . ĠWe Ġfound Ġthat ĠE q Em bs Ġprovide Ġbetter Ġmodels Ġwhen Ġcompared Ġto Ġexisting Ġword Ġembed ding Ġapproaches . ĠE q Em bs Ġalso Ġprovide Ġcoherent Ġsemantic Ġrepresentations Ġof Ġequations Ġand Ġcan Ġcapture Ġsemantic Ġsimilarity Ġto Ġother Ġequations Ġand Ġto Ġwords . </s> Related ĠWork </s> Word Ġembed d ings Ġwere Ġfirst Ġintroduced Ġin ĠB IB REF 2 Ġ, ĠB IB REF 3 Ġand Ġthere Ġhave Ġbeen Ġmany Ġvariants ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġ, ĠB IB REF 7 Ġ. ĠCommon Ġfor Ġall Ġof Ġthem Ġis Ġthe Ġidea Ġthat Ġwords Ġcan Ġbe Ġrepresented Ġby Ġlatent Ġfeature Ġvectors . ĠThese Ġfeature Ġvectors Ġare Ġoptimized Ġto Ġmaximize Ġthe Ġconditional Ġprobability Ġof Ġthe Ġdataset . ĠRecently ĠB IB REF 1 Ġextended Ġthe Ġidea Ġof Ġword Ġembed d ings Ġto Ġother Ġtypes Ġof Ġdata . ĠE q Emb Ġexpand Ġthe Ġidea Ġof Ġword Ġembed d ings Ġto Ġa Ġnew Ġtype Ġof Ġdata Ġpoints ĠâĢĵ Ġequations . </s> There Ġhave Ġbeen Ġdifferent Ġproposed Ġapproaches Ġfor Ġrepresenting Ġmathematical Ġequations . ĠB IB REF 8 Ġintroduced Ġthe Ġsymbol Ġlayout Ġtree , Ġa Ġrepresentation Ġthat Ġenc odes Ġthe Ġspatial Ġrelationship Ġof Ġvariables Ġand Ġoperators Ġfor Ġthe Ġpurpose Ġof Ġindex ing Ġand Ġretrieving Ġmathematical Ġequations . ĠOur Ġwork Ġalso Ġfalls Ġinto Ġthe Ġframework Ġof Ġmathematical Ġlanguage Ġprocessing Ġ( ML P ) ĠB IB REF 9 Ġwhose Ġfirst Ġstep Ġis Ġconverting Ġmathematical Ġsolutions Ġinto Ġa Ġseries Ġof Ġnumerical Ġfeatures . </s> Equ ation ĠEmb edd ings ĠModels </s> E q Emb Ġare Ġbased Ġon Ġword Ġembed d ings ĠB IB REF 5 Ġor Ġspecifically ĠBern ou lli Ġembed d ings Ġ( b - em bs ) ĠB IB REF 1 Ġ. ĠWord Ġembed d ings Ġmodels Ġthe Ġprobability Ġof Ġa Ġword ĠIN LINE FORM 0 Ġgiven Ġits Ġcontext ĠIN LINE FORM 1 Ġas Ġa Ġconditional Ġdistribution ĠIN LINE FORM 2 Ġwhere Ġthe Ġcontext Ġis Ġdefined Ġas Ġthe Ġset Ġof Ġwords ĠIN LINE FORM 3 Ġin Ġa Ġwindow Ġof Ġsize ĠIN LINE FORM 4 Ġthat Ġsurrounds Ġit . ĠIn Ġword Ġembed d ings Ġeach Ġword Ġis Ġassigned Ġto Ġtwo Ġtypes Ġof Ġlatent Ġfeature Ġvectors , Ġthe Ġembed ding Ġ( ĠIN LINE FORM 5 Ġ) Ġand Ġcontext Ġ( ĠIN LINE FORM 6 Ġ) Ġvectors , Ġboth Ġof Ġwhich Ġare ĠIN LINE FORM 7 Ġdimensional . </s> B - emb Ġis Ġan Ġexponential Ġfamily Ġembed ding Ġmodel Ġwhere Ġthe Ġconditional Ġdistribution Ġis Ġa ĠBern ou lli : ĠDIS PLAY FORM 0 </s> The Ġparameter ĠIN LINE FORM 0 Ġis Ġdefined Ġusing Ġthe Ġword Ġembed ding ĠIN LINE FORM 1 Ġand Ġthe Ġword Ġcontext ĠIN LINE FORM 2 Ġvectors : ĠDIS PLAY FORM 0 </s> where ĠIN LINE FORM 0 Ġis Ġthe Ġlog istic Ġfunction . </s> Equ ation ĠEmb edd ings </s> Given Ġa Ġdataset Ġof Ġwords Ġand Ġequations Ġthe Ġgoal Ġof Ġthe ĠE q Emb Ġmodels Ġis Ġto Ġderive Ġa Ġsemantic Ġrepresentation Ġof Ġeach Ġequation . ĠE q Emb Ġmodel Ġequations Ġin Ġthe Ġcontext Ġof Ġwords . ĠE q Emb Ġis Ġbased Ġon Ġthe Ġidea Ġthat Ġa Ġgood Ġsemantic Ġrepresentation Ġof Ġequations Ġcould Ġbe Ġdiscovered Ġby Ġexpanding Ġthe Ġoriginal Ġword Ġcontext Ġto Ġinclude Ġany Ġequations Ġthat Ġappear Ġin Ġa Ġpossibly Ġlarger Ġwindow Ġaround Ġit . </s> We Ġassign Ġembed d ings Ġto Ġwords Ġ( ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ) Ġand Ġequations Ġ( ĠIN LINE FORM 2 Ġ, ĠIN LINE FORM 3 Ġ). ĠThe Ġobjective Ġfunction Ġcontains Ġcondition als Ġover Ġthe Ġobserved Ġwords Ġand Ġequations : ĠDIS PLAY FORM 0 </s> This Ġis Ġa Ġsum Ġof Ġtwo Ġsets Ġof Ġconditional Ġdistributions , Ġthe Ġfirst Ġover Ġobserved Ġwords Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġthe Ġsecond Ġover Ġobserved Ġequations Ġ( ĠIN LINE FORM 1 Ġ). ĠIn Ġword Ġembed ding Ġmodels , ĠIN LINE FORM 2 Ġand ĠIN LINE FORM 3 Ġare Ġreferred Ġto Ġas Ġembed ding Ġand Ġcontext Ġvectors . ĠHere Ġwe Ġuse Ġa Ġdifferent Ġterminology : Ġthe Ġinteraction ĠIN LINE FORM 4 Ġand Ġfeature Ġvector ĠIN LINE FORM 5 Ġ. </s> In Ġword Ġembed d ings , Ġthe Ġcontext Ġof Ġthe Ġword ĠIN LINE FORM 0 Ġis Ġdefined Ġto Ġindex Ġthe Ġsurrounding Ġwords Ġin Ġa Ġsmall Ġwindow Ġaround Ġit . ĠHere Ġthe Ġcontext Ġof Ġthe Ġword ĠIN LINE FORM 1 Ġwill Ġbe Ġthe Ġoriginal Ġcontext Ġ( ĠIN LINE FORM 2 Ġ) Ġand Ġany Ġequations Ġ( ĠIN LINE FORM 3 Ġ) Ġthat Ġare Ġin Ġa Ġpossibly Ġlarger Ġwindow Ġaround Ġit . ĠThis Ġis Ġreferred Ġto Ġas Ġthe Ġword - equ ation Ġcontext Ġwindow . </s> Both Ġcondition als Ġare ĠBern ou lli Ġdistributions . ĠThe Ġfirst Ġconditional Ġis Ġdefined Ġover Ġthe Ġwords Ġin Ġthe Ġcollection . ĠIt Ġhas Ġthe Ġfollowing Ġparameter : ĠDIS PLAY FORM 0 </s> The Ġword Ġcontext Ġfunction Ġis : ĠDIS PLAY FORM 0 </s> This Ġfunction Ġencompasses Ġthe Ġwords Ġin Ġthe Ġoriginal Ġword Ġcontext Ġ( ĠIN LINE FORM 0 Ġ) Ġand Ġany Ġequations Ġ( ĠIN LINE FORM 1 Ġ) Ġthat Ġappear Ġin Ġa Ġpossibly Ġlarger Ġwindow Ġ( ĠIN LINE FORM 2 Ġ) Ġaround Ġit . </s> The Ġsecond Ġterm Ġin Ġthe Ġobjective Ġcorresponds Ġto Ġthe Ġsum Ġof Ġthe Ġlog Ġconditional Ġprobabilities Ġof Ġeach Ġequation . ĠIts Ġparameter Ġis : ĠDIS PLAY FORM 0 </s> Similar Ġto Ġword Ġembed d ings , Ġequation Ġcontext ĠIN LINE FORM 0 Ġcontains Ġwords Ġthat Ġare Ġin Ġa Ġcontext Ġwindow Ġaround Ġthe Ġequation : ĠDIS PLAY FORM 0 </s> The Ġequation Ġcontext Ġcan Ġhave Ġa Ġlarger Ġwindow Ġthan Ġthe Ġword Ġcontext . ĠEqu ation Ġfeature Ġvectors Ġ( ĠIN LINE FORM 0 Ġ) Ġare Ġonly Ġassociated Ġwith Ġthe Ġfirst Ġterm Ġof Ġthe Ġobjective Ġfunction . ĠThis Ġfunction Ġcontains Ġthe Ġwords Ġwhere Ġthe Ġequation Ġappears Ġin Ġtheir Ġlarger Ġcontext ĠIN LINE FORM 1 Ġ. </s> The Ġleft Ġside Ġof ĠFigure ĠFIG REF 1 Ġshows Ġan Ġexample Ġequation Ġin Ġa Ġscientific Ġarticle . ĠWith Ġa Ġword Ġcontext Ġof Ġsize ĠIN LINE FORM 0 Ġwe Ġmodel Ġthe Ġwords Ġin Ġthe Ġarticle Ġwhile Ġignoring Ġequations . ĠFor Ġexample Ġwhen Ġmodeling Ġthe Ġword Ġ" embed ding " Ġ( high light ed Ġin Ġgreen ) Ġwith Ġcontext Ġwindow Ġsize Ġof Ġ4 Ġ( i . e . ĠIN LINE FORM 1 Ġ), Ġthe Ġcontext Ġcontains Ġthe Ġwords Ġthat Ġappear Ġtwo Ġwords Ġbefore Ġ(" current " Ġand Ġ" word ") Ġand Ġafter Ġ(" rec urrent " Ġand Ġ" version ") Ġthis Ġword . ĠWith Ġa Ġword - equ ation Ġcontext Ġwindow Ġof Ġsize ĠIN LINE FORM 2 Ġ= 16 , Ġthe Ġterm Ġfor Ġthe Ġword Ġ" embed ding " Ġwould Ġhave Ġthe Ġfeature Ġvector Ġof Ġthe Ġequation Ġas Ġone Ġof Ġits Ġcomponents . </s> Equ ation ĠUnit ĠEmb edd ings </s> Building Ġon Ġour Ġprevious Ġmethod , Ġwe Ġdefine Ġa Ġnew Ġmodel Ġwhich Ġwe Ġcall Ġequation Ġunit Ġembed d ings Ġ( E q Emb - U ). ĠE q Emb - U Ġmodel Ġequations Ġby Ġtreating Ġthem Ġas Ġsentences Ġwhere Ġthe Ġwords Ġare Ġthe Ġequation Ġvariables , Ġsymbols Ġand Ġoperators Ġwhich Ġwe Ġrefer Ġto Ġas Ġunits . ĠThe Ġfirst Ġstep Ġin Ġrepresenting Ġequations Ġusing Ġequation Ġunits Ġis Ġto Ġtoken ize Ġthem . ĠWe Ġuse Ġthe Ġapproach Ġoutlined Ġin ĠB IB REF 8 Ġwhich Ġrepresents Ġequations Ġinto Ġa Ġsyntax Ġlayout Ġtree Ġ( SL T ), Ġa Ġsequence Ġof ĠSL T Ġtu ples Ġeach Ġof Ġwhich Ġcontains Ġthe Ġspatial Ġrelationship Ġinformation Ġbetween Ġtwo Ġequation Ġsymbols Ġfound Ġwithin Ġa Ġparticular Ġwindow Ġof Ġequation Ġsymbols . ĠFigure ĠFIG REF 11 Ġshows Ġexample ĠSL T Ġrepresentations Ġof Ġthree Ġequations . </s> Each Ġequation ĠIN LINE FORM 0 Ġis Ġa Ġsequence Ġof Ġequation Ġunits ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġsimilar Ġto Ġa Ġsentence Ġwhere Ġthe Ġwords Ġare Ġthe Ġequation Ġunits . ĠFor Ġeach Ġequation Ġunit ĠIN LINE FORM 3 Ġwe Ġassign Ġinteraction ĠIN LINE FORM 4 Ġand Ġfeature ĠIN LINE FORM 5 Ġvectors . </s> We Ġassume Ġthat Ġthe Ġcontext Ġof Ġthe Ġword ĠIN LINE FORM 0 Ġwill Ġbe Ġthe Ġoriginal Ġcontext Ġ( ĠIN LINE FORM 1 Ġ) Ġand Ġthe Ġequation Ġunits Ġ( ĠIN LINE FORM 2 Ġ) Ġof Ġany Ġequations Ġthat Ġare Ġin Ġthe Ġword - equ ation Ġcontext Ġwindow . ĠIn Ġaddition Ġfor Ġeach Ġequation Ġunit Ġwe Ġdefine Ġits Ġunit Ġcontext ĠIN LINE FORM 3 Ġto Ġbe Ġthe Ġset Ġof Ġsurrounding Ġequation Ġunits Ġin Ġa Ġsmall Ġwindow ĠIN LINE FORM 4 Ġaround Ġit : ĠDIS PLAY FORM 0 </s> The Ġobjective Ġis Ġover Ġtwo Ġcondition als , Ġone Ġfor Ġeach Ġcontext Ġtype : ĠDIS PLAY FORM 0 </s> The Ġtwo Ġparameters Ġare : ĠDIS PLAY FORM 0 </s> We Ġdefine Ġequation - level Ġrepresentations Ġby Ġaveraging Ġthe Ġrepresentations Ġof Ġtheir Ġconstituent Ġunits : ĠDIS PLAY FORM 0 </s> Com put ation </s> We Ġuse Ġsto ch astic Ġgradient Ġdescent Ġwith ĠAd ag rad ĠB IB REF 10 Ġto Ġfit Ġthe Ġembed ding Ġand Ġcontext Ġvectors . ĠFollowing ĠB IB REF 1 Ġ, Ġwe Ġreduce Ġthe Ġcomputational Ġcomplexity Ġby Ġsplitting Ġthe Ġgradient Ġinto Ġtwo Ġterms . ĠThe Ġfirst Ġterm Ġcontains Ġthe Ġnon - zero Ġentries Ġ( ĠIN LINE FORM 0 Ġ); Ġthe Ġsecond Ġterm Ġcontains Ġthe Ġzero Ġentries Ġ( ĠIN LINE FORM 1 Ġ). ĠWe Ġcompute Ġthe Ġexact Ġgradient Ġfor Ġthe Ġnon - zero Ġpoints ; ĠWe Ġsubs ample Ġfor Ġthe Ġzero Ġdata Ġpoints . ĠThis Ġis Ġsimilar Ġto Ġnegative Ġsampling ĠB IB REF 5 Ġ, Ġwhich Ġalso Ġdown - weights Ġthe Ġcontributions Ġof Ġthe Ġzero Ġpoints . ĠUnlike ĠB IB REF 1 Ġwhich Ġuses ĠIN LINE FORM 2 Ġregular ization Ġto Ġprotect Ġagainst Ġover fitting Ġwhen Ġfitting Ġthe Ġembed ding Ġvectors Ġwe Ġuse Ġearly Ġstopping Ġbased Ġon Ġvalidation Ġaccuracy , Ġfor Ġthe Ġsame Ġeffect . </s> E mp ir ical ĠStudy </s> We Ġstudied Ġthe Ġperformance Ġof ĠE q Emb Ġon Ġarticles Ġfrom Ġthe Ġar X iv . ĠE q Emb Ġmodels Ġprovide Ġbetter Ġfits Ġthan Ġexisting Ġembed ding Ġapproaches , Ġand Ġinfer Ġmeaningful Ġsemantic Ġrelationships Ġbetween Ġequations Ġand Ġwords Ġin Ġthe Ġcollection . </s> We Ġpresent Ġa Ġcomparison Ġof Ġthe Ġproposed Ġmodels Ġto Ġexisting Ġword Ġembed d ings Ġapproaches . ĠThese Ġare : Ġthe ĠBern ou lli Ġembed d ings Ġ( b - emb ) ĠB IB REF 1 Ġ, Ġcontinuous Ġbag - of - words Ġ( CB OW ) ĠB IB REF 5 Ġ, ĠDist ributed ĠMemory Ġversion Ġof ĠPar agraph ĠVector Ġ( P V - DM ) ĠB IB REF 11 Ġand Ġthe ĠGlobal ĠV ect ors Ġ( G lo Ve ) ĠB IB REF 6 Ġmodel . </s> Dat as ets </s> Our Ġdatasets Ġare Ġscientific Ġarticles Ġthat Ġwere Ġpublished Ġon Ġar X iv . ĠThe Ġsets Ġcontain Ġarticles Ġ( in ĠLa TeX Ġformat ) Ġfrom Ġfour Ġcomputer Ġscience Ġdomains : ĠN LP , ĠIR , ĠAI , Ġand ĠML . ĠThey Ġwere Ġcreated Ġby Ġfiltering Ġar X iv Ġarticles Ġbased Ġon Ġtheir Ġprimary Ġand Ġsecondary Ġcategories . ĠWe Ġused Ġthe Ġfollowing Ġcategories Ġfor Ġthe Ġfour Ġcollections : Ġcs . cl Ġfor ĠN LP ; Ġcs . ir Ġfor ĠIR ; Ġcs . ai Ġfor ĠAI Ġand Ġstat . ml , Ġstat . co , Ġstat . me Ġor Ġcs . l g Ġfor ĠML . </s> Table ĠT AB REF 22 Ġshows Ġthe Ġnumber Ġof Ġdocuments Ġalong Ġwith Ġthe Ġnumber Ġof Ġunique Ġwords , Ġequations Ġand Ġequation Ġunits Ġfor Ġeach Ġcollection . ĠThe Ġequations Ġare Ġdisplay Ġequations Ġthat Ġwere Ġenumer ated Ġin Ġthe ĠLa TeX Ġversion Ġof Ġthe Ġarticles . ĠUnlike Ġinline Ġequations , Ġwhich Ġin Ġmany Ġinstances Ġrepresent Ġvariables Ġwith Ġgeneral Ġmeaning Ġ( e . g . ĠIN LINE FORM 0 Ġ, ĠIN LINE FORM 1 Ġ, ĠIN LINE FORM 2 Ġ, Ġetc .) Ġand Ġeven Ġnumerical Ġvalues , Ġdisplay Ġequations Ġtypically Ġrepresent Ġmathematical Ġconcepts Ġwith Ġmore Ġspecific Ġsemantics . ĠFor Ġthe Ġempirical Ġstudy Ġwe Ġused Ġa Ġrandom Ġsubset Ġof Ġ2 k Ġsing let ons Ġfrom Ġthe Ġtotal Ġcollection , Ġalong Ġwith Ġall Ġequations Ġthat Ġoccur Ġmore Ġthan Ġonce . ĠFor Ġthe Ġqualitative Ġanalysis , Ġwe Ġused Ġall Ġequations . </s> We Ġextracted Ġwords Ġby Ġtoken izing Ġarticles Ġusing Ġthe ĠNL T K Ġpackage ĠB IB REF 12 Ġand Ġrestricted Ġthe Ġvocabulary Ġto Ġnoun Ġphrases Ġand Ġadject ives . ĠThe Ġvocabulary Ġwas Ġselected Ġby : </s> rem oving Ġcommon Ġstop words </s> t reating Ġthe Ġtop Ġ25 Ġmost Ġfrequent Ġwords Ġas Ġstop Ġwords Ġand Ġremoving Ġthem </s> including Ġwords Ġwhose Ġterm Ġfrequency Ġis Ġgreater Ġthan Ġor Ġequal Ġto Ġ10 Ġand Ġwhose Ġcharacter Ġlength Ġis Ġgreater Ġthan Ġor Ġequal Ġto Ġ4 </s> including Ġthe Ġtop Ġ50 Ġmost Ġfrequent Ġabbrevi ations Ġwhose Ġcharacter Ġlength Ġis Ġ3 Ġ( an Ġexception Ġto Ġour Ġprevious Ġrule ) </s> When Ġtoken izing Ġequations , Ġwe Ġfirst Ġcreate Ġan Ġeffective Ġvocabulary Ġof Ġequation Ġunits . ĠWe Ġconvert Ġequations Ġinto ĠSL T Ġformat Ġand Ġcollect Ġcollection Ġwide Ġfrequency Ġstatistics Ġover Ġthe Ġequation Ġunits . ĠThe Ġvocabulary Ġcontains Ġall Ġequation Ġunits Ġwhose Ġfrequency Ġcount Ġis Ġgreater Ġthan ĠIN LINE FORM 0 Ġ. </s> Exper imental ĠSetup </s> We Ġanalyzed ĠE q Emb Ġmodels Ġperformance Ġusing Ġa Ġheld Ġout Ġset Ġof Ġwords Ġthat Ġwe Ġgenerate Ġfor Ġeach Ġequation Ġin Ġour Ġcollections . ĠHeld Ġout Ġsets Ġare Ġconstructed Ġusing Ġthe Ġfollowing Ġprocedure : ĠWe Ġtraverse Ġover Ġthe Ġcollections Ġand Ġfor Ġevery Ġdiscovered Ġequation Ġwe Ġrandomly Ġsample Ġwords Ġfrom Ġits Ġcontext Ġset . ĠThe Ġheld Ġout Ġset Ġcontains Ġthe Ġsampled Ġwords Ġand Ġtheir Ġcontext Ġwindow Ġwhich Ġalso Ġincludes Ġthe Ġequation . ĠFor Ġeach Ġheld Ġout Ġword Ġwe Ġalso Ġgenerate Ġa Ġset Ġof Ġnegative Ġsamples Ġfor Ġthe Ġgiven Ġword Ġcontext . ĠWe Ġperform Ġthe Ġsame Ġprocedure Ġto Ġform Ġa Ġvalidation Ġset . ĠFor Ġeach Ġof Ġthe ĠIN LINE FORM 0 Ġequations Ġin Ġa Ġcollection , Ġtwo Ġheld Ġout Ġwords ĠIN LINE FORM 1 Ġare Ġsampled . ĠFor Ġa Ġcontext Ġwindow Ġof Ġsize Ġ4 Ġthe Ġsampled Ġword Ġcontext Ġis Ġdefined Ġas ĠIN LINE FORM 2 Ġ. </s> During Ġtraining Ġwe Ġcompute Ġthe Ġpredictive Ġlog - like lihood Ġon Ġthe Ġvalidation Ġset Ġof Ġwords Ġusing Ġthe Ġfitted Ġmodel Ġafter Ġeach Ġiteration Ġover Ġthe Ġcollection . ĠA Ġfitted Ġmodel Ġis Ġa Ġcollection Ġof Ġinteraction Ġand Ġfeature Ġvectors Ġfor Ġeach Ġequation Ġand Ġword . ĠGiven Ġa Ġfitted Ġmodel , Ġthe Ġlog Ġprobability Ġof Ġa Ġheld Ġout Ġword Ġis Ġcomputed Ġusing Ġthe Ġfollowing Ġformula : ĠDIS PLAY FORM 0 </s> which Ġis Ġthe Ġsoft max Ġfunction Ġcomputed Ġover Ġa Ġset Ġof Ġnegative Ġsamples ĠIN LINE FORM 0 Ġand Ġthe Ġheld Ġout Ġword . ĠIn Ġparticular , Ġwe Ġran Ġthe Ġmodel Ġ20 Ġtimes Ġacross Ġthe Ġcollection . ĠAfter Ġeach Ġcollection Ġiteration ĠIN LINE FORM 1 Ġwe Ġobserve Ġwhether Ġthe Ġpredictive Ġlog - like lihood Ġcontinues Ġto Ġimprove Ġcompared Ġto Ġthe Ġprevious Ġ( ĠIN LINE FORM 2 Ġ) Ġiteration . ĠWe Ġstop Ġat Ġthe ĠIN LINE FORM 3 Ġ- th Ġiteration Ġwhen Ġthat Ġis Ġno Ġlonger Ġthe Ġcase . </s> When Ġmodeling Ġequations Ġusing ĠE q Emb Ġwe Ġperform Ġtwo Ġpasses Ġover Ġthe Ġcollection . ĠIn Ġthe Ġfirst Ġpass Ġwe Ġonly Ġmodel Ġwords Ġwhile Ġignoring Ġequations . ĠIn Ġthe Ġsecond Ġpass Ġwe Ġonly Ġmodel Ġequations Ġwhile Ġholding Ġfixed Ġthe Ġinteraction Ġand Ġfeature Ġvectors Ġof Ġall Ġwords . ĠIn Ġcontext Ġof ĠE q Emb Ġwe Ġtreat Ġequations Ġas Ġsingle ton Ġwords Ġand Ġthe Ġbroader Ġquestion Ġthat Ġwe Ġare Ġtrying Ġto Ġanswer Ġis Ġwhether Ġwe Ġcan Ġlearn Ġsomething Ġabout Ġthe Ġmeaning Ġof Ġthe Ġsingle ton Ġwords Ġgiven Ġthe Ġfixed Ġword Ġinteraction Ġand Ġfeature Ġvectors . </s> In Ġour Ġanalysis Ġwe Ġevaluated Ġthe Ġperformance Ġof Ġthe ĠE q Emb Ġmodels Ġacross Ġdifferent Ġsizes Ġfor Ġthe Ġword Ġcontext Ġ( W ), Ġword - equ ation Ġcontext Ġ( E ) Ġand Ġembed ding Ġvector Ġsize Ġ( K ). ĠModel Ġperformance Ġwas Ġcompared Ġwith Ġ4 Ġexisting Ġembed ding Ġmodels : Ġb - emb , ĠCB OW , ĠGl o Ve Ġand ĠPV - DM . ĠWe Ġused Ġthe Ġg ens im ĠB IB REF 13 Ġimplementation Ġof Ġthe ĠCB OW Ġand ĠPV - DM Ġmodels . ĠWhen Ġmodeling Ġequations Ġusing Ġthe Ġfirst Ġ3 Ġembed ding Ġmodels Ġwe Ġtreat Ġequations Ġas Ġregular Ġwords Ġin Ġthe Ġcollection . ĠIn Ġcase Ġof Ġthe ĠPV - DM Ġmodel Ġwe Ġparse Ġthe Ġarticle Ġso Ġthat Ġequations Ġand Ġtheir Ġsurrounding Ġcontext Ġof Ġlength Ġequivalent Ġto Ġthe Ġword - equ ation Ġcontext Ġwindow Ġare Ġlabeled Ġas Ġa Ġseparate Ġparagraph . ĠWe Ġalso Ġassign Ġparagraph Ġlabels Ġto Ġthe Ġarticle Ġtext Ġoccurring Ġbetween Ġequation Ġparagraphs . </s> Results </s> Table ĠT AB REF 23 Ġshows Ġthe Ġperformance Ġcomparison Ġresults Ġacross Ġthe Ġdifferent Ġembed d ings Ġmodels . ĠFor Ġeach Ġmodel , Ġperformance Ġresults Ġare Ġshown Ġon Ġ4 Ġlatent Ġdimension Ġvalues Ġ( K = 25 , Ġ50 , Ġ75 Ġand Ġ100 ). ĠFor Ġeach Ġdimension Ġwe Ġran Ġexperiments Ġby Ġvarying Ġthe Ġcontext Ġwindow Ġsize Ġfor Ġwords Ġ( Word ĠContext = 4 , Ġ8 Ġand Ġ16 ). ĠIn Ġaddition Ġfor Ġthe ĠE q Emb , ĠE q Emb - U Ġand ĠPV - DM Ġmodels Ġwe Ġalso Ġvaried Ġthe Ġword - equ ation Ġwindow Ġsize Ġ( E = 8 Ġand Ġ16 ). ĠCompar isons Ġacross Ġmodels Ġare Ġperformed Ġusing Ġthe Ġpseudo Ġlog - like lihood Ġmeasure ĠB IB REF 14 Ġ. ĠFor Ġa Ġgiven Ġheld - out Ġword ĠIN LINE FORM 0 Ġand Ġa Ġset Ġof Ġnegative Ġsamples ĠIN LINE FORM 1 Ġthe Ġpseudo Ġlog - like lihood Ġis Ġdefined Ġas : ĠDIS PLAY FORM 0 </s> We Ġtreat Ġthis Ġa Ġdownstream Ġtask . ĠFor Ġeach Ġmodel Ġtype Ġand Ġlatent Ġdimension Ġconfiguration , Ġwe Ġuse Ġthe Ġvalidation Ġset Ġto Ġselect Ġthe Ġbest Ġmodel Ġconfiguration Ġ( i . e . Ġcombination Ġof Ġcontext Ġwindow Ġsizes ). ĠWe Ġreport Ġvalues Ġon Ġboth Ġdatasets . </s> Across Ġall Ġcollections ĠE q Emb Ġoutper form Ġprevious Ġembed ding Ġmodels Ġand ĠE q Emb - U Ġfurther Ġimproves Ġperformance . </s> Word ĠRepresent ation Ġof ĠEqu ations </s> E q Emb Ġhelp Ġobtain Ġword Ġdescriptions Ġof Ġthe Ġequations . ĠTable ĠT AB REF 25 Ġshows Ġexample Ġequation Ġand Ġthe Ġ5 Ġmost Ġsimilar Ġwords Ġobtained Ġusing Ġ4 Ġdifferent Ġembed ding Ġapproaches Ġwhich Ġinclude ĠCB OW , ĠPV - DM , ĠGl o Ve Ġand ĠE q Emb . ĠFor Ġthe Ġquery Ġequation Ġwe Ġobtain Ġmost Ġsimilar Ġwords Ġby Ġcomputing ĠCos ine Ġdistance Ġbetween Ġthe Ġembed ding Ġvector Ġ( ĠIN LINE FORM 0 Ġ) Ġrepresentation Ġof Ġthe Ġquery Ġequation Ġand Ġthe Ġcontext Ġvector Ġrepresentation Ġof Ġthe Ġwords Ġ( ĠIN LINE FORM 1 Ġ). </s> With Ġthe Ġembed ding Ġrepresentation Ġof Ġwords Ġand Ġequations Ġwe Ġcould Ġalso Ġperform Ġequation Ġsearch Ġusing Ġwords Ġas Ġqueries . ĠFor Ġa Ġset Ġof Ġquery Ġwords Ġwe Ġgenerate Ġits Ġembed ding Ġrepresentation Ġby Ġtaking Ġthe Ġaverage Ġof Ġthe Ġembed ding Ġrepresentation Ġof Ġeach Ġword Ġand Ġcompute ĠCos ine Ġdistance Ġacross Ġall Ġthe Ġequations Ġembed d ings . ĠTable ĠT AB REF 25 Ġshows Ġan Ġexample Ġquery , Ġwhich Ġconsists Ġof Ġthree Ġwords , Ġand Ġits Ġ5 Ġnearest Ġequations Ġdiscovered Ġusing ĠE q Emb . ĠFor Ġa Ġgiven Ġword Ġquery , ĠE q Emb Ġare Ġable Ġto Ġretrieve Ġquery Ġrelevant Ġequations . </s> Discover ing ĠSem antically ĠSimilar ĠEqu ations </s> In Ġaddition Ġto Ġwords , ĠE q Emb Ġmodels Ġcan Ġcapture Ġthe Ġsemantic Ġsimilarity Ġbetween Ġequations Ġin Ġthe Ġcollection . ĠWe Ġperformed Ġqualitative Ġanalysis Ġof Ġthe Ġmodel Ġperformance Ġusing Ġall Ġdiscovered Ġequations Ġacross Ġthe Ġ4 Ġcollection . ĠTable ĠT AB REF 24 Ġshows Ġthe Ġquery Ġequation Ġused Ġin Ġthe Ġprevious Ġanalysis Ġand Ġits Ġ5 Ġmost Ġsimilar Ġequations Ġdiscovered Ġusing ĠE q Emb - U . ĠFor Ġqualitative Ġcomparisons Ġacross Ġthe Ġother Ġembed ding Ġmodels , Ġin ĠAppendix ĠA Ġwe Ġprovide Ġresults Ġover Ġthe Ġsame Ġquery Ġusing ĠCB OW , ĠPV - DM , ĠGl o Ve Ġand ĠE q Emb . ĠIn ĠAppendix ĠA Ġreader Ġshould Ġnotice Ġthe Ġdifference Ġin Ġperformance Ġbetween ĠE q Emb - U Ġand ĠE q Emb Ġcompared Ġto Ġexisting Ġembed ding Ġmodels Ġwhich Ġfail Ġto Ġdiscover Ġsem antically Ġsimilar Ġequations . Ġtab : ire x ample 1 , tab : nl pex ample 2 Ġshow Ġtwo Ġadditional Ġexample Ġequation Ġand Ġits Ġ5 Ġmost Ġsimilar Ġequations Ġand Ġwords Ġdiscovered Ġusing Ġthe ĠE q Emb Ġmodel . ĠSimilar Ġwords Ġwere Ġranked Ġby Ġcomputing ĠCos ine Ġdistance Ġbetween Ġthe Ġembed ding Ġvector Ġ( ĠIN LINE FORM 0 Ġ) Ġrepresentation Ġof Ġthe Ġquery Ġequation Ġand Ġthe Ġcontext Ġvector Ġrepresentation Ġof Ġthe Ġwords Ġ( ĠIN LINE FORM 1 Ġ). ĠSimilar Ġequations Ġwere Ġdiscovered Ġusing ĠEucl ide an Ġdistance Ġcomputed Ġbetween Ġthe Ġcontext Ġvector Ġrepresentations Ġof Ġthe Ġequations Ġ( ĠIN LINE FORM 2 Ġ). ĠWe Ġgive Ġadditional Ġexample Ġresults Ġin ĠAppendix ĠB . </s> Conclusion </s> We Ġpresented Ġun super vised Ġapproaches Ġfor Ġsemantic Ġrepresentations Ġof Ġmathematical Ġequations Ġusing Ġtheir Ġsurrounding Ġwords . ĠAcross Ġ4 Ġdifferent Ġcollections Ġwe Ġshowed Ġthat Ġout Ġmethods Ġoffer Ġmore Ġeffective Ġmodeling Ġcompared Ġto Ġexisting Ġembed ding Ġmodels . ĠWe Ġalso Ġdemonstrate Ġthat Ġthey Ġcan Ġcapture Ġthe Ġsemantic Ġsimilarity Ġbetween Ġequations Ġand Ġthe Ġwords Ġin Ġthe Ġcollection . ĠIn Ġthe Ġfuture Ġwe Ġplan Ġto Ġexplore Ġhow ĠE q Emb Ġcould Ġbe Ġexpend Ġto Ġrepresent Ġother Ġobjects Ġsuch Ġas Ġimages , Ġcapt ions Ġand Ġinline Ġfigures .