a simple word-level encoder, The encoder is essentially the same as tweet2vec, with the input as words instead of characters.
EXTRACTIVE
The encoder is essentially the same as tweet2vec, with the input as words instead of characters.
EXTRACTIVE
Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets. The input tweet is first split into tokens along white-spaces. A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum. The encoder is essentially the same as tweet2vec, with the input as words instead of characters. A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token.
Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets. The input tweet is first split into tokens along white-spaces. A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum. The encoder is essentially the same as tweet2vec, with the input as words instead of characters. A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token.
1605.03481
Introduction </s> We Ġunderstand Ġfrom ĠZip f 's ĠLaw Ġthat Ġin Ġany Ġnatural Ġlanguage Ġcorpus Ġa Ġmajority Ġof Ġthe Ġvocabulary Ġword Ġtypes Ġwill Ġeither Ġbe Ġabsent Ġor Ġoccur Ġin Ġlow Ġfrequency . ĠEst imating Ġthe Ġstatistical Ġproperties Ġof Ġthese Ġrare Ġword Ġtypes Ġis Ġnaturally Ġa Ġdifficult Ġtask . ĠThis Ġis Ġanalogous Ġto Ġthe Ġcurse Ġof Ġdimension ality Ġwhen Ġwe Ġdeal Ġwith Ġsequences Ġof Ġtokens Ġ- Ġmost Ġsequences Ġwill Ġoccur Ġonly Ġonce Ġin Ġthe Ġtraining Ġdata . ĠNeural Ġnetwork Ġarchitectures Ġovercome Ġthis Ġproblem Ġby Ġdefining Ġnon - linear Ġcompos itional Ġmodels Ġover Ġvector Ġspace Ġrepresentations Ġof Ġtokens Ġand Ġhence Ġassign Ġnon - zero Ġprobability Ġeven Ġto Ġsequences Ġnot Ġseen Ġduring Ġtraining ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġexplore Ġa Ġsimilar Ġapproach Ġto Ġlearning Ġdistributed Ġrepresentations Ġof Ġsocial Ġmedia Ġposts Ġby Ġcomposing Ġthem Ġfrom Ġtheir Ġconstituent Ġcharacters , Ġwith Ġthe Ġgoal Ġof Ġgeneral izing Ġto Ġout - of - voc abulary Ġwords Ġas Ġwell Ġas Ġsequences Ġat Ġtest Ġtime . </s> Traditional ĠNeural ĠNetwork ĠLanguage ĠModels Ġ( NN L Ms ) Ġtreat Ġwords Ġas Ġthe Ġbasic Ġunits Ġof Ġlanguage Ġand Ġassign Ġindependent Ġvectors Ġto Ġeach Ġword Ġtype . ĠTo Ġconst rain Ġmemory Ġrequirements , Ġthe Ġvocabulary Ġsize Ġis Ġfixed Ġbefore - hand ; Ġtherefore , Ġrare Ġand Ġout - of - voc abulary Ġwords Ġare Ġall Ġgrouped Ġtogether Ġunder Ġa Ġcommon Ġtype Ġ` UN KNOWN '. ĠThis Ġchoice Ġis Ġmotivated Ġby Ġthe Ġassumption Ġof Ġarbit rar iness Ġin Ġlanguage , Ġwhich Ġmeans Ġthat Ġsurface Ġforms Ġof Ġwords Ġhave Ġlittle Ġto Ġdo Ġwith Ġtheir Ġsemantic Ġroles . ĠRecently , ĠB IB REF 2 Ġchallenge Ġthis Ġassumption Ġand Ġpresent Ġa Ġbid irection al ĠLong ĠShort ĠTerm ĠMemory Ġ( L ST M ) ĠB IB REF 3 Ġfor Ġcomposing Ġword Ġvectors Ġfrom Ġtheir Ġconstituent Ġcharacters Ġwhich Ġcan Ġmemor ize Ġthe Ġarbitrary Ġaspects Ġof Ġword Ġorth ography Ġas Ġwell Ġas Ġgeneral ize Ġto Ġrare Ġand Ġout - of - voc abulary Ġwords . </s> Enc our aged Ġby Ġtheir Ġfindings , Ġwe Ġextend Ġtheir Ġapproach Ġto Ġa Ġmuch Ġlarger Ġunic ode Ġcharacter Ġset , Ġand Ġmodel Ġlong Ġsequences Ġof Ġtext Ġas Ġfunctions Ġof Ġtheir Ġconstituent Ġcharacters Ġ( including Ġwhite - space ). ĠWe Ġfocus Ġon Ġsocial Ġmedia Ġposts Ġfrom Ġthe Ġwebsite ĠTwitter , Ġwhich Ġare Ġan Ġexcellent Ġtesting Ġground Ġfor Ġcharacter Ġbased Ġmodels Ġdue Ġto Ġthe Ġnoisy Ġnature Ġof Ġtext . ĠHeavy Ġuse Ġof Ġslang Ġand Ġabundant Ġmiss pell ings Ġmeans Ġthat Ġthere Ġare Ġmany Ġorth ographically Ġand Ġsem antically Ġsimilar Ġtokens , Ġand Ġspecial Ġcharacters Ġsuch Ġas Ġem oj is Ġare Ġalso Ġimmensely Ġpopular Ġand Ġcarry Ġuseful Ġsemantic Ġinformation . ĠIn Ġour Ġmoderately Ġsized Ġtraining Ġdataset Ġof Ġ2 Ġmillion Ġtweets , Ġthere Ġwere Ġabout Ġ0 . 92 Ġmillion Ġunique Ġword Ġtypes . ĠIt Ġwould Ġbe Ġexpensive Ġto Ġcapture Ġall Ġthese Ġphenomena Ġin Ġa Ġword Ġbased Ġmodel Ġin Ġterms Ġof Ġboth Ġthe Ġmemory Ġrequirement Ġ( for Ġthe Ġincreased Ġvocabulary ) Ġand Ġthe Ġamount Ġof Ġtraining Ġdata Ġrequired Ġfor Ġeffective Ġlearning . ĠAdditional Ġbenefits Ġof Ġthe Ġcharacter Ġbased Ġapproach Ġinclude Ġlanguage Ġindependence Ġof Ġthe Ġmethods , Ġand Ġno Ġrequirement Ġof ĠN LP Ġpre processing Ġsuch Ġas Ġword - se gment ation . </s> A Ġcrucial Ġstep Ġin Ġlearning Ġgood Ġtext Ġrepresentations Ġis Ġto Ġchoose Ġan Ġappropriate Ġobjective Ġfunction Ġto Ġoptimize . ĠUn super vised Ġapproaches Ġattempt Ġto Ġreconstruct Ġthe Ġoriginal Ġtext Ġfrom Ġits Ġlatent Ġrepresentation ĠB IB REF 4 Ġ, ĠB IB REF 0 Ġ. ĠSocial Ġmedia Ġposts Ġhowever , Ġcome Ġwith Ġtheir Ġown Ġform Ġof Ġsupervision Ġannot ated Ġby Ġmillions Ġof Ġusers , Ġin Ġthe Ġform Ġof Ġhasht ags Ġwhich Ġlink Ġposts Ġabout Ġthe Ġsame Ġtopic Ġtogether . ĠA Ġnatural Ġassumption Ġis Ġthat Ġthe Ġposts Ġwith Ġthe Ġsame Ġhasht ags Ġshould Ġhave Ġembed d ings Ġwhich Ġare Ġclose Ġto Ġeach Ġother . ĠHence , Ġwe Ġformulate Ġour Ġtraining Ġobjective Ġto Ġmaximize Ġcross - ent ropy Ġloss Ġat Ġthe Ġtask Ġof Ġpredicting Ġhasht ags Ġfor Ġa Ġpost Ġfrom Ġits Ġlatent Ġrepresentation . </s> We Ġpropose Ġa ĠBi - direction al ĠG ated ĠRec urrent ĠUnit Ġ( Bi - GR U ) ĠB IB REF 5 Ġneural Ġnetwork Ġfor Ġlearning Ġtweet Ġrepresentations . ĠTreat ing Ġwhite - space Ġas Ġa Ġspecial Ġcharacter Ġitself , Ġthe Ġmodel Ġdoes Ġa Ġforward Ġand Ġbackward Ġpass Ġover Ġthe Ġentire Ġsequence , Ġand Ġthe Ġfinal ĠGR U Ġstates Ġare Ġlin early Ġcombined Ġto Ġget Ġthe Ġtweet Ġembed ding . ĠPoster ior Ġprobabilities Ġover Ġhasht ags Ġare Ġcomputed Ġby Ġprojecting Ġthis Ġembed ding Ġto Ġa Ġsoft max Ġoutput Ġlayer . ĠCompared Ġto Ġa Ġword - level Ġbaseline Ġthis Ġmodel Ġshows Ġimproved Ġperformance Ġat Ġpredicting Ġhasht ags Ġfor Ġa Ġheld - out Ġset Ġof Ġposts . ĠInspired Ġby Ġrecent Ġwork Ġin Ġlearning Ġvector Ġspace Ġtext Ġrepresentations , Ġwe Ġname Ġour Ġmodel Ġtweet 2 vec . </s> Related ĠWork </s> Using Ġneural Ġnetworks Ġto Ġlearn Ġdistributed Ġrepresentations Ġof Ġwords Ġdates Ġback Ġto ĠB IB REF 0 Ġ. ĠMore Ġrecently , ĠB IB REF 4 Ġreleased Ġword 2 vec Ġ- Ġa Ġcollection Ġof Ġword Ġvectors Ġtrained Ġusing Ġa Ġrecurrent Ġneural Ġnetwork . ĠThese Ġword Ġvectors Ġare Ġin Ġwidespread Ġuse Ġin Ġthe ĠN LP Ġcommunity , Ġand Ġthe Ġoriginal Ġwork Ġhas Ġsince Ġbeen Ġextended Ġto Ġsentences ĠB IB REF 1 Ġ, Ġdocuments Ġand Ġparagraphs ĠB IB REF 6 Ġ, Ġtopics ĠB IB REF 7 Ġand Ġqueries ĠB IB REF 8 Ġ. ĠAll Ġthese Ġmethods Ġrequire Ġstoring Ġan Ġextremely Ġlarge Ġtable Ġof Ġvectors Ġfor Ġall Ġword Ġtypes Ġand Ġcannot Ġbe Ġeasily Ġgeneralized Ġto Ġunseen Ġwords Ġat Ġtest Ġtime ĠB IB REF 2 Ġ. ĠThey Ġalso Ġrequire Ġpre processing Ġto Ġfind Ġword Ġboundaries Ġwhich Ġis Ġnon - t riv ial Ġfor Ġa Ġsocial Ġnetwork Ġdomain Ġlike ĠTwitter . </s> In ĠB IB REF 2 Ġ, Ġthe Ġauthors Ġpresent Ġa Ġcompos itional Ġcharacter Ġmodel Ġbased Ġon Ġbid irection al ĠL ST Ms Ġas Ġa Ġpotential Ġsolution Ġto Ġthese Ġproblems . ĠA Ġmajor Ġbenefit Ġof Ġthis Ġapproach Ġis Ġthat Ġlarge Ġword Ġlookup Ġtables Ġcan Ġbe Ġcompact ed Ġinto Ġcharacter Ġlookup Ġtables Ġand Ġthe Ġcompos itional Ġmodel Ġscales Ġto Ġlarge Ġdata Ġsets Ġbetter Ġthan Ġother Ġstate - of - the - art Ġapproaches . ĠWhile ĠB IB REF 2 Ġgenerate Ġword Ġembed d ings Ġfrom Ġcharacter Ġrepresentations , Ġwe Ġpropose Ġto Ġgenerate Ġvector Ġrepresentations Ġof Ġentire Ġtweets Ġfrom Ġcharacters Ġin Ġour Ġtweet 2 vec Ġmodel . </s> Our Ġwork Ġadds Ġto Ġthe Ġgrowing Ġbody Ġof Ġwork Ġshowing Ġthe Ġapplic ability Ġof Ġcharacter Ġmodels Ġfor Ġa Ġvariety Ġof ĠN LP Ġtasks Ġsuch Ġas ĠNamed ĠEntity ĠRecogn ition ĠB IB REF 9 Ġ, ĠPOS Ġtagging ĠB IB REF 10 Ġ, Ġtext Ġclassification ĠB IB REF 11 Ġand Ġlanguage Ġmodeling ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> Previously , ĠB IB REF 14 Ġdealt Ġwith Ġthe Ġproblem Ġof Ġestimating Ġrare Ġword Ġrepresentations Ġby Ġbuilding Ġthem Ġfrom Ġtheir Ġconstituent Ġmorp hem es . ĠWhile Ġthey Ġshow Ġimproved Ġperformance Ġover Ġword - based Ġmodels , Ġtheir Ġapproach Ġrequires Ġa Ġmor phe me Ġparser Ġfor Ġpre processing Ġwhich Ġmay Ġnot Ġperform Ġwell Ġon Ġnoisy Ġtext Ġlike ĠTwitter . ĠAlso Ġthe Ġspace Ġof Ġall Ġmorp hem es , Ġthough Ġsmaller Ġthan Ġthe Ġspace Ġof Ġall Ġwords , Ġis Ġstill Ġlarge Ġenough Ġthat Ġmodelling Ġall Ġmorp hem es Ġis Ġimpractical . </s> Hash tag Ġprediction Ġfor Ġsocial Ġmedia Ġhas Ġbeen Ġaddressed Ġearlier , Ġfor Ġexample Ġin ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠB IB REF 15 Ġalso Ġuse Ġa Ġneural Ġarchitecture , Ġbut Ġcompose Ġtext Ġembed d ings Ġfrom Ġa Ġlookup Ġtable Ġof Ġwords . ĠThey Ġalso Ġshow Ġthat Ġthe Ġlearned Ġembed d ings Ġcan Ġgeneral ize Ġto Ġan Ġunrelated Ġtask Ġof Ġdocument Ġrecommendation , Ġjustifying Ġthe Ġuse Ġof Ġhasht ags Ġas Ġsupervision Ġfor Ġlearning Ġtext Ġrepresentations . </s> Tweet 2 V ec </s> Bi - GR U ĠEnc oder : ĠFigure Ġ1 Ġshows Ġour Ġmodel Ġfor Ġencoding Ġtweets . ĠIt Ġuses Ġa Ġsimilar Ġstructure Ġto Ġthe ĠC 2 W Ġmodel Ġin ĠB IB REF 2 Ġ, Ġwith ĠL ST M Ġunits Ġreplaced Ġwith ĠGR U Ġunits . </s> The Ġinput Ġto Ġthe Ġnetwork Ġis Ġdefined Ġby Ġan Ġalphabet Ġof Ġcharacters Ġ$ C $ Ġ( this Ġmay Ġinclude Ġthe Ġentire Ġunic ode Ġcharacter Ġset ). ĠThe Ġinput Ġtweet Ġis Ġbroken Ġinto Ġa Ġstream Ġof Ġcharacters Ġ$ c _ 1 , Ġc _ 2 , Ġ... Ġc _ m $ Ġeach Ġof Ġwhich Ġis Ġrepresented Ġby Ġa Ġ1 - by - Ġ$ | C | $ Ġencoding . ĠThese Ġone - hot Ġvectors Ġare Ġthen Ġprojected Ġto Ġa Ġcharacter Ġspace Ġby Ġmultiplying Ġwith Ġthe Ġmatrix Ġ$ P _ C Ġ\ in Ġ\ math bb Ġ{ R } ^{ | C | Ġ\ times Ġd _ c } $ Ġ, Ġwhere Ġ$ d _ c $ Ġis Ġthe Ġdimension Ġof Ġthe Ġcharacter Ġvector Ġspace . ĠLet Ġ$ x _ 1 , Ġx _ 2 , Ġ... Ġx _ m $ Ġbe Ġthe Ġsequence Ġof Ġcharacter Ġvectors Ġfor Ġthe Ġinput Ġtweet Ġafter Ġthe Ġlookup . ĠThe Ġenc oder Ġconsists Ġof Ġa Ġforward - GR U Ġand Ġa Ġbackward - GR U . ĠBoth Ġhave Ġthe Ġsame Ġarchitecture , Ġexcept Ġthe Ġbackward - GR U Ġprocesses Ġthe Ġsequence Ġin Ġreverse Ġorder . ĠEach Ġof Ġthe ĠGR U Ġunits Ġprocess Ġthese Ġvectors Ġsequ entially , Ġand Ġstarting Ġwith Ġthe Ġinitial Ġstate Ġ$ h _ 0 $ Ġcompute Ġthe Ġsequence Ġ$ h _ 1 , Ġh _ 2 , Ġ... Ġh _ m $ Ġas Ġfollows : Ġ$ Ġr _ t Ġ& = Ġ\ s igma Ġ( W _ r Ġx _ t Ġ+ ĠU _ r Ġh _{ t - 1 } Ġ+ Ġb _ r ), Ġ\\ Ġz _ t Ġ& = Ġ\ s igma Ġ( W _ z Ġx _ t Ġ+ ĠU _ z Ġh _{ t - 1 } Ġ+ Ġb _ z ), Ġ\\ Ġ\ t ilde { h } _ t Ġ& = Ġtan h ( W _ h Ġx _ t Ġ+ ĠU _ h Ġ( r _ t Ġ\ od ot Ġh _{ t - 1 }) Ġ+ Ġb _ h ), Ġ\\ Ġh _ t Ġ& = Ġ( 1 - z _ t ) Ġ\ od ot Ġh _{ t - 1 } Ġ+ Ġz _ t Ġ\ od ot Ġ\ t ilde { h } _ t . Ġ$ </s> Here Ġ$ r _ t $ Ġ, Ġ$ z _ t $ Ġare Ġcalled Ġthe Ġreset Ġand Ġupdate Ġgates Ġrespectively , Ġand Ġ$\ t ilde { h } _ t $ Ġis Ġthe Ġcandidate Ġoutput Ġstate Ġwhich Ġis Ġconverted Ġto Ġthe Ġactual Ġoutput Ġstate Ġ$ h _ t $ Ġ. Ġ$ W _ r , ĠW _ z , ĠW _ h $ Ġare Ġ$ d _ h Ġ\ times Ġd _ c $ Ġmat rices Ġand Ġ$ U _ r , ĠU _ z , ĠU _ h $ Ġare Ġ$ d _ h Ġ\ times Ġd _ h $ Ġmat rices , Ġwhere Ġ$ d _ h $ Ġis Ġthe Ġhidden Ġstate Ġdimension Ġof Ġthe ĠGR U . ĠThe Ġfinal Ġstates Ġ$ h _ m ^ f $ Ġfrom Ġthe Ġforward - GR U , Ġand Ġ$ z _ t $ 0 Ġfrom Ġthe Ġbackward ĠGR U Ġare Ġcombined Ġusing Ġa Ġfully - connected Ġlayer Ġto Ġthe Ġgive Ġthe Ġfinal Ġtweet Ġembed ding Ġ$ z _ t $ 1 Ġ: </s> $$ e _ t Ġ= ĠW ^ f Ġh _ m ^ f Ġ+ ĠW ^ b Ġh _ 0 ^ b $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> Here Ġ$ W ^ f , ĠW ^ b $ Ġare Ġ$ d _ t Ġ\ times Ġd _ h $ Ġand Ġ$ b $ Ġis Ġ$ d _ t Ġ\ times Ġ1 $ Ġbias Ġterm , Ġwhere Ġ$ d _ t $ Ġis Ġthe Ġdimension Ġof Ġthe Ġfinal Ġtweet Ġembed ding . ĠIn Ġour Ġexperiments Ġwe Ġset Ġ$ d _ t = d _ h $ Ġ. ĠAll Ġparameters Ġare Ġlearned Ġusing Ġgradient Ġdescent . </s> Soft max : ĠFinally , Ġthe Ġtweet Ġembed ding Ġis Ġpassed Ġthrough Ġa Ġlinear Ġlayer Ġwhose Ġoutput Ġis Ġthe Ġsame Ġsize Ġas Ġthe Ġnumber Ġof Ġhasht ags Ġ$ L $ Ġin Ġthe Ġdata Ġset . ĠWe Ġuse Ġa Ġsoft max Ġlayer Ġto Ġcompute Ġthe Ġposterior Ġhashtag Ġprobabilities : </s> $$ P ( y = j Ġ| e ) Ġ= Ġ\ frac { exp ( w _ j ^ Te Ġ+ Ġb _ j ) }{ \ sum Ġ_ { i = 1 } ^ L Ġexp ( w _ i ^ Te Ġ+ Ġb _ j ) }. $$ Ġ Ġ Ġ( E q . Ġ4 ) </s> Object ive ĠFunction : ĠWe Ġoptimize Ġthe Ġcateg orical Ġcross - ent ropy Ġloss Ġbetween Ġpredicted Ġand Ġtrue Ġhasht ags : </s> $$ J Ġ= Ġ\ frac { 1 }{ B } Ġ\ sum Ġ_ { i = 1 } ^{ B } Ġ\ sum Ġ_ { j = 1 } ^{ L } Ġ- t _{ i , j } log ( p _{ i , j }) Ġ+ Ġ\ lambda Ġ\ Vert Ġ\ The ta Ġ\ Vert Ġ^ 2 . $$ Ġ Ġ Ġ( E q . Ġ5 ) </s> Here Ġ$ B $ Ġis Ġthe Ġbatch Ġsize , Ġ$ L $ Ġis Ġthe Ġnumber Ġof Ġclasses , Ġ$ p _{ i , j } $ Ġis Ġthe Ġpredicted Ġprobability Ġthat Ġthe Ġ$ i $ Ġ- th Ġtweet Ġhas Ġhashtag Ġ$ j $ Ġ, Ġand Ġ$ t _{ i , j } Ġ\ in Ġ\ l brace Ġ0 , 1 \ r brace Ġ$ Ġdenotes Ġthe Ġground Ġtruth Ġof Ġwhether Ġthe Ġ$ j $ Ġ- th Ġhashtag Ġis Ġin Ġthe Ġ$ i $ Ġ- th Ġtweet . ĠWe Ġuse ĠL 2 - regular ization Ġweighted Ġby Ġ$\ lambda Ġ$ Ġ. </s> Word ĠLevel ĠBas eline </s> Since Ġour Ġobjective Ġis Ġto Ġcompare Ġcharacter - based Ġand Ġword - based Ġapproaches , Ġwe Ġhave Ġalso Ġimplemented Ġa Ġsimple Ġword - level Ġenc oder Ġfor Ġtweets . ĠThe Ġinput Ġtweet Ġis Ġfirst Ġsplit Ġinto Ġtokens Ġalong Ġwhite - sp aces . ĠA Ġmore Ġsophisticated Ġtoken izer Ġmay Ġbe Ġused , Ġbut Ġfor Ġa Ġfair Ġcomparison Ġwe Ġwanted Ġto Ġkeep Ġlanguage Ġspecific Ġpre processing Ġto Ġa Ġminimum . ĠThe Ġenc oder Ġis Ġessentially Ġthe Ġsame Ġas Ġtweet 2 vec , Ġwith Ġthe Ġinput Ġas Ġwords Ġinstead Ġof Ġcharacters . ĠA Ġlookup Ġtable Ġstores Ġword Ġvectors Ġfor Ġthe Ġ$ V $ Ġ( 20 K Ġhere ) Ġmost Ġcommon Ġwords , Ġand Ġthe Ġrest Ġare Ġgrouped Ġtogether Ġunder Ġthe Ġ` UN K ' Ġtoken . </s> Data </s> Our Ġdataset Ġconsists Ġof Ġa Ġlarge Ġcollection Ġof Ġglobal Ġposts Ġfrom ĠTwitter Ġbetween Ġthe Ġdates Ġof ĠJune Ġ1 , Ġ2013 Ġto ĠJune Ġ5 , Ġ2013 . ĠOnly ĠEnglish Ġlanguage Ġposts Ġ( as Ġdetected Ġby Ġthe Ġlang Ġfield Ġin ĠTwitter ĠAPI ) Ġand Ġposts Ġwith Ġat Ġleast Ġone Ġhashtag Ġare Ġretained . ĠWe Ġremoved Ġinf requent Ġhasht ags Ġ( Ġ$ < 500 $ Ġposts ) Ġsince Ġthey Ġdo Ġnot Ġhave Ġenough Ġdata Ġfor Ġgood Ġgeneral ization . ĠWe Ġalso Ġremoved Ġvery Ġfrequent Ġtags Ġ( Ġ$ > 19 K $ Ġposts ) Ġwhich Ġwere Ġalmost Ġalways Ġfrom Ġautomatically Ġgenerated Ġposts Ġ( ex : Ġ# android game ) Ġwhich Ġare Ġtrivial Ġto Ġpredict . ĠThe Ġfinal Ġdataset Ġcontains Ġ2 Ġmillion Ġtweets Ġfor Ġtraining , Ġ10 K Ġfor Ġvalidation Ġand Ġ50 K Ġfor Ġtesting , Ġwith Ġa Ġtotal Ġof Ġ20 39 Ġdistinct Ġhasht ags . ĠWe Ġuse Ġsimple Ġregex Ġto Ġpre process Ġthe Ġpost Ġtext Ġand Ġremove Ġhasht ags Ġ( since Ġthese Ġare Ġto Ġbe Ġpredicted ) Ġand ĠHTML Ġtags , Ġand Ġreplace Ġuser - names Ġand ĠURLs Ġwith Ġspecial Ġtokens . ĠWe Ġalso Ġremoved Ġret we ets Ġand Ġconvert Ġthe Ġtext Ġto Ġlower - case . </s> Im plementation ĠDetails </s> Word Ġvectors Ġand Ġcharacter Ġvectors Ġare Ġboth Ġset Ġto Ġsize Ġ$ d _ L = 150 $ Ġfor Ġtheir Ġrespective Ġmodels . ĠThere Ġwere Ġ28 29 Ġunique Ġcharacters Ġin Ġthe Ġtraining Ġset Ġand Ġwe Ġmodel Ġeach Ġof Ġthese Ġindependently Ġin Ġa Ġcharacter Ġlook - up Ġtable . ĠEmb edd ing Ġsizes Ġwere Ġchosen Ġsuch Ġthat Ġeach Ġmodel Ġhad Ġroughly Ġthe Ġsame Ġnumber Ġof Ġparameters Ġ( Table Ġ2 Ġ). ĠTraining Ġis Ġperformed Ġusing Ġmini - batch Ġgradient Ġdescent Ġwith ĠN ester ov 's Ġmomentum . ĠWe Ġuse Ġa Ġbatch Ġsize Ġ$ B = 64 $ Ġ, Ġinitial Ġlearning Ġrate Ġ$\ eta Ġ_ 0 = 0 . 01 $ Ġand Ġmomentum Ġparameter Ġ$\ mu Ġ_ 0 = 0 . 9 $ Ġ. ĠL 2 - regular ization Ġwith Ġ$\ lambda Ġ= 0 . 001 $ Ġwas Ġapplied Ġto Ġall Ġmodels . ĠInitial Ġweights Ġwere Ġdrawn Ġfrom Ġ0 - mean Ġga uss ians Ġwith Ġ$\ s igma Ġ= 0 . 1 $ Ġand Ġinitial Ġbiases Ġwere Ġset Ġto Ġ0 . ĠThe Ġhyper param eters Ġwere Ġtuned Ġone Ġat Ġa Ġtime Ġkeeping Ġothers Ġfixed , Ġand Ġvalues Ġwith Ġthe Ġlowest Ġvalidation Ġcost Ġwere Ġchosen . ĠThe Ġresultant Ġcombination Ġwas Ġused Ġto Ġtrain Ġthe Ġmodels Ġuntil Ġperformance Ġon Ġvalidation Ġset Ġstopped Ġincreasing . ĠDuring Ġtraining , Ġthe Ġlearning Ġrate Ġis Ġhal ved Ġevery time Ġthe Ġvalidation Ġset Ġprecision Ġincreases Ġby Ġless Ġthan Ġ0 . 01 Ġ% Ġfrom Ġone Ġepoch Ġto Ġthe Ġnext . ĠThe Ġmodels Ġconverge Ġin Ġabout Ġ20 Ġepoch s . ĠCode Ġfor Ġtraining Ġboth Ġthe Ġmodels Ġis Ġpublicly Ġavailable Ġon Ġgithub . </s> Results </s> We Ġtest Ġthe Ġcharacter Ġand Ġword - level Ġvariants Ġby Ġpredicting Ġhasht ags Ġfor Ġa Ġheld - out Ġtest Ġset Ġof Ġposts . ĠSince Ġthere Ġmay Ġbe Ġmore Ġthan Ġone Ġcorrect Ġhashtag Ġper Ġpost , Ġwe Ġgenerate Ġa Ġranked Ġlist Ġof Ġtags Ġfor Ġeach Ġpost Ġfrom Ġthe Ġoutput Ġposter iors , Ġand Ġreport Ġaverage Ġprecision @ 1 , Ġrecall @ 10 Ġand Ġmean Ġrank Ġof Ġthe Ġcorrect Ġhasht ags . ĠThese Ġare Ġlisted Ġin ĠTable Ġ3 Ġ. </s> To Ġsee Ġthe Ġperformance Ġof Ġeach Ġmodel Ġon Ġposts Ġcontaining Ġrare Ġwords Ġ( RW ) Ġand Ġfrequent Ġwords Ġ( FW ) Ġwe Ġselected Ġtwo Ġtest Ġsets Ġeach Ġcontaining Ġ2 , 000 Ġposts . ĠWe Ġpopulated Ġthese Ġsets Ġwith Ġposts Ġwhich Ġhad Ġthe Ġmaximum Ġand Ġminimum Ġnumber Ġof Ġout - of - voc abulary Ġwords Ġrespectively , Ġwhere Ġvocabulary Ġis Ġdefined Ġby Ġthe Ġ20 K Ġmost Ġfrequent Ġwords . ĠOverall , Ġtweet 2 vec Ġoutper forms Ġthe Ġword Ġmodel , Ġdoing Ġsignificantly Ġbetter Ġon ĠRW Ġtest Ġset Ġand Ġcompar ably Ġon ĠFW Ġset . ĠThis Ġimproved Ġperformance Ġcomes Ġat Ġthe Ġcost Ġof Ġincreased Ġtraining Ġtime Ġ( see ĠTable Ġ2 Ġ), Ġsince Ġmoving Ġfrom Ġwords Ġto Ġcharacters Ġresults Ġin Ġlonger Ġinput Ġsequences Ġto Ġthe ĠGR U . </s> We Ġalso Ġstudy Ġthe Ġeffect Ġof Ġmodel Ġsize Ġon Ġthe Ġperformance Ġof Ġthese Ġmodels . ĠFor Ġthe Ġword Ġmodel Ġwe Ġset Ġvocabulary Ġsize Ġ$ V $ Ġto Ġ8 K , Ġ15 K Ġand Ġ20 K Ġrespectively . ĠFor Ġtweet 2 vec Ġwe Ġset Ġthe ĠGR U Ġhidden Ġstate Ġsize Ġto Ġ300 , Ġ400 Ġand Ġ500 Ġrespectively . ĠFigure Ġ2 Ġshows Ġprecision Ġ1 Ġof Ġthe Ġtwo Ġmodels Ġas Ġthe Ġnumber Ġof Ġparameters Ġis Ġincreased , Ġfor Ġeach Ġtest Ġset Ġdescribed Ġabove . ĠThere Ġis Ġnot Ġmuch Ġvariation Ġin Ġthe Ġperformance , Ġand Ġmoreover Ġtweet 2 vec Ġalways Ġoutper forms Ġthe Ġword Ġbased Ġmodel Ġfor Ġthe Ġsame Ġnumber Ġof Ġparameters . </s> Table Ġ4 Ġcompares Ġthe Ġmodels Ġas Ġcomplexity Ġof Ġthe Ġtask Ġis Ġincreased . ĠWe Ġcreated Ġ3 Ġdatasets Ġ( small , Ġmedium Ġand Ġlarge ) Ġwith Ġan Ġincreasing Ġnumber Ġof Ġhasht ags Ġto Ġbe Ġpredicted . ĠThis Ġwas Ġdone Ġby Ġvarying Ġthe Ġlower Ġthreshold Ġof Ġthe Ġminimum Ġnumber Ġof Ġtags Ġper Ġpost Ġfor Ġit Ġto Ġbe Ġincluded Ġin Ġthe Ġdataset . ĠOnce Ġagain Ġwe Ġobserve Ġthat Ġtweet 2 vec Ġoutper forms Ġits Ġword - based Ġcounterpart Ġfor Ġeach Ġof Ġthe Ġthree Ġsettings . </s> Finally , Ġtable Ġ1 Ġshows Ġsome Ġpredictions Ġfrom Ġthe Ġword Ġlevel Ġmodel Ġand Ġtweet 2 vec . ĠWe Ġselected Ġthese Ġto Ġhighlight Ġsome Ġstrengths Ġof Ġthe Ġcharacter Ġbased Ġapproach Ġ- Ġit Ġis Ġrobust Ġto Ġword Ġsegment ation Ġerrors Ġand Ġspelling Ġmistakes , Ġeffectively Ġinterpre ts Ġem oj is Ġand Ġother Ġspecial Ġcharacters Ġto Ġmake Ġpredictions , Ġand Ġalso Ġperforms Ġcompar ably Ġto Ġthe Ġword - based Ġapproach Ġfor Ġin - voc abulary Ġtokens . </s> Conclusion </s> We Ġhave Ġpresented Ġtweet 2 vec Ġ- Ġa Ġcharacter Ġlevel Ġenc oder Ġfor Ġsocial Ġmedia Ġposts Ġtrained Ġusing Ġsupervision Ġfrom Ġassociated Ġhasht ags . ĠOur Ġresult Ġshows Ġthat Ġtweet 2 vec Ġoutper forms Ġthe Ġword Ġbased Ġapproach , Ġdoing Ġsignificantly Ġbetter Ġwhen Ġthe Ġinput Ġpost Ġcontains Ġmany Ġrare Ġwords . ĠWe Ġhave Ġfocused Ġonly Ġon ĠEnglish Ġlanguage Ġposts , Ġbut Ġthe Ġcharacter Ġmodel Ġrequires Ġno Ġlanguage Ġspecific Ġpre processing Ġand Ġcan Ġbe Ġextended Ġto Ġother Ġlanguages . ĠFor Ġfuture Ġwork , Ġone Ġnatural Ġextension Ġwould Ġbe Ġto Ġuse Ġa Ġcharacter - level Ġdec oder Ġfor Ġpredicting Ġthe Ġhasht ags . ĠThis Ġwill Ġallow Ġgeneration Ġof Ġhasht ags Ġnot Ġseen Ġin Ġthe Ġtraining Ġdataset . ĠAlso , Ġit Ġwill Ġbe Ġinteresting Ġto Ġsee Ġhow Ġour Ġtweet 2 vec Ġembed d ings Ġcan Ġbe Ġused Ġin Ġdomains Ġwhere Ġthere Ġis Ġa Ġneed Ġfor Ġsemantic Ġunderstanding Ġof Ġsocial Ġmedia , Ġsuch Ġas Ġtracking Ġinfectious Ġdiseases ĠB IB REF 17 Ġ. ĠHence , Ġwe Ġprovide Ġan Ġoff - the - she lf Ġenc oder Ġtrained Ġon Ġmedium Ġdataset Ġdescribed Ġabove Ġto Ġcompute Ġvector - space Ġrepresentations Ġof Ġtweets Ġalong Ġwith Ġour Ġcode Ġon Ġgithub . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank ĠAlex ĠSm ola , ĠYun ĠFu , ĠH s iao - Yu ĠFish ĠT ung , ĠRus lan ĠSal akh ut din ov , Ġand ĠBarn ab as ĠPoc zos Ġfor Ġuseful Ġdiscussions . ĠWe Ġwould Ġalso Ġlike Ġto Ġthank ĠJu er gen ĠP fe ffer Ġfor Ġproviding Ġaccess Ġto Ġthe ĠTwitter Ġdata , Ġand Ġthe Ġreviewers Ġfor Ġtheir Ġcomments .
Introduction We understand from Zipf's Law that in any natural language corpus a majority of the vocabulary word types will either be absent or occur in low frequency. Estimating the statistical properties of these rare word types is naturally a difficult task. This is analogous to the curse of dimensionality when we deal with sequences of tokens - most sequences will occur only once in the training data. Neural network architectures overcome this problem by defining non-linear compositional models over vector space representations of tokens and hence assign non-zero probability even to sequences not seen during training BIBREF0 , BIBREF1 . In this work, we explore a similar approach to learning distributed representations of social media posts by composing them from their constituent characters, with the goal of generalizing to out-of-vocabulary words as well as sequences at test time. Traditional Neural Network Language Models (NNLMs) treat words as the basic units of language and assign independent vectors to each word type. To constrain memory requirements, the vocabulary size is fixed before-hand; therefore, rare and out-of-vocabulary words are all grouped together under a common type `UNKNOWN'. This choice is motivated by the assumption of arbitrariness in language, which means that surface forms of words have little to do with their semantic roles. Recently, BIBREF2 challenge this assumption and present a bidirectional Long Short Term Memory (LSTM) BIBREF3 for composing word vectors from their constituent characters which can memorize the arbitrary aspects of word orthography as well as generalize to rare and out-of-vocabulary words. Encouraged by their findings, we extend their approach to a much larger unicode character set, and model long sequences of text as functions of their constituent characters (including white-space). We focus on social media posts from the website Twitter, which are an excellent testing ground for character based models due to the noisy nature of text. Heavy use of slang and abundant misspellings means that there are many orthographically and semantically similar tokens, and special characters such as emojis are also immensely popular and carry useful semantic information. In our moderately sized training dataset of 2 million tweets, there were about 0.92 million unique word types. It would be expensive to capture all these phenomena in a word based model in terms of both the memory requirement (for the increased vocabulary) and the amount of training data required for effective learning. Additional benefits of the character based approach include language independence of the methods, and no requirement of NLP preprocessing such as word-segmentation. A crucial step in learning good text representations is to choose an appropriate objective function to optimize. Unsupervised approaches attempt to reconstruct the original text from its latent representation BIBREF4 , BIBREF0 . Social media posts however, come with their own form of supervision annotated by millions of users, in the form of hashtags which link posts about the same topic together. A natural assumption is that the posts with the same hashtags should have embeddings which are close to each other. Hence, we formulate our training objective to maximize cross-entropy loss at the task of predicting hashtags for a post from its latent representation. We propose a Bi-directional Gated Recurrent Unit (Bi-GRU) BIBREF5 neural network for learning tweet representations. Treating white-space as a special character itself, the model does a forward and backward pass over the entire sequence, and the final GRU states are linearly combined to get the tweet embedding. Posterior probabilities over hashtags are computed by projecting this embedding to a softmax output layer. Compared to a word-level baseline this model shows improved performance at predicting hashtags for a held-out set of posts. Inspired by recent work in learning vector space text representations, we name our model tweet2vec. Related Work Using neural networks to learn distributed representations of words dates back to BIBREF0 . More recently, BIBREF4 released word2vec - a collection of word vectors trained using a recurrent neural network. These word vectors are in widespread use in the NLP community, and the original work has since been extended to sentences BIBREF1 , documents and paragraphs BIBREF6 , topics BIBREF7 and queries BIBREF8 . All these methods require storing an extremely large table of vectors for all word types and cannot be easily generalized to unseen words at test time BIBREF2 . They also require preprocessing to find word boundaries which is non-trivial for a social network domain like Twitter. In BIBREF2 , the authors present a compositional character model based on bidirectional LSTMs as a potential solution to these problems. A major benefit of this approach is that large word lookup tables can be compacted into character lookup tables and the compositional model scales to large data sets better than other state-of-the-art approaches. While BIBREF2 generate word embeddings from character representations, we propose to generate vector representations of entire tweets from characters in our tweet2vec model. Our work adds to the growing body of work showing the applicability of character models for a variety of NLP tasks such as Named Entity Recognition BIBREF9 , POS tagging BIBREF10 , text classification BIBREF11 and language modeling BIBREF12 , BIBREF13 . Previously, BIBREF14 dealt with the problem of estimating rare word representations by building them from their constituent morphemes. While they show improved performance over word-based models, their approach requires a morpheme parser for preprocessing which may not perform well on noisy text like Twitter. Also the space of all morphemes, though smaller than the space of all words, is still large enough that modelling all morphemes is impractical. Hashtag prediction for social media has been addressed earlier, for example in BIBREF15 , BIBREF16 . BIBREF15 also use a neural architecture, but compose text embeddings from a lookup table of words. They also show that the learned embeddings can generalize to an unrelated task of document recommendation, justifying the use of hashtags as supervision for learning text representations. Tweet2Vec Bi-GRU Encoder: Figure 1 shows our model for encoding tweets. It uses a similar structure to the C2W model in BIBREF2 , with LSTM units replaced with GRU units. The input to the network is defined by an alphabet of characters $C$ (this may include the entire unicode character set). The input tweet is broken into a stream of characters $c_1, c_2, ... c_m$ each of which is represented by a 1-by- $|C|$ encoding. These one-hot vectors are then projected to a character space by multiplying with the matrix $P_C \in \mathbb {R}^{|C| \times d_c}$ , where $d_c$ is the dimension of the character vector space. Let $x_1, x_2, ... x_m$ be the sequence of character vectors for the input tweet after the lookup. The encoder consists of a forward-GRU and a backward-GRU. Both have the same architecture, except the backward-GRU processes the sequence in reverse order. Each of the GRU units process these vectors sequentially, and starting with the initial state $h_0$ compute the sequence $h_1, h_2, ... h_m$ as follows: $ r_t &= \sigma (W_r x_t + U_r h_{t-1} + b_r), \\ z_t &= \sigma (W_z x_t + U_z h_{t-1} + b_z), \\ \tilde{h}_t &= tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h), \\ h_t &= (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t. $ Here $r_t$ , $z_t$ are called the reset and update gates respectively, and $\tilde{h}_t$ is the candidate output state which is converted to the actual output state $h_t$ . $W_r, W_z, W_h$ are $d_h \times d_c$ matrices and $U_r, U_z, U_h$ are $d_h \times d_h$ matrices, where $d_h$ is the hidden state dimension of the GRU. The final states $h_m^f$ from the forward-GRU, and $z_t$0 from the backward GRU are combined using a fully-connected layer to the give the final tweet embedding $z_t$1 : $$e_t = W^f h_m^f + W^b h_0^b$$   (Eq. 3) Here $W^f, W^b$ are $d_t \times d_h$ and $b$ is $d_t \times 1$ bias term, where $d_t$ is the dimension of the final tweet embedding. In our experiments we set $d_t=d_h$ . All parameters are learned using gradient descent. Softmax: Finally, the tweet embedding is passed through a linear layer whose output is the same size as the number of hashtags $L$ in the data set. We use a softmax layer to compute the posterior hashtag probabilities: $$P(y=j |e) = \frac{exp(w_j^Te + b_j)}{\sum _{i=1}^L exp(w_i^Te + b_j)}.$$   (Eq. 4) Objective Function: We optimize the categorical cross-entropy loss between predicted and true hashtags: $$J = \frac{1}{B} \sum _{i=1}^{B} \sum _{j=1}^{L} -t_{i,j}log(p_{i,j}) + \lambda \Vert \Theta \Vert ^2.$$   (Eq. 5) Here $B$ is the batch size, $L$ is the number of classes, $p_{i,j}$ is the predicted probability that the $i$ -th tweet has hashtag $j$ , and $t_{i,j} \in \lbrace 0,1\rbrace $ denotes the ground truth of whether the $j$ -th hashtag is in the $i$ -th tweet. We use L2-regularization weighted by $\lambda $ . Word Level Baseline Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets. The input tweet is first split into tokens along white-spaces. A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum. The encoder is essentially the same as tweet2vec, with the input as words instead of characters. A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token. Data Our dataset consists of a large collection of global posts from Twitter between the dates of June 1, 2013 to June 5, 2013. Only English language posts (as detected by the lang field in Twitter API) and posts with at least one hashtag are retained. We removed infrequent hashtags ( $<500$ posts) since they do not have enough data for good generalization. We also removed very frequent tags ( $>19K$ posts) which were almost always from automatically generated posts (ex: #androidgame) which are trivial to predict. The final dataset contains 2 million tweets for training, 10K for validation and 50K for testing, with a total of 2039 distinct hashtags. We use simple regex to preprocess the post text and remove hashtags (since these are to be predicted) and HTML tags, and replace user-names and URLs with special tokens. We also removed retweets and convert the text to lower-case. Implementation Details Word vectors and character vectors are both set to size $d_L=150$ for their respective models. There were 2829 unique characters in the training set and we model each of these independently in a character look-up table. Embedding sizes were chosen such that each model had roughly the same number of parameters (Table 2 ). Training is performed using mini-batch gradient descent with Nesterov's momentum. We use a batch size $B=64$ , initial learning rate $\eta _0=0.01$ and momentum parameter $\mu _0=0.9$ . L2-regularization with $\lambda =0.001$ was applied to all models. Initial weights were drawn from 0-mean gaussians with $\sigma =0.1$ and initial biases were set to 0. The hyperparameters were tuned one at a time keeping others fixed, and values with the lowest validation cost were chosen. The resultant combination was used to train the models until performance on validation set stopped increasing. During training, the learning rate is halved everytime the validation set precision increases by less than 0.01 % from one epoch to the next. The models converge in about 20 epochs. Code for training both the models is publicly available on github. Results We test the character and word-level variants by predicting hashtags for a held-out test set of posts. Since there may be more than one correct hashtag per post, we generate a ranked list of tags for each post from the output posteriors, and report average precision@1, recall@10 and mean rank of the correct hashtags. These are listed in Table 3 . To see the performance of each model on posts containing rare words (RW) and frequent words (FW) we selected two test sets each containing 2,000 posts. We populated these sets with posts which had the maximum and minimum number of out-of-vocabulary words respectively, where vocabulary is defined by the 20K most frequent words. Overall, tweet2vec outperforms the word model, doing significantly better on RW test set and comparably on FW set. This improved performance comes at the cost of increased training time (see Table 2 ), since moving from words to characters results in longer input sequences to the GRU. We also study the effect of model size on the performance of these models. For the word model we set vocabulary size $V$ to 8K, 15K and 20K respectively. For tweet2vec we set the GRU hidden state size to 300, 400 and 500 respectively. Figure 2 shows precision 1 of the two models as the number of parameters is increased, for each test set described above. There is not much variation in the performance, and moreover tweet2vec always outperforms the word based model for the same number of parameters. Table 4 compares the models as complexity of the task is increased. We created 3 datasets (small, medium and large) with an increasing number of hashtags to be predicted. This was done by varying the lower threshold of the minimum number of tags per post for it to be included in the dataset. Once again we observe that tweet2vec outperforms its word-based counterpart for each of the three settings. Finally, table 1 shows some predictions from the word level model and tweet2vec. We selected these to highlight some strengths of the character based approach - it is robust to word segmentation errors and spelling mistakes, effectively interprets emojis and other special characters to make predictions, and also performs comparably to the word-based approach for in-vocabulary tokens. Conclusion We have presented tweet2vec - a character level encoder for social media posts trained using supervision from associated hashtags. Our result shows that tweet2vec outperforms the word based approach, doing significantly better when the input post contains many rare words. We have focused only on English language posts, but the character model requires no language specific preprocessing and can be extended to other languages. For future work, one natural extension would be to use a character-level decoder for predicting the hashtags. This will allow generation of hashtags not seen in the training dataset. Also, it will be interesting to see how our tweet2vec embeddings can be used in domains where there is a need for semantic understanding of social media, such as tracking infectious diseases BIBREF17 . Hence, we provide an off-the-shelf encoder trained on medium dataset described above to compute vector-space representations of tweets along with our code on github. Acknowledgments We would like to thank Alex Smola, Yun Fu, Hsiao-Yu Fish Tung, Ruslan Salakhutdinov, and Barnabas Poczos for useful discussions. We would also like to thank Juergen Pfeffer for providing access to the Twitter data, and the reviewers for their comments.
What is the word-level baseline?
127d5ddfabec5c58832e5865cbd8ed0978c25a13
What Ġis Ġthe Ġword - level Ġbaseline ?
a Ġsimple Ġword - level Ġenc oder , ĠThe Ġenc oder Ġis Ġessentially Ġthe Ġsame Ġas Ġtweet 2 vec , Ġwith Ġthe Ġinput Ġas Ġwords Ġinstead Ġof Ġcharacters .
<s> What Ġis Ġthe Ġword - level Ġbaseline ? </s> Introduction </s> We Ġunderstand Ġfrom ĠZip f 's ĠLaw Ġthat Ġin Ġany Ġnatural Ġlanguage Ġcorpus Ġa Ġmajority Ġof Ġthe Ġvocabulary Ġword Ġtypes Ġwill Ġeither Ġbe Ġabsent Ġor Ġoccur Ġin Ġlow Ġfrequency . ĠEst imating Ġthe Ġstatistical Ġproperties Ġof Ġthese Ġrare Ġword Ġtypes Ġis Ġnaturally Ġa Ġdifficult Ġtask . ĠThis Ġis Ġanalogous Ġto Ġthe Ġcurse Ġof Ġdimension ality Ġwhen Ġwe Ġdeal Ġwith Ġsequences Ġof Ġtokens Ġ- Ġmost Ġsequences Ġwill Ġoccur Ġonly Ġonce Ġin Ġthe Ġtraining Ġdata . ĠNeural Ġnetwork Ġarchitectures Ġovercome Ġthis Ġproblem Ġby Ġdefining Ġnon - linear Ġcompos itional Ġmodels Ġover Ġvector Ġspace Ġrepresentations Ġof Ġtokens Ġand Ġhence Ġassign Ġnon - zero Ġprobability Ġeven Ġto Ġsequences Ġnot Ġseen Ġduring Ġtraining ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġexplore Ġa Ġsimilar Ġapproach Ġto Ġlearning Ġdistributed Ġrepresentations Ġof Ġsocial Ġmedia Ġposts Ġby Ġcomposing Ġthem Ġfrom Ġtheir Ġconstituent Ġcharacters , Ġwith Ġthe Ġgoal Ġof Ġgeneral izing Ġto Ġout - of - voc abulary Ġwords Ġas Ġwell Ġas Ġsequences Ġat Ġtest Ġtime . </s> Traditional ĠNeural ĠNetwork ĠLanguage ĠModels Ġ( NN L Ms ) Ġtreat Ġwords Ġas Ġthe Ġbasic Ġunits Ġof Ġlanguage Ġand Ġassign Ġindependent Ġvectors Ġto Ġeach Ġword Ġtype . ĠTo Ġconst rain Ġmemory Ġrequirements , Ġthe Ġvocabulary Ġsize Ġis Ġfixed Ġbefore - hand ; Ġtherefore , Ġrare Ġand Ġout - of - voc abulary Ġwords Ġare Ġall Ġgrouped Ġtogether Ġunder Ġa Ġcommon Ġtype Ġ` UN KNOWN '. ĠThis Ġchoice Ġis Ġmotivated Ġby Ġthe Ġassumption Ġof Ġarbit rar iness Ġin Ġlanguage , Ġwhich Ġmeans Ġthat Ġsurface Ġforms Ġof Ġwords Ġhave Ġlittle Ġto Ġdo Ġwith Ġtheir Ġsemantic Ġroles . ĠRecently , ĠB IB REF 2 Ġchallenge Ġthis Ġassumption Ġand Ġpresent Ġa Ġbid irection al ĠLong ĠShort ĠTerm ĠMemory Ġ( L ST M ) ĠB IB REF 3 Ġfor Ġcomposing Ġword Ġvectors Ġfrom Ġtheir Ġconstituent Ġcharacters Ġwhich Ġcan Ġmemor ize Ġthe Ġarbitrary Ġaspects Ġof Ġword Ġorth ography Ġas Ġwell Ġas Ġgeneral ize Ġto Ġrare Ġand Ġout - of - voc abulary Ġwords . </s> Enc our aged Ġby Ġtheir Ġfindings , Ġwe Ġextend Ġtheir Ġapproach Ġto Ġa Ġmuch Ġlarger Ġunic ode Ġcharacter Ġset , Ġand Ġmodel Ġlong Ġsequences Ġof Ġtext Ġas Ġfunctions Ġof Ġtheir Ġconstituent Ġcharacters Ġ( including Ġwhite - space ). ĠWe Ġfocus Ġon Ġsocial Ġmedia Ġposts Ġfrom Ġthe Ġwebsite ĠTwitter , Ġwhich Ġare Ġan Ġexcellent Ġtesting Ġground Ġfor Ġcharacter Ġbased Ġmodels Ġdue Ġto Ġthe Ġnoisy Ġnature Ġof Ġtext . ĠHeavy Ġuse Ġof Ġslang Ġand Ġabundant Ġmiss pell ings Ġmeans Ġthat Ġthere Ġare Ġmany Ġorth ographically Ġand Ġsem antically Ġsimilar Ġtokens , Ġand Ġspecial Ġcharacters Ġsuch Ġas Ġem oj is Ġare Ġalso Ġimmensely Ġpopular Ġand Ġcarry Ġuseful Ġsemantic Ġinformation . ĠIn Ġour Ġmoderately Ġsized Ġtraining Ġdataset Ġof Ġ2 Ġmillion Ġtweets , Ġthere Ġwere Ġabout Ġ0 . 92 Ġmillion Ġunique Ġword Ġtypes . ĠIt Ġwould Ġbe Ġexpensive Ġto Ġcapture Ġall Ġthese Ġphenomena Ġin Ġa Ġword Ġbased Ġmodel Ġin Ġterms Ġof Ġboth Ġthe Ġmemory Ġrequirement Ġ( for Ġthe Ġincreased Ġvocabulary ) Ġand Ġthe Ġamount Ġof Ġtraining Ġdata Ġrequired Ġfor Ġeffective Ġlearning . ĠAdditional Ġbenefits Ġof Ġthe Ġcharacter Ġbased Ġapproach Ġinclude Ġlanguage Ġindependence Ġof Ġthe Ġmethods , Ġand Ġno Ġrequirement Ġof ĠN LP Ġpre processing Ġsuch Ġas Ġword - se gment ation . </s> A Ġcrucial Ġstep Ġin Ġlearning Ġgood Ġtext Ġrepresentations Ġis Ġto Ġchoose Ġan Ġappropriate Ġobjective Ġfunction Ġto Ġoptimize . ĠUn super vised Ġapproaches Ġattempt Ġto Ġreconstruct Ġthe Ġoriginal Ġtext Ġfrom Ġits Ġlatent Ġrepresentation ĠB IB REF 4 Ġ, ĠB IB REF 0 Ġ. ĠSocial Ġmedia Ġposts Ġhowever , Ġcome Ġwith Ġtheir Ġown Ġform Ġof Ġsupervision Ġannot ated Ġby Ġmillions Ġof Ġusers , Ġin Ġthe Ġform Ġof Ġhasht ags Ġwhich Ġlink Ġposts Ġabout Ġthe Ġsame Ġtopic Ġtogether . ĠA Ġnatural Ġassumption Ġis Ġthat Ġthe Ġposts Ġwith Ġthe Ġsame Ġhasht ags Ġshould Ġhave Ġembed d ings Ġwhich Ġare Ġclose Ġto Ġeach Ġother . ĠHence , Ġwe Ġformulate Ġour Ġtraining Ġobjective Ġto Ġmaximize Ġcross - ent ropy Ġloss Ġat Ġthe Ġtask Ġof Ġpredicting Ġhasht ags Ġfor Ġa Ġpost Ġfrom Ġits Ġlatent Ġrepresentation . </s> We Ġpropose Ġa ĠBi - direction al ĠG ated ĠRec urrent ĠUnit Ġ( Bi - GR U ) ĠB IB REF 5 Ġneural Ġnetwork Ġfor Ġlearning Ġtweet Ġrepresentations . ĠTreat ing Ġwhite - space Ġas Ġa Ġspecial Ġcharacter Ġitself , Ġthe Ġmodel Ġdoes Ġa Ġforward Ġand Ġbackward Ġpass Ġover Ġthe Ġentire Ġsequence , Ġand Ġthe Ġfinal ĠGR U Ġstates Ġare Ġlin early Ġcombined Ġto Ġget Ġthe Ġtweet Ġembed ding . ĠPoster ior Ġprobabilities Ġover Ġhasht ags Ġare Ġcomputed Ġby Ġprojecting Ġthis Ġembed ding Ġto Ġa Ġsoft max Ġoutput Ġlayer . ĠCompared Ġto Ġa Ġword - level Ġbaseline Ġthis Ġmodel Ġshows Ġimproved Ġperformance Ġat Ġpredicting Ġhasht ags Ġfor Ġa Ġheld - out Ġset Ġof Ġposts . ĠInspired Ġby Ġrecent Ġwork Ġin Ġlearning Ġvector Ġspace Ġtext Ġrepresentations , Ġwe Ġname Ġour Ġmodel Ġtweet 2 vec . </s> Related ĠWork </s> Using Ġneural Ġnetworks Ġto Ġlearn Ġdistributed Ġrepresentations Ġof Ġwords Ġdates Ġback Ġto ĠB IB REF 0 Ġ. ĠMore Ġrecently , ĠB IB REF 4 Ġreleased Ġword 2 vec Ġ- Ġa Ġcollection Ġof Ġword Ġvectors Ġtrained Ġusing Ġa Ġrecurrent Ġneural Ġnetwork . ĠThese Ġword Ġvectors Ġare Ġin Ġwidespread Ġuse Ġin Ġthe ĠN LP Ġcommunity , Ġand Ġthe Ġoriginal Ġwork Ġhas Ġsince Ġbeen Ġextended Ġto Ġsentences ĠB IB REF 1 Ġ, Ġdocuments Ġand Ġparagraphs ĠB IB REF 6 Ġ, Ġtopics ĠB IB REF 7 Ġand Ġqueries ĠB IB REF 8 Ġ. ĠAll Ġthese Ġmethods Ġrequire Ġstoring Ġan Ġextremely Ġlarge Ġtable Ġof Ġvectors Ġfor Ġall Ġword Ġtypes Ġand Ġcannot Ġbe Ġeasily Ġgeneralized Ġto Ġunseen Ġwords Ġat Ġtest Ġtime ĠB IB REF 2 Ġ. ĠThey Ġalso Ġrequire Ġpre processing Ġto Ġfind Ġword Ġboundaries Ġwhich Ġis Ġnon - t riv ial Ġfor Ġa Ġsocial Ġnetwork Ġdomain Ġlike ĠTwitter . </s> In ĠB IB REF 2 Ġ, Ġthe Ġauthors Ġpresent Ġa Ġcompos itional Ġcharacter Ġmodel Ġbased Ġon Ġbid irection al ĠL ST Ms Ġas Ġa Ġpotential Ġsolution Ġto Ġthese Ġproblems . ĠA Ġmajor Ġbenefit Ġof Ġthis Ġapproach Ġis Ġthat Ġlarge Ġword Ġlookup Ġtables Ġcan Ġbe Ġcompact ed Ġinto Ġcharacter Ġlookup Ġtables Ġand Ġthe Ġcompos itional Ġmodel Ġscales Ġto Ġlarge Ġdata Ġsets Ġbetter Ġthan Ġother Ġstate - of - the - art Ġapproaches . ĠWhile ĠB IB REF 2 Ġgenerate Ġword Ġembed d ings Ġfrom Ġcharacter Ġrepresentations , Ġwe Ġpropose Ġto Ġgenerate Ġvector Ġrepresentations Ġof Ġentire Ġtweets Ġfrom Ġcharacters Ġin Ġour Ġtweet 2 vec Ġmodel . </s> Our Ġwork Ġadds Ġto Ġthe Ġgrowing Ġbody Ġof Ġwork Ġshowing Ġthe Ġapplic ability Ġof Ġcharacter Ġmodels Ġfor Ġa Ġvariety Ġof ĠN LP Ġtasks Ġsuch Ġas ĠNamed ĠEntity ĠRecogn ition ĠB IB REF 9 Ġ, ĠPOS Ġtagging ĠB IB REF 10 Ġ, Ġtext Ġclassification ĠB IB REF 11 Ġand Ġlanguage Ġmodeling ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ. </s> Previously , ĠB IB REF 14 Ġdealt Ġwith Ġthe Ġproblem Ġof Ġestimating Ġrare Ġword Ġrepresentations Ġby Ġbuilding Ġthem Ġfrom Ġtheir Ġconstituent Ġmorp hem es . ĠWhile Ġthey Ġshow Ġimproved Ġperformance Ġover Ġword - based Ġmodels , Ġtheir Ġapproach Ġrequires Ġa Ġmor phe me Ġparser Ġfor Ġpre processing Ġwhich Ġmay Ġnot Ġperform Ġwell Ġon Ġnoisy Ġtext Ġlike ĠTwitter . ĠAlso Ġthe Ġspace Ġof Ġall Ġmorp hem es , Ġthough Ġsmaller Ġthan Ġthe Ġspace Ġof Ġall Ġwords , Ġis Ġstill Ġlarge Ġenough Ġthat Ġmodelling Ġall Ġmorp hem es Ġis Ġimpractical . </s> Hash tag Ġprediction Ġfor Ġsocial Ġmedia Ġhas Ġbeen Ġaddressed Ġearlier , Ġfor Ġexample Ġin ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ. ĠB IB REF 15 Ġalso Ġuse Ġa Ġneural Ġarchitecture , Ġbut Ġcompose Ġtext Ġembed d ings Ġfrom Ġa Ġlookup Ġtable Ġof Ġwords . ĠThey Ġalso Ġshow Ġthat Ġthe Ġlearned Ġembed d ings Ġcan Ġgeneral ize Ġto Ġan Ġunrelated Ġtask Ġof Ġdocument Ġrecommendation , Ġjustifying Ġthe Ġuse Ġof Ġhasht ags Ġas Ġsupervision Ġfor Ġlearning Ġtext Ġrepresentations . </s> Tweet 2 V ec </s> Bi - GR U ĠEnc oder : ĠFigure Ġ1 Ġshows Ġour Ġmodel Ġfor Ġencoding Ġtweets . ĠIt Ġuses Ġa Ġsimilar Ġstructure Ġto Ġthe ĠC 2 W Ġmodel Ġin ĠB IB REF 2 Ġ, Ġwith ĠL ST M Ġunits Ġreplaced Ġwith ĠGR U Ġunits . </s> The Ġinput Ġto Ġthe Ġnetwork Ġis Ġdefined Ġby Ġan Ġalphabet Ġof Ġcharacters Ġ$ C $ Ġ( this Ġmay Ġinclude Ġthe Ġentire Ġunic ode Ġcharacter Ġset ). ĠThe Ġinput Ġtweet Ġis Ġbroken Ġinto Ġa Ġstream Ġof Ġcharacters Ġ$ c _ 1 , Ġc _ 2 , Ġ... Ġc _ m $ Ġeach Ġof Ġwhich Ġis Ġrepresented Ġby Ġa Ġ1 - by - Ġ$ | C | $ Ġencoding . ĠThese Ġone - hot Ġvectors Ġare Ġthen Ġprojected Ġto Ġa Ġcharacter Ġspace Ġby Ġmultiplying Ġwith Ġthe Ġmatrix Ġ$ P _ C Ġ\ in Ġ\ math bb Ġ{ R } ^{ | C | Ġ\ times Ġd _ c } $ Ġ, Ġwhere Ġ$ d _ c $ Ġis Ġthe Ġdimension Ġof Ġthe Ġcharacter Ġvector Ġspace . ĠLet Ġ$ x _ 1 , Ġx _ 2 , Ġ... Ġx _ m $ Ġbe Ġthe Ġsequence Ġof Ġcharacter Ġvectors Ġfor Ġthe Ġinput Ġtweet Ġafter Ġthe Ġlookup . ĠThe Ġenc oder Ġconsists Ġof Ġa Ġforward - GR U Ġand Ġa Ġbackward - GR U . ĠBoth Ġhave Ġthe Ġsame Ġarchitecture , Ġexcept Ġthe Ġbackward - GR U Ġprocesses Ġthe Ġsequence Ġin Ġreverse Ġorder . ĠEach Ġof Ġthe ĠGR U Ġunits Ġprocess Ġthese Ġvectors Ġsequ entially , Ġand Ġstarting Ġwith Ġthe Ġinitial Ġstate Ġ$ h _ 0 $ Ġcompute Ġthe Ġsequence Ġ$ h _ 1 , Ġh _ 2 , Ġ... Ġh _ m $ Ġas Ġfollows : Ġ$ Ġr _ t Ġ& = Ġ\ s igma Ġ( W _ r Ġx _ t Ġ+ ĠU _ r Ġh _{ t - 1 } Ġ+ Ġb _ r ), Ġ\\ Ġz _ t Ġ& = Ġ\ s igma Ġ( W _ z Ġx _ t Ġ+ ĠU _ z Ġh _{ t - 1 } Ġ+ Ġb _ z ), Ġ\\ Ġ\ t ilde { h } _ t Ġ& = Ġtan h ( W _ h Ġx _ t Ġ+ ĠU _ h Ġ( r _ t Ġ\ od ot Ġh _{ t - 1 }) Ġ+ Ġb _ h ), Ġ\\ Ġh _ t Ġ& = Ġ( 1 - z _ t ) Ġ\ od ot Ġh _{ t - 1 } Ġ+ Ġz _ t Ġ\ od ot Ġ\ t ilde { h } _ t . Ġ$ </s> Here Ġ$ r _ t $ Ġ, Ġ$ z _ t $ Ġare Ġcalled Ġthe Ġreset Ġand Ġupdate Ġgates Ġrespectively , Ġand Ġ$\ t ilde { h } _ t $ Ġis Ġthe Ġcandidate Ġoutput Ġstate Ġwhich Ġis Ġconverted Ġto Ġthe Ġactual Ġoutput Ġstate Ġ$ h _ t $ Ġ. Ġ$ W _ r , ĠW _ z , ĠW _ h $ Ġare Ġ$ d _ h Ġ\ times Ġd _ c $ Ġmat rices Ġand Ġ$ U _ r , ĠU _ z , ĠU _ h $ Ġare Ġ$ d _ h Ġ\ times Ġd _ h $ Ġmat rices , Ġwhere Ġ$ d _ h $ Ġis Ġthe Ġhidden Ġstate Ġdimension Ġof Ġthe ĠGR U . ĠThe Ġfinal Ġstates Ġ$ h _ m ^ f $ Ġfrom Ġthe Ġforward - GR U , Ġand Ġ$ z _ t $ 0 Ġfrom Ġthe Ġbackward ĠGR U Ġare Ġcombined Ġusing Ġa Ġfully - connected Ġlayer Ġto Ġthe Ġgive Ġthe Ġfinal Ġtweet Ġembed ding Ġ$ z _ t $ 1 Ġ: </s> $$ e _ t Ġ= ĠW ^ f Ġh _ m ^ f Ġ+ ĠW ^ b Ġh _ 0 ^ b $$ Ġ Ġ Ġ( E q . Ġ3 ) </s> Here Ġ$ W ^ f , ĠW ^ b $ Ġare Ġ$ d _ t Ġ\ times Ġd _ h $ Ġand Ġ$ b $ Ġis Ġ$ d _ t Ġ\ times Ġ1 $ Ġbias Ġterm , Ġwhere Ġ$ d _ t $ Ġis Ġthe Ġdimension Ġof Ġthe Ġfinal Ġtweet Ġembed ding . ĠIn Ġour Ġexperiments Ġwe Ġset Ġ$ d _ t = d _ h $ Ġ. ĠAll Ġparameters Ġare Ġlearned Ġusing Ġgradient Ġdescent . </s> Soft max : ĠFinally , Ġthe Ġtweet Ġembed ding Ġis Ġpassed Ġthrough Ġa Ġlinear Ġlayer Ġwhose Ġoutput Ġis Ġthe Ġsame Ġsize Ġas Ġthe Ġnumber Ġof Ġhasht ags Ġ$ L $ Ġin Ġthe Ġdata Ġset . ĠWe Ġuse Ġa Ġsoft max Ġlayer Ġto Ġcompute Ġthe Ġposterior Ġhashtag Ġprobabilities : </s> $$ P ( y = j Ġ| e ) Ġ= Ġ\ frac { exp ( w _ j ^ Te Ġ+ Ġb _ j ) }{ \ sum Ġ_ { i = 1 } ^ L Ġexp ( w _ i ^ Te Ġ+ Ġb _ j ) }. $$ Ġ Ġ Ġ( E q . Ġ4 ) </s> Object ive ĠFunction : ĠWe Ġoptimize Ġthe Ġcateg orical Ġcross - ent ropy Ġloss Ġbetween Ġpredicted Ġand Ġtrue Ġhasht ags : </s> $$ J Ġ= Ġ\ frac { 1 }{ B } Ġ\ sum Ġ_ { i = 1 } ^{ B } Ġ\ sum Ġ_ { j = 1 } ^{ L } Ġ- t _{ i , j } log ( p _{ i , j }) Ġ+ Ġ\ lambda Ġ\ Vert Ġ\ The ta Ġ\ Vert Ġ^ 2 . $$ Ġ Ġ Ġ( E q . Ġ5 ) </s> Here Ġ$ B $ Ġis Ġthe Ġbatch Ġsize , Ġ$ L $ Ġis Ġthe Ġnumber Ġof Ġclasses , Ġ$ p _{ i , j } $ Ġis Ġthe Ġpredicted Ġprobability Ġthat Ġthe Ġ$ i $ Ġ- th Ġtweet Ġhas Ġhashtag Ġ$ j $ Ġ, Ġand Ġ$ t _{ i , j } Ġ\ in Ġ\ l brace Ġ0 , 1 \ r brace Ġ$ Ġdenotes Ġthe Ġground Ġtruth Ġof Ġwhether Ġthe Ġ$ j $ Ġ- th Ġhashtag Ġis Ġin Ġthe Ġ$ i $ Ġ- th Ġtweet . ĠWe Ġuse ĠL 2 - regular ization Ġweighted Ġby Ġ$\ lambda Ġ$ Ġ. </s> Word ĠLevel ĠBas eline </s> Since Ġour Ġobjective Ġis Ġto Ġcompare Ġcharacter - based Ġand Ġword - based Ġapproaches , Ġwe Ġhave Ġalso Ġimplemented Ġa Ġsimple Ġword - level Ġenc oder Ġfor Ġtweets . ĠThe Ġinput Ġtweet Ġis Ġfirst Ġsplit Ġinto Ġtokens Ġalong Ġwhite - sp aces . ĠA Ġmore Ġsophisticated Ġtoken izer Ġmay Ġbe Ġused , Ġbut Ġfor Ġa Ġfair Ġcomparison Ġwe Ġwanted Ġto Ġkeep Ġlanguage Ġspecific Ġpre processing Ġto Ġa Ġminimum . ĠThe Ġenc oder Ġis Ġessentially Ġthe Ġsame Ġas Ġtweet 2 vec , Ġwith Ġthe Ġinput Ġas Ġwords Ġinstead Ġof Ġcharacters . ĠA Ġlookup Ġtable Ġstores Ġword Ġvectors Ġfor Ġthe Ġ$ V $ Ġ( 20 K Ġhere ) Ġmost Ġcommon Ġwords , Ġand Ġthe Ġrest Ġare Ġgrouped Ġtogether Ġunder Ġthe Ġ` UN K ' Ġtoken . </s> Data </s> Our Ġdataset Ġconsists Ġof Ġa Ġlarge Ġcollection Ġof Ġglobal Ġposts Ġfrom ĠTwitter Ġbetween Ġthe Ġdates Ġof ĠJune Ġ1 , Ġ2013 Ġto ĠJune Ġ5 , Ġ2013 . ĠOnly ĠEnglish Ġlanguage Ġposts Ġ( as Ġdetected Ġby Ġthe Ġlang Ġfield Ġin ĠTwitter ĠAPI ) Ġand Ġposts Ġwith Ġat Ġleast Ġone Ġhashtag Ġare Ġretained . ĠWe Ġremoved Ġinf requent Ġhasht ags Ġ( Ġ$ < 500 $ Ġposts ) Ġsince Ġthey Ġdo Ġnot Ġhave Ġenough Ġdata Ġfor Ġgood Ġgeneral ization . ĠWe Ġalso Ġremoved Ġvery Ġfrequent Ġtags Ġ( Ġ$ > 19 K $ Ġposts ) Ġwhich Ġwere Ġalmost Ġalways Ġfrom Ġautomatically Ġgenerated Ġposts Ġ( ex : Ġ# android game ) Ġwhich Ġare Ġtrivial Ġto Ġpredict . ĠThe Ġfinal Ġdataset Ġcontains Ġ2 Ġmillion Ġtweets Ġfor Ġtraining , Ġ10 K Ġfor Ġvalidation Ġand Ġ50 K Ġfor Ġtesting , Ġwith Ġa Ġtotal Ġof Ġ20 39 Ġdistinct Ġhasht ags . ĠWe Ġuse Ġsimple Ġregex Ġto Ġpre process Ġthe Ġpost Ġtext Ġand Ġremove Ġhasht ags Ġ( since Ġthese Ġare Ġto Ġbe Ġpredicted ) Ġand ĠHTML Ġtags , Ġand Ġreplace Ġuser - names Ġand ĠURLs Ġwith Ġspecial Ġtokens . ĠWe Ġalso Ġremoved Ġret we ets Ġand Ġconvert Ġthe Ġtext Ġto Ġlower - case . </s> Im plementation ĠDetails </s> Word Ġvectors Ġand Ġcharacter Ġvectors Ġare Ġboth Ġset Ġto Ġsize Ġ$ d _ L = 150 $ Ġfor Ġtheir Ġrespective Ġmodels . ĠThere Ġwere Ġ28 29 Ġunique Ġcharacters Ġin Ġthe Ġtraining Ġset Ġand Ġwe Ġmodel Ġeach Ġof Ġthese Ġindependently Ġin Ġa Ġcharacter Ġlook - up Ġtable . ĠEmb edd ing Ġsizes Ġwere Ġchosen Ġsuch Ġthat Ġeach Ġmodel Ġhad Ġroughly Ġthe Ġsame Ġnumber Ġof Ġparameters Ġ( Table Ġ2 Ġ). ĠTraining Ġis Ġperformed Ġusing Ġmini - batch Ġgradient Ġdescent Ġwith ĠN ester ov 's Ġmomentum . ĠWe Ġuse Ġa Ġbatch Ġsize Ġ$ B = 64 $ Ġ, Ġinitial Ġlearning Ġrate Ġ$\ eta Ġ_ 0 = 0 . 01 $ Ġand Ġmomentum Ġparameter Ġ$\ mu Ġ_ 0 = 0 . 9 $ Ġ. ĠL 2 - regular ization Ġwith Ġ$\ lambda Ġ= 0 . 001 $ Ġwas Ġapplied Ġto Ġall Ġmodels . ĠInitial Ġweights Ġwere Ġdrawn Ġfrom Ġ0 - mean Ġga uss ians Ġwith Ġ$\ s igma Ġ= 0 . 1 $ Ġand Ġinitial Ġbiases Ġwere Ġset Ġto Ġ0 . ĠThe Ġhyper param eters Ġwere Ġtuned Ġone Ġat Ġa Ġtime Ġkeeping Ġothers Ġfixed , Ġand Ġvalues Ġwith Ġthe Ġlowest Ġvalidation Ġcost Ġwere Ġchosen . ĠThe Ġresultant Ġcombination Ġwas Ġused Ġto Ġtrain Ġthe Ġmodels Ġuntil Ġperformance Ġon Ġvalidation Ġset Ġstopped Ġincreasing . ĠDuring Ġtraining , Ġthe Ġlearning Ġrate Ġis Ġhal ved Ġevery time Ġthe Ġvalidation Ġset Ġprecision Ġincreases Ġby Ġless Ġthan Ġ0 . 01 Ġ% Ġfrom Ġone Ġepoch Ġto Ġthe Ġnext . ĠThe Ġmodels Ġconverge Ġin Ġabout Ġ20 Ġepoch s . ĠCode Ġfor Ġtraining Ġboth Ġthe Ġmodels Ġis Ġpublicly Ġavailable Ġon Ġgithub . </s> Results </s> We Ġtest Ġthe Ġcharacter Ġand Ġword - level Ġvariants Ġby Ġpredicting Ġhasht ags Ġfor Ġa Ġheld - out Ġtest Ġset Ġof Ġposts . ĠSince Ġthere Ġmay Ġbe Ġmore Ġthan Ġone Ġcorrect Ġhashtag Ġper Ġpost , Ġwe Ġgenerate Ġa Ġranked Ġlist Ġof Ġtags Ġfor Ġeach Ġpost Ġfrom Ġthe Ġoutput Ġposter iors , Ġand Ġreport Ġaverage Ġprecision @ 1 , Ġrecall @ 10 Ġand Ġmean Ġrank Ġof Ġthe Ġcorrect Ġhasht ags . ĠThese Ġare Ġlisted Ġin ĠTable Ġ3 Ġ. </s> To Ġsee Ġthe Ġperformance Ġof Ġeach Ġmodel Ġon Ġposts Ġcontaining Ġrare Ġwords Ġ( RW ) Ġand Ġfrequent Ġwords Ġ( FW ) Ġwe Ġselected Ġtwo Ġtest Ġsets Ġeach Ġcontaining Ġ2 , 000 Ġposts . ĠWe Ġpopulated Ġthese Ġsets Ġwith Ġposts Ġwhich Ġhad Ġthe Ġmaximum Ġand Ġminimum Ġnumber Ġof Ġout - of - voc abulary Ġwords Ġrespectively , Ġwhere Ġvocabulary Ġis Ġdefined Ġby Ġthe Ġ20 K Ġmost Ġfrequent Ġwords . ĠOverall , Ġtweet 2 vec Ġoutper forms Ġthe Ġword Ġmodel , Ġdoing Ġsignificantly Ġbetter Ġon ĠRW Ġtest Ġset Ġand Ġcompar ably Ġon ĠFW Ġset . ĠThis Ġimproved Ġperformance Ġcomes Ġat Ġthe Ġcost Ġof Ġincreased Ġtraining Ġtime Ġ( see ĠTable Ġ2 Ġ), Ġsince Ġmoving Ġfrom Ġwords Ġto Ġcharacters Ġresults Ġin Ġlonger Ġinput Ġsequences Ġto Ġthe ĠGR U . </s> We Ġalso Ġstudy Ġthe Ġeffect Ġof Ġmodel Ġsize Ġon Ġthe Ġperformance Ġof Ġthese Ġmodels . ĠFor Ġthe Ġword Ġmodel Ġwe Ġset Ġvocabulary Ġsize Ġ$ V $ Ġto Ġ8 K , Ġ15 K Ġand Ġ20 K Ġrespectively . ĠFor Ġtweet 2 vec Ġwe Ġset Ġthe ĠGR U Ġhidden Ġstate Ġsize Ġto Ġ300 , Ġ400 Ġand Ġ500 Ġrespectively . ĠFigure Ġ2 Ġshows Ġprecision Ġ1 Ġof Ġthe Ġtwo Ġmodels Ġas Ġthe Ġnumber Ġof Ġparameters Ġis Ġincreased , Ġfor Ġeach Ġtest Ġset Ġdescribed Ġabove . ĠThere Ġis Ġnot Ġmuch Ġvariation Ġin Ġthe Ġperformance , Ġand Ġmoreover Ġtweet 2 vec Ġalways Ġoutper forms Ġthe Ġword Ġbased Ġmodel Ġfor Ġthe Ġsame Ġnumber Ġof Ġparameters . </s> Table Ġ4 Ġcompares Ġthe Ġmodels Ġas Ġcomplexity Ġof Ġthe Ġtask Ġis Ġincreased . ĠWe Ġcreated Ġ3 Ġdatasets Ġ( small , Ġmedium Ġand Ġlarge ) Ġwith Ġan Ġincreasing Ġnumber Ġof Ġhasht ags Ġto Ġbe Ġpredicted . ĠThis Ġwas Ġdone Ġby Ġvarying Ġthe Ġlower Ġthreshold Ġof Ġthe Ġminimum Ġnumber Ġof Ġtags Ġper Ġpost Ġfor Ġit Ġto Ġbe Ġincluded Ġin Ġthe Ġdataset . ĠOnce Ġagain Ġwe Ġobserve Ġthat Ġtweet 2 vec Ġoutper forms Ġits Ġword - based Ġcounterpart Ġfor Ġeach Ġof Ġthe Ġthree Ġsettings . </s> Finally , Ġtable Ġ1 Ġshows Ġsome Ġpredictions Ġfrom Ġthe Ġword Ġlevel Ġmodel Ġand Ġtweet 2 vec . ĠWe Ġselected Ġthese Ġto Ġhighlight Ġsome Ġstrengths Ġof Ġthe Ġcharacter Ġbased Ġapproach Ġ- Ġit Ġis Ġrobust Ġto Ġword Ġsegment ation Ġerrors Ġand Ġspelling Ġmistakes , Ġeffectively Ġinterpre ts Ġem oj is Ġand Ġother Ġspecial Ġcharacters Ġto Ġmake Ġpredictions , Ġand Ġalso Ġperforms Ġcompar ably Ġto Ġthe Ġword - based Ġapproach Ġfor Ġin - voc abulary Ġtokens . </s> Conclusion </s> We Ġhave Ġpresented Ġtweet 2 vec Ġ- Ġa Ġcharacter Ġlevel Ġenc oder Ġfor Ġsocial Ġmedia Ġposts Ġtrained Ġusing Ġsupervision Ġfrom Ġassociated Ġhasht ags . ĠOur Ġresult Ġshows Ġthat Ġtweet 2 vec Ġoutper forms Ġthe Ġword Ġbased Ġapproach , Ġdoing Ġsignificantly Ġbetter Ġwhen Ġthe Ġinput Ġpost Ġcontains Ġmany Ġrare Ġwords . ĠWe Ġhave Ġfocused Ġonly Ġon ĠEnglish Ġlanguage Ġposts , Ġbut Ġthe Ġcharacter Ġmodel Ġrequires Ġno Ġlanguage Ġspecific Ġpre processing Ġand Ġcan Ġbe Ġextended Ġto Ġother Ġlanguages . ĠFor Ġfuture Ġwork , Ġone Ġnatural Ġextension Ġwould Ġbe Ġto Ġuse Ġa Ġcharacter - level Ġdec oder Ġfor Ġpredicting Ġthe Ġhasht ags . ĠThis Ġwill Ġallow Ġgeneration Ġof Ġhasht ags Ġnot Ġseen Ġin Ġthe Ġtraining Ġdataset . ĠAlso , Ġit Ġwill Ġbe Ġinteresting Ġto Ġsee Ġhow Ġour Ġtweet 2 vec Ġembed d ings Ġcan Ġbe Ġused Ġin Ġdomains Ġwhere Ġthere Ġis Ġa Ġneed Ġfor Ġsemantic Ġunderstanding Ġof Ġsocial Ġmedia , Ġsuch Ġas Ġtracking Ġinfectious Ġdiseases ĠB IB REF 17 Ġ. ĠHence , Ġwe Ġprovide Ġan Ġoff - the - she lf Ġenc oder Ġtrained Ġon Ġmedium Ġdataset Ġdescribed Ġabove Ġto Ġcompute Ġvector - space Ġrepresentations Ġof Ġtweets Ġalong Ġwith Ġour Ġcode Ġon Ġgithub . </s> Acknowled gments </s> We Ġwould Ġlike Ġto Ġthank ĠAlex ĠSm ola , ĠYun ĠFu , ĠH s iao - Yu ĠFish ĠT ung , ĠRus lan ĠSal akh ut din ov , Ġand ĠBarn ab as ĠPoc zos Ġfor Ġuseful Ġdiscussions . ĠWe Ġwould Ġalso Ġlike Ġto Ġthank ĠJu er gen ĠP fe ffer Ġfor Ġproviding Ġaccess Ġto Ġthe ĠTwitter Ġdata , Ġand Ġthe Ġreviewers Ġfor Ġtheir Ġcomments .