Energy with accuracy of 0.538
ABSTRACTIVE
Energy
ABSTRACTIVE
FLOAT SELECTED: Table 8: Sector-level performance comparison.
FLOAT SELECTED: Table 7: Our volatility model performance compared with GARCH(1,1). Best performance in bold. Our model has superior performance across the three evaluation metrics and taking into consideration the state-of-the-art volatility proxies, namely Garman-Klass (σ̂PK) and Parkinson (σ̂PK).
1812.10479
Introduction </s> Natural ĠLanguage ĠProcessing Ġ( N LP ) Ġhas Ġincreasingly Ġattracted Ġthe Ġattention Ġof Ġthe Ġfinancial Ġcommunity . ĠThis Ġtrend Ġcan Ġbe Ġexplained Ġby Ġat Ġleast Ġthree Ġmajor Ġfactors . ĠThe Ġfirst Ġfactor Ġrefers Ġto Ġthe Ġbusiness Ġperspective . ĠIt Ġis Ġthe Ġeconomics Ġof Ġgaining Ġcompetitive Ġadvantage Ġusing Ġalternative Ġsources Ġof Ġdata Ġand Ġgoing Ġbeyond Ġhistorical Ġstock Ġprices , Ġthus , Ġtrading Ġby Ġanalyzing Ġmarket Ġnews Ġautomatically . ĠThe Ġsecond Ġfactor Ġis Ġthe Ġmajor Ġadvancements Ġin Ġthe Ġtechnologies Ġto Ġcollect , Ġstore , Ġand Ġquery Ġmassive Ġamounts Ġof Ġuser - generated Ġdata Ġalmost Ġin Ġreal - time . ĠThe Ġthird Ġfactor Ġrefers Ġto Ġthe Ġprogress Ġmade Ġby Ġthe ĠN LP Ġcommunity Ġin Ġunderstanding Ġun struct ured Ġtext . ĠOver Ġthe Ġlast Ġdecades Ġthe Ġnumber Ġof Ġstudies Ġusing ĠN LP Ġfor Ġfinancial Ġforecasting Ġhas Ġexperienced Ġexponential Ġgrowth . ĠAccording Ġto ĠB IB REF 0 Ġ, Ġuntil Ġ2008 , Ġless Ġthan Ġfive Ġresearch Ġarticles Ġwere Ġpublished Ġper Ġyear Ġmentioning Ġboth ĠâĢ ľ stock Ġmarket âĢ Ŀ Ġand ĠâĢ ľ text Ġmining âĢ Ŀ Ġor ĠâĢ ľ sent iment Ġanalysis âĢ Ŀ Ġkeywords . ĠIn Ġ2012 , Ġthis Ġnumber Ġincreased Ġto Ġslightly Ġmore Ġthan Ġten Ġarticles Ġper Ġyear . ĠThe Ġlast Ġnumbers Ġavailable Ġfor Ġ2016 Ġindicates Ġthis Ġhas Ġincreased Ġto Ġsixty Ġarticles Ġper Ġyear . </s> The Ġability Ġto Ġmechanically Ġharvest Ġthe Ġsentiment Ġfrom Ġtexts Ġusing ĠN LP Ġhas Ġshed Ġlight Ġon Ġconflicting Ġtheories Ġof Ġfinancial Ġeconomics . ĠHistorically , Ġthere Ġhas Ġbeen Ġtwo Ġdiffering Ġviews Ġon Ġwhether Ġdisagreement Ġamong Ġmarket Ġparticipants Ġinduces Ġmore Ġtrades . ĠThe ĠâĢ ľ non - trade Ġtheorem âĢ Ŀ ĠB IB REF 1 Ġstates Ġthat Ġassuming Ġall Ġmarket Ġparticipants Ġhave Ġcommon Ġknowledge Ġabout Ġa Ġmarket Ġevent , Ġthe Ġlevel Ġof Ġdisagreement Ġamong Ġthe Ġparticipants Ġdoes Ġnot Ġincrease Ġthe Ġnumber Ġof Ġtrades Ġbut Ġonly Ġleads Ġto Ġa Ġrevision Ġof Ġthe Ġmarket Ġquotes . ĠIn Ġcontrast , Ġthe Ġtheoretically Ġframework Ġproposed Ġin ĠB IB REF 2 Ġadvocates Ġthat Ġdisagreement Ġamong Ġmarket Ġparticipants Ġincreases Ġtrading Ġvolume . ĠUsing Ġtextual Ġdata Ġfrom ĠYahoo Ġand ĠR aging Bull . com Ġmessage Ġboards Ġto Ġmeasure Ġthe Ġdisp ersion Ġof Ġopinions Ġ( positive Ġor Ġnegative ) Ġamong Ġtraders , Ġit Ġwas Ġshown Ġin ĠB IB REF 3 Ġthat Ġdisagreement Ġamong Ġusers ' Ġmessages Ġhelps Ġto Ġpredict Ġsubsequent Ġtrading Ġvolume Ġand Ġvolatility . ĠSimilar Ġrelation Ġbetween Ġdisagreement Ġand Ġincreased Ġtrading Ġvolume Ġwas Ġfound Ġin ĠB IB REF 4 Ġusing ĠTwitter Ġposts . ĠAdditionally , Ġtextual Ġanalysis Ġis Ġadding Ġto Ġthe Ġtheories Ġof Ġmedium - term / long - term Ġmomentum / re vers al Ġin Ġstock Ġmarkets ĠB IB REF 5 Ġ. ĠThe Ġunified ĠHong Ġand ĠStein Ġmodel ĠB IB REF 6 Ġon Ġstock 's Ġmomentum / re vers al Ġproposes Ġthat Ġinvestors Ġunder react Ġto Ġnews , Ġcausing Ġslow Ġprice Ġdr ifts , Ġand Ġover react Ġto Ġprice Ġshocks Ġnot Ġaccompanied Ġby Ġnews , Ġhence Ġinducing Ġrevers als . ĠThis Ġtheoretical Ġpred icated Ġbehaviour Ġbetween Ġprice Ġand Ġnews Ġwas Ġsystematically Ġestimated Ġand Ġsupported Ġin ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġusing Ġfinancial Ġmedia Ġheadlines Ġand Ġin ĠB IB REF 9 Ġusing Ġthe ĠConsumer ĠConf idence ĠIndex Â® Ġpublished Ġby ĠThe ĠConference ĠBoard ĠB IB REF 10 Ġ. ĠSimilarly , ĠB IB REF 11 Ġuses Ġthe ĠHarvard ĠIV - 4 Ġsentiment Ġlex icon Ġto Ġcount Ġthe Ġoccurrence Ġof Ġwords Ġwith Ġpositive Ġand Ġnegative Ġcon notation Ġof Ġthe ĠWall ĠStreet ĠJournal Ġshowing Ġthat Ġnegative Ġsentiment Ġis Ġa Ġgood Ġpredictor Ġof Ġprice Ġreturns Ġand Ġtrading Ġvolumes . </s> Acc urate Ġmodels Ġfor Ġforecasting Ġboth Ġprice Ġreturns Ġand Ġvolatility Ġare Ġequally Ġimportant Ġin Ġthe Ġfinancial Ġdomain . ĠVol atility Ġmeasures Ġhow Ġwildly Ġthe Ġasset Ġis Ġexpected Ġto Ġoscill ate Ġin Ġa Ġgiven Ġtime Ġperiod Ġand Ġis Ġrelated Ġto Ġthe Ġsecond Ġmoment Ġof Ġthe Ġprice Ġreturn Ġdistribution . ĠIn Ġgeneral Ġterms , Ġforecasting Ġprice Ġreturns Ġis Ġrelevant Ġto Ġtake Ġspeculative Ġpositions . ĠThe Ġvolatility , Ġon Ġthe Ġother Ġhand , Ġmeasures Ġthe Ġrisk Ġof Ġthese Ġpositions . ĠOn Ġa Ġdaily Ġbasis , Ġfinancial Ġinstitutions Ġneed Ġto Ġassess Ġthe Ġshort - term Ġrisk Ġof Ġtheir Ġportfolios . ĠMe asuring Ġthe Ġrisk Ġis Ġessential Ġin Ġmany Ġaspects . ĠIt Ġis Ġimperative Ġfor Ġregulatory Ġcapital Ġdisclosures Ġrequired Ġby Ġbanking Ġsupervision Ġbodies . ĠMoreover , Ġit Ġis Ġuseful Ġto Ġdynamically Ġadjust Ġposition Ġsizing Ġaccordingly Ġto Ġmarket Ġconditions , Ġthus , Ġmaintaining Ġthe Ġrisk Ġwithin Ġreasonable Ġlevels . </s> Although , Ġit Ġis Ġcrucial Ġto Ġpredict Ġthe Ġshort - term Ġvolatility Ġfrom Ġthe Ġfinancial Ġmarkets Ġapplication Ġperspective , Ġmuch Ġof Ġthe Ġcurrent ĠN LP Ġresearch Ġon Ġvolatility Ġforecasting Ġfocus Ġon Ġthe Ġvolatility Ġprediction Ġfor Ġvery Ġlong - term Ġhor izons Ġ( see ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ). ĠPred omin ately , Ġthese Ġworks Ġare Ġbuilt Ġon Ġextensions Ġof Ġthe Ġbag - of - words Ġrepresentation Ġthat Ġhas Ġthe Ġmain Ġdrawback Ġof Ġnot Ġcapturing Ġword Ġorder . ĠFinancial Ġforecasting , Ġhowever , Ġrequires Ġthe Ġability Ġto Ġcapture Ġsemantics Ġthat Ġis Ġdependent Ġupon Ġword Ġorder . ĠFor Ġexample , Ġthe Ġheadline ĠâĢ ľ Qual comm Ġsu es ĠApple Ġfor Ġcontract Ġbreach âĢ Ŀ Ġand ĠâĢ ľ Apple Ġsu es ĠQualcomm Ġfor Ġcontract Ġbreach âĢ Ŀ Ġtrigger Ġdifferent Ġresponses Ġfor Ġeach Ġstock Ġand Ġfor Ġthe Ġmarket Ġaggreg ated Ġindex , Ġhowever , Ġthey Ġshare Ġthe Ġsame Ġbag - of - words Ġrepresentation . ĠAdditionally , Ġthese Ġworks Ġuse Ġfeatures Ġfrom Ġa Ġpret rained Ġsentiment Ġanaly is Ġmodel Ġto Ġtrain Ġthe Ġfinancial Ġforecasting Ġmodel . ĠA Ġkey Ġlimitation Ġof Ġthis Ġprocess Ġis Ġthat Ġit Ġrequires Ġa Ġlabelled Ġsentiment Ġdataset . ĠAdditionally , Ġthe Ġerror Ġpropagation Ġis Ġnot Ġend - to - end . ĠIn Ġthis Ġwork , Ġwe Ġfill Ġin Ġthe Ġgaps Ġof Ġvolatility Ġprediction Ġresearch Ġin Ġthe Ġfollowing Ġmanner : </s> Related Ġwork </s> Previous Ġwork Ġin ĠB IB REF 12 Ġincorporates Ġsections Ġof Ġthe ĠâĢ ľ Form Ġ10 - K âĢ Ŀ Ġto Ġpredict Ġthe Ġvolatility Ġtwelve Ġmonths Ġafter Ġthe Ġreport Ġis Ġreleased . ĠThey Ġtrain Ġa ĠSupport ĠVector ĠReg ression Ġmodel Ġon Ġtop Ġof Ġsparse Ġrepresentation Ġ( bag - of - words ) Ġwith Ġstandard Ġterm Ġweight ing Ġ( e . g . ĠTerm - F requency ). ĠThis Ġwork Ġwas Ġextended Ġin ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġby Ġemploying Ġthe ĠL ough ran - Mc Donald ĠSent iment ĠWord ĠLists ĠB IB REF 20 Ġ, Ġwhich Ġcontain Ġthree Ġlists Ġwhere Ġwords Ġare Ġgrouped Ġby Ġtheir Ġsentiments Ġ( positive , Ġnegative Ġand Ġneutral ). ĠIn Ġall Ġthese Ġworks , Ġthe Ġtextual Ġrepresentation Ġis Ġengineered Ġusing Ġthe Ġfollowing Ġsteps : Ġ1 ) ĠFor Ġeach Ġsentiment Ġgroup , Ġthe Ġlist Ġis Ġexpanded Ġby Ġretrieving Ġ20 Ġmost Ġsimilar Ġwords Ġfor Ġeach Ġword Ġusing ĠWord 2 V ec Ġword Ġembed d ings ĠB IB REF 21 Ġ. Ġ2 ) ĠFinally , Ġeach Ġ10 - K Ġdocument Ġis Ġrepresented Ġusing Ġthe Ġexpanded Ġlists Ġof Ġwords . ĠThe Ġweight Ġof Ġeach Ġword Ġin Ġthis Ġsparse Ġrepresentation Ġis Ġdefined Ġusing ĠInformation ĠRet ri eval Ġ( IR ) Ġmethods Ġsuch Ġas Ġterm - frequency Ġ( tf ) Ġand Ġterm - frequency Ġwith Ġinverted Ġdocument Ġfrequency Ġ( tf id f ). ĠParticularly , ĠB IB REF 16 Ġshows Ġthat Ġresults Ġcan Ġbe Ġimproved Ġusing Ġenhanced ĠIR Ġmethods Ġand Ġprojecting Ġeach Ġsparse Ġfeature Ġinto Ġa Ġdense Ġspace Ġusing ĠPrincipal ĠComponent ĠAnalysis Ġ( PC A ). </s> The Ġworks Ġdescribed Ġabove Ġ( ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ) Ġtarget Ġlong - hor izon Ġvolatility Ġpredictions Ġ( one Ġyear Ġor Ġquarterly ĠB IB REF 16 Ġ). ĠIn Ġparticular , ĠB IB REF 16 Ġand ĠB IB REF 15 Ġuses Ġmarket Ġdata Ġ( price ) Ġfeatures Ġalong Ġwith Ġthe Ġtextual Ġrepresentation Ġof Ġthe Ġ10 - K Ġreports . ĠThese Ġexisting Ġworks Ġthat Ġemploy Ġmulti - mod al Ġlearning ĠB IB REF 22 Ġare Ġbased Ġon Ġa Ġlate Ġfusion Ġapproach . ĠFor Ġexample , Ġstacking Ġen semb les Ġto Ġtake Ġinto Ġaccount Ġthe Ġprice Ġand Ġtext Ġpredictions ĠB IB REF 16 Ġ. ĠIn Ġcontrast , Ġour Ġend - to - end Ġtrained Ġmodel Ġcan Ġlearn Ġthe Ġjoint Ġdistribution Ġof Ġboth Ġprice Ġand Ġtext . </s> P redict ing Ġthe Ġprice Ġdirection Ġrather Ġthan Ġthe Ġvolatility Ġwas Ġthe Ġfocus Ġin ĠB IB REF 23 Ġ. ĠThey Ġextracted Ġsentiment Ġwords Ġfrom ĠTwitter Ġposts Ġto Ġbuild Ġa Ġtime Ġseries Ġof Ġcollective ĠProfile Ġof ĠMood ĠStates Ġ( P OM S ). ĠTheir Ġresults Ġshow Ġthat Ġcollective Ġmood Ġaccurately Ġpredicts Ġthe Ġdirection Ġof ĠDown ĠJones Ġstock Ġindex Ġ( 86 . 7 % Ġaccuracy ). ĠIn ĠB IB REF 24 Ġhand crafted Ġtext Ġrepresentations Ġincluding Ġterm Ġcount , Ġnoun - phrase Ġtags Ġand Ġextracted Ġnamed Ġentities Ġare Ġemployed Ġfor Ġpredicting Ġstock Ġmarket Ġdirection Ġusing ĠSupport ĠVector ĠMachine Ġ( S VM ). ĠAn Ġextension Ġof ĠLat ent ĠDir ich let ĠAll ocation Ġ( L DA ) Ġis Ġproposed Ġin ĠB IB REF 25 Ġto Ġlearn Ġa Ġjoint Ġlatent Ġspace Ġof Ġtopics Ġand Ġsentiments . </s> Our Ġdeep Ġlearning Ġmodels Ġbear Ġa Ġclose Ġresemblance Ġto Ġworks Ġfocused Ġon Ġdirectional Ġprice Ġforecasting ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠIn ĠB IB REF 26 Ġ, Ġheadline Ġnews Ġare Ġprocessed Ġusing ĠStanford ĠOpen IE Ġto Ġgenerate Ġtri ples Ġthat Ġare Ġfed Ġinto Ġa ĠNeural ĠT ensor ĠNetwork Ġto Ġcreate Ġthe Ġfinal Ġheadline Ġrepresentation . ĠIn ĠB IB REF 27 Ġ, Ġa Ġcharacter - level Ġembed ding Ġis Ġpre - trained Ġin Ġan Ġun super vised Ġmanner . ĠThe Ġcharacter Ġembed ding Ġis Ġused Ġas Ġinput Ġto Ġa Ġsequence Ġmodel Ġto Ġlearn Ġthe Ġheadline Ġrepresentation . ĠParticularly , Ġboth Ġworks Ġaverage Ġall Ġheadline Ġrepresentations Ġin Ġa Ġgiven Ġday , Ġrather Ġthan Ġattempting Ġto Ġweight Ġthe Ġmost Ġrelevant Ġones . ĠIn Ġthis Ġwork , Ġwe Ġpropose Ġa Ġneural Ġattention Ġmechanism Ġto Ġcapture Ġthe ĠNews ĠRe lev ance Ġand Ġprovide Ġexperimental Ġevidence Ġthat Ġit Ġis Ġa Ġkey Ġcomponent Ġof Ġthe Ġend - to - end Ġlearning Ġprocess . ĠOur Ġattention Ġextends Ġthe Ġprevious Ġdeep Ġlearning Ġmethods Ġfrom ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. </s> Despite Ġthe Ġfact Ġthat Ġend - to - end Ġdeep Ġlearning Ġmodels Ġhave Ġattained Ġstate - of - the - art Ġperformance , Ġthe Ġlarge Ġnumber Ġof Ġparameters Ġmake Ġthem Ġprone Ġto Ġover fitting . ĠAdditionally , Ġend - to - end Ġmodels Ġare Ġtrained Ġfrom Ġscratch Ġrequiring Ġlarge Ġdatasets Ġand Ġcomputational Ġresources . ĠTransfer Ġlearning Ġ( TL ) Ġallev iates Ġthis Ġproblem Ġby Ġadapting Ġrepresentations Ġlearnt Ġfrom Ġa Ġdifferent Ġand Ġpotentially Ġweak ly Ġrelated Ġsource Ġdomain Ġto Ġthe Ġnew Ġtarget Ġdomain . ĠFor Ġexample , Ġin Ġcomputer Ġvision Ġtasks Ġthe Ġconv olution al Ġfeatures Ġlearnt Ġfrom ĠImage Net ĠB IB REF 28 Ġdataset Ġ( source Ġdomain ) Ġhave Ġbeen Ġsuccessfully Ġtransferred Ġto Ġmultiple Ġdomain Ġtarget Ġtasks Ġwith Ġmuch Ġsmaller Ġdatasets Ġsuch Ġas Ġobject Ġclassification Ġand Ġscene Ġrecognition ĠB IB REF 29 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġconsider ĠTL Ġin Ġour Ġexperiments Ġfor Ġtwo Ġmain Ġreasons . ĠFirst , Ġit Ġaddress Ġthe Ġquestion Ġwhether Ġour Ġproposed Ġdataset Ġis Ġsuitable Ġfor Ġend - to - end Ġtraining Ġsince Ġthe Ġperformance Ġof Ġthe Ġtransferred Ġrepresentations Ġcan Ġbe Ġcompared Ġwith Ġend - to - end Ġlearning . ĠSecond , Ġit Ġis Ġstill Ġto Ġbe Ġinvestigated Ġwhich Ġdataset Ġtransfers Ġbetter Ġto Ġthe Ġforecasting Ġproblem . ĠRecently , Ġthe ĠN LP Ġcommunity Ġhas Ġfocused Ġon Ġuniversal Ġrepresentations Ġof Ġsentences ĠB IB REF 17 Ġ, ĠB IB REF 19 Ġ, Ġwhich Ġare Ġdense Ġrepresentations Ġthat Ġcarry Ġthe Ġmeaning Ġof Ġa Ġfull Ġsentence . ĠB IB REF 17 Ġfound Ġthat Ġtransferring Ġthe Ġsentence Ġrepresentation Ġtrained Ġon Ġthe ĠStanford ĠNatural ĠLanguage ĠIn ference Ġ( SN LI ) ĠB IB REF 30 Ġdataset Ġachieves Ġstate - of - the - art Ġsentence Ġrepresentations Ġto Ġmultiple ĠN LP Ġtasks Ġ( e . g . Ġsentiment Ġanalysis , Ġquestion - type Ġand Ġopinion Ġpol arity ). ĠFollowing ĠB IB REF 17 Ġ, Ġin Ġthis Ġwork , Ġwe Ġinvestigate Ġthe Ġsuit ability Ġof ĠSN LI Ġand ĠReuters ĠRC V 1 ĠB IB REF 31 Ġdatasets Ġto Ġtransfer Ġlearning Ġto Ġthe Ġvolatility Ġforecasting Ġtask . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthe Ġhierarchical Ġattention Ġmechanism Ġat Ġheadline Ġlevel , Ġproposed Ġin Ġour Ġwork , Ġhas Ġnot Ġbeing Ġapplied Ġto Ġvolatility Ġprediction Ġso Ġfar ; Ġneither Ġhas Ġbeen Ġinvestigated Ġthe Ġability Ġto Ġtransfer Ġsentence Ġenc od ers Ġfrom Ġsource Ġdatasets Ġto Ġthe Ġtarget Ġforecasting Ġproblem Ġ( Transfer ĠLearning ). </s> Our Ġdataset </s> Our Ġcorpus Ġcovers Ġa Ġbroad Ġrange Ġof Ġnews Ġincluding Ġnews Ġaround Ġearnings Ġdates Ġand Ġcomple ments Ġthe Ġ10 - K Ġreports Ġcontent . ĠAs Ġan Ġillustration , Ġthe Ġheadlines ĠâĢ ľ Wal mart Ġwarns Ġthat Ġstrong ĠU . S . Ġdollar Ġwill Ġcost Ġ$ 15 B Ġin Ġsales âĢ Ŀ Ġand ĠâĢ ľ Pro ct er Ġ& ĠGamble ĠCo Ġraises ĠFY Ġorganic Ġsales Ġgrowth Ġforecast Ġafter Ġsales Ġbeat âĢ Ŀ Ġdescribe Ġthe Ġcompany Ġfinancial Ġconditions Ġand Ġperformance Ġfrom Ġthe Ġmanagement Ġpoint Ġof Ġview ĠâĢĵ Ġthese Ġare Ġalso Ġtypical Ġcontent Ġpresent Ġin ĠSection Ġ7 Ġof Ġthe Ġ10 - K Ġreports . </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġsteps Ġinvolved Ġin Ġcompiling Ġour Ġdataset Ġof Ġfinancial Ġnews Ġat Ġstock Ġlevel , Ġwhich Ġcomprises Ġa Ġbroad Ġrange Ġof Ġbusiness Ġsectors . </s> S ect ors Ġand Ġstocks </s> The Ġfirst Ġstep Ġin Ġcompiling Ġour Ġcorpus Ġwas Ġto Ġchoose Ġthe Ġconstituents Ġstocks . ĠOur Ġgoal Ġwas Ġto Ġconsider Ġstocks Ġin Ġa Ġbroad Ġrange Ġof Ġsectors , Ġaiming Ġa Ġdivers ified Ġfinancial Ġdomain Ġcorpus . ĠWe Ġfound Ġthat ĠExchange ĠTr aded ĠFunds Ġ( ETF ) Ġprovide Ġa Ġmechanical Ġway Ġto Ġaggregate Ġthe Ġmost Ġrelevant Ġstocks Ġin Ġa Ġgiven Ġindustry / sector . ĠAn ĠETF Ġis Ġa Ġfund Ġthat Ġowns Ġassets , Ġe . g . Ġstock Ġshares Ġor Ġcurrencies , Ġbut , Ġunlike Ġmutual Ġfunds Ġare Ġtraded Ġin Ġstock Ġexchanges . ĠThese ĠETF s Ġare Ġextremely Ġliquid Ġand Ġtrack Ġdifferent Ġinvestment Ġthemes . ĠWe Ġdecided Ġto Ġuse ĠSPD R ĠSet cor ĠFunds Ġconstituents Ġstocks Ġin Ġour Ġwork Ġsince Ġthe Ġcompany Ġis Ġthe Ġlargest Ġprovider Ġof Ġsector Ġfunds Ġin Ġthe ĠUnited ĠStates . ĠWe Ġincluded Ġin Ġour Ġanalysis Ġthe Ġtop Ġ5 Ġ( five ) Ġsector ĠETF s Ġby Ġfinancial Ġtrading Ġvolume Ġ( as Ġin ĠJan / 2018 ). ĠAmong Ġthe Ġmost Ġtraded Ġsectors Ġwe Ġalso Ġfiltered Ġout Ġthe Ġsectors Ġthat Ġwere Ġsimilar Ġto Ġeach Ġother . ĠFor Ġexample , Ġthe ĠConsumer ĠStaples Ġand ĠConsumer ĠDisc retion ary Ġsectors Ġare Ġboth Ġpart Ġof Ġthe Ġparent ĠConsumer Ġcategory . ĠFor Ġeach Ġof Ġthe Ġtop Ġ5 Ġsectors Ġwe Ġselected Ġthe Ġtop Ġ10 Ġholdings , Ġwhich Ġare Ġdeemed Ġthe Ġmost Ġrelevant Ġstocks . Ġt bl : stock un iverse , Ġdetails Ġour Ġdataset Ġsectors Ġand Ġits Ġrespective Ġstocks . </s> Stock Ġspecific Ġdata </s> We Ġassume Ġthat Ġan Ġindividual Ġstock Ġnews Ġas Ġthe Ġone Ġthat Ġexplicitly Ġmention Ġthe Ġstock Ġname Ġor Ġany Ġof Ġits Ġsurface Ġforms Ġin Ġthe Ġheadline . ĠAs Ġan Ġillustration , Ġin Ġorder Ġto Ġcollect Ġall Ġnews Ġfor Ġthe Ġstock Ġcode ĠPG , ĠPro ct er Ġ& ĠGamble Ġcompany Ġname , Ġwe Ġsearch Ġall Ġthe Ġheadlines Ġwith Ġany Ġof Ġthese Ġwords : ĠPro ct er & Gam ble ĠOR ĠPro ct er Ġand ĠGamble ĠOR ĠP & G . ĠIn Ġthis Ġexample , Ġthe Ġfirst Ġword Ġis Ġjust Ġthe Ġcompany Ġname Ġand Ġthe Ġremaining Ġwords Ġare Ġthe Ġcompany Ġsurface Ġforms . </s> We Ġautomatically Ġderived Ġthe Ġsurface Ġforms Ġfor Ġeach Ġstock Ġby Ġstarting Ġwith Ġa Ġseed Ġof Ġsurface Ġforms Ġextracted Ġfrom Ġthe ĠDB pedia ĠKnowledge ĠBase Ġ( KB ). ĠWe Ġthen Ġapplied Ġthe Ġfollowing Ġprocedure : </s> Rel ate Ġeach Ġcompany Ġname Ġwith Ġthe ĠKB Ġentity Ġunique Ġidentifier . </s> Ret rieve Ġall Ġvalues Ġof Ġthe Ġwiki Page Red irect s Ġproperty . ĠThe Ġproperty Ġholds Ġthe Ġnames Ġof Ġdifferent Ġpages Ġthat Ġpoints Ġto Ġthe Ġsame Ġentity / company Ġname . ĠThis Ġstep Ġsets Ġthe Ġinitial Ġseed Ġof Ġsurface Ġforms . </s> Man ually , Ġfilter Ġout Ġsome Ġnoisy Ġproperty Ġvalues . ĠFor Ġinstance , Ġfrom Ġthe ĠPro ct er Ġ& ĠG lam ble Ġentity Ġpage Ġwe Ġwere Ġable Ġto Ġautomatically Ġextract Ġd br : Pro ct er _ and _ gam ble Ġand Ġd br : P _ & _ G , Ġbut Ġhad Ġto Ġmanually Ġexclude Ġthe Ġnoisy Ġassociations Ġd br : Female _ p ads Ġand Ġd br : California _ Natural . </s> The Ġresult Ġof Ġthe Ġsteps Ġabove Ġis Ġa Ġdictionary Ġof Ġsurface Ġforms Ġ$ wd _{ sc } $ Ġ. </s> Stock Ġheadlines </s> Our Ġcorpus Ġis Ġbuilt Ġat Ġstock Ġcode Ġlevel Ġby Ġcollecting Ġheadlines Ġfrom Ġthe ĠReuters ĠArchive . ĠThis Ġarchive Ġgroups Ġthe Ġheadlines Ġby Ġdate , Ġstarting Ġfrom Ġ1 ĠJanuary Ġ2007 . ĠEach Ġheadline Ġis Ġa Ġhtml Ġlink Ġ(< a Ġhref > Ġtag ) Ġto Ġthe Ġfull Ġbody Ġof Ġthe Ġnews , Ġwhere Ġthe Ġanchor Ġtext Ġis Ġthe Ġheadline Ġcontent Ġfollowed Ġby Ġthe Ġrelease Ġtime . ĠFor Ġexample , Ġthe Ġpage Ġdated Ġ16 ĠDec Ġ2016 Ġhas Ġthe Ġheadline ĠâĢ ľ Pro ct er Ġ& ĠGamble Ġappoint s ĠNelson ĠP elt z Ġto Ġboard Ġ5 : 26 PM ĠUTC âĢ Ŀ . </s> For Ġeach Ġof Ġthe Ġ50 Ġstocks Ġ( 5 Ġsectors Ġtimes Ġ10 Ġstocks Ġper Ġsector ) Ġselected Ġusing Ġthe Ġcriteria Ġdescribed Ġin Ġsub : cor p us sec stock , Ġwe Ġretrieved Ġall Ġthe Ġheadlines Ġfrom Ġthe ĠReuters ĠArchive Ġraging Ġfrom Ġ01 / 01 / 2007 Ġto Ġ30 / 12 / 2017 . ĠThis Ġprocess Ġtakes Ġthe Ġfollowing Ġsteps : </s> For Ġa Ġgiven Ġstock Ġcode Ġ( Ġ$ sc $ Ġ) Ġretrieve Ġall Ġsurface Ġforms Ġ$ wd _{ sc } $ Ġ. </s> For Ġeach Ġday , Ġstore Ġonly Ġthe Ġheadlines Ġcontent Ġmatching Ġany Ġword Ġin Ġ$ wd _{ sc } $ Ġ. ĠFor Ġeach Ġstored Ġheadline Ġwe Ġalso Ġstore Ġthe Ġtime Ġand Ġtime zone . </s> Con vert Ġthe Ġnews Ġdate Ġand Ġtime Ġto ĠEastern ĠDaylight ĠTime Ġ( ED T ). </s> C ategor ize Ġthe Ġnews Ġrelease Ġtime . ĠWe Ġconsider Ġthe Ġfollowing Ġcategory Ġset : Ġ{ before Ġmarket , Ġduring Ġmarket Ġ, Ġafter Ġmarket , Ġholidays , Ġweekends }. Ġduring Ġmarket Ġcontains Ġnews Ġbetween Ġ9 : 30 AM Ġand Ġ4 : 00 PM . Ġbefore Ġmarket Ġbefore Ġ9 : 30 AM Ġand Ġafter Ġmarket Ġafter Ġ4 : 00 PM . </s> The Ġtime Ġcategories Ġprevents Ġany Ġmis al ignment Ġbetween Ġtext Ġand Ġstock Ġprice Ġdata . ĠMoreover , Ġit Ġprevents Ġdata Ġleakage Ġand , Ġconsequently , Ġunrealistic Ġpredictive Ġmodel Ġperformance . ĠIn Ġgeneral , Ġnews Ġreleased Ġafter Ġ4 : 00 PM ĠEDT Ġcan Ġdrastically Ġchange Ġmarket Ġexpectations Ġand Ġthe Ġreturns Ġcalculated Ġusing Ġclose Ġto Ġclose Ġprices Ġas Ġin Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel Ġ( see Ġeq : cl osing return ). ĠFollowing ĠB IB REF 3 Ġ, Ġto Ġdeal Ġwith Ġnews Ġmis al ignment , Ġnews Ġissued Ġafter Ġ4 : 00 PM Ġ( after Ġmarket ) Ġare Ġgrouped Ġwith Ġthe Ġpre - market Ġ( before Ġmarket ) Ġon Ġthe Ġfollowing Ġtrading Ġday . </s> t bl : stock time cat Ġshows Ġthe Ġdistribution Ġof Ġnews Ġper Ġsector Ġfor Ġeach Ġtime Ġcategory . ĠWe Ġcan Ġsee Ġa Ġhigh Ġconcentration Ġof Ġnews Ġreleased Ġbefore Ġthe Ġmarket Ġopens Ġ( 55 % Ġon Ġaverage ). ĠIn Ġcontrast , Ġusing Ġa Ġcorpus Ġcompiled Ġfrom Ġmessage Ġboards , Ġa Ġlarge Ġoccurrence Ġof Ġnews Ġduring Ġmarket Ġhours Ġwas Ġfound ĠB IB REF 3 Ġ. ĠThis Ġbehaviour Ġindicating Ġday Ġtraders ' Ġactivity . ĠOur Ġcorpus Ġcomprise Ġfinancial Ġnews Ġagency Ġheadlines , Ġa Ġcontent Ġmore Ġfocused Ġon Ġcorporate Ġevents Ġ( e . g . Ġlawsuits , Ġmer ges Ġ& Ġacquisitions , Ġresearch Ġ& Ġdevelopment ) Ġand Ġon Ġeconomic Ġnews Ġ( see Ġt bl : stock head lines ex ma ples Ġfor Ġa Ġsample Ġof Ġour Ġdataset ). ĠThese Ġheadlines Ġare Ġmostly Ġfactual . ĠOn Ġthe Ġother Ġhand , Ġuser - generated Ġcontent Ġsuch Ġas ĠTwitter Ġand Ġmessage Ġboards Ġ( as Ġin ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ) Ġtends Ġto Ġbe Ġmore Ġsubjective . </s> U . S . Ġmacro economic Ġindicators Ġsuch Ġas ĠRetail ĠSales , ĠJob less ĠClaims Ġand ĠGDP Ġare Ġmostly Ġreleased Ġaround Ġ8 : 30 AM Ġ( one Ġhour Ġbefore Ġthe Ġmarket Ġopens ). ĠThese Ġnumbers Ġare Ġkey Ġdrivers Ġof Ġmarket Ġactivity Ġand , Ġas Ġsuch , Ġhave Ġa Ġhigh Ġmedia Ġcoverage . ĠSpecific Ġsections Ġof Ġthese Ġeconomic Ġreports Ġimpact Ġseveral Ġstocks Ġand Ġsectors . ĠAnother Ġfactor Ġthat Ġcontribute Ġto Ġthe Ġhigh Ġactivity Ġof Ġnews Ġoutside Ġregular Ġtrading Ġhours Ġare Ġcompany Ġearnings Ġreports . ĠThese Ġare Ġrarely Ġreleased Ġduring Ġtrading Ġhours . ĠFinally , Ġbefore Ġthe Ġmarket Ġopens Ġnews Ġagencies Ġprovide Ġa Ġsummary Ġof Ġthe Ġinternational Ġmarkets Ġdevelopments , Ġe . g . Ġthe Ġkey Ġfacts Ġduring Ġthe ĠAsian Ġand ĠAustralian Ġtrading Ġhours . ĠAll Ġthese Ġfactors Ġcontribute Ġto Ġthe Ġhigh Ġconcentration Ġof Ġpre - market Ġnews . </s> Background </s> We Ġstart Ġthis Ġsection Ġby Ġreviewing Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel , Ġwhich Ġis Ġa Ġstrong Ġbenchmark Ġused Ġto Ġevaluate Ġour Ġneural Ġmodel . ĠWe Ġthen Ġreview Ġthe Ġsource Ġdatasets Ġproposed Ġin Ġthe Ġliterature Ġthat Ġwere Ġtrained Ġindependently Ġand Ġtransf ered Ġto Ġour Ġvolatility Ġprediction Ġmodel . ĠFinally , Ġwe Ġreview Ġthe Ġgeneral Ġarchitectures Ġof Ġsequence Ġmodelling Ġand Ġattention Ġmechanisms . </s> G ARCH Ġmodel </s> Financial Ġinstitutions Ġuse Ġthe Ġconcept Ġof ĠâĢ ľ Value Ġat Ġrisk âĢ Ŀ Ġto Ġmeasure Ġthe Ġexpected Ġvolatility Ġof Ġtheir Ġportfolios . ĠThe Ġwidespread Ġe con ometric Ġmodel Ġfor Ġvolatility Ġforecasting Ġis Ġthe ĠGeneral ized ĠAut ore gressive ĠCond itional ĠH eter os ked astic ity Ġ( G ARCH ) ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ. ĠPrevious Ġresearch Ġshows Ġthat Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel Ġis Ġhard Ġto Ġbeat . ĠFor Ġexample , ĠB IB REF 34 Ġcompared ĠG ARCH ( 1 , 1 ) Ġwith Ġ330 Ġdifferent Ġe con ometric Ġvolatility Ġmodels Ġshowing Ġthat Ġthey Ġare Ġnot Ġsignificantly Ġbetter Ġthan ĠG ARCH ( 1 , 1 ). ĠLet Ġ$ p _ t $ Ġbe Ġthe Ġprice Ġof Ġan Ġstock Ġat Ġthe Ġend Ġof Ġa Ġtrading Ġperiod Ġwith Ġclosing Ġreturns Ġ$ r _ t $ Ġgiven Ġby </s> $$ r _ t Ġ= Ġ\ frac { p _ t }{ p _{ t - 1 }} Ġ- Ġ1 Ġ$$ Ġ Ġ Ġ( E q . Ġ29 ) </s> The ĠG ARCH Ġprocess Ġexplicitly Ġmodels Ġthe Ġtime - v ary ing Ġvolatility Ġof Ġasset Ġreturns . ĠIn Ġthe ĠG ARCH ( 1 , 1 ) Ġspecification Ġthe Ġreturns Ġseries Ġ$ r _ t $ Ġfollow Ġthe Ġprocess : </s> $$ r _ t Ġ& = Ġ\ mu Ġ+ Ġ\ ep sil on Ġ_ t Ġ Ġ\\ Ġ\ ep sil on Ġ_ t Ġ& = Ġ\ s igma Ġ_ t Ġz _ t Ġ Ġ\\ Ġ\ s igma Ġ^ 2 _ t Ġ& = Ġa _ 0 Ġ+ Ġa _ 1 Ġ\ ep sil on Ġ_ { t - 1 } ^ 2 Ġ+ Ġb _ 1 Ġ\ s igma Ġ_ { t - 1 } ^ 2 $$ Ġ Ġ Ġ( E q . Ġ30 ) </s> where Ġ$\ mu Ġ$ Ġis Ġa Ġconstant Ġ( return Ġdrift ) Ġand Ġ$ z _ t $ Ġis Ġa Ġsequence Ġof Ġi . i . d . Ġrandom Ġvariables Ġwith Ġmean Ġzero Ġand Ġunit Ġvariance . ĠIt Ġis Ġworth Ġnoting Ġthat Ġalthough Ġthe Ġconditional Ġmean Ġreturn Ġdescribed Ġin Ġeq : g arch cond mean Ġhas Ġa Ġconstant Ġvalue , Ġthe Ġconditional Ġvolatility Ġ$\ s igma Ġ_ t $ Ġis Ġtime - dependent Ġand Ġmodeled Ġby Ġeq : att . </s> The Ġone - step Ġahead Ġexpected Ġvolatility Ġforecast Ġcan Ġbe Ġcomputed Ġdirectly Ġfrom Ġeq : g arch cond vari ance Ġand Ġis Ġgiven Ġby </s> $$ E _ T [ \ s igma Ġ_ { T + 1 } ^ 2 ] Ġ= Ġa _ 0 Ġ+ Ġa _ 1 ĠE _ T [ \ ep sil on Ġ^ 2 ] Ġ+ Ġb _ 1 ĠE _ T [ \ s igma Ġ_ { T } ^ 2 ] Ġ$$ Ġ Ġ Ġ( E q . Ġ32 ) </s> In Ġgeneral , Ġthe Ġ$ t ^ {\ prime Ġ} $ Ġ- steps Ġahead Ġexpected Ġvolatility Ġ$ E _ T [ \ s igma Ġ_ { T + t ^ {\ prime Ġ}} ^ 2 ] $ Ġcan Ġbe Ġeasily Ġexpressed Ġin Ġterms Ġof Ġthe Ġprevious Ġstep Ġexpected Ġvolatility . ĠIt Ġis Ġeasy Ġto Ġprove Ġby Ġinduction Ġthat Ġthe Ġforecast Ġfor Ġany Ġhorizon Ġcan Ġbe Ġrepresented Ġin Ġterms Ġof Ġthe Ġone - step Ġahead Ġforecast Ġand Ġis Ġgiven Ġby </s> $$ E _ T [ \ s igma Ġ_ { T + t ^ {\ prime Ġ}} ^ 2 ] Ġ- Ġ\ s igma Ġ_ u ^ 2 Ġ= Ġ( a _ 1 Ġ+ Ġb _ 1 ) ^{ ( t ^ {\ prime Ġ} Ġ- 1 )} Ġ\ left ( E _ T [ \ s igma Ġ_ { T + 1 } ^ 2 ] Ġ- Ġ\ s igma Ġ_ u ^ 2 \ right ) $$ Ġ Ġ Ġ( E q . Ġ33 ) </s> where Ġ$\ s igma Ġ_ u $ Ġis Ġthe Ġunconditional Ġvolatility : </s> $$ \ s igma Ġ_ u Ġ= Ġ\ sq rt { a _ 0 Ġ/ Ġ( 1 Ġ- Ġa _ 1 Ġ- Ġb _ 1 )} Ġ$$ Ġ Ġ Ġ( E q . Ġ34 ) </s> From Ġthe Ġequation Ġabove Ġwe Ġcan Ġsee Ġthat Ġfor Ġlong Ġhor izons , Ġi . e . Ġ$ t ^ \ prime Ġ\ right arrow Ġ\ in fty Ġ$ Ġ, Ġthe Ġvolatility Ġforecast Ġin Ġeq : fore cast rec ursive Ġconver ges Ġto Ġthe Ġunconditional Ġvolatility Ġin Ġeq : un var . </s> All Ġthe Ġworks Ġreviewed Ġin Ġsec : introdu ction Ġ( ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ) Ġconsider ĠG ARCH ( 1 , 1 ) Ġbenchmark . ĠHowever , Ġgiven Ġthe Ġlong Ġhorizon Ġof Ġtheir Ġpredictions Ġ( e . g . Ġquarterly Ġor Ġannual ), Ġthe Ġmodels Ġare Ġevaluated Ġusing Ġthe Ġunconditional Ġvolatility Ġ$\ s igma Ġ_ u $ Ġin Ġeq : un var . ĠIn Ġthis Ġwork , Ġwe Ġfocus Ġon Ġthe Ġshort - term Ġvolatility Ġprediction Ġand Ġuse Ġthe ĠG ARCH ( 1 , 1 ) Ġone - day Ġahead Ġconditional Ġvolatility Ġprediction Ġin Ġeq : fore cast one period Ġto Ġevaluate Ġour Ġmodels . </s> Let Ġ$\ s igma Ġ_ { t + 1 } $ Ġdenote Ġthe Ġex - post ĠâĢ ľ true âĢ Ŀ Ġdaily Ġvolatility Ġat Ġa Ġgiven Ġtime Ġ$ t $ Ġ. ĠThe Ġperformance Ġon Ġa Ġset Ġwith Ġ$ N $ Ġdaily Ġsamples Ġcan Ġbe Ġevaluated Ġusing Ġthe Ġstandard ĠMean ĠSqu ared ĠError Ġ( Ġ$ M SE $ Ġ) Ġand ĠMean ĠAbsolute ĠError Ġ( Ġ$ MA E $ Ġ) </s> $$ M SE Ġ& = Ġ\ frac { 1 }{ N } Ġ\ sum Ġ_ { t = 1 } ^{ N } Ġ\ left ( ĠE _ t [ \ s igma Ġ_ { t + 1 } ] Ġ- Ġ\ s igma Ġ_ { t + 1 }\ right ) ^ 2 Ġ Ġ\\ ĠMA E Ġ& = Ġ\ frac { 1 }{ N } Ġ\ sum Ġ_ { t = 1 } ^{ N }\ left | E _ t [ \ s igma Ġ_ { t + 1 } ] Ġ- Ġ\ s igma Ġ_ { t + 1 } Ġ\ right | $$ Ġ Ġ Ġ( E q . Ġ36 ) </s> Additionally , Ġfollowing ĠB IB REF 35 Ġ, Ġthe Ġmodels Ġare Ġalso Ġevaluated Ġusing Ġthe Ġcoefficient Ġof Ġdetermination Ġ$ R ^ 2 $ Ġof Ġthe Ġregression </s> $$ \ s igma Ġ_ { t + 1 } Ġ= Ġa Ġ+ Ġb ĠE _ t [ \ s igma Ġ_ { t + 1 } ] Ġ+ Ġe _ t $$ Ġ Ġ Ġ( E q . Ġ37 ) </s> where </s> $$ R ^ 2 Ġ= Ġ1 Ġ- Ġ\ frac {\ sum Ġ_ { t = 1 } ^{ N } e ^{ 2 } _{ t }} {\ sum Ġ_ { t = 1 } ^{ N }\ left ( E _ t [ \ s igma Ġ_ { t + 1 } ] Ġ- Ġ\ frac { 1 }{ N } Ġ\ sum Ġ_ { t = 1 } ^{ N } E _ t [ \ s igma Ġ_ { t + 1 } ] \ right ) ^{ 2 }} $$ Ġ Ġ Ġ( E q . Ġ38 ) </s> One Ġof Ġthe Ġchallenges Ġin Ġevaluating ĠG ARCH Ġmodels Ġis Ġthe Ġfact Ġthat Ġthe Ġex - post Ġvolatility Ġ$\ s igma Ġ_ { t + 1 } $ Ġis Ġnot Ġdirectly Ġobserved . ĠApparently , Ġthe Ġsquared Ġdaily Ġreturns Ġ$ r _{ t + 1 } ^{ 2 } $ Ġin Ġeq : cl osing return Ġcould Ġstand Ġas Ġa Ġgood Ġproxy Ġfor Ġthe Ġex - post Ġvolatility . ĠHowever , Ġthe Ġsquared Ġreturns Ġyield Ġvery Ġnoisy Ġmeasurements . ĠThis Ġis Ġa Ġdirect Ġconsequence Ġof Ġthe Ġterm Ġ$ z ^ t $ Ġthat Ġconnects Ġthe Ġsquared Ġreturn Ġto Ġthe Ġlatent Ġvolatility Ġfactor Ġin Ġeq : g arch wh it eno ise . ĠThe Ġuse Ġof Ġint rad ay Ġprices Ġto Ġestimate Ġthe Ġex - post Ġdaily Ġvol ay ility Ġwas Ġfirst Ġproposed Ġin ĠB IB REF 35 Ġ. ĠThey Ġargue Ġthat Ġvolatility Ġestim ators Ġusing Ġint rad ay Ġprices Ġis Ġthe Ġproper Ġway Ġto Ġevaluate Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel , Ġas Ġopposed Ġto Ġsquared Ġdaily Ġreturns . ĠFor Ġexample , Ġconsidering Ġthe ĠDeutsche ĠMark Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel Ġ$ R ^ 2 $ Ġimproves Ġfrom Ġ$ 0 . 047 $ Ġ( squ ared Ġreturns ) Ġto Ġ$ 0 . 33 $ Ġ( int rad ay Ġreturns ) ĠB IB REF 35 Ġ. </s> It Ġis Ġclear Ġfrom Ġthe Ġprevious Ġsection Ġthat Ġany Ġvolatility Ġmodel Ġevaluation Ġusing Ġthe Ġnoisy Ġsquared Ġreturns Ġas Ġthe Ġex - post Ġvolatility Ġproxy Ġwill Ġlead Ġto Ġvery Ġpoor Ġperformance . ĠTherefore , Ġhigh - frequency Ġint rad ay Ġdata Ġis Ġfundamental Ġto Ġshort - term Ġvolatility Ġperformance Ġevaluation . ĠHowever , Ġint rad ay Ġdata Ġis Ġdifficult Ġto Ġacquire Ġand Ġcostly . ĠFortunately , Ġthere Ġare Ġstatistically Ġefficient Ġdaily Ġvolatility Ġestim ators Ġthat Ġonly Ġdepend Ġon Ġthe Ġopen , Ġhigh , Ġlow Ġand Ġclose Ġprices . ĠThese Ġprice ĠâĢ ľ r anges âĢ Ŀ Ġare Ġwidely Ġavailable . ĠIn Ġthis Ġsection , Ġwe Ġdiscuss Ġthese Ġestim ators . </s> Let Ġ$ O _ t $ Ġ, Ġ$ H _ t $ Ġ, Ġ$ L _ t $ Ġ, Ġ$ C _ t $ Ġbe Ġthe Ġopen , Ġhigh , Ġlow Ġand Ġclose Ġprices Ġof Ġan Ġasset Ġin Ġa Ġgiven Ġday Ġ$ t $ Ġ. ĠAssuming Ġthat Ġthe Ġdaily Ġprice Ġfollows Ġa Ġgeometric ĠBrown ian Ġmotion Ġwith Ġzero Ġdrift Ġand Ġconstant Ġdaily Ġvolatility Ġ$\ s igma Ġ$ Ġ, ĠParkinson Ġ( 1980 ) Ġderived Ġthe Ġfirst Ġdaily Ġvolatility Ġestim ator </s> $$ \ wide hat {\ s igma Ġ_ { PK , t } ^ 2 } Ġ= Ġ\ frac {\ ln Ġ\ left (\ frac { H _ t }{ L _ t }\ right ) ^ 2 }{ 4 \ ln Ġ( 2 )} Ġ$$ Ġ Ġ Ġ( E q . Ġ41 ) </s> which Ġrepresents Ġthe Ġdaily Ġvolatility Ġin Ġterms Ġof Ġits Ġprice Ġrange . ĠHence , Ġit Ġcontains Ġinformation Ġabout Ġthe Ġprice Ġpath . ĠGiven Ġthis Ġproperty , Ġit Ġis Ġexpected Ġthat Ġ$\ s igma Ġ_ { PK } $ Ġis Ġless Ġnoisy Ġthan Ġthe Ġvolatility Ġcalculated Ġusing Ġsquared Ġreturns . ĠThe ĠParkinson 's Ġvolatility Ġestim ator Ġwas Ġextended Ġby ĠGar man - K lass Ġ( 1980 ) Ġwhich Ġincorporates Ġadditional Ġinformation Ġabout Ġthe Ġopening Ġ( Ġ$ O _ t $ Ġ) Ġand Ġclosing Ġ( Ġ$ C _ t $ Ġ) Ġprices Ġand Ġis Ġdefined Ġas </s> $$ \ wide hat {\ s igma Ġ_ { G K , t } ^{ 2 }} Ġ= Ġ\ frac { 1 }{ 2 } Ġ\ ln Ġ\ left (\ frac { H _ t }{ L _ t }\ right ) ^ 2 Ġ- Ġ( 2 \ ln Ġ( 2 ) Ġ- Ġ1 ) Ġ\ ln Ġ\ left (\ frac { C _ t }{ O _ t }\ right ) ^ 2 Ġ$$ Ġ Ġ Ġ( E q . Ġ42 ) </s> The Ġrelative Ġnoisy Ġof Ġdifferent Ġestim ators Ġ$\ hat {\ s igma Ġ} $ Ġcan Ġbe Ġmeasured Ġin Ġterms Ġof Ġits Ġrelative Ġefficiency Ġto Ġthe Ġdaily Ġvolatility Ġ$\ s igma Ġ$ Ġand Ġis Ġdefined Ġas </s> $$ e \ left (\ wide hat {\ s igma Ġ^ { 2 } }, Ġ\ s igma Ġ^ 2 \ right ) Ġ\ equ iv Ġ\ frac { Var [ \ s igma Ġ^ 2 ] }{ Var [ \ wide hat {\ s igma Ġ^ { 2 }} ]} $$ Ġ Ġ Ġ( E q . Ġ43 ) </s> where Ġ$ Var [ \ cd ot Ġ] $ Ġis Ġthe Ġvariance Ġoperator . ĠIt Ġfollows Ġdirectly Ġfrom Ġeq : g arch wh it eno ise Ġthat Ġthe Ġsquared Ġreturn Ġhas Ġefficiency Ġ1 Ġand Ġtherefore , Ġvery Ġnoisy . ĠB IB REF 36 Ġreports ĠParkinson Ġ( Ġ$\ wide hat {\ s igma Ġ_ { PK , t } ^ 2 } $ Ġ) Ġvolatility Ġestim ator Ġhas Ġ4 . 9 Ġrelative Ġefficiency Ġand ĠGar man - K lass Ġ( Ġ$\ wide hat {\ s igma Ġ_ { G K , t } ^ 2 } $ Ġ) Ġ7 . 4 . ĠAdditionally , Ġall Ġthe Ġdescribed Ġestim ators Ġare Ġunbiased . </s> Many Ġalternative Ġestim ators Ġto Ġdaily Ġvolatility Ġhave Ġbeen Ġproposed Ġin Ġthe Ġliterature . ĠHowever , Ġexperiments Ġin ĠB IB REF 36 Ġrate Ġthe ĠGar man - K lass Ġvolatility Ġestim ator Ġas Ġthe Ġbest Ġvolatility Ġestim ator Ġbased Ġonly Ġon Ġopen , Ġhigh , Ġlow Ġand Ġclose Ġprices . ĠIn Ġthis Ġwork , Ġwe Ġtrain Ġour Ġmodels Ġto Ġpredict Ġthe Ġstate - of - the - art ĠGar man - K lass Ġestim ator . ĠMoreover , Ġwe Ġevaluate Ġour Ġmodels Ġand ĠG ARCH ( 1 , 1 ) Ġusing Ġthe Ġmetrics Ġdescribed Ġin Ġsub : eval ution , Ġbut Ġwith Ġthe Ġappropriate Ġvolatility Ġproxies , Ġi . e . ĠParkinson Ġand ĠGar man - K lass Ġestim ators . </s> Transfer ĠLearning Ġfrom Ġother Ġsource Ġdomains </s> Vector Ġrepresentations Ġof Ġwords , Ġalso Ġknown Ġas ĠWord Ġembed d ings ĠB IB REF 21 Ġ, ĠB IB REF 37 Ġ, Ġthat Ġrepresent Ġa Ġword Ġas Ġa Ġdense Ġvector Ġhas Ġbecome Ġthe Ġstandard Ġbuilding Ġblocks Ġof Ġalmost Ġall ĠN LP Ġtasks . ĠThese Ġembed d ings Ġare Ġtrained Ġon Ġlarge Ġunl abel ed Ġcorpus Ġand Ġare Ġable Ġto Ġcapture Ġcontext Ġand Ġsimilarity Ġamong Ġwords . </s> Some Ġattempts Ġhave Ġbeen Ġmade Ġto Ġlearn Ġvector Ġrepresentations Ġof Ġa Ġfull Ġsentence , Ġrather Ġthan Ġonly Ġa Ġsingle Ġword , Ġusing Ġun super vised Ġapproaches Ġsimilar Ġin Ġnature Ġto Ġword Ġembed d ings . ĠRecently , ĠB IB REF 17 Ġshowed Ġstate - of - the - art Ġperformance Ġwhen Ġa Ġsentence Ġenc oder Ġis Ġtrained Ġend - to - end Ġon Ġa Ġsupervised Ġsource Ġtask Ġand Ġtransferred Ġto Ġother Ġtarget Ġtasks . ĠInspired Ġby Ġthis Ġwork , Ġwe Ġinvestigate Ġthe Ġperformance Ġof Ġsentence Ġenc od ers Ġtrained Ġon Ġthe ĠText Ġcategor ization Ġand ĠNatural ĠLanguage ĠIn ference Ġ( N LI ) Ġtasks Ġand Ġuse Ġthese Ġenc od ers Ġin Ġour Ġmain Ġshort - term Ġvolatility Ġprediction Ġtask . </s> A Ġgeneric Ġsentence Ġenc oder Ġ$ S _ e $ Ġreceives Ġthe Ġsentence Ġwords Ġas Ġinput Ġand Ġreturns Ġa Ġvector Ġrepresenting Ġthe Ġsentence . ĠThis Ġcan Ġbe Ġexpressed Ġas Ġa Ġmapping </s> $$ S _ e Ġ\ col on Ġ\ math bb Ġ{ R } ^{ T ^{ S } Ġ\ times Ġd _ w } Ġ\ right arrow Ġ\ math bb Ġ{ R } ^{ d _ S } $$ Ġ Ġ Ġ( E q . Ġ45 ) </s> from Ġa Ġvariable Ġsize Ġsequence Ġof Ġwords Ġto Ġa Ġsentence Ġvector Ġ$ S $ Ġof Ġfixed - size Ġ$ d _ S $ Ġ, Ġwhere Ġ$ T ^{ S } $ Ġis Ġthe Ġsentence Ġnumber Ġof Ġwords Ġand Ġ$ d _ w $ Ġis Ġthe Ġpre - trained Ġword Ġembed ding Ġdimension . </s> In Ġthe Ġfollowing Ġsections , Ġwe Ġdescribe Ġthe Ġdatasets Ġand Ġarchitectures Ġto Ġtrain Ġthe Ġsentence Ġenc od ers Ġof Ġthe Ġauxiliary Ġtransfer Ġlearning Ġtasks . </s> The ĠReuters ĠCorpus ĠVolume ĠI Ġ( RC V 1 ) Ġis Ġcorpus Ġcontaining Ġ8 06 , 79 1 Ġnews Ġarticles Ġin Ġthe ĠEnglish Ġlanguage Ġcollected Ġfrom Ġ20 / 08 / 1996 Ġto Ġ19 / 08 / 1997 ĠB IB REF 31 Ġ. ĠThe Ġtopic Ġof Ġeach Ġnews Ġwas Ġhuman - annot ated Ġusing Ġa Ġhierarchical Ġstructure . ĠAt Ġthe Ġtop Ġof Ġthe Ġhierarchy , Ġlies Ġthe Ġcoarse - gr ained Ġcategories : ĠCC AT Ġ( Cor porate ), ĠEC AT Ġ( Econom ics ), ĠGC AT Ġ( Government ), Ġand ĠMC AT Ġ( Mark ets ). ĠA Ġnews Ġarticle Ġcan Ġbe Ġassigned Ġto Ġmore Ġthan Ġone Ġcategory Ġmeaning Ġthat Ġthe Ġtext Ġcategor ization Ġtask Ġis Ġmut il abel . ĠEach Ġnews Ġis Ġstored Ġin Ġa Ġseparate ĠXML Ġfile . Ġl st : rc v 1 x m lex ample Ġshows Ġthe Ġtypical Ġstructure Ġof Ġan Ġarticle . </s> <? xml Ġversion =" 1 . 0 " Ġencoding =" iso - 88 59 - 1 " Ġ? > </s> < news item Ġitem id =" 6 159 " Ġid =" root " Ġdate =" 1996 - 08 - 21 " Ġxml : lang =" en "> </s> < head line > Col omb ia Ġraises Ġinternal Ġcoffee Ġprice .</ head line > </s> < date line > B OG OTA Ġ1996 - 08 - 21 </ date line > </s> < cop yright >( c ) ĠReuters ĠLimited Ġ1996 </ cop yright > </s> < metadata > </s> < codes Ġclass =" b ip : top ics : 1 . 0 "> </s> < code Ġcode =" C 13 "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" C 31 "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" CC AT "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" M 14 "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" M 141 "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" MC AT "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> </ codes > </s> </ metadata > </s> </ news item > </s> The ĠRC V 1 Ġdataset Ġis Ġnot Ġreleased Ġwith Ġa Ġstandard Ġtrain , Ġvalidation , Ġtest Ġsplit . ĠIn Ġthis Ġwork , Ġwe Ġseparated Ġ15 % Ġof Ġsamples Ġas Ġa Ġtest Ġset Ġfor Ġevaluation Ġpurposes . ĠThe Ġremaining Ġsamples Ġwere Ġfurther Ġsplit Ġleaving Ġ70 % Ġand Ġ15 % Ġfor Ġtraining Ġand Ġvalidation , Ġrespectively . </s> Regarding Ġthe Ġcategories Ġdistribution , Ġwe Ġfound Ġthat , Ġfrom Ġthe Ġoriginal Ġ126 Ġcategories , Ġ23 Ġcategories Ġwere Ġnever Ġassigned Ġto Ġany Ġnews ; Ġtherefore , Ġwere Ġdisreg arded . ĠFrom Ġthe Ġ103 Ġclasses Ġleft Ġwe Ġfound Ġa Ġhigh Ġimbalance Ġamong Ġthe Ġlabels Ġwith Ġa Ġlarge Ġnumber Ġof Ġunder represented Ġcategories Ġhaving Ġless Ġthan Ġ12 Ġsamples . ĠThe Ġvery Ġlow Ġnumber Ġof Ġsamples Ġfor Ġthese Ġminority Ġclasses Ġbrings Ġa Ġgreat Ġchallenge Ġto Ġdiscriminate Ġthe Ġvery Ġfine - gr ained Ġcategories . ĠAim ing Ġto Ġalleviate Ġthis Ġproblem , Ġwe Ġgrouped Ġinto Ġa Ġsame Ġclass Ġall Ġcategories Ġbelow Ġthe Ġsecond Ġhierarchical Ġlevel . ĠFor Ġexample , Ġgiven Ġthe Ġroot Ġnode ĠCC AT Ġ( Cor porate ) Ġwe Ġgrouped ĠC 151 Ġ( ACC OUN TS / EAR NING S ), ĠC 15 11 Ġ( ANN UAL ĠRES ULTS ) Ġand ĠC 152 Ġ( COM MENT / FOR EC AS TS ) Ġinto Ġthe Ġdirect Ġchild Ġnode ĠC 15 Ġ( PER FOR MAN CE ). ĠUsing Ġthis Ġprocedure Ġthe Ġoriginal Ġ103 Ġcategories Ġwhere Ġreduced Ġto Ġ55 . ĠOne Ġof Ġthe Ġbenefits Ġof Ġthis Ġprocedure Ġwas Ġthat Ġthe Ġless Ġrepresented Ġclasses Ġend Ġup Ġhaving Ġaround Ġthousand Ġsamples Ġcompared Ġwith Ġonly Ġ12 Ġsamples Ġin Ġthe Ġoriginal Ġdataset . </s> fig : rc v 1 arch , Ġshows Ġthe Ġarchitecture Ġfor Ġthe Ġend - to - end Ġtext Ġcategor ization Ġtask . ĠOn Ġthe Ġbottom Ġof Ġthe Ġarchitecture Ġ$ S _ e $ Ġreceives Ġword Ġembed d ings Ġand Ġoutputs Ġa Ġsentence Ġvector Ġ$ S $ Ġ. ĠThe Ġ$ S $ Ġvector Ġpass Ġthrough Ġa Ġfully Ġconnected Ġ( FC ) Ġlayer Ġwith Ġs igm oid Ġactivation Ġfunction Ġthat Ġoutputs Ġa Ġvector Ġ$\ hat { y } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 55 } $ Ġwith Ġeach Ġelement Ġ$\ hat { y } _ j Ġ\ in Ġ[ 0 , 1 ] $ Ġ. </s> The Ġarchitecture Ġdescribed Ġabove Ġis Ġtrained Ġunder Ġthe Ġassumption Ġthat Ġeach Ġcategory Ġis Ġindependent Ġbut Ġnot Ġmutually Ġexclusive Ġsince Ġa Ġsample Ġcan Ġhave Ġmore Ġthan Ġone Ġcategory Ġassigned Ġ( mult il abel Ġclassification ). ĠThe Ġloss Ġper Ġsample Ġis Ġthe Ġaverage Ġlog Ġloss Ġacross Ġall Ġlabels : </s> $$ \ math cal Ġ{ L } (\ hat { y }, Ġy ) Ġ= Ġ- Ġ\ sum Ġ_ { i = 1 } ^{ 55 }\ left ( Ġy _ i Ġ\ log Ġ( \ hat { y } _ i ) Ġ+ Ġ( 1 - y _{ i }) Ġ\ log Ġ( 1 - \ hat { y } _{ i }) Ġ\ right ) $$ Ġ Ġ Ġ( E q . Ġ48 ) </s> where Ġthe Ġindex Ġ$ i $ Ġruns Ġover Ġthe Ġelements Ġof Ġthe Ġpredicted Ġand Ġtrue Ġvectors . </s> Given Ġthe Ġhigh Ġcategories Ġimbalance , Ġduring Ġthe Ġtraining Ġwe Ġmonitor Ġthe Ġ$ F _ 1 $ Ġmetric Ġof Ġthe Ġvalidation Ġset Ġand Ġchoose Ġthe Ġmodel Ġwith Ġthe Ġhighest Ġvalue . </s> Stan ford ĠNatural ĠLanguage ĠIn ference Ġ( SN LI ) Ġdataset ĠB IB REF 30 Ġconsist Ġof Ġ570 , 000 Ġpairs Ġof Ġsentences . ĠEach Ġpair Ġhas Ġa Ġpremise Ġand Ġa Ġhypothesis , Ġmanually Ġlabeled Ġwith Ġone Ġof Ġthe Ġthree Ġlabels : Ġentail ment , Ġcontradiction , Ġor Ġneutral . ĠThe ĠSN LI Ġhas Ġmany Ġdesired Ġproperties . ĠThe Ġlabels Ġare Ġequally Ġbalanced , Ġas Ġopposed Ġto Ġthe ĠRC V 1 Ġdataset . ĠAdditionally , Ġlanguage Ġinference Ġis Ġa Ġcomplex Ġtask Ġthat Ġrequires Ġa Ġdeeper Ġunderstanding Ġof Ġthe Ġsentence Ġmeaning Ġmaking Ġthis Ġdataset Ġsuitable Ġfor Ġlearning Ġsupervised Ġsentence Ġenc od ers Ġthat Ġgeneral ize Ġwell Ġto Ġother Ġtasks ĠB IB REF 17 Ġ. Ġt bl : sn lie x ma ples , Ġshows Ġexamples Ġof ĠSN LI Ġdataset Ġsentence Ġpairs Ġand Ġits Ġrespective Ġlabels . </s> In Ġorder Ġto Ġlearn Ġsentence Ġenc od ers Ġthat Ġcan Ġbe Ġtransf ered Ġto Ġother Ġtasks Ġunamb ig uously , Ġwe Ġconsider Ġa Ġneural Ġnetwork Ġarchitecture Ġfor Ġthe Ġsentence Ġenc oder Ġwith Ġshared Ġparameters Ġbetween Ġthe Ġpremise Ġand Ġhypothesis Ġpairs Ġas Ġin ĠB IB REF 17 Ġ. </s> fig : sn li arch , Ġdescribes Ġthe Ġneural Ġnetwork Ġarchitecture . ĠAfter Ġeach Ġpremise Ġand Ġhypothesis Ġis Ġencoded Ġinto Ġ$ S _ p $ Ġand Ġ$ S _ h $ Ġ, Ġrespectively , Ġwe Ġhave Ġa Ġfusion Ġlayer . ĠThis Ġlayer Ġhas Ġno Ġtrain able Ġweights Ġand Ġjust Ġconc aten ate Ġeach Ġsentence Ġembed ding . ĠFollowing ĠB IB REF 17 Ġ, Ġwe Ġadd Ġtwo Ġmore Ġmatching Ġmethods : Ġthe Ġabsolute Ġdifference Ġ$\ vert ĠS _ p Ġ- ĠS _ h Ġ\ vert Ġ$ Ġand Ġthe Ġelement - wise Ġ$ S _ p Ġ\ od ot ĠS _ h $ Ġ. ĠFinally , Ġin Ġorder Ġto Ġlearn Ġthe Ġpair Ġrepresentation , Ġ$ S _ ph $ Ġis Ġfeed Ġinto Ġand ĠFC Ġlayer Ġwith Ġrect ified Ġlinear Ġunit Ġ( Re LU ) Ġactivation Ġfunction , Ġwhich Ġis Ġexpressed Ġas Ġ$ f ( x ) Ġ= Ġ\ log Ġ( 1 Ġ+ Ġe ^ x ) $ Ġ. ĠThe Ġlast Ġsoft max Ġlayer Ġoutputs Ġthe Ġprobability Ġof Ġeach Ġclass . </s> Finally , Ġthe ĠNL I Ġclass ifier Ġweights Ġare Ġoptimized Ġin Ġorder Ġto Ġminimize Ġthe Ġcateg orical Ġlog Ġloss Ġper Ġsample </s> $$ \ math cal Ġ{ L } (\ hat { y }, Ġy ) Ġ= Ġ- Ġ\ sum Ġ_ { j = 1 } ^{ 3 } y _ i Ġ\ log Ġ( \ hat { y } _ i ) $$ Ġ Ġ Ġ( E q . Ġ52 ) </s> During Ġthe Ġtraining , Ġwe Ġmonitor Ġthe Ġvalidation Ġset Ġaccuracy Ġand Ġchoose Ġthe Ġmodel Ġwith Ġthe Ġhighest Ġmetric Ġvalue . </s> Sequ ence ĠModels </s> We Ġstart Ġthis Ġsection Ġby Ġreviewing Ġthe ĠRec urrent ĠNeural ĠNetwork Ġ( R NN ) Ġarchitecture Ġand Ġits Ġapplication Ġto Ġencode Ġa Ġsequence Ġof Ġwords . </s> R NN 's Ġare Ġcapable Ġof Ġhandling Ġvariable - length Ġsequences , Ġthis Ġbeing Ġa Ġdirect Ġconsequence Ġof Ġits Ġrecurrent Ġcell , Ġwhich Ġshares Ġthe Ġsame Ġparameters Ġacross Ġall Ġsequence Ġelements . ĠIn Ġthis Ġwork , Ġwe Ġadopt Ġthe ĠLong ĠShort - Term ĠMemory Ġ( L ST M ) Ġcell ĠB IB REF 38 Ġwith Ġforget Ġgates Ġ$ f _ t $ ĠB IB REF 39 Ġ. ĠThe ĠL ST M Ġcell Ġis Ġendowed Ġwith Ġa Ġmemory Ġstate Ġthat Ġcan Ġlearn Ġrepresentations Ġthat Ġdepend Ġon Ġthe Ġorder Ġof Ġthe Ġwords Ġin Ġa Ġsentence . ĠThis Ġmakes ĠL ST M Ġmore Ġfit Ġto Ġfind Ġrelations Ġthat Ġcould Ġnot Ġbe Ġcaptured Ġusing Ġstandard Ġbag - of - words Ġrepresentations . </s> Let Ġ$ x _ 1 , Ġx _ 2 , Ġ\ cd ots Ġ, Ġx _ T $ Ġbe Ġa Ġseries Ġof Ġobservations Ġof Ġlength Ġ$ T $ Ġ, Ġwhere Ġ$ x _ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ w } $ Ġ. ĠIn Ġgeneral Ġterms , Ġthe ĠL ST M Ġcell Ġreceives Ġa Ġprevious Ġhidden Ġstate Ġ$ h _{ t - 1 } $ Ġthat Ġis Ġcombined Ġwith Ġthe Ġcurrent Ġobservation Ġ$ x _ t $ Ġand Ġa Ġmemory Ġstate Ġ$ C _ t $ Ġto Ġoutput Ġa Ġnew Ġhidden Ġstate Ġ$ h _ t $ Ġ. ĠThis Ġinternal Ġmemory Ġstate Ġ$ C _{ t } $ Ġis Ġupdated Ġdepending Ġon Ġits Ġprevious Ġstate Ġand Ġthree Ġmod ulating Ġgates : Ġinput , Ġforget , Ġand Ġoutput . ĠForm ally , Ġfor Ġeach Ġstep Ġ$ t $ Ġthe Ġupdating Ġprocess Ġgoes Ġas Ġfollows Ġ( see Ġfig : l st m cell Ġfor Ġa Ġhigh Ġlevel Ġschematic Ġview ): ĠFirst , Ġwe Ġcalculate Ġthe Ġinput Ġ$ i _ t $ Ġ, Ġforget Ġ$ T $ 0 Ġ, Ġand Ġoutput Ġ$ T $ 1 Ġgates : </s> $$ i _ t Ġ& = Ġ\ s igma Ġ_ s \ left ( W _ i Ġx _ t Ġ+ ĠU _ i Ġh _{ t - 1 } Ġ+ Ġb _ i \ right ) Ġ\\ Ġf _ t Ġ& = Ġ\ s igma Ġ_ s \ left ( W _ f Ġx _ t Ġ+ ĠU _ f Ġh _{ t - 1 } Ġ+ Ġb _ f \ right ) Ġ\\ Ġo _ t Ġ& = Ġ\ s igma Ġ_ s \ left ( W _ o Ġx _ t Ġ+ ĠU _ o Ġh _{ t - 1 } Ġ+ Ġb _ o \ right ) $$ Ġ Ġ Ġ( E q . Ġ54 ) </s> where Ġ$\ s igma Ġ_ s $ Ġis Ġthe Ġs igm oid Ġactivation . ĠSecond , Ġa Ġcandidate Ġmemory Ġstate Ġ$\ wid et ilde { C } _ t $ Ġis Ġgenerated : </s> $$ \ wid et ilde { C } _ t Ġ= Ġ\ tan h Ġ\ left ( W _ c Ġx _ t Ġ+ ĠU _ c Ġh _{ t - 1 } Ġ+ Ġb _ c \ right ) $$ Ġ Ġ Ġ( E q . Ġ55 ) </s> Now Ġwe Ġare Ġin Ġa Ġposition Ġto Ġset Ġthe Ġfinal Ġmemory Ġstate Ġ$ C _ t $ Ġ. ĠIts Ġvalue Ġis Ġmod ulated Ġbased Ġon Ġthe Ġinput Ġand Ġforget Ġgates Ġof Ġeq : input for get g ates Ġand Ġis Ġgiven Ġby : </s> $$ C _ t Ġ= Ġi _ t Ġ\ od ot Ġ\ wid et ilde { C } _ t Ġ+ Ġf _ t Ġ\ od ot ĠC _{ t - 1 } $$ Ġ Ġ Ġ( E q . Ġ56 ) </s> Finally , Ġbased Ġon Ġthe Ġmemory Ġstate Ġand Ġoutput Ġgate Ġof Ġeq : input for get g ates , Ġwe Ġhave Ġthe Ġoutput Ġhidden Ġstate </s> $$ h _ t Ġ= Ġo _ t Ġ\ od ot Ġ\ tan h Ġ\ left ( C _ t \ right ) $$ Ġ Ġ Ġ( E q . Ġ57 ) </s> Regarding Ġthe Ġtrain able Ġweights , Ġlet Ġ$ n $ Ġbe Ġthe ĠL ST M Ġcell Ġnumber Ġof Ġunits . ĠIt Ġfollows Ġthat Ġ$ W $ Ġ' s Ġand Ġ$ U $ Ġ' s Ġmat rices Ġof Ġthe Ġaff ine Ġtransformations Ġhave Ġ${ n Ġ\ times Ġd _ w } $ Ġand Ġ${ n Ġ\ times Ġn } $ Ġdimensions , Ġrespectively . ĠIts Ġbias Ġterms Ġ$ b $ Ġ' s Ġare Ġvectors Ġof Ġsize Ġ$ n $ Ġ. ĠConsequently , Ġthe Ġtotal Ġnumber Ġof Ġparameters Ġis Ġ$ 4 Ġ( n Ġd _ w Ġ+ Ġn ^ 2 Ġ+ Ġn ) $ Ġand Ġdoes Ġnot Ġdepend Ġon Ġthe Ġsequence Ġnumber Ġof Ġtime Ġsteps Ġ$ T $ Ġ. </s> We Ġsee Ġthat Ġthe ĠL ST M Ġnetworks Ġare Ġable Ġto Ġcapture Ġtemporal Ġdependencies Ġin Ġsequences Ġof Ġarbitrary Ġlength . ĠOne Ġstraightforward Ġapplication Ġis Ġto Ġmodel Ġthe ĠSent ence Ġenc oder Ġdiscussed Ġin Ġsec : transfer learning , Ġwhich Ġoutputs Ġa Ġsentence Ġvector Ġrepresentation Ġusing Ġits Ġwords Ġas Ġinput . </s> Given Ġa Ġsequence Ġof Ġwords Ġ$\ left \ l brace Ġw _ t \ right \ r brace Ġ_ { t = 1 } ^{ T } $ Ġwe Ġaim Ġto Ġlearn Ġthe Ġwords Ġhidden Ġstate Ġ$\ left \ l brace Ġh _ t \ right \ r brace Ġ_ { t = 1 } ^{ T } $ Ġin Ġa Ġway Ġthat Ġeach Ġword Ġcaptures Ġthe Ġinfluence Ġof Ġits Ġpast Ġand Ġfuture Ġwords . ĠThe ĠBid irection al ĠL ST M Ġ( Bi L ST M ) Ġproposed Ġin ĠB IB REF 40 Ġis Ġan ĠL ST M Ġthat ĠâĢ ľ reads âĢ Ŀ Ġa Ġsentence , Ġor Ġany Ġsequence Ġin Ġgeneral , Ġfrom Ġthe Ġbeginning Ġto Ġthe Ġend Ġ( forward ) Ġand Ġthe Ġother Ġway Ġaround Ġ( back ward ). ĠThe Ġnew Ġstate Ġ$ h _ t $ Ġis Ġthe Ġconc aten ation </s> $$ h _ t Ġ= Ġ[ \ over right arrow { h _ t }, Ġ\ over left arrow { h _ t } ] $$ Ġ Ġ Ġ( E q . Ġ59 ) </s> where </s> $$ \ over right arrow { h _ t } Ġ& = Ġ\ text { L ST M }\ left ( w _ 1 , Ġ\ cd ots Ġ, Ġw _ T \ right ) Ġ\\ Ġ\ over left arrow { h _ t } Ġ& = Ġ\ text { L ST M }\ left ( w _ T , Ġ\ cd ots Ġ, Ġw _ 1 \ right ) Ġ\\ $$ Ġ Ġ Ġ( E q . Ġ60 ) </s> Because Ġsentences Ġhave Ġdifferent Ġlengths , Ġwe Ġneed Ġto Ġconvert Ġthe Ġ$ T $ Ġconc aten ated Ġhidden Ġstates Ġof Ġthe ĠBi L ST M Ġinto Ġa Ġfixed - length Ġsentence Ġrepresentation . ĠOne Ġstraightforward Ġoperation Ġis Ġto Ġapply Ġany Ġform Ġof Ġpool ing . ĠAttention Ġmechanism Ġis Ġan Ġalternative Ġapproach Ġwhere Ġthe Ġsentence Ġis Ġrepresented Ġas Ġan Ġweighted Ġaverage Ġof Ġhidden Ġstates Ġwhere Ġthe Ġweights Ġare Ġlearnt Ġend - to - end . </s> In Ġthe Ġnext Ġsections Ġwe Ġdescribe Ġthe Ġsentence Ġenc od ers Ġusing Ġpool ing Ġand Ġattention Ġlayers . </s> The Ġmax - pool ing Ġlayer Ġaims Ġto Ġextract Ġthe Ġmost Ġsalient Ġword Ġfeatures Ġall Ġover Ġthe Ġsentence . ĠForm ally , Ġit Ġoutputs Ġa Ġsentence Ġvector Ġrepresentation Ġ$ S _{ MP } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 n } $ Ġsuch Ġthat </s> $$ S _{ MP } Ġ= Ġ\ max Ġ_ { t = 1 } ^{ T } Ġh _ t $$ Ġ Ġ Ġ( E q . Ġ62 ) </s> where Ġ$ h _ t $ Ġis Ġdefined Ġin Ġeq : ht con cat Ġand Ġthe Ġ$\ max Ġ$ Ġoperator Ġis Ġapplied Ġover Ġthe Ġtime Ġsteps Ġdimension . Ġfig : b ilst mm ax pool Ġillustrates Ġthe ĠBi L ST M Ġmax - pool ing Ġ( MP ) Ġsentence Ġenc oder . </s> The Ġefficacy Ġof Ġthe Ġmax - pool ing Ġlayer Ġwas Ġassessed Ġin Ġmany ĠN LP Ġstudies . ĠB IB REF 41 Ġemployed Ġa Ġmax - pool ing Ġlayer Ġon Ġtop Ġof Ġword Ġrepresentations Ġand Ġargues Ġthat Ġit Ġperforms Ġbetter Ġthan Ġmean Ġpool ing . ĠExperimental Ġresults Ġin ĠB IB REF 17 Ġshow Ġthat Ġamong Ġthree Ġtypes Ġof Ġpool ing Ġ( max , Ġmean Ġand Ġlast ) Ġthe Ġmax - pool ing Ġprovides Ġthe Ġmost Ġuniversal Ġsentence Ġrepresentations Ġin Ġterms Ġof Ġtransferring Ġperformance Ġto Ġother Ġtasks . ĠGround ed Ġon Ġthese Ġstudies , Ġin Ġthis Ġwork , Ġwe Ġchoose Ġthe ĠBi L ST M Ġmax - pool ing Ġas Ġour Ġpool ing Ġlayer Ġof Ġchoice . </s> Att ention Ġmechanisms Ġwere Ġintroduced Ġin Ġthe Ġdeep Ġlearning Ġliterature Ġto Ġovercome Ġsome Ġsimpl ifications Ġimposed Ġby Ġpool ing Ġoperators . ĠWhen Ġwe Ġhumans Ġread Ġa Ġsentence , Ġwe Ġare Ġable Ġto Ġspot Ġits Ġmost Ġrelevant Ġparts Ġin Ġa Ġgiven Ġcontext Ġand Ġdisregard Ġinformation Ġthat Ġis Ġredundant Ġor Ġmisleading . ĠThe Ġattention Ġmodel Ġaims Ġto Ġmimic Ġthis Ġbehaviour . </s> Att ention Ġlayers Ġwere Ġproposed Ġfor Ġdifferent ĠN LP Ġtasks . ĠFor Ġexample , ĠNL I , Ġwith Ġcross - att ention Ġbetween Ġpremise Ġand Ġhypothesis , ĠQuestion Ġ& ĠAn sw ering Ġand ĠMachine ĠTranslation Ġ( MT ). ĠSpecifically Ġin Ġthe ĠMachine ĠTranslation Ġtask , Ġeach Ġword Ġin Ġthe Ġtarget Ġsentence Ġlearns Ġto Ġattend Ġthe Ġrelevant Ġwords Ġof Ġthe Ġsource Ġsentence Ġin Ġorder Ġto Ġgenerate Ġthe Ġsentence Ġtranslation . </s> A Ġsentence Ġenc oder Ġwith Ġattention Ġ( or Ġself - att ent ive ) ĠB IB REF 42 Ġ, ĠB IB REF 43 Ġ, ĠB IB REF 44 Ġassigns Ġdifferent Ġweights Ġto Ġthe Ġown Ġwords Ġof Ġthe Ġsentence ; Ġtherefore , Ġconverting Ġthe Ġhidden Ġstates Ġinto Ġa Ġsingle Ġsentence Ġvector Ġrepresentation . </s> Considering Ġthe Ġword Ġhidden Ġvectors Ġset Ġ$\ l brace Ġh _ 1 , Ġ\ cd ots Ġ, Ġh _ T \ r brace Ġ$ Ġwhere Ġ$ h _ t Ġ\ in Ġ\ math bb Ġ{ R } ^ n $ Ġ, Ġthe Ġattention Ġmechanism Ġis Ġdefined Ġby Ġthe Ġequations : </s> $$ \ t ilde { h } _ t Ġ& = Ġ\ s igma Ġ\ left ( W Ġh _ t Ġ+ Ġb Ġ\ right ) Ġ\\ Ġ\ alpha Ġ_ { t } Ġ& = Ġ\ frac {\ exp Ġ({ v ^ {\ inter cal Ġ} Ġ\ cd ot Ġ\ t ilde { h } _ t } Ġ) }{ \ sum Ġ_ { t } Ġ\ exp Ġ({ v Ġ\ cd ot Ġ\ t ilde { h } _ t }) } Ġ\\ ĠS _{ A _ w } Ġ& = Ġ\ sum Ġ_ { t } Ġ\ alpha Ġ_ { t } Ġh _ t $$ Ġ Ġ Ġ( E q . Ġ66 ) </s> where Ġ$ W Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ a Ġ\ times Ġn } $ Ġ, Ġ$ b Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ a Ġ\ times Ġ1 } $ Ġ, Ġand Ġ$ v Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ a Ġ\ times Ġ1 } $ Ġare Ġtrain able Ġparameters . </s> We Ġcan Ġsee Ġthat Ġthe Ġsentence Ġrepresentation Ġ$ S _{ A _ w } $ Ġis Ġa Ġweighted Ġaverage Ġof Ġthe Ġhidden Ġstates . Ġfig : b ilst min ner att Ġprovides Ġa Ġschematic Ġview Ġof Ġthe ĠBi L ST M Ġattention , Ġwhere Ġwe Ġcan Ġaccount Ġthe Ġattention Ġdescribed Ġin Ġeq : att Ġas Ġa Ġtwo Ġlayer Ġmodel Ġwith Ġa Ġdense Ġlayer Ġ( Ġ$ d _ a $ Ġunits ) Ġfollowed Ġby Ġanother Ġdense Ġthat Ġpredicts Ġ$\ alpha Ġ_ t $ Ġ( single Ġunit ). </s> Method ology </s> In Ġthis Ġsection , Ġwe Ġfirst Ġintroduce Ġour Ġproblem Ġin Ġa Ġdeep Ġmultim od al Ġlearning Ġframework . ĠWe Ġthen Ġpresent Ġour Ġneural Ġarchitecture , Ġwhich Ġis Ġable Ġto Ġaddress Ġthe Ġproblems Ġof Ġnews Ġrelevance Ġand Ġnovelty . ĠFinally , Ġwe Ġreview Ġthe Ġmethods Ġapplied Ġto Ġlearn Ġcommon alities Ġbetween Ġstocks Ġ( global Ġfeatures ). </s> Problem Ġstatement </s> Our Ġproblem Ġis Ġto Ġpredict Ġthe Ġdaily Ġstock Ġvolatility . ĠAs Ġdiscussed Ġin Ġsubs ub : range vol est im ators , Ġthe ĠG aman - K lass Ġestim ator Ġ$\ wide hat {\ s igma Ġ_ { G K , t }} $ Ġin Ġeq : vol g k Ġis Ġa Ġvery Ġefficient Ġshort - term Ġvolatility Ġproxy , Ġthus , Ġit Ġis Ġadopted Ġas Ġour Ġtarget Ġvariable . </s> Our Ġgoal Ġis Ġto Ġlearn Ġa Ġmapping Ġbetween Ġthe Ġnext Ġday Ġvolatility Ġ$\ s igma Ġ_ { t + 1 } $ Ġand Ġhistorical Ġmultim od al Ġdata Ġavailable Ġup Ġto Ġday Ġ$ t $ Ġ. ĠTo Ġthis Ġaim , Ġwe Ġuse Ġa Ġsliding Ġwindow Ġapproach Ġwith Ġwindow Ġsize Ġ$ T $ Ġ. ĠThat Ġis , Ġfor Ġeach Ġstock Ġ$ sc $ Ġa Ġsample Ġon Ġday Ġ$ t $ Ġis Ġexpressed Ġas Ġa Ġsequence Ġof Ġhistorical Ġprices Ġ$ P ^{ sc } _ t $ Ġand Ġcorpus Ġheadlines Ġ$ N ^{ sc } _ t $ Ġ. ĠThe Ġprice Ġsequence Ġis Ġa Ġvector Ġof ĠDaily ĠPrices Ġ( DP ) Ġand Ġexpressed Ġas </s> $$ P ^{ sc } _ t Ġ= Ġ\ left [ DP ^{ sc } _{ t - T }, ĠDP ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDP ^{ sc } _ t Ġ\ right ] $$ Ġ Ġ Ġ( E q . Ġ69 ) </s> where Ġ$ DP ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġis Ġa Ġvector Ġof Ġprice Ġfeatures . ĠIn Ġorder Ġto Ġavoid Ġtask - specific Ġfeature Ġengineering , Ġthe Ġdaily Ġprice Ġfeatures Ġare Ġexpressed Ġas Ġthe Ġsimple Ġreturns : </s> $$ DP ^{ sc } _ t Ġ= Ġ\ left [ Ġ\ frac { O ^{ sc } _{ t }} { C ^{ sc } _{ t - 1 }} Ġ- Ġ1 , Ġ\ frac { H ^{ sc } _{ t }} { C ^{ sc } _{ t - 1 }} Ġ- Ġ1 , Ġ\ frac { L ^{ sc } _{ t }} { C ^{ sc } _{ t - 1 }} Ġ- Ġ1 , Ġ\ frac { C ^{ sc } _{ t }} { C ^{ sc } _{ t - 1 }} Ġ- Ġ1 Ġ\ right ] $$ Ġ Ġ Ġ( E q . Ġ70 ) </s> The Ġsequence Ġof Ġhistorical Ġcorpus Ġheadlines Ġ$ N ^{ sc } _ t $ Ġis Ġexpressed Ġas </s> $$ N ^{ sc } _ t Ġ= Ġ\ left [ n ^{ sc } _{ t - T }, Ġn ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, Ġn ^{ sc } _{ t } Ġ\ right ] $$ Ġ Ġ Ġ( E q . Ġ71 ) </s> where Ġ$ n ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġis Ġa Ġset Ġcontaining Ġall Ġheadlines Ġthat Ġinfluence Ġthe Ġmarket Ġon Ġa Ġgiven Ġday Ġ$ t ^ {\ prime Ġ} $ Ġ. </s> Aim ing Ġto Ġalign Ġprices Ġand Ġnews Ġmodes , Ġwe Ġconsider Ġthe Ġexplicit Ġalignment Ġmethod Ġdiscussed Ġin Ġsub sec : stock head lines . ĠThat Ġis , Ġ$ n ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġcontains Ġall Ġstock Ġheadlines Ġbefore Ġthe Ġmarket Ġopens Ġ( Ġ$\ text tt Ġ{ before Ġmarket } _{ t } $ Ġ), Ġduring Ġthe Ġtrading Ġhours </s> ( Ġ$\ text tt Ġ{ during Ġmarket } _{ t } $ Ġ), Ġand Ġprevious Ġday Ġafter - markets </s> ( Ġ$\ text tt Ġ{ after Ġmarket } _{ t - 1 } $ Ġ). </s> As Ġa Ġtext Ġpre processing Ġstep , Ġwe Ġtoken ize Ġthe Ġheadlines Ġand Ġconvert Ġeach Ġword Ġto Ġan Ġinteger Ġthat Ġrefers Ġto Ġits Ġrespective Ġpre - trained Ġword Ġembed ding . ĠThis Ġprocess Ġis Ġdescribed Ġas Ġfollows : ĠFirst , Ġfor Ġall Ġstocks Ġof Ġour Ġcorpus Ġwe Ġtoken ize Ġeach Ġheadline Ġand Ġextract Ġthe Ġcorpus Ġvocabulary Ġset Ġ$ V $ Ġ. ĠWe Ġthen Ġbuild Ġthe Ġembed ding Ġmatrix Ġ$ E _ w Ġ\ in Ġ\ math bb Ġ{ R } ^ {\ vert ĠV Ġ\ vert Ġ\ times Ġd _ w } $ Ġ, Ġwhere Ġeach Ġrow Ġis Ġa Ġword Ġembed ding Ġvector Ġ$ d _ w $ Ġdimensions . ĠWords Ġthat Ġdo Ġnot Ġhave Ġa Ġcorresponding Ġembed ding , Ġi . e . Ġout Ġof Ġvocabulary Ġwords , Ġare Ġskipped . </s> Finally , Ġthe Ġinput Ġsample Ġof Ġthe Ġtext Ġmode Ġis Ġa Ġtens or Ġof Ġintegers Ġwith Ġ$ T Ġ\ times Ġl _ n Ġ\ times Ġl _ s $ Ġdimensions , Ġwhere Ġ$ l _ n $ Ġis Ġthe Ġmaximum Ġnumber Ġof Ġnews Ġoccurring Ġin Ġa Ġgiven Ġday Ġand Ġ$ l _ s $ Ġis Ġthe Ġmaximum Ġlength Ġof Ġa Ġcorpus Ġsentence . ĠRegarding Ġthe Ġprice Ġmode , Ġwe Ġhave Ġa Ġ$ T Ġ\ times Ġ4 $ Ġtens or Ġof Ġfloating Ġnumbers . </s> Global Ġfeatures Ġand Ġstock Ġembed ding </s> Given Ġthe Ġprice Ġand Ġnews Ġhistories Ġfor Ġeach Ġstock Ġ$ sc $ Ġwe Ġcould Ġdirectly Ġlearn Ġone Ġmodel Ġper Ġstock . ĠHowever , Ġthis Ġapproach Ġsuffers Ġfrom Ġtwo Ġmain Ġdrawbacks . ĠFirst , Ġthe Ġmarket Ġactivity Ġof Ġone Ġspecific Ġstock Ġis Ġexpected Ġto Ġimpact Ġother Ġstocks , Ġwhich Ġis Ġa Ġwidely Ġaccepted Ġpattern Ġnamed ĠâĢ ľ sp ill over Ġeffect âĢ Ŀ . ĠSecond , Ġsince Ġour Ġprice Ġdata Ġis Ġsampled Ġon Ġa Ġdaily Ġbasis , Ġwe Ġwould Ġtrain Ġthe Ġstock Ġmodel Ġrelying Ġon Ġa Ġsmall Ġnumber Ġof Ġsamples . ĠOne Ġpossible Ġsolution Ġto Ġmodel Ġthe Ġcommon ality Ġamong Ġstocks Ġwould Ġbe Ġfeature Ġenrichment . ĠFor Ġexample , Ġwhen Ġmodeling Ġa Ġgiven Ġstock Ġ$ X $ Ġwe Ġwould Ġenrich Ġits Ġnews Ġand Ġprice Ġfeatures Ġby Ġconc aten ating Ġfeatures Ġfrom Ġstock Ġ$ Y $ Ġand Ġ$ Z $ Ġ. ĠAlthough Ġthe Ġfeature Ġenrichment Ġis Ġable Ġto Ġmodel Ġthe Ġeffect Ġof Ġother Ġstocks , Ġit Ġstill Ġwould Ġconsider Ġonly Ġone Ġsample Ġper Ġday . </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġmethod Ġthat Ġlearns Ġan Ġglobal Ġmodel . </s> The Ġglobal Ġmodel Ġis Ġimplemented Ġusing Ġthe Ġfollowing Ġmethods : </s> Multi - Stock Ġbatch Ġsamples : ĠSince Ġour Ġmodels Ġare Ġtrained Ġusing ĠSt och astic ĠGrad ient ĠDes cent , Ġwe Ġpropose Ġat Ġeach Ġmini - batch Ġiteration Ġto Ġsample Ġfrom Ġa Ġbatch Ġset Ġcontaining Ġany Ġstock Ġof Ġour Ġstocks Ġuniverse . ĠAs Ġa Ġconsequence , Ġthe Ġmapping Ġbetween Ġvolatility Ġand Ġmultim od al Ġdata Ġis Ġnow Ġable Ġto Ġlearn Ġcommon Ġexplanatory Ġfactors Ġamong Ġstocks . ĠMoreover , Ġadopting Ġthis Ġapproach Ġincreases Ġthe Ġtotal Ġnumber Ġof Ġtraining Ġsamples , Ġwhich Ġis Ġnow Ġthe Ġsum Ġof Ġthe Ġnumber Ġof Ġsamples Ġper Ġstock . </s> Stock ĠEmb edd ing : ĠUt il izing Ġthe ĠMulti - Stock Ġbatch Ġsamples Ġabove , Ġwe Ġtackle Ġthe Ġproblem Ġof Ġmodeling Ġcommon ality Ġamong Ġstocks . ĠHowever , Ġit Ġis Ġreasonable Ġto Ġassume Ġthat Ġstocks Ġhave Ġpart Ġof Ġits Ġdynamic Ġdriven Ġby Ġidiosyncr atic Ġfactors . ĠNevertheless , Ġwe Ġcould Ġaggregate Ġstocks Ġper Ġsector Ġor Ġrely Ġon Ġany Ġmeasure Ġof Ġsimilarity Ġamong Ġstocks . ĠIn Ġorder Ġto Ġincorporate Ġinformation Ġspecific Ġto Ġeach Ġstock , Ġwe Ġpropose Ġto Ġequip Ġour Ġmodel Ġwith Ġa ĠâĢ ľ stock Ġembed ding âĢ Ŀ Ġmode Ġthat Ġis Ġlearnt Ġjointly Ġwith Ġprice Ġand Ġnews Ġmodes . ĠThat Ġis Ġto Ġsay , Ġwe Ġleave Ġthe Ġtask Ġof Ġdistinguishing Ġthe Ġspecific Ġdynamic Ġof Ġeach Ġstock Ġto Ġbe Ġlearnt Ġby Ġthe Ġneural Ġnetwork . ĠSpecifically , Ġthis Ġstock Ġembed ding Ġis Ġmodeled Ġusing Ġa Ġdiscrete Ġencoding Ġas Ġinput , Ġi . e . Ġ$\ math cal Ġ{ I } ^{ sc } _ t $ Ġis Ġa Ġvector Ġwith Ġsize Ġequal Ġto Ġthe Ġnumber Ġof Ġstocks Ġof Ġthe Ġstocks Ġuniverse Ġand Ġhas Ġelement Ġ1 Ġfor Ġthe Ġi - th Ġcoordinate Ġand Ġ0 Ġelsewhere , Ġthus , Ġindicating Ġthe Ġstock Ġof Ġeach Ġsample . </s> Form ally , Ġwe Ġcan Ġexpress Ġthe Ġone Ġmodel Ġper Ġstock Ġapproach Ġas Ġthe Ġmapping </s> $$ \ begin { split } Ġ\ s igma Ġ^ { sc } _{ t + 1 } Ġ= Ġf ^{ sc } Ġ( ĠDN ^{ sc } _{ t - T }, ĠDN ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDN ^{ sc } _ t Ġ; Ġ\\ ĠDP ^{ sc } _{ t - T }, ĠDP ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDP ^{ sc } _ t Ġ) Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ75 ) </s> where Ġ$ DN ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġis Ġa Ġfixed - vector Ġrepresenting Ġall Ġnews Ġreleased Ġon Ġa Ġgiven Ġday Ġfor Ġthe Ġstock Ġ$ sc $ Ġand Ġ$ DP ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġis Ġdefined Ġin Ġeq : pr ic em ode vec . </s> The Ġglobal Ġmodel Ġattempts Ġto Ġlearn Ġa Ġsingle Ġmapping Ġ$ f $ Ġthat Ġat Ġeach Ġmini - batch Ġiteration Ġrandomly Ġaggreg ates Ġsamples Ġacross Ġall Ġthe Ġuniverse Ġof Ġstocks , Ġrather Ġthan Ġone Ġmapping Ġ$ f ^{ sc } $ Ġper Ġstock . ĠThe Ġglobal Ġmodel Ġis Ġexpressed Ġas </s> $$ \ begin { split } Ġ\ s igma Ġ^ { sc } _{ t + 1 } Ġ= Ġf Ġ( ĠDN ^{ sc } _{ t - T }, ĠDN ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDN ^{ sc } _ t Ġ; Ġ\\ ĠDP ^{ sc } _{ t - T }, ĠDP ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDP ^{ sc } _ t Ġ; Ġ\\ Ġ\ math cal Ġ{ I } ^{ sc } _ t ) Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ77 ) </s> In Ġthe Ġnext Ġsection , Ġwe Ġdescribe Ġour Ġhierarchical Ġneural Ġmodel Ġand Ġhow Ġthe Ġnews , Ġprice Ġand Ġstock Ġembed ding Ġare Ġfused Ġinto Ġa Ġjoint Ġrepresentation . </s> Our Ġmultim od al Ġhierarchical Ġnetwork </s> In Ġbroad Ġterms , Ġour Ġhierarchical Ġneural Ġarchitecture Ġis Ġdescribed Ġas Ġfollows . ĠFirst , Ġeach Ġheadline Ġreleased Ġon Ġa Ġgiven Ġday Ġ$ t $ Ġis Ġencoded Ġinto Ġa Ġfixed - size Ġvector Ġ$ S _ t $ Ġusing Ġa Ġsentence Ġenc oder . ĠWe Ġthen Ġapply Ġour Ġdaily ĠNew ĠRe lev ance ĠAttention Ġ( N RA ) Ġmechanism Ġthat Ġattends Ġeach Ġnews Ġbased Ġon Ġits Ġcontent Ġand Ġconverts Ġa Ġvariable Ġsize Ġof Ġnews Ġreleased Ġon Ġa Ġgiven Ġday Ġinto Ġa Ġsingle Ġvector Ġden oted Ġby ĠDaily ĠNews Ġ( Ġ$ DN $ Ġ). ĠWe Ġnote Ġthat Ġthis Ġrepresentation Ġtake Ġaccount Ġof Ġthe Ġoverall Ġeffect Ġof Ġall Ġnews Ġreleased Ġon Ġa Ġgiven Ġday . ĠThis Ġprocess Ġis Ġillustrated Ġin Ġfig : DN enc oder . ĠWe Ġnow Ġare Ġin Ġa Ġposition Ġto Ġconsider Ġthe Ġtemporal Ġeffect Ġof Ġthe Ġpast Ġ$ T $ Ġdays Ġof Ġmarket Ġnews Ġand Ġprice Ġfeatures . Ġfig : n nt imes eries arch Ġillustrates Ġthe Ġneural Ġnetwork Ġarchitecture Ġfrom Ġthe Ġtemporal Ġsequence Ġto Ġthe Ġfinal Ġvolatility Ġprediction . ĠFor Ġeach Ġstock Ġcode Ġ$ sc $ Ġthe Ġtemporal Ġencoding Ġfor Ġnews Ġis Ġden oted Ġby ĠMarket ĠNews Ġ$ MN ^{ sc } _ t $ Ġand Ġfor Ġthe Ġprice Ġby ĠMarket ĠPrice Ġ$ MP ^{ sc } _ t $ Ġand Ġare Ġa Ġfunction Ġof Ġthe Ġpast Ġ$ T $ ĠDaily ĠNews Ġrepresentations Ġ${ \ l brace ĠDN ^{ sc } _{ t - T }, Ġ\ cd ots Ġ, ĠDN ^{ sc } _ t Ġ\ r brace Ġ} $ Ġ( Text Ġmode ) Ġand ĠDaily ĠPrices Ġfeatures Ġ$ S _ t $ 0 Ġ( Price Ġmode ), Ġwhere Ġeach ĠDaily ĠPrice Ġ$ S _ t $ 1 Ġfeature Ġis Ġgiven Ġby Ġeq : pr ic em ode vec Ġand Ġthe Ġ$ S _ t $ 2 Ġrepresentation Ġis Ġcalculated Ġusing ĠDaily ĠNew ĠRe lev ance ĠAttention . ĠAfter Ġthe Ġtemporal Ġeffects Ġof Ġ$ S _ t $ 3 Ġpast Ġdays Ġof Ġmarket Ġactivity Ġwere Ġalready Ġencoded Ġinto Ġthe ĠMarket ĠNews Ġ$ S _ t $ 4 Ġand ĠMarket ĠPrice Ġ$ S _ t $ 5 Ġ, Ġwe Ġconc aten ate Ġfeature - wise Ġ$ S _ t $ 6 Ġ, Ġ$ S _ t $ 7 Ġand Ġthe ĠStock Ġembed ding Ġ$ S _ t $ 8 Ġ. ĠThe Ġstock Ġembed ding Ġ$ S _ t $ 9 Ġrepresents Ġthe Ġstock Ġcode Ġof Ġthe Ġsample Ġon Ġa Ġgiven Ġday Ġ$ t $ Ġ. ĠFinally , Ġwe Ġhave Ġa ĠFully ĠConnect ed Ġ( FC ) Ġlayer Ġthat Ġlearns Ġthe ĠJoint ĠRepresent ation Ġof Ġall Ġmodes . ĠThis Ġfixed - sized Ġjoint Ġrepresentation Ġis Ġfed Ġinto Ġa ĠFC Ġlayer Ġwith Ġlinear Ġactivation Ġthat Ġpredicts Ġthe Ġnext Ġday Ġvolatility Ġ$\ hat {\ s igma Ġ} _{ t + 1 } $ Ġ. </s> Below , Ġwe Ġdetail , Ġfor Ġeach Ġmode Ġseparately , Ġthe Ġlayers Ġof Ġour Ġhierarchical Ġmodel . </s> âĢĵ ĠText Ġmode </s> Word ĠEmb edd ing ĠRet ri eval </s> Standard Ġembed ding Ġlayer Ġwith Ġno Ġtrain able Ġparameters . ĠIt Ġreceives Ġa Ġvector Ġof Ġword Ġindices Ġas Ġinput Ġand Ġreturns Ġa Ġmatrix Ġof Ġword Ġembed d ings . </s> News ĠEnc oder </s> This Ġlayer Ġenc odes Ġall Ġnews Ġon Ġa Ġgiven Ġday Ġand Ġoutputs Ġa Ġset Ġnews Ġembed d ings Ġ$\ l brace ĠS ^{ 1 } _ t , Ġ\ cd ots Ġ, ĠS ^{ l _ n } _ t Ġ\ r brace Ġ$ Ġ. ĠEach Ġencoded Ġsentence Ġhas Ġdimension Ġ$ d _ S $ Ġ, Ġwhich Ġis Ġa Ġhyper param eter Ġof Ġour Ġmodel . ĠThis Ġlayer Ġconstitutes Ġa Ġkey Ġcomponent Ġof Ġour Ġneural Ġarchitectures Ġand , Ġas Ġsuch , Ġwe Ġevaluate Ġour Ġmodels Ġconsidering Ġsentence Ġenc od ers Ġtrained Ġend - to - end , Ġusing Ġthe ĠBi L ST M Ġattention Ġ( sub sec : b ilst min ner att ) Ġand ĠBi L ST M Ġmax - pool ing Ġ( sub sec : b ilst mm ax pool ) Ġarchitectures , Ġand Ġalso Ġtransferred Ġfrom Ġthe ĠRC V 1 Ġand ĠSN LI Ġas Ġfixed Ġfeatures . </s> Daily Ġnews Ġrelevance Ġattention </s> Our Ġproposed Ġnews Ġrelevance Ġattention Ġmechanism Ġfor Ġall Ġnews Ġreleased Ġon Ġa Ġgiven Ġday . ĠThe Ġattention Ġmechanism Ġis Ġintroduced Ġto Ġtackle Ġinformation Ġoverload . ĠIt Ġwas Ġdesigned Ġto ĠâĢ ľ filter Ġout âĢ Ŀ Ġredundant Ġor Ġmisleading Ġnews Ġand Ġfocus Ġon Ġthe Ġrelevant Ġones Ġbased Ġsolely Ġon Ġthe Ġnews Ġcontent . ĠForm ally , Ġthe Ġlayer Ġoutputs Ġa ĠDaily ĠNews Ġ( DN ) Ġembed ding Ġ$ DN ^{ sc } _ t Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ l _ n } Ġ\ beta Ġ_ i ĠS ^{ sc ^{ i }} _ t $ Ġ, Ġwhich Ġis Ġa Ġlinear Ġcombination Ġof Ġall Ġencoded Ġnews Ġon Ġa Ġgiven Ġday Ġ$ t $ Ġ. ĠThis Ġnews - level Ġattention Ġuses Ġthe Ġsame Ġequations Ġas Ġin Ġeq : att , Ġbut Ġwith Ġtrain able Ġweights Ġ$\ l brace ĠW _{ R }, Ġb _{ R }, Ġv _{ R }\ r brace Ġ$ Ġ, Ġi . e . Ġthe Ġweights Ġare Ġsegregated Ġfrom Ġthe Ġsentence Ġenc oder . Ġfig : DN enc oder , Ġillustrates Ġour Ġrelevance Ġattention . ĠNote Ġthat Ġthis Ġlayer Ġwas Ġdeliberately Ġdeveloped Ġto Ġbe Ġinvari ant Ġto Ġheadlines Ġperm utation , Ġas Ġis Ġthe Ġcase Ġwith Ġthe Ġlinear Ġcombination Ġformula Ġabove . ĠThe Ġreason Ġis Ġthat Ġour Ġprice Ġdata Ġis Ġsampled Ġdaily Ġand , Ġas Ġa Ġconsequence , Ġwe Ġare Ġnot Ġable Ġto Ġdiscriminate Ġthe Ġmarket Ġreaction Ġfor Ġeach Ġint rad ay Ġnews . </s> News ĠTem poral ĠContext </s> Sequ ence Ġlayer Ġwith Ġdaily Ġnews Ġembed d ings Ġ$ DN ^{ sc } _ t $ Ġas Ġtime Ġsteps . ĠThis Ġlayer Ġaims Ġto Ġlearn Ġthe Ġtemporal Ġcontext Ġof Ġnews , Ġi . e . Ġthe Ġrelationship Ġbetween Ġthe Ġnews Ġat Ġday Ġ$ t $ Ġand Ġthe Ġ$ T $ Ġpast Ġdays . ĠIt Ġreceives Ġas Ġinput Ġa Ġchron ologically Ġordered Ġsequence Ġof Ġ$ T $ Ġpast ĠDaily ĠNews Ġembed d ings Ġ${ \ l brace ĠDN ^{ sc } _{ t - T }, Ġ\ cd ots Ġ, ĠDN ^{ sc } _ t Ġ\ r brace Ġ} $ Ġand Ġoutputs Ġthe Ġnews Ġmode Ġencoding ĠMarket ĠNews Ġ$ MN ^{ sc } _ t Ġ\ in Ġd _{ MN } $ Ġ. ĠThe Ġsequence Ġwith Ġ$ T $ Ġtime Ġsteps Ġis Ġencoded Ġusing Ġa ĠBi L ST M Ġattention . ĠThe Ġlayer Ġwas Ġdesigned Ġto Ġcapture Ġthe Ġtemporal Ġorder Ġthat Ġnews Ġare Ġreleased Ġand Ġthe Ġcurrent Ġnews Ġnovelty . Ġi . e . Ġnews Ġthat Ġwere Ġrepeated Ġin Ġthe Ġpast Ġcan Ġbe ĠâĢ ľ for gotten âĢ Ŀ Ġbased Ġon Ġthe Ġmod ulating Ġgates Ġof Ġthe ĠL ST M Ġnetwork . </s> âĢĵ ĠPrice Ġmode </s> Price ĠEnc oder </s> Sequ ence Ġlayer Ġanalogous Ġto ĠNews ĠTem poral ĠContext , Ġbut Ġfor Ġthe Ġprice Ġmode . ĠThe Ġinput Ġis Ġthe Ġordered Ġsequence ĠDaily ĠPrices Ġ${ \ l brace ĠDP ^{ sc } _{ t - T }, Ġ\ cd ots Ġ, ĠDP ^{ sc } _ t Ġ\ r brace Ġ} $ Ġof Ġsize Ġ$ T $ Ġ, Ġwhere Ġeach Ġelement Ġthe Ġprice Ġfeature Ġdefined Ġin Ġeq : pr ic em ode vec . ĠParticularly , Ġthe Ġarchitecture Ġconsists Ġof Ġtwo Ġstacked ĠL ST M 's . ĠThe Ġfirst Ġone Ġoutputs Ġfor Ġeach Ġprice Ġfeature Ġtime Ġstep Ġa Ġhidden Ġvector Ġthat Ġtakes Ġthe Ġtemporal Ġcontext Ġinto Ġaccount . ĠThen Ġthese Ġhidden Ġvectors Ġare Ġagain Ġpassed Ġto Ġa Ġsecond Ġindependent ĠL ST M . ĠThe Ġlayer Ġoutputs Ġthe Ġprice Ġmode Ġencoding ĠMarket ĠPrice Ġ$ MP ^{ sc } _ t Ġ\ in Ġd _{ MP } $ Ġ. ĠThis Ġencoding Ġis Ġthe Ġlast Ġhidden Ġvector Ġof Ġthe Ġsecond ĠL ST M ĠMarket . </s> âĢĵ ĠStock Ġembed ding </s> Stock ĠEnc oder </s> Stock Ġdense Ġrepresentation . ĠThe Ġlayer Ġreceives Ġthe Ġdiscrete Ġencoding Ġ$\ math cal Ġ{ I } ^{ sc } _ t $ Ġindicating Ġthe Ġsample Ġstock Ġcode Ġpass Ġthrough Ġa ĠFC Ġlayer Ġand Ġoutputs Ġa Ġstock Ġembed ding Ġ$ E _{ sc } $ Ġ. </s> âĢĵ ĠJoint ĠRepresent ation </s> Mer ging </s> Feature - wise ĠNews , ĠPrice , Ġand ĠStock Ġmodes Ġconc aten ation . ĠNo Ġtrain able Ġparameters . </s> J oint ĠRepresent ation ĠEnc oder </s> FC Ġlayer Ġof Ġsize Ġ$ d _{ JR } $ Ġ. </s> Mult im od al Ġlearning Ġwith Ġmissing Ġmodes </s> During Ġthe Ġtraining Ġwe Ġfeed Ġinto Ġour Ġneural Ġmodel Ġthe Ġprice , Ġnews , Ġand Ġstock Ġindicator Ġdata . ĠThe Ġprice Ġand Ġstock Ġindicator Ġmodes Ġdata Ġoccur Ġin Ġall Ġdays . ĠHowever , Ġat Ġthe Ġindividual Ġstock Ġlevel Ġwe Ġcan Ġhave Ġdays Ġthat Ġthe Ġcompany Ġis Ġnot Ġcovered Ġby Ġthe Ġmedia . ĠThis Ġfeature Ġimposes Ġchallenges Ġto Ġour Ġmultim od al Ġtraining Ġsince Ġneural Ġnetworks Ġare Ġnot Ġable Ġto Ġhandle Ġmissing Ġmodes Ġwithout Ġspecial Ġintervention . ĠA Ġstraightforward Ġsolution Ġwould Ġbe Ġto Ġconsider Ġonly Ġdays Ġwith Ġnews Ġreleased , Ġdisreg arding Ġthe Ġremaining Ġsamples . ĠHowever , Ġthis Ġapproach Ġhas Ġtwo Ġmain Ġdrawbacks . ĠFirst , Ġthe ĠâĢ ľ missing Ġnews âĢ Ŀ Ġdo Ġnot Ġhappen Ġat Ġrandom , Ġor Ġare Ġattributed Ġto Ġmeasurement Ġfailure Ġas Ġis , Ġfor Ġexample , Ġthe Ġcase Ġof Ġmultim od al Ġtasks Ġusing Ġmechanical Ġsensors Ġdata . ĠConversely , Ġas Ġhighlighted Ġin ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġthe Ġsame Ġprice Ġbehaviour Ġresults Ġin Ġdistinct Ġmarket Ġreactions Ġwhen Ġaccompanied Ġor Ġnot Ġby Ġnews . ĠIn Ġother Ġwords , Ġspecifically Ġto Ġfinancial Ġforecasting Ġproblems Ġthe Ġabsence Ġor Ġexistence Ġof Ġnews Ġare Ġhighly Ġinformative . </s> Some Ġmethods Ġwere Ġproposed Ġin Ġthe Ġmultim od al Ġliterature Ġto Ġeffectively Ġtreat Ġinformative Ġmissing Ġmodes Ġor ĠâĢ ľ in form ative Ġmissing ness âĢ Ŀ , Ġwhich Ġis Ġa Ġcharacteristic Ġrefere ed Ġin Ġthe Ġliterature Ġas Ġlearning Ġwith Ġmissing Ġmod alities ĠB IB REF 22 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġdirectly Ġmodel Ġthe Ġnews Ġmissing ness Ġas Ġa Ġfeature Ġof Ġour Ġtext Ġmodel Ġtemporal Ġsequence Ġby Ġusing Ġthe Ġmethod Ġinitially Ġproposed Ġin ĠB IB REF 45 Ġ, ĠB IB REF 46 Ġfor Ġclinical Ġdata Ġwith Ġmissing Ġmeasurements Ġand Ġapplied Ġin Ġthe Ġcontext Ġof Ġfinancial Ġforecasting Ġin ĠB IB REF 47 Ġ. ĠSpecifically , Ġwe Ġimplement Ġthe ĠZ eros Ġ& ĠIm put ation Ġ( ZI ) Ġmethod ĠB IB REF 46 Ġin Ġorder Ġto Ġjointly Ġlearn Ġthe Ġprice Ġmode Ġand Ġnews Ġrelationship Ġacross Ġall Ġdays Ġof Ġmarket Ġactivity . </s> The ĠZ I Ġimplementation Ġis Ġdescribed Ġas Ġfollows : ĠBefore Ġthe Ġdaily Ġnews Ġsequence Ġis Ġprocessed Ġby Ġthe Ġtext Ġtemporal Ġlayer Ġ( described Ġin Ġit m : new st cl ayer ) Ġwe Ġinput Ġa Ġ0 Ġvector Ġfor Ġall Ġtime Ġsteps Ġwith Ġmissing Ġnews Ġand Ġleave Ġthe Ġnews Ġencoding Ġunchanged Ġotherwise . ĠThis Ġstep Ġis Ġcalled Ġzero Ġimp utation . ĠIn Ġaddition , Ġwe Ġconc aten ate Ġfeature - wise Ġan Ġindicator Ġvector Ġwith Ġvalue Ġ1 Ġfor Ġall Ġvectors Ġwith Ġzero Ġimp utation Ġand Ġ0 Ġfor Ġthe Ġdays Ġwith Ġnews . </s> As Ġdescribed Ġin ĠB IB REF 47 Ġ, Ġthe ĠZ I Ġmethod Ġend ow Ġa Ġtemporal Ġsequence Ġmodel Ġwith Ġthe Ġability Ġto Ġlearn Ġdifferent Ġrepresentations Ġdepending Ġon Ġthe Ġnews Ġhistory Ġand Ġits Ġrelative Ġtime Ġposition . ĠMoreover , Ġit Ġallows Ġour Ġmodel Ġto Ġpredict Ġthe Ġvolatility Ġfor Ġall Ġdays Ġof Ġour Ġtime Ġseries Ġand , Ġat Ġthe Ġsame Ġtime , Ġtake Ġinto Ġaccount Ġthe Ġcurrent Ġand Ġpast Ġnews Ġinformative Ġmissing ness . ĠFurthermore , Ġthe Ġlearnt Ġpositional Ġnews Ġencoding Ġworks Ġdifferently Ġthan Ġa Ġtypical ĠâĢ ľ mask ing âĢ Ŀ , Ġwhere Ġdays Ġwithout Ġnews Ġare Ġnot Ġpassed Ġthrough Ġthe ĠL ST M Ġcell . ĠMask ing Ġthe Ġtime Ġsteps Ġwould Ġbe Ġlosing Ġinformation Ġabout Ġthe Ġpresence Ġor Ġabsence Ġof Ġnews Ġcon com itant Ġwith Ġprices . </s> Exper imental Ġresults Ġand Ġdiscussions </s> We Ġaim Ġto Ġevaluate Ġour Ġhierarchical Ġneural Ġmodel Ġin Ġthe Ġlight Ġof Ġthree Ġmain Ġaspects . ĠFirst , Ġwe Ġasses Ġthe Ġimportance Ġof Ġthe Ġdifferent Ġsentence Ġenc od ers Ġto Ġour Ġend - to - end Ġmodels Ġand Ġhow Ġit Ġcompares Ġto Ġtransferring Ġthe Ġsentence Ġenc oder Ġfrom Ġour Ġtwo Ġauxiliary ĠTL Ġtasks . ĠSecond , Ġwe Ġab late Ġour Ġproposed Ġnews Ġrelevance Ġattention Ġ( N RA ) Ġcomponent Ġto Ġevaluate Ġits Ġimportance . ĠFinally , Ġwe Ġconsider Ġa Ġmodel Ġthat Ġtakes Ġinto Ġconsideration Ġonly Ġthe Ġprice Ġmode Ġ( un im od al ), Ġi . e . Ġignoring Ġany Ġarchitecture Ġrelated Ġto Ġthe Ġtext Ġmode . </s> Before Ġwe Ġdefine Ġthe Ġbas elines Ġto Ġasses Ġthe Ġthree Ġaspects Ġdescribed Ġabove , Ġwe Ġreview Ġin Ġthe Ġnext Ġsection Ġthe Ġscores Ġof Ġthe Ġtrained ĠTL Ġtasks . </s> A ux iliary Ġtransfer Ġlearning Ġtasks </s> This Ġsection Ġreports Ġthe Ġperformance Ġof Ġthe Ġauxiliary ĠTL Ġtasks Ġconsidered Ġin Ġthis Ġwork . ĠOur Ġultimate Ġgoal Ġis Ġto Ġindicate Ġthat Ġour Ġscores Ġare Ġin Ġline Ġwith Ġprevious Ġworks ĠAll Ġthe Ġarchitectures Ġpresented Ġin Ġsec : transfer learning Ġare Ġtrained Ġfor Ġa Ġmaximum Ġof Ġ50 Ġepoch s Ġusing Ġmini - batch ĠSG D Ġwith ĠAdam Ġoptim izer ĠB IB REF 48 Ġ. ĠMoreover , Ġat Ġthe Ġend Ġof Ġeach Ġepoch , Ġwe Ġevaluate Ġthe Ġvalidation Ġscores , Ġwhich Ġare Ġaccuracy Ġ( Stan for ĠSN LI Ġdataset ) Ġand ĠF 1 Ġ( RC V 1 Ġdataset ), Ġand Ġsave Ġthe Ġweights Ġwith Ġthe Ġbest Ġvalues . ĠAim ing Ġto Ġse ep ed Ġup Ġtraining , Ġwe Ġimplement Ġearly Ġstopping Ġwith Ġpatience Ġset Ġto Ġ8 Ġepoch s . ĠThat Ġis , Ġif Ġthe Ġvalidation Ġscores Ġdo Ġnot Ġimprove Ġfor Ġmore Ġthan Ġ10 Ġepoch s Ġwe Ġhalt Ġthe Ġtraining . ĠFinally , Ġwe Ġuse ĠGl ove Ġpre - trained Ġword Ġembed d ings ĠB IB REF 37 Ġas Ġfixed Ġfeatures . </s> t bl : tle val uation Ġcompares Ġour Ġtest Ġscores Ġwith Ġstate - of - the - art Ġ( S OTA ) Ġresults Ġreported Ġin Ġprevious Ġworks . ĠWe Ġcan Ġsee Ġthat Ġour Ġscores Ġfor Ġthe ĠSN LI Ġtask Ġare Ġvery Ġclose Ġto Ġstate - of - the - art . </s> Regarding Ġthe ĠRC V 1 Ġdataset , Ġour Ġresults Ġconsider Ġonly Ġthe Ġheadline Ġcontent Ġfor Ġtraining , Ġwhile Ġthe Ġrefere ed Ġworks Ġconsider Ġboth Ġthe Ġnews Ġheadline Ġand Ġmessage Ġbody . ĠThe Ġreason Ġfor Ġtraining Ġusing Ġonly Ġthe Ġheadlines Ġis Ġthat Ġboth Ġtasks Ġare Ġlearnt Ġwith Ġthe Ġsole Ġpurpose Ġof Ġtransferring Ġthe Ġsentence Ġenc od ers Ġto Ġour Ġmain Ġvolatility Ġprediction Ġtask , Ġwhose Ġtextual Ġinput Ġis Ġrestricted Ġto Ġheadlines . </s> Training Ġsetup </s> During Ġthe Ġtraining Ġof Ġour Ġhierarchical Ġneural Ġmodel Ġdescribed Ġin Ġsub : H AN Ġwe Ġtook Ġspecial Ġcare Ġto Ġguard Ġagainst Ġover fitting . ĠTo Ġthis Ġaim , Ġwe Ġcompletely Ġseparate Ġ2016 Ġand Ġ2017 Ġas Ġthe Ġtest Ġset Ġand Ġreport Ġour Ġresults Ġon Ġthis ĠâĢ ľ un seen âĢ Ŀ Ġset . ĠThe Ġremaining Ġdata Ġis Ġfurther Ġsplit Ġinto Ġtraining Ġ( 2007 Ġto Ġ2013 ) Ġand Ġvalidation Ġ( 2014 Ġto Ġ2015 ). ĠThe Ġmodel Ġconvergence Ġduring Ġtraining Ġis Ġmonitored Ġin Ġthe Ġvalidation Ġset . ĠWe Ġmonitor Ġthe Ġvalidation Ġscore Ġof Ġour Ġmodel Ġat Ġthe Ġend Ġof Ġeach Ġepoch Ġand Ġstore Ġthe Ġnetwork Ġweights Ġif Ġthe Ġvalidation Ġscores Ġimproves Ġbetween Ġtwo Ġconsecutive Ġepoch s . ĠAdditionally , Ġwe Ġuse Ġmini - batch ĠSG D Ġwith ĠAdam Ġoptim izer Ġand Ġearly Ġstopping Ġwith Ġpatience Ġset Ġto Ġeight Ġepoch s . ĠThe Ġhyper param eter Ġtun ning Ġis Ġperformed Ġusing Ġgrid Ġsearch . </s> All Ġtraining Ġis Ġperformed Ġusing Ġthe Ġproposed Ġglobal Ġmodel Ġapproach Ġdescribed Ġin Ġsub : global model , Ġwhich Ġlearns Ġa Ġmodel Ġthat Ġtakes Ġinto Ġaccount Ġthe Ġfeatures Ġof Ġall Ġthe Ġ40 Ġstocks Ġof Ġour Ġcorpus . ĠUsing Ġthis Ġapproach Ġour Ġtraining Ġset Ġhas Ġa Ġtotal Ġof Ġ97 , 9 03 Ġsamples . ĠMoreover , Ġduring Ġthe ĠSG D Ġmini - batch Ġsampling Ġthe Ġpast Ġ$ T $ Ġdays Ġof Ġprice Ġand Ġnews Ġhistory Ġtens ors Ġand Ġeach Ġstock Ġsample Ġstock Ġindicator Ġare Ġrandomly Ġselected Ġfrom Ġthe Ġset Ġof Ġall Ġ40 Ġstocks . </s> St ocks Ġuniverse Ġresult </s> In Ġorder Ġto Ġevaluate Ġthe Ġcontributions Ġof Ġeach Ġcomponent Ġof Ġour Ġneural Ġmodel Ġdescribed Ġin Ġsub : H AN Ġand Ġthe Ġeffect Ġof Ġusing Ġtextual Ġdata Ġto Ġpredict Ġthe Ġvolatility , Ġwe Ġreport Ġour Ġresults Ġusing Ġthe Ġfollowing Ġbas elines : </s> - ĠNews Ġ( un im od al Ġprice Ġonly ): ĠThis Ġbaseline Ġcompletely Ġabl ates Ġ( i . e . Ġremoves ) Ġany Ġarchitecture Ġrelated Ġto Ġthe Ġnews Ġmode , Ġconsidering Ġonly Ġthe Ġprice Ġencoding Ġand Ġthe Ġstock Ġembed ding Ġcomponents . ĠUsing Ġthis Ġab lation Ġwe Ġaim Ġto Ġevaluate Ġthe Ġinfluence Ġof Ġnews Ġto Ġthe Ġvolatility Ġprediction Ġproblem . </s> + ĠNews Ġ( End - to - end ĠSent ence ĠEnc od ers ) Ġ- ĠNRA : ĠThis Ġbaseline Ġabl ates Ġour Ġproposed Ġnew Ġrelevance Ġattention Ġ( N RA ) Ġcomponent , Ġand Ġinstead , Ġmakes Ġuse Ġof Ġthe Ġsame ĠDaily ĠA ver aging Ġmethod Ġin ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ, Ġwhere Ġall Ġfixed - sized Ġheadline Ġrepresentations Ġon Ġa Ġgiven Ġday Ġare Ġaveraged Ġwithout Ġtaking Ġinto Ġaccount Ġthe Ġrelevance Ġof Ġeach Ġnews . ĠWe Ġevaluate Ġthis Ġbaseline Ġfor Ġboth ĠBi L ST M Ġattention Ġ( Att ) Ġand ĠBi L ST M Ġmax - pool ing Ġ( MP ) Ġsentence Ġenc od ers . ĠHere , Ġour Ġgoal Ġis Ġto Ġasses Ġthe Ġtrue Ġcontribution Ġof Ġour ĠNRA Ġcomponent Ġin Ġthe Ġcase ĠS OTA Ġsentence Ġenc od ers Ġare Ġtaken Ġinto Ġaccount . </s> + ĠNews Ġ( End - to - End ĠW - L ĠAtt ĠSent ence ĠEnc oder ) Ġ+ ĠNRA : ĠThe ĠWord - Level ĠAttention Ġ( W - L ĠAtt ) Ġsentence Ġenc oder Ġimplements Ġan Ġattention Ġmechanism Ġdirectly Ġon Ġtop Ġof Ġword Ġembed d ings , Ġand , Ġas Ġsuch , Ġdoes Ġnot Ġconsider Ġthe Ġorder Ġof Ġwords Ġin Ġa Ġsentence . ĠThis Ġbaseline Ġcomple ments Ġthe Ġprevious Ġone , Ġi . e . Ġit Ġevaluates Ġthe Ġinfluence Ġof Ġthe Ġsentence Ġenc oder Ġwhen Ġour Ġfull Ġspecification Ġis Ġconsidered . </s> + ĠNews Ġ( TL ĠSent ence ĠEnc od ers ) Ġ+ ĠNRA : ĠMakes Ġuse Ġof Ġsentence Ġenc od ers Ġof Ġour Ġtwo Ġauxiliary ĠTL Ġtasks Ġas Ġfixed Ġfeatures . ĠThis Ġbaseline Ġaims Ġto Ġaddress Ġthe Ġfollowing Ġquestions , Ġnamely : ĠWhat Ġdataset Ġand Ġmodels Ġare Ġmore Ġsuitable Ġto Ġtransfer Ġto Ġour Ġspecific Ġvolatility Ġforecasting Ġproblem ; ĠHow ĠEnd - to - End Ġmodels , Ġwhich Ġare Ġtrained Ġon Ġtop Ġof Ġword Ġembed d ings , Ġperform Ġcompared Ġto Ġsentence Ġenc od ers Ġtransferred Ġfrom Ġother Ġtasks . </s> t bl : com par ative all se ctors Ġsummarizes Ġthe Ġtest Ġscores Ġfor Ġthe Ġabl ations Ġdiscussed Ġabove . ĠOur Ġbest Ġmodel Ġis Ġthe Ġ+ ĠNews Ġ( Bi L ST M ĠAtt ) Ġ+ ĠNRA , Ġwhich Ġis Ġtrained Ġend - to - end Ġand Ġuses Ġour Ġfull Ġarchitecture . ĠThe Ġsecond Ġbest Ġmodel , Ġi . e . Ġ+ ĠNews Ġ( Bi L ST M ĠMP ) Ġ+ ĠNRA , Ġranks Ġslightly Ġlower Ġand Ġonly Ġdiffers Ġform Ġthe Ġbest Ġmodel Ġin Ġterms Ġof Ġthe Ġsentence Ġenc oder . ĠThe Ġformer Ġsentence Ġenc oder Ġuses Ġan Ġattention Ġlayer Ġ( sub sec : b ilst min ner att ) Ġand Ġthe Ġthe Ġlast Ġa Ġmax - pool ing Ġlayer Ġ( sub sec : b ilst mm ax pool ), Ġwhere Ġboth Ġlayers Ġare Ġplaced Ġon Ġtop Ġof Ġthe ĠL ST M Ġhidden Ġstates Ġof Ġeach Ġword . </s> Import antly , Ġour Ġexperiments Ġshow Ġthat Ġusing Ġnews Ġand Ġprice Ġ( mult im od al ) Ġto Ġpredict Ġthe Ġvolatility Ġimproves Ġthe Ġscores Ġby Ġ11 % Ġ( M SE ) Ġand Ġ9 % Ġ( MA E ) Ġwhen Ġcompared Ġwith Ġthe ĠâĢĵ ĠNews Ġ( price Ġonly Ġunim od al ) Ġmodel Ġthat Ġconsiders Ġonly Ġprice Ġfeatures Ġas Ġexplanatory Ġvariables . </s> When Ġcomparing Ġthe Ġperformance Ġof ĠEnd - to - End Ġmodels Ġand Ġthe ĠTL Ġauxiliary Ġtasks Ġthe Ġfollowing Ġcan Ġbe Ġobserved : ĠThe Ġend - to - end Ġmodels Ġtrained Ġwith Ġthe Ġtwo ĠS OTA Ġsentence Ġenc od ers Ġperform Ġbetter Ġthan Ġtransferring Ġsentence Ġenc oder Ġfrom Ġboth Ġauxiliary Ġtasks . ĠHowever , Ġour Ġexperiments Ġshow Ġthat Ġthe Ġsame Ġdoes Ġnot Ġhold Ġfor Ġmodels Ġtrained Ġend - to - end Ġrelying Ġon Ġthe Ġsimpler ĠW L - Att Ġsentence Ġenc oder , Ġwhich Ġignores Ġthe Ġorder Ġof Ġwords Ġin Ġa Ġsentence . ĠIn Ġother Ġwords , Ġconsidering Ġthe Ġappropriate ĠTL Ġtask , Ġit Ġis Ġpreferable Ġto Ġtransfer Ġa ĠS OTA Ġsentence Ġenc oder Ġtrained Ġon Ġa Ġlarger Ġdataset Ġthan Ġlearning Ġa Ġless Ġrobust Ġsentence Ġenc oder Ġin Ġan Ġend - to - end Ġfashion . ĠMoreover , Ġinitially , Ġwe Ġthought Ġthat Ġbeing Ġthe ĠRC V 1 Ġa Ġfinancial Ġdomain Ġcorpus Ġit Ġwould Ġdemonstrate Ġa Ġsuperior Ġperformance Ġwhen Ġcompared Ġto Ġthe ĠSN LI Ġdataset . ĠStill , Ġthe ĠSN LI Ġtransfers Ġbetter Ġthan ĠRC V 1 . ĠWe Ġhypothes ize Ġthat Ġthe Ġtext Ġcategor ization Ġtask Ġ( RC V 1 Ġdataset ) Ġis Ġnot Ġable Ġto Ġcapture Ġcomplex Ġsentence Ġstructures Ġat Ġthe Ġsame Ġlevel Ġrequired Ġto Ġperform Ġnatural Ġlanguage Ġinference . ĠParticularly Ġto Ġthe Ġvolatility Ġforecasting Ġproblem , Ġour ĠTL Ġresults Ġcorrobor ates Ġthe Ġsame Ġfindings Ġin ĠB IB REF 17 Ġ, Ġwhere Ġit Ġwas Ġshown Ġthat ĠSN LI Ġdataset Ġatt ains Ġthe Ġbest Ġsentence Ġencoding Ġfor Ġa Ġbroad Ġrange Ġof Ġpure ĠN LP Ġtasks , Ġincluding , Ġamong Ġother , Ġtext Ġcategor ization Ġand Ġsentiment Ġanalysis . </s> Sign ificantly , Ġexperimental Ġresults Ġin Ġt bl : com par ative all se ctors Ġclearly Ġdemonstrate Ġthat Ġour Ġproposed Ġnews Ġrelevance Ġattention Ġ( N RA ) Ġoutper forms Ġthe ĠNews ĠA ver aging Ġmethod Ġproposed Ġin Ġprevious Ġstudies ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠEven Ġwhen Ġevaluating Ġour ĠNRA Ġcomponent Ġin Ġconjunction Ġwith Ġthe Ġmore Ġelementary ĠW - L ĠAtt Ġsentence Ġenc oder Ġit Ġsurpass Ġthe Ġresults Ġof Ġsophisticated Ġsentence Ġenc oder Ġusing Ġa ĠNews ĠA ver aging Ġapproach . ĠIn Ġother Ġwords , Ġour Ġresults Ġstrongly Ġpoints Ġto Ġthe Ġadvantage Ġof Ġdiscriminating Ġnoisy Ġfrom Ġimpacting Ġnews Ġand Ġthe Ġeffectiveness Ġof Ġlearning Ġto Ġattend Ġthe Ġmost Ġrelevant Ġnews . </s> Having Ġanalyzed Ġour Ġbest Ġmodel , Ġwe Ġnow Ġturn Ġto Ġits Ġcomparative Ġperformance Ġwith Ġrespect Ġto Ġthe Ġwidely Ġregarded ĠG ARCH ( 1 , 1 ) Ġmodel Ġdescribed Ġin Ġsec : G ARCH . </s> We Ġasses Ġour Ġmodel Ġperformance Ġrelative Ġto ĠG ARCH ( 1 , 1 ) Ġusing Ġstandard Ġloss Ġmetrics Ġ( M SE Ġand ĠMA E ) Ġand Ġthe Ġregression - based Ġaccuracy Ġspecified Ġin Ġeq : reg ression loss Ġand Ġmeasured Ġin Ġterms Ġof Ġthe Ġcoefficient Ġof Ġdetermination Ġ$ R ^ 2 $ Ġ. ĠIn Ġaddition , Ġwe Ġevaluate Ġour Ġmodel Ġacross Ġtwo Ġdifferent Ġvolatility Ġproxies : ĠGar man - K lass Ġ( Ġ$\ wide hat {\ s igma Ġ_ { G K }} $ Ġ) Ġ( eq : vol g k ) Ġand ĠParkinson Ġ( Ġ$\ wide hat {\ s igma Ġ_ { PK }} $ Ġ) Ġ( eq : vol p k ). ĠWe Ġnote Ġthat , Ġas Ġreviewed Ġin Ġsub : eval ution , Ġthese Ġtwo Ġvolatility Ġproxies Ġare Ġstatically Ġefficient Ġand Ġproper Ġestim ators Ġof Ġthe Ġnext Ġday Ġvolatility . </s> t bl : g arch all se ctors Ġreports Ġthe Ġcomparative Ġperformance Ġamong Ġour Ġbest ĠPrice Ġ+ ĠNews Ġmodel Ġ(+ ĠNews ĠBi L ST M Ġ( MP ) Ġ+ ĠNRA ), Ġour ĠPrice Ġonly Ġ( un im od al ) Ġmodel Ġand ĠG ARCH ( 1 , 1 ). ĠThe Ġresults Ġclearly Ġdemonstrate Ġthe Ġsuperiority Ġof Ġour Ġmodel , Ġbeing Ġmore Ġaccurate Ġthan ĠGR ACH Ġfor Ġboth Ġvolatility Ġproxies . ĠWe Ġnote Ġthat Ġevaluating Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel Ġrelying Ġon Ġstandard ĠM SE Ġand ĠMA E Ġerror Ġmetrics Ġshould Ġbe Ġtaken Ġwith Ġa Ġgrain Ġof Ġsalt . ĠB IB REF 35 Ġprovides Ġthe Ġbackground Ġtheory Ġand Ġarguments Ġsupporting Ġ$ R ^ 2 $ Ġas Ġthe Ġmetric Ġof Ġchoice Ġto Ġevaluate Ġthe Ġpredictive Ġpower Ġof Ġa Ġvolatility Ġmodel . ĠIn Ġany Ġcase , Ġthe Ġout performance Ġor Ġour Ġmodel Ġwith Ġrespect Ġto ĠG ARCH ( 1 , 1 ) Ġperme ates Ġall Ġthree Ġmetrics , Ġname Ġ$ R ^ 2 $ Ġ, Ġ$ M SE $ Ġand Ġ$ MA E $ Ġ. </s> S ector - level Ġresults </s> Company Ġsectors Ġare Ġexpected Ġto Ġhave Ġdifferent Ġrisk Ġlevels , Ġin Ġthe Ġsense Ġthat Ġeach Ġsector Ġis Ġdriven Ġby Ġdifferent Ġtypes Ġof Ġnews Ġand Ġeconomic Ġcycles . ĠMoreover , Ġby Ġperforming Ġa Ġsector - level Ġanalysis Ġwe Ġwere Ġinitially Ġinterested Ġin Ġunderstanding Ġif Ġthe Ġout performance Ġof Ġour Ġmodel Ġwith Ġrespect Ġto ĠG ARCH ( 1 , 1 ) Ġwas Ġthe Ġresult Ġof Ġa Ġlearning Ġbias Ġto Ġa Ġgiven Ġsector Ġor Ġif , Ġas Ġturned Ġout Ġto Ġbe Ġthe Ġcase , Ġthe Ġsuperior Ġperformance Ġof Ġour Ġmodel Ġspreads Ġacross Ġa Ġdivers ified Ġportfolio Ġof Ġsectors . </s> In Ġorder Ġto Ġevaluate Ġthe Ġperformance Ġper Ġsector , Ġwe Ġfirst Ġseparate Ġthe Ġconstituents Ġstocks Ġfor Ġeach Ġsector Ġin Ġt bl : stock un iverse . ĠThen , Ġwe Ġcalculate Ġthe Ġsame Ġmetrics Ġdiscussed Ġin Ġthe Ġprevious Ġsection Ġfor Ġeach Ġsector Ġindividually . </s> t bl : gar che ach sector Ġreports Ġour Ġexperimental Ġresults Ġsegregated Ġby Ġsector . ĠWe Ġobserve Ġthat Ġthe ĠGR ACH Ġmodel Ġaccuracy , Ġmeasured Ġusing Ġthe Ġ$ R ^ 2 $ Ġscore , Ġhas Ġa Ġhigh Ġdegree Ġof Ġvariability Ġamong Ġsectors . ĠFor Ġexample , Ġthe Ġaccuracy Ġranges Ġfrom Ġ0 . 15 Ġto Ġ0 . 44 Ġfor Ġthe ĠHealth Care Ġand ĠEnergy Ġsector , Ġrespectively . ĠThis Ġhigh Ġdegree Ġof Ġvariability Ġis Ġin Ġagreement Ġwith Ġprevious Ġresults Ġreported Ġin ĠB IB REF 16 Ġ, Ġbut Ġin Ġthe Ġcontext Ġof Ġlong - term Ġ( quarter ly ) Ġvolatility Ġpredictions . ĠAlthough Ġthe ĠG ARCH ( 1 , 1 ) Ġaccuracy Ġis Ġsector - dependent , Ġwithout Ġany Ġexception , Ġour Ġmodel Ġusing Ġprice Ġand Ġnews Ġas Ġinput Ġclearly Ġoutper forms ĠGR ACH Ġsector - wise . ĠThis Ġfact Ġallow Ġus Ġto Ġdraw Ġthe Ġfollowing Ġconclusions : </s> Our Ġmodel Ġout performance Ġis Ġpersistent Ġacross Ġsectors , Ġi . e . Ġthe Ġcharacteristics Ġof Ġthe Ġresults Ġreported Ġin Ġt bl : g arch all se ctors Ġperme ates Ġall Ġsectors , Ġrather Ġthan Ġbeing Ġcomposed Ġof Ġa Ġmix Ġof Ġoutper forming Ġand Ġunder performing Ġsector Ġcontributions . ĠThis Ġfact Ġprovides Ġa Ġstrong Ġevidence Ġthat Ġour Ġmodel Ġis Ġmore Ġaccurate Ġthan ĠG ARCH ( 1 , 1 ). </s> The Ġproposed ĠGlobal Ġmodel Ġapproach Ġdiscussed Ġin Ġsub : global model Ġis Ġable Ġto Ġgeneral ize Ġwell , Ġi . e . Ġthe Ġpatterns Ġlearnt Ġare Ġnot Ġbiased Ġto Ġa Ġgiven Ġsector Ġor Ġstock . </s> One Ġof Ġthe Ġlimitations Ġof Ġour Ġwork Ġis Ġto Ġrely Ġon Ġproxies Ġfor Ġthe Ġvolatility Ġestimation . ĠAlthough Ġthese Ġproxies Ġare Ġhandy Ġif Ġonly Ġopen , Ġhigh , Ġlow Ġand Ġclose Ġdaily Ġprice Ġdata Ġis Ġavailable , Ġhaving Ġhigh Ġfrequency Ġprice Ġdata Ġwe Ġcould Ġestimate Ġthe Ġdaily Ġvolatility Ġusing Ġthe Ġsum Ġof Ġsquared Ġint rad ay Ġreturns Ġto Ġmeasure Ġthe Ġtrue Ġdaily Ġlatent Ġvolatility . ĠFor Ġexample , Ġin Ġevaluating Ġthe Ġperformance Ġfor Ġthe Ġone - day - ahead ĠG ARCH ( 1 , 1 ) ĠYen / D ollar Ġexchange Ġrate ĠB IB REF 35 Ġreports Ġ$ R ^ 2 $ Ġvalues Ġof Ġ0 . 237 Ġand Ġ0 . 392 Ġusing Ġhourly Ġand Ġfive Ġminutes Ġsampled Ġint rad ay Ġreturns , Ġrespectively . ĠHowever , Ġwe Ġbelieve Ġthat Ġutilizing Ġint rad ay Ġdata Ġwould Ġfurther Ġimprove Ġour Ġmodel Ġperformance . </s> Since Ġour Ġexperimental Ġresults Ġdemonstrate Ġthe Ġkey Ġaspect Ġof Ġthe Ġnews Ġrelevance Ġattention Ġto Ġmodel Ġarchitecture Ġwe Ġobserve Ġthat Ġint rad ay Ġdata Ġwould Ġarguably Ġam el ior ate Ġthe Ġlearning Ġprocess . ĠHaving Ġint rad ay Ġdata Ġwould Ġallow Ġus Ġto Ġpair Ġeach Ġindividual Ġnews Ġrelease Ġwith Ġthe Ġinstantaneous Ġmarket Ġprice Ġreaction . ĠUsing Ġdaily Ġdata Ġwe Ġare Ġlosing Ġpart Ġof Ġthis Ġinformation Ġby Ġonly Ġmeasuring Ġthe Ġaggregate Ġeffect Ġof Ġall Ġnews Ġto Ġthe Ġone - day - ahead Ġprediction . </s> Conclusion </s> We Ġstudy Ġthe Ġjoint Ġeffect Ġof Ġstock Ġnews Ġand Ġprices Ġon Ġthe Ġdaily Ġvolatility Ġforecasting Ġproblem . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġwork Ġis Ġone Ġof Ġthe Ġfirst Ġstudies Ġaiming Ġto Ġpredict Ġshort - term Ġ( daily ) Ġrather Ġthan Ġlong - term Ġ( quarter ly Ġor Ġyearly ) Ġvolatility Ġtaking Ġnews Ġand Ġprice Ġas Ġexplanatory Ġvariables Ġand Ġusing Ġa Ġcomprehensive Ġdataset Ġof Ġnews Ġheadlines Ġat Ġthe Ġindividual Ġstock Ġlevel . ĠOur Ġhierarchical Ġend - to - end Ġmodel Ġbenefits Ġfrom Ġstate - of - the - art Ġapproaches Ġto Ġencode Ġtext Ġinformation Ġand Ġto Ġdeal Ġwith Ġtwo Ġmain Ġchallenges Ġin Ġcorrel ating Ġnews Ġwith Ġmarket Ġreaction : Ġnews Ġrelevance Ġand Ġnovelty . ĠThat Ġis , Ġto Ġaddress Ġthe Ġproblem Ġof Ġhow Ġto Ġattend Ġthe Ġmost Ġimportant Ġnews Ġbased Ġpurely Ġon Ġits Ġcontent Ġ( news Ġrelevance Ġattention ) Ġand Ġto Ġtake Ġinto Ġaccount Ġthe Ġtemporal Ġinformation Ġof Ġpast Ġnews Ġ( tem poral Ġcontext ). ĠAdditionally , Ġwe Ġpropose Ġa Ġmulti - stock Ġmini - batch Ġ+ Ġstock Ġembed ding Ġmethod Ġsuitable Ġto Ġmodel Ġcommon ality Ġamong Ġstocks . </s> The Ġexperimental Ġresults Ġshow Ġthat Ġour Ġmultim od al Ġapproach Ġoutper forms Ġthe ĠG ARCH ( 1 , 1 ) Ġvolatility Ġmodel , Ġwhich Ġis Ġthe Ġmost Ġprevalent Ġe con ometric Ġmodel Ġfor Ġdaily Ġvolatility Ġpredictions . ĠThe Ġout performance Ġbeing Ġsector - wise Ġand Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġcombining Ġprice Ġand Ġnews Ġfor Ġshort - term Ġvolatility Ġforecasting . ĠThe Ġfact Ġthat Ġwe Ġoutper form ĠG ARCH ( 1 , 1 ) Ġfor Ġall Ġanalyzed Ġsectors Ġconfirms Ġthe Ġrobust ness Ġof Ġour Ġproposed Ġarchitecture Ġand Ġevid ences Ġthat Ġour Ġglobal Ġmodel Ġapproach Ġgeneral izes Ġwell . </s> We Ġab lated Ġ( i . e . Ġremoved ) Ġdifferent Ġcomponents Ġof Ġour Ġneural Ġarchitecture Ġto Ġassess Ġits Ġmost Ġrelevant Ġparts . ĠTo Ġthis Ġaim , Ġwe Ġreplaced Ġour Ġproposed Ġnews Ġrelevance Ġattention Ġlayer , Ġwhich Ġaims Ġto Ġattend Ġthe Ġmost Ġimportant Ġnews Ġon Ġa Ġgiven Ġday , Ġwith Ġa Ġsimpler Ġarchitecture Ġproposed Ġin Ġthe Ġliterature , Ġwhich Ġaverages Ġthe Ġdaily Ġnews . ĠWe Ġfound Ġthat Ġour Ġattention Ġlayer Ġimproves Ġthe Ġresults . ĠAdditionally , Ġwe Ġab lated Ġall Ġthe Ġarchitecture Ġrelated Ġto Ġthe Ġnews Ġmode Ġand Ġfound Ġthat Ġnews Ġenhances Ġthe Ġforecasting Ġaccuracy . </s> Finally , Ġwe Ġevaluated Ġdifferent Ġsentence Ġenc od ers , Ġincluding Ġthose Ġtransf ered Ġfrom Ġother ĠN LP Ġtasks , Ġand Ġconcluded Ġthat Ġthey Ġachieve Ġbetter Ġperformance Ġas Ġcompared Ġto Ġa Ġplain ĠWord - level Ġattention Ġsentence Ġenc oder Ġtrained Ġend - to - end . ĠHowever , Ġthey Ġdo Ġnot Ġbeat Ġstate
Introduction Natural Language Processing (NLP) has increasingly attracted the attention of the financial community. This trend can be explained by at least three major factors. The first factor refers to the business perspective. It is the economics of gaining competitive advantage using alternative sources of data and going beyond historical stock prices, thus, trading by analyzing market news automatically. The second factor is the major advancements in the technologies to collect, store, and query massive amounts of user-generated data almost in real-time. The third factor refers to the progress made by the NLP community in understanding unstructured text. Over the last decades the number of studies using NLP for financial forecasting has experienced exponential growth. According to BIBREF0 , until 2008, less than five research articles were published per year mentioning both “stock market” and “text mining” or “sentiment analysis” keywords. In 2012, this number increased to slightly more than ten articles per year. The last numbers available for 2016 indicates this has increased to sixty articles per year. The ability to mechanically harvest the sentiment from texts using NLP has shed light on conflicting theories of financial economics. Historically, there has been two differing views on whether disagreement among market participants induces more trades. The “non-trade theorem” BIBREF1 states that assuming all market participants have common knowledge about a market event, the level of disagreement among the participants does not increase the number of trades but only leads to a revision of the market quotes. In contrast, the theoretically framework proposed in BIBREF2 advocates that disagreement among market participants increases trading volume. Using textual data from Yahoo and RagingBull.com message boards to measure the dispersion of opinions (positive or negative) among traders, it was shown in BIBREF3 that disagreement among users' messages helps to predict subsequent trading volume and volatility. Similar relation between disagreement and increased trading volume was found in BIBREF4 using Twitter posts. Additionally, textual analysis is adding to the theories of medium-term/long-term momentum/reversal in stock markets BIBREF5 . The unified Hong and Stein model BIBREF6 on stock's momentum/reversal proposes that investors underreact to news, causing slow price drifts, and overreact to price shocks not accompanied by news, hence inducing reversals. This theoretical predicated behaviour between price and news was systematically estimated and supported in BIBREF7 , BIBREF8 using financial media headlines and in BIBREF9 using the Consumer Confidence Index® published by The Conference Board BIBREF10 . Similarly, BIBREF11 uses the Harvard IV-4 sentiment lexicon to count the occurrence of words with positive and negative connotation of the Wall Street Journal showing that negative sentiment is a good predictor of price returns and trading volumes. Accurate models for forecasting both price returns and volatility are equally important in the financial domain. Volatility measures how wildly the asset is expected to oscillate in a given time period and is related to the second moment of the price return distribution. In general terms, forecasting price returns is relevant to take speculative positions. The volatility, on the other hand, measures the risk of these positions. On a daily basis, financial institutions need to assess the short-term risk of their portfolios. Measuring the risk is essential in many aspects. It is imperative for regulatory capital disclosures required by banking supervision bodies. Moreover, it is useful to dynamically adjust position sizing accordingly to market conditions, thus, maintaining the risk within reasonable levels. Although, it is crucial to predict the short-term volatility from the financial markets application perspective, much of the current NLP research on volatility forecasting focus on the volatility prediction for very long-term horizons (see BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 ). Predominately, these works are built on extensions of the bag-of-words representation that has the main drawback of not capturing word order. Financial forecasting, however, requires the ability to capture semantics that is dependent upon word order. For example, the headline “Qualcomm sues Apple for contract breach” and “Apple sues Qualcomm for contract breach” trigger different responses for each stock and for the market aggregated index, however, they share the same bag-of-words representation. Additionally, these works use features from a pretrained sentiment analyis model to train the financial forecasting model. A key limitation of this process is that it requires a labelled sentiment dataset. Additionally, the error propagation is not end-to-end. In this work, we fill in the gaps of volatility prediction research in the following manner: Related work Previous work in BIBREF12 incorporates sections of the “Form 10-K” to predict the volatility twelve months after the report is released. They train a Support Vector Regression model on top of sparse representation (bag-of-words) with standard term weighting (e.g. Term-Frequency). This work was extended in BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 by employing the Loughran-McDonald Sentiment Word Lists BIBREF20 , which contain three lists where words are grouped by their sentiments (positive, negative and neutral). In all these works, the textual representation is engineered using the following steps: 1) For each sentiment group, the list is expanded by retrieving 20 most similar words for each word using Word2Vec word embeddings BIBREF21 . 2) Finally, each 10-K document is represented using the expanded lists of words. The weight of each word in this sparse representation is defined using Information Retrieval (IR) methods such as term-frequency (tf) and term-frequency with inverted document frequency (tfidf). Particularly, BIBREF16 shows that results can be improved using enhanced IR methods and projecting each sparse feature into a dense space using Principal Component Analysis (PCA). The works described above ( BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 ) target long-horizon volatility predictions (one year or quarterly BIBREF16 ). In particular, BIBREF16 and BIBREF15 uses market data (price) features along with the textual representation of the 10-K reports. These existing works that employ multi-modal learning BIBREF22 are based on a late fusion approach. For example, stacking ensembles to take into account the price and text predictions BIBREF16 . In contrast, our end-to-end trained model can learn the joint distribution of both price and text. Predicting the price direction rather than the volatility was the focus in BIBREF23 . They extracted sentiment words from Twitter posts to build a time series of collective Profile of Mood States (POMS). Their results show that collective mood accurately predicts the direction of Down Jones stock index (86.7% accuracy). In BIBREF24 handcrafted text representations including term count, noun-phrase tags and extracted named entities are employed for predicting stock market direction using Support Vector Machine (SVM). An extension of Latent Dirichlet Allocation (LDA) is proposed in BIBREF25 to learn a joint latent space of topics and sentiments. Our deep learning models bear a close resemblance to works focused on directional price forecasting BIBREF26 , BIBREF27 . In BIBREF26 , headline news are processed using Stanford OpenIE to generate triples that are fed into a Neural Tensor Network to create the final headline representation. In BIBREF27 , a character-level embedding is pre-trained in an unsupervised manner. The character embedding is used as input to a sequence model to learn the headline representation. Particularly, both works average all headline representations in a given day, rather than attempting to weight the most relevant ones. In this work, we propose a neural attention mechanism to capture the News Relevance and provide experimental evidence that it is a key component of the end-to-end learning process. Our attention extends the previous deep learning methods from BIBREF26 , BIBREF27 . Despite the fact that end-to-end deep learning models have attained state-of-the-art performance, the large number of parameters make them prone to overfitting. Additionally, end-to-end models are trained from scratch requiring large datasets and computational resources. Transfer learning (TL) alleviates this problem by adapting representations learnt from a different and potentially weakly related source domain to the new target domain. For example, in computer vision tasks the convolutional features learnt from ImageNet BIBREF28 dataset (source domain) have been successfully transferred to multiple domain target tasks with much smaller datasets such as object classification and scene recognition BIBREF29 . In this work, we consider TL in our experiments for two main reasons. First, it address the question whether our proposed dataset is suitable for end-to-end training since the performance of the transferred representations can be compared with end-to-end learning. Second, it is still to be investigated which dataset transfers better to the forecasting problem. Recently, the NLP community has focused on universal representations of sentences BIBREF17 , BIBREF19 , which are dense representations that carry the meaning of a full sentence. BIBREF17 found that transferring the sentence representation trained on the Stanford Natural Language Inference (SNLI) BIBREF30 dataset achieves state-of-the-art sentence representations to multiple NLP tasks (e.g. sentiment analysis, question-type and opinion polarity). Following BIBREF17 , in this work, we investigate the suitability of SNLI and Reuters RCV1 BIBREF31 datasets to transfer learning to the volatility forecasting task. To the best of our knowledge, the hierarchical attention mechanism at headline level, proposed in our work, has not being applied to volatility prediction so far; neither has been investigated the ability to transfer sentence encoders from source datasets to the target forecasting problem (Transfer Learning). Our dataset Our corpus covers a broad range of news including news around earnings dates and complements the 10-K reports content. As an illustration, the headlines “Walmart warns that strong U.S. dollar will cost $15B in sales” and “Procter & Gamble Co raises FY organic sales growth forecast after sales beat” describe the company financial conditions and performance from the management point of view – these are also typical content present in Section 7 of the 10-K reports. In this section, we describe the steps involved in compiling our dataset of financial news at stock level, which comprises a broad range of business sectors. Sectors and stocks The first step in compiling our corpus was to choose the constituents stocks. Our goal was to consider stocks in a broad range of sectors, aiming a diversified financial domain corpus. We found that Exchange Traded Funds (ETF) provide a mechanical way to aggregate the most relevant stocks in a given industry/sector. An ETF is a fund that owns assets, e.g. stock shares or currencies, but, unlike mutual funds are traded in stock exchanges. These ETFs are extremely liquid and track different investment themes. We decided to use SPDR Setcor Funds constituents stocks in our work since the company is the largest provider of sector funds in the United States. We included in our analysis the top 5 (five) sector ETFs by financial trading volume (as in Jan/2018). Among the most traded sectors we also filtered out the sectors that were similar to each other. For example, the Consumer Staples and Consumer Discretionary sectors are both part of the parent Consumer category. For each of the top 5 sectors we selected the top 10 holdings, which are deemed the most relevant stocks. tbl:stockuniverse, details our dataset sectors and its respective stocks. Stock specific data We assume that an individual stock news as the one that explicitly mention the stock name or any of its surface forms in the headline. As an illustration, in order to collect all news for the stock code PG, Procter & Gamble company name, we search all the headlines with any of these words: Procter&Gamble OR Procter and Gamble OR P&G. In this example, the first word is just the company name and the remaining words are the company surface forms. We automatically derived the surface forms for each stock by starting with a seed of surface forms extracted from the DBpedia Knowledge Base (KB). We then applied the following procedure: Relate each company name with the KB entity unique identifier. Retrieve all values of the wikiPageRedirects property. The property holds the names of different pages that points to the same entity/company name. This step sets the initial seed of surface forms. Manually, filter out some noisy property values. For instance, from the Procter & Glamble entity page we were able to automatically extract dbr:Procter_and_gamble and dbr:P_&_G, but had to manually exclude the noisy associations dbr:Female_pads and dbr:California_Natural. The result of the steps above is a dictionary of surface forms $wd_{sc}$ . Stock headlines Our corpus is built at stock code level by collecting headlines from the Reuters Archive. This archive groups the headlines by date, starting from 1 January 2007. Each headline is a html link (<a href> tag) to the full body of the news, where the anchor text is the headline content followed by the release time. For example, the page dated 16 Dec 2016 has the headline “Procter & Gamble appoints Nelson Peltz to board 5:26PM UTC”. For each of the 50 stocks (5 sectors times 10 stocks per sector) selected using the criteria described in sub:corpussecstock, we retrieved all the headlines from the Reuters Archive raging from 01/01/2007 to 30/12/2017. This process takes the following steps: For a given stock code ( $sc$ ) retrieve all surface forms $wd_{sc}$ . For each day, store only the headlines content matching any word in $wd_{sc}$ . For each stored headline we also store the time and timezone. Convert the news date and time to Eastern Daylight Time (EDT). Categorize the news release time. We consider the following category set: {before market, during market , after market, holidays, weekends}. during market contains news between 9:30AM and 4:00PM. before market before 9:30AM and after market after 4:00PM. The time categories prevents any misalignment between text and stock price data. Moreover, it prevents data leakage and, consequently, unrealistic predictive model performance. In general, news released after 4:00PM EDT can drastically change market expectations and the returns calculated using close to close prices as in the GARCH(1,1) model (see eq:closingreturn). Following BIBREF3 , to deal with news misalignment, news issued after 4:00PM (after market) are grouped with the pre-market (before market) on the following trading day. tbl:stocktimecat shows the distribution of news per sector for each time category. We can see a high concentration of news released before the market opens (55% on average). In contrast, using a corpus compiled from message boards, a large occurrence of news during market hours was found BIBREF3 . This behaviour indicating day traders' activity. Our corpus comprise financial news agency headlines, a content more focused on corporate events (e.g. lawsuits, merges & acquisitions, research & development) and on economic news (see tbl:stockheadlinesexmaples for a sample of our dataset). These headlines are mostly factual. On the other hand, user-generated content such as Twitter and message boards (as in BIBREF3 , BIBREF4 ) tends to be more subjective. U.S. macroeconomic indicators such as Retail Sales, Jobless Claims and GDP are mostly released around 8:30AM (one hour before the market opens). These numbers are key drivers of market activity and, as such, have a high media coverage. Specific sections of these economic reports impact several stocks and sectors. Another factor that contribute to the high activity of news outside regular trading hours are company earnings reports. These are rarely released during trading hours. Finally, before the market opens news agencies provide a summary of the international markets developments, e.g. the key facts during the Asian and Australian trading hours. All these factors contribute to the high concentration of pre-market news. Background We start this section by reviewing the GARCH(1,1) model, which is a strong benchmark used to evaluate our neural model. We then review the source datasets proposed in the literature that were trained independently and transfered to our volatility prediction model. Finally, we review the general architectures of sequence modelling and attention mechanisms. GARCH model Financial institutions use the concept of “Value at risk” to measure the expected volatility of their portfolios. The widespread econometric model for volatility forecasting is the Generalized Autoregressive Conditional Heteroskedasticity (GARCH) BIBREF32 , BIBREF33 . Previous research shows that the GARCH(1,1) model is hard to beat. For example, BIBREF34 compared GARCH(1,1) with 330 different econometric volatility models showing that they are not significantly better than GARCH(1,1). Let $p_t$ be the price of an stock at the end of a trading period with closing returns $r_t$ given by $$r_t = \frac{p_t}{p_{t-1}} - 1 $$   (Eq. 29) The GARCH process explicitly models the time-varying volatility of asset returns. In the GARCH(1,1) specification the returns series $r_t$ follow the process: $$r_t &= \mu + \epsilon _t  \\ \epsilon _t &= \sigma _t z_t  \\ \sigma ^2_t &= a_0 + a_1 \epsilon _{t-1}^2 + b_1 \sigma _{t-1}^2$$   (Eq. 30) where $\mu $ is a constant (return drift) and $z_t$ is a sequence of i.i.d. random variables with mean zero and unit variance. It is worth noting that although the conditional mean return described in eq:garchcondmean has a constant value, the conditional volatility $\sigma _t$ is time-dependent and modeled by eq:att. The one-step ahead expected volatility forecast can be computed directly from eq:garchcondvariance and is given by $$E_T[\sigma _{T+1}^2] = a_0 + a_1 E_T[\epsilon ^2] + b_1 E_T[\sigma _{T}^2] $$   (Eq. 32) In general, the $t^{\prime }$ -steps ahead expected volatility $E_T[\sigma _{T+t^{\prime }}^2]$ can be easily expressed in terms of the previous step expected volatility. It is easy to prove by induction that the forecast for any horizon can be represented in terms of the one-step ahead forecast and is given by $$E_T[\sigma _{T+t^{\prime }}^2] - \sigma _u^2 = (a_1 + b_1)^{(t^{\prime } -1)} \left(E_T[\sigma _{T+1}^2] - \sigma _u^2\right)$$   (Eq. 33) where $\sigma _u$ is the unconditional volatility: $$\sigma _u = \sqrt{a_0 / (1 - a_1 - b_1)} $$   (Eq. 34) From the equation above we can see that for long horizons, i.e. $t^\prime \rightarrow \infty $ , the volatility forecast in eq:forecastrecursive converges to the unconditional volatility in eq:unvar. All the works reviewed in sec:introduction ( BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 ) consider GARCH(1,1) benchmark. However, given the long horizon of their predictions (e.g. quarterly or annual), the models are evaluated using the unconditional volatility $\sigma _u$ in eq:unvar. In this work, we focus on the short-term volatility prediction and use the GARCH(1,1) one-day ahead conditional volatility prediction in eq:forecastoneperiod to evaluate our models. Let $\sigma _{t+1}$ denote the ex-post “true” daily volatility at a given time $t$ . The performance on a set with $N$ daily samples can be evaluated using the standard Mean Squared Error ( $MSE$ ) and Mean Absolute Error ( $MAE$ ) $$MSE &= \frac{1}{N} \sum _{t=1}^{N} \left( E_t[\sigma _{t+1}] - \sigma _{t+1}\right)^2  \\ MAE &= \frac{1}{N} \sum _{t=1}^{N}\left|E_t[\sigma _{t+1}] - \sigma _{t+1} \right|$$   (Eq. 36) Additionally, following BIBREF35 , the models are also evaluated using the coefficient of determination $R^2$ of the regression $$\sigma _{t+1} = a + b E_t[\sigma _{t+1}] + e_t$$   (Eq. 37) where $$R^2 = 1 - \frac{\sum _{t=1}^{N}e^{2}_{t}}{\sum _{t=1}^{N}\left(E_t[\sigma _{t+1}] - \frac{1}{N} \sum _{t=1}^{N}E_t[\sigma _{t+1}]\right)^{2}}$$   (Eq. 38) One of the challenges in evaluating GARCH models is the fact that the ex-post volatility $\sigma _{t+1}$ is not directly observed. Apparently, the squared daily returns $r_{t+1}^{2}$ in eq:closingreturn could stand as a good proxy for the ex-post volatility. However, the squared returns yield very noisy measurements. This is a direct consequence of the term $z^t$ that connects the squared return to the latent volatility factor in eq:garchwhitenoise. The use of intraday prices to estimate the ex-post daily volayility was first proposed in BIBREF35 . They argue that volatility estimators using intraday prices is the proper way to evaluate the GARCH(1,1) model, as opposed to squared daily returns. For example, considering the Deutsche Mark the GARCH(1,1) model $R^2$ improves from $0.047$ (squared returns) to $0.33$ (intraday returns) BIBREF35 . It is clear from the previous section that any volatility model evaluation using the noisy squared returns as the ex-post volatility proxy will lead to very poor performance. Therefore, high-frequency intraday data is fundamental to short-term volatility performance evaluation. However, intraday data is difficult to acquire and costly. Fortunately, there are statistically efficient daily volatility estimators that only depend on the open, high, low and close prices. These price “ranges” are widely available. In this section, we discuss these estimators. Let $O_t$ , $H_t$ , $L_t$ , $C_t$ be the open, high, low and close prices of an asset in a given day $t$ . Assuming that the daily price follows a geometric Brownian motion with zero drift and constant daily volatility $\sigma $ , Parkinson (1980) derived the first daily volatility estimator $$\widehat{\sigma _{PK,t}^2} = \frac{\ln \left(\frac{H_t}{L_t}\right)^2}{4\ln (2)} $$   (Eq. 41) which represents the daily volatility in terms of its price range. Hence, it contains information about the price path. Given this property, it is expected that $\sigma _{PK}$ is less noisy than the volatility calculated using squared returns. The Parkinson's volatility estimator was extended by Garman-Klass (1980) which incorporates additional information about the opening ( $O_t$ ) and closing ( $C_t$ ) prices and is defined as $$\widehat{\sigma _{GK,t}^{2}} = \frac{1}{2} \ln \left(\frac{H_t}{L_t}\right)^2 - (2\ln (2) - 1) \ln \left(\frac{C_t}{O_t}\right)^2 $$   (Eq. 42) The relative noisy of different estimators $\hat{\sigma }$ can be measured in terms of its relative efficiency to the daily volatility $\sigma $ and is defined as $$e\left(\widehat{\sigma ^{2}}, \sigma ^2\right) \equiv \frac{Var[\sigma ^2]}{Var[\widehat{\sigma ^{2}}]}$$   (Eq. 43) where $Var[\cdot ]$ is the variance operator. It follows directly from eq:garchwhitenoise that the squared return has efficiency 1 and therefore, very noisy. BIBREF36 reports Parkinson ( $\widehat{\sigma _{PK,t}^2}$ ) volatility estimator has 4.9 relative efficiency and Garman-Klass ( $\widehat{\sigma _{GK,t}^2}$ ) 7.4. Additionally, all the described estimators are unbiased. Many alternative estimators to daily volatility have been proposed in the literature. However, experiments in BIBREF36 rate the Garman-Klass volatility estimator as the best volatility estimator based only on open, high, low and close prices. In this work, we train our models to predict the state-of-the-art Garman-Klass estimator. Moreover, we evaluate our models and GARCH(1,1) using the metrics described in sub:evalution, but with the appropriate volatility proxies, i.e. Parkinson and Garman-Klass estimators. Transfer Learning from other source domains Vector representations of words, also known as Word embeddings BIBREF21 , BIBREF37 , that represent a word as a dense vector has become the standard building blocks of almost all NLP tasks. These embeddings are trained on large unlabeled corpus and are able to capture context and similarity among words. Some attempts have been made to learn vector representations of a full sentence, rather than only a single word, using unsupervised approaches similar in nature to word embeddings. Recently, BIBREF17 showed state-of-the-art performance when a sentence encoder is trained end-to-end on a supervised source task and transferred to other target tasks. Inspired by this work, we investigate the performance of sentence encoders trained on the Text categorization and Natural Language Inference (NLI) tasks and use these encoders in our main short-term volatility prediction task. A generic sentence encoder $S_e$ receives the sentence words as input and returns a vector representing the sentence. This can be expressed as a mapping $$S_e \colon \mathbb {R}^{T^{S} \times d_w} \rightarrow \mathbb {R}^{d_S}$$   (Eq. 45) from a variable size sequence of words to a sentence vector $S$ of fixed-size $d_S$ , where $T^{S}$ is the sentence number of words and $d_w$ is the pre-trained word embedding dimension. In the following sections, we describe the datasets and architectures to train the sentence encoders of the auxiliary transfer learning tasks. The Reuters Corpus Volume I (RCV1) is corpus containing 806,791 news articles in the English language collected from 20/08/1996 to 19/08/1997 BIBREF31 . The topic of each news was human-annotated using a hierarchical structure. At the top of the hierarchy, lies the coarse-grained categories: CCAT (Corporate), ECAT (Economics), GCAT (Government), and MCAT (Markets). A news article can be assigned to more than one category meaning that the text categorization task is mutilabel. Each news is stored in a separate XML file. lst:rcv1xmlexample shows the typical structure of an article. <?xml version="1.0" encoding="iso-8859-1" ?> <newsitem itemid="6159" id="root" date="1996-08-21" xml:lang="en"> <headline>Colombia raises internal coffee price.</headline> <dateline>BOGOTA 1996-08-21</dateline> <copyright>(c) Reuters Limited 1996</copyright> <metadata> <codes class="bip:topics:1.0"> <code code="C13"> <editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/> </code> <code code="C31"> <editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/> </code> <code code="CCAT"> <editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/> </code> <code code="M14"> <editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/> </code> <code code="M141"> <editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/> </code> <code code="MCAT"> <editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/> </code> </codes> </metadata> </newsitem> The RCV1 dataset is not released with a standard train, validation, test split. In this work, we separated 15% of samples as a test set for evaluation purposes. The remaining samples were further split leaving 70% and 15% for training and validation, respectively. Regarding the categories distribution, we found that, from the original 126 categories, 23 categories were never assigned to any news; therefore, were disregarded. From the 103 classes left we found a high imbalance among the labels with a large number of underrepresented categories having less than 12 samples. The very low number of samples for these minority classes brings a great challenge to discriminate the very fine-grained categories. Aiming to alleviate this problem, we grouped into a same class all categories below the second hierarchical level. For example, given the root node CCAT (Corporate) we grouped C151 (ACCOUNTS/EARNINGS), C1511 (ANNUAL RESULTS) and C152 (COMMENT/FORECASTS) into the direct child node C15 (PERFORMANCE). Using this procedure the original 103 categories where reduced to 55. One of the benefits of this procedure was that the less represented classes end up having around thousand samples compared with only 12 samples in the original dataset. fig:rcv1arch, shows the architecture for the end-to-end text categorization task. On the bottom of the architecture $S_e$ receives word embeddings and outputs a sentence vector $S$ . The $S$ vector pass through a fully connected (FC) layer with sigmoid activation function that outputs a vector $\hat{y} \in \mathbb {R}^{55}$ with each element $\hat{y}_j \in [0,1]$ . The architecture described above is trained under the assumption that each category is independent but not mutually exclusive since a sample can have more than one category assigned (multilabel classification). The loss per sample is the average log loss across all labels: $$\mathcal {L}(\hat{y}, y) = - \sum _{i=1}^{55}\left( y_i \log (\hat{y}_i) + (1-y_{i}) \log (1-\hat{y}_{i}) \right)$$   (Eq. 48) where the index $i$ runs over the elements of the predicted and true vectors. Given the high categories imbalance, during the training we monitor the $F_1$ metric of the validation set and choose the model with the highest value. Stanford Natural Language Inference (SNLI) dataset BIBREF30 consist of 570,000 pairs of sentences. Each pair has a premise and a hypothesis, manually labeled with one of the three labels: entailment, contradiction, or neutral. The SNLI has many desired properties. The labels are equally balanced, as opposed to the RCV1 dataset. Additionally, language inference is a complex task that requires a deeper understanding of the sentence meaning making this dataset suitable for learning supervised sentence encoders that generalize well to other tasks BIBREF17 . tbl:snliexmaples, shows examples of SNLI dataset sentence pairs and its respective labels. In order to learn sentence encoders that can be transfered to other tasks unambiguously, we consider a neural network architecture for the sentence encoder with shared parameters between the premise and hypothesis pairs as in BIBREF17 . fig:snliarch, describes the neural network architecture. After each premise and hypothesis is encoded into $S_p$ and $S_h$ , respectively, we have a fusion layer. This layer has no trainable weights and just concatenate each sentence embedding. Following BIBREF17 , we add two more matching methods: the absolute difference $\vert S_p - S_h \vert $ and the element-wise $S_p \odot S_h$ . Finally, in order to learn the pair representation, $S_ph$ is feed into and FC layer with rectified linear unit (ReLU) activation function, which is expressed as $f(x) = \log (1 + e^x)$ . The last softmax layer outputs the probability of each class. Finally, the NLI classifier weights are optimized in order to minimize the categorical log loss per sample $$\mathcal {L}(\hat{y}, y) = - \sum _{j=1}^{3}y_i \log (\hat{y}_i)$$   (Eq. 52) During the training, we monitor the validation set accuracy and choose the model with the highest metric value. Sequence Models We start this section by reviewing the Recurrent Neural Network (RNN) architecture and its application to encode a sequence of words. RNN's are capable of handling variable-length sequences, this being a direct consequence of its recurrent cell, which shares the same parameters across all sequence elements. In this work, we adopt the Long Short-Term Memory (LSTM) cell BIBREF38 with forget gates $f_t$ BIBREF39 . The LSTM cell is endowed with a memory state that can learn representations that depend on the order of the words in a sentence. This makes LSTM more fit to find relations that could not be captured using standard bag-of-words representations. Let $x_1, x_2, \cdots , x_T$ be a series of observations of length $T$ , where $x_t \in \mathbb {R}^{d_w}$ . In general terms, the LSTM cell receives a previous hidden state $h_{t-1}$ that is combined with the current observation $x_t$ and a memory state $C_t$ to output a new hidden state $h_t$ . This internal memory state $C_{t}$ is updated depending on its previous state and three modulating gates: input, forget, and output. Formally, for each step $t$ the updating process goes as follows (see fig:lstmcell for a high level schematic view): First, we calculate the input $i_t$ , forget $T$0 , and output $T$1 gates: $$i_t &= \sigma _s\left(W_i x_t + U_i h_{t-1} + b_i\right) \\ f_t &= \sigma _s\left(W_f x_t + U_f h_{t-1} + b_f\right) \\ o_t &= \sigma _s\left(W_o x_t + U_o h_{t-1} + b_o\right)$$   (Eq. 54) where $\sigma _s$ is the sigmoid activation. Second, a candidate memory state $\widetilde{C}_t$ is generated: $$\widetilde{C}_t = \tanh \left(W_c x_t + U_c h_{t-1} + b_c\right)$$   (Eq. 55) Now we are in a position to set the final memory state $C_t$ . Its value is modulated based on the input and forget gates of eq:inputforgetgates and is given by: $$C_t = i_t \odot \widetilde{C}_t + f_t \odot C_{t-1}$$   (Eq. 56) Finally, based on the memory state and output gate of eq:inputforgetgates, we have the output hidden state $$h_t = o_t \odot \tanh \left(C_t\right)$$   (Eq. 57) Regarding the trainable weights, let $n$ be the LSTM cell number of units. It follows that $W$ 's and $U$ 's matrices of the affine transformations have ${n \times d_w}$ and ${n \times n}$ dimensions, respectively. Its bias terms $b$ 's are vectors of size $n$ . Consequently, the total number of parameters is $4 (n d_w + n^2 + n)$ and does not depend on the sequence number of time steps $T$ . We see that the LSTM networks are able to capture temporal dependencies in sequences of arbitrary length. One straightforward application is to model the Sentence encoder discussed in sec:transferlearning, which outputs a sentence vector representation using its words as input. Given a sequence of words $\left\lbrace w_t\right\rbrace _{t=1}^{T}$ we aim to learn the words hidden state $\left\lbrace h_t\right\rbrace _{t=1}^{T}$ in a way that each word captures the influence of its past and future words. The Bidirectional LSTM (BiLSTM) proposed in BIBREF40 is an LSTM that “reads” a sentence, or any sequence in general, from the beginning to the end (forward) and the other way around (backward). The new state $h_t$ is the concatenation $$h_t = [\overrightarrow{h_t}, \overleftarrow{h_t}]$$   (Eq. 59) where $$\overrightarrow{h_t} &= \text{LSTM}\left(w_1, \cdots , w_T\right) \\ \overleftarrow{h_t} &= \text{LSTM}\left(w_T, \cdots , w_1\right) \\$$   (Eq. 60) Because sentences have different lengths, we need to convert the $T$ concatenated hidden states of the BiLSTM into a fixed-length sentence representation. One straightforward operation is to apply any form of pooling. Attention mechanism is an alternative approach where the sentence is represented as an weighted average of hidden states where the weights are learnt end-to-end. In the next sections we describe the sentence encoders using pooling and attention layers. The max-pooling layer aims to extract the most salient word features all over the sentence. Formally, it outputs a sentence vector representation $S_{MP} \in \mathbb {R}^{2n}$ such that $$S_{MP} = \max _{t=1}^{T} h_t$$   (Eq. 62) where $h_t$ is defined in eq:htconcat and the $\max $ operator is applied over the time steps dimension. fig:bilstmmaxpool illustrates the BiLSTM max-pooling (MP) sentence encoder. The efficacy of the max-pooling layer was assessed in many NLP studies. BIBREF41 employed a max-pooling layer on top of word representations and argues that it performs better than mean pooling. Experimental results in BIBREF17 show that among three types of pooling (max, mean and last) the max-pooling provides the most universal sentence representations in terms of transferring performance to other tasks. Grounded on these studies, in this work, we choose the BiLSTM max-pooling as our pooling layer of choice. Attention mechanisms were introduced in the deep learning literature to overcome some simplifications imposed by pooling operators. When we humans read a sentence, we are able to spot its most relevant parts in a given context and disregard information that is redundant or misleading. The attention model aims to mimic this behaviour. Attention layers were proposed for different NLP tasks. For example, NLI, with cross-attention between premise and hypothesis, Question & Answering and Machine Translation (MT). Specifically in the Machine Translation task, each word in the target sentence learns to attend the relevant words of the source sentence in order to generate the sentence translation. A sentence encoder with attention (or self-attentive) BIBREF42 , BIBREF43 , BIBREF44 assigns different weights to the own words of the sentence; therefore, converting the hidden states into a single sentence vector representation. Considering the word hidden vectors set $\lbrace h_1, \cdots , h_T\rbrace $ where $h_t \in \mathbb {R}^n$ , the attention mechanism is defined by the equations: $$\tilde{h}_t &= \sigma \left(W h_t + b \right) \\ \alpha _{t} &= \frac{\exp ({v^{\intercal } \cdot \tilde{h}_t} )}{\sum _{t} \exp ({v \cdot \tilde{h}_t})} \\ S_{A_w} &= \sum _{t} \alpha _{t} h_t$$   (Eq. 66) where $W \in \mathbb {R}^{d_a \times n}$ , $b \in \mathbb {R}^{d_a \times 1}$ , and $v \in \mathbb {R}^{d_a \times 1}$ are trainable parameters. We can see that the sentence representation $S_{A_w}$ is a weighted average of the hidden states. fig:bilstminneratt provides a schematic view of the BiLSTM attention, where we can account the attention described in eq:att as a two layer model with a dense layer ( $d_a$ units) followed by another dense that predicts $\alpha _t$ (single unit). Methodology In this section, we first introduce our problem in a deep multimodal learning framework. We then present our neural architecture, which is able to address the problems of news relevance and novelty. Finally, we review the methods applied to learn commonalities between stocks (global features). Problem statement Our problem is to predict the daily stock volatility. As discussed in subsub:rangevolestimators, the Gaman-Klass estimator $\widehat{\sigma _{GK,t}}$ in eq:volgk is a very efficient short-term volatility proxy, thus, it is adopted as our target variable. Our goal is to learn a mapping between the next day volatility $\sigma _{t+1}$ and historical multimodal data available up to day $t$ . To this aim, we use a sliding window approach with window size $T$ . That is, for each stock $sc$ a sample on day $t$ is expressed as a sequence of historical prices $P^{sc}_t$ and corpus headlines $N^{sc}_t$ . The price sequence is a vector of Daily Prices (DP) and expressed as $$P^{sc}_t = \left[DP^{sc}_{t-T}, DP^{sc}_{t-T+1}, \cdots , DP^{sc}_t \right]$$   (Eq. 69) where $DP^{sc}_{t^{\prime }}$ is a vector of price features. In order to avoid task-specific feature engineering, the daily price features are expressed as the simple returns: $$DP^{sc}_t = \left[ \frac{O^{sc}_{t}}{C^{sc}_{t-1}} - 1, \frac{H^{sc}_{t}}{C^{sc}_{t-1}} - 1, \frac{L^{sc}_{t}}{C^{sc}_{t-1}} - 1, \frac{C^{sc}_{t}}{C^{sc}_{t-1}} - 1 \right]$$   (Eq. 70) The sequence of historical corpus headlines $N^{sc}_t$ is expressed as $$N^{sc}_t = \left[n^{sc}_{t-T}, n^{sc}_{t-T+1}, \cdots , n^{sc}_{t} \right]$$   (Eq. 71) where $n^{sc}_{t^{\prime }}$ is a set containing all headlines that influence the market on a given day $t^{\prime }$ . Aiming to align prices and news modes, we consider the explicit alignment method discussed in subsec:stockheadlines. That is, $n^{sc}_{t^{\prime }}$ contains all stock headlines before the market opens ( $\texttt {before market}_{t}$ ), during the trading hours ( $\texttt {during market}_{t}$ ), and previous day after-markets ( $\texttt {after market}_{t-1}$ ). As a text preprocessing step, we tokenize the headlines and convert each word to an integer that refers to its respective pre-trained word embedding. This process is described as follows: First, for all stocks of our corpus we tokenize each headline and extract the corpus vocabulary set $V$ . We then build the embedding matrix $E_w \in \mathbb {R}^{\vert V \vert \times d_w}$ , where each row is a word embedding vector $d_w$ dimensions. Words that do not have a corresponding embedding, i.e. out of vocabulary words, are skipped. Finally, the input sample of the text mode is a tensor of integers with $T \times l_n \times l_s$ dimensions, where $l_n$ is the maximum number of news occurring in a given day and $l_s$ is the maximum length of a corpus sentence. Regarding the price mode, we have a $T \times 4$ tensor of floating numbers. Global features and stock embedding Given the price and news histories for each stock $sc$ we could directly learn one model per stock. However, this approach suffers from two main drawbacks. First, the market activity of one specific stock is expected to impact other stocks, which is a widely accepted pattern named “spillover effect”. Second, since our price data is sampled on a daily basis, we would train the stock model relying on a small number of samples. One possible solution to model the commonality among stocks would be feature enrichment. For example, when modeling a given stock $X$ we would enrich its news and price features by concatenating features from stock $Y$ and $Z$ . Although the feature enrichment is able to model the effect of other stocks, it still would consider only one sample per day. In this work, we propose a method that learns an global model. The global model is implemented using the following methods: Multi-Stock batch samples: Since our models are trained using Stochastic Gradient Descent, we propose at each mini-batch iteration to sample from a batch set containing any stock of our stocks universe. As a consequence, the mapping between volatility and multimodal data is now able to learn common explanatory factors among stocks. Moreover, adopting this approach increases the total number of training samples, which is now the sum of the number of samples per stock. Stock Embedding: Utilizing the Multi-Stock batch samples above, we tackle the problem of modeling commonality among stocks. However, it is reasonable to assume that stocks have part of its dynamic driven by idiosyncratic factors. Nevertheless, we could aggregate stocks per sector or rely on any measure of similarity among stocks. In order to incorporate information specific to each stock, we propose to equip our model with a “stock embedding” mode that is learnt jointly with price and news modes. That is to say, we leave the task of distinguishing the specific dynamic of each stock to be learnt by the neural network. Specifically, this stock embedding is modeled using a discrete encoding as input, i.e. $\mathcal {I}^{sc}_t$ is a vector with size equal to the number of stocks of the stocks universe and has element 1 for the i-th coordinate and 0 elsewhere, thus, indicating the stock of each sample. Formally, we can express the one model per stock approach as the mapping $$\begin{split} \sigma ^{sc}_{t+1} = f^{sc} ( DN^{sc}_{t-T}, DN^{sc}_{t-T+1}, \cdots , DN^{sc}_t ; \\ DP^{sc}_{t-T}, DP^{sc}_{t-T+1}, \cdots , DP^{sc}_t ) \end{split}$$   (Eq. 75) where $DN^{sc}_{t^{\prime }}$ is a fixed-vector representing all news released on a given day for the stock $sc$ and $DP^{sc}_{t^{\prime }}$ is defined in eq:pricemodevec. The global model attempts to learn a single mapping $f$ that at each mini-batch iteration randomly aggregates samples across all the universe of stocks, rather than one mapping $f^{sc}$ per stock. The global model is expressed as $$\begin{split} \sigma ^{sc}_{t+1} = f ( DN^{sc}_{t-T}, DN^{sc}_{t-T+1}, \cdots , DN^{sc}_t ; \\ DP^{sc}_{t-T}, DP^{sc}_{t-T+1}, \cdots , DP^{sc}_t ; \\ \mathcal {I}^{sc}_t) \end{split}$$   (Eq. 77) In the next section, we describe our hierarchical neural model and how the news, price and stock embedding are fused into a joint representation. Our multimodal hierarchical network In broad terms, our hierarchical neural architecture is described as follows. First, each headline released on a given day $t$ is encoded into a fixed-size vector $S_t$ using a sentence encoder. We then apply our daily New Relevance Attention (NRA) mechanism that attends each news based on its content and converts a variable size of news released on a given day into a single vector denoted by Daily News ( $DN$ ). We note that this representation take account of the overall effect of all news released on a given day. This process is illustrated in fig:DNencoder. We now are in a position to consider the temporal effect of the past $T$ days of market news and price features. fig:nntimeseriesarch illustrates the neural network architecture from the temporal sequence to the final volatility prediction. For each stock code $sc$ the temporal encoding for news is denoted by Market News $MN^{sc}_t$ and for the price by Market Price $MP^{sc}_t$ and are a function of the past $T$ Daily News representations ${\lbrace DN^{sc}_{t-T}, \cdots , DN^{sc}_t \rbrace }$ (Text mode) and Daily Prices features $S_t$0 (Price mode), where each Daily Price $S_t$1 feature is given by eq:pricemodevec and the $S_t$2 representation is calculated using Daily New Relevance Attention. After the temporal effects of $S_t$3 past days of market activity were already encoded into the Market News $S_t$4 and Market Price $S_t$5 , we concatenate feature-wise $S_t$6 , $S_t$7 and the Stock embedding $S_t$8 . The stock embedding $S_t$9 represents the stock code of the sample on a given day $t$ . Finally, we have a Fully Connected (FC) layer that learns the Joint Representation of all modes. This fixed-sized joint representation is fed into a FC layer with linear activation that predicts the next day volatility $\hat{\sigma }_{t+1}$ . Below, we detail, for each mode separately, the layers of our hierarchical model. – Text mode Word Embedding Retrieval Standard embedding layer with no trainable parameters. It receives a vector of word indices as input and returns a matrix of word embeddings. News Encoder This layer encodes all news on a given day and outputs a set news embeddings $\lbrace S^{1}_t, \cdots , S^{l_n}_t \rbrace $ . Each encoded sentence has dimension $d_S$ , which is a hyperparameter of our model. This layer constitutes a key component of our neural architectures and, as such, we evaluate our models considering sentence encoders trained end-to-end, using the BiLSTM attention (subsec:bilstminneratt) and BiLSTM max-pooling (subsec:bilstmmaxpool) architectures, and also transferred from the RCV1 and SNLI as fixed features. Daily news relevance attention Our proposed news relevance attention mechanism for all news released on a given day. The attention mechanism is introduced to tackle information overload. It was designed to “filter out” redundant or misleading news and focus on the relevant ones based solely on the news content. Formally, the layer outputs a Daily News (DN) embedding $DN^{sc}_t = \sum _{i=1}^{l_n} \beta _i S^{sc^{i}}_t$ , which is a linear combination of all encoded news on a given day $t$ . This news-level attention uses the same equations as in eq:att, but with trainable weights $\lbrace W_{R}, b_{R}, v_{R}\rbrace $ , i.e. the weights are segregated from the sentence encoder. fig:DNencoder, illustrates our relevance attention. Note that this layer was deliberately developed to be invariant to headlines permutation, as is the case with the linear combination formula above. The reason is that our price data is sampled daily and, as a consequence, we are not able to discriminate the market reaction for each intraday news. News Temporal Context Sequence layer with daily news embeddings $DN^{sc}_t$ as time steps. This layer aims to learn the temporal context of news, i.e. the relationship between the news at day $t$ and the $T$ past days. It receives as input a chronologically ordered sequence of $T$ past Daily News embeddings ${\lbrace DN^{sc}_{t-T}, \cdots , DN^{sc}_t \rbrace }$ and outputs the news mode encoding Market News $MN^{sc}_t \in d_{MN}$ . The sequence with $T$ time steps is encoded using a BiLSTM attention. The layer was designed to capture the temporal order that news are released and the current news novelty. i.e. news that were repeated in the past can be “forgotten” based on the modulating gates of the LSTM network. – Price mode Price Encoder Sequence layer analogous to News Temporal Context, but for the price mode. The input is the ordered sequence Daily Prices ${\lbrace DP^{sc}_{t-T}, \cdots , DP^{sc}_t \rbrace }$ of size $T$ , where each element the price feature defined in eq:pricemodevec. Particularly, the architecture consists of two stacked LSTM's. The first one outputs for each price feature time step a hidden vector that takes the temporal context into account. Then these hidden vectors are again passed to a second independent LSTM. The layer outputs the price mode encoding Market Price $MP^{sc}_t \in d_{MP}$ . This encoding is the last hidden vector of the second LSTM Market. – Stock embedding Stock Encoder Stock dense representation. The layer receives the discrete encoding $\mathcal {I}^{sc}_t$ indicating the sample stock code pass through a FC layer and outputs a stock embedding $E_{sc}$ . – Joint Representation Merging Feature-wise News, Price, and Stock modes concatenation. No trainable parameters. Joint Representation Encoder FC layer of size $d_{JR}$ . Multimodal learning with missing modes During the training we feed into our neural model the price, news, and stock indicator data. The price and stock indicator modes data occur in all days. However, at the individual stock level we can have days that the company is not covered by the media. This feature imposes challenges to our multimodal training since neural networks are not able to handle missing modes without special intervention. A straightforward solution would be to consider only days with news released, disregarding the remaining samples. However, this approach has two main drawbacks. First, the “missing news” do not happen at random, or are attributed to measurement failure as is, for example, the case of multimodal tasks using mechanical sensors data. Conversely, as highlighted in BIBREF7 , BIBREF8 the same price behaviour results in distinct market reactions when accompanied or not by news. In other words, specifically to financial forecasting problems the absence or existence of news are highly informative. Some methods were proposed in the multimodal literature to effectively treat informative missing modes or “informative missingness”, which is a characteristic refereed in the literature as learning with missing modalities BIBREF22 . In this work, we directly model the news missingness as a feature of our text model temporal sequence by using the method initially proposed in BIBREF45 , BIBREF46 for clinical data with missing measurements and applied in the context of financial forecasting in BIBREF47 . Specifically, we implement the Zeros & Imputation (ZI) method BIBREF46 in order to jointly learn the price mode and news relationship across all days of market activity. The ZI implementation is described as follows: Before the daily news sequence is processed by the text temporal layer (described in itm:newstclayer) we input a 0 vector for all time steps with missing news and leave the news encoding unchanged otherwise. This step is called zero imputation. In addition, we concatenate feature-wise an indicator vector with value 1 for all vectors with zero imputation and 0 for the days with news. As described in BIBREF47 , the ZI method endow a temporal sequence model with the ability to learn different representations depending on the news history and its relative time position. Moreover, it allows our model to predict the volatility for all days of our time series and, at the same time, take into account the current and past news informative missingness. Furthermore, the learnt positional news encoding works differently than a typical “masking”, where days without news are not passed through the LSTM cell. Masking the time steps would be losing information about the presence or absence of news concomitant with prices. Experimental results and discussions We aim to evaluate our hierarchical neural model in the light of three main aspects. First, we asses the importance of the different sentence encoders to our end-to-end models and how it compares to transferring the sentence encoder from our two auxiliary TL tasks. Second, we ablate our proposed news relevance attention (NRA) component to evaluate its importance. Finally, we consider a model that takes into consideration only the price mode (unimodal), i.e. ignoring any architecture related to the text mode. Before we define the baselines to asses the three aspects described above, we review in the next section the scores of the trained TL tasks. Auxiliary transfer learning tasks This section reports the performance of the auxiliary TL tasks considered in this work. Our ultimate goal is to indicate that our scores are in line with previous works All the architectures presented in sec:transferlearning are trained for a maximum of 50 epochs using mini-batch SGD with Adam optimizer BIBREF48 . Moreover, at the end of each epoch, we evaluate the validation scores, which are accuracy (Stanfor SNLI dataset) and F1 (RCV1 dataset), and save the weights with the best values. Aiming to seeped up training, we implement early stopping with patience set to 8 epochs. That is, if the validation scores do not improve for more than 10 epochs we halt the training. Finally, we use Glove pre-trained word embeddings BIBREF37 as fixed features. tbl:tlevaluation compares our test scores with state-of-the-art (SOTA) results reported in previous works. We can see that our scores for the SNLI task are very close to state-of-the-art. Regarding the RCV1 dataset, our results consider only the headline content for training, while the refereed works consider both the news headline and message body. The reason for training using only the headlines is that both tasks are learnt with the sole purpose of transferring the sentence encoders to our main volatility prediction task, whose textual input is restricted to headlines. Training setup During the training of our hierarchical neural model described in sub:HAN we took special care to guard against overfitting. To this aim, we completely separate 2016 and 2017 as the test set and report our results on this “unseen” set. The remaining data is further split into training (2007 to 2013) and validation (2014 to 2015). The model convergence during training is monitored in the validation set. We monitor the validation score of our model at the end of each epoch and store the network weights if the validation scores improves between two consecutive epochs. Additionally, we use mini-batch SGD with Adam optimizer and early stopping with patience set to eight epochs. The hyperparameter tunning is performed using grid search. All training is performed using the proposed global model approach described in sub:globalmodel, which learns a model that takes into account the features of all the 40 stocks of our corpus. Using this approach our training set has a total of 97,903 samples. Moreover, during the SGD mini-batch sampling the past $T$ days of price and news history tensors and each stock sample stock indicator are randomly selected from the set of all 40 stocks. Stocks universe result In order to evaluate the contributions of each component of our neural model described in sub:HAN and the effect of using textual data to predict the volatility, we report our results using the following baselines: - News (unimodal price only): This baseline completely ablates (i.e. removes) any architecture related to the news mode, considering only the price encoding and the stock embedding components. Using this ablation we aim to evaluate the influence of news to the volatility prediction problem. + News (End-to-end Sentence Encoders) - NRA: This baseline ablates our proposed new relevance attention (NRA) component, and instead, makes use of the same Daily Averaging method in BIBREF26 , BIBREF27 , where all fixed-sized headline representations on a given day are averaged without taking into account the relevance of each news. We evaluate this baseline for both BiLSTM attention (Att) and BiLSTM max-pooling (MP) sentence encoders. Here, our goal is to asses the true contribution of our NRA component in the case SOTA sentence encoders are taken into account. + News (End-to-End W-L Att Sentence Encoder) + NRA: The Word-Level Attention (W-L Att) sentence encoder implements an attention mechanism directly on top of word embeddings, and, as such, does not consider the order of words in a sentence. This baseline complements the previous one, i.e. it evaluates the influence of the sentence encoder when our full specification is considered. + News (TL Sentence Encoders) + NRA: Makes use of sentence encoders of our two auxiliary TL tasks as fixed features. This baseline aims to address the following questions, namely: What dataset and models are more suitable to transfer to our specific volatility forecasting problem; How End-to-End models, which are trained on top of word embeddings, perform compared to sentence encoders transferred from other tasks. tbl:comparativeallsectors summarizes the test scores for the ablations discussed above. Our best model is the + News (BiLSTM Att) + NRA, which is trained end-to-end and uses our full architecture. The second best model, i.e. + News (BiLSTM MP) + NRA, ranks slightly lower and only differs form the best model in terms of the sentence encoder. The former sentence encoder uses an attention layer (subsec:bilstminneratt) and the the last a max-pooling layer (subsec:bilstmmaxpool), where both layers are placed on top of the LSTM hidden states of each word. Importantly, our experiments show that using news and price (multimodal) to predict the volatility improves the scores by 11% (MSE) and 9% (MAE) when compared with the – News (price only unimodal) model that considers only price features as explanatory variables. When comparing the performance of End-to-End models and the TL auxiliary tasks the following can be observed: The end-to-end models trained with the two SOTA sentence encoders perform better than transferring sentence encoder from both auxiliary tasks. However, our experiments show that the same does not hold for models trained end-to-end relying on the simpler WL-Att sentence encoder, which ignores the order of words in a sentence. In other words, considering the appropriate TL task, it is preferable to transfer a SOTA sentence encoder trained on a larger dataset than learning a less robust sentence encoder in an end-to-end fashion. Moreover, initially, we thought that being the RCV1 a financial domain corpus it would demonstrate a superior performance when compared to the SNLI dataset. Still, the SNLI transfers better than RCV1. We hypothesize that the text categorization task (RCV1 dataset) is not able to capture complex sentence structures at the same level required to perform natural language inference. Particularly to the volatility forecasting problem, our TL results corroborates the same findings in BIBREF17 , where it was shown that SNLI dataset attains the best sentence encoding for a broad range of pure NLP tasks, including, among other, text categorization and sentiment analysis. Significantly, experimental results in tbl:comparativeallsectors clearly demonstrate that our proposed news relevance attention (NRA) outperforms the News Averaging method proposed in previous studies BIBREF26 , BIBREF27 . Even when evaluating our NRA component in conjunction with the more elementary W-L Att sentence encoder it surpass the results of sophisticated sentence encoder using a News Averaging approach. In other words, our results strongly points to the advantage of discriminating noisy from impacting news and the effectiveness of learning to attend the most relevant news. Having analyzed our best model, we now turn to its comparative performance with respect to the widely regarded GARCH(1,1) model described in sec:GARCH. We asses our model performance relative to GARCH(1,1) using standard loss metrics (MSE and MAE) and the regression-based accuracy specified in eq:regressionloss and measured in terms of the coefficient of determination $R^2$ . In addition, we evaluate our model across two different volatility proxies: Garman-Klass ( $\widehat{\sigma _{GK}}$ ) (eq:volgk) and Parkinson ( $\widehat{\sigma _{PK}}$ ) (eq:volpk). We note that, as reviewed in sub:evalution, these two volatility proxies are statically efficient and proper estimators of the next day volatility. tbl:garchallsectors reports the comparative performance among our best Price + News model (+ News BiLSTM (MP) + NRA), our Price only (unimodal) model and GARCH(1,1). The results clearly demonstrate the superiority of our model, being more accurate than GRACH for both volatility proxies. We note that evaluating the GARCH(1,1) model relying on standard MSE and MAE error metrics should be taken with a grain of salt. BIBREF35 provides the background theory and arguments supporting $R^2$ as the metric of choice to evaluate the predictive power of a volatility model. In any case, the outperformance or our model with respect to GARCH(1,1) permeates all three metrics, name $R^2$ , $MSE$ and $MAE$ . Sector-level results Company sectors are expected to have different risk levels, in the sense that each sector is driven by different types of news and economic cycles. Moreover, by performing a sector-level analysis we were initially interested in understanding if the outperformance of our model with respect to GARCH(1,1) was the result of a learning bias to a given sector or if, as turned out to be the case, the superior performance of our model spreads across a diversified portfolio of sectors. In order to evaluate the performance per sector, we first separate the constituents stocks for each sector in tbl:stockuniverse. Then, we calculate the same metrics discussed in the previous section for each sector individually. tbl:garcheachsector reports our experimental results segregated by sector. We observe that the GRACH model accuracy, measured using the $R^2$ score, has a high degree of variability among sectors. For example, the accuracy ranges from 0.15 to 0.44 for the HealthCare and Energy sector, respectively. This high degree of variability is in agreement with previous results reported in BIBREF16 , but in the context of long-term (quarterly) volatility predictions. Although the GARCH(1,1) accuracy is sector-dependent, without any exception, our model using price and news as input clearly outperforms GRACH sector-wise. This fact allow us to draw the following conclusions: Our model outperformance is persistent across sectors, i.e. the characteristics of the results reported in tbl:garchallsectors permeates all sectors, rather than being composed of a mix of outperforming and underperforming sector contributions. This fact provides a strong evidence that our model is more accurate than GARCH(1,1). The proposed Global model approach discussed in sub:globalmodel is able to generalize well, i.e. the patterns learnt are not biased to a given sector or stock. One of the limitations of our work is to rely on proxies for the volatility estimation. Although these proxies are handy if only open, high, low and close daily price data is available, having high frequency price data we could estimate the daily volatility using the sum of squared intraday returns to measure the true daily latent volatility. For example, in evaluating the performance for the one-day-ahead GARCH(1,1) Yen/Dollar exchange rate BIBREF35 reports $R^2$ values of 0.237 and 0.392 using hourly and five minutes sampled intraday returns, respectively. However, we believe that utilizing intraday data would further improve our model performance. Since our experimental results demonstrate the key aspect of the news relevance attention to model architecture we observe that intraday data would arguably ameliorate the learning process. Having intraday data would allow us to pair each individual news release with the instantaneous market price reaction. Using daily data we are losing part of this information by only measuring the aggregate effect of all news to the one-day-ahead prediction. Conclusion We study the joint effect of stock news and prices on the daily volatility forecasting problem. To the best of our knowledge, this work is one of the first studies aiming to predict short-term (daily) rather than long-term (quarterly or yearly) volatility taking news and price as explanatory variables and using a comprehensive dataset of news headlines at the individual stock level. Our hierarchical end-to-end model benefits from state-of-the-art approaches to encode text information and to deal with two main challenges in correlating news with market reaction: news relevance and novelty. That is, to address the problem of how to attend the most important news based purely on its content (news relevance attention) and to take into account the temporal information of past news (temporal context). Additionally, we propose a multi-stock mini-batch + stock embedding method suitable to model commonality among stocks. The experimental results show that our multimodal approach outperforms the GARCH(1,1) volatility model, which is the most prevalent econometric model for daily volatility predictions. The outperformance being sector-wise and demonstrates the effectiveness of combining price and news for short-term volatility forecasting. The fact that we outperform GARCH(1,1) for all analyzed sectors confirms the robustness of our proposed architecture and evidences that our global model approach generalizes well. We ablated (i.e. removed) different components of our neural architecture to assess its most relevant parts. To this aim, we replaced our proposed news relevance attention layer, which aims to attend the most important news on a given day, with a simpler architecture proposed in the literature, which averages the daily news. We found that our attention layer improves the results. Additionally, we ablated all the architecture related to the news mode and found that news enhances the forecasting accuracy. Finally, we evaluated different sentence encoders, including those transfered from other NLP tasks, and concluded that they achieve better performance as compared to a plain Word-level attention sentence encoder trained end-to-end. However, they do not beat state-of-the-art sentence encoders trained end-to-end. In order to contribute to the literature of Universal Sentence Encoders, we evaluated the performance of transferring sentence encoders from two different tasks to the volatility prediction problem. We showed that models trained on the Natural Language Inference (NLI) task are more suitable to forecasting problems than a financial domain dataset (Reuters RCV1). By analyzing different architectures, we showed that a BiLSTM with max-pooling for the SNLI dataset provides the best sentence encoder. In the future, we plan to make use of intraday prices to better assess the predictive power of our proposed models. Additionally, we would further extend our analysis to other stock market sectors.
Which stock market sector achieved the best performance?
b634ff1607ce5756655e61b9a6f18bc736f84c83
Which Ġstock Ġmarket Ġsector Ġachieved Ġthe Ġbest Ġperformance ?
Energy Ġwith Ġaccuracy Ġof Ġ0 . 538
<s> Which Ġstock Ġmarket Ġsector Ġachieved Ġthe Ġbest Ġperformance ? </s> Introduction </s> Natural ĠLanguage ĠProcessing Ġ( N LP ) Ġhas Ġincreasingly Ġattracted Ġthe Ġattention Ġof Ġthe Ġfinancial Ġcommunity . ĠThis Ġtrend Ġcan Ġbe Ġexplained Ġby Ġat Ġleast Ġthree Ġmajor Ġfactors . ĠThe Ġfirst Ġfactor Ġrefers Ġto Ġthe Ġbusiness Ġperspective . ĠIt Ġis Ġthe Ġeconomics Ġof Ġgaining Ġcompetitive Ġadvantage Ġusing Ġalternative Ġsources Ġof Ġdata Ġand Ġgoing Ġbeyond Ġhistorical Ġstock Ġprices , Ġthus , Ġtrading Ġby Ġanalyzing Ġmarket Ġnews Ġautomatically . ĠThe Ġsecond Ġfactor Ġis Ġthe Ġmajor Ġadvancements Ġin Ġthe Ġtechnologies Ġto Ġcollect , Ġstore , Ġand Ġquery Ġmassive Ġamounts Ġof Ġuser - generated Ġdata Ġalmost Ġin Ġreal - time . ĠThe Ġthird Ġfactor Ġrefers Ġto Ġthe Ġprogress Ġmade Ġby Ġthe ĠN LP Ġcommunity Ġin Ġunderstanding Ġun struct ured Ġtext . ĠOver Ġthe Ġlast Ġdecades Ġthe Ġnumber Ġof Ġstudies Ġusing ĠN LP Ġfor Ġfinancial Ġforecasting Ġhas Ġexperienced Ġexponential Ġgrowth . ĠAccording Ġto ĠB IB REF 0 Ġ, Ġuntil Ġ2008 , Ġless Ġthan Ġfive Ġresearch Ġarticles Ġwere Ġpublished Ġper Ġyear Ġmentioning Ġboth ĠâĢ ľ stock Ġmarket âĢ Ŀ Ġand ĠâĢ ľ text Ġmining âĢ Ŀ Ġor ĠâĢ ľ sent iment Ġanalysis âĢ Ŀ Ġkeywords . ĠIn Ġ2012 , Ġthis Ġnumber Ġincreased Ġto Ġslightly Ġmore Ġthan Ġten Ġarticles Ġper Ġyear . ĠThe Ġlast Ġnumbers Ġavailable Ġfor Ġ2016 Ġindicates Ġthis Ġhas Ġincreased Ġto Ġsixty Ġarticles Ġper Ġyear . </s> The Ġability Ġto Ġmechanically Ġharvest Ġthe Ġsentiment Ġfrom Ġtexts Ġusing ĠN LP Ġhas Ġshed Ġlight Ġon Ġconflicting Ġtheories Ġof Ġfinancial Ġeconomics . ĠHistorically , Ġthere Ġhas Ġbeen Ġtwo Ġdiffering Ġviews Ġon Ġwhether Ġdisagreement Ġamong Ġmarket Ġparticipants Ġinduces Ġmore Ġtrades . ĠThe ĠâĢ ľ non - trade Ġtheorem âĢ Ŀ ĠB IB REF 1 Ġstates Ġthat Ġassuming Ġall Ġmarket Ġparticipants Ġhave Ġcommon Ġknowledge Ġabout Ġa Ġmarket Ġevent , Ġthe Ġlevel Ġof Ġdisagreement Ġamong Ġthe Ġparticipants Ġdoes Ġnot Ġincrease Ġthe Ġnumber Ġof Ġtrades Ġbut Ġonly Ġleads Ġto Ġa Ġrevision Ġof Ġthe Ġmarket Ġquotes . ĠIn Ġcontrast , Ġthe Ġtheoretically Ġframework Ġproposed Ġin ĠB IB REF 2 Ġadvocates Ġthat Ġdisagreement Ġamong Ġmarket Ġparticipants Ġincreases Ġtrading Ġvolume . ĠUsing Ġtextual Ġdata Ġfrom ĠYahoo Ġand ĠR aging Bull . com Ġmessage Ġboards Ġto Ġmeasure Ġthe Ġdisp ersion Ġof Ġopinions Ġ( positive Ġor Ġnegative ) Ġamong Ġtraders , Ġit Ġwas Ġshown Ġin ĠB IB REF 3 Ġthat Ġdisagreement Ġamong Ġusers ' Ġmessages Ġhelps Ġto Ġpredict Ġsubsequent Ġtrading Ġvolume Ġand Ġvolatility . ĠSimilar Ġrelation Ġbetween Ġdisagreement Ġand Ġincreased Ġtrading Ġvolume Ġwas Ġfound Ġin ĠB IB REF 4 Ġusing ĠTwitter Ġposts . ĠAdditionally , Ġtextual Ġanalysis Ġis Ġadding Ġto Ġthe Ġtheories Ġof Ġmedium - term / long - term Ġmomentum / re vers al Ġin Ġstock Ġmarkets ĠB IB REF 5 Ġ. ĠThe Ġunified ĠHong Ġand ĠStein Ġmodel ĠB IB REF 6 Ġon Ġstock 's Ġmomentum / re vers al Ġproposes Ġthat Ġinvestors Ġunder react Ġto Ġnews , Ġcausing Ġslow Ġprice Ġdr ifts , Ġand Ġover react Ġto Ġprice Ġshocks Ġnot Ġaccompanied Ġby Ġnews , Ġhence Ġinducing Ġrevers als . ĠThis Ġtheoretical Ġpred icated Ġbehaviour Ġbetween Ġprice Ġand Ġnews Ġwas Ġsystematically Ġestimated Ġand Ġsupported Ġin ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġusing Ġfinancial Ġmedia Ġheadlines Ġand Ġin ĠB IB REF 9 Ġusing Ġthe ĠConsumer ĠConf idence ĠIndex Â® Ġpublished Ġby ĠThe ĠConference ĠBoard ĠB IB REF 10 Ġ. ĠSimilarly , ĠB IB REF 11 Ġuses Ġthe ĠHarvard ĠIV - 4 Ġsentiment Ġlex icon Ġto Ġcount Ġthe Ġoccurrence Ġof Ġwords Ġwith Ġpositive Ġand Ġnegative Ġcon notation Ġof Ġthe ĠWall ĠStreet ĠJournal Ġshowing Ġthat Ġnegative Ġsentiment Ġis Ġa Ġgood Ġpredictor Ġof Ġprice Ġreturns Ġand Ġtrading Ġvolumes . </s> Acc urate Ġmodels Ġfor Ġforecasting Ġboth Ġprice Ġreturns Ġand Ġvolatility Ġare Ġequally Ġimportant Ġin Ġthe Ġfinancial Ġdomain . ĠVol atility Ġmeasures Ġhow Ġwildly Ġthe Ġasset Ġis Ġexpected Ġto Ġoscill ate Ġin Ġa Ġgiven Ġtime Ġperiod Ġand Ġis Ġrelated Ġto Ġthe Ġsecond Ġmoment Ġof Ġthe Ġprice Ġreturn Ġdistribution . ĠIn Ġgeneral Ġterms , Ġforecasting Ġprice Ġreturns Ġis Ġrelevant Ġto Ġtake Ġspeculative Ġpositions . ĠThe Ġvolatility , Ġon Ġthe Ġother Ġhand , Ġmeasures Ġthe Ġrisk Ġof Ġthese Ġpositions . ĠOn Ġa Ġdaily Ġbasis , Ġfinancial Ġinstitutions Ġneed Ġto Ġassess Ġthe Ġshort - term Ġrisk Ġof Ġtheir Ġportfolios . ĠMe asuring Ġthe Ġrisk Ġis Ġessential Ġin Ġmany Ġaspects . ĠIt Ġis Ġimperative Ġfor Ġregulatory Ġcapital Ġdisclosures Ġrequired Ġby Ġbanking Ġsupervision Ġbodies . ĠMoreover , Ġit Ġis Ġuseful Ġto Ġdynamically Ġadjust Ġposition Ġsizing Ġaccordingly Ġto Ġmarket Ġconditions , Ġthus , Ġmaintaining Ġthe Ġrisk Ġwithin Ġreasonable Ġlevels . </s> Although , Ġit Ġis Ġcrucial Ġto Ġpredict Ġthe Ġshort - term Ġvolatility Ġfrom Ġthe Ġfinancial Ġmarkets Ġapplication Ġperspective , Ġmuch Ġof Ġthe Ġcurrent ĠN LP Ġresearch Ġon Ġvolatility Ġforecasting Ġfocus Ġon Ġthe Ġvolatility Ġprediction Ġfor Ġvery Ġlong - term Ġhor izons Ġ( see ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ). ĠPred omin ately , Ġthese Ġworks Ġare Ġbuilt Ġon Ġextensions Ġof Ġthe Ġbag - of - words Ġrepresentation Ġthat Ġhas Ġthe Ġmain Ġdrawback Ġof Ġnot Ġcapturing Ġword Ġorder . ĠFinancial Ġforecasting , Ġhowever , Ġrequires Ġthe Ġability Ġto Ġcapture Ġsemantics Ġthat Ġis Ġdependent Ġupon Ġword Ġorder . ĠFor Ġexample , Ġthe Ġheadline ĠâĢ ľ Qual comm Ġsu es ĠApple Ġfor Ġcontract Ġbreach âĢ Ŀ Ġand ĠâĢ ľ Apple Ġsu es ĠQualcomm Ġfor Ġcontract Ġbreach âĢ Ŀ Ġtrigger Ġdifferent Ġresponses Ġfor Ġeach Ġstock Ġand Ġfor Ġthe Ġmarket Ġaggreg ated Ġindex , Ġhowever , Ġthey Ġshare Ġthe Ġsame Ġbag - of - words Ġrepresentation . ĠAdditionally , Ġthese Ġworks Ġuse Ġfeatures Ġfrom Ġa Ġpret rained Ġsentiment Ġanaly is Ġmodel Ġto Ġtrain Ġthe Ġfinancial Ġforecasting Ġmodel . ĠA Ġkey Ġlimitation Ġof Ġthis Ġprocess Ġis Ġthat Ġit Ġrequires Ġa Ġlabelled Ġsentiment Ġdataset . ĠAdditionally , Ġthe Ġerror Ġpropagation Ġis Ġnot Ġend - to - end . ĠIn Ġthis Ġwork , Ġwe Ġfill Ġin Ġthe Ġgaps Ġof Ġvolatility Ġprediction Ġresearch Ġin Ġthe Ġfollowing Ġmanner : </s> Related Ġwork </s> Previous Ġwork Ġin ĠB IB REF 12 Ġincorporates Ġsections Ġof Ġthe ĠâĢ ľ Form Ġ10 - K âĢ Ŀ Ġto Ġpredict Ġthe Ġvolatility Ġtwelve Ġmonths Ġafter Ġthe Ġreport Ġis Ġreleased . ĠThey Ġtrain Ġa ĠSupport ĠVector ĠReg ression Ġmodel Ġon Ġtop Ġof Ġsparse Ġrepresentation Ġ( bag - of - words ) Ġwith Ġstandard Ġterm Ġweight ing Ġ( e . g . ĠTerm - F requency ). ĠThis Ġwork Ġwas Ġextended Ġin ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġby Ġemploying Ġthe ĠL ough ran - Mc Donald ĠSent iment ĠWord ĠLists ĠB IB REF 20 Ġ, Ġwhich Ġcontain Ġthree Ġlists Ġwhere Ġwords Ġare Ġgrouped Ġby Ġtheir Ġsentiments Ġ( positive , Ġnegative Ġand Ġneutral ). ĠIn Ġall Ġthese Ġworks , Ġthe Ġtextual Ġrepresentation Ġis Ġengineered Ġusing Ġthe Ġfollowing Ġsteps : Ġ1 ) ĠFor Ġeach Ġsentiment Ġgroup , Ġthe Ġlist Ġis Ġexpanded Ġby Ġretrieving Ġ20 Ġmost Ġsimilar Ġwords Ġfor Ġeach Ġword Ġusing ĠWord 2 V ec Ġword Ġembed d ings ĠB IB REF 21 Ġ. Ġ2 ) ĠFinally , Ġeach Ġ10 - K Ġdocument Ġis Ġrepresented Ġusing Ġthe Ġexpanded Ġlists Ġof Ġwords . ĠThe Ġweight Ġof Ġeach Ġword Ġin Ġthis Ġsparse Ġrepresentation Ġis Ġdefined Ġusing ĠInformation ĠRet ri eval Ġ( IR ) Ġmethods Ġsuch Ġas Ġterm - frequency Ġ( tf ) Ġand Ġterm - frequency Ġwith Ġinverted Ġdocument Ġfrequency Ġ( tf id f ). ĠParticularly , ĠB IB REF 16 Ġshows Ġthat Ġresults Ġcan Ġbe Ġimproved Ġusing Ġenhanced ĠIR Ġmethods Ġand Ġprojecting Ġeach Ġsparse Ġfeature Ġinto Ġa Ġdense Ġspace Ġusing ĠPrincipal ĠComponent ĠAnalysis Ġ( PC A ). </s> The Ġworks Ġdescribed Ġabove Ġ( ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ) Ġtarget Ġlong - hor izon Ġvolatility Ġpredictions Ġ( one Ġyear Ġor Ġquarterly ĠB IB REF 16 Ġ). ĠIn Ġparticular , ĠB IB REF 16 Ġand ĠB IB REF 15 Ġuses Ġmarket Ġdata Ġ( price ) Ġfeatures Ġalong Ġwith Ġthe Ġtextual Ġrepresentation Ġof Ġthe Ġ10 - K Ġreports . ĠThese Ġexisting Ġworks Ġthat Ġemploy Ġmulti - mod al Ġlearning ĠB IB REF 22 Ġare Ġbased Ġon Ġa Ġlate Ġfusion Ġapproach . ĠFor Ġexample , Ġstacking Ġen semb les Ġto Ġtake Ġinto Ġaccount Ġthe Ġprice Ġand Ġtext Ġpredictions ĠB IB REF 16 Ġ. ĠIn Ġcontrast , Ġour Ġend - to - end Ġtrained Ġmodel Ġcan Ġlearn Ġthe Ġjoint Ġdistribution Ġof Ġboth Ġprice Ġand Ġtext . </s> P redict ing Ġthe Ġprice Ġdirection Ġrather Ġthan Ġthe Ġvolatility Ġwas Ġthe Ġfocus Ġin ĠB IB REF 23 Ġ. ĠThey Ġextracted Ġsentiment Ġwords Ġfrom ĠTwitter Ġposts Ġto Ġbuild Ġa Ġtime Ġseries Ġof Ġcollective ĠProfile Ġof ĠMood ĠStates Ġ( P OM S ). ĠTheir Ġresults Ġshow Ġthat Ġcollective Ġmood Ġaccurately Ġpredicts Ġthe Ġdirection Ġof ĠDown ĠJones Ġstock Ġindex Ġ( 86 . 7 % Ġaccuracy ). ĠIn ĠB IB REF 24 Ġhand crafted Ġtext Ġrepresentations Ġincluding Ġterm Ġcount , Ġnoun - phrase Ġtags Ġand Ġextracted Ġnamed Ġentities Ġare Ġemployed Ġfor Ġpredicting Ġstock Ġmarket Ġdirection Ġusing ĠSupport ĠVector ĠMachine Ġ( S VM ). ĠAn Ġextension Ġof ĠLat ent ĠDir ich let ĠAll ocation Ġ( L DA ) Ġis Ġproposed Ġin ĠB IB REF 25 Ġto Ġlearn Ġa Ġjoint Ġlatent Ġspace Ġof Ġtopics Ġand Ġsentiments . </s> Our Ġdeep Ġlearning Ġmodels Ġbear Ġa Ġclose Ġresemblance Ġto Ġworks Ġfocused Ġon Ġdirectional Ġprice Ġforecasting ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠIn ĠB IB REF 26 Ġ, Ġheadline Ġnews Ġare Ġprocessed Ġusing ĠStanford ĠOpen IE Ġto Ġgenerate Ġtri ples Ġthat Ġare Ġfed Ġinto Ġa ĠNeural ĠT ensor ĠNetwork Ġto Ġcreate Ġthe Ġfinal Ġheadline Ġrepresentation . ĠIn ĠB IB REF 27 Ġ, Ġa Ġcharacter - level Ġembed ding Ġis Ġpre - trained Ġin Ġan Ġun super vised Ġmanner . ĠThe Ġcharacter Ġembed ding Ġis Ġused Ġas Ġinput Ġto Ġa Ġsequence Ġmodel Ġto Ġlearn Ġthe Ġheadline Ġrepresentation . ĠParticularly , Ġboth Ġworks Ġaverage Ġall Ġheadline Ġrepresentations Ġin Ġa Ġgiven Ġday , Ġrather Ġthan Ġattempting Ġto Ġweight Ġthe Ġmost Ġrelevant Ġones . ĠIn Ġthis Ġwork , Ġwe Ġpropose Ġa Ġneural Ġattention Ġmechanism Ġto Ġcapture Ġthe ĠNews ĠRe lev ance Ġand Ġprovide Ġexperimental Ġevidence Ġthat Ġit Ġis Ġa Ġkey Ġcomponent Ġof Ġthe Ġend - to - end Ġlearning Ġprocess . ĠOur Ġattention Ġextends Ġthe Ġprevious Ġdeep Ġlearning Ġmethods Ġfrom ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. </s> Despite Ġthe Ġfact Ġthat Ġend - to - end Ġdeep Ġlearning Ġmodels Ġhave Ġattained Ġstate - of - the - art Ġperformance , Ġthe Ġlarge Ġnumber Ġof Ġparameters Ġmake Ġthem Ġprone Ġto Ġover fitting . ĠAdditionally , Ġend - to - end Ġmodels Ġare Ġtrained Ġfrom Ġscratch Ġrequiring Ġlarge Ġdatasets Ġand Ġcomputational Ġresources . ĠTransfer Ġlearning Ġ( TL ) Ġallev iates Ġthis Ġproblem Ġby Ġadapting Ġrepresentations Ġlearnt Ġfrom Ġa Ġdifferent Ġand Ġpotentially Ġweak ly Ġrelated Ġsource Ġdomain Ġto Ġthe Ġnew Ġtarget Ġdomain . ĠFor Ġexample , Ġin Ġcomputer Ġvision Ġtasks Ġthe Ġconv olution al Ġfeatures Ġlearnt Ġfrom ĠImage Net ĠB IB REF 28 Ġdataset Ġ( source Ġdomain ) Ġhave Ġbeen Ġsuccessfully Ġtransferred Ġto Ġmultiple Ġdomain Ġtarget Ġtasks Ġwith Ġmuch Ġsmaller Ġdatasets Ġsuch Ġas Ġobject Ġclassification Ġand Ġscene Ġrecognition ĠB IB REF 29 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġconsider ĠTL Ġin Ġour Ġexperiments Ġfor Ġtwo Ġmain Ġreasons . ĠFirst , Ġit Ġaddress Ġthe Ġquestion Ġwhether Ġour Ġproposed Ġdataset Ġis Ġsuitable Ġfor Ġend - to - end Ġtraining Ġsince Ġthe Ġperformance Ġof Ġthe Ġtransferred Ġrepresentations Ġcan Ġbe Ġcompared Ġwith Ġend - to - end Ġlearning . ĠSecond , Ġit Ġis Ġstill Ġto Ġbe Ġinvestigated Ġwhich Ġdataset Ġtransfers Ġbetter Ġto Ġthe Ġforecasting Ġproblem . ĠRecently , Ġthe ĠN LP Ġcommunity Ġhas Ġfocused Ġon Ġuniversal Ġrepresentations Ġof Ġsentences ĠB IB REF 17 Ġ, ĠB IB REF 19 Ġ, Ġwhich Ġare Ġdense Ġrepresentations Ġthat Ġcarry Ġthe Ġmeaning Ġof Ġa Ġfull Ġsentence . ĠB IB REF 17 Ġfound Ġthat Ġtransferring Ġthe Ġsentence Ġrepresentation Ġtrained Ġon Ġthe ĠStanford ĠNatural ĠLanguage ĠIn ference Ġ( SN LI ) ĠB IB REF 30 Ġdataset Ġachieves Ġstate - of - the - art Ġsentence Ġrepresentations Ġto Ġmultiple ĠN LP Ġtasks Ġ( e . g . Ġsentiment Ġanalysis , Ġquestion - type Ġand Ġopinion Ġpol arity ). ĠFollowing ĠB IB REF 17 Ġ, Ġin Ġthis Ġwork , Ġwe Ġinvestigate Ġthe Ġsuit ability Ġof ĠSN LI Ġand ĠReuters ĠRC V 1 ĠB IB REF 31 Ġdatasets Ġto Ġtransfer Ġlearning Ġto Ġthe Ġvolatility Ġforecasting Ġtask . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthe Ġhierarchical Ġattention Ġmechanism Ġat Ġheadline Ġlevel , Ġproposed Ġin Ġour Ġwork , Ġhas Ġnot Ġbeing Ġapplied Ġto Ġvolatility Ġprediction Ġso Ġfar ; Ġneither Ġhas Ġbeen Ġinvestigated Ġthe Ġability Ġto Ġtransfer Ġsentence Ġenc od ers Ġfrom Ġsource Ġdatasets Ġto Ġthe Ġtarget Ġforecasting Ġproblem Ġ( Transfer ĠLearning ). </s> Our Ġdataset </s> Our Ġcorpus Ġcovers Ġa Ġbroad Ġrange Ġof Ġnews Ġincluding Ġnews Ġaround Ġearnings Ġdates Ġand Ġcomple ments Ġthe Ġ10 - K Ġreports Ġcontent . ĠAs Ġan Ġillustration , Ġthe Ġheadlines ĠâĢ ľ Wal mart Ġwarns Ġthat Ġstrong ĠU . S . Ġdollar Ġwill Ġcost Ġ$ 15 B Ġin Ġsales âĢ Ŀ Ġand ĠâĢ ľ Pro ct er Ġ& ĠGamble ĠCo Ġraises ĠFY Ġorganic Ġsales Ġgrowth Ġforecast Ġafter Ġsales Ġbeat âĢ Ŀ Ġdescribe Ġthe Ġcompany Ġfinancial Ġconditions Ġand Ġperformance Ġfrom Ġthe Ġmanagement Ġpoint Ġof Ġview ĠâĢĵ Ġthese Ġare Ġalso Ġtypical Ġcontent Ġpresent Ġin ĠSection Ġ7 Ġof Ġthe Ġ10 - K Ġreports . </s> In Ġthis Ġsection , Ġwe Ġdescribe Ġthe Ġsteps Ġinvolved Ġin Ġcompiling Ġour Ġdataset Ġof Ġfinancial Ġnews Ġat Ġstock Ġlevel , Ġwhich Ġcomprises Ġa Ġbroad Ġrange Ġof Ġbusiness Ġsectors . </s> S ect ors Ġand Ġstocks </s> The Ġfirst Ġstep Ġin Ġcompiling Ġour Ġcorpus Ġwas Ġto Ġchoose Ġthe Ġconstituents Ġstocks . ĠOur Ġgoal Ġwas Ġto Ġconsider Ġstocks Ġin Ġa Ġbroad Ġrange Ġof Ġsectors , Ġaiming Ġa Ġdivers ified Ġfinancial Ġdomain Ġcorpus . ĠWe Ġfound Ġthat ĠExchange ĠTr aded ĠFunds Ġ( ETF ) Ġprovide Ġa Ġmechanical Ġway Ġto Ġaggregate Ġthe Ġmost Ġrelevant Ġstocks Ġin Ġa Ġgiven Ġindustry / sector . ĠAn ĠETF Ġis Ġa Ġfund Ġthat Ġowns Ġassets , Ġe . g . Ġstock Ġshares Ġor Ġcurrencies , Ġbut , Ġunlike Ġmutual Ġfunds Ġare Ġtraded Ġin Ġstock Ġexchanges . ĠThese ĠETF s Ġare Ġextremely Ġliquid Ġand Ġtrack Ġdifferent Ġinvestment Ġthemes . ĠWe Ġdecided Ġto Ġuse ĠSPD R ĠSet cor ĠFunds Ġconstituents Ġstocks Ġin Ġour Ġwork Ġsince Ġthe Ġcompany Ġis Ġthe Ġlargest Ġprovider Ġof Ġsector Ġfunds Ġin Ġthe ĠUnited ĠStates . ĠWe Ġincluded Ġin Ġour Ġanalysis Ġthe Ġtop Ġ5 Ġ( five ) Ġsector ĠETF s Ġby Ġfinancial Ġtrading Ġvolume Ġ( as Ġin ĠJan / 2018 ). ĠAmong Ġthe Ġmost Ġtraded Ġsectors Ġwe Ġalso Ġfiltered Ġout Ġthe Ġsectors Ġthat Ġwere Ġsimilar Ġto Ġeach Ġother . ĠFor Ġexample , Ġthe ĠConsumer ĠStaples Ġand ĠConsumer ĠDisc retion ary Ġsectors Ġare Ġboth Ġpart Ġof Ġthe Ġparent ĠConsumer Ġcategory . ĠFor Ġeach Ġof Ġthe Ġtop Ġ5 Ġsectors Ġwe Ġselected Ġthe Ġtop Ġ10 Ġholdings , Ġwhich Ġare Ġdeemed Ġthe Ġmost Ġrelevant Ġstocks . Ġt bl : stock un iverse , Ġdetails Ġour Ġdataset Ġsectors Ġand Ġits Ġrespective Ġstocks . </s> Stock Ġspecific Ġdata </s> We Ġassume Ġthat Ġan Ġindividual Ġstock Ġnews Ġas Ġthe Ġone Ġthat Ġexplicitly Ġmention Ġthe Ġstock Ġname Ġor Ġany Ġof Ġits Ġsurface Ġforms Ġin Ġthe Ġheadline . ĠAs Ġan Ġillustration , Ġin Ġorder Ġto Ġcollect Ġall Ġnews Ġfor Ġthe Ġstock Ġcode ĠPG , ĠPro ct er Ġ& ĠGamble Ġcompany Ġname , Ġwe Ġsearch Ġall Ġthe Ġheadlines Ġwith Ġany Ġof Ġthese Ġwords : ĠPro ct er & Gam ble ĠOR ĠPro ct er Ġand ĠGamble ĠOR ĠP & G . ĠIn Ġthis Ġexample , Ġthe Ġfirst Ġword Ġis Ġjust Ġthe Ġcompany Ġname Ġand Ġthe Ġremaining Ġwords Ġare Ġthe Ġcompany Ġsurface Ġforms . </s> We Ġautomatically Ġderived Ġthe Ġsurface Ġforms Ġfor Ġeach Ġstock Ġby Ġstarting Ġwith Ġa Ġseed Ġof Ġsurface Ġforms Ġextracted Ġfrom Ġthe ĠDB pedia ĠKnowledge ĠBase Ġ( KB ). ĠWe Ġthen Ġapplied Ġthe Ġfollowing Ġprocedure : </s> Rel ate Ġeach Ġcompany Ġname Ġwith Ġthe ĠKB Ġentity Ġunique Ġidentifier . </s> Ret rieve Ġall Ġvalues Ġof Ġthe Ġwiki Page Red irect s Ġproperty . ĠThe Ġproperty Ġholds Ġthe Ġnames Ġof Ġdifferent Ġpages Ġthat Ġpoints Ġto Ġthe Ġsame Ġentity / company Ġname . ĠThis Ġstep Ġsets Ġthe Ġinitial Ġseed Ġof Ġsurface Ġforms . </s> Man ually , Ġfilter Ġout Ġsome Ġnoisy Ġproperty Ġvalues . ĠFor Ġinstance , Ġfrom Ġthe ĠPro ct er Ġ& ĠG lam ble Ġentity Ġpage Ġwe Ġwere Ġable Ġto Ġautomatically Ġextract Ġd br : Pro ct er _ and _ gam ble Ġand Ġd br : P _ & _ G , Ġbut Ġhad Ġto Ġmanually Ġexclude Ġthe Ġnoisy Ġassociations Ġd br : Female _ p ads Ġand Ġd br : California _ Natural . </s> The Ġresult Ġof Ġthe Ġsteps Ġabove Ġis Ġa Ġdictionary Ġof Ġsurface Ġforms Ġ$ wd _{ sc } $ Ġ. </s> Stock Ġheadlines </s> Our Ġcorpus Ġis Ġbuilt Ġat Ġstock Ġcode Ġlevel Ġby Ġcollecting Ġheadlines Ġfrom Ġthe ĠReuters ĠArchive . ĠThis Ġarchive Ġgroups Ġthe Ġheadlines Ġby Ġdate , Ġstarting Ġfrom Ġ1 ĠJanuary Ġ2007 . ĠEach Ġheadline Ġis Ġa Ġhtml Ġlink Ġ(< a Ġhref > Ġtag ) Ġto Ġthe Ġfull Ġbody Ġof Ġthe Ġnews , Ġwhere Ġthe Ġanchor Ġtext Ġis Ġthe Ġheadline Ġcontent Ġfollowed Ġby Ġthe Ġrelease Ġtime . ĠFor Ġexample , Ġthe Ġpage Ġdated Ġ16 ĠDec Ġ2016 Ġhas Ġthe Ġheadline ĠâĢ ľ Pro ct er Ġ& ĠGamble Ġappoint s ĠNelson ĠP elt z Ġto Ġboard Ġ5 : 26 PM ĠUTC âĢ Ŀ . </s> For Ġeach Ġof Ġthe Ġ50 Ġstocks Ġ( 5 Ġsectors Ġtimes Ġ10 Ġstocks Ġper Ġsector ) Ġselected Ġusing Ġthe Ġcriteria Ġdescribed Ġin Ġsub : cor p us sec stock , Ġwe Ġretrieved Ġall Ġthe Ġheadlines Ġfrom Ġthe ĠReuters ĠArchive Ġraging Ġfrom Ġ01 / 01 / 2007 Ġto Ġ30 / 12 / 2017 . ĠThis Ġprocess Ġtakes Ġthe Ġfollowing Ġsteps : </s> For Ġa Ġgiven Ġstock Ġcode Ġ( Ġ$ sc $ Ġ) Ġretrieve Ġall Ġsurface Ġforms Ġ$ wd _{ sc } $ Ġ. </s> For Ġeach Ġday , Ġstore Ġonly Ġthe Ġheadlines Ġcontent Ġmatching Ġany Ġword Ġin Ġ$ wd _{ sc } $ Ġ. ĠFor Ġeach Ġstored Ġheadline Ġwe Ġalso Ġstore Ġthe Ġtime Ġand Ġtime zone . </s> Con vert Ġthe Ġnews Ġdate Ġand Ġtime Ġto ĠEastern ĠDaylight ĠTime Ġ( ED T ). </s> C ategor ize Ġthe Ġnews Ġrelease Ġtime . ĠWe Ġconsider Ġthe Ġfollowing Ġcategory Ġset : Ġ{ before Ġmarket , Ġduring Ġmarket Ġ, Ġafter Ġmarket , Ġholidays , Ġweekends }. Ġduring Ġmarket Ġcontains Ġnews Ġbetween Ġ9 : 30 AM Ġand Ġ4 : 00 PM . Ġbefore Ġmarket Ġbefore Ġ9 : 30 AM Ġand Ġafter Ġmarket Ġafter Ġ4 : 00 PM . </s> The Ġtime Ġcategories Ġprevents Ġany Ġmis al ignment Ġbetween Ġtext Ġand Ġstock Ġprice Ġdata . ĠMoreover , Ġit Ġprevents Ġdata Ġleakage Ġand , Ġconsequently , Ġunrealistic Ġpredictive Ġmodel Ġperformance . ĠIn Ġgeneral , Ġnews Ġreleased Ġafter Ġ4 : 00 PM ĠEDT Ġcan Ġdrastically Ġchange Ġmarket Ġexpectations Ġand Ġthe Ġreturns Ġcalculated Ġusing Ġclose Ġto Ġclose Ġprices Ġas Ġin Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel Ġ( see Ġeq : cl osing return ). ĠFollowing ĠB IB REF 3 Ġ, Ġto Ġdeal Ġwith Ġnews Ġmis al ignment , Ġnews Ġissued Ġafter Ġ4 : 00 PM Ġ( after Ġmarket ) Ġare Ġgrouped Ġwith Ġthe Ġpre - market Ġ( before Ġmarket ) Ġon Ġthe Ġfollowing Ġtrading Ġday . </s> t bl : stock time cat Ġshows Ġthe Ġdistribution Ġof Ġnews Ġper Ġsector Ġfor Ġeach Ġtime Ġcategory . ĠWe Ġcan Ġsee Ġa Ġhigh Ġconcentration Ġof Ġnews Ġreleased Ġbefore Ġthe Ġmarket Ġopens Ġ( 55 % Ġon Ġaverage ). ĠIn Ġcontrast , Ġusing Ġa Ġcorpus Ġcompiled Ġfrom Ġmessage Ġboards , Ġa Ġlarge Ġoccurrence Ġof Ġnews Ġduring Ġmarket Ġhours Ġwas Ġfound ĠB IB REF 3 Ġ. ĠThis Ġbehaviour Ġindicating Ġday Ġtraders ' Ġactivity . ĠOur Ġcorpus Ġcomprise Ġfinancial Ġnews Ġagency Ġheadlines , Ġa Ġcontent Ġmore Ġfocused Ġon Ġcorporate Ġevents Ġ( e . g . Ġlawsuits , Ġmer ges Ġ& Ġacquisitions , Ġresearch Ġ& Ġdevelopment ) Ġand Ġon Ġeconomic Ġnews Ġ( see Ġt bl : stock head lines ex ma ples Ġfor Ġa Ġsample Ġof Ġour Ġdataset ). ĠThese Ġheadlines Ġare Ġmostly Ġfactual . ĠOn Ġthe Ġother Ġhand , Ġuser - generated Ġcontent Ġsuch Ġas ĠTwitter Ġand Ġmessage Ġboards Ġ( as Ġin ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ) Ġtends Ġto Ġbe Ġmore Ġsubjective . </s> U . S . Ġmacro economic Ġindicators Ġsuch Ġas ĠRetail ĠSales , ĠJob less ĠClaims Ġand ĠGDP Ġare Ġmostly Ġreleased Ġaround Ġ8 : 30 AM Ġ( one Ġhour Ġbefore Ġthe Ġmarket Ġopens ). ĠThese Ġnumbers Ġare Ġkey Ġdrivers Ġof Ġmarket Ġactivity Ġand , Ġas Ġsuch , Ġhave Ġa Ġhigh Ġmedia Ġcoverage . ĠSpecific Ġsections Ġof Ġthese Ġeconomic Ġreports Ġimpact Ġseveral Ġstocks Ġand Ġsectors . ĠAnother Ġfactor Ġthat Ġcontribute Ġto Ġthe Ġhigh Ġactivity Ġof Ġnews Ġoutside Ġregular Ġtrading Ġhours Ġare Ġcompany Ġearnings Ġreports . ĠThese Ġare Ġrarely Ġreleased Ġduring Ġtrading Ġhours . ĠFinally , Ġbefore Ġthe Ġmarket Ġopens Ġnews Ġagencies Ġprovide Ġa Ġsummary Ġof Ġthe Ġinternational Ġmarkets Ġdevelopments , Ġe . g . Ġthe Ġkey Ġfacts Ġduring Ġthe ĠAsian Ġand ĠAustralian Ġtrading Ġhours . ĠAll Ġthese Ġfactors Ġcontribute Ġto Ġthe Ġhigh Ġconcentration Ġof Ġpre - market Ġnews . </s> Background </s> We Ġstart Ġthis Ġsection Ġby Ġreviewing Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel , Ġwhich Ġis Ġa Ġstrong Ġbenchmark Ġused Ġto Ġevaluate Ġour Ġneural Ġmodel . ĠWe Ġthen Ġreview Ġthe Ġsource Ġdatasets Ġproposed Ġin Ġthe Ġliterature Ġthat Ġwere Ġtrained Ġindependently Ġand Ġtransf ered Ġto Ġour Ġvolatility Ġprediction Ġmodel . ĠFinally , Ġwe Ġreview Ġthe Ġgeneral Ġarchitectures Ġof Ġsequence Ġmodelling Ġand Ġattention Ġmechanisms . </s> G ARCH Ġmodel </s> Financial Ġinstitutions Ġuse Ġthe Ġconcept Ġof ĠâĢ ľ Value Ġat Ġrisk âĢ Ŀ Ġto Ġmeasure Ġthe Ġexpected Ġvolatility Ġof Ġtheir Ġportfolios . ĠThe Ġwidespread Ġe con ometric Ġmodel Ġfor Ġvolatility Ġforecasting Ġis Ġthe ĠGeneral ized ĠAut ore gressive ĠCond itional ĠH eter os ked astic ity Ġ( G ARCH ) ĠB IB REF 32 Ġ, ĠB IB REF 33 Ġ. ĠPrevious Ġresearch Ġshows Ġthat Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel Ġis Ġhard Ġto Ġbeat . ĠFor Ġexample , ĠB IB REF 34 Ġcompared ĠG ARCH ( 1 , 1 ) Ġwith Ġ330 Ġdifferent Ġe con ometric Ġvolatility Ġmodels Ġshowing Ġthat Ġthey Ġare Ġnot Ġsignificantly Ġbetter Ġthan ĠG ARCH ( 1 , 1 ). ĠLet Ġ$ p _ t $ Ġbe Ġthe Ġprice Ġof Ġan Ġstock Ġat Ġthe Ġend Ġof Ġa Ġtrading Ġperiod Ġwith Ġclosing Ġreturns Ġ$ r _ t $ Ġgiven Ġby </s> $$ r _ t Ġ= Ġ\ frac { p _ t }{ p _{ t - 1 }} Ġ- Ġ1 Ġ$$ Ġ Ġ Ġ( E q . Ġ29 ) </s> The ĠG ARCH Ġprocess Ġexplicitly Ġmodels Ġthe Ġtime - v ary ing Ġvolatility Ġof Ġasset Ġreturns . ĠIn Ġthe ĠG ARCH ( 1 , 1 ) Ġspecification Ġthe Ġreturns Ġseries Ġ$ r _ t $ Ġfollow Ġthe Ġprocess : </s> $$ r _ t Ġ& = Ġ\ mu Ġ+ Ġ\ ep sil on Ġ_ t Ġ Ġ\\ Ġ\ ep sil on Ġ_ t Ġ& = Ġ\ s igma Ġ_ t Ġz _ t Ġ Ġ\\ Ġ\ s igma Ġ^ 2 _ t Ġ& = Ġa _ 0 Ġ+ Ġa _ 1 Ġ\ ep sil on Ġ_ { t - 1 } ^ 2 Ġ+ Ġb _ 1 Ġ\ s igma Ġ_ { t - 1 } ^ 2 $$ Ġ Ġ Ġ( E q . Ġ30 ) </s> where Ġ$\ mu Ġ$ Ġis Ġa Ġconstant Ġ( return Ġdrift ) Ġand Ġ$ z _ t $ Ġis Ġa Ġsequence Ġof Ġi . i . d . Ġrandom Ġvariables Ġwith Ġmean Ġzero Ġand Ġunit Ġvariance . ĠIt Ġis Ġworth Ġnoting Ġthat Ġalthough Ġthe Ġconditional Ġmean Ġreturn Ġdescribed Ġin Ġeq : g arch cond mean Ġhas Ġa Ġconstant Ġvalue , Ġthe Ġconditional Ġvolatility Ġ$\ s igma Ġ_ t $ Ġis Ġtime - dependent Ġand Ġmodeled Ġby Ġeq : att . </s> The Ġone - step Ġahead Ġexpected Ġvolatility Ġforecast Ġcan Ġbe Ġcomputed Ġdirectly Ġfrom Ġeq : g arch cond vari ance Ġand Ġis Ġgiven Ġby </s> $$ E _ T [ \ s igma Ġ_ { T + 1 } ^ 2 ] Ġ= Ġa _ 0 Ġ+ Ġa _ 1 ĠE _ T [ \ ep sil on Ġ^ 2 ] Ġ+ Ġb _ 1 ĠE _ T [ \ s igma Ġ_ { T } ^ 2 ] Ġ$$ Ġ Ġ Ġ( E q . Ġ32 ) </s> In Ġgeneral , Ġthe Ġ$ t ^ {\ prime Ġ} $ Ġ- steps Ġahead Ġexpected Ġvolatility Ġ$ E _ T [ \ s igma Ġ_ { T + t ^ {\ prime Ġ}} ^ 2 ] $ Ġcan Ġbe Ġeasily Ġexpressed Ġin Ġterms Ġof Ġthe Ġprevious Ġstep Ġexpected Ġvolatility . ĠIt Ġis Ġeasy Ġto Ġprove Ġby Ġinduction Ġthat Ġthe Ġforecast Ġfor Ġany Ġhorizon Ġcan Ġbe Ġrepresented Ġin Ġterms Ġof Ġthe Ġone - step Ġahead Ġforecast Ġand Ġis Ġgiven Ġby </s> $$ E _ T [ \ s igma Ġ_ { T + t ^ {\ prime Ġ}} ^ 2 ] Ġ- Ġ\ s igma Ġ_ u ^ 2 Ġ= Ġ( a _ 1 Ġ+ Ġb _ 1 ) ^{ ( t ^ {\ prime Ġ} Ġ- 1 )} Ġ\ left ( E _ T [ \ s igma Ġ_ { T + 1 } ^ 2 ] Ġ- Ġ\ s igma Ġ_ u ^ 2 \ right ) $$ Ġ Ġ Ġ( E q . Ġ33 ) </s> where Ġ$\ s igma Ġ_ u $ Ġis Ġthe Ġunconditional Ġvolatility : </s> $$ \ s igma Ġ_ u Ġ= Ġ\ sq rt { a _ 0 Ġ/ Ġ( 1 Ġ- Ġa _ 1 Ġ- Ġb _ 1 )} Ġ$$ Ġ Ġ Ġ( E q . Ġ34 ) </s> From Ġthe Ġequation Ġabove Ġwe Ġcan Ġsee Ġthat Ġfor Ġlong Ġhor izons , Ġi . e . Ġ$ t ^ \ prime Ġ\ right arrow Ġ\ in fty Ġ$ Ġ, Ġthe Ġvolatility Ġforecast Ġin Ġeq : fore cast rec ursive Ġconver ges Ġto Ġthe Ġunconditional Ġvolatility Ġin Ġeq : un var . </s> All Ġthe Ġworks Ġreviewed Ġin Ġsec : introdu ction Ġ( ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ, ĠB IB REF 16 Ġ) Ġconsider ĠG ARCH ( 1 , 1 ) Ġbenchmark . ĠHowever , Ġgiven Ġthe Ġlong Ġhorizon Ġof Ġtheir Ġpredictions Ġ( e . g . Ġquarterly Ġor Ġannual ), Ġthe Ġmodels Ġare Ġevaluated Ġusing Ġthe Ġunconditional Ġvolatility Ġ$\ s igma Ġ_ u $ Ġin Ġeq : un var . ĠIn Ġthis Ġwork , Ġwe Ġfocus Ġon Ġthe Ġshort - term Ġvolatility Ġprediction Ġand Ġuse Ġthe ĠG ARCH ( 1 , 1 ) Ġone - day Ġahead Ġconditional Ġvolatility Ġprediction Ġin Ġeq : fore cast one period Ġto Ġevaluate Ġour Ġmodels . </s> Let Ġ$\ s igma Ġ_ { t + 1 } $ Ġdenote Ġthe Ġex - post ĠâĢ ľ true âĢ Ŀ Ġdaily Ġvolatility Ġat Ġa Ġgiven Ġtime Ġ$ t $ Ġ. ĠThe Ġperformance Ġon Ġa Ġset Ġwith Ġ$ N $ Ġdaily Ġsamples Ġcan Ġbe Ġevaluated Ġusing Ġthe Ġstandard ĠMean ĠSqu ared ĠError Ġ( Ġ$ M SE $ Ġ) Ġand ĠMean ĠAbsolute ĠError Ġ( Ġ$ MA E $ Ġ) </s> $$ M SE Ġ& = Ġ\ frac { 1 }{ N } Ġ\ sum Ġ_ { t = 1 } ^{ N } Ġ\ left ( ĠE _ t [ \ s igma Ġ_ { t + 1 } ] Ġ- Ġ\ s igma Ġ_ { t + 1 }\ right ) ^ 2 Ġ Ġ\\ ĠMA E Ġ& = Ġ\ frac { 1 }{ N } Ġ\ sum Ġ_ { t = 1 } ^{ N }\ left | E _ t [ \ s igma Ġ_ { t + 1 } ] Ġ- Ġ\ s igma Ġ_ { t + 1 } Ġ\ right | $$ Ġ Ġ Ġ( E q . Ġ36 ) </s> Additionally , Ġfollowing ĠB IB REF 35 Ġ, Ġthe Ġmodels Ġare Ġalso Ġevaluated Ġusing Ġthe Ġcoefficient Ġof Ġdetermination Ġ$ R ^ 2 $ Ġof Ġthe Ġregression </s> $$ \ s igma Ġ_ { t + 1 } Ġ= Ġa Ġ+ Ġb ĠE _ t [ \ s igma Ġ_ { t + 1 } ] Ġ+ Ġe _ t $$ Ġ Ġ Ġ( E q . Ġ37 ) </s> where </s> $$ R ^ 2 Ġ= Ġ1 Ġ- Ġ\ frac {\ sum Ġ_ { t = 1 } ^{ N } e ^{ 2 } _{ t }} {\ sum Ġ_ { t = 1 } ^{ N }\ left ( E _ t [ \ s igma Ġ_ { t + 1 } ] Ġ- Ġ\ frac { 1 }{ N } Ġ\ sum Ġ_ { t = 1 } ^{ N } E _ t [ \ s igma Ġ_ { t + 1 } ] \ right ) ^{ 2 }} $$ Ġ Ġ Ġ( E q . Ġ38 ) </s> One Ġof Ġthe Ġchallenges Ġin Ġevaluating ĠG ARCH Ġmodels Ġis Ġthe Ġfact Ġthat Ġthe Ġex - post Ġvolatility Ġ$\ s igma Ġ_ { t + 1 } $ Ġis Ġnot Ġdirectly Ġobserved . ĠApparently , Ġthe Ġsquared Ġdaily Ġreturns Ġ$ r _{ t + 1 } ^{ 2 } $ Ġin Ġeq : cl osing return Ġcould Ġstand Ġas Ġa Ġgood Ġproxy Ġfor Ġthe Ġex - post Ġvolatility . ĠHowever , Ġthe Ġsquared Ġreturns Ġyield Ġvery Ġnoisy Ġmeasurements . ĠThis Ġis Ġa Ġdirect Ġconsequence Ġof Ġthe Ġterm Ġ$ z ^ t $ Ġthat Ġconnects Ġthe Ġsquared Ġreturn Ġto Ġthe Ġlatent Ġvolatility Ġfactor Ġin Ġeq : g arch wh it eno ise . ĠThe Ġuse Ġof Ġint rad ay Ġprices Ġto Ġestimate Ġthe Ġex - post Ġdaily Ġvol ay ility Ġwas Ġfirst Ġproposed Ġin ĠB IB REF 35 Ġ. ĠThey Ġargue Ġthat Ġvolatility Ġestim ators Ġusing Ġint rad ay Ġprices Ġis Ġthe Ġproper Ġway Ġto Ġevaluate Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel , Ġas Ġopposed Ġto Ġsquared Ġdaily Ġreturns . ĠFor Ġexample , Ġconsidering Ġthe ĠDeutsche ĠMark Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel Ġ$ R ^ 2 $ Ġimproves Ġfrom Ġ$ 0 . 047 $ Ġ( squ ared Ġreturns ) Ġto Ġ$ 0 . 33 $ Ġ( int rad ay Ġreturns ) ĠB IB REF 35 Ġ. </s> It Ġis Ġclear Ġfrom Ġthe Ġprevious Ġsection Ġthat Ġany Ġvolatility Ġmodel Ġevaluation Ġusing Ġthe Ġnoisy Ġsquared Ġreturns Ġas Ġthe Ġex - post Ġvolatility Ġproxy Ġwill Ġlead Ġto Ġvery Ġpoor Ġperformance . ĠTherefore , Ġhigh - frequency Ġint rad ay Ġdata Ġis Ġfundamental Ġto Ġshort - term Ġvolatility Ġperformance Ġevaluation . ĠHowever , Ġint rad ay Ġdata Ġis Ġdifficult Ġto Ġacquire Ġand Ġcostly . ĠFortunately , Ġthere Ġare Ġstatistically Ġefficient Ġdaily Ġvolatility Ġestim ators Ġthat Ġonly Ġdepend Ġon Ġthe Ġopen , Ġhigh , Ġlow Ġand Ġclose Ġprices . ĠThese Ġprice ĠâĢ ľ r anges âĢ Ŀ Ġare Ġwidely Ġavailable . ĠIn Ġthis Ġsection , Ġwe Ġdiscuss Ġthese Ġestim ators . </s> Let Ġ$ O _ t $ Ġ, Ġ$ H _ t $ Ġ, Ġ$ L _ t $ Ġ, Ġ$ C _ t $ Ġbe Ġthe Ġopen , Ġhigh , Ġlow Ġand Ġclose Ġprices Ġof Ġan Ġasset Ġin Ġa Ġgiven Ġday Ġ$ t $ Ġ. ĠAssuming Ġthat Ġthe Ġdaily Ġprice Ġfollows Ġa Ġgeometric ĠBrown ian Ġmotion Ġwith Ġzero Ġdrift Ġand Ġconstant Ġdaily Ġvolatility Ġ$\ s igma Ġ$ Ġ, ĠParkinson Ġ( 1980 ) Ġderived Ġthe Ġfirst Ġdaily Ġvolatility Ġestim ator </s> $$ \ wide hat {\ s igma Ġ_ { PK , t } ^ 2 } Ġ= Ġ\ frac {\ ln Ġ\ left (\ frac { H _ t }{ L _ t }\ right ) ^ 2 }{ 4 \ ln Ġ( 2 )} Ġ$$ Ġ Ġ Ġ( E q . Ġ41 ) </s> which Ġrepresents Ġthe Ġdaily Ġvolatility Ġin Ġterms Ġof Ġits Ġprice Ġrange . ĠHence , Ġit Ġcontains Ġinformation Ġabout Ġthe Ġprice Ġpath . ĠGiven Ġthis Ġproperty , Ġit Ġis Ġexpected Ġthat Ġ$\ s igma Ġ_ { PK } $ Ġis Ġless Ġnoisy Ġthan Ġthe Ġvolatility Ġcalculated Ġusing Ġsquared Ġreturns . ĠThe ĠParkinson 's Ġvolatility Ġestim ator Ġwas Ġextended Ġby ĠGar man - K lass Ġ( 1980 ) Ġwhich Ġincorporates Ġadditional Ġinformation Ġabout Ġthe Ġopening Ġ( Ġ$ O _ t $ Ġ) Ġand Ġclosing Ġ( Ġ$ C _ t $ Ġ) Ġprices Ġand Ġis Ġdefined Ġas </s> $$ \ wide hat {\ s igma Ġ_ { G K , t } ^{ 2 }} Ġ= Ġ\ frac { 1 }{ 2 } Ġ\ ln Ġ\ left (\ frac { H _ t }{ L _ t }\ right ) ^ 2 Ġ- Ġ( 2 \ ln Ġ( 2 ) Ġ- Ġ1 ) Ġ\ ln Ġ\ left (\ frac { C _ t }{ O _ t }\ right ) ^ 2 Ġ$$ Ġ Ġ Ġ( E q . Ġ42 ) </s> The Ġrelative Ġnoisy Ġof Ġdifferent Ġestim ators Ġ$\ hat {\ s igma Ġ} $ Ġcan Ġbe Ġmeasured Ġin Ġterms Ġof Ġits Ġrelative Ġefficiency Ġto Ġthe Ġdaily Ġvolatility Ġ$\ s igma Ġ$ Ġand Ġis Ġdefined Ġas </s> $$ e \ left (\ wide hat {\ s igma Ġ^ { 2 } }, Ġ\ s igma Ġ^ 2 \ right ) Ġ\ equ iv Ġ\ frac { Var [ \ s igma Ġ^ 2 ] }{ Var [ \ wide hat {\ s igma Ġ^ { 2 }} ]} $$ Ġ Ġ Ġ( E q . Ġ43 ) </s> where Ġ$ Var [ \ cd ot Ġ] $ Ġis Ġthe Ġvariance Ġoperator . ĠIt Ġfollows Ġdirectly Ġfrom Ġeq : g arch wh it eno ise Ġthat Ġthe Ġsquared Ġreturn Ġhas Ġefficiency Ġ1 Ġand Ġtherefore , Ġvery Ġnoisy . ĠB IB REF 36 Ġreports ĠParkinson Ġ( Ġ$\ wide hat {\ s igma Ġ_ { PK , t } ^ 2 } $ Ġ) Ġvolatility Ġestim ator Ġhas Ġ4 . 9 Ġrelative Ġefficiency Ġand ĠGar man - K lass Ġ( Ġ$\ wide hat {\ s igma Ġ_ { G K , t } ^ 2 } $ Ġ) Ġ7 . 4 . ĠAdditionally , Ġall Ġthe Ġdescribed Ġestim ators Ġare Ġunbiased . </s> Many Ġalternative Ġestim ators Ġto Ġdaily Ġvolatility Ġhave Ġbeen Ġproposed Ġin Ġthe Ġliterature . ĠHowever , Ġexperiments Ġin ĠB IB REF 36 Ġrate Ġthe ĠGar man - K lass Ġvolatility Ġestim ator Ġas Ġthe Ġbest Ġvolatility Ġestim ator Ġbased Ġonly Ġon Ġopen , Ġhigh , Ġlow Ġand Ġclose Ġprices . ĠIn Ġthis Ġwork , Ġwe Ġtrain Ġour Ġmodels Ġto Ġpredict Ġthe Ġstate - of - the - art ĠGar man - K lass Ġestim ator . ĠMoreover , Ġwe Ġevaluate Ġour Ġmodels Ġand ĠG ARCH ( 1 , 1 ) Ġusing Ġthe Ġmetrics Ġdescribed Ġin Ġsub : eval ution , Ġbut Ġwith Ġthe Ġappropriate Ġvolatility Ġproxies , Ġi . e . ĠParkinson Ġand ĠGar man - K lass Ġestim ators . </s> Transfer ĠLearning Ġfrom Ġother Ġsource Ġdomains </s> Vector Ġrepresentations Ġof Ġwords , Ġalso Ġknown Ġas ĠWord Ġembed d ings ĠB IB REF 21 Ġ, ĠB IB REF 37 Ġ, Ġthat Ġrepresent Ġa Ġword Ġas Ġa Ġdense Ġvector Ġhas Ġbecome Ġthe Ġstandard Ġbuilding Ġblocks Ġof Ġalmost Ġall ĠN LP Ġtasks . ĠThese Ġembed d ings Ġare Ġtrained Ġon Ġlarge Ġunl abel ed Ġcorpus Ġand Ġare Ġable Ġto Ġcapture Ġcontext Ġand Ġsimilarity Ġamong Ġwords . </s> Some Ġattempts Ġhave Ġbeen Ġmade Ġto Ġlearn Ġvector Ġrepresentations Ġof Ġa Ġfull Ġsentence , Ġrather Ġthan Ġonly Ġa Ġsingle Ġword , Ġusing Ġun super vised Ġapproaches Ġsimilar Ġin Ġnature Ġto Ġword Ġembed d ings . ĠRecently , ĠB IB REF 17 Ġshowed Ġstate - of - the - art Ġperformance Ġwhen Ġa Ġsentence Ġenc oder Ġis Ġtrained Ġend - to - end Ġon Ġa Ġsupervised Ġsource Ġtask Ġand Ġtransferred Ġto Ġother Ġtarget Ġtasks . ĠInspired Ġby Ġthis Ġwork , Ġwe Ġinvestigate Ġthe Ġperformance Ġof Ġsentence Ġenc od ers Ġtrained Ġon Ġthe ĠText Ġcategor ization Ġand ĠNatural ĠLanguage ĠIn ference Ġ( N LI ) Ġtasks Ġand Ġuse Ġthese Ġenc od ers Ġin Ġour Ġmain Ġshort - term Ġvolatility Ġprediction Ġtask . </s> A Ġgeneric Ġsentence Ġenc oder Ġ$ S _ e $ Ġreceives Ġthe Ġsentence Ġwords Ġas Ġinput Ġand Ġreturns Ġa Ġvector Ġrepresenting Ġthe Ġsentence . ĠThis Ġcan Ġbe Ġexpressed Ġas Ġa Ġmapping </s> $$ S _ e Ġ\ col on Ġ\ math bb Ġ{ R } ^{ T ^{ S } Ġ\ times Ġd _ w } Ġ\ right arrow Ġ\ math bb Ġ{ R } ^{ d _ S } $$ Ġ Ġ Ġ( E q . Ġ45 ) </s> from Ġa Ġvariable Ġsize Ġsequence Ġof Ġwords Ġto Ġa Ġsentence Ġvector Ġ$ S $ Ġof Ġfixed - size Ġ$ d _ S $ Ġ, Ġwhere Ġ$ T ^{ S } $ Ġis Ġthe Ġsentence Ġnumber Ġof Ġwords Ġand Ġ$ d _ w $ Ġis Ġthe Ġpre - trained Ġword Ġembed ding Ġdimension . </s> In Ġthe Ġfollowing Ġsections , Ġwe Ġdescribe Ġthe Ġdatasets Ġand Ġarchitectures Ġto Ġtrain Ġthe Ġsentence Ġenc od ers Ġof Ġthe Ġauxiliary Ġtransfer Ġlearning Ġtasks . </s> The ĠReuters ĠCorpus ĠVolume ĠI Ġ( RC V 1 ) Ġis Ġcorpus Ġcontaining Ġ8 06 , 79 1 Ġnews Ġarticles Ġin Ġthe ĠEnglish Ġlanguage Ġcollected Ġfrom Ġ20 / 08 / 1996 Ġto Ġ19 / 08 / 1997 ĠB IB REF 31 Ġ. ĠThe Ġtopic Ġof Ġeach Ġnews Ġwas Ġhuman - annot ated Ġusing Ġa Ġhierarchical Ġstructure . ĠAt Ġthe Ġtop Ġof Ġthe Ġhierarchy , Ġlies Ġthe Ġcoarse - gr ained Ġcategories : ĠCC AT Ġ( Cor porate ), ĠEC AT Ġ( Econom ics ), ĠGC AT Ġ( Government ), Ġand ĠMC AT Ġ( Mark ets ). ĠA Ġnews Ġarticle Ġcan Ġbe Ġassigned Ġto Ġmore Ġthan Ġone Ġcategory Ġmeaning Ġthat Ġthe Ġtext Ġcategor ization Ġtask Ġis Ġmut il abel . ĠEach Ġnews Ġis Ġstored Ġin Ġa Ġseparate ĠXML Ġfile . Ġl st : rc v 1 x m lex ample Ġshows Ġthe Ġtypical Ġstructure Ġof Ġan Ġarticle . </s> <? xml Ġversion =" 1 . 0 " Ġencoding =" iso - 88 59 - 1 " Ġ? > </s> < news item Ġitem id =" 6 159 " Ġid =" root " Ġdate =" 1996 - 08 - 21 " Ġxml : lang =" en "> </s> < head line > Col omb ia Ġraises Ġinternal Ġcoffee Ġprice .</ head line > </s> < date line > B OG OTA Ġ1996 - 08 - 21 </ date line > </s> < cop yright >( c ) ĠReuters ĠLimited Ġ1996 </ cop yright > </s> < metadata > </s> < codes Ġclass =" b ip : top ics : 1 . 0 "> </s> < code Ġcode =" C 13 "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" C 31 "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" CC AT "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" M 14 "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" M 141 "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> < code Ġcode =" MC AT "> </s> < edit detail Ġattribution =" Reuters ĠB IP ĠC oding ĠGroup " Ġaction =" confirmed " Ġdate =" 1996 - 08 - 21 "/> </s> </ code > </s> </ codes > </s> </ metadata > </s> </ news item > </s> The ĠRC V 1 Ġdataset Ġis Ġnot Ġreleased Ġwith Ġa Ġstandard Ġtrain , Ġvalidation , Ġtest Ġsplit . ĠIn Ġthis Ġwork , Ġwe Ġseparated Ġ15 % Ġof Ġsamples Ġas Ġa Ġtest Ġset Ġfor Ġevaluation Ġpurposes . ĠThe Ġremaining Ġsamples Ġwere Ġfurther Ġsplit Ġleaving Ġ70 % Ġand Ġ15 % Ġfor Ġtraining Ġand Ġvalidation , Ġrespectively . </s> Regarding Ġthe Ġcategories Ġdistribution , Ġwe Ġfound Ġthat , Ġfrom Ġthe Ġoriginal Ġ126 Ġcategories , Ġ23 Ġcategories Ġwere Ġnever Ġassigned Ġto Ġany Ġnews ; Ġtherefore , Ġwere Ġdisreg arded . ĠFrom Ġthe Ġ103 Ġclasses Ġleft Ġwe Ġfound Ġa Ġhigh Ġimbalance Ġamong Ġthe Ġlabels Ġwith Ġa Ġlarge Ġnumber Ġof Ġunder represented Ġcategories Ġhaving Ġless Ġthan Ġ12 Ġsamples . ĠThe Ġvery Ġlow Ġnumber Ġof Ġsamples Ġfor Ġthese Ġminority Ġclasses Ġbrings Ġa Ġgreat Ġchallenge Ġto Ġdiscriminate Ġthe Ġvery Ġfine - gr ained Ġcategories . ĠAim ing Ġto Ġalleviate Ġthis Ġproblem , Ġwe Ġgrouped Ġinto Ġa Ġsame Ġclass Ġall Ġcategories Ġbelow Ġthe Ġsecond Ġhierarchical Ġlevel . ĠFor Ġexample , Ġgiven Ġthe Ġroot Ġnode ĠCC AT Ġ( Cor porate ) Ġwe Ġgrouped ĠC 151 Ġ( ACC OUN TS / EAR NING S ), ĠC 15 11 Ġ( ANN UAL ĠRES ULTS ) Ġand ĠC 152 Ġ( COM MENT / FOR EC AS TS ) Ġinto Ġthe Ġdirect Ġchild Ġnode ĠC 15 Ġ( PER FOR MAN CE ). ĠUsing Ġthis Ġprocedure Ġthe Ġoriginal Ġ103 Ġcategories Ġwhere Ġreduced Ġto Ġ55 . ĠOne Ġof Ġthe Ġbenefits Ġof Ġthis Ġprocedure Ġwas Ġthat Ġthe Ġless Ġrepresented Ġclasses Ġend Ġup Ġhaving Ġaround Ġthousand Ġsamples Ġcompared Ġwith Ġonly Ġ12 Ġsamples Ġin Ġthe Ġoriginal Ġdataset . </s> fig : rc v 1 arch , Ġshows Ġthe Ġarchitecture Ġfor Ġthe Ġend - to - end Ġtext Ġcategor ization Ġtask . ĠOn Ġthe Ġbottom Ġof Ġthe Ġarchitecture Ġ$ S _ e $ Ġreceives Ġword Ġembed d ings Ġand Ġoutputs Ġa Ġsentence Ġvector Ġ$ S $ Ġ. ĠThe Ġ$ S $ Ġvector Ġpass Ġthrough Ġa Ġfully Ġconnected Ġ( FC ) Ġlayer Ġwith Ġs igm oid Ġactivation Ġfunction Ġthat Ġoutputs Ġa Ġvector Ġ$\ hat { y } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 55 } $ Ġwith Ġeach Ġelement Ġ$\ hat { y } _ j Ġ\ in Ġ[ 0 , 1 ] $ Ġ. </s> The Ġarchitecture Ġdescribed Ġabove Ġis Ġtrained Ġunder Ġthe Ġassumption Ġthat Ġeach Ġcategory Ġis Ġindependent Ġbut Ġnot Ġmutually Ġexclusive Ġsince Ġa Ġsample Ġcan Ġhave Ġmore Ġthan Ġone Ġcategory Ġassigned Ġ( mult il abel Ġclassification ). ĠThe Ġloss Ġper Ġsample Ġis Ġthe Ġaverage Ġlog Ġloss Ġacross Ġall Ġlabels : </s> $$ \ math cal Ġ{ L } (\ hat { y }, Ġy ) Ġ= Ġ- Ġ\ sum Ġ_ { i = 1 } ^{ 55 }\ left ( Ġy _ i Ġ\ log Ġ( \ hat { y } _ i ) Ġ+ Ġ( 1 - y _{ i }) Ġ\ log Ġ( 1 - \ hat { y } _{ i }) Ġ\ right ) $$ Ġ Ġ Ġ( E q . Ġ48 ) </s> where Ġthe Ġindex Ġ$ i $ Ġruns Ġover Ġthe Ġelements Ġof Ġthe Ġpredicted Ġand Ġtrue Ġvectors . </s> Given Ġthe Ġhigh Ġcategories Ġimbalance , Ġduring Ġthe Ġtraining Ġwe Ġmonitor Ġthe Ġ$ F _ 1 $ Ġmetric Ġof Ġthe Ġvalidation Ġset Ġand Ġchoose Ġthe Ġmodel Ġwith Ġthe Ġhighest Ġvalue . </s> Stan ford ĠNatural ĠLanguage ĠIn ference Ġ( SN LI ) Ġdataset ĠB IB REF 30 Ġconsist Ġof Ġ570 , 000 Ġpairs Ġof Ġsentences . ĠEach Ġpair Ġhas Ġa Ġpremise Ġand Ġa Ġhypothesis , Ġmanually Ġlabeled Ġwith Ġone Ġof Ġthe Ġthree Ġlabels : Ġentail ment , Ġcontradiction , Ġor Ġneutral . ĠThe ĠSN LI Ġhas Ġmany Ġdesired Ġproperties . ĠThe Ġlabels Ġare Ġequally Ġbalanced , Ġas Ġopposed Ġto Ġthe ĠRC V 1 Ġdataset . ĠAdditionally , Ġlanguage Ġinference Ġis Ġa Ġcomplex Ġtask Ġthat Ġrequires Ġa Ġdeeper Ġunderstanding Ġof Ġthe Ġsentence Ġmeaning Ġmaking Ġthis Ġdataset Ġsuitable Ġfor Ġlearning Ġsupervised Ġsentence Ġenc od ers Ġthat Ġgeneral ize Ġwell Ġto Ġother Ġtasks ĠB IB REF 17 Ġ. Ġt bl : sn lie x ma ples , Ġshows Ġexamples Ġof ĠSN LI Ġdataset Ġsentence Ġpairs Ġand Ġits Ġrespective Ġlabels . </s> In Ġorder Ġto Ġlearn Ġsentence Ġenc od ers Ġthat Ġcan Ġbe Ġtransf ered Ġto Ġother Ġtasks Ġunamb ig uously , Ġwe Ġconsider Ġa Ġneural Ġnetwork Ġarchitecture Ġfor Ġthe Ġsentence Ġenc oder Ġwith Ġshared Ġparameters Ġbetween Ġthe Ġpremise Ġand Ġhypothesis Ġpairs Ġas Ġin ĠB IB REF 17 Ġ. </s> fig : sn li arch , Ġdescribes Ġthe Ġneural Ġnetwork Ġarchitecture . ĠAfter Ġeach Ġpremise Ġand Ġhypothesis Ġis Ġencoded Ġinto Ġ$ S _ p $ Ġand Ġ$ S _ h $ Ġ, Ġrespectively , Ġwe Ġhave Ġa Ġfusion Ġlayer . ĠThis Ġlayer Ġhas Ġno Ġtrain able Ġweights Ġand Ġjust Ġconc aten ate Ġeach Ġsentence Ġembed ding . ĠFollowing ĠB IB REF 17 Ġ, Ġwe Ġadd Ġtwo Ġmore Ġmatching Ġmethods : Ġthe Ġabsolute Ġdifference Ġ$\ vert ĠS _ p Ġ- ĠS _ h Ġ\ vert Ġ$ Ġand Ġthe Ġelement - wise Ġ$ S _ p Ġ\ od ot ĠS _ h $ Ġ. ĠFinally , Ġin Ġorder Ġto Ġlearn Ġthe Ġpair Ġrepresentation , Ġ$ S _ ph $ Ġis Ġfeed Ġinto Ġand ĠFC Ġlayer Ġwith Ġrect ified Ġlinear Ġunit Ġ( Re LU ) Ġactivation Ġfunction , Ġwhich Ġis Ġexpressed Ġas Ġ$ f ( x ) Ġ= Ġ\ log Ġ( 1 Ġ+ Ġe ^ x ) $ Ġ. ĠThe Ġlast Ġsoft max Ġlayer Ġoutputs Ġthe Ġprobability Ġof Ġeach Ġclass . </s> Finally , Ġthe ĠNL I Ġclass ifier Ġweights Ġare Ġoptimized Ġin Ġorder Ġto Ġminimize Ġthe Ġcateg orical Ġlog Ġloss Ġper Ġsample </s> $$ \ math cal Ġ{ L } (\ hat { y }, Ġy ) Ġ= Ġ- Ġ\ sum Ġ_ { j = 1 } ^{ 3 } y _ i Ġ\ log Ġ( \ hat { y } _ i ) $$ Ġ Ġ Ġ( E q . Ġ52 ) </s> During Ġthe Ġtraining , Ġwe Ġmonitor Ġthe Ġvalidation Ġset Ġaccuracy Ġand Ġchoose Ġthe Ġmodel Ġwith Ġthe Ġhighest Ġmetric Ġvalue . </s> Sequ ence ĠModels </s> We Ġstart Ġthis Ġsection Ġby Ġreviewing Ġthe ĠRec urrent ĠNeural ĠNetwork Ġ( R NN ) Ġarchitecture Ġand Ġits Ġapplication Ġto Ġencode Ġa Ġsequence Ġof Ġwords . </s> R NN 's Ġare Ġcapable Ġof Ġhandling Ġvariable - length Ġsequences , Ġthis Ġbeing Ġa Ġdirect Ġconsequence Ġof Ġits Ġrecurrent Ġcell , Ġwhich Ġshares Ġthe Ġsame Ġparameters Ġacross Ġall Ġsequence Ġelements . ĠIn Ġthis Ġwork , Ġwe Ġadopt Ġthe ĠLong ĠShort - Term ĠMemory Ġ( L ST M ) Ġcell ĠB IB REF 38 Ġwith Ġforget Ġgates Ġ$ f _ t $ ĠB IB REF 39 Ġ. ĠThe ĠL ST M Ġcell Ġis Ġendowed Ġwith Ġa Ġmemory Ġstate Ġthat Ġcan Ġlearn Ġrepresentations Ġthat Ġdepend Ġon Ġthe Ġorder Ġof Ġthe Ġwords Ġin Ġa Ġsentence . ĠThis Ġmakes ĠL ST M Ġmore Ġfit Ġto Ġfind Ġrelations Ġthat Ġcould Ġnot Ġbe Ġcaptured Ġusing Ġstandard Ġbag - of - words Ġrepresentations . </s> Let Ġ$ x _ 1 , Ġx _ 2 , Ġ\ cd ots Ġ, Ġx _ T $ Ġbe Ġa Ġseries Ġof Ġobservations Ġof Ġlength Ġ$ T $ Ġ, Ġwhere Ġ$ x _ t Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ w } $ Ġ. ĠIn Ġgeneral Ġterms , Ġthe ĠL ST M Ġcell Ġreceives Ġa Ġprevious Ġhidden Ġstate Ġ$ h _{ t - 1 } $ Ġthat Ġis Ġcombined Ġwith Ġthe Ġcurrent Ġobservation Ġ$ x _ t $ Ġand Ġa Ġmemory Ġstate Ġ$ C _ t $ Ġto Ġoutput Ġa Ġnew Ġhidden Ġstate Ġ$ h _ t $ Ġ. ĠThis Ġinternal Ġmemory Ġstate Ġ$ C _{ t } $ Ġis Ġupdated Ġdepending Ġon Ġits Ġprevious Ġstate Ġand Ġthree Ġmod ulating Ġgates : Ġinput , Ġforget , Ġand Ġoutput . ĠForm ally , Ġfor Ġeach Ġstep Ġ$ t $ Ġthe Ġupdating Ġprocess Ġgoes Ġas Ġfollows Ġ( see Ġfig : l st m cell Ġfor Ġa Ġhigh Ġlevel Ġschematic Ġview ): ĠFirst , Ġwe Ġcalculate Ġthe Ġinput Ġ$ i _ t $ Ġ, Ġforget Ġ$ T $ 0 Ġ, Ġand Ġoutput Ġ$ T $ 1 Ġgates : </s> $$ i _ t Ġ& = Ġ\ s igma Ġ_ s \ left ( W _ i Ġx _ t Ġ+ ĠU _ i Ġh _{ t - 1 } Ġ+ Ġb _ i \ right ) Ġ\\ Ġf _ t Ġ& = Ġ\ s igma Ġ_ s \ left ( W _ f Ġx _ t Ġ+ ĠU _ f Ġh _{ t - 1 } Ġ+ Ġb _ f \ right ) Ġ\\ Ġo _ t Ġ& = Ġ\ s igma Ġ_ s \ left ( W _ o Ġx _ t Ġ+ ĠU _ o Ġh _{ t - 1 } Ġ+ Ġb _ o \ right ) $$ Ġ Ġ Ġ( E q . Ġ54 ) </s> where Ġ$\ s igma Ġ_ s $ Ġis Ġthe Ġs igm oid Ġactivation . ĠSecond , Ġa Ġcandidate Ġmemory Ġstate Ġ$\ wid et ilde { C } _ t $ Ġis Ġgenerated : </s> $$ \ wid et ilde { C } _ t Ġ= Ġ\ tan h Ġ\ left ( W _ c Ġx _ t Ġ+ ĠU _ c Ġh _{ t - 1 } Ġ+ Ġb _ c \ right ) $$ Ġ Ġ Ġ( E q . Ġ55 ) </s> Now Ġwe Ġare Ġin Ġa Ġposition Ġto Ġset Ġthe Ġfinal Ġmemory Ġstate Ġ$ C _ t $ Ġ. ĠIts Ġvalue Ġis Ġmod ulated Ġbased Ġon Ġthe Ġinput Ġand Ġforget Ġgates Ġof Ġeq : input for get g ates Ġand Ġis Ġgiven Ġby : </s> $$ C _ t Ġ= Ġi _ t Ġ\ od ot Ġ\ wid et ilde { C } _ t Ġ+ Ġf _ t Ġ\ od ot ĠC _{ t - 1 } $$ Ġ Ġ Ġ( E q . Ġ56 ) </s> Finally , Ġbased Ġon Ġthe Ġmemory Ġstate Ġand Ġoutput Ġgate Ġof Ġeq : input for get g ates , Ġwe Ġhave Ġthe Ġoutput Ġhidden Ġstate </s> $$ h _ t Ġ= Ġo _ t Ġ\ od ot Ġ\ tan h Ġ\ left ( C _ t \ right ) $$ Ġ Ġ Ġ( E q . Ġ57 ) </s> Regarding Ġthe Ġtrain able Ġweights , Ġlet Ġ$ n $ Ġbe Ġthe ĠL ST M Ġcell Ġnumber Ġof Ġunits . ĠIt Ġfollows Ġthat Ġ$ W $ Ġ' s Ġand Ġ$ U $ Ġ' s Ġmat rices Ġof Ġthe Ġaff ine Ġtransformations Ġhave Ġ${ n Ġ\ times Ġd _ w } $ Ġand Ġ${ n Ġ\ times Ġn } $ Ġdimensions , Ġrespectively . ĠIts Ġbias Ġterms Ġ$ b $ Ġ' s Ġare Ġvectors Ġof Ġsize Ġ$ n $ Ġ. ĠConsequently , Ġthe Ġtotal Ġnumber Ġof Ġparameters Ġis Ġ$ 4 Ġ( n Ġd _ w Ġ+ Ġn ^ 2 Ġ+ Ġn ) $ Ġand Ġdoes Ġnot Ġdepend Ġon Ġthe Ġsequence Ġnumber Ġof Ġtime Ġsteps Ġ$ T $ Ġ. </s> We Ġsee Ġthat Ġthe ĠL ST M Ġnetworks Ġare Ġable Ġto Ġcapture Ġtemporal Ġdependencies Ġin Ġsequences Ġof Ġarbitrary Ġlength . ĠOne Ġstraightforward Ġapplication Ġis Ġto Ġmodel Ġthe ĠSent ence Ġenc oder Ġdiscussed Ġin Ġsec : transfer learning , Ġwhich Ġoutputs Ġa Ġsentence Ġvector Ġrepresentation Ġusing Ġits Ġwords Ġas Ġinput . </s> Given Ġa Ġsequence Ġof Ġwords Ġ$\ left \ l brace Ġw _ t \ right \ r brace Ġ_ { t = 1 } ^{ T } $ Ġwe Ġaim Ġto Ġlearn Ġthe Ġwords Ġhidden Ġstate Ġ$\ left \ l brace Ġh _ t \ right \ r brace Ġ_ { t = 1 } ^{ T } $ Ġin Ġa Ġway Ġthat Ġeach Ġword Ġcaptures Ġthe Ġinfluence Ġof Ġits Ġpast Ġand Ġfuture Ġwords . ĠThe ĠBid irection al ĠL ST M Ġ( Bi L ST M ) Ġproposed Ġin ĠB IB REF 40 Ġis Ġan ĠL ST M Ġthat ĠâĢ ľ reads âĢ Ŀ Ġa Ġsentence , Ġor Ġany Ġsequence Ġin Ġgeneral , Ġfrom Ġthe Ġbeginning Ġto Ġthe Ġend Ġ( forward ) Ġand Ġthe Ġother Ġway Ġaround Ġ( back ward ). ĠThe Ġnew Ġstate Ġ$ h _ t $ Ġis Ġthe Ġconc aten ation </s> $$ h _ t Ġ= Ġ[ \ over right arrow { h _ t }, Ġ\ over left arrow { h _ t } ] $$ Ġ Ġ Ġ( E q . Ġ59 ) </s> where </s> $$ \ over right arrow { h _ t } Ġ& = Ġ\ text { L ST M }\ left ( w _ 1 , Ġ\ cd ots Ġ, Ġw _ T \ right ) Ġ\\ Ġ\ over left arrow { h _ t } Ġ& = Ġ\ text { L ST M }\ left ( w _ T , Ġ\ cd ots Ġ, Ġw _ 1 \ right ) Ġ\\ $$ Ġ Ġ Ġ( E q . Ġ60 ) </s> Because Ġsentences Ġhave Ġdifferent Ġlengths , Ġwe Ġneed Ġto Ġconvert Ġthe Ġ$ T $ Ġconc aten ated Ġhidden Ġstates Ġof Ġthe ĠBi L ST M Ġinto Ġa Ġfixed - length Ġsentence Ġrepresentation . ĠOne Ġstraightforward Ġoperation Ġis Ġto Ġapply Ġany Ġform Ġof Ġpool ing . ĠAttention Ġmechanism Ġis Ġan Ġalternative Ġapproach Ġwhere Ġthe Ġsentence Ġis Ġrepresented Ġas Ġan Ġweighted Ġaverage Ġof Ġhidden Ġstates Ġwhere Ġthe Ġweights Ġare Ġlearnt Ġend - to - end . </s> In Ġthe Ġnext Ġsections Ġwe Ġdescribe Ġthe Ġsentence Ġenc od ers Ġusing Ġpool ing Ġand Ġattention Ġlayers . </s> The Ġmax - pool ing Ġlayer Ġaims Ġto Ġextract Ġthe Ġmost Ġsalient Ġword Ġfeatures Ġall Ġover Ġthe Ġsentence . ĠForm ally , Ġit Ġoutputs Ġa Ġsentence Ġvector Ġrepresentation Ġ$ S _{ MP } Ġ\ in Ġ\ math bb Ġ{ R } ^{ 2 n } $ Ġsuch Ġthat </s> $$ S _{ MP } Ġ= Ġ\ max Ġ_ { t = 1 } ^{ T } Ġh _ t $$ Ġ Ġ Ġ( E q . Ġ62 ) </s> where Ġ$ h _ t $ Ġis Ġdefined Ġin Ġeq : ht con cat Ġand Ġthe Ġ$\ max Ġ$ Ġoperator Ġis Ġapplied Ġover Ġthe Ġtime Ġsteps Ġdimension . Ġfig : b ilst mm ax pool Ġillustrates Ġthe ĠBi L ST M Ġmax - pool ing Ġ( MP ) Ġsentence Ġenc oder . </s> The Ġefficacy Ġof Ġthe Ġmax - pool ing Ġlayer Ġwas Ġassessed Ġin Ġmany ĠN LP Ġstudies . ĠB IB REF 41 Ġemployed Ġa Ġmax - pool ing Ġlayer Ġon Ġtop Ġof Ġword Ġrepresentations Ġand Ġargues Ġthat Ġit Ġperforms Ġbetter Ġthan Ġmean Ġpool ing . ĠExperimental Ġresults Ġin ĠB IB REF 17 Ġshow Ġthat Ġamong Ġthree Ġtypes Ġof Ġpool ing Ġ( max , Ġmean Ġand Ġlast ) Ġthe Ġmax - pool ing Ġprovides Ġthe Ġmost Ġuniversal Ġsentence Ġrepresentations Ġin Ġterms Ġof Ġtransferring Ġperformance Ġto Ġother Ġtasks . ĠGround ed Ġon Ġthese Ġstudies , Ġin Ġthis Ġwork , Ġwe Ġchoose Ġthe ĠBi L ST M Ġmax - pool ing Ġas Ġour Ġpool ing Ġlayer Ġof Ġchoice . </s> Att ention Ġmechanisms Ġwere Ġintroduced Ġin Ġthe Ġdeep Ġlearning Ġliterature Ġto Ġovercome Ġsome Ġsimpl ifications Ġimposed Ġby Ġpool ing Ġoperators . ĠWhen Ġwe Ġhumans Ġread Ġa Ġsentence , Ġwe Ġare Ġable Ġto Ġspot Ġits Ġmost Ġrelevant Ġparts Ġin Ġa Ġgiven Ġcontext Ġand Ġdisregard Ġinformation Ġthat Ġis Ġredundant Ġor Ġmisleading . ĠThe Ġattention Ġmodel Ġaims Ġto Ġmimic Ġthis Ġbehaviour . </s> Att ention Ġlayers Ġwere Ġproposed Ġfor Ġdifferent ĠN LP Ġtasks . ĠFor Ġexample , ĠNL I , Ġwith Ġcross - att ention Ġbetween Ġpremise Ġand Ġhypothesis , ĠQuestion Ġ& ĠAn sw ering Ġand ĠMachine ĠTranslation Ġ( MT ). ĠSpecifically Ġin Ġthe ĠMachine ĠTranslation Ġtask , Ġeach Ġword Ġin Ġthe Ġtarget Ġsentence Ġlearns Ġto Ġattend Ġthe Ġrelevant Ġwords Ġof Ġthe Ġsource Ġsentence Ġin Ġorder Ġto Ġgenerate Ġthe Ġsentence Ġtranslation . </s> A Ġsentence Ġenc oder Ġwith Ġattention Ġ( or Ġself - att ent ive ) ĠB IB REF 42 Ġ, ĠB IB REF 43 Ġ, ĠB IB REF 44 Ġassigns Ġdifferent Ġweights Ġto Ġthe Ġown Ġwords Ġof Ġthe Ġsentence ; Ġtherefore , Ġconverting Ġthe Ġhidden Ġstates Ġinto Ġa Ġsingle Ġsentence Ġvector Ġrepresentation . </s> Considering Ġthe Ġword Ġhidden Ġvectors Ġset Ġ$\ l brace Ġh _ 1 , Ġ\ cd ots Ġ, Ġh _ T \ r brace Ġ$ Ġwhere Ġ$ h _ t Ġ\ in Ġ\ math bb Ġ{ R } ^ n $ Ġ, Ġthe Ġattention Ġmechanism Ġis Ġdefined Ġby Ġthe Ġequations : </s> $$ \ t ilde { h } _ t Ġ& = Ġ\ s igma Ġ\ left ( W Ġh _ t Ġ+ Ġb Ġ\ right ) Ġ\\ Ġ\ alpha Ġ_ { t } Ġ& = Ġ\ frac {\ exp Ġ({ v ^ {\ inter cal Ġ} Ġ\ cd ot Ġ\ t ilde { h } _ t } Ġ) }{ \ sum Ġ_ { t } Ġ\ exp Ġ({ v Ġ\ cd ot Ġ\ t ilde { h } _ t }) } Ġ\\ ĠS _{ A _ w } Ġ& = Ġ\ sum Ġ_ { t } Ġ\ alpha Ġ_ { t } Ġh _ t $$ Ġ Ġ Ġ( E q . Ġ66 ) </s> where Ġ$ W Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ a Ġ\ times Ġn } $ Ġ, Ġ$ b Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ a Ġ\ times Ġ1 } $ Ġ, Ġand Ġ$ v Ġ\ in Ġ\ math bb Ġ{ R } ^{ d _ a Ġ\ times Ġ1 } $ Ġare Ġtrain able Ġparameters . </s> We Ġcan Ġsee Ġthat Ġthe Ġsentence Ġrepresentation Ġ$ S _{ A _ w } $ Ġis Ġa Ġweighted Ġaverage Ġof Ġthe Ġhidden Ġstates . Ġfig : b ilst min ner att Ġprovides Ġa Ġschematic Ġview Ġof Ġthe ĠBi L ST M Ġattention , Ġwhere Ġwe Ġcan Ġaccount Ġthe Ġattention Ġdescribed Ġin Ġeq : att Ġas Ġa Ġtwo Ġlayer Ġmodel Ġwith Ġa Ġdense Ġlayer Ġ( Ġ$ d _ a $ Ġunits ) Ġfollowed Ġby Ġanother Ġdense Ġthat Ġpredicts Ġ$\ alpha Ġ_ t $ Ġ( single Ġunit ). </s> Method ology </s> In Ġthis Ġsection , Ġwe Ġfirst Ġintroduce Ġour Ġproblem Ġin Ġa Ġdeep Ġmultim od al Ġlearning Ġframework . ĠWe Ġthen Ġpresent Ġour Ġneural Ġarchitecture , Ġwhich Ġis Ġable Ġto Ġaddress Ġthe Ġproblems Ġof Ġnews Ġrelevance Ġand Ġnovelty . ĠFinally , Ġwe Ġreview Ġthe Ġmethods Ġapplied Ġto Ġlearn Ġcommon alities Ġbetween Ġstocks Ġ( global Ġfeatures ). </s> Problem Ġstatement </s> Our Ġproblem Ġis Ġto Ġpredict Ġthe Ġdaily Ġstock Ġvolatility . ĠAs Ġdiscussed Ġin Ġsubs ub : range vol est im ators , Ġthe ĠG aman - K lass Ġestim ator Ġ$\ wide hat {\ s igma Ġ_ { G K , t }} $ Ġin Ġeq : vol g k Ġis Ġa Ġvery Ġefficient Ġshort - term Ġvolatility Ġproxy , Ġthus , Ġit Ġis Ġadopted Ġas Ġour Ġtarget Ġvariable . </s> Our Ġgoal Ġis Ġto Ġlearn Ġa Ġmapping Ġbetween Ġthe Ġnext Ġday Ġvolatility Ġ$\ s igma Ġ_ { t + 1 } $ Ġand Ġhistorical Ġmultim od al Ġdata Ġavailable Ġup Ġto Ġday Ġ$ t $ Ġ. ĠTo Ġthis Ġaim , Ġwe Ġuse Ġa Ġsliding Ġwindow Ġapproach Ġwith Ġwindow Ġsize Ġ$ T $ Ġ. ĠThat Ġis , Ġfor Ġeach Ġstock Ġ$ sc $ Ġa Ġsample Ġon Ġday Ġ$ t $ Ġis Ġexpressed Ġas Ġa Ġsequence Ġof Ġhistorical Ġprices Ġ$ P ^{ sc } _ t $ Ġand Ġcorpus Ġheadlines Ġ$ N ^{ sc } _ t $ Ġ. ĠThe Ġprice Ġsequence Ġis Ġa Ġvector Ġof ĠDaily ĠPrices Ġ( DP ) Ġand Ġexpressed Ġas </s> $$ P ^{ sc } _ t Ġ= Ġ\ left [ DP ^{ sc } _{ t - T }, ĠDP ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDP ^{ sc } _ t Ġ\ right ] $$ Ġ Ġ Ġ( E q . Ġ69 ) </s> where Ġ$ DP ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġis Ġa Ġvector Ġof Ġprice Ġfeatures . ĠIn Ġorder Ġto Ġavoid Ġtask - specific Ġfeature Ġengineering , Ġthe Ġdaily Ġprice Ġfeatures Ġare Ġexpressed Ġas Ġthe Ġsimple Ġreturns : </s> $$ DP ^{ sc } _ t Ġ= Ġ\ left [ Ġ\ frac { O ^{ sc } _{ t }} { C ^{ sc } _{ t - 1 }} Ġ- Ġ1 , Ġ\ frac { H ^{ sc } _{ t }} { C ^{ sc } _{ t - 1 }} Ġ- Ġ1 , Ġ\ frac { L ^{ sc } _{ t }} { C ^{ sc } _{ t - 1 }} Ġ- Ġ1 , Ġ\ frac { C ^{ sc } _{ t }} { C ^{ sc } _{ t - 1 }} Ġ- Ġ1 Ġ\ right ] $$ Ġ Ġ Ġ( E q . Ġ70 ) </s> The Ġsequence Ġof Ġhistorical Ġcorpus Ġheadlines Ġ$ N ^{ sc } _ t $ Ġis Ġexpressed Ġas </s> $$ N ^{ sc } _ t Ġ= Ġ\ left [ n ^{ sc } _{ t - T }, Ġn ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, Ġn ^{ sc } _{ t } Ġ\ right ] $$ Ġ Ġ Ġ( E q . Ġ71 ) </s> where Ġ$ n ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġis Ġa Ġset Ġcontaining Ġall Ġheadlines Ġthat Ġinfluence Ġthe Ġmarket Ġon Ġa Ġgiven Ġday Ġ$ t ^ {\ prime Ġ} $ Ġ. </s> Aim ing Ġto Ġalign Ġprices Ġand Ġnews Ġmodes , Ġwe Ġconsider Ġthe Ġexplicit Ġalignment Ġmethod Ġdiscussed Ġin Ġsub sec : stock head lines . ĠThat Ġis , Ġ$ n ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġcontains Ġall Ġstock Ġheadlines Ġbefore Ġthe Ġmarket Ġopens Ġ( Ġ$\ text tt Ġ{ before Ġmarket } _{ t } $ Ġ), Ġduring Ġthe Ġtrading Ġhours </s> ( Ġ$\ text tt Ġ{ during Ġmarket } _{ t } $ Ġ), Ġand Ġprevious Ġday Ġafter - markets </s> ( Ġ$\ text tt Ġ{ after Ġmarket } _{ t - 1 } $ Ġ). </s> As Ġa Ġtext Ġpre processing Ġstep , Ġwe Ġtoken ize Ġthe Ġheadlines Ġand Ġconvert Ġeach Ġword Ġto Ġan Ġinteger Ġthat Ġrefers Ġto Ġits Ġrespective Ġpre - trained Ġword Ġembed ding . ĠThis Ġprocess Ġis Ġdescribed Ġas Ġfollows : ĠFirst , Ġfor Ġall Ġstocks Ġof Ġour Ġcorpus Ġwe Ġtoken ize Ġeach Ġheadline Ġand Ġextract Ġthe Ġcorpus Ġvocabulary Ġset Ġ$ V $ Ġ. ĠWe Ġthen Ġbuild Ġthe Ġembed ding Ġmatrix Ġ$ E _ w Ġ\ in Ġ\ math bb Ġ{ R } ^ {\ vert ĠV Ġ\ vert Ġ\ times Ġd _ w } $ Ġ, Ġwhere Ġeach Ġrow Ġis Ġa Ġword Ġembed ding Ġvector Ġ$ d _ w $ Ġdimensions . ĠWords Ġthat Ġdo Ġnot Ġhave Ġa Ġcorresponding Ġembed ding , Ġi . e . Ġout Ġof Ġvocabulary Ġwords , Ġare Ġskipped . </s> Finally , Ġthe Ġinput Ġsample Ġof Ġthe Ġtext Ġmode Ġis Ġa Ġtens or Ġof Ġintegers Ġwith Ġ$ T Ġ\ times Ġl _ n Ġ\ times Ġl _ s $ Ġdimensions , Ġwhere Ġ$ l _ n $ Ġis Ġthe Ġmaximum Ġnumber Ġof Ġnews Ġoccurring Ġin Ġa Ġgiven Ġday Ġand Ġ$ l _ s $ Ġis Ġthe Ġmaximum Ġlength Ġof Ġa Ġcorpus Ġsentence . ĠRegarding Ġthe Ġprice Ġmode , Ġwe Ġhave Ġa Ġ$ T Ġ\ times Ġ4 $ Ġtens or Ġof Ġfloating Ġnumbers . </s> Global Ġfeatures Ġand Ġstock Ġembed ding </s> Given Ġthe Ġprice Ġand Ġnews Ġhistories Ġfor Ġeach Ġstock Ġ$ sc $ Ġwe Ġcould Ġdirectly Ġlearn Ġone Ġmodel Ġper Ġstock . ĠHowever , Ġthis Ġapproach Ġsuffers Ġfrom Ġtwo Ġmain Ġdrawbacks . ĠFirst , Ġthe Ġmarket Ġactivity Ġof Ġone Ġspecific Ġstock Ġis Ġexpected Ġto Ġimpact Ġother Ġstocks , Ġwhich Ġis Ġa Ġwidely Ġaccepted Ġpattern Ġnamed ĠâĢ ľ sp ill over Ġeffect âĢ Ŀ . ĠSecond , Ġsince Ġour Ġprice Ġdata Ġis Ġsampled Ġon Ġa Ġdaily Ġbasis , Ġwe Ġwould Ġtrain Ġthe Ġstock Ġmodel Ġrelying Ġon Ġa Ġsmall Ġnumber Ġof Ġsamples . ĠOne Ġpossible Ġsolution Ġto Ġmodel Ġthe Ġcommon ality Ġamong Ġstocks Ġwould Ġbe Ġfeature Ġenrichment . ĠFor Ġexample , Ġwhen Ġmodeling Ġa Ġgiven Ġstock Ġ$ X $ Ġwe Ġwould Ġenrich Ġits Ġnews Ġand Ġprice Ġfeatures Ġby Ġconc aten ating Ġfeatures Ġfrom Ġstock Ġ$ Y $ Ġand Ġ$ Z $ Ġ. ĠAlthough Ġthe Ġfeature Ġenrichment Ġis Ġable Ġto Ġmodel Ġthe Ġeffect Ġof Ġother Ġstocks , Ġit Ġstill Ġwould Ġconsider Ġonly Ġone Ġsample Ġper Ġday . </s> In Ġthis Ġwork , Ġwe Ġpropose Ġa Ġmethod Ġthat Ġlearns Ġan Ġglobal Ġmodel . </s> The Ġglobal Ġmodel Ġis Ġimplemented Ġusing Ġthe Ġfollowing Ġmethods : </s> Multi - Stock Ġbatch Ġsamples : ĠSince Ġour Ġmodels Ġare Ġtrained Ġusing ĠSt och astic ĠGrad ient ĠDes cent , Ġwe Ġpropose Ġat Ġeach Ġmini - batch Ġiteration Ġto Ġsample Ġfrom Ġa Ġbatch Ġset Ġcontaining Ġany Ġstock Ġof Ġour Ġstocks Ġuniverse . ĠAs Ġa Ġconsequence , Ġthe Ġmapping Ġbetween Ġvolatility Ġand Ġmultim od al Ġdata Ġis Ġnow Ġable Ġto Ġlearn Ġcommon Ġexplanatory Ġfactors Ġamong Ġstocks . ĠMoreover , Ġadopting Ġthis Ġapproach Ġincreases Ġthe Ġtotal Ġnumber Ġof Ġtraining Ġsamples , Ġwhich Ġis Ġnow Ġthe Ġsum Ġof Ġthe Ġnumber Ġof Ġsamples Ġper Ġstock . </s> Stock ĠEmb edd ing : ĠUt il izing Ġthe ĠMulti - Stock Ġbatch Ġsamples Ġabove , Ġwe Ġtackle Ġthe Ġproblem Ġof Ġmodeling Ġcommon ality Ġamong Ġstocks . ĠHowever , Ġit Ġis Ġreasonable Ġto Ġassume Ġthat Ġstocks Ġhave Ġpart Ġof Ġits Ġdynamic Ġdriven Ġby Ġidiosyncr atic Ġfactors . ĠNevertheless , Ġwe Ġcould Ġaggregate Ġstocks Ġper Ġsector Ġor Ġrely Ġon Ġany Ġmeasure Ġof Ġsimilarity Ġamong Ġstocks . ĠIn Ġorder Ġto Ġincorporate Ġinformation Ġspecific Ġto Ġeach Ġstock , Ġwe Ġpropose Ġto Ġequip Ġour Ġmodel Ġwith Ġa ĠâĢ ľ stock Ġembed ding âĢ Ŀ Ġmode Ġthat Ġis Ġlearnt Ġjointly Ġwith Ġprice Ġand Ġnews Ġmodes . ĠThat Ġis Ġto Ġsay , Ġwe Ġleave Ġthe Ġtask Ġof Ġdistinguishing Ġthe Ġspecific Ġdynamic Ġof Ġeach Ġstock Ġto Ġbe Ġlearnt Ġby Ġthe Ġneural Ġnetwork . ĠSpecifically , Ġthis Ġstock Ġembed ding Ġis Ġmodeled Ġusing Ġa Ġdiscrete Ġencoding Ġas Ġinput , Ġi . e . Ġ$\ math cal Ġ{ I } ^{ sc } _ t $ Ġis Ġa Ġvector Ġwith Ġsize Ġequal Ġto Ġthe Ġnumber Ġof Ġstocks Ġof Ġthe Ġstocks Ġuniverse Ġand Ġhas Ġelement Ġ1 Ġfor Ġthe Ġi - th Ġcoordinate Ġand Ġ0 Ġelsewhere , Ġthus , Ġindicating Ġthe Ġstock Ġof Ġeach Ġsample . </s> Form ally , Ġwe Ġcan Ġexpress Ġthe Ġone Ġmodel Ġper Ġstock Ġapproach Ġas Ġthe Ġmapping </s> $$ \ begin { split } Ġ\ s igma Ġ^ { sc } _{ t + 1 } Ġ= Ġf ^{ sc } Ġ( ĠDN ^{ sc } _{ t - T }, ĠDN ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDN ^{ sc } _ t Ġ; Ġ\\ ĠDP ^{ sc } _{ t - T }, ĠDP ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDP ^{ sc } _ t Ġ) Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ75 ) </s> where Ġ$ DN ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġis Ġa Ġfixed - vector Ġrepresenting Ġall Ġnews Ġreleased Ġon Ġa Ġgiven Ġday Ġfor Ġthe Ġstock Ġ$ sc $ Ġand Ġ$ DP ^{ sc } _{ t ^ {\ prime Ġ}} $ Ġis Ġdefined Ġin Ġeq : pr ic em ode vec . </s> The Ġglobal Ġmodel Ġattempts Ġto Ġlearn Ġa Ġsingle Ġmapping Ġ$ f $ Ġthat Ġat Ġeach Ġmini - batch Ġiteration Ġrandomly Ġaggreg ates Ġsamples Ġacross Ġall Ġthe Ġuniverse Ġof Ġstocks , Ġrather Ġthan Ġone Ġmapping Ġ$ f ^{ sc } $ Ġper Ġstock . ĠThe Ġglobal Ġmodel Ġis Ġexpressed Ġas </s> $$ \ begin { split } Ġ\ s igma Ġ^ { sc } _{ t + 1 } Ġ= Ġf Ġ( ĠDN ^{ sc } _{ t - T }, ĠDN ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDN ^{ sc } _ t Ġ; Ġ\\ ĠDP ^{ sc } _{ t - T }, ĠDP ^{ sc } _{ t - T + 1 }, Ġ\ cd ots Ġ, ĠDP ^{ sc } _ t Ġ; Ġ\\ Ġ\ math cal Ġ{ I } ^{ sc } _ t ) Ġ\ end { split } $$ Ġ Ġ Ġ( E q . Ġ77 ) </s> In Ġthe Ġnext Ġsection , Ġwe Ġdescribe Ġour Ġhierarchical Ġneural Ġmodel Ġand Ġhow Ġthe Ġnews , Ġprice Ġand Ġstock Ġembed ding Ġare Ġfused Ġinto Ġa Ġjoint Ġrepresentation . </s> Our Ġmultim od al Ġhierarchical Ġnetwork </s> In Ġbroad Ġterms , Ġour Ġhierarchical Ġneural Ġarchitecture Ġis Ġdescribed Ġas Ġfollows . ĠFirst , Ġeach Ġheadline Ġreleased Ġon Ġa Ġgiven Ġday Ġ$ t $ Ġis Ġencoded Ġinto Ġa Ġfixed - size Ġvector Ġ$ S _ t $ Ġusing Ġa Ġsentence Ġenc oder . ĠWe Ġthen Ġapply Ġour Ġdaily ĠNew ĠRe lev ance ĠAttention Ġ( N RA ) Ġmechanism Ġthat Ġattends Ġeach Ġnews Ġbased Ġon Ġits Ġcontent Ġand Ġconverts Ġa Ġvariable Ġsize Ġof Ġnews Ġreleased Ġon Ġa Ġgiven Ġday Ġinto Ġa Ġsingle Ġvector Ġden oted Ġby ĠDaily ĠNews Ġ( Ġ$ DN $ Ġ). ĠWe Ġnote Ġthat Ġthis Ġrepresentation Ġtake Ġaccount Ġof Ġthe Ġoverall Ġeffect Ġof Ġall Ġnews Ġreleased Ġon Ġa Ġgiven Ġday . ĠThis Ġprocess Ġis Ġillustrated Ġin Ġfig : DN enc oder . ĠWe Ġnow Ġare Ġin Ġa Ġposition Ġto Ġconsider Ġthe Ġtemporal Ġeffect Ġof Ġthe Ġpast Ġ$ T $ Ġdays Ġof Ġmarket Ġnews Ġand Ġprice Ġfeatures . Ġfig : n nt imes eries arch Ġillustrates Ġthe Ġneural Ġnetwork Ġarchitecture Ġfrom Ġthe Ġtemporal Ġsequence Ġto Ġthe Ġfinal Ġvolatility Ġprediction . ĠFor Ġeach Ġstock Ġcode Ġ$ sc $ Ġthe Ġtemporal Ġencoding Ġfor Ġnews Ġis Ġden oted Ġby ĠMarket ĠNews Ġ$ MN ^{ sc } _ t $ Ġand Ġfor Ġthe Ġprice Ġby ĠMarket ĠPrice Ġ$ MP ^{ sc } _ t $ Ġand Ġare Ġa Ġfunction Ġof Ġthe Ġpast Ġ$ T $ ĠDaily ĠNews Ġrepresentations Ġ${ \ l brace ĠDN ^{ sc } _{ t - T }, Ġ\ cd ots Ġ, ĠDN ^{ sc } _ t Ġ\ r brace Ġ} $ Ġ( Text Ġmode ) Ġand ĠDaily ĠPrices Ġfeatures Ġ$ S _ t $ 0 Ġ( Price Ġmode ), Ġwhere Ġeach ĠDaily ĠPrice Ġ$ S _ t $ 1 Ġfeature Ġis Ġgiven Ġby Ġeq : pr ic em ode vec Ġand Ġthe Ġ$ S _ t $ 2 Ġrepresentation Ġis Ġcalculated Ġusing ĠDaily ĠNew ĠRe lev ance ĠAttention . ĠAfter Ġthe Ġtemporal Ġeffects Ġof Ġ$ S _ t $ 3 Ġpast Ġdays Ġof Ġmarket Ġactivity Ġwere Ġalready Ġencoded Ġinto Ġthe ĠMarket ĠNews Ġ$ S _ t $ 4 Ġand ĠMarket ĠPrice Ġ$ S _ t $ 5 Ġ, Ġwe Ġconc aten ate Ġfeature - wise Ġ$ S _ t $ 6 Ġ, Ġ$ S _ t $ 7 Ġand Ġthe ĠStock Ġembed ding Ġ$ S _ t $ 8 Ġ. ĠThe Ġstock Ġembed ding Ġ$ S _ t $ 9 Ġrepresents Ġthe Ġstock Ġcode Ġof Ġthe Ġsample Ġon Ġa Ġgiven Ġday Ġ$ t $ Ġ. ĠFinally , Ġwe Ġhave Ġa ĠFully ĠConnect ed Ġ( FC ) Ġlayer Ġthat Ġlearns Ġthe ĠJoint ĠRepresent ation Ġof Ġall Ġmodes . ĠThis Ġfixed - sized Ġjoint Ġrepresentation Ġis Ġfed Ġinto Ġa ĠFC Ġlayer Ġwith Ġlinear Ġactivation Ġthat Ġpredicts Ġthe Ġnext Ġday Ġvolatility Ġ$\ hat {\ s igma Ġ} _{ t + 1 } $ Ġ. </s> Below , Ġwe Ġdetail , Ġfor Ġeach Ġmode Ġseparately , Ġthe Ġlayers Ġof Ġour Ġhierarchical Ġmodel . </s> âĢĵ ĠText Ġmode </s> Word ĠEmb edd ing ĠRet ri eval </s> Standard Ġembed ding Ġlayer Ġwith Ġno Ġtrain able Ġparameters . ĠIt Ġreceives Ġa Ġvector Ġof Ġword Ġindices Ġas Ġinput Ġand Ġreturns Ġa Ġmatrix Ġof Ġword Ġembed d ings . </s> News ĠEnc oder </s> This Ġlayer Ġenc odes Ġall Ġnews Ġon Ġa Ġgiven Ġday Ġand Ġoutputs Ġa Ġset Ġnews Ġembed d ings Ġ$\ l brace ĠS ^{ 1 } _ t , Ġ\ cd ots Ġ, ĠS ^{ l _ n } _ t Ġ\ r brace Ġ$ Ġ. ĠEach Ġencoded Ġsentence Ġhas Ġdimension Ġ$ d _ S $ Ġ, Ġwhich Ġis Ġa Ġhyper param eter Ġof Ġour Ġmodel . ĠThis Ġlayer Ġconstitutes Ġa Ġkey Ġcomponent Ġof Ġour Ġneural Ġarchitectures Ġand , Ġas Ġsuch , Ġwe Ġevaluate Ġour Ġmodels Ġconsidering Ġsentence Ġenc od ers Ġtrained Ġend - to - end , Ġusing Ġthe ĠBi L ST M Ġattention Ġ( sub sec : b ilst min ner att ) Ġand ĠBi L ST M Ġmax - pool ing Ġ( sub sec : b ilst mm ax pool ) Ġarchitectures , Ġand Ġalso Ġtransferred Ġfrom Ġthe ĠRC V 1 Ġand ĠSN LI Ġas Ġfixed Ġfeatures . </s> Daily Ġnews Ġrelevance Ġattention </s> Our Ġproposed Ġnews Ġrelevance Ġattention Ġmechanism Ġfor Ġall Ġnews Ġreleased Ġon Ġa Ġgiven Ġday . ĠThe Ġattention Ġmechanism Ġis Ġintroduced Ġto Ġtackle Ġinformation Ġoverload . ĠIt Ġwas Ġdesigned Ġto ĠâĢ ľ filter Ġout âĢ Ŀ Ġredundant Ġor Ġmisleading Ġnews Ġand Ġfocus Ġon Ġthe Ġrelevant Ġones Ġbased Ġsolely Ġon Ġthe Ġnews Ġcontent . ĠForm ally , Ġthe Ġlayer Ġoutputs Ġa ĠDaily ĠNews Ġ( DN ) Ġembed ding Ġ$ DN ^{ sc } _ t Ġ= Ġ\ sum Ġ_ { i = 1 } ^{ l _ n } Ġ\ beta Ġ_ i ĠS ^{ sc ^{ i }} _ t $ Ġ, Ġwhich Ġis Ġa Ġlinear Ġcombination Ġof Ġall Ġencoded Ġnews Ġon Ġa Ġgiven Ġday Ġ$ t $ Ġ. ĠThis Ġnews - level Ġattention Ġuses Ġthe Ġsame Ġequations Ġas Ġin Ġeq : att , Ġbut Ġwith Ġtrain able Ġweights Ġ$\ l brace ĠW _{ R }, Ġb _{ R }, Ġv _{ R }\ r brace Ġ$ Ġ, Ġi . e . Ġthe Ġweights Ġare Ġsegregated Ġfrom Ġthe Ġsentence Ġenc oder . Ġfig : DN enc oder , Ġillustrates Ġour Ġrelevance Ġattention . ĠNote Ġthat Ġthis Ġlayer Ġwas Ġdeliberately Ġdeveloped Ġto Ġbe Ġinvari ant Ġto Ġheadlines Ġperm utation , Ġas Ġis Ġthe Ġcase Ġwith Ġthe Ġlinear Ġcombination Ġformula Ġabove . ĠThe Ġreason Ġis Ġthat Ġour Ġprice Ġdata Ġis Ġsampled Ġdaily Ġand , Ġas Ġa Ġconsequence , Ġwe Ġare Ġnot Ġable Ġto Ġdiscriminate Ġthe Ġmarket Ġreaction Ġfor Ġeach Ġint rad ay Ġnews . </s> News ĠTem poral ĠContext </s> Sequ ence Ġlayer Ġwith Ġdaily Ġnews Ġembed d ings Ġ$ DN ^{ sc } _ t $ Ġas Ġtime Ġsteps . ĠThis Ġlayer Ġaims Ġto Ġlearn Ġthe Ġtemporal Ġcontext Ġof Ġnews , Ġi . e . Ġthe Ġrelationship Ġbetween Ġthe Ġnews Ġat Ġday Ġ$ t $ Ġand Ġthe Ġ$ T $ Ġpast Ġdays . ĠIt Ġreceives Ġas Ġinput Ġa Ġchron ologically Ġordered Ġsequence Ġof Ġ$ T $ Ġpast ĠDaily ĠNews Ġembed d ings Ġ${ \ l brace ĠDN ^{ sc } _{ t - T }, Ġ\ cd ots Ġ, ĠDN ^{ sc } _ t Ġ\ r brace Ġ} $ Ġand Ġoutputs Ġthe Ġnews Ġmode Ġencoding ĠMarket ĠNews Ġ$ MN ^{ sc } _ t Ġ\ in Ġd _{ MN } $ Ġ. ĠThe Ġsequence Ġwith Ġ$ T $ Ġtime Ġsteps Ġis Ġencoded Ġusing Ġa ĠBi L ST M Ġattention . ĠThe Ġlayer Ġwas Ġdesigned Ġto Ġcapture Ġthe Ġtemporal Ġorder Ġthat Ġnews Ġare Ġreleased Ġand Ġthe Ġcurrent Ġnews Ġnovelty . Ġi . e . Ġnews Ġthat Ġwere Ġrepeated Ġin Ġthe Ġpast Ġcan Ġbe ĠâĢ ľ for gotten âĢ Ŀ Ġbased Ġon Ġthe Ġmod ulating Ġgates Ġof Ġthe ĠL ST M Ġnetwork . </s> âĢĵ ĠPrice Ġmode </s> Price ĠEnc oder </s> Sequ ence Ġlayer Ġanalogous Ġto ĠNews ĠTem poral ĠContext , Ġbut Ġfor Ġthe Ġprice Ġmode . ĠThe Ġinput Ġis Ġthe Ġordered Ġsequence ĠDaily ĠPrices Ġ${ \ l brace ĠDP ^{ sc } _{ t - T }, Ġ\ cd ots Ġ, ĠDP ^{ sc } _ t Ġ\ r brace Ġ} $ Ġof Ġsize Ġ$ T $ Ġ, Ġwhere Ġeach Ġelement Ġthe Ġprice Ġfeature Ġdefined Ġin Ġeq : pr ic em ode vec . ĠParticularly , Ġthe Ġarchitecture Ġconsists Ġof Ġtwo Ġstacked ĠL ST M 's . ĠThe Ġfirst Ġone Ġoutputs Ġfor Ġeach Ġprice Ġfeature Ġtime Ġstep Ġa Ġhidden Ġvector Ġthat Ġtakes Ġthe Ġtemporal Ġcontext Ġinto Ġaccount . ĠThen Ġthese Ġhidden Ġvectors Ġare Ġagain Ġpassed Ġto Ġa Ġsecond Ġindependent ĠL ST M . ĠThe Ġlayer Ġoutputs Ġthe Ġprice Ġmode Ġencoding ĠMarket ĠPrice Ġ$ MP ^{ sc } _ t Ġ\ in Ġd _{ MP } $ Ġ. ĠThis Ġencoding Ġis Ġthe Ġlast Ġhidden Ġvector Ġof Ġthe Ġsecond ĠL ST M ĠMarket . </s> âĢĵ ĠStock Ġembed ding </s> Stock ĠEnc oder </s> Stock Ġdense Ġrepresentation . ĠThe Ġlayer Ġreceives Ġthe Ġdiscrete Ġencoding Ġ$\ math cal Ġ{ I } ^{ sc } _ t $ Ġindicating Ġthe Ġsample Ġstock Ġcode Ġpass Ġthrough Ġa ĠFC Ġlayer Ġand Ġoutputs Ġa Ġstock Ġembed ding Ġ$ E _{ sc } $ Ġ. </s> âĢĵ ĠJoint ĠRepresent ation </s> Mer ging </s> Feature - wise ĠNews , ĠPrice , Ġand ĠStock Ġmodes Ġconc aten ation . ĠNo Ġtrain able Ġparameters . </s> J oint ĠRepresent ation ĠEnc oder </s> FC Ġlayer Ġof Ġsize Ġ$ d _{ JR } $ Ġ. </s> Mult im od al Ġlearning Ġwith Ġmissing Ġmodes </s> During Ġthe Ġtraining Ġwe Ġfeed Ġinto Ġour Ġneural Ġmodel Ġthe Ġprice , Ġnews , Ġand Ġstock Ġindicator Ġdata . ĠThe Ġprice Ġand Ġstock Ġindicator Ġmodes Ġdata Ġoccur Ġin Ġall Ġdays . ĠHowever , Ġat Ġthe Ġindividual Ġstock Ġlevel Ġwe Ġcan Ġhave Ġdays Ġthat Ġthe Ġcompany Ġis Ġnot Ġcovered Ġby Ġthe Ġmedia . ĠThis Ġfeature Ġimposes Ġchallenges Ġto Ġour Ġmultim od al Ġtraining Ġsince Ġneural Ġnetworks Ġare Ġnot Ġable Ġto Ġhandle Ġmissing Ġmodes Ġwithout Ġspecial Ġintervention . ĠA Ġstraightforward Ġsolution Ġwould Ġbe Ġto Ġconsider Ġonly Ġdays Ġwith Ġnews Ġreleased , Ġdisreg arding Ġthe Ġremaining Ġsamples . ĠHowever , Ġthis Ġapproach Ġhas Ġtwo Ġmain Ġdrawbacks . ĠFirst , Ġthe ĠâĢ ľ missing Ġnews âĢ Ŀ Ġdo Ġnot Ġhappen Ġat Ġrandom , Ġor Ġare Ġattributed Ġto Ġmeasurement Ġfailure Ġas Ġis , Ġfor Ġexample , Ġthe Ġcase Ġof Ġmultim od al Ġtasks Ġusing Ġmechanical Ġsensors Ġdata . ĠConversely , Ġas Ġhighlighted Ġin ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġthe Ġsame Ġprice Ġbehaviour Ġresults Ġin Ġdistinct Ġmarket Ġreactions Ġwhen Ġaccompanied Ġor Ġnot Ġby Ġnews . ĠIn Ġother Ġwords , Ġspecifically Ġto Ġfinancial Ġforecasting Ġproblems Ġthe Ġabsence Ġor Ġexistence Ġof Ġnews Ġare Ġhighly Ġinformative . </s> Some Ġmethods Ġwere Ġproposed Ġin Ġthe Ġmultim od al Ġliterature Ġto Ġeffectively Ġtreat Ġinformative Ġmissing Ġmodes Ġor ĠâĢ ľ in form ative Ġmissing ness âĢ Ŀ , Ġwhich Ġis Ġa Ġcharacteristic Ġrefere ed Ġin Ġthe Ġliterature Ġas Ġlearning Ġwith Ġmissing Ġmod alities ĠB IB REF 22 Ġ. ĠIn Ġthis Ġwork , Ġwe Ġdirectly Ġmodel Ġthe Ġnews Ġmissing ness Ġas Ġa Ġfeature Ġof Ġour Ġtext Ġmodel Ġtemporal Ġsequence Ġby Ġusing Ġthe Ġmethod Ġinitially Ġproposed Ġin ĠB IB REF 45 Ġ, ĠB IB REF 46 Ġfor Ġclinical Ġdata Ġwith Ġmissing Ġmeasurements Ġand Ġapplied Ġin Ġthe Ġcontext Ġof Ġfinancial Ġforecasting Ġin ĠB IB REF 47 Ġ. ĠSpecifically , Ġwe Ġimplement Ġthe ĠZ eros Ġ& ĠIm put ation Ġ( ZI ) Ġmethod ĠB IB REF 46 Ġin Ġorder Ġto Ġjointly Ġlearn Ġthe Ġprice Ġmode Ġand Ġnews Ġrelationship Ġacross Ġall Ġdays Ġof Ġmarket Ġactivity . </s> The ĠZ I Ġimplementation Ġis Ġdescribed Ġas Ġfollows : ĠBefore Ġthe Ġdaily Ġnews Ġsequence Ġis Ġprocessed Ġby Ġthe Ġtext Ġtemporal Ġlayer Ġ( described Ġin Ġit m : new st cl ayer ) Ġwe Ġinput Ġa Ġ0 Ġvector Ġfor Ġall Ġtime Ġsteps Ġwith Ġmissing Ġnews Ġand Ġleave Ġthe Ġnews Ġencoding Ġunchanged Ġotherwise . ĠThis Ġstep Ġis Ġcalled Ġzero Ġimp utation . ĠIn Ġaddition , Ġwe Ġconc aten ate Ġfeature - wise Ġan Ġindicator Ġvector Ġwith Ġvalue Ġ1 Ġfor Ġall Ġvectors Ġwith Ġzero Ġimp utation Ġand Ġ0 Ġfor Ġthe Ġdays Ġwith Ġnews . </s> As Ġdescribed Ġin ĠB IB REF 47 Ġ, Ġthe ĠZ I Ġmethod Ġend ow Ġa Ġtemporal Ġsequence Ġmodel Ġwith Ġthe Ġability Ġto Ġlearn Ġdifferent Ġrepresentations Ġdepending Ġon Ġthe Ġnews Ġhistory Ġand Ġits Ġrelative Ġtime Ġposition . ĠMoreover , Ġit Ġallows Ġour Ġmodel Ġto Ġpredict Ġthe Ġvolatility Ġfor Ġall Ġdays Ġof Ġour Ġtime Ġseries Ġand , Ġat Ġthe Ġsame Ġtime , Ġtake Ġinto Ġaccount Ġthe Ġcurrent Ġand Ġpast Ġnews Ġinformative Ġmissing ness . ĠFurthermore , Ġthe Ġlearnt Ġpositional Ġnews Ġencoding Ġworks Ġdifferently Ġthan Ġa Ġtypical ĠâĢ ľ mask ing âĢ Ŀ , Ġwhere Ġdays Ġwithout Ġnews Ġare Ġnot Ġpassed Ġthrough Ġthe ĠL ST M Ġcell . ĠMask ing Ġthe Ġtime Ġsteps Ġwould Ġbe Ġlosing Ġinformation Ġabout Ġthe Ġpresence Ġor Ġabsence Ġof Ġnews Ġcon com itant Ġwith Ġprices . </s> Exper imental Ġresults Ġand Ġdiscussions </s> We Ġaim Ġto Ġevaluate Ġour Ġhierarchical Ġneural Ġmodel Ġin Ġthe Ġlight Ġof Ġthree Ġmain Ġaspects . ĠFirst , Ġwe Ġasses Ġthe Ġimportance Ġof Ġthe Ġdifferent Ġsentence Ġenc od ers Ġto Ġour Ġend - to - end Ġmodels Ġand Ġhow Ġit Ġcompares Ġto Ġtransferring Ġthe Ġsentence Ġenc oder Ġfrom Ġour Ġtwo Ġauxiliary ĠTL Ġtasks . ĠSecond , Ġwe Ġab late Ġour Ġproposed Ġnews Ġrelevance Ġattention Ġ( N RA ) Ġcomponent Ġto Ġevaluate Ġits Ġimportance . ĠFinally , Ġwe Ġconsider Ġa Ġmodel Ġthat Ġtakes Ġinto Ġconsideration Ġonly Ġthe Ġprice Ġmode Ġ( un im od al ), Ġi . e . Ġignoring Ġany Ġarchitecture Ġrelated Ġto Ġthe Ġtext Ġmode . </s> Before Ġwe Ġdefine Ġthe Ġbas elines Ġto Ġasses Ġthe Ġthree Ġaspects Ġdescribed Ġabove , Ġwe Ġreview Ġin Ġthe Ġnext Ġsection Ġthe Ġscores Ġof Ġthe Ġtrained ĠTL Ġtasks . </s> A ux iliary Ġtransfer Ġlearning Ġtasks </s> This Ġsection Ġreports Ġthe Ġperformance Ġof Ġthe Ġauxiliary ĠTL Ġtasks Ġconsidered Ġin Ġthis Ġwork . ĠOur Ġultimate Ġgoal Ġis Ġto Ġindicate Ġthat Ġour Ġscores Ġare Ġin Ġline Ġwith Ġprevious Ġworks ĠAll Ġthe Ġarchitectures Ġpresented Ġin Ġsec : transfer learning Ġare Ġtrained Ġfor Ġa Ġmaximum Ġof Ġ50 Ġepoch s Ġusing Ġmini - batch ĠSG D Ġwith ĠAdam Ġoptim izer ĠB IB REF 48 Ġ. ĠMoreover , Ġat Ġthe Ġend Ġof Ġeach Ġepoch , Ġwe Ġevaluate Ġthe Ġvalidation Ġscores , Ġwhich Ġare Ġaccuracy Ġ( Stan for ĠSN LI Ġdataset ) Ġand ĠF 1 Ġ( RC V 1 Ġdataset ), Ġand Ġsave Ġthe Ġweights Ġwith Ġthe Ġbest Ġvalues . ĠAim ing Ġto Ġse ep ed Ġup Ġtraining , Ġwe Ġimplement Ġearly Ġstopping Ġwith Ġpatience Ġset Ġto Ġ8 Ġepoch s . ĠThat Ġis , Ġif Ġthe Ġvalidation Ġscores Ġdo Ġnot Ġimprove Ġfor Ġmore Ġthan Ġ10 Ġepoch s Ġwe Ġhalt Ġthe Ġtraining . ĠFinally , Ġwe Ġuse ĠGl ove Ġpre - trained Ġword Ġembed d ings ĠB IB REF 37 Ġas Ġfixed Ġfeatures . </s> t bl : tle val uation Ġcompares Ġour Ġtest Ġscores Ġwith Ġstate - of - the - art Ġ( S OTA ) Ġresults Ġreported Ġin Ġprevious Ġworks . ĠWe Ġcan Ġsee Ġthat Ġour Ġscores Ġfor Ġthe ĠSN LI Ġtask Ġare Ġvery Ġclose Ġto Ġstate - of - the - art . </s> Regarding Ġthe ĠRC V 1 Ġdataset , Ġour Ġresults Ġconsider Ġonly Ġthe Ġheadline Ġcontent Ġfor Ġtraining , Ġwhile Ġthe Ġrefere ed Ġworks Ġconsider Ġboth Ġthe Ġnews Ġheadline Ġand Ġmessage Ġbody . ĠThe Ġreason Ġfor Ġtraining Ġusing Ġonly Ġthe Ġheadlines Ġis Ġthat Ġboth Ġtasks Ġare Ġlearnt Ġwith Ġthe Ġsole Ġpurpose Ġof Ġtransferring Ġthe Ġsentence Ġenc od ers Ġto Ġour Ġmain Ġvolatility Ġprediction Ġtask , Ġwhose Ġtextual Ġinput Ġis Ġrestricted Ġto Ġheadlines . </s> Training Ġsetup </s> During Ġthe Ġtraining Ġof Ġour Ġhierarchical Ġneural Ġmodel Ġdescribed Ġin Ġsub : H AN Ġwe Ġtook Ġspecial Ġcare Ġto Ġguard Ġagainst Ġover fitting . ĠTo Ġthis Ġaim , Ġwe Ġcompletely Ġseparate Ġ2016 Ġand Ġ2017 Ġas Ġthe Ġtest Ġset Ġand Ġreport Ġour Ġresults Ġon Ġthis ĠâĢ ľ un seen âĢ Ŀ Ġset . ĠThe Ġremaining Ġdata Ġis Ġfurther Ġsplit Ġinto Ġtraining Ġ( 2007 Ġto Ġ2013 ) Ġand Ġvalidation Ġ( 2014 Ġto Ġ2015 ). ĠThe Ġmodel Ġconvergence Ġduring Ġtraining Ġis Ġmonitored Ġin Ġthe Ġvalidation Ġset . ĠWe Ġmonitor Ġthe Ġvalidation Ġscore Ġof Ġour Ġmodel Ġat Ġthe Ġend Ġof Ġeach Ġepoch Ġand Ġstore Ġthe Ġnetwork Ġweights Ġif Ġthe Ġvalidation Ġscores Ġimproves Ġbetween Ġtwo Ġconsecutive Ġepoch s . ĠAdditionally , Ġwe Ġuse Ġmini - batch ĠSG D Ġwith ĠAdam Ġoptim izer Ġand Ġearly Ġstopping Ġwith Ġpatience Ġset Ġto Ġeight Ġepoch s . ĠThe Ġhyper param eter Ġtun ning Ġis Ġperformed Ġusing Ġgrid Ġsearch . </s> All Ġtraining Ġis Ġperformed Ġusing Ġthe Ġproposed Ġglobal Ġmodel Ġapproach Ġdescribed Ġin Ġsub : global model , Ġwhich Ġlearns Ġa Ġmodel Ġthat Ġtakes Ġinto Ġaccount Ġthe Ġfeatures Ġof Ġall Ġthe Ġ40 Ġstocks Ġof Ġour Ġcorpus . ĠUsing Ġthis Ġapproach Ġour Ġtraining Ġset Ġhas Ġa Ġtotal Ġof Ġ97 , 9 03 Ġsamples . ĠMoreover , Ġduring Ġthe ĠSG D Ġmini - batch Ġsampling Ġthe Ġpast Ġ$ T $ Ġdays Ġof Ġprice Ġand Ġnews Ġhistory Ġtens ors Ġand Ġeach Ġstock Ġsample Ġstock Ġindicator Ġare Ġrandomly Ġselected Ġfrom Ġthe Ġset Ġof Ġall Ġ40 Ġstocks . </s> St ocks Ġuniverse Ġresult </s> In Ġorder Ġto Ġevaluate Ġthe Ġcontributions Ġof Ġeach Ġcomponent Ġof Ġour Ġneural Ġmodel Ġdescribed Ġin Ġsub : H AN Ġand Ġthe Ġeffect Ġof Ġusing Ġtextual Ġdata Ġto Ġpredict Ġthe Ġvolatility , Ġwe Ġreport Ġour Ġresults Ġusing Ġthe Ġfollowing Ġbas elines : </s> - ĠNews Ġ( un im od al Ġprice Ġonly ): ĠThis Ġbaseline Ġcompletely Ġabl ates Ġ( i . e . Ġremoves ) Ġany Ġarchitecture Ġrelated Ġto Ġthe Ġnews Ġmode , Ġconsidering Ġonly Ġthe Ġprice Ġencoding Ġand Ġthe Ġstock Ġembed ding Ġcomponents . ĠUsing Ġthis Ġab lation Ġwe Ġaim Ġto Ġevaluate Ġthe Ġinfluence Ġof Ġnews Ġto Ġthe Ġvolatility Ġprediction Ġproblem . </s> + ĠNews Ġ( End - to - end ĠSent ence ĠEnc od ers ) Ġ- ĠNRA : ĠThis Ġbaseline Ġabl ates Ġour Ġproposed Ġnew Ġrelevance Ġattention Ġ( N RA ) Ġcomponent , Ġand Ġinstead , Ġmakes Ġuse Ġof Ġthe Ġsame ĠDaily ĠA ver aging Ġmethod Ġin ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ, Ġwhere Ġall Ġfixed - sized Ġheadline Ġrepresentations Ġon Ġa Ġgiven Ġday Ġare Ġaveraged Ġwithout Ġtaking Ġinto Ġaccount Ġthe Ġrelevance Ġof Ġeach Ġnews . ĠWe Ġevaluate Ġthis Ġbaseline Ġfor Ġboth ĠBi L ST M Ġattention Ġ( Att ) Ġand ĠBi L ST M Ġmax - pool ing Ġ( MP ) Ġsentence Ġenc od ers . ĠHere , Ġour Ġgoal Ġis Ġto Ġasses Ġthe Ġtrue Ġcontribution Ġof Ġour ĠNRA Ġcomponent Ġin Ġthe Ġcase ĠS OTA Ġsentence Ġenc od ers Ġare Ġtaken Ġinto Ġaccount . </s> + ĠNews Ġ( End - to - End ĠW - L ĠAtt ĠSent ence ĠEnc oder ) Ġ+ ĠNRA : ĠThe ĠWord - Level ĠAttention Ġ( W - L ĠAtt ) Ġsentence Ġenc oder Ġimplements Ġan Ġattention Ġmechanism Ġdirectly Ġon Ġtop Ġof Ġword Ġembed d ings , Ġand , Ġas Ġsuch , Ġdoes Ġnot Ġconsider Ġthe Ġorder Ġof Ġwords Ġin Ġa Ġsentence . ĠThis Ġbaseline Ġcomple ments Ġthe Ġprevious Ġone , Ġi . e . Ġit Ġevaluates Ġthe Ġinfluence Ġof Ġthe Ġsentence Ġenc oder Ġwhen Ġour Ġfull Ġspecification Ġis Ġconsidered . </s> + ĠNews Ġ( TL ĠSent ence ĠEnc od ers ) Ġ+ ĠNRA : ĠMakes Ġuse Ġof Ġsentence Ġenc od ers Ġof Ġour Ġtwo Ġauxiliary ĠTL Ġtasks Ġas Ġfixed Ġfeatures . ĠThis Ġbaseline Ġaims Ġto Ġaddress Ġthe Ġfollowing Ġquestions , Ġnamely : ĠWhat Ġdataset Ġand Ġmodels Ġare Ġmore Ġsuitable Ġto Ġtransfer Ġto Ġour Ġspecific Ġvolatility Ġforecasting Ġproblem ; ĠHow ĠEnd - to - End Ġmodels , Ġwhich Ġare Ġtrained Ġon Ġtop Ġof Ġword Ġembed d ings , Ġperform Ġcompared Ġto Ġsentence Ġenc od ers Ġtransferred Ġfrom Ġother Ġtasks . </s> t bl : com par ative all se ctors Ġsummarizes Ġthe Ġtest Ġscores Ġfor Ġthe Ġabl ations Ġdiscussed Ġabove . ĠOur Ġbest Ġmodel Ġis Ġthe Ġ+ ĠNews Ġ( Bi L ST M ĠAtt ) Ġ+ ĠNRA , Ġwhich Ġis Ġtrained Ġend - to - end Ġand Ġuses Ġour Ġfull Ġarchitecture . ĠThe Ġsecond Ġbest Ġmodel , Ġi . e . Ġ+ ĠNews Ġ( Bi L ST M ĠMP ) Ġ+ ĠNRA , Ġranks Ġslightly Ġlower Ġand Ġonly Ġdiffers Ġform Ġthe Ġbest Ġmodel Ġin Ġterms Ġof Ġthe Ġsentence Ġenc oder . ĠThe Ġformer Ġsentence Ġenc oder Ġuses Ġan Ġattention Ġlayer Ġ( sub sec : b ilst min ner att ) Ġand Ġthe Ġthe Ġlast Ġa Ġmax - pool ing Ġlayer Ġ( sub sec : b ilst mm ax pool ), Ġwhere Ġboth Ġlayers Ġare Ġplaced Ġon Ġtop Ġof Ġthe ĠL ST M Ġhidden Ġstates Ġof Ġeach Ġword . </s> Import antly , Ġour Ġexperiments Ġshow Ġthat Ġusing Ġnews Ġand Ġprice Ġ( mult im od al ) Ġto Ġpredict Ġthe Ġvolatility Ġimproves Ġthe Ġscores Ġby Ġ11 % Ġ( M SE ) Ġand Ġ9 % Ġ( MA E ) Ġwhen Ġcompared Ġwith Ġthe ĠâĢĵ ĠNews Ġ( price Ġonly Ġunim od al ) Ġmodel Ġthat Ġconsiders Ġonly Ġprice Ġfeatures Ġas Ġexplanatory Ġvariables . </s> When Ġcomparing Ġthe Ġperformance Ġof ĠEnd - to - End Ġmodels Ġand Ġthe ĠTL Ġauxiliary Ġtasks Ġthe Ġfollowing Ġcan Ġbe Ġobserved : ĠThe Ġend - to - end Ġmodels Ġtrained Ġwith Ġthe Ġtwo ĠS OTA Ġsentence Ġenc od ers Ġperform Ġbetter Ġthan Ġtransferring Ġsentence Ġenc oder Ġfrom Ġboth Ġauxiliary Ġtasks . ĠHowever , Ġour Ġexperiments Ġshow Ġthat Ġthe Ġsame Ġdoes Ġnot Ġhold Ġfor Ġmodels Ġtrained Ġend - to - end Ġrelying Ġon Ġthe Ġsimpler ĠW L - Att Ġsentence Ġenc oder , Ġwhich Ġignores Ġthe Ġorder Ġof Ġwords Ġin Ġa Ġsentence . ĠIn Ġother Ġwords , Ġconsidering Ġthe Ġappropriate ĠTL Ġtask , Ġit Ġis Ġpreferable Ġto Ġtransfer Ġa ĠS OTA Ġsentence Ġenc oder Ġtrained Ġon Ġa Ġlarger Ġdataset Ġthan Ġlearning Ġa Ġless Ġrobust Ġsentence Ġenc oder Ġin Ġan Ġend - to - end Ġfashion . ĠMoreover , Ġinitially , Ġwe Ġthought Ġthat Ġbeing Ġthe ĠRC V 1 Ġa Ġfinancial Ġdomain Ġcorpus Ġit Ġwould Ġdemonstrate Ġa Ġsuperior Ġperformance Ġwhen Ġcompared Ġto Ġthe ĠSN LI Ġdataset . ĠStill , Ġthe ĠSN LI Ġtransfers Ġbetter Ġthan ĠRC V 1 . ĠWe Ġhypothes ize Ġthat Ġthe Ġtext Ġcategor ization Ġtask Ġ( RC V 1 Ġdataset ) Ġis Ġnot Ġable Ġto Ġcapture Ġcomplex Ġsentence Ġstructures Ġat Ġthe Ġsame Ġlevel Ġrequired Ġto Ġperform Ġnatural Ġlanguage Ġinference . ĠParticularly Ġto Ġthe Ġvolatility Ġforecasting Ġproblem , Ġour ĠTL Ġresults Ġcorrobor ates Ġthe Ġsame Ġfindings Ġin ĠB IB REF 17 Ġ, Ġwhere Ġit Ġwas Ġshown Ġthat ĠSN LI Ġdataset Ġatt ains Ġthe Ġbest Ġsentence Ġencoding Ġfor Ġa Ġbroad Ġrange Ġof Ġpure ĠN LP Ġtasks , Ġincluding , Ġamong Ġother , Ġtext Ġcategor ization Ġand Ġsentiment Ġanalysis . </s> Sign ificantly , Ġexperimental Ġresults Ġin Ġt bl : com par ative all se ctors Ġclearly Ġdemonstrate Ġthat Ġour Ġproposed Ġnews Ġrelevance Ġattention Ġ( N RA ) Ġoutper forms Ġthe ĠNews ĠA ver aging Ġmethod Ġproposed Ġin Ġprevious Ġstudies ĠB IB REF 26 Ġ, ĠB IB REF 27 Ġ. ĠEven Ġwhen Ġevaluating Ġour ĠNRA Ġcomponent Ġin Ġconjunction Ġwith Ġthe Ġmore Ġelementary ĠW - L ĠAtt Ġsentence Ġenc oder Ġit Ġsurpass Ġthe Ġresults Ġof Ġsophisticated Ġsentence Ġenc oder Ġusing Ġa ĠNews ĠA ver aging Ġapproach . ĠIn Ġother Ġwords , Ġour Ġresults Ġstrongly Ġpoints Ġto Ġthe Ġadvantage Ġof Ġdiscriminating Ġnoisy Ġfrom Ġimpacting Ġnews Ġand Ġthe Ġeffectiveness Ġof Ġlearning Ġto Ġattend Ġthe Ġmost Ġrelevant Ġnews . </s> Having Ġanalyzed Ġour Ġbest Ġmodel , Ġwe Ġnow Ġturn Ġto Ġits Ġcomparative Ġperformance Ġwith Ġrespect Ġto Ġthe Ġwidely Ġregarded ĠG ARCH ( 1 , 1 ) Ġmodel Ġdescribed Ġin Ġsec : G ARCH . </s> We Ġasses Ġour Ġmodel Ġperformance Ġrelative Ġto ĠG ARCH ( 1 , 1 ) Ġusing Ġstandard Ġloss Ġmetrics Ġ( M SE Ġand ĠMA E ) Ġand Ġthe Ġregression - based Ġaccuracy Ġspecified Ġin Ġeq : reg ression loss Ġand Ġmeasured Ġin Ġterms Ġof Ġthe Ġcoefficient Ġof Ġdetermination Ġ$ R ^ 2 $ Ġ. ĠIn Ġaddition , Ġwe Ġevaluate Ġour Ġmodel Ġacross Ġtwo Ġdifferent Ġvolatility Ġproxies : ĠGar man - K lass Ġ( Ġ$\ wide hat {\ s igma Ġ_ { G K }} $ Ġ) Ġ( eq : vol g k ) Ġand ĠParkinson Ġ( Ġ$\ wide hat {\ s igma Ġ_ { PK }} $ Ġ) Ġ( eq : vol p k ). ĠWe Ġnote Ġthat , Ġas Ġreviewed Ġin Ġsub : eval ution , Ġthese Ġtwo Ġvolatility Ġproxies Ġare Ġstatically Ġefficient Ġand Ġproper Ġestim ators Ġof Ġthe Ġnext Ġday Ġvolatility . </s> t bl : g arch all se ctors Ġreports Ġthe Ġcomparative Ġperformance Ġamong Ġour Ġbest ĠPrice Ġ+ ĠNews Ġmodel Ġ(+ ĠNews ĠBi L ST M Ġ( MP ) Ġ+ ĠNRA ), Ġour ĠPrice Ġonly Ġ( un im od al ) Ġmodel Ġand ĠG ARCH ( 1 , 1 ). ĠThe Ġresults Ġclearly Ġdemonstrate Ġthe Ġsuperiority Ġof Ġour Ġmodel , Ġbeing Ġmore Ġaccurate Ġthan ĠGR ACH Ġfor Ġboth Ġvolatility Ġproxies . ĠWe Ġnote Ġthat Ġevaluating Ġthe ĠG ARCH ( 1 , 1 ) Ġmodel Ġrelying Ġon Ġstandard ĠM SE Ġand ĠMA E Ġerror Ġmetrics Ġshould Ġbe Ġtaken Ġwith Ġa Ġgrain Ġof Ġsalt . ĠB IB REF 35 Ġprovides Ġthe Ġbackground Ġtheory Ġand Ġarguments Ġsupporting Ġ$ R ^ 2 $ Ġas Ġthe Ġmetric Ġof Ġchoice Ġto Ġevaluate Ġthe Ġpredictive Ġpower Ġof Ġa Ġvolatility Ġmodel . ĠIn Ġany Ġcase , Ġthe Ġout performance Ġor Ġour Ġmodel Ġwith Ġrespect Ġto ĠG ARCH ( 1 , 1 ) Ġperme ates Ġall Ġthree Ġmetrics , Ġname Ġ$ R ^ 2 $ Ġ, Ġ$ M SE $ Ġand Ġ$ MA E $ Ġ. </s> S ector - level Ġresults </s> Company Ġsectors Ġare Ġexpected Ġto Ġhave Ġdifferent Ġrisk Ġlevels , Ġin Ġthe Ġsense Ġthat Ġeach Ġsector Ġis Ġdriven Ġby Ġdifferent Ġtypes Ġof Ġnews Ġand Ġeconomic Ġcycles . ĠMoreover , Ġby Ġperforming Ġa Ġsector - level Ġanalysis Ġwe Ġwere Ġinitially Ġinterested Ġin Ġunderstanding Ġif Ġthe Ġout performance Ġof Ġour Ġmodel Ġwith Ġrespect Ġto ĠG ARCH ( 1 , 1 ) Ġwas Ġthe Ġresult Ġof Ġa Ġlearning Ġbias Ġto Ġa Ġgiven Ġsector Ġor Ġif , Ġas Ġturned Ġout Ġto Ġbe Ġthe Ġcase , Ġthe Ġsuperior Ġperformance Ġof Ġour Ġmodel Ġspreads Ġacross Ġa Ġdivers ified Ġportfolio Ġof Ġsectors . </s> In Ġorder Ġto Ġevaluate Ġthe Ġperformance Ġper Ġsector , Ġwe Ġfirst Ġseparate Ġthe Ġconstituents Ġstocks Ġfor Ġeach Ġsector Ġin Ġt bl : stock un iverse . ĠThen , Ġwe Ġcalculate Ġthe Ġsame Ġmetrics Ġdiscussed Ġin Ġthe Ġprevious Ġsection Ġfor Ġeach Ġsector Ġindividually . </s> t bl : gar che ach sector Ġreports Ġour Ġexperimental Ġresults Ġsegregated Ġby Ġsector . ĠWe Ġobserve Ġthat Ġthe ĠGR ACH Ġmodel Ġaccuracy , Ġmeasured Ġusing Ġthe Ġ$ R ^ 2 $ Ġscore , Ġhas Ġa Ġhigh Ġdegree Ġof Ġvariability Ġamong Ġsectors . ĠFor Ġexample , Ġthe Ġaccuracy Ġranges Ġfrom Ġ0 . 15 Ġto Ġ0 . 44 Ġfor Ġthe ĠHealth Care Ġand ĠEnergy Ġsector , Ġrespectively . ĠThis Ġhigh Ġdegree Ġof Ġvariability Ġis Ġin Ġagreement Ġwith Ġprevious Ġresults Ġreported Ġin ĠB IB REF 16 Ġ, Ġbut Ġin Ġthe Ġcontext Ġof Ġlong - term Ġ( quarter ly ) Ġvolatility Ġpredictions . ĠAlthough Ġthe ĠG ARCH ( 1 , 1 ) Ġaccuracy Ġis Ġsector - dependent , Ġwithout Ġany Ġexception , Ġour Ġmodel Ġusing Ġprice Ġand Ġnews Ġas Ġinput Ġclearly Ġoutper forms ĠGR ACH Ġsector - wise . ĠThis Ġfact Ġallow Ġus Ġto Ġdraw Ġthe Ġfollowing Ġconclusions : </s> Our Ġmodel Ġout performance Ġis Ġpersistent Ġacross Ġsectors , Ġi . e . Ġthe Ġcharacteristics Ġof Ġthe Ġresults Ġreported Ġin Ġt bl : g arch all se ctors Ġperme ates Ġall Ġsectors , Ġrather Ġthan Ġbeing Ġcomposed Ġof Ġa Ġmix Ġof Ġoutper forming Ġand Ġunder performing Ġsector Ġcontributions . ĠThis Ġfact Ġprovides Ġa Ġstrong Ġevidence Ġthat Ġour Ġmodel Ġis Ġmore Ġaccurate Ġthan ĠG ARCH ( 1 , 1 ). </s> The Ġproposed ĠGlobal Ġmodel Ġapproach Ġdiscussed Ġin Ġsub : global model Ġis Ġable Ġto Ġgeneral ize Ġwell , Ġi . e . Ġthe Ġpatterns Ġlearnt Ġare Ġnot Ġbiased Ġto Ġa Ġgiven Ġsector Ġor Ġstock . </s> One Ġof Ġthe Ġlimitations Ġof Ġour Ġwork Ġis Ġto Ġrely Ġon Ġproxies Ġfor Ġthe Ġvolatility Ġestimation . ĠAlthough Ġthese Ġproxies Ġare Ġhandy Ġif Ġonly Ġopen , Ġhigh , Ġlow Ġand Ġclose Ġdaily Ġprice Ġdata Ġis Ġavailable , Ġhaving Ġhigh Ġfrequency Ġprice Ġdata Ġwe Ġcould Ġestimate Ġthe Ġdaily Ġvolatility Ġusing Ġthe Ġsum Ġof Ġsquared Ġint rad ay Ġreturns Ġto Ġmeasure Ġthe Ġtrue Ġdaily Ġlatent Ġvolatility . ĠFor Ġexample , Ġin Ġevaluating Ġthe Ġperformance Ġfor Ġthe Ġone - day - ahead ĠG ARCH ( 1 , 1 ) ĠYen / D ollar Ġexchange Ġrate ĠB IB REF 35 Ġreports Ġ$ R ^ 2 $ Ġvalues Ġof Ġ0 . 237 Ġand Ġ0 . 392 Ġusing Ġhourly Ġand Ġfive Ġminutes Ġsampled Ġint rad ay Ġreturns , Ġrespectively . ĠHowever , Ġwe Ġbelieve Ġthat Ġutilizing Ġint rad ay Ġdata Ġwould Ġfurther Ġimprove Ġour Ġmodel Ġperformance . </s> Since Ġour Ġexperimental Ġresults Ġdemonstrate Ġthe Ġkey Ġaspect Ġof Ġthe Ġnews Ġrelevance Ġattention Ġto Ġmodel Ġarchitecture Ġwe Ġobserve Ġthat Ġint rad ay Ġdata Ġwould Ġarguably Ġam el ior ate Ġthe Ġlearning Ġprocess . ĠHaving Ġint rad ay Ġdata Ġwould Ġallow Ġus Ġto Ġpair Ġeach Ġindividual Ġnews Ġrelease Ġwith Ġthe Ġinstantaneous Ġmarket Ġprice Ġreaction . ĠUsing Ġdaily Ġdata Ġwe Ġare Ġlosing Ġpart Ġof Ġthis Ġinformation Ġby Ġonly Ġmeasuring Ġthe Ġaggregate Ġeffect Ġof Ġall Ġnews Ġto Ġthe Ġone - day - ahead Ġprediction . </s> Conclusion </s> We Ġstudy Ġthe Ġjoint Ġeffect Ġof Ġstock Ġnews Ġand Ġprices Ġon Ġthe Ġdaily Ġvolatility Ġforecasting Ġproblem . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġthis Ġwork Ġis Ġone Ġof Ġthe Ġfirst Ġstudies Ġaiming Ġto Ġpredict Ġshort - term Ġ( daily ) Ġrather Ġthan Ġlong - term Ġ( quarter ly Ġor Ġyearly ) Ġvolatility Ġtaking Ġnews Ġand Ġprice Ġas Ġexplanatory Ġvariables Ġand Ġusing Ġa Ġcomprehensive Ġdataset Ġof Ġnews Ġheadlines Ġat Ġthe Ġindividual Ġstock Ġlevel . ĠOur Ġhierarchical Ġend - to - end Ġmodel Ġbenefits Ġfrom Ġstate - of - the - art Ġapproaches Ġto Ġencode Ġtext Ġinformation Ġand Ġto Ġdeal Ġwith Ġtwo Ġmain Ġchallenges Ġin Ġcorrel ating Ġnews Ġwith Ġmarket Ġreaction : Ġnews Ġrelevance Ġand Ġnovelty . ĠThat Ġis , Ġto Ġaddress Ġthe Ġproblem Ġof Ġhow Ġto Ġattend Ġthe Ġmost Ġimportant Ġnews Ġbased Ġpurely Ġon Ġits Ġcontent Ġ( news Ġrelevance Ġattention ) Ġand Ġto Ġtake Ġinto Ġaccount Ġthe Ġtemporal Ġinformation Ġof Ġpast Ġnews Ġ( tem poral Ġcontext ). ĠAdditionally , Ġwe Ġpropose Ġa Ġmulti - stock Ġmini - batch Ġ+ Ġstock Ġembed ding Ġmethod Ġsuitable Ġto Ġmodel Ġcommon ality Ġamong Ġstocks . </s> The Ġexperimental Ġresults Ġshow Ġthat Ġour Ġmultim od al Ġapproach Ġoutper forms Ġthe ĠG ARCH ( 1 , 1 ) Ġvolatility Ġmodel , Ġwhich Ġis Ġthe Ġmost Ġprevalent Ġe con ometric Ġmodel Ġfor Ġdaily Ġvolatility Ġpredictions . ĠThe Ġout performance Ġbeing Ġsector - wise Ġand Ġdemonstrates Ġthe Ġeffectiveness Ġof Ġcombining Ġprice Ġand Ġnews Ġfor Ġshort - term Ġvolatility Ġforecasting . ĠThe Ġfact Ġthat Ġwe Ġoutper form ĠG ARCH ( 1 , 1 ) Ġfor Ġall Ġanalyzed Ġsectors Ġconfirms Ġthe Ġrobust ness Ġof Ġour Ġproposed Ġarchitecture Ġand Ġevid ences Ġthat Ġour Ġglobal Ġmodel Ġapproach Ġgeneral izes Ġwell . </s> We Ġab lated Ġ( i . e . Ġremoved ) Ġdifferent Ġcomponents Ġof Ġour Ġneural Ġarchitecture Ġto Ġassess Ġits Ġmost Ġrelevant Ġparts . ĠTo Ġthis Ġaim , Ġwe Ġreplaced Ġour Ġproposed Ġnews Ġrelevance Ġattention Ġlayer , Ġwhich Ġaims Ġto Ġattend Ġthe Ġmost Ġimportant Ġnews Ġon Ġa Ġgiven Ġday , Ġwith Ġa Ġsimpler Ġarchitecture Ġproposed Ġin Ġthe Ġliterature , Ġwhich Ġaverages Ġthe Ġdaily Ġnews . ĠWe Ġfound Ġthat Ġour Ġattention Ġlayer Ġimproves Ġthe Ġresults . ĠAdditionally , Ġwe Ġab lated Ġall Ġthe Ġarchitecture Ġrelated Ġto Ġthe Ġnews Ġmode Ġand Ġfound Ġthat Ġnews Ġenhances Ġthe Ġforecasting Ġaccuracy . </s> Finally , Ġwe Ġevaluated Ġdifferent Ġsentence Ġenc od ers , Ġincluding Ġthose Ġtransf ered Ġfrom Ġother ĠN LP Ġtasks , Ġand Ġconcluded Ġthat Ġthey Ġachieve Ġbetter Ġperformance Ġas Ġcompared Ġto Ġa Ġplain ĠWord - level Ġattention Ġsentence Ġenc oder Ġtrained Ġend - to - end . ĠHowever , Ġthey Ġdo Ġnot Ġbeat Ġstate