Dimension size, window size, architecture, algorithm, epochs, hidden dimension size, learning rate, loss function, optimizer algorithm.
ABSTRACTIVE
Hyperparameters explored were: dimension size, window size, architecture, algorithm and epochs.
ABSTRACTIVE
FLOAT SELECTED: Table 1: Hyper-parameter choices FLOAT SELECTED: Table 2: Network hyper-parameters
To form the vocabulary, words occurring less than 5 times in the corpora were dropped, stop words removed using the natural language toolkit (NLTK) (BIBREF22) and data pre-processing carried out. Table TABREF2 describes most hyper-parameters explored for each dataset. In all, 80 runs (of about 160 minutes) were conducted for the 15MB Wiki Abstract dataset with 80 serialized models totaling 15.136GB while 80 runs (for over 320 hours) were conducted for the 711MB SW dataset, with 80 serialized models totaling over 145GB. Experiments for all combinations for 300 dimensions were conducted on the 3.9GB training set of the BW corpus and additional runs for other dimensions for the window 8 + skipgram + heirarchical softmax combination to verify the trend of quality of word vectors as dimensions are increased. FLOAT SELECTED: Table 1: Hyper-parameter choices
2003.11645
Introduction </s> There Ġhave Ġbeen Ġmany Ġimplementations Ġof Ġthe Ġword 2 vec Ġmodel Ġin Ġeither Ġof Ġthe Ġtwo Ġarchitectures Ġit Ġprovides : Ġcontinuous Ġskip gram Ġand ĠCB o W Ġ( BI B REF 0 ). ĠSimilar Ġdistributed Ġmodels Ġof Ġword Ġor Ġsub word Ġembed d ings Ġ( or Ġvector Ġrepresentations ) Ġfind Ġusage Ġin Ġs ota , Ġdeep Ġneural Ġnetworks Ġlike ĠB ERT Ġand Ġits Ġsuccessors Ġ( BI B REF 1 , ĠB IB REF 2 , ĠB IB REF 3 ). ĠThese Ġdeep Ġnetworks Ġgenerate Ġcontextual Ġrepresentations Ġof Ġwords Ġafter Ġbeen Ġtrained Ġfor Ġextended Ġperiods Ġon Ġlarge Ġcorpor a , Ġun super vised , Ġusing Ġthe Ġattention Ġmechanisms Ġ( BI B REF 4 ). </s> It Ġhas Ġbeen Ġobserved Ġthat Ġvarious Ġhyper - param eter Ġcombinations Ġhave Ġbeen Ġused Ġin Ġdifferent Ġresearch Ġinvolving Ġword 2 vec Ġwith Ġthe Ġpossibility Ġof Ġmany Ġof Ġthem Ġbeing Ġsub - opt imal Ġ( BI B REF 5 , ĠB IB REF 6 , ĠB IB REF 7 ). ĠTherefore , Ġthe Ġauthors Ġseek Ġto Ġaddress Ġthe Ġresearch Ġquestion : Ġwhat Ġis Ġthe Ġoptimal Ġcombination Ġof Ġword 2 vec Ġhyper - param eters Ġfor Ġintrinsic Ġand Ġextr ins ic ĠN LP Ġpurposes ? ĠThere Ġare Ġastron om ically Ġhigh Ġnumbers Ġof Ġcombinations Ġof Ġhyper - param eters Ġpossible Ġfor Ġneural Ġnetworks , Ġeven Ġwith Ġjust Ġa Ġfew Ġlayers . ĠHence , Ġthe Ġscope Ġof Ġour Ġextensive Ġwork Ġover Ġthree Ġcorpor a Ġis Ġon Ġdimension Ġsize , Ġtraining Ġepoch s , Ġwindow Ġsize Ġand Ġvocabulary Ġsize Ġfor Ġthe Ġtraining Ġalgorithms Ġ( h ier arch ical Ġsoft max Ġand Ġnegative Ġsampling ) Ġof Ġboth Ġskip gram Ġand ĠCB o W . ĠThe Ġcorpor a Ġused Ġfor Ġword Ġembed d ings Ġare ĠEnglish ĠWiki ĠNews ĠAbstract Ġby ĠB IB REF 8 Ġof Ġabout Ġ15 MB , ĠEnglish ĠWiki ĠSimple Ġ( SW ) ĠArticles Ġby ĠB IB REF 9 Ġof Ġabout Ġ7 11 MB Ġand Ġthe ĠBillion ĠWord Ġ( BW ) Ġof Ġ3 . 9 GB Ġby ĠB IB REF 10 . ĠThe Ġcorpus Ġused Ġfor Ġsentiment Ġanalysis Ġis Ġthe ĠIM Db Ġdataset Ġof Ġmovie Ġreviews Ġby ĠB IB REF 11 Ġwhile Ġthat Ġfor ĠN ER Ġis ĠGron ingen ĠMeaning ĠBank Ġ( G MB ) Ġby ĠB IB REF 12 , Ġcontaining Ġ47 , 9 59 Ġsentence Ġsamples . ĠThe ĠIM Db Ġdataset Ġused Ġhas Ġa Ġtotal Ġof Ġ25 , 000 Ġsentences Ġwith Ġhalf Ġbeing Ġpositive Ġsentiments Ġand Ġthe Ġother Ġhalf Ġbeing Ġnegative Ġsentiments . ĠThe ĠGM B Ġdataset Ġhas Ġ17 Ġlabels , Ġwith Ġ9 Ġmain Ġlabels Ġand Ġ2 Ġcontext Ġtags . ĠIt Ġis Ġhowever Ġun balanced Ġdue Ġto Ġthe Ġhigh Ġpercentage Ġof Ġtokens Ġwith Ġthe Ġlabel Ġ' O '. ĠThis Ġskew Ġin Ġthe ĠGM B Ġdataset Ġis Ġtypical Ġwith ĠN ER Ġdatasets . </s> The Ġobjective Ġof Ġthis Ġwork Ġis Ġto Ġdetermine Ġthe Ġoptimal Ġcombinations Ġof Ġword 2 vec Ġhyper - param eters Ġfor Ġintrinsic Ġevaluation Ġ( sem antic Ġand Ġsynt actic Ġanalog ies ) Ġand Ġextr ins ic Ġevaluation Ġtasks Ġ( BI B REF 13 , ĠB IB REF 14 ), Ġlike ĠSA Ġand ĠN ER . ĠIt Ġis Ġnot Ġour Ġobjective Ġin Ġthis Ġwork Ġto Ġrecord Ġs ota Ġresults . ĠSome Ġof Ġthe Ġmain Ġcontributions Ġof Ġthis Ġresearch Ġare Ġthe Ġempirical Ġestablishment Ġof Ġoptimal Ġcombinations Ġof Ġword 2 vec Ġhyper - param eters Ġfor ĠN LP Ġtasks , Ġdiscovering Ġthe Ġbehaviour Ġof Ġquality Ġof Ġvectors Ġviz - a - v iz Ġincreasing Ġdimensions Ġand Ġthe Ġconfirmation Ġof Ġembed d ings Ġbeing Ġtask - specific Ġfor Ġthe Ġdownstream . ĠThe Ġrest Ġof Ġthis Ġpaper Ġis Ġorganised Ġas Ġfollows : Ġthe Ġliterature Ġreview Ġthat Ġbriefly Ġsurveys Ġdistributed Ġrepresentation Ġof Ġwords , Ġparticularly Ġword 2 vec ; Ġthe Ġmethodology Ġemployed Ġin Ġthis Ġresearch Ġwork ; Ġthe Ġresults Ġobtained Ġand Ġthe Ġconclusion . </s> Liter ature ĠReview </s> Breaking Ġaway Ġfrom Ġthe Ġnon - dist ributed Ġ( high - dimensional , Ġsparse ) Ġrepresentations Ġof Ġwords , Ġtypical Ġof Ġtraditional Ġbag - of - words Ġor Ġone - hot - enc oding Ġ( BI B REF 15 ), ĠB IB REF 0 Ġcreated Ġword 2 vec . ĠWord 2 V ec Ġconsists Ġof Ġtwo Ġshallow Ġneural Ġnetwork Ġarchitectures : Ġcontinuous Ġskip gram Ġand ĠCB o W . ĠIt Ġuses Ġdistributed Ġ( low - dimensional , Ġdense ) Ġrepresentations Ġof Ġwords Ġthat Ġgroup Ġsimilar Ġwords . ĠThis Ġnew Ġmodel Ġtraded Ġthe Ġcomplexity Ġof Ġdeep Ġneural Ġnetwork Ġarchitectures , Ġby Ġother Ġresearchers , Ġfor Ġmore Ġefficient Ġtraining Ġover Ġlarge Ġcorpor a . ĠIts Ġarchitectures Ġhave Ġtwo Ġtraining Ġalgorithms : Ġnegative Ġsampling Ġand Ġhierarchical Ġsoft max Ġ( BI B REF 16 ). ĠThe Ġreleased Ġmodel Ġwas Ġtrained Ġon ĠGoogle Ġnews Ġdataset Ġof Ġ100 Ġbillion Ġwords . ĠImplement ations Ġof Ġthe Ġmodel Ġhave Ġbeen Ġundertaken Ġby Ġresearchers Ġin Ġthe Ġprogramming Ġlanguages ĠPython Ġand ĠC ++ , Ġthough Ġthe Ġoriginal Ġwas Ġdone Ġin ĠC Ġ( BI B REF 17 ). </s> Contin uous Ġskip gram Ġpredicts Ġ( by Ġmaximizing Ġclassification Ġof ) Ġwords Ġbefore Ġand Ġafter Ġthe Ġcenter Ġword , Ġfor Ġa Ġgiven Ġrange . ĠSince Ġdistant Ġwords Ġare Ġless Ġconnected Ġto Ġa Ġcenter Ġword Ġin Ġa Ġsentence , Ġless Ġweight Ġis Ġassigned Ġto Ġsuch Ġdistant Ġwords Ġin Ġtraining . ĠCB o W , Ġon Ġthe Ġother Ġhand , Ġuses Ġwords Ġfrom Ġthe Ġhistory Ġand Ġfuture Ġin Ġa Ġsequence , Ġwith Ġthe Ġobjective Ġof Ġcorrectly Ġclass ifying Ġthe Ġtarget Ġword Ġin Ġthe Ġmiddle . ĠIt Ġworks Ġby Ġprojecting Ġall Ġhistory Ġor Ġfuture Ġwords Ġwithin Ġa Ġchosen Ġwindow Ġinto Ġthe Ġsame Ġposition , Ġaveraging Ġtheir Ġvectors . ĠHence , Ġthe Ġorder Ġof Ġwords Ġin Ġthe Ġhistory Ġor Ġfuture Ġdoes Ġnot Ġinfluence Ġthe Ġaveraged Ġvector . ĠThis Ġis Ġsimilar Ġto Ġthe Ġtraditional Ġbag - of - words , Ġwhich Ġis Ġoblivious Ġof Ġthe Ġorder Ġof Ġwords Ġin Ġits Ġsequence . ĠA Ġlog - linear Ġclass ifier Ġis Ġused Ġin Ġboth Ġarchitectures Ġ( BI B REF 0 ). ĠIn Ġfurther Ġwork , Ġthey Ġextended Ġthe Ġmodel Ġto Ġbe Ġable Ġto Ġdo Ġphrase Ġrepresentations Ġand Ġsubs ample Ġfrequent Ġwords Ġ( BI B REF 16 ). ĠBeing Ġa ĠN N LM , Ġword 2 vec Ġassigns Ġprobabilities Ġto Ġwords Ġin Ġa Ġsequence , Ġlike Ġother ĠN NL Ms Ġsuch Ġas Ġfeed forward Ġnetworks Ġor Ġrecurrent Ġneural Ġnetworks Ġ( BI B REF 15 ). ĠEarlier Ġmodels Ġlike Ġlatent Ġdir ich let Ġallocation Ġ( L DA ) Ġand Ġlatent Ġsemantic Ġanalysis Ġ( L SA ) Ġexist Ġand Ġeffectively Ġachieve Ġlow Ġdimensional Ġvectors Ġby Ġmatrix Ġfactor ization Ġ( BI B REF 18 , ĠB IB REF 19 ). </s> It 's Ġbeen Ġshown Ġthat Ġword Ġvectors Ġare Ġbeneficial Ġfor ĠN LP Ġtasks Ġ( BI B REF 15 ), Ġsuch Ġas Ġsentiment Ġanalysis Ġand Ġnamed Ġentity Ġrecognition . ĠBesides , ĠB IB REF 0 Ġshowed Ġwith Ġvector Ġspace Ġalgebra Ġthat Ġrelationships Ġamong Ġwords Ġcan Ġbe Ġevaluated , Ġexpressing Ġthe Ġquality Ġof Ġvectors Ġproduced Ġfrom Ġthe Ġmodel . ĠThe Ġfamous , Ġsemantic Ġexample : Ġvector (" King ") Ġ- Ġvector (" Man ") Ġ+ Ġvector (" Woman ") Ġ$\ app rox Ġ$ Ġvector (" Queen ") Ġcan Ġbe Ġverified Ġusing Ġcos ine Ġdistance . ĠAnother Ġtype Ġof Ġsemantic Ġmeaning Ġis Ġthe Ġrelationship Ġbetween Ġa Ġcapital Ġcity Ġand Ġits Ġcorresponding Ġcountry . ĠSynt actic Ġrelationship Ġexamples Ġinclude Ġplural Ġverbs Ġand Ġpast Ġtense , Ġamong Ġothers . ĠComb ination Ġof Ġboth Ġsynt actic Ġand Ġsemantic Ġanalyses Ġis Ġpossible Ġand Ġprovided Ġ( t ot aling Ġover Ġ19 , 000 Ġquestions ) Ġas ĠGoogle Ġanalogy Ġtest Ġset Ġby ĠB IB REF 0 . ĠWord Similar ity - 353 Ġtest Ġset Ġis Ġanother Ġanalysis Ġtool Ġfor Ġword Ġvectors Ġ( BI B REF 20 ). ĠUnlike ĠGoogle Ġanalogy Ġscore , Ġwhich Ġis Ġbased Ġon Ġvector Ġspace Ġalgebra , ĠWord Similar ity Ġis Ġbased Ġon Ġhuman Ġexpert - ass igned Ġsemantic Ġsimilarity Ġon Ġtwo Ġsets Ġof ĠEnglish Ġword Ġpairs . ĠBoth Ġtools Ġrank Ġfrom Ġ0 Ġ( t otally Ġdiss imilar ) Ġto Ġ1 Ġ( very Ġmuch Ġsimilar Ġor Ġexact , Ġin ĠGoogle Ġanalogy Ġcase ). </s> A Ġtypical Ġartificial Ġneural Ġnetwork Ġ( ANN ) Ġhas Ġvery Ġmany Ġhyper - param eters Ġwhich Ġmay Ġbe Ġtuned . ĠHyper - param eters Ġare Ġvalues Ġwhich Ġmay Ġbe Ġmanually Ġadjusted Ġand Ġinclude Ġvector Ġdimension Ġsize , Ġtype Ġof Ġalgorithm Ġand Ġlearning Ġrate Ġ( BI B REF 19 ). ĠB IB REF 0 Ġtried Ġvarious Ġhyper - param eters Ġwith Ġboth Ġarchitectures Ġof Ġtheir Ġmodel , Ġranging Ġfrom Ġ50 Ġto Ġ1 , 000 Ġdimensions , Ġ30 , 000 Ġto Ġ3 , 000 , 000 Ġvocabulary Ġsizes , Ġ1 Ġto Ġ3 Ġepoch s , Ġamong Ġothers . ĠIn Ġour Ġwork , Ġwe Ġextended Ġresearch Ġto Ġ3 , 000 Ġdimensions . ĠDifferent Ġobservations Ġwere Ġnoted Ġfrom Ġthe Ġmany Ġtrials . ĠThey Ġobserved Ġdiminishing Ġreturns Ġafter Ġa Ġcertain Ġpoint , Ġdespite Ġadditional Ġdimensions Ġor Ġlarger , Ġun struct ured Ġtraining Ġdata . ĠHowever , Ġquality Ġincreased Ġwhen Ġboth Ġdimensions Ġand Ġdata Ġsize Ġwere Ġincreased Ġtogether . ĠAlthough ĠB IB REF 16 Ġpointed Ġout Ġthat Ġchoice Ġof Ġoptimal Ġhyper - param eter Ġconfigurations Ġdepends Ġon Ġthe ĠN LP Ġproblem Ġat Ġhand , Ġthey Ġidentified Ġthe Ġmost Ġimportant Ġfactors Ġare Ġarchitecture , Ġdimension Ġsize , Ġsubs am pling Ġrate , Ġand Ġthe Ġwindow Ġsize . ĠIn Ġaddition , Ġit Ġhas Ġbeen Ġobserved Ġthat Ġvariables Ġlike Ġsize Ġof Ġdatasets Ġimprove Ġthe Ġquality Ġof Ġword Ġvectors Ġand , Ġpotentially , Ġperformance Ġon Ġdownstream Ġtasks Ġ( BI B REF 21 , ĠB IB REF 0 ). </s> Method ology </s> The Ġmodels Ġwere Ġgenerated Ġin Ġa Ġshared Ġcluster Ġrunning ĠUbuntu Ġ16 Ġwith Ġ32 ĠCPUs Ġof Ġ32 x ĠIntel ĠXeon Ġ41 10 Ġat Ġ2 . 1 GHz . ĠG ens im Ġ( BI B REF 17 ) Ġpython Ġlibrary Ġimplementation Ġof Ġword 2 vec Ġwas Ġused Ġwith Ġparallel ization Ġto Ġutilize Ġall Ġ32 ĠCPUs . ĠThe Ġdownstream Ġexperiments Ġwere Ġrun Ġon Ġa ĠTesla ĠGPU Ġon Ġa Ġshared ĠDG X Ġcluster Ġrunning ĠUbuntu Ġ18 . ĠPy tor ch Ġdeep Ġlearning Ġframework Ġwas Ġused . ĠG ens im Ġwas Ġchosen Ġbecause Ġof Ġits Ġrelative Ġstability , Ġpopular Ġsupport Ġand Ġto Ġminimize Ġthe Ġtime Ġrequired Ġin Ġwriting Ġand Ġtesting Ġa Ġnew Ġimplementation Ġin Ġpython Ġfrom Ġscratch . </s> To Ġform Ġthe Ġvocabulary , Ġwords Ġoccurring Ġless Ġthan Ġ5 Ġtimes Ġin Ġthe Ġcorpor a Ġwere Ġdropped , Ġstop Ġwords Ġremoved Ġusing Ġthe Ġnatural Ġlanguage Ġtool kit Ġ( N LT K ) Ġ( BI B REF 22 ) Ġand Ġdata Ġpre - processing Ġcarried Ġout . ĠTable ĠT AB REF 2 Ġdescribes Ġmost Ġhyper - param eters Ġexplored Ġfor Ġeach Ġdataset . ĠIn Ġall , Ġ80 Ġruns Ġ( of Ġabout Ġ160 Ġminutes ) Ġwere Ġconducted Ġfor Ġthe Ġ15 MB ĠWiki ĠAbstract Ġdataset Ġwith Ġ80 Ġserial ized Ġmodels Ġtotaling Ġ15 . 136 GB Ġwhile Ġ80 Ġruns Ġ( for Ġover Ġ320 Ġhours ) Ġwere Ġconducted Ġfor Ġthe Ġ7 11 MB ĠSW Ġdataset , Ġwith Ġ80 Ġserial ized Ġmodels Ġtotaling Ġover Ġ145 GB . ĠExper iments Ġfor Ġall Ġcombinations Ġfor Ġ300 Ġdimensions Ġwere Ġconducted Ġon Ġthe Ġ3 . 9 GB Ġtraining Ġset Ġof Ġthe ĠBW Ġcorpus Ġand Ġadditional Ġruns Ġfor Ġother Ġdimensions Ġfor Ġthe Ġwindow Ġ8 Ġ+ Ġskip gram Ġ+ Ġheir arch ical Ġsoft max Ġcombination Ġto Ġverify Ġthe Ġtrend Ġof Ġquality Ġof Ġword Ġvectors Ġas Ġdimensions Ġare Ġincreased . </s> Google Ġ( sem antic Ġand Ġsynt actic ) Ġanalogy Ġtests Ġand ĠWord Similar ity - 353 Ġ( with ĠSpear man Ġcorrelation ) Ġby ĠB IB REF 20 Ġwere Ġchosen Ġfor Ġintrinsic Ġevaluations . ĠThey Ġmeasure Ġthe Ġquality Ġof Ġword Ġvectors . ĠThe Ġanalogy Ġscores Ġare Ġaverages Ġof Ġboth Ġsemantic Ġand Ġsynt actic Ġtests . ĠN ER Ġand ĠSA Ġwere Ġchosen Ġfor Ġextr ins ic Ġevaluations . ĠThe ĠGM B Ġdataset Ġfor ĠN ER Ġwas Ġtrained Ġin Ġan ĠL ST M Ġnetwork , Ġwhich Ġhad Ġan Ġembed ding Ġlayer Ġfor Ġinput . ĠThe Ġnetwork Ġdiagram Ġis Ġshown Ġin Ġfig . ĠFIG REF 4 . ĠThe ĠIM Db Ġdataset Ġfor ĠSA Ġwas Ġtrained Ġin Ġa ĠBi L ST M Ġnetwork , Ġwhich Ġalso Ġused Ġan Ġembed ding Ġlayer Ġfor Ġinput . ĠIts Ġnetwork Ġdiagram Ġis Ġgiven Ġin Ġfig . ĠFIG REF 4 . ĠIt Ġincludes Ġan Ġadditional Ġhidden Ġlinear Ġlayer . ĠHyper - param eter Ġdetails Ġof Ġthe Ġtwo Ġnetworks Ġfor Ġthe Ġdownstream Ġtasks Ġare Ġgiven Ġin Ġtable ĠT AB REF 3 . ĠThe Ġmetrics Ġfor Ġextr ins ic Ġevaluation Ġinclude ĠF 1 , Ġprecision , Ġrecall Ġand Ġaccuracy Ġscores . ĠIn Ġboth Ġtasks , Ġthe Ġdefault Ġpy tor ch Ġembed ding Ġwas Ġtested Ġbefore Ġbeing Ġreplaced Ġby Ġpre - trained Ġembed d ings Ġreleased Ġby ĠB IB REF 0 Ġand Ġours . ĠIn Ġeach Ġcase , Ġthe Ġdataset Ġwas Ġshuff led Ġbefore Ġtraining Ġand Ġsplit Ġin Ġthe Ġratio Ġ70 : 15 : 15 Ġfor Ġtraining , Ġvalidation Ġ( dev ) Ġand Ġtest Ġsets . ĠB atch Ġsize Ġof Ġ64 Ġwas Ġused . ĠFor Ġeach Ġtask , Ġexperiments Ġfor Ġeach Ġembed ding Ġwas Ġconducted Ġfour Ġtimes Ġand Ġan Ġaverage Ġvalue Ġcalculated Ġand Ġreported Ġin Ġthe Ġnext Ġsection </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 5 Ġsummarizes Ġkey Ġresults Ġfrom Ġthe Ġintrinsic Ġevaluations Ġfor Ġ300 Ġdimensions . ĠTable ĠT AB REF 6 Ġreveals Ġthe Ġtraining Ġtime Ġ( in Ġhours ) Ġand Ġaverage Ġembed ding Ġloading Ġtime Ġ( in Ġseconds ) Ġrepresentative Ġof Ġthe Ġvarious Ġmodels Ġused . ĠTables ĠT AB REF 11 Ġand ĠT AB REF 12 Ġsummarize Ġkey Ġresults Ġfor Ġthe Ġextr ins ic Ġevaluations . ĠFigures ĠFIG REF 7 , ĠFIG REF 9 , ĠFIG REF 10 , ĠFIG REF 13 Ġand ĠFIG REF 14 Ġpresent Ġline Ġgraph Ġof Ġthe Ġeight Ġcombinations Ġfor Ġdifferent Ġdimension Ġsizes Ġfor ĠSimple ĠWiki , Ġtrend Ġof ĠSimple ĠWiki Ġand ĠBillion ĠWord Ġcorpor a Ġover Ġseveral Ġdimension Ġsizes , Ġanalogy Ġscore Ġcomparison Ġfor Ġmodels Ġacross Ġdatasets , ĠN ER Ġmean ĠF 1 Ġscores Ġon Ġthe ĠGM B Ġdataset Ġand ĠSA Ġmean ĠF 1 Ġscores Ġon Ġthe ĠIM Db Ġdataset , Ġrespectively . ĠComb ination Ġof Ġthe Ġskip gram Ġusing Ġhierarchical Ġsoft max Ġand Ġwindow Ġsize Ġof Ġ8 Ġfor Ġ300 Ġdimensions Ġoutper formed Ġothers Ġin Ġanalogy Ġscores Ġfor Ġthe ĠWiki ĠAbstract . ĠHowever , Ġits Ġresults Ġare Ġso Ġpoor , Ġbecause Ġof Ġthe Ġtiny Ġfile Ġsize , Ġthey 're Ġnot Ġworth Ġreporting Ġhere . ĠHence , Ġwe 'll Ġfocus Ġon Ġresults Ġfrom Ġthe ĠSimple ĠWiki Ġand ĠBillion ĠWord Ġcorpor a . </s> Best Ġcombination Ġchanges Ġwhen Ġcorpus Ġsize Ġincreases , Ġas Ġwill Ġbe Ġnoticed Ġfrom Ġtable ĠT AB REF 5 . ĠIn Ġterms Ġof Ġanalogy Ġscore , Ġfor Ġ10 Ġepoch s , Ġw 8 s 0 h 0 Ġperforms Ġbest Ġwhile Ġw 8 s 1 h 0 Ġperforms Ġbest Ġin Ġterms Ġof ĠWord Sim Ġand Ġcorresponding ĠSpear man Ġcorrelation . ĠMeanwhile , Ġincreasing Ġthe Ġcorpus Ġsize Ġto ĠBW , Ġw 4 s 1 h 0 Ġperforms Ġbest Ġin Ġterms Ġof Ġanalogy Ġscore Ġwhile Ġw 8 s 1 h 0 Ġmaintains Ġits Ġposition Ġas Ġthe Ġbest Ġin Ġterms Ġof ĠWord Sim Ġand ĠSpear man Ġcorrelation . ĠBesides Ġconsidering Ġquality Ġmetrics , Ġit Ġcan Ġbe Ġobserved Ġfrom Ġtable ĠT AB REF 6 Ġthat Ġcomparative Ġratio Ġof Ġvalues Ġbetween Ġthe Ġmodels Ġis Ġnot Ġcomm ens urate Ġwith Ġthe Ġresults Ġin Ġintrinsic Ġor Ġextr ins ic Ġvalues , Ġespecially Ġwhen Ġwe Ġconsider Ġthe Ġamount Ġof Ġtime Ġand Ġenergy Ġspent , Ġsince Ġmore Ġtraining Ġtime Ġresults Ġin Ġmore Ġenergy Ġconsumption Ġ( BI B REF 23 ). </s> Information Ġon Ġthe Ġlength Ġof Ġtraining Ġtime Ġfor Ġthe Ġreleased ĠMik ol ov Ġmodel Ġis Ġnot Ġreadily Ġavailable . ĠHowever , Ġit 's Ġinteresting Ġto Ġnote Ġthat Ġtheir Ġpresumed Ġbest Ġmodel , Ġwhich Ġwas Ġreleased Ġis Ġalso Ġs 1 h 0 . ĠIts Ġanalogy Ġscore , Ġwhich Ġwe Ġtested Ġand Ġreport , Ġis Ġconfirmed Ġin Ġtheir Ġpaper . ĠIt Ġbeats Ġour Ġbest Ġmodels Ġin Ġonly Ġanalogy Ġscore Ġ( even Ġfor ĠSimple ĠWiki ), Ġperforming Ġworse Ġin Ġothers . ĠThis Ġis Ġins pite Ġof Ġusing Ġa Ġmuch Ġbigger Ġcorpus Ġof Ġ3 , 000 , 000 Ġvocabulary Ġsize Ġand Ġ100 Ġbillion Ġwords Ġwhile ĠSimple ĠWiki Ġhad Ġvocabulary Ġsize Ġof Ġ367 , 8 11 Ġand Ġis Ġ7 11 MB . ĠIt Ġis Ġvery Ġlikely Ġour Ġanalogy Ġscores Ġwill Ġimprove Ġwhen Ġwe Ġuse Ġa Ġmuch Ġlarger Ġcorpus , Ġas Ġcan Ġbe Ġobserved Ġfrom Ġtable ĠT AB REF 5 , Ġwhich Ġinvolves Ġjust Ġone Ġbillion Ġwords . </s> Although Ġthe Ġtwo Ġbest Ġcombinations Ġin Ġanalogy Ġ( w 8 s 0 h 0 Ġ& Ġw 4 s 0 h 0 ) Ġfor ĠSW , Ġas Ġshown Ġin Ġfig . ĠFIG REF 7 , Ġdecreased Ġonly Ġslightly Ġcompared Ġto Ġothers Ġwith Ġincreasing Ġdimensions , Ġthe Ġincreased Ġtraining Ġtime Ġand Ġmuch Ġlarger Ġserial ized Ġmodel Ġsize Ġrender Ġany Ġpossible Ġminimal Ġscore Ġadvantage Ġover Ġhigher Ġdimensions Ġundesirable . ĠAs Ġcan Ġbe Ġobserved Ġin Ġfig . ĠFIG REF 9 , Ġfrom Ġ100 Ġdimensions , Ġscores Ġimprove Ġbut Ġstart Ġto Ġdrop Ġafter Ġover Ġ300 Ġdimensions Ġfor ĠSW Ġand Ġafter Ġover Ġ400 Ġdimensions Ġfor ĠBW . ĠMore Ġbecomes Ġworse ! ĠThis Ġtrend Ġis Ġtrue Ġfor Ġall Ġcombinations Ġfor Ġall Ġtests . ĠPoly nom ial Ġinterpol ation Ġmay Ġbe Ġused Ġto Ġdetermine Ġthe Ġoptimal Ġdimension Ġin Ġboth Ġcorpor a . ĠOur Ġmodels Ġare Ġavailable Ġfor Ġconfirmation Ġand Ġsource Ġcodes Ġare Ġavailable Ġon Ġgithub . </s> With Ġregards Ġto ĠN ER , Ġmost Ġpret rained Ġembed d ings Ġoutper formed Ġthe Ġdefault Ġpy tor ch Ġembed ding , Ġwith Ġour ĠBW Ġw 4 s 1 h 0 Ġmodel Ġ( which Ġis Ġbest Ġin ĠBW Ġanalogy Ġscore ) Ġperforming Ġbest Ġin ĠF 1 Ġscore Ġand Ġclosely Ġfollowed Ġby ĠB IB REF 0 Ġmodel . ĠOn Ġthe Ġother Ġhand , Ġwith Ġregards Ġto ĠSA , Ġpy tor ch Ġembed ding Ġoutper formed Ġthe Ġpret rained Ġembed d ings Ġbut Ġwas Ġclosely Ġfollowed Ġby Ġour ĠSW Ġw 8 s 0 h 0 Ġmodel Ġ( which Ġalso Ġhad Ġthe Ġbest ĠSW Ġanalogy Ġscore ). ĠB IB REF 0 Ġperformed Ġsecond Ġworst Ġof Ġall , Ġdespite Ġoriginating Ġfrom Ġa Ġvery Ġhuge Ġcorpus . ĠThe Ġcombinations Ġw 8 s 0 h 0 Ġ& Ġw 4 s 0 h 0 Ġof ĠSW Ġperformed Ġreasonably Ġwell Ġin Ġboth Ġextr ins ic Ġtasks , Ġjust Ġas Ġthe Ġdefault Ġpy tor ch Ġembed ding Ġdid . </s> Conclusion </s> This Ġwork Ġanalyses , Ġempir ically , Ġoptimal Ġcombinations Ġof Ġhyper - param eters Ġfor Ġembed d ings , Ġspecifically Ġfor Ġword 2 vec . ĠIt Ġfurther Ġshows Ġthat Ġfor Ġdownstream Ġtasks , Ġlike ĠN ER Ġand ĠSA , Ġthere 's Ġno Ġsilver Ġbullet ! ĠHowever , Ġsome Ġcombinations Ġshow Ġstrong Ġperformance Ġacross Ġtasks . ĠPerformance Ġof Ġembed d ings Ġis Ġtask - specific Ġand Ġhigh Ġanalogy Ġscores Ġdo Ġnot Ġnecessarily Ġcorrelate Ġpositively Ġwith Ġperformance Ġon Ġdownstream Ġtasks . ĠThis Ġpoint Ġon Ġcorrelation Ġis Ġsomewhat Ġsimilar Ġto Ġresults Ġby ĠB IB REF 24 Ġand ĠB IB REF 14 . ĠIt Ġwas Ġdiscovered Ġthat Ġincreasing Ġdimension Ġsize Ġdep reci ates Ġperformance Ġafter Ġa Ġpoint . ĠIf Ġstrong Ġconsiderations Ġof Ġsaving Ġtime , Ġenergy Ġand Ġthe Ġenvironment Ġare Ġmade , Ġthen Ġreasonably Ġsmaller Ġcorpor a Ġmay Ġsuffice Ġor Ġeven Ġbe Ġbetter Ġin Ġsome Ġcases . ĠThe Ġon - going Ġdrive Ġby Ġmany Ġresearchers Ġto Ġuse Ġever - growing Ġdata Ġto Ġtrain Ġdeep Ġneural Ġnetworks Ġcan Ġbenefit Ġfrom Ġthe Ġfindings Ġof Ġthis Ġwork . ĠIndeed , Ġhyper - param eter Ġchoices Ġare Ġvery Ġimportant Ġin Ġneural Ġnetwork Ġsystems Ġ( BI B REF 19 ). </s> Future Ġwork Ġthat Ġmay Ġbe Ġinvestigated Ġare Ġperformance Ġof Ġother Ġarchitectures Ġof Ġword Ġor Ġsub - word Ġembed d ings , Ġthe Ġperformance Ġand Ġcomparison Ġof Ġembed d ings Ġapplied Ġto Ġlanguages Ġother Ġthan ĠEnglish Ġand Ġhow Ġembed d ings Ġperform Ġin Ġother Ġdownstream Ġtasks . ĠIn Ġaddition , Ġsince Ġthe Ġactual Ġreason Ġfor Ġthe Ġchanges Ġin Ġbest Ġmodel Ġas Ġcorpus Ġsize Ġincreases Ġis Ġnot Ġclear , Ġthis Ġwill Ġalso Ġbe Ġsuitable Ġfor Ġfurther Ġresearch . </s> The Ġwork Ġon Ġthis Ġproject Ġis Ġpartially Ġfunded Ġby ĠV inn ova Ġunder Ġthe Ġproject Ġnumber Ġ2019 - 029 96 Ġ" Spr Ã¥ k mod eller Ġf Ã¶r Ġs ven s ka Ġmy nd ig heter " </s> Ac ron y ms
Introduction There have been many implementations of the word2vec model in either of the two architectures it provides: continuous skipgram and CBoW (BIBREF0). Similar distributed models of word or subword embeddings (or vector representations) find usage in sota, deep neural networks like BERT and its successors (BIBREF1, BIBREF2, BIBREF3). These deep networks generate contextual representations of words after been trained for extended periods on large corpora, unsupervised, using the attention mechanisms (BIBREF4). It has been observed that various hyper-parameter combinations have been used in different research involving word2vec with the possibility of many of them being sub-optimal (BIBREF5, BIBREF6, BIBREF7). Therefore, the authors seek to address the research question: what is the optimal combination of word2vec hyper-parameters for intrinsic and extrinsic NLP purposes? There are astronomically high numbers of combinations of hyper-parameters possible for neural networks, even with just a few layers. Hence, the scope of our extensive work over three corpora is on dimension size, training epochs, window size and vocabulary size for the training algorithms (hierarchical softmax and negative sampling) of both skipgram and CBoW. The corpora used for word embeddings are English Wiki News Abstract by BIBREF8 of about 15MB, English Wiki Simple (SW) Articles by BIBREF9 of about 711MB and the Billion Word (BW) of 3.9GB by BIBREF10. The corpus used for sentiment analysis is the IMDb dataset of movie reviews by BIBREF11 while that for NER is Groningen Meaning Bank (GMB) by BIBREF12, containing 47,959 sentence samples. The IMDb dataset used has a total of 25,000 sentences with half being positive sentiments and the other half being negative sentiments. The GMB dataset has 17 labels, with 9 main labels and 2 context tags. It is however unbalanced due to the high percentage of tokens with the label 'O'. This skew in the GMB dataset is typical with NER datasets. The objective of this work is to determine the optimal combinations of word2vec hyper-parameters for intrinsic evaluation (semantic and syntactic analogies) and extrinsic evaluation tasks (BIBREF13, BIBREF14), like SA and NER. It is not our objective in this work to record sota results. Some of the main contributions of this research are the empirical establishment of optimal combinations of word2vec hyper-parameters for NLP tasks, discovering the behaviour of quality of vectors viz-a-viz increasing dimensions and the confirmation of embeddings being task-specific for the downstream. The rest of this paper is organised as follows: the literature review that briefly surveys distributed representation of words, particularly word2vec; the methodology employed in this research work; the results obtained and the conclusion. Literature Review Breaking away from the non-distributed (high-dimensional, sparse) representations of words, typical of traditional bag-of-words or one-hot-encoding (BIBREF15), BIBREF0 created word2vec. Word2Vec consists of two shallow neural network architectures: continuous skipgram and CBoW. It uses distributed (low-dimensional, dense) representations of words that group similar words. This new model traded the complexity of deep neural network architectures, by other researchers, for more efficient training over large corpora. Its architectures have two training algorithms: negative sampling and hierarchical softmax (BIBREF16). The released model was trained on Google news dataset of 100 billion words. Implementations of the model have been undertaken by researchers in the programming languages Python and C++, though the original was done in C (BIBREF17). Continuous skipgram predicts (by maximizing classification of) words before and after the center word, for a given range. Since distant words are less connected to a center word in a sentence, less weight is assigned to such distant words in training. CBoW, on the other hand, uses words from the history and future in a sequence, with the objective of correctly classifying the target word in the middle. It works by projecting all history or future words within a chosen window into the same position, averaging their vectors. Hence, the order of words in the history or future does not influence the averaged vector. This is similar to the traditional bag-of-words, which is oblivious of the order of words in its sequence. A log-linear classifier is used in both architectures (BIBREF0). In further work, they extended the model to be able to do phrase representations and subsample frequent words (BIBREF16). Being a NNLM, word2vec assigns probabilities to words in a sequence, like other NNLMs such as feedforward networks or recurrent neural networks (BIBREF15). Earlier models like latent dirichlet allocation (LDA) and latent semantic analysis (LSA) exist and effectively achieve low dimensional vectors by matrix factorization (BIBREF18, BIBREF19). It's been shown that word vectors are beneficial for NLP tasks (BIBREF15), such as sentiment analysis and named entity recognition. Besides, BIBREF0 showed with vector space algebra that relationships among words can be evaluated, expressing the quality of vectors produced from the model. The famous, semantic example: vector("King") - vector("Man") + vector("Woman") $\approx $ vector("Queen") can be verified using cosine distance. Another type of semantic meaning is the relationship between a capital city and its corresponding country. Syntactic relationship examples include plural verbs and past tense, among others. Combination of both syntactic and semantic analyses is possible and provided (totaling over 19,000 questions) as Google analogy test set by BIBREF0. WordSimilarity-353 test set is another analysis tool for word vectors (BIBREF20). Unlike Google analogy score, which is based on vector space algebra, WordSimilarity is based on human expert-assigned semantic similarity on two sets of English word pairs. Both tools rank from 0 (totally dissimilar) to 1 (very much similar or exact, in Google analogy case). A typical artificial neural network (ANN) has very many hyper-parameters which may be tuned. Hyper-parameters are values which may be manually adjusted and include vector dimension size, type of algorithm and learning rate (BIBREF19). BIBREF0 tried various hyper-parameters with both architectures of their model, ranging from 50 to 1,000 dimensions, 30,000 to 3,000,000 vocabulary sizes, 1 to 3 epochs, among others. In our work, we extended research to 3,000 dimensions. Different observations were noted from the many trials. They observed diminishing returns after a certain point, despite additional dimensions or larger, unstructured training data. However, quality increased when both dimensions and data size were increased together. Although BIBREF16 pointed out that choice of optimal hyper-parameter configurations depends on the NLP problem at hand, they identified the most important factors are architecture, dimension size, subsampling rate, and the window size. In addition, it has been observed that variables like size of datasets improve the quality of word vectors and, potentially, performance on downstream tasks (BIBREF21, BIBREF0). Methodology The models were generated in a shared cluster running Ubuntu 16 with 32 CPUs of 32x Intel Xeon 4110 at 2.1GHz. Gensim (BIBREF17) python library implementation of word2vec was used with parallelization to utilize all 32 CPUs. The downstream experiments were run on a Tesla GPU on a shared DGX cluster running Ubuntu 18. Pytorch deep learning framework was used. Gensim was chosen because of its relative stability, popular support and to minimize the time required in writing and testing a new implementation in python from scratch. To form the vocabulary, words occurring less than 5 times in the corpora were dropped, stop words removed using the natural language toolkit (NLTK) (BIBREF22) and data pre-processing carried out. Table TABREF2 describes most hyper-parameters explored for each dataset. In all, 80 runs (of about 160 minutes) were conducted for the 15MB Wiki Abstract dataset with 80 serialized models totaling 15.136GB while 80 runs (for over 320 hours) were conducted for the 711MB SW dataset, with 80 serialized models totaling over 145GB. Experiments for all combinations for 300 dimensions were conducted on the 3.9GB training set of the BW corpus and additional runs for other dimensions for the window 8 + skipgram + heirarchical softmax combination to verify the trend of quality of word vectors as dimensions are increased. Google (semantic and syntactic) analogy tests and WordSimilarity-353 (with Spearman correlation) by BIBREF20 were chosen for intrinsic evaluations. They measure the quality of word vectors. The analogy scores are averages of both semantic and syntactic tests. NER and SA were chosen for extrinsic evaluations. The GMB dataset for NER was trained in an LSTM network, which had an embedding layer for input. The network diagram is shown in fig. FIGREF4. The IMDb dataset for SA was trained in a BiLSTM network, which also used an embedding layer for input. Its network diagram is given in fig. FIGREF4. It includes an additional hidden linear layer. Hyper-parameter details of the two networks for the downstream tasks are given in table TABREF3. The metrics for extrinsic evaluation include F1, precision, recall and accuracy scores. In both tasks, the default pytorch embedding was tested before being replaced by pre-trained embeddings released by BIBREF0 and ours. In each case, the dataset was shuffled before training and split in the ratio 70:15:15 for training, validation (dev) and test sets. Batch size of 64 was used. For each task, experiments for each embedding was conducted four times and an average value calculated and reported in the next section Results and Discussion Table TABREF5 summarizes key results from the intrinsic evaluations for 300 dimensions. Table TABREF6 reveals the training time (in hours) and average embedding loading time (in seconds) representative of the various models used. Tables TABREF11 and TABREF12 summarize key results for the extrinsic evaluations. Figures FIGREF7, FIGREF9, FIGREF10, FIGREF13 and FIGREF14 present line graph of the eight combinations for different dimension sizes for Simple Wiki, trend of Simple Wiki and Billion Word corpora over several dimension sizes, analogy score comparison for models across datasets, NER mean F1 scores on the GMB dataset and SA mean F1 scores on the IMDb dataset, respectively. Combination of the skipgram using hierarchical softmax and window size of 8 for 300 dimensions outperformed others in analogy scores for the Wiki Abstract. However, its results are so poor, because of the tiny file size, they're not worth reporting here. Hence, we'll focus on results from the Simple Wiki and Billion Word corpora. Best combination changes when corpus size increases, as will be noticed from table TABREF5. In terms of analogy score, for 10 epochs, w8s0h0 performs best while w8s1h0 performs best in terms of WordSim and corresponding Spearman correlation. Meanwhile, increasing the corpus size to BW, w4s1h0 performs best in terms of analogy score while w8s1h0 maintains its position as the best in terms of WordSim and Spearman correlation. Besides considering quality metrics, it can be observed from table TABREF6 that comparative ratio of values between the models is not commensurate with the results in intrinsic or extrinsic values, especially when we consider the amount of time and energy spent, since more training time results in more energy consumption (BIBREF23). Information on the length of training time for the released Mikolov model is not readily available. However, it's interesting to note that their presumed best model, which was released is also s1h0. Its analogy score, which we tested and report, is confirmed in their paper. It beats our best models in only analogy score (even for Simple Wiki), performing worse in others. This is inspite of using a much bigger corpus of 3,000,000 vocabulary size and 100 billion words while Simple Wiki had vocabulary size of 367,811 and is 711MB. It is very likely our analogy scores will improve when we use a much larger corpus, as can be observed from table TABREF5, which involves just one billion words. Although the two best combinations in analogy (w8s0h0 & w4s0h0) for SW, as shown in fig. FIGREF7, decreased only slightly compared to others with increasing dimensions, the increased training time and much larger serialized model size render any possible minimal score advantage over higher dimensions undesirable. As can be observed in fig. FIGREF9, from 100 dimensions, scores improve but start to drop after over 300 dimensions for SW and after over 400 dimensions for BW. More becomes worse! This trend is true for all combinations for all tests. Polynomial interpolation may be used to determine the optimal dimension in both corpora. Our models are available for confirmation and source codes are available on github. With regards to NER, most pretrained embeddings outperformed the default pytorch embedding, with our BW w4s1h0 model (which is best in BW analogy score) performing best in F1 score and closely followed by BIBREF0 model. On the other hand, with regards to SA, pytorch embedding outperformed the pretrained embeddings but was closely followed by our SW w8s0h0 model (which also had the best SW analogy score). BIBREF0 performed second worst of all, despite originating from a very huge corpus. The combinations w8s0h0 & w4s0h0 of SW performed reasonably well in both extrinsic tasks, just as the default pytorch embedding did. Conclusion This work analyses, empirically, optimal combinations of hyper-parameters for embeddings, specifically for word2vec. It further shows that for downstream tasks, like NER and SA, there's no silver bullet! However, some combinations show strong performance across tasks. Performance of embeddings is task-specific and high analogy scores do not necessarily correlate positively with performance on downstream tasks. This point on correlation is somewhat similar to results by BIBREF24 and BIBREF14. It was discovered that increasing dimension size depreciates performance after a point. If strong considerations of saving time, energy and the environment are made, then reasonably smaller corpora may suffice or even be better in some cases. The on-going drive by many researchers to use ever-growing data to train deep neural networks can benefit from the findings of this work. Indeed, hyper-parameter choices are very important in neural network systems (BIBREF19). Future work that may be investigated are performance of other architectures of word or sub-word embeddings, the performance and comparison of embeddings applied to languages other than English and how embeddings perform in other downstream tasks. In addition, since the actual reason for the changes in best model as corpus size increases is not clear, this will also be suitable for further research. The work on this project is partially funded by Vinnova under the project number 2019-02996 "Språkmodeller för svenska myndigheter" Acronyms
What hyperparameters are explored?
27275fe9f6a9004639f9ac33c3a5767fea388a98
What Ġhyper param eters Ġare Ġexplored ?
Dim ension Ġsize , Ġwindow Ġsize , Ġarchitecture , Ġalgorithm , Ġepoch s , Ġhidden Ġdimension Ġsize , Ġlearning Ġrate , Ġloss Ġfunction , Ġoptim izer Ġalgorithm .
<s> What Ġhyper param eters Ġare Ġexplored ? </s> Introduction </s> There Ġhave Ġbeen Ġmany Ġimplementations Ġof Ġthe Ġword 2 vec Ġmodel Ġin Ġeither Ġof Ġthe Ġtwo Ġarchitectures Ġit Ġprovides : Ġcontinuous Ġskip gram Ġand ĠCB o W Ġ( BI B REF 0 ). ĠSimilar Ġdistributed Ġmodels Ġof Ġword Ġor Ġsub word Ġembed d ings Ġ( or Ġvector Ġrepresentations ) Ġfind Ġusage Ġin Ġs ota , Ġdeep Ġneural Ġnetworks Ġlike ĠB ERT Ġand Ġits Ġsuccessors Ġ( BI B REF 1 , ĠB IB REF 2 , ĠB IB REF 3 ). ĠThese Ġdeep Ġnetworks Ġgenerate Ġcontextual Ġrepresentations Ġof Ġwords Ġafter Ġbeen Ġtrained Ġfor Ġextended Ġperiods Ġon Ġlarge Ġcorpor a , Ġun super vised , Ġusing Ġthe Ġattention Ġmechanisms Ġ( BI B REF 4 ). </s> It Ġhas Ġbeen Ġobserved Ġthat Ġvarious Ġhyper - param eter Ġcombinations Ġhave Ġbeen Ġused Ġin Ġdifferent Ġresearch Ġinvolving Ġword 2 vec Ġwith Ġthe Ġpossibility Ġof Ġmany Ġof Ġthem Ġbeing Ġsub - opt imal Ġ( BI B REF 5 , ĠB IB REF 6 , ĠB IB REF 7 ). ĠTherefore , Ġthe Ġauthors Ġseek Ġto Ġaddress Ġthe Ġresearch Ġquestion : Ġwhat Ġis Ġthe Ġoptimal Ġcombination Ġof Ġword 2 vec Ġhyper - param eters Ġfor Ġintrinsic Ġand Ġextr ins ic ĠN LP Ġpurposes ? ĠThere Ġare Ġastron om ically Ġhigh Ġnumbers Ġof Ġcombinations Ġof Ġhyper - param eters Ġpossible Ġfor Ġneural Ġnetworks , Ġeven Ġwith Ġjust Ġa Ġfew Ġlayers . ĠHence , Ġthe Ġscope Ġof Ġour Ġextensive Ġwork Ġover Ġthree Ġcorpor a Ġis Ġon Ġdimension Ġsize , Ġtraining Ġepoch s , Ġwindow Ġsize Ġand Ġvocabulary Ġsize Ġfor Ġthe Ġtraining Ġalgorithms Ġ( h ier arch ical Ġsoft max Ġand Ġnegative Ġsampling ) Ġof Ġboth Ġskip gram Ġand ĠCB o W . ĠThe Ġcorpor a Ġused Ġfor Ġword Ġembed d ings Ġare ĠEnglish ĠWiki ĠNews ĠAbstract Ġby ĠB IB REF 8 Ġof Ġabout Ġ15 MB , ĠEnglish ĠWiki ĠSimple Ġ( SW ) ĠArticles Ġby ĠB IB REF 9 Ġof Ġabout Ġ7 11 MB Ġand Ġthe ĠBillion ĠWord Ġ( BW ) Ġof Ġ3 . 9 GB Ġby ĠB IB REF 10 . ĠThe Ġcorpus Ġused Ġfor Ġsentiment Ġanalysis Ġis Ġthe ĠIM Db Ġdataset Ġof Ġmovie Ġreviews Ġby ĠB IB REF 11 Ġwhile Ġthat Ġfor ĠN ER Ġis ĠGron ingen ĠMeaning ĠBank Ġ( G MB ) Ġby ĠB IB REF 12 , Ġcontaining Ġ47 , 9 59 Ġsentence Ġsamples . ĠThe ĠIM Db Ġdataset Ġused Ġhas Ġa Ġtotal Ġof Ġ25 , 000 Ġsentences Ġwith Ġhalf Ġbeing Ġpositive Ġsentiments Ġand Ġthe Ġother Ġhalf Ġbeing Ġnegative Ġsentiments . ĠThe ĠGM B Ġdataset Ġhas Ġ17 Ġlabels , Ġwith Ġ9 Ġmain Ġlabels Ġand Ġ2 Ġcontext Ġtags . ĠIt Ġis Ġhowever Ġun balanced Ġdue Ġto Ġthe Ġhigh Ġpercentage Ġof Ġtokens Ġwith Ġthe Ġlabel Ġ' O '. ĠThis Ġskew Ġin Ġthe ĠGM B Ġdataset Ġis Ġtypical Ġwith ĠN ER Ġdatasets . </s> The Ġobjective Ġof Ġthis Ġwork Ġis Ġto Ġdetermine Ġthe Ġoptimal Ġcombinations Ġof Ġword 2 vec Ġhyper - param eters Ġfor Ġintrinsic Ġevaluation Ġ( sem antic Ġand Ġsynt actic Ġanalog ies ) Ġand Ġextr ins ic Ġevaluation Ġtasks Ġ( BI B REF 13 , ĠB IB REF 14 ), Ġlike ĠSA Ġand ĠN ER . ĠIt Ġis Ġnot Ġour Ġobjective Ġin Ġthis Ġwork Ġto Ġrecord Ġs ota Ġresults . ĠSome Ġof Ġthe Ġmain Ġcontributions Ġof Ġthis Ġresearch Ġare Ġthe Ġempirical Ġestablishment Ġof Ġoptimal Ġcombinations Ġof Ġword 2 vec Ġhyper - param eters Ġfor ĠN LP Ġtasks , Ġdiscovering Ġthe Ġbehaviour Ġof Ġquality Ġof Ġvectors Ġviz - a - v iz Ġincreasing Ġdimensions Ġand Ġthe Ġconfirmation Ġof Ġembed d ings Ġbeing Ġtask - specific Ġfor Ġthe Ġdownstream . ĠThe Ġrest Ġof Ġthis Ġpaper Ġis Ġorganised Ġas Ġfollows : Ġthe Ġliterature Ġreview Ġthat Ġbriefly Ġsurveys Ġdistributed Ġrepresentation Ġof Ġwords , Ġparticularly Ġword 2 vec ; Ġthe Ġmethodology Ġemployed Ġin Ġthis Ġresearch Ġwork ; Ġthe Ġresults Ġobtained Ġand Ġthe Ġconclusion . </s> Liter ature ĠReview </s> Breaking Ġaway Ġfrom Ġthe Ġnon - dist ributed Ġ( high - dimensional , Ġsparse ) Ġrepresentations Ġof Ġwords , Ġtypical Ġof Ġtraditional Ġbag - of - words Ġor Ġone - hot - enc oding Ġ( BI B REF 15 ), ĠB IB REF 0 Ġcreated Ġword 2 vec . ĠWord 2 V ec Ġconsists Ġof Ġtwo Ġshallow Ġneural Ġnetwork Ġarchitectures : Ġcontinuous Ġskip gram Ġand ĠCB o W . ĠIt Ġuses Ġdistributed Ġ( low - dimensional , Ġdense ) Ġrepresentations Ġof Ġwords Ġthat Ġgroup Ġsimilar Ġwords . ĠThis Ġnew Ġmodel Ġtraded Ġthe Ġcomplexity Ġof Ġdeep Ġneural Ġnetwork Ġarchitectures , Ġby Ġother Ġresearchers , Ġfor Ġmore Ġefficient Ġtraining Ġover Ġlarge Ġcorpor a . ĠIts Ġarchitectures Ġhave Ġtwo Ġtraining Ġalgorithms : Ġnegative Ġsampling Ġand Ġhierarchical Ġsoft max Ġ( BI B REF 16 ). ĠThe Ġreleased Ġmodel Ġwas Ġtrained Ġon ĠGoogle Ġnews Ġdataset Ġof Ġ100 Ġbillion Ġwords . ĠImplement ations Ġof Ġthe Ġmodel Ġhave Ġbeen Ġundertaken Ġby Ġresearchers Ġin Ġthe Ġprogramming Ġlanguages ĠPython Ġand ĠC ++ , Ġthough Ġthe Ġoriginal Ġwas Ġdone Ġin ĠC Ġ( BI B REF 17 ). </s> Contin uous Ġskip gram Ġpredicts Ġ( by Ġmaximizing Ġclassification Ġof ) Ġwords Ġbefore Ġand Ġafter Ġthe Ġcenter Ġword , Ġfor Ġa Ġgiven Ġrange . ĠSince Ġdistant Ġwords Ġare Ġless Ġconnected Ġto Ġa Ġcenter Ġword Ġin Ġa Ġsentence , Ġless Ġweight Ġis Ġassigned Ġto Ġsuch Ġdistant Ġwords Ġin Ġtraining . ĠCB o W , Ġon Ġthe Ġother Ġhand , Ġuses Ġwords Ġfrom Ġthe Ġhistory Ġand Ġfuture Ġin Ġa Ġsequence , Ġwith Ġthe Ġobjective Ġof Ġcorrectly Ġclass ifying Ġthe Ġtarget Ġword Ġin Ġthe Ġmiddle . ĠIt Ġworks Ġby Ġprojecting Ġall Ġhistory Ġor Ġfuture Ġwords Ġwithin Ġa Ġchosen Ġwindow Ġinto Ġthe Ġsame Ġposition , Ġaveraging Ġtheir Ġvectors . ĠHence , Ġthe Ġorder Ġof Ġwords Ġin Ġthe Ġhistory Ġor Ġfuture Ġdoes Ġnot Ġinfluence Ġthe Ġaveraged Ġvector . ĠThis Ġis Ġsimilar Ġto Ġthe Ġtraditional Ġbag - of - words , Ġwhich Ġis Ġoblivious Ġof Ġthe Ġorder Ġof Ġwords Ġin Ġits Ġsequence . ĠA Ġlog - linear Ġclass ifier Ġis Ġused Ġin Ġboth Ġarchitectures Ġ( BI B REF 0 ). ĠIn Ġfurther Ġwork , Ġthey Ġextended Ġthe Ġmodel Ġto Ġbe Ġable Ġto Ġdo Ġphrase Ġrepresentations Ġand Ġsubs ample Ġfrequent Ġwords Ġ( BI B REF 16 ). ĠBeing Ġa ĠN N LM , Ġword 2 vec Ġassigns Ġprobabilities Ġto Ġwords Ġin Ġa Ġsequence , Ġlike Ġother ĠN NL Ms Ġsuch Ġas Ġfeed forward Ġnetworks Ġor Ġrecurrent Ġneural Ġnetworks Ġ( BI B REF 15 ). ĠEarlier Ġmodels Ġlike Ġlatent Ġdir ich let Ġallocation Ġ( L DA ) Ġand Ġlatent Ġsemantic Ġanalysis Ġ( L SA ) Ġexist Ġand Ġeffectively Ġachieve Ġlow Ġdimensional Ġvectors Ġby Ġmatrix Ġfactor ization Ġ( BI B REF 18 , ĠB IB REF 19 ). </s> It 's Ġbeen Ġshown Ġthat Ġword Ġvectors Ġare Ġbeneficial Ġfor ĠN LP Ġtasks Ġ( BI B REF 15 ), Ġsuch Ġas Ġsentiment Ġanalysis Ġand Ġnamed Ġentity Ġrecognition . ĠBesides , ĠB IB REF 0 Ġshowed Ġwith Ġvector Ġspace Ġalgebra Ġthat Ġrelationships Ġamong Ġwords Ġcan Ġbe Ġevaluated , Ġexpressing Ġthe Ġquality Ġof Ġvectors Ġproduced Ġfrom Ġthe Ġmodel . ĠThe Ġfamous , Ġsemantic Ġexample : Ġvector (" King ") Ġ- Ġvector (" Man ") Ġ+ Ġvector (" Woman ") Ġ$\ app rox Ġ$ Ġvector (" Queen ") Ġcan Ġbe Ġverified Ġusing Ġcos ine Ġdistance . ĠAnother Ġtype Ġof Ġsemantic Ġmeaning Ġis Ġthe Ġrelationship Ġbetween Ġa Ġcapital Ġcity Ġand Ġits Ġcorresponding Ġcountry . ĠSynt actic Ġrelationship Ġexamples Ġinclude Ġplural Ġverbs Ġand Ġpast Ġtense , Ġamong Ġothers . ĠComb ination Ġof Ġboth Ġsynt actic Ġand Ġsemantic Ġanalyses Ġis Ġpossible Ġand Ġprovided Ġ( t ot aling Ġover Ġ19 , 000 Ġquestions ) Ġas ĠGoogle Ġanalogy Ġtest Ġset Ġby ĠB IB REF 0 . ĠWord Similar ity - 353 Ġtest Ġset Ġis Ġanother Ġanalysis Ġtool Ġfor Ġword Ġvectors Ġ( BI B REF 20 ). ĠUnlike ĠGoogle Ġanalogy Ġscore , Ġwhich Ġis Ġbased Ġon Ġvector Ġspace Ġalgebra , ĠWord Similar ity Ġis Ġbased Ġon Ġhuman Ġexpert - ass igned Ġsemantic Ġsimilarity Ġon Ġtwo Ġsets Ġof ĠEnglish Ġword Ġpairs . ĠBoth Ġtools Ġrank Ġfrom Ġ0 Ġ( t otally Ġdiss imilar ) Ġto Ġ1 Ġ( very Ġmuch Ġsimilar Ġor Ġexact , Ġin ĠGoogle Ġanalogy Ġcase ). </s> A Ġtypical Ġartificial Ġneural Ġnetwork Ġ( ANN ) Ġhas Ġvery Ġmany Ġhyper - param eters Ġwhich Ġmay Ġbe Ġtuned . ĠHyper - param eters Ġare Ġvalues Ġwhich Ġmay Ġbe Ġmanually Ġadjusted Ġand Ġinclude Ġvector Ġdimension Ġsize , Ġtype Ġof Ġalgorithm Ġand Ġlearning Ġrate Ġ( BI B REF 19 ). ĠB IB REF 0 Ġtried Ġvarious Ġhyper - param eters Ġwith Ġboth Ġarchitectures Ġof Ġtheir Ġmodel , Ġranging Ġfrom Ġ50 Ġto Ġ1 , 000 Ġdimensions , Ġ30 , 000 Ġto Ġ3 , 000 , 000 Ġvocabulary Ġsizes , Ġ1 Ġto Ġ3 Ġepoch s , Ġamong Ġothers . ĠIn Ġour Ġwork , Ġwe Ġextended Ġresearch Ġto Ġ3 , 000 Ġdimensions . ĠDifferent Ġobservations Ġwere Ġnoted Ġfrom Ġthe Ġmany Ġtrials . ĠThey Ġobserved Ġdiminishing Ġreturns Ġafter Ġa Ġcertain Ġpoint , Ġdespite Ġadditional Ġdimensions Ġor Ġlarger , Ġun struct ured Ġtraining Ġdata . ĠHowever , Ġquality Ġincreased Ġwhen Ġboth Ġdimensions Ġand Ġdata Ġsize Ġwere Ġincreased Ġtogether . ĠAlthough ĠB IB REF 16 Ġpointed Ġout Ġthat Ġchoice Ġof Ġoptimal Ġhyper - param eter Ġconfigurations Ġdepends Ġon Ġthe ĠN LP Ġproblem Ġat Ġhand , Ġthey Ġidentified Ġthe Ġmost Ġimportant Ġfactors Ġare Ġarchitecture , Ġdimension Ġsize , Ġsubs am pling Ġrate , Ġand Ġthe Ġwindow Ġsize . ĠIn Ġaddition , Ġit Ġhas Ġbeen Ġobserved Ġthat Ġvariables Ġlike Ġsize Ġof Ġdatasets Ġimprove Ġthe Ġquality Ġof Ġword Ġvectors Ġand , Ġpotentially , Ġperformance Ġon Ġdownstream Ġtasks Ġ( BI B REF 21 , ĠB IB REF 0 ). </s> Method ology </s> The Ġmodels Ġwere Ġgenerated Ġin Ġa Ġshared Ġcluster Ġrunning ĠUbuntu Ġ16 Ġwith Ġ32 ĠCPUs Ġof Ġ32 x ĠIntel ĠXeon Ġ41 10 Ġat Ġ2 . 1 GHz . ĠG ens im Ġ( BI B REF 17 ) Ġpython Ġlibrary Ġimplementation Ġof Ġword 2 vec Ġwas Ġused Ġwith Ġparallel ization Ġto Ġutilize Ġall Ġ32 ĠCPUs . ĠThe Ġdownstream Ġexperiments Ġwere Ġrun Ġon Ġa ĠTesla ĠGPU Ġon Ġa Ġshared ĠDG X Ġcluster Ġrunning ĠUbuntu Ġ18 . ĠPy tor ch Ġdeep Ġlearning Ġframework Ġwas Ġused . ĠG ens im Ġwas Ġchosen Ġbecause Ġof Ġits Ġrelative Ġstability , Ġpopular Ġsupport Ġand Ġto Ġminimize Ġthe Ġtime Ġrequired Ġin Ġwriting Ġand Ġtesting Ġa Ġnew Ġimplementation Ġin Ġpython Ġfrom Ġscratch . </s> To Ġform Ġthe Ġvocabulary , Ġwords Ġoccurring Ġless Ġthan Ġ5 Ġtimes Ġin Ġthe Ġcorpor a Ġwere Ġdropped , Ġstop Ġwords Ġremoved Ġusing Ġthe Ġnatural Ġlanguage Ġtool kit Ġ( N LT K ) Ġ( BI B REF 22 ) Ġand Ġdata Ġpre - processing Ġcarried Ġout . ĠTable ĠT AB REF 2 Ġdescribes Ġmost Ġhyper - param eters Ġexplored Ġfor Ġeach Ġdataset . ĠIn Ġall , Ġ80 Ġruns Ġ( of Ġabout Ġ160 Ġminutes ) Ġwere Ġconducted Ġfor Ġthe Ġ15 MB ĠWiki ĠAbstract Ġdataset Ġwith Ġ80 Ġserial ized Ġmodels Ġtotaling Ġ15 . 136 GB Ġwhile Ġ80 Ġruns Ġ( for Ġover Ġ320 Ġhours ) Ġwere Ġconducted Ġfor Ġthe Ġ7 11 MB ĠSW Ġdataset , Ġwith Ġ80 Ġserial ized Ġmodels Ġtotaling Ġover Ġ145 GB . ĠExper iments Ġfor Ġall Ġcombinations Ġfor Ġ300 Ġdimensions Ġwere Ġconducted Ġon Ġthe Ġ3 . 9 GB Ġtraining Ġset Ġof Ġthe ĠBW Ġcorpus Ġand Ġadditional Ġruns Ġfor Ġother Ġdimensions Ġfor Ġthe Ġwindow Ġ8 Ġ+ Ġskip gram Ġ+ Ġheir arch ical Ġsoft max Ġcombination Ġto Ġverify Ġthe Ġtrend Ġof Ġquality Ġof Ġword Ġvectors Ġas Ġdimensions Ġare Ġincreased . </s> Google Ġ( sem antic Ġand Ġsynt actic ) Ġanalogy Ġtests Ġand ĠWord Similar ity - 353 Ġ( with ĠSpear man Ġcorrelation ) Ġby ĠB IB REF 20 Ġwere Ġchosen Ġfor Ġintrinsic Ġevaluations . ĠThey Ġmeasure Ġthe Ġquality Ġof Ġword Ġvectors . ĠThe Ġanalogy Ġscores Ġare Ġaverages Ġof Ġboth Ġsemantic Ġand Ġsynt actic Ġtests . ĠN ER Ġand ĠSA Ġwere Ġchosen Ġfor Ġextr ins ic Ġevaluations . ĠThe ĠGM B Ġdataset Ġfor ĠN ER Ġwas Ġtrained Ġin Ġan ĠL ST M Ġnetwork , Ġwhich Ġhad Ġan Ġembed ding Ġlayer Ġfor Ġinput . ĠThe Ġnetwork Ġdiagram Ġis Ġshown Ġin Ġfig . ĠFIG REF 4 . ĠThe ĠIM Db Ġdataset Ġfor ĠSA Ġwas Ġtrained Ġin Ġa ĠBi L ST M Ġnetwork , Ġwhich Ġalso Ġused Ġan Ġembed ding Ġlayer Ġfor Ġinput . ĠIts Ġnetwork Ġdiagram Ġis Ġgiven Ġin Ġfig . ĠFIG REF 4 . ĠIt Ġincludes Ġan Ġadditional Ġhidden Ġlinear Ġlayer . ĠHyper - param eter Ġdetails Ġof Ġthe Ġtwo Ġnetworks Ġfor Ġthe Ġdownstream Ġtasks Ġare Ġgiven Ġin Ġtable ĠT AB REF 3 . ĠThe Ġmetrics Ġfor Ġextr ins ic Ġevaluation Ġinclude ĠF 1 , Ġprecision , Ġrecall Ġand Ġaccuracy Ġscores . ĠIn Ġboth Ġtasks , Ġthe Ġdefault Ġpy tor ch Ġembed ding Ġwas Ġtested Ġbefore Ġbeing Ġreplaced Ġby Ġpre - trained Ġembed d ings Ġreleased Ġby ĠB IB REF 0 Ġand Ġours . ĠIn Ġeach Ġcase , Ġthe Ġdataset Ġwas Ġshuff led Ġbefore Ġtraining Ġand Ġsplit Ġin Ġthe Ġratio Ġ70 : 15 : 15 Ġfor Ġtraining , Ġvalidation Ġ( dev ) Ġand Ġtest Ġsets . ĠB atch Ġsize Ġof Ġ64 Ġwas Ġused . ĠFor Ġeach Ġtask , Ġexperiments Ġfor Ġeach Ġembed ding Ġwas Ġconducted Ġfour Ġtimes Ġand Ġan Ġaverage Ġvalue Ġcalculated Ġand Ġreported Ġin Ġthe Ġnext Ġsection </s> Results Ġand ĠDiscussion </s> Table ĠT AB REF 5 Ġsummarizes Ġkey Ġresults Ġfrom Ġthe Ġintrinsic Ġevaluations Ġfor Ġ300 Ġdimensions . ĠTable ĠT AB REF 6 Ġreveals Ġthe Ġtraining Ġtime Ġ( in Ġhours ) Ġand Ġaverage Ġembed ding Ġloading Ġtime Ġ( in Ġseconds ) Ġrepresentative Ġof Ġthe Ġvarious Ġmodels Ġused . ĠTables ĠT AB REF 11 Ġand ĠT AB REF 12 Ġsummarize Ġkey Ġresults Ġfor Ġthe Ġextr ins ic Ġevaluations . ĠFigures ĠFIG REF 7 , ĠFIG REF 9 , ĠFIG REF 10 , ĠFIG REF 13 Ġand ĠFIG REF 14 Ġpresent Ġline Ġgraph Ġof Ġthe Ġeight Ġcombinations Ġfor Ġdifferent Ġdimension Ġsizes Ġfor ĠSimple ĠWiki , Ġtrend Ġof ĠSimple ĠWiki Ġand ĠBillion ĠWord Ġcorpor a Ġover Ġseveral Ġdimension Ġsizes , Ġanalogy Ġscore Ġcomparison Ġfor Ġmodels Ġacross Ġdatasets , ĠN ER Ġmean ĠF 1 Ġscores Ġon Ġthe ĠGM B Ġdataset Ġand ĠSA Ġmean ĠF 1 Ġscores Ġon Ġthe ĠIM Db Ġdataset , Ġrespectively . ĠComb ination Ġof Ġthe Ġskip gram Ġusing Ġhierarchical Ġsoft max Ġand Ġwindow Ġsize Ġof Ġ8 Ġfor Ġ300 Ġdimensions Ġoutper formed Ġothers Ġin Ġanalogy Ġscores Ġfor Ġthe ĠWiki ĠAbstract . ĠHowever , Ġits Ġresults Ġare Ġso Ġpoor , Ġbecause Ġof Ġthe Ġtiny Ġfile Ġsize , Ġthey 're Ġnot Ġworth Ġreporting Ġhere . ĠHence , Ġwe 'll Ġfocus Ġon Ġresults Ġfrom Ġthe ĠSimple ĠWiki Ġand ĠBillion ĠWord Ġcorpor a . </s> Best Ġcombination Ġchanges Ġwhen Ġcorpus Ġsize Ġincreases , Ġas Ġwill Ġbe Ġnoticed Ġfrom Ġtable ĠT AB REF 5 . ĠIn Ġterms Ġof Ġanalogy Ġscore , Ġfor Ġ10 Ġepoch s , Ġw 8 s 0 h 0 Ġperforms Ġbest Ġwhile Ġw 8 s 1 h 0 Ġperforms Ġbest Ġin Ġterms Ġof ĠWord Sim Ġand Ġcorresponding ĠSpear man Ġcorrelation . ĠMeanwhile , Ġincreasing Ġthe Ġcorpus Ġsize Ġto ĠBW , Ġw 4 s 1 h 0 Ġperforms Ġbest Ġin Ġterms Ġof Ġanalogy Ġscore Ġwhile Ġw 8 s 1 h 0 Ġmaintains Ġits Ġposition Ġas Ġthe Ġbest Ġin Ġterms Ġof ĠWord Sim Ġand ĠSpear man Ġcorrelation . ĠBesides Ġconsidering Ġquality Ġmetrics , Ġit Ġcan Ġbe Ġobserved Ġfrom Ġtable ĠT AB REF 6 Ġthat Ġcomparative Ġratio Ġof Ġvalues Ġbetween Ġthe Ġmodels Ġis Ġnot Ġcomm ens urate Ġwith Ġthe Ġresults Ġin Ġintrinsic Ġor Ġextr ins ic Ġvalues , Ġespecially Ġwhen Ġwe Ġconsider Ġthe Ġamount Ġof Ġtime Ġand Ġenergy Ġspent , Ġsince Ġmore Ġtraining Ġtime Ġresults Ġin Ġmore Ġenergy Ġconsumption Ġ( BI B REF 23 ). </s> Information Ġon Ġthe Ġlength Ġof Ġtraining Ġtime Ġfor Ġthe Ġreleased ĠMik ol ov Ġmodel Ġis Ġnot Ġreadily Ġavailable . ĠHowever , Ġit 's Ġinteresting Ġto Ġnote Ġthat Ġtheir Ġpresumed Ġbest Ġmodel , Ġwhich Ġwas Ġreleased Ġis Ġalso Ġs 1 h 0 . ĠIts Ġanalogy Ġscore , Ġwhich Ġwe Ġtested Ġand Ġreport , Ġis Ġconfirmed Ġin Ġtheir Ġpaper . ĠIt Ġbeats Ġour Ġbest Ġmodels Ġin Ġonly Ġanalogy Ġscore Ġ( even Ġfor ĠSimple ĠWiki ), Ġperforming Ġworse Ġin Ġothers . ĠThis Ġis Ġins pite Ġof Ġusing Ġa Ġmuch Ġbigger Ġcorpus Ġof Ġ3 , 000 , 000 Ġvocabulary Ġsize Ġand Ġ100 Ġbillion Ġwords Ġwhile ĠSimple ĠWiki Ġhad Ġvocabulary Ġsize Ġof Ġ367 , 8 11 Ġand Ġis Ġ7 11 MB . ĠIt Ġis Ġvery Ġlikely Ġour Ġanalogy Ġscores Ġwill Ġimprove Ġwhen Ġwe Ġuse Ġa Ġmuch Ġlarger Ġcorpus , Ġas Ġcan Ġbe Ġobserved Ġfrom Ġtable ĠT AB REF 5 , Ġwhich Ġinvolves Ġjust Ġone Ġbillion Ġwords . </s> Although Ġthe Ġtwo Ġbest Ġcombinations Ġin Ġanalogy Ġ( w 8 s 0 h 0 Ġ& Ġw 4 s 0 h 0 ) Ġfor ĠSW , Ġas Ġshown Ġin Ġfig . ĠFIG REF 7 , Ġdecreased Ġonly Ġslightly Ġcompared Ġto Ġothers Ġwith Ġincreasing Ġdimensions , Ġthe Ġincreased Ġtraining Ġtime Ġand Ġmuch Ġlarger Ġserial ized Ġmodel Ġsize Ġrender Ġany Ġpossible Ġminimal Ġscore Ġadvantage Ġover Ġhigher Ġdimensions Ġundesirable . ĠAs Ġcan Ġbe Ġobserved Ġin Ġfig . ĠFIG REF 9 , Ġfrom Ġ100 Ġdimensions , Ġscores Ġimprove Ġbut Ġstart Ġto Ġdrop Ġafter Ġover Ġ300 Ġdimensions Ġfor ĠSW Ġand Ġafter Ġover Ġ400 Ġdimensions Ġfor ĠBW . ĠMore Ġbecomes Ġworse ! ĠThis Ġtrend Ġis Ġtrue Ġfor Ġall Ġcombinations Ġfor Ġall Ġtests . ĠPoly nom ial Ġinterpol ation Ġmay Ġbe Ġused Ġto Ġdetermine Ġthe Ġoptimal Ġdimension Ġin Ġboth Ġcorpor a . ĠOur Ġmodels Ġare Ġavailable Ġfor Ġconfirmation Ġand Ġsource Ġcodes Ġare Ġavailable Ġon Ġgithub . </s> With Ġregards Ġto ĠN ER , Ġmost Ġpret rained Ġembed d ings Ġoutper formed Ġthe Ġdefault Ġpy tor ch Ġembed ding , Ġwith Ġour ĠBW Ġw 4 s 1 h 0 Ġmodel Ġ( which Ġis Ġbest Ġin ĠBW Ġanalogy Ġscore ) Ġperforming Ġbest Ġin ĠF 1 Ġscore Ġand Ġclosely Ġfollowed Ġby ĠB IB REF 0 Ġmodel . ĠOn Ġthe Ġother Ġhand , Ġwith Ġregards Ġto ĠSA , Ġpy tor ch Ġembed ding Ġoutper formed Ġthe Ġpret rained Ġembed d ings Ġbut Ġwas Ġclosely Ġfollowed Ġby Ġour ĠSW Ġw 8 s 0 h 0 Ġmodel Ġ( which Ġalso Ġhad Ġthe Ġbest ĠSW Ġanalogy Ġscore ). ĠB IB REF 0 Ġperformed Ġsecond Ġworst Ġof Ġall , Ġdespite Ġoriginating Ġfrom Ġa Ġvery Ġhuge Ġcorpus . ĠThe Ġcombinations Ġw 8 s 0 h 0 Ġ& Ġw 4 s 0 h 0 Ġof ĠSW Ġperformed Ġreasonably Ġwell Ġin Ġboth Ġextr ins ic Ġtasks , Ġjust Ġas Ġthe Ġdefault Ġpy tor ch Ġembed ding Ġdid . </s> Conclusion </s> This Ġwork Ġanalyses , Ġempir ically , Ġoptimal Ġcombinations Ġof Ġhyper - param eters Ġfor Ġembed d ings , Ġspecifically Ġfor Ġword 2 vec . ĠIt Ġfurther Ġshows Ġthat Ġfor Ġdownstream Ġtasks , Ġlike ĠN ER Ġand ĠSA , Ġthere 's Ġno Ġsilver Ġbullet ! ĠHowever , Ġsome Ġcombinations Ġshow Ġstrong Ġperformance Ġacross Ġtasks . ĠPerformance Ġof Ġembed d ings Ġis Ġtask - specific Ġand Ġhigh Ġanalogy Ġscores Ġdo Ġnot Ġnecessarily Ġcorrelate Ġpositively Ġwith Ġperformance Ġon Ġdownstream Ġtasks . ĠThis Ġpoint Ġon Ġcorrelation Ġis Ġsomewhat Ġsimilar Ġto Ġresults Ġby ĠB IB REF 24 Ġand ĠB IB REF 14 . ĠIt Ġwas Ġdiscovered Ġthat Ġincreasing Ġdimension Ġsize Ġdep reci ates Ġperformance Ġafter Ġa Ġpoint . ĠIf Ġstrong Ġconsiderations Ġof Ġsaving Ġtime , Ġenergy Ġand Ġthe Ġenvironment Ġare Ġmade , Ġthen Ġreasonably Ġsmaller Ġcorpor a Ġmay Ġsuffice Ġor Ġeven Ġbe Ġbetter Ġin Ġsome Ġcases . ĠThe Ġon - going Ġdrive Ġby Ġmany Ġresearchers Ġto Ġuse Ġever - growing Ġdata Ġto Ġtrain Ġdeep Ġneural Ġnetworks Ġcan Ġbenefit Ġfrom Ġthe Ġfindings Ġof Ġthis Ġwork . ĠIndeed , Ġhyper - param eter Ġchoices Ġare Ġvery Ġimportant Ġin Ġneural Ġnetwork Ġsystems Ġ( BI B REF 19 ). </s> Future Ġwork Ġthat Ġmay Ġbe Ġinvestigated Ġare Ġperformance Ġof Ġother Ġarchitectures Ġof Ġword Ġor Ġsub - word Ġembed d ings , Ġthe Ġperformance Ġand Ġcomparison Ġof Ġembed d ings Ġapplied Ġto Ġlanguages Ġother Ġthan ĠEnglish Ġand Ġhow Ġembed d ings Ġperform Ġin Ġother Ġdownstream Ġtasks . ĠIn Ġaddition , Ġsince Ġthe Ġactual Ġreason Ġfor Ġthe Ġchanges Ġin Ġbest Ġmodel Ġas Ġcorpus Ġsize Ġincreases Ġis Ġnot Ġclear , Ġthis Ġwill Ġalso Ġbe Ġsuitable Ġfor Ġfurther Ġresearch . </s> The Ġwork Ġon Ġthis Ġproject Ġis Ġpartially Ġfunded Ġby ĠV inn ova Ġunder Ġthe Ġproject Ġnumber Ġ2019 - 029 96 Ġ" Spr Ã¥ k mod eller Ġf Ã¶r Ġs ven s ka Ġmy nd ig heter " </s> Ac ron y ms