They use the first principal component of a word's contextualized representation in a given layer as its static embedding.
ABSTRACTIVE
 by taking the first principal component (PC) of its contextualized representations in a given layer
EXTRACTIVE
FLOAT SELECTED: Table 1: The performance of various static embeddings on word embedding benchmark tasks. The best result for each task is in bold. For the contextualizing models (ELMo, BERT, GPT-2), we use the first principal component of a word’s contextualized representations in a given layer as its static embedding. The static embeddings created using ELMo and BERT’s contextualized representations often outperform GloVe and FastText vectors.
As noted earlier, we can create static embeddings for each word by taking the first principal component (PC) of its contextualized representations in a given layer. In Table TABREF34, we plot the performance of these PC static embeddings on several benchmark tasks. These tasks cover semantic similarity, analogy solving, and concept categorization: SimLex999 BIBREF21, MEN BIBREF22, WS353 BIBREF23, RW BIBREF24, SemEval-2012 BIBREF25, Google analogy solving BIBREF0 MSR analogy solving BIBREF26, BLESS BIBREF27 and AP BIBREF28. We leave out layers 3 - 10 in Table TABREF34 because their performance is between those of Layers 2 and 11.
1909.00512
Introduction </s> The Ġapplication Ġof Ġdeep Ġlearning Ġmethods Ġto ĠN LP Ġis Ġmade Ġpossible Ġby Ġrepresenting Ġwords Ġas Ġvectors Ġin Ġa Ġlow - dimensional Ġcontinuous Ġspace . ĠTrad itionally , Ġthese Ġword Ġembed d ings Ġwere Ġstatic : Ġeach Ġword Ġhad Ġa Ġsingle Ġvector , Ġregardless Ġof Ġcontext ĠB IB REF 0 , ĠB IB REF 1 . ĠThis Ġposed Ġseveral Ġproblems , Ġmost Ġnotably Ġthat Ġall Ġsenses Ġof Ġa Ġpoly sem ous Ġword Ġhad Ġto Ġshare Ġthe Ġsame Ġrepresentation . ĠMore Ġrecent Ġwork , Ġnamely Ġdeep Ġneural Ġlanguage Ġmodels Ġsuch Ġas ĠEL Mo ĠB IB REF 2 Ġand ĠB ERT ĠB IB REF 3 , Ġhave Ġsuccessfully Ġcreated Ġcontextual ized Ġword Ġrepresentations , Ġword Ġvectors Ġthat Ġare Ġsensitive Ġto Ġthe Ġcontext Ġin Ġwhich Ġthey Ġappear . ĠRepl acing Ġstatic Ġembed d ings Ġwith Ġcontextual ized Ġrepresentations Ġhas Ġyielded Ġsignificant Ġimprovements Ġon Ġa Ġdiverse Ġarray Ġof ĠN LP Ġtasks , Ġranging Ġfrom Ġquestion - ans w ering Ġto Ġcore ference Ġresolution . </s> The Ġsuccess Ġof Ġcontextual ized Ġword Ġrepresentations Ġsuggests Ġthat Ġdespite Ġbeing Ġtrained Ġwith Ġonly Ġa Ġlanguage Ġmodelling Ġtask , Ġthey Ġlearn Ġhighly Ġtransfer able Ġand Ġtask - agn ostic Ġproperties Ġof Ġlanguage . ĠIn Ġfact , Ġlinear Ġprobing Ġmodels Ġtrained Ġon Ġfrozen Ġcontextual ized Ġrepresentations Ġcan Ġpredict Ġlinguistic Ġproperties Ġof Ġwords Ġ( e . g ., Ġpart - of - speech Ġtags ) Ġalmost Ġas Ġwell Ġas Ġstate - of - the - art Ġmodels ĠB IB REF 4 , ĠB IB REF 5 . ĠStill , Ġthese Ġrepresentations Ġremain Ġpoorly Ġunderstood . ĠFor Ġone , Ġjust Ġhow Ġcontextual Ġare Ġthese Ġcontextual ized Ġword Ġrepresentations ? ĠAre Ġthere Ġinfinitely Ġmany Ġcontext - specific Ġrepresentations Ġthat ĠB ERT Ġand ĠEL Mo Ġcan Ġassign Ġto Ġeach Ġword , Ġor Ġare Ġwords Ġessentially Ġassigned Ġone Ġof Ġa Ġfinite Ġnumber Ġof Ġword - sense Ġrepresentations ? </s> We Ġanswer Ġthis Ġquestion Ġby Ġstudying Ġthe Ġgeometry Ġof Ġthe Ġrepresentation Ġspace Ġfor Ġeach Ġlayer Ġof ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . ĠOur Ġanalysis Ġyields Ġsome Ġsurprising Ġfindings : </s> In Ġall Ġlayers Ġof Ġall Ġthree Ġmodels , Ġthe Ġcontextual ized Ġword Ġrepresentations Ġof Ġall Ġwords Ġare Ġnot Ġisot rop ic : Ġthey Ġare Ġnot Ġuniformly Ġdistributed Ġwith Ġrespect Ġto Ġdirection . ĠInstead , Ġthey Ġare Ġan is otropic , Ġoccupying Ġa Ġnarrow Ġcone Ġin Ġthe Ġvector Ġspace . ĠThe Ġan is ot ropy Ġin ĠG PT - 2 's Ġlast Ġlayer Ġis Ġso Ġextreme Ġthat Ġtwo Ġrandom Ġwords Ġwill Ġon Ġaverage Ġhave Ġalmost Ġperfect Ġcos ine Ġsimilarity ! ĠGiven Ġthat Ġisot ropy Ġhas Ġboth Ġtheoretical Ġand Ġempirical Ġbenefits Ġfor Ġstatic Ġembed d ings ĠB IB REF 6 , Ġthe Ġextent Ġof Ġan is ot ropy Ġin Ġcontextual ized Ġrepresentations Ġis Ġsurprising . </s> Occ ur rences Ġof Ġthe Ġsame Ġword Ġin Ġdifferent Ġcontexts Ġhave Ġnon - ident ical Ġvector Ġrepresentations . ĠWhere Ġvector Ġsimilarity Ġis Ġdefined Ġas Ġcos ine Ġsimilarity , Ġthese Ġrepresentations Ġare Ġmore Ġdiss imilar Ġto Ġeach Ġother Ġin Ġupper Ġlayers . ĠThis Ġsuggests Ġthat , Ġmuch Ġlike Ġhow Ġupper Ġlayers Ġof ĠL ST Ms Ġproduce Ġmore Ġtask - specific Ġrepresentations ĠB IB REF 4 , Ġupper Ġlayers Ġof Ġcontextual izing Ġmodels Ġproduce Ġmore Ġcontext - specific Ġrepresentations . </s> Context - specific ity Ġmanifests Ġvery Ġdifferently Ġin ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . ĠIn ĠEL Mo , Ġrepresentations Ġof Ġwords Ġin Ġthe Ġsame Ġsentence Ġgrow Ġmore Ġsimilar Ġto Ġeach Ġother Ġas Ġcontext - specific ity Ġincreases Ġin Ġupper Ġlayers ; Ġin ĠB ERT , Ġthey Ġbecome Ġmore Ġdiss imilar Ġto Ġeach Ġother Ġin Ġupper Ġlayers Ġbut Ġare Ġstill Ġmore Ġsimilar Ġthan Ġrandomly Ġsampled Ġwords Ġare Ġon Ġaverage ; Ġin ĠG PT - 2 , Ġhowever , Ġwords Ġin Ġthe Ġsame Ġsentence Ġare Ġno Ġmore Ġsimilar Ġto Ġeach Ġother Ġthan Ġtwo Ġrandomly Ġchosen Ġwords . </s> After Ġadjusting Ġfor Ġthe Ġeffect Ġof Ġan is ot ropy , Ġon Ġaverage , Ġless Ġthan Ġ5 % Ġof Ġthe Ġvariance Ġin Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġcan Ġbe Ġexplained Ġby Ġtheir Ġfirst Ġprincipal Ġcomponent . ĠThis Ġholds Ġacross Ġall Ġlayers Ġof Ġall Ġmodels . ĠThis Ġsuggests Ġthat Ġcontextual ized Ġrepresentations Ġdo Ġnot Ġcorrespond Ġto Ġa Ġfinite Ġnumber Ġof Ġword - sense Ġrepresentations , Ġand Ġeven Ġin Ġthe Ġbest Ġpossible Ġscenario , Ġstatic Ġembed d ings Ġwould Ġbe Ġa Ġpoor Ġreplacement Ġfor Ġcontextual ized Ġones . ĠStill , Ġstatic Ġembed d ings Ġcreated Ġby Ġtaking Ġthe Ġfirst Ġprincipal Ġcomponent Ġof Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġoutper form ĠGl o Ve Ġand ĠFast Text Ġembed d ings Ġon Ġmany Ġword Ġvector Ġbenchmarks . </s> These Ġinsights Ġhelp Ġjustify Ġwhy Ġthe Ġuse Ġof Ġcontextual ized Ġrepresentations Ġhas Ġled Ġto Ġsuch Ġsignificant Ġimprovements Ġon Ġmany ĠN LP Ġtasks . </s> Related ĠWork Ġ: :: ĠStatic ĠWord ĠEmb edd ings </s> Skip - gram Ġwith Ġnegative Ġsampling Ġ( SG NS ) ĠB IB REF 0 Ġand ĠGl o Ve ĠB IB REF 1 Ġare Ġamong Ġthe Ġbest Ġknown Ġmodels Ġfor Ġgenerating Ġstatic Ġword Ġembed d ings . ĠThough Ġthey Ġlearn Ġembed d ings Ġiter atively Ġin Ġpractice , Ġit Ġhas Ġbeen Ġproven Ġthat Ġin Ġtheory , Ġthey Ġboth Ġimplicitly Ġfactor ize Ġa Ġword - context Ġmatrix Ġcontaining Ġa Ġco - occ urrence Ġstatistic ĠB IB REF 7 , ĠB IB REF 8 . ĠBecause Ġthey Ġcreate Ġa Ġsingle Ġrepresentation Ġfor Ġeach Ġword , Ġa Ġnotable Ġproblem Ġwith Ġstatic Ġword Ġembed d ings Ġis Ġthat Ġall Ġsenses Ġof Ġa Ġpoly sem ous Ġword Ġmust Ġshare Ġa Ġsingle Ġvector . </s> Related ĠWork Ġ: :: ĠContext ual ized ĠWord ĠRepresent ations </s> Given Ġthe Ġlimitations Ġof Ġstatic Ġword Ġembed d ings , Ġrecent Ġwork Ġhas Ġtried Ġto Ġcreate Ġcontext - sensitive Ġword Ġrepresentations . ĠEL Mo ĠB IB REF 2 , ĠB ERT ĠB IB REF 3 , Ġand ĠG PT - 2 ĠB IB REF 9 Ġare Ġdeep Ġneural Ġlanguage Ġmodels Ġthat Ġare Ġfine - tun ed Ġto Ġcreate Ġmodels Ġfor Ġa Ġwide Ġrange Ġof Ġdownstream ĠN LP Ġtasks . ĠTheir Ġinternal Ġrepresentations Ġof Ġwords Ġare Ġcalled Ġcontextual ized Ġword Ġrepresentations Ġbecause Ġthey Ġare Ġa Ġfunction Ġof Ġthe Ġentire Ġinput Ġsentence . ĠThe Ġsuccess Ġof Ġthis Ġapproach Ġsuggests Ġthat Ġthese Ġrepresentations Ġcapture Ġhighly Ġtransfer able Ġand Ġtask - agn ostic Ġproperties Ġof Ġlanguage ĠB IB REF 4 . </s> EL Mo Ġcreates Ġcontextual ized Ġrepresentations Ġof Ġeach Ġtoken Ġby Ġconc aten ating Ġthe Ġinternal Ġstates Ġof Ġa Ġ2 - layer Ġbi L ST M Ġtrained Ġon Ġa Ġbid irection al Ġlanguage Ġmodelling Ġtask ĠB IB REF 2 . ĠIn Ġcontrast , ĠB ERT Ġand ĠG PT - 2 Ġare Ġbi - direction al Ġand Ġun i - direction al Ġtransformer - based Ġlanguage Ġmodels Ġrespectively . ĠEach Ġtransformer Ġlayer Ġof Ġ12 - layer ĠB ERT Ġ( base , Ġc ased ) Ġand Ġ12 - layer ĠG PT - 2 Ġcreates Ġa Ġcontextual ized Ġrepresentation Ġof Ġeach Ġtoken Ġby Ġattending Ġto Ġdifferent Ġparts Ġof Ġthe Ġinput Ġsentence ĠB IB REF 3 , ĠB IB REF 9 . ĠB ERT ĠâĢĵ Ġand Ġsubsequent Ġiterations Ġon ĠB ERT ĠB IB REF 10 , ĠB IB REF 11 ĠâĢĵ Ġhave Ġachieved Ġstate - of - the - art Ġperformance Ġon Ġvarious Ġdownstream ĠN LP Ġtasks , Ġranging Ġfrom Ġquestion - ans w ering Ġto Ġsentiment Ġanalysis . </s> Related ĠWork Ġ: :: ĠPro bing ĠT asks </s> Prior Ġanalysis Ġof Ġcontextual ized Ġword Ġrepresentations Ġhas Ġlargely Ġbeen Ġrestricted Ġto Ġprobing Ġtasks ĠB IB REF 12 , ĠB IB REF 5 . ĠThis Ġinvolves Ġtraining Ġlinear Ġmodels Ġto Ġpredict Ġsynt actic Ġ( e . g ., Ġpart - of - speech Ġtag ) Ġand Ġsemantic Ġ( e . g ., Ġword Ġrelation ) Ġproperties Ġof Ġwords . ĠPro bing Ġmodels Ġare Ġbased Ġon Ġthe Ġpremise Ġthat Ġif Ġa Ġsimple Ġlinear Ġmodel Ġcan Ġbe Ġtrained Ġto Ġaccurately Ġpredict Ġa Ġlinguistic Ġproperty , Ġthen Ġthe Ġrepresentations Ġimplicitly Ġencode Ġthis Ġinformation Ġto Ġbegin Ġwith . ĠWhile Ġthese Ġanalyses Ġhave Ġfound Ġthat Ġcontextual ized Ġrepresentations Ġencode Ġsemantic Ġand Ġsynt actic Ġinformation , Ġthey Ġcannot Ġanswer Ġhow Ġcontextual Ġthese Ġrepresentations Ġare , Ġand Ġto Ġwhat Ġextent Ġthey Ġcan Ġbe Ġreplaced Ġwith Ġstatic Ġword Ġembed d ings , Ġif Ġat Ġall . ĠOur Ġwork Ġin Ġthis Ġpaper Ġis Ġthus Ġmarkedly Ġdifferent Ġfrom Ġmost Ġdisse ctions Ġof Ġcontextual ized Ġrepresentations . ĠIt Ġis Ġmore Ġsimilar Ġto ĠB IB REF 13 , Ġwhich Ġstudied Ġthe Ġgeometry Ġof Ġstatic Ġword Ġembed ding Ġspaces . </s> App roach Ġ: :: ĠContext ual izing ĠModels </s> The Ġcontextual izing Ġmodels Ġwe Ġstudy Ġin Ġthis Ġpaper Ġare ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . ĠWe Ġchoose Ġthe Ġbase Ġc ased Ġversion Ġof ĠB ERT Ġbecause Ġit Ġis Ġmost Ġcomparable Ġto ĠG PT - 2 Ġwith Ġrespect Ġto Ġnumber Ġof Ġlayers Ġand Ġdimension ality . ĠThe Ġmodels Ġwe Ġwork Ġwith Ġare Ġall Ġpre - trained Ġon Ġtheir Ġrespective Ġlanguage Ġmodelling Ġtasks . ĠAlthough ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 Ġhave Ġ2 , Ġ12 , Ġand Ġ12 Ġhidden Ġlayers Ġrespectively , Ġwe Ġalso Ġinclude Ġthe Ġinput Ġlayer Ġof Ġeach Ġcontextual izing Ġmodel Ġas Ġits Ġ0 th Ġlayer . ĠThis Ġis Ġbecause Ġthe Ġ0 th Ġlayer Ġis Ġnot Ġcontextual ized , Ġmaking Ġit Ġa Ġuseful Ġbaseline Ġagainst Ġwhich Ġto Ġcompare Ġthe Ġcontextual ization Ġdone Ġby Ġsubsequent Ġlayers . </s> App roach Ġ: :: ĠData </s> To Ġanalyze Ġcontextual ized Ġword Ġrepresentations , Ġwe Ġneed Ġinput Ġsentences Ġto Ġfeed Ġinto Ġour Ġpre - trained Ġmodels . ĠOur Ġinput Ġdata Ġcome Ġfrom Ġthe ĠSem E val ĠSem antic ĠText ual ĠSimilar ity Ġtasks Ġfrom Ġyears Ġ2012 Ġ- Ġ2016 ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 17 . ĠWe Ġuse Ġthese Ġdatasets Ġbecause Ġthey Ġcontain Ġsentences Ġin Ġwhich Ġthe Ġsame Ġwords Ġappear Ġin Ġdifferent Ġcontexts . ĠFor Ġexample , Ġthe Ġword Ġ` dog ' Ġappears Ġin ĠâĢ ľ A Ġp anda Ġdog Ġis Ġrunning Ġon Ġthe Ġroad . âĢ Ŀ Ġand ĠâĢ ľ A Ġdog Ġis Ġtrying Ġto Ġget Ġbacon Ġoff Ġhis Ġback . âĢ Ŀ ĠIf Ġa Ġmodel Ġgenerated Ġthe Ġsame Ġrepresentation Ġfor Ġ` dog ' Ġin Ġboth Ġthese Ġsentences , Ġwe Ġcould Ġinfer Ġthat Ġthere Ġwas Ġno Ġcontextual ization ; Ġcon versely , Ġif Ġthe Ġtwo Ġrepresentations Ġwere Ġdifferent , Ġwe Ġcould Ġinfer Ġthat Ġthey Ġwere Ġcontextual ized Ġto Ġsome Ġextent . ĠUsing Ġthese Ġdatasets , Ġwe Ġmap Ġwords Ġto Ġthe Ġlist Ġof Ġsentences Ġthey Ġappear Ġin Ġand Ġtheir Ġindex Ġwithin Ġthese Ġsentences . ĠWe Ġdo Ġnot Ġconsider Ġwords Ġthat Ġappear Ġin Ġless Ġthan Ġ5 Ġunique Ġcontexts Ġin Ġour Ġanalysis . </s> App roach Ġ: :: ĠMeasures Ġof ĠContext uality </s> We Ġmeasure Ġhow Ġcontextual Ġa Ġword Ġrepresentation Ġis Ġusing Ġthree Ġdifferent Ġmetrics : Ġself - similar ity , Ġintra - sent ence Ġsimilarity , Ġand Ġmaximum Ġexplain able Ġvariance . </s> App roach Ġ: :: ĠMeasures Ġof ĠContext uality Ġ: :: ĠDefinition Ġ1 </s> Let Ġ$ w $ Ġbe Ġa Ġword Ġthat Ġappears Ġin Ġsentences Ġ$\ l brace Ġs _ 1 , Ġ... , Ġs _ n Ġ\ r brace Ġ$ Ġat Ġindices Ġ$\ l brace Ġi _ 1 , Ġ... , Ġi _ n Ġ\ r brace Ġ$ Ġrespectively , Ġsuch Ġthat Ġ$ w Ġ= Ġs _ 1 [ i _ 1 ] Ġ= Ġ... Ġ= Ġs _ n [ i _ n ] $. ĠLet Ġ$ f _{ \ ell Ġ} ( s , i ) $ Ġbe Ġa Ġfunction Ġthat Ġmaps Ġ$ s [ i ] $ Ġto Ġits Ġrepresentation Ġin Ġlayer Ġ$\ ell Ġ$ Ġof Ġmodel Ġ$ f $. ĠThe Ġself Ġsimilarity Ġof Ġ$ w $ Ġin Ġlayer Ġ$\ ell Ġ$ Ġis </s> where Ġ$\ cos Ġ$ Ġdenotes Ġthe Ġcos ine Ġsimilarity . ĠIn Ġother Ġwords , Ġthe Ġself - similar ity Ġof Ġa Ġword Ġ$ w $ Ġin Ġlayer Ġ$\ ell Ġ$ Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġits Ġcontextual ized Ġrepresentations Ġacross Ġits Ġ$ n $ Ġunique Ġcontexts . ĠIf Ġlayer Ġ$\ ell Ġ$ Ġdoes Ġnot Ġcontextual ize Ġthe Ġrepresentations Ġat Ġall , Ġthen Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) Ġ= Ġ1 $ Ġ( i . e ., Ġthe Ġrepresentations Ġare Ġidentical Ġacross Ġall Ġcontexts ). ĠThe Ġmore Ġcontextual ized Ġthe Ġrepresentations Ġare Ġfor Ġ$ w $, Ġthe Ġlower Ġwe Ġwould Ġexpect Ġits Ġself - similar ity Ġto Ġbe . </s> App roach Ġ: :: ĠMeasures Ġof ĠContext uality Ġ: :: ĠDefinition Ġ2 </s> Let Ġ$ s $ Ġbe Ġa Ġsentence Ġthat Ġis Ġa Ġsequence Ġ$\ left < Ġw _ 1 , Ġ... , Ġw _ n Ġ\ right > $ Ġof Ġ$ n $ Ġwords . ĠLet Ġ$ f _{ \ ell Ġ} ( s , i ) $ Ġbe Ġa Ġfunction Ġthat Ġmaps Ġ$ s [ i ] $ Ġto Ġits Ġrepresentation Ġin Ġlayer Ġ$\ ell Ġ$ Ġof Ġmodel Ġ$ f $. ĠThe Ġintra - sent ence Ġsimilarity Ġof Ġ$ s $ Ġin Ġlayer Ġ$\ ell Ġ$ Ġis </s> Put Ġmore Ġsimply , Ġthe Ġintra - sent ence Ġsimilarity Ġof Ġa Ġsentence Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġits Ġword Ġrepresentations Ġand Ġthe Ġsentence Ġvector , Ġwhich Ġis Ġjust Ġthe Ġmean Ġof Ġthose Ġword Ġvectors . ĠThis Ġmeasure Ġcaptures Ġhow Ġcontext - specific ity Ġmanifests Ġin Ġthe Ġvector Ġspace . ĠFor Ġexample , Ġif Ġboth Ġ$\ text it Ġ{ Int ra Sim } _ \ ell Ġ( s ) $ Ġand Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) $ Ġare Ġlow Ġ$\ for all Ġ\ Ġw Ġ\ in Ġs $, Ġthen Ġthe Ġmodel Ġcontextual izes Ġwords Ġin Ġthat Ġlayer Ġby Ġgiving Ġeach Ġone Ġa Ġcontext - specific Ġrepresentation Ġthat Ġis Ġstill Ġdistinct Ġfrom Ġall Ġother Ġword Ġrepresentations Ġin Ġthe Ġsentence . ĠIf Ġ$\ text it Ġ{ Int ra Sim } _ \ ell Ġ( s ) $ Ġis Ġhigh Ġbut Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) $ Ġis Ġlow , Ġthis Ġsuggests Ġa Ġless Ġnuanced Ġcontextual ization , Ġwhere Ġwords Ġin Ġa Ġsentence Ġare Ġcontextual ized Ġsimply Ġby Ġmaking Ġtheir Ġrepresentations Ġconverge Ġin Ġvector Ġspace . </s> App roach Ġ: :: ĠMeasures Ġof ĠContext uality Ġ: :: ĠDefinition Ġ3 </s> Let Ġ$ w $ Ġbe Ġa Ġword Ġthat Ġappears Ġin Ġsentences Ġ$\ l brace Ġs _ 1 , Ġ... , Ġs _ n Ġ\ r brace Ġ$ Ġat Ġindices Ġ$\ l brace Ġi _ 1 , Ġ... , Ġi _ n Ġ\ r brace Ġ$ Ġrespectively , Ġsuch Ġthat Ġ$ w Ġ= Ġs _ 1 [ i _ 1 ] Ġ= Ġ... Ġ= Ġs _ n [ i _ n ] $. ĠLet Ġ$ f _{ \ ell Ġ} ( s , i ) $ Ġbe Ġa Ġfunction Ġthat Ġmaps Ġ$ s [ i ] $ Ġto Ġits Ġrepresentation Ġin Ġlayer Ġ$\ ell Ġ$ Ġof Ġmodel Ġ$ f $. ĠWhere Ġ$ [ Ġf _{ \ ell Ġ} ( s _ 1 , Ġi _ 1 ) Ġ... Ġf _{ \ ell Ġ} ( s _ n , Ġi _ n ) Ġ] $ Ġis Ġthe Ġoccurrence Ġmatrix Ġof Ġ$ w $ Ġand Ġ$\ s igma Ġ_ 1 Ġ... Ġ\ s igma Ġ_ m $ Ġare Ġthe Ġfirst Ġ$ m $ Ġsingular Ġvalues Ġof Ġthis Ġmatrix , Ġthe Ġmaximum Ġexplain able Ġvariance Ġis </s> $ \ text it Ġ{ ME V } _ \ ell Ġ( w ) $ Ġis Ġthe Ġproportion Ġof Ġvariance Ġin Ġ$ w $ ' s Ġcontextual ized Ġrepresentations Ġfor Ġa Ġgiven Ġlayer Ġthat Ġcan Ġbe Ġexplained Ġby Ġtheir Ġfirst Ġprincipal Ġcomponent . ĠIt Ġgives Ġus Ġan Ġupper Ġbound Ġon Ġhow Ġwell Ġa Ġstatic Ġembed ding Ġcould Ġreplace Ġa Ġword 's Ġcontextual ized Ġrepresentations . ĠThe Ġcloser Ġ$\ text it Ġ{ ME V } _ \ ell Ġ( w ) $ Ġis Ġto Ġ0 , Ġthe Ġpoorer Ġa Ġreplacement Ġa Ġstatic Ġembed ding Ġwould Ġbe ; Ġif Ġ$\ text it Ġ{ ME V } _ \ ell Ġ( w ) Ġ= Ġ1 $, Ġthen Ġa Ġstatic Ġembed ding Ġwould Ġbe Ġa Ġperfect Ġreplacement Ġfor Ġthe Ġcontextual ized Ġrepresentations . </s> App roach Ġ: :: ĠAdjust ing Ġfor ĠAn is ot ropy </s> It Ġis Ġimportant Ġto Ġconsider Ġisot ropy Ġ( or Ġthe Ġlack Ġthereof ) Ġwhen Ġdiscussing Ġcontext uality . ĠFor Ġexample , Ġif Ġword Ġvectors Ġwere Ġperfectly Ġisot rop ic Ġ( i . e ., Ġdirection ally Ġuniform ), Ġthen Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) Ġ= Ġ0 . 95 $ Ġwould Ġsuggest Ġthat Ġ$ w $ ' s Ġrepresentations Ġwere Ġpoorly Ġcontextual ized . ĠHowever , Ġconsider Ġthe Ġscenario Ġwhere Ġword Ġvectors Ġare Ġso Ġan is otropic Ġthat Ġany Ġtwo Ġwords Ġhave Ġon Ġaverage Ġa Ġcos ine Ġsimilarity Ġof Ġ0 . 99 . ĠThen Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) Ġ= Ġ0 . 95 $ Ġwould Ġactually Ġsuggest Ġthe Ġopposite ĠâĢĵ Ġthat Ġ$ w $ ' s Ġrepresentations Ġwere Ġwell Ġcontextual ized . ĠThis Ġis Ġbecause Ġrepresentations Ġof Ġ$ w $ Ġin Ġdifferent Ġcontexts Ġwould Ġon Ġaverage Ġbe Ġmore Ġdiss imilar Ġto Ġeach Ġother Ġthan Ġtwo Ġrandomly Ġchosen Ġwords . </s> To Ġadjust Ġfor Ġthe Ġeffect Ġof Ġan is ot ropy , Ġwe Ġuse Ġthree Ġan is otropic Ġbas elines , Ġone Ġfor Ġeach Ġof Ġour Ġcontext uality Ġmeasures . ĠFor Ġself - similar ity Ġand Ġintra - sent ence Ġsimilarity , Ġthe Ġbaseline Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġthe Ġrepresentations Ġof Ġuniformly Ġrandomly Ġsampled Ġwords Ġfrom Ġdifferent Ġcontexts . ĠThe Ġmore Ġan is otropic Ġthe Ġword Ġrepresentations Ġare Ġin Ġa Ġgiven Ġlayer , Ġthe Ġcloser Ġthis Ġbaseline Ġis Ġto Ġ1 . ĠFor Ġmaximum Ġexplain able Ġvariance Ġ( ME V ), Ġthe Ġbaseline Ġis Ġthe Ġproportion Ġof Ġvariance Ġin Ġuniformly Ġrandomly Ġsampled Ġword Ġrepresentations Ġthat Ġis Ġexplained Ġby Ġtheir Ġfirst Ġprincipal Ġcomponent . ĠThe Ġmore Ġan is otropic Ġthe Ġrepresentations Ġin Ġa Ġgiven Ġlayer , Ġthe Ġcloser Ġthis Ġbaseline Ġis Ġto Ġ1 : Ġeven Ġfor Ġa Ġrandom Ġassortment Ġof Ġwords , Ġthe Ġprincipal Ġcomponent Ġwould Ġbe Ġable Ġto Ġexplain Ġa Ġlarge Ġproportion Ġof Ġthe Ġvariance . </s> Since Ġcontext uality Ġmeasures Ġare Ġcalculated Ġfor Ġeach Ġlayer Ġof Ġa Ġcontextual izing Ġmodel , Ġwe Ġcalculate Ġseparate Ġbas elines Ġfor Ġeach Ġlayer Ġas Ġwell . ĠWe Ġthen Ġsubtract Ġfrom Ġeach Ġmeasure Ġits Ġrespective Ġbaseline Ġto Ġget Ġthe Ġan is ot ropy - adjusted Ġcont ex uality Ġmeasure . ĠFor Ġexample , Ġthe Ġan is ot ropy - adjusted Ġself - similar ity Ġis </s> where Ġ$\ math cal Ġ{ O } $ Ġis Ġthe Ġset Ġof Ġall Ġword Ġoccurrences Ġand Ġ$ f _{ \ ell Ġ} (\ cd ot Ġ) $ Ġmaps Ġa Ġword Ġoccurrence Ġto Ġits Ġrepresentation Ġin Ġlayer Ġ$\ ell Ġ$ Ġof Ġmodel Ġ$ f $. ĠUnless Ġotherwise Ġstated , Ġreferences Ġto Ġcontext uality Ġmeasures Ġin Ġthe Ġrest Ġof Ġthe Ġpaper Ġrefer Ġto Ġthe Ġan is ot ropy - adjusted Ġmeasures , Ġwhere Ġboth Ġthe Ġraw Ġmeasure Ġand Ġbaseline Ġare Ġestimated Ġwith Ġ1 K Ġuniformly Ġrandomly Ġsampled Ġword Ġrepresentations . </s> Find ings Ġ: :: Ġ( An ) Is ot ropy Ġ: :: ĠContext ual ized Ġrepresentations Ġare Ġan is otropic Ġin Ġall Ġnon - input Ġlayers . </s> If Ġword Ġrepresentations Ġfrom Ġa Ġparticular Ġlayer Ġwere Ġisot rop ic Ġ( i . e ., Ġdirection ally Ġuniform ), Ġthen Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġuniformly Ġrandomly Ġsampled Ġwords Ġwould Ġbe Ġ0 ĠB IB REF 18 . ĠThe Ġcloser Ġthis Ġaverage Ġis Ġto Ġ1 , Ġthe Ġmore Ġan is otropic Ġthe Ġrepresentations . ĠThe Ġgeometric Ġinterpretation Ġof Ġan is ot ropy Ġis Ġthat Ġthe Ġword Ġrepresentations Ġall Ġoccupy Ġa Ġnarrow Ġcone Ġin Ġthe Ġvector Ġspace Ġrather Ġthan Ġbeing Ġuniform Ġin Ġall Ġdirections ; Ġthe Ġgreater Ġthe Ġan is ot ropy , Ġthe Ġnarrower Ġthis Ġcone ĠB IB REF 13 . ĠAs Ġseen Ġin ĠFigure ĠFIG REF 20 , Ġthis Ġimplies Ġthat Ġin Ġalmost Ġall Ġlayers Ġof ĠB ERT , ĠEL Mo Ġand ĠG PT - 2 , Ġthe Ġrepresentations Ġof Ġall Ġwords Ġoccupy Ġa Ġnarrow Ġcone Ġin Ġthe Ġvector Ġspace . ĠThe Ġonly Ġexception Ġis ĠEL Mo 's Ġinput Ġlayer , Ġwhich Ġproduces Ġstatic Ġcharacter - level Ġembed d ings Ġwithout Ġusing Ġcontextual Ġor Ġeven Ġpositional Ġinformation ĠB IB REF 2 . ĠIt Ġshould Ġbe Ġnoted Ġthat Ġnot Ġall Ġstatic Ġembed d ings Ġare Ġnecessarily Ġisot rop ic , Ġhowever ; ĠB IB REF 13 Ġfound Ġthat Ġskip gram Ġembed d ings , Ġwhich Ġare Ġalso Ġstatic , Ġare Ġnot Ġisot rop ic . </s> Find ings Ġ: :: Ġ( An ) Is ot ropy Ġ: :: ĠContext ual ized Ġrepresentations Ġare Ġgenerally Ġmore Ġan is otropic Ġin Ġhigher Ġlayers . </s> As Ġseen Ġin ĠFigure ĠFIG REF 20 , Ġfor ĠG PT - 2 , Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġuniformly Ġrandomly Ġwords Ġis Ġroughly Ġ0 . 6 Ġin Ġlayers Ġ2 Ġthrough Ġ8 Ġbut Ġincreases Ġexponentially Ġfrom Ġlayers Ġ8 Ġthrough Ġ12 . ĠIn Ġfact , Ġword Ġrepresentations Ġin ĠG PT - 2 's Ġlast Ġlayer Ġare Ġso Ġan is otropic Ġthat Ġany Ġtwo Ġwords Ġhave Ġon Ġaverage Ġan Ġalmost Ġperfect Ġcos ine Ġsimilarity ! ĠThis Ġpattern Ġholds Ġfor ĠB ERT Ġand ĠEL Mo Ġas Ġwell , Ġthough Ġthere Ġare Ġexceptions : Ġfor Ġexample , Ġthe Ġan is ot ropy Ġin ĠB ERT 's Ġpen ultimate Ġlayer Ġis Ġmuch Ġhigher Ġthan Ġin Ġits Ġfinal Ġlayer . </s> Is ot ropy Ġhas Ġboth Ġtheoretical Ġand Ġempirical Ġbenefits Ġfor Ġstatic Ġword Ġembed d ings . ĠIn Ġtheory , Ġit Ġallows Ġfor Ġstronger ĠâĢ ľ self - normal ization âĢ Ŀ Ġduring Ġtraining ĠB IB REF 18 , Ġand Ġin Ġpractice , Ġsubtract ing Ġthe Ġmean Ġvector Ġfrom Ġstatic Ġembed d ings Ġleads Ġto Ġimprovements Ġon Ġseveral Ġdownstream ĠN LP Ġtasks ĠB IB REF 6 . ĠThus Ġthe Ġextreme Ġdegree Ġof Ġan is ot ropy Ġseen Ġin Ġcontextual ized Ġword Ġrepresentations ĠâĢĵ Ġparticularly Ġin Ġhigher Ġlayers ĠâĢĵ Ġis Ġsurprising . ĠAs Ġseen Ġin ĠFigure ĠFIG REF 20 , Ġfor Ġall Ġthree Ġmodels , Ġthe Ġcontextual ized Ġhidden Ġlayer Ġrepresentations Ġare Ġalmost Ġall Ġmore Ġan is otropic Ġthan Ġthe Ġinput Ġlayer Ġrepresentations , Ġwhich Ġdo Ġnot Ġincorporate Ġcontext . ĠThis Ġsuggests Ġthat Ġhigh Ġan is ot ropy Ġis Ġinherent Ġto , Ġor Ġleast Ġa Ġby - product Ġof , Ġthe Ġprocess Ġof Ġcontextual ization . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠContext ual ized Ġword Ġrepresentations Ġare Ġmore Ġcontext - specific Ġin Ġhigher Ġlayers . </s> Rec all Ġfrom ĠDefinition Ġ1 Ġthat Ġthe Ġself - similar ity Ġof Ġa Ġword , Ġin Ġa Ġgiven Ġlayer Ġof Ġa Ġgiven Ġmodel , Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġits Ġrepresentations Ġin Ġdifferent Ġcontexts , Ġadjusted Ġfor Ġan is ot ropy . ĠIf Ġthe Ġself - similar ity Ġis Ġ1 , Ġthen Ġthe Ġrepresentations Ġare Ġnot Ġcontext - specific Ġat Ġall ; Ġif Ġthe Ġself - similar ity Ġis Ġ0 , Ġthat Ġthe Ġrepresentations Ġare Ġmaxim ally Ġcontext - specific . ĠIn ĠFigure ĠFIG REF 24 , Ġwe Ġplot Ġthe Ġaverage Ġself - similar ity Ġof Ġuniformly Ġrandomly Ġsampled Ġwords Ġin Ġeach Ġlayer Ġof ĠB ERT , ĠEL Mo , Ġand ĠG PT - 2 . ĠFor Ġexample , Ġthe Ġself - similar ity Ġis Ġ1 . 0 Ġin ĠEL Mo 's Ġinput Ġlayer Ġbecause Ġrepresentations Ġin Ġthat Ġlayer Ġare Ġstatic Ġcharacter - level Ġembed d ings . </s> In Ġall Ġthree Ġmodels , Ġthe Ġhigher Ġthe Ġlayer , Ġthe Ġlower Ġthe Ġself - similar ity Ġis Ġon Ġaverage . ĠIn Ġother Ġwords , Ġthe Ġhigher Ġthe Ġlayer , Ġthe Ġmore Ġcontext - specific Ġthe Ġcontextual ized Ġrepresentations . ĠThis Ġfinding Ġmakes Ġintuitive Ġsense . ĠIn Ġimage Ġclassification Ġmodels , Ġlower Ġlayers Ġrecognize Ġmore Ġgeneric Ġfeatures Ġsuch Ġas Ġedges Ġwhile Ġupper Ġlayers Ġrecognize Ġmore Ġclass - specific Ġfeatures ĠB IB REF 19 . ĠSimilarly , Ġupper Ġlayers Ġof ĠL ST Ms Ġtrained Ġon ĠN LP Ġtasks Ġlearn Ġmore Ġtask - specific Ġrepresentations ĠB IB REF 4 . ĠTherefore , Ġit Ġfollows Ġthat Ġupper Ġlayers Ġof Ġneural Ġlanguage Ġmodels Ġlearn Ġmore Ġcontext - specific Ġrepresentations , Ġso Ġas Ġto Ġpredict Ġthe Ġnext Ġword Ġfor Ġa Ġgiven Ġcontext Ġmore Ġaccurately . ĠOf Ġall Ġthree Ġmodels , Ġrepresentations Ġin ĠG PT - 2 Ġare Ġthe Ġmost Ġcontext - specific , Ġwith Ġthose Ġin ĠG PT - 2 's Ġlast Ġlayer Ġbeing Ġalmost Ġmaxim ally Ġcontext - specific . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠStop words Ġ( e . g ., Ġ` the ', Ġ` of ', Ġ` to ') Ġhave Ġamong Ġthe Ġmost Ġcontext - specific Ġrepresentations . </s> Across Ġall Ġlayers , Ġstop words Ġhave Ġamong Ġthe Ġlowest Ġself - similar ity Ġof Ġall Ġwords , Ġimplying Ġthat Ġtheir Ġcontextual ized Ġrepresentations Ġare Ġamong Ġthe Ġmost Ġcontext - specific . ĠFor Ġexample , Ġthe Ġwords Ġwith Ġthe Ġlowest Ġaverage Ġself - similar ity Ġacross ĠEL Mo 's Ġlayers Ġare Ġ` and ', Ġ` of ', Ġ` ' s ', Ġ` the ', Ġand Ġ` to '. ĠThis Ġis Ġrelatively Ġsurprising , Ġgiven Ġthat Ġthese Ġwords Ġare Ġnot Ġpoly sem ous . ĠThis Ġfinding Ġsuggests Ġthat Ġthe Ġvariety Ġof Ġcontexts Ġa Ġword Ġappears Ġin , Ġrather Ġthan Ġits Ġinherent Ġpoly se my , Ġis Ġwhat Ġdrives Ġvariation Ġin Ġits Ġcontextual ized Ġrepresentations . ĠThis Ġanswers Ġone Ġof Ġthe Ġquestions Ġwe Ġposed Ġin Ġthe Ġintroduction : ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 Ġare Ġnot Ġsimply Ġassigning Ġone Ġof Ġa Ġfinite Ġnumber Ġof Ġword - sense Ġrepresentations Ġto Ġeach Ġword ; Ġotherwise , Ġthere Ġwould Ġnot Ġbe Ġso Ġmuch Ġvariation Ġin Ġthe Ġrepresentations Ġof Ġwords Ġwith Ġso Ġfew Ġword Ġsenses . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠContext - specific ity Ġmanifests Ġvery Ġdifferently Ġin ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . </s> As Ġnoted Ġearlier , Ġcontextual ized Ġrepresentations Ġare Ġmore Ġcontext - specific Ġin Ġupper Ġlayers Ġof ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . ĠHowever , Ġhow Ġdoes Ġthis Ġincreased Ġcontext - specific ity Ġmanifest Ġin Ġthe Ġvector Ġspace ? ĠDo Ġword Ġrepresentations Ġin Ġthe Ġsame Ġsentence Ġconverge Ġto Ġa Ġsingle Ġpoint , Ġor Ġdo Ġthey Ġremain Ġdistinct Ġfrom Ġone Ġanother Ġwhile Ġstill Ġbeing Ġdistinct Ġfrom Ġtheir Ġrepresentations Ġin Ġother Ġcontexts ? ĠTo Ġanswer Ġthis Ġquestion , Ġwe Ġcan Ġmeasure Ġa Ġsentence 's Ġintra - sent ence Ġsimilarity . ĠRecall Ġfrom ĠDefinition Ġ2 Ġthat Ġthe Ġintra - sent ence Ġsimilarity Ġof Ġa Ġsentence , Ġin Ġa Ġgiven Ġlayer Ġof Ġa Ġgiven Ġmodel , Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġeach Ġof Ġits Ġword Ġrepresentations Ġand Ġtheir Ġmean , Ġadjusted Ġfor Ġan is ot ropy . ĠIn ĠFigure ĠFIG REF 25 , Ġwe Ġplot Ġthe Ġaverage Ġintra - sent ence Ġsimilarity Ġof Ġ500 Ġuniformly Ġrandomly Ġsampled Ġsentences . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠIn ĠEL Mo , Ġwords Ġin Ġthe Ġsame Ġsentence Ġare Ġmore Ġsimilar Ġto Ġone Ġanother Ġin Ġupper Ġlayers . </s> As Ġword Ġrepresentations Ġin Ġa Ġsentence Ġbecome Ġmore Ġcontext - specific Ġin Ġupper Ġlayers , Ġthe Ġintra - sent ence Ġsimilarity Ġalso Ġrises . ĠThis Ġsuggests Ġthat , Ġin Ġpractice , ĠEL Mo Ġends Ġup Ġextending Ġthe Ġintuition Ġbehind ĠF irth 's ĠB IB REF 20 Ġdistribution al Ġhypothesis Ġto Ġthe Ġsentence Ġlevel : Ġthat Ġbecause Ġwords Ġin Ġthe Ġsame Ġsentence Ġshare Ġthe Ġsame Ġcontext , Ġtheir Ġcontextual ized Ġrepresentations Ġshould Ġalso Ġbe Ġsimilar . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠIn ĠB ERT , Ġwords Ġin Ġthe Ġsame Ġsentence Ġare Ġmore Ġdiss imilar Ġto Ġone Ġanother Ġin Ġupper Ġlayers . </s> As Ġword Ġrepresentations Ġin Ġa Ġsentence Ġbecome Ġmore Ġcontext - specific Ġin Ġupper Ġlayers , Ġthey Ġdrift Ġaway Ġfrom Ġone Ġanother , Ġalthough Ġthere Ġare Ġexceptions Ġ( see Ġlayer Ġ12 Ġin ĠFigure ĠFIG REF 25 ). ĠHowever , Ġin Ġall Ġlayers , Ġthe Ġaverage Ġsimilarity Ġbetween Ġwords Ġin Ġthe Ġsame Ġsentence Ġis Ġstill Ġgreater Ġthan Ġthe Ġaverage Ġsimilarity Ġbetween Ġrandomly Ġchosen Ġwords Ġ( i . e ., Ġthe Ġan is ot ropy Ġbaseline ). ĠThis Ġsuggests Ġa Ġmore Ġnuanced Ġcontextual ization Ġthan Ġin ĠEL Mo , Ġwith ĠB ERT Ġrecognizing Ġthat Ġalthough Ġthe Ġsurrounding Ġsentence Ġinforms Ġa Ġword 's Ġmeaning , Ġtwo Ġwords Ġin Ġthe Ġsame Ġsentence Ġdo Ġnot Ġnecessarily Ġhave Ġa Ġsimilar Ġmeaning Ġbecause Ġthey Ġshare Ġthe Ġsame Ġcontext . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠIn ĠG PT - 2 , Ġword Ġrepresentations Ġin Ġthe Ġsame Ġsentence Ġare Ġno Ġmore Ġsimilar Ġto Ġeach Ġother Ġthan Ġrandomly Ġsampled Ġwords . </s> On Ġaverage , Ġthe Ġun adjusted Ġintra - sent ence Ġsimilarity Ġis Ġroughly Ġthe Ġsame Ġas Ġthe Ġan is otropic Ġbaseline , Ġso Ġas Ġseen Ġin ĠFigure ĠFIG REF 25 , Ġthe Ġan is ot ropy - adjusted Ġintra - sent ence Ġsimilarity Ġis Ġclose Ġto Ġ0 Ġin Ġmost Ġlayers Ġof ĠG PT - 2 . ĠIn Ġfact , Ġthe Ġintra - sent ence Ġsimilarity Ġis Ġhighest Ġin Ġthe Ġinput Ġlayer , Ġwhich Ġdoes Ġnot Ġcontextual ize Ġwords Ġat Ġall . ĠThis Ġis Ġin Ġcontrast Ġto ĠEL Mo Ġand ĠB ERT , Ġwhere Ġthe Ġaverage Ġintra - sent ence Ġsimilarity Ġis Ġabove Ġ0 . 20 Ġfor Ġall Ġbut Ġone Ġlayer . </s> As Ġnoted Ġearlier Ġwhen Ġdiscussing ĠB ERT , Ġthis Ġbehavior Ġstill Ġmakes Ġintuitive Ġsense : Ġtwo Ġwords Ġin Ġthe Ġsame Ġsentence Ġdo Ġnot Ġnecessarily Ġhave Ġa Ġsimilar Ġmeaning Ġsimply Ġbecause Ġthey Ġshare Ġthe Ġsame Ġcontext . ĠThe Ġsuccess Ġof ĠG PT - 2 Ġsuggests Ġthat Ġunlike Ġan is ot ropy , Ġwhich Ġaccompanies Ġcontext - specific ity Ġin Ġall Ġthree Ġmodels , Ġa Ġhigh Ġintra - sent ence Ġsimilarity Ġis Ġnot Ġinherent Ġto Ġcontextual ization . ĠWords Ġin Ġthe Ġsame Ġsentence Ġcan Ġhave Ġhighly Ġcontextual ized Ġrepresentations Ġwithout Ġthose Ġrepresentations Ġbeing Ġany Ġmore Ġsimilar Ġto Ġeach Ġother Ġthan Ġtwo Ġrandom Ġword Ġrepresentations . ĠIt Ġis Ġunclear , Ġhowever , Ġwhether Ġthese Ġdifferences Ġin Ġintra - sent ence Ġsimilarity Ġcan Ġbe Ġtraced Ġback Ġto Ġdifferences Ġin Ġmodel Ġarchitecture ; Ġwe Ġleave Ġthis Ġquestion Ġas Ġfuture Ġwork . </s> Find ings Ġ: :: ĠStatic Ġvs . ĠContext ual ized Ġ: :: ĠOn Ġaverage , Ġless Ġthan Ġ5 % Ġof Ġthe Ġvariance Ġin Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġcan Ġbe Ġexplained Ġby Ġa Ġstatic Ġembed ding . </s> Rec all Ġfrom ĠDefinition Ġ3 Ġthat Ġthe Ġmaximum Ġexplain able Ġvariance Ġ( ME V ) Ġof Ġa Ġword , Ġfor Ġa Ġgiven Ġlayer Ġof Ġa Ġgiven Ġmodel , Ġis Ġthe Ġproportion Ġof Ġvariance Ġin Ġits Ġcontextual ized Ġrepresentations Ġthat Ġcan Ġbe Ġexplained Ġby Ġtheir Ġfirst Ġprincipal Ġcomponent . ĠThis Ġgives Ġus Ġan Ġupper Ġbound Ġon Ġhow Ġwell Ġa Ġstatic Ġembed ding Ġcould Ġreplace Ġa Ġword 's Ġcontextual ized Ġrepresentations . ĠBecause Ġcontextual ized Ġrepresentations Ġare Ġan is otropic Ġ( see Ġsection ĠSEC REF 21 ), Ġmuch Ġof Ġthe Ġvariation Ġacross Ġall Ġwords Ġcan Ġbe Ġexplained Ġby Ġa Ġsingle Ġvector . ĠWe Ġadjust Ġfor Ġan is ot ropy Ġby Ġcalculating Ġthe Ġproportion Ġof Ġvariance Ġexplained Ġby Ġthe Ġfirst Ġprincipal Ġcomponent Ġof Ġuniformly Ġrandomly Ġsampled Ġword Ġrepresentations Ġand Ġsubtract ing Ġthis Ġproportion Ġfrom Ġthe Ġraw ĠME V . ĠIn ĠFigure ĠFIG REF 29 , Ġwe Ġplot Ġthe Ġaverage Ġan is ot ropy - adjusted ĠME V Ġacross Ġuniformly Ġrandomly Ġsampled Ġwords . </s> In Ġno Ġlayer Ġof ĠEL Mo , ĠB ERT , Ġor ĠG PT - 2 Ġcan Ġmore Ġthan Ġ5 % Ġof Ġthe Ġvariance Ġin Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġbe Ġexplained Ġby Ġa Ġstatic Ġembed ding , Ġon Ġaverage . ĠThough Ġnot Ġvisible Ġin ĠFigure ĠFIG REF 29 , Ġthe Ġraw ĠME V Ġof Ġmany Ġwords Ġis Ġactually Ġbelow Ġthe Ġan is ot ropy Ġbaseline : Ġi . e ., Ġa Ġgreater Ġproportion Ġof Ġthe Ġvariance Ġacross Ġall Ġwords Ġcan Ġbe Ġexplained Ġby Ġa Ġsingle Ġvector Ġthan Ġcan Ġthe Ġvariance Ġacross Ġall Ġrepresentations Ġof Ġa Ġsingle Ġword . ĠNote Ġthat Ġthe Ġ5 % Ġthreshold Ġrepresents Ġthe Ġbest - case Ġscenario , Ġand Ġthere Ġis Ġno Ġtheoretical Ġguarantee Ġthat Ġa Ġword Ġvector Ġobtained Ġusing ĠGl o Ve , Ġfor Ġexample , Ġwould Ġbe Ġsimilar Ġto Ġthe Ġstatic Ġembed ding Ġthat Ġmaxim izes ĠME V . ĠThis Ġsuggests Ġthat Ġcontextual izing Ġmodels Ġare Ġnot Ġsimply Ġassigning Ġone Ġof Ġa Ġfinite Ġnumber Ġof Ġword - sense Ġrepresentations Ġto Ġeach Ġword ĠâĢĵ Ġotherwise , Ġthe Ġproportion Ġof Ġvariance Ġexplained Ġwould Ġbe Ġmuch Ġhigher . ĠEven Ġthe Ġaverage Ġraw ĠME V Ġis Ġbelow Ġ5 % Ġfor Ġall Ġlayers Ġof ĠEL Mo Ġand ĠB ERT ; Ġonly Ġfor ĠG PT - 2 Ġis Ġthe Ġraw ĠME V Ġnon - neg lig ible , Ġbeing Ġaround Ġ30 % Ġon Ġaverage Ġfor Ġlayers Ġ2 Ġto Ġ11 Ġdue Ġto Ġextremely Ġhigh Ġan is ot ropy . </s> Find ings Ġ: :: ĠStatic Ġvs . ĠContext ual ized Ġ: :: ĠPrincipal Ġcomponents Ġof Ġcontextual ized Ġrepresentations Ġin Ġlower Ġlayers Ġoutper form ĠGl o Ve Ġand ĠFast Text Ġon Ġmany Ġbenchmarks . </s> As Ġnoted Ġearlier , Ġwe Ġcan Ġcreate Ġstatic Ġembed d ings Ġfor Ġeach Ġword Ġby Ġtaking Ġthe Ġfirst Ġprincipal Ġcomponent Ġ( PC ) Ġof Ġits Ġcontextual ized Ġrepresentations Ġin Ġa Ġgiven Ġlayer . ĠIn ĠTable ĠT AB REF 34 , Ġwe Ġplot Ġthe Ġperformance Ġof Ġthese ĠPC Ġstatic Ġembed d ings Ġon Ġseveral Ġbenchmark Ġtasks . ĠThese Ġtasks Ġcover Ġsemantic Ġsimilarity , Ġanalogy Ġsolving , Ġand Ġconcept Ġcategor ization : ĠSim Lex 999 ĠB IB REF 21 , ĠMEN ĠB IB REF 22 , ĠWS 353 ĠB IB REF 23 , ĠRW ĠB IB REF 24 , ĠSem E val - 2012 ĠB IB REF 25 , ĠGoogle Ġanalogy Ġsolving ĠB IB REF 0 ĠMS R Ġanalogy Ġsolving ĠB IB REF 26 , ĠBL ESS ĠB IB REF 27 Ġand ĠAP ĠB IB REF 28 . ĠWe Ġleave Ġout Ġlayers Ġ3 Ġ- Ġ10 Ġin ĠTable ĠT AB REF 34 Ġbecause Ġtheir Ġperformance Ġis Ġbetween Ġthose Ġof ĠL ayers Ġ2 Ġand Ġ11 . </s> The Ġbest - performing ĠPC Ġstatic Ġembed d ings Ġbelong Ġto Ġthe Ġfirst Ġlayer Ġof ĠB ERT , Ġalthough Ġthose Ġfrom Ġthe Ġother Ġlayers Ġof ĠB ERT Ġand ĠEL Mo Ġalso Ġoutper form ĠGl o Ve Ġand ĠFast Text Ġon Ġmost Ġbenchmarks . ĠFor Ġall Ġthree Ġcontextual izing Ġmodels , ĠPC Ġstatic Ġembed d ings Ġcreated Ġfrom Ġlower Ġlayers Ġare Ġmore Ġeffective Ġthose Ġcreated Ġfrom Ġupper Ġlayers . ĠThose Ġcreated Ġusing ĠG PT - 2 Ġalso Ġperform Ġmarkedly Ġworse Ġthan Ġtheir Ġcounterparts Ġfrom ĠEL Mo Ġand ĠB ERT . ĠGiven Ġthat Ġupper Ġlayers Ġare Ġmuch Ġmore Ġcontext - specific Ġthan Ġlower Ġlayers , Ġand Ġgiven Ġthat ĠG PT - 2 's Ġrepresentations Ġare Ġmore Ġcontext - specific Ġthan ĠEL Mo Ġand ĠB ERT 's Ġ( see ĠFigure ĠFIG REF 24 ), Ġthis Ġsuggests Ġthat Ġthe ĠPCs Ġof Ġhighly Ġcontext - specific Ġrepresentations Ġare Ġless Ġeffective Ġon Ġtraditional Ġbenchmarks . ĠThose Ġderived Ġfrom Ġless Ġcontext - specific Ġrepresentations , Ġsuch Ġas Ġthose Ġfrom ĠLayer Ġ1 Ġof ĠB ERT , Ġare Ġmuch Ġmore Ġeffective . </s> Future ĠWork </s> Our Ġfindings Ġoffer Ġsome Ġnew Ġdirections Ġfor Ġfuture Ġwork . ĠFor Ġone , Ġas Ġnoted Ġearlier Ġin Ġthe Ġpaper , ĠB IB REF 6 Ġfound Ġthat Ġmaking Ġstatic Ġembed d ings Ġmore Ġisot rop ic ĠâĢĵ Ġby Ġsubtract ing Ġtheir Ġmean Ġfrom Ġeach Ġembed ding ĠâĢĵ Ġleads Ġto Ġsurprisingly Ġlarge Ġimprovements Ġin Ġperformance Ġon Ġdownstream Ġtasks . ĠGiven Ġthat Ġisot ropy Ġhas Ġbenefits Ġfor Ġstatic Ġembed d ings , Ġit Ġmay Ġalso Ġhave Ġbenefits Ġfor Ġcontextual ized Ġword Ġrepresentations , Ġalthough Ġthe Ġlatter Ġhave Ġalready Ġyielded Ġsignificant Ġimprovements Ġdespite Ġbeing Ġhighly Ġan is otropic . ĠTherefore , Ġadding Ġan Ġan is ot ropy Ġpenalty Ġto Ġthe Ġlanguage Ġmodelling Ġobjective ĠâĢĵ Ġto Ġencourage Ġthe Ġcontextual ized Ġrepresentations Ġto Ġbe Ġmore Ġisot rop ic ĠâĢĵ Ġmay Ġyield Ġeven Ġbetter Ġresults . </s> Another Ġdirection Ġfor Ġfuture Ġwork Ġis Ġgenerating Ġstatic Ġword Ġrepresentations Ġfrom Ġcontextual ized Ġones . ĠWhile Ġthe Ġlatter Ġoffer Ġsuperior Ġperformance , Ġthere Ġare Ġoften Ġchallenges Ġto Ġdeploying Ġlarge Ġmodels Ġsuch Ġas ĠB ERT Ġin Ġproduction , Ġboth Ġwith Ġrespect Ġto Ġmemory Ġand Ġrun - time . ĠIn Ġcontrast , Ġstatic Ġrepresentations Ġare Ġmuch Ġeasier Ġto Ġdeploy . ĠOur Ġwork Ġin Ġsection Ġ4 . 3 Ġsuggests Ġthat Ġnot Ġonly Ġit Ġis Ġpossible Ġto Ġextract Ġstatic Ġrepresentations Ġfrom Ġcontextual izing Ġmodels , Ġbut Ġthat Ġthese Ġextracted Ġvectors Ġoften Ġperform Ġmuch Ġbetter Ġon Ġa Ġdiverse Ġarray Ġof Ġtasks Ġcompared Ġto Ġtraditional Ġstatic Ġembed d ings Ġsuch Ġas ĠGl o Ve Ġand ĠFast Text . ĠThis Ġmay Ġbe Ġa Ġmeans Ġof Ġextracting Ġsome Ġuse Ġfrom Ġcontextual izing Ġmodels Ġwithout Ġinc urring Ġthe Ġfull Ġcost Ġof Ġusing Ġthem Ġin Ġproduction . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġinvestigated Ġhow Ġcontextual Ġcontextual ized Ġword Ġrepresentations Ġtruly Ġare . ĠFor Ġone , Ġwe Ġfound Ġthat Ġupper Ġlayers Ġof ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 Ġproduce Ġmore Ġcontext - specific Ġrepresentations Ġthan Ġlower Ġlayers . ĠThis Ġincreased Ġcontext - specific ity Ġis Ġalways Ġaccompanied Ġby Ġincreased Ġan is ot ropy . ĠHowever , Ġcontext - specific ity Ġalso Ġmanifests Ġdifferently Ġacross Ġthe Ġthree Ġmodels ; Ġthe Ġan is ot ropy - adjusted Ġsimilarity Ġbetween Ġwords Ġin Ġthe Ġsame Ġsentence Ġis Ġhighest Ġin ĠEL Mo Ġbut Ġalmost Ġnon - existent Ġin ĠG PT - 2 . ĠWe Ġultimately Ġfound Ġthat Ġafter Ġadjusting Ġfor Ġan is ot ropy , Ġon Ġaverage , Ġless Ġthan Ġ5 % Ġof Ġthe Ġvariance Ġin Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġcould Ġbe Ġexplained Ġby Ġa Ġstatic Ġembed ding . ĠThis Ġmeans Ġthat Ġeven Ġin Ġthe Ġbest - case Ġscenario , Ġin Ġall Ġlayers Ġof Ġall Ġmodels , Ġstatic Ġword Ġembed d ings Ġwould Ġbe Ġa Ġpoor Ġreplacement Ġfor Ġcontextual ized Ġones . ĠThese Ġinsights Ġhelp Ġexplain Ġsome Ġof Ġthe Ġremarkable Ġsuccess Ġthat Ġcontextual ized Ġrepresentations Ġhave Ġhad Ġon Ġa Ġdiverse Ġarray Ġof ĠN LP Ġtasks . </s> Acknowled gments </s> We Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġinsightful Ġcomments . ĠWe Ġthank Ġthe ĠNatural ĠSciences Ġand ĠEngineering ĠResearch ĠCouncil Ġof ĠCanada Ġ( NS ERC ) Ġfor Ġtheir Ġfinancial Ġsupport .
Introduction The application of deep learning methods to NLP is made possible by representing words as vectors in a low-dimensional continuous space. Traditionally, these word embeddings were static: each word had a single vector, regardless of context BIBREF0, BIBREF1. This posed several problems, most notably that all senses of a polysemous word had to share the same representation. More recent work, namely deep neural language models such as ELMo BIBREF2 and BERT BIBREF3, have successfully created contextualized word representations, word vectors that are sensitive to the context in which they appear. Replacing static embeddings with contextualized representations has yielded significant improvements on a diverse array of NLP tasks, ranging from question-answering to coreference resolution. The success of contextualized word representations suggests that despite being trained with only a language modelling task, they learn highly transferable and task-agnostic properties of language. In fact, linear probing models trained on frozen contextualized representations can predict linguistic properties of words (e.g., part-of-speech tags) almost as well as state-of-the-art models BIBREF4, BIBREF5. Still, these representations remain poorly understood. For one, just how contextual are these contextualized word representations? Are there infinitely many context-specific representations that BERT and ELMo can assign to each word, or are words essentially assigned one of a finite number of word-sense representations? We answer this question by studying the geometry of the representation space for each layer of ELMo, BERT, and GPT-2. Our analysis yields some surprising findings: In all layers of all three models, the contextualized word representations of all words are not isotropic: they are not uniformly distributed with respect to direction. Instead, they are anisotropic, occupying a narrow cone in the vector space. The anisotropy in GPT-2's last layer is so extreme that two random words will on average have almost perfect cosine similarity! Given that isotropy has both theoretical and empirical benefits for static embeddings BIBREF6, the extent of anisotropy in contextualized representations is surprising. Occurrences of the same word in different contexts have non-identical vector representations. Where vector similarity is defined as cosine similarity, these representations are more dissimilar to each other in upper layers. This suggests that, much like how upper layers of LSTMs produce more task-specific representations BIBREF4, upper layers of contextualizing models produce more context-specific representations. Context-specificity manifests very differently in ELMo, BERT, and GPT-2. In ELMo, representations of words in the same sentence grow more similar to each other as context-specificity increases in upper layers; in BERT, they become more dissimilar to each other in upper layers but are still more similar than randomly sampled words are on average; in GPT-2, however, words in the same sentence are no more similar to each other than two randomly chosen words. After adjusting for the effect of anisotropy, on average, less than 5% of the variance in a word's contextualized representations can be explained by their first principal component. This holds across all layers of all models. This suggests that contextualized representations do not correspond to a finite number of word-sense representations, and even in the best possible scenario, static embeddings would be a poor replacement for contextualized ones. Still, static embeddings created by taking the first principal component of a word's contextualized representations outperform GloVe and FastText embeddings on many word vector benchmarks. These insights help justify why the use of contextualized representations has led to such significant improvements on many NLP tasks. Related Work ::: Static Word Embeddings Skip-gram with negative sampling (SGNS) BIBREF0 and GloVe BIBREF1 are among the best known models for generating static word embeddings. Though they learn embeddings iteratively in practice, it has been proven that in theory, they both implicitly factorize a word-context matrix containing a co-occurrence statistic BIBREF7, BIBREF8. Because they create a single representation for each word, a notable problem with static word embeddings is that all senses of a polysemous word must share a single vector. Related Work ::: Contextualized Word Representations Given the limitations of static word embeddings, recent work has tried to create context-sensitive word representations. ELMo BIBREF2, BERT BIBREF3, and GPT-2 BIBREF9 are deep neural language models that are fine-tuned to create models for a wide range of downstream NLP tasks. Their internal representations of words are called contextualized word representations because they are a function of the entire input sentence. The success of this approach suggests that these representations capture highly transferable and task-agnostic properties of language BIBREF4. ELMo creates contextualized representations of each token by concatenating the internal states of a 2-layer biLSTM trained on a bidirectional language modelling task BIBREF2. In contrast, BERT and GPT-2 are bi-directional and uni-directional transformer-based language models respectively. Each transformer layer of 12-layer BERT (base, cased) and 12-layer GPT-2 creates a contextualized representation of each token by attending to different parts of the input sentence BIBREF3, BIBREF9. BERT – and subsequent iterations on BERT BIBREF10, BIBREF11 – have achieved state-of-the-art performance on various downstream NLP tasks, ranging from question-answering to sentiment analysis. Related Work ::: Probing Tasks Prior analysis of contextualized word representations has largely been restricted to probing tasks BIBREF12, BIBREF5. This involves training linear models to predict syntactic (e.g., part-of-speech tag) and semantic (e.g., word relation) properties of words. Probing models are based on the premise that if a simple linear model can be trained to accurately predict a linguistic property, then the representations implicitly encode this information to begin with. While these analyses have found that contextualized representations encode semantic and syntactic information, they cannot answer how contextual these representations are, and to what extent they can be replaced with static word embeddings, if at all. Our work in this paper is thus markedly different from most dissections of contextualized representations. It is more similar to BIBREF13, which studied the geometry of static word embedding spaces. Approach ::: Contextualizing Models The contextualizing models we study in this paper are ELMo, BERT, and GPT-2. We choose the base cased version of BERT because it is most comparable to GPT-2 with respect to number of layers and dimensionality. The models we work with are all pre-trained on their respective language modelling tasks. Although ELMo, BERT, and GPT-2 have 2, 12, and 12 hidden layers respectively, we also include the input layer of each contextualizing model as its 0th layer. This is because the 0th layer is not contextualized, making it a useful baseline against which to compare the contextualization done by subsequent layers. Approach ::: Data To analyze contextualized word representations, we need input sentences to feed into our pre-trained models. Our input data come from the SemEval Semantic Textual Similarity tasks from years 2012 - 2016 BIBREF14, BIBREF15, BIBREF16, BIBREF17. We use these datasets because they contain sentences in which the same words appear in different contexts. For example, the word `dog' appears in “A panda dog is running on the road.” and “A dog is trying to get bacon off his back.” If a model generated the same representation for `dog' in both these sentences, we could infer that there was no contextualization; conversely, if the two representations were different, we could infer that they were contextualized to some extent. Using these datasets, we map words to the list of sentences they appear in and their index within these sentences. We do not consider words that appear in less than 5 unique contexts in our analysis. Approach ::: Measures of Contextuality We measure how contextual a word representation is using three different metrics: self-similarity, intra-sentence similarity, and maximum explainable variance. Approach ::: Measures of Contextuality ::: Definition 1 Let $w$ be a word that appears in sentences $\lbrace s_1, ..., s_n \rbrace $ at indices $\lbrace i_1, ..., i_n \rbrace $ respectively, such that $w = s_1[i_1] = ... = s_n[i_n]$. Let $f_{\ell }(s,i)$ be a function that maps $s[i]$ to its representation in layer $\ell $ of model $f$. The self similarity of $w$ in layer $\ell $ is where $\cos $ denotes the cosine similarity. In other words, the self-similarity of a word $w$ in layer $\ell $ is the average cosine similarity between its contextualized representations across its $n$ unique contexts. If layer $\ell $ does not contextualize the representations at all, then $\textit {SelfSim}_\ell (w) = 1$ (i.e., the representations are identical across all contexts). The more contextualized the representations are for $w$, the lower we would expect its self-similarity to be. Approach ::: Measures of Contextuality ::: Definition 2 Let $s$ be a sentence that is a sequence $\left< w_1, ..., w_n \right>$ of $n$ words. Let $f_{\ell }(s,i)$ be a function that maps $s[i]$ to its representation in layer $\ell $ of model $f$. The intra-sentence similarity of $s$ in layer $\ell $ is Put more simply, the intra-sentence similarity of a sentence is the average cosine similarity between its word representations and the sentence vector, which is just the mean of those word vectors. This measure captures how context-specificity manifests in the vector space. For example, if both $\textit {IntraSim}_\ell (s)$ and $\textit {SelfSim}_\ell (w)$ are low $\forall \ w \in s$, then the model contextualizes words in that layer by giving each one a context-specific representation that is still distinct from all other word representations in the sentence. If $\textit {IntraSim}_\ell (s)$ is high but $\textit {SelfSim}_\ell (w)$ is low, this suggests a less nuanced contextualization, where words in a sentence are contextualized simply by making their representations converge in vector space. Approach ::: Measures of Contextuality ::: Definition 3 Let $w$ be a word that appears in sentences $\lbrace s_1, ..., s_n \rbrace $ at indices $\lbrace i_1, ..., i_n \rbrace $ respectively, such that $w = s_1[i_1] = ... = s_n[i_n]$. Let $f_{\ell }(s,i)$ be a function that maps $s[i]$ to its representation in layer $\ell $ of model $f$. Where $[ f_{\ell }(s_1, i_1) ... f_{\ell }(s_n, i_n) ]$ is the occurrence matrix of $w$ and $\sigma _1 ... \sigma _m$ are the first $m$ singular values of this matrix, the maximum explainable variance is $\textit {MEV}_\ell (w)$ is the proportion of variance in $w$'s contextualized representations for a given layer that can be explained by their first principal component. It gives us an upper bound on how well a static embedding could replace a word's contextualized representations. The closer $\textit {MEV}_\ell (w)$ is to 0, the poorer a replacement a static embedding would be; if $\textit {MEV}_\ell (w) = 1$, then a static embedding would be a perfect replacement for the contextualized representations. Approach ::: Adjusting for Anisotropy It is important to consider isotropy (or the lack thereof) when discussing contextuality. For example, if word vectors were perfectly isotropic (i.e., directionally uniform), then $\textit {SelfSim}_\ell (w) = 0.95$ would suggest that $w$'s representations were poorly contextualized. However, consider the scenario where word vectors are so anisotropic that any two words have on average a cosine similarity of 0.99. Then $\textit {SelfSim}_\ell (w) = 0.95$ would actually suggest the opposite – that $w$'s representations were well contextualized. This is because representations of $w$ in different contexts would on average be more dissimilar to each other than two randomly chosen words. To adjust for the effect of anisotropy, we use three anisotropic baselines, one for each of our contextuality measures. For self-similarity and intra-sentence similarity, the baseline is the average cosine similarity between the representations of uniformly randomly sampled words from different contexts. The more anisotropic the word representations are in a given layer, the closer this baseline is to 1. For maximum explainable variance (MEV), the baseline is the proportion of variance in uniformly randomly sampled word representations that is explained by their first principal component. The more anisotropic the representations in a given layer, the closer this baseline is to 1: even for a random assortment of words, the principal component would be able to explain a large proportion of the variance. Since contextuality measures are calculated for each layer of a contextualizing model, we calculate separate baselines for each layer as well. We then subtract from each measure its respective baseline to get the anisotropy-adjusted contexuality measure. For example, the anisotropy-adjusted self-similarity is where $\mathcal {O}$ is the set of all word occurrences and $f_{\ell }(\cdot )$ maps a word occurrence to its representation in layer $\ell $ of model $f$. Unless otherwise stated, references to contextuality measures in the rest of the paper refer to the anisotropy-adjusted measures, where both the raw measure and baseline are estimated with 1K uniformly randomly sampled word representations. Findings ::: (An)Isotropy ::: Contextualized representations are anisotropic in all non-input layers. If word representations from a particular layer were isotropic (i.e., directionally uniform), then the average cosine similarity between uniformly randomly sampled words would be 0 BIBREF18. The closer this average is to 1, the more anisotropic the representations. The geometric interpretation of anisotropy is that the word representations all occupy a narrow cone in the vector space rather than being uniform in all directions; the greater the anisotropy, the narrower this cone BIBREF13. As seen in Figure FIGREF20, this implies that in almost all layers of BERT, ELMo and GPT-2, the representations of all words occupy a narrow cone in the vector space. The only exception is ELMo's input layer, which produces static character-level embeddings without using contextual or even positional information BIBREF2. It should be noted that not all static embeddings are necessarily isotropic, however; BIBREF13 found that skipgram embeddings, which are also static, are not isotropic. Findings ::: (An)Isotropy ::: Contextualized representations are generally more anisotropic in higher layers. As seen in Figure FIGREF20, for GPT-2, the average cosine similarity between uniformly randomly words is roughly 0.6 in layers 2 through 8 but increases exponentially from layers 8 through 12. In fact, word representations in GPT-2's last layer are so anisotropic that any two words have on average an almost perfect cosine similarity! This pattern holds for BERT and ELMo as well, though there are exceptions: for example, the anisotropy in BERT's penultimate layer is much higher than in its final layer. Isotropy has both theoretical and empirical benefits for static word embeddings. In theory, it allows for stronger “self-normalization” during training BIBREF18, and in practice, subtracting the mean vector from static embeddings leads to improvements on several downstream NLP tasks BIBREF6. Thus the extreme degree of anisotropy seen in contextualized word representations – particularly in higher layers – is surprising. As seen in Figure FIGREF20, for all three models, the contextualized hidden layer representations are almost all more anisotropic than the input layer representations, which do not incorporate context. This suggests that high anisotropy is inherent to, or least a by-product of, the process of contextualization. Findings ::: Context-Specificity ::: Contextualized word representations are more context-specific in higher layers. Recall from Definition 1 that the self-similarity of a word, in a given layer of a given model, is the average cosine similarity between its representations in different contexts, adjusted for anisotropy. If the self-similarity is 1, then the representations are not context-specific at all; if the self-similarity is 0, that the representations are maximally context-specific. In Figure FIGREF24, we plot the average self-similarity of uniformly randomly sampled words in each layer of BERT, ELMo, and GPT-2. For example, the self-similarity is 1.0 in ELMo's input layer because representations in that layer are static character-level embeddings. In all three models, the higher the layer, the lower the self-similarity is on average. In other words, the higher the layer, the more context-specific the contextualized representations. This finding makes intuitive sense. In image classification models, lower layers recognize more generic features such as edges while upper layers recognize more class-specific features BIBREF19. Similarly, upper layers of LSTMs trained on NLP tasks learn more task-specific representations BIBREF4. Therefore, it follows that upper layers of neural language models learn more context-specific representations, so as to predict the next word for a given context more accurately. Of all three models, representations in GPT-2 are the most context-specific, with those in GPT-2's last layer being almost maximally context-specific. Findings ::: Context-Specificity ::: Stopwords (e.g., `the', `of', `to') have among the most context-specific representations. Across all layers, stopwords have among the lowest self-similarity of all words, implying that their contextualized representations are among the most context-specific. For example, the words with the lowest average self-similarity across ELMo's layers are `and', `of', `'s', `the', and `to'. This is relatively surprising, given that these words are not polysemous. This finding suggests that the variety of contexts a word appears in, rather than its inherent polysemy, is what drives variation in its contextualized representations. This answers one of the questions we posed in the introduction: ELMo, BERT, and GPT-2 are not simply assigning one of a finite number of word-sense representations to each word; otherwise, there would not be so much variation in the representations of words with so few word senses. Findings ::: Context-Specificity ::: Context-specificity manifests very differently in ELMo, BERT, and GPT-2. As noted earlier, contextualized representations are more context-specific in upper layers of ELMo, BERT, and GPT-2. However, how does this increased context-specificity manifest in the vector space? Do word representations in the same sentence converge to a single point, or do they remain distinct from one another while still being distinct from their representations in other contexts? To answer this question, we can measure a sentence's intra-sentence similarity. Recall from Definition 2 that the intra-sentence similarity of a sentence, in a given layer of a given model, is the average cosine similarity between each of its word representations and their mean, adjusted for anisotropy. In Figure FIGREF25, we plot the average intra-sentence similarity of 500 uniformly randomly sampled sentences. Findings ::: Context-Specificity ::: In ELMo, words in the same sentence are more similar to one another in upper layers. As word representations in a sentence become more context-specific in upper layers, the intra-sentence similarity also rises. This suggests that, in practice, ELMo ends up extending the intuition behind Firth's BIBREF20 distributional hypothesis to the sentence level: that because words in the same sentence share the same context, their contextualized representations should also be similar. Findings ::: Context-Specificity ::: In BERT, words in the same sentence are more dissimilar to one another in upper layers. As word representations in a sentence become more context-specific in upper layers, they drift away from one another, although there are exceptions (see layer 12 in Figure FIGREF25). However, in all layers, the average similarity between words in the same sentence is still greater than the average similarity between randomly chosen words (i.e., the anisotropy baseline). This suggests a more nuanced contextualization than in ELMo, with BERT recognizing that although the surrounding sentence informs a word's meaning, two words in the same sentence do not necessarily have a similar meaning because they share the same context. Findings ::: Context-Specificity ::: In GPT-2, word representations in the same sentence are no more similar to each other than randomly sampled words. On average, the unadjusted intra-sentence similarity is roughly the same as the anisotropic baseline, so as seen in Figure FIGREF25, the anisotropy-adjusted intra-sentence similarity is close to 0 in most layers of GPT-2. In fact, the intra-sentence similarity is highest in the input layer, which does not contextualize words at all. This is in contrast to ELMo and BERT, where the average intra-sentence similarity is above 0.20 for all but one layer. As noted earlier when discussing BERT, this behavior still makes intuitive sense: two words in the same sentence do not necessarily have a similar meaning simply because they share the same context. The success of GPT-2 suggests that unlike anisotropy, which accompanies context-specificity in all three models, a high intra-sentence similarity is not inherent to contextualization. Words in the same sentence can have highly contextualized representations without those representations being any more similar to each other than two random word representations. It is unclear, however, whether these differences in intra-sentence similarity can be traced back to differences in model architecture; we leave this question as future work. Findings ::: Static vs. Contextualized ::: On average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding. Recall from Definition 3 that the maximum explainable variance (MEV) of a word, for a given layer of a given model, is the proportion of variance in its contextualized representations that can be explained by their first principal component. This gives us an upper bound on how well a static embedding could replace a word's contextualized representations. Because contextualized representations are anisotropic (see section SECREF21), much of the variation across all words can be explained by a single vector. We adjust for anisotropy by calculating the proportion of variance explained by the first principal component of uniformly randomly sampled word representations and subtracting this proportion from the raw MEV. In Figure FIGREF29, we plot the average anisotropy-adjusted MEV across uniformly randomly sampled words. In no layer of ELMo, BERT, or GPT-2 can more than 5% of the variance in a word's contextualized representations be explained by a static embedding, on average. Though not visible in Figure FIGREF29, the raw MEV of many words is actually below the anisotropy baseline: i.e., a greater proportion of the variance across all words can be explained by a single vector than can the variance across all representations of a single word. Note that the 5% threshold represents the best-case scenario, and there is no theoretical guarantee that a word vector obtained using GloVe, for example, would be similar to the static embedding that maximizes MEV. This suggests that contextualizing models are not simply assigning one of a finite number of word-sense representations to each word – otherwise, the proportion of variance explained would be much higher. Even the average raw MEV is below 5% for all layers of ELMo and BERT; only for GPT-2 is the raw MEV non-negligible, being around 30% on average for layers 2 to 11 due to extremely high anisotropy. Findings ::: Static vs. Contextualized ::: Principal components of contextualized representations in lower layers outperform GloVe and FastText on many benchmarks. As noted earlier, we can create static embeddings for each word by taking the first principal component (PC) of its contextualized representations in a given layer. In Table TABREF34, we plot the performance of these PC static embeddings on several benchmark tasks. These tasks cover semantic similarity, analogy solving, and concept categorization: SimLex999 BIBREF21, MEN BIBREF22, WS353 BIBREF23, RW BIBREF24, SemEval-2012 BIBREF25, Google analogy solving BIBREF0 MSR analogy solving BIBREF26, BLESS BIBREF27 and AP BIBREF28. We leave out layers 3 - 10 in Table TABREF34 because their performance is between those of Layers 2 and 11. The best-performing PC static embeddings belong to the first layer of BERT, although those from the other layers of BERT and ELMo also outperform GloVe and FastText on most benchmarks. For all three contextualizing models, PC static embeddings created from lower layers are more effective those created from upper layers. Those created using GPT-2 also perform markedly worse than their counterparts from ELMo and BERT. Given that upper layers are much more context-specific than lower layers, and given that GPT-2's representations are more context-specific than ELMo and BERT's (see Figure FIGREF24), this suggests that the PCs of highly context-specific representations are less effective on traditional benchmarks. Those derived from less context-specific representations, such as those from Layer 1 of BERT, are much more effective. Future Work Our findings offer some new directions for future work. For one, as noted earlier in the paper, BIBREF6 found that making static embeddings more isotropic – by subtracting their mean from each embedding – leads to surprisingly large improvements in performance on downstream tasks. Given that isotropy has benefits for static embeddings, it may also have benefits for contextualized word representations, although the latter have already yielded significant improvements despite being highly anisotropic. Therefore, adding an anisotropy penalty to the language modelling objective – to encourage the contextualized representations to be more isotropic – may yield even better results. Another direction for future work is generating static word representations from contextualized ones. While the latter offer superior performance, there are often challenges to deploying large models such as BERT in production, both with respect to memory and run-time. In contrast, static representations are much easier to deploy. Our work in section 4.3 suggests that not only it is possible to extract static representations from contextualizing models, but that these extracted vectors often perform much better on a diverse array of tasks compared to traditional static embeddings such as GloVe and FastText. This may be a means of extracting some use from contextualizing models without incurring the full cost of using them in production. Conclusion In this paper, we investigated how contextual contextualized word representations truly are. For one, we found that upper layers of ELMo, BERT, and GPT-2 produce more context-specific representations than lower layers. This increased context-specificity is always accompanied by increased anisotropy. However, context-specificity also manifests differently across the three models; the anisotropy-adjusted similarity between words in the same sentence is highest in ELMo but almost non-existent in GPT-2. We ultimately found that after adjusting for anisotropy, on average, less than 5% of the variance in a word's contextualized representations could be explained by a static embedding. This means that even in the best-case scenario, in all layers of all models, static word embeddings would be a poor replacement for contextualized ones. These insights help explain some of the remarkable success that contextualized representations have had on a diverse array of NLP tasks. Acknowledgments We thank the anonymous reviewers for their insightful comments. We thank the Natural Sciences and Engineering Research Council of Canada (NSERC) for their financial support.
How do they calculate a static embedding for each word?
891c2001d6baaaf0da4e65b647402acac621a7d2
How Ġdo Ġthey Ġcalculate Ġa Ġstatic Ġembed ding Ġfor Ġeach Ġword ?
They Ġuse Ġthe Ġfirst Ġprincipal Ġcomponent Ġof Ġa Ġword 's Ġcontextual ized Ġrepresentation Ġin Ġa Ġgiven Ġlayer Ġas Ġits Ġstatic Ġembed ding .
<s> How Ġdo Ġthey Ġcalculate Ġa Ġstatic Ġembed ding Ġfor Ġeach Ġword ? </s> Introduction </s> The Ġapplication Ġof Ġdeep Ġlearning Ġmethods Ġto ĠN LP Ġis Ġmade Ġpossible Ġby Ġrepresenting Ġwords Ġas Ġvectors Ġin Ġa Ġlow - dimensional Ġcontinuous Ġspace . ĠTrad itionally , Ġthese Ġword Ġembed d ings Ġwere Ġstatic : Ġeach Ġword Ġhad Ġa Ġsingle Ġvector , Ġregardless Ġof Ġcontext ĠB IB REF 0 , ĠB IB REF 1 . ĠThis Ġposed Ġseveral Ġproblems , Ġmost Ġnotably Ġthat Ġall Ġsenses Ġof Ġa Ġpoly sem ous Ġword Ġhad Ġto Ġshare Ġthe Ġsame Ġrepresentation . ĠMore Ġrecent Ġwork , Ġnamely Ġdeep Ġneural Ġlanguage Ġmodels Ġsuch Ġas ĠEL Mo ĠB IB REF 2 Ġand ĠB ERT ĠB IB REF 3 , Ġhave Ġsuccessfully Ġcreated Ġcontextual ized Ġword Ġrepresentations , Ġword Ġvectors Ġthat Ġare Ġsensitive Ġto Ġthe Ġcontext Ġin Ġwhich Ġthey Ġappear . ĠRepl acing Ġstatic Ġembed d ings Ġwith Ġcontextual ized Ġrepresentations Ġhas Ġyielded Ġsignificant Ġimprovements Ġon Ġa Ġdiverse Ġarray Ġof ĠN LP Ġtasks , Ġranging Ġfrom Ġquestion - ans w ering Ġto Ġcore ference Ġresolution . </s> The Ġsuccess Ġof Ġcontextual ized Ġword Ġrepresentations Ġsuggests Ġthat Ġdespite Ġbeing Ġtrained Ġwith Ġonly Ġa Ġlanguage Ġmodelling Ġtask , Ġthey Ġlearn Ġhighly Ġtransfer able Ġand Ġtask - agn ostic Ġproperties Ġof Ġlanguage . ĠIn Ġfact , Ġlinear Ġprobing Ġmodels Ġtrained Ġon Ġfrozen Ġcontextual ized Ġrepresentations Ġcan Ġpredict Ġlinguistic Ġproperties Ġof Ġwords Ġ( e . g ., Ġpart - of - speech Ġtags ) Ġalmost Ġas Ġwell Ġas Ġstate - of - the - art Ġmodels ĠB IB REF 4 , ĠB IB REF 5 . ĠStill , Ġthese Ġrepresentations Ġremain Ġpoorly Ġunderstood . ĠFor Ġone , Ġjust Ġhow Ġcontextual Ġare Ġthese Ġcontextual ized Ġword Ġrepresentations ? ĠAre Ġthere Ġinfinitely Ġmany Ġcontext - specific Ġrepresentations Ġthat ĠB ERT Ġand ĠEL Mo Ġcan Ġassign Ġto Ġeach Ġword , Ġor Ġare Ġwords Ġessentially Ġassigned Ġone Ġof Ġa Ġfinite Ġnumber Ġof Ġword - sense Ġrepresentations ? </s> We Ġanswer Ġthis Ġquestion Ġby Ġstudying Ġthe Ġgeometry Ġof Ġthe Ġrepresentation Ġspace Ġfor Ġeach Ġlayer Ġof ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . ĠOur Ġanalysis Ġyields Ġsome Ġsurprising Ġfindings : </s> In Ġall Ġlayers Ġof Ġall Ġthree Ġmodels , Ġthe Ġcontextual ized Ġword Ġrepresentations Ġof Ġall Ġwords Ġare Ġnot Ġisot rop ic : Ġthey Ġare Ġnot Ġuniformly Ġdistributed Ġwith Ġrespect Ġto Ġdirection . ĠInstead , Ġthey Ġare Ġan is otropic , Ġoccupying Ġa Ġnarrow Ġcone Ġin Ġthe Ġvector Ġspace . ĠThe Ġan is ot ropy Ġin ĠG PT - 2 's Ġlast Ġlayer Ġis Ġso Ġextreme Ġthat Ġtwo Ġrandom Ġwords Ġwill Ġon Ġaverage Ġhave Ġalmost Ġperfect Ġcos ine Ġsimilarity ! ĠGiven Ġthat Ġisot ropy Ġhas Ġboth Ġtheoretical Ġand Ġempirical Ġbenefits Ġfor Ġstatic Ġembed d ings ĠB IB REF 6 , Ġthe Ġextent Ġof Ġan is ot ropy Ġin Ġcontextual ized Ġrepresentations Ġis Ġsurprising . </s> Occ ur rences Ġof Ġthe Ġsame Ġword Ġin Ġdifferent Ġcontexts Ġhave Ġnon - ident ical Ġvector Ġrepresentations . ĠWhere Ġvector Ġsimilarity Ġis Ġdefined Ġas Ġcos ine Ġsimilarity , Ġthese Ġrepresentations Ġare Ġmore Ġdiss imilar Ġto Ġeach Ġother Ġin Ġupper Ġlayers . ĠThis Ġsuggests Ġthat , Ġmuch Ġlike Ġhow Ġupper Ġlayers Ġof ĠL ST Ms Ġproduce Ġmore Ġtask - specific Ġrepresentations ĠB IB REF 4 , Ġupper Ġlayers Ġof Ġcontextual izing Ġmodels Ġproduce Ġmore Ġcontext - specific Ġrepresentations . </s> Context - specific ity Ġmanifests Ġvery Ġdifferently Ġin ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . ĠIn ĠEL Mo , Ġrepresentations Ġof Ġwords Ġin Ġthe Ġsame Ġsentence Ġgrow Ġmore Ġsimilar Ġto Ġeach Ġother Ġas Ġcontext - specific ity Ġincreases Ġin Ġupper Ġlayers ; Ġin ĠB ERT , Ġthey Ġbecome Ġmore Ġdiss imilar Ġto Ġeach Ġother Ġin Ġupper Ġlayers Ġbut Ġare Ġstill Ġmore Ġsimilar Ġthan Ġrandomly Ġsampled Ġwords Ġare Ġon Ġaverage ; Ġin ĠG PT - 2 , Ġhowever , Ġwords Ġin Ġthe Ġsame Ġsentence Ġare Ġno Ġmore Ġsimilar Ġto Ġeach Ġother Ġthan Ġtwo Ġrandomly Ġchosen Ġwords . </s> After Ġadjusting Ġfor Ġthe Ġeffect Ġof Ġan is ot ropy , Ġon Ġaverage , Ġless Ġthan Ġ5 % Ġof Ġthe Ġvariance Ġin Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġcan Ġbe Ġexplained Ġby Ġtheir Ġfirst Ġprincipal Ġcomponent . ĠThis Ġholds Ġacross Ġall Ġlayers Ġof Ġall Ġmodels . ĠThis Ġsuggests Ġthat Ġcontextual ized Ġrepresentations Ġdo Ġnot Ġcorrespond Ġto Ġa Ġfinite Ġnumber Ġof Ġword - sense Ġrepresentations , Ġand Ġeven Ġin Ġthe Ġbest Ġpossible Ġscenario , Ġstatic Ġembed d ings Ġwould Ġbe Ġa Ġpoor Ġreplacement Ġfor Ġcontextual ized Ġones . ĠStill , Ġstatic Ġembed d ings Ġcreated Ġby Ġtaking Ġthe Ġfirst Ġprincipal Ġcomponent Ġof Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġoutper form ĠGl o Ve Ġand ĠFast Text Ġembed d ings Ġon Ġmany Ġword Ġvector Ġbenchmarks . </s> These Ġinsights Ġhelp Ġjustify Ġwhy Ġthe Ġuse Ġof Ġcontextual ized Ġrepresentations Ġhas Ġled Ġto Ġsuch Ġsignificant Ġimprovements Ġon Ġmany ĠN LP Ġtasks . </s> Related ĠWork Ġ: :: ĠStatic ĠWord ĠEmb edd ings </s> Skip - gram Ġwith Ġnegative Ġsampling Ġ( SG NS ) ĠB IB REF 0 Ġand ĠGl o Ve ĠB IB REF 1 Ġare Ġamong Ġthe Ġbest Ġknown Ġmodels Ġfor Ġgenerating Ġstatic Ġword Ġembed d ings . ĠThough Ġthey Ġlearn Ġembed d ings Ġiter atively Ġin Ġpractice , Ġit Ġhas Ġbeen Ġproven Ġthat Ġin Ġtheory , Ġthey Ġboth Ġimplicitly Ġfactor ize Ġa Ġword - context Ġmatrix Ġcontaining Ġa Ġco - occ urrence Ġstatistic ĠB IB REF 7 , ĠB IB REF 8 . ĠBecause Ġthey Ġcreate Ġa Ġsingle Ġrepresentation Ġfor Ġeach Ġword , Ġa Ġnotable Ġproblem Ġwith Ġstatic Ġword Ġembed d ings Ġis Ġthat Ġall Ġsenses Ġof Ġa Ġpoly sem ous Ġword Ġmust Ġshare Ġa Ġsingle Ġvector . </s> Related ĠWork Ġ: :: ĠContext ual ized ĠWord ĠRepresent ations </s> Given Ġthe Ġlimitations Ġof Ġstatic Ġword Ġembed d ings , Ġrecent Ġwork Ġhas Ġtried Ġto Ġcreate Ġcontext - sensitive Ġword Ġrepresentations . ĠEL Mo ĠB IB REF 2 , ĠB ERT ĠB IB REF 3 , Ġand ĠG PT - 2 ĠB IB REF 9 Ġare Ġdeep Ġneural Ġlanguage Ġmodels Ġthat Ġare Ġfine - tun ed Ġto Ġcreate Ġmodels Ġfor Ġa Ġwide Ġrange Ġof Ġdownstream ĠN LP Ġtasks . ĠTheir Ġinternal Ġrepresentations Ġof Ġwords Ġare Ġcalled Ġcontextual ized Ġword Ġrepresentations Ġbecause Ġthey Ġare Ġa Ġfunction Ġof Ġthe Ġentire Ġinput Ġsentence . ĠThe Ġsuccess Ġof Ġthis Ġapproach Ġsuggests Ġthat Ġthese Ġrepresentations Ġcapture Ġhighly Ġtransfer able Ġand Ġtask - agn ostic Ġproperties Ġof Ġlanguage ĠB IB REF 4 . </s> EL Mo Ġcreates Ġcontextual ized Ġrepresentations Ġof Ġeach Ġtoken Ġby Ġconc aten ating Ġthe Ġinternal Ġstates Ġof Ġa Ġ2 - layer Ġbi L ST M Ġtrained Ġon Ġa Ġbid irection al Ġlanguage Ġmodelling Ġtask ĠB IB REF 2 . ĠIn Ġcontrast , ĠB ERT Ġand ĠG PT - 2 Ġare Ġbi - direction al Ġand Ġun i - direction al Ġtransformer - based Ġlanguage Ġmodels Ġrespectively . ĠEach Ġtransformer Ġlayer Ġof Ġ12 - layer ĠB ERT Ġ( base , Ġc ased ) Ġand Ġ12 - layer ĠG PT - 2 Ġcreates Ġa Ġcontextual ized Ġrepresentation Ġof Ġeach Ġtoken Ġby Ġattending Ġto Ġdifferent Ġparts Ġof Ġthe Ġinput Ġsentence ĠB IB REF 3 , ĠB IB REF 9 . ĠB ERT ĠâĢĵ Ġand Ġsubsequent Ġiterations Ġon ĠB ERT ĠB IB REF 10 , ĠB IB REF 11 ĠâĢĵ Ġhave Ġachieved Ġstate - of - the - art Ġperformance Ġon Ġvarious Ġdownstream ĠN LP Ġtasks , Ġranging Ġfrom Ġquestion - ans w ering Ġto Ġsentiment Ġanalysis . </s> Related ĠWork Ġ: :: ĠPro bing ĠT asks </s> Prior Ġanalysis Ġof Ġcontextual ized Ġword Ġrepresentations Ġhas Ġlargely Ġbeen Ġrestricted Ġto Ġprobing Ġtasks ĠB IB REF 12 , ĠB IB REF 5 . ĠThis Ġinvolves Ġtraining Ġlinear Ġmodels Ġto Ġpredict Ġsynt actic Ġ( e . g ., Ġpart - of - speech Ġtag ) Ġand Ġsemantic Ġ( e . g ., Ġword Ġrelation ) Ġproperties Ġof Ġwords . ĠPro bing Ġmodels Ġare Ġbased Ġon Ġthe Ġpremise Ġthat Ġif Ġa Ġsimple Ġlinear Ġmodel Ġcan Ġbe Ġtrained Ġto Ġaccurately Ġpredict Ġa Ġlinguistic Ġproperty , Ġthen Ġthe Ġrepresentations Ġimplicitly Ġencode Ġthis Ġinformation Ġto Ġbegin Ġwith . ĠWhile Ġthese Ġanalyses Ġhave Ġfound Ġthat Ġcontextual ized Ġrepresentations Ġencode Ġsemantic Ġand Ġsynt actic Ġinformation , Ġthey Ġcannot Ġanswer Ġhow Ġcontextual Ġthese Ġrepresentations Ġare , Ġand Ġto Ġwhat Ġextent Ġthey Ġcan Ġbe Ġreplaced Ġwith Ġstatic Ġword Ġembed d ings , Ġif Ġat Ġall . ĠOur Ġwork Ġin Ġthis Ġpaper Ġis Ġthus Ġmarkedly Ġdifferent Ġfrom Ġmost Ġdisse ctions Ġof Ġcontextual ized Ġrepresentations . ĠIt Ġis Ġmore Ġsimilar Ġto ĠB IB REF 13 , Ġwhich Ġstudied Ġthe Ġgeometry Ġof Ġstatic Ġword Ġembed ding Ġspaces . </s> App roach Ġ: :: ĠContext ual izing ĠModels </s> The Ġcontextual izing Ġmodels Ġwe Ġstudy Ġin Ġthis Ġpaper Ġare ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . ĠWe Ġchoose Ġthe Ġbase Ġc ased Ġversion Ġof ĠB ERT Ġbecause Ġit Ġis Ġmost Ġcomparable Ġto ĠG PT - 2 Ġwith Ġrespect Ġto Ġnumber Ġof Ġlayers Ġand Ġdimension ality . ĠThe Ġmodels Ġwe Ġwork Ġwith Ġare Ġall Ġpre - trained Ġon Ġtheir Ġrespective Ġlanguage Ġmodelling Ġtasks . ĠAlthough ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 Ġhave Ġ2 , Ġ12 , Ġand Ġ12 Ġhidden Ġlayers Ġrespectively , Ġwe Ġalso Ġinclude Ġthe Ġinput Ġlayer Ġof Ġeach Ġcontextual izing Ġmodel Ġas Ġits Ġ0 th Ġlayer . ĠThis Ġis Ġbecause Ġthe Ġ0 th Ġlayer Ġis Ġnot Ġcontextual ized , Ġmaking Ġit Ġa Ġuseful Ġbaseline Ġagainst Ġwhich Ġto Ġcompare Ġthe Ġcontextual ization Ġdone Ġby Ġsubsequent Ġlayers . </s> App roach Ġ: :: ĠData </s> To Ġanalyze Ġcontextual ized Ġword Ġrepresentations , Ġwe Ġneed Ġinput Ġsentences Ġto Ġfeed Ġinto Ġour Ġpre - trained Ġmodels . ĠOur Ġinput Ġdata Ġcome Ġfrom Ġthe ĠSem E val ĠSem antic ĠText ual ĠSimilar ity Ġtasks Ġfrom Ġyears Ġ2012 Ġ- Ġ2016 ĠB IB REF 14 , ĠB IB REF 15 , ĠB IB REF 16 , ĠB IB REF 17 . ĠWe Ġuse Ġthese Ġdatasets Ġbecause Ġthey Ġcontain Ġsentences Ġin Ġwhich Ġthe Ġsame Ġwords Ġappear Ġin Ġdifferent Ġcontexts . ĠFor Ġexample , Ġthe Ġword Ġ` dog ' Ġappears Ġin ĠâĢ ľ A Ġp anda Ġdog Ġis Ġrunning Ġon Ġthe Ġroad . âĢ Ŀ Ġand ĠâĢ ľ A Ġdog Ġis Ġtrying Ġto Ġget Ġbacon Ġoff Ġhis Ġback . âĢ Ŀ ĠIf Ġa Ġmodel Ġgenerated Ġthe Ġsame Ġrepresentation Ġfor Ġ` dog ' Ġin Ġboth Ġthese Ġsentences , Ġwe Ġcould Ġinfer Ġthat Ġthere Ġwas Ġno Ġcontextual ization ; Ġcon versely , Ġif Ġthe Ġtwo Ġrepresentations Ġwere Ġdifferent , Ġwe Ġcould Ġinfer Ġthat Ġthey Ġwere Ġcontextual ized Ġto Ġsome Ġextent . ĠUsing Ġthese Ġdatasets , Ġwe Ġmap Ġwords Ġto Ġthe Ġlist Ġof Ġsentences Ġthey Ġappear Ġin Ġand Ġtheir Ġindex Ġwithin Ġthese Ġsentences . ĠWe Ġdo Ġnot Ġconsider Ġwords Ġthat Ġappear Ġin Ġless Ġthan Ġ5 Ġunique Ġcontexts Ġin Ġour Ġanalysis . </s> App roach Ġ: :: ĠMeasures Ġof ĠContext uality </s> We Ġmeasure Ġhow Ġcontextual Ġa Ġword Ġrepresentation Ġis Ġusing Ġthree Ġdifferent Ġmetrics : Ġself - similar ity , Ġintra - sent ence Ġsimilarity , Ġand Ġmaximum Ġexplain able Ġvariance . </s> App roach Ġ: :: ĠMeasures Ġof ĠContext uality Ġ: :: ĠDefinition Ġ1 </s> Let Ġ$ w $ Ġbe Ġa Ġword Ġthat Ġappears Ġin Ġsentences Ġ$\ l brace Ġs _ 1 , Ġ... , Ġs _ n Ġ\ r brace Ġ$ Ġat Ġindices Ġ$\ l brace Ġi _ 1 , Ġ... , Ġi _ n Ġ\ r brace Ġ$ Ġrespectively , Ġsuch Ġthat Ġ$ w Ġ= Ġs _ 1 [ i _ 1 ] Ġ= Ġ... Ġ= Ġs _ n [ i _ n ] $. ĠLet Ġ$ f _{ \ ell Ġ} ( s , i ) $ Ġbe Ġa Ġfunction Ġthat Ġmaps Ġ$ s [ i ] $ Ġto Ġits Ġrepresentation Ġin Ġlayer Ġ$\ ell Ġ$ Ġof Ġmodel Ġ$ f $. ĠThe Ġself Ġsimilarity Ġof Ġ$ w $ Ġin Ġlayer Ġ$\ ell Ġ$ Ġis </s> where Ġ$\ cos Ġ$ Ġdenotes Ġthe Ġcos ine Ġsimilarity . ĠIn Ġother Ġwords , Ġthe Ġself - similar ity Ġof Ġa Ġword Ġ$ w $ Ġin Ġlayer Ġ$\ ell Ġ$ Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġits Ġcontextual ized Ġrepresentations Ġacross Ġits Ġ$ n $ Ġunique Ġcontexts . ĠIf Ġlayer Ġ$\ ell Ġ$ Ġdoes Ġnot Ġcontextual ize Ġthe Ġrepresentations Ġat Ġall , Ġthen Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) Ġ= Ġ1 $ Ġ( i . e ., Ġthe Ġrepresentations Ġare Ġidentical Ġacross Ġall Ġcontexts ). ĠThe Ġmore Ġcontextual ized Ġthe Ġrepresentations Ġare Ġfor Ġ$ w $, Ġthe Ġlower Ġwe Ġwould Ġexpect Ġits Ġself - similar ity Ġto Ġbe . </s> App roach Ġ: :: ĠMeasures Ġof ĠContext uality Ġ: :: ĠDefinition Ġ2 </s> Let Ġ$ s $ Ġbe Ġa Ġsentence Ġthat Ġis Ġa Ġsequence Ġ$\ left < Ġw _ 1 , Ġ... , Ġw _ n Ġ\ right > $ Ġof Ġ$ n $ Ġwords . ĠLet Ġ$ f _{ \ ell Ġ} ( s , i ) $ Ġbe Ġa Ġfunction Ġthat Ġmaps Ġ$ s [ i ] $ Ġto Ġits Ġrepresentation Ġin Ġlayer Ġ$\ ell Ġ$ Ġof Ġmodel Ġ$ f $. ĠThe Ġintra - sent ence Ġsimilarity Ġof Ġ$ s $ Ġin Ġlayer Ġ$\ ell Ġ$ Ġis </s> Put Ġmore Ġsimply , Ġthe Ġintra - sent ence Ġsimilarity Ġof Ġa Ġsentence Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġits Ġword Ġrepresentations Ġand Ġthe Ġsentence Ġvector , Ġwhich Ġis Ġjust Ġthe Ġmean Ġof Ġthose Ġword Ġvectors . ĠThis Ġmeasure Ġcaptures Ġhow Ġcontext - specific ity Ġmanifests Ġin Ġthe Ġvector Ġspace . ĠFor Ġexample , Ġif Ġboth Ġ$\ text it Ġ{ Int ra Sim } _ \ ell Ġ( s ) $ Ġand Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) $ Ġare Ġlow Ġ$\ for all Ġ\ Ġw Ġ\ in Ġs $, Ġthen Ġthe Ġmodel Ġcontextual izes Ġwords Ġin Ġthat Ġlayer Ġby Ġgiving Ġeach Ġone Ġa Ġcontext - specific Ġrepresentation Ġthat Ġis Ġstill Ġdistinct Ġfrom Ġall Ġother Ġword Ġrepresentations Ġin Ġthe Ġsentence . ĠIf Ġ$\ text it Ġ{ Int ra Sim } _ \ ell Ġ( s ) $ Ġis Ġhigh Ġbut Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) $ Ġis Ġlow , Ġthis Ġsuggests Ġa Ġless Ġnuanced Ġcontextual ization , Ġwhere Ġwords Ġin Ġa Ġsentence Ġare Ġcontextual ized Ġsimply Ġby Ġmaking Ġtheir Ġrepresentations Ġconverge Ġin Ġvector Ġspace . </s> App roach Ġ: :: ĠMeasures Ġof ĠContext uality Ġ: :: ĠDefinition Ġ3 </s> Let Ġ$ w $ Ġbe Ġa Ġword Ġthat Ġappears Ġin Ġsentences Ġ$\ l brace Ġs _ 1 , Ġ... , Ġs _ n Ġ\ r brace Ġ$ Ġat Ġindices Ġ$\ l brace Ġi _ 1 , Ġ... , Ġi _ n Ġ\ r brace Ġ$ Ġrespectively , Ġsuch Ġthat Ġ$ w Ġ= Ġs _ 1 [ i _ 1 ] Ġ= Ġ... Ġ= Ġs _ n [ i _ n ] $. ĠLet Ġ$ f _{ \ ell Ġ} ( s , i ) $ Ġbe Ġa Ġfunction Ġthat Ġmaps Ġ$ s [ i ] $ Ġto Ġits Ġrepresentation Ġin Ġlayer Ġ$\ ell Ġ$ Ġof Ġmodel Ġ$ f $. ĠWhere Ġ$ [ Ġf _{ \ ell Ġ} ( s _ 1 , Ġi _ 1 ) Ġ... Ġf _{ \ ell Ġ} ( s _ n , Ġi _ n ) Ġ] $ Ġis Ġthe Ġoccurrence Ġmatrix Ġof Ġ$ w $ Ġand Ġ$\ s igma Ġ_ 1 Ġ... Ġ\ s igma Ġ_ m $ Ġare Ġthe Ġfirst Ġ$ m $ Ġsingular Ġvalues Ġof Ġthis Ġmatrix , Ġthe Ġmaximum Ġexplain able Ġvariance Ġis </s> $ \ text it Ġ{ ME V } _ \ ell Ġ( w ) $ Ġis Ġthe Ġproportion Ġof Ġvariance Ġin Ġ$ w $ ' s Ġcontextual ized Ġrepresentations Ġfor Ġa Ġgiven Ġlayer Ġthat Ġcan Ġbe Ġexplained Ġby Ġtheir Ġfirst Ġprincipal Ġcomponent . ĠIt Ġgives Ġus Ġan Ġupper Ġbound Ġon Ġhow Ġwell Ġa Ġstatic Ġembed ding Ġcould Ġreplace Ġa Ġword 's Ġcontextual ized Ġrepresentations . ĠThe Ġcloser Ġ$\ text it Ġ{ ME V } _ \ ell Ġ( w ) $ Ġis Ġto Ġ0 , Ġthe Ġpoorer Ġa Ġreplacement Ġa Ġstatic Ġembed ding Ġwould Ġbe ; Ġif Ġ$\ text it Ġ{ ME V } _ \ ell Ġ( w ) Ġ= Ġ1 $, Ġthen Ġa Ġstatic Ġembed ding Ġwould Ġbe Ġa Ġperfect Ġreplacement Ġfor Ġthe Ġcontextual ized Ġrepresentations . </s> App roach Ġ: :: ĠAdjust ing Ġfor ĠAn is ot ropy </s> It Ġis Ġimportant Ġto Ġconsider Ġisot ropy Ġ( or Ġthe Ġlack Ġthereof ) Ġwhen Ġdiscussing Ġcontext uality . ĠFor Ġexample , Ġif Ġword Ġvectors Ġwere Ġperfectly Ġisot rop ic Ġ( i . e ., Ġdirection ally Ġuniform ), Ġthen Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) Ġ= Ġ0 . 95 $ Ġwould Ġsuggest Ġthat Ġ$ w $ ' s Ġrepresentations Ġwere Ġpoorly Ġcontextual ized . ĠHowever , Ġconsider Ġthe Ġscenario Ġwhere Ġword Ġvectors Ġare Ġso Ġan is otropic Ġthat Ġany Ġtwo Ġwords Ġhave Ġon Ġaverage Ġa Ġcos ine Ġsimilarity Ġof Ġ0 . 99 . ĠThen Ġ$\ text it Ġ{ Self Sim } _ \ ell Ġ( w ) Ġ= Ġ0 . 95 $ Ġwould Ġactually Ġsuggest Ġthe Ġopposite ĠâĢĵ Ġthat Ġ$ w $ ' s Ġrepresentations Ġwere Ġwell Ġcontextual ized . ĠThis Ġis Ġbecause Ġrepresentations Ġof Ġ$ w $ Ġin Ġdifferent Ġcontexts Ġwould Ġon Ġaverage Ġbe Ġmore Ġdiss imilar Ġto Ġeach Ġother Ġthan Ġtwo Ġrandomly Ġchosen Ġwords . </s> To Ġadjust Ġfor Ġthe Ġeffect Ġof Ġan is ot ropy , Ġwe Ġuse Ġthree Ġan is otropic Ġbas elines , Ġone Ġfor Ġeach Ġof Ġour Ġcontext uality Ġmeasures . ĠFor Ġself - similar ity Ġand Ġintra - sent ence Ġsimilarity , Ġthe Ġbaseline Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġthe Ġrepresentations Ġof Ġuniformly Ġrandomly Ġsampled Ġwords Ġfrom Ġdifferent Ġcontexts . ĠThe Ġmore Ġan is otropic Ġthe Ġword Ġrepresentations Ġare Ġin Ġa Ġgiven Ġlayer , Ġthe Ġcloser Ġthis Ġbaseline Ġis Ġto Ġ1 . ĠFor Ġmaximum Ġexplain able Ġvariance Ġ( ME V ), Ġthe Ġbaseline Ġis Ġthe Ġproportion Ġof Ġvariance Ġin Ġuniformly Ġrandomly Ġsampled Ġword Ġrepresentations Ġthat Ġis Ġexplained Ġby Ġtheir Ġfirst Ġprincipal Ġcomponent . ĠThe Ġmore Ġan is otropic Ġthe Ġrepresentations Ġin Ġa Ġgiven Ġlayer , Ġthe Ġcloser Ġthis Ġbaseline Ġis Ġto Ġ1 : Ġeven Ġfor Ġa Ġrandom Ġassortment Ġof Ġwords , Ġthe Ġprincipal Ġcomponent Ġwould Ġbe Ġable Ġto Ġexplain Ġa Ġlarge Ġproportion Ġof Ġthe Ġvariance . </s> Since Ġcontext uality Ġmeasures Ġare Ġcalculated Ġfor Ġeach Ġlayer Ġof Ġa Ġcontextual izing Ġmodel , Ġwe Ġcalculate Ġseparate Ġbas elines Ġfor Ġeach Ġlayer Ġas Ġwell . ĠWe Ġthen Ġsubtract Ġfrom Ġeach Ġmeasure Ġits Ġrespective Ġbaseline Ġto Ġget Ġthe Ġan is ot ropy - adjusted Ġcont ex uality Ġmeasure . ĠFor Ġexample , Ġthe Ġan is ot ropy - adjusted Ġself - similar ity Ġis </s> where Ġ$\ math cal Ġ{ O } $ Ġis Ġthe Ġset Ġof Ġall Ġword Ġoccurrences Ġand Ġ$ f _{ \ ell Ġ} (\ cd ot Ġ) $ Ġmaps Ġa Ġword Ġoccurrence Ġto Ġits Ġrepresentation Ġin Ġlayer Ġ$\ ell Ġ$ Ġof Ġmodel Ġ$ f $. ĠUnless Ġotherwise Ġstated , Ġreferences Ġto Ġcontext uality Ġmeasures Ġin Ġthe Ġrest Ġof Ġthe Ġpaper Ġrefer Ġto Ġthe Ġan is ot ropy - adjusted Ġmeasures , Ġwhere Ġboth Ġthe Ġraw Ġmeasure Ġand Ġbaseline Ġare Ġestimated Ġwith Ġ1 K Ġuniformly Ġrandomly Ġsampled Ġword Ġrepresentations . </s> Find ings Ġ: :: Ġ( An ) Is ot ropy Ġ: :: ĠContext ual ized Ġrepresentations Ġare Ġan is otropic Ġin Ġall Ġnon - input Ġlayers . </s> If Ġword Ġrepresentations Ġfrom Ġa Ġparticular Ġlayer Ġwere Ġisot rop ic Ġ( i . e ., Ġdirection ally Ġuniform ), Ġthen Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġuniformly Ġrandomly Ġsampled Ġwords Ġwould Ġbe Ġ0 ĠB IB REF 18 . ĠThe Ġcloser Ġthis Ġaverage Ġis Ġto Ġ1 , Ġthe Ġmore Ġan is otropic Ġthe Ġrepresentations . ĠThe Ġgeometric Ġinterpretation Ġof Ġan is ot ropy Ġis Ġthat Ġthe Ġword Ġrepresentations Ġall Ġoccupy Ġa Ġnarrow Ġcone Ġin Ġthe Ġvector Ġspace Ġrather Ġthan Ġbeing Ġuniform Ġin Ġall Ġdirections ; Ġthe Ġgreater Ġthe Ġan is ot ropy , Ġthe Ġnarrower Ġthis Ġcone ĠB IB REF 13 . ĠAs Ġseen Ġin ĠFigure ĠFIG REF 20 , Ġthis Ġimplies Ġthat Ġin Ġalmost Ġall Ġlayers Ġof ĠB ERT , ĠEL Mo Ġand ĠG PT - 2 , Ġthe Ġrepresentations Ġof Ġall Ġwords Ġoccupy Ġa Ġnarrow Ġcone Ġin Ġthe Ġvector Ġspace . ĠThe Ġonly Ġexception Ġis ĠEL Mo 's Ġinput Ġlayer , Ġwhich Ġproduces Ġstatic Ġcharacter - level Ġembed d ings Ġwithout Ġusing Ġcontextual Ġor Ġeven Ġpositional Ġinformation ĠB IB REF 2 . ĠIt Ġshould Ġbe Ġnoted Ġthat Ġnot Ġall Ġstatic Ġembed d ings Ġare Ġnecessarily Ġisot rop ic , Ġhowever ; ĠB IB REF 13 Ġfound Ġthat Ġskip gram Ġembed d ings , Ġwhich Ġare Ġalso Ġstatic , Ġare Ġnot Ġisot rop ic . </s> Find ings Ġ: :: Ġ( An ) Is ot ropy Ġ: :: ĠContext ual ized Ġrepresentations Ġare Ġgenerally Ġmore Ġan is otropic Ġin Ġhigher Ġlayers . </s> As Ġseen Ġin ĠFigure ĠFIG REF 20 , Ġfor ĠG PT - 2 , Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġuniformly Ġrandomly Ġwords Ġis Ġroughly Ġ0 . 6 Ġin Ġlayers Ġ2 Ġthrough Ġ8 Ġbut Ġincreases Ġexponentially Ġfrom Ġlayers Ġ8 Ġthrough Ġ12 . ĠIn Ġfact , Ġword Ġrepresentations Ġin ĠG PT - 2 's Ġlast Ġlayer Ġare Ġso Ġan is otropic Ġthat Ġany Ġtwo Ġwords Ġhave Ġon Ġaverage Ġan Ġalmost Ġperfect Ġcos ine Ġsimilarity ! ĠThis Ġpattern Ġholds Ġfor ĠB ERT Ġand ĠEL Mo Ġas Ġwell , Ġthough Ġthere Ġare Ġexceptions : Ġfor Ġexample , Ġthe Ġan is ot ropy Ġin ĠB ERT 's Ġpen ultimate Ġlayer Ġis Ġmuch Ġhigher Ġthan Ġin Ġits Ġfinal Ġlayer . </s> Is ot ropy Ġhas Ġboth Ġtheoretical Ġand Ġempirical Ġbenefits Ġfor Ġstatic Ġword Ġembed d ings . ĠIn Ġtheory , Ġit Ġallows Ġfor Ġstronger ĠâĢ ľ self - normal ization âĢ Ŀ Ġduring Ġtraining ĠB IB REF 18 , Ġand Ġin Ġpractice , Ġsubtract ing Ġthe Ġmean Ġvector Ġfrom Ġstatic Ġembed d ings Ġleads Ġto Ġimprovements Ġon Ġseveral Ġdownstream ĠN LP Ġtasks ĠB IB REF 6 . ĠThus Ġthe Ġextreme Ġdegree Ġof Ġan is ot ropy Ġseen Ġin Ġcontextual ized Ġword Ġrepresentations ĠâĢĵ Ġparticularly Ġin Ġhigher Ġlayers ĠâĢĵ Ġis Ġsurprising . ĠAs Ġseen Ġin ĠFigure ĠFIG REF 20 , Ġfor Ġall Ġthree Ġmodels , Ġthe Ġcontextual ized Ġhidden Ġlayer Ġrepresentations Ġare Ġalmost Ġall Ġmore Ġan is otropic Ġthan Ġthe Ġinput Ġlayer Ġrepresentations , Ġwhich Ġdo Ġnot Ġincorporate Ġcontext . ĠThis Ġsuggests Ġthat Ġhigh Ġan is ot ropy Ġis Ġinherent Ġto , Ġor Ġleast Ġa Ġby - product Ġof , Ġthe Ġprocess Ġof Ġcontextual ization . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠContext ual ized Ġword Ġrepresentations Ġare Ġmore Ġcontext - specific Ġin Ġhigher Ġlayers . </s> Rec all Ġfrom ĠDefinition Ġ1 Ġthat Ġthe Ġself - similar ity Ġof Ġa Ġword , Ġin Ġa Ġgiven Ġlayer Ġof Ġa Ġgiven Ġmodel , Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġits Ġrepresentations Ġin Ġdifferent Ġcontexts , Ġadjusted Ġfor Ġan is ot ropy . ĠIf Ġthe Ġself - similar ity Ġis Ġ1 , Ġthen Ġthe Ġrepresentations Ġare Ġnot Ġcontext - specific Ġat Ġall ; Ġif Ġthe Ġself - similar ity Ġis Ġ0 , Ġthat Ġthe Ġrepresentations Ġare Ġmaxim ally Ġcontext - specific . ĠIn ĠFigure ĠFIG REF 24 , Ġwe Ġplot Ġthe Ġaverage Ġself - similar ity Ġof Ġuniformly Ġrandomly Ġsampled Ġwords Ġin Ġeach Ġlayer Ġof ĠB ERT , ĠEL Mo , Ġand ĠG PT - 2 . ĠFor Ġexample , Ġthe Ġself - similar ity Ġis Ġ1 . 0 Ġin ĠEL Mo 's Ġinput Ġlayer Ġbecause Ġrepresentations Ġin Ġthat Ġlayer Ġare Ġstatic Ġcharacter - level Ġembed d ings . </s> In Ġall Ġthree Ġmodels , Ġthe Ġhigher Ġthe Ġlayer , Ġthe Ġlower Ġthe Ġself - similar ity Ġis Ġon Ġaverage . ĠIn Ġother Ġwords , Ġthe Ġhigher Ġthe Ġlayer , Ġthe Ġmore Ġcontext - specific Ġthe Ġcontextual ized Ġrepresentations . ĠThis Ġfinding Ġmakes Ġintuitive Ġsense . ĠIn Ġimage Ġclassification Ġmodels , Ġlower Ġlayers Ġrecognize Ġmore Ġgeneric Ġfeatures Ġsuch Ġas Ġedges Ġwhile Ġupper Ġlayers Ġrecognize Ġmore Ġclass - specific Ġfeatures ĠB IB REF 19 . ĠSimilarly , Ġupper Ġlayers Ġof ĠL ST Ms Ġtrained Ġon ĠN LP Ġtasks Ġlearn Ġmore Ġtask - specific Ġrepresentations ĠB IB REF 4 . ĠTherefore , Ġit Ġfollows Ġthat Ġupper Ġlayers Ġof Ġneural Ġlanguage Ġmodels Ġlearn Ġmore Ġcontext - specific Ġrepresentations , Ġso Ġas Ġto Ġpredict Ġthe Ġnext Ġword Ġfor Ġa Ġgiven Ġcontext Ġmore Ġaccurately . ĠOf Ġall Ġthree Ġmodels , Ġrepresentations Ġin ĠG PT - 2 Ġare Ġthe Ġmost Ġcontext - specific , Ġwith Ġthose Ġin ĠG PT - 2 's Ġlast Ġlayer Ġbeing Ġalmost Ġmaxim ally Ġcontext - specific . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠStop words Ġ( e . g ., Ġ` the ', Ġ` of ', Ġ` to ') Ġhave Ġamong Ġthe Ġmost Ġcontext - specific Ġrepresentations . </s> Across Ġall Ġlayers , Ġstop words Ġhave Ġamong Ġthe Ġlowest Ġself - similar ity Ġof Ġall Ġwords , Ġimplying Ġthat Ġtheir Ġcontextual ized Ġrepresentations Ġare Ġamong Ġthe Ġmost Ġcontext - specific . ĠFor Ġexample , Ġthe Ġwords Ġwith Ġthe Ġlowest Ġaverage Ġself - similar ity Ġacross ĠEL Mo 's Ġlayers Ġare Ġ` and ', Ġ` of ', Ġ` ' s ', Ġ` the ', Ġand Ġ` to '. ĠThis Ġis Ġrelatively Ġsurprising , Ġgiven Ġthat Ġthese Ġwords Ġare Ġnot Ġpoly sem ous . ĠThis Ġfinding Ġsuggests Ġthat Ġthe Ġvariety Ġof Ġcontexts Ġa Ġword Ġappears Ġin , Ġrather Ġthan Ġits Ġinherent Ġpoly se my , Ġis Ġwhat Ġdrives Ġvariation Ġin Ġits Ġcontextual ized Ġrepresentations . ĠThis Ġanswers Ġone Ġof Ġthe Ġquestions Ġwe Ġposed Ġin Ġthe Ġintroduction : ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 Ġare Ġnot Ġsimply Ġassigning Ġone Ġof Ġa Ġfinite Ġnumber Ġof Ġword - sense Ġrepresentations Ġto Ġeach Ġword ; Ġotherwise , Ġthere Ġwould Ġnot Ġbe Ġso Ġmuch Ġvariation Ġin Ġthe Ġrepresentations Ġof Ġwords Ġwith Ġso Ġfew Ġword Ġsenses . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠContext - specific ity Ġmanifests Ġvery Ġdifferently Ġin ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . </s> As Ġnoted Ġearlier , Ġcontextual ized Ġrepresentations Ġare Ġmore Ġcontext - specific Ġin Ġupper Ġlayers Ġof ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 . ĠHowever , Ġhow Ġdoes Ġthis Ġincreased Ġcontext - specific ity Ġmanifest Ġin Ġthe Ġvector Ġspace ? ĠDo Ġword Ġrepresentations Ġin Ġthe Ġsame Ġsentence Ġconverge Ġto Ġa Ġsingle Ġpoint , Ġor Ġdo Ġthey Ġremain Ġdistinct Ġfrom Ġone Ġanother Ġwhile Ġstill Ġbeing Ġdistinct Ġfrom Ġtheir Ġrepresentations Ġin Ġother Ġcontexts ? ĠTo Ġanswer Ġthis Ġquestion , Ġwe Ġcan Ġmeasure Ġa Ġsentence 's Ġintra - sent ence Ġsimilarity . ĠRecall Ġfrom ĠDefinition Ġ2 Ġthat Ġthe Ġintra - sent ence Ġsimilarity Ġof Ġa Ġsentence , Ġin Ġa Ġgiven Ġlayer Ġof Ġa Ġgiven Ġmodel , Ġis Ġthe Ġaverage Ġcos ine Ġsimilarity Ġbetween Ġeach Ġof Ġits Ġword Ġrepresentations Ġand Ġtheir Ġmean , Ġadjusted Ġfor Ġan is ot ropy . ĠIn ĠFigure ĠFIG REF 25 , Ġwe Ġplot Ġthe Ġaverage Ġintra - sent ence Ġsimilarity Ġof Ġ500 Ġuniformly Ġrandomly Ġsampled Ġsentences . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠIn ĠEL Mo , Ġwords Ġin Ġthe Ġsame Ġsentence Ġare Ġmore Ġsimilar Ġto Ġone Ġanother Ġin Ġupper Ġlayers . </s> As Ġword Ġrepresentations Ġin Ġa Ġsentence Ġbecome Ġmore Ġcontext - specific Ġin Ġupper Ġlayers , Ġthe Ġintra - sent ence Ġsimilarity Ġalso Ġrises . ĠThis Ġsuggests Ġthat , Ġin Ġpractice , ĠEL Mo Ġends Ġup Ġextending Ġthe Ġintuition Ġbehind ĠF irth 's ĠB IB REF 20 Ġdistribution al Ġhypothesis Ġto Ġthe Ġsentence Ġlevel : Ġthat Ġbecause Ġwords Ġin Ġthe Ġsame Ġsentence Ġshare Ġthe Ġsame Ġcontext , Ġtheir Ġcontextual ized Ġrepresentations Ġshould Ġalso Ġbe Ġsimilar . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠIn ĠB ERT , Ġwords Ġin Ġthe Ġsame Ġsentence Ġare Ġmore Ġdiss imilar Ġto Ġone Ġanother Ġin Ġupper Ġlayers . </s> As Ġword Ġrepresentations Ġin Ġa Ġsentence Ġbecome Ġmore Ġcontext - specific Ġin Ġupper Ġlayers , Ġthey Ġdrift Ġaway Ġfrom Ġone Ġanother , Ġalthough Ġthere Ġare Ġexceptions Ġ( see Ġlayer Ġ12 Ġin ĠFigure ĠFIG REF 25 ). ĠHowever , Ġin Ġall Ġlayers , Ġthe Ġaverage Ġsimilarity Ġbetween Ġwords Ġin Ġthe Ġsame Ġsentence Ġis Ġstill Ġgreater Ġthan Ġthe Ġaverage Ġsimilarity Ġbetween Ġrandomly Ġchosen Ġwords Ġ( i . e ., Ġthe Ġan is ot ropy Ġbaseline ). ĠThis Ġsuggests Ġa Ġmore Ġnuanced Ġcontextual ization Ġthan Ġin ĠEL Mo , Ġwith ĠB ERT Ġrecognizing Ġthat Ġalthough Ġthe Ġsurrounding Ġsentence Ġinforms Ġa Ġword 's Ġmeaning , Ġtwo Ġwords Ġin Ġthe Ġsame Ġsentence Ġdo Ġnot Ġnecessarily Ġhave Ġa Ġsimilar Ġmeaning Ġbecause Ġthey Ġshare Ġthe Ġsame Ġcontext . </s> Find ings Ġ: :: ĠContext - Specific ity Ġ: :: ĠIn ĠG PT - 2 , Ġword Ġrepresentations Ġin Ġthe Ġsame Ġsentence Ġare Ġno Ġmore Ġsimilar Ġto Ġeach Ġother Ġthan Ġrandomly Ġsampled Ġwords . </s> On Ġaverage , Ġthe Ġun adjusted Ġintra - sent ence Ġsimilarity Ġis Ġroughly Ġthe Ġsame Ġas Ġthe Ġan is otropic Ġbaseline , Ġso Ġas Ġseen Ġin ĠFigure ĠFIG REF 25 , Ġthe Ġan is ot ropy - adjusted Ġintra - sent ence Ġsimilarity Ġis Ġclose Ġto Ġ0 Ġin Ġmost Ġlayers Ġof ĠG PT - 2 . ĠIn Ġfact , Ġthe Ġintra - sent ence Ġsimilarity Ġis Ġhighest Ġin Ġthe Ġinput Ġlayer , Ġwhich Ġdoes Ġnot Ġcontextual ize Ġwords Ġat Ġall . ĠThis Ġis Ġin Ġcontrast Ġto ĠEL Mo Ġand ĠB ERT , Ġwhere Ġthe Ġaverage Ġintra - sent ence Ġsimilarity Ġis Ġabove Ġ0 . 20 Ġfor Ġall Ġbut Ġone Ġlayer . </s> As Ġnoted Ġearlier Ġwhen Ġdiscussing ĠB ERT , Ġthis Ġbehavior Ġstill Ġmakes Ġintuitive Ġsense : Ġtwo Ġwords Ġin Ġthe Ġsame Ġsentence Ġdo Ġnot Ġnecessarily Ġhave Ġa Ġsimilar Ġmeaning Ġsimply Ġbecause Ġthey Ġshare Ġthe Ġsame Ġcontext . ĠThe Ġsuccess Ġof ĠG PT - 2 Ġsuggests Ġthat Ġunlike Ġan is ot ropy , Ġwhich Ġaccompanies Ġcontext - specific ity Ġin Ġall Ġthree Ġmodels , Ġa Ġhigh Ġintra - sent ence Ġsimilarity Ġis Ġnot Ġinherent Ġto Ġcontextual ization . ĠWords Ġin Ġthe Ġsame Ġsentence Ġcan Ġhave Ġhighly Ġcontextual ized Ġrepresentations Ġwithout Ġthose Ġrepresentations Ġbeing Ġany Ġmore Ġsimilar Ġto Ġeach Ġother Ġthan Ġtwo Ġrandom Ġword Ġrepresentations . ĠIt Ġis Ġunclear , Ġhowever , Ġwhether Ġthese Ġdifferences Ġin Ġintra - sent ence Ġsimilarity Ġcan Ġbe Ġtraced Ġback Ġto Ġdifferences Ġin Ġmodel Ġarchitecture ; Ġwe Ġleave Ġthis Ġquestion Ġas Ġfuture Ġwork . </s> Find ings Ġ: :: ĠStatic Ġvs . ĠContext ual ized Ġ: :: ĠOn Ġaverage , Ġless Ġthan Ġ5 % Ġof Ġthe Ġvariance Ġin Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġcan Ġbe Ġexplained Ġby Ġa Ġstatic Ġembed ding . </s> Rec all Ġfrom ĠDefinition Ġ3 Ġthat Ġthe Ġmaximum Ġexplain able Ġvariance Ġ( ME V ) Ġof Ġa Ġword , Ġfor Ġa Ġgiven Ġlayer Ġof Ġa Ġgiven Ġmodel , Ġis Ġthe Ġproportion Ġof Ġvariance Ġin Ġits Ġcontextual ized Ġrepresentations Ġthat Ġcan Ġbe Ġexplained Ġby Ġtheir Ġfirst Ġprincipal Ġcomponent . ĠThis Ġgives Ġus Ġan Ġupper Ġbound Ġon Ġhow Ġwell Ġa Ġstatic Ġembed ding Ġcould Ġreplace Ġa Ġword 's Ġcontextual ized Ġrepresentations . ĠBecause Ġcontextual ized Ġrepresentations Ġare Ġan is otropic Ġ( see Ġsection ĠSEC REF 21 ), Ġmuch Ġof Ġthe Ġvariation Ġacross Ġall Ġwords Ġcan Ġbe Ġexplained Ġby Ġa Ġsingle Ġvector . ĠWe Ġadjust Ġfor Ġan is ot ropy Ġby Ġcalculating Ġthe Ġproportion Ġof Ġvariance Ġexplained Ġby Ġthe Ġfirst Ġprincipal Ġcomponent Ġof Ġuniformly Ġrandomly Ġsampled Ġword Ġrepresentations Ġand Ġsubtract ing Ġthis Ġproportion Ġfrom Ġthe Ġraw ĠME V . ĠIn ĠFigure ĠFIG REF 29 , Ġwe Ġplot Ġthe Ġaverage Ġan is ot ropy - adjusted ĠME V Ġacross Ġuniformly Ġrandomly Ġsampled Ġwords . </s> In Ġno Ġlayer Ġof ĠEL Mo , ĠB ERT , Ġor ĠG PT - 2 Ġcan Ġmore Ġthan Ġ5 % Ġof Ġthe Ġvariance Ġin Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġbe Ġexplained Ġby Ġa Ġstatic Ġembed ding , Ġon Ġaverage . ĠThough Ġnot Ġvisible Ġin ĠFigure ĠFIG REF 29 , Ġthe Ġraw ĠME V Ġof Ġmany Ġwords Ġis Ġactually Ġbelow Ġthe Ġan is ot ropy Ġbaseline : Ġi . e ., Ġa Ġgreater Ġproportion Ġof Ġthe Ġvariance Ġacross Ġall Ġwords Ġcan Ġbe Ġexplained Ġby Ġa Ġsingle Ġvector Ġthan Ġcan Ġthe Ġvariance Ġacross Ġall Ġrepresentations Ġof Ġa Ġsingle Ġword . ĠNote Ġthat Ġthe Ġ5 % Ġthreshold Ġrepresents Ġthe Ġbest - case Ġscenario , Ġand Ġthere Ġis Ġno Ġtheoretical Ġguarantee Ġthat Ġa Ġword Ġvector Ġobtained Ġusing ĠGl o Ve , Ġfor Ġexample , Ġwould Ġbe Ġsimilar Ġto Ġthe Ġstatic Ġembed ding Ġthat Ġmaxim izes ĠME V . ĠThis Ġsuggests Ġthat Ġcontextual izing Ġmodels Ġare Ġnot Ġsimply Ġassigning Ġone Ġof Ġa Ġfinite Ġnumber Ġof Ġword - sense Ġrepresentations Ġto Ġeach Ġword ĠâĢĵ Ġotherwise , Ġthe Ġproportion Ġof Ġvariance Ġexplained Ġwould Ġbe Ġmuch Ġhigher . ĠEven Ġthe Ġaverage Ġraw ĠME V Ġis Ġbelow Ġ5 % Ġfor Ġall Ġlayers Ġof ĠEL Mo Ġand ĠB ERT ; Ġonly Ġfor ĠG PT - 2 Ġis Ġthe Ġraw ĠME V Ġnon - neg lig ible , Ġbeing Ġaround Ġ30 % Ġon Ġaverage Ġfor Ġlayers Ġ2 Ġto Ġ11 Ġdue Ġto Ġextremely Ġhigh Ġan is ot ropy . </s> Find ings Ġ: :: ĠStatic Ġvs . ĠContext ual ized Ġ: :: ĠPrincipal Ġcomponents Ġof Ġcontextual ized Ġrepresentations Ġin Ġlower Ġlayers Ġoutper form ĠGl o Ve Ġand ĠFast Text Ġon Ġmany Ġbenchmarks . </s> As Ġnoted Ġearlier , Ġwe Ġcan Ġcreate Ġstatic Ġembed d ings Ġfor Ġeach Ġword Ġby Ġtaking Ġthe Ġfirst Ġprincipal Ġcomponent Ġ( PC ) Ġof Ġits Ġcontextual ized Ġrepresentations Ġin Ġa Ġgiven Ġlayer . ĠIn ĠTable ĠT AB REF 34 , Ġwe Ġplot Ġthe Ġperformance Ġof Ġthese ĠPC Ġstatic Ġembed d ings Ġon Ġseveral Ġbenchmark Ġtasks . ĠThese Ġtasks Ġcover Ġsemantic Ġsimilarity , Ġanalogy Ġsolving , Ġand Ġconcept Ġcategor ization : ĠSim Lex 999 ĠB IB REF 21 , ĠMEN ĠB IB REF 22 , ĠWS 353 ĠB IB REF 23 , ĠRW ĠB IB REF 24 , ĠSem E val - 2012 ĠB IB REF 25 , ĠGoogle Ġanalogy Ġsolving ĠB IB REF 0 ĠMS R Ġanalogy Ġsolving ĠB IB REF 26 , ĠBL ESS ĠB IB REF 27 Ġand ĠAP ĠB IB REF 28 . ĠWe Ġleave Ġout Ġlayers Ġ3 Ġ- Ġ10 Ġin ĠTable ĠT AB REF 34 Ġbecause Ġtheir Ġperformance Ġis Ġbetween Ġthose Ġof ĠL ayers Ġ2 Ġand Ġ11 . </s> The Ġbest - performing ĠPC Ġstatic Ġembed d ings Ġbelong Ġto Ġthe Ġfirst Ġlayer Ġof ĠB ERT , Ġalthough Ġthose Ġfrom Ġthe Ġother Ġlayers Ġof ĠB ERT Ġand ĠEL Mo Ġalso Ġoutper form ĠGl o Ve Ġand ĠFast Text Ġon Ġmost Ġbenchmarks . ĠFor Ġall Ġthree Ġcontextual izing Ġmodels , ĠPC Ġstatic Ġembed d ings Ġcreated Ġfrom Ġlower Ġlayers Ġare Ġmore Ġeffective Ġthose Ġcreated Ġfrom Ġupper Ġlayers . ĠThose Ġcreated Ġusing ĠG PT - 2 Ġalso Ġperform Ġmarkedly Ġworse Ġthan Ġtheir Ġcounterparts Ġfrom ĠEL Mo Ġand ĠB ERT . ĠGiven Ġthat Ġupper Ġlayers Ġare Ġmuch Ġmore Ġcontext - specific Ġthan Ġlower Ġlayers , Ġand Ġgiven Ġthat ĠG PT - 2 's Ġrepresentations Ġare Ġmore Ġcontext - specific Ġthan ĠEL Mo Ġand ĠB ERT 's Ġ( see ĠFigure ĠFIG REF 24 ), Ġthis Ġsuggests Ġthat Ġthe ĠPCs Ġof Ġhighly Ġcontext - specific Ġrepresentations Ġare Ġless Ġeffective Ġon Ġtraditional Ġbenchmarks . ĠThose Ġderived Ġfrom Ġless Ġcontext - specific Ġrepresentations , Ġsuch Ġas Ġthose Ġfrom ĠLayer Ġ1 Ġof ĠB ERT , Ġare Ġmuch Ġmore Ġeffective . </s> Future ĠWork </s> Our Ġfindings Ġoffer Ġsome Ġnew Ġdirections Ġfor Ġfuture Ġwork . ĠFor Ġone , Ġas Ġnoted Ġearlier Ġin Ġthe Ġpaper , ĠB IB REF 6 Ġfound Ġthat Ġmaking Ġstatic Ġembed d ings Ġmore Ġisot rop ic ĠâĢĵ Ġby Ġsubtract ing Ġtheir Ġmean Ġfrom Ġeach Ġembed ding ĠâĢĵ Ġleads Ġto Ġsurprisingly Ġlarge Ġimprovements Ġin Ġperformance Ġon Ġdownstream Ġtasks . ĠGiven Ġthat Ġisot ropy Ġhas Ġbenefits Ġfor Ġstatic Ġembed d ings , Ġit Ġmay Ġalso Ġhave Ġbenefits Ġfor Ġcontextual ized Ġword Ġrepresentations , Ġalthough Ġthe Ġlatter Ġhave Ġalready Ġyielded Ġsignificant Ġimprovements Ġdespite Ġbeing Ġhighly Ġan is otropic . ĠTherefore , Ġadding Ġan Ġan is ot ropy Ġpenalty Ġto Ġthe Ġlanguage Ġmodelling Ġobjective ĠâĢĵ Ġto Ġencourage Ġthe Ġcontextual ized Ġrepresentations Ġto Ġbe Ġmore Ġisot rop ic ĠâĢĵ Ġmay Ġyield Ġeven Ġbetter Ġresults . </s> Another Ġdirection Ġfor Ġfuture Ġwork Ġis Ġgenerating Ġstatic Ġword Ġrepresentations Ġfrom Ġcontextual ized Ġones . ĠWhile Ġthe Ġlatter Ġoffer Ġsuperior Ġperformance , Ġthere Ġare Ġoften Ġchallenges Ġto Ġdeploying Ġlarge Ġmodels Ġsuch Ġas ĠB ERT Ġin Ġproduction , Ġboth Ġwith Ġrespect Ġto Ġmemory Ġand Ġrun - time . ĠIn Ġcontrast , Ġstatic Ġrepresentations Ġare Ġmuch Ġeasier Ġto Ġdeploy . ĠOur Ġwork Ġin Ġsection Ġ4 . 3 Ġsuggests Ġthat Ġnot Ġonly Ġit Ġis Ġpossible Ġto Ġextract Ġstatic Ġrepresentations Ġfrom Ġcontextual izing Ġmodels , Ġbut Ġthat Ġthese Ġextracted Ġvectors Ġoften Ġperform Ġmuch Ġbetter Ġon Ġa Ġdiverse Ġarray Ġof Ġtasks Ġcompared Ġto Ġtraditional Ġstatic Ġembed d ings Ġsuch Ġas ĠGl o Ve Ġand ĠFast Text . ĠThis Ġmay Ġbe Ġa Ġmeans Ġof Ġextracting Ġsome Ġuse Ġfrom Ġcontextual izing Ġmodels Ġwithout Ġinc urring Ġthe Ġfull Ġcost Ġof Ġusing Ġthem Ġin Ġproduction . </s> Conclusion </s> In Ġthis Ġpaper , Ġwe Ġinvestigated Ġhow Ġcontextual Ġcontextual ized Ġword Ġrepresentations Ġtruly Ġare . ĠFor Ġone , Ġwe Ġfound Ġthat Ġupper Ġlayers Ġof ĠEL Mo , ĠB ERT , Ġand ĠG PT - 2 Ġproduce Ġmore Ġcontext - specific Ġrepresentations Ġthan Ġlower Ġlayers . ĠThis Ġincreased Ġcontext - specific ity Ġis Ġalways Ġaccompanied Ġby Ġincreased Ġan is ot ropy . ĠHowever , Ġcontext - specific ity Ġalso Ġmanifests Ġdifferently Ġacross Ġthe Ġthree Ġmodels ; Ġthe Ġan is ot ropy - adjusted Ġsimilarity Ġbetween Ġwords Ġin Ġthe Ġsame Ġsentence Ġis Ġhighest Ġin ĠEL Mo Ġbut Ġalmost Ġnon - existent Ġin ĠG PT - 2 . ĠWe Ġultimately Ġfound Ġthat Ġafter Ġadjusting Ġfor Ġan is ot ropy , Ġon Ġaverage , Ġless Ġthan Ġ5 % Ġof Ġthe Ġvariance Ġin Ġa Ġword 's Ġcontextual ized Ġrepresentations Ġcould Ġbe Ġexplained Ġby Ġa Ġstatic Ġembed ding . ĠThis Ġmeans Ġthat Ġeven Ġin Ġthe Ġbest - case Ġscenario , Ġin Ġall Ġlayers Ġof Ġall Ġmodels , Ġstatic Ġword Ġembed d ings Ġwould Ġbe Ġa Ġpoor Ġreplacement Ġfor Ġcontextual ized Ġones . ĠThese Ġinsights Ġhelp Ġexplain Ġsome Ġof Ġthe Ġremarkable Ġsuccess Ġthat Ġcontextual ized Ġrepresentations Ġhave Ġhad Ġon Ġa Ġdiverse Ġarray Ġof ĠN LP Ġtasks . </s> Acknowled gments </s> We Ġthank Ġthe Ġanonymous Ġreviewers Ġfor Ġtheir Ġinsightful Ġcomments . ĠWe Ġthank Ġthe ĠNatural ĠSciences Ġand ĠEngineering ĠResearch ĠCouncil Ġof ĠCanada Ġ( NS ERC ) Ġfor Ġtheir Ġfinancial Ġsupport .