Answer with content missing: (Whole Method and Results sections) Self-paced reading times widely benefit ERP prediction, while eye-tracking data seems to have more limited benefit to just the ELAN, LAN, and PNP ERP components.
Select:
- ELAN, LAN
- PNP ERP
ABSTRACTIVE
Unanswerable
NONE
This work is most closely related to the paper from which we get the ERP data: BIBREF0 . In that work, the authors relate the surprisal of a word, i.e. the (negative log) probability of the word appearing in its context, to each of the ERP signals we consider here. The authors do not directly train a model to predict ERPs. Instead, models of the probability distribution of each word in context are used to compute a surprisal for each word, which is input into a mixed effects regression along with word frequency, word length, word position in the sentence, and sentence position in the experiment. The effect of the surprisal is assessed using a likelihood-ratio test. In BIBREF7 , the authors take an approach similar to BIBREF0 . The authors compare the explanatory power of surprisal (as computed by an LSTM or a Recurrent Neural Network Grammar (RNNG) language model) to a measure of syntactic complexity they call “distance" that counts the number of parser actions in the RNNG language model. The authors find that surprisal (as predicted by the RNNG) and distance are both significant factors in a mixed effects regression which predicts the P600, while the surprisal as computed by an LSTM is not. Unlike BIBREF0 and BIBREF7 , we do not use a linking function (e.g. surprisal) to relate a language model to ERPs. We thus lose the interpretability provided by the linking function, but we are able to predict a significant proportion of the variance for all of the ERP components, where prior work could not. We interpret our results through characterization of the ERPs in terms of how they relate to each other and to eye-tracking data rather than through a linking function. The authors in BIBREF8 also use a recurrent neural network to predict neural activity directly. In that work the authors predict magnetoencephalography (MEG) activity, a close cousin to EEG, recorded while participants read a chapter of Harry Potter and the Sorcerer’s Stone BIBREF9 . Their approach to characterization of processing at each MEG sensor location is to determine whether it is best predicted by the context vector of the recurrent network (prior to the current word being processed), the embedding of the current word, or the probability of the current word given the context. In future work we also intend to add these types of studies to the ERP predictions. Discussion
1904.01548
Introduction </s> The Ġcognitive Ġprocesses Ġinvolved Ġin Ġhuman Ġlanguage Ġcomprehension Ġare Ġcomplex Ġand Ġonly Ġpartially Ġidentified . ĠAccording Ġto Ġthe Ġdual - stream Ġmodel Ġof Ġspeech Ġcomprehension ĠB IB REF 1 Ġ, Ġsound Ġwaves Ġare Ġfirst Ġconverted Ġto Ġphon eme - like Ġfeatures Ġand Ġfurther Ġprocessed Ġby Ġa Ġvent ral Ġstream Ġthat Ġmaps Ġthose Ġfeatures Ġonto Ġwords Ġand Ġsemantic Ġstructures , Ġand Ġa Ġdorsal Ġstream Ġthat Ġ( among Ġother Ġthings ) Ġsupports Ġaudio - short Ġterm Ġmemory . ĠThe Ġmapping Ġof Ġwords Ġonto Ġmeaning Ġis Ġthought Ġto Ġbe Ġsub served Ġby Ġwidely Ġdistributed Ġregions Ġof Ġthe Ġbrain Ġthat Ġspecialize Ġin Ġparticular Ġmod alities ĠâĢĶ Ġfor Ġexample Ġvisual Ġaspects Ġof Ġthe Ġword Ġbanana Ġreside Ġin Ġthe Ġocc ip ital Ġlobe Ġof Ġthe Ġbrain Ġand Ġare Ġactivated Ġwhen Ġthe Ġword Ġbanana Ġis Ġheard ĠB IB REF 2 ĠâĢĶ Ġand Ġthe Ġdifferent Ġrepresentation Ġmod alities Ġare Ġthought Ġto Ġbe Ġintegrated Ġinto Ġa Ġsingle Ġcoherent Ġlatent Ġrepresentation Ġin Ġthe Ġanterior Ġtemporal Ġlobe ĠB IB REF 3 Ġ. ĠWhile Ġthis Ġpart Ġof Ġmeaning Ġrepresentation Ġin Ġhuman Ġlanguage Ġcomprehension Ġis Ġsomewhat Ġunderstood , Ġmuch Ġless Ġis Ġknown Ġabout Ġhow Ġthe Ġmeanings Ġof Ġwords Ġare Ġintegrated Ġtogether Ġto Ġform Ġthe Ġmeaning Ġof Ġsentences Ġand Ġdisc ourses . ĠOne Ġtool Ġresearchers Ġuse Ġto Ġstudy Ġthe Ġintegration Ġof Ġmeaning Ġacross Ġwords Ġis Ġelectro ence p hel ography Ġ( EE G ), Ġwhich Ġmeasures Ġthe Ġelectrical Ġactivity Ġof Ġlarge Ġnumbers Ġof Ġneurons Ġacting Ġin Ġconcert . ĠEEG Ġhas Ġthe Ġtemporal Ġresolution Ġnecessary Ġto Ġstudy Ġthe Ġprocesses Ġinvolved Ġin Ġmeaning Ġintegration , Ġand Ġcertain Ġstereotyp ed Ġelectrical Ġresponses Ġto Ġword Ġpresentations , Ġknown Ġas Ġevent - related Ġpotential s Ġ( ER Ps ), Ġhave Ġbeen Ġidentified Ġwith Ġsome Ġof Ġthe Ġprocesses Ġthought Ġto Ġcontribute Ġto Ġcomprehension . </s> In Ġthis Ġwork , Ġwe Ġconsider Ġsix ĠER P Ġcomponents Ġthat Ġhave Ġbeen Ġassociated Ġin Ġthe Ġcognitive Ġneuroscience Ġand Ġpsych ol ingu istics Ġliterature Ġwith Ġlanguage Ġprocessing Ġand Ġwhich Ġwe Ġanalyze Ġin Ġthe Ġdata Ġfrom ĠB IB REF 0 Ġ( see ĠFigure ĠFIG REF 1 Ġfor Ġspatial Ġand Ġtemporal Ġdefinitions Ġof Ġthese ĠER P Ġcomponents ). ĠThree Ġof Ġthese ĠâĢĶ Ġthe ĠN 400 , ĠE PN P , Ġand ĠP NP Ġresponses ĠâĢĶ Ġare Ġprimarily Ġconsidered Ġmarkers Ġfor Ġsemantic Ġprocessing , Ġwhile Ġthe Ġother Ġthree ĠâĢĶ Ġthe ĠP 600 , ĠEL AN , Ġand ĠLAN Ġresponses ĠâĢĶ Ġare Ġprimarily Ġconsidered Ġmarkers Ġfor Ġsynt actic Ġprocessing . ĠHowever , Ġthe Ġneat Ġdivision Ġof Ġthe ĠER P Ġresponses Ġinto Ġeither Ġsemantic Ġor Ġsynt actic Ġcategories Ġis Ġcontroversial . ĠThe ĠN 400 Ġresponse Ġhas Ġbeen Ġvery Ġwell Ġstudied Ġ( for Ġan Ġoverview Ġsee ĠB IB REF 4 Ġ) Ġand Ġit Ġis Ġwell Ġestablished Ġthat Ġit Ġis Ġassociated Ġwith Ġsemantic Ġcomplexity , Ġbut Ġthe Ġfeatures Ġof Ġlanguage Ġthat Ġtrigger Ġthe Ġother ĠER P Ġresponses Ġwe Ġconsider Ġhere Ġare Ġpoorly Ġunderstood . ĠWe Ġpropose Ġto Ġuse Ġa Ġneural Ġnetwork Ġpret rained Ġas Ġa Ġlanguage Ġmodel Ġto Ġprobe Ġwhat Ġfeatures Ġof Ġlanguage Ġdrive Ġthese ĠER P Ġresponses , Ġand Ġin Ġturn Ġto Ġprobe Ġwhat Ġfeatures Ġof Ġlanguage Ġmed iate Ġthe Ġcognitive Ġprocesses Ġthat Ġunder lie Ġhuman Ġlanguage Ġcomprehension , Ġand Ġespecially Ġthe Ġintegration Ġof Ġmeaning Ġacross Ġwords . </s> Background </s> While Ġa Ġfull Ġdiscussion Ġof Ġeach ĠER P Ġcomponent Ġand Ġthe Ġfeatures Ġof Ġlanguage Ġthought Ġto Ġtrigger Ġeach Ġare Ġbeyond Ġthe Ġscope Ġof Ġthis Ġdocument Ġ( for Ġreviews Ġsee Ġe . g . ĠB IB REF 0 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġand ĠB IB REF 6 Ġ), Ġwe Ġintroduce Ġsome Ġbasic Ġfeatures Ġof ĠER P Ġcomponents Ġto Ġhelp Ġin Ġthe Ġdiscussion Ġlater . ĠER P Ġcomponents Ġare Ġelectrical Ġpotential Ġresponses Ġmeasured Ġwith Ġrespect Ġto Ġa Ġbaseline Ġthat Ġare Ġtriggered Ġby Ġan Ġevent Ġ( in Ġour Ġcase Ġthe Ġpresentation Ġof Ġa Ġnew Ġword Ġto Ġa Ġparticipant Ġin Ġan Ġexperiment ). ĠThe Ġname Ġof Ġeach ĠER P Ġcomponent Ġreflects Ġwhether Ġthe Ġpotential Ġis Ġpositive Ġor Ġnegative Ġrelative Ġto Ġthe Ġbaseline . ĠThe ĠN 400 Ġis Ġso - named Ġbecause Ġit Ġis ĠNegative Ġrelative Ġto Ġa Ġbaseline Ġ( the Ġbaseline Ġis Ġtypically Ġrecorded Ġjust Ġbefore Ġa Ġword Ġis Ġpresented Ġat Ġan Ġelectrode Ġthat Ġis Ġnot Ġaffected Ġby Ġthe ĠER P Ġresponse ) Ġand Ġbecause Ġit Ġpeaks Ġin Ġmagnitude Ġat Ġabout Ġ400 ms Ġafter Ġa Ġword Ġis Ġpresented Ġto Ġa Ġparticipant Ġin Ġan Ġexperiment . ĠThe ĠP 600 Ġis ĠPositive Ġrelative Ġto Ġa Ġbaseline Ġand Ġpeaks Ġaround Ġ600 ms Ġafter Ġa Ġword Ġis Ġpresented Ġto Ġa Ġparticipant Ġ( though Ġits Ġoverall Ġduration Ġis Ġmuch Ġlonger Ġand Ġless Ġspecific Ġin Ġtime Ġthan Ġthe ĠN 400 ). ĠThe Ġpost - N 400 Ġpos itivity Ġis Ġso - named Ġbecause Ġit Ġis Ġpart Ġof Ġa Ġbip has ic Ġresponse ; Ġit Ġis Ġa Ġpos itivity Ġthat Ġoccurs Ġafter Ġthe Ġnegativity Ġassociated Ġwith Ġthe ĠN 400 . ĠThe Ġearly Ġpost - N 400 Ġpos itivity Ġ( EP NP ) Ġis Ġalso Ġpart Ġof Ġa Ġbip has ic Ġresponse , Ġbut Ġthe Ġpos itivity Ġhas Ġan Ġear iler Ġonset Ġthan Ġthe Ġstandard ĠP NP . ĠFinally , Ġthe ĠLAN Ġand ĠEL AN Ġare Ġthe Ġleft - an terior Ġnegativity Ġand Ġearly Ġleft - an terior Ġnegativity Ġrespectively . ĠThese Ġare Ġnamed Ġfor Ġtheir Ġtiming , Ġspatial Ġdistribution Ġon Ġthe Ġscalp , Ġand Ġdirection Ġof Ġdifference Ġfrom Ġthe Ġbaseline . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat ĠER P Ġcomponents Ġcan Ġpotentially Ġcancel Ġand Ġmask Ġeach Ġother , Ġand Ġthat Ġit Ġis Ġdifficult Ġto Ġprecisely Ġlocal ize Ġthe Ġneural Ġactivity Ġthat Ġcauses Ġthe Ġchanges Ġin Ġelectrical Ġpotential Ġat Ġthe Ġelectrodes Ġwhere Ġthose Ġchanges Ġare Ġmeasured . </s> Related ĠWork </s> This Ġwork Ġis Ġmost Ġclosely Ġrelated Ġto Ġthe Ġpaper Ġfrom Ġwhich Ġwe Ġget Ġthe ĠER P Ġdata : ĠB IB REF 0 Ġ. ĠIn Ġthat Ġwork , Ġthe Ġauthors Ġrelate Ġthe Ġsurpr isal Ġof Ġa Ġword , Ġi . e . Ġthe Ġ( negative Ġlog ) Ġprobability Ġof Ġthe Ġword Ġappearing Ġin Ġits Ġcontext , Ġto Ġeach Ġof Ġthe ĠER P Ġsignals Ġwe Ġconsider Ġhere . ĠThe Ġauthors Ġdo Ġnot Ġdirectly Ġtrain Ġa Ġmodel Ġto Ġpredict ĠER Ps . ĠInstead , Ġmodels Ġof Ġthe Ġprobability Ġdistribution Ġof Ġeach Ġword Ġin Ġcontext Ġare Ġused Ġto Ġcompute Ġa Ġsurpr isal Ġfor Ġeach Ġword , Ġwhich Ġis Ġinput Ġinto Ġa Ġmixed Ġeffects Ġregression Ġalong Ġwith Ġword Ġfrequency , Ġword Ġlength , Ġword Ġposition Ġin Ġthe Ġsentence , Ġand Ġsentence Ġposition Ġin Ġthe Ġexperiment . ĠThe Ġeffect Ġof Ġthe Ġsurpr isal Ġis Ġassessed Ġusing Ġa Ġlikelihood - rat io Ġtest . ĠIn ĠB IB REF 7 Ġ, Ġthe Ġauthors Ġtake Ġan Ġapproach Ġsimilar Ġto ĠB IB REF 0 Ġ. ĠThe Ġauthors Ġcompare Ġthe Ġexplanatory Ġpower Ġof Ġsurpr isal Ġ( as Ġcomputed Ġby Ġan ĠL ST M Ġor Ġa ĠRec urrent ĠNeural ĠNetwork ĠGram mar Ġ( R NN G ) Ġlanguage Ġmodel ) Ġto Ġa Ġmeasure Ġof Ġsynt actic Ġcomplexity Ġthey Ġcall ĠâĢ ľ distance " Ġthat Ġcounts Ġthe Ġnumber Ġof Ġparser Ġactions Ġin Ġthe ĠR NN G Ġlanguage Ġmodel . ĠThe Ġauthors Ġfind Ġthat Ġsurpr isal Ġ( as Ġpredicted Ġby Ġthe ĠR NN G ) Ġand Ġdistance Ġare Ġboth Ġsignificant Ġfactors Ġin Ġa Ġmixed Ġeffects Ġregression Ġwhich Ġpredicts Ġthe ĠP 600 , Ġwhile Ġthe Ġsurpr isal Ġas Ġcomputed Ġby Ġan ĠL ST M Ġis Ġnot . ĠUnlike ĠB IB REF 0 Ġand ĠB IB REF 7 Ġ, Ġwe Ġdo Ġnot Ġuse Ġa Ġlinking Ġfunction Ġ( e . g . Ġsurpr isal ) Ġto Ġrelate Ġa Ġlanguage Ġmodel Ġto ĠER Ps . ĠWe Ġthus Ġlose Ġthe Ġinterpret ability Ġprovided Ġby Ġthe Ġlinking Ġfunction , Ġbut Ġwe Ġare Ġable Ġto Ġpredict Ġa Ġsignificant Ġproportion Ġof Ġthe Ġvariance Ġfor Ġall Ġof Ġthe ĠER P Ġcomponents , Ġwhere Ġprior Ġwork Ġcould Ġnot . ĠWe Ġinterpret Ġour Ġresults Ġthrough Ġcharacterization Ġof Ġthe ĠER Ps Ġin Ġterms Ġof Ġhow Ġthey Ġrelate Ġto Ġeach Ġother Ġand Ġto Ġeye - tracking Ġdata Ġrather Ġthan Ġthrough Ġa Ġlinking Ġfunction . ĠThe Ġauthors Ġin ĠB IB REF 8 Ġalso Ġuse Ġa Ġrecurrent Ġneural Ġnetwork Ġto Ġpredict Ġneural Ġactivity Ġdirectly . ĠIn Ġthat Ġwork Ġthe Ġauthors Ġpredict Ġmagnet o ence phal ography Ġ( M EG ) Ġactivity , Ġa Ġclose Ġcousin Ġto ĠEEG , Ġrecorded Ġwhile Ġparticipants Ġread Ġa Ġchapter Ġof ĠHarry ĠPotter Ġand Ġthe ĠSorcerer âĢ Ļ s ĠStone ĠB IB REF 9 Ġ. ĠTheir Ġapproach Ġto Ġcharacterization Ġof Ġprocessing Ġat Ġeach ĠM EG Ġsensor Ġlocation Ġis Ġto Ġdetermine Ġwhether Ġit Ġis Ġbest Ġpredicted Ġby Ġthe Ġcontext Ġvector Ġof Ġthe Ġrecurrent Ġnetwork Ġ( pri or Ġto Ġthe Ġcurrent Ġword Ġbeing Ġprocessed ), Ġthe Ġembed ding Ġof Ġthe Ġcurrent Ġword , Ġor Ġthe Ġprobability Ġof Ġthe Ġcurrent Ġword Ġgiven Ġthe Ġcontext . ĠIn Ġfuture Ġwork Ġwe Ġalso Ġintend Ġto Ġadd Ġthese Ġtypes Ġof Ġstudies Ġto Ġthe ĠER P Ġpredictions . </s> Discussion </s> In Ġthis Ġwork Ġwe Ġfind Ġthat Ġall Ġsix Ġof Ġthe ĠER P Ġcomponents Ġfrom ĠB IB REF 0 Ġcan Ġbe Ġpredicted Ġabove Ġchance Ġby Ġa Ġmodel Ġwhich Ġhas Ġbeen Ġpret rained Ġusing Ġa Ġlanguage Ġmodeling Ġobjective Ġand Ġthen Ġdirectly Ġtrained Ġto Ġpredict Ġthe Ġcomponents . ĠThis Ġis Ġin Ġcontrast Ġto Ġprior Ġwork Ġwhich Ġhas Ġsuccessfully Ġlinked Ġlanguage Ġmodels Ġto Ġthe ĠN 400 ĠB IB REF 0 Ġand ĠP 600 ĠB IB REF 7 Ġbut Ġnot Ġthe Ġother ĠER P Ġcomponents . ĠWe Ġalso Ġnote Ġthat Ġcontrary Ġto ĠB IB REF 7 Ġ, Ġwe Ġfind Ġthat Ġan ĠL ST M Ġdoes Ġcontain Ġinformation Ġthat Ġcan Ġbe Ġused Ġto Ġpredict ĠEEG Ġdata , Ġand Ġin Ġparticular Ġthat Ġit Ġcan Ġpredict Ġthe ĠP 600 . ĠWe Ġspeculate Ġthat Ġthe Ġanalysis Ġused Ġin ĠB IB REF 7 Ġdid Ġnot Ġfind Ġreliable Ġeffects Ġbecause Ġthe Ġlanguage Ġmodels Ġwere Ġrelated Ġto Ġthe ĠEEG Ġdata Ġthrough Ġfunctions Ġchosen Ġa Ġprior i Ġ( the Ġsurpr isal , Ġand Ġthe Ġ` distance ' Ġmetric ). ĠThese Ġfunctions , Ġthough Ġinterpret able , Ġmight Ġbe Ġinterpret able Ġat Ġthe Ġcost Ġof Ġlosing Ġmuch Ġof Ġthe Ġinformation Ġin Ġthe Ġrepresentations Ġlearned Ġby Ġthe Ġnetwork . </s> In Ġaddition , Ġwe Ġshow Ġthrough Ġour Ġmultit ask Ġlearning Ġanalysis Ġthat Ġinformation Ġis Ġshared Ġbetween ĠER P Ġcomponents , Ġand Ġbetween ĠER P Ġcomponents Ġand Ġbehavioral Ġdata . ĠAlthough Ġthese Ġrelationships Ġmust Ġbe Ġviewed Ġwith Ġcaution Ġuntil Ġthey Ġcan Ġbe Ġverified Ġacross Ġmultiple Ġdatasets Ġand Ġwith Ġmore Ġvariation Ġin Ġneural Ġnetwork Ġarchitectures , Ġhere Ġwe Ġconsider Ġsome Ġpotential Ġreasons Ġfor Ġour Ġfindings . ĠThe Ġbroad Ġpoint Ġwe Ġwish Ġto Ġmake Ġis Ġthat Ġby Ġbetter Ġunderstanding Ġwhich ĠER P Ġcomponents Ġshare Ġinformation Ġwith Ġeach Ġother Ġand Ġwith Ġbehavioral Ġdata Ġthrough Ġthe Ġtype Ġof Ġanalysis Ġwe Ġpresent Ġhere Ġ( mult it ask Ġlearning ) Ġor Ġother Ġmeans , Ġwe Ġcan Ġbetter Ġunderstand Ġwhat Ġdrives Ġeach ĠER P Ġcomponent Ġand Ġin Ġturn Ġthe Ġprocesses Ġinvolved Ġin Ġhuman Ġlanguage Ġcomprehension . </s> Conclusion </s> We Ġhave Ġshown Ġthat ĠER P Ġcomponents Ġcan Ġbe Ġpredicted Ġfrom Ġneural Ġnetworks Ġpret rained Ġas Ġlanguage Ġmodels Ġand Ġfine - tun ed Ġto Ġdirectly Ġpredict Ġthose Ġcomponents . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġprior Ġwork Ġhas Ġnot Ġsuccessfully Ġused Ġstatistical Ġmodels Ġto Ġpredict Ġall Ġof Ġthese Ġcomponents . ĠFurthermore , Ġwe Ġhave Ġshown Ġthat Ġmultit ask Ġlearning Ġbenefits Ġthe Ġprediction Ġof ĠER P Ġcomponents Ġand Ġcan Ġsuggest Ġhow Ġcomponents Ġrelate Ġto Ġeach Ġother . ĠAt Ġpresent , Ġthese Ġjoint - training Ġbenefit Ġrelationships Ġare Ġonly Ġsuggestive , Ġbut Ġif Ġthese Ġrelationships Ġultimately Ġlead Ġto Ġinsights Ġabout Ġwhat Ġdrives Ġeach ĠER P Ġcomponent , Ġthen Ġthe Ġcomponents Ġbecome Ġmore Ġuseful Ġtools Ġfor Ġstudying Ġhuman Ġlanguage Ġcomprehension . ĠBy Ġusing Ġmultit ask Ġlearning Ġas Ġa Ġmethod Ġof Ġcharacterization , Ġwe Ġhave Ġfound Ġsome Ġexpected Ġrelationships Ġ( LAN + P 600 Ġand ĠEL AN + P 600 ) Ġand Ġseveral Ġmore Ġsurprising Ġrelationships . ĠWe Ġbelieve Ġthat Ġthis Ġis Ġexactly Ġthe Ġkind Ġof Ġfinding Ġthat Ġmakes Ġmultit ask Ġlearning Ġan Ġinteresting Ġexplor atory Ġtechnique Ġin Ġthis Ġarea . ĠAdditionally , Ġwe Ġhave Ġshown Ġthat Ġinformation Ġcan Ġbe Ġshared Ġbetween Ġheter ogeneous Ġtypes Ġof Ġdata Ġ( eye - tracking , Ġself - paced Ġreading , Ġand ĠER P Ġcomponents ) Ġin Ġthe Ġdomain Ġof Ġhuman Ġlanguage Ġprocessing Ġprediction , Ġand Ġin Ġparticular Ġbetween Ġbehavioral Ġand Ġneural Ġdata . ĠGiven Ġthe Ġsmall Ġdatasets Ġassociated Ġwith Ġhuman Ġlanguage Ġprocessing , Ġusing Ġheter ogeneous Ġdata Ġis Ġa Ġpotentially Ġmajor Ġadvantage Ġof Ġa Ġmultit ask Ġapproach . ĠIn Ġfuture Ġwork , Ġwe Ġwill Ġfurther Ġexplore Ġwhat Ġinformation Ġis Ġencoded Ġinto Ġthe Ġmodel Ġrepresentations Ġwhen Ġneural Ġand Ġbehavioral Ġdata Ġare Ġused Ġto Ġtrain Ġneural Ġnetworks , Ġand Ġhow Ġthese Ġrepresentations Ġdiffer Ġfrom Ġthe Ġrepresentations Ġin Ġa Ġmodel Ġtrained Ġon Ġlanguage Ġalone . </s> Acknowled gments </s> We Ġthank Ġour Ġreviewers Ġfor Ġtheir Ġvaluable Ġfeedback . ĠThis Ġwork Ġis Ġsupported Ġin Ġpart Ġby ĠNational ĠInstitutes Ġof ĠHealth Ġgrant Ġnumber ĠU 01 NS 09 89 69 . </s> App endix </s> Here Ġwe Ġpresent Ġa Ġvisualization Ġ( Figure ĠFIG REF 21 Ġ) Ġof Ġthe Ġresults Ġpresented Ġin ĠTable ĠT AB REF 9 Ġof Ġthe Ġmain Ġpaper , Ġand Ġa Ġvisualization Ġ( Figure ĠFIG REF 22 Ġ) Ġof Ġa Ġmore Ġcomplete Ġset Ġof Ġresults Ġfrom Ġwhich Ġthe Ġinformation Ġin ĠTable ĠT AB REF 16 Ġof Ġthe Ġmain Ġpaper Ġis Ġdrawn . ĠWe Ġalso Ġshow Ġsupplemental Ġresults Ġfor Ġvariants Ġof Ġour Ġprimary Ġanalysis Ġon Ġmultit ask Ġlearning Ġwith Ġeye - tracking , Ġself - paced Ġreading Ġtime Ġand ĠER P Ġdata . ĠIn Ġthe Ġvariants Ġwe Ġmodify Ġthe Ġinput Ġrepresentation Ġto Ġour Ġdec oder Ġnetwork Ġto Ġsee Ġwhether Ġthe Ġrelationships Ġbetween Ġthe Ġbehavioral Ġdata Ġand Ġneural Ġactivity Ġappear Ġto Ġbe Ġconsistent Ġwith Ġdifferent Ġchoices Ġof Ġenc oder Ġarchitectures . ĠAdditional Ġ( and Ġmore Ġvaried ) Ġchoices Ġor Ġarchitectures Ġare Ġleft Ġto Ġfuture Ġwork . ĠThe Ġresults Ġin ĠTable ĠT AB REF 23 Ġreflect Ġusing Ġonly Ġthe Ġforward - enc oder Ġ( rather Ġthan Ġthe Ġbi - L ST M ) Ġin Ġthe Ġenc oder Ġnetwork , Ġwhile Ġthe Ġresults Ġin ĠTable ĠT AB REF 24 Ġreflect Ġusing Ġonly Ġthe Ġword Ġembed d ings Ġ( i . e . Ġbypass ing Ġthe ĠL ST M Ġentirely ). ĠWhile Ġthe Ġresults Ġare Ġclearly Ġworse Ġfor Ġeach Ġof Ġthese Ġchoices Ġof Ġarchitecture Ġthan Ġfor Ġusing Ġa Ġbi - L ST M Ġenc oder , Ġthe Ġrelationships Ġbetween Ġthe Ġbehavioral Ġdata Ġand Ġthe ĠER P Ġsignals Ġis Ġqual itatively Ġsimilar . ĠFinally , ĠT AB REF 25 Ġshows Ġthe ĠPearson Ġcorrelation Ġcoefficient Ġbetween Ġdifferent Ġmeasures . ĠWe Ġnote Ġthat Ġthe Ġpatterns Ġof Ġcorrelation Ġare Ġdifferent Ġthan Ġthe Ġpatterns Ġof Ġwhich Ġmeasures Ġbenefit Ġfrom Ġjoint Ġtraining Ġwith Ġeach Ġother .
Introduction The cognitive processes involved in human language comprehension are complex and only partially identified. According to the dual-stream model of speech comprehension BIBREF1 , sound waves are first converted to phoneme-like features and further processed by a ventral stream that maps those features onto words and semantic structures, and a dorsal stream that (among other things) supports audio-short term memory. The mapping of words onto meaning is thought to be subserved by widely distributed regions of the brain that specialize in particular modalities — for example visual aspects of the word banana reside in the occipital lobe of the brain and are activated when the word banana is heard BIBREF2 — and the different representation modalities are thought to be integrated into a single coherent latent representation in the anterior temporal lobe BIBREF3 . While this part of meaning representation in human language comprehension is somewhat understood, much less is known about how the meanings of words are integrated together to form the meaning of sentences and discourses. One tool researchers use to study the integration of meaning across words is electroencephelography (EEG), which measures the electrical activity of large numbers of neurons acting in concert. EEG has the temporal resolution necessary to study the processes involved in meaning integration, and certain stereotyped electrical responses to word presentations, known as event-related potentials (ERPs), have been identified with some of the processes thought to contribute to comprehension. In this work, we consider six ERP components that have been associated in the cognitive neuroscience and psycholinguistics literature with language processing and which we analyze in the data from BIBREF0 (see Figure FIGREF1 for spatial and temporal definitions of these ERP components). Three of these — the N400, EPNP, and PNP responses — are primarily considered markers for semantic processing, while the other three — the P600, ELAN, and LAN responses — are primarily considered markers for syntactic processing. However, the neat division of the ERP responses into either semantic or syntactic categories is controversial. The N400 response has been very well studied (for an overview see BIBREF4 ) and it is well established that it is associated with semantic complexity, but the features of language that trigger the other ERP responses we consider here are poorly understood. We propose to use a neural network pretrained as a language model to probe what features of language drive these ERP responses, and in turn to probe what features of language mediate the cognitive processes that underlie human language comprehension, and especially the integration of meaning across words. Background While a full discussion of each ERP component and the features of language thought to trigger each are beyond the scope of this document (for reviews see e.g. BIBREF0 , BIBREF2 , BIBREF4 , BIBREF5 , and BIBREF6 ), we introduce some basic features of ERP components to help in the discussion later. ERP components are electrical potential responses measured with respect to a baseline that are triggered by an event (in our case the presentation of a new word to a participant in an experiment). The name of each ERP component reflects whether the potential is positive or negative relative to the baseline. The N400 is so-named because it is Negative relative to a baseline (the baseline is typically recorded just before a word is presented at an electrode that is not affected by the ERP response) and because it peaks in magnitude at about 400ms after a word is presented to a participant in an experiment. The P600 is Positive relative to a baseline and peaks around 600ms after a word is presented to a participant (though its overall duration is much longer and less specific in time than the N400). The post-N400 positivity is so-named because it is part of a biphasic response; it is a positivity that occurs after the negativity associated with the N400. The early post-N400 positivity (EPNP) is also part of a biphasic response, but the positivity has an eariler onset than the standard PNP. Finally, the LAN and ELAN are the left-anterior negativity and early left-anterior negativity respectively. These are named for their timing, spatial distribution on the scalp, and direction of difference from the baseline. It is important to note that ERP components can potentially cancel and mask each other, and that it is difficult to precisely localize the neural activity that causes the changes in electrical potential at the electrodes where those changes are measured. Related Work This work is most closely related to the paper from which we get the ERP data: BIBREF0 . In that work, the authors relate the surprisal of a word, i.e. the (negative log) probability of the word appearing in its context, to each of the ERP signals we consider here. The authors do not directly train a model to predict ERPs. Instead, models of the probability distribution of each word in context are used to compute a surprisal for each word, which is input into a mixed effects regression along with word frequency, word length, word position in the sentence, and sentence position in the experiment. The effect of the surprisal is assessed using a likelihood-ratio test. In BIBREF7 , the authors take an approach similar to BIBREF0 . The authors compare the explanatory power of surprisal (as computed by an LSTM or a Recurrent Neural Network Grammar (RNNG) language model) to a measure of syntactic complexity they call “distance" that counts the number of parser actions in the RNNG language model. The authors find that surprisal (as predicted by the RNNG) and distance are both significant factors in a mixed effects regression which predicts the P600, while the surprisal as computed by an LSTM is not. Unlike BIBREF0 and BIBREF7 , we do not use a linking function (e.g. surprisal) to relate a language model to ERPs. We thus lose the interpretability provided by the linking function, but we are able to predict a significant proportion of the variance for all of the ERP components, where prior work could not. We interpret our results through characterization of the ERPs in terms of how they relate to each other and to eye-tracking data rather than through a linking function. The authors in BIBREF8 also use a recurrent neural network to predict neural activity directly. In that work the authors predict magnetoencephalography (MEG) activity, a close cousin to EEG, recorded while participants read a chapter of Harry Potter and the Sorcerer’s Stone BIBREF9 . Their approach to characterization of processing at each MEG sensor location is to determine whether it is best predicted by the context vector of the recurrent network (prior to the current word being processed), the embedding of the current word, or the probability of the current word given the context. In future work we also intend to add these types of studies to the ERP predictions. Discussion In this work we find that all six of the ERP components from BIBREF0 can be predicted above chance by a model which has been pretrained using a language modeling objective and then directly trained to predict the components. This is in contrast to prior work which has successfully linked language models to the N400 BIBREF0 and P600 BIBREF7 but not the other ERP components. We also note that contrary to BIBREF7 , we find that an LSTM does contain information that can be used to predict EEG data, and in particular that it can predict the P600. We speculate that the analysis used in BIBREF7 did not find reliable effects because the language models were related to the EEG data through functions chosen a priori (the surprisal, and the `distance' metric). These functions, though interpretable, might be interpretable at the cost of losing much of the information in the representations learned by the network. In addition, we show through our multitask learning analysis that information is shared between ERP components, and between ERP components and behavioral data. Although these relationships must be viewed with caution until they can be verified across multiple datasets and with more variation in neural network architectures, here we consider some potential reasons for our findings. The broad point we wish to make is that by better understanding which ERP components share information with each other and with behavioral data through the type of analysis we present here (multitask learning) or other means, we can better understand what drives each ERP component and in turn the processes involved in human language comprehension. Conclusion We have shown that ERP components can be predicted from neural networks pretrained as language models and fine-tuned to directly predict those components. To the best of our knowledge, prior work has not successfully used statistical models to predict all of these components. Furthermore, we have shown that multitask learning benefits the prediction of ERP components and can suggest how components relate to each other. At present, these joint-training benefit relationships are only suggestive, but if these relationships ultimately lead to insights about what drives each ERP component, then the components become more useful tools for studying human language comprehension. By using multitask learning as a method of characterization, we have found some expected relationships (LAN+P600 and ELAN+P600) and several more surprising relationships. We believe that this is exactly the kind of finding that makes multitask learning an interesting exploratory technique in this area. Additionally, we have shown that information can be shared between heterogeneous types of data (eye-tracking, self-paced reading, and ERP components) in the domain of human language processing prediction, and in particular between behavioral and neural data. Given the small datasets associated with human language processing, using heterogeneous data is a potentially major advantage of a multitask approach. In future work, we will further explore what information is encoded into the model representations when neural and behavioral data are used to train neural networks, and how these representations differ from the representations in a model trained on language alone. Acknowledgments We thank our reviewers for their valuable feedback. This work is supported in part by National Institutes of Health grant number U01NS098969. Appendix Here we present a visualization (Figure FIGREF21 ) of the results presented in Table TABREF9 of the main paper, and a visualization (Figure FIGREF22 ) of a more complete set of results from which the information in Table TABREF16 of the main paper is drawn. We also show supplemental results for variants of our primary analysis on multitask learning with eye-tracking, self-paced reading time and ERP data. In the variants we modify the input representation to our decoder network to see whether the relationships between the behavioral data and neural activity appear to be consistent with different choices of encoder architectures. Additional (and more varied) choices or architectures are left to future work. The results in Table TABREF23 reflect using only the forward-encoder (rather than the bi-LSTM) in the encoder network, while the results in Table TABREF24 reflect using only the word embeddings (i.e. bypassing the LSTM entirely). While the results are clearly worse for each of these choices of architecture than for using a bi-LSTM encoder, the relationships between the behavioral data and the ERP signals is qualitatively similar. Finally, TABREF25 shows the Pearson correlation coefficient between different measures. We note that the patterns of correlation are different than the patterns of which measures benefit from joint training with each other.
Which two pairs of ERPs from the literature benefit from joint training?
7d2f812cb345bb3ab91eb8cbbdeefd4b58f65569
Which Ġtwo Ġpairs Ġof ĠER Ps Ġfrom Ġthe Ġliterature Ġbenefit Ġfrom Ġjoint Ġtraining ?
Answer Ġwith Ġcontent Ġmissing : Ġ( Wh ole ĠMethod Ġand ĠResults Ġsections ) ĠSelf - paced Ġreading Ġtimes Ġwidely Ġbenefit ĠER P Ġprediction , Ġwhile Ġeye - tracking Ġdata Ġseems Ġto Ġhave Ġmore Ġlimited Ġbenefit Ġto Ġjust Ġthe ĠEL AN , ĠLAN , Ġand ĠP NP ĠER P Ġcomponents . Ċ Select : Ċ - ĠEL AN , ĠLAN Ċ - ĠP NP ĠER P
<s> Which Ġtwo Ġpairs Ġof ĠER Ps Ġfrom Ġthe Ġliterature Ġbenefit Ġfrom Ġjoint Ġtraining ? </s> Introduction </s> The Ġcognitive Ġprocesses Ġinvolved Ġin Ġhuman Ġlanguage Ġcomprehension Ġare Ġcomplex Ġand Ġonly Ġpartially Ġidentified . ĠAccording Ġto Ġthe Ġdual - stream Ġmodel Ġof Ġspeech Ġcomprehension ĠB IB REF 1 Ġ, Ġsound Ġwaves Ġare Ġfirst Ġconverted Ġto Ġphon eme - like Ġfeatures Ġand Ġfurther Ġprocessed Ġby Ġa Ġvent ral Ġstream Ġthat Ġmaps Ġthose Ġfeatures Ġonto Ġwords Ġand Ġsemantic Ġstructures , Ġand Ġa Ġdorsal Ġstream Ġthat Ġ( among Ġother Ġthings ) Ġsupports Ġaudio - short Ġterm Ġmemory . ĠThe Ġmapping Ġof Ġwords Ġonto Ġmeaning Ġis Ġthought Ġto Ġbe Ġsub served Ġby Ġwidely Ġdistributed Ġregions Ġof Ġthe Ġbrain Ġthat Ġspecialize Ġin Ġparticular Ġmod alities ĠâĢĶ Ġfor Ġexample Ġvisual Ġaspects Ġof Ġthe Ġword Ġbanana Ġreside Ġin Ġthe Ġocc ip ital Ġlobe Ġof Ġthe Ġbrain Ġand Ġare Ġactivated Ġwhen Ġthe Ġword Ġbanana Ġis Ġheard ĠB IB REF 2 ĠâĢĶ Ġand Ġthe Ġdifferent Ġrepresentation Ġmod alities Ġare Ġthought Ġto Ġbe Ġintegrated Ġinto Ġa Ġsingle Ġcoherent Ġlatent Ġrepresentation Ġin Ġthe Ġanterior Ġtemporal Ġlobe ĠB IB REF 3 Ġ. ĠWhile Ġthis Ġpart Ġof Ġmeaning Ġrepresentation Ġin Ġhuman Ġlanguage Ġcomprehension Ġis Ġsomewhat Ġunderstood , Ġmuch Ġless Ġis Ġknown Ġabout Ġhow Ġthe Ġmeanings Ġof Ġwords Ġare Ġintegrated Ġtogether Ġto Ġform Ġthe Ġmeaning Ġof Ġsentences Ġand Ġdisc ourses . ĠOne Ġtool Ġresearchers Ġuse Ġto Ġstudy Ġthe Ġintegration Ġof Ġmeaning Ġacross Ġwords Ġis Ġelectro ence p hel ography Ġ( EE G ), Ġwhich Ġmeasures Ġthe Ġelectrical Ġactivity Ġof Ġlarge Ġnumbers Ġof Ġneurons Ġacting Ġin Ġconcert . ĠEEG Ġhas Ġthe Ġtemporal Ġresolution Ġnecessary Ġto Ġstudy Ġthe Ġprocesses Ġinvolved Ġin Ġmeaning Ġintegration , Ġand Ġcertain Ġstereotyp ed Ġelectrical Ġresponses Ġto Ġword Ġpresentations , Ġknown Ġas Ġevent - related Ġpotential s Ġ( ER Ps ), Ġhave Ġbeen Ġidentified Ġwith Ġsome Ġof Ġthe Ġprocesses Ġthought Ġto Ġcontribute Ġto Ġcomprehension . </s> In Ġthis Ġwork , Ġwe Ġconsider Ġsix ĠER P Ġcomponents Ġthat Ġhave Ġbeen Ġassociated Ġin Ġthe Ġcognitive Ġneuroscience Ġand Ġpsych ol ingu istics Ġliterature Ġwith Ġlanguage Ġprocessing Ġand Ġwhich Ġwe Ġanalyze Ġin Ġthe Ġdata Ġfrom ĠB IB REF 0 Ġ( see ĠFigure ĠFIG REF 1 Ġfor Ġspatial Ġand Ġtemporal Ġdefinitions Ġof Ġthese ĠER P Ġcomponents ). ĠThree Ġof Ġthese ĠâĢĶ Ġthe ĠN 400 , ĠE PN P , Ġand ĠP NP Ġresponses ĠâĢĶ Ġare Ġprimarily Ġconsidered Ġmarkers Ġfor Ġsemantic Ġprocessing , Ġwhile Ġthe Ġother Ġthree ĠâĢĶ Ġthe ĠP 600 , ĠEL AN , Ġand ĠLAN Ġresponses ĠâĢĶ Ġare Ġprimarily Ġconsidered Ġmarkers Ġfor Ġsynt actic Ġprocessing . ĠHowever , Ġthe Ġneat Ġdivision Ġof Ġthe ĠER P Ġresponses Ġinto Ġeither Ġsemantic Ġor Ġsynt actic Ġcategories Ġis Ġcontroversial . ĠThe ĠN 400 Ġresponse Ġhas Ġbeen Ġvery Ġwell Ġstudied Ġ( for Ġan Ġoverview Ġsee ĠB IB REF 4 Ġ) Ġand Ġit Ġis Ġwell Ġestablished Ġthat Ġit Ġis Ġassociated Ġwith Ġsemantic Ġcomplexity , Ġbut Ġthe Ġfeatures Ġof Ġlanguage Ġthat Ġtrigger Ġthe Ġother ĠER P Ġresponses Ġwe Ġconsider Ġhere Ġare Ġpoorly Ġunderstood . ĠWe Ġpropose Ġto Ġuse Ġa Ġneural Ġnetwork Ġpret rained Ġas Ġa Ġlanguage Ġmodel Ġto Ġprobe Ġwhat Ġfeatures Ġof Ġlanguage Ġdrive Ġthese ĠER P Ġresponses , Ġand Ġin Ġturn Ġto Ġprobe Ġwhat Ġfeatures Ġof Ġlanguage Ġmed iate Ġthe Ġcognitive Ġprocesses Ġthat Ġunder lie Ġhuman Ġlanguage Ġcomprehension , Ġand Ġespecially Ġthe Ġintegration Ġof Ġmeaning Ġacross Ġwords . </s> Background </s> While Ġa Ġfull Ġdiscussion Ġof Ġeach ĠER P Ġcomponent Ġand Ġthe Ġfeatures Ġof Ġlanguage Ġthought Ġto Ġtrigger Ġeach Ġare Ġbeyond Ġthe Ġscope Ġof Ġthis Ġdocument Ġ( for Ġreviews Ġsee Ġe . g . ĠB IB REF 0 Ġ, ĠB IB REF 2 Ġ, ĠB IB REF 4 Ġ, ĠB IB REF 5 Ġ, Ġand ĠB IB REF 6 Ġ), Ġwe Ġintroduce Ġsome Ġbasic Ġfeatures Ġof ĠER P Ġcomponents Ġto Ġhelp Ġin Ġthe Ġdiscussion Ġlater . ĠER P Ġcomponents Ġare Ġelectrical Ġpotential Ġresponses Ġmeasured Ġwith Ġrespect Ġto Ġa Ġbaseline Ġthat Ġare Ġtriggered Ġby Ġan Ġevent Ġ( in Ġour Ġcase Ġthe Ġpresentation Ġof Ġa Ġnew Ġword Ġto Ġa Ġparticipant Ġin Ġan Ġexperiment ). ĠThe Ġname Ġof Ġeach ĠER P Ġcomponent Ġreflects Ġwhether Ġthe Ġpotential Ġis Ġpositive Ġor Ġnegative Ġrelative Ġto Ġthe Ġbaseline . ĠThe ĠN 400 Ġis Ġso - named Ġbecause Ġit Ġis ĠNegative Ġrelative Ġto Ġa Ġbaseline Ġ( the Ġbaseline Ġis Ġtypically Ġrecorded Ġjust Ġbefore Ġa Ġword Ġis Ġpresented Ġat Ġan Ġelectrode Ġthat Ġis Ġnot Ġaffected Ġby Ġthe ĠER P Ġresponse ) Ġand Ġbecause Ġit Ġpeaks Ġin Ġmagnitude Ġat Ġabout Ġ400 ms Ġafter Ġa Ġword Ġis Ġpresented Ġto Ġa Ġparticipant Ġin Ġan Ġexperiment . ĠThe ĠP 600 Ġis ĠPositive Ġrelative Ġto Ġa Ġbaseline Ġand Ġpeaks Ġaround Ġ600 ms Ġafter Ġa Ġword Ġis Ġpresented Ġto Ġa Ġparticipant Ġ( though Ġits Ġoverall Ġduration Ġis Ġmuch Ġlonger Ġand Ġless Ġspecific Ġin Ġtime Ġthan Ġthe ĠN 400 ). ĠThe Ġpost - N 400 Ġpos itivity Ġis Ġso - named Ġbecause Ġit Ġis Ġpart Ġof Ġa Ġbip has ic Ġresponse ; Ġit Ġis Ġa Ġpos itivity Ġthat Ġoccurs Ġafter Ġthe Ġnegativity Ġassociated Ġwith Ġthe ĠN 400 . ĠThe Ġearly Ġpost - N 400 Ġpos itivity Ġ( EP NP ) Ġis Ġalso Ġpart Ġof Ġa Ġbip has ic Ġresponse , Ġbut Ġthe Ġpos itivity Ġhas Ġan Ġear iler Ġonset Ġthan Ġthe Ġstandard ĠP NP . ĠFinally , Ġthe ĠLAN Ġand ĠEL AN Ġare Ġthe Ġleft - an terior Ġnegativity Ġand Ġearly Ġleft - an terior Ġnegativity Ġrespectively . ĠThese Ġare Ġnamed Ġfor Ġtheir Ġtiming , Ġspatial Ġdistribution Ġon Ġthe Ġscalp , Ġand Ġdirection Ġof Ġdifference Ġfrom Ġthe Ġbaseline . ĠIt Ġis Ġimportant Ġto Ġnote Ġthat ĠER P Ġcomponents Ġcan Ġpotentially Ġcancel Ġand Ġmask Ġeach Ġother , Ġand Ġthat Ġit Ġis Ġdifficult Ġto Ġprecisely Ġlocal ize Ġthe Ġneural Ġactivity Ġthat Ġcauses Ġthe Ġchanges Ġin Ġelectrical Ġpotential Ġat Ġthe Ġelectrodes Ġwhere Ġthose Ġchanges Ġare Ġmeasured . </s> Related ĠWork </s> This Ġwork Ġis Ġmost Ġclosely Ġrelated Ġto Ġthe Ġpaper Ġfrom Ġwhich Ġwe Ġget Ġthe ĠER P Ġdata : ĠB IB REF 0 Ġ. ĠIn Ġthat Ġwork , Ġthe Ġauthors Ġrelate Ġthe Ġsurpr isal Ġof Ġa Ġword , Ġi . e . Ġthe Ġ( negative Ġlog ) Ġprobability Ġof Ġthe Ġword Ġappearing Ġin Ġits Ġcontext , Ġto Ġeach Ġof Ġthe ĠER P Ġsignals Ġwe Ġconsider Ġhere . ĠThe Ġauthors Ġdo Ġnot Ġdirectly Ġtrain Ġa Ġmodel Ġto Ġpredict ĠER Ps . ĠInstead , Ġmodels Ġof Ġthe Ġprobability Ġdistribution Ġof Ġeach Ġword Ġin Ġcontext Ġare Ġused Ġto Ġcompute Ġa Ġsurpr isal Ġfor Ġeach Ġword , Ġwhich Ġis Ġinput Ġinto Ġa Ġmixed Ġeffects Ġregression Ġalong Ġwith Ġword Ġfrequency , Ġword Ġlength , Ġword Ġposition Ġin Ġthe Ġsentence , Ġand Ġsentence Ġposition Ġin Ġthe Ġexperiment . ĠThe Ġeffect Ġof Ġthe Ġsurpr isal Ġis Ġassessed Ġusing Ġa Ġlikelihood - rat io Ġtest . ĠIn ĠB IB REF 7 Ġ, Ġthe Ġauthors Ġtake Ġan Ġapproach Ġsimilar Ġto ĠB IB REF 0 Ġ. ĠThe Ġauthors Ġcompare Ġthe Ġexplanatory Ġpower Ġof Ġsurpr isal Ġ( as Ġcomputed Ġby Ġan ĠL ST M Ġor Ġa ĠRec urrent ĠNeural ĠNetwork ĠGram mar Ġ( R NN G ) Ġlanguage Ġmodel ) Ġto Ġa Ġmeasure Ġof Ġsynt actic Ġcomplexity Ġthey Ġcall ĠâĢ ľ distance " Ġthat Ġcounts Ġthe Ġnumber Ġof Ġparser Ġactions Ġin Ġthe ĠR NN G Ġlanguage Ġmodel . ĠThe Ġauthors Ġfind Ġthat Ġsurpr isal Ġ( as Ġpredicted Ġby Ġthe ĠR NN G ) Ġand Ġdistance Ġare Ġboth Ġsignificant Ġfactors Ġin Ġa Ġmixed Ġeffects Ġregression Ġwhich Ġpredicts Ġthe ĠP 600 , Ġwhile Ġthe Ġsurpr isal Ġas Ġcomputed Ġby Ġan ĠL ST M Ġis Ġnot . ĠUnlike ĠB IB REF 0 Ġand ĠB IB REF 7 Ġ, Ġwe Ġdo Ġnot Ġuse Ġa Ġlinking Ġfunction Ġ( e . g . Ġsurpr isal ) Ġto Ġrelate Ġa Ġlanguage Ġmodel Ġto ĠER Ps . ĠWe Ġthus Ġlose Ġthe Ġinterpret ability Ġprovided Ġby Ġthe Ġlinking Ġfunction , Ġbut Ġwe Ġare Ġable Ġto Ġpredict Ġa Ġsignificant Ġproportion Ġof Ġthe Ġvariance Ġfor Ġall Ġof Ġthe ĠER P Ġcomponents , Ġwhere Ġprior Ġwork Ġcould Ġnot . ĠWe Ġinterpret Ġour Ġresults Ġthrough Ġcharacterization Ġof Ġthe ĠER Ps Ġin Ġterms Ġof Ġhow Ġthey Ġrelate Ġto Ġeach Ġother Ġand Ġto Ġeye - tracking Ġdata Ġrather Ġthan Ġthrough Ġa Ġlinking Ġfunction . ĠThe Ġauthors Ġin ĠB IB REF 8 Ġalso Ġuse Ġa Ġrecurrent Ġneural Ġnetwork Ġto Ġpredict Ġneural Ġactivity Ġdirectly . ĠIn Ġthat Ġwork Ġthe Ġauthors Ġpredict Ġmagnet o ence phal ography Ġ( M EG ) Ġactivity , Ġa Ġclose Ġcousin Ġto ĠEEG , Ġrecorded Ġwhile Ġparticipants Ġread Ġa Ġchapter Ġof ĠHarry ĠPotter Ġand Ġthe ĠSorcerer âĢ Ļ s ĠStone ĠB IB REF 9 Ġ. ĠTheir Ġapproach Ġto Ġcharacterization Ġof Ġprocessing Ġat Ġeach ĠM EG Ġsensor Ġlocation Ġis Ġto Ġdetermine Ġwhether Ġit Ġis Ġbest Ġpredicted Ġby Ġthe Ġcontext Ġvector Ġof Ġthe Ġrecurrent Ġnetwork Ġ( pri or Ġto Ġthe Ġcurrent Ġword Ġbeing Ġprocessed ), Ġthe Ġembed ding Ġof Ġthe Ġcurrent Ġword , Ġor Ġthe Ġprobability Ġof Ġthe Ġcurrent Ġword Ġgiven Ġthe Ġcontext . ĠIn Ġfuture Ġwork Ġwe Ġalso Ġintend Ġto Ġadd Ġthese Ġtypes Ġof Ġstudies Ġto Ġthe ĠER P Ġpredictions . </s> Discussion </s> In Ġthis Ġwork Ġwe Ġfind Ġthat Ġall Ġsix Ġof Ġthe ĠER P Ġcomponents Ġfrom ĠB IB REF 0 Ġcan Ġbe Ġpredicted Ġabove Ġchance Ġby Ġa Ġmodel Ġwhich Ġhas Ġbeen Ġpret rained Ġusing Ġa Ġlanguage Ġmodeling Ġobjective Ġand Ġthen Ġdirectly Ġtrained Ġto Ġpredict Ġthe Ġcomponents . ĠThis Ġis Ġin Ġcontrast Ġto Ġprior Ġwork Ġwhich Ġhas Ġsuccessfully Ġlinked Ġlanguage Ġmodels Ġto Ġthe ĠN 400 ĠB IB REF 0 Ġand ĠP 600 ĠB IB REF 7 Ġbut Ġnot Ġthe Ġother ĠER P Ġcomponents . ĠWe Ġalso Ġnote Ġthat Ġcontrary Ġto ĠB IB REF 7 Ġ, Ġwe Ġfind Ġthat Ġan ĠL ST M Ġdoes Ġcontain Ġinformation Ġthat Ġcan Ġbe Ġused Ġto Ġpredict ĠEEG Ġdata , Ġand Ġin Ġparticular Ġthat Ġit Ġcan Ġpredict Ġthe ĠP 600 . ĠWe Ġspeculate Ġthat Ġthe Ġanalysis Ġused Ġin ĠB IB REF 7 Ġdid Ġnot Ġfind Ġreliable Ġeffects Ġbecause Ġthe Ġlanguage Ġmodels Ġwere Ġrelated Ġto Ġthe ĠEEG Ġdata Ġthrough Ġfunctions Ġchosen Ġa Ġprior i Ġ( the Ġsurpr isal , Ġand Ġthe Ġ` distance ' Ġmetric ). ĠThese Ġfunctions , Ġthough Ġinterpret able , Ġmight Ġbe Ġinterpret able Ġat Ġthe Ġcost Ġof Ġlosing Ġmuch Ġof Ġthe Ġinformation Ġin Ġthe Ġrepresentations Ġlearned Ġby Ġthe Ġnetwork . </s> In Ġaddition , Ġwe Ġshow Ġthrough Ġour Ġmultit ask Ġlearning Ġanalysis Ġthat Ġinformation Ġis Ġshared Ġbetween ĠER P Ġcomponents , Ġand Ġbetween ĠER P Ġcomponents Ġand Ġbehavioral Ġdata . ĠAlthough Ġthese Ġrelationships Ġmust Ġbe Ġviewed Ġwith Ġcaution Ġuntil Ġthey Ġcan Ġbe Ġverified Ġacross Ġmultiple Ġdatasets Ġand Ġwith Ġmore Ġvariation Ġin Ġneural Ġnetwork Ġarchitectures , Ġhere Ġwe Ġconsider Ġsome Ġpotential Ġreasons Ġfor Ġour Ġfindings . ĠThe Ġbroad Ġpoint Ġwe Ġwish Ġto Ġmake Ġis Ġthat Ġby Ġbetter Ġunderstanding Ġwhich ĠER P Ġcomponents Ġshare Ġinformation Ġwith Ġeach Ġother Ġand Ġwith Ġbehavioral Ġdata Ġthrough Ġthe Ġtype Ġof Ġanalysis Ġwe Ġpresent Ġhere Ġ( mult it ask Ġlearning ) Ġor Ġother Ġmeans , Ġwe Ġcan Ġbetter Ġunderstand Ġwhat Ġdrives Ġeach ĠER P Ġcomponent Ġand Ġin Ġturn Ġthe Ġprocesses Ġinvolved Ġin Ġhuman Ġlanguage Ġcomprehension . </s> Conclusion </s> We Ġhave Ġshown Ġthat ĠER P Ġcomponents Ġcan Ġbe Ġpredicted Ġfrom Ġneural Ġnetworks Ġpret rained Ġas Ġlanguage Ġmodels Ġand Ġfine - tun ed Ġto Ġdirectly Ġpredict Ġthose Ġcomponents . ĠTo Ġthe Ġbest Ġof Ġour Ġknowledge , Ġprior Ġwork Ġhas Ġnot Ġsuccessfully Ġused Ġstatistical Ġmodels Ġto Ġpredict Ġall Ġof Ġthese Ġcomponents . ĠFurthermore , Ġwe Ġhave Ġshown Ġthat Ġmultit ask Ġlearning Ġbenefits Ġthe Ġprediction Ġof ĠER P Ġcomponents Ġand Ġcan Ġsuggest Ġhow Ġcomponents Ġrelate Ġto Ġeach Ġother . ĠAt Ġpresent , Ġthese Ġjoint - training Ġbenefit Ġrelationships Ġare Ġonly Ġsuggestive , Ġbut Ġif Ġthese Ġrelationships Ġultimately Ġlead Ġto Ġinsights Ġabout Ġwhat Ġdrives Ġeach ĠER P Ġcomponent , Ġthen Ġthe Ġcomponents Ġbecome Ġmore Ġuseful Ġtools Ġfor Ġstudying Ġhuman Ġlanguage Ġcomprehension . ĠBy Ġusing Ġmultit ask Ġlearning Ġas Ġa Ġmethod Ġof Ġcharacterization , Ġwe Ġhave Ġfound Ġsome Ġexpected Ġrelationships Ġ( LAN + P 600 Ġand ĠEL AN + P 600 ) Ġand Ġseveral Ġmore Ġsurprising Ġrelationships . ĠWe Ġbelieve Ġthat Ġthis Ġis Ġexactly Ġthe Ġkind Ġof Ġfinding Ġthat Ġmakes Ġmultit ask Ġlearning Ġan Ġinteresting Ġexplor atory Ġtechnique Ġin Ġthis Ġarea . ĠAdditionally , Ġwe Ġhave Ġshown Ġthat Ġinformation Ġcan Ġbe Ġshared Ġbetween Ġheter ogeneous Ġtypes Ġof Ġdata Ġ( eye - tracking , Ġself - paced Ġreading , Ġand ĠER P Ġcomponents ) Ġin Ġthe Ġdomain Ġof Ġhuman Ġlanguage Ġprocessing Ġprediction , Ġand Ġin Ġparticular Ġbetween Ġbehavioral Ġand Ġneural Ġdata . ĠGiven Ġthe Ġsmall Ġdatasets Ġassociated Ġwith Ġhuman Ġlanguage Ġprocessing , Ġusing Ġheter ogeneous Ġdata Ġis Ġa Ġpotentially Ġmajor Ġadvantage Ġof Ġa Ġmultit ask Ġapproach . ĠIn Ġfuture Ġwork , Ġwe Ġwill Ġfurther Ġexplore Ġwhat Ġinformation Ġis Ġencoded Ġinto Ġthe Ġmodel Ġrepresentations Ġwhen Ġneural Ġand Ġbehavioral Ġdata Ġare Ġused Ġto Ġtrain Ġneural Ġnetworks , Ġand Ġhow Ġthese Ġrepresentations Ġdiffer Ġfrom Ġthe Ġrepresentations Ġin Ġa Ġmodel Ġtrained Ġon Ġlanguage Ġalone . </s> Acknowled gments </s> We Ġthank Ġour Ġreviewers Ġfor Ġtheir Ġvaluable Ġfeedback . ĠThis Ġwork Ġis Ġsupported Ġin Ġpart Ġby ĠNational ĠInstitutes Ġof ĠHealth Ġgrant Ġnumber ĠU 01 NS 09 89 69 . </s> App endix </s> Here Ġwe Ġpresent Ġa Ġvisualization Ġ( Figure ĠFIG REF 21 Ġ) Ġof Ġthe Ġresults Ġpresented Ġin ĠTable ĠT AB REF 9 Ġof Ġthe Ġmain Ġpaper , Ġand Ġa Ġvisualization Ġ( Figure ĠFIG REF 22 Ġ) Ġof Ġa Ġmore Ġcomplete Ġset Ġof Ġresults Ġfrom Ġwhich Ġthe Ġinformation Ġin ĠTable ĠT AB REF 16 Ġof Ġthe Ġmain Ġpaper Ġis Ġdrawn . ĠWe Ġalso Ġshow Ġsupplemental Ġresults Ġfor Ġvariants Ġof Ġour Ġprimary Ġanalysis Ġon Ġmultit ask Ġlearning Ġwith Ġeye - tracking , Ġself - paced Ġreading Ġtime Ġand ĠER P Ġdata . ĠIn Ġthe Ġvariants Ġwe Ġmodify Ġthe Ġinput Ġrepresentation Ġto Ġour Ġdec oder Ġnetwork Ġto Ġsee Ġwhether Ġthe Ġrelationships Ġbetween Ġthe Ġbehavioral Ġdata Ġand Ġneural Ġactivity Ġappear Ġto Ġbe Ġconsistent Ġwith Ġdifferent Ġchoices Ġof Ġenc oder Ġarchitectures . ĠAdditional Ġ( and Ġmore Ġvaried ) Ġchoices Ġor Ġarchitectures Ġare Ġleft Ġto Ġfuture Ġwork . ĠThe Ġresults Ġin ĠTable ĠT AB REF 23 Ġreflect Ġusing Ġonly Ġthe Ġforward - enc oder Ġ( rather Ġthan Ġthe Ġbi - L ST M ) Ġin Ġthe Ġenc oder Ġnetwork , Ġwhile Ġthe Ġresults Ġin ĠTable ĠT AB REF 24 Ġreflect Ġusing Ġonly Ġthe Ġword Ġembed d ings Ġ( i . e . Ġbypass ing Ġthe ĠL ST M Ġentirely ). ĠWhile Ġthe Ġresults Ġare Ġclearly Ġworse Ġfor Ġeach Ġof Ġthese Ġchoices Ġof Ġarchitecture Ġthan Ġfor Ġusing Ġa Ġbi - L ST M Ġenc oder , Ġthe Ġrelationships Ġbetween Ġthe Ġbehavioral Ġdata Ġand Ġthe ĠER P Ġsignals Ġis Ġqual itatively Ġsimilar . ĠFinally , ĠT AB REF 25 Ġshows Ġthe ĠPearson Ġcorrelation Ġcoefficient Ġbetween Ġdifferent Ġmeasures . ĠWe Ġnote Ġthat Ġthe Ġpatterns Ġof Ġcorrelation Ġare Ġdifferent Ġthan Ġthe Ġpatterns Ġof Ġwhich Ġmeasures Ġbenefit Ġfrom Ġjoint Ġtraining Ġwith Ġeach Ġother .