7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)
EXTRACTIVE
KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)
EXTRACTIVE
We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw). The dataset consists of 14 participants, with each prompt presented 11 times to each individual. Since our intention is to classify the phonological categories from human thoughts, we discard the facial and audio information and only consider the EEG data corresponding to imagined speech. It is noteworthy that given the mixed nature of EEG signals, it is reportedly challenging to attain a pairwise EEG-phoneme mapping BIBREF18 . In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i.e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels.
We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw). The dataset consists of 14 participants, with each prompt presented 11 times to each individual. Since our intention is to classify the phonological categories from human thoughts, we discard the facial and audio information and only consider the EEG data corresponding to imagined speech. It is noteworthy that given the mixed nature of EEG signals, it is reportedly challenging to attain a pairwise EEG-phoneme mapping BIBREF18 . In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i.e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels.
1904.04358
Introduction </s> Dec oding Ġintended Ġspeech Ġor Ġmotor Ġactivity Ġfrom Ġbrain Ġsignals Ġis Ġone Ġof Ġthe Ġmajor Ġresearch Ġareas Ġin ĠBrain ĠComputer ĠInterface Ġ( BC I ) Ġsystems ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠIn Ġparticular , Ġspeech - related ĠBC I Ġtechnologies Ġattempt Ġto Ġprovide Ġeffective Ġvocal Ġcommunication Ġstrategies Ġfor Ġcontrolling Ġexternal Ġdevices Ġthrough Ġspeech Ġcommands Ġinterpreted Ġfrom Ġbrain Ġsignals ĠB IB REF 2 Ġ. ĠNot Ġonly Ġdo Ġthey Ġprovide Ġneuro - pro st hetic Ġhelp Ġfor Ġpeople Ġwith Ġspeaking Ġdisabilities Ġand Ġneuro - mus cular Ġdisorders Ġlike Ġlocked - in - sy nd rome , Ġnas oph aryn ge al Ġcancer , Ġand Ġam yt otropic Ġlateral Ġsclerosis Ġ( ALS ), Ġbut Ġalso Ġequip Ġpeople Ġwith Ġa Ġbetter Ġmedium Ġto Ġcommunicate Ġand Ġexpress Ġthoughts , Ġthereby Ġimproving Ġthe Ġquality Ġof Ġrehabilitation Ġand Ġclinical Ġneurolog y ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠSuch Ġdevices Ġalso Ġhave Ġapplications Ġin Ġentertainment , Ġpreventive Ġtreatments , Ġpersonal Ġcommunication , Ġgames , Ġetc . ĠFurthermore , ĠBC I Ġtechnologies Ġcan Ġbe Ġutilized Ġin Ġsilent Ġcommunication , Ġas Ġin Ġnoisy Ġenvironments , Ġor Ġsituations Ġwhere Ġany Ġsort Ġof Ġaudio - visual Ġcommunication Ġis Ġinf eas ible . </s> Among Ġthe Ġvarious Ġbrain Ġactivity - monitor ing Ġmod alities Ġin ĠBC I , Ġelectro ence phal ography Ġ( EE G ) ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġhas Ġdemonstrated Ġpromising Ġpotential Ġto Ġdifferentiate Ġbetween Ġvarious Ġbrain Ġactivities Ġthrough Ġmeasurement Ġof Ġrelated Ġelectric Ġfields . ĠEEG Ġis Ġnon - in vasive , Ġportable , Ġlow Ġcost , Ġand Ġprovides Ġsatisfactory Ġtemporal Ġresolution . ĠThis Ġmakes ĠEEG Ġsuitable Ġto Ġrealize ĠBC I Ġsystems . ĠEEG Ġdata , Ġhowever , Ġis Ġchallenging : Ġthese Ġdata Ġare Ġhigh Ġdimensional , Ġhave Ġpoor ĠSN R , Ġand Ġsuffer Ġfrom Ġlow Ġspatial Ġresolution Ġand Ġa Ġmultitude Ġof Ġartifacts . ĠFor Ġthese Ġreasons , Ġit Ġis Ġnot Ġparticularly Ġobvious Ġhow Ġto Ġdecode Ġthe Ġdesired Ġinformation Ġfrom Ġraw ĠEEG Ġsignals . ĠAlthough Ġthe Ġarea Ġof ĠBC I Ġbased Ġspeech Ġintent Ġrecognition Ġhas Ġreceived Ġincreasing Ġattention Ġamong Ġthe Ġresearch Ġcommunity Ġin Ġthe Ġpast Ġfew Ġyears , Ġmost Ġresearch Ġhas Ġfocused Ġon Ġclassification Ġof Ġindividual Ġspeech Ġcategories Ġin Ġterms Ġof Ġdiscrete Ġvow els , Ġphon em es Ġand Ġwords ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠThis Ġincludes Ġcategor ization Ġof Ġimagined ĠEEG Ġsignal Ġinto Ġbinary Ġvowel Ġcategories Ġlike Ġ/ a /, Ġ/ u / Ġand Ġrest ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ; Ġbinary Ġsyll able Ġclasses Ġlike Ġ/ ba / Ġand Ġ/ ku / ĠB IB REF 1 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ; Ġa Ġhandful Ġof Ġcontrol Ġwords Ġlike Ġ' up ', Ġ' down ', Ġ' left ', Ġ' right ' Ġand Ġ' select ' ĠB IB REF 15 Ġor Ġothers Ġlike Ġ' water ', Ġ' help ', Ġ' thanks ', Ġ' food ', Ġ' stop ' ĠB IB REF 13 Ġ, ĠChinese Ġcharacters ĠB IB REF 14 Ġ, Ġetc . ĠSuch Ġworks Ġmostly Ġinvolve Ġtraditional Ġsignal Ġprocessing Ġor Ġmanual Ġfeature Ġhand craft ing Ġalong Ġwith Ġlinear Ġclass ifiers Ġ( e . g ., ĠSV Ms ). ĠIn Ġour Ġrecent Ġwork ĠB IB REF 16 Ġ, Ġwe Ġintroduced Ġdeep Ġlearning Ġmodels Ġfor Ġclassification Ġof Ġvow els Ġand Ġwords Ġthat Ġachieved Ġ23 . 45 % Ġimprovement Ġof Ġaccuracy Ġover Ġthe Ġbaseline . </s> Production Ġof Ġartic ulatory Ġspeech Ġis Ġan Ġextremely Ġcomplicated Ġprocess , Ġthereby Ġrendering Ġunderstanding Ġof Ġthe Ġdiscrim in ative ĠEEG Ġmanifold Ġcorresponding Ġto Ġimagined Ġspeech Ġhighly Ġchallenging . ĠAs Ġa Ġresult , Ġmost Ġof Ġthe Ġexisting Ġapproaches Ġfailed Ġto Ġachieve Ġsatisfactory Ġaccuracy Ġon Ġdecoding Ġspeech Ġtokens Ġfrom Ġthe Ġspeech Ġimagery ĠEEG Ġdata . ĠPerhaps , Ġfor Ġthese Ġreasons , Ġvery Ġlittle Ġwork Ġhas Ġbeen Ġdevoted Ġto Ġrelating Ġthe Ġbrain Ġsignals Ġto Ġthe Ġunderlying Ġartic ulation . ĠThe Ġfew Ġexceptions Ġinclude ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠIn ĠB IB REF 17 Ġ, ĠZhao Ġet Ġal . Ġused Ġmanually Ġhand crafted Ġfeatures Ġfrom ĠEEG Ġdata , Ġcombined Ġwith Ġspeech Ġaudio Ġand Ġfacial Ġfeatures Ġto Ġachieve Ġclassification Ġof Ġthe Ġphon ological Ġcategories Ġvarying Ġbased Ġon Ġthe Ġartic ulatory Ġsteps . ĠHowever , Ġthe Ġimagined Ġspeech Ġclassification Ġaccuracy Ġbased Ġon ĠEEG Ġdata Ġalone , Ġas Ġreported Ġin ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġare Ġnot Ġsatisfactory Ġin Ġterms Ġof Ġaccuracy Ġand Ġreliability . ĠWe Ġnow Ġturn Ġto Ġdescribing Ġour Ġproposed Ġmodels . </s> Prop osed ĠFramework </s> C ognitive Ġlearning Ġprocess Ġunderlying Ġartic ulatory Ġspeech Ġproduction Ġinvolves Ġincorporation Ġof Ġintermediate Ġfeedback Ġloops Ġand Ġutilization Ġof Ġpast Ġinformation Ġstored Ġin Ġthe Ġform Ġof Ġmemory Ġas Ġwell Ġas Ġhierarchical Ġcombination Ġof Ġseveral Ġfeature Ġextract ors . ĠTo Ġthis Ġend , Ġwe Ġdevelop Ġour Ġmixed Ġneural Ġnetwork Ġarchitecture Ġcomposed Ġof Ġthree Ġsupervised Ġand Ġa Ġsingle Ġun super vised Ġlearning Ġstep , Ġdiscussed Ġin Ġthe Ġnext Ġsubsections Ġand Ġshown Ġin ĠFig . ĠFIG REF 1 Ġ. ĠWe Ġformulate Ġthe Ġproblem Ġof Ġcategor izing ĠEEG Ġdata Ġbased Ġon Ġspeech Ġimagery Ġas Ġa Ġnon - linear Ġmapping ĠIN LINE FORM 0 Ġof Ġa Ġmult ivariate Ġtime - series Ġinput Ġsequence ĠIN LINE FORM 1 Ġto Ġfixed Ġoutput ĠIN LINE FORM 2 Ġ, Ġi . e , Ġmat hematically ĠIN LINE FORM 3 Ġ: ĠIN LINE FORM 4 Ġ, Ġwhere Ġc Ġand Ġt Ġdenote Ġthe ĠEEG Ġchannels Ġand Ġtime Ġinst ants Ġrespectively . </s> Pre processing Ġstep </s> We Ġfollow Ġsimilar Ġpre - processing Ġsteps Ġon Ġraw ĠEEG Ġdata Ġas Ġreported Ġin ĠB IB REF 17 Ġ( ocular Ġartifact Ġremoval Ġusing Ġblind Ġsource Ġseparation , Ġband pass Ġfiltering Ġand Ġsubtract ing Ġmean Ġvalue Ġfrom Ġeach Ġchannel ) Ġexcept Ġthat Ġwe Ġdo Ġnot Ġperform ĠLa pl ac ian Ġfiltering Ġstep Ġsince Ġsuch Ġhigh - pass Ġfiltering Ġmay Ġdecrease Ġinformation Ġcontent Ġfrom Ġthe Ġsignals Ġin Ġthe Ġselected Ġbandwidth . </s> J oint Ġvariability Ġof Ġelectrodes </s> Mult ich annel ĠEEG Ġdata Ġis Ġhigh Ġdimensional Ġmult ivariate Ġtime Ġseries Ġdata Ġwhose Ġdimension ality Ġdepends Ġon Ġthe Ġnumber Ġof Ġelectrodes . ĠIt Ġis Ġa Ġmajor Ġhurdle Ġto Ġoptim ally Ġencode Ġinformation Ġfrom Ġthese ĠEEG Ġdata Ġinto Ġlower Ġdimensional Ġspace . ĠIn Ġfact , Ġour Ġinvestigation Ġbased Ġon Ġa Ġdevelopment Ġset Ġ( as Ġwe Ġexplain Ġlater ) Ġshowed Ġthat Ġwell - known Ġdeep Ġneural Ġnetworks Ġ( e . g ., Ġfully Ġconnected Ġnetworks Ġsuch Ġas Ġconv olution al Ġneural Ġnetworks , Ġrecurrent Ġneural Ġnetworks Ġand Ġaut oen cod ers ) Ġfail Ġto Ġindividually Ġlearn Ġsuch Ġcomplex Ġfeature Ġrepresentations Ġfrom Ġsingle - trial ĠEEG Ġdata . ĠBesides , Ġwe Ġfound Ġthat Ġinstead Ġof Ġusing Ġthe Ġraw Ġmulti - channel Ġhigh - dimensional ĠEEG Ġrequiring Ġlarge Ġtraining Ġtimes Ġand Ġresource Ġrequirements , Ġit Ġis Ġadvantageous Ġto Ġfirst Ġreduce Ġits Ġdimension ality Ġby Ġcapturing Ġthe Ġinformation Ġtransfer Ġamong Ġthe Ġelectrodes . ĠInstead Ġof Ġthe Ġconventional Ġapproach Ġof Ġselecting Ġa Ġhandful Ġof Ġchannels Ġas ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġwe Ġaddress Ġthis Ġby Ġcomputing Ġthe Ġchannel Ġcross - c ov ari ance , Ġresulting Ġin Ġpositive , Ġsemi - def inite Ġmat rices Ġencoding Ġthe Ġconnectivity Ġof Ġthe Ġelectrodes . ĠWe Ġdefine Ġchannel Ġcross - c ov ari ance Ġ( CC V ) Ġbetween Ġany Ġtwo Ġelectrodes ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġas : ĠIN LINE FORM 2 Ġ. ĠNext , Ġwe Ġreject Ġthe Ġchannels Ġwhich Ġhave Ġsignificantly Ġlower Ġcross - c ov ari ance Ġthan Ġauto - c ov ari ance Ġvalues Ġ( where Ġauto - c ov ari ance Ġimplies ĠCC V Ġon Ġsame Ġelectrode ). ĠWe Ġfound Ġthis Ġmeasure Ġto Ġbe Ġessential Ġas Ġthe Ġhigher Ġcognitive Ġprocesses Ġunderlying Ġspeech Ġplanning Ġand Ġsynthesis Ġinvolve Ġfrequent Ġinformation Ġexchange Ġbetween Ġdifferent Ġparts Ġof Ġthe Ġbrain . ĠHence , Ġsuch Ġmat rices Ġoften Ġcontain Ġmore Ġdiscrim in ative Ġfeatures Ġand Ġhidden Ġinformation Ġthan Ġmere Ġraw Ġsignals . ĠThis Ġis Ġessentially Ġdifferent Ġthan Ġour Ġprevious Ġwork ĠB IB REF 16 Ġwhere Ġwe Ġextract Ġper - channel Ġ1 - D Ġcovari ance Ġinformation Ġand Ġfeed Ġit Ġto Ġthe Ġnetworks . ĠWe Ġpresent Ġour Ġsample Ġ2 - D ĠEEG Ġcross - c ov ari ance Ġmat rices Ġ( of Ġtwo Ġindividuals ) Ġin ĠFig . ĠFIG REF 2 Ġ. </s> CNN Ġ& ĠL ST M </s> In Ġorder Ġto Ġdecode Ġspatial Ġconnections Ġbetween Ġthe Ġelectrodes Ġfrom Ġthe Ġchannel Ġcovari ance Ġmatrix , Ġwe Ġuse Ġa ĠCNN ĠB IB REF 19 Ġ, Ġin Ġparticular Ġa Ġfour - lay ered Ġ2 D ĠCNN Ġstacking Ġtwo Ġconv olution al Ġand Ġtwo Ġfully Ġconnected Ġhidden Ġlayers . ĠThe ĠIN LINE FORM 0 Ġfeature Ġmap Ġat Ġa Ġgiven ĠCNN Ġlayer Ġwith Ġinput ĠIN LINE FORM 1 Ġ, Ġweight Ġmatrix ĠIN LINE FORM 2 Ġand Ġbias ĠIN LINE FORM 3 Ġis Ġobtained Ġas : ĠIN LINE FORM 4 Ġ. ĠAt Ġthis Ġfirst Ġlevel Ġof Ġhierarchy , Ġthe Ġnetwork Ġis Ġtrained Ġwith Ġthe Ġcorresponding Ġlabels Ġas Ġtarget Ġoutputs , Ġoptimizing Ġa Ġcross - ent ropy Ġcost Ġfunction . ĠIn Ġparallel , Ġwe Ġapply Ġa Ġfour - lay ered Ġrecurrent Ġneural Ġnetwork Ġon Ġthe Ġchannel Ġcovari ance Ġmat rices Ġto Ġexplore Ġthe Ġhidden Ġtemporal Ġfeatures Ġof Ġthe Ġelectrodes . ĠName ly , Ġwe Ġexploit Ġan ĠL ST M ĠB IB REF 20 Ġconsisting Ġof Ġtwo Ġfully Ġconnected Ġhidden Ġlayers , Ġstacked Ġwith Ġtwo ĠL ST M Ġlayers Ġand Ġtrained Ġin Ġa Ġsimilar Ġmanner Ġas ĠCNN . </s> Deep Ġaut oen c oder Ġfor Ġspat io - tem poral Ġinformation </s> As Ġwe Ġfound Ġthe Ġindividually - trained Ġparallel Ġnetworks Ġ( CNN Ġand ĠL ST M ) Ġto Ġbe Ġuseful Ġ( see ĠTable ĠT AB REF 12 Ġ), Ġwe Ġsuspected Ġthe Ġcombination Ġof Ġthese Ġtwo Ġnetworks Ġcould Ġprovide Ġa Ġmore Ġpowerful Ġdiscrim in ative Ġspatial Ġand Ġtemporal Ġrepresentation Ġof Ġthe Ġdata Ġthan Ġeach Ġindependent Ġnetwork . ĠAs Ġsuch , Ġwe Ġconc aten ate Ġthe Ġlast Ġfully - connected Ġlayer Ġfrom Ġthe ĠCNN Ġwith Ġits Ġcounterpart Ġin Ġthe ĠL ST M Ġto Ġcompose Ġa Ġsingle Ġfeature Ġvector Ġbased Ġon Ġthese Ġtwo Ġpen ultimate Ġlayers . ĠUltimately , Ġthis Ġforms Ġa Ġjoint Ġspat io - tem poral Ġencoding Ġof Ġthe Ġcross - c ov ari ance Ġmatrix . </s> In Ġorder Ġto Ġfurther Ġreduce Ġthe Ġdimension ality Ġof Ġthe Ġspat io - tem poral Ġenc od ings Ġand Ġcancel Ġbackground Ġnoise Ġeffects ĠB IB REF 21 Ġ, Ġwe Ġtrain Ġan Ġun super vised Ġdeep Ġaut oen oder Ġ( DA E ) Ġon Ġthe Ġfused Ġheter ogeneous Ġfeatures Ġproduced Ġby Ġthe Ġcombined ĠCNN Ġand ĠL ST M Ġinformation . ĠThe ĠD AE Ġforms Ġour Ġsecond Ġlevel Ġof Ġhierarchy , Ġwith Ġ3 Ġencoding Ġand Ġ3 Ġdecoding Ġlayers , Ġand Ġmean Ġsquared Ġerror Ġ( M SE ) Ġas Ġthe Ġcost Ġfunction . </s> Class ification Ġwith ĠExtreme ĠGrad ient ĠBoost </s> At Ġthe Ġthird Ġlevel Ġof Ġhierarchy , Ġthe Ġdiscrete Ġlatent Ġvector Ġrepresentation Ġof Ġthe Ġdeep Ġaut oen c oder Ġis Ġfed Ġinto Ġan ĠExtreme ĠGrad ient ĠBoost Ġbased Ġclassification Ġlayer ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġmotivated Ġby ĠB IB REF 21 Ġ. ĠIt Ġis Ġa Ġregular ized Ġgradient Ġboosted Ġdecision Ġtree Ġthat Ġperforms Ġwell Ġon Ġstructured Ġproblems . ĠSince Ġour ĠEEG - ph on ological Ġpair wise Ġclassification Ġhas Ġan Ġinternal Ġstructure Ġinvolving Ġindividual Ġphon em es Ġand Ġwords , Ġit Ġseems Ġto Ġbe Ġa Ġreasonable Ġchoice Ġof Ġclass ifier . ĠThe Ġclass ifier Ġreceives Ġits Ġinput Ġfrom Ġthe Ġlatent Ġvectors Ġof Ġthe Ġdeep Ġaut oen c oder Ġand Ġis Ġtrained Ġin Ġa Ġsupervised Ġmanner Ġto Ġoutput Ġthe Ġfinal Ġpredicted Ġclasses Ġcorresponding Ġto Ġthe Ġspeech Ġimagery . </s> Dat as et </s> We Ġevaluate Ġour Ġmodel Ġon Ġa Ġpublicly Ġavailable Ġdataset , ĠK ARA ĠONE ĠB IB REF 17 Ġ, Ġcomposed Ġof Ġmultim od al Ġdata Ġfor Ġstimulus - based , Ġimagined Ġand Ġarticulated Ġspeech Ġstate Ġcorresponding Ġto Ġ7 Ġphon emic / sy ll ab ic Ġ( Ġ/ iy /, Ġ/ p iy /, Ġ/ t iy /, Ġ/ d iy /, Ġ/ u w /, Ġ/ m /, Ġ/ n / Ġ) Ġas Ġwell Ġas Ġ4 Ġwords ( pat , Ġpot , Ġknew Ġand Ġgn aw ). ĠThe Ġdataset Ġconsists Ġof Ġ14 Ġparticipants , Ġwith Ġeach Ġprompt Ġpresented Ġ11 Ġtimes Ġto Ġeach Ġindividual . ĠSince Ġour Ġintention Ġis Ġto Ġclassify Ġthe Ġphon ological Ġcategories Ġfrom Ġhuman Ġthoughts , Ġwe Ġdiscard Ġthe Ġfacial Ġand Ġaudio Ġinformation Ġand Ġonly Ġconsider Ġthe ĠEEG Ġdata Ġcorresponding Ġto Ġimagined Ġspeech . ĠIt Ġis Ġnoteworthy Ġthat Ġgiven Ġthe Ġmixed Ġnature Ġof ĠEEG Ġsignals , Ġit Ġis Ġreportedly Ġchallenging Ġto Ġattain Ġa Ġpair wise ĠEEG - ph on eme Ġmapping ĠB IB REF 18 Ġ. ĠIn Ġorder Ġto Ġexplore Ġthe Ġproblem Ġspace , Ġwe Ġthus Ġspecifically Ġtarget Ġfive Ġbinary Ġclassification Ġproblems Ġaddressed Ġin ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġi . e Ġpresence / abs ence Ġof Ġconson ants , Ġphon emic Ġnasal , Ġbil ab ial , Ġhigh - front Ġvow els Ġand Ġhigh - back Ġvow els . </s> Training Ġand Ġhyper param eter Ġselection </s> We Ġperformed Ġtwo Ġsets Ġof Ġexperiments Ġwith Ġthe Ġsingle - trial ĠEEG Ġdata . ĠIn ĠPH ASE - ONE , Ġour Ġgoals Ġwas Ġto Ġidentify Ġthe Ġbest Ġarchitectures Ġand Ġhyper param eters Ġfor Ġour Ġnetworks Ġwith Ġa Ġreasonable Ġnumber Ġof Ġruns . ĠFor ĠPH ASE - ONE , Ġwe Ġrandomly Ġshuff led Ġand Ġdivided Ġthe Ġdata Ġ( 19 13 Ġsignals Ġfrom Ġ14 Ġindividuals ) Ġinto Ġtrain Ġ( 80 %), Ġdevelopment Ġ( 10 %) Ġand Ġtest Ġsets Ġ( 10 %). ĠIn ĠPH ASE - TW O , Ġin Ġorder Ġto Ġperform Ġa Ġfair Ġcomparison Ġwith Ġthe Ġprevious Ġmethods Ġreported Ġon Ġthe Ġsame Ġdataset , Ġwe Ġperform Ġa Ġleave - one - subject Ġout Ġcross - valid ation Ġexperiment Ġusing Ġthe Ġbest Ġsettings Ġwe Ġlearn Ġfrom ĠPH ASE - ONE . </s> The Ġarchitectural Ġparameters Ġand Ġhyper param eters Ġlisted Ġin ĠTable ĠT AB REF 6 Ġwere Ġselected Ġthrough Ġan Ġexhaustive Ġgrid - search Ġbased Ġon Ġthe Ġvalidation Ġset Ġof ĠPH ASE - ONE . ĠWe Ġconducted Ġa Ġseries Ġof Ġempirical Ġstudies Ġstarting Ġfrom Ġsingle Ġhidden - lay ered Ġnetworks Ġfor Ġeach Ġof Ġthe Ġblocks Ġand , Ġbased Ġon Ġthe Ġvalidation Ġaccuracy , Ġwe Ġincreased Ġthe Ġdepth Ġof Ġeach Ġgiven Ġnetwork Ġand Ġselected Ġthe Ġoptimal Ġparam etric Ġset Ġfrom Ġall Ġpossible Ġcombinations Ġof Ġparameters . ĠFor Ġthe Ġgradient Ġboosting Ġclassification , Ġwe Ġfixed Ġthe Ġmaximum Ġdepth Ġat Ġ10 , Ġnumber Ġof Ġestim ators Ġat Ġ5000 , Ġlearning Ġrate Ġat Ġ0 . 1 , Ġregular ization Ġcoefficient Ġat Ġ0 . 3 , Ġsubs ample Ġratio Ġat Ġ0 . 8 , Ġand Ġcolumn - sample / iter ation Ġat Ġ0 . 4 . ĠWe Ġdid Ġnot Ġfind Ġany Ġnotable Ġchange Ġof Ġaccuracy Ġwhile Ġvarying Ġother Ġhyper param eters Ġwhile Ġtraining Ġgradient Ġboost Ġclass ifier . </s> Performance Ġanalysis Ġand Ġdiscussion </s> To Ġdemonstrate Ġthe Ġsignificance Ġof Ġthe Ġhierarchical ĠCNN - L ST M - DA E Ġmethod , Ġwe Ġconducted Ġseparate Ġexperiments Ġwith Ġthe Ġindividual Ġnetworks Ġin ĠPH ASE - ONE Ġof Ġexperiments Ġand Ġsummarized Ġthe Ġresults Ġin ĠTable ĠT AB REF 12 ĠFrom Ġthe Ġaverage Ġaccuracy Ġscores , Ġwe Ġobserve Ġthat Ġthe Ġmixed Ġnetwork Ġperforms Ġmuch Ġbetter Ġthan Ġindividual Ġblocks Ġwhich Ġis Ġin Ġagreement Ġwith Ġthe Ġfindings Ġin ĠB IB REF 21 Ġ. ĠA Ġdetailed Ġanalysis Ġon Ġrepeated Ġruns Ġfurther Ġshows Ġthat Ġin Ġmost Ġof Ġthe Ġcases , ĠL ST M Ġalone Ġdoes Ġnot Ġperform Ġbetter Ġthan Ġchance . ĠCNN , Ġon Ġthe Ġother Ġhand , Ġis Ġheavily Ġbiased Ġtowards Ġthe Ġclass Ġlabel Ġwhich Ġsees Ġmore Ġtraining Ġdata Ġcorresponding Ġto Ġit . ĠThough Ġthe Ġsituation Ġimproves Ġwith Ġcombined ĠCNN - L ST M , Ġour Ġanalysis Ġclearly Ġshows Ġthe Ġnecessity Ġof Ġa Ġbetter Ġencoding Ġscheme Ġto Ġutilize Ġthe Ġcombined Ġfeatures Ġrather Ġthan Ġmere Ġconc aten ation Ġof Ġthe Ġpen ultimate Ġfeatures Ġof Ġboth Ġnetworks . </s> The Ġvery Ġfact Ġthat Ġour Ġcombined Ġnetwork Ġimproves Ġthe Ġclassification Ġaccuracy Ġby Ġa Ġmean Ġmargin Ġof Ġ14 . 45 % Ġthan Ġthe ĠCNN - L ST M Ġnetwork Ġindeed Ġreveals Ġthat Ġthe Ġaut oen c oder Ġcontributes Ġtowards Ġfiltering Ġout Ġthe Ġunrelated Ġand Ġnoisy Ġfeatures Ġfrom Ġthe Ġconc aten ated Ġpen ultimate Ġfeature Ġset . ĠIt Ġalso Ġproves Ġthat Ġthe Ġcombined Ġsupervised Ġand Ġun super vised Ġneural Ġnetworks , Ġtrained Ġhierarch ically , Ġcan Ġlearn Ġthe Ġdiscrim in ative Ġmanifold Ġbetter Ġthan Ġthe Ġindividual Ġnetworks Ġand Ġit Ġis Ġcrucial Ġfor Ġimproving Ġthe Ġclassification Ġaccuracy . ĠIn Ġaddition Ġto Ġaccuracy , Ġwe Ġalso Ġprovide Ġthe Ġk appa Ġcoefficients ĠB IB REF 24 Ġof Ġour Ġmethod Ġin ĠFig . ĠFIG REF 14 Ġ. ĠHere , Ġa Ġhigher Ġmean Ġk appa Ġvalue Ġcorresponding Ġto Ġa Ġtask Ġimplies Ġthat Ġthe Ġnetwork Ġis Ġable Ġto Ġfind Ġbetter Ġdiscrim in ative Ġinformation Ġfrom Ġthe ĠEEG Ġdata Ġbeyond Ġrandom Ġdecisions . ĠThe Ġmaximum Ġabove - chance Ġaccuracy Ġ( 75 . 92 %) Ġis Ġrecorded Ġfor Ġpresence / abs ence Ġof Ġthe Ġvowel Ġtask Ġand Ġthe Ġminimum Ġ( 49 . 14 %) Ġis Ġrecorded Ġfor Ġthe ĠIN LINE FORM 0 Ġ. </s> To Ġfurther Ġinvestigate Ġthe Ġfeature Ġrepresentation Ġachieved Ġby Ġour Ġmodel , Ġwe Ġplot ĠT - dist ributed ĠSt och astic ĠNeighbor ĠEmb edd ing Ġ( t S NE ) Ġcorresponding Ġto ĠIN LINE FORM 0 Ġand ĠV / C Ġclassification Ġtasks Ġin ĠFig . ĠFIG REF 8 Ġ. ĠWe Ġparticularly Ġselect Ġthese Ġtwo Ġtasks Ġas Ġour Ġmodel Ġexhibits Ġrespectively Ġminimum Ġand Ġmaximum Ġperformance Ġfor Ġthese Ġtwo . ĠThe Ġt S NE Ġvisualization Ġreveals Ġthat Ġthe Ġsecond Ġset Ġof Ġfeatures Ġare Ġmore Ġeasily Ġsepar able Ġthan Ġthe Ġfirst Ġone , Ġthereby Ġgiving Ġa Ġrationale Ġfor Ġour Ġperformance . </s> Next , Ġwe Ġprovide Ġperformance Ġcomparison Ġof Ġthe Ġproposed Ġapproach Ġwith Ġthe Ġbaseline Ġmethods Ġfor ĠPH ASE - TW O Ġof Ġour Ġstudy Ġ( cross - valid ation Ġexperiment ) Ġin ĠTable ĠT AB REF 15 Ġ. ĠSince Ġthe Ġmodel Ġencounters Ġthe Ġunseen Ġdata Ġof Ġa Ġnew Ġsubject Ġfor Ġtesting , Ġand Ġgiven Ġthe Ġhigh Ġinter - subject Ġvariability Ġof Ġthe ĠEEG Ġdata , Ġa Ġreduction Ġin Ġthe Ġaccuracy Ġwas Ġexpected . ĠHowever , Ġour Ġnetwork Ġstill Ġmanaged Ġto Ġachieve Ġan Ġimprovement Ġof Ġ18 . 91 , Ġ9 . 95 , Ġ67 . 15 , Ġ2 . 83 Ġand Ġ13 . 70 Ġ% Ġover ĠB IB REF 17 Ġ. ĠBesides , Ġour Ġbest Ġmodel Ġshows Ġmore Ġreliability Ġcompared Ġto Ġprevious Ġworks : ĠThe Ġstandard Ġdeviation Ġof Ġour Ġmodel 's Ġclassification Ġaccuracy Ġacross Ġall Ġthe Ġtasks Ġis Ġreduced Ġfrom Ġ22 . 59 % ĠB IB REF 17 Ġand Ġ17 . 52 % ĠB IB REF 18 Ġto Ġa Ġmere Ġ5 . 41 %. </s> Conclusion Ġand Ġfuture Ġdirection </s> In Ġan Ġattempt Ġto Ġmove Ġa Ġstep Ġtowards Ġunderstanding Ġthe Ġspeech Ġinformation Ġencoded Ġin Ġbrain Ġsignals , Ġwe Ġdeveloped Ġa Ġnovel Ġmixed Ġdeep Ġneural Ġnetwork Ġscheme Ġfor Ġa Ġnumber Ġof Ġbinary Ġclassification Ġtasks Ġfrom Ġspeech Ġimagery ĠEEG Ġdata . ĠUnlike Ġprevious Ġapproaches Ġwhich Ġmostly Ġdeal Ġwith Ġsubject - dependent Ġclassification Ġof ĠEEG Ġinto Ġdiscrete Ġvowel Ġor Ġword Ġlabels , Ġthis Ġwork Ġinvestigates Ġa Ġsubject - inv ari ant Ġmapping Ġof ĠEEG Ġdata Ġwith Ġdifferent Ġphon ological Ġcategories , Ġvarying Ġwidely Ġin Ġterms Ġof Ġunderlying Ġartic ulator Ġmotions Ġ( eg : Ġinvolvement Ġor Ġnon - inv olve ment Ġof Ġlips Ġand Ġvel um , Ġvariation Ġof Ġtongue Ġmovements Ġetc ). ĠOur Ġmodel Ġtakes Ġan Ġadvantage Ġof Ġfeature Ġextraction Ġcapability Ġof ĠCNN , ĠL ST M Ġas Ġwell Ġas Ġthe Ġdeep Ġlearning Ġbenefit Ġof Ġdeep Ġaut oen cod ers . ĠWe Ġtook ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġas Ġthe Ġbaseline Ġworks Ġinvestigating Ġthe Ġsame Ġproblem Ġand Ġcompared Ġour Ġperformance Ġwith Ġtheirs . ĠOur Ġproposed Ġmethod Ġhighly Ġoutper forms Ġthe Ġexisting Ġmethods Ġacross Ġall Ġthe Ġfive Ġbinary Ġclassification Ġtasks Ġby Ġa Ġlarge Ġaverage Ġmargin Ġof Ġ22 . 51 %. </s> Acknowled gments </s> This Ġwork Ġwas Ġfunded Ġby Ġthe ĠNatural ĠSciences Ġand ĠEngineering ĠResearch ĠCouncil Ġ( NS ERC ) Ġof ĠCanada Ġand ĠCanadian ĠInstitutes Ġfor ĠHealth ĠResearch Ġ( CI HR ).
Introduction Decoding intended speech or motor activity from brain signals is one of the major research areas in Brain Computer Interface (BCI) systems BIBREF0 , BIBREF1 . In particular, speech-related BCI technologies attempt to provide effective vocal communication strategies for controlling external devices through speech commands interpreted from brain signals BIBREF2 . Not only do they provide neuro-prosthetic help for people with speaking disabilities and neuro-muscular disorders like locked-in-syndrome, nasopharyngeal cancer, and amytotropic lateral sclerosis (ALS), but also equip people with a better medium to communicate and express thoughts, thereby improving the quality of rehabilitation and clinical neurology BIBREF3 , BIBREF4 . Such devices also have applications in entertainment, preventive treatments, personal communication, games, etc. Furthermore, BCI technologies can be utilized in silent communication, as in noisy environments, or situations where any sort of audio-visual communication is infeasible. Among the various brain activity-monitoring modalities in BCI, electroencephalography (EEG) BIBREF5 , BIBREF6 has demonstrated promising potential to differentiate between various brain activities through measurement of related electric fields. EEG is non-invasive, portable, low cost, and provides satisfactory temporal resolution. This makes EEG suitable to realize BCI systems. EEG data, however, is challenging: these data are high dimensional, have poor SNR, and suffer from low spatial resolution and a multitude of artifacts. For these reasons, it is not particularly obvious how to decode the desired information from raw EEG signals. Although the area of BCI based speech intent recognition has received increasing attention among the research community in the past few years, most research has focused on classification of individual speech categories in terms of discrete vowels, phonemes and words BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 . This includes categorization of imagined EEG signal into binary vowel categories like /a/, /u/ and rest BIBREF7 , BIBREF8 , BIBREF9 ; binary syllable classes like /ba/ and /ku/ BIBREF1 , BIBREF10 , BIBREF11 , BIBREF12 ; a handful of control words like 'up', 'down', 'left', 'right' and 'select' BIBREF15 or others like 'water', 'help', 'thanks', 'food', 'stop' BIBREF13 , Chinese characters BIBREF14 , etc. Such works mostly involve traditional signal processing or manual feature handcrafting along with linear classifiers (e.g., SVMs). In our recent work BIBREF16 , we introduced deep learning models for classification of vowels and words that achieved 23.45% improvement of accuracy over the baseline. Production of articulatory speech is an extremely complicated process, thereby rendering understanding of the discriminative EEG manifold corresponding to imagined speech highly challenging. As a result, most of the existing approaches failed to achieve satisfactory accuracy on decoding speech tokens from the speech imagery EEG data. Perhaps, for these reasons, very little work has been devoted to relating the brain signals to the underlying articulation. The few exceptions include BIBREF17 , BIBREF18 . In BIBREF17 , Zhao et al. used manually handcrafted features from EEG data, combined with speech audio and facial features to achieve classification of the phonological categories varying based on the articulatory steps. However, the imagined speech classification accuracy based on EEG data alone, as reported in BIBREF17 , BIBREF18 , are not satisfactory in terms of accuracy and reliability. We now turn to describing our proposed models. Proposed Framework Cognitive learning process underlying articulatory speech production involves incorporation of intermediate feedback loops and utilization of past information stored in the form of memory as well as hierarchical combination of several feature extractors. To this end, we develop our mixed neural network architecture composed of three supervised and a single unsupervised learning step, discussed in the next subsections and shown in Fig. FIGREF1 . We formulate the problem of categorizing EEG data based on speech imagery as a non-linear mapping INLINEFORM0 of a multivariate time-series input sequence INLINEFORM1 to fixed output INLINEFORM2 , i.e, mathematically INLINEFORM3 : INLINEFORM4 , where c and t denote the EEG channels and time instants respectively. Preprocessing step We follow similar pre-processing steps on raw EEG data as reported in BIBREF17 (ocular artifact removal using blind source separation, bandpass filtering and subtracting mean value from each channel) except that we do not perform Laplacian filtering step since such high-pass filtering may decrease information content from the signals in the selected bandwidth. Joint variability of electrodes Multichannel EEG data is high dimensional multivariate time series data whose dimensionality depends on the number of electrodes. It is a major hurdle to optimally encode information from these EEG data into lower dimensional space. In fact, our investigation based on a development set (as we explain later) showed that well-known deep neural networks (e.g., fully connected networks such as convolutional neural networks, recurrent neural networks and autoencoders) fail to individually learn such complex feature representations from single-trial EEG data. Besides, we found that instead of using the raw multi-channel high-dimensional EEG requiring large training times and resource requirements, it is advantageous to first reduce its dimensionality by capturing the information transfer among the electrodes. Instead of the conventional approach of selecting a handful of channels as BIBREF17 , BIBREF18 , we address this by computing the channel cross-covariance, resulting in positive, semi-definite matrices encoding the connectivity of the electrodes. We define channel cross-covariance (CCV) between any two electrodes INLINEFORM0 and INLINEFORM1 as: INLINEFORM2 . Next, we reject the channels which have significantly lower cross-covariance than auto-covariance values (where auto-covariance implies CCV on same electrode). We found this measure to be essential as the higher cognitive processes underlying speech planning and synthesis involve frequent information exchange between different parts of the brain. Hence, such matrices often contain more discriminative features and hidden information than mere raw signals. This is essentially different than our previous work BIBREF16 where we extract per-channel 1-D covariance information and feed it to the networks. We present our sample 2-D EEG cross-covariance matrices (of two individuals) in Fig. FIGREF2 . CNN & LSTM In order to decode spatial connections between the electrodes from the channel covariance matrix, we use a CNN BIBREF19 , in particular a four-layered 2D CNN stacking two convolutional and two fully connected hidden layers. The INLINEFORM0 feature map at a given CNN layer with input INLINEFORM1 , weight matrix INLINEFORM2 and bias INLINEFORM3 is obtained as: INLINEFORM4 . At this first level of hierarchy, the network is trained with the corresponding labels as target outputs, optimizing a cross-entropy cost function. In parallel, we apply a four-layered recurrent neural network on the channel covariance matrices to explore the hidden temporal features of the electrodes. Namely, we exploit an LSTM BIBREF20 consisting of two fully connected hidden layers, stacked with two LSTM layers and trained in a similar manner as CNN. Deep autoencoder for spatio-temporal information As we found the individually-trained parallel networks (CNN and LSTM) to be useful (see Table TABREF12 ), we suspected the combination of these two networks could provide a more powerful discriminative spatial and temporal representation of the data than each independent network. As such, we concatenate the last fully-connected layer from the CNN with its counterpart in the LSTM to compose a single feature vector based on these two penultimate layers. Ultimately, this forms a joint spatio-temporal encoding of the cross-covariance matrix. In order to further reduce the dimensionality of the spatio-temporal encodings and cancel background noise effects BIBREF21 , we train an unsupervised deep autoenoder (DAE) on the fused heterogeneous features produced by the combined CNN and LSTM information. The DAE forms our second level of hierarchy, with 3 encoding and 3 decoding layers, and mean squared error (MSE) as the cost function. Classification with Extreme Gradient Boost At the third level of hierarchy, the discrete latent vector representation of the deep autoencoder is fed into an Extreme Gradient Boost based classification layer BIBREF22 , BIBREF23 motivated by BIBREF21 . It is a regularized gradient boosted decision tree that performs well on structured problems. Since our EEG-phonological pairwise classification has an internal structure involving individual phonemes and words, it seems to be a reasonable choice of classifier. The classifier receives its input from the latent vectors of the deep autoencoder and is trained in a supervised manner to output the final predicted classes corresponding to the speech imagery. Dataset We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw). The dataset consists of 14 participants, with each prompt presented 11 times to each individual. Since our intention is to classify the phonological categories from human thoughts, we discard the facial and audio information and only consider the EEG data corresponding to imagined speech. It is noteworthy that given the mixed nature of EEG signals, it is reportedly challenging to attain a pairwise EEG-phoneme mapping BIBREF18 . In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i.e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels. Training and hyperparameter selection We performed two sets of experiments with the single-trial EEG data. In PHASE-ONE, our goals was to identify the best architectures and hyperparameters for our networks with a reasonable number of runs. For PHASE-ONE, we randomly shuffled and divided the data (1913 signals from 14 individuals) into train (80%), development (10%) and test sets (10%). In PHASE-TWO, in order to perform a fair comparison with the previous methods reported on the same dataset, we perform a leave-one-subject out cross-validation experiment using the best settings we learn from PHASE-ONE. The architectural parameters and hyperparameters listed in Table TABREF6 were selected through an exhaustive grid-search based on the validation set of PHASE-ONE. We conducted a series of empirical studies starting from single hidden-layered networks for each of the blocks and, based on the validation accuracy, we increased the depth of each given network and selected the optimal parametric set from all possible combinations of parameters. For the gradient boosting classification, we fixed the maximum depth at 10, number of estimators at 5000, learning rate at 0.1, regularization coefficient at 0.3, subsample ratio at 0.8, and column-sample/iteration at 0.4. We did not find any notable change of accuracy while varying other hyperparameters while training gradient boost classifier. Performance analysis and discussion To demonstrate the significance of the hierarchical CNN-LSTM-DAE method, we conducted separate experiments with the individual networks in PHASE-ONE of experiments and summarized the results in Table TABREF12 From the average accuracy scores, we observe that the mixed network performs much better than individual blocks which is in agreement with the findings in BIBREF21 . A detailed analysis on repeated runs further shows that in most of the cases, LSTM alone does not perform better than chance. CNN, on the other hand, is heavily biased towards the class label which sees more training data corresponding to it. Though the situation improves with combined CNN-LSTM, our analysis clearly shows the necessity of a better encoding scheme to utilize the combined features rather than mere concatenation of the penultimate features of both networks. The very fact that our combined network improves the classification accuracy by a mean margin of 14.45% than the CNN-LSTM network indeed reveals that the autoencoder contributes towards filtering out the unrelated and noisy features from the concatenated penultimate feature set. It also proves that the combined supervised and unsupervised neural networks, trained hierarchically, can learn the discriminative manifold better than the individual networks and it is crucial for improving the classification accuracy. In addition to accuracy, we also provide the kappa coefficients BIBREF24 of our method in Fig. FIGREF14 . Here, a higher mean kappa value corresponding to a task implies that the network is able to find better discriminative information from the EEG data beyond random decisions. The maximum above-chance accuracy (75.92%) is recorded for presence/absence of the vowel task and the minimum (49.14%) is recorded for the INLINEFORM0 . To further investigate the feature representation achieved by our model, we plot T-distributed Stochastic Neighbor Embedding (tSNE) corresponding to INLINEFORM0 and V/C classification tasks in Fig. FIGREF8 . We particularly select these two tasks as our model exhibits respectively minimum and maximum performance for these two. The tSNE visualization reveals that the second set of features are more easily separable than the first one, thereby giving a rationale for our performance. Next, we provide performance comparison of the proposed approach with the baseline methods for PHASE-TWO of our study (cross-validation experiment) in Table TABREF15 . Since the model encounters the unseen data of a new subject for testing, and given the high inter-subject variability of the EEG data, a reduction in the accuracy was expected. However, our network still managed to achieve an improvement of 18.91, 9.95, 67.15, 2.83 and 13.70 % over BIBREF17 . Besides, our best model shows more reliability compared to previous works: The standard deviation of our model's classification accuracy across all the tasks is reduced from 22.59% BIBREF17 and 17.52% BIBREF18 to a mere 5.41%. Conclusion and future direction In an attempt to move a step towards understanding the speech information encoded in brain signals, we developed a novel mixed deep neural network scheme for a number of binary classification tasks from speech imagery EEG data. Unlike previous approaches which mostly deal with subject-dependent classification of EEG into discrete vowel or word labels, this work investigates a subject-invariant mapping of EEG data with different phonological categories, varying widely in terms of underlying articulator motions (eg: involvement or non-involvement of lips and velum, variation of tongue movements etc). Our model takes an advantage of feature extraction capability of CNN, LSTM as well as the deep learning benefit of deep autoencoders. We took BIBREF17 , BIBREF18 as the baseline works investigating the same problem and compared our performance with theirs. Our proposed method highly outperforms the existing methods across all the five binary classification tasks by a large average margin of 22.51%. Acknowledgments This work was funded by the Natural Sciences and Engineering Research Council (NSERC) of Canada and Canadian Institutes for Health Research (CIHR).
What data was presented to the subjects to elicit event-related responses?
e6583c60b13b87fc37af75ffc975e7e316d4f4e0
What Ġdata Ġwas Ġpresented Ġto Ġthe Ġsubjects Ġto Ġelicit Ġevent - related Ġresponses ?
7 Ġphon emic / sy ll ab ic Ġ( Ġ/ iy /, Ġ/ p iy /, Ġ/ t iy /, Ġ/ d iy /, Ġ/ u w /, Ġ/ m /, Ġ/ n / Ġ) Ġas Ġwell Ġas Ġ4 Ġwords ( pat , Ġpot , Ġknew Ġand Ġgn aw )
<s> What Ġdata Ġwas Ġpresented Ġto Ġthe Ġsubjects Ġto Ġelicit Ġevent - related Ġresponses ? </s> Introduction </s> Dec oding Ġintended Ġspeech Ġor Ġmotor Ġactivity Ġfrom Ġbrain Ġsignals Ġis Ġone Ġof Ġthe Ġmajor Ġresearch Ġareas Ġin ĠBrain ĠComputer ĠInterface Ġ( BC I ) Ġsystems ĠB IB REF 0 Ġ, ĠB IB REF 1 Ġ. ĠIn Ġparticular , Ġspeech - related ĠBC I Ġtechnologies Ġattempt Ġto Ġprovide Ġeffective Ġvocal Ġcommunication Ġstrategies Ġfor Ġcontrolling Ġexternal Ġdevices Ġthrough Ġspeech Ġcommands Ġinterpreted Ġfrom Ġbrain Ġsignals ĠB IB REF 2 Ġ. ĠNot Ġonly Ġdo Ġthey Ġprovide Ġneuro - pro st hetic Ġhelp Ġfor Ġpeople Ġwith Ġspeaking Ġdisabilities Ġand Ġneuro - mus cular Ġdisorders Ġlike Ġlocked - in - sy nd rome , Ġnas oph aryn ge al Ġcancer , Ġand Ġam yt otropic Ġlateral Ġsclerosis Ġ( ALS ), Ġbut Ġalso Ġequip Ġpeople Ġwith Ġa Ġbetter Ġmedium Ġto Ġcommunicate Ġand Ġexpress Ġthoughts , Ġthereby Ġimproving Ġthe Ġquality Ġof Ġrehabilitation Ġand Ġclinical Ġneurolog y ĠB IB REF 3 Ġ, ĠB IB REF 4 Ġ. ĠSuch Ġdevices Ġalso Ġhave Ġapplications Ġin Ġentertainment , Ġpreventive Ġtreatments , Ġpersonal Ġcommunication , Ġgames , Ġetc . ĠFurthermore , ĠBC I Ġtechnologies Ġcan Ġbe Ġutilized Ġin Ġsilent Ġcommunication , Ġas Ġin Ġnoisy Ġenvironments , Ġor Ġsituations Ġwhere Ġany Ġsort Ġof Ġaudio - visual Ġcommunication Ġis Ġinf eas ible . </s> Among Ġthe Ġvarious Ġbrain Ġactivity - monitor ing Ġmod alities Ġin ĠBC I , Ġelectro ence phal ography Ġ( EE G ) ĠB IB REF 5 Ġ, ĠB IB REF 6 Ġhas Ġdemonstrated Ġpromising Ġpotential Ġto Ġdifferentiate Ġbetween Ġvarious Ġbrain Ġactivities Ġthrough Ġmeasurement Ġof Ġrelated Ġelectric Ġfields . ĠEEG Ġis Ġnon - in vasive , Ġportable , Ġlow Ġcost , Ġand Ġprovides Ġsatisfactory Ġtemporal Ġresolution . ĠThis Ġmakes ĠEEG Ġsuitable Ġto Ġrealize ĠBC I Ġsystems . ĠEEG Ġdata , Ġhowever , Ġis Ġchallenging : Ġthese Ġdata Ġare Ġhigh Ġdimensional , Ġhave Ġpoor ĠSN R , Ġand Ġsuffer Ġfrom Ġlow Ġspatial Ġresolution Ġand Ġa Ġmultitude Ġof Ġartifacts . ĠFor Ġthese Ġreasons , Ġit Ġis Ġnot Ġparticularly Ġobvious Ġhow Ġto Ġdecode Ġthe Ġdesired Ġinformation Ġfrom Ġraw ĠEEG Ġsignals . ĠAlthough Ġthe Ġarea Ġof ĠBC I Ġbased Ġspeech Ġintent Ġrecognition Ġhas Ġreceived Ġincreasing Ġattention Ġamong Ġthe Ġresearch Ġcommunity Ġin Ġthe Ġpast Ġfew Ġyears , Ġmost Ġresearch Ġhas Ġfocused Ġon Ġclassification Ġof Ġindividual Ġspeech Ġcategories Ġin Ġterms Ġof Ġdiscrete Ġvow els , Ġphon em es Ġand Ġwords ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ, ĠB IB REF 13 Ġ, ĠB IB REF 14 Ġ, ĠB IB REF 15 Ġ. ĠThis Ġincludes Ġcategor ization Ġof Ġimagined ĠEEG Ġsignal Ġinto Ġbinary Ġvowel Ġcategories Ġlike Ġ/ a /, Ġ/ u / Ġand Ġrest ĠB IB REF 7 Ġ, ĠB IB REF 8 Ġ, ĠB IB REF 9 Ġ; Ġbinary Ġsyll able Ġclasses Ġlike Ġ/ ba / Ġand Ġ/ ku / ĠB IB REF 1 Ġ, ĠB IB REF 10 Ġ, ĠB IB REF 11 Ġ, ĠB IB REF 12 Ġ; Ġa Ġhandful Ġof Ġcontrol Ġwords Ġlike Ġ' up ', Ġ' down ', Ġ' left ', Ġ' right ' Ġand Ġ' select ' ĠB IB REF 15 Ġor Ġothers Ġlike Ġ' water ', Ġ' help ', Ġ' thanks ', Ġ' food ', Ġ' stop ' ĠB IB REF 13 Ġ, ĠChinese Ġcharacters ĠB IB REF 14 Ġ, Ġetc . ĠSuch Ġworks Ġmostly Ġinvolve Ġtraditional Ġsignal Ġprocessing Ġor Ġmanual Ġfeature Ġhand craft ing Ġalong Ġwith Ġlinear Ġclass ifiers Ġ( e . g ., ĠSV Ms ). ĠIn Ġour Ġrecent Ġwork ĠB IB REF 16 Ġ, Ġwe Ġintroduced Ġdeep Ġlearning Ġmodels Ġfor Ġclassification Ġof Ġvow els Ġand Ġwords Ġthat Ġachieved Ġ23 . 45 % Ġimprovement Ġof Ġaccuracy Ġover Ġthe Ġbaseline . </s> Production Ġof Ġartic ulatory Ġspeech Ġis Ġan Ġextremely Ġcomplicated Ġprocess , Ġthereby Ġrendering Ġunderstanding Ġof Ġthe Ġdiscrim in ative ĠEEG Ġmanifold Ġcorresponding Ġto Ġimagined Ġspeech Ġhighly Ġchallenging . ĠAs Ġa Ġresult , Ġmost Ġof Ġthe Ġexisting Ġapproaches Ġfailed Ġto Ġachieve Ġsatisfactory Ġaccuracy Ġon Ġdecoding Ġspeech Ġtokens Ġfrom Ġthe Ġspeech Ġimagery ĠEEG Ġdata . ĠPerhaps , Ġfor Ġthese Ġreasons , Ġvery Ġlittle Ġwork Ġhas Ġbeen Ġdevoted Ġto Ġrelating Ġthe Ġbrain Ġsignals Ġto Ġthe Ġunderlying Ġartic ulation . ĠThe Ġfew Ġexceptions Ġinclude ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ. ĠIn ĠB IB REF 17 Ġ, ĠZhao Ġet Ġal . Ġused Ġmanually Ġhand crafted Ġfeatures Ġfrom ĠEEG Ġdata , Ġcombined Ġwith Ġspeech Ġaudio Ġand Ġfacial Ġfeatures Ġto Ġachieve Ġclassification Ġof Ġthe Ġphon ological Ġcategories Ġvarying Ġbased Ġon Ġthe Ġartic ulatory Ġsteps . ĠHowever , Ġthe Ġimagined Ġspeech Ġclassification Ġaccuracy Ġbased Ġon ĠEEG Ġdata Ġalone , Ġas Ġreported Ġin ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġare Ġnot Ġsatisfactory Ġin Ġterms Ġof Ġaccuracy Ġand Ġreliability . ĠWe Ġnow Ġturn Ġto Ġdescribing Ġour Ġproposed Ġmodels . </s> Prop osed ĠFramework </s> C ognitive Ġlearning Ġprocess Ġunderlying Ġartic ulatory Ġspeech Ġproduction Ġinvolves Ġincorporation Ġof Ġintermediate Ġfeedback Ġloops Ġand Ġutilization Ġof Ġpast Ġinformation Ġstored Ġin Ġthe Ġform Ġof Ġmemory Ġas Ġwell Ġas Ġhierarchical Ġcombination Ġof Ġseveral Ġfeature Ġextract ors . ĠTo Ġthis Ġend , Ġwe Ġdevelop Ġour Ġmixed Ġneural Ġnetwork Ġarchitecture Ġcomposed Ġof Ġthree Ġsupervised Ġand Ġa Ġsingle Ġun super vised Ġlearning Ġstep , Ġdiscussed Ġin Ġthe Ġnext Ġsubsections Ġand Ġshown Ġin ĠFig . ĠFIG REF 1 Ġ. ĠWe Ġformulate Ġthe Ġproblem Ġof Ġcategor izing ĠEEG Ġdata Ġbased Ġon Ġspeech Ġimagery Ġas Ġa Ġnon - linear Ġmapping ĠIN LINE FORM 0 Ġof Ġa Ġmult ivariate Ġtime - series Ġinput Ġsequence ĠIN LINE FORM 1 Ġto Ġfixed Ġoutput ĠIN LINE FORM 2 Ġ, Ġi . e , Ġmat hematically ĠIN LINE FORM 3 Ġ: ĠIN LINE FORM 4 Ġ, Ġwhere Ġc Ġand Ġt Ġdenote Ġthe ĠEEG Ġchannels Ġand Ġtime Ġinst ants Ġrespectively . </s> Pre processing Ġstep </s> We Ġfollow Ġsimilar Ġpre - processing Ġsteps Ġon Ġraw ĠEEG Ġdata Ġas Ġreported Ġin ĠB IB REF 17 Ġ( ocular Ġartifact Ġremoval Ġusing Ġblind Ġsource Ġseparation , Ġband pass Ġfiltering Ġand Ġsubtract ing Ġmean Ġvalue Ġfrom Ġeach Ġchannel ) Ġexcept Ġthat Ġwe Ġdo Ġnot Ġperform ĠLa pl ac ian Ġfiltering Ġstep Ġsince Ġsuch Ġhigh - pass Ġfiltering Ġmay Ġdecrease Ġinformation Ġcontent Ġfrom Ġthe Ġsignals Ġin Ġthe Ġselected Ġbandwidth . </s> J oint Ġvariability Ġof Ġelectrodes </s> Mult ich annel ĠEEG Ġdata Ġis Ġhigh Ġdimensional Ġmult ivariate Ġtime Ġseries Ġdata Ġwhose Ġdimension ality Ġdepends Ġon Ġthe Ġnumber Ġof Ġelectrodes . ĠIt Ġis Ġa Ġmajor Ġhurdle Ġto Ġoptim ally Ġencode Ġinformation Ġfrom Ġthese ĠEEG Ġdata Ġinto Ġlower Ġdimensional Ġspace . ĠIn Ġfact , Ġour Ġinvestigation Ġbased Ġon Ġa Ġdevelopment Ġset Ġ( as Ġwe Ġexplain Ġlater ) Ġshowed Ġthat Ġwell - known Ġdeep Ġneural Ġnetworks Ġ( e . g ., Ġfully Ġconnected Ġnetworks Ġsuch Ġas Ġconv olution al Ġneural Ġnetworks , Ġrecurrent Ġneural Ġnetworks Ġand Ġaut oen cod ers ) Ġfail Ġto Ġindividually Ġlearn Ġsuch Ġcomplex Ġfeature Ġrepresentations Ġfrom Ġsingle - trial ĠEEG Ġdata . ĠBesides , Ġwe Ġfound Ġthat Ġinstead Ġof Ġusing Ġthe Ġraw Ġmulti - channel Ġhigh - dimensional ĠEEG Ġrequiring Ġlarge Ġtraining Ġtimes Ġand Ġresource Ġrequirements , Ġit Ġis Ġadvantageous Ġto Ġfirst Ġreduce Ġits Ġdimension ality Ġby Ġcapturing Ġthe Ġinformation Ġtransfer Ġamong Ġthe Ġelectrodes . ĠInstead Ġof Ġthe Ġconventional Ġapproach Ġof Ġselecting Ġa Ġhandful Ġof Ġchannels Ġas ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġwe Ġaddress Ġthis Ġby Ġcomputing Ġthe Ġchannel Ġcross - c ov ari ance , Ġresulting Ġin Ġpositive , Ġsemi - def inite Ġmat rices Ġencoding Ġthe Ġconnectivity Ġof Ġthe Ġelectrodes . ĠWe Ġdefine Ġchannel Ġcross - c ov ari ance Ġ( CC V ) Ġbetween Ġany Ġtwo Ġelectrodes ĠIN LINE FORM 0 Ġand ĠIN LINE FORM 1 Ġas : ĠIN LINE FORM 2 Ġ. ĠNext , Ġwe Ġreject Ġthe Ġchannels Ġwhich Ġhave Ġsignificantly Ġlower Ġcross - c ov ari ance Ġthan Ġauto - c ov ari ance Ġvalues Ġ( where Ġauto - c ov ari ance Ġimplies ĠCC V Ġon Ġsame Ġelectrode ). ĠWe Ġfound Ġthis Ġmeasure Ġto Ġbe Ġessential Ġas Ġthe Ġhigher Ġcognitive Ġprocesses Ġunderlying Ġspeech Ġplanning Ġand Ġsynthesis Ġinvolve Ġfrequent Ġinformation Ġexchange Ġbetween Ġdifferent Ġparts Ġof Ġthe Ġbrain . ĠHence , Ġsuch Ġmat rices Ġoften Ġcontain Ġmore Ġdiscrim in ative Ġfeatures Ġand Ġhidden Ġinformation Ġthan Ġmere Ġraw Ġsignals . ĠThis Ġis Ġessentially Ġdifferent Ġthan Ġour Ġprevious Ġwork ĠB IB REF 16 Ġwhere Ġwe Ġextract Ġper - channel Ġ1 - D Ġcovari ance Ġinformation Ġand Ġfeed Ġit Ġto Ġthe Ġnetworks . ĠWe Ġpresent Ġour Ġsample Ġ2 - D ĠEEG Ġcross - c ov ari ance Ġmat rices Ġ( of Ġtwo Ġindividuals ) Ġin ĠFig . ĠFIG REF 2 Ġ. </s> CNN Ġ& ĠL ST M </s> In Ġorder Ġto Ġdecode Ġspatial Ġconnections Ġbetween Ġthe Ġelectrodes Ġfrom Ġthe Ġchannel Ġcovari ance Ġmatrix , Ġwe Ġuse Ġa ĠCNN ĠB IB REF 19 Ġ, Ġin Ġparticular Ġa Ġfour - lay ered Ġ2 D ĠCNN Ġstacking Ġtwo Ġconv olution al Ġand Ġtwo Ġfully Ġconnected Ġhidden Ġlayers . ĠThe ĠIN LINE FORM 0 Ġfeature Ġmap Ġat Ġa Ġgiven ĠCNN Ġlayer Ġwith Ġinput ĠIN LINE FORM 1 Ġ, Ġweight Ġmatrix ĠIN LINE FORM 2 Ġand Ġbias ĠIN LINE FORM 3 Ġis Ġobtained Ġas : ĠIN LINE FORM 4 Ġ. ĠAt Ġthis Ġfirst Ġlevel Ġof Ġhierarchy , Ġthe Ġnetwork Ġis Ġtrained Ġwith Ġthe Ġcorresponding Ġlabels Ġas Ġtarget Ġoutputs , Ġoptimizing Ġa Ġcross - ent ropy Ġcost Ġfunction . ĠIn Ġparallel , Ġwe Ġapply Ġa Ġfour - lay ered Ġrecurrent Ġneural Ġnetwork Ġon Ġthe Ġchannel Ġcovari ance Ġmat rices Ġto Ġexplore Ġthe Ġhidden Ġtemporal Ġfeatures Ġof Ġthe Ġelectrodes . ĠName ly , Ġwe Ġexploit Ġan ĠL ST M ĠB IB REF 20 Ġconsisting Ġof Ġtwo Ġfully Ġconnected Ġhidden Ġlayers , Ġstacked Ġwith Ġtwo ĠL ST M Ġlayers Ġand Ġtrained Ġin Ġa Ġsimilar Ġmanner Ġas ĠCNN . </s> Deep Ġaut oen c oder Ġfor Ġspat io - tem poral Ġinformation </s> As Ġwe Ġfound Ġthe Ġindividually - trained Ġparallel Ġnetworks Ġ( CNN Ġand ĠL ST M ) Ġto Ġbe Ġuseful Ġ( see ĠTable ĠT AB REF 12 Ġ), Ġwe Ġsuspected Ġthe Ġcombination Ġof Ġthese Ġtwo Ġnetworks Ġcould Ġprovide Ġa Ġmore Ġpowerful Ġdiscrim in ative Ġspatial Ġand Ġtemporal Ġrepresentation Ġof Ġthe Ġdata Ġthan Ġeach Ġindependent Ġnetwork . ĠAs Ġsuch , Ġwe Ġconc aten ate Ġthe Ġlast Ġfully - connected Ġlayer Ġfrom Ġthe ĠCNN Ġwith Ġits Ġcounterpart Ġin Ġthe ĠL ST M Ġto Ġcompose Ġa Ġsingle Ġfeature Ġvector Ġbased Ġon Ġthese Ġtwo Ġpen ultimate Ġlayers . ĠUltimately , Ġthis Ġforms Ġa Ġjoint Ġspat io - tem poral Ġencoding Ġof Ġthe Ġcross - c ov ari ance Ġmatrix . </s> In Ġorder Ġto Ġfurther Ġreduce Ġthe Ġdimension ality Ġof Ġthe Ġspat io - tem poral Ġenc od ings Ġand Ġcancel Ġbackground Ġnoise Ġeffects ĠB IB REF 21 Ġ, Ġwe Ġtrain Ġan Ġun super vised Ġdeep Ġaut oen oder Ġ( DA E ) Ġon Ġthe Ġfused Ġheter ogeneous Ġfeatures Ġproduced Ġby Ġthe Ġcombined ĠCNN Ġand ĠL ST M Ġinformation . ĠThe ĠD AE Ġforms Ġour Ġsecond Ġlevel Ġof Ġhierarchy , Ġwith Ġ3 Ġencoding Ġand Ġ3 Ġdecoding Ġlayers , Ġand Ġmean Ġsquared Ġerror Ġ( M SE ) Ġas Ġthe Ġcost Ġfunction . </s> Class ification Ġwith ĠExtreme ĠGrad ient ĠBoost </s> At Ġthe Ġthird Ġlevel Ġof Ġhierarchy , Ġthe Ġdiscrete Ġlatent Ġvector Ġrepresentation Ġof Ġthe Ġdeep Ġaut oen c oder Ġis Ġfed Ġinto Ġan ĠExtreme ĠGrad ient ĠBoost Ġbased Ġclassification Ġlayer ĠB IB REF 22 Ġ, ĠB IB REF 23 Ġmotivated Ġby ĠB IB REF 21 Ġ. ĠIt Ġis Ġa Ġregular ized Ġgradient Ġboosted Ġdecision Ġtree Ġthat Ġperforms Ġwell Ġon Ġstructured Ġproblems . ĠSince Ġour ĠEEG - ph on ological Ġpair wise Ġclassification Ġhas Ġan Ġinternal Ġstructure Ġinvolving Ġindividual Ġphon em es Ġand Ġwords , Ġit Ġseems Ġto Ġbe Ġa Ġreasonable Ġchoice Ġof Ġclass ifier . ĠThe Ġclass ifier Ġreceives Ġits Ġinput Ġfrom Ġthe Ġlatent Ġvectors Ġof Ġthe Ġdeep Ġaut oen c oder Ġand Ġis Ġtrained Ġin Ġa Ġsupervised Ġmanner Ġto Ġoutput Ġthe Ġfinal Ġpredicted Ġclasses Ġcorresponding Ġto Ġthe Ġspeech Ġimagery . </s> Dat as et </s> We Ġevaluate Ġour Ġmodel Ġon Ġa Ġpublicly Ġavailable Ġdataset , ĠK ARA ĠONE ĠB IB REF 17 Ġ, Ġcomposed Ġof Ġmultim od al Ġdata Ġfor Ġstimulus - based , Ġimagined Ġand Ġarticulated Ġspeech Ġstate Ġcorresponding Ġto Ġ7 Ġphon emic / sy ll ab ic Ġ( Ġ/ iy /, Ġ/ p iy /, Ġ/ t iy /, Ġ/ d iy /, Ġ/ u w /, Ġ/ m /, Ġ/ n / Ġ) Ġas Ġwell Ġas Ġ4 Ġwords ( pat , Ġpot , Ġknew Ġand Ġgn aw ). ĠThe Ġdataset Ġconsists Ġof Ġ14 Ġparticipants , Ġwith Ġeach Ġprompt Ġpresented Ġ11 Ġtimes Ġto Ġeach Ġindividual . ĠSince Ġour Ġintention Ġis Ġto Ġclassify Ġthe Ġphon ological Ġcategories Ġfrom Ġhuman Ġthoughts , Ġwe Ġdiscard Ġthe Ġfacial Ġand Ġaudio Ġinformation Ġand Ġonly Ġconsider Ġthe ĠEEG Ġdata Ġcorresponding Ġto Ġimagined Ġspeech . ĠIt Ġis Ġnoteworthy Ġthat Ġgiven Ġthe Ġmixed Ġnature Ġof ĠEEG Ġsignals , Ġit Ġis Ġreportedly Ġchallenging Ġto Ġattain Ġa Ġpair wise ĠEEG - ph on eme Ġmapping ĠB IB REF 18 Ġ. ĠIn Ġorder Ġto Ġexplore Ġthe Ġproblem Ġspace , Ġwe Ġthus Ġspecifically Ġtarget Ġfive Ġbinary Ġclassification Ġproblems Ġaddressed Ġin ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġ, Ġi . e Ġpresence / abs ence Ġof Ġconson ants , Ġphon emic Ġnasal , Ġbil ab ial , Ġhigh - front Ġvow els Ġand Ġhigh - back Ġvow els . </s> Training Ġand Ġhyper param eter Ġselection </s> We Ġperformed Ġtwo Ġsets Ġof Ġexperiments Ġwith Ġthe Ġsingle - trial ĠEEG Ġdata . ĠIn ĠPH ASE - ONE , Ġour Ġgoals Ġwas Ġto Ġidentify Ġthe Ġbest Ġarchitectures Ġand Ġhyper param eters Ġfor Ġour Ġnetworks Ġwith Ġa Ġreasonable Ġnumber Ġof Ġruns . ĠFor ĠPH ASE - ONE , Ġwe Ġrandomly Ġshuff led Ġand Ġdivided Ġthe Ġdata Ġ( 19 13 Ġsignals Ġfrom Ġ14 Ġindividuals ) Ġinto Ġtrain Ġ( 80 %), Ġdevelopment Ġ( 10 %) Ġand Ġtest Ġsets Ġ( 10 %). ĠIn ĠPH ASE - TW O , Ġin Ġorder Ġto Ġperform Ġa Ġfair Ġcomparison Ġwith Ġthe Ġprevious Ġmethods Ġreported Ġon Ġthe Ġsame Ġdataset , Ġwe Ġperform Ġa Ġleave - one - subject Ġout Ġcross - valid ation Ġexperiment Ġusing Ġthe Ġbest Ġsettings Ġwe Ġlearn Ġfrom ĠPH ASE - ONE . </s> The Ġarchitectural Ġparameters Ġand Ġhyper param eters Ġlisted Ġin ĠTable ĠT AB REF 6 Ġwere Ġselected Ġthrough Ġan Ġexhaustive Ġgrid - search Ġbased Ġon Ġthe Ġvalidation Ġset Ġof ĠPH ASE - ONE . ĠWe Ġconducted Ġa Ġseries Ġof Ġempirical Ġstudies Ġstarting Ġfrom Ġsingle Ġhidden - lay ered Ġnetworks Ġfor Ġeach Ġof Ġthe Ġblocks Ġand , Ġbased Ġon Ġthe Ġvalidation Ġaccuracy , Ġwe Ġincreased Ġthe Ġdepth Ġof Ġeach Ġgiven Ġnetwork Ġand Ġselected Ġthe Ġoptimal Ġparam etric Ġset Ġfrom Ġall Ġpossible Ġcombinations Ġof Ġparameters . ĠFor Ġthe Ġgradient Ġboosting Ġclassification , Ġwe Ġfixed Ġthe Ġmaximum Ġdepth Ġat Ġ10 , Ġnumber Ġof Ġestim ators Ġat Ġ5000 , Ġlearning Ġrate Ġat Ġ0 . 1 , Ġregular ization Ġcoefficient Ġat Ġ0 . 3 , Ġsubs ample Ġratio Ġat Ġ0 . 8 , Ġand Ġcolumn - sample / iter ation Ġat Ġ0 . 4 . ĠWe Ġdid Ġnot Ġfind Ġany Ġnotable Ġchange Ġof Ġaccuracy Ġwhile Ġvarying Ġother Ġhyper param eters Ġwhile Ġtraining Ġgradient Ġboost Ġclass ifier . </s> Performance Ġanalysis Ġand Ġdiscussion </s> To Ġdemonstrate Ġthe Ġsignificance Ġof Ġthe Ġhierarchical ĠCNN - L ST M - DA E Ġmethod , Ġwe Ġconducted Ġseparate Ġexperiments Ġwith Ġthe Ġindividual Ġnetworks Ġin ĠPH ASE - ONE Ġof Ġexperiments Ġand Ġsummarized Ġthe Ġresults Ġin ĠTable ĠT AB REF 12 ĠFrom Ġthe Ġaverage Ġaccuracy Ġscores , Ġwe Ġobserve Ġthat Ġthe Ġmixed Ġnetwork Ġperforms Ġmuch Ġbetter Ġthan Ġindividual Ġblocks Ġwhich Ġis Ġin Ġagreement Ġwith Ġthe Ġfindings Ġin ĠB IB REF 21 Ġ. ĠA Ġdetailed Ġanalysis Ġon Ġrepeated Ġruns Ġfurther Ġshows Ġthat Ġin Ġmost Ġof Ġthe Ġcases , ĠL ST M Ġalone Ġdoes Ġnot Ġperform Ġbetter Ġthan Ġchance . ĠCNN , Ġon Ġthe Ġother Ġhand , Ġis Ġheavily Ġbiased Ġtowards Ġthe Ġclass Ġlabel Ġwhich Ġsees Ġmore Ġtraining Ġdata Ġcorresponding Ġto Ġit . ĠThough Ġthe Ġsituation Ġimproves Ġwith Ġcombined ĠCNN - L ST M , Ġour Ġanalysis Ġclearly Ġshows Ġthe Ġnecessity Ġof Ġa Ġbetter Ġencoding Ġscheme Ġto Ġutilize Ġthe Ġcombined Ġfeatures Ġrather Ġthan Ġmere Ġconc aten ation Ġof Ġthe Ġpen ultimate Ġfeatures Ġof Ġboth Ġnetworks . </s> The Ġvery Ġfact Ġthat Ġour Ġcombined Ġnetwork Ġimproves Ġthe Ġclassification Ġaccuracy Ġby Ġa Ġmean Ġmargin Ġof Ġ14 . 45 % Ġthan Ġthe ĠCNN - L ST M Ġnetwork Ġindeed Ġreveals Ġthat Ġthe Ġaut oen c oder Ġcontributes Ġtowards Ġfiltering Ġout Ġthe Ġunrelated Ġand Ġnoisy Ġfeatures Ġfrom Ġthe Ġconc aten ated Ġpen ultimate Ġfeature Ġset . ĠIt Ġalso Ġproves Ġthat Ġthe Ġcombined Ġsupervised Ġand Ġun super vised Ġneural Ġnetworks , Ġtrained Ġhierarch ically , Ġcan Ġlearn Ġthe Ġdiscrim in ative Ġmanifold Ġbetter Ġthan Ġthe Ġindividual Ġnetworks Ġand Ġit Ġis Ġcrucial Ġfor Ġimproving Ġthe Ġclassification Ġaccuracy . ĠIn Ġaddition Ġto Ġaccuracy , Ġwe Ġalso Ġprovide Ġthe Ġk appa Ġcoefficients ĠB IB REF 24 Ġof Ġour Ġmethod Ġin ĠFig . ĠFIG REF 14 Ġ. ĠHere , Ġa Ġhigher Ġmean Ġk appa Ġvalue Ġcorresponding Ġto Ġa Ġtask Ġimplies Ġthat Ġthe Ġnetwork Ġis Ġable Ġto Ġfind Ġbetter Ġdiscrim in ative Ġinformation Ġfrom Ġthe ĠEEG Ġdata Ġbeyond Ġrandom Ġdecisions . ĠThe Ġmaximum Ġabove - chance Ġaccuracy Ġ( 75 . 92 %) Ġis Ġrecorded Ġfor Ġpresence / abs ence Ġof Ġthe Ġvowel Ġtask Ġand Ġthe Ġminimum Ġ( 49 . 14 %) Ġis Ġrecorded Ġfor Ġthe ĠIN LINE FORM 0 Ġ. </s> To Ġfurther Ġinvestigate Ġthe Ġfeature Ġrepresentation Ġachieved Ġby Ġour Ġmodel , Ġwe Ġplot ĠT - dist ributed ĠSt och astic ĠNeighbor ĠEmb edd ing Ġ( t S NE ) Ġcorresponding Ġto ĠIN LINE FORM 0 Ġand ĠV / C Ġclassification Ġtasks Ġin ĠFig . ĠFIG REF 8 Ġ. ĠWe Ġparticularly Ġselect Ġthese Ġtwo Ġtasks Ġas Ġour Ġmodel Ġexhibits Ġrespectively Ġminimum Ġand Ġmaximum Ġperformance Ġfor Ġthese Ġtwo . ĠThe Ġt S NE Ġvisualization Ġreveals Ġthat Ġthe Ġsecond Ġset Ġof Ġfeatures Ġare Ġmore Ġeasily Ġsepar able Ġthan Ġthe Ġfirst Ġone , Ġthereby Ġgiving Ġa Ġrationale Ġfor Ġour Ġperformance . </s> Next , Ġwe Ġprovide Ġperformance Ġcomparison Ġof Ġthe Ġproposed Ġapproach Ġwith Ġthe Ġbaseline Ġmethods Ġfor ĠPH ASE - TW O Ġof Ġour Ġstudy Ġ( cross - valid ation Ġexperiment ) Ġin ĠTable ĠT AB REF 15 Ġ. ĠSince Ġthe Ġmodel Ġencounters Ġthe Ġunseen Ġdata Ġof Ġa Ġnew Ġsubject Ġfor Ġtesting , Ġand Ġgiven Ġthe Ġhigh Ġinter - subject Ġvariability Ġof Ġthe ĠEEG Ġdata , Ġa Ġreduction Ġin Ġthe Ġaccuracy Ġwas Ġexpected . ĠHowever , Ġour Ġnetwork Ġstill Ġmanaged Ġto Ġachieve Ġan Ġimprovement Ġof Ġ18 . 91 , Ġ9 . 95 , Ġ67 . 15 , Ġ2 . 83 Ġand Ġ13 . 70 Ġ% Ġover ĠB IB REF 17 Ġ. ĠBesides , Ġour Ġbest Ġmodel Ġshows Ġmore Ġreliability Ġcompared Ġto Ġprevious Ġworks : ĠThe Ġstandard Ġdeviation Ġof Ġour Ġmodel 's Ġclassification Ġaccuracy Ġacross Ġall Ġthe Ġtasks Ġis Ġreduced Ġfrom Ġ22 . 59 % ĠB IB REF 17 Ġand Ġ17 . 52 % ĠB IB REF 18 Ġto Ġa Ġmere Ġ5 . 41 %. </s> Conclusion Ġand Ġfuture Ġdirection </s> In Ġan Ġattempt Ġto Ġmove Ġa Ġstep Ġtowards Ġunderstanding Ġthe Ġspeech Ġinformation Ġencoded Ġin Ġbrain Ġsignals , Ġwe Ġdeveloped Ġa Ġnovel Ġmixed Ġdeep Ġneural Ġnetwork Ġscheme Ġfor Ġa Ġnumber Ġof Ġbinary Ġclassification Ġtasks Ġfrom Ġspeech Ġimagery ĠEEG Ġdata . ĠUnlike Ġprevious Ġapproaches Ġwhich Ġmostly Ġdeal Ġwith Ġsubject - dependent Ġclassification Ġof ĠEEG Ġinto Ġdiscrete Ġvowel Ġor Ġword Ġlabels , Ġthis Ġwork Ġinvestigates Ġa Ġsubject - inv ari ant Ġmapping Ġof ĠEEG Ġdata Ġwith Ġdifferent Ġphon ological Ġcategories , Ġvarying Ġwidely Ġin Ġterms Ġof Ġunderlying Ġartic ulator Ġmotions Ġ( eg : Ġinvolvement Ġor Ġnon - inv olve ment Ġof Ġlips Ġand Ġvel um , Ġvariation Ġof Ġtongue Ġmovements Ġetc ). ĠOur Ġmodel Ġtakes Ġan Ġadvantage Ġof Ġfeature Ġextraction Ġcapability Ġof ĠCNN , ĠL ST M Ġas Ġwell Ġas Ġthe Ġdeep Ġlearning Ġbenefit Ġof Ġdeep Ġaut oen cod ers . ĠWe Ġtook ĠB IB REF 17 Ġ, ĠB IB REF 18 Ġas Ġthe Ġbaseline Ġworks Ġinvestigating Ġthe Ġsame Ġproblem Ġand Ġcompared Ġour Ġperformance Ġwith Ġtheirs . ĠOur Ġproposed Ġmethod Ġhighly Ġoutper forms Ġthe Ġexisting Ġmethods Ġacross Ġall Ġthe Ġfive Ġbinary Ġclassification Ġtasks Ġby Ġa Ġlarge Ġaverage Ġmargin Ġof Ġ22 . 51 %. </s> Acknowled gments </s> This Ġwork Ġwas Ġfunded Ġby Ġthe ĠNatural ĠSciences Ġand ĠEngineering ĠResearch ĠCouncil Ġ( NS ERC ) Ġof ĠCanada Ġand ĠCanadian ĠInstitutes Ġfor ĠHealth ĠResearch Ġ( CI HR ).