Unanswerable
NONE
No
BOOLEAN
We explored leveraging multimodality for the NLU module in the SDS pipeline. As our AMIE in-cabin dataset has video and audio recordings, we investigated 3 modalities for the NLU: text, audio, and video. For text (language) modality, our previous work BIBREF1 presents the details of our best-performing Hierarchical & Joint Bi-LSTM models BIBREF3, BIBREF4, BIBREF5, BIBREF6 (H-Joint-2, see SECREF5) and the results for utterance-level intent recognition and word-level slot filling via transcribed and recognized (ASR output) textual data, using word embeddings (GloVe BIBREF7) as features. This study explores the following multimodal features:
1909.13714
Introduction </s> Understanding Ġpassenger Ġint ents Ġfrom Ġspoken Ġinteractions Ġand Ġcar 's Ġvision Ġ( both Ġinside Ġand Ġoutside Ġthe Ġvehicle ) Ġare Ġimportant Ġbuilding Ġblocks Ġtowards Ġdeveloping Ġcontextual Ġdialog Ġsystems Ġfor Ġnatural Ġinteractions Ġin Ġautonomous Ġvehicles Ġ( AV ). ĠIn Ġthis Ġstudy , Ġwe Ġcontinued Ġexploring ĠAM IE Ġ( Autom ated - veh icle ĠMult im od al ĠIn - c abin ĠExperience ), Ġthe Ġin - c abin Ġagent Ġresponsible Ġfor Ġhandling Ġcertain Ġmultim od al Ġpassenger - veh icle Ġinteractions . ĠWhen Ġthe Ġpassengers Ġgive Ġinstructions Ġto ĠAM IE , Ġthe Ġagent Ġshould Ġparse Ġsuch Ġcommands Ġproperly Ġconsidering Ġavailable Ġthree Ġmod alities Ġ( language / text , Ġaudio , Ġvideo ) Ġand Ġtrigger Ġthe Ġappropriate Ġfunctionality Ġof Ġthe ĠAV Ġsystem . ĠWe Ġhad Ġcollected Ġa Ġmultim od al Ġin - c abin Ġdataset Ġwith Ġmulti - turn Ġdialog ues Ġbetween Ġthe Ġpassengers Ġand ĠAM IE Ġusing Ġa ĠWizard - of - O z Ġscheme Ġvia Ġrealistic Ġscaven ger Ġhunt Ġgame . ĠIn Ġour Ġprevious Ġexplor ations ĠB IB REF 0 , ĠB IB REF 1 , Ġwe Ġexperimented Ġwith Ġvarious ĠR NN - based Ġmodels Ġto Ġdetect Ġutter ance - level Ġint ents Ġ( set Ġdestination , Ġchange Ġroute , Ġgo Ġfaster , Ġgo Ġslower , Ġstop , Ġpark , Ġpull Ġover , Ġdrop Ġoff , Ġopen Ġdoor , Ġand Ġothers ) Ġalong Ġwith Ġintent Ġkeywords Ġand Ġrelevant Ġslots Ġ( location , Ġposition / direction , Ġobject , Ġgesture / g aze , Ġtime - gu id ance , Ġperson ) Ġassociated Ġwith Ġthe Ġaction Ġto Ġbe Ġperformed Ġin Ġour ĠAV Ġscenarios . ĠIn Ġthis Ġrecent Ġwork , Ġwe Ġpropose Ġto Ġdiscuss Ġthe Ġbenefits Ġof Ġmultim od al Ġunderstanding Ġof Ġin - c abin Ġutter ances Ġby Ġincorporating Ġverbal / language Ġinput Ġ( text Ġand Ġspeech Ġembed d ings ) Ġtogether Ġwith Ġthe Ġnon - verbal / ac oustic Ġand Ġvisual Ġinput Ġfrom Ġinside Ġand Ġoutside Ġthe Ġvehicle Ġ( i . e ., Ġpassenger Ġgestures Ġand Ġgaze Ġfrom Ġin - c abin Ġvideo Ġstream , Ġreferred Ġobjects Ġoutside Ġof Ġthe Ġvehicle Ġfrom Ġthe Ġroad Ġview Ġcamera Ġstream ). ĠOur Ġexperimental Ġresults Ġoutper formed Ġtext - only Ġbas elines Ġand Ġwith Ġmultim od ality , Ġwe Ġachieved Ġimproved Ġperformances Ġfor Ġutter ance - level Ġintent Ġdetection Ġand Ġslot Ġfilling . </s> Method ology </s> We Ġexplored Ġleveraging Ġmultim od ality Ġfor Ġthe ĠNL U Ġmodule Ġin Ġthe ĠS DS Ġpipeline . ĠAs Ġour ĠAM IE Ġin - c abin Ġdataset Ġhas Ġvideo Ġand Ġaudio Ġrecordings , Ġwe Ġinvestigated Ġ3 Ġmod alities Ġfor Ġthe ĠNL U : Ġtext , Ġaudio , Ġand Ġvideo . ĠFor Ġtext Ġ( language ) Ġmod ality , Ġour Ġprevious Ġwork ĠB IB REF 1 Ġpresents Ġthe Ġdetails Ġof Ġour Ġbest - performing ĠHier arch ical Ġ& ĠJoint ĠBi - L ST M Ġmodels ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġ( H - J oint - 2 , Ġsee ĠSEC REF 5 ) Ġand Ġthe Ġresults Ġfor Ġutter ance - level Ġintent Ġrecognition Ġand Ġword - level Ġslot Ġfilling Ġvia Ġtransc ribed Ġand Ġrecognized Ġ( AS R Ġoutput ) Ġtextual Ġdata , Ġusing Ġword Ġembed d ings Ġ( G lo Ve ĠB IB REF 7 ) Ġas Ġfeatures . ĠThis Ġstudy Ġexplores Ġthe Ġfollowing Ġmultim od al Ġfeatures : </s> Spe ech ĠEmb edd ings : ĠWe Ġincorporated Ġpre - trained Ġspeech Ġembed d ings Ġ( Spe ech 2 V ec ĠB IB REF 8 ) Ġas Ġfeatures , Ġtrained Ġon Ġa Ġcorpus Ġof Ġ500 Ġhours Ġof Ġspeech Ġfrom ĠLib ri Spe ech . ĠSpeech 2 V ec Ġis Ġconsidered Ġas Ġa Ġspeech Ġversion Ġof ĠWord 2 V ec ĠB IB REF 9 Ġwhich Ġis Ġcompared Ġwith ĠWord 2 V ec Ġvectors Ġtrained Ġon Ġthe Ġtranscript Ġof Ġthe Ġsame Ġspeech Ġcorpus . ĠWe Ġexperimented Ġwith Ġconc aten ating Ġword Ġand Ġspeech Ġembed d ings Ġby Ġusing Ġpre - trained ĠGl o Ve Ġembed d ings Ġ( 6 B Ġtokens , Ġ400 K Ġvoc ab , Ġdim = 100 ), ĠSpeech 2 V ec Ġembed d ings Ġ( 37 . 6 K Ġvoc ab , Ġdim = 100 ), Ġand Ġits ĠWord 2 V ec Ġcounterpart Ġ( 37 . 6 K Ġvoc ab , Ġdim = 100 ). </s> Audio ĠFeatures : ĠUsing Ġopen S MI LE ĠB IB REF 10 , Ġ15 82 Ġaudio Ġfeatures Ġare Ġextracted Ġfor Ġeach Ġutter ance Ġusing Ġthe Ġsegment ed Ġaudio Ġclips Ġfrom Ġin - c abin ĠAM IE Ġdataset . ĠThese Ġare Ġthe ĠINTER SP E ECH Ġ2010 ĠPar aling u istic ĠChallenge Ġfeatures Ġ( IS 10 ) Ġincluding ĠPC M Ġloud ness , ĠMF CC , Ġlog ĠMel ĠFre q . ĠBand , ĠL SP , Ġetc . ĠB IB REF 11 . </s> Video ĠFeatures : ĠUsing Ġthe Ġfeature Ġextraction Ġprocess Ġdescribed Ġin ĠB IB REF 12 , Ġwe Ġextracted Ġintermediate ĠCNN Ġfeatures Ġfor Ġeach Ġsegment ed Ġvideo Ġclip Ġfrom ĠAM IE Ġdataset . ĠFor Ġany Ġgiven Ġinput Ġvideo Ġclip Ġ( se gment ed Ġfor Ġeach Ġutter ance ), Ġone Ġframe Ġper Ġsecond Ġis Ġsampled Ġand Ġits Ġvisual Ġdescriptor Ġis Ġextracted Ġfrom Ġthe Ġactiv ations Ġof Ġthe Ġintermediate Ġconv olution Ġlayers Ġof Ġa Ġpre - trained ĠCNN . ĠWe Ġused Ġthe Ġpre - trained ĠIn ception - Res Net - v 2 Ġmodel ĠB IB REF 13 Ġand Ġgenerated Ġ4096 - dim Ġfeatures Ġfor Ġeach Ġsample . ĠWe Ġexperimented Ġwith Ġadding Ġ2 Ġsources Ġof Ġvisual Ġinformation : Ġ( i ) Ġcabin / pass enger Ġview Ġfrom Ġthe ĠBack Driver ĠRGB Ġcamera Ġrecordings , Ġ( ii ) Ġroad / outside Ġview Ġfrom Ġthe ĠDash Cam ĠRGB Ġvideo Ġstreams . </s> Exper imental ĠResults </s> For Ġincorporating Ġspeech Ġembed d ings Ġexperiments , Ġperformance Ġresults Ġof ĠNL U Ġmodels Ġon Ġin - c abin Ġdata Ġwith Ġvarious Ġfeature Ġconc aten ations Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 3 , Ġusing Ġour Ġprevious Ġhierarchical Ġjoint Ġmodel Ġ( H - J oint - 2 ). ĠWhen Ġused Ġin Ġisolation , ĠWord 2 V ec Ġand ĠSpeech 2 V ec Ġachieves Ġcomparable Ġperformances , Ġwhich Ġcannot Ġreach ĠGl o Ve Ġperformance . ĠThis Ġwas Ġexpected Ġas Ġthe Ġpre - trained ĠSpeech 2 V ec Ġvectors Ġhave Ġlower Ġvocabulary Ġcoverage Ġthan ĠGl o Ve . ĠYet , Ġwe Ġobserved Ġthat Ġconc aten ating ĠGl o Ve Ġ+ ĠSpeech 2 V ec , Ġand Ġfurther ĠGl o Ve Ġ+ ĠWord 2 V ec Ġ+ ĠSpeech 2 V ec Ġyields Ġbetter ĠNL U Ġresults : ĠF 1 - score Ġincreased Ġfrom Ġ0 . 89 Ġto Ġ0 . 91 Ġfor Ġintent Ġrecognition , Ġfrom Ġ0 . 96 Ġto Ġ0 . 97 Ġfor Ġslot Ġfilling . </s> For Ġmultim od al Ġ( audio Ġ& Ġvideo ) Ġfeatures Ġexploration , Ġperformance Ġresults Ġof Ġthe Ġcompared Ġmodels Ġwith Ġvarying Ġmod ality / feature Ġconc aten ations Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 4 . ĠSince Ġthese Ġaudio / video Ġfeatures Ġare Ġextracted Ġper Ġutter ance Ġ( on Ġsegment ed Ġaudio Ġ& Ġvideo Ġclips ), Ġwe Ġexperimented Ġwith Ġthe Ġutter ance - level Ġintent Ġrecognition Ġtask Ġonly , Ġusing Ġhierarchical Ġjoint Ġlearning Ġ( H - J oint - 2 ). ĠWe Ġinvestigated Ġthe Ġaudio - visual Ġfeature Ġadditions Ġon Ġtop Ġof Ġtext - only Ġand Ġtext + speech Ġembed ding Ġmodels . ĠAdding Ġopen S MI LE / IS 10 Ġfeatures Ġfrom Ġaudio , Ġas Ġwell Ġas Ġincorporating Ġintermediate ĠCNN / In ception - Res Net - v 2 Ġfeatures Ġfrom Ġvideo Ġbrought Ġslight Ġimprovements Ġto Ġour Ġintent Ġmodels , Ġreaching Ġ0 . 92 ĠF 1 - score . ĠThese Ġinitial Ġresults Ġusing Ġfeature Ġconc aten ations Ġmay Ġneed Ġfurther Ġexplor ations , Ġespecially Ġfor Ġcertain Ġintent - types Ġsuch Ġas Ġstop Ġ( audio Ġintensity ) Ġor Ġrelevant Ġslots Ġsuch Ġas Ġpassenger Ġgestures / g aze Ġ( from Ġcabin Ġvideo ) Ġand Ġoutside Ġobjects Ġ( from Ġroad Ġvideo ). </s> Conclusion </s> In Ġthis Ġstudy , Ġwe Ġpresent Ġour Ġinitial Ġexplor ations Ġtowards Ġmultim od al Ġunderstanding Ġof Ġpassenger Ġutter ances Ġin Ġautonomous Ġvehicles . ĠWe Ġbriefly Ġshow Ġthat Ġour Ġexperimental Ġresults Ġoutper formed Ġcertain Ġbas elines Ġand Ġwith Ġmultim od ality , Ġwe Ġachieved Ġimproved Ġoverall ĠF 1 - sc ores Ġof Ġ0 . 92 Ġfor Ġutter ance - level Ġintent Ġdetection Ġand Ġ0 . 97 Ġfor Ġword - level Ġslot Ġfilling . ĠThis Ġongoing Ġresearch Ġhas Ġa Ġpotential Ġimpact Ġof Ġexploring Ġreal - world Ġchallenges Ġwith Ġhuman - veh icle - scene Ġinteractions Ġfor Ġautonomous Ġdriving Ġsupport Ġwith Ġspoken Ġutter ances . </s> App end ices </s> AM IE ĠIn - c abin ĠDat as et : ĠWe Ġobtained Ġ13 31 Ġutter ances Ġhaving Ġcommands Ġto ĠAM IE Ġagent Ġfrom Ġour Ġin - c abin Ġdataset . ĠAn notation Ġresults Ġfor Ġutter ance - level Ġintent Ġtypes , Ġslots Ġand Ġintent Ġkeywords Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 7 Ġand ĠTable ĠT AB REF 8 . </s> H ier arch ical Ġ& ĠJoint ĠModel Ġ( H - J oint - 2 ): Ġ2 - level Ġhierarchical Ġjoint Ġlearning Ġmodel Ġthat Ġdetects / ext ract s Ġintent Ġkeywords Ġ& Ġslots Ġusing Ġseq 2 seq ĠBi - L ST Ms Ġfirst Ġ( Level - 1 ), Ġthen Ġonly Ġthe Ġwords Ġthat Ġare Ġpredicted Ġas Ġintent Ġkeywords Ġ& Ġvalid Ġslots Ġare Ġfed Ġinto ĠJoint - 2 Ġmodel Ġ( Level - 2 ), Ġwhich Ġis Ġanother Ġseq 2 seq ĠBi - L ST M Ġnetwork Ġfor Ġutter ance - level Ġintent Ġdetection Ġ( j oint ly Ġtrained Ġwith Ġslots Ġ& Ġintent Ġkeywords ) ĠB IB REF 1 .
Introduction Understanding passenger intents from spoken interactions and car's vision (both inside and outside the vehicle) are important building blocks towards developing contextual dialog systems for natural interactions in autonomous vehicles (AV). In this study, we continued exploring AMIE (Automated-vehicle Multimodal In-cabin Experience), the in-cabin agent responsible for handling certain multimodal passenger-vehicle interactions. When the passengers give instructions to AMIE, the agent should parse such commands properly considering available three modalities (language/text, audio, video) and trigger the appropriate functionality of the AV system. We had collected a multimodal in-cabin dataset with multi-turn dialogues between the passengers and AMIE using a Wizard-of-Oz scheme via realistic scavenger hunt game. In our previous explorations BIBREF0, BIBREF1, we experimented with various RNN-based models to detect utterance-level intents (set destination, change route, go faster, go slower, stop, park, pull over, drop off, open door, and others) along with intent keywords and relevant slots (location, position/direction, object, gesture/gaze, time-guidance, person) associated with the action to be performed in our AV scenarios. In this recent work, we propose to discuss the benefits of multimodal understanding of in-cabin utterances by incorporating verbal/language input (text and speech embeddings) together with the non-verbal/acoustic and visual input from inside and outside the vehicle (i.e., passenger gestures and gaze from in-cabin video stream, referred objects outside of the vehicle from the road view camera stream). Our experimental results outperformed text-only baselines and with multimodality, we achieved improved performances for utterance-level intent detection and slot filling. Methodology We explored leveraging multimodality for the NLU module in the SDS pipeline. As our AMIE in-cabin dataset has video and audio recordings, we investigated 3 modalities for the NLU: text, audio, and video. For text (language) modality, our previous work BIBREF1 presents the details of our best-performing Hierarchical & Joint Bi-LSTM models BIBREF3, BIBREF4, BIBREF5, BIBREF6 (H-Joint-2, see SECREF5) and the results for utterance-level intent recognition and word-level slot filling via transcribed and recognized (ASR output) textual data, using word embeddings (GloVe BIBREF7) as features. This study explores the following multimodal features: Speech Embeddings: We incorporated pre-trained speech embeddings (Speech2Vec BIBREF8) as features, trained on a corpus of 500 hours of speech from LibriSpeech. Speech2Vec is considered as a speech version of Word2Vec BIBREF9 which is compared with Word2Vec vectors trained on the transcript of the same speech corpus. We experimented with concatenating word and speech embeddings by using pre-trained GloVe embeddings (6B tokens, 400K vocab, dim=100), Speech2Vec embeddings (37.6K vocab, dim=100), and its Word2Vec counterpart (37.6K vocab, dim=100). Audio Features: Using openSMILE BIBREF10, 1582 audio features are extracted for each utterance using the segmented audio clips from in-cabin AMIE dataset. These are the INTERSPEECH 2010 Paralinguistic Challenge features (IS10) including PCM loudness, MFCC, log Mel Freq. Band, LSP, etc. BIBREF11. Video Features: Using the feature extraction process described in BIBREF12, we extracted intermediate CNN features for each segmented video clip from AMIE dataset. For any given input video clip (segmented for each utterance), one frame per second is sampled and its visual descriptor is extracted from the activations of the intermediate convolution layers of a pre-trained CNN. We used the pre-trained Inception-ResNet-v2 model BIBREF13 and generated 4096-dim features for each sample. We experimented with adding 2 sources of visual information: (i) cabin/passenger view from the BackDriver RGB camera recordings, (ii) road/outside view from the DashCam RGB video streams. Experimental Results For incorporating speech embeddings experiments, performance results of NLU models on in-cabin data with various feature concatenations can be found in Table TABREF3, using our previous hierarchical joint model (H-Joint-2). When used in isolation, Word2Vec and Speech2Vec achieves comparable performances, which cannot reach GloVe performance. This was expected as the pre-trained Speech2Vec vectors have lower vocabulary coverage than GloVe. Yet, we observed that concatenating GloVe + Speech2Vec, and further GloVe + Word2Vec + Speech2Vec yields better NLU results: F1-score increased from 0.89 to 0.91 for intent recognition, from 0.96 to 0.97 for slot filling. For multimodal (audio & video) features exploration, performance results of the compared models with varying modality/feature concatenations can be found in Table TABREF4. Since these audio/video features are extracted per utterance (on segmented audio & video clips), we experimented with the utterance-level intent recognition task only, using hierarchical joint learning (H-Joint-2). We investigated the audio-visual feature additions on top of text-only and text+speech embedding models. Adding openSMILE/IS10 features from audio, as well as incorporating intermediate CNN/Inception-ResNet-v2 features from video brought slight improvements to our intent models, reaching 0.92 F1-score. These initial results using feature concatenations may need further explorations, especially for certain intent-types such as stop (audio intensity) or relevant slots such as passenger gestures/gaze (from cabin video) and outside objects (from road video). Conclusion In this study, we present our initial explorations towards multimodal understanding of passenger utterances in autonomous vehicles. We briefly show that our experimental results outperformed certain baselines and with multimodality, we achieved improved overall F1-scores of 0.92 for utterance-level intent detection and 0.97 for word-level slot filling. This ongoing research has a potential impact of exploring real-world challenges with human-vehicle-scene interactions for autonomous driving support with spoken utterances. Appendices AMIE In-cabin Dataset: We obtained 1331 utterances having commands to AMIE agent from our in-cabin dataset. Annotation results for utterance-level intent types, slots and intent keywords can be found in Table TABREF7 and Table TABREF8. Hierarchical & Joint Model (H-Joint-2): 2-level hierarchical joint learning model that detects/extracts intent keywords & slots using seq2seq Bi-LSTMs first (Level-1), then only the words that are predicted as intent keywords & valid slots are fed into Joint-2 model (Level-2), which is another seq2seq Bi-LSTM network for utterance-level intent detection (jointly trained with slots & intent keywords) BIBREF1.
Is collected multimodal in cabin dataset public?
5563a3538d311c979c2fb83c1cc9afc66ff6fffc
Is Ġcollected Ġmultim od al Ġin Ġcabin Ġdataset Ġpublic ?
Un answer able
<s> Is Ġcollected Ġmultim od al Ġin Ġcabin Ġdataset Ġpublic ? </s> Introduction </s> Understanding Ġpassenger Ġint ents Ġfrom Ġspoken Ġinteractions Ġand Ġcar 's Ġvision Ġ( both Ġinside Ġand Ġoutside Ġthe Ġvehicle ) Ġare Ġimportant Ġbuilding Ġblocks Ġtowards Ġdeveloping Ġcontextual Ġdialog Ġsystems Ġfor Ġnatural Ġinteractions Ġin Ġautonomous Ġvehicles Ġ( AV ). ĠIn Ġthis Ġstudy , Ġwe Ġcontinued Ġexploring ĠAM IE Ġ( Autom ated - veh icle ĠMult im od al ĠIn - c abin ĠExperience ), Ġthe Ġin - c abin Ġagent Ġresponsible Ġfor Ġhandling Ġcertain Ġmultim od al Ġpassenger - veh icle Ġinteractions . ĠWhen Ġthe Ġpassengers Ġgive Ġinstructions Ġto ĠAM IE , Ġthe Ġagent Ġshould Ġparse Ġsuch Ġcommands Ġproperly Ġconsidering Ġavailable Ġthree Ġmod alities Ġ( language / text , Ġaudio , Ġvideo ) Ġand Ġtrigger Ġthe Ġappropriate Ġfunctionality Ġof Ġthe ĠAV Ġsystem . ĠWe Ġhad Ġcollected Ġa Ġmultim od al Ġin - c abin Ġdataset Ġwith Ġmulti - turn Ġdialog ues Ġbetween Ġthe Ġpassengers Ġand ĠAM IE Ġusing Ġa ĠWizard - of - O z Ġscheme Ġvia Ġrealistic Ġscaven ger Ġhunt Ġgame . ĠIn Ġour Ġprevious Ġexplor ations ĠB IB REF 0 , ĠB IB REF 1 , Ġwe Ġexperimented Ġwith Ġvarious ĠR NN - based Ġmodels Ġto Ġdetect Ġutter ance - level Ġint ents Ġ( set Ġdestination , Ġchange Ġroute , Ġgo Ġfaster , Ġgo Ġslower , Ġstop , Ġpark , Ġpull Ġover , Ġdrop Ġoff , Ġopen Ġdoor , Ġand Ġothers ) Ġalong Ġwith Ġintent Ġkeywords Ġand Ġrelevant Ġslots Ġ( location , Ġposition / direction , Ġobject , Ġgesture / g aze , Ġtime - gu id ance , Ġperson ) Ġassociated Ġwith Ġthe Ġaction Ġto Ġbe Ġperformed Ġin Ġour ĠAV Ġscenarios . ĠIn Ġthis Ġrecent Ġwork , Ġwe Ġpropose Ġto Ġdiscuss Ġthe Ġbenefits Ġof Ġmultim od al Ġunderstanding Ġof Ġin - c abin Ġutter ances Ġby Ġincorporating Ġverbal / language Ġinput Ġ( text Ġand Ġspeech Ġembed d ings ) Ġtogether Ġwith Ġthe Ġnon - verbal / ac oustic Ġand Ġvisual Ġinput Ġfrom Ġinside Ġand Ġoutside Ġthe Ġvehicle Ġ( i . e ., Ġpassenger Ġgestures Ġand Ġgaze Ġfrom Ġin - c abin Ġvideo Ġstream , Ġreferred Ġobjects Ġoutside Ġof Ġthe Ġvehicle Ġfrom Ġthe Ġroad Ġview Ġcamera Ġstream ). ĠOur Ġexperimental Ġresults Ġoutper formed Ġtext - only Ġbas elines Ġand Ġwith Ġmultim od ality , Ġwe Ġachieved Ġimproved Ġperformances Ġfor Ġutter ance - level Ġintent Ġdetection Ġand Ġslot Ġfilling . </s> Method ology </s> We Ġexplored Ġleveraging Ġmultim od ality Ġfor Ġthe ĠNL U Ġmodule Ġin Ġthe ĠS DS Ġpipeline . ĠAs Ġour ĠAM IE Ġin - c abin Ġdataset Ġhas Ġvideo Ġand Ġaudio Ġrecordings , Ġwe Ġinvestigated Ġ3 Ġmod alities Ġfor Ġthe ĠNL U : Ġtext , Ġaudio , Ġand Ġvideo . ĠFor Ġtext Ġ( language ) Ġmod ality , Ġour Ġprevious Ġwork ĠB IB REF 1 Ġpresents Ġthe Ġdetails Ġof Ġour Ġbest - performing ĠHier arch ical Ġ& ĠJoint ĠBi - L ST M Ġmodels ĠB IB REF 3 , ĠB IB REF 4 , ĠB IB REF 5 , ĠB IB REF 6 Ġ( H - J oint - 2 , Ġsee ĠSEC REF 5 ) Ġand Ġthe Ġresults Ġfor Ġutter ance - level Ġintent Ġrecognition Ġand Ġword - level Ġslot Ġfilling Ġvia Ġtransc ribed Ġand Ġrecognized Ġ( AS R Ġoutput ) Ġtextual Ġdata , Ġusing Ġword Ġembed d ings Ġ( G lo Ve ĠB IB REF 7 ) Ġas Ġfeatures . ĠThis Ġstudy Ġexplores Ġthe Ġfollowing Ġmultim od al Ġfeatures : </s> Spe ech ĠEmb edd ings : ĠWe Ġincorporated Ġpre - trained Ġspeech Ġembed d ings Ġ( Spe ech 2 V ec ĠB IB REF 8 ) Ġas Ġfeatures , Ġtrained Ġon Ġa Ġcorpus Ġof Ġ500 Ġhours Ġof Ġspeech Ġfrom ĠLib ri Spe ech . ĠSpeech 2 V ec Ġis Ġconsidered Ġas Ġa Ġspeech Ġversion Ġof ĠWord 2 V ec ĠB IB REF 9 Ġwhich Ġis Ġcompared Ġwith ĠWord 2 V ec Ġvectors Ġtrained Ġon Ġthe Ġtranscript Ġof Ġthe Ġsame Ġspeech Ġcorpus . ĠWe Ġexperimented Ġwith Ġconc aten ating Ġword Ġand Ġspeech Ġembed d ings Ġby Ġusing Ġpre - trained ĠGl o Ve Ġembed d ings Ġ( 6 B Ġtokens , Ġ400 K Ġvoc ab , Ġdim = 100 ), ĠSpeech 2 V ec Ġembed d ings Ġ( 37 . 6 K Ġvoc ab , Ġdim = 100 ), Ġand Ġits ĠWord 2 V ec Ġcounterpart Ġ( 37 . 6 K Ġvoc ab , Ġdim = 100 ). </s> Audio ĠFeatures : ĠUsing Ġopen S MI LE ĠB IB REF 10 , Ġ15 82 Ġaudio Ġfeatures Ġare Ġextracted Ġfor Ġeach Ġutter ance Ġusing Ġthe Ġsegment ed Ġaudio Ġclips Ġfrom Ġin - c abin ĠAM IE Ġdataset . ĠThese Ġare Ġthe ĠINTER SP E ECH Ġ2010 ĠPar aling u istic ĠChallenge Ġfeatures Ġ( IS 10 ) Ġincluding ĠPC M Ġloud ness , ĠMF CC , Ġlog ĠMel ĠFre q . ĠBand , ĠL SP , Ġetc . ĠB IB REF 11 . </s> Video ĠFeatures : ĠUsing Ġthe Ġfeature Ġextraction Ġprocess Ġdescribed Ġin ĠB IB REF 12 , Ġwe Ġextracted Ġintermediate ĠCNN Ġfeatures Ġfor Ġeach Ġsegment ed Ġvideo Ġclip Ġfrom ĠAM IE Ġdataset . ĠFor Ġany Ġgiven Ġinput Ġvideo Ġclip Ġ( se gment ed Ġfor Ġeach Ġutter ance ), Ġone Ġframe Ġper Ġsecond Ġis Ġsampled Ġand Ġits Ġvisual Ġdescriptor Ġis Ġextracted Ġfrom Ġthe Ġactiv ations Ġof Ġthe Ġintermediate Ġconv olution Ġlayers Ġof Ġa Ġpre - trained ĠCNN . ĠWe Ġused Ġthe Ġpre - trained ĠIn ception - Res Net - v 2 Ġmodel ĠB IB REF 13 Ġand Ġgenerated Ġ4096 - dim Ġfeatures Ġfor Ġeach Ġsample . ĠWe Ġexperimented Ġwith Ġadding Ġ2 Ġsources Ġof Ġvisual Ġinformation : Ġ( i ) Ġcabin / pass enger Ġview Ġfrom Ġthe ĠBack Driver ĠRGB Ġcamera Ġrecordings , Ġ( ii ) Ġroad / outside Ġview Ġfrom Ġthe ĠDash Cam ĠRGB Ġvideo Ġstreams . </s> Exper imental ĠResults </s> For Ġincorporating Ġspeech Ġembed d ings Ġexperiments , Ġperformance Ġresults Ġof ĠNL U Ġmodels Ġon Ġin - c abin Ġdata Ġwith Ġvarious Ġfeature Ġconc aten ations Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 3 , Ġusing Ġour Ġprevious Ġhierarchical Ġjoint Ġmodel Ġ( H - J oint - 2 ). ĠWhen Ġused Ġin Ġisolation , ĠWord 2 V ec Ġand ĠSpeech 2 V ec Ġachieves Ġcomparable Ġperformances , Ġwhich Ġcannot Ġreach ĠGl o Ve Ġperformance . ĠThis Ġwas Ġexpected Ġas Ġthe Ġpre - trained ĠSpeech 2 V ec Ġvectors Ġhave Ġlower Ġvocabulary Ġcoverage Ġthan ĠGl o Ve . ĠYet , Ġwe Ġobserved Ġthat Ġconc aten ating ĠGl o Ve Ġ+ ĠSpeech 2 V ec , Ġand Ġfurther ĠGl o Ve Ġ+ ĠWord 2 V ec Ġ+ ĠSpeech 2 V ec Ġyields Ġbetter ĠNL U Ġresults : ĠF 1 - score Ġincreased Ġfrom Ġ0 . 89 Ġto Ġ0 . 91 Ġfor Ġintent Ġrecognition , Ġfrom Ġ0 . 96 Ġto Ġ0 . 97 Ġfor Ġslot Ġfilling . </s> For Ġmultim od al Ġ( audio Ġ& Ġvideo ) Ġfeatures Ġexploration , Ġperformance Ġresults Ġof Ġthe Ġcompared Ġmodels Ġwith Ġvarying Ġmod ality / feature Ġconc aten ations Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 4 . ĠSince Ġthese Ġaudio / video Ġfeatures Ġare Ġextracted Ġper Ġutter ance Ġ( on Ġsegment ed Ġaudio Ġ& Ġvideo Ġclips ), Ġwe Ġexperimented Ġwith Ġthe Ġutter ance - level Ġintent Ġrecognition Ġtask Ġonly , Ġusing Ġhierarchical Ġjoint Ġlearning Ġ( H - J oint - 2 ). ĠWe Ġinvestigated Ġthe Ġaudio - visual Ġfeature Ġadditions Ġon Ġtop Ġof Ġtext - only Ġand Ġtext + speech Ġembed ding Ġmodels . ĠAdding Ġopen S MI LE / IS 10 Ġfeatures Ġfrom Ġaudio , Ġas Ġwell Ġas Ġincorporating Ġintermediate ĠCNN / In ception - Res Net - v 2 Ġfeatures Ġfrom Ġvideo Ġbrought Ġslight Ġimprovements Ġto Ġour Ġintent Ġmodels , Ġreaching Ġ0 . 92 ĠF 1 - score . ĠThese Ġinitial Ġresults Ġusing Ġfeature Ġconc aten ations Ġmay Ġneed Ġfurther Ġexplor ations , Ġespecially Ġfor Ġcertain Ġintent - types Ġsuch Ġas Ġstop Ġ( audio Ġintensity ) Ġor Ġrelevant Ġslots Ġsuch Ġas Ġpassenger Ġgestures / g aze Ġ( from Ġcabin Ġvideo ) Ġand Ġoutside Ġobjects Ġ( from Ġroad Ġvideo ). </s> Conclusion </s> In Ġthis Ġstudy , Ġwe Ġpresent Ġour Ġinitial Ġexplor ations Ġtowards Ġmultim od al Ġunderstanding Ġof Ġpassenger Ġutter ances Ġin Ġautonomous Ġvehicles . ĠWe Ġbriefly Ġshow Ġthat Ġour Ġexperimental Ġresults Ġoutper formed Ġcertain Ġbas elines Ġand Ġwith Ġmultim od ality , Ġwe Ġachieved Ġimproved Ġoverall ĠF 1 - sc ores Ġof Ġ0 . 92 Ġfor Ġutter ance - level Ġintent Ġdetection Ġand Ġ0 . 97 Ġfor Ġword - level Ġslot Ġfilling . ĠThis Ġongoing Ġresearch Ġhas Ġa Ġpotential Ġimpact Ġof Ġexploring Ġreal - world Ġchallenges Ġwith Ġhuman - veh icle - scene Ġinteractions Ġfor Ġautonomous Ġdriving Ġsupport Ġwith Ġspoken Ġutter ances . </s> App end ices </s> AM IE ĠIn - c abin ĠDat as et : ĠWe Ġobtained Ġ13 31 Ġutter ances Ġhaving Ġcommands Ġto ĠAM IE Ġagent Ġfrom Ġour Ġin - c abin Ġdataset . ĠAn notation Ġresults Ġfor Ġutter ance - level Ġintent Ġtypes , Ġslots Ġand Ġintent Ġkeywords Ġcan Ġbe Ġfound Ġin ĠTable ĠT AB REF 7 Ġand ĠTable ĠT AB REF 8 . </s> H ier arch ical Ġ& ĠJoint ĠModel Ġ( H - J oint - 2 ): Ġ2 - level Ġhierarchical Ġjoint Ġlearning Ġmodel Ġthat Ġdetects / ext ract s Ġintent Ġkeywords Ġ& Ġslots Ġusing Ġseq 2 seq ĠBi - L ST Ms Ġfirst Ġ( Level - 1 ), Ġthen Ġonly Ġthe Ġwords Ġthat Ġare Ġpredicted Ġas Ġintent Ġkeywords Ġ& Ġvalid Ġslots Ġare Ġfed Ġinto ĠJoint - 2 Ġmodel Ġ( Level - 2 ), Ġwhich Ġis Ġanother Ġseq 2 seq ĠBi - L ST M Ġnetwork Ġfor Ġutter ance - level Ġintent Ġdetection Ġ( j oint ly Ġtrained Ġwith Ġslots Ġ& Ġintent Ġkeywords ) ĠB IB REF 1 .