TITLE: Look at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errors 

ABSTRACT: People interacting with voice assistants are often frustrated by voice
assistants' frequent errors and inability to respond to backchannel cues. We
introduce an open-source video dataset of 21 participants' interactions with a
voice assistant, and explore the possibility of using this dataset to enable
automatic error recognition to inform self-repair. The dataset includes clipped
and labeled videos of participants' faces during free-form interactions with
the voice assistant from the smart speaker's perspective. To validate our
dataset, we emulated a machine learning classifier by asking crowdsourced
workers to recognize voice assistant errors from watching soundless video clips
of participants' reactions. We found trends suggesting it is possible to
determine the voice assistant's performance from a participant's facial
reaction alone. This work posits elicited datasets of interactive responses as
a key step towards improving error recognition for repair for voice assistants
in a wide variety of applications.
People interacting with voice assistants are often frustrated by voice
assistants' frequent errors and inability to respond to backchannel cues. We
introduce an open-source video dataset of 21 participants' interactions with a
voice assistant, and explore the possibility of using this dataset to enable
automatic error recognition to inform self-repair. The dataset includes clipped
and labeled videos of participants' faces during free-form interactions with
the voice assistant from the smart speaker's perspective. To validate our
dataset, we emulated a machine learning classifier by asking crowdsourced
workers to recognize voice assistant errors from watching soundless video clips
of participants' reactions. We found trends suggesting it is possible to
determine the voice assistant's performance from a participant's facial
reaction alone. This work posits elicited datasets of interactive responses as
a key step towards improving error recognition for repair for voice assistants
in a wide variety of applications.
Look at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errors
http://arxiv.org/abs/2104.07153v1