TITLE: Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model 

ABSTRACT: While Machine Comprehension (MC) has attracted extensive research interests
in recent years, existing approaches mainly belong to the category of Machine
Reading Comprehension task which mines textual inputs (paragraphs and
questions) to predict the answers (choices or text spans). However, there are a
lot of MC tasks that accept audio input in addition to the textual input, e.g.
English listening comprehension test. In this paper, we target the problem of
Audio-Oriented Multimodal Machine Comprehension, and its goal is to answer
questions based on the given audio and textual information. To solve this
problem, we propose a Dynamic Inter- and Intra-modality Attention (DIIA) model
to effectively fuse the two modalities (audio and textual). DIIA can work as an
independent component and thus be easily integrated into existing MC models.
Moreover, we further develop a Multimodal Knowledge Distillation (MKD) module
to enable our multimodal MC model to accurately predict the answers based only
on either the text or the audio. As a result, the proposed approach can handle
various tasks including: Audio-Oriented Multimodal Machine Comprehension,
Machine Reading Comprehension and Machine Listening Comprehension, in a single
model, making fair comparisons possible between our model and the existing
unimodal MC models. Experimental results and analysis prove the effectiveness
of the proposed approaches. First, the proposed DIIA boosts the baseline models
by up to 21.08% in terms of accuracy; Second, under the unimodal scenarios, the
MKD module allows our multimodal MC model to significantly outperform the
unimodal models by up to 18.87%, which are trained and tested with only audio
or textual data.
While Machine Comprehension (MC) has attracted extensive research interests
in recent years, existing approaches mainly belong to the category of Machine
Reading Comprehension task which mines textual inputs (paragraphs and
questions) to predict the answers (choices or text spans). However, there are a
lot of MC tasks that accept audio input in addition to the textual input, e.g.
English listening comprehension test. In this paper, we target the problem of
Audio-Oriented Multimodal Machine Comprehension, and its goal is to answer
questions based on the given audio and textual information. To solve this
problem, we propose a Dynamic Inter- and Intra-modality Attention (DIIA) model
to effectively fuse the two modalities (audio and textual). DIIA can work as an
independent component and thus be easily integrated into existing MC models.
Moreover, we further develop a Multimodal Knowledge Distillation (MKD) module
to enable our multimodal MC model to accurately predict the answers based only
on either the text or the audio. As a result, the proposed approach can handle
various tasks including: Audio-Oriented Multimodal Machine Comprehension,
Machine Reading Comprehension and Machine Listening Comprehension, in a single
model, making fair comparisons possible between our model and the existing
unimodal MC models. Experimental results and analysis prove the effectiveness
of the proposed approaches. First, the proposed DIIA boosts the baseline models
by up to 21.08% in terms of accuracy; Second, under the unimodal scenarios, the
MKD module allows our multimodal MC model to significantly outperform the
unimodal models by up to 18.87%, which are trained and tested with only audio
or textual data.
Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model
http://arxiv.org/abs/2107.01571v1