Audio-video fusion strategies for active speaker detection in meetings
Meetings are a common activity in professional contexts, and it remains
challenging to endow vocal assistants with advanced functionalities to
facilitate meeting management. In this context, a task like active speaker
detection can provide useful insights to model interaction between meeting
participants. Motivated by our application context related to advanced meeting
assistant, we want to combine audio and visual information to achieve the best
possible performance. In this paper, we propose two different types of fusion
for the detection of the active speaker, combining two visual modalities and an
audio modality through neural networks. For comparison purpose, classical
unsupervised approaches for audio feature extraction are also used. We expect
visual data centered on the face of each participant to be very appropriate for
detecting voice activity, based on the detection of lip and facial gestures.
Thus, our baseline system uses visual data and we chose a 3D Convolutional
Neural Network architecture, which is effective for simultaneously encoding
appearance and movement. To improve this system, we supplemented the visual
information by processing the audio stream with a CNN or an unsupervised
speaker diarization system. We have further improved this system by adding
visual modality information using motion through optical flow. We evaluated our
proposal with a public and state-of-the-art benchmark: the AMI corpus. We
analysed the contribution of each system to the merger carried out in order to
determine if a given participant is currently speaking. We also discussed the
results we obtained. Besides, we have shown that, for our application context,
adding motion information greatly improves performance. Finally, we have shown
that attention-based fusion improves performance while reducing the standard
deviation.