MultiMediate '22: Backchannel Detection and Agreement Estimation in Group Interactions

Backchannels, i.e. short interjections of the listener, serve important meta-conversational purposes like signifying attention or indicating agreement. Despite their key role, automatic analysis of backchannels in group interactions has been largely neglected so far. The MultiMediate challenge addresses, for the first time, the tasks of backchannel detection and agreement estimation from backchannels in group conversations. This paper describes the MultiMediate challenge and presents a novel set of annotations consisting of 7234 backchannel instances for the MPIIGroupInteraction dataset. Each backchannel was additionally annotated with the extent by which it expresses agreement towards the current speaker. In addition to a an analysis of the collected annotations, we present baseline results for both challenge tasks.


INTRODUCTION
Backchannels, i.e. short interjections made by listeners, are at the core of the bilateral nature of dialogue [24]. That is, dialogue in which a listener's responses affect the speaker's speech acts. They may consist of a verbal phrase like "oh yes", a sound (e.g. "hm"), or bodily gestures, such as head nods or hand movements, or combinations thereof. Backchanneling serves the important functions of confirming listeners' attention and comprehension [33], as well as regulating turn-taking [60]. In addition, it is used to communicate agreement or disagreement with the current speaker and is therefore an important indicator of participants' opinions and intra-group relations [12]. The inability to appropriately perform backchanneling can have severe impact on dialogue, e.g. by distracting the speaker [10,51].
As a result, automatic analysis of backchanneling behaviour is crucial and has significant potential for artificial systems designed to passively monitor or actively mediate human dialogue [6,52,53,61]. For example, artificial mediators [14,17,25] could analyse the frequency of backchannels to gauge participants' engagement and encourage those who are disengaged from the discussion. Agreement or disagreement expressed in backchannels could also help artificial mediators to better understand the opinions of participants. This may enable mediators to give a voice to participants who find it difficult to express diverging opinions. Despite this potential, automatic analysis of backchannels in group interactions remains largely unexplored due to the lack of suitable datasets.
With MultiMediate '22, we present the first challenge for automatic backchannel analysis in group interactions. To this end, we introduce the first publicly available dataset of backchanneling behaviour in group discussions. We fully annotated the MPI-IGroupInteraction dataset [49] with 7234 backchannel instances. Furthermore, each backchannel instance was rated with the extent to which it expresses agreement with the current speaker. We present analyses of our novel annotations as well as evaluations of baseline approaches for MultiMediate '22. All collected annotations, baseline implementations, and raw feature representations are made publicly available for further use. 1

PREVIOUS WORK 2.1 Backchannel and Agreement Datasets
While a number of different datasets were recorded for the purpose of backchannel analysis [10,22,44,55], they are rarely public and not sufficient to ensure progress on automatic backchannel analysis in group interactions. In Table 1 we list recent publicly available datasets that were used in BC research, the majority of which consists of spoken dialogues. Note that while these datasets are publicly available, the backchannel annotations collected on them are often not (column "Pub." in Table 1). Furthermore, all existing datasets (except the Canal9 Corpus [54,67] for which no publicly available backchannel annotations exist) consist of dyadic interactions and are thus not fitting for group behaviour analysis.
In terms of annotations of backchannels, some studies use semiautomatic labeling of potential events [9,18], or query a dataset for specific BC keywords [50]. Backchannel annotations are most often based on a set list of backchannel events ("is there a "uh-huh" here?") [18,27,45,50,63], and only rarely on the holistic perception of a backchannel ("is there a backchannel here?") [5,11,35].
While a relationship between backchannels and agreement has been discussed [60], few studies have systematically investigated it. [55] included agreement in their annotations of backchannels, but did not analyse it; [30,36] distinguish backchannels from agreement signals. Some studies of agreement include backchannels [19,20], but to the best of our knowledge, no study of backchanneling investigated agreement transmitted through the backchannels themselves.
In our work, we present the first publicly available annotations of backchannel occurances and agreements expressed via backchannels in group interactions. Our full dataset is manually annotated and with over 33 hours of annotated human behaviour across training, validation and test sets, its size is equal to the largest dyadic BC dataset currently available (see Table 1).

Computational Models
The prediction (i.e. anticipation) of backchannels is a highly active area of research in social signal processing. One major motivation is the goal to generate natural backchanneling behaviour in artificial agents. There is a variety of traditional machine learning methods used for predicting backchannels [27,29,31,40,44,64]. Recently, LSTM networks became the most frequent choice [1,2,34,58], along with residual networks [31]. Multitask learning also seems to be a particularly successful approach [34,37,39]. The most common features used in BC prediction are prosodic [2,34,57,58] while some research also makes use of linguistic features such as word embeddings [1,2,57,58] or syntactic (part of speech tags), semantic (concreteness, valence) and discourse features [16,18,40]. Research shows that adding lexical features to accoustic ones improves results [50,57]. [33] offers a review of the most important backchannel-inviting cues which may also be used as features. In contrast to backchannel prediction, we define the task of backchannel detection as categorizing observed behaviour as to whether a backchannel was shown or not. While to the best of our knowledge, this task was not studied in isolation in previous work, backchannel detection can appear as a part of the multi-class problem of dialogue act classification (DAC) [3,59]. DAC is primarily addressed by text analysis [32,42,56] and only few works incorporated multi-modal information like emotional expressions [21,59]. Importantly, no previous work addressed the task of backchannel detection in group interactions from multi-modal behaviour.
With MultiMediate '22 we aim to attract researchers to two challenging problems centered around backchannels in group discussions: backchannel detection and estimation of the amount of agreement expressed in a backchannel. In line with human studies that underline the importance of non-verbal backchannel cues [5,41], we provide annotations of backchannel behaviour that can take place both via speech and in the visual domain. We contribute to improved comparability between approaches by using an unpublished test set for evaluation.

CHALLENGE DESCRIPTION
As in MultiMediate '21 [47], our challenge is based on the MPI-IGroupInteraction dataset [48,49]. This dataset has been used for diverse tasks, including low rapport detection [49], emergent leadership detection [46], eye contact detection [28,48], next speaker prediction [15], and body language detection [7]. For MultiMediate '22 we collected novel backchannel annotations on the whole dataset. Test samples (excluding ground truth) are released to participants before the challenge deadline. Participants in turn submit their predictions for evaluation by the challenge organisers. We first describe MPIIGroupInteraction and subsequently discuss annotation procedures and task definitions for backchannel detection and agreement estimation.

Dataset
Training data. We use the publicly available recordings of MPI-IGroupInteraction [49] as training data. The dataset comprises 22 conversations between three to four people on a maximally controversial topic, lasting 20 minutes each. While discussing the topic, interactants were recorded with four microphones and eight framesynchronised video cameras (see Figure 1).
Evaluation data. We follow MultiMediate '21 [47] and use six yet unpublished discussions that were recorded during the creation of MPIIGroupInteraction [49] for testing. These six discussions followed the same procedure as the training discussions with the only exception that the topic was not chosen to be maximally controversial. Instead, a topic was randomly selected for each group and participants were asked to take on opposing views for themselves.

Backchannel Detection Task
Backchannel Annotations. In a first step annotators where asked to label the occurrences of backchanneling behaviour with respect to different modalities: auditive, and visual. Visual annotations take only the aspects observable in the video into account (e.g. nodding or head shaking). Auditive backchanneling behaviour, in turn, relies only on cues from the audio signal, encompassing verbal (e.g. "yes", "oh really?") and paraverbal behaviour (e.g. "uhm hum", "aha"). Each of three annotators labeled a specific portion of the dataset with respect to backchannels for each modality. Figure 2 (left) shows the average of annotated backchannel events annotated per participant in each modality. On average, we observed 28 annotations per participant for the auditive modality, and 60 for the visual modality. Figure 2 (middle) shows the average duration of annotated backchannel events for each modality. Visual backchannel events are on average longer (1.8 seconds) compared to auditive backchannel events (0.56 seconds). The modality specific labels were subsequently joined using the logical OR operator. Hence, overlapping backchannels in different modalities (e.g. a person nods while simultaneously saying "yes") are merged to a new, modality independent backchannel label. To provide negative examples we also calculated an equal amount of non-backchannel samples per session. Overall this results in 14468 labels partitioned into the following splits: 6716 Train, 2854 Val, 4898 Test.
Task definition. Given an observation window of 10 seconds participants have to detect if there is a backchannel present in the sample. Every sample has been created such that the end of an annotated backchannel has been used as the end the sample or that a sample consists of 10 seconds with no annotated backchannels. The performance metric for the task is accuracy.

Backchannel Agreement Estimation Task
Agreement Annotations. In the second step, all backchannel instances that where annotated for the backchannel detection test were labeled with respect to their level of expressed agreement on a scale from -1 (total disagreement) to 1 (total agreement) using a step size of 0.1. In case annotators found that the instance was erroneously labeled as a backchannel, they indicated this fact with an extra label. When all annotators agreed that an instance was wrongly labeled as a backchannel the respective sample has been removed from the dataset during the sampling process. This annotation step has been performed by each annotator for all backchannel labels (see section 3.2) The groundtruth labels were then created by averaging all three annotations per sample. Overall this results in 7234 labels partitioned into the following splits: 3358 Train, 1427 Val, 2449 Test. To quantify the reliability of agreement annotations, we compute the average Spearman when comparing a left-out annotator to the average of the two remaining ones is. We observe a of 0.62, indicating substantial agreement. Figure 2 (right) shows the distribution of aggregated agreement annotations. The distribution is centered in the positive range between 0 and 0.5. The detection of disagreement represents a special challenge due to the small number of samples below 0.
Task definition. For the backchannel agreement estimation tasks participants have to predict the average expressed agreement per sample on scale from -1 to 1. The performance metric for the task is mean squared error.

Features
We extracted the same set of features for backchannel detection and agreement estimation from backchannels. Features were extracted from the last second of the 10 second input window and aggregated by computing the mean or the mean of absolute differences ("mean delta") of adjacent frames over this second.

Head Features.
We extracted features from participants' head and face using OpenFace 2.0 [8]. These include mean and mean delta of AU intensity estimates, mean delta of head orientation and translation, as well as mean delta of gaze angles for both eyes. In total, we extracted 46 features based on OpenFace 2.0.

Pose Features.
We extract body pose estimates using Open-Pose [23] and employ a set of angular features that proved successful for group interaction analysis [13]. These features consist of angles between body parts, e.g. the angle defined by the line between left shoulder and left elbow and the line between left shoulder and right shoulder. We limit ourselves to the "Upper Body" and "Head" features described in [13], as lower body pose estimates tend to be unreliable on the MPIIGroupInteraction dataset. We compute the mean delta of these features, resulting in 8 body pose features.

Voice Features.
We extracted the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [26] on the last second of the input window. This set consists of 88 acoustic parameters that are commonly applied to tasks like depression, mood, and emotion recognition [66], or Alzheimer's Dementia recognition [43].

Prediction Approach
For backchannel detection, we trained a binary Support Vector Classifier (SVC) with rbf kernel. For the agreement estimation task, we trained a Support Vector Regressor (SVR) with rbf kernel. We employed 10-fold cross-validation on the training set to choose and parameters of the SVC/SVR. For test set evaluations, we trained on training and validation sets; for evaluations on the validation set we only trained on the MultiMediate '22 training set.

Results
We present evaluation results for different combinations of feature sets in Table 2. For fairness reasons, we only evaluate two feature sets for each task on the training set: the best performing feature set on the validation set, as well as all included features.

Backchannel Detection.
The best feature set on the validation set consisted of a combination of head and pose features, reaching 0.639 accuracy on validation and 0.596 accuracy on the test set. This clearly outperformed the trivial baseline of a random predictor at 0.5 accuracy and was marginally better than all featuresets combined. Notably, our experiments on the validation set revealed that each individual feature set achieved above-random performance on backchannel detection, even though OpenFace 2.0 based head features were clearly leading. An ablation of the head feature set revealed that head pose alone (i.e. mean delta of translation and rotation of the head) reaches 0.636 accuracy. This is likely due to the strong association between backchanneling and nodding.

Agreement Estimation.
For agreement estimation, head pose features performed best on the validation set (0.075 MSE) and reached 0.061 MSE on the test set. In both cases this improves above the trivial baseline of using the mean on the training set as a predictor. However, featuresets that do not include head pose or  gaze features fail to outperform the trivial baseline. This is in contrast to the backchannel detection task and indicates the difficulty of backchannel agreement estimation.

CONCLUSION
We introduced MultiMediate '22, the first challenge addressing backchannel detection and agreement estimation from backchannels in well-defined conditions and evaluated baseline approaches for each task. In addition we introduced a novel publicly available dataset of backchannel annotations in group interactions that is a valuable resource for research on backchannel detection and agreement estimation, even beyond the MultiMediate challenge.