Unsupervised Domain Adaptation for Dysarthric Speech Detection via Domain Adversarial Training and Mutual Information Minimization

Dysarthric speech detection (DSD) systems aim to detect characteristics of the neuromotor disorder from speech. Such systems are particularly susceptible to domain mismatch where the training and testing data come from the source and target domains respectively, but the two domains may differ in terms of speech stimuli, disease etiology, etc. It is hard to acquire labelled data in the target domain, due to high costs of annotating sizeable datasets. This paper makes a first attempt to formulate cross-domain DSD as an unsupervised domain adaptation (UDA) problem. We use labelled source-domain data and unlabelled target-domain data, and propose a multi-task learning strategy, including dysarthria presence classification (DPC), domain adversarial training (DAT) and mutual information minimization (MIM), which aim to learn dysarthria-discriminative and domain-invariant biomarker embeddings. Specifically, DPC helps biomarker embeddings capture critical indicators of dysarthria; DAT forces biomarker embeddings to be indistinguishable in source and target domains; and MIM further reduces the correlation between biomarker embeddings and domain-related cues. By treating the UASPEECH and TORGO corpora respectively as the source and target domains, experiments show that the incorporation of UDA attains absolute increases of 22.2% and 20.0% respectively in utterance-level weighted average recall and speaker-level accuracy.


Introduction
Dysarthria encapsulates various speech disorders caused by a set of neurodegenerative conditions and diseases, such as cerebral palsy, Parkinson's disease or amyotrophic lateral sclerosis, which lead to poor control of muscles including lips, tongue, jaw, velum and throat [1]. Therefore, patients with dysarthria often produce harsh and breathy speech with unstable prosody and imprecise articulation. To facilitate the clinical diagnosis and treatment of neurological diseases, early onset detection of dysarthric speech may serve as a promising tool.
Current research on dysarthric speech detection (DSD) mainly focuses on building models trained and validated on the data from the same domain, where high DSD accuracy can be achieved. However, DSD models are less robust under domain mismatch conditions, i.e., DSD models trained by the data from the source domain will suffer from marked performance degradation when they are exposed to data from an unseen target domain with different statistical distributions. The difference may be the types of stimuli, phonetic context, vocal quality, disease etiology, recording environments and devices, etc. Training on *Work done during internship at Huawei Noah's Ark Lab labelled data from the target domain will improve DSD accuracy. However, such data is often too costly to acquire [2]. Therefore, leveraging available, labelled source data along with unlabelled target data to create a DSD model that generalizes well to target domain is desirable, and can be treated as an unsupervised domain adaptation (UDA) problem [3], as no supervision information is available in the target domain.
To alleviate the domain mismatch issues, domain adversarial training (DAT) and mutual information minimization (MIM) are proposed to extract domain-invariant biomarker embeddings that are used to identify the dysarthria for accurate DSD. The UDA framework consists of three learning tasks: The primary task employs a biomarker encoder to extract biomarker embeddings for dysarthria presence classification (DPC). The second task applies DAT to force biomarker embeddings to be indistinguishable in the source and target domains by deceiving a domain discriminator, so that the biomarker embeddings from source and target domains have similar distributions. The last task strives to minimize the mutual information between the biomarker embeddings and the counterpart domain embeddings that are extracted by a domain encoder, which further removes domain cues in biomarker embeddings. The proposed UDA framework facilitates the learning of biomarker embeddings that are invariant across domains while capturing critical information used for dysarthria detection.
This work paves the way towards the under-explored problem of cross-domain DSD that is widely encountered in practice. The main contribution lies in the novel approach by combining DPC, DAT and MIM for cross-domain DSD that is formulated as an UDA problem for the first time. Extensive experiments have been conducted to verify the effectiveness of proposed methods by using different kinds of neural networks.

Related work
Diagnosis of speech symptoms is commonly used in the identification of dysarthria. Traditional approaches are involved with clinicians or speech-language pathologists conducting a series of subjective listening tests, e.g., Frenchay Dysarthria Assessment [4], which may be affected by subjectivity among assessors. This motivates researchers to turn to objective evaluation of dysarthria based on building statistical DSD models, which are economical with potentials for remote patient rehabilitation monitoring [5]. To develop an efficient DSD system, previous work mostly design handcrafted acoustic features as biomarkers that capture dysarthric patterns, including prosodic, spectral, phonological and glottal features [6][7][8][9][10]. Besides, automatic feature extraction from raw speech via a learnable frontend is proposed in [11]. Though significant progress has been achieved, effectiveness of previous methods requires further verification under cross-domain conditions. A few previ- Figure 1: Diagram of the proposed UDA framework for DSD based on multi-task learning, which includes dysarthria presence classification (DPC), domain adversarial training (DAT) and mutual information minimization (MIM). 'GRL' denotes the gradient reversal layer.
ous efforts investigate this problem by carefully designing and selecting domain-robust features for cross-language [12] and cross-dataset [13] scenarios. In contrast, this paper focuses on automatically extracting dysarthria-discriminative and domaininvariant biomarkers from simple acoustic features, e.g., melspectrograms, which is formulated as an UDA problem.
UDA has been explored in many speech tasks including automatic speech recognition [14,15], speech emotion recognition [16] and speaker recognition [17], where DAT [18] is widely used to remove domain variations and project the data of different domains into the same subspace. There is still much room to apply DAT for the DSD task. Besides, inspired by information theory [19], MIM is proposed to reduce the dependency between biomarker embeddings and domain-related information. The combination of DAT and MIM forces biomarker embeddings to achieve robustness for detection of dysarthria.

Proposed approach
Assuming that there are I and J utterances with corresponding mel-spectrograms {M s i } and {M t j } in the source domain and target domain, respectively. Each source mel-spectrogram M s i is associated with a binary label y s i denoting whether the corresponding speech is dysarthric, while no such label is provided in the target domain. Given the data {M s i , y s i }(i = 1, 2, ..., I) and {M t j }(j = 1, 2, ..., J), the goal is to build a DSD system that generalizes well to the target domain. To achieve robustness in a DSD system, we propose a multi-task learning based UDA framework as shown in Figure 1, which consists of three learning tasks: dysarthria presence classification, domain adversarial training and mutual information minimization.

Dysarthria presence classification (DPC)
This primary task performs binary classification for the presence or absence of dysarthria by using the labelled source data. Specifically, a biomarker encoder θ bio takes in mel-spectrogram M s i to derive a single vector x s i , which is denoted as biomarker embedding and fed into the dysarthria classifier ψ bio , to give the dysarthria presence posterior f (M s i ; θ bio , ψ bio ). During training, the biomarker encoder and dysarthria classifier are optimized to minimize the cross-entropy loss as follows:

Domain adversarial training (DAT)
The second task applies DAT to render biomarker embeddings indistinguishable in the source and target domains, by introduc-

Algorithm 1. Training process for UDA based DSD system
Input: source data {M s i , y s i }, target data {M t j }, learning rate α, β and γ 1. for each training iteration do 2. freeze θ bio , ψ bio , θ dom , ψ dom and φ, compute discrimination loss (2) using {M s i } and {M t j }, then update ψ dis : ing a gradient reversal layer (GRL) and a domain discriminator ψ dis , as shown in Figure 1. During training, parameters of the domain discriminator and biomarker encoder are updated alternatively. On the one hand, by freezing θ bio , the domain discriminator is trained to determine whether the input biomarker embeddings are from the source or target domains by minimizing the discrimination loss [18]: On the other hand, by freezing ψ dis , the biomarker encoder is trained to maximize the above discrimination loss to deceive the discriminator, which is realized via GRL that passes the data during forward propagation and inverts the sign of the gradient during backward propagation. The alternative processes in training force the domain discriminator and biomarker encoder to compete against each other in an adversarial manner [20], which encourages the distribution of biomarker embeddings across domains to be similar, so that the dysarthria-related cues learned from the source domain in the DPC task remain effective in the target domain.

Mutual information minimization
The last task strives to reduce the dependency between the biomarker embeddings and domain-related information via MIM. To extract domain-related information, a domain encoder θ dom and a domain classifier ψ dom are utilized. The domain encoder takes in M s i and M t j to extract domain emebddings z s i and z t j respectively, which are used for domain prediction via the domain classifier. Therefore, θ dom and ψ dom are jointly trained by minimizing the domain classification loss that is similar with (2) as: The embeddings z s i and z t j are domain-dependent and can be used to represent domain-related information.
Then the mutual information between the biomarker embeddings x (x s i or x t j ) and domain embeddings z (z s i or z t j ) is used to measure the dependency as Kullback-Leibler (KL) divergence between their joint and marginal distribution: H(x, z) = DKL(p(x, z); p(x)p(z)). As the computation of mutual information is challenging for high-dimensional continuous variables with unknown probability distributions, variational contrastive log-ratio upper bound (vCLUB) [21] is used to calculate the mutual information loss as: where q φ (x|z) is the variational approximation of the groundtruth posterior of x given z and can be parameterized by a network φ. During training, θ bio and θ dom are optimized to minimize Lmi (4), while the variational approximation network φ is optimized to maximize the log-likelihood:

Integrating the learning tasks
By combining the three learning tasks, the total DSD training loss used for updating θ bio , ψ bio , θ dom and ψ dom is: where λ k (k=1, 2, 3, 4) are positive constant weights. The final training process is summarized in Algorithm 1, where the welltrained biomarker encoder θ bio and dysarthria classifier ψ bio are retained to perform the detection of dysarthria.

Experimental setup
To verify the effectiveness of proposed methods, the UASPEECH [22] and TORGO [23] corpora are used for experimentation. UASPEECH contains 15 dysarthric speakers (11 males and 4 females) with cerebral palsy, and 13 healthy speakers (9 males and 4 females). Each speaker has three blocks of utterances with isolated words, where the speech stimuli in each block contain 100 uncommon words and 155 repetitive words (i.e., 10 digits, 26 alphabets, 19 computer commands and 100 common words), which are recorded by 7-channel microphone arrays, we select the data of M6-channel for experiments. TORGO contains 7 dysarthric speakers (4 males and 3 females) with cerebral palsy or amyotrophic lateral sclerosis and 7 healthy speakers (4 males and 3 females). Different from UASPEECH, the speech stimuli include not only words, but also non-words and sentences. Words are mainly chosen from the word intelligibility section of the Frenchay Dysarthria Assessment [4] and Yorkston-Beukelman Assessment [24]. Nonwords involve 5-10 repetitions of /iy-p-ah/, /ah-p-iy/, and /pah-t-ah-k-ah/ along with high-pitch and low-pitch vowels. Sentences are formed by Grandfather passage from the Nemours database [25], 162 sentences from the sentence intelligibility section of Yorkston-Beukelman Assessment, 460 sentences of MOCHA database [26] and spontaneously elicited descriptive texts. Due to discrepancies of speech stimuli types, phonetic context, articulation patterns, recording environments and devices, UASPEECH and TORGO can be treated as two different domains with distinct data distributions.
All speech signals are sampled at 16kHz, 80-band melspectrogram is calculated with hanning window of 25ms and hop length of 10ms. Utterance-level z-score normalization for mel-spectrograms is performed before feeding them into the DSD system. The biomarker encoder and domain encoder adopt the same architecture that contains Convolution Banks and Recurrent Neural Network [27] with Attention [28] (CBRNN-A). There are 8 convolution banks with kernel size varying from 1 to 8, one-layer long-shot term memory (LSTM) with 128 units is employed for Recurrent Neural Network, attention module includes two linear layers (100 and 1 unit) with a softmax layer to obtain a vector that is used to weight the linear combination of LSTM outputs to form the final biomarker embedding and domain embedding. Both dysarthria and domain classifiers contain a linear layer with the sigmoid function, and the domain discriminator contains two linear layers with hidden size of 128 and 1. The variational approximation q φ (x|z) in (4) is parameterized by the Gaussian distribution as q φ (x|z) = N (x|µ(z), diag(σ 2 (z))) with mean µ(z) and variance σ 2 (z) inferred by two-way linear layers with a hidden size of 256. All networks are trained by the Adam optimizer [29] for 8 epochs with learning rate α, β and γ set to 1e-4, 1e-4 and 1e-3 respectively, and the weights λ1, λ2, λ3 and λ4 in loss (6) are set to 1, 1e-1, 1 and 1e-4 respectively.
We compare CBRNN-A with other networks that also use mel-spectrograms to detect dysarthria, including Recurrent Convolutional Neural Network (RCNN) [10] and Recurrent Neural Network with Attention (RNN-A) [11]. We adopt the leave-one-subject-out cross validation scheme, i.e., all speakers are used for training except the one that is left out for testing. Three evaluation metrics are used: (1) Utterance-level weighted average recall (WAR), denoting the ratio of utterances that are correctly classified; (2) Utterance-level unweighted average recall (UAR), denoting the accuracy per class averaged by total number of classes; (3) Speaker-level accuracy (ACC), denoting the ratio of speakers for which more than 50% of the individual's utterances are classified correctly.

Within-domain DSD performance
We first evaluate within-domain DSD performance, where the training and testing are both performed in a target domain, assuming that labelled data is provided, i.e., the ideal condition. The results are shown in Table 1, we can see that the proposed CBRNN-A outperforms RCNN and RNN-A with higher utterance-level WAR, UAR and speaker-level ACC for both UASPEECH and TORGO corpora. This shows the effectiveness of CBRNN-A by using multiple convolution banks with varied kernel size to capture articulation patterns at different scales for accurate DSD.

Cross-domain DSD performance
Next, we consider two domain mismatch conditions: 'UASPEECH → TORGO' and 'TORGO → UASPEECH', where the former treats UASPEECH as the source domain and  TORGO as the target domain, and vice versa for the latter. Results are shown in Table 2. First, for 'UASPEECH → TORGO', the performance of all DSD systems without DAT and MIM drops significantly. As TORGO has data imbalance where the ratio between healthy and dysarthric utterances is around 2:1, and healthy utterances are often incorrectly classified, less than 50% WAR-utterance and only 50% ACC-speaker are achieved. This shows the susceptibility of DSD systems to domain mismatch issues. Second, detection accuracy can be improved by using DAT or MIM, and the combination of DAT and MIM can greatly boost DSD performance for different kinds of networks, where the proposed CBRNN-A outperforms RCNN and RNN-A when DAT and MIM are used, showing the effectiveness of proposed methods for learning dysarthria-discriminative and domain-invariant biomarker embeddings for robust dysarthria detection. Third, compared with 'TORGO → UASPEECH', larger improvements can be achieved under 'UASPEECH → TORGO' condition by using DAT and MIM, e.g., the absolute values of WAR-utterance and ACC-speaker are increased with 22.2% and 20.0% respectively by using CBRNN-A. As UASPEECH contains utterances with limited words, while TORGO contains richer words and unseen speech stimuli types including non-words and sentences, the DSD systems trained on UASPEECH generalize poorly to TORGO. This can be verified by the utternace-level WAR results for words, non-words and sentences as shown in Table  3, CBRNN-A performs worst for sentences, followed by words and non-words. With the proposed DAT and MIM, 34.2%, 22.7% and 7.0% absolute increase in WAR can be achieved on average for sentences, words and non-words respectively. As TORGO contains richer speech stimuli, DSD systems trained on TORGO have better generalization capability, which can be further enhanced by the proposed DAT and MIM approaches.

Visualization of biomarker embeddings
To acquire an intuition regarding how the UDA framework extracts effective biomarker embeddings, we consider 'UASPEECH → TORGO' condition. Principal component analysis (PCA) is performed on biomarker embeddings extracted by the DSD systems trained without and with DAT & MIM. The first and second components of PCA results are illustrated in Figure 2, where different colors denote different domains, different shapes denote the presence or absence of dysarthria. We observe that the biomarker embeddings from UASPEECH and TORGO tend to be separate when DAT & MIM is not used, while biomarker embeddings are mixed when DAT & MIM is used, indicating that without additional regularizations, the biomarker embeddings contain domain cues, but these can be effectively removed by proposed DAT and MIM. Besides, it can be seen that dysarthric and healthy biomarker embeddings of DSD systems with DAT & MIM are more dysarthria-discriminative with more obvious cluster formation than those of DSD systems without DAT & MIM, which further proves the superiority of proposed methods to achieve higher detection accuracy across domains.

Conclusions
This paper studies an under-explored field of DSD, i.e., crossdomain DSD where the DSD system is trained and tested on different domains with distinct data distributions. We propose a multi-task learning strategy, where the primary task performs DPC using labelled source data, while DAT and MIM tasks leverage large amounts of additional, unlabelled target-domain data that can be easily acquired to align the domain distributions. The proposed approach can obtain domaininvariant biomarker embeddings that contain critical indicators of dysarthria presence for accurate and robust detection. This is verified by extensive experiments with different kinds of network architectures. Our future study will focus on applying and improving the proposed UDA methods for more challenging domain mismatch conditions, e.g., cross-language condition.