The target set comes from a specific accent.
Datasets: We report results on two public datasets, viz., L2-Arctic and IndicTTS, containing English speech in non-native accents. L2-Arctic has English speech from 24 speakers spanning six non-native accents: Hindi (HIN), Vietnamese (VTN), Chinese (CHN), Korean (KOR), Arabic (ARB) and Spanish (ESP). IndicTTS consists of speech from 8 Indian speakers, each with a different accent depending on their native language: Kannada (Kn), Tamil (Ta), Malayalam (Ml), Hindi (Hi), Rajasthani (Ra), Assamese (As) and Manipuri (Ma). For each target speaker and accent, we created data splits by partitioning \(70\%\)  of the data into a ground set (\(\mathcal {V}\) ) and a very small target set \(\mathcal {T}\)  (with sizes varying from 10 to 50) and we created test/development sets from the remaining 30% using a 27/3 split, respectively. For L2-Arctic, the test sets for a target speaker and a target accent contained roughly 300 utterances and 1.2K utterances, respectively. When grouped by target speakers, \(|\mathcal {V}| \approx 740\) , and for a specific target accent, \(|\mathcal {V}| \approx 3000\) . For IndicTTS, the average sizes of the ground set and the test set are 4.3K and 1.9K utterances respectively. Given that this dataset contains speech from only one speaker per accent, it was not used for accent personalization.
<FIGURE>Features. We represent each utterance as a 39-dimensional feature vector of MFCC coefficients averaged over the duration of the utterance. Fig. REF  shows a t-SNE plot of MFCC features of IndicTTS test set utterances coloured using speaker IDs. We observe good speaker-specific clusters that are well-separated from each other.
ASR Models: Following [1]}, our pre-trained model is based on the QuartzNet-15x5 [2]} architecture. It is trained on LibriSpeech [3]} for 400 epochs using the CTC-loss [4]} and yields a WER of 3.90 on the test-clean split of LibriSpeech. The QuartzNet-15x5 architecture is fully convolutional with residual connections. This model is finetuned with our selected subsets of accented speech to minimize CTC loss using the NovoGrad optimizer [5]} for 100 epochs with a batch size of 16, a linearly decaying learning rate of \(10^{-5}\)  and early stopping based on the dev set. In all our experiments, we report results averaged over three runs using three different seeds.

Experiments and Results
We experiment with personalized ASR systems for a specific target speaker and a specific target accent. For each setting, we compare the following approaches: (i) random selection (baseline), (ii) SMI-based selection using three SMI functions, FLMI, GCMI and LogDMI and (iii) a skyline oracle system that uses the speaker/accent metadata and randomly samples from the targeted accent/speaker.
Unless specified otherwise, we use target sets of size 10 and a transcription budget of 360 sec for L2-Arctic and 492 sec for IndicTTS, amounting to an average of 100 utterances in each dataset.
<TABLE><TABLE><TABLE>Personalized ASR for a target accent: Table REF  shows our main results with accent personalization on L2-Arctic test sets. We observe consistent WER reductions using the SMI functions compared to random selection with FLMI performing the best, followed by LogDMI and GCMI. In our skyline, we finetune using a subset created by randomly selecting an equal number of test samples from each speaker of the target accent. We find that the best SMI-based systems outperform the skyline for nearly all accents (except Arabic).
We further examine the distribution of selected samples across accents using each of the three SMI functions. FLMI selects utterances almost exclusively from the target accent (\(99.4\%\)  on average), while LogDMI and GCMI select \(93.5\%\)  and \(89.8\%\)  samples from the target accent, respectively. When subject to a limited budget, we hypothesize that both FLMI and LogDMI perform better than GCMI, since the latter does not explicitly reward diversity and might disproportionally favour specific speakers within an accent. (Distinctions between the SMI functions are diminished with increasing the budget as we will see in Fig REF .)
Personalized ASR for a target speaker: Tables REF  and REF  show our main results by personalizing to a target speaker in L2-Arctic and IndicTTS, respectively. Again, we see consistent WER reductions using all three SMI functions compared to the random selection baseline. The skyline picks utterances exclusively from the target speaker. Our best SMI-based systems, LogDMI and FLMI perform comparably to the skyline.
Both FLMI and GCMI almost exclusively select speech samples from the target speaker (\(99.8\%\) ), and LogDMI picks \(94.8\%\)  samples in the selected subset from the target speaker. When personalizing to a target speaker under a limited budget, GCMI largely selects from the target speaker (given its objective which does not explicitly reward diversity) unlike in the target accent setting.
Rankings among SMI functions based on WER reductions are less consistent when the budgets are increased, as seen in Fig. REF  that shows the WER improvements using GCMI, FLMI and LogDMI for a specific accent (Chinese) and target speaker (Assamese female) by varying durations of the selected subsets. However, they always consistently outperform random selection. For Chinese-accented English, we observe that FLMI yields a WER reduction of 3.15 with 6 minutes of data, while a similar reduction needs 24 minutes using random selection.
<FIGURE>
<FIGURE>

Conclusion
In this work, we propose a targeted subset selection approach using submodular mutual information functions that help identify speech utterances that best match an unlabeled target set of limited size. We evaluate our approach by limiting the target set to either a single speaker or a single accent. Our method consistently outperforms random selection methods and even beats an oracle system when the target set is from a specific accent. Future work will explore extensions of this approach to adapt to speakers with speech impediments.
