Personalized One-Shot Lipreading for an ALS Patient

Lipreading or visually recognizing speech from the mouth movements of a speaker is a challenging and mentally taxing task. Unfortunately, multiple medical conditions force people to depend on this skill in their day-to-day lives for essential communication. Patients suffering from Amyotrophic Lateral Sclerosis (ALS) often lose muscle control, consequently their ability to generate speech and communicate via lip movements. Existing large datasets do not focus on medical patients or curate personalized vocabulary relevant to an individual. Collecting a large-scale dataset of a patient, needed to train mod-ern data-hungry deep learning models is, however, extremely challenging. In this work, we propose a personalized network to lipread an ALS patient using only one-shot examples. We depend on synthetically generated lip movements to augment the one-shot scenario. A Variational Encoder based domain adaptation technique is used to bridge the real-synthetic domain gap. Our approach significantly improves and achieves high top-5accuracy with 83.2% accuracy compared to 62.6% achieved by comparable methods for the patient. Apart from evaluating our approach on the ALS patient, we also extend it to people with hearing impairment relying extensively on lip movements to communicate.


Introduction
: We propose a personalized one-shot lipreading framework to tackle a real-world medical challenge of lipreading a patient suffering from ALS 1 . The patient communicates primarily by mouthing the words. In our case of limited available real data, we use a combination of synthetic data augmentation technique and a domain adaptation technique called Variational Encoders to build a robust word-level lipreading model for the patient.
Lipreading is mentally taxing and can affect the communication quality. For instance, speakers with hearing impairment may lack holistic audio feedback [29] and ALS patients may have lesser control over their mouth muscles [4,16]. This may cause them to have irregular and unreliable mouth movements making it difficult for people to lipread them. Applications and automated algorithms capable of lipreading a person can thus significantly improve the day-to-day communication of people dependent on lipreading. Motivated by this need, we tackle the real-world challenge of lipreading a patient suffering from ALS and people with hearing impairment. ALS is a progressive nervous system disease that affects nerve cells in the brain and spinal cord, causing loss of muscle control [32]. An ALS patient may lose their voice and rely solely on mouth movements for communication [12,21].
Current Works and Limitations: Current deep learning techniques are inherently datahungry. Collecting large amounts of data specifically from a patient is, however, not an option. Mouthing words is a tiring maneuver for people suffering from ALS, and thus a patient undergoes physical and mental stress during such data collection exercises. Manually labeling words mouthed by a person is time-consuming. It is thus crucial to use the minimum amount of manually labeled data to build lipreading models that can work well on a person.
Recent years have seen much progress in word-level [8,17,28] and sentence-level [3,31] lipreading. Oxford's Visual Geometry Group released large-scale in-the-wild datasets such as Lipreading in the Wild (LRW) [5] and Lipreading sentences (LRS) [1,2] consisting of 1000+ speakers. LRW, the most relevant dataset to our task, is a word-level lipreading dataset made of 1000 examples for 500 English words but turns out to be somewhat limiting: (1) The speakers in the dataset do not have any speaking disability thus making perfect mouth movements. (2) It is curated by cropping words from long speech segments resulting in fastpaced speech with co-articulation in the videos. (3) It contains a large amount of head motion and variations like the different characteristics of the mouth region, both of which are unnecessary for lipreading specific medical patients. Thus, SOTA models like LipReading without Pains (LRwP) [8] and Lipreading using Temporal Convolutional Networks (LTCN) [17] trained on LRW do not directly adapt to speakers with speaking disabilities. Figure 2: Speakers in our study. In left to right order, the first speaker suffers from ALS and relies solely on lip movements. The next two speakers primarily use sign language while making imperfect lip movements. Next speaker uses deaf speech along with sign language for daily communication. The following speaker is the 46th president of the USA, Joe Biden.
LRW only supports a limited pre-curated vocabulary missing out on medically essential words like 'nauseous' or 'backache'. It also lacks a personalized vocabulary relevant to a person's daily communication. Deploying systems to enable a persons' communication would need highly accurate models on their specific lip movements for their particular vocabulary. The problem of personalized lip reading has also been explored in [22]. According to [22], lip movements vary across speakers. Observing a single speaker for an extended period could lead to better speaker-specific lipreading models. They collect ∼20 hours of data per speaker to train a personalized lipreading model generating speech purely from an individual speaker's lip movements. Collecting such a large dataset is, however, not always an option.
Lack of medical data has been studied [9,11,26] widely in the past. Taking inspiration from these, we formally tackle one-shot lipreading in a personalized setting. We first synthetically generate data using a SOTA lipsync network [23]. We then use a SOTA lipreading network [8] as our backbone and use synthetically generated data along with very limited real examples to train a word-level lipreading network. Our approach includes an important domain adaptation step using a novel network -Variational Encoders -a modified VAE for bringing a vast number of synthetic examples closer to the real domain using only one real instance per class. We specifically tackle the use case of an ALS patient and also explore the same for four other speakers (refer to Fig. 2) including the 46th President of the United States, Joe Biden, as an additional example to show that our approach can easily be extended to speakers with no disability. Our contributions in this work are threefold: 1. We tackle a real-world medical challenge of lipreading speakers with ALS and hearing impairment by developing highly accurate personalized models for each speaker.
2. We propose Variational Encoders, a novel network-based on VAE. Instead of autoencoding, they exploit the loss of the downstream task for generating task-relevant latent distributions. The learned distributions are then used for domain adaptation.
3. To the best of our knowledge, we are the first to propose lipreading in one-shot setting.
In this vein, we curate a medical dataset involving speakers with medical conditions.
2 One-shot personalized lipreading framework As mentioned previously we aim to build a personalized lip-reading model for each speaker using only single real examples. Fig. 3 presents the pipeline for personalized lipreading.  Figure 3: Personalized lipreading -We start by curating personalized vocabulary and collect unlabeled videos for each speaker. TTS models are used to generate speech utterances for the curated words. The speaker's unlabeled videos and the generated speech utterances are given as input to Wav2Lip that generates synthetic data to augment one-shot data. Variational Encoders then use the synthetic and the one-shot data to train the model for lipreading.
The use of synthetic data to augment low data in the medical domain [9,11,26] has gained traction with improving generative models. Similarly, we augment the one-shot examples collected for each speaker by generating synthetic data for each of them. To achieve this, we use a SOTA talking face generation model, Wav2Lip [23] pretrained on the largescale LRS2 [6] dataset. Given a speaker's video, Wav2Lip preserves the speaker's pose, facial expressions, and mouth characteristics like beard and skin color while modifying the speaker's lip movements according to a guiding speech. We generate word-level speech utterances using SOTA TTS models FastSpeech2 [24] and GlowTTS [14] as a replacement for the speaker's voice. Using these TTS models allows us to generate variations in the speech in terms of the speed of the spoken word, pitch, and energy. Additionally, we collect unlabeled face videos for each speaker which, along with the generated speech utterances, is used to create 1 hour of speaker-specific synthetic data on an average. The augmented dataset is then used to train an LRwP and LTCN based architecture for the classes curated per speaker.
A combination of the speaker-specific synthetic videos and a single real video per class are used to train our model. The synthetic data helps the model learn the general underlying word-level characteristics for the new classes. However, the properties of personal style of lip-movements for a word -could be because of the medical condition -is not captured in the synthetic dataset. We utilize the one-shot examples for introducing the properties of personal speaking style in the model. Single examples per class are however, not enough to capture the underlying style variations of a speaker. A person may not utter the same word, exactly, each time. To tackle this, we use our novel approach -Variational Encoders.  Figure 4: Variational Encoders -Encoder S and Encoder T denote the source and target encoders respectively. Encoder T generates a latent distribution using the target examples. The latent distribution is trained against the downstream classification task. Encoder S aims to learn robust domain invariant features by minimizing the distance between its learned embedding and points randomly sampled from the generated target distribution. latent representations preserve important domain characteristics. However, using a decoder to reconstruct the input accurately is a non-trivial task, especially for videos needing spatial and continuous temporal reconstruction. We propose Variational Encoders, a modified VAE that uses the loss of the downstream task to generate task-relevant latent distributions.

Variational Encoders: Mapping words into distributions
How do Variational Encoders differ from a standard VAE? Similar to VAEs, Variational Encoders (refer to Fig. 4) generate latent distributions given a sample. A key difference with a standard VAE is -instead of autoencoding, it uses the final multi-class classification loss of the downstream task to generate a task-relevant latent distribution that represents the "class" of input instead of a generic input feature. The learned distribution is used to sample a variation of the input example. We hypothesize that this sample introduces the missing speaker-specific style variation for the input class. The sampled variation encourages the encoder to see and align to the potential interpretation of the input class and learn robust representations. In summary, Variational Encoder retains all the benefits of a VAE, like generating domain invariant features while removing the complicated video reconstruction loss.

Network Architecture
Given a source domain S and a target domain T, domain adaptation aims to bridge the gap between the two domains by generating domain invariant features. We denote S and T as, where N S is the number of source samples and N T is the number of target samples. Variational Encoder is a paired domain adaptation network that assumes the target domain is labeled. However, the number of examples in the target domain is expected to be at max k where k ≈ 1. Thus, we assume N S >> N T . The adaptation network is made of two encoders -source and target encoders -and a single classifier. For a given inputx i belonging to either domain, the encoder learns to generates an embedding ex i for the input. A distribution p(x i ) is then generated for the input denoted by µx i and σx i from the learnt embedding ex i . A random point is sampled from the generated distribution denoted as zx i = µx i + σx i ε.
The encoder network for both domains are identical. Each encoder generates the embedding e, and the latent distribution p. To introduce domain invariance, we use KL divergence and L1 loss. The KL divergence is computed between the two generated distributions p(x i ) = (µ x i , σ x i ) and p(x i ) = (µx i , σx i ). In addition, a point zx i is randomly sampled from the target distribution p(x i ). L1 loss is then applied between zx i and the source embedding e x i . This minimizes the distance between the source embedding against several randomly sampled points from the target distribution thus acting as a pseudo for multiple target examples.
Gradient stopping: The L1 Loss between the source embedding and the sampled target embedding will force the target distribution down to a single point, losing the essence of a distribution. To prevent that, we stop the gradient from flowing back through the target encoder. This way, only the source encoder is regularized against the variations sampled from the learnt target distribution, while leaving the target encoder unaffected. We denote the sampled target embedding as zx i detach , where detach denotes that the sample zx i does not have any gradient. The combined loss to bridge the two domains is given as, where β is a hyper-parameter and i denotes both the samples belong to the same class.
For the downstream task of classification, we employ a common classifier that is trained on both domains at the same time. A single classifier allows both encoders to have a common base-loss helping them generate features relevant to the common downstream task. The downstream classifier also trains on large amounts of variations and learns robust representations. The classifier receives a randomly sampled point zx i from the learnt distribution px i as input for the target domain. For the source encoder, however, the classifier receives the learnt embedding e x i instead of a randomly sampled latent point z x i . The motivation behind a target distribution is to encourage the target encoder to hallucinate variations of the single real examples available for each class. However, during inference, we want to obtain a definite point for the input to avoid any uncertainties. The classification loss is given as, where g can be any classification loss such as cross entropy or negative log likelihood. y denote the same label for both the source x i and targetx i input. The combined loss for the entire network is then given as (see Fig. 4), where α and γ are hyper-parameters. All the network components are trained end-to-end.

Experiments
Dataset: We first collect a set of unlabeled videos for each speaker that are used later to generate the speaker-specific synthetic data using Wav2Lip. Unlabeled videos of the ALS patient are recorded without any manual intervention. For other speakers, the videos are randomly selected from their respective YouTube channels. As the next step, we curate the personalized vocabulary for each of the speakers. For the ALS patient, we curate a list of 200 words with the help of his family. We modify the existing list for the remaining speakers by removing irrelevant words and adding the most occurring keywords in the transcription of the collected videos. Splits of the curated dataset for each speaker is presented in Table 1. We obtain two sets of manually curated real data from the ALS patient's family. We use the first set for model training and the second set as the intermediate test set. We collect additional data by deploying a website that records the patient's word-level mouthings and displays the inference on our best model (see Fig. 1). The website needs external help to start and stop recording the patient. The helper can then either select one of the displayed words as the correct label or manually assign the correct label. Through this exercise, we collect an additional 320 examples. Out of these, we use 200 data points, in addition to the original 200 train data points, to train a model in a two-shot setting. We report our test results of oneshot and two-shot models on the combined set of the intermediate test set and the remaining additional 120 examples (see Table 1). To simulate the same setting, we maintain one real example for training the rest of the speakers. To generate the train and test examples, we use the transcriptions with timestamps available on YouTube for the selected videos.
Preprocessing: Our preprocessing steps are similar to LRwP. The lip landmarks are first detected using dlib [15]. The lip is then cropped out such that it is horizontally and vertically centered in the cropped image. The image is converted to gray-scale and resized to a fixed dimension of 88 × 88. A maximum sequence length of 64 frames is used. A batch size of 16 on a multi-GPU NVIDIA GeForce RTX 2080 setup is used. The models are trained up to 200 epochs using a cosine scheduler and Adam optimizer, with a 3e − 5 learning rate and a weight decay of 1e − 4. The encoders in the experiments are adopted from LRwP that uses Resnet18 and BiGRU, and LTCN that uses Resnet18 and Temporal Convolutional Networks. Table 2 presents a comprehensive overview of all the experiments conducted on the 5 speakers. Spk-1 uses deaf-speech and sign language, Spk-2 and Spk-3 use sign language as their primary mode of communication while mouthing words with imperfect lip movements. Table 3 presents additional experiments performed on the dataset of the ALS patient. We initialize our models with the weights of LRwP or LTCN, both of them pretrained on LRW. We observed that using the pretrained weights leads to faster convergence.

Training strategy
Baseline (Exp-cl−r): We begin by training our model directly on the one-shot examples. Since we are the first to perform one-shot lipreading on a personalized vocabulary, we treat this model as our baseline. Table 2 Exp-cl−r presents the performance of the baseline model on each of the 5 speakers. The average accuracy of the speakers at top-1 and top-5 is only 33.0% and 51.6% respectively. The accuracy of the current SOTA lipreading model on the LRW dataset is 88.5% at top-1. This presents us with a huge scope for improvement.  Table 2: Evaluation of our models against each speaker reported in %. All metrics are evaluated on the curated test set made of only real-data. Spk-1 uses a combination of deafspeech and sign language, Spk-2 and Spk-3 use sign language for communication. Patient denote the ALS patient in our study. r, s, and aug indicate real, synthetic, and augmented-real datapoints. cl indicates standard classification while VE is the proposed technique.  Table 3: Accuracy of additional experiments performed on the ALS patient in %. The test dataset used for both, One-shot and Two-shot experiments is the same. FastSpeech2 and GlowTTS are represented by fs tts and glow tts, respectively. All experiments are conducted using LRwP as the backbone on the combined real, synthetic, and augmented real datasets.
Data Augmentation using Synthetic Data (Exp-cl−r+s): We augment the one-shot examples with a potentially unlimited number of synthetic examples. We train our models with varying amounts of synthetic data. We plot a graph of accuracy against the combined synthetic and one-shot examples to determine the optimal amount of synthetic data needed for each speaker against the one-shot examples as shown in the supplementary, Fig. 1. The optimal number of synthetic examples per speaker is reported in Table 1.
As shown in Table 2 Exp-cl−r+s, we observe a significant jump in the accuracy consistently for every speaker with an overall improvement of 9% and 7% at top-1 and 5, respectively on LRwP, and 7% and 12% at top-1 and 5, respectively on LTCN. Although the accuracy improved over the baseline model, we observe that the model overfits on the synthetic data after a few epochs. Since we use only one example per class for the real domain, the model cannot foresee the target (real) variations that it may encounter during testing. We, therefore, try to introduce variations using Variational Encoders.
Variational Encoders (Exp-ve−r+s): To train the network, we use the one-shot examples as the source domain. For the target domain, we combine the real and synthetic dataset. We observed that using only the synthetic data in the source encoder makes the training highly unstable and the network does not converge. Instead, we allow the source encoder to first fit and then improve.
As seen from Table 2 Exp-ve−r+s, Variational Encoders consistently achieves the highest accuracy at top-5 across LRwP and LTCN. There is an overall improvement of ∼ 19% over the baseline and ∼ 12% over Exp-cl−r+s. The improvement at top-1 accuracy is comparatively much lower, ∼ 15% over the baseline and only ∼ 6% over Exp-cl−r+s. We use PCA visualizations to analyze the latent representations learnt by the LRwP based source encoder, as shown in Fig. 5 (right). We observe better separation for each class compared to the separation of Exp-cl−r+s (middle). We also observe better disentanglement between nonhomophenes such as 'bathroom', 'appreciate', and 'coughing' compared to Exp-cl−r+s. For Varitional Encoders we observe that homophenes 'coughing', 'cooking', 'something' are closer together. This suggests that the model trained on Variational Encoders can get confused for homophenes bringing the accuracy at top-1 down and at top-5 higher. For Exp-cl−r+s however, all the classes seem equally apart.
Ad-hoc Data Augmentation (Exp-cl−r+s+aug and Exp-ve−r+s+aug): In addition to the implicit variations introduced by Variational Encoders during the model training, we introduce explicit real-domain variations by augmenting the one-shot dataset. First, we use moviepy [20] library to speed up and speed down the one-shot videos by a factor of 1.2× and 0.8×. The videos in our work are unconstrained, that is, the actual mouthing could be spread across several frames placed temporally anywhere in the video. Thus, we increase the video frames sequence length to 85 and add temporal variations during training by padding the videos with random number (between 0 to 20) of empty frames at the start and end.
We observe improvement in the performance across speakers for both, classification (Exp-cl−r+s+aug) and Varitional Encoders (Exp-ve−r+s+aug). We observe that for Spk-1 and Spk-2, the performance degrades. Upon further analysis, we find that the videos maintain constant pace and have fewer overall variations. Thus, the added variations during training behave as noise driving the performance down. The overall performance improves by 7% and 3% at top-1 on Exp-cl−r+s+aug and Exp-ve−r+s+aug over Exp-cl−r+s and Expve−r+s respectively. We observe the best performance on one-shot setting with these ad-hoc additions for classification. For Variational Encoders, the improvements are less significant, especially at top-5, indicating the technique itself makes up for these ad-hoc additions.
Additional Experiments on the ALS Patient's Dataset: To observe the affect of using different TTS models for generating synthetic data, we perform an ablation by eliminating one TTS model at a time for data generation and compare its performance against the data generated by combining both the TTS models. As shown in Table 3, the performance with both the TTS combined gives us the best performance. This indicates that the variations introduced by different TTS models are important for generalization.
Lastly, we train an additional model with the additional data obtained from the patient's family to evaluate the performance difference between one-shot and two-shot setting. As seen from Table 3, the performance improves by ∼ 7% at top-5 against the best performing one-shot model on both training methodologies. Specifically for Variational Encoders, the performance improves to 71.64% and 89.36% at top-1 and top-5. This suggests that the network learns a better real-domain latent distribution using the extra real examples.
Evaluation on Joe Biden: Although Joe Biden represents speakers without disabilities, he speaks in an American accent, while LRW is composed of British speakers. As can be observed from Table 2 Exp-cl−r, the performance for Joe Biden on the baseline model is similar to the performance for the deaf speakers. A different accent can also be thought of as a different style of speaking. Thus the pretrained model does not directly adapt to Joe Biden. We observe in his case, adding synthetic data leads to significant improvements in the accuracy over the baseline with an average gain of 21% at top-1 and top-5 (Exp-cl−r+s). Variational Encoders fail to bring expected improvements compared to the other speakers. With LRwP, at top-1, the accuracy drops by 3%, and we see a marginal improvement at top-5. We note that the TTS used to generate synthetic data is of an American accent. This suggests that the synthetic data captures the variations of the real domain exceptionally well in his case. Variational Encoders, on the other hand, adds more noise than valuable variations.

Conclusion and Future Work
In this work, we leap from previous lipreading approaches and propose a one-shot personalized lipreading framework to aid patients suffering from ALS or hearing disabilities. Due to the extreme scarcity of personalized data available for a medical patient, we generate synthetic data to augment our training process. We train our network with Variational Encoders, a domain adaptation technique, to bridge the gap between the synthetically generated examples and the available one-shot real examples for each class. Our method proves to be highly effective, and we achieve over 83% top-5 accuracy for the ALS patient. We also report the performance of speaker-specific models trained for multiple speakers with hearing impairment and a speaker with no disability. In the future, we would like to improve the accuracy of our model while also increasing the vocabulary it can handle. We believe our work achieves essential milestones for the lipreading community and can enhance the communication between people dependent on lipreading. It encourages research in the direction of few-shot lipreading that is an important real-world challenge that can have far-reaching applications including and beyond medical lipreading. Although, Variational Encoders has been proposed for few-shot lipreading, its utility can be explored in other areas with a few-shot settings.

Ablation Experiments
In this section, we report ablation experiments to inspect and gain further insights on the different components of our network. Our experimental setup is identical to the setup mentioned in Section 3 of the main paper. For better readability during comparisons, we have copied the results of

Variational Encoders on Only Single Real Examples
As reported in Section 3.1 of the main paper, our network is trained on the combined data of synthetic and one-shot examples. In this section, we remove the synthetic data to see its benefit. Instead of passing synthetic and one-shot examples, we only use one-shot examples as input to both encoders. Since the Variational Encoder itself acts as a data augmentation network, we assume the network to improve over the baseline (Table 1 Exp-cl−r). As shown in Table 1 Exp-ve−r, the performance of the reduced network improves by an average of 6% and 4% at top-1 and top-5 respectively over the baseline. However, the performance is still lower than the rest of the experiments mentioned in the main paper (Table 2 Exp-cl−r+s and Exp-ve−r+s). This suggests that Variational Encoders with just one example cannot surpass methods with additional data augmentation techniques. But, even with only one example, Variational Encoders is capable of bringing significant improvements to the model. Thus, our approach can be a potential data augmentation technique impacting other tasks requiring a low-data setting.

Effect of ∆ dist
As reported in Equation (4) of the main paper, ∆ ve is a combination of ∆ dist that introduces domain invariance by reducing the distance between the feature space of source and target encoder, and ∆ entropy that allows the encoders to generate task-relevant latent embeddings and distributions using the loss of the downstream task. In this section, we remove ∆ dist and observe the behavior of the network using only ∆ entropy . This allows us to determine the effect of introducing domain invariance. Loss of the reduced network is thus given by, As can be seen from Table 1 Exp-ve−r+s−mod, the accuracy of the reduced network drops by 1% and 5% across the speakers at top-1 and top-5 respectively over Exp-ve−r+s. We observe, ∆ entropy plays a substantial role in our network. Generating variations using Variational Encoders lets the downstream classifier see various examples of the real domain making it robust. We observe the accuracy drop at top-5 is higher than top-1. The drop suggests that ∆ dist may introduce domain invariance across related mouth movements (such as homophenes). 2 Need for Variational Encoders

Other Data Augmentation Techniques
In this section we report additional approaches that we tried to augment the one-shot examples. An expert human lipreader can only lipread 40% of a given sentence [? ]. Thus, manually labeling unlabeled videos is challenging. We therefore try to automatically label the unlabeled videos.
Best Match using Syncnet -Syncnet [? ] is an audio-video synchronization model to synchronize mouth movements and a given speech utterance. It generates a confidence score for a given audio and video segment. We exploit this method to find segments of unlabeled videos closely matching the generated TTS audios. We use different confidence thresholds to label the videos and vary the length of video segments based on the audio length.
Pseudo Labeling -Pseudo Labeling [? ? ] is a technique that iteratively labels the unlabeled data in a semi-supervised setting. A model is trained on the labeled data and is used to label the unlabeled data known as pseudo labels. The model is trained further on the pseudo labels. These steps are repeated until the pseudo labels converge. It assumes the first model trained on the labeled data to be accurate so the pseudo labels are close to the actual labels. Pseudo labels can be noisy, but as the training progresses, labels converge to stable and accurate predictions. We first segment the continuous unlabeled videos into smaller word-level video segments. We use lip-landmarks detected by dlib to measure the rate of change of lip-movements and determine the pauses between words. We then segment the videos on these pauses. We then iteratively label these videos using the pseudo labeling technique.
We observed that these methods led to noisy labels. The performance for each model mentioned Table 2 of the main paper dropped by an average of 15% and 7% on adding pseudo labels and syncnet labels respectively.

Test Loss on One-Shot Examples
Training a classifier on one-shot examples restricts the model from seeing many variations of the real domain and thus tends to underfit the real domain. We show this phenomenon in Fig. 2 that plots the test loss of Exp-cl−r+s (of the main paper) and Exp-ve−r+s. In Expcl−r+s, we see loss on the test set flattens and increases after Epoch-12. In Exp-ve−r+s we see the loss keeps decreasing slowly. This suggests that Variational Encoders introduces meaningful variations of the real domain.