tPLCnet: Real-time Deep Packet Loss Concealment in the Time Domain Using a Short Temporal Context

This paper introduces a real-time time-domain packet loss concealment (PLC) neural-network (tPLCnet). It efficiently predicts lost frames from a short context buffer in a sequence-to-one (seq2one) fashion. Because of its seq2one structure, a continuous inference of the model is not required since it can be triggered when packet loss is actually detected. It is trained on 64h of open-source speech data and packet-loss traces of real calls provided by the Audio PLC Challenge. The model with the lowest complexity described in this paper reaches a robust PLC performance and consistent improvements over the zero-filling baseline for all metrics. A configuration with higher complexity is submitted to the PLC Challenge and shows a performance increase of 1.07 compared to the zero-filling baseline in terms of PLC-MOS on the blind test set and reaches a competitive 3rd place in the challenge ranking.


Introduction
Several crucial processing steps for virtual communication have immensely profited from deep learning, e.g., speech enhancement (SE) [1,2] and acoustic echo cancellation (AEC) [3,4]. The underlying models can be created with deep-learning tools alone or based on a combination with "classic" signalprocessing approaches such as adaptive filters and with deeplearning models [3,5]. Another important challenge is to improve communication systems with effective packet loss concealment (PLC) using deep learning. The task of PLC is to conceal or predict lost packets during packet-based transmissions. A well-performing PLC system is especially important for poor network connections, since more packets are lost or overly delayed and therefore discarded. A robust PLC system could also decrease listening fatigue and effort in situations with poor audio quality and therefore result in overall improved communication channels. PLC systems should work efficiently so they can be used for many device classes without fully utilizing the processing power and sacrificing battery run-time on mobile devices.
Packet loss concealment is a sequence problem since missing frames from an audio sequence must be predicted from past information. An early deep approach [6] utilized feed-forward neural networks for predicting the magnitude and phase from a number of past frames using mean-squared-error based loss in time-frequency domain. Another approach is a causal sequence to sequence model such as RNNs as used for SE [7,2,8]. In related research, RNN-based models have been applied to PLC [9,10,11]. In [10] a recurrent model with online adaptation is proposed by exploiting the recent past of the signal. [9] used an LSTM-based PLC model to increase the performance of speech emotion recognition trained with a concor-dance correlation coefficient as cost function. A convolutional recurrent structure for time-domain PLC was introduced in [11] trained with a mean absolute error cost in time domain. Alternative deep approaches utilizing generative adversarial networks (GAN) [12,13,14], which were shown in the past to be able to predict speech frames.
A comparison between these studies is difficult because of the lack of a common training and test set, and therefore Microsoft proposed a deep PLC Challenge for Interspeech 2022 [15]. The authors provided a large amount of speech data, traces of lost frames from real calls and a deep model for predicting mean opinion scores (MOS), indicating PLC performance of enhanced audio. The challenge evaluations will be performed as in previous challenges organized by Microsoft [16,17,18] with a crowd-sourcing approach on the Amazon Mechanical Turk.
For the first time, to the best of our knowledge, we propose in the context of the PLC Challenge a time-domain sequenceto-one (seq2one) RNN model for PLC trained with a combined magnitude and complex mean absolute error (MAE) loss in time-frequency domain. In contrast to previous RNN approaches [9,10,11], the model predicts only lost frames from a short context buffer similar as in [6] in a non-continuous way instead of running inference for each incoming audio frame to update internal states. This seq2one approach can decrease, depending on the complexity of the model, the computational load of the host system and probably increase the battery run-time of a mobile device.

Deep PLC Strategy
The proposed PLC seq2one system is based on a context buffer of length n that contains non-overlapping frames of length m. For m 10 ms (160 samples at 16 kHz sampling rate) are chosen. It is therefore possible to use one look-ahead frame and still comply with the PLC-Challenge rules, which specify a maximum delay of 20 ms [15]. The use of look-ahead frames has been shown to be beneficial in [11] to connect predicted frames better to available audio. When no packet loss is detected, the current and the look-ahead frames are concatenated to a frame with 320 samples and subsequently hann windowed. When a packet loss is detected, the dPLC model predicts a 320 sample long frame that is windowed as well. With an overlap-add procedure the current 10 ms output frame is predicted which is time-aligned to the current input frame. The current output frame is written back to the context buffer. This procedure, especially the windowing and overlap-add, was chosen to reduce artifacts on the edges of lost packets and connect the predicted content better to the available frames and so pursues a similar goal as the cross-fading shown in [6]. The proposed PLC sys-  Figure 1: Illustration of the proposed PLC system. Incoming audio is written to a context buffer. When no frame is lost, the current frame and the look-ahead frame are concatenated and hann-windowed. If the the frame on position x or x + 1 is lost, the dPLC predicts an audio frame which is also hannwindowed. The resulting windowed frames are processed by an overlap-add procedure returning the current frame y time aligned to input frame x.
tem is also illustrated in Figure 1.

Model Architectures
The model is build as seq2one model predicting one time domain frame from a context buffer. The general architecture is inspired by the NSnet2 architecture [7] and an architecture proposed in [11]. It was adapted to seq2one task and working completely in the time domain. First, the frames of the context buffer are processed by a fully-connected (FC) layer with relu activation that maps the time-domain frames to a non negative feature space, as has been proposed with the TasNet approach [19,20,21]. Second, all frames are processed by another FC layer with leakyrelu activation. The resulting intermediate representation sequence is processed by two consecutive Conv1D layers with leaky relu-activation and kernel size 4 and 2, to extract local patterns from the sequence. The sequence is padded so that the resulting sequence has still the same length. The sequence is subsequently processed by two bidirectional Gated Recurrent Unit (GRU) [22] layers, which enables the use of information about the whole sequence. The final hidden state of the second BGRU is forwarded to two FC mapping layers with leakyrelu activation. A last FC layer without activation predicts the time-domain frame. To construct a feed-forward baseline (FF-Baseline) the Conv1Ds and BGRUs layers are replaced by a flatten operation to remove the sequence dimension and 3 FC layers with 512 units and leaky-relu activation.
In total, three different size configurations are tested for the proposed tPLCnet to explore the scaling properties of the model (Small, Medium and Large). The number of units or filters are listed in Table 1. Additionally, Table 2 shows the number of Multiply Accumulates (MACs) and execution time for predict-ing one 10 ms frame on an Intel i5-7267U dual-core processor clocked at 3.1 GHz.

Loss Functions
Recent studies have shown that the loss function is crucial to perceived audio quality in speech enhancement [23], but at the same time, relatively simple loss functions are surpassing more complex ones [24]. To quantify the effect loss functions for PLC, our study compared three loss functions, which are all applied to the whole enhanced utterance. [11] proposed to use a simple mean absolute error (MAE) in time domain which shown promising results.
where the loss calculated as the mean absolute difference of the clean audio x and the predicted audiox. Due to its definition in time domain, MAE is phase sensitive and is -in comparison to the mean squared error -less sensitive to outliers. In [23], it was shown that the MAE in time-frequency domain (TF-domain) produces competitive results for speech enhancement. MAE has also been explored for speech synthesis [25], which in some aspects is related to PLC. To calculate the MAE in TF-domain, the clean and predicted utterances are subject to an STFT which results in the complex TF representations X(t, f ) andX(t, f ). The magnitude MAE loss is given by Results of related studies indicate that integrating a phase component in the loss function can be beneficial [26,23], which can be implemented by using a complex distance. The combined TF-domain MAE loss is given by where α is a weighting factor for the amount of phase sensitivity. In this study, α = 0.1 is chosen, which was found to be a good trade-off between some phase sensitivity and a good magnitude reconstruction in preliminary experiments. The level of the speech is sampled from a normal distribution with a mean of -26 dB and a standard deviation 10 dB. The known test set contains around 2.7 h of clean speech, degraded speech and PLC traces. As in the training set, each file is around 10 s long. The data is divided in three subsets by the organizers depending on the maximum burst loss length of the traces. The categories are: low loss up to 120 ms (52 % of the data), medium loss from 120 to 320 ms (32 % of the data) and high loss from 320 to 1000 ms (16 % of the data).
The blind test set has the same properties, but does not contain the clean speech files. Further details on the data can be found in the challenge description [15].

Training Setup
The models are trained over 200 epochs with a batch size of 16 and a learning rate of 5e-4. The learning rate is multiplied by 0.8 when the loss on the test set does not decrease for three consecutive epochs. The Adam optimizer is used in combination with a gradient norm clipping of 3. The context buffer size was set to a value of 6 as a trade-off between complexity and performance. During training, the four most recent frames of the context buffer are sampled from the degraded signal, as well as the last two frames from the clean signal. We found that this strategy improves performance compared to training on the degraded audio only. The training for the medium-size model takes around 6 h on an Nvidia RTX A5000.

Objective and Challenge Evaluation
The first objective metric used is the wide band perceptual evaluation of speech quality (PESQ) [27], which was originally built to evaluate the quality of transmitted speech. The second one is DNS-MOS P.835 [28] which is a single-ended/blind measure based on a deep-learning approach for estimating Mean Opin-ions Scores (MOS) of speech quality (nSIG), background noise quality (nBAK) and an overall quality (nOVR). DNS-MOS was trained with ratings from previous deep-noise suppression challenges organized by Microsoft [18]. The last measure PLC-MOS [15] is provided by the challenge organizers. PLC-MOS utilizes a similar approach as DNS-MOS but can be used as double-ended or single-ended measure. It predicts MOS scores indicating the quality of the PLC system.
The challenge organizers evaluate the improved audio created from blind test with four metrics: Double-ended PLC-MOS, second, the DNS-MOS P.808 [29], which returns an overall quality rating. Third, a crowd-sourced Comparison Category Rating (CCR MOS) is used, for which the participants are asked to rate the enhanced audio compared to the clean reference. This evaluation is run on the Amazon Mechanical Turk. Fourth, the word accuracy calculated by the Microsoft Azure cognitive services speech recognition is used. The final score is calculated from CCR MOS and word accuracy.

Experiments and Results
In the following the experiments and results are described: Objective results on complexity comparison: Table 2 shows the objective results for the zero filling baseline, the feedforward baseline trained on the same setup, as well as three complexity configurations of the tPLCnet (cf. Table 1). The FF baseline already reaches a good performance increase for all metrics. Performance increases with model size for all metrics. The total improvement in terms of PLC-MOS for tPLCnetLarge is 1.08. The PESQ improvement over the zero filling baseline is 0.58 and for DNS-MOS nOVR 0.82. The PESQ and PLC-MOS improvements are constant over all subsets.
Objective results on loss functions and loss function configurations : We compared MAEtime, MAEmag and MAEcomb; for the latter, three different window sizes with 50 % overlap were tested to identify an optimal configuration for the PLC task. All results were acquired with tPLCnet Medium . The results are shown in Table 3 Comparison of context buffer lengths: The length of the context buffer is a major factor for complexity in terms of operations for our seq2one model, since the whole buffer must be processed each time by the convolutional and the BGRU lay- ers. Hence, we explore the effect of different buffer lengths on performance. Figure Table 4. tPLCnetLarge with MAE comb using a loss window length of 32 ms ranked 3rd in the challenge. It improves on all metrics except for word accuracy. The 1st and 2nd ranked entries improve on all metrics (including word accuracy). Our tPLCnet achieves the second-best score in terms of PLC-MOS.

Discussion
In the following some aspects of the results are discussed. While an increase in complexity of tPLCnet also results in a performance increase, the gain seems to saturate compared to the negative aspects of increased complexity. The FF baseline already produces a robust PLC performance, indicating that a large portion of the performance comes from our general approach in combination with the training setup. This implies that for real on-device applications the PLC model can be chosen reasonable depending on the available resources and it must not be the largest possible model. In [8] a similar saturation was found for NSNET2 architecture for SE, i.e., a larger amount of RNN units heavily increases the complexity, but only slightly increases the noise reduction performance.
Regarding the comparison of loss functions, the spectral MAE emerged as the best choice, with the time-domain MAE achieving clearly lower scores. The magnitude-only loss reaches comparable performance to the combined version which is in line with observations in [23]. Interestingly, the highest PLC-MOS is observed for a window size of 32 ms, while the best choice for other metrics was a value of 20 ms. Also, the higher spectral resolution achieved with the 64 ms window seems not to be helpful for PLC.
The evaluation of the length of the context buffer shows that a short context buffer with only 4 frames already reaches a good PLC performance while very long context buffers even seem detrimental. A similar effect for longer input buffers was observed in [6]. This suggests that the main information required for robust PLC is temporally localized around the lost frame and long-term dependencies only weakly influence overall performance. However, the observed performance differences between the context buffer lengths are not very large, which also means a short buffer length can be chosen without sacrificing performance.
One observation from the challenge result is that a high PLC-MOS does not necessarily result in a high CCR-MOS. Our model reached a higher PLC-MOS than the 2nd place, but the the 2nd ranked model reached a significantly better result in the crowd-sourced CCR-MOS. We assume this can be caused by addtional speech enhancement used by the 2nd-ranked model, which modifies the spectrum such that PLC-MOS is decreased but perceived audio quality is increased. Word accuracies for the baseline and the tPLCnet are very close to each other (0.861 and 0.859, respectively), i.e., the PLC improvement perceived by the raters does not help the speech recognition system. We assume this is due to a slight deterioration of clean speech in the vicinity of the edges of lost frames, since our algorithm smooths the transition between these areas. It also seems that the speech recognition system used by challenge organizers can already handle missing packets relatively well.

Conclusions
This study introduced tPLCnet, a straightforward real-time approach for predicting lost packets from a short context buffer in the time-domain. It achieves state-of-the-art performance and ranks 3rd in the deep PLC Challenge. The training was performed with open-source data which resulted in a robust and reproducible approach which can be easily applied in the realworld.