Sample Drop Detection for Distant-speech Recognition with Asynchronous Devices Distributed in Space

In many applications of multi-microphone multi-device processing, the synchronization among different input channels can be affected by the lack of a common clock and isolated drops of samples. In this work, we address the issue of sample drop detection in the context of a conversational speech scenario, recorded by a set of microphones distributed in space. The goal is to design a neural-based model that given a short window in the time domain, detects whether one or more devices have been subjected to a sample drop event. The candidate time windows are selected from a set of large time intervals, possibly including a sample drop, and by using a preprocessing step. The latter is based on the application of normalized cross-correlation between signals acquired by different devices. The architecture of the neural network relies on a CNN-LSTM encoder, followed by multi-head attention. The experiments are conducted using both artificial and real data. Our proposed approach obtained F1 score of 88% on an evaluation set extracted from the CHiME-5 corpus. A comparable performance was found in a larger set of experiments conducted on a set of multi-channel artificial scenes.


INTRODUCTION
Distant multi-microphone conversational Automatic Speech Recognition (ASR) in real home environments has attracted a considerable interest in the research community in the past decade. There is a large number of ASR systems deployed successfully in applications which require robustness to different sources of degradation of the input signal. Moreover, distant-ASR introduces many new complex challenges to solve, mainly due to environmental acoustics and unpredictable noisy conditions, often represented by interfering speakers [1]. In order to improve the recognition performance, it is very common to adopt a multi-microphone multi-device setting which allows to observe the scene from different locations in space. For this purpose, the challenges and corpora such as CHiME, DIRHA, and AMI/AMIDA [2,3,4] were created to address a wide spectrum of research topics related to scenarios such as automatic transcription of speech in office and home environments, with spontaneous speech input, different noisy and reverberant conditions as well as microphone and device placement configurations [5,6]. In the specific case of the CHiME-5 challenge, we are in the presence of simultaneous recordings of different real conversational scenarios from multiple microphone arrays, distributed in rather large spaces. The recording sessions in this case are done by using 6 Kinect microphone arrays and binaural in-ear microphone pairs for each of the 4 speakers. The signals recorded by each Kinect microphone array are sample-synchronous within the device. However, the different devices can be subjected to asynchrony [7], due to both small clock speed variations and sample drop events. The latter aspect causes misalignment between different signals, creating additional hurdles for tasks such as voice activity detection, beamforming, and any other multi-channel processing that would require synchrony at sample or frame level [8,9]. It is also worth noting that such misalignment is additional to an intrinsic one that characterizes audio signals acquired by microphones largely spaced one to another, i.e., a shift due to propagation time delays related to the geometry of the problem. This can strongly depend on the spatial location of each sound source, which is typically represented by both active speakers and coherent noise sources, and is continuously changing in experimental scenarios such as CHiME-5. In principle, clock drift, sample drop, and sound source localization [10] could be jointly addressed with the aim of obtaining a rather accurate re-alignment between all signals. Under controlled conditions, e.g., using artificial multimicrophone audio datasets, this joint goal can be addressed, also thanks to fully available ground-truth information. On the contrary, in a real context the problem becomes highly challenging, and generally it is not supported by accurate ground-truth information, such as speaker location and drop duration itself. Just to give a rough idea about the relevance of the problem, in the case of the CHiME-5 challenge, the shift among different devices due to sample drops exceeds one second in many sessions, while the duration of each drop can range from a few milliseconds to more than hundred milliseconds.
As discussed in the following, in order to identify temporal intervals that can be affected by sample drop, one can apply standard cross-correlation techniques, which generally need to process rather large context windows (e.g., 10-15 seconds duration), in order both to accurately detect and to eventually quantify the drop duration. A crucial aspect is represented by the time accuracy with which the drop is detected. For this purpose, the analysis on a shorter context window is strongly required. The main focus of this paper is on the sample drop detection within a short context window, which is tackled by adopting a neural classifier. The design of the architecture has been conceived to respond to a two-fold problem. First, the model has to learn a latent representation of the signal that allows for the comparison between the signal from a device affected by a sample drop event, and the signals coming from the other devices. Secondly, the system has to attain robustness to the resulting time shift.
The paper is structured as follows. After a short overview of the relation to the prior works, Section 3 describes the binary classification task, cross-correlation based method and the neural solution. We then present the dataset, the conducted experiments and the relative results, in Section 4, followed by our conclusions.

RELATION TO PRIOR WORK
The relevance of the sample drop detection task in applications of multi-device multi-microphone processing is not restricted merely to the research related to CHiME-5. Nowadays, many ASR systems rely on multiple devices distributed in the environment, which transmit eventually the signals to the Cloud. Sometimes, these devices are low-cost and characterized by possible interrupts in their operational activity. The communication step itself can also be affected by packet loss, for instance in the case of limited internet access. Furthermore, there are two different research fields to which the general concept of the detection of missing data can relate. However, they both differ from our approach. According to the best of the authors knowledge, there is no similar prior work for the sample drop detection in distributed microphone arrays for distant-ASR.
A first group of studies is done for packet loss concealment algorithms, and the perceptual evaluation of the speech transmission quality in the communication networks. The term discontinuity defined as one of the three orthogonal speech quality dimensions [11], describes the isolated distortion introduced by packet loss due to the transmission errors or network delay. The predictive model designed with both handcrafted features and deep learning methods estimates the packet loss rate and explains the effect of the transmission errors on the speech quality [12,13].
Another related field is in the framework of missing feature techniques, where it is stated that a robust ASR system must acknowledge that some of the spectro-temporal regions are dominated by noise, and hence incomplete or missing. These techniques normally consider a first step for the estimation of a mask which divides the input signal into reliable and unreliable regions [14,15,16]. In order to deal with the unreliable regions normally the two marginalization and imputation methods are proposed [17,6].  Fig. 1. Signals and related spectrograms referred to recordings acquired by two asynchronous devices for CHiME-5. The signal of case b) is affected by a sample drop, which is highlighted by the shorter vowel sequence in the middle of the sentence (see blue rectangles) and by a different spectrographic alignment after that time instant.

CLASSIFICATION TASK
Let x(t) be a source signal, and y m,k (t) the corresponding signal acquired at time sample t by the m-th microphone of the device k, with m ∈ {1, · · · , M } and k ∈ {1, · · · , K}, where M = 4 and K = 6 in CHiME-5. Denote by Y m,k (·) the Short Time Fourier Transform (STFT) of the acquired signal. Given a context window of length , within a time interval of length T , with T , our goal is to decide whether a device has been subjected to a sample drop event, as depicted in Figure 1.
We denote by C(·, ·) a function which correlates or compares |Y m,hyp (·)| of a hypothesis device with the remaining reference devices |Y m,k (·)|, for all k = hyp. The binary classification task can be defined as follows: where F and Comb are a binary classifier and a combination strategy, e.g. averaging, respectively. Furthermore, we assume that the input to the function C(·, ·) is the log-magnitude spectrum 20 log 10 |Y m,k (·)|, in decibel.

Cross-correlation method
The normalized cross-correlation method adopted in this work, derives from a standard approach to pattern matching and feature detection often deployed in application fields based on image processing [18,19,20]. Given the log-magnitude spectrum 20 log 10 |Y m,k (.)|, it is possible to represent the signal in terms of spectrogram, whose portions are then processed as images, namely to find the best match shift between a pattern (see Figure 2) extracted from a signal of the device hyp, and a longer temporal sequence, related to the device ref , which is very likely covering the absolute time range of the former one. Due to different propagation time delays from active sound sources to the microphones of each device, and to minor drift effects due to clock mismatch, in general the two spectrographic representations are not synchronized in time. Under noisy and reverberant conditions, they can differ substantially, in particular in the case of distant devices. Nevertheless, a global similarity is often preserved in terms of temporal relationships among most significant speech contents (e.g., formants and voice onsets), which leads to a higher correlation value corresponding to the correct time shift. In the case of a sample drop that affects one sequence, this match drastically changes, due to an entire vertical slice of the spectrogram which has been lost in one of the two channels. However, this is the key aspect of the proposed method, since a new correlation peak is often found at a time shift that differs from the previous one (observed before the loss) by the exact duration of the loss. In order to increase the robustness of the method, we combine the analysis outputs obtained for each device by correlating the corresponding signal with each of the other K − 1 device signals. As a result, this processing produces K cumulative functions, one per device.
The described method is effective both to identify rather large intervals (e.g., 10-15 seconds) that are likely to include a possible drop, and to provide an estimate of its duration. However, in our preliminary work we observed some limitations when one aims to derive a more accurate location in time (e.g., less than one second resolution) of this drop. For this reason, in this work we explore the application of this normalized cross-correlation method for a preprocessing before the application of a neural classifier whose goal is to eventually decide between loss and no-loss, in a short context window of a few seconds.

Neural-based method
As mentioned in the last Section, the intervention of the neural model is carried out over a short context window, once the cross-correlation method has identified a set of candidate large time intervals.
The cross-correlation method works upon raw features like the log-magnitude spectrum, whereas deep learning approaches build intermediate abstractions of the features throughout a stack of neural layers [21]. These abstractions are usually more robust to low-level feature changes (like temporal shifts) and also facilitate a learnable preprocessing to solve a task of a high-abstraction level like classification. Hence we leverage the capacity of neural networks to process two separate log-magnitude spectrum sources, the hypothesis and the reference, and to classify whether there is sample loss or not in the hypothesis. To that end, we design a network that has two branches processing the raw spectral inputs, and relates them in the end with an attention mechanism. The proposed model is depicted in Figure 3. Given the hypothesis input and the reference input, both of length , we inject them into the two encoder branches. Each input branch is composed by a convolutional neural network (CNN) frontend, specially suitable to detect local temporal correlations in the spectral frames, and a long-short term memory (LSTM) [22] block that exploits specially the long-term sequential dependencies of the input sequence. The convolutional front-end also decimates the sequence lengths of the inputs by 4 through a couple of max-pooling layers that halve temporarily the feature maps right after each of the first two convolutional blocks. This type of model is known as siamese network [23], where both branches are constrained to share the same weights as they process the same type of input data. The optimization problem in the siamese network then leads to learning two separate embeddings which have positive inner products for sequences of the same class but negative inner products for those of different classes [24]. Once we obtained the two hidden representations of the signals, we forward them through the multi-head attention (MHA) component, which outputs a weighted sum of the reference computed by a compatibility function of the hypothesis to itself [25]. As a final step, we took the last time step values and forwarded it through a multi-layer perceptron (MLP), right before the application of a sigmoid function σ.
During training, the optimization procedure is not taking the multi-device scenario into account. The learning problem is reduced to the detection of the sample drop between pairs of hypotheses and references. Starting with the Equation 1 we define the function C to be a composition between a multi-head attention block and a linear layer. By setting F = σ, we have: During the test time, for each device j we forwarded five segment pairs using all devices k = j. The distinct output values of the network are then combined together by taking the average (or median) value, and leading to: (3)

EXPERIMENTS
The CHiME-5 corpus does not offer a reliable ground-truth for the sample drop detection task. Therefore, the overall training of the network has been carried out by using artificial data, and in two distinct stages. Concerning the evaluation of the performance of the model, we did not limit us to the mere usage of artificial data, and used also an evaluation set extracted from the real data.

Noisy-reverberated LibriSpeech
The first training round of the neural model was conducted using two different noisy and reverberated datasets, both derived from the LibriSpeech train-clean-100 corpus [26], containing 28539 speech utterances. Each utterance was filtered by using a different room Impulse Response (IR), characterized by a reverberation time typically ranging between 0.4 and 0.8 seconds. IRs were computed by using a modified version of the image method [27,28]. Moreover, the two datasets differ both due to a different environmental noise, and to a different sound propagation time shift. Overall, SNRs are generally in the range between 5 and 25 dB. Finally, a random sample loss event was simulated on one of the two versions, and from each of them segment pairs of hypothesis and reference were extracted.

Artificial multi-device mini-scenes
The artificial dataset incorporates 1182 mini-scenes that were generated using clean Librispeech signals, and an accurate and realistic simulation of multi-microphone acquisition in a noisy and reverberant environment, as described in [28]. Each scene refers to a different distribution of the six Kinect devices in space, with varying room size, location of speakers, and noise sources, as well as directivity of source, microphone polar pattern, and reflection coefficient of each wall. The mini-scenes were created without any sample drops. Each scene was then postprocessed to simulate the introduction of none, one, or more drops, for a total number of 880 drops. The data set was split into 782, 100 and 300 mini-scenes for drop detection training, development and evaluation, respectively. The scene duration ranges from 5 to 30 seconds. This material has then been organized to expose the neural model to a balanced set of drop/no-drop examples.

Real dataset
In order to evaluate the proposed model in a more realistic context, it is necessary to test the performance also on a real dataset. In this regard, it is worth noting that there exists no real corpus for studies on sample drop detection. The evaluation set was created by extracting 65 segments, from which 31 with sample drop, from three sessions of the train portion of the CHiME-5 corpus, namely sessions 03, 07, and 08. Some of these segments are characterized by a loss that was found through a very careful visual inspection of the related spectrograms, as outlined in Section 3.1.

Setting
The duration of the loss for the two different contaminated versions of LibriSpeech, was drawn from a left-truncated normal distribution N (600, 150), with a cut value of 50. The samples belonging to that interval were then eliminated from one of the segments. A one second context window containing this loss was isolated, by positioning the loss point randomly within the context window. The parameters of the pre-trained model on this data were then used for a further training step on the mini-scenes. The performance of the final model was tested on two separate evaluation sets belonging to both artificial and real data. The labels of the one-second-long segment pairs are distributed equally in both training and evaluation sets. Regarding the network's architecture, two different experiments were carried out. First, we concatenated h hyp and h ref defined in Equation 2 and applied the sigmoid function on the last time step. Then we added the attention block and operated in the same way for the binary output. We also experimented with the average activations over the time values, but it proved to be less effective. We trained the models for 20 epochs on minibatches of size 50 and 30 for the pre-trained and final models respectively. We used the Adam optimizer [29] with the default PyTorch [30] parameters and learning rate 5 · 10 −5 . Regarding the network structure, fully connected and convolutional layers comprise 512 units, with kernel length N = 5 in the case of convolutions. The LSTM layer contains 1024 cells. The model structure hence amounts to approximately 15 M of parameters in total.

Results
The results obtained from different experiments are reported in Table 1. A quick overview of the numbers reveals the importance of having a two-stage training procedure, since the pre-trained model on the contaminated LibriSpeech or the model trained directly on the mini-scenes without a pre-training phase have both a poor performance. The addition of the attention mechanism is also another important aspect. The difference of F1 score of the final model over mini-scenes is much smaller than the one over CHiME-5, in case of application of multi-head attention. Furthermore, the big gap between the result of the pre-trained model and others indicates the importance of having an appropriate data for multi-device case. For the Fast Fourier Transform (FFT) analysis frame length we tried 32 and 64 milliseconds (ms) with 50% overlap. The former choice leads to better results, specially in case of real data. During the test time we combined the output of the network for different pair devices by using major voting, averaging and the median value. The best choice resulted to be averaging and median for mini-scenes and CHiME-5, respectively. Table 1. Precision (P), Recall (R) and F1 Score [%] for the neural model pre-trained only on contaminated LibriSpeech (Pre-NN), and the final neural model (NN) with and without attention and pretraining step, on artificial multi-device mini-scenes and CHiME-5 corpus, using FFT analysis frame length of 32 ms.

CONCLUSIONS
In this paper, we investigated a possible approach for the detection of sample drops in the context of multi-microphone devices distributed in space, a very challenging technical issue that recently interested many researchers working on the CHiME-5 challenge.
The proposed approach consists in combining a normalized cross-correlation processing and a neural classifier, in order to detect short context windows that are characterized by a possible loss. Experimental results show that a classification performance of 88% F1 score is obtained both on simulated and on real evaluation data sets. In the near future, we plan to improve this performance along different research directions, such as cross-processing all the microphone signals acquired by the available devices.
Though our current main focus is on the CHiME-5 data set the proposed solution can be applied to other similar contexts, which are affected by loss of segments in the audio input signals. Our work also represents a preliminary step towards a joint combination of sample drop detection and quantification of the duration of the lost segment, which is a research issue under study. Moreover, we envisage a third processing step that includes a possible reconstruction of the lost information, based on exploiting redundant information available from higher-quality input channels not affected by the sample drop. For all of these foreseen directions, we also plan to verify soon a possible improvement in terms of recognition performance on the CHiME-5 task provided by the application of the proposed method and of the resulting array synchronization.