Computer Science
Computer Science-2021.gz:143442
Computer Science
32ce435ea1333d9c1ca101862a7d2eb36c564193
Joint Weakly Supervised AT and AED Using Deep Feature Distillation and Adaptive Focal Loss

A good joint training framework is very helpful to improve the performances of weakly supervised audio tagging (AT) and acoustic event detection (AED) simultaneously. In this study, we propose three methods to improve the best teacherstudent framework of DCASE2019 Task 4 for both AT and AED tasks. A frame-level target-events based deep feature distillation is first proposed, it aims to leverage the potential of limited strong-labeled data in weakly supervised framework to learn better intermediate feature maps. Then we propose an adaptive focal loss and two-stage training strategy to enable an effective and more accurate model training, in which the contribution of difficult-to-classify and easy-to-classify acoustic events to the total cost function can be automatically adjusted. Furthermore, an event-specific post processing is designed to improve the prediction of target event time-stamps. Our experiments are performed on the public DCASE2019 Task4 dataset, and results show that our approach achieves competitive performances in both AT (49.8% F1-score) and AED (81.2% F1-score) tasks.

Abstract-A good joint training framework is very helpful to improve the performances of weakly supervised audio tagging (AT) and acoustic event detection (AED) simultaneously. In this study, we propose three methods to improve the best teacherstudent framework of DCASE2019 Task 4 for both AT and AED tasks. A frame-level target-events based deep feature distillation is first proposed, it aims to leverage the potential of limited strong-labeled data in weakly supervised framework to learn better intermediate feature maps. Then we propose an adaptive focal loss and two-stage training strategy to enable an effective and more accurate model training, in which the contribution of difficult-to-classify and easy-to-classify acoustic events to the total cost function can be automatically adjusted. Furthermore, an event-specific post processing is designed to improve the prediction of target event time-stamps. Our experiments are performed on the public DCASE2019 Task4 dataset, and results show that our approach achieves competitive performances in both AT (49.8% F1-score) and AED (81.2% F1-score) tasks.
Index Terms-Acoustic event detection, feature distillation, adaptive focal loss, event-specific post processing

I. INTRODUCTION
A UDIO tagging (AT) refers to predict the category of acoustic events that occur in a audio clip, while acoustic event detection (AED) not only needs to identify whether certain events occur, but also requires their onset and offset time-stamps. Both AED and AT can be applied in many areas such as smart home [1], health monitoring [2] and multimedia retrieval [3], [4], etc. As labeling large-scale training data with detailed annotations is high cost and time-consuming, the challenges of Detection and Classification of Acoustic Scenes and Events (DCASE) from 2018 to 2020 Task 4 [5]- [7] have been launched to exploit a large amount of unbalanced and unlabeled training data together with a small weakly annotated training set to improve AED system performance. These challenges have attracted increasing research attention [8]- [11]. In this study, we also focus on this weakly supervised task to improve both of the AT and AED system performances.
Many previous works have been proposed to improve AED systems from variety aspects, such as proposing new networks in [12], new features in [13], better post-processing methods in [14], [15] and training data enhancement techniques in [16], [17], etc. However, for the weakly supervised AED tasks, the key issue is how to design a better modeling framework to fully exploit the large-scale weakly and unlabeled training data. Therefore, many recent works focus on the semisupervised or unsupervised learning approaches, such as, in DCASE2018 Task 4, the baseline used two convolutional recurrent neural network (CRNN) with a two-pass training to predict labels of unlabeled clips [18]; The best system of this challenge was based on the Mean-teacher model [19], [20], where two same CRNN networks were taken as the teacher and student model with weight-averaged consistency targets to exploit the large amount of unlabeled data; In DCASE2019 Task 4, the Guided Learning (GL) [21] convolutional system achieved the best results. The GL is also a teacher-student (T-S) framework, but unlike the homologous T-S model in Meanteacher, GL uses two different model structures to enable an appropriate trade-off between AT and AED. Two models are trained synchronously and forced to learn from the unlabeled data with tags generated by each other.
In this study, our work is also based on the Guided Learning. However, we modify it to learn the models for AED and AT tasks as two independent branches in a joint framework. Three new methods are proposed to improve this framework. Motivated by the knowledge distillation techniques in recent machine learning community [22]- [24], a frame-level targetevents based deep feature distillation is first proposed. It aims to leverage the maximum potential of limited stronglabeled data to learn better intermediate feature maps in the weakly supervised framework. In addition, to address the class imbalance of large-scale weakly labeled and unlabeled training data and different level of classification and detection difficulty of each target event, we propose a two-stage training strategy with an adaptive focal loss together to enable an effective and more accurate model training. During the training process, the contribution of difficult-to-classify and easy-to-classify acoustic events to the total cost function is dynamically adjusted in each iteration, which makes the contribution of those easy-toclassify events are down-weighted and the model can rapidly focus on hard events in later stages. Furthermore, an eventspecific post processing is designed to fix the prediction errors that result from outliers. Results on the DCASE2019 Task 4 challenge show that our proposed methods can achieve competitive performances in both AT and AED tasks.

A. Joint framework
The proposed joint architecture for both weakly supervised AT and AED is shown in Fig.1  The teacher model has five double-layer CNN blocks with a larger time compression scale that professional for a better audio tagging, while the student model only has three singlelayer CNN blocks with no temporal compression scale for a better event boundary detection.
Compared with the Guided Learning model in [21] that is only composed of two different convolution networks (CNN Blocks in (a) and (b)), we propose to add two additional bi-directional Gated Recurrent Unit (BGRU) layers after the CNN blocks to extract the temporal information of CNN representations for a better audio tagging. And different from both the GL and CRNN frameworks, besides the proposed deep feature distillation (module (c)), the event-specific post processing (module (d)) and the two-stage model training strategy with adaptive focal loss, we also divide the AED and AT tasks into two independent branches as shown in (a) and (b). The AED branch uses a fully connected layer with larger hidden states (the 'Linear' block) followed by ten small size separate fully connected layers (FC) with sigmoid activation to model the outputs of CNN blocks for detection. However, in the AT branch, the outputs of both the CNN blocks and BGRU are concatenated as the input features of 'Linear' layer, followed by an attention pooling module [20] with a consistency loss to perform the events classification. Furthermore, as shown in the block between student model and event-specific post processing, we apply an element-wise multiplication between the one-hot audio tagging prediction labels and the frame-level detection posteriors to ensure the consistency of AED and AT results.

B. Target-events based deep feature distillation (TFD)
Feature-level knowledge distillation (KD) is one popular technique that used in machine learning to improve a smaller student model given large teacher model guiding [25], [26].
Most related works choose to distill the intermediate information between teacher and student model using the whole feature maps directly, including the intra-utterance similarity preserving KD that proposed in [27] for audio tagging, the similarity matrix is also calculated on the whole feature maps.
In this study, our goal is to improve the target-events boundary detection performances in weakly supervised AED task, how to maximally exploit the frame-level alignment information of the available limited strong-labeled data is crucial. Instead of using the whole feature maps to perform the KD, we propose to force the distillation performed only on the frame-level deep feature vectors that belong to targetevents. The specific operations are illustrated in module (c) of Fig.1 and we define the distillation loss function as, where M ∈ R 1×T is the frame-level label mask matrix with j-th element M j ∈ [0, 1], and 1 means j-th frame is labeled as one type of target events in the limited strong-labeled dataset. A = [1, 1, ..., 1] T ×1 , T is the frame size, and · denote the element-wise and normal matrix multiplication respectively. D ∈ R 1×T is a frame-level similarity matrix, where D j , j = 1, 2, ..., T represents j-th frame Euclidean distance between the transformed deep feature vectors of CNN blocks outputs in teacher and student model, and it is defined as, where (F t ·W t ) j and (F s ·W s ) j is j-th column of transformed feature maps F t and F s respectively, W t and W s are their transformation matrix as shown in module (c) of Fig.1. By introducing the distillation loss L T F D into the total training loss of weakly supervised framework, we hope it can maximally leverage the information of limited strong-labeled data to regulate the teacher-student model training, especially for guiding the early training stages of T-S model, because all the information captured by the L T F D is derived from those target-events with golden-standard time-stamps.

C. Two-stage training with adaptive focal loss
Under the weakly supervised AED task like DCASE2019 Task 4, three types of dataset with different level of annotations are provided, i.e., a limited strong labeled set with time-stamps for target acoustic events, a small weakly annotated set with only the multiple events presence labels (without time-stamps), and a large amount of unlabeled training dataset. To make full use of all the available datasets, recent work in [21] proposed a Guided Learning (GL) strategy, in which the total connectional binary cross entropy (BCE) loss of the weakly supervised T-S training consists of two parts as follows: where L t,s weak = L t weak + L s weak , L t,s strong = L t strong + L s strong are the clip-level and frame-level supervised loss as shown in Fig.1 for the AT and AED respectively. The last two terms are the clip-level consistency losses performed on all types of training dataset. L t→s con denotes using teacher predictions to guide the student model training, while L s→t con denotes using the student to fine-tune the teacher model with a small weight α. During the earlier T-S training, only the first three terms are used, the last term is normally added when the teacher model becomes relatively stable. More details can be found in [21].
Motivated by the principle of focal loss in [28], here we aim to improve the GL training by combining the above BCE loss with an adaptive focal loss that defined as follows: where γ is a scaling factor to control the loss contribution of posterior probability p ij for i-th clip, j-th target-event category. K is the total size of audio clips with both weakly and strong labels in a minbatch, C is the number of targetevent categories. As illustrated in Fig.1 (f), the L af tends to give a higher weight to the difficult-to-classify (small p ij ) events than the easy-to-classify ones, and it is dynamically adapted in each training epoch. Moreover, to guarantee the basic ability of BCE loss for achieving a relatively stable T-S model, we find that the adaptive focal loss should be utilized in a curriculum learning [29] way. Therefore, we propose a two-stage training strategy to train the whole teacher-student model of Fig.1

as follows:
Stage 1, performing the normal GL training together with the proposed TFD in first s epochs using: where β ∈ [0, 1] is used to control the distillation depth. Stage 2, performing the fine-tuning epochs by replacing the L t,s weak , L t,s strong in L using their corresponding adaptive focal loss L t,s af −weak , L t,s af −strong respectively as, L s2 = L + β · L T F D L = L t,s af −weak + L t,s af −strong + L t→s con + α · L s→t con (6) By performing the adaptive focal loss together with twostage weakly supervised training, we hope that the class imbalance issue in training data, and various difficulty level of AED between multiple target events will be dynamically regulated and focused during model training. Because in Stage 1, the discriminations between those easy-to-classify acoustic events are well-learned with L s1 , and it results in relatively stable teacher and student models. As the training progresses in Stage 2, the adaptive focal loss can automatically downweight the contribution of easy events during training, and this makes the training rapidly focus the model on difficultto-classify events. By using the two-stage training, the L T F D , L af and the BCE losses are well integrated to boost the whole weakly supervised system performances.

D. Event-specific post processing (ESP)
Median filtering (MF) has proved effective in smoothing the noisy outputs of the student model for AED tasks [6]. However, DCASE2019 Task4 is a multiple target events detection instead of single one, the subsequent duration of each event in audio clip varies significantly. Conventional MF with fixed window size is no longer suitable for this task. Recent works in [21], [30] used group of median filters with adaptive window size by calculating the average duration of events with strong labels on the development set. However, each event duration is not an uniform distribution, using the average duration to optimize the MF window size may not the optimal. So we propose to use event-specific MF window size as: where W c , c = 1, 2, ..., C is the MF window size of class c, N c is the segment index for the inflection point of cumulative distribution of short-to-long sorted segments of c-class target event. L i is duration of i-th segment for event c. η is a scaling factor and set to 1/3 in our experiments. All strong labeled training clips are used to compute W c . As shown in Fig.1 (d), we perform the proposed ESP on both the frame-level posterior and one-hot predictions using the same W c .

A. Setups
Our experiments are performed on the dataset of DCASE 2019 Task4 Challenge [6]. It is a sound event detection task in domestic environments. The training set includes 1,473 weakly labeled, 13,390 unlabeled and 2,045 strongly labeled audio clips. 1,077 and 692 strongly labeled clips are taken as the development and evaluation clips respectively.
We extract 64 log mel-band magnitudes as features, each 10-second clip is transformed into 600 frames. Details of our T-S model structure and its parameters for each block is shown in Fig.1 and section II. We take both the Mean-Teacher (MT) [20] and Guided Learning [21] as baselines, because the MT was the official baseline, while GL was the best single system for DCASE2019 Task4 and our proposal is improving the GL system. α = 1 − λ e−s , β = e −5(1−x) 2 , where e is the current epoch and x ∈ [0, 1]. The γ, λ and s is set to 2, 0.996 and 30 respectively in the two-stage training. F 1-score [31] is used to measure the system performances. The AED uses an eventbased F 1 that computed with a 200ms collar on onsets and a 200ms/20% of the events length collar on offsets, and a segment-based F 1 on 1 s segment is taken as a secondary measure. The overall (macro-average) F 1-score at clip level is used to measure the AT performances.

B. Results and Discussion
All techniques are examined on DCASE 2019 Task4 evaluation set and results are shown in Table I. 'MT' and 'GL' are two baselines. 'PTS' is the backbone T-S model of our proposed framework without the proposed TFD, two-stage training with adaptive focal loss (AFL) and the ESP post processing techniques. Technical improvements of PTS over GL is presented in section II-A. 'DFD-MSE' and 'DFD-D' represent performing the deep feature KD on two transformed feature maps F t · W t , F s · W s using the conventional mean squared error (MSE) [27], and on the frame-level Euclidean distance similarity matrix D respectively. Both of them are performed on all the training data because they do not need any stronglabel information. 'DFD-D(w+u)' means only performing the 'DFD-D' on the weakly and unlabeled training clips. In Table I, we see that the last system with all the proposed techniques achieves the best results for both AED and AT tasks, it outperforms the GL system significantly by absolute 9.3%, 9.4% and 11.0% F 1-score in event-based, segmentbased AED and AT respectively. And the best single system GL in DCASE2019 Task4 challenge outperforms the official baseline MT significantly. By comparing system 2 with 1, the proposed backbone T-S network only achieves 0.4% and 1.6% AED and 3.2% AT F 1 improvements. However, when comparing system 5, 7 and 8 with system 2, continuous performance gains are obtained by adding the proposed TFD, AFL and ESP techniques into the backbone model. Moreover, to validate the proposed TFD, system 3, 4 are built for comparison, we see that the conventional deep feature distillations performed on the whole feature maps are not better than the target-event based one, even they can be trained using all the training data. By comparing system 4 with 3, we see that performing KD using frame-level Euclidean similarity is better than using MSE. Furthermore, we also try to combine the TFD and DFD-D(w+u) together, it's interesting to find that adding weakly and unlabeled data doesn't bring additional performance gain as we expected. Fig.2 and Table II show the detail examinations of the proposed event-specific post processing. From Fig.2, we see that there is a big duration gap for one event when it occurs in different clips. The average duration (green line) of each event deviates far from the ESP duration (black line) that computed according to the event-based cumulative distributions. We argue that the ESP duration is better to reflect the real distribution than the average one that used in most previous works [21], [30] to perform the MF post processing. The class-wise performances are in Table II. The 'Avg' represents using adaptive MF window size with average duration of each event class. 'ESP' is our event-specific post processing. 'AED/AT' is the event-based and class-wise F 1 for acoustic event detection and audio tagging. It's clear that the proposed ESP is very effective to improve the system performances for most target events, especially for those short-duration ones as the blender, cat and dishes. Compared with the 'Avg' based MF, the proposed ESP improves the overall AED/AT F 1-score from 47.1/78.0% to 49.8/81.2%.

IV. CONCLUSION
This letter proposes a new joint framework with two independent branches for weakly supervised AED and AT tasks. Based on the framework, a frame-level target-events based deep feature distillation, a two-stage training strategy with adaptive focal loss, and an event-specific post processing are further proposed to enhance the architecture and system training. Experiments on the DCASE2019 Task 4 challenge show that our proposed framework achieves competitive performances in both AT and AED tasks. Generalizing the proposals to other acoustic event detection tasks is our future work.