Masked cross self-attention encoding for deep speaker embedding

In general, speaker verification tasks require the extraction of speaker embedding from a deep neural network. As speaker embedding may contain additional information such as noise besides speaker information, its variability controlling is needed. Our previous model have used multiple pooling based on shortcut connections to amplify speaker information by deepening the dimension; however, the problem of variability remains. In this paper, we propose a masked cross self-attention encoding (MCSAE) for deep speaker embedding. This method controls the variability of speaker embedding by focusing on each masked output of multiple pooling on each other. The output of the MCSAE is used to construct the deep speaker embedding. Experimental results on VoxCeleb data set demonstrate that the proposed approach improves performance as compared with previous state-of-the-art models.


INTRODUCTION
Speech recognition (SR) aims to identify linguistic information from input speech; in contrast, speaker recognition's objective is to identify speaker information from input speech. Speaker verification (SV) is a problem for determining whether the enrolled speaker's speech is true or false for the test speaker's speech.
Traditionally, the Gaussian mixture model and the universal background model (GMM-UBM) have been used to model various speakers [1]. GMM-UBM models the enrolled speaker via maximum a posteriori adaptation (MAP). The supervector extracted from the support vector machine (GMM-SVM) based on a Naive Bayes classifier has been used as a speaker feature in the GMM-based model [2]. Based on eigenvalue decomposition, a joint factor analysis (JFA) method has also been proposed to separate the supervector from the channel and speaker factors [3].
However, all the previous methods required an enormous amount of data for an enrolled speaker, and the disadvantage of the GMM-based supervector is rapidly increasing its dimensions depending on the Gaussian mixture. To solve this issue, an i-vector capable of processing speech data in each utterance has become one of the most efficient feature extraction methods in traditional SV along with the probabilistic linear discriminant analysis (PLDA) [4,5].
Since the arrival of deep learning, supervectors are extracted directly from the pre-trained deep neural network (DNN). Feature-level speech data are trained on a DNN based speaker classifier, and the activation value of the last hidden layer of the model has been used as utterance-level speaker representations called d-vector [6]. Additionally, speaker embedding encoding methods using various DNN based models have been proposed. In the time delayed neural network-based model (TDNN), x-vector which uses statistical pooling and encodes fixed dimensional statistics vectors while controlling the variable-length of input data has been proposed [7]. Among the convolutional neural networkbased models (CNN), ResNet [8], which stands out in the image field, has been used for speaker embedding [9][10][11][12][13][14].
In this paper, we propose a masked cross self-attention encoding (MCSAE) for deep speaker embedding. It is based on a deep speaker embedding encoding by multiple pooling using shortcut connections [14]. The proposed methods aim to encode the speaker embedding maximizing the speaker information while controlling the speaker variability. The main idea of proposed methods is as follows: 1) The new encoding method is to emphasize speaker information by using self-attention mechanism on each output of multiple pooing based on shortcut connection. 2) By applying random masking, the variability of speaker embedding is minimized.
3) Deep speaker embedding is generated using the attention matrix as a result of MCSAE.
We will introduce baseline model and prior attention mechanism studies in Section 2. In Section 3, we describe the proposed MCSAE method for deep speaker embedding and show the results in Section 4. Finally, the conclusions are drawn in Section 5.

Self-attention Mechanism
The principle of the attention mechanism is to focus on specific information. In machine translation, self-attention with scaled dot-product attention and multi-head attention has been proposed [16]. The formula for the scaled dotproduct attention is: The input comprises query ( ), key ( ), value ( ). To train the relationship between and , scaling is applied to compute similarity using dot product operations on all and elements, and each of them is divided by √ ( is dimension of . Next, after applying the softmax for normalization, the weights for are obtained. That is, the more similar is to , the higher its value, and thus, more attention will be paid to . Based on this process, multi-heads are determined by performing several linear projections on each input. In SV, self-attentive pooling, (SAP) which is applied to xvector and ResNet based models, outperforms conventional temporal average pooling (TAP) or global average pooling (GAP) [9,24]. In speaker embedding encoding, each input sequence generated a different weighted utterance level representation using SAP. In addition, the performance of SV is improved by constructing an self multi-head attention pooling (MHA), which is a way to split the encoded representations of the input sequence [26].

Cross Attention and Masking Methods
In image-text matching, stacked cross attention has been proposed in an attempts to identify the appropriate text that appears in an input image [18]. By constructing the input as an image-text and a text-image, cross attention is applied to each of the two pairs to obtain, in order that more accurate weights than one attention does.
In person re-identification, masking is applied to the input to solve the problem of dissimilarity neglected between the source and the target [19]. By constructing a masking matrix of 1 or -1 according to the label, an attention mechanism for person identity is constructed.

Overview
The proposed model builds on our previous research, the deep speaker embedding encoding method based on multiple pooling using shortcut connections [14]. The modified ResNet model using shortcut connection (SC-ResNet) is trained for speaker classification in an end-to-end manner.
The output bottleneck features of the GAP in each residual block are combined to create a utterance-level speaker representation.
However, as the other information is also trained in the speaker embedding encoding process, it is necessary to control the variability of speaker embedding. Therefore, the proposed MCSAE uses a masking technique to minimize speaker variability and a self-attention based attention mechanism to emphasize speaker information.

Model Architecture
The proposed model architecture uses the SC-ResNet-18 model [14] as a baseline and adds an attention block for MCSAE after each pooling layer, as shown in Fig.1 and Table  1. Like the standard ResNet-18, this model has total 8 residual blocks. Each residual block consists of a convolution layer, batch normalization (BN), and ReLU activation functions, with the except the conv-1 layer. Between each conv layer, a pooling layer with additional shortcut connections for bottleneck feature extraction is used. In the training process, each output of the residual blocks encoding the speaker embedding in order, include the low representation information to the high representation information. The output embeddings ( , +1 ) of the two previous pooling layers are used as inputs to the ℎ attention block. As shown below, , which refers to each segment matrix for the attention matrix , is generated by applying the masking method and cross self-attention method to the attention block: Here, , the output of each attention block, is used for constructing an attention matrix of 1 × 512 size using matrix product calculation in a matmul layer as follow: To match the dimension, an embedding 1 of 1 × 64 size extracted from the pooling-1 layer is used for the matrix product. Using the 1 matrix allows dimensional matching without increasing the parameters.
In the concat layer, embedding 5 of 1 × 512 size extracted from pooling-5 layer is concatenated with attention matrix . As a result, embedding of 1 × 1024 size is finally constructed after normalization. The embedding 5 is standard embedding in ResNet without the attention mechanism. The generated embedding is encoded into fully-

Attention Blocks (MCSAE)
Spk. Embedding connected layers (fc layer) to the output layer representing the speaker classes as: Through this process, a 1024-dimensional deep speaker embedding is extracted upon applying BN and ReLU from the last hidden layer of the trained model (fc-3 layer).

Cross Self-attention
Attention blocks employ two main proposed methods: 1) cross self-attention 2) random masking. They aim to encode the segment matrix 1 which constructs the attention matrix . The proposed attention method is based on the scaled dotproduct attention used in the self-attention mechanism [16].
We assume that the embedding is a step preceding +1 , and that these are closely related to each other, which is emphasized further by the attention mechanism. That is, because contains feature-level information when compared to +1 , and +1 contains utterance-level information when compared to , these can complement each other.
As depicted in Fig. 2, the attention block consists of two input pairs to perform cross attention. The first attention input consists of (query vector, ), +1 (key vector, ) and +1 (value vector, ). After the scaled dot-product operation between and , self-attention is performed between and the target as: In the dot-product operation between and , the random masking is applied to . Next, scaling to the value of √ ( is dimension of is performed and normalization is applied using the softmax function. The computed matrix is multiplied by , self-attention is finally conducted. To match the dimension, transposed and are applied. The second attention input consists of +1 ( ), ( ) and ( ). As is the attention target, the scaled dot attention mechanism is performed as done earlier. The matrix is encoded using matrix multiplication for the output of masked cross self-attention as:

Random Masking
A random masking is applied as depicted in Fig. 3, because it is not known as which speaker or other information is located at the point of input. In other words, this method aims to minimize the variability of speaker embedding.

Fig. 3. Proposed random masking process
The masking map consists of 0 or 1 randomly, according to the value of the adaptive scaling vector, which determines the range of masking updated by training. As the value increases, the range of masking widens. Next, masking is performed to input feature map or +1 and element-wise multiplication. The masked value is filled with 0. When masking is applied to both inputs feature map and +1 , degradation problems are caused during training. Therefore, cross attention is performed by applying masking only to the one side.

Data Set
In this paper, we trained the proposed model using the VoxCeleb1 data set [27]. These are large scale textindependent speaker verification data sets collected from YouTube. The VoxCeleb1 training data set contains over 100,000 utterances from 1,211 celebrities. We evaluated all the experiments by the VoxCeleb1 test data set of 40 speakers and 37,220 pairs of official test protocol.

Experiment Setup
The input feature vectors are 64-dimensional log melfilterbank energies of 25ms width and 10ms shift size, which are mean-normalized over a sliding window of up to 3s. Preprocessing methods such as voice activity detection or data augmentation are not applied. For each training step, an integer T within [300,800] interval is randomly extracted from each utterance and is applied using cropping or padding [9]. Therefore, we used an input size of 64 × . For training parameters, we used standard the stochastic gradient descent optimizer with a momentum of 0.9, weight decay of 0.0001 at an initial learning rate of 0.1 reduced by 0.1 decay factor on plateau. The 64 mini-batch size is used, and early stopping in 100 epochs is performed. The initial adaptive scaling vector is 0.5 in masking of MCSAE. Our proposed model is trained on end-to-end manners without usage of any additional methods after extracting the speaker embedding such as [14]. From the trained model, we extracted a 1024-dimensional deep speaker embedding and evaluated it using cosine similarity metrics and equal error rate (EER) performance.

Experiment Results
We compared our proposed model to the state-of-the-art models employing the VoxCeleb1 test data set, as shown in Table 2. We focused on EER (%) performance of speaker embedding encoding methods with softmax loss. Except TDNN based model with attentive statistics pooling (ASP) [25], previous models were CNN based model. They proposed to various approaches using the pooling methods such as TAP/GAP [9,12], learnable dictionary encoding (LDE), SAP [9], spatial pyramid encoding (SPE) [13]. Also, they applied to orthogonal vector pooling (OVP) [12], SAP applied MHA [26] based on attention mechanism. SC-ResNet-18 is our baseline model not applying MCSAE [14]. Experimental results show that the performance is improved (EER of 2.07%) compared to the baseline model. Also, SC-ResNet-34-MCSAE model achieved our highest performance (EER of 1.84%).
We also tested to investigate the impact of each module on the MCSAE as shown in Table 3. We performed the experiments under the conditions that each modules applied as: 1) w/ self-attention, 2) w/ cross self-attention, 3) w/ random masking. Experimental results demonstrate that performance improved when all the proposed modules are used together without parameter size issues.

CONCLUSIONS
Controlling the variability of the speaker embedding is important for speaker verification tasks. In this paper, we propose masked cross self-attention encoding (MCSAE) for deep speaker embedding. The cross self-attention mechanism improves the concentration of speaker information in the output of multiple pooling. The random masking method also minimizes the other information during encoding. Experimental results upon employing the VoxCeleb data set demonstrate that proposed model improves EER performance compared to the previous state-of-the-art models.