BACKGROUND

1. Field

The present disclosure relates to methods and apparatuses for performing sequence to sequence (Seq2Seq) speech recognition, particularly, multi-task training for attention based Seq2Seq speech recognition.

2. Description of the Related Art

In related art speech recognition systems, acoustic signal and lexicons are modeled independently with two separate models such as an acoustic model and a language model, which are then combined only during decoding stage of training. The related art training, using the acoustic model, involves expansive expert knowledge and manual work. This process is simplified by an attention based Seq2Seq speech recognition model, which models acoustic and lexicon information simultaneously. However, such related art speech recognition systems fall behind the state-of-art hybrid systems in terms of recognition accuracy. Hybrid speech recognition systems are similar to the related art speech recognition systems, but with a neural network component in the acoustic model part.

Furthermore, related art end to end speech recognition systems include two major branches: CTC model and Seq2Seq model. The CTC model focuses on the frame classification and the lexicon dependency is modeled indirectly from CTC loss. The Seq2Seq model focuses more on lexicon dependencies and generates only a soft alignment on acoustic signals. These two models have been improved independent of each other until the recent work in NPL document 1, in which, these two models are combined to produce better recognition performance on a small data set.

However, the related art solution for combining CTC and Seq2Seq model is the simple interpolation of loss functions, in which, the interpolation weight has to be decided manually with several model building/testing cycles. For instance, in the related art speech recognition system, an interpolation weight w is given to combine the CTC loss (ctcL) and the original cross-entropy loss used by the attention model (attL) as follows: L=w*ctcL+(1−w)*attL. Here, the related art training tries to minimize the loss at every mini-step with predefined fixed weights. As such, usually several models with different weights are trained and tested to decide which weight should be selected. This process is not only cumbersome, but takes a long time and is inefficient, since this process requires individually training and testing the models with various different weights.

NPL DOCUMENTS

NPL document 1: S. Kim, T. Hori, and S. Watanabe, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” in https://arxiv.org/abs/1609.06773, 2017.

SUMMARY

Provided are methods and apparatuses that improve the related art end to end recognition system by automatically and independently balancing the importance of two loss functions. Moreover, the methods and apparatuses of the present disclosure transform the encoder hidden layers before they are used by the attention model training, which significantly improves the attention model training quality and leads to much better performance improvement. Also, the time required for training and modeling is decreased because only one model needs to be trained as compared to the many different models with different weights (ws) in the related art.

According to an aspect of the present disclosure, there is provided a method of performing sequence to sequence (Seq2Seq) speech recognition training by at least one processor, the Seq2Seq speech recognition training method comprising: acquiring, by the at least one processor, a training set comprising a plurality of pairs of input data and target data corresponding to the input data; encoding, by an encoder implemented by the at least one processor, the input data into a sequence of hidden states; performing, by the at least one processor, a connectionist temporal classification (CTC) model training based on the sequence of hidden states; performing, by the at least one processor, an attention model training based on the sequence of hidden states; and decoding, by a decoder implemented by the at least one processor, the sequence of hidden states to generate target labels by independently performing the CTC model training and the attention model training.

The Seq2Seq speech recognition training method may further comprise: additionally transforming the sequence of hidden states using additional layers to enable content match between query and context; and performing the attention model training based on the additionally transformed sequence of hidden states.

The Seq2Seq speech recognition training method may further comprise: performing the CTC model training based on a CTC loss function.

The Seq2Seq speech recognition training method may further comprise: performing the attention model training based on a cross entropy loss function.

The independently performing the CTC model training and the attention model training may comprise: performing the CTC model training to minimize CTC loss during a first time period; and performing the attention model training to minimize cross entropy loss during a second period different from the first period.

The independently performing the CTC model training and the attention model training may comprise: performing a mini-batch based alternate training in which one of the CTC model training and the attention model training is randomly selected for optimization in each mini-batch.

The decoding by the decoder may comprise: generating a query information based on a previous target label and a previous prediction; generating a context information by calculating a soft alignment over all steps of the additionally transformed sequence of hidden states based on the query; and generating a target label based on the query information and the context information.

The context information may be a summary of speech signals encoded in hidden layers of the encoder.

The context information may be generated using scalar energy computed based on content similarity between the additionally transformed sequence of hidden states at each time step and the query information.

The CTC loss function may be defined as a mean of normalized edit distance between hypothesis H(x) and the corresponding targets,

Loss⁡(H,S)=1S⁢∑x,t∈S⁢editDistance⁡(H⁡(x),t)t,
where S=(x, t) is the training set containing all pairs of input x and its corresponding target t.

According to another exemplary embodiment, there is provided a sequence to sequence (Seq2Seq) speech recognition training apparatus comprising: at least one memory operable to store program code; and at least one processor operable to read said program code and operate as instructed by said program code to: acquire a training set comprising a plurality of pairs of input data and target data corresponding to the input data; encode the input data into a sequence of hidden states; perform a connectionist temporal classification (CTC) model training based on the sequence of hidden states; perform an attention model training based on the sequence of hidden states; and decode the sequence of hidden states to generate target labels by independently performing the CTC model training and the attention model training.

The at least one processor may be further configured to: additionally transform the sequence of hidden states using additional layers to enable content match between query and context; and perform the attention model training based on the additionally transformed sequence of hidden states.

The at least one processor may be further configured to: perform the CTC model training based on a CTC loss function.

The at least one processor may be further configured to: perform the attention model training based on a cross entropy loss function.

The independently performing the CTC model training and the attention model training may comprise: performing the CTC model training to minimize CTC loss during a first time period; and performing the attention model training to minimize cross entropy loss during a second period different from the first period.

The independently performing the CTC model training and the attention model training may comprise: performing a mini-batch based alternate training in which one of the CTC model training and the attention model training is randomly selected for optimization in each mini-batch.

The decoding the sequence of hidden states to generate the target label may comprise: generating a query information based on a previous target label and a previous prediction; generating a context information by calculating a soft alignment over all steps of the additionally transformed sequence of hidden states based on the query; and generating a target label based on the query information and the context information.

The context information may be a summary of speech signals encoded in hidden layers of the encoder.

The context information may be generated using scalar energy computed based on content similarity between the additionally transformed sequence of hidden states at each time step and the query information.

The CTC loss function may be defined as a mean of normalized edit distance between hypothesis H(x) and the corresponding targets,

Loss⁡(H,S)=1S⁢∑x,t∈S⁢editDistance⁡(H⁡(x),t)t,
where S=(x, t) is the training set containing all pairs of input x and its corresponding target t.

According to another embodiment, there is provided a non-transitory computer readable medium having stored thereon a program for performing sequence to sequence (Seq2Seq) speech recognition training method, the Seq2Seq speech recognition training method comprising: acquiring, by at least one processor, a training set comprising a plurality of pairs of input data and target data corresponding to the input data; encoding, by an encoder implemented by the at least one processor, the input data into a sequence of hidden states; performing, by the at least one processor, a connectionist temporal classification (CTC) model training based on the sequence of hidden states; performing, by the at least one processor, an attention model training based on the sequence of hidden states; and decoding, by a decoder implemented by the at least one processor, the sequence of hidden states to generate target labels by independently performing the CTC model training and the attention model training.

