End-to-end automatic speech recognition (ASR) commonly transcribes audio signals into sequences of characters while its performance is evaluated by measuring the word-error rate (WER). This suggests that predicting sequences of words directly may be helpful instead. However, training with word-level supervision can be more difficult due to the sparsity of examples per label class. In this paper we analyze an end-to-end ASR model that combines a word-and-character representation in a multi-task learning (MTL) framework. We show that it improves on the WER and study how the word-level model can benefit from character-level supervision by analyzing the learned inductive preference bias of each model component empirically. We find that by adding character-level supervision, the MTL model interpolates between recognizing more frequent words (preferred by the word-level model) and shorter words (preferred by the character-level model). End-to-end automatic speech recognition (ASR) allows for learning a direct mapping from audio signals to character outputs. Usually, a language model re-scores the predicted transcripts during inference to correct spelling mistakes BID16 . If we map the audio input directly to words, we can use a simpler decoding mechanism and reduce the prediction time. Unfortunately, word-level models can only be trained on known words. Out-of-vocabulary (OOV) words have to be mapped to an unknown token. Furthermore, decomposing transcripts into sequences of words decreases the available number of examples per label class. These shortcomings make it difficult to train on the word-level BID2 .Recent works have shown that multi-task learning (MTL) BID8 on the word-and character-level can improve the word-error rate (WER) of common end-to-end speech recognition architectures BID2 BID3 BID18 BID21 BID22 BID24 BID29 . MTL can be interpreted as learning an inductive bias with favorable generalization properties BID6 . In this work we aim at characterizing the nature of this inductive bias in word-character-level MTL models by analyzing the distribution of words that they recognize. Thereby , we seek to shed light on the learning process and possibly inform the design of better models. We will focus on connectionist temporal classification (CTC) BID15 . However , the analysis can also prove beneficial to other modeling paradigms, such as RNN Transducers BID14 or Encoder-Decoder models, e.g., BID5 BID9 .Contributions . We show that , contrary to earlier negative results BID2 BID27 , it is in fact possible to train a word-level model from scratch on a relatively small dataset and that its performance can be further improved by adding character-level supervision. Through an empirical analysis we show that the resulting MTL model combines the preference biases of word-and character-level models. We hypothesize that this can partially explain why word-character MTL improves on only using a single decomposition, such as phonemes, characters or words.Several works have explored using words instead of characters or phonemes as outputs of the end-toend ASR model BID2 BID27 . Soltau et al. BID27 found that in order to solve the problem of observing only few labels per word, they needed to use a large dataset of 120, 000 hours to train a word-level model directly. Accordingly, Audhkhasi et al. BID2 reported difficulty to train a model on words from scratch and instead fine-tuned a pre-trained character-level model after replacing the last dense layer with a word embedding.MTL enables a straightforward joint training procedure to integrate transcript information on multiple levels of granularity. Treating word-and character-level transcription as two distinct tasks allows for combining their losses in a parallel BID21 BID22 BID28 BID29 or hierarchical structure BID13 BID20 BID24 . Augmenting the commonly-used CTC loss with an attention mechanism can help with aligning the predictions on both character-and word-level BID3 BID12 BID22 . All these MTL methods improve a standard CTC baseline.Finding the right granularity of the word decomposition is in itself a difficult problem. While Li et al. BID22 used different fixed decompositions of words, sub-words and characters, it is also possible to optimize over alignments and decompositions jointly BID23 . Orthogonal to these works different authors have explored how to minimize WER directly by computing approximate gradients BID25 BID32 .When and why does MTL work? Earlier theoretical work argued that the auxiliary task provides a favorable inductive bias to the main task BID6 . Within natural language processing on text several works verified empirically that this inductive bias is favorable if there is a certain notion of relatedness between the tasks BID4 BID7 BID26 . Here, we investigate how to characterize the inductive bias learned via MTL for speech recognition. In contrast to earlier studies in the literature, we found that, even on a relatively small dataset, training on a word-level can be feasible. Furthermore, we found that combining a word-level model with character-level supervision in MTL can improve results noticeably. To gain a better understanding of this, we characterized the inductive bias of word-character MTL in ASR by comparing the distributions of recognized words at the beginning of training. We found that adding character-level supervision to a word-level interpolates between recognizing more frequent words (preferred by the word-level model) and shorter words (preferred by the character-level model). This effect could be even more pronounced on harder datasets than WSJ, such as medical communication data where many long words are infrequent, but very important. Further analysis of word distributions in terms of pitch, noise and acoustic variability could provide additional insight.
Multi-task learning improves word-and-character-level speech recognition by interpolating the preference biases of its components: frequency- and word length-preference.
{"src": "scitldr-AIC_train", "paper_id": "B1GySqOojm"}