TECHNICAL FIELD

This disclosure relates to contextual biasing for speech recognition.

BACKGROUND

Recognizing the context of speech is the goal of automatic speech recognition (ASR) systems. The ability to recognize context in speech, however, is challenging given the wide variety of words that people may speak and the many variations in accents and pronunciation. In many cases, the types of words and phrases that a person speaks varies depending on the context the person finds himself or herself in.

Contextual automated speech recognition (ASR) involves biasing speech recognition towards a given context, such as towards a user's own playlist, contacts, or geographic place names. Context information usually includes a list of relevant phrases to be recognized, which often includes rare phrases or even foreign words which are seen infrequently in training. To perform contextual biasing, conventional ASR systems sometimes model contextual information in an independent contextual language model (LM), using an n-gram weighted finite state transducer (WFST), and compose the independent contextual LM with a baseline LM for on-the-fly (OTF) rescoring.

Recently, end-to-end (E2E) models have shown great promise for ASR, exhibiting improved word error rates (WERs) and latency metrics as compared to conventional on-device models. These E2E models, which fold the acoustic model (AM), pronunciation model (PM), and LMs into a single network to directly learn speech-to-text mapping, have shown competitive results compared to conventional ASR systems which have a separate AM, PM, and LMs. Representative E2E models include word-based connectionist temporal classification (CTC) models, recurrent neural network transducer (RNN-T) models, and attention-based models such as Listen, Attend, and Spell (LAS). Because E2E models maintain a limited number of recognition candidates during beam-search decoding, contextual ASR can be challenging for E2E models.

SUMMARY

Implementations herein are directed toward biasing a speech recognition model toward a set of phrases relevant to a current context. The set of phrases are dynamic in that the phrases may change as the context changes. The speech recognition model may further take into account the pronunciations of the phrases used for biasing in addition to written representations of the phrases. For example, the model can use text information (e.g., grapheme sequences) and pronunciation information (e.g., phoneme sequences) representing biasing phrases to select which phrases are most applicable to speech being recognized. This allows the contextual biasing aspect of the speech model to be aware of the pronunciations of individual biasing phrases, thereby enabling the model to achieve much higher accuracy when recognizing rare words and words with unusual pronunciations. The contextual biasing techniques discussed herein can be used with various types of speech recognition models, including end-to-end models that can generate transcription data without a separate acoustic model, pronunciation model, and language model.

To provide high speech recognition accuracy, general purpose automatic speech recognition (ASR) systems may use information indicating the context in which speech occurs. The context may be related to, for example, the user's contacts, calendar appointments, open apps, and location. One example of an ASR system contains separate acoustic, pronunciation, and language models. Other ASR systems combine the acoustic, pronunciation, and language models as a single neural network. A single neural network model improves simplicity and quality, and optimizes word error rate (WER).

For conventional neural network-based ASR models, it is challenging to recognize words that infrequently appear in a language. The distribution of words in a language typically follows a Zipfian distribution, where a small number of words are used very frequently, and vast numbers of words are rarely used. It can be difficult to obtain a large enough set of training data with audio and corresponding text to effectively provide examples of rare words. Even if sufficient training data is available, adding more and more training examples often yields improvements of lower and lower magnitude. Additionally, it is challenging for these models to recognize words with unusual pronunciations relative to their spelling.

Incorporating contextual biasing into a neural network-based ASR model can improve recognition for rare words and words with unusual pronunciations. One useful application is to better recognize proper names (i.e., proper nouns such as people names, song names, city names, etc.), which may be relatively rare in training data and/or may not follow typical pronunciation rules. For example, a smart phone or other user device often stores a user's contacts. When a user is using a messaging or phone calling application, this information can be used as context to help the ASR system recognize names spoken by a user. In a similar manner, a list of songs in a music library of a user can be used to bias speech recognition, for example, when the user is using a media player application. Implementations herein are directed toward applying contextual biasing to an ASR model by injecting information associated with both written forms and pronunciations of biasing phrases into the ASR model. Contextually biasing the ASR model incorporates knowledge of rare word pronunciations even if the words were not observed during training of the ASR model.

One aspect of the disclosure provides a method for biasing speech recognition that includes receiving, at data processing hardware, audio data encoding an utterance, and obtaining, by the data processing hardware, a set of one or more biasing phrases corresponding to a context of the utterance, each biasing phrase in the set of one or more biasing phrases includes one or more words. The method also includes processing, by the data processing hardware, using a speech recognition model, acoustic features derived from the audio data and grapheme and phoneme data derived from the set of one or more biasing phrases to generate an output of the speech recognition model. The method also includes determining, by the data processing hardware, a transcription for the utterance based on the output of the speech recognition model.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the speech recognition model includes a first encoder, a first attention module, a grapheme encoder, a phoneme encoder, a second attention module, and a decoder. The first encoder is configured to receive, as input, the acoustic features and generate, as output, audio encodings from the acoustic features. The first attention module is configured to receive, as input, the audio encodings output from the first encoder and generate, as output, first attention outputs. The grapheme encoder is configured to receive, as input, grapheme data indicating graphemes of each word in the set of one or more biasing phrases and generate, as output, grapheme encodings. The phoneme encoder is configured to receive, as input, phoneme data indicating phonemes of each word in the set of one or more biasing phrases and generate, as output, phoneme encodings. The second attention module is configured to receive, as input, a representation of the grapheme encodings output from the grapheme encoder and the phoneme encodings output from the phoneme encoder and generate, as output, second attention outputs. The decoder is configured to determine likelihoods of sequences of speech elements based on the first attention outputs and the second attention outputs.

In some examples, for each particular word of each biasing phrase in the set of one or more biasing phrases: the grapheme encoder is configured to generate a corresponding grapheme encoding for the particular word; the phoneme encoder is configured to generate a corresponding phoneme encoding for the particular word; and the second attention module is configured to encode a corresponding second attention output that comprises a corresponding contextual biasing vector for the particular word based on the corresponding grapheme and phoneme encodings for the particular word. In addition examples, the representation of the grapheme encodings output from the grapheme encoder and the phoneme encodings output from the phoneme encoder includes a projection vector representing a concatenation between the grapheme encodings and the phoneme encodings. Additionally, the first encoder, the first attention module, the grapheme encoder, the phoneme encoder, the second attention module, and the decoder may be trained jointly to predict a sequence of graphemes from a sequence of acoustic feature frames.

In some implementations, the first attention module is configured to compute attention as a function of a previous hidden state of the decoder and a full sequence of audio encodings output by the first encoder, while the second attention module is configured to compute attention as a function of the previous hidden states of the decoder and a full sequence of projection vectors representing characteristics of both the grapheme and phoneme data derived from the biasing phrases in the set of one or more biasing phrases. In additional implementations, the second attention module is configured to receive a contextual biasing vector that does not correspond to any of the biasing phrases in the set of one or more biasing phrases, the contextual biasing vector representing an option to not bias the output of the speech recognition model. In some examples, the speech elements include graphemes. In other examples, the speech elements include words or wordpieces. Optionally, the speech elements may include phonemes.

The set of one or more biasing phrases may include one or more contact names personalized for a particular user. Additionally or alternatively, the set of one or more biasing phrases may include one or more calendar events personalized for the particular user. In some examples, the method also includes determining, by the data processing hardware, the context of the utterance based on at least one of: a location of a user that spoke the utterance; one or more applications open on a user device associated with the user that spoke the utterance; or a current date and/or time of the utterance. The context of the utterance may additionally or alternatively be based on one or more previous commands issued by the user.

In some implementations, the speech recognition model includes a decoder configured to determine a hidden state and the output of the speech recognition model based on an embedding vector, a previous hidden state of the decoder; a first vector; and a second vector. Here, the embedding vector is for a previous grapheme output by the speech recognition model, while the first vector is output by a first attention module and the second vector is output by a second attention module.

Another aspect of the disclosure provides a system for biasing speech recognition based on a context of an utterance. The system includes data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving audio data encoding an utterance and obtaining a set of one or more biasing phrases corresponding to a context of the utterance, each biasing phrase in the set of one or more biasing phrases includes one or more words. The operations also include processing, using a speech recognition model, acoustic features derived from the audio data and grapheme and phoneme data derived from the set of one or more biasing phrases to generate an output of the speech recognition model. The operations also include determining a transcription for the utterance based on the output of the speech recognition model.

This aspect may include one or more of the following optional features. In some implementations, the speech recognition model includes a first encoder, a first attention module, a grapheme encoder, a phoneme encoder, a second attention module, and a decoder. The first encoder is configured to receive, as input, the acoustic features and generate, as output, audio encodings from the acoustic features. The first attention module is configured to receive, as input, the audio encodings output from the first encoder and generate, as output, first attention outputs. The grapheme encoder is configured to receive, as input, grapheme data indicating graphemes of each word in the set of one or more biasing phrases and generate, as output, grapheme encodings. The phoneme encoder is configured to receive, as input, phoneme data indicating phonemes of each word in the set of one or more biasing phrases and generate, as output, phoneme encodings. The second attention module is configured to receive, as input, a representation of the grapheme encodings output from the grapheme encoder and the phoneme encodings output from the phoneme encoder and generate, as output, second attention outputs. The decoder is configured to determine likelihoods of sequences of speech elements based on the first attention outputs and the second attention outputs.

In some examples, for each particular word of each biasing phrase in the set of one or more biasing phrases: the grapheme encoder is configured to generate a corresponding grapheme encoding for the particular word; the phoneme encoder is configured to generate a corresponding phoneme encoding for the particular word; and the second attention module is configured to encode a corresponding second attention output that comprises a corresponding contextual biasing vector for the particular word based on the corresponding grapheme and phoneme encodings for the particular word. In addition examples, the representation of the grapheme encodings output from the grapheme encoder and the phoneme encodings output from the phoneme encoder includes a projection vector representing a concatenation between the grapheme encodings and the phoneme encodings. Additionally, the first encoder, the first attention module, the grapheme encoder, the phoneme encoder, the second attention module, and the decoder may be trained jointly to predict a sequence of graphemes from a sequence of acoustic feature frames.

In some implementations, the first attention module is configured to compute attention as a function of a previous hidden state of the decoder and a full sequence of audio encodings output by the first encoder, while the second attention module is configured to compute attention as a function of the previous hidden states of the decoder and a full sequence of projection vectors representing characteristics of both the grapheme and phoneme data derived from the biasing phrases in the set of one or more biasing phrases. In additional implementations, the second attention module is configured to receive a contextual biasing vector that does not correspond to any of the biasing phrases in the set of one or more biasing phrases, the contextual biasing vector representing an option to not bias the output of the speech recognition model. In some examples, the speech elements include graphemes. In other examples, the speech elements include words or wordpieces. Optionally, the speech elements may include phonemes.

The set of one or more biasing phrases may include one or more contact names personalized for a particular user. Additionally or alternatively, the set of one or more biasing phrases may include one or more calendar events personalized for the particular user. In some examples, the operations also include determining the context of the utterance based on at least one of: a location of a user that spoke the utterance; one or more applications open on a user device associated with the user that spoke the utterance; or a current date and/or time of the utterance. The context of the utterance may additionally or alternatively be based on one or more previous commands issued by the user.

In some implementations, the speech recognition model includes a decoder configured to determine a hidden state and the output of the speech recognition model based on an embedding vector, a previous hidden state of the decoder; a first vector; and a second vector. Here, the embedding vector is for a previous grapheme output by the speech recognition model, while the first vector is output by a first attention module and the second vector is output by a second attention module.

The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

