TECHNICAL FIELD

This disclosure relates to a clockwork hierarchal variational encoder for predicting prosody.

BACKGROUND

Speech synthesis systems use text-to-speech (TTS) models to generate speech from textual input. The generated/synthesized speech should accurately convey the message (intelligibility) while sounding like human speech (naturalness) with an intended prosody (expressiveness). While traditional concatenative and parametric synthesis models are capable of providing intelligible speech and recent advances in neural modeling of speech have significantly improved the naturalness of synthesized speech, most existing TTS models are ineffective at modeling prosody, thereby causing synthesized speech used by important applications to lack expressiveness. For instance, it is desirable for applications such as conversational assistants and long-form readers to produce realistic speech by imputing prosody features not conveyed in textual input, such as intonation, stress, and rhythm and style. For example, a simple statement can be spoken in many different ways depending on whether the statement is a question, an answer to a question, there is uncertainty in the statement, or to convey any other meaning about the environment or context which is unspecified by the input text.

SUMMARY

One aspect of the disclosure provides a method of representing an intended prosody in synthesized speech. The method includes receiving, at data processing hardware, a text utterance having at least one word, and selecting, by the data processing hardware, an utterance embedding for the text utterance. Each word in the text utterance has at least one syllable and each syllable has at least one phoneme. The utterance embedding represents an intended prosody. For each syllable, using the selected utterance embedding, the method also includes: predicting, by the data processing hardware, a duration of the syllable by encoding linguistic features of each phoneme of the syllable with a corresponding prosodic syllable embedding for the syllable; predicting, by the data processing hardware, a pitch contour of the syllable based on the predicted duration for the syllable; and generating, by the data processing hardware, a plurality of fixed-length predicted pitch frames based on the predicted duration for the syllable. Each fixed-length predicted pitch frame represents part of the predicted pitch contour of the syllable.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, a network representing a hierarchical linguistic structure of the text utterance includes a first level including each syllable of the text utterance, a second level including each phoneme of the text utterance, and a third level including each fixed-length predicted pitch frame for each syllable of the text utterance. In these implementations, the first level of the network may include a long short-term memory (LSTM) processing cell representing each syllable of the text utterance, the second level of the network may include a LSTM processing cell representing each phoneme of the text utterance, and the third level of the network may include a LSTM processing cell representing each fixed-length predicted pitch frame. Here, the LSTM processing cells of the second level clock relative to and faster than the LSTM processing cells of the first level, while the LSTM processing cells of the third level clock relative to and faster than the LSTM processing cells of the second level.

In some examples, predicting the duration of the syllable includes: for each phoneme associated with the syllable, predicting a duration of the corresponding phoneme by encoding the linguistic features of the corresponding phoneme with the corresponding prosodic syllable embedding for the syllable; and determining the duration of the syllable by summing the predicted durations for each phoneme associated with the syllable. In these examples, predicting the pitch contour of the syllable based on the predicted duration for the syllable may include combining the corresponding prosodic syllable embedding for the syllable with each encoding of the corresponding prosodic syllable embedding and the phone-level linguistic features of each corresponding phoneme associated with the syllable.

In some implementations, the method also includes, for each syllable, using the selected utterance embedding: predicting, by the data processing hardware, an energy contour of each phoneme in the syllable based on a predicted duration for the phoneme; and for each phoneme associated with the syllable, generating, by the data processing hardware, a plurality of fixed-length predicted energy frames based on the predicted duration for the phoneme. Here, each fixed-length energy frame represents the predicted energy contour of the corresponding phoneme. In these implementations, a hierarchical linguistic structure represents the text utterance and the hierarchical linguistic structure includes a first level including each syllable of the text utterance, a second level including each phoneme of the text utterance, a third level including each fixed-length predicted pitch frame for each syllable of the text utterance, and a fourth level parallel to the third level and including each fixed-length predicted energy frame for each phoneme of the text utterance. The first level may include a long short-term memory (LSTM) processing cell representing each syllable of the text utterance, the second level may include a LSTM processing cell representing each phoneme of the text utterance, the third level may include a LSTM processing cell representing each fixed-length predicted pitch frame, and the fourth level may include a LSTM processing cell representing each fixed-length predicted energy frame. Here, the LSTM processing cells of the second level clock relative to and faster than the LSTM processing cells of the first level, the LSTM processing cells of the third level clock relative to and faster than the LSTM processing cells of the second level, and the LSTM processing cells of the fourth level clock at the same speed as the LSTM processing cells of the third level and clock relative to and faster than the LSTM processing cells of the second level.

In some examples, the third level of the hierarchical linguistic structure includes a feed-forward layer that predicts the predicted pitch frames for each syllable in a single pass and/or the fourth level of the hierarchical linguistic structure includes a feed-forward layer that predicts the predicted energy frames for each phoneme in a single pass. Moreover, the lengths of the fixed-length predicted energy frames and the fixed-length predicted pitch frames may be the same. Additionally or alternatively, a total number of fixed-length predicted energy frames generated for each phoneme of the received text utterance may be equal to a total number of the fixed-length predicted pitch frames generated for each syllable of the received text utterance.

In some implementations, the method also includes: receiving, by the data processing hardware, training data including a plurality of reference audio signals, each reference audio signal including a spoken utterance of human speech and having a corresponding prosody; and training, by the data processing hardware, a deep neural network for a prosody model by encoding each reference audio signal into a corresponding fixed-length utterance embedding representing the corresponding prosody of the reference audio signal. The utterance embedding may include a fixed-length numerical vector.

Another aspect of the disclosure provides a system for representing an intended prosody in synthesized speech. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving a text utterance having at least one word, and selecting an utterance embedding for the text utterance. Each word in the text utterance has at least one syllable and each syllable has at least one phoneme. The utterance embedding represents an intended prosody. For each syllable, using the selected utterance embedding, the operations also include: predicting a duration of the syllable by encoding linguistic features of each phoneme of the syllable with a corresponding prosodic syllable embedding for the syllable; predicting a pitch contour of the syllable based on the predicted duration for the syllable; and generating a plurality of fixed-length predicted pitch frames based on the predicted duration for the syllable. Each fixed-length predicted pitch frame represents part of the predicted pitch contour of the syllable.

This aspect may include one or more of the following optional features. In some implementations, a network representing a hierarchical linguistic structure of the text utterance includes a first level including each syllable of the text utterance, a second level including each phoneme of the text utterance, and a third level including each fixed-length predicted pitch frame for each syllable of the text utterance. In these implementations, the first level of the network may include a long short-term memory (LSTM) processing cell representing each syllable of the text utterance, the second level of the network may include a LSTM processing cell representing each phoneme of the text utterance, and the third level of the network may include a LSTM processing cell representing each fixed-length predicted pitch frame. Here, the LSTM processing cells of the second level clock relative to and faster than the LSTM processing cells of the first level, while the LSTM processing cells of the third level clock relative to and faster than the LSTM processing cells of the second level.

In some examples, predicting the duration of the syllable includes: for each phoneme associated with the syllable, predicting a duration of the corresponding phoneme by encoding the linguistic features of the corresponding phoneme with the corresponding prosodic syllable embedding for the syllable; and determining the duration of the syllable by summing the predicted durations for each phoneme associated with the syllable. In these examples, predicting the pitch contour of the syllable based on the predicted duration for the syllable may include combining the corresponding prosodic syllable embedding for the syllable with each encoding of the corresponding prosodic syllable embedding and the phone-level linguistic features of each corresponding phoneme associated with the syllable.

In some implementations, the operations also include, for each syllable, using the selected utterance embedding: predicting an energy contour of each phoneme in the syllable based on a predicted duration for the phoneme; and for each phoneme associated with the syllable, generating a plurality of fixed-length predicted energy frames based on the predicted duration for the phoneme. Here, each fixed-length energy frame represents the predicted energy contour of the corresponding phoneme. In these implementations, a hierarchical linguistic structure represents the text utterance and the hierarchical linguistic structure includes a first level including each syllable of the text utterance, a second level including each phoneme of the text utterance, a third level including each fixed-length predicted pitch frame for each syllable of the text utterance, and a fourth level parallel to the third level and including each fixed-length predicted energy frame for each phoneme of the text utterance. The first level may include a long short-term memory (LSTM) processing cell representing each syllable of the text utterance, the second level may include a LSTM processing cell representing each phoneme of the text utterance, the third level may include a LSTM processing cell representing each fixed-length predicted pitch frame, and the fourth level may include a LSTM processing cell representing each fixed-length predicted energy frame. Here, the LSTM processing cells of the second level clock relative to and faster than the LSTM processing cells of the first level, the LSTM processing cells of the third level clock relative to and faster than the LSTM processing cells of the second level, and the LSTM processing cells of the fourth level clock at the same speed as the LSTM processing cells of the third level and clock relative to and faster than the LSTM processing cells of the second level.

In some examples, the third level of the hierarchical linguistic structure includes a feed-forward layer that predicts the predicted pitch frames for each syllable in a single pass and/or the fourth level of the hierarchical linguistic structure includes a feed-forward layer that predicts the predicted energy frames for each phoneme in a single pass. Moreover, the lengths of the fixed-length predicted energy frames and the fixed-length predicted pitch frames may be the same. Additionally or alternatively, a total number of fixed-length predicted energy frames generated for each phoneme of the received text utterance may be equal to a total number of the fixed-length predicted pitch frames generated for each syllable of the received text utterance.

In some implementations, the operations also includes: receiving training data including a plurality of reference audio signals, each reference audio signal including a spoken utterance of human speech and having a corresponding prosody; and training a deep neural network for a prosody model by encoding each reference audio signal into a corresponding fixed-length utterance embedding representing the corresponding prosody of the reference audio signal. The utterance embedding may include a fixed-length numerical vector.

The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

