BACKGROUND

Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as “automated assistants” (also referred to as “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” “virtual assistants,” etc.). For example, humans (which when they interact with automated assistants may be referred to as “users”) may provide commands, queries, and/or requests (collectively referred to herein as “queries”) using free form natural language input which may include vocal utterances converted into text and then processed and/or typed free form natural language input. Automated assistants may perform various types of processing on the natural language input, such as natural language processing, syntactic process, semantic processing, etc., in order to identify and respond to the user's intent.

In many cases, before the automated assistant can interpret and respond to a user's request, it must first be “invoked” in response to various events. In some cases an automated assistant can be invoked in response to a button being operated and/or pressed for some predetermined time period, etc. Additionally or alternatively, some automated assistants may be invoked using predefined oral invocation words/phrases that are often referred to as “hot words” or “wake words.” Thus, many automated assistants operate in what will be referred to herein as a “limited hot word listening state” or “inactive listening state” in which they are always “listening” to audio data sampled by a microphone for a limited (or finite, or “default”) set of hot words. Any utterances captured in the audio data other than the default set of hot words are ignored. Once the automated assistant is invoked with one or more of the default set of hot words, it may operate in what will be referred to herein as an “active listening state” wherein for at least some time interval after invocation, the automated assistant performs speech-to-text (“STT”, also referred to as “automatic speech recognition,” or “ASR”) processing of audio data sampled by a microphone to generate textual input, which in turn is semantically processed to determine a user's intent (and to resolve that intent).

Operating the automated assistant in the inactive listening state provides a variety of benefits. Limiting the number of hot words being “listened for” allows for conservation of power and/or computing resources. For example, an on-device machine learning model may be trained to generate output that indicates when one or more hot words are detected. Implementing such a model may require only minimal computing resources and/or power, which is particularly beneficial for assistant devices that are often resource-constrained. Storing such a trained model locally on a client device also provides advantages relating to privacy. For example, some users do not want STT/ASR processing to be performed automatically on everything they say within earshot of a computing device operating an automated assistant. Additionally, the on-device model also prevents data indicative of user utterances that are not intended to be processed by the automated assistant from being provided to a semantic processor, which often operates at least in part on the cloud.

Along with these benefits, operating the automated assistant in the limited hot word listening state also presents various challenges. To avoid inadvertent invocation of the automated assistant, hot words are typically selected to be words or phrases that are not often uttered in everyday conversation (e.g., “long tail” words or phrases). However, there are various scenarios in which requiring users to utter long tail hot words before invoking an automated assistant to perform some action can be cumbersome. Some automated assistants may provide an option for a “continued listening” mode after a user utters a command, so that the user need not “reawaken” the automated assistant with hot word(s) before performing a subsequent command. However, transitioning the automated assistant into a continued listening mode means that the automated assistant may be performing far more STT processing for far more utterances, potentially wasting power and/or computing resources. Additionally, and as noted above, some users prefer that only utterances addressed to the automated assistant are STT processed.

SUMMARY

Techniques are described herein for selectively processing a user's utterances captured prior to and after an event that invokes an automated assistant to determine the user's intent and/or any parameters required for resolving the user's intent. Consequently, it is possible, for instance, for the user to provide a spoken utterance—which may include a request, a command, a search query, a question, etc.—prior to invoking the automated assistant. Once the automated assistant is invoked, it can retroactively analyze data indicative of the user's prior spoken utterance—e.g., the recorded raw audio, an embedding generated from the raw audio, output of STT and/or natural language processing based on the raw audio, etc.—to determine the user's intent, and respond accordingly. The use of retroactive analysis can avoid unnecessary processing of utterances which are not ultimately associated with a user request to an automated assistant. In this manner, processing and network overheads may be reduced. Moreover, processing of utterances may be distributed between a local device and one or more remote servers while limiting the networked transmission of utterances not intended for the assistant, thereby improving data security while overcoming the technical limitations in implementing all utterances processing locally.

In various implementations, a memory buffer (also referred to as a “retroactive memory buffer”) of a client device at which a user engages with an automated assistant may at least temporarily store some predetermined time interval (e.g., ten seconds) worth of data indicative of sound captured by one or more microphones. In some implementations, the data indicative of the sound captured by the microphone(s) may be written into the retroactive memory buffer in response to various stimuli, such as noise, voices, etc. In some such implementations, if there is little or no ambient sound, no data indicative of sound captured by microphones may be stored in the retroactive memory buffer.

Prior to detection of an event that invokes the automated assistant, the automated assistant may continuously and repeatedly overwrite the retroactive memory buffer. In some implementations the retroactive memory buffer may be a circular memory buffer that is periodically overwritten, e.g., using sliding window techniques, although this is not required. In some implementations, the memory buffer may be configured to store some predetermined time interval's worth of data, such as ten seconds of raw audio, text or other data generated from ten seconds of raw audio, etc. In some implementations, the retroactive memory buffer may temporarily store raw audio data captured by the microphone(s), an embedding generated from the raw audio data, or other intermediate features, such as a spectrogram representation. Additionally or alternatively, in some implementations, rudimentary or even comprehensive ASR/STT processing may be performed on captured raw audio, and the extracted text generated by the ASR/STT processing may be temporarily stored in the retroactive memory buffer. Additionally or alternatively, in some implementations, NLP may be performed on the extracted text, and the output of the NLP, which may include annotated text and/or a user intent, may be temporarily stored in the retroactive memory buffer.

Upon detection of an event that invokes the automated assistant, the automated assistant may transition from the inactive listening state into an “active listening state” in which additional processing is performed on the data temporarily stored in the retroactive memory buffer, which is referred to herein as the “pre-event input stream.” Additionally or alternatively, additional processing may be performed on data indicative of sound captured by microphone(s) after occurrence of the event; this data is referred to herein as the “post-event input stream.” In various implementations, the pre-event and post-event input streams may each be analyzed to determine respective measures of fitness (or suitability) for triggering responsive action by the automated assistant. Measures of fitness for triggering responsive action by the automated assistant may be determined in various ways.

In some implementations, determining the respective measures of fitness may include determining respective similarity measures between the pre-event and post-event input streams and one or more known command syntaxes. If one of the pre-event and post-event input stream includes text that is syntactically similar to a command, and the other does not, then the one input stream may be selected for further processing, e.g., to cause the automated assistant to take responsive action. For example, the phrase “Turn on the lights” is far more similar syntactically to the command syntax “Turn <off/on> the <device>” than the phrase, “I wonder when the next full moon is?”

Additionally or alternatively, it has been observed that users tend to address automated assistants differently than other humans. The users tend to annunciate more clearly, speak more slowly, speak more loudly, etc. Moreover, when people speak to each other, they may not necessarily speak in the direction of a microphone that is accessible to an automated assistant, and as a consequence, user-to-user utterances may tend to have less clarity and/or more noise than user-to-assistant utterances. Accordingly, in various implementations, determining the respective measures of fitness may include analyzing one or more features of the user's voice, such as cadence, annunciation, volume, tempo, etc. to determine its fitness for triggering responsive action. In some implementations, this may include comparing the one or features of the user's voice to one or more voice features observed from utterances containing commands for automated assistants. In some such implementations this comparison may be implemented by way of a machine learning model that is trained—e.g., using training data comprising utterances labeled as being directed to (or not directed to) an automated assistant—to generate output indicative of whether the input is directed to an automated assistant. For example, the pre-event and post-event input streams may be processed by a machine learning model to generate an output indicative of whether their content is directed to an automated assistant and one or both of the pre-event and post-event input streams may be selected in dependence on the output. In some implementations, visual cues may also be considered to determine whether a person is addressing the automated assistant. For example, if the person is looking at the camera when making an utterance, whatever the person says may be more likely to be considered an invoking event. In some implementations, the machine learning model mentioned earlier or an additional machine learning model may use such visual cues to improve an assessment of likelihood that the utterance is directed towards the assistant.

As noted previously, an automated assistant may be invoked by various events, such as user selection of various input elements and/or one or more hot words/phrases. In some implementations, the audio data captured by the microphone(s) may be monitored, e.g., in real time using a digital signal processor (“DSP”), to detect one or more hot words. Upon detection of the hot word(s), audio data previously stored in the retroactive memory buffer of the client device at which the user is invoking the automated assistant may be analyzed, e.g., alone or in parallel with audio data captured after the event. In some such implementations, the local memory buffer may be a DSP buffer integral with a DSP chip or chipset, although this is not required. When the automated assistant transitions into the active listening state, it may access this DSP buffer to perform, for instance, speech recognition, natural language processing, etc., on the temporarily-stored audio data and identify one or more tokens.

Various hot words or phrases may be used to trigger transition of an automated assistant into the active listening state. In some implementations, some hot word(s) may trigger analysis of both the pre-event and post-event input streams, other hot word(s) may trigger analysis of only the pre-event input stream, and yet other hot word(s) may trigger analysis of only the post-event input stream. In some implementations, there may be no distinction between retroactive and processing.

As a first non-limiting example, in some implementations, particular hot word(s) may trigger the automated assistant to perform additional processing (e.g., ASR, NLP, etc.) on the entire content of the memory buffer, and use the output of this processing to, for instance, perform a search. By extracting entities and other terms/phrases that a user might search for, both “word definitions” and factual searches may be supported. Suppose that a first user in a room says, to a second user, “Hmm, I wonder what parsimonious means?”, and the second user then says, “sleuth that.” In some implementations, the phrase “sleuth that” may operate as a trigger that causes an automated assistant implemented at least in part on a nearby client device to transition into an active listening state. In the active listening state, the automated assistant may obtain the contents of the retroactive memory buffer (audio data comprising a recording of the first user saying “Hmm, I wonder what parsimonious means?”), process them as described above (e.g., ASR, various NLP and/or entity annotations), and submit the result as a search.

In various implementations, hot word(s) usable to transition an automated assistant into the active listening state may include various cues that help the automated assistant determine which action to take. For example, in some implementations, one or more specific hot word(s) may be used to trigger the automated assistant to create a reminder based on retroactive spoken input contained in the memory buffer. For example, suppose a user says, “I need to get some milk on the way home. Remind me about that.” The phrase “Remind me about that” may operate as a trigger to cause the automated assistant to transition into the active listening state and process the user's prior utterance, “I need to get some milk on the way home,” into a reminder that will be provided to the user later, e.g., when it is determined (e.g., using a position coordinate trigger) that the user is driving home from work. And it need not be the same user that provides the reminder text and the hot word(s). For example, a first user may say, to a second user (in person and/or over the phone), “Can you get some milk on the way home?” The second user may then say, “Remind me about that,” to cause the first user's utterance to be processed into a reminder for the second user to receive at a later, contextually appropriate time.

As another example of cues that can be used by the automated assistant in active listening state, in some implementations, referential keywords such as pronouns, determiners, etc., that are contained in the hot word(s) may be identified and mapped to token(s) extracted from the audio data contained in the memory buffer. For example, if multiple entities are identified in the audio data, then the hot word(s) may be analyzed to determine which entity should be the subject or scope of the responsive action taken by the automated assistant. As one example, suppose a first user says to a second user during a conversation, “Yeah, I think Liz Taylor was married to Richard Burton.” Suppose the first or second user then says, “Sleuth her.” Of the two entities identified (e.g., using a knowledge graph) in the utterance—Liz Taylor and Richard Burton—only Liz Taylor is female. Accordingly, the resulting search assembled and submitted by the automated assistant may be scoped to Liz Taylor. In various implementations, the other identified entity, Richard Burton, may be ignored or used as a weaker signal to help further scope the search results.

As a general example, suppose a user says, “I'm going to the store. Remind me to pick up some apples.” The utterance “I'm going to the store” is stored in the retroactive memory buffer as the pre-event stream, the hot word phrase “Remind me to . . . ” is the event that triggers the automated assistant, and the phrase “ . . . pick up some apples” is captured following the hot word phrase and constitutes the post-event input stream. The pre-event input stream indicative of the phrase “I'm going to the store” is somewhat ambiguous insofar as it doesn't clearly state what the automated assistant is supposed to do, likely doesn't match any known or common command syntax, and is a statement, rather than a command. Thus, when analyzed by automated assistant, this pre-event statement may yield a user intent such as “directions to store” that has a relatively low confidence measure, and therefore has a relatively low measure of fitness for triggering an automated assistant to take responsive action. Moreover, the trigger phrase “remind me” indicates that a reminder is to be set, and the pre-trigger statement “I'm going to the store” is not a typical reminder phrase; it simply states what the user is going to do. On the other hand, the post-event input stream indicative of the phrase “ . . . pick up some apples” is a command, and therefore is more suitable as a reminder. Consequently, the measure of fitness for user intent “Remind me to <pick up some apples>” is greater, and therefore would likely be selected by the automated assistant.

Another example of an event that may invoke an automated assistant into the active listening state is context. As a non-limiting example, some automated assistants may be invoked upon a particular application being launched. For example, an application such as a mapping/navigation application may, when launched, invoke an automated assistant into the active listening state. Consequently, when a user opens the mapping/navigation app, e.g., while driving, the user can then issue commands (e.g., “find directions to the nearest coffee shop”) without having to provide any additional inputs, and the automated assistant will respond accordingly.

In various implementations, at least some processing capability, such as ASR, NLP, entity tagging, etc., may be built into the client device itself, so that the onboard processing can be used to attempt to analyze speech without sending potentially sensitive information to an online (i.e. “cloud-based”) semantic processor. However, the constrained resources of most client devices may limit the functionality of such onboard capabilities. Additionally or alternatively, the user's utterance may not be clear, e.g., due to background noise, mumbling, accents, limited vocabulary, etc. Thus it still may be necessary in some instances to send audio data (or data indicative thereof, such as an embedding) to the online semantic processor.

Accordingly, in various implementations, automated assistants configured with selected aspects of the present disclosure may be configured to solicit permission from a user (or at least notify the user) that something they said is about to be sent to an online semantic processor, so that the user has a chance to stop that from happening. It also may prevent a user from being startled by receiving natural language output from an automated assistant when none was requested.

For example, in some implementations, the client device on which the automated assistant operates may activate one or more lights (e.g., light emitting diodes) to indicate that something the user said is about to be processed online. Additionally or alternatively, in some implementations, the automated assistant may generate and provide natural language output conveying the request or recording it is about to submit to the online semantic processor, and may even ask the user to confirm this is OK. In some implementations, such notifications may be provided when, for instance, a confidence associated with the detected hot word(s) is below some threshold. If, on the other hand, the confidence measure is above the threshold—indicating the user spoke the hot word(s) clearly and they mapped to known hot word(s)—the user's utterance or data indicative thereof (e.g., an embedding, annotated text, etc.) may be sent to the online semantic processor for resolution.

In some implementations, a method performed by one or more processors is provided that includes: executing an automated assistant in an inactive listening state at least in part on a computing device operated by a user; while in the inactive listening state, detecting an event that triggers transition of the automated assistant from the inactive listening state into an active listening state, wherein first data indicative of sound captured by one or more microphones prior to the event is temporarily stored, as a pre-event input stream, in a memory buffer of the computing device operated by the user; obtaining, as a post-event input stream, second data indicative of sound captured by one or more of the microphones after detecting the event; while in the active listening state, determining respective measures of fitness of the pre-event and post-event input streams for triggering responsive action by the automated assistant; based on the respective measures of fitness, selecting the pre-event input stream or post-event input stream; and causing the automated assistant to perform one or more responsive actions based on the selected input stream.

These and other implementations of technology disclosed herein may optionally include one or more of the following features.

In various implementations, the method may further include monitoring sound captured by one or more of the microphones for one or more hot words, wherein the event that triggers transition of the automated assistant from the inactive listening state into the active listening state comprises detection of one or more of the hot words during the monitoring. In various implementations, the method may further include performing speaker recognition processing on at least a portion of the sound captured by the one or more microphones to determine an identity of the user, wherein transition of the automated assistant from the inactive listening state into the active listening state is further triggered in response to a determination that the identity of the user satisfies one or more criteria.

In various implementations, the one or more criteria may include the identity of the user matching an owner of the computing device. In various implementations, the monitoring may be performed using a digital signal processor integral with the computing device operated by the user. In various implementations, the digital signal processor is integral with a digital signal processing chip, and wherein the memory buffer comprises a digital signal processing buffer onboard the digital signal processing chip. In various implementations, one or more of the hot words is a pronoun, and the method further includes resolving the pronoun to one or more tokens extracted from the selected input stream.

In various implementations, the memory buffer may be a circular memory buffer that is periodically overwritten by newly captured audio data. In various implementations, the memory buffer may be configured to temporarily store audio data captured over a predetermined time interval. In various implementations, determining the respective measures of fitness may include determining respective similarity measures between the pre-event and post-event input streams and one or more known command syntaxes. In various implementations, determining the respective measures of fitness includes analyzing one or more features of the user's voice in one or both of the pre-event and post-event input streams. In various implementations, the analyzing includes comparing the one or features of the user's voice to one or more voice features observed from utterances containing commands for automated assistants. In various implementations, the analyzing includes applying the one or features of the user's voice as input across a machine learning model, wherein the machine learning model is trained to generate output indicative of whether the input is directed to an automated assistant.

In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

