BACKGROUND

Humans can engage in human-to-computer dialogs with interactive software applications referred to herein as “automated assistants” (also referred to as “digital agents,” “interactive personal assistants,” “intelligent personal assistants,” “assistant applications,” “conversational agents,” etc.). For example, humans (which when they interact with automated assistants may be referred to as “users”) may provide commands and/or requests to an automated assistant using spoken natural language input (i.e., spoken utterances), which may in some cases be converted into text and then processed, and/or by providing textual (e.g., typed) natural language input (i.e., typed utterances). An automated assistant responds to a request by providing responsive user interface output, which can include audible and/or visual user interface output.

As mentioned above, many automated assistants are configured to be interacted with via spoken utterances. A user may submit queries and/or commands to an automated assistant interface of a client device via a spoken utterance, verbally indicating what information the user has interest in being provided and/or an action that the user has interest in being performed. Typically, the spoken utterance is detected by microphone(s) of the client device and captured as audio data. The client device may operate in a “speech recognition state” where for at least some time interval after invocation, the automated assistant performs speech-to-text (“STT”) processing of audio data sampled by a microphone to generate textual input. The input in turn is semantically processed to determine a user's intent and to fulfill that intent. Fulfilling the intent can include determining an appropriate response, and causing the response to be rendered at the client device.

The response to a query may be presented to the user in either or both an audible and a visual format. For example, a smartphone may display text and images on its display in response to a particular query. Additionally or alternatively, the user device may use a speaker to present the user with an audible answer to their question in a digital voice. Under certain circumstances, a user expecting an audible response may instead receive only a visual response and/or an audible response may be provided for audible rendering, but not audibly perceived by the user. For instance, a user may unknowingly have an audio response setting of their phone set to a muted level, which can prevent the user from perceiving the audible response. As a result, the user can repeat the query in another attempt to receive an audible response. The repeated query must again be processed (e.g., STT, semantic understanding) and a response again generated. This can burden available hardware resources (e.g., battery, processing resources, memory resources) and, when data is transmitted between a client device and server in processing the query and/or in generating the response, can burden network resources.

SUMMARY

Some implementations disclosed herein relate to determining whether audible perception, of an audible response to a query (e.g., a voice query directed to an automated assistant), is critical to resolution of the query. Some of those implementations determine whether audible perception of the audible response to the query is critical based on one or more objective metrics generated based on historical analysis of past submission of that query (and/or of similar queries) and/or past instances of that response (and/or of similar responses). For example, a metric of a query can be generated based on a quantity of occurrences, across a population of users, of certain repeat submissions of that query (and/or similar queries) to an automated assistant. For instance, the certain repeat submissions can be those where: (a) an initial submission of the query was by a corresponding user and occurred when the response volume setting of a corresponding device that received the query and/or rendered (or at least attempted to render) the response was muted (e.g., less than a threshold, such as 25% of a maximum volume setting); and (b) a manual increase in the response volume setting (e.g., to non-muted) occurred between the initial submission and the repeat submission of the query by the corresponding user. Optionally, the repeat submissions can be restricted to those occurring within a threshold time (e.g., within 30 seconds of) a corresponding initial submission. Generally, the larger the quantity of repeat submissions, the more likely the query (and/or similar queries) will be found to be ones whose audible response is critical to resolution of the query.

Some implementations additionally or alternatively relate to, responsive to determining that audible perception of an audible response is critical, performing one or more operations to increase the likelihood that the audible response is audibly perceived in response to a submission of the query. As one example, the operation(s) can include fully automatically (i.e., without requiring any user input) adjusting the response volume setting of a device at which the query was submitted. For instance, the response volume setting of the device can be fully automatically adjusted responsive to the response volume setting being muted (e.g., less than a threshold, such as 30% of a maximum volume setting) and responsive to determining that audible perception of the audible response is critical. The automatic adjustment can be to a non-muted setting, such as to at least 50% of a maximum volume setting or to other adjusted non-muted setting. As another example, the operation(s) can include providing a visual prompt (e.g., a graphical element on a display) to recommend adjusting the volume, then automatically adjusting the volume responsive to affirmative user interface input provided responsive to the visual prompt (e.g., a touch of the graphical element on the display, or a spoken “yes” or “increase volume”). Optionally, the rendering of the audible response is postponed for at least a threshold duration of time and/or until the user has responded (affirmatively or negatively) to the visual prompt. This can prevent rendering of the audible response prior to the response volume setting being increased, mitigating necessitating repeating of the query to audible perceive the entirety of the audible response. In some implementations, whether a fully automatic adjustment or, instead, a visual prompt for adjustment is provided, can be based on a magnitude of a corresponding metric. For example, if the metric satisfies a first threshold but fails to satisfy a second threshold, a visual prompt for adjustment can be provided. However, if the metric satisfies both the first and second thresholds, a fully automatic adjustment can be performed. As yet another example, the operation(s) can include performing device arbitration to select a secondary device, determined to be farther away from the user than a primary device (but optionally within a threshold distance of the user), based on the secondary device being at a greater (e.g., non-muted) response volume setting than the primary device. Put another way, when it is determined that audible perception of an audible response is critical, a device for rendering the audible response can be selected from an ecosystem of available devices based at least in part on that device having a non-muted response volume setting.

In these and other manners, the objective metric can be utilized to mitigate occurrences of users not audibly perceiving audible responses when audible perception is deemed critical. This, in turn, can mitigate occurrences of repeat submissions of the query. By mitigating repeat submission of the query, overall duration of user-assistant dialogs is decreased. Moreover, computational and/or network resources are preserved by mitigating the repeat submission of the query. For example, for voice-based queries another occurrence of speech-to-text processing is mitigated, another occurrence of transmitting audio data (or transcribed text) to a server is mitigated, and/or another occurrence of transmitting and rendering a response to the query is mitigated. It is noted that, in various implementations, the objective metric is determined based on historical data as described herein. In some of those implementations, by determining whether to automatically adjust the response volume setting based on the objective metric, it is ensured that, at least over multiple queries from a plurality of users, occurrences of repeat submissions of a query are mitigated.

It is also noted that utilizing the objective metric can also prevent unnecessarily automatically adjusting the response volume setting when audible responses are not critical, thereby preventing a greater amount of power to be utilized in rendering the response at a higher volume. Further, in some implementations, when the objective metric indicates an audible response is not critical, a visual response is being provided, and a display for rendering the visual response is available, the audible response may not even be generated and/or transmitted for rendering—conserving computational and network resources.

As mentioned above, in various implementations the metric, utilized in determining whether audible perception of an audible response is critical, can be generated based on a quantity of occurrences of certain repeat submissions of that query. In some of those implementations, the metric can be based on the raw quantity of occurrences. For example, the metric can equal the raw quantity of occurrences in analyzed historical data. In some versions of those implementations, the raw quantity of occurrences can optionally be normalized utilizing one or more normalization techniques. For example, the raw quantity can be normalized based on an overall quantity of occurrences of the query (repeated or otherwise). For instance, the raw quantity can be divided by the overall quantity of occurrences, and the result utilized as the metric. In these and other manners, the quantity of certain repeat submissions of queries that are highly popular will need to be greater to find criticality as compared to the quantity of repeat submission of queries that are less popular. As another example of normalization, the raw quantity can be normalized based on an overall quantity of occurrences of repeats of the query (optionally within a threshold time period of one another), without an intervening manual increase in the response volume setting. In these and other manners, the quantity of certain repeat submissions of queries that are often repeated quickly (e.g., “how much time is left on the timer”) will need to be greater to find criticality as compared to the quantity of repeat submission of queries that are not often repeated quickly (e.g., “when did California become a state”).

Regardless of the technique(s) utilized to generate the metric(s), the metric(s) can be utilized in some implementations in determining whether an audible response to the query is critical. For example, queries whose metric(s) satisfy threshold(s) can be considered critical. In some implementations, the threshold for a metric is manually chosen. In some implementations, the threshold is chosen based on it being the mean, median, third quartile or other value of all generated metrics. It is noted that a metric can be associated with a single query or with multiple queries. When associated with multiple queries, the multiple queries can be represented as a regular expression, a query type, a query embedding, and/or other representation. For example, a metric can be generated based on analysis of historical repeat queries that conform to “what does a bad [automotive part] sound like?”, where “[automotive part]” represents a plurality of disparate automotive parts, and can be assigned to “what does a bad [automotive part] sound like”. As another example, a metric can be generated based on analysis of historical repeat queries that have corresponding query embeddings that are close to one another in embedding space, and can be assigned to those query embeddings or to an average of the query embeddings. The query embedding of a query can be generated by processing the query using a machine learning model such as, for example, a Word2Vec model.

It is noted that generating a metric for a query and/or utilizing the metric to determine whether an audible response to the query can include generating a metric for the audible response to the query. For example, the metric can be generated for an audible response based on a quantity of occurrences, across a population of users, of certain repeat provisions of that audible response (and/or similar responses) by an automated assistant. For instance, the certain repeat submissions can be those where: (a) an initial provision of the audible response was to a corresponding user and occurred when the response volume setting of a corresponding device that rendered (or at least attempted to render) the audible response was muted (e.g., less than a threshold, such as 25% of a maximum volume setting); and (b) a manual increase in the response volume setting (e.g., to non-muted) occurred between the initial provision of the audible response and the repeat provision of the audible response. Although this metric is generated based on responses, and can even be assigned to responses, it encompasses generating a metric for a query as that phrase is utilized herein. This is at least because determining whether an audible response to a query is critical can be made with reference to a determined response to the query.

In some implementations, generated metrics can be utilized to adapt the underlying content of audible responses. For example, responsive to the metric indicating audible perception of an audible response is critical, the audible response can be made to be more robust (i.e., include more content). On the other hand, responsive to the metric indicating audible perception of an audible response is not critical, the audible response can be abridged (i.e., include less content). For instance, if it is determined that audible perception of a given audible response that includes speech synthesis of text is not critical, the text can be shortened (e.g., using a text summarization algorithm) and speech synthesis of the shortened text provided in lieu of the speech synthesis of the non-abridged text. This can result in less data being transmitted (e.g., when the text and/or the synthesized speech is transmitted from a server to a client) and/or to a shorter duration of audio being rendered at a client device. In some implementations, where a query is for a third party agent (i.e., the third party agent generates the response to the query), metrics associated with the query can be provided to one or more parties associated with the third party agent. Providing such metrics can enable the third party agents to adapt the underlying content of the audible responses (e.g., as described above).

The above description is provided as an overview of only some implementations disclosed herein. Those and other implementations are described in more detail here.

In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the methods described herein. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the methods described herein.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

