BACKGROUND

Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as “automated assistants” (also referred to as “digital agents,” “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “assistant applications,” “conversational agents,” etc.). For example, humans (which when they interact with automated assistants may be referred to as “users”) may provide commands and/or requests to an automated assistant using spoken natural language input (i.e., spoken utterances), which may in some cases be converted into text and then processed, and/or by providing textual (e.g., typed) natural language input. An automated assistant may respond to a request by providing responsive user interface output, which can include audible and/or visual user interface output.

Automated assistants can have limited availability when a user is operating other applications. As a result, a user may attempt to invoke an automated assistant to perform certain functions that the user associates with other applications, but ultimately terminate a dialog session with the automated assistant when the automated assistant cannot continue. For example, the limited availability or limited functionality of automated assistants may mean that users are unable to control other applications via voice commands processed by the automated assistant. This can waste computational resources, such as network and processing bandwidth, because any processing of spoken utterances during the dialog session would not have resulted in performance of any action(s). Furthermore, because of this deficiency, as a user interacts with their respective automated assistant, the user may avoid operating other applications that may otherwise provide efficiency for various tasks performed by the user. An example of such a task is the control of a separate hardware system, such as a heating system, air-conditioning system or other climate control system, via an application installed on the user's computing device. Such avoidance can lead to inefficiencies for any devices that might otherwise be assisted by such applications, such as smart thermostats and other application-controlled devices within the hardware system, as well as for any persons that might benefit from such applications or their control of associated devices.

Moreover, when a user does elect to interact with their automated assistant to perform certain tasks, the limited functionality of the automated assistant may inadvertently negate an ongoing dialog session as a result of the user opening another application. The opening of the other application can be assumed by many systems to be an indication that the user is no longer interested in furthering an ongoing dialog session, which can waste computational resources when the user is actually intending to perform an action related to the dialog session. In such instances, the dialog session may be canceled, thereby leading to the user repeating any previous spoken utterances in order to re-invoke the automated assistant, which can impede any progress of the user initializing certain application actions using such systems.

SUMMARY

Implementations set forth herein relate to one or more systems for allowing a user to invoke an automated assistant to initialize performance of one or more actions by a particular application (or a separate application), simultaneous to the user interacting with the particular application. The particular application that the user is interacting with can be associated with data that characterizes a variety of different actions capable of being performed by the particular application. Furthermore, separate applications can also be associated with the data, which can also characterize various actions capable of being performed by those separate applications. When the user is interacting with the particular application, the user can provide a spoken utterance in furtherance of completing one or more actions that the particular application, or one or more of the separate applications, can complete.

As an example, the particular application can be an alarm system application that the user is accessing via a computing device. The alarm system application may, for example, be installed on the computing device. In the discussion below, the computing device will be referred to in the context of a tablet computing device, but it will be appreciated that the computing device could alternatively be a smartphone, a smartwatch etc. The user can be using the tablet computing device to access the alarm system application in order to view video that has been captured by one or more security cameras that are in communication with the alarm system application. While viewing the videos, the user may desire to secure their alarm system. In order to do this, the user can provide a spoken utterance simultaneous to interacting with the video interface of the alarm system application, i.e., without having to close the video they are viewing or otherwise navigate to a separate interface of the alarm system application in order to secure the alarm system. For example, the user can provide a spoken utterance such as, “secure the alarm system.” In some implementations, the spoken utterance can be processed at the tablet computing device and/or a remote computing device, such as a server device, in order to identify one or more actions that the user is requesting the automated assistant to initialize performance of. For instance, automatic speech recognition and/or natural language understanding of the spoken utterance can be performed on-device. Automatic speech recognition (ASR) can be performed on-device in order to detect certain terms that can correspond to particular actions capable of being performed via the device. Alternatively, or additionally, natural language understanding (NLU) can be performed on-device in order to identify certain intent(s) capable of being performed via the device.

Input data characterizing the natural language content can be used in order to determine one or more particular actions that the user is requesting to initialize via the automated assistant. For instance, the tablet computing device can access application data and/or store application data for each application that is accessible via the tablet computing device. In some implementations, the application can be accessed in response to the user invoking the automated assistant via non-voice activity (e.g., button push, physical interaction with a device, indirect input such as a gesture) and/or via voice activity (e.g., hot word, invocation phrase, detecting particular term(s) in a spoken utterance, and/or detecting that a spoken utterance corresponds to an intent(s)).

The application data may be accessed by the automated assistant in response to an invocation gesture alone, such as the voice/non-voice activity referred to above, before the remainder of a spoken user request/utterance following the invocation gesture is received and/or processed by the assistant. This may allow the assistant to obtain application data for any application which is currently running on the tablet computing device before the device has finished receiving/processing the complete request from the user, i.e., following the invocation gesture. As such, once the complete request has been received and processed, the assistant is in a position to immediately determine whether the request can be actioned by an application currently running on the device. In some cases, the application data that is obtained in this manner may be limited to application data for an application which is currently running in the foreground of a multitask operating environment on the device.

In order to access the application data, the automated assistant application can transmit a request (e.g., via an operating system) to one or more applications in response to the non-voice activity and/or the voice activity. In response, the one or more applications can provide application data characterizing contextual actions and/or global actions. The contextual actions can be identified and/or executable based on a current state of a respective application (e.g., an active application that the user is accessing) that performs the contextual actions, and the global actions can be identified and/or executable regardless of the current state of the respective application. By allowing the applications to provided application data in this way, the automated assistant can operate from accurate indexes of actions, thereby enabling the automated assistant to initialize related actions in response to a particular input. Furthermore, in some implementations, the application data can be accessed by the automated assistant without any network transmissions, as a result of ASR and/or NLU being performed on device and in combination with the action selection.

The input data can be compared to the application data for one or more different applications in order to identify one or more actions that the user is intending to initialize. In some implementations, one or more actions could be ranked and/or otherwise prioritized in order to identify a most suitable action to initialize in response to the spoken utterance. Prioritizing the one or more actions can be based on content of the spoken utterance, current, past, and/or expected usage of one or more applications, contextual data that characterizes a context in which the user provided the spoken utterance, whether the action corresponds to an active application or not, and/or any other information that can be used to prioritize one or more actions over one or more other actions. In some implementations, action(s) corresponding to the active application (i.e., an application that is executing in a foreground of a graphical user interface) can be prioritized and/or ranked higher than actions corresponding to non-active applications.

In some implementations, the application data can include structured data in the form of, for example, a schema, which can characterize a variety of different actions capable of being performed by a particular application that the application data corresponds to. For example, an alarm system application can be associated with particular application data characterizing a schema that includes a variety of different entries characterizing one or more different actions capable of being performed via the alarm system application. Furthermore, a thermostat application can be associated with other application data characterizing another schema that includes entries characterizing one or more other actions capable of being performed via the thermostat application. In some implementations, despite these two applications being different, the schema of each application can include an entry that characterizes an “on” action.

Each entry can include properties of the “on” action, a natural language description of the “on” action, a file pathway for data associated with the “on” action, and/or any other information that can be relevant to an application action. For example, an entry in the schema for the alarm system application can include a pathway for a file to execute in order to secure the alarm system. Furthermore, a separate entry in a schema for the thermostat application can include information characterizing a current status of the thermostat application and/or a thermostat that corresponds to the thermostat application. Information provided for each entry can be compared to content of a spoken utterance from the user, and/or contextual data corresponding to a context in which the spoken utterance was provided to the user. This comparison can be performed in order to rank and/or prioritize one or more actions over other actions.

For example, contextual data generated by the computing device can characterize one or more applications that are currently active at the computing device. Therefore, because the alarm system application was active at the time the user provided the spoken utterance, the “on” action characterized by the schema for the alarm system application can be prioritized over the “on” action characterized by the schema for the thermostat application. In some instances, as described in more detail below, the schema or other application data for an application which is running in the foreground of a multitasking environment may be prioritized over schemas/other application data for all other applications on the device. In this scenario, when the application which is running in the foreground changes from a first application to a second application, the application data for the second application may take priority over the application data for the first application when determining which of the applications should be used to implement the action specified in the user request.

In some implementations, application data for a particular application can characterize a variety of different actions capable of being performed by that particular application. When a spoken utterance or other input is received by an automated assistant, multiple different actions characterized by the schema can be ranked and/or prioritized. Thereafter, a highest priority action can be executed in response to the spoken utterance. As an example, the user can be operating a restaurant reservation application, which can be associated with application data characterizing a schema that identifies a variety of different actions capable of being performed by the restaurant application. While the user is interacting with the restaurant reservation application, for example when the restaurant reservation application is running in the foreground of a multitasking environment, the user can navigate to a particular interface of the application for selecting a particular restaurant at which to make reservations. While interacting with the particular interface of the application, the user can provide a spoken utterance such as, “Make the reservation for 7:30 P.M.”

In response to receiving the spoken utterance, an automated assistant can cause the spoken utterance to be processed in order to identify one or more actions to initialize based on the spoken utterance. For example, content data characterizing natural language content of the received spoken utterance can be generated at a device that received the spoken utterance. The content data can be compared to application data that characterizes a variety of different actions capable of being performed via the restaurant reservation application. In some implementations, the application data can characterize one or more contextual actions capable of being performed by the application while the application is exhibiting a current status, and/or one or more global actions capable of being performed by the application regardless of the current status.

Based on the comparison, one or more actions identified in the application data can be ranked and/or prioritized in order to determine a suitable action to initialize in response to the spoken utterance. For example, the application data can characterize a “reservation time” action and a “notification time” action, each capable of being performed by the restaurant reservation application. A correspondence between content of the received spoken utterance and the actions can be determined in order to prioritize one action over the other. In some implementations, the application data can include information further characterizing each action, and this information can be compared to the content of the spoken utterance in order to determine a strength of correlation between the content of the spoken utterance and the information characterizing each action. For instance, because the content of the spoken utterance includes the term “reservation,” the “reservation time” action can be prioritized over the notification time action.

In some implementations, a status of the application can be considered when selecting an action to initialize in response to a spoken utterance. For example, and in accordance with the previous example, when the user provided the spoken utterance, the restaurant reservation application may not have established a stored reservation at the time the spoken utterance was received. When the content of the spoken utterance is compared to the application data, the status of the restaurant reservation application can be accessed and also compared to the application data. The application data can characterize an application status for one or more actions capable of being performed by the reservation application. For example, the reservation time action can be correlated to a draft reservation status, whereas the notification time action can be correlated to a stored reservation status. Therefore, when there is no stored reservation, but the user is creating a draft reservation, the reservation time action can be prioritized over the notification time action, at least in response to the user providing the spoken utterance, “Make the reservation for 7:30 P.M.”

In another example, the user can be operating a thermostat application, which can be associated with application data characterizing a schema that identifies a variety of different actions capable of being performed by the thermostat application. While the user is interacting with the thermostat application, for example when the thermostat application is running in the foreground of a multitasking environment, the user can navigate to a particular interface of the application for selecting a particular time at which to set an indoor temperature to a particular value. While interacting with the particular interface of the application, the user can provide a spoken utterance such as, “Set temperature to 68 degrees at 7:00 A.M.”

In response to receiving the spoken utterance, an automated assistant can cause the spoken utterance to be processed in order to identify one or more actions to initialize based on the spoken utterance. For example, content data characterizing natural language content of the received spoken utterance can be generated at a device that received the spoken utterance. The content data can be compared to application data that characterizes a variety of different actions capable of being performed via the thermostat application. In some implementations, the application data can characterize one or more contextual actions capable of being performed by the application while the application is exhibiting a current status, and/or one or more global actions capable of being performed by the application regardless of the current status.

Based on the comparison, one or more actions identified in the application data can be ranked and/or prioritized in order to determine a suitable action to initialize in response to the spoken utterance. For example, the application data can characterize a “set temperature” action and a “eco mode” action, each capable of being performed by the thermostat application. A correspondence between content of the received spoken utterance and the actions can be determined in order to prioritize one action over the other. In some implementations, the application data can include information further characterizing each action, and this information can be compared to the content of the spoken utterance in order to determine a strength of correlation between the content of the spoken utterance and the information characterizing each action. For instance, because the content of the spoken utterance includes the term “set temperature,” the “set temperature” action can be prioritized over the “eco mode” action.

In some implementations, a variety of different actions from different applications can be considered when responding to a spoken utterance that is provided by a user when an active application is executing at a computing device. For instance, when a user provides a spoken utterance such as, “Read my new message,” while a non-messaging application (e.g., a stock application) is being rendered, the automated assistant can interpret the spoken utterance as being most correlated to an automated assistant action of reading new email messages to the user. However, when the user provides the spoken utterance when a social media application is executing in the background of the non-messaging application, the automated assistant can determine whether a status of the social media application is associated with a “new message.” If the status and/or context of the background application is associated with a “new message,” the automated assistant can initialize performance of a message-related action via the background application. However, if the status and/or context of the background application is not associated with a “new message,” the automated assistant can perform the automated assistant action of reading any new email messages to the user, and/or if there are no new email messages, the automated assistant can provide a response such as, “There are no new messages.”

By providing an automated assistant that can initialize other application actions in this way, other corresponding applications would not need to be pre-loaded with modules for voice control, but, rather, can rely on the automated assistant for ASR and/or NLU. This can conserve client-side resources that might otherwise by consumed by having multiple different applications pre-loaded with ASR and/or NLU modules, which can consume a variety of different computational resources. For instance, operating multiple different applications that each have their own respective ASR and/or NLU modules can consume processing bandwidth and/or storage resources. Therefore, utilization of the techniques discussed herein can eliminate waste of such computational resources. Furthermore, these techniques allow for a single interface (e.g., a microphone and/or other interface for interacting with an automated assistant) to control an active application, a background application, and/or an automated assistant. This can eliminate waste of computational resources that might otherwise be consumed launching separate applications and/or connecting with remote servers to process inputs.

The above description is provided as an overview of some implementations of the present disclosure. Further description of those implementations, and other implementations, are described in more detail below.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations may include a system of one or more computers and/or one or more robots that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

