BACKGROUND

Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as “automated assistants” (also referred to as “digital agents,” “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “assistant applications,” “conversational agents,” etc.). For example, humans (which when they interact with automated assistants may be referred to as “users”) may provide commands and/or requests to an automated assistant using spoken natural language input (i.e. utterances), which may in some cases be converted into text and then processed, and/or by providing textual (e.g., typed) natural language input.

Some automated assistants can be used to control internet of things (“IoT”) devices. However, many IoT devices can come with their own corresponding applications. As a result, a user that has acquired multiple different IoT devices may be tasked with installing a variety of third party applications on their personal computing devices. Because applications can typically require memory, network bandwidth, updates, and/or other resources, having a variety of similar applications on a single device can be inefficient. Although a user may rely on an automated assistant to mediate interactions between the user and a particular IoT device, such interactions can frequently require spoken utterances from the user to be converted from audio to text. In order to effectuate conversion of audio to text, audio data oftentimes must be sent over a network, thereby consuming significant network bandwidth. Furthermore, mediating interactions in such a way can prove unreliable when a user is surrounded by other people that are talking, or is otherwise in environment with background noise. As a result, a dialog session between a user and an automated assistant may waste computational resources and network resources when a request for an IoT device to perform a particular function is ultimately not identified, or otherwise not fulfilled.

SUMMARY

The present disclosure is generally directed to methods, apparatus, and computer-readable media (transitory and non-transitory) for providing composite graphical assistant interfaces through which to control multiple different devices. As network connectivity of common devices, such as lightbulbs and other appliances, becomes more prevalent, the ways of interfacing with such devices over a network can increase. This can be in part due to an increasing number of manufacturers making similar devices. For instance, there may be dozens of smart lightbulb manufacturers, and a majority of the lightbulb manufacturers may offer applications or interfaces from which to control their respective lightbulbs. As a result, a user that installs lightbulbs from different manufacturers may download different applications to their personal computing devices for controlling the lightbulbs. This can waste memory, as installing multiple applications can occupy a substantial amount of device memory, and can also waste network bandwidth, as having multiple applications on a device can cause the device to check for updates, share data, and/or push notifications for a respective manufacturer. In order to overcome these and/or other limitations, some implementations described herein provide various techniques for an automated assistant to generate composite graphical assistant interfaces for controlling various devices, despite the devices being provided by different third parties.

A composite graphical assistant interface can be an interface compiled from one or more graphical elements in order to effectuate control of one or more devices. The composite graphical assistant interface can be employed by an automated assistant, which can act as an intermediary between the user and any connected devices. For example, a user can provide a spoken utterance such as, “Assistant, turn on my kitchen lights.” The spoken utterance can be received at a client device, which can include one or more microphones and a display panel, and in response, the automated assistant can cause any lights in a kitchen of a home of the user to turn on. Furthermore, the automated assistant can identify a type of “internet of things” (IoT) device that the user has requested the automated assistant to control, and identify one or more control parameters associated with the type of IoT device. The control parameters can correspond to slot values, which can be supplied to the IoT device for modifying a state of the IoT device, therefore the control parameters can be identified by analyzing acceptable slot values for the IoT device. For instance, an IoT device such as a smart lightbulb can receive slot values such as “on” and “off,” which can correspond to an illumination state of the lightbulb, and/or a color values such as “red,” “green,” and “blue,” which can correspond to color states of the lightbulb. Furthermore, the IoT device can receive slot values such as any number between 0 and 100, which can correspond to brightness states for the lightbulb. When the automated assistant has identified all possible slot values and/or resulting states of an IoT device, the automated assistant can identify suitable graphical control elements from which to compile a composite graphical assistant interface.

For instance, in some implementations, the automated assistant can identify a total number of slot values for a given function of a particular IoT device and correlate the function to a graphical control element. The automated assistant can select a graphical control element that has two states, similar to a two-position electrical switch, when selecting the graphical control element for a function with two slot values (e.g., “on” and “off”). Furthermore, the automated assistant can select a graphical control element that has color-coded inputs, such as an array of colored buttons, when selecting the graphical control element for a function with names of colors as slot values (e.g., “red,” “green,” and “blue,”). Moreover, the automated assistant can select a graphical control element that resembles a dial control, such as a potentiometer, when selecting the graphical control element for a function that has at least a threshold number of slot values (e.g., more than two, more than five, more than ten, etc.).

When the automated assistant has selected one or more suitable graphical control elements from which to generate the composite graphical assistant interface, the automated assistant can then cause the client device to display the composite graphical assistant interface. For instance, in response to receiving the spoken utterance, “Assistant, turn on my kitchen lights,” the automated assistant can cause the kitchen lights to turn on and also cause the client device to display the composite graphical assistant interface with the selected graphical control elements. The composite graphical assistant interface would therefore be displayed at the display panel of the client device with graphical control elements such as a two-position switch for turning on and off the lights, a color-coded control input for controlling a color of the lights, and/or dial control for adjusting a brightness of the lights. In these and other manners, rather than having to access a unique application for each respective device and/or each respective device manufacturer, the automated assistant can generate similar interfaces with familiar graphical control elements the user can engage with to modify states of certain devices. Also, by compiling composite graphical assistant interfaces in such a manner, network bandwidth can be preserved, as the client device would not necessarily have to communicate with a remote server to process spoken utterances.

Moreover, in various implementations, and as described above and elsewhere herein, a graphical assistant interface to control function(s) of a connected device(s) is rendered in response to a spoken command to control the connected device(s). For example, in response to a spoken input of “adjust the thermostat to 72 degrees”, an interactive assistant interface can be rendered that reflects the change of the thermostat set point to 72 degrees, that enables touch interactions to further adjust the controlled set point function, and that enables touch interactions to further adjust additional function(s) (e.g., humidity functions, fan on/off functions, etc.). In these and other manners, the presented assistant interface further assists the user in continuing performance of the control of the connected device. Further, the presented interface can enable more network resource and/or computational resource efficient further adjustment of the functions as compared to, for example, further adjustments effectuated through further spoken inputs.

In some implementations, states of a device can be reflected at the composite graphical assistant interface presented by the automated assistant. For example, in response to receiving the spoken utterance, “Assistant, please turn on my kitchen lights,” the automated assistant can cause the aforementioned composite graphical assistant interface to be presented at a display panel of a client device. Furthermore, after the automated assistant has caused the kitchen lights to turn on, the automated assistant can identify a resulting state of the kitchen lights (e.g., State: [lights:“on”], . . . ) and configure the graphical control element of the composite graphical assistant interface to represent the state of the kitchen lights. For instance, the graphical control element for controlling the kitchen lights can be presenting in an “on” state in order to reflect the change of state that the user caused by issuing the spoken utterance to the automated assistant. In some implementations, other states of the device can also be reflected at the composite graphical assistant interface. For example, when a brightness level of the kitchen lights is at 50% when the user provides the spoken utterance, the automated assistant can cause the dial that controls the brightness level of the kitchen lights to be positioned at 50%. In other words, in response to the automated assistant receiving a particular spoken utterance that modifies a state of an IoT device, the automated assistant can retrieve (or receive) additional state information from the IoT device in order to accurately reflect the state of the IoT device to a user. Furthermore, in response to the automated assistant receiving the particular spoken utterance, the automated assistant can determine other functions capable of being performed by the IoT device. This can allow the automated assistant to compile a more comprehensive composite graphical assistant interface that is reflective of multiple controls that can employed for modifying a state of the IoT device.

By providing a graphical assistance interface that is reflective of a user's previous interactions with an automated assistant, improved efficiency of user interaction may be facilitated. In particular, related assistant tasks, such as control of IoT devices, may be facilitated with a reduced set of interactions, thereby reducing the consumption of computer and network resources associated with those interactions. Moreover, in some examples, an initial interaction may be carried out through an alternative modality—such as through recognition of an utterance—which is particularly suited to certain controls (e.g. binary selection of an on or off state) but may not be as efficient for related functions (such as fine control of a preferred level or balance). Appropriate modalities for each function can therefore be presented in an automated and efficient manner. Furthermore, the provision of a graphical interface may enhance discoverability of functions related to an initial user input.

The above description is provided as an overview of some implementations of the present disclosure. Further description of those implementations, and other implementations, are described in more detail below.

In some implementations, a method implemented by one or more processors is set forth as including operations such as determining that a spoken utterance, detected via an automated assistant interface, includes a request to modify a state of a connected device that is linked with a client device. The spoken utterance is provided by a user. The method can further include identifying, in response to receiving the request, a function that corresponds to the received request; causing, based on identifying the function, the connected device to perform the function that corresponds to the received request; and determining a current state of the connected device. The current state of the connected device can be affected by performance of the function. The method can also include identifying one or more other functions capable of being performed by the connected device. The current state of the connected device can be further affected by performance of the one or more other functions. The method can further include determining, based on identifying the function and the one or more other functions, a first graphical control element for controlling the function performed by the connected device and a second graphical control element for controlling the one or more other functions. The method can also include determining, based on the current state of the connected device, a configuration for each graphical control element of the first graphical control element and the second graphical control element; and causing, in response to receiving the request, a display panel associated with the client device to provide a composite graphical assistant interface for controlling the connected device. The composite graphical assistant interface can include the first graphical control element and the second graphical control element, which are arranged according to the determined configuration for each graphical control element.

In some implementations, the method can include determining that the user has selected, at the display panel, the second graphical control element, which corresponds to an additional function of the one or more other functions; causing, in response to determining that the user has selected the second graphical control element, the connected device to perform the additional function, wherein performance of the additional function causes the connected device to exhibit a modified state; and causing, in response to the connected device performing the additional function, the second graphical control element to reconfigure according to the modified state.

In some implementations the display panel includes a touch interface through which the user can directly control the first graphical control element and the second graphical control element using one or more gesture inputs. In some implementations, the method can include determining a third graphical control element for controlling an additional function of the one or more functions, and a selectable link configured to link to the third graphical control element. Causing the display panel associated with the first client device to provide the composite graphical assistant interface can include causing the selectable link to be provided at the composite graphical assistant interface.

In some implementations, the method can include determining that the user has selected the selectable link at the display panel; and causing, in response to determining the user has selected the selectable link, the composite graphical assistant interface to provide the third graphical control element.

In some implementations, wherein the client device lacks an integrated display panel, and the method can further include determining that the client device lacks the integrated display panel; and identifying, in response to determining that the client device lacks the integrated display panel, an additional connected device that incorporates the display panel and is linked to the client device. Causing the display panel to provide the composite graphical assistant interface can include causing the display panel of the additional connected device to render the composite graphical assistant interface. In some implementations, identifying the additional connected device includes: selecting the additional connected device, from a plurality of candidate devices, based on a stored device topology that indicates a proximity of the additional connected device relative to the client device.

In other implementations, a method implemented by one or more processors is set forth as including operations such as determining that a spoken utterance, detected via an automated assistant interface, includes a request for a connected device to perform a function. The spoken utterance can be provided by a user and the connected device is linked to a client device. The method can also include identifying, in response to receiving the request, the function that corresponds to the received request; causing, based on identifying the function, the connected device to perform the function that corresponds to the received request; and determining, based on historical interaction data that is accessible to the client device, one or more other functions previously performed by the connected device according to a previous request from the user. The method can further include determining, based on identifying the one or more other functions, a graphical control element for controlling the one or more other functions; and causing, in response to receiving the request, a display panel associated with the client device to provide a composite graphical assistant interface for controlling the connected device. The composite graphical assistant interface can include the graphical control element.

In some implementations, the method can include determining a profile for the user based on a voice characteristic associated with the spoken utterance. Causing the composite graphical assistant interface to be provided at the client device can include causing the composite graphical assistant interface to include a plurality of graphical control elements that are arranged based on the determined profile.

In some implementations, the plurality of graphical control elements omits at least one available graphical control element based on a preference of the user indicated by the determined profile. In some implementations, the method can include requesting, from a remote server device, state data that indicates a current state of the connected device. The current state of the connected device can be affected by the one or more functions previously performed by the connected device. The method can further include receiving the state data from the remote server device; and determining a configuration for the graphical control element based on the state data. The graphical control element can be arranged at the composite graphical assistant interface according to the determined configuration.

In some implementations, the method can include receiving, in response to requesting the state data, command data corresponding to an additional function for modifying the current state of the connected device. In some implementations, the method can include determining an additional graphical control element for controlling the additional function and a selectable link. Causing the display panel associated with the client device to provide the composite graphical assistant interface can include causing the composite graphical assistant interface to include the selectable link. The selectable link can be configured to reveal the additional graphical control element in response to receiving a user selection of the selectable link.

In yet other implementations, a method implemented by one or more processors, is set forth as including operation such as determining that a user has selected a graphical control element at a first composite graphical assistant interface provided at a display panel associated with a client device. The first composite graphical assistant interface can be a modality through which the user can control a connected device that is in communication with the client device. The method can further include causing, in response to determining that the user has selected the graphical control element, the connected device to perform a function that corresponds to the graphical control element. The connected device can be configured to perform a plurality of different functions. The method can also include determining, in response to determining that the user has selected the graphical control element, a user preference that characterizes a historical inclination of the user to control one or more other functions of the plurality of different functions. The method can further include identifying, based on determining the user preference, the one or more other functions of the plurality of different functions; and determining, based on identifying the one or more other functions, one or more graphical control elements for controlling the one or more other functions of the connected device. The method can also include causing, in response to determining that the user has selected the graphical control element, the display panel associated with the client device to supplant the first composite graphical assistant interface with a second composite graphical assistant interface that includes the one or more graphical control elements.

In some implementations, the method can include determining a state of the connected device. The state can be affected by performance of the function and the one or more other functions. The method can also include determining, based on the state of the connected device, an arrangement for each graphical control element of the one or more graphical control elements. In some implementations, the state of the connected device can be based on a resulting condition effectuated by performance or nonperformance of each function of the one or more other functions.

In some implementations, the state of the connected device can be based on a resulting condition effectuated by performance of the function and the second composite graphical assistant interface includes a graphical interface element that indicates the resulting condition. The second composite graphical assistant interface can exclude the graphical control element that corresponds to the function. In some implementations, the user preference characterizes the historical inclination of the user to control one or more other functions of the plurality of different functions based at least on a previous interaction between the user and an automated assistant, which is accessible via an automated assistant interface of the client device. In some implementations, the previous interaction corresponds to a dialog session between the user and the automated assistant, prior to the user selecting the graphical control element; wherein, during the dialog session, the user provided a request to cause the connected device to perform the function.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations may include a system of one or more computers and/or one or more robots that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

