BACKGROUND

Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as “automated assistants” (also referred to as “digital agents,” “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “assistant applications,” “conversational agents,” etc.). For example, humans (which when they interact with automated assistants may be referred to as “users”) may provide commands and/or requests to an automated assistant using spoken natural language input (i.e. utterances), which may in some cases be converted into text and then processed, and/or by providing textual (e.g., typed) natural language input. An automated assistant responds to a request by providing responsive user interface output, which can include audible and/or visual user interface output.

In many cases, users provide spoken utterances to cause an automated assistant to perform various actions. However, extensive computational and/or network resources can be utilized in processing spoken utterances to perform automated assistant actions. For instance, an automated assistant action of turning on a smart light can be accomplished by transmitting a corresponding command to the smart light directly, or to a third-party server that processes the command, then transmits a corresponding command to the smart light. However, in performing such an action in response to a spoken utterance of “Assistant, turn on smart light X”, speech-to-text processing is performed on audio data that captures the spoken utterance, natural language processing is performed based on the resulting text, and then the corresponding command is determined. Further, where speech-to-text and/or natural language processing is performed remotely, audio data and/or other data will need to be transmitted over one or more networks.

SUMMARY

Implementations set forth herein relate to causing an automated assistant to perform one or more actions responsive to a user interacting with a peripheral assistant control device. The peripheral assistant control device can be paired (e.g., via Bluetooth or other communications protocol) with an assistant computing device that includes an automated assistant interface and that alone and/or through interaction with remote automated assistant components, enables performance of various automated assistant functions. In response to user interaction with the peripheral assistant control device, output can be provided by the peripheral assistant control device to the paired assistant computing device. In response to receiving the output, the paired assistant computing device can initiate the performance of one or more automated assistant actions/functions. Implementations described herein may serve to increase the utility of peripheral assistant control devices such that fewer peripheral assistant control devices may be required to enable performance of a given number of automated assistant functions. In addition, some implementations may facilitate/reduce the complexity of configuration of the peripheral assistant control devices. Furthermore, since the peripheral assistant control devices may reduce the amount of speech recognition processing (and associated bandwidth usage etc.) that is required by the automated assistant system, increasing the utility/functionality of the peripheral assistant control devices may serve to further reduce the processing, bandwidth etc. that is utilized by the automated assistant system as a whole.

As described herein, the automated assistant action(s) performed can be based on user interface input previously provided by the user in configuring the peripheral assistant control device. Moreover, the action(s) performed can vary in dependence on one or more conditions, such as which of multiple users is determined to be currently active, where the peripheral assistant control device is currently located (which can optionally be inferred based on which of multiple assistant computing devices the peripheral assistant control device is paired with), and/or the current state of one or more smart devices and/or other devices (e.g., as determined based on a device topology). As also described herein, the automated assistant action(s) performed can include one or more actions that extend beyond action(s) that were specifically requested by a user during configuration of the peripheral assistant control device.

The peripheral assistant control device itself can lack direct access to an automated assistant, and may lack, or contain less powerful versions of, various hardware resources that are present on the paired assistant computing device. For example, the peripheral assistant control device can lack a microphone and a Wi-Fi module, whereas the paired assistant computing device includes both. Further, while the peripheral assistant control device can include a processor, it can be less powerful than processor(s) included at the paired assistant computing device. In some implementations, the peripheral assistant control device can include a virtual button provided at a graphical user interface, and/or any other interface capable of presenting a virtual button. Alternatively, or additionally, the peripheral assistant control device can include a physical button with one or more user interfaces through which a user can provide one or more different inputs. The one or more different inputs can cause the peripheral assistant control device to provide, over a wireless communications channel, one or more different outputs, respectively.

One or more users can provide user interface input to configure a peripheral assistant control device. For example, user interface input can be provided at an assistant computing device and/or at the peripheral assistant control device to initially pair the assistant computing device and the peripheral assistant control device. Further user interface input can be provided to explicitly assign certain automated assistant function(s) to an interaction with the peripheral assistant control device. For example, a user can provide a spoken utterance of “Assistant, when I push this button turn on the living room lights”. In various implementations, although the user only specifies certain automated assistant function(s) to be assigned to interaction with the peripheral assistant control device, one or more additional automated assistant function(s) can be further automatically suggested (and assigned if affirmative user interface input received), or further automatically assigned (without first suggesting).

As one example, for the spoken utterance of “Assistant, when I push this button turn on the living room lights” the automated assistant action of causing an on/off property of the “living room” lights to be commanded to an “on” state is explicitly indicated by the spoken utterance and, as a result, can be assigned to the peripheral assistant control device. Although not explicitly indicated by the spoken utterance, the further automated assistant function of causing an on/off property of the “living room” lights to be commanded to an “off” state can be further assigned. Further, causing an on/off property of the “living room” lights to be commanded to an “on” state can be further assigned to a first condition of the on/off property of the “living room” lights being in an “off” state—and causing an on/off property of the “living room” lights to be commanded to an “off” state can be further assigned to a second condition of the on/off property of the “living room” lights being in an “on” state. Thus, in response to receiving output from the peripheral assistant control device in response to an interaction, a current state of the on/off property of the “living room” lights can be determined (e.g., by querying the lights and/or using a device topology (with recently updated device states)), and the lights commanded to either the on state or the off state, in dependence on whether the current condition is the first condition or the second condition.

As another example, for the spoken utterance of “Assistant, when I push this button play some bluegrass music” the automated assistant action of causing the paired assistant computing device (and/or other device(s), if specified) to play “bluegrass music” is explicitly indicated by the spoken utterance and, as a result, can be assigned to the peripheral assistant control device. Although not explicitly indicated by the spoken utterance, the further automated assistant function of causing a currently playing “bluegrass song” to be replaced by playing of another “bluegrass song” (e.g., a “next song” automated assistant function) can be further assigned. Further, the play “bluegrass music” automated assistant function can be further assigned to a first condition of the assistant computing device not currently rendering “bluegrass music” (or, optionally, not rendering any music) and the “next song” automated assistant function can be further assigned to a second condition of “bluegrass music” (or, optionally, any music) being currently rendered at the assistant computing device. Thus, in response to receiving output from the peripheral assistant control device in response to an interaction, the play “bluegrass music” automated assistant function can be initiated by the assistant computing device if the first condition is satisfied, whereas the “next song” automated assistant function can instead be initiated by the assistant computing device if the second condition is satisfied.

Accordingly, even though a user can provide the same user interaction (e.g., a button press) at the peripheral assistant control device on two separate occasions, and the same output transmitted from the peripheral assistant control device to a paired assistant computing device in response, different automated assistant functions can resultantly occur at the two separate occasions, due to different conditions at the two separate occasions.

Automatically suggesting and/or automatically assigning one or more additional automated assistant function(s) to the peripheral assistant control device results in various technical benefits. For example, it enables the additional automated assistant functions to be performed responsive to certain user interactions with the peripheral automated assistant device. The additional automated assistant functions can be performed responsive to receiving the output provided by the peripheral automated assistant device, and determining certain condition(s) are satisfied—which can be significantly more efficient (computationally and/or network wise) than processing a corresponding spoken utterance. Also, for example, it can reduce an amount of user input that would otherwise be required by the user to explicitly assign the additional automated assistant functions. Moreover, automatically altering, in dependence on a current condition, the automated assistant function that is initiated responsive to receiving output from the peripheral assistant control device also results in various technical benefits. For example, initiating the correct automated assistant function for the current condition ensures that processor and/or network resources utilized to perform that automated assistant function are not wasted. For instance, techniques that do not initiate different automated assistant functions for different conditions can wastefully generate and transmit commands that are duplicative and will be ignored (e.g., generating and transmitting a “turn lights on” command even though they are already “on”).

In some additional and/or alternative implementations, different automated assistant functions can be assigned to the same interaction with the peripheral automated assistant control device, and which of the automated assistant functions is imitated in response to a given instance of the interaction can depend on where the peripheral assistant control device is currently located. For example, first automated assistant function(s) of turning on and/or off the “living room” lights can be initiated when the peripheral assistant control device is located in or near the “living room”, whereas a second automated assistant function of “play some music on basement devices” can be initiated when the peripheral assistant control device is located in or near the “basement”. Various techniques can be utilized to determine where the device is currently located. For example, where the device is currently located can be based on which of multiple assistant computing devices the button is currently paired with. The pairing of the peripheral assistant control device can be switched between different assistant computing devices responsive to user interface input (at the respective assistant computing device and/or at the peripheral assistant control device), or can be automatically switched so as to pair with the assistant computing device having the strongest (or only detectable) communications signal (e.g., Bluetooth) with the peripheral assistant control device.

In some implementations, explicit user interface input can be provided that indicates which automated assistant function(s) should be performed for which locations of the peripheral assistant control device. For example, the user can provide a spoken utterance such as, “Assistant, when the interface device is in the living room, have it control the television, but when the interface device is in the kitchen, have it control playback of podcasts.” In response, the automated assistant can establish corresponding configuration settings for the peripheral assistant control device. For example, using data that is indicative of a location of the peripheral assistant control device, the automated assistant can correlate an output of the peripheral assistant control device to a particular function based on where the peripheral assistant control device is indicated to be located. For example, the peripheral assistant control device can be determined to be in the “living room” based on it being paired with an automated assistant computing device that is assigned, in a device topology, to a “living room” location. Additional and/or alternative techniques can be utilized to determine location of the peripheral assistant control device, such as techniques that determine the location based on network strength data, data from another device that the peripheral assistant control device is paired with or otherwise in communication with, data identifying a particular device that the peripheral assistant control device is paired with or otherwise in communication with, elevation data, motion data, audio data, image data, and/or any other data from which a location of a device can be determined.

In some implementations, user interface input can specify an automated assistant function of control of a type of smart device, and can specify that it is be assigned to the peripheral assistant control device. However, the user interface input may not specify the particular smart devices to be controlled. For example, the user can provide a spoken utterance such as, “Assistant, have the interface device control turn the lights on and off for the room it is in”. In some of those implementations, particular automated assistant function(s) can be assigned, that are each directed to control of one or more particular smart devices of the type, and that are each associated with a corresponding condition (such as location of the peripheral assistant control device). For example, a first automated assistant function of turning on/off the “living room” lights can be assigned to a first condition of the peripheral assistant control device being in the “living room”, and a second automated assistant function of turning on/off the “kitchen” lights can be assigned to a second condition of the peripheral assistant control device being in the “kitchen”. When the peripheral assistant control device is determined to be in the “kitchen” (e.g., based on being paired with a “kitchen” assistant computing device), output received responsive to interaction with the peripheral assistant control device can cause control of the on/off state of lights that are also assigned to the “kitchen” (e.g., in a device topology). On the other hand, when the peripheral assistant control device is determined to be in the “living room” (e.g., based on being paired with a “living room” assistant computing device), output received responsive to interaction with the peripheral assistant control device can cause control of the on/off state of lights that are also assigned to the “living room” (e.g., in a device topology).

Assigning different automated assistant functions in dependence on where the peripheral assistant control device is currently located results in various technical benefits. For example, it enables the functionality of the peripheral assistant control device to be dynamically adapted based on location, without necessitating that a user provide resource consuming further interface input to explicitly reconfigure the functionality each time the peripheral assistant control device is moved. Also, for example, it enables a single peripheral assistant control device, and optionally a single interaction, to have multiple functionalities in dependence on location—thus obviating the need for multiple separate peripheral assistant control devices for the multiple functionalities.

In some additional and/or alternative implementations, different automated assistant functions can be assigned to the same interaction with the peripheral automated assistant control device, and which of the automated assistant functions is imitated in response to a given instance of the interaction can depend on which of multiple users is currently an active user. For example, a first automated assistant function(s) of turning on and/or off the “living room” lights can be initiated when a first user is an active user, whereas second automated assistant function(s) of “play some music” can be initiated when a second user is an active user. Various techniques can be utilized to determine which of multiple users is currently an active user. For example, the user that most recently interacted with an assistant computing device with which the peripheral assistant control device is paired, can be considered the currently active user (optionally contingent on that recent interaction being within the last 5 minutes, or within another time threshold). A particular user can be determined to have interacted with an assistant computing device based on verifying that user based on, for example, matching spoken input from the interaction with a stored speaker embedding for the user. Additional and/or alternative verification techniques can be utilized, such as a user provided pin, user provided passphrase, and/or other biometric identification techniques (with permission from a corresponding user).

Assigning different automated assistant functions in dependence on which user is a currently active user of the peripheral assistant control device results in various technical benefits. For example, it enables the functionality of the peripheral assistant control device to be dynamically adapted based on a currently active user, without necessitating that a user provide resource consuming further interface input to explicitly reconfigure the functionality each time the active user changes. Also, for example, it enables a single peripheral assistant control device, and optionally a single interaction, to have multiple functionalities in dependence on user—thus obviating the need for multiple separate peripheral assistant control devices for the multiple functionalities.

In some implementations, conditions for when the peripheral assistant control device is assigned control of particular functions can be established automatically in response to an automated assistant detecting a presence of the peripheral assistant control device. In some implementations, the peripheral assistant control device can communicate via Bluetooth and, when the user turns on the peripheral assistant control device, the device can pair with an existing client device owned by the user, or otherwise associated with the user. In response to the peripheral assistant control device pairing with the existing client device, the existing client device can invoke an automated assistant, which can provide a natural language output for directing further configuration of the peripheral assistant control device. For example, the automated assistant can cause the existing client device to render an audible natural language output, such as, “George, I've detected a new interface device in the living room. Would you like the new interface device to control the lights in the living room?” This natural language output can be generated based on the automated assistant identifying the location of the peripheral assistant control device, other existing devices in the living room that can be controlled via the automated assistant, and/or any other device that is in communication with the existing client device.

In response to the natural language output from the automated assistant, the user can confirm the configuration proposed by the automated assistant and/or request a different configuration for the peripheral assistant control device. For example, the user can provide a natural language input such as, “Assistant, when I am in the living room, have the interface device control the television, but when I am not in the living room, have the interface device control the lights in the living room” (the “interface device” can refer to the peripheral assistant control device). In response to receiving the natural language input from the user, the automated assistant can cause the peripheral assistant control device to be conditionally assigned control over the identified functions. Specifically, in order for the peripheral assistant control device to be assigned control over the lights, the automated assistant should determine that the user is not present in the living room. Additionally, in order for the peripheral assistant control device to be assigned control over the television, the automated assistant should determine that the user is actually present in the living room. Such determinations can be based on data that is accessible to the existing client device, the automated assistant, and/or any other associated device, as discussed herein.

The above description is provided as an overview of some implementations of the present disclosure. Further description of those implementations, and other implementations, are described in more detail below.

In some implementations, a method implemented by one or more processors as including operations such as determining, based on data that is accessible via a computing device, whether a first user or a second user is a currently active user of a peripheral assistant control device, wherein the peripheral assistant control device is in wireless communication with the computing device and an automated assistant is accessible via the computing device. The operations can further include receiving, from the peripheral assistant control device, other data that is generated in response to a touch interface of the peripheral assistant control device receiving a touch input, wherein receipt of the other data that is generated in response to interaction with the touch interface causes the automated assistant to perform a function, selected from at least a first automated assistant function and a second automated assistant function, according to whether the first user or the second user is determined to be the currently active user of the peripheral assistant control device. The operations can further include, when the first user is determined to be the currently active user: causing, in response to receiving the other data and determining that the first user is the currently active user, the automated assistant to execute the first automated assistant function. The operations can further include, when the second user is determined to be the currently active user: causing, in response to receiving the other data and determining that the second user is the currently active user, the automated assistant to execute the second automated assistant function, which is different from the first automated assistant function.

In some implementations, the first automated assistant function is also executable in response to a first corresponding spoken request to an automated assistant interface of the computing device, and the second automated assistant function is also executable in response to a second corresponding spoken request to the automated assistant interface of the computing device. In some implementations, the method can further include, prior to determining the currently active user of the peripheral assistant control device: assigning the first automated assistant function and the second automated assistant function to the peripheral assistant control device in response to the first user and/or the second user providing one or more spoken utterances requesting the automated assistant to assign the first automated assistant function and the second automated assistant function to the peripheral assistant control device. In some implementations, the method can further include, prior to determining the currently active user of the peripheral assistant control device: assigning the first automated assistant function to the peripheral assistant control device and to the first user based on determining that a first spoken utterance requests assignment of the first automated assistant function to the peripheral assistant control device, and based on determining that the first spoken utterance was provided by the first user. In some implementations, the method can further include assigning the second automated assistant function to the peripheral assistant control device and to the second user based on determining that a second spoken utterance requests assignment of the first automated assistant function to the peripheral assistant control device, and based on determining that the second spoken utterance was provided by the second user. In some implementations, the touch interface of the peripheral assistant control device includes a touch sensor that is responsive to tactile gestures. In some implementations, the peripheral assistant control device provides the other data exclusively in response to the interaction with the touch interface, and the touch interface is exclusively responsive to one or more tactile gestures.

In some implementations, the data that is accessible via the computing device comprises audio data detected via one or more microphones of the computing device prior to receiving the other data from the peripheral assistant control device; and wherein determining, based on the data that is accessible via the computing device, whether the first user or the second user is the currently active user of the peripheral assistant control device comprises: determining, based on processing the audio data, that the audio data matches a stored voice embedding for the first user; and determining that the first user is the active user based on the audio data matching the stored voice embedding for the first user. In some implementations, determining that the first user is the active user is further based on: determining that the audio data was detected within a threshold amount of time of receiving the other data from the peripheral assistant control device. In some implementations, wherein the first automated assistant function comprises altering a property of a smart device, and wherein causing the automated assistant to execute the first automated assistant function comprises: determining a current state of the property of the smart device, and generating, based on the current state, a command that causes the property of the smart device to transition to an alternate state that differs from the current state; and transmitting the command to the smart device or to a server associated with the smart device. In some implementations, the method can further include receiving input that requests assignment, to the peripheral assistant control device, of only a function that causes the property of the smart device to transition to the current state; and automatically assigning the first automated assistant function to the smart device in response to receiving the input, wherein the first automated assistant function alters the property of the smart device between at least the current state and an alternate state instead of only causing the property of the smart device to transition to the current state.

In yet other implementations, a method implemented by one or more processors, the method comprising: determining, based on data that is accessible via a computing device, whether a peripheral assistant control device is located at a first area or a second area, wherein the peripheral assistant control device is in wireless communication with the computing device, and an automated assistant is accessible via the computing device. The method can further include receiving, from the peripheral assistant control device, other data that is generated in response to a touch interface of the peripheral assistant control device receiving a touch input, wherein interaction with the touch interface causes the automated assistant to perform a function, selected from at least a first automated assistant function and a second automated assistant function, according to whether the peripheral assistant control device is determined to be located at the first area or the second area. The method can further include, when the peripheral assistant control device is determined to be at the first area: causing, in response to receiving the other data and determining that the peripheral assistant control device is determined to be at the first area, the automated assistant to execute the first automated assistant function. The method can further include, when the peripheral assistant control device is determined to be at the second area: causing, in response to receiving the other data and determining that the peripheral assistant control device is determined to be at the second area, the automated assistant to execute the second automated assistant function, which is different from the first automated assistant function.

In some implementations, the data that is accessible via the computing device indicates that the peripheral assistant control device is directly paired, via a wireless communication protocol, with only the computing device, and wherein determining whether the peripheral assistant control device is located at the first area or the second area comprises determining that the peripheral assistant control device is located at the first area based on the paired computing device being assigned, in a device topology, to the first area. In some implementations, the method can further include, prior to determining whether the peripheral assistant control device is located at the first area or the second area: receiving input that requests assignment, to the peripheral assistant control device, of control of a type of smart device. In some implementations, the method can further include generating, based on the input, the first automated assistant function to cause control of one or more first smart devices of the type based on the first smart devices being of the type and being assigned, in a device topology, to the first area. In some implementations, the method can further include generating, based on the input, the second automated assistant function to cause control of one or more second smart devices of the type based on the second smart devices being of the type and being assigned, in a device topology, to the second area.

In some implementations, the method can further include, prior to determining whether the peripheral assistant control device is located at the first area or the second area: assigning the first automated assistant function to the first area based on user interface input that request assignment of the first automated assistant function to the peripheral assistant control device and to the first area. In some implementations, the method can further include, prior to determining whether the peripheral assistant control device is located at the first area or the second area: assigning the second automated assistant function to the second area based on user interface input that request assignment of the second automated assistant function to the peripheral assistant control device and to the second area. In some implementations, the computing device is one device of multiple different assistant computing devices defined in a device topology that is accessible to the automated assistant, and wherein determining whether the peripheral assistant control device is located at the first area or the second area includes determining which of the multiple assistant computing devices the peripheral assistant control device is paired with and/or is most proximal to.

In yet other implementations, a method implemented by one or more processors is set forth as including operations such as receiving audio data corresponding a spoken utterance provided by a user to an automated assistant interface of a computing device, wherein the computing device provides access to an automated assistant and is connected to a peripheral assistant control device. The method can further include determining, based on the audio data, that the spoken utterance includes a request for controlling one or more functions in response to an interaction with a touch interface of the peripheral assistant control device, wherein the peripheral assistant control device provides an output to the computing device, or another computing device, in response to the interaction with the touch interface. The method can further include determining, based on the audio data, that the spoken utterance at least partially identifies the one or more functions for correlating to the output of the peripheral assistant control device. The method can further include identifying, based on data that is associated with the one or more functions, one or more other functions that are executable via the computing device or the automated assistant, wherein the spoken utterance is void of content specifying the one or more other functions. The method can further include generating correlation data that correlates the output of the peripheral assistant control device with the one or more functions when at least a first condition is satisfied, and correlates the output of the peripheral assistant control device with the one or more other functions when at least a second condition is satisfied.

In some implementations, the method can further include determining, subsequent to generating the correlation data, that the peripheral assistant control device has provided the output in response to the user interacting with the touch interface. In some implementations, the method can further include determining, based on other data that is associated with the user, whether the first condition or the second condition is satisfied. In some implementations, the method can further include, when the first condition is satisfied: causing the one or more functions to be executed in response to the user interacting with the touch interface of the peripheral assistant control device. In some implementations, the method can further include, when the second condition is satisfied: causing the one or more other functions to be executed in response to the user interacting with the touch interface of the peripheral assistant control device.

In some implementations, the one or more functions include an initiating media action that causes rendering of media, and the first condition is satisfied when media is not being rendered at the computing device and/or another device associated with the user, and wherein the second condition is satisfied when media is being rendered at the computing device or the other device associated with the user. In some implementations, the one or more other functions include a next media item action that causes a new media item to be rendered in replacement of a current media item currently being rendered. In some implementations, the one or more functions, when executed, cause the computing device or another device to operate according to a setting, and the second condition is satisfied when the computing device or the other device is operating according to the setting, and wherein the one or more other functions, when executed, cause the computing device or the other device to operate according to another setting, and the first condition is satisfied when the computing device or the other device is operating according to the other setting.

In some implementations, the one or more functions, when executed, cause the computing device to operate according to the setting, and wherein the one or more other functions, when executed, cause the other device to operate according to the other setting. In some implementations, the one or more functions cause the computing device to transition operating modes from a first operating mode to a second operating mode, and wherein the one or more other functions cause the computing device to transition the operating modes from the second operating mode to the first operating mode. In some implementations, the first operating mode causes the computing device to provide a first output via a modality of the computing device and the second operating mode causes the computing device to provide a second output, which is different from the first output, via the modality of the computing device. In some implementations, the first condition includes a property of a smart device being in a first state, and wherein the second condition includes the property of the smart device being in a second state. In some implementations, the one or more functions comprise causing the property of the smart device to transition from the first state to the second state, and wherein the one or more other functions comprise causing the property of the smart device to transition from the second state to the first state. In some implementations, the method can further include determining that the peripheral assistant control device has provided the output in response to the user interacting with the touch interface; In some implementations, the method can further include determining, in response to the output being provided and based on a device topology that defines a current state of the property of the smart device, whether the first condition or the second condition.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations may include a system of one or more computers and/or one or more robots that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

