BACKGROUND

Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as “automated assistants” (also referred to as “digital agents,” “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “conversational agents,” etc.). For example, humans (which when they interact with automated assistants may be referred to as “users”) may provide commands and/or requests using spoken natural language input (i.e., utterances) which may in some cases be converted into text and then processed, and/or by providing textual (e.g., typed) natural language input.

In some instances, an automated assistant may respond to a particular user according to whether a voice signature of the user corresponds to a restricted account that is accessible upon voice verification. However, as a number of users near an automated assistant device increases, background noise and other voices can be detected by the automated assistant, causing the automated assistant to lose track of whether the particular user is still present. This can be especially problematic when a particular user has been authenticated via voice verification but subsequently leaves a vicinity of the automated assistant device. As a result, a display panel of the automated assistant device may continue rendering restricted data that the particular authenticated user was accessing, despite the user no longer being present. Furthermore, when an automated assistant continues to render data for an initial user that has left a vicinity of the automated assistant device, such rendering can waste computational resources. Moreover, wasteful latency can be exhibited by the automated assistant device when another user enters the vicinity of the automated assistant device, but the automated assistant device continues rendering data with the expectation that the initial user is still present.

SUMMARY

Implementations set forth herein relate to control of computing hardware to render particular outputs. The particular outputs may be rendered by the computing hardware for particular authenticated users and may comprise access-restricted data. The computing hardware, which is discussed below in the context of a computing device, is used to determine users which are active at the computing device by, for example, controlling a camera to capture aspects of an environment around the device. An active-user queue for dynamically adjusting access permissions at the device according to, for example, a detected presence of one or more users near the particular computing device is established. Initially, when no user is considered active, the active-user queue can be established in response to a user entering a field of view of a camera of the computing device. Multiple image frames captured via the camera can be processed to determine whether an initial user is within the viewing window and/or whether any other users have entered the viewing window. The initial user can be considered active as long as they are exclusively detected in the viewing window. However, should the other user enter the viewing window, the other user can be assigned priority access to the device when one or more conditions are satisfied. By assigning access priorities in this way, access-restricted data, associated with particular users, can be protected from users that may access the computing device subsequent to the initial user leaving the field of view of the camera. In some implementations, this can ensure that systems, such as machinery or other safety-sensitive apparatuses, which are controllable using the access-restricted data rendered for a particular user of the device cannot be taken over and controlled by unauthorized users once the particular user is no longer active at, or otherwise controlling, the device. Furthermore, latency that is otherwise introduced via authentication procedures can be mitigated, thereby preserving computational resources that can be wasted during periods of latency. Yet further, in various implementations, personalized or otherwise access-restricted content for a given user can be prefetched (e.g., from a remote server) and/or pre-compiled (e.g., prepared for rendering on-device) responsive to the given user entering the user queue for the device, but before the given user has priority in the user queue for the device. In response to the given user subsequently having priority in the user queue for the device, the prefetched and/or precompiled personalized content or otherwise access-restricted data can be quickly rendered to the given user. This reduces the latency with which the personalized content or otherwise access-restricted data is rendered to the given user.

In some implementations, a first user and a second user can be present in a home (or other environment such as a manufacturing environment) that includes a computing device that provides access to an automated assistant. The first user can enter a vicinity of the computing device, for example by entering a room (such as a kitchen) that includes the computing device, in order to view access-restricted data accessible via an account of the first user. The access-restricted data may, for example, be associated with a particular automated apparatus in the environment. An example of the access-restricted data is a schedule that is accessible via an account of the first user, such as an editable schedule of operations to be performed by the automated apparatus. The automated apparatus in this context may be a safety or security-sensitive apparatus, such as a cleaning robot, a 3D printer, a smart appliance or, e.g., an automated manufacturing apparatus, to which access control may be restricted (e.g. restricted from access by minors or untrained persons). When the first user enters a field of view of a camera of the computing device, the computing device can process one or more image frames in order to verify that the first user has entered the field of view. When the first user has been verified by the computing device, an automated assistant can cause the computing device to render access-restricted content that is associated with an account of the first user. For example, prior to the first user entering the vicinity of the computing device, a display panel of the computing device can be rendering unrestricted content and/or non-personalized content, such as a non-editable version of the operations schedule for the automated apparatus that is accessible via the computing device, or e.g. publicly available images and/or other information. In response to the computing device verifying the presence of the first user within the field of view, the computing device can render at least some amount of content that is personalized to the user, such as access-restricted content. An example is the access-restricted editable schedule of operations for the automated apparatus and/or an editing interface for altering/amending the schedule. Optionally, and as described herein, the rendering of the content that is personalized to the user can be further responsive to determining one or more conditions are present that indicate the user is paying attention to the computing device (which may be a client device), such as determining, based on vision data from a vision component of the computing device, that a gaze of the user is directed to the computing device (e.g., for at least a threshold duration), that the user is within a threshold distance of the computing device, that a head pose and/or body pose of the user is directed to the computing device, and/or other condition(s) are present. In addition to, or as an alternative to, the access-restricted content referred to above, the computing device may also render other access-restricted content (e.g., a portion of a schedule of the user) and/or content that is publicly accessible, but determined to be of relevance to the user.

In some implementations, in response to verifying the presence of the first user, the automated assistant can cause queue data to be generated for assigning the priority to the first user at the device, as well as defining an amount of time that the user will be assigned priority at the device. The amount of time can be N minutes, seconds, and/or milliseconds, and/or any time value that can be assigned to a timer. As the timer is counting down, the amount of time for the first user to be assigned priority can eventually expire, which can result in no user having priority at the device, thereby causing the computing device to no longer render any access-restricted content. However, with prior permission from the first user and/or the second user, image data generated using the camera of the computing device can be processed to determine whether the first user remains present in the field of view of the camera. When the automated assistant determines that the first user is still in the field of view of the camera, the automated assistant can cause the timer to be reset (e.g., reset back to the initial amount of time “N”), in order to extend an amount of time that the user is assigned priority at the device.

While the first user is assigned the priority and the timer has not expired, the second user can enter the field of view of the camera by entering the vicinity of the device (e.g. the kitchen area). One or more image frames characterizing the second user can be processed in order to verify the presence of the second user, and, based on the verification, the second user can be assigned priority at the device over the first user. In some implementations, an indication of the change in priority can be indicated by the computing device using one or more graphical elements at the display interface, for example, avatars that are displayed over the top of each other, indicating that a particular user has priority over another user at the device. As changes in assignment of priority occur, the computing device can cease rendering personalized content associated with the first account of the first user, and, instead, render personalized content associated with a second account of the second user. For example, when the second user is assigned priority, the computing device can render access-restricted content for the second user. An example of such access-restricted content for the second user is content similar to that described above for the first user, which may allow the second user to control operational aspects of one or more automated apparatuses for which the second user is authorized. Additionally or alternatively, the computing device can render other graphical content, such as content characterizing one or more events identified on a stored calendar created by the second user.

In some implementations, the other timer for the second user can be initialized when the second user is verified and/or is assigned priority at the device over the first user. Furthermore, the other timer can be reset each time the second user is determined to be remaining in the field of view of the camera. However, when the second user is determined to no longer be in the field of view of the camera, the other timer can optionally be set to a reduced time (e.g., the other timer can have M minutes, seconds, and/or milliseconds deducted from the other timer, where “M” is any number value). When the other timer expires as a result of the second user not remaining in the field of view of the camera, the computing device can cease rendering access-restricted content associated with the second account of the second user. Accordingly, personalized content can be at least selectively rendered to corresponding users when they have priority in the user queue. Moreover, utilization of the countdown timer prevents prematurely assigning an alternative user as having priority in the user queue due to the current priority user only temporarily leaving a field of view of a vision component of the computing device and/or the current priority user only temporarily not being detected in image frames (despite being present in the field of view) due to turning his/her face, being occluded, and/or other condition(s). In these and other manners, personalized content for the alternative user is only rendered responsive to the countdown timer expiring for the current priority user and the alternative user becoming the new priority user. This prevents, for example, too frequent switching between rendering different personalized content, which can unnecessarily consume resources of the computing device and/or make rendered content more difficult to comprehend.

In some implementations, while the first user and the second user are determined to be within the field of view of the camera, a touch input can be provided to a touch display of the computing device. In order to determine how to respond to the touch input, the computing device can compare a distance, between the first user and the camera, to a proximity threshold. Furthermore, the computing device can also compare another distance, between the second user and the camera, to the proximity threshold. When the distance of the first user is determined to satisfy the proximity threshold, and the other distance of the second user is determined to also satisfy the proximity threshold, the automated assistant can respond to the touch input by providing further content that is based on the second account of the second user. This can be based on an assumption that the second user has acquiesced to the touch input because of their proximity to the camera. Alternatively, when the distance of the first user is determined to satisfy the proximity threshold, but the other distance of the second user is determined to not satisfy the proximity threshold, the automated assistant can respond by providing other content that is not based on the second account of the second user (e.g. by not rendering access-restricted content associated with the second account of the second user). This operation can be based on another assumption that the second user is not acquiescing to the touch input, because they are not located within a particular distance of the camera. Furthermore, when neither the first user or the second user are located at a particular distance that satisfies the proximity threshold, the automated assistant can assume that the touch input was provided by someone that is located outside the field of view of the camera. Therefore, in response to such a touch input, the automated assistant can respond by providing the other content that is not based on the second account of the user (and/or e.g. by providing content that is not based on the first account of the first user). When distance of a user is determined, various techniques can be utilized (e.g. at the computing device) to determine the distance. For example, the distance can be estimated based on distance between eyes of the user in image(s), a relative head size of the user in image(s), and/or other attribute(s) of the user in image(s).

In some implementations, priority at the computing device can change according to which user is determined to be paying attention to the computing device. For instance, when the first user is in the vicinity of the computing device (e.g. the kitchen area in which the device is located) without any other persons and is determined, by the computing device, to be viewing the computing device, the first user can be assigned priority at the computing device. Furthermore, a timer for the priority for the first user can also be initialized at this time. Thereafter, as long as the timer has not expired, the automated assistant can render access-restricted content associated with the first account of the first user. However, during this time, the first user may turn away from the computing device in order to attend to something else in the environment (e.g., turning off a stove). One or more images frames captured via the camera can be processed to determine that the first user has turned away, and, in response, an amount of time remaining on the timer can be reduced by M minutes, seconds, milliseconds, and/or any other time value, where “M” is any number value.

In some implementations, when the first user or the second user is assigned priority at the computing device and/or determined to be paying attention to the computing device, the automated assistant can cause “action items” to be rendered by the computing device. An action item can correspond to an action that the computing device, the automated assistant, and/or any other application is requesting the user to perform and/or respond to. For example, an action item can correspond to a software update that needs to be confirmed by a user before the software update is installed. Additionally, or alternatively, the action item can correspond to a notification that the automated assistant, the computing device, and/or any other application is requesting a user to acknowledge. For example, when the first user is assigned priority and also determined to be paying attention to the computing device and/or the automated assistant, the automated assistant can cause a message notification to be rendered at a touch display panel of the computing device. The message notification can operate as an “action item” because, by rendering the message notification, the automated assistant is requesting the first user to perform an “action” of acknowledging the message notification by, for example, tapping on an area of the touch display panel in which the message notification is being rendered. When the first user is determined to be no longer paying attention to the computing device and/or the “action item,” the amount of time on the timer for the priority assigned to the first user can be reduced. Furthermore, in response to determining that the first user is no longer paying attention to the computing device and/or the action item, the automated assistant can cause the “action item” to no longer be rendered at the touch display of the computing device.

As used herein, “access-restricted content” of a given user is non publicly accessible content that is stored in computer memory in association with an account of the given user, and over which the user has access-control. In addition to those mentioned above, some non-limiting examples include content from a private calendar of the user, content from an email account of the user, and reminders from a private account of the user. In various implementations, the given user can provide permission to enable all or portions of (e.g., a snippet of or an abstraction of) access-restricted content to be at least selectively rendered via a shared computing device. For example, the given user can provide permission to enable rendering of at least portions of access-restricted content at a given client device generally, or only when the given user is recognized, by the given client device, using facial recognition and/or voice recognition. However, in some of those implementations access to additional portion(s) of the access-restricted content and/or the ability to edit the access-restricted content can only be provided in response to verifying that the given user is the one requesting access to the additional portion(s) and/or requesting the editing.

For example, voice recognition can be utilized to confirm that a spoken utterance, that requests access to the additional portion(s), is provided by the given user. Also, for example, facial recognition can be utilized to confirm that a touch-event at the client device (e.g., a touch selection of an abstraction of user-restricted content of the given user), that requests access to the additional portion(s) and/or the editing, is actually from the given user. For instance, as described herein, facial recognition and distance estimation can be utilized in combination to determine that the given user is within a threshold distance of the client device at a time of the touch event and optionally that any other user(s) are not within the threshold distance (or another threshold distance). As one particular example, the abstraction of the access-restricted content can be a graphical element that includes only “upcoming appointment tomorrow @ 12:00”. If a touch-event directed to the graphical element is received, additional details of the appointment (e.g., a location of the appointment, a title for the appointment, etc.) will only be rendered in response if it is determined that the given user is within a threshold distance of the client device at a time of the touch event and optionally that any other user(s) are not within the threshold distance (or another threshold distance). In these and other manners, security of access-restricted content is maintained. Additionally, security is maintained while enabling rendering of aspects of the access-restricted content, with permission from the given user, under various conditions. Thus, utility of shared client devices can be improved by rendering aspects of access-restricted content from multiple accounts of the client devices (optionally at the same time), providing corresponding users with efficient overviews of the access-restricted content. Moreover, more detailed access to the access-restricted content can be enabled through touch-input (e.g., a single touch input) and utilization of facial recognition and user distance techniques disclosed herein. Such a single touch-input can be quicker to provide than a spoken utterance, thereby making the user-assistant interaction more efficient and decreasing the overall duration of the user-assistant interaction. Such a single touch-input is also effective in noisy conditions where speaker recognition can fail and/or also enables users with speech difficulties to effectively access access-restricted content.

As used herein, “personalized content” of a given user encompasses access-restricted content, but also encompasses content that is publicly accessible, but is rendered to a given user based on determining that the content is relevant to the given user. Public content can be determined to be relevant to a given user based on comparing attribute(s) of the content with attribute(s) of the given user, which can be based on historical interactions of the given user indicating interest in those attribute(s). For example, a given user may have previously viewed multiple articles related to Sports Team A and be provided with personalized content, that is a new publicly accessibly article, based on the new publicly accessible article being about Sports Team A. As with access-restricted content, in various implementations, the given user can provide permission to enable all or portions of personalized content (generally, or personalized content that is not also access-restricted content) to be at least selectively rendered via a shared computing device. For example, the given user can provide permission to enable rendering of at least portions of personalized content at a given client device generally, or only when the given user is recognized, by the given client device, using facial recognition and/or voice recognition.

The above description is provided as an overview of some implementations of the present disclosure. Further description of those implementations, and other implementations, are described in more detail below.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations may include a system of one or more computers that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

