BACKGROUND

Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as “automated assistants” (also referred to as “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” “virtual assistants,” etc.). For example, humans (which when they interact with automated assistants may be referred to as “users”) may provide commands, queries, and/or requests (collectively referred to herein as “queries”) using free form natural language input which may include vocal utterances converted into text and then processed and/or typed free form natural language input.

Different users may control and/or have permission to access different resources via an automated assistant. For example, a trusted user may have permission to cause the automated assistant to perform various actions that untrusted users may not necessarily be able to perform, such as controlling smart appliances (e.g., lights, thermostats, locks, etc.). As another example, an automated assistant may have the ability to present various content to users. Some of this content, such as personal documents, calendar data, etc., may be protected, and the automated assistant will only present it upon recognition/authentication of the requesting user. The automated assistant may deny unrecognized or unauthorized users access to the same protected content. Other content may not necessarily be protected, but may be inappropriate for some users. For example, children may be prevented from asking an automated assistant to initiate playback of content for which parental discretion is advised.

Configuring existing automated assistants to distinguish between individuals currently requires manual interaction with a graphical user interface, e.g., to turn on a “voice match” feature. Consequently, other individuals who may lack sufficient knowledge or motivation to access such an interface may never enroll with the voice match feature. Further, using voice matching technology (also referred to herein as “speaker recognition processing”) by itself to distinguish between different individuals may not be sufficiently reliable, especially in a noisy environment, or in scenarios in which multiple speakers have similar voices/accents.

In many cases, before the automated assistant can interpret and respond to a user's request, it must first be “invoked,” e.g., using predefined oral invocation phrases that are often referred to as “hot words” or “wake words.” Thus, many automated assistants operate in what will be referred to herein as a “limited hot word listening state” or “default listening state” in which they are always “listening” to audio data sampled by a microphone for a limited (or finite, or “default”) set of hot words. Any utterances captured in the audio data other than the default set of hot words are ignored. Once the automated assistant is invoked with one or more of the default set of hot words, it may operate in what will be referred to herein as a “speech recognition state” wherein for at least some time interval after invocation, the automated assistant performs speech-to-text (“STT”) processing of audio data sampled by a microphone to generate textual input, which in turn is semantically processed to determine and fulfill a user's intent. Existing automated assistants typically can only be invoked using one or more of the default hot words, which are the same regardless of whether the requesting user is recognized.

SUMMARY

Techniques are described herein to improve security processes in automated assistants by selective enrollment, wherein enrollment with an automated assistant by a user unlocks one or more features of the automated assistant that were unavailable to the user prior to enrollment. In particular techniques are described for dialog-based enrollment of individual users for single- and/or multi-modal recognition by an automated assistant, as well as determining how to respond to a particular user's request based on the particular user being enrolled and/or recognized. Rather than requiring operation of a graphical user interface for individual enrollment, dialog-based enrollment enables users to enroll themselves (or others) by way of a human-to-computer dialog with the automated assistant. Example implementations described herein improve security by placing users in different trust levels, wherein access to functions of the automated assistant that might be considered sensitive, such as controlling appliances and/or accessing protected data, are restricted based on trust level.

Techniques described herein may often (but not exclusively) be employed on what will be referred to herein as “assistant devices.” Assistant devices are computing devices that are designed primarily to facilitate human-to-computer dialogs between user(s) and an automated assistant. Many assistant devices take the form of standalone interactive speakers, which are becoming increasingly ubiquitous. Standalone interactive speakers are often placed in heavily trafficked locations such as kitchens, living rooms, conference rooms, etc., and are often interacted with by multiple different people (e.g., family members, co-workers, guests, etc.).

While it may be possible to enroll any individual who ever interacts with the assistant device, doing so may have various disadvantages. Information that is usable to recognize individuals (referred to herein as “distinguishing attributes of a user”), such as “voice profiles” and “visual profiles” described herein, may need to be stored locally on the assistant device. In many cases, assistant devices are resource-constrained (e.g., relatively little memory and/or processing power) for economical and/or technical reasons. Thus, storing data indicative of distinguishing attributes of a large number of users may require an inordinate amount of the assistant device's limited memory. Moreover, if a particular individual's interaction with the assistant device is likely to be minimal—e.g., they are a transient guest visiting a household in which the assistant device is deployed—it may be wasteful to enroll that individual. Additionally, such a transient guest may not wish to be enrolled, e.g., because they'd prefer that data indicative of their distinguishing attributes not be maintained on someone else's assistant device.

Accordingly, in various implementations, before a heretofore unknown individual is enrolled with an automated assistant using techniques described herein, the automated assistant may determine whether the individual satisfies one or more “automated assistant enrollment criteria.” These criteria may include, for instance, the individual engaging in a threshold number of distinct human-to-computer dialog sessions with the automated assistant on the same assistant device or on one or more of a coordinated ecosystem of computing devices controlled by a “host” user (e.g., person who controls/configures the ecosystem of devices, such as the owner, head of household, etc.). Additionally or alternatively, these criteria may include a threshold number of dialog turns occurring between the individual and the automated assistant.

To determine whether the individual satisfies the automated assistant enrollment criteria, one or more distinguishing attributes of the individual may be identified, e.g., based on signals generated by one or more hardware sensors integral with or otherwise communicatively coupled with the assistant device. These hardware sensors may include, for example, vision sensors (e.g., cameras, passive infrared sensors, etc.), pressure sensors (e.g., microphone, ultrasonic sensors, etc.), or wireless receivers that can detect wireless signals (e.g., Wi-Fi, Bluetooth, ZigBee, Z-Wave, RFID, visual indicia) emitted by a mobile device carried by the individual. Based on the identified distinguishing attribute(s) of the individual, historical interaction data (e.g., a log maintained by or on behalf of the automated assistant) may be analyzed to identify one or more prior human-to-computer dialog sessions in which the same individual exchanged dialog with the automated assistant (e.g., using the same assistant device or another computing device in the same coordinated ecosystem of computing devices). Based on the analysis, if the automated assistant enrollment criteria are satisfied, then the automated assistant may initiate what will be referred to herein as a “human-to-computer dialog enrollment routine.”

During a human-to-computer dialog enrollment routine, the automated assistant may provide natural language output that includes instructions for the user to perform various actions that facilitate recognition of the user in the future, e.g., by capturing and storing data indicative of distinguishing attributes of the user. For example, during a visual enrollment routine, the automated assistant may instruct the user to reposition the user's face to multiple different poses, and capture, using a vision sensor, the user's face in the multiple different poses. Capturing multiple diverse and distinct images of the user's face may enable creation of a “visual profile” of the user. This visual profile may be usable to detect/recognize the user in the future, e.g., using facial recognition processing. In some implementations, the visual profile of the user may include some combination of the multiple images and/or some combination of features extracted from the multiple images. Additionally or alternatively, in some implementations, the visual profile may be “baked into” a machine learning classifier/model (e.g., a convolutional neural network). Future images may be applied as input across such a classifier/model, and output generated based on the model may be indicative of the user's identity.

In addition to or instead of visual enrollment, in some implementations, an automated assistant configured with selected aspects of the present disclosure may trigger a voice enrollment routine. During a voice enrollment routine, the automated assistant may instruct the user to speak various words and/or phrases. These words or phrases may be selected for their suitability for generating a “voice profile” of the user. The user's utterances of these words/phrases may be used to build the voice profile, and the voice profile may be useable, e.g., in conjunction with subsequently captured audio data, to perform speaker recognition. Like visual profiles, voice profiles can take various forms, such as data indicative of utterances of the user, features extracted from utterances of the user, parameters of a trained machine learning classifier/model, etc.

In some implementations, once the user enrolls, an identity of the user (e.g., a unique identifier, the user's name, etc.) may be stored in one or more databases (e.g., local to the assistant device or in remote cloud infrastructure) in association with data indicative of one or more of the distinguishing attributes of the user. In some implementations, these distinguishing features may be stored as an “enrollment” embedding generated from vision/pressure sensor data that is applied as input across a machine learning model, such as various types of neural networks. These distinguishing feature(s) may be detected later, e.g., during subsequent human-to-computer dialog sessions between the user and the automated assistant, and used to determine the user's identity, in effect authenticating the user to the automated assistant.

For example, in some implementations, vision sensor data and/or pressure sensor data that captures a not-yet-recognized individual may be applied across the same machine learning model to generate a new embedding. The new embedding may be compared to a previously-stored enrollment embeddings (e.g., determining Euclidian distances between them) to determine whether the proximate individual's embedding is sufficiently similar to one of the existing enrollment embeddings to reliably match the proximate individual to the previously enrolled individual.

In various implementations, enrollment by the user may unlock one or more features of the automated assistant that were unavailable to the user prior to enrollment. These features may be available to the user upon recognition of the user based on their enrollment. For example, in some implementations, one or more dynamic or custom hot words may be activated such that the user, when later recognized (e.g., using speaker and/or facial recognition), is able to invoke the automated assistant using these dynamic hot words, in addition to or instead of the default hot words that are available to unrecognized users. Additionally or alternatively, in various implementations, other features of (or associated with) the automated assistant may be unlocked to an enrolled user. These may include, for instance, the ability to cause automated assistant to perform one or more actions that might not otherwise be performable at the request of an unenrolled user, such as altering one or more parameters of a smart appliance, accessing protected data, order goods and/or services, making payments, and so forth.

In some implementations, recognition of an enrolled user may generate a confidence measure. For example, in some implementations, users may be requested to enroll for both speaker recognition and facial recognition. Later, when such a user approaches an assistant device, it may be the case that hardware sensors of or associated with the assistant device are unable to capture sufficient data to perform both speaker and facial recognition with a high degree of confidence, e.g., because the camera is malfunctioning, the computing device lacks a camera altogether, the user mumbles or speaks too softly to enable confident speaker recognition, etc. In some such implementations, the user may nonetheless be recognized with a limited degree of confidence. Such a user may be granted limited access to various features of the automated assistant, instead of the full access they might be granted if they were recognized with greater confidence.

For example, in some implementations, detected users may be placed in “levels” or “bins” of trust. A first, or highest, level of trust may be assigned to a user for which facial and/or speaker recognition (or recognition based on a user-emitted wireless signal) generated a confidence measure that satisfies a first threshold. A second level of trust may be assigned to a user for which facial and/or speaker recognition generated a confidence measure that satisfies a second threshold, but not the first threshold. A third level of trust may be assigned to a user for which facial and/or speaker recognition generated a confidence measure that satisfies a third threshold, but not the first or second thresholds. And so on until the user is not recognized at all, in which case they may be assigned a lowest level of trust (e.g., “guest”). In various implementations, each level of trust may unlock various features of the automated assistant for the user. For example, a user assigned to the first level of trust (i.e., voice/speaker recognition generated a relatively high confidence measure) may gain unfettered access to functions of the automated assistant that might be considered sensitive, such as controlling appliances and/or accessing protected data. By contrast, a user assigned to the lowest level of trust may be considered a “guest,” and may be denied access altogether or only allowed access to features of the automated assistant that are considered non-sensitive (e.g., weather forecast, sports scores, movies schedules, etc.).

In some implementations, a method performed by one or more processors is provided that includes: executing an automated assistant at least in part on one or more computing devices; processing one or more sensor signals generated by one or more hardware sensors integral with one or more of the computing devices; based on the processing, identifying one or more distinguishing attributes of a user within range of the one or more hardware sensors; based on the one or more distinguishing attributes, analyzing historical interaction data to identify one or more prior human-to-computer dialog sessions in which the user exchanged dialog with the automated assistant using one or more of the computing devices; based on the identified one or more prior human-to-computer dialog sessions, determining that the user satisfies an automated assistant enrollment criterion; and in response to determining that the user satisfies the automated assistant enrollment criterion, engaging in a human-to-computer dialog enrollment routine in which the user is solicited to enroll with the automated assistant, wherein enrollment by the user includes storing an identity of the user in one or more databases in association with data indicative of one or more of the distinguishing attributes of the user, and wherein enrollment by the user unlocks one or more features of the automated assistant that were unavailable to the user prior to enrollment.

In various implementations, the one or more hardware sensors may include a vision sensor, and the one or more distinguishing attributes may include a visual profile of the user. In various implementations, the visual profile of the user may be usable in conjunction with sensor signals generated by the vision sensor or another vision sensor to identify the user using facial recognition processing.

In various implementations, the one or more hardware sensors may include a microphone, and the one or more distinguishing attributes may include a voice profile of the user. In various implementations, the voice profile of the user may be usable in conjunction with a sensor signal generated by the microphone or another microphone to identify the user using speaker recognition processing.

In various implementations, the one or more distinguishing attributes may include a signal emitted by a mobile device carried by the user. In various implementations, the one or more unlocked features may include activation of one or more hot words that are usable to invoke the automated assistant. In various implementations, the one or more unlocked features may include one or more responsive actions performable by the automated assistant. In various implementations, the one or more unlocked features may include access to protected content.

In various implementations, the automated assistant enrollment criterion may include a threshold number of human-to-computer dialog sessions between the user and the automated assistant using one or more of the computing devices. In various implementations, the automated assistant enrollment criterion may include a threshold number of dialog turns in human-to-computer dialog sessions between the user and the automated assistant using one or more of the computing devices.

In various implementations, the human-to-computer dialog routine may include: instructing the user to reposition the user's face to multiple poses; and capturing, using a vision sensor, the user's face in the multiple poses.

In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.

It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

