TECHNICAL FIELD

This relates generally to analysis of visual information, including but not limited to, recognizing persons in a video stream.

BACKGROUND

Video surveillance produces a large amount of continuous video data over the course of hours, days, and even months. Such video data includes many long and uneventful portions that are of no significance or interest to a reviewer. In some existing video surveillance systems, motion detection is used to trigger alerts or video recording. However, using motion detection as the only means for selecting video segments for user review may still produce too many video segments that are of no interest to the reviewer. For example, some detected motions are generated by normal activities that routinely occur at the monitored location, and it is tedious and time consuming for a reviewer to manually scan through all of the normal activities recorded on video to identify a small number of activities that warrant special attention. In addition, when the sensitivity of the motion detection is set too high for the location being monitored, trivial movements (e.g., movements of tree leaves, shifting of the sunlight, etc.) can account for a large amount of video being recorded and/or reviewed. On the other hand, when the sensitivity of the motion detection is set too low for the location being monitored, the surveillance system may fail to record and present video data on some important and useful events.

It is a challenge to accurately identify and categorize meaningful segments of a video stream, and to convey this information to a user in an efficient, intuitive, and convenient manner. Human-friendly techniques for discovering, categorizing, and notifying users of events of interest are in great need.

SUMMARY

Accordingly, there is a need for systems and/or devices with more efficient, accurate, and intuitive methods for entity (e.g., person) and event identification, categorization, and presentation. Such systems, devices, and methods optionally complement or replace conventional systems, devices, and methods for entity and event identification, categorization, and/or presentation.

Users of home monitoring systems can potentially be bombarded with alerts and notifications about unimportant and trivial events if the systems do not employ some recognition/identification and categorization processes. The large number of unnecessary or incomplete information places a larger burden on the users and makes it more difficult to identify and respond to important events. For example, a user who receives motion notifications every time a household member walks in front of the camera may not be able to distinguish the motion notification for a burglar sneaking through the room. Thus, it is beneficial to be able to recognize the type of event that occurred and the persons involved. For example, rather than sending a notification stating that “motion was detected,” the system could send a notification detailing that “an unknown person was detected sneaking through the living room.” The user can see at a glance the nature of the event and request more details (e.g., a clip of the event) and/or generate a response (e.g., alert the authorities).

In addition, recognition of individual people in events enables the systems to associate events with the particular persons involved (e.g., the event could be denoted as including the persons involved). For example, a user may wish to see a summary of all events involving his kids and could bring up a list of all events associated with them. As another example, a user may wish to find a particular event involving a particular person and the system would enable the user to narrow the search to only events involving the particular person.

In one aspect, some implementations include a method for recognizing a person in a video stream. In some implementations, the method includes: (1) obtaining a live video stream; (2) detecting one or more persons in the live video stream; (3) determining, from analysis of the live video stream, personally identifiable information that identifies a first person of the detected one or more persons; (4) determining, based on at least some of the personally identifiable information, that the first person is not a known person to the computing system; (5) in accordance with the determination that the first person is not a known person: (a) storing at least some of the personally identifiable information; and (b) requesting a user to classify the first person; and (6) in accordance with (i) a determination that a predetermined amount of time has elapsed since the request was transmitted and a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, deleting the stored personally identifiable information.

In some implementations, the determination that the first person is not a known person is made by comparing the personally identifiable information (PII) with previously stored PII of one or more known persons. In some implementations, only a preset number of images are stored (e.g., 100, 200, or 300 images). In some implementations, if the preset number of images (e.g., 300) has already been stored for a particular person, a previously stored image is deleted and the new image is saved (e.g., the oldest previously stored image is deleted, or the previously stored image with the lowest determined quality is deleted). In some implementations, if the PII closely matches previously stored PII for the person, the image is not saved (e.g., too cumulative).

In some implementations, the predetermined amount of time is 1 day, 1 week, 2 weeks, or 1 month. In some implementations, the predetermined amount of time is equal to a subscription storage length associated with the computing system.

In some implementations, the classifications include one or more of: a known person, a trusted person, an authorized person, a stranger, an unknown person, etc. In some implementations, the user designates a person as trusted, known, or unknown. In some implementations, the designation for untrusted persons is stored as well. In some implementations, the designation is associated with the image and/or the PII of the first person.

In some implementations, the computing system comprises one or more cameras communicatively coupled to a remote server system. In some implementations, storing at least some of the PII includes storing an image and corresponding PII for comparison with PII of subsequently detected persons. In some implementations, for each detected person, the method includes: (1) storing an image of the detected person and corresponding PII; (2) in accordance with a determination that the detected person is a known person to the computing system, associating the stored image and corresponding feature characterization information (also sometimes called personally identifiable information) with a label for the known person; and (3) in accordance with a determination that the detected person is not a known person to the computing system, generating a new label for the stored image and corresponding feature characterization information.

In some implementations, determining the personally identifiable information includes: (1) selecting one or images of the first person from the live video stream; and (2) characterizing a plurality of features of the first person based on the one or more images. In some implementations, the plurality of features is characterized based on an analysis of only a portion of the image (e.g., the portion containing the person or the person's face). In some implementations, a camera device determines the portion and sends the portion of the image, but not the remainder of the image, to a server system for further analysis and recognition. In some implementations, the method further includes acquiring from the camera a higher quality version of the portion of the image (e.g., with more pixels) and performing the characterization on the higher quality version. In some implementations, the plurality of features includes one or more features that can personally identify the first person. In some implementations, the plurality of features includes one or more facial features of the first person. In some implementations, the method includes: (1) generating a facial signature for the first person based on the characterization of the plurality of features; and (2) determining that the first person is not a known person is based on a comparison of the generated facial signature with one or more previously stored facial signatures.

In some implementations, the method further comprises: (1) identifying a pose of the first person; and (2) in accordance with a determination that an image of the first person in the identified pose has not previously been stored, storing (i) the image of the first person in the identified pose, and (ii) corresponding pose information. In some implementations, the method includes determining and storing quality information for the image, timing information for the image, location information for the image, and the like.

In some implementations, the method further includes: (1) receiving, from the user, a designation that the first person is not a person; and (2) responsive to the designation, storing the designation in association with the personally identifiable information of the first person. In some implementations, the first person is an image of a person (e.g., a picture or a scene on a display). In some implementations, the first person is an animal rather than a person. In some implementations, the first person is a reflection or shadow of a person. In some implementations, future detections of the first person are not notified as a result of the first person being designated as not a person. In some implementations, future detections of the first person are notified as activity not involving a person.

In some implementations, the method further includes, in accordance with the determination that the first person is not a known person, generating a timely notification of detection of an unknown person. In some implementations, a timely notification is a notification sent within 10 seconds, 1 minute, 10 minutes, or 1 hour. In some implementations, the notification comprises a real-time notification sent within 5 seconds, 10 seconds, or 30 seconds. In some implementations, the notification includes a cropped portion of the image of the first person and a high resolution (e.g., 4K) version of the image showing the camera's field of view.

In some implementations, the method further includes: (1) detecting a second person in the live video stream; (2) characterizing a plurality of features of the second person based on one or more images of the second person in the live video stream; and (3) determining whether the second person is a same person as the first person by comparing the plurality of characterized features of the second person with the stored PII of the first person.

In another aspect, in accordance with some implementations a computing system includes: (1) one or more processors; and (2) memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors. In some implementations, the one or more programs include instructions for: (1) obtaining a live video stream; (2) detecting one or more persons in the live video stream; (3) determining, from analysis of the live video stream, personally identifiable information that identifies a first person of the detected one or more persons; (4) determining, based on at least some of the personally identifiable information, whether the first person is a known person to the computing system; (5) in accordance with a determination that the first person is not a known person: (a) storing at least some of the personally identifiable information; and (b) requesting a user to classify the first person; and (6) in accordance with (i) a determination that a predetermined amount of time has elapsed since the request was transmitted and a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, deleting the stored personally identifiable information.

In some implementations, detecting the one or more persons includes detecting the one or more persons within a first predefined threshold time (e.g., 5 seconds, 10, seconds, or 30 seconds).

In some implementations: (1) the one or more programs further include instructions for comparing the personally identifiable information of the first person with stored personally identifiable information of one or more persons previously classified by the user; and (2) determining that the first person is not a known person comprises determining that the personally identifiable information of the first person does not match, within a predetermined threshold, the stored personally identifiable information of the one or more persons previously classified by the user.

In some implementations, the one or more programs further include instructions for: (1) determining that the first person is a known person to the computing system; and (2) in accordance with the determination that the first person is a known person, forgoing requesting the user to classify the first person.

In some implementations, the one or more programs further include instructions for forgoing generating a timely notification in accordance with the determination that the first person is a known person. In some implementations, the first person is determined to be a known person without having gone through an enrollment process (e.g., the person is determined to be a known person based on prior candid/unposed images).

In some implementations, the one or more programs further include instructions for generating a timely notification identifying the first person in accordance with the determination that the first person is a known person. In some implementations, the one or more programs further include instructions for: (1) determining a location of the first person; (2) determining whether the first person has previously been authorized by the user to access the location; and (3) generating the timely notification identifying the first person and the location in accordance with a determination that the first person has not previously been authorized to access the location. For example, a child may be allowed to play in the living room, but should not access the workshop, garage, or home office.

In another aspect, some implementations include a non-transitory computer-readable storage medium storing one or more programs. In some implementations, the one or more programs include instructions, which when executed by a computing system, cause the system to: (1) obtain a live video stream; (2) detect one or more persons in the live video stream; (3) determine, from analysis of the live video stream, personally identifiable information that identifies a first person of the detected one or more persons; (4) determine, based on at least some of the personally identifiable information, whether the first person is a known person to the computing system; (5) in accordance with a determination that the first person is not a known person: (a) store at least some of the personally identifiable information; and (b) request a user to classify the first person; and (6) in accordance with (i) a determination that a predetermined amount of time has elapsed since the request was transmitted and a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, delete the stored personally identifiable information.

In some implementations, the one or more programs further include instructions, which when executed by the computing system, cause the system to: (1) detect the first person by performing facial detection on a first frame in the live video stream; and (2) track the first person through time by analyzing a plurality of video frames in the live video stream, the plurality of video frames subsequent to the first frame. In some implementations, the detecting comprises analyzing the entire frame, whereas the tracking comprises analyzing only a portion of the frames (e.g., only an area at or around the location where the person was detected). In some implementations, the instructions include obtaining one or more additional live video streams; detecting one or more persons in the one or more additional live video streams; and determining whether any of the one or more persons in the one or more additional live video streams are known to the computing system. In some implementations, the instructions include determining, for each detected person of the one or more detected persons in the live video stream and the one or more detected persons in the one or more additional live video streams, whether the detected person is a same person as another of the one or more detected persons in the live video stream or the one or more detected persons in the one or more additional live video streams; and in accordance with a determination that a detected person is a same person as another detected person, tracking the detected person in all corresponding live video streams.

In some implementations, the one or more programs further include instructions, which when executed by the computing system, cause the system to: (1) for each video frame of the first frame and the plurality of video frames, determine an identification quality factor for the video frame; (2) select one or more video frames of the first frame and the plurality of video frames based on the determined identification quality factors; and (3) determine personally identifiable information based on an analysis of the selected one or more images.

In some implementations, the one or more programs further comprise instructions, which when executed by the computing system, cause the system to: (1) determine an activity of the first person based on the tracking of the first person; and (2) send a notification with identification of the first person and the determined activity. In some implementations, the system sends the notification to a user. In some implementations, based on the determined activity, the system sends a notification to the first person (e.g., in addition to, or alternative to, sending a notification to the user). In some implementations, the system instructs the camera that recorded the live video stream (or a nearby device) to provide an audible and/or visual alert upon determining that the activity is a nefarious activity (e.g., stealing a package, looking in a window, breaking a window, kicking a door, etc.). In some implementations, the audible alert comprises a verbal warning to the first person. In some implementations, the verbal warning includes reference to one or more detected qualities of the first person (e.g., the color and/or type of the first person's clothing). For example, the first person is wearing a hat and red jacket and the verbal warning says “You, with the hat and the red jacket, stop what you are doing.”

In some implementations, the one or more programs further include instructions, which when executed by the computing system, cause the system to provide a notification to the first person in accordance with the determination that the first person is not a known person. In some implementations, the system instructs the camera that recorded the live video stream (or another device within audible range of the first person) to emit an audible alert, such as a siren or verbal warning. In some implementations, the system instructs the camera that recorded the live video stream (or another device within visual sight of the first person) to provide a visual alert, such as activation of a light or light ring (e.g., a red light ring). In some implementations, the system determines a time of day for the detection of the first person and provides a notification based on the time of day (e.g., provides a notification upon determining that it is night-time).

In yet another aspect, in accordance with some implementations a method includes: (1) obtaining a plurality of images collected by one or more video cameras in a smart home environment, each image of the plurality of images including a detected person; (2) for each image of the plurality of images, obtaining personally identifiable information of the detected person, the personally identifiable information generated from analysis of the image; (3) grouping the plurality of images into a first group of a plurality of groups based on the personally identifiable information, each group of the plurality of groups representing a unique one of a plurality of detected persons; (4) receiving from a user a request to remove a first image from the first group; and (5) in response to the request: (a) removing the first image from the first group; and (b) disassociating the corresponding personally identifiable information from the first group. In some implementations, the plurality of images includes a plurality of image portions cropped from video feeds of the video cameras, each image portion including a detected person. In some implementations, the plurality of images is grouped based on similarities in the personally identifiable information (PII).

In some implementations: (1) the method further includes presenting to the user one or more images of the first group, including presenting the first image; and (2) the user request to remove the first image corresponds to a user selection of the first image and a subsequent user action.

In some implementations, the method further includes: (1) obtaining a new image including the detected person; (2) obtaining new personally identifiable information of the detected person, the new personally identifiable information generated from analysis of the new image; (3) determining that the new image includes the detected person based on a comparison of the new personally identifiable information with personally identifiable information associated with the first group; and (4) in accordance with the determination that the new image includes the detected person: (a) including the new image in the first group; and (b) associating the new personally identifiable information with the first group.

In some implementations, the method further includes, for each group of the plurality of groups, selecting an image of the group as a profile image for the group.

In some implementations, the profile image is selected based on a determined mood of the detected person in each image. In some implementations, only images of the person while happy are considered for the profile image. In some implementations, the image is selected based on a quality factor for each image. In some implementations, the image is selected based on a pose of the person in each image (e.g., only frontal images are considered).

In some implementations, the method further includes displaying a listing of the one or more groups, the listing including display of the profile image for each group. In some implementations, the listing includes one or more labels for each group (e.g., user labels such as the person's name and/or automated labels such as “Unidentified Person 1”.

In some implementations, the method further includes displaying an event timeline corresponding to at least one of the one or more video cameras, including: (1) displaying on the timeline indication of one or more detected events; and (2) for at least one of the one or more detected events, displaying one or more profile images corresponding to one or more persons in the detected event.

In yet another aspect, some implementations include a server system having one or more processors and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors. In some implementations, the one or more programs including instructions for: (1) obtaining a plurality of images collected by one or more video cameras in a smart home environment, each image of the plurality of images including a detected person; (2) for each image of the plurality of images, obtaining personally identifiable information of the detected person, the personally identifiable information generated from analysis of the image; (3) grouping the plurality of images into a first group of a plurality of groups based on the personally identifiable information, each group of the plurality of groups representing a unique one of a plurality of detected persons; (4) receiving from a user a request to remove a first image from the first group; and (5) in response to the request: (a) removing the first image from the first group; and (b) disassociating the corresponding personally identifiable information from the first group.

In some implementations, the plurality of images are grouped into the first group in response to a determination that the personally identifiable information corresponding to each image in the plurality of images is within a similarity threshold to other personally identifiable information for the first group. In some implementations, in accordance with a determination that the PII is within a threshold amount to at least one PII of the first group, the image and PII is deemed to be of the first group. In some implementations, in accordance with a determination that the PII is within a threshold amount to an average or mean PII set for the group, the image and PII is deemed to be of the first group.

In some implementations, the one or more programs further include instructions for, for a first image of the plurality of images, adjusting the personally identifiable information based on a determined pose of the person in the image. In some implementations, the pose is normalized when determining the PII (e.g., a person's facial features are extrapolated from a profile-view image of the person).

In some implementations, the one or more programs further include instructions for, for a first image of the plurality of images, adjusting the personally identifiable information based on a camera operating mode. For example, the PII is adjusted to account for the camera being in a low-light mode (e.g., when determining skin tone, hair color, eye color, etc.). In some implementations, PII from low-light mode images is only compared with other PII from low-light mode images.

In some implementations: (1) the request to remove the first image comprises a request to assign the first image to a different group of the plurality of groups; (2) removing the first image from the first group comprises assigning the first image to the different group; and (3) disassociating the personally identifiable information of the first image comprises associating the personally identifiable information with the different group. For example, the user indicates that one of the images was incorrectly deemed to include the same detected person and instead includes a different person known to the user. In some implementations, the different group comprises a new group.

In some implementations, the one or more programs further include instructions for: (1) receiving from the user a request to remove a group of the plurality of groups; and (2) in response to the request: (a) deleting all images in the group; and (b) deleting corresponding personally identifiable information.

In some implementations, the personally identifiable information comprises facial feature information of the detected person. In some implementations, the personally identifiable information comprises a feature signature for the detected person, the feature signature generated based on analysis of the image.

In yet another aspect, some implementations include a non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to: (1) obtain a plurality of images collected by one or more video cameras in a smart home environment, each image of the plurality of images including a detected person; (2) for each image of the plurality of images, obtain personally identifiable information of the detected person, the personally identifiable information generated from analysis of the image; (3) group the plurality of images into a first group of a plurality of groups based on the personally identifiable information, each group of the plurality of groups representing a unique one of a plurality of detected persons; (4) receive from a user a request to remove a first image from the first group; and (5) in response to the request: (a) remove the first image from the first group; and (b) disassociate the corresponding personally identifiable information from the first group.

In some implementations, the one or more programs further comprise instructions, which when executed by the computing system, cause the system to: (1) receive, from the user, identification information of the detected person for the first group; and (2) remove the first group from the plurality of groups based on the received identification information.

In some implementations: (1) the identification information indicates that the detected person is a stranger to the user; and (2) removing the first group comprises deleting (i) all images in the first group, and (ii) the corresponding personally identifiable information.

In some implementations: (1) the identification information indicates an identity of the detected person; and (2) removing the first group comprises combining the first group with another group representing the same identity. For example, the user identifies the group has images of Joe and the group is combined with another group of images for Joe.

In some implementations: (1) the request to remove the first image comprises a request to delete the first image; (2) removing the first image from the group comprises deleting the first image; and (3) disassociating the personally identifiable information of the first image comprises deleting the personally identifiable information.

In yet another aspect, some implementations include a camera system configured to perform any of the methods described herein. In some implementations, the camera system includes means for performing any of the actions described herein. In some implementations, the camera system includes one or more cameras and a server system.

Thus, systems are provided with more efficient and effective methods for monitoring and facilitating review of events and persons in video streams, thereby increasing the accuracy, effectiveness, efficiency, and user satisfaction with such systems. Such systems and methods may complement or replace conventional systems and methods for event and person monitoring and presentation.

