FIELD

This application is related to image capture and image processing. More specifically, this application relates to systems and methods of automatically guiding image capture and automatically adjusting settings to visually optimize image composition and/or to apply a particular style.

BACKGROUND

In photography, certain rules or guidelines for image composition can help a photographer frame subjects within an image in a way that makes the image more visually appealing. However, many photographers are not familiar with the many different rules and guidelines for image composition, how to best apply these rules and guidelines for different types of photographs, or when to disregard certain rules and guidelines.

Cameras can apply a variety of image capture and image processing settings to alter how an image looks. Some camera settings are determined and applied before or during capture of the photograph, such as ISO, exposure time, aperture size, f/stop, shutter speed, focus, and gain. Other camera settings can configure post-processing of a photograph, such as alterations to contrast, brightness, saturation, sharpness, levels, curves, or colors. Different camera settings can emphasize different aspects of an image. However, the large number of different camera settings can be confusing for users. Users may not be aware of which settings are helpful in which scenarios, and may not understand how to tweak certain camera settings to be helpful in those scenarios.

SUMMARY

Systems and techniques are described herein for generating and outputting guidance of image capture. An image capture device captures a first image. Based on the first image, the image capture device identifies changes to properties of the image capture device. The changes cause a visual difference between the first image and a second image to be captured by the image capture device after capture of the first image. The image capture device can identify the changes based on settings for the properties under which other images than the first image were captured. For example, the other images may be images depicting the same subject as the first image depicts, or a similar subject to the subject depicted in the first image. In some examples, the changes can be based on a machine learning model trained on these other images. The image capture device generates and outputs the guidance to indicate the changes to produce the visual difference by the time the image capture device captures the second image. The properties can include positioning of the image capture device (to affect image composition), image capture settings, and/or image processing settings.

In one example, an apparatus for guiding image capture is provided. The apparatus includes one or more connectors coupled to one or more image sensors, wherein the one or more connectors receive image data from the one or more image sensors. The apparatus includes one or more memory units storing instructions and one or more processors that execute the instructions. Execution of the instructions by the one or more processors causes the one or more processors to perform a method of guiding image capture. The method includes receiving a first image of a scene captured by an image sensor. The method includes identifying a subject depicted in the first image. The method includes inputting the first image into a machine learning model, the machine learning model being trained using a plurality of training images with identified subjects. The method includes identifying, using the machine learning model, one or more changes to one or more properties associated with image capture that cause a visual difference between the first image and a second image to be captured by the image sensor after capturing the first image. The method includes outputting guidance indicative of the one or more changes to produce the visual difference before the image sensor captures the second image.

In another example, a method of guiding image capture is provided. The method includes receiving a first image of a scene captured by an image sensor of an image capture device. The method includes identifying a subject depicted in the first image. The method includes inputting the first image into a machine learning model, the machine learning model being trained using a plurality of training images with identified subjects. The method includes identifying, using the machine learning model, one or more changes to one or more properties of the image capture device that cause a visual difference between the first image and a second image to be captured by the image sensor after capturing the first image. The method includes outputting guidance indicative of the one or more changes to produce the visual difference before the image sensor captures the second image.

In another example, an non-transitory computer readable storage medium having embodied thereon a program is provided. The program is executable by a processor to perform a method of guiding image capture. The method includes receiving a first image of a scene captured by an image sensor of an image capture device. The method includes identifying a subject depicted in the first image. The method includes inputting the first image into a machine learning model, the machine learning model being trained using a plurality of training images with identified subjects. The method includes identifying, using the machine learning model, one or more changes to one or more properties of the image capture device that cause a visual difference between the first image and a second image to be captured by the image sensor after capturing the first image. The method includes outputting guidance indicative of the one or more changes to produce the visual difference before the image sensor captures the second image.

In another example, an apparatus for guiding image capture is provided. The apparatus includes means for receiving a first image of a scene captured by an image sensor of an image capture device. The apparatus includes means for identifying a subject depicted in the first image. The apparatus includes means for inputting the first image into a machine learning model, the machine learning model being trained using a plurality of training images with identified subjects. The apparatus includes means for identifying, using the machine learning model, one or more changes to one or more properties of the image capture device that cause a visual difference between the first image and a second image to be captured by the image sensor after capturing the first image. The apparatus includes means for outputting guidance indicative of the one or more changes to produce the visual difference before the image sensor captures the second image.

In some aspects, identifying the subject depicted in the first image includes performing at least one of feature detection, object detection, face detection, feature recognition, object recognition, facial recognition, and generation of a saliency map. In some aspects, the methods, apparatuses, and computer-readable medium described above further comprise: receiving the second image from the image sensor after outputting the guidance; and outputting the second image, wherein outputting the second image includes at least one of displaying the second image using a display and transmitting the second image using a transmitter.

In some aspects, identifying the one or more changes to the one or more properties of the image capture device includes identifying a movement of the image capture device from a first position to a second position, wherein outputting the guidance includes outputting an indicator for moving the image capture device from the first position to the second position. In some aspects, the second position is identified using the machine learning model. In some aspects, the indicator includes at least one of a visual indicator, an audio indicator, and a vibrational indicator. In some aspects, indicator identifies at least one of a translational direction of the movement, a translational distance of the movement, a rotational direction of the movement, and a rotational angle of the movement.

In some aspects, the indicator identifies a translational direction from the first position to the second position. In some aspects, the indicator identifies a translational distance from the first position to the second position. In some aspects, the indicator identifies a rotational direction from the first position to the second position. In some aspects, the indicator identifies a rotational angle from the first position to the second position. In some aspects, the indicator includes one or more location coordinates of the second position. In some aspects, the visual difference between the first image and the second image levels a horizon in the second image, wherein the horizon is not level as depicted in the first image.

In some aspects, the methods, apparatuses, and computer-readable medium described above further comprise: receiving pose sensor measurement data from one or more pose sensors; and determining a pose of the apparatus based on the pose sensor measurement data, wherein identifying the movement of the apparatus from the first position to a second position is based on the pose of the apparatus, wherein the pose of the apparatus includes at least one of a location of the apparatus and an orientation of the apparatus. In some aspects, the one or more pose sensors include at least one of an accelerometer, a gyroscope, a magnetometer, an inertial measurement unit, a global navigation satellite system (GNSS) receiver, and an altimeter.

In some aspects, the methods, apparatuses, and computer-readable medium described above further comprise: determining a position of the subject in the first image; and determining a direction that the subject is facing in the first image, wherein identifying the movement of the image capture device from the first position to the second position is based on the position of the subject in the first image and the direction that the subject is facing in the first image. In some aspects, determining the direction that the subject is facing in the first image is based on relative positioning of two features of the subject. In some aspects, determining the direction that the subject is facing in the first image is based on positioning of a plurality of features of the subject within the first image relative to one another. In some aspects, the subject is a person, and the plurality of features of the subject include at least one of an ear of the person, a cheek of the person, an eye of the person, an eyebrow of the person, a nose of the person, a mouth of the person, a chin of the person, and an appendage of the person.

In some aspects, determining the direction that the subject is facing in the first image is based on a movement direction in which the subject moves between the first image and a third image captured by the image sensor. In some aspects, the methods, apparatuses, and computer-readable medium described above further comprise: receiving a third image captured by the image sensor, the third image depicting the subject; and determining a direction of movement of the subject based on the position of the subject in the first image and a position of the subject in the third image, wherein determining the direction that the subject is facing in the first image is based on the direction of movement of the subject. In some aspects, the visual difference between the first image and the second image includes an increase in negative space adjacent to the subject in the direction that the subject is facing.

In some aspects, the methods, apparatuses, and computer-readable medium described above further comprise: receive a third image of the scene captured by a second image sensor, wherein the first image of the scene and the third image of the scene are captured within a time window, wherein the second image sensor has a wider field of view than the image sensor, wherein the guidance is based on a depiction of a portion of the scene in the third image that is not depicted in the first image.

In some aspects, the guidance indicates that the image capture device is to remain still between capture of the first image and capture of the second image.

In some aspects, the plurality of training images includes a training image that depicts at least one of the subject and a second subject sharing one or more similarities with the subject, wherein the one or more changes to the one or more properties indicated by the guidance are based on one or more settings for the one or more properties used for capture of the training image. In some aspects, the one or more similarities shared between the second subject and the subject include one or more saliency values associated with the second subject being within a predetermined range of one or more saliency values associated with the subject. In some aspects, the visual difference between the first image and the second image includes the second image being more similar to the training image than the first image is to the training image. In some aspects,

In some aspects, the one or more changes to the one or more properties associated with image capture include applying an image capture setting before the image sensor captures the second image, wherein the image capture setting corresponds to at least one of zoom, focus, exposure time, aperture size, ISO, depth of field, analog gain, and f/stop. In some aspects, outputting the guidance includes outputting an indicator identifying the one or more changes to the one or more properties associated with image capture corresponding to applying the image capture setting. In some aspects, outputting the guidance includes automatically applying the one or more changes to the one or more properties associated with image capture corresponding to applying the image capture setting.

In some aspects, the methods, apparatuses, and computer-readable medium described above further comprise: receive the second image captured by the image sensor, wherein the one or more changes to the one or more properties associated with image capture include applying an image processing setting to the second image, wherein the image processing setting corresponds to at least one of brightness, contrast, saturation, gamma, levels, histogram, color adjustments, blur, sharpness, levels, curves, filtering, and cropping. In some aspects, outputting the guidance includes outputting an indicator identifying the one or more changes to the one or more properties associated with image capture corresponding to applying the image processing setting. In some aspects, outputting the guidance includes automatically applying the one or more changes to the one or more properties associated with image capture corresponding to applying the image processing setting.

In some aspects, the apparatus comprises a camera, a mobile device (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a wireless communication device, a wearable device, a head-mounted display (HMD), an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus further includes the image sensor. In some aspects, the apparatus further includes one or more connectors coupled to the image sensor, wherein the one or more processors receive the first image from the image sensor through the one or more connectors. In some aspects, the apparatus further includes a display for displaying at least the second image. In some aspects, the apparatus further includes a display for displaying one or more images, notifications, and/or other displayable data.

This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and embodiments, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

