TECHNICAL FIELD

This disclosure relates to techniques for identifying unwanted objects in an image, and more particularly, to techniques for identifying unwanted people in an image using machine learning.

BACKGROUND

Distractors are objects or regions in images that draw attention away from the main subjects of an image. Examples of distractors include signs that are present in a photo of a landscape, the presence of an unwanted person in a portrait photo or in a landmark photo, or an incomplete portion of an object appearing at an edge of a photo.

One technique for identifying distractors and removing them from images employs a machine learning model trained on ground-truth (human annotated) images to identify distractors. The model creates a distractor map of an image that identifies the objects in an image and predicts a degree to which each object is likely a distractor. An adjustable threshold is set in some scenarios to remove those objects predicted to be distractors to a degree that exceeds the threshold.

A problem arises, however, when the model removes a person from a photo who was positioned relatively far from the camera, even though the photographer intended for the person to be in the photo. The opposite problem arises when an unwanted person close to the camera is not removed, because the model does not classify the person as a distractor.

Another technique for identifying distractors from images assumes that objects appearing in a photo at a size significantly smaller than other objects are the distractors since larger objects tend to be more salient and thus more likely to be a main subject.

A problem arises, however, when an unwanted person appears between the intended subject and the camera, thus appearing much larger. This leads to instances of wrongly identifying an unwanted person as a main subject and retaining these people in the image.

SUMMARY

Systems, methods, and software are described herein for identifying unwanted objects (e.g., human distractors) in images. A distractor classifier implemented in hardware, software, and/or firmware on one or more computing devices functions to identify the distractors within images.

An exemplary embodiment of the distractor classifier comprises a combination of two or more networks including an object detector network, a heatmap network and a hybrid classifier. The heatmap network operates on an input image to generate a heatmap (e.g., global information or global data). In an embodiment, the global data is generated from all pixels of the input image. The heatmap includes a heatmap for a main subject, a heatmap for a distractor, and optionally a heatmap for the background. The object detector identifies objects in the input image. Each of the objects is cropped within the input image to generate a corresponding cropped image (e.g., local information or local data). In an embodiment, the local data is generated from pixels of the object. Regions within the heatmaps that correspond to the objects are identified, and each of the regions is cropped within each of the heatmaps to generate cropped heatmaps. The hybrid classifier then operates on the cropped images and the cropped heatmaps to classify each of the objects as being either a main subject or a distractor.

According to an embodiment, features of an object cropped from the image are generated by an object detector network, the features including first data local to the corresponding object. Features of a cropped heatmap of the object are generated by a heatmap network, the features of the cropped heatmap include second data global to the image

