FIELD

Aspects of embodiments of the present disclosure relate to the field of computer vision, including specifically a system and method for providing dominant scene classification by semantic segmentation.

BACKGROUND

Scene classification refers to the process of perceiving a natural scene and understanding the content of the scene. A human may perceive a scene with their eyes and identify salient aspects of the scene (e.g., people in front of a landmark or a tourist attraction). In the context of computer vision, scene classification may include capturing one or more images of a scene using a camera and identifying the elements of the scene. Semantic segmentation refers to the process of identifying portions or regions of the images that correspond to particular classes of objects (e.g., people, buildings, and trees).

SUMMARY

Aspects of embodiments of the present disclosure relate to performing dominant scene classification of images of scenes. Dominant scene classification includes identifying the subject or subjects of a scene. Some aspects of embodiments of the present disclosure utilize semantic segmentation to perform the dominant scene classification.

According to one embodiment, a method for computing a dominant class of a scene includes: receiving an input image of a scene; generating a segmentation map of the input image, the segmentation map being labeled with a plurality of corresponding classes of a plurality of classes; computing a plurality of area ratios based on the segmentation map, each of the area ratios corresponding to a different class of the plurality of classes of the segmentation map; and outputting a detected dominant class of the scene based on the plurality of ranked labels based on the area ratios.

The method may further include determining the detected dominant class based on a highest ranked label of the plurality of area ratios.

The method may further include using an atrous spatial pyramid pooling module to receive an output of a plurality of atrous convolutional layers, and the segmentation map may be computed based on an output of the atrous spatial pyramid pooling module.

The computing the area ratios may include: spatially weighting the segmentation map by multiplying each location of the segmentation map by a corresponding one of a plurality of spatial importance weights; and summing the spatially weighted segmentation map to compute a spatially weighted area ratio for each of the classes, wherein the spatial importance weights are a weighted combination of Gaussian filters having highest weight in a region corresponding to a middle third of the input image.

The computing the area ratios may further include class weighting the area ratios by multiplying the area ratio for each class by a corresponding class importance weight of a plurality of class importance weights, and wherein the plurality of class importance weights may include a foreground group of classes having higher weights than a background group of classes.

The foreground group of classes may include a text class and a person class, and the background group of classes may include a sky class and a tree class.

The method may further include: receiving a sequence of input images before the input image; computing a softmax over each pixel of each image of the sequence of input images; performing temporal filtering over each pixel across each image of the sequence of input images to compute a filtered softmax volume; and computing a maximum across the filtered softmax volume to compute the segmentation map.

The temporal filtering may be performed with a triple-exponential smoothing filter.

The method may further include: generating a sequence of weighted area ratios for a sequence of segmentation maps computed from the sequence of input images; and performing temporal filtering over the sequence of weighted area ratios, wherein the plurality of ranked labels is computed based on the sequence of weighted area ratios.

The detected dominant class may be selected by: evaluating a hysteresis condition that is met when a previously detected dominant class is a second highest ranked label of the plurality of ranked labels and when a difference in weighted area ratio between a highest ranked label and the second highest ranked label is less than a threshold; in response to determining that the hysteresis condition is met, maintaining the previously detected dominant class as the dominant class; and in response to determining that the hysteresis condition is not met, setting the highest ranked label as the detected dominant class.

Each pixel of the segmentation map may be associated with one or more corresponding confidence values, each of the one or more corresponding confidence values corresponding to a different one of the plurality of classes, and the method may further include thresholding the segmentation map by selecting values from locations of the segmentation map where corresponding locations of the confidence map exceed a threshold corresponding to a class of the location of the segmentation map.

The segmentation map may be computed from a plurality of logits output by a convolutional neural network, the logits including spatial dimensions and a feature dimension, and the one or more confidence values form a confidence map may be generated by: computing a softmax along the feature dimension of the logits; and computing a maximum of the softmax along the feature dimension of the logits to compute the confidence values corresponding to each location of the confidence map.

The segmentation map may be generated by a convolutional neural network wherein the convolutional neural network may include a global classification head configured to compute a global classification of a class of the input image, and the convolutional neural network may be trained with a loss function including a weighted sum of: a first loss associated with the detected dominant class; and a second loss associated with the global classification computed by the global classification head.

The global classification head may be configured to receive input from an output of the convolutional neural network.

The method may further include using an atrous spatial pyramid pooling module configured to receive an output of a plurality of atrous convolutional layers, wherein the segmentation map may be computed based on an output of the atrous spatial pyramid pooling module, and wherein the global classification head may be configured to receive input from the output of the atrous spatial pyramid pooling module.

The segmentation map may be generated by a convolutional neural network trained to recognize a text class of the plurality of classes with training data including images of text and corresponding labels, and the corresponding labels may include bounding boxes surrounding text.

A class of the plurality of classes may include a plurality of subclasses, and the method may further include assigning a subclass to a region in the segmentation map corresponding to the class by: detecting a color of each of a plurality of pixels of the input image in the region corresponding to the class; assigning one of the plurality of subclasses to each of the pixels based on the color of the pixel; and assigning the subclass to the region based on majority voting among the subclasses assigned to the pixels of the region.

The class may be water and the subclasses may include: low saturation water; green water; blue water; and other water.

The method may further include: identifying a portion of the input image of the scene corresponding to the detected dominant class; and configuring camera settings of a digital camera module in accordance with the identified portion of the input image of the scene.

The digital camera module may be a component of a mobile device.

According to one embodiment, a system includes: a processor; and memory storing instructions that, when executed by the processor, cause the processor to compute a dominant class of a scene by: receiving an input image of a scene; generating a segmentation map of the input image, the segmentation map being labeled with a corresponding class of a plurality of classes; computing a plurality of area ratios based on the segmentation map, each of the area ratios corresponding to a different class of the plurality of classes of the segmentation map; and outputting a detected dominant class of the scene based on the plurality of ranked labels.

The memory may further store instructions for computing the area ratios by: spatially weighting the segmentation map by multiplying each location of the segmentation map by a corresponding one of a plurality of spatial importance weights; and summing the spatially weighted segmentation map to compute a spatially weighted area ratio for each of the classes, wherein the spatial importance weights may be a weighted combination of Gaussian filters having highest weight in a region corresponding to a middle third of the input image.

The memory may further store instructions for computing the area ratios by class weighting the area ratios by multiplying the area ratio for each class by a corresponding class importance weight of a plurality of class importance weights, and wherein the plurality of class importance weights may include a foreground group of classes having higher weights than a background group of classes.

The foreground group of classes may include a text class and a person class, and the background group of classes may include a sky class and a tree class.

Each pixel of the segmentation map may be associated with one or more corresponding confidence values, each of the one or more corresponding confidence values corresponding to a different one of the plurality of classes, and wherein the memory may further store instructions for thresholding the segmentation map by selecting values from locations of the segmentation map where corresponding locations of the confidence map exceed a threshold corresponding to a class of the location of the segmentation map.

The system may further include a digital camera module, wherein the memory further stores instructions that, when executed by the processor, cause the processor to: identify a portion of the input image of the scene corresponding to the detected dominant class; and configure camera settings of the digital camera module in accordance with the identified portion of the input image of the scene.

