CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean Patent Application No. 10-2019-0027201, filed Mar. 8, 2019, which is hereby incorporated by reference for all purposes as if fully set forth herein.

BACKGROUND

Field

Exemplary embodiments generally relate to a class labeling system for autonomous driving, and more particularly, to a class labeling system for autonomous driving that accurately recognizes and automatically labels classes for autonomous driving of a vehicle based on deep learning.

Discussion

In autonomous driving, image recognition recognizes various objects, such as vehicles, pedestrians, traffic lights, lanes, crosswalks, etc., and is aimed at various properties, such as the relative location, speed, and size of each object. Due to the diversity of objects and properties, image recognition has more difficulty as compared to other recognition fields. It, however, is predicted that the difficulty in the field of image recognition in autonomous driving will also be able to be solved (or at least reduced) by deep learning technology if there is sufficient diversity and retention of learning data. However, it is difficult to acquire learning data because of different geographical and legal characteristics between countries. By way of example, it is difficult to directly utilize learning data, such as cityscape or KITTI, in domestic roads because the learning data is very different from domestic road conditions.

Despite the importance of data acquisition, a strategy to acquire a recognition object range and a learning data technology acquisition for autonomous driving is unclear or limited to the development of some major technologies, with the exception of several advanced companies. In particular, data is inefficiently labeled using a manual annotation tool in most cases. In this case, considerable resources are consumed due to the trouble of re-annotating according to a required labeling method or a change in laws and regulations.

It is recognized that ApolloScape and Berkeley DeepDrive (BDD) datasets have been published through a semi-automated annotation tool. The semi-automated annotation tool is innovative to label the most widely used cityscape and KITTI, but it still lacks the objects of recognition because it is only intended to detect major objects. For instance, the semi-automated annotation tool excludes information necessary for driving, such as a traffic sign or a vehicle lamp, from the detection objects, compared to humans, and continuously consumes considerable human and time resources in annotating.

A conventional image processing method for autonomous driving is disclosed in Korean Patent Application Publication No. 10-2018-0097966, published Sep. 3, 2018, and entitled “Image Processing Method for Autonomous Driving and Apparatus Thereof.”

The above information disclosed in this section is only for understanding the background of the inventive concepts, and, therefore, may contain information that does not form prior art.

SUMMARY

Some aspects provide a class labeling system for autonomous driving that is capable of accurately recognizing and automatically labeling classes for autonomous driving of a vehicle based on deep learning.

Additional aspects will be set forth in the detailed description which follows, and, in part, will be apparent from the disclosure, or may be learned by practice of the inventive concepts.

According to some aspects, a class labeling system for autonomous driving includes a detection module, a segmentation module, and a lane road boundary detection module. The detection module is configured to detect objects for autonomous driving from an image captured by a camera to generate a bounding box for each of the objects and detect property information about the object. The segmentation module is configured to determine classes for each pixel of the bounding box detected by the detection module and process at least one of the classes as don't care. The lane road boundary detection module is configured to detect at least one of lane and road boundaries using the bounding box detected by the detection module.

In an embodiment, the detection module may include a detection unit, a classification task unit, an instance segmentation task unit, a distance value regression task unit, and a tracking task unit. The detection unit may be configured to detect the objects by analyzing the image captured by the camera and detect the bounding box for each of the objects. The classification task unit may be configured to detect a sub-property of each object detected with the bounding box by the detection unit. The instance segmentation task unit may be configured to crop the bounding box detected by the detection unit to detect a pixel related to the object in the bounding box. The distance value regression task unit may be configured to use a distance value input from a light detection and ranging (LiDAR) to detect a distance to the object in the bounding box detected by the detection unit. The tracking task unit may be configured to predict a location of at least one of a vehicle and a pedestrian detected with the bounding box by the detection unit.

In an embodiment, the detection unit may be configured to detect in-image coordinates for locations of objects to receive the bounding box.

In an embodiment, the in-image coordinates may be expressed by a vertex of the bounding box, a width of the bounding box, and a height of the bounding box.

In an embodiment, the vertex may be a left upper end of the bounding box.

In an embodiment, the classification task unit may be configured to detect a sub-property of a sign from among the objects.

In an embodiment, the distance value regression task unit may be configured to extract point cloud coordinate values related to the in-image coordinate values of the bounding box, and determine, among the in-image coordinate values, a value having a minimum distance from an object to a vehicle as the distance value of the object.

In an embodiment, at least one of the classification task unit, the instance segmentation task unit, the distance value regression task unit, and the tracking task unit may be configured to perform shift data augmentation in multiple directions and regions to learn each network.

In an embodiment, at least one of the classification task unit, the instance segmentation task unit, the distance value regression task unit, and the tracking task unit may be configured to learn a network using at least one of zero shot learning, one shot learning, and low shot learning techniques.

In an embodiment, at least one of the classification task unit, the instance segmentation task unit, the distance value regression task unit, and the tracking task unit may be configured to learn a network by an ensemble technique using at least one of left rotation, right rotation, and flip rotation.

In an embodiment, the segmentation module may include a segmentation unit configured to determine the classes of each pixel of the image captured by the camera, and a don't care processing unit configured to process at least one of the classes determined by the segmentation unit as don't care.

In an embodiment, the don't care processing unit may be configured to: detect a bounding box determined as don't care from an original image to generate a segmentation image; learn a dataset of the segmentation image and a dataset labeled by a human labeler to compare performance evaluation results; and perform repetitive learning until the dataset of the segmentation image is improved more than the dataset labeled by the human labeler in recognition performance according to the performance evaluation results.

In an embodiment, the lane road boundary detection module may include a free instance segmentation unit and a B-spline detection unit. The free instance segmentation unit is may be configured to output a pixel-wise class and an instance output from the image captured by the camera using an instance deep learning network. The B-spline detection unit may be configured to detect a B-spline for lane or road shapes through the pixel-wise class and the instance output that are output from the free instance segmentation unit.

According to various exemplary embodiments, a class labeling system for autonomous driving can automatically obtain high-quality learning data for autonomous driving. In addition, the class labeling system for autonomous driving can prevent (or at least mitigate) the trouble of re-annotating according to a labeling method or a change in laws and regulations by performing automated labeling without using an existing manual annotation tool, and can significantly increase the production speed of learning data as compared to using the existing manual annotation tool. Further, the class labeling system for autonomous driving can recognize all objects for autonomous driving, even in common road environments, such as paved, unpaved, suburban, urban roads, etc., and a variety of weather conditions. Moreover, the class labeling system for autonomous driving can minimize (or at least reduce) the time to regenerate annotation data even when a data specification has changed through a deep learning-based automatic inferencing technique.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.

