FIELD

The present invention relates in general to the area of artificial intelligence. In particular, the present invention relates to a method for training a machine learning system, to a method for operating a machine learning system, and to a machine learning system.

BACKGROUND INFORMATION

Driving a motor vehicle in roadway traffic regularly requires reliable visual recognition of a large number of objects, such as other traffic participants, traffic signs, roadway boundaries, or obstacles of any kind. Increasingly, motor vehicles are being further developed in the direction of partly or completely automated driving, in which the recognition of objects, previously done by a driver, is artificially emulated. Due to the high degree of complexity of recognition processes, as a rule machine learning systems and/or artificial intelligence systems are used here, which carry out a classification of the objects that are recognizable in the image data, for example on the basis of image data recorded by a camera of the motor vehicle, in order ultimately to be able to automatically identify the objects. Such machine learning systems for recognizing objects can include for example a neural network.

Object recognition using machine learning systems can be further improved by evaluating data from other sensors, in addition to image data. Such systems are also referred to as multipath systems. Such a system can acquire and interpret the surrounding environment for example on the basis of visual information from various sources. Here, a learning system, such as a neural network that acquires the surrounding environment on the basis of image information from a single camera, can be supplemented by a module that acquires and interprets 3D information and/or temporal movement information. In this way, the object recognition as a whole can be improved.

However, in order to enable reliable recognition of objects in input data and/or sensor data using machine learning systems and/or artificial intelligence systems, these systems must be trained with a large quantity of training data. This can be a challenge, in particular in multipath systems.

SUMMARY

Using specific example embodiments of the present invention, a machine learning system can advantageously be comprehensively trained so that overall the machine learning system, and/or an object recognition based on the machine learning system, can be improved.

According to a first aspect of the present invention, an example method is provided for training a machine learning system. In a step of the method, image data are fed into a machine learning system with processing of at least a part of the image data. The method includes the following steps:synthetic generation and/or artificial production at least a part of at least one depth map that has a plurality of depth information values, each of the depth information values correlating with a distance to an object;feeding the at least one depth map into the machine learning system, with processing of at least a part of the depth information values of the at least one depth map by the machine learning system; andtraining the machine learning system based on the processed image data and based on the processed depth information values of the at least one depth map, with adaptation, varying, and/or modification of at least one parameter value of at least one parameter of the machine learning system, the adapted at least one parameter value influencing an interpretation of input data by the machine learning system and/or an output of the machine learning system.

In general, the machine learning system, hereinafter also designated “ML system” or simply “system,” can designate any realization of an artificial intelligence system. In particular, the ML system can be designed as a classifier, for example as a neural network. The ML system can be configured as a multipath system and can be configured to process, analyze, and/or interpret depth maps as well, in addition to image data. For this purpose, the ML system can be realized as a one-part system, or can for example include a plurality of modules each of which can process different data. For example, the ML system can include a first module for processing image data and a second module for processing depth maps.

Here and in the following, the term “feeding in” of data to the ML system can designate a providing of data to the ML system, for example via a suitable data connection and/or a suitable interface of the ML system.

In addition, “training of the ML system” can designate a feeding in and processing of the corresponding data, such as the image data and/or the depth information values, to any data processing device of the ML system. In particular, the data for training the ML system can be propagated through the ML system in the forward direction, in a forward propagation, and/or in the backward direction, in a backward propagation. Here, the parameter value of the at least one parameter of the ML system can be adapted, modified, and/or varied iteratively and/or successively in such a way that a reaction to, and/or the interpretation of, any further input data is improved and/or optimized with respect to an intended use of the ML system. If, for example, the ML system is used for object recognition, then, using the method, the parameter value of the at least one parameter can be adapted, modified, and/or varied in such a way that ultimately the precision of the object recognition is improved. In this context, the “interpretation of the input data by the ML system” can mean that the system processes the input data and provides an output that is at least partly influenced in particular by the parameter value of the at least one parameter. If the ML system is for example a classifier, then the system can for example output at least one class designation, a class, and/or at least one probability value for a class of objects. Thus, in general the parameter can designate a variable, in particular a mathematical variable, based on which the ML system analyzes or interprets the input data supplied to it, such as sensor data, image data, and/or depth maps.

The image data named above and/or the at least one depth map can designate in particular training data of the ML system. For training purposes, the image data and/or the depth map can also be labeled. In other words, the image data and/or the depth map can also contain an item of information relating for example to the objects in the image data and/or in the depth map. Based on the labels of the image data and/or of the depth map, the ML system can in addition ascertain a recognition error, for training purposes, and the at least one parameter value of the at least one parameter can be adapted with minimization of the recognition error, in order to train the ML system and/or to optimize the object recognition.

The depth map can for example designate a disparity map that can represent data of an arbitrary sensor for the acquisition of distance information. The depth map can, as it were, represent, contain, and/or include items of information relating to a distance to an object, i.e., distance information. The depth map can also contain spatial information, three-dimensional information, and/or 3D information. The depth information values of the depth map can here designate distance data.

The input data can designate any sensor data, such as image data of a camera and/or a depth map of a distance sensor, that are evaluated, analyzed, and/or interpreted by the ML system, for example for object recognition. In particular, the input data can be unlabeled, and/or can designate data that are supplied to the ML system after the training of the ML system for the actual object recognition.

The synthetic generation can be understood as artificial production. Synthetically generated and/or artificially produced depth information values can therefore designate distance data that have not been acquired by a sensor and/or distance sensor, but rather have been produced for example manually and/or by machine.

In sum, according to the first aspect of the present invention, an example training method is provided for an ML system, such as a neural network, in which both image data, such as images of a camera, and also an at least partly synthetically generated depth map are supplied to the system. The parameter value of the at least one parameter of the ML system is adapted based on the image data and the depth map for the training of the ML system. In particular, in the training method according to the present invention a plurality of, or all, parameters of the ML system can be adapted so that for example an object recognition by the ML system is improved. Here, the ML system can be trained using image data, and exclusively with synthetically generated depth maps. Alternatively, in addition to image data and synthetically generated depth maps, real depth maps that have been acquired by a sensor can also be used for the training. Synthetically generated image data can also be supplied to the system for training.

Through the synthetic generation of the depth map, an artificial scenario can advantageously be produced and trained in which, for example, an interpretation of the image data and an interpretation of the depth map by the ML system would result in different reactions and/or results. In addition, compared to image data, a depth map can be synthetically produced with low outlay, quickly, and in a cost-efficient manner, because it is less complex compared to image data. In this way, depth maps can be produced in a cost-efficient fashion that may contradict the image data, may deviate from them, and/or may result in different reactions or interpretations by the ML system. In this way, the reaction of the ML system to unusual and rarely occurring scenarios, such as optical illusions, can advantageously be trained. For example, a scenario can be simulated in which a camera is used to make an image of an advertising poster on which a street is depicted. An interpretation of only this depiction of a street could have the result that the ML system interprets it as a real street. A sensor for acquiring distance information would however recognize the poster as a solid object at a fixed distance from the sensor, and would record corresponding depth information values and/or a depth map. Using the example method according to the present invention, such scenarios can advantageously be artificially produced and trained, so that the trained ML system can recognize and correctly interpret such scenarios during use. Overall, the ML system, in particular object recognition by the ML system, can be significantly improved by the method according to the present invention.

The present invention can be regarded in particular as being based on the findings described in the following. A learning system is generally improved by the processing of large amounts of training data. However, it is not possible to reproduce the entire world and/or all scenarios in the training data. Therefore, objects and/or scenarios may occur that are unknown to the system. The present invention therefore makes it possible to use additional synthetically produced depth maps to train a correct assessment of objects and/or scenarios that are not contained in a conventional training data set of real data. In a training of the ML system exclusively with real data, the parameters of the system, such as weights of a neural network, could be ascertained and/or selected in such a way that the system can derive complex information from the image data and can derive information from the depth maps that is less complex in comparison to the image data, because more information may be present in the image data compared to the depth maps. During use of the trained ML system, this can have the result that the system follows an interpretation of the image data, for example when image data and depth maps are interpreted that deviate from one another, because such deviations in real data sets are typically underrepresented, occur rarely, and may therefore be insufficiently trained. The use of a multipath system, which for example recognizes objects based on data from different sensors, such as image data and depth maps, can in general enable a redundant protection of the object recognition. Because it is relatively improbable that two different recognition paths of the system, for example a first path based on the image data and a second path based on the depth maps, will contain errors, each of the paths can be used as a plausibility check for the respective other path, and in this way the object recognition as a whole can be improved. Such scenarios, in which an interpretation of the image data by the ML system and an interpretation of the depth map by the ML system deviate from one another, can advantageously be trained by the method according to the present invention.

In other words, using the example method according to the present invention, a correct interpretation can be trained of objects having ambiguous appearance. Overall, therefore, the training process of learning systems can be expanded and improved. Scenarios with synthetically produced depth information are trained that occur only rarely and can therefore be trained into the system only with difficulty and at considerable expense. The core of the present invention is therefore the use of synthetically produced depth maps to train the system. The greatest benefit is seen in the handling of so-called corner cases, or rarely occurring special cases in which the image data would permit a different inference than the depth map, such as in the case of the optical illusions as described above. Such cases are very rare, and can be trained only with difficulty simply by feeding real scenes into the system. In general, therefore, the present invention can decisively improve a learning system and/or an ML system that has to reliably decide, on the basis of visual data and/or image data, whether it is dealing with a relevant object.

According to a specific embodiment of the present invention, the method additionally has the step of assigning the image data to the at least one depth map. Alternatively or in addition, the parameter value of the machine learning system is adapted as a function of the processed image data and as a function of the processed depth information values. In other words, the parameter value of the at least one parameter can be calibrated to both kinds of data, i.e., both the image data and the depth map. In this way, a multipath system can advantageously be comprehensively trained that analyzes and/or interprets a surrounding environment both based on image data, i.e., based on visual information, and also based on a depth map, i.e., based on spatial information.

According to a specific embodiment of the present invention, the depth map includes a matrix, an array, and/or a list having entries, each entry representing a pixel of a device for acquiring depth information, spatial information, and/or 3D information. A value of each entry is a depth information value for indicating a distance between the device and an object. The depth map can in particular designate a disparity map, and/or the depth information values can designate disparity values and/or distance data.

According to a specific embodiment of the present invention, the at least one depth map represents data of a stereo camera, a multiview camera, a distance measuring device, a radar-based distance measuring device, an ultrasound-based distance measuring device, and/or a laser-based distance measuring device. In general, the depth map can represent data of any sensor for acquiring distance information, distance data, and/or spatial information. In addition, the image data can designate data of any optical sensor, such as a camera, an RGB camera, a color camera, a grayscale camera, and/or an infrared camera.

According to a specific embodiment of the present invention, the step of synthetic generation of the at least one part of the at least one depth map includes in addition the substeps of defining, specifying, and/or determining a plurality of depth information values of the depth map and of storing the plurality of defined depth information values in the depth map. Here, in particular at least 1%, for example at least 5%, of all depth information values of the synthetically generated depth map can be defined. Through the specification of at least approximately 1% of the depth information values, it can be ensured that an adequately large and/or massive object can be artificially produced in the depth map, so that when a real depth map is fed into the trained ML system, for example statistical noise in the data is not recognized as an object. Overall, in this way the training process, and the object recognition, with the trained system can be further improved.

According to a specific embodiment of the present invention, the defined depth information values are values of at least one subset of entries of the depth map, which subset represents a contiguous pixel region of pixels of a device for acquiring depth information, so that by defining the depth information values, an item of distance information is produced regarding a geometrically contiguous object in the depth map. In this way, a massive and/or relatively large object can be artificially produced in the depth map that can represent a real object. The object artificially produced in the depth map can have any shape, contour, and/or size. The object can also be produced at any position in the depth map. In addition, a plurality of objects can be produced in a single depth map, for example at different positions.

According to a specific embodiment of the present invention, the geometrically contiguous object is contained exclusively in the at least one depth map, so that a discrepancy and/or a deviation between the image data and the at least one depth map is produced by the synthetic generation of the at least one part of the at least one depth map. In this context, the discrepancy between the depth map and the image data can mean that the object is present only in the depth map. Alternatively or in addition, the discrepancy can mean that different objects are present in the image data and in the depth map. For example, in the image data a roadway may be recognizable that however originates only from an advertising poster, whereas in the depth map the poster may be recognizable as a massive object at a certain distance.

According to a specific embodiment of the present invention, the parameter value of the at least one parameter of the machine learning system is adapted in such a way that, given a discrepancy between the image data and the at least one depth map, an interpretation of the depth map by the machine learning system is preferred over an interpretation of the image data by the machine learning system. In this way, it can be ensured that if the depth map, or the distance information contained therein, after interpretation by the ML system, permits the conjecture that, for example, an obstacle is situated in a travel path of a motor vehicle that however is not recognized, or is recognized insufficiently, by the ML system in the image data, the ML system provides a corresponding output that can cause the vehicle to execute a braking process and/or an evasive maneuver. In this way, the degree of reliability and/or precision of the object recognition by the trained ML system can be increased overall. When the ML system trained in this way is used in a vehicle, this can also significantly increase safety.

According to a specific embodiment of the present invention, the plurality of depth information values is defined and/or selected in such a way that a distance between the device for acquiring depth information and the object is in a range between 5 cm and 500 m, in particular between 5 cm and 200 m. In this way, the distance can correspond to a fictive distance of a fictive device for acquiring distance information from the object. For example, the distance from the object can be in a safety-relevant range of a vehicle, so that the ML system can be comprehensively trained for use in a vehicle. The corresponding distance of the synthetically generated object can be selected as a function of which type of device for acquiring distance information is represented by the depth map. If, for example, the depth map represents data of an ultrasound-based distance measuring device, then the object synthetically produced in the depth map can be produced at a smaller distance than would be the case for example given a radar-based distance measuring device, in order in this way to take into account the smaller range of the ultrasound-based distance measuring device.

According to a specific embodiment of the present invention, the method in addition has the following steps:synthetic generation of, in each case, at least a part of a plurality of depth maps, with definition of a plurality of depth information values of each depth map; andtraining of the machine learning system with processing of the plurality of synthetically generated depth maps by the machine learning system, the defined depth information values of each depth map respectively representing a contiguous pixel region of pixels of a device for acquiring depth information, so that through the definition of the depth information values of each depth map, in each case an item of distance information is produced relating to a geometrically contiguous object in the respective depth map.

In general, the training of the ML system can be improved by synthetic generation and training with processing of different depth maps. In this way as well, many different scenarios having many different objects can be trained, so that in this way the object recognition using the trained ML system can be substantially improved.

According to a specific embodiment of the present invention, the objects produced in the synthetically generated depth maps differ from one another with respect to a contour, a dimension, a position in the respective depth maps, and/or with respect to a distance. In other words, in the depth maps objects differing from one another can be produced, which can enable the training of different scenarios and/or a recognition of different objects. The objects produced in the depth maps can have for example a round, oval, cornered, polygonal, rectangular, triangular, or any other contour and/or geometry. The different objects can in particular be generated randomly. For example, specified parameters of the objects, such as dimensions, sizes, geometries, positions in the depth maps or the like, can be randomly varied, for example using a random number generator. Alternatively or in addition, the objects produced in the depth maps can originate from scanned real objects. In this way, large numbers of different depth maps having different objects can be produced efficiently and quickly and used for the training of the ML system.

According to a specific embodiment of the present invention, the machine learning system is an artificial neural network, in particular a multilayer artificial neural network. Alternatively or in addition, the at least one parameter of the machine learning system is a weight of a node of an artificial neural network. The neural network can be for example a linear, nonlinear, recurrent, and/or convolutional neural network.

A second aspect of the present invention relates to the use of at least one at least partly synthetically generated depth map in combination with image data for the training of a machine learning system, in particular for training an ML system as described above and in the following.

A third aspect of the present invention relates to a method for operating a machine learning system for a motor vehicle, the machine learning system being trained using a method as described above and in the following. The method for operating the ML system can designate, as it were, a method for recognizing objects using the ML system.

Features, elements, and/or steps of the method for training the ML system can be features, elements, and/or steps of the method for operating the trained ML system, and vice versa.

A fourth aspect of the present invention relates to a machine learning system for recognizing objects for a motor vehicle, the machine learning system being trained using a method as described above and in the following.

Features, elements, and/or steps of the method for training and/or of the method for operating the machine learning system can be features, elements, and/or properties of the machine learning system, and vice versa. In other words, everything disclosed in relation to an aspect of the present invention holds equally for all other aspects of the present invention.

