STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

Not Applicable.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to reconstructing three-dimensional video from two-dimensional video data using particle filtering and generating training data for autonomous vehicles.

2. Description of the Related Art

Collecting a sufficient amount of three dimensional (3D) training data for autonomous vehicles to handle rare, but critical, traffic events (e.g., collisions) may take decades of deployment. Autonomous vehicles collect large quantities of training data by operating, or being operated, in their target environment. This data is used to teach the vehicle how to adjust to and interact with the physical world [Ref. 6]. However, research suffers from a lack of realistic training data, especially of rare and unusual events such as traffic accidents [Ref. 17]. To collect sufficient training instances of such events, autonomous vehicles need to run and record hundreds of billions of miles in the wild, corresponding to decades of operating a car on the roads.

Therefore there is a need to generate realistic training data for autonomous vehicles other than operating autonomous vehicles in the real world and collecting data during operation of the autonomous vehicles.

SUMMARY OF THE INVENTION

In one aspect, the present disclosure provides a method for reconstructing three-dimensional video from two-dimensional video data using particle filtering and thereby generating training data for autonomous vehicles.

In another aspect, the present disclosure provides a method in a data processing system including at least one processor and at least one memory, the at least one memory including instructions executed by the at least one processor to implement a vehicle reconstruction system. The method (a) receiving a set of annotations associated with a video frame including a view of at least a portion of a vehicle, each annotation including at least one two-dimensional line, (b) removing at least one outlier from the set of annotations, (c) determining an estimated vehicle model based on the set of annotations, and (d) providing the estimated vehicle model to a driving simulator.

In the method, one of the annotations can be based on input attributed to a human.

In the method, each two-dimensional line can be overlaid on the video frame.

The method can further include receiving a set of bounding boxes associated with the video frame, each bounding box associated with an annotation included in the set of annotations, and the removing at least one outlier from the set of annotations can include determining a target bounding box included in the set of bounding boxes is an outlier, and removing the at least one two-dimensional line included in the annotation associated with the target bounding box from the set of annotations. The determining the target bounding box is an outlier can include determining a second target bounding box included in the set of bounding boxes is a median, determining a percentage of the target bounding box that does not overlap the second target bounding box, and determining that the percentage is below a predetermined threshold. The percentage cam be about fifty percent.

In the method, the at least one two-dimensional line of each annotation in the set of annotations can include a first two-dimensional line associated with a first dimension, a second two-dimensional line associated with a second dimension, and a third two-dimensional line associated with a third dimension. The removing at least one outlier from the set of annotations can include determining a first target two-dimensional line associated with the first dimension and included in the set of annotations is a median, determining that a second target two-dimensional line associated with the first dimension and included in the set of annotations is not within a threshold range of the first target two-dimensional line, and removing the second target two-dimensional line from the set of annotations. The method can still further include determining an interquartile range of all two-dimensional lines included in the set of annotations and associated with the first dimension, and determining the threshold range based on the interquartile range.

In the method, the estimated vehicle model can include a three dimensional volume, a position value, and an orientation value.

In yet another aspect, the present disclosure provides a method in a data processing system including at least one processor and at least one memory, the at least one memory including instructions executed by the at least one processor to implement a vehicle reconstruction system. The method includes (a) receiving a set of annotations including at least two subsets of annotations, each subset of annotations being associated with a video frame included in a set of video frames, each video frame including a view of at least a portion of a vehicle, and each annotation including at least one two-dimensional line, (b) removing at least one outlier from the set of annotations, (c) determining a set of estimated vehicle models based on the set of annotations, each estimated model associated with a video frame in the set of video frames, and (d) providing the set of estimated vehicle models to a driving simulator.

In the method, each estimated vehicle model can be associated with a time step, and each estimated vehicle model can be determined based on a second estimated vehicle model associated with a previous time step.

In the method, each estimated vehicle model can be determined using a recursive Bayesian method. The recursive Bayesian method can be particle filtering. The particle filtering can include generating, for each subset of annotations associated, a plurality of particles, each particle being associated with a cuboid, and determining, for each particle, a probability value based on the at least one two-dimensional line.

In the method, each estimated vehicle model can include a three dimensional volume, a position value, and an orientation value.

In a further aspect, the present disclosure provides a method in a data processing system including at least one processor and at least one memory, the at least one memory including instructions executed by the at least one processor to implement a vehicle reconstruction system. The method includes (a) causing a video frame including a view of at least a portion of a vehicle to be displayed to a plurality of users, (b) causing an option to provide annotations corresponding to the video frame to be displayed to a plurality of users, (c) causing an option to not provide annotations corresponding to the video frame to be displayed to a plurality of users, (d) receiving a number of annotations corresponding to the video frame from the plurality of users, (e) calculating an estimated vehicle model based on the number of annotations, and (f) providing the estimated vehicle model to a driving behavior analyzer.

The method can further include causing an instructional module to be displayed to the plurality of users, the instructional module including instructions to provide at least one first dimensional line along a first dimension and to provide at least one second dimensional line along a second dimension. The instructional module can further include instructions to provide at least one third dimensional line along a third dimension.

The can further include receiving a selection to provide annotations from a group of users, the group of users including at least a portion of the plurality of users, receiving the number of annotations from the group of users. The method can still further include removing at least one outlier from the number of annotations.

These and other features, aspects, and advantages of the present invention will become better understood upon consideration of the following detailed description, drawings and appended claims.

