FIELD

The present disclosure relates to Simultaneous Localization and Mapping (SLAM).

BACKGROUND

The next generation of safety and vehicle automation will rely on precise positioning or localization, yet Global Positioning System (GPS)-based positioning is hampered by frequent blockages of the GPS signal, even in normal driving situations. One of the main technical problems faced by the positioning is the map creation (i.e., mapping). In order to have a successful mapping, a positioning or localization is needed. Therefore, it is mandatory to perform the so-called SLAM.

Conventional SLAM methods with stereo cameras are, for instance, the ORB-SLAM2 and the Stereo LSD-SLAM, which belong to the feature-based and direct methods, respectively (medium.com/@j.zijlmans/lsd-slam-vs-orb-slam2-a-literature-based-comparison-20732df431d).

Extensions of SLAM combining sensors is still an open research topic. For instance, in a first approach (R. Mur-Artal and J. D. Tara's, “Visual-Inertial Monocular SLAM With Map Reuse”, IEEE Robotics and Automation Letters, vol. 2, no. 2, pp. 796-803, April 2017) a monocular ORB-SLAM was combined with an Inertial Measurement Unit (IMU). In a second approach (M. Lhuillier, “Incremental Fusion of Structure-from-Motion and GPS Using Constrained Bundle Adjustments”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 12, pp. 2489-2495, December 2012), a bundle adjustment (BA) technique—similar to ORB-SLAM—was combined with a GPS.

In the above first approach, the ORB-SLAM architecture is modified to include the IMU reading in the Tracking, Local Bundle Adjustment (LBA) and Full Bundle Adjustment. Results show that the monocular ORB-SLAM with known scale performs better than visual-inertial monocular ORB-SLAM.

In the above second approach, there are two BAs, i.e., the Standard BA and Fusion BA. The Standard BA minimizes the reprojection error of 3D point features, and the results of this BA are used in the Fusion BA, where the GPS is incorporated. The approach includes three proposed methods, each with its own focus. The first method is minimizing both the reprojection and GPS errors as a sum of squared differences, the other two methods are using barrier term optimization for either the reprojection error or the GPS error. The results show that the barrier term optimization works better than the first method.

However, so far the problem of combining multiple sensors (e.g. GPS and IMU) with SLAM has not been solved. Additionally, a distributed processing architecture for SLAM has not yet been exploited. Accordingly, so far no advantage of so-called crowdsourcing algorithms has taken place.

SUMMARY

In view of the above-mentioned problem, embodiments of the disclosure improve the conventional SLAM approaches. Embodiments of the present disclosure improve the feature-based mapping process, e.g., in terms of modularity, and extend it to include multiple sensors, particularly to include both GPS and IMU. Embodiments of the present disclosure further provide real-time distributed operation. In particular, embodiments of the present disclosure can take advantage of a crowdsourcing algorithm, in which for example multiple devices (e.g., vehicles) contribute to the map creation, by uploading respective sensor readings to, e.g., a cloud.

A first aspect of the present disclosure provides a device for performing SLAM. The device comprises at least one processor configured to: preprocess, in a first processing stage, a received data sequence including multiple images recorded by a camera and sensor readings from multiple sensors, in order to obtain a frame sequence. Each frame includes a visual feature set related to one of the images at a determined time instance and sensor readings from that time instance. The at least one processor is further configured to sequentially process, in a second processing stage, each frame of the frame sequence based on the visual feature set and the sensor readings included in that frame, in order to generate a sequence mapping graph, and merge, in a third processing stage, the sequence mapping graph with at least one other graph, in order to generate or update a full graph.

A “time instance” may be a determined point or an interval in time, or may be a point in time or time interval relative to some starting point in time. For example, the “time instance” may be a time interval between consecutive images in the data sequence. That is, a first image in the data sequence may be taken at a first point in time, and a second image in the data sequence may be taken at a second point in time, and a third image at a third point in time. In this case, the “time instance” of the second image may be the time interval between the first point in time and the second point in time. The sensor readings from that “time instance”, i.e. from the same “time instance” of the second image, in this case include all sensor readings between the first point in time and the second point in time. Notably, it is possible that there are no sensor readings during a “time instance”, e.g. if the sensors work at a slower frequency than a camera taking the images. Further on, the “time instance” of the third image starts at the second point in time.

The modularity of the device, i.e. the separation into the three processing stages, enables implementing real-time and distributed SLAM. The separation of particularly the second and third processing stages allows obtaining information from multiple data sequences. The third processing stage can correct sensor readings, e.g. can correct for GPS bias, given that multiple overlapping data sequences (e.g. from different devices) are available. The sensor readings allow fusing, e.g., GPS and IMU readings with the visual information obtained by one or more cameras. The structure of the frames further result in less data that needs to be passed to the next processing stage (compared to passing the entire image).

In an implementation form of the first aspect, a visual feature set includes an image feature set comprising one or more 2D key points extracted from the related image, descriptors corresponding to the 2D key points, and disparity and/or depth information of the 2D key points.

Thus, the frames include less information than the entire image, but include enough characteristic visual information to allow performing the sequence mapping. The benefit is that the preprocessed frames, which are to be used in the second processing stage, can be stored, which facilitates e.g. debugging processes. Also, less information needs to be transmitted, e.g., in a distributed device from one processor to the other. The disparity and/or depth information enables the calculation of 3D key points based on the respective 2D key points.

In a further implementation form of the first aspect, the at least one processor is configured to, in the first processing stage: extract an image from the data sequence, rectify the image, extract 2D key points from the rectified image, and generate an image feature set based on the extracted 2D key points.

In a further implementation form of the first aspect, the at least one processor is configured to, in the first processing stage: assign one or more semantic labels to the pixels of the rectified image, and filter the image feature set based on the semantic labels to remove 2D key points from the image feature set related to objects labelled as dynamic objects.

This removes unwanted 2D key points and consequently helps to reduce processing load.

In a further implementation form of the first aspect, the at least one processor is further configured to, in the first processing stage: generate a visual feature set by adding a bag-of-words descriptor to the filtered image feature set, and optionally a hash table for searching the 2D key points, and generate a frame by combining the visual feature set with sensor readings from the same time instance of the image.

This adds more information to the visual feature set. The hash table can further increase speed and robustness of the process. Sensor information can furthermore be implemented in this way.

In a further implementation form of the first aspect, the at least one processor is configured to, in the second processing stage: perform camera tracking based on a visual feature set included in a frame by matching 2D key points in the visual feature set to locally stored 3D key points, in order to obtain a camera pose associated with the frame.

In a further implementation form of the first aspect, the at least one processor is configured to: determine whether the frame is a key frame based on a number of matched 2D key points.

In a further implementation form of the first aspect, the at least one processor is further configured to, in the second processing stage, if it determines that the frame is a key frame: perform a first Local Bundle Adjustment, LBA, based on the camera pose, in order to obtain visual odometry information and a LBA graph, calculate a fused camera pose based on the visual odometry information and the sensor readings included in the frame, and perform a second LBA based on the fused camera pose and the LBA graph, in order to obtain the sequence mapping graph.

A feature matching process in the first LBA is accordingly not affected by, e.g., noisy sensor readings, which prevents inlier loss and tracking failures. The second LBA allows fusing, e.g., GPS and/or IMU with visual odometry data.

In a further implementation form of the first aspect, the at least one processor is further configured to, in the third processing stage: detect the presence of one or more loops and/or overlapping areas shared among the sequence mapping graph and the at least one further graph, merge the sequence mapping graph and the at least one further graph, in order to obtain an intermediate graph, and perform a graph optimization on the intermediate graph based on the detected loops and/or overlapping areas, in order to obtain the full graph.

For instance, a GPS bias can be corrected, if loops or overlapping areas are available.

In a further implementation form of the first aspect, the at least two of the three processing stages are performed in different processors.

This enables a distributed processing of the SLAM.

In a further implementation form of the first aspect, the device is a distributed device and includes at least one terminal device and at least one network device, a processor of the terminal device is configured to perform the first processing stage and transmit the obtained frame sequence to the network device, and a processor of the network device is configured to perform the second and third processing stages.

This enables implementing, e.g., a vehicle automation system. The preprocessing can be done in a real-time manner in a vehicle.

In a further implementation form of the first aspect, the processor of the terminal device is further configured to: perform a real-time localization based on the frame sequence obtained in the first processing stage.

Thus, the real-time localization is separated from the graph (map) update. Notably, for the sake of simplicity because usually there is a rigid transformation between a camera pose and a car pose (if the camera is installed on the car), both terms may be used interchangeably because either pose can directly be obtained given the other pose by means of the rigid transformation.

In a further implementation form of the first aspect, the processor of the terminal device is further configured to: perform the real-time localization according to the second processing stage described above, wherein the second LBA is replaced by a fusion tracking procedure based on the fused camera pose, the LBA graph and a current full graph, in order to obtain a current camera pose.

In a further implementation form of the first aspect, the terminal device is located in a vehicle and the vehicle includes the at least one camera and multiple sensors.

A second aspect of the present disclosure provides a method for performing SLAM. The method comprises: preprocessing, in a first processing stage, a received data sequence including multiple images recorded by a camera and sensor readings from multiple sensors, in order to obtain a frame sequence. Each frame includes a visual feature set related to one of the images at a determined time instance and sensor readings from that time instance The method further comprises: sequentially processing, in a second processing stage, each frame of the frame sequence based on the visual feature set and the sensor readings included in that frame, in order to generate a sequence mapping graph, and merging, in a third processing stage, the sequence mapping graph with at least one other graph, in order to generate or update a full graph.

In an implementation form of the second aspect, a visual feature set includes an image feature set comprising one or more 2D key points extracted from the related image, descriptors corresponding to the 2D key points, and disparity and/or depth information of the 2D key points.

In a further implementation form of the second aspect, the method comprises, in the first processing stage: extracting an image from the data sequence, rectifying the image, extracting 2D key points from the rectified image, and generating an image feature set based on the extracted 2D key points.

In a further implementation form of the second aspect, the method comprises, in the first processing stage: assigning one or more semantic labels to the pixels of the rectified image, and filtering the image feature set based on the semantic labels to remove 2D key points from the image feature set related to objects labelled as dynamic objects.

In a further implementation form of the second aspect, the method further comprises, in the first processing stage: generating a visual feature set by adding a bag-of-words descriptor to the filtered image feature set, and optionally a hash table for searching the 2D key points, and generate a frame by combining the visual feature set with sensor readings from the same time instance of the image.

In a further implementation form of the second aspect, the method comprises, in the second processing stage: performing camera tracking based on a visual feature set included in a frame by matching 2D key points in the visual feature set to locally stored 3D key points, in order to obtain a camera pose associated with the frame.

In a further implementation form of the second aspect, the method comprises: determining whether the frame is a key frame based on a number of matched 2D key points.

In a further implementation form of the second aspect, the method further comprises, in the second processing stage, if determining that the frame is a key frame: performing a first LBA based on the camera pose, in order to obtain visual odometry information and a LBA graph, calculating a fused camera pose based on the visual odometry information and the sensor readings included in the frame, and performing a second LBA based on the fused camera pose and the LBA graph, in order to obtain the sequence mapping graph.

In a further implementation form of the second aspect, the method further comprises, in the third processing stage: detecting the presence of one or more loops and/or overlapping areas shared among the sequence mapping graph and the at least one further graph, merging the sequence mapping graph and the at least one further graph, in order to obtain an intermediate graph, and performing a graph optimization on the intermediate graph based on the detected loops and/or overlapping areas, in order to obtain the full graph.

In a further implementation form of the second aspect, the at least two of the three processing stages are performed in different processors.

In a further implementation form of the second aspect, the method is performed in a distributed device including at least one terminal device and at least one network device. The terminal device performs the first processing stage and transmit the obtained frame sequence to the network device, and the network device performs the second and third processing stages.

In a further implementation form of the second aspect, the terminal device further performs a real-time localization based on the frame sequence obtained in the first processing stage.

In a further implementation form of the second aspect, the terminal further performs the real-time localization according to the second processing stage described above, wherein the second LBA is replaced by a fusion tracking procedure based on the fused camera pose, the LBA graph and a current full graph, in order to obtain a current camera pose.

In a further implementation form of the second aspect, the terminal device is located in a vehicle and the vehicle includes the at least one camera and multiple sensors.

With the method of the second aspect and its implementation forms, the advantages and effects described above for the device of the first aspect and its respective implementation forms are achieved.

Embodiments of the present disclosure are based on several improvements, such as the exemplary improvements described below:Software Architecture: The processing architecture of the device (separation into three processing stages) enables a modular approach that can be used for real-time distributed SLAM. In this way, the full graph (global map) can always be updated while doing localization, and only the extracted features need to be transmitted instead of the whole image (i.e., lower data rate).The separation of sequence mapping and final graph optimization: the process of mapping a sequence, which is focusing on having an accurate output based on visual information is separated from the full graph (global map) update, which will perform the final overall optimization taking into account the availability of multiple overlapping sequences. In this way, all information from all sequences can be properly used because of the process separation instead of having a single process in performing the graph update. Additionally, the graph update can correct GPS bias, since this is observable due to the availability of multiple overlapping sequences.Multiple sensor fusion: two Bundle Adjustments during the Sequence Mapping are used, wherein the first LBA is called a Standard LBA, and the second one fuses the output of the first LBA and a fused camera pose based on e.g. GPS, IMU readings and visual odometry information. The aim is to guarantee two separate independent processes of reprojection error optimization (done in the first LBA) and the GPS/IMU fusion prior to another optimization in the second LBA. In this way, the feature matching process in the first LBA is not affected by noisy GPS/IMU readings. In contrast to the above-described second approach, also IMU reading are fused.

It has to be noted that all devices, elements, units, and means described in the present disclosure could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present disclosure as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of exemplary embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof.

