Challenges of SLAM in extremely unstructured environments: the DLR Planetary Stereo, Solid-State LiDAR, Inertial Dataset

We present the DLR Planetary Stereo, Solid-State LiDAR, Inertial (S3LI) dataset, recorded on Mt. Etna, Sicily, an environment analogous to the Moon and Mars, using a hand-held sensor suite with attributes suitable for implementation on a space-like mobile rover. The environment is characterized by challenging conditions regarding both the visual and structural appearance: severe visual aliasing poses significant limitations to the ability of visual SLAM systems to perform place recognition, while the absence of outstanding structural details, joined with the limited Field-of-View of the utilized Solid-State LiDAR sensor, challenges traditional LiDAR SLAM for the task of pose estimation using point clouds alone. With this data, that covers more than 4 kilometers of travel on soft volcanic slopes, we aim to: 1) provide a tool to expose limitations of state-of-the-art SLAM systems with respect to environments, which are not present in widely available datasets and 2) motivate the development of novel localization and mapping approaches, that rely efficiently on the complementary capabilities of the two sensors. The dataset is accessible at the following url: https://rmc.dlr.de/s3li_dataset


I. INTRODUCTION
Simultaneous Localization and Mapping (SLAM) techniques can nowadays be considered mature enough to find applications in many fields such as autonomous driving [1], [2], automated construction [3] or agriculture [4], [5]. While typically implemented with the use of multi-camera setups or RGB-D sensors, the recent democratization of 3D LiDAR sensors allowed easy integration of range sensing on mobile robots, enabling robust and large-scale mapping especially in urban or man-made scenarios.
The wide variety of published visual or LiDAR SLAM approaches [6]- [8] might suggest that the SLAM problem could be considered solved. However, we argue that this is far from the truth, specifically when dealing with environments that are unstructured and severely aliased [9], [10]. This is especially true in the case of planetary analogous environments, which are usually located in extreme locations such as deserts or volcanic surfaces. In this case, the lack of uniquely identifiable visual or structural features prevents from performing place recognition in a robust and repeatable manner. Moreover, when dealing with LiDAR SLAM, the lack of vertical structures leads to missing constraints for a robust convergence of ICP-style algorithms [11].
Typical datasets fail to address this extreme case. Among the most popular, KITTI [12], Oxford RobotCar [13], KAIST [14] and 4Season [15]   in urban scenarios. While presenting sequences with significant challenges, e.g., dynamic objects or large time intervals spanning seasons, the point of view of a road vehicle, as well as the panoramic 3D LiDAR scans captured within highly structured scenarios, facilitate the tasks of computing odometry and performing place recognition.
Other datasets, such as TUM RGB-D [16], TUM VI [17] or the ETH3D SLAM benchmark [18] provide sequences recorded with hand-held stereo or RGB-D cameras principally in indoor environments. Their sequences tend to be short and frequently re-observing places from similar viewpoints and under very limited changes in the visual appearance.
To overcome the limitations related to the practical realization of the sensor setup, but also to model several motion characteristics and environments, many synthetic datasets have been recently released such as ICL-NUIM [19] or TartanAir [20]. While this has many advantages for the development of SLAM algorithms, e.g., the diversity of motion patterns and scenes is beneficial towards training of learningbased approaches, we believe that, especially for robots operating in the field, unstructured natural environments still offer the most challenging conditions for the implementation of visual or LiDAR-based SLAM.
Several datasets have been proposed which are recorded from the perspective of a mobile robot operating in the field [21]. The Chilean Underground Mine dataset [22] provides sequences recorded in a mine with a stereo camera, a 3D LiDAR and a radar. The Visual-Inertial Canoe dataset [23] provides sequences where the main challenge is represented by reflections on the water. The Katwijk beach dataset [24] is recorded on a beach with artificial landmarks from a planetary-like rover equipped with a stereo camera that has similar characteristics to the one of the ExoMars rover. However challenging, the environment of caves and mines is, as was also shown during the DARPA SubT Challenge [25]- [27], particularly suited for LiDAR-based SLAM, as are the stone replicas on the flat sandy surface of the Katwijk beach. The MADMAX dataset [28], collected in the Morocco desert using a hand-held stereo unit with accurate ground truth, offers stereo sequences which are particularly challenging for traditional place recognition, but does not provide LiDAR data, which could be beneficial as it can provide information for localization invariant to the visual appearance. Similarly, [29] provides two sequences recorded on Mt. Etna with a mobile rover, which was only equipped with a stereo camera mounted on a pan-tilt unit. We fill this gap and present here a dataset collected on the volcanic surface of Mt. Etna, Italy, a planetary analogous environment characterized by extreme visual aliasing and lack of distinct structural features. The dataset has been collected with a hand-held sensor assembly that includes a stereo monochrome camera setup, a Solid State LiDAR and an IMU. The setup replicates the typical height and point of view of the sensors mounted on a mobile robot. Although influenced by the walking pattern, the recordings look smooth and free of sharp or high-frequency motions. Seven sequences, 8 to 30 minutes long and covering distances up to 1.3 kilometers, have been recorded in order to evaluate the ability of SLAM algorithms: 1) to accurately estimate odometry under harsh lighting conditions and lack of unique features, 2) to rely on different sensing modalities, depending on the structure and appearance of the environment, 3) to use visual and structural information to robustly perform place recognition under severe aliasing. In summary, this paper presents the following contributions: • We release a dataset to challenge SLAM algorithms in an natural, extremely unstructured environment, that resembles the conditions of planetary-like scenarios. • We evaluate a selection of state-of-the-art visual, and visual-inertial SLAM algorithms. • We provide examples of using the data in unconventional manners, leveraging the interaction of cameras and LiDARs towards more robust SLAM in severely unstructured scenes.

II. THE DATASET
The dataset was recorded on the volcanic environment of Mt. Etna, Sicily, an active stratovolcano, which is denoted as a planetary analogous site for Moon and Mars due to its peculiar geological features [30]. The dataset includes 7 sequences, that were recorded on a slope at 2650 meters above sea level and in proximity to the Cisternazza crater (see Fig. 1). Each sequence is located far from hiking trails and provides views of an extreme environment characterized by a surface of smooth dark lava ash, due to the frequent eruptions that happened overnight at the time of recording the data, extreme visual contrast, due to the darkness of the soil and the brightness of the sky, and scarcity of geological features that might facilitate localization with respect to prior observations of the same places. In comparison with other similar datasets, listed in Table I, we provide an honest multisensory look at an environment, analogous to potential sites for robotic planetary exploration, without alterations, e.g. by introducing artificial landmarks, and without a prior selection based on properties that help specific sensing modalities, i.e., flatness of the soil, presence of outstanding structural details, or presence of distinct visual textures on the ground.

A. Sequences
The 7 sequences provided can be categorized according to the challenges they pose for localization algorithms. s3li traverse 1, s3li traverse 2 and s3li crater test the ability of localization methods or SLAM algorithms to provide accurate pose estimation during traverses. The first two observe mainly ash slopes, with the sporadic presence of rocks or boulders, while the third is a long traverse, that starts and ends at the initial location, around the rim of the crater, providing long range features for which depth is unknown. s3li loops and s3li crater inout are, instead, recorded with the purpose of evaluating the ability of SLAM algorithms to perform place recognition, exploiting either the visual or structural similarity. While the first revisits mainly unstructured slopes, that might be recognized through carefully tuned visual loop closure detectors, the second includes short repeating traverses within the rim of the crater, that could be exploited with both sensing modalities. s3li mapping instead tests the ability of SLAM algorithms to produce a complete and consistent map of the environment relying on stereo as well as LiDAR depth measurements, as a confined area  is explored multiple times and there should be plenty of loop closure opportunities. Finally, s3li landmarks allows testing of segmentation techniques for natural landmarks, with the aim of either semantic mapping or segment-based loop closure detection [31]. An overview of the sequences, including D-GNSS tracks, is provided in Table II.

B. The hand-held setup (S3LI)
The sensor setup used to collect the data, depicted in Fig. 3, is designed to be compliant to a potential implemen-tation on a planetary-like rover [32] and comprises: Stereo Camera: two AVT Mako cameras are configured in a stereo setup with baseline 20 centimeters. The cameras are triggered at a frequency of 30 Hz to capture monochromatic images with resolution of 688 × 512 pixels using automatic exposure control to cope with the harsh lighting conditions on the mountain. Their timestamps are synchronized to the system time using the PTP protocol, where cameras function as slave to the master clock of the laptop. Compared to other • Loopy sequence around the rocky ridges observed also in s3li traverse 1 and s3li traverse 2, plenty of opportunities for visual loop closure detection and/or for visual or structure-based segmentation of natural features. • Less monotonous visual appearance than s3li traverse 1 and s3li traverse 2, a few salient rocky features are observed in "orbiting" motions. • Structure is less interesting than visual appearance. • Long traverse around the crater, that also includes a short travel below the rim to observe some geological formations. They are revisited after completing the first traverse, offering opportunities for loop closure detection. • Challenging visual appearance characterized by sharp changes in illumination due to the exploration of the insides of the crater, often lying in the shadows • Interesting sequence to map geological formations with the LiDAR, to be exploited also towards structure-based place recognition. • Sequence of limited spatial extent targeted at mapping, where the sensors observe the environment to cover as much area as possible. The terrain is characterized by ash slopes with few small boulders. • Plenty of loop closure opportunities are available, however the contrast between the dark lava ash and harsh lighting challenges traditional visual methods. • LiDAR measurements cover a dense area and can be interesting for complementing the mapping ability of the stereo camera as well as for providing solid ground constraints in the context of multi-modal SLAM. • Sequence of traverses between large and isolated rock formations, some of which are re-observed to allow for an easy recognition of structures and visual features. • As for s3li mapping, the visual appearance is challenging due to the harsh contrast between dark lava ash and the bright sky. During the traverses, the ground offers limited visual details. • The rock formations are observed from many viewpoints allowing for a complete reconstruction. Salient structures can easily be segmented for semantic mapping or matching. datasets based on stereo images [28], [29], we do not include depth images, as the computation of disparity is not intended as a sensor output, but as part of the utilization of the stereo images. Many SLAM architectures [33]- [35], in fact, although relying on a stereo image stream, do not make use of dense disparity images, but instead perform sparse feature matching when needed. Nevertheless, as an example and for the purpose of visualization, Fig. 2 presents a collection of disparity images obtained with OpenCV's SGBM stereo algorithm. Due to the automatic exposure control that, in combination with the extreme intensity differences between the dark soil and the bright sky, often compresses the ground details into dark patches, we recommend the usage of automatic contrast enhancement algorithms such as OpenCV's CLAHE.
LiDAR: we use a Blickfeld Cube-1 LiDAR, which employs a MEMS-actuated beam deflection mirror, instead of traditional 360 • LiDARs, as this type of construction is more suited for potential space applications due to increased mechanical robustness and reduced weight and power consumption. For all sequences, the LiDAR is configured to capture point clouds with a maximum number of 17400 points with a Field-of-View of about 70 • H × 30 • V, which results in a scan rate of 4.7 Hz. Each data point is timestamped with respect to the origin of the scan and contains additional intensity information that can be used, for instance, to identify low-quality measurements. The synchronization of the scan timestamps with respect to the time of the laptop is realized with NTP. IMU: an XSens MTi-G 10, connected via USB, provides linear acceleration and angular velocity at a rate of 400 Hz. GPS receiver: an Ublox f9p GNSS receiver mounted on the hand-held setup provides accurate differential estimates in conjunction with a base station. The data logs are processed in a later stage using the RTKLIB to obtain ground truth positions at a frequency of 5 Hz and centimeter-level accuracy.

C. Sensor calibration
Stereo camera and IMU: The intrinsic and extrinsic parameters of the stereo camera pair are calibrated using Calde/Callab [37], while the IMU to camera calibration is performed with Kalibr [38]. In order to facilitate the playback of data, the stereo images are provided already rectified. In fact, as the lenses do not have a particularly wide field of view, the images corrected for distortion do not lose a significant amount of visual detail. LiDAR-Camera extrinsic calibration: the LiDAR is extrinsically calibrated with respect to the left camera. As the solid-state LiDAR that we employed is characterized by quite a large minimum measurable depth of about 4 meters, traditional techniques for LiDAR to camera calibration proved either non-effective, e.g., in the case of a * In bold are highlighted the best results, from those algorithms that completed the sequence checkerboard pattern printed on planar target [39], or too impractical to perform on the field, e.g., methodologies that rely on non-planar geometries [40], [41]. For this reason, the transformation between the camera and LiDAR reference frames is determined by fixing the translation component to the one obtained by a CAD model of the sensor assembly, while the rotation component is optimized in an automatic edge alignment scheme [42], [43]. More specifically, n pairs of left camera images and LiDAR scans are manually selected from recording of an indoor environment, characterized by structural regularity and scarcity of evident textures. From each image, strong Canny edges are selected to obtain binary images, where the identified edges are an input to the optimization scheme. From each corresponding LiDAR scan, we estimate the local curvatures, using the PCL library, and edge points, which lie in correspondence to depth discontinuities between neighboring points in the sequence of measurements of the LiDAR beam. Formally, the extrinsic calibration is solved as a non-linear least squares problem, where the goal is to find the optimal rotation between LiDAR and left camera, that aligns the projections of LiDAR points, lying on edges or discontinuities, to edges in the image: arg min where n is the number of image-scan pairs, m i is the number of correspondences for the i-th pair, π is the camera projection function, p L ik and x ik are, respectively, 3D LiDAR and 2D image points on edges, R Cl L and t L Cl are the rotation (to be optimized) and translation (known form CAD model) of the LiDAR in the left camera reference frame, ρ c is a Cauchy robust loss function, and ω ik is a weight proportional to the local curvature of the point cloud. The association between LiDAR and image edge points is performed iteratively in a nearest-neighbor approach, and the problem is solved a few consecutive times reducing the value of parameter c for the Cauchy loss.

A. Visual SLAM
A major application area of this dataset is to evaluate the performances of SLAM algorithms in environments with challenging visual appearance. While the texture of the terrain can ease the task of, for instance, sparse optical flow, feature matching and place recognition are particularly challenged by the severe visual aliasing. The algorithms compared here are ORB SLAM3 [33], VINS Fusion [36], OPEN VINS [35] and BASALT [34] in stereo or stereoinertial configuration. The estimated path is correlated to the D-GNSS ground truth using time correspondences. After this step, we align all pairs of matching positions using Horn's algorithm, and compute the root mean square of all the residuals. To be comparable amongst all sequences, the RMSE is normalized by the length of the associated ground truth trajectory. Furthermore, as the RMSE is affected by the length of the estimated trajectory (shorter trajectories exhibit less translational and rotational drift) we provide, as an additional performance indicator, the proportion of each sequence for which poses are estimated from Visual SLAM. This value is computed as the length of the ground truth, associated to SLAM estimates, over its total length. Table II-C reports these metrics for all algorithms and for every sequence of the dataset. Compared to the others, ORB SLAM3, where camera tracking depends heavily on ORB descriptors matching, exhibits very good accuracy in all datasets, but frequently fails to estimate the whole sequence, especially when the observed environment is characterized almost completely by a very repetitive terrain, without the presence of rocks or features of unique appearance. The other algorithms, where the localization front-end depends on tracking strong corners through optical flow, do not suffer from this specific problem, but are generally more prone to pose drift.
As expected, the lowest errors are scored from stereoinertial configurations, as the inertial measurements help significantly to deal with high accelerations and rotational velocities, typical of the walking motion patterns when recording with hand-held devices. However, compared to their stereo counterparts, the delicate procedures to initialize IMU parameters leads often to lose significant portions of the initial parts of the sequences. Furthermore, we experienced random behaviours of this step, especially with VINS Fusion and OPEN VINS, often initializing the state estimation at very different points after the beginning of sequences. For this reason, Table II-C reports the best results obtained during different runs of the same sequences, and omits results for which less than 30% of the trajectory was estimated.
Overall, there is no apparent winner among all the algorithms compared, in terms of lowest RMSE values, although for robustness, indicated by the proportions of sequences successfully processed, the approach of BASALT would be the most successful. Hence, this dataset can help the development of Visual SLAM approaches, considering the challenges provided by this type of environment, which are often missing in traditional datasets for Visual SLAM.

B. LiDAR Mapping
As Figs. 2 and 4 show, each LiDAR scan covers a small portion of the environment, which is located in front of the sensor setup within a relatively narrow Field-of-View. Furthermore, the geometry of the environment, which is characterized by soft slopes and small objects, does not provide structures of great extent in the vertical direction as it is, instead, very common in datasets targeted at autonomous driving [13], [44]. For these reasons, traditional LiDAR SLAM can not be employed successfully on this data. Compared to the stereo camera, however, where the depth uncertainty grows quadratically with the distance, the LiDAR sensor allows to measure accurately the shape of farther landscapes that might not be possible to reach, e.g., with a mobile system. It is, therefore, of great interest to develop SLAM pipelines where the LiDAR and visual sensing approaches are combined.
Meant purely as an example, we developed a simple mapping pipeline, that assembles LiDAR scans synchronized to pose estimated from a Visual SLAM approach. For this, we used OPEN VINS. Fig. 5 shows four examples of LiDAR maps from the s3li traverse 1, s3li landmarks, s3li traverse 2 and s3li crater inout sequences, respectively. Observing these maps, it is evident that the predominant geometries are slopes, with the exception of sparse and relatively small rocks and stones. Other more evident structures, i.e., big rock formations, are visible in Fig.5(b), which are re-observed twice. Fig 5(d), instead, shows the reconstruction of the rim of the Cisternazza crater, to an extent and detail that is not possible to achieve with a stereo camera alone. This dataset, therefore, intends to challenge visual-LiDAR SLAM algorithms to robustly perform pose estimation and mapping by developing novel approaches for place recognition that depart from traditional global LiDAR descriptors [45] or segmentation of structures with strong ground plane assumptions [46].
In order to illustrate what may lead to such methods, we employed the Instance Stereo Transformer (INSTR) [47], to detect instances of unknown objects in a scene (see Fig. 6 for two examples). Note that the network has not been trained on this specific data. The segmentation masks can be used as an input for isolating objects in, otherwise complicated to process, LiDAR scans and accurately generate geometric models with the goal of realizing landmark-based [46] or metric-semantic [48] SLAM.

IV. CONCLUSIONS
We presented a dataset, recorded in a planetary analogue environment, that comprises stereo, LiDAR and inertial measurements, and allows to assess localization and mapping frameworks under challenging visual and structural appearance. We, furthermore, evaluated a variety of visual-inertial SLAM algorithms on the sequences, highlighting strengths and limitations, and provided examples of potential usecases for this data, that depart from traditional visual-LiDAR SLAM approaches. This dataset, therefore, provides a mean to evaluate and overcome limitations of traditional SLAM algorithms, that are usually developed and tested on datasets which lack elements of visual aliasing and ambiguous geometries, typical of extremely unstructured environments.