2) Unifying evaluation benchmark and metric. Finding suitable evaluating benchmark and metric is always a concern for SLAM systems.
This is especially the case for DNN based systems. The predictions from DNNs are affected by the characteristics of both training and test data, including the dataset size, hyperparameters (batch size and learning rate etc.), and the difference in testing scenarios. Therefore, it is hard to fairly compare them when considering dataset differences, training/testing configuration, or evaluation metric adopted in each work.
For example, the KITTI dataset is a common choice to evaluate visual odometry, but previous works split training and testing data in different ways (e.g. [1]}, [2]}, [3]} used Sequence 00, 02, 08, 09 as training set, and Sequence 03, 04, 05, 06, 07, 10 as testing set, while [4]}, [5]} used Sequence 00 - 08 as training, and left 09, and 10 as testing set). Some of them are even based on different evaluation metrics (e.g. [1]}, [2]}, [3]} applied the KITTI official evaluation metric, while [9]}, [10]} applied absolute trajectory error (ATE) as their evaluation metric). All these factors bring difficulties to a direct and fair comparison across them. Moreover, the KITTI dataset is relatively simple (the vehicle only moves in 2D translation) and in small size. It is not convincing if only results on KITTI benchmark are provided without a comprehensive evaluation in the long-term real-world experiment. In fact, there is a growing need in creating a benchmark for a though system evaluation covering various environments, self-motions and dynamics.
