No Kahan: The summation of TSDF is conducted directly, i.e., without using the Kahan algorithm.

The evaluation is conducted on both ordinary sequences ICL-NUIM and our fast-motion ones FastCaMo-Synth. Table REF  compares the tracking accuracy of our method and the various baselines.
It can be seen that “No Overlap” causes the most accuracy drop among the three baselines, suggesting its importance in likelihood estimation. For Office_2 and Hotel_0, “No Normalize” produces large tracking error which can be attributed to exactly the cases of Figure REF (d), i.e., a very small portion of depth map lying within the valid range of TSDF due to very large rotation.
Although relatively less significant, “No Kahan” does affect the final tracking accuracy, hinting that numerical precision is a non-ignorable factor.
Multi-res PST alternation – Section .
To balance between the particle density of PST and the computational cost of fitness evaluation, we opt to work with three different combinations of PST resolution and depth map down-sampling rate: (1024, \(1/8\) ), (3072, \(1/16\) ), and (10240, \(1/32\) ). The three combinations are used alternatively across iterations.
In Figure REF , we evaluate such multi-res alternation scheme by comparing the accuracy and efficiency of camera tracking against various single resolution options on FastCaMo-Synth.
From the figure, the success rate of our alternation scheme is close to or higher than those high-res settings for almost all ATE thresholds, while our computational cost is at the same level as the low-res options. This verifies the effectiveness of our alternation scheme. It is the nature of swarm-guided PFO that enables this improvement: It is unnecessary to use a high-res PST at every step since 1) the best solution found by high-res PSTs would be passed along and inherited by the following iteration steps, and 2) the high-res depth maps associated with low-res PSTs yields more noise-robust likelihood estimation.
<FIGURE><FIGURE>
Quantitative Comparisons
We quantitatively evaluate our method against several state-of-the-art methods for both ordinary and fast-motion sequences.
Comparison on ordinary benchmarks.
In most existing datasets, the camera motion is usually slower than 1m/s. For those sequences, our method is able to achieve a comparable accuracy of camera tracking.
Table REF  compares ATE on the four sequences of ICL-NUIM between our method and the state-of-the-art online (DVO-SLAM [1]}, RGB-D SLAM [2]}, MRSMap [3]}, Kintinuous [4]}, VoxelHashing [5]}, ElasticFusion [6]}, BundleFusion [7]}) and offline (Redwood [8]}) methods.
Our method achieves comparable accuracy to the best-performing method BundleFusion which involves a global pose optimization by bundle adjustment.
The kt3 sequence is the most challenging one where there is barely no color or geometric features in some frames. Our method attains a decent accuracy even without any global optimization.
Similar comparative results can also be observed for TUM RGB-D; see the supplemental material.
<TABLE><TABLE>
Comparison on fast-motion benchmark – ETH3D.
The advantage of our method is best reflected on fast-motion sequences.
We first conduct a comparison on the three camera shake sequences of ETH3D-CS. As reported in Table REF , the average angular velocity of camera motion of these sequences is nearly 10 times faster than ICL-NUIM and TUM RGB-D. Our method is compared to BAD-SLAM [9]}, DVO-SLAM, ORB-SLAM2 [10]}, ElasticFusion and BundleFusion. ETH3D-CS contains IMU data which was not used by any of the methods being compared. The results are reported in Table REF . Our method was able to reconstruct all the three sequences with an acceptable tracking accuracy while none of the other methods could succeed on all.
camera_shake_3 is the fastest sequence on which all other method failed while ours achieves \(4.6\) cm ATE.
In the plots of Figure REF , we provide a breakdown study of per-frame pose accuracy for each camera shake sequence. In particular, we measure the frame-wise pose error using Translation Error (TE) and plot
the percentage of frames whose pose TE is below different thresholds.
BundleFusion drops frames which are lost tracking. We count the dropped frames as failed for all TE thresholds since the mechanism of frame dropping in BundleFusion is more involved than TE. Therefore, the comparison with BundleFusion may not be absolutely fair; we provide the results serving only as a reference.
It can be seen that our method achieves consistently more accurate per-frame pose estimation on all the three sequences.
<FIGURE>
Comparison on fast-motion benchmark – FastCaMo.
Table REF  provides a comparison on FastCaMo-Synth. Based on the ground-truth trajectories of this synthetic dataset, we compare our method with ElasticFusion and BundleFusion on the ten sequences. Note that, in what follows, we mainly compare to these two methods in the various fast-motion tests since they are the representative state-of-the-arts in point-based and volumetric fusion, respectively.
Compare to the two baselines, our method demonstrates significantly higher tracking accuracy for all the sequences except Room_1. The Room_1 sequence is quite challenging since the empty room contains barely no color or geometric feature. BundleFusion is able to reconstruct a small portion of the room with good tracking accuracy while dropping \(88.8\%\)  of the frames. On the contrary, our method and ElasticFusion do not drop frame. Similar situation is also found in Room_0, Office_2 and Office_3, on which our method achieves successful reconstruction. Figure REF  shows three examples of visual reconstruction results for FastCaMo-Synth.
<TABLE><TABLE>Similar to Figure REF , Figure REF  provides breakdown analyses of frame-wise pose accuracy for the ten sequences. These plots also reflect the difficulty of the above-mentioned sequences; see the slowly growing percentage. Our method again performs consistently better than the two baseline methods. For Room_1, our curve is notably higher than BundleFusion indicating that our method attains an overall better tracking when all frames are counted in.
In Table REF , we also conduct a comparison on 12 real captured sequences of FastCaMo-Real. See the results on the other 12 sequences in the supplemental material. Since these sequences only possess ground-truth reconstruction by LiDAR, we evaluate the reconstruction quality, i.e., the completeness and accuracy w.r.t. the ground-truth surfaces. As the reconstruction accuracy measures the RMS error over only the overlapping (inlier) regions between the reconstructed and the ground-truth surfaces, the numbers of the three methods are close to each other although ours does not involve any post-processing of global pose optimization or loop closure as the two alternative. In Table REF , we set the threshold of inlier to 15cm. When the threshold is set to 5cm, the average error is \(1\sim 3\) cm, with about \(10\%\)  drop in completeness. The reconstruction quality is best exposed by completeness on which our method is consistently and significantly better than the two alternatives. The visual results of reconstruction for this dataset can be found in Figure REF .
<FIGURE>
Comparison with [11]}.
Our method is closely related to [11]} in which a similar objective was defined based on the summation of TSDF over the unprojected 3D points of a depth map (Eq. (REF )). This makes their method correspondence-free too. Different from our method, however, they pursue a gradient descent approach and employ the Gauss-Newton method to minimize the summation of TSDF. This nonlinear least square approach finds difficulty in handling large rotation as shown in many related works [13]}. Moreover, the gradient can be undefined when the unprojected 3D points lie out of the valid range of TSDF when the camera undergoes a large transformation.
In Figure REF  (left column), we plot the objective function values (Eq. (REF ); the lower the better) of all frames and all sequences at different iteration steps.
For our method, we plot both the average values and the range of values of all particles.
For their method, we simply plot the actual objective function values at each iteration step.
Their values are mostly larger than the medium half of our particles (see the slim boxes).
This demonstrates that our method minimizes the objective with a significantly faster convergence for both of the two datasets.

Qualitative Results
Visual comparison of reconstruction.
We provide visual results on both synthetic and real fast-motion datasets.
Figure REF  shows the reconstructions on three sequences of FastCaMo-Synth (see the full set of results in the supplemental material). For each sequence, we show the reconstruction results along with the tracked camera trajectories for ROSEFusion (left), ElasticFusion (middle) and BundleFusion (right).
The tracked trajectories are overlaid on top of the ground-truth ones for visual contrasting. It can be seen that our trajectory matches well to the ground-truth. For BundleFusion, the trajectory segments corresponding to those remaining frames (did not get dropped) also conform well against the ground-truth.
<FIGURE>Figure REF  is a gallery of visual results for FastCaMo-Real, in a similar layout to Figure REF .
The linear velocity of camera movement (estimated based on our tracking) is color-coded (refer to Figure REF  for color bar) along the trajectories.
Note how our method is able to successfully reconstruct most of the sequences on which the other two methods failed. The Stairwell sequence is especially challenging since it contains repetitive structure but no prominent color features. Nevertheless, our method succeeds on it with good reconstruction quality. See also Table REF  for quantitative measurement. The Gym sequence is also quite challenging due to the large mirror on the wall. The mirror reflectance causes drastically missing/erroneous depth such that our method produces some drift along the wall which is any loop closure or global optimization mechanism.
<FIGURE>
Visualization of optimization process.
In Figure REF , we present a visualization of the per-frame pose optimization process.
Given the current frame (the green point cloud) with an initial pose, the goal is to optimize its pose to align it against the previous frame (the red point cloud). For each example, the initial and final configurations are shown at the two ends and the progressive evolution of PST is visualized in-between. In the upper part of the evolution sequence, we visualize the landscape of objective (Eq. (REF )) via 2D Isomap embedding [14]} of the 6D solution space, as well as the exploration path of our PST. The lower part shows the evolution of the 6D PST visualized as a 3D ellipsoid. The axis lengths of the 3D ellipsoid depict the rescaling factors along the three dimensions corresponding to translation and the color map encodes the scales along three rotational dimensions. The color map on the ellipsoid can be seen as the Gauss map of camera orientations of all particles. Note that the rotation here refers to relative rotation w.r.t. the pose of the previous step. Using relative rotation makes the color maps well aligned and the color change easy to observe.
The visualization clearly demonstrates the power of our random optimization scheme. Although the optimization landscape is highly non-convex due to large inter-frame transformations, our PFO guided by PST can always finds a good local optimum efficiently. It enables good quality frame-to-frame alignment without needing photometric or geometric feature detection and correspondence.
<FIGURE>
Complexity and Runtime Analysis
The time complexity of our random optimization is \(O(NM)\)  (\(N\)  particles and \(M\)  depth pixels) for each iteration. In our method, all particle fitness evaluation could be performed in parallel on the GPU (Section ).
We have implemented our core algorithm in C++ and CUDA. Both the main optimization pipeline and the volumetric fusion run on a workstation with an Intel® CoreTM i7-5930K CPU @ 3.50GHz \(\times \)  12 with 32G RAM and an Nvidia GeForce RTX 2080 SUPER GPU with 8G memory. To enable flexible scanning, we implemented a front-end program running on a laptop for RGB-D capturing, compressing, and streaming to the workstation via WiFi.
Our pipeline runs with a framerate of 30Hz for all shown test sequences. In fact, we control the time budget of random optimization process within 30ms to maintain a 30Hz framerate. This time slot allows the optimization to run for at least 20 iterations which is well-sufficient for all the sequences tested as our method converges with 4 iterations for ordinary sequences and 10 for fast-motion ones. The time for volumetric depth fusion is 3ms per frame.
The readers are welcomed to watch our accompanying video.

Limitation and Failure Cases
Our method has several limitations. Firstly, since our method is purely geometrically-based, it would naturally fail when no prominent geometric feature can be found in a serial of consecutive frames. Figure REF (a) shows such a cases where the camera is scanning towards a flat wall (fr3_nst of TUM RGB-D). When the camera motion is not too fast to be contaminated by motion blur, one can always complement our method with photometric based pose optimization. Secondly, when depth is severely missing due to reflectance or absorbance of light, our method would fail to track and fuse; see Figure REF (b). Thirdly, while it is hard to gauge the upper limit of camera moving speed, our method could still fail under extremely fast motion (e.g. \(\sim 5\) m/s in the example of Figure REF (c)). Finally, a more fundamental issue is that our current method does not include a loop closure detection and global pose optimization. Although we have tested scanning an indoor space of about 300m\(^2\)  for 10K frames without introducing observable drift, it is almost impossible to track without drifting when the sequence becomes extremely long especially some challenging frames were encountered along the way. The Gym sequence in Figure REF  is a failure example of such kind. Nonetheless, our method can of course be enhanced by loop closure and/or global pose optimization.
<FIGURE>
Discussion and conclusions
With our work, we wish to bring it to the community's attention that online dense reconstruction under fast camera motion is a practically useful problem and might demand a new paradigm of solution. The traditional approaches are mostly based on feature matching plus gradient descent. They do find difficulties in handling fast-motion camera tracking due to motion blur of color images and highly nonlinear/non-convex optimization landscape.
Our attempt achieves a satisfactory outcome thanks to two major design choices. First, we pursue a depth-only approach and define an effective objective function based on depth-to-TSDF conformance, thus avoiding explicit feature detection and correspondence.
Second, we propose to solve the per-frame optimization based on a novel particle-filter-based random optimization with swarm guidance, leading to a robust and efficient algorithm.
Our work serves merely as a starting point, which we hope would inspire a rich set future directions:

How to integrate loop closure detection and global pose optimization into the random optimization framework? It is interesting to study random optimization based pose graph optimization or bundle adjustment when large transformations are involved.

How to unify traditional approach and random optimization into a single framework? When camera motion slows down, it is of course a good option to enhance camera tracking with photometric features. Here, how to bridge and switch smoothly between the two technique is worth of investigating.

How to fuse multi-modal input to further improve the robustness of fast-motion camera tracking?

How to realize autonomous reconstruction with mobile robots or drones with our technique? There are several technical issues to address, e.g., how to integrate our method with online motion planning [15]}, [16]}, [17]}, [18]}.

It is interesting to extend our random optimization to distributed bundle adjustment [19]}, [20]} for collaborative reconstruction by a network of robots or drones. Here, the small overlap between distributed nodes also calls for a robust pose optimization.

Another promising and interesting direction is to explore the combination of random optimization and deep priors for online and/or semantic scene reconstruction [21]}, [22]}, [23]}.

We thank the anonymous reviewers for their valuable comments and suggestions. We are grateful to Yuefeng Xi and Yao Chen for their effort in the preparation of the FastCaMo dataset. This work was supported in part by National Key Research and Development Program of China (2018AAA0102200), NSFC (61532003, 62002376) and NUDT Research Grants (ZK19-30).
