Computer Science
Computer Science-2021.gz:143276
Computer Science
177fd056dda5943e01378b94d7f8ba4177985afb
HybVIO: Pushing the Limits of Real-time Visual-inertial Odometry

We present HybVIO, a novel hybrid approach for combining filtering-based visual-inertial odometry (VIO) with optimization-based SLAM. The core of our method is highly robust, independent VIO with improved IMU bias modeling, outlier rejection, stationarity detection, and feature track selection, which is adjustable to run on embedded hardware. Long-term consistency is achieved with a loosely-coupled SLAM module. In academic benchmarks, our solution yields excellent performance in all categories, especially in the real-time use case, where we outperform the current state-of-the-art. We also demonstrate the feasibility of VIO for vehicular tracking on consumer-grade hardware using a custom dataset, and show good performance in comparison to current commercial VISLAM alternatives. An open-source implementation of the HybVIO method is available at https://github.com/SpectacularAI/HybVIO


Introduction
Visual-inertial odometry (VIO) refers to the tracking of the position and orientation of a device using one or more cameras and an inertial measurement unit (IMU), which, in this context, is assumed to comprise of at least an accelerometer and a gyroscope. A closely related term is visual-inertial SLAM (VISLAM), which is typically used to describe methods that possess a longer memory than VIO: simultaneously with tracking, they produce a map of the environment, which can be used to correct accumulated drift in the case the device revisits a previously mapped area. Without additional inputs, these methods can only estimate the location relative to the starting point but provide no global position information. In the visual-inertial context, the orientation of the device also has one unsolvable degree of freedom: the rotation about the gravity axis, or equivalently, the initial compass heading of the device.
VISLAM is the basic building block of infrastructurefree augmented reality applications. VIO, especially when fused with satellite navigation (GNSS), can be applied to tracking of various types of both industrial and personal vehicles, where it can maintain accurate tracking during The latter remains the leader in the EuRoC post-processing category, while our method yields the best online results.

Related work
Our VIO module is a stereoscopic extension of PIVO [34] and, consequently, a member of the MSCKF family of VIO methods that stems from [18]. Other recent methods belonging to the same class include the hybrid-EKF-SLAM (cf. [16]) method LARVIO [26] and S-MSCKF [37], which extends the original MSCKF to stereo cameras. These methods, following their EKF-SLAM predecessors (e.g., [8]), use an Extended Kalman Filter (EKF) to keep track of the VIO state. They track the Bayesian conditional mean (CM) of the VIO state and keep it in memory together with its full covariance matrix, which limits the practical dimension of the state vector in real-time use cases. An alternative to the above filtering-based methods are optimization-based approaches, which compute a Maximum A Posteriori (MAP) estimate in place of the conditional mean, and instead of storing a full covariance matrix, they may use sparse Bayesian factor graphs. The optimization-based methods are often stated to be more accurate than filtering-based methods and many recent publications prefer this approach. Notable examples include OKVIS [15], MARS-VINS [23], ORB-SLAM3 [5], BASALT [41] and Kimera-VIO [28].
However, there are also disadvantages compared to filtering-based methods, namely, the lack of uncertainty quantification capabilities and the difficulty of marginalizing the active state on all the past data. Our method includes elements from both approaches, as filtering-based VIO is loosely coupled with optimization-based SLAM module. Previously, good results for post-processed trajectories have been reported with hybrid filtering-optimization approaches in, e.g., [27] and [1]. However, our approach differs from these tightly-coupled solutions. In addition to the more common sparse approaches above, various alternatives have been proposed (see, e.g., [2,7,9,38,42]). For a more extensive survey of recent and historical methods, we refer the reader to [5] and the references therein.

VIO state definition
Similarly to [34] and [18], we construct the VIO state vector at time step t k , using the poses π (j) k ) ∈ R 3 × R 4 of the IMU sensor at the latest input sample (j = 0) and a fixed-size window of recent camera frames (j = 1, . . . , n a ). The other elements in Eq. (1) are the current velocity v k ∈ R 3 , a vector of IMU biases b k = (b a k , b ω k , diag(T a k )) (see Eq. (3)), and an IMU-camera time shift parameter τ k ∈ R, utilized as described in [25].
In the EKF framework, the probability distribution of the state, given all the observations y 1:k until time t k , is modeled as Gaussian, x k |y 1:k ∼ N (m k|k , P k|k ). We model the orientation quaternions as Gaussians in R 4 and restore their unit length after each EKF update step.

IMU propagation model
The VIO system is initialized to (m 1|1 , P 1|1 ), where the current orientation q (0) 1|1 is based on the first IMU samples equally to [35]. The other components of m 1|1 are fixed (zero or one), and P 1|1 is a fixed diagonal matrix. No other measures are required to initialize the system.
Following [34], IMU propagation is performed on each synchronized pair (ω k , a k ) of gyroscope and accelerometer samples as an EKF prediction step of the form . The function f k updates the pose and velocity by the mechanization equation where the bias-corrected IMU measurements are computed asã k = T a k a k − b a k andω k = ω k − b ω k . In our model, the multiplicative correction T a k ∈ R 3×3 is a diagonal matrix. Contrary to the approach used in [18], this does not involve linearization errors that could cause the orientation quaternion to lose its unit length, since Ω[·] ∈ R 4×4 (cf. [39] or Eq. (A3)) is an orthogonal matrix.
As an extension to [34], we assume the following model for the IMU biases: where the parameters α, σ in the Ornstein-Uhlenbeck [40] random walks ϵ k ∼ N (0, σ 2 2α [1 − exp(−2α∆t k )]) can be adjusted to match the characteristics of the IMU sensor.

Feature tracking
Similarly to [18], our visual updates are based on the constraints induced by viewing certain point features from multiple camera frames. We first utilize the Good Features to Track (GFTT) algorithm [33] (or, alternatively, FAST [29], cf. Table 1) for detecting an initial set of features, which are subsequently tracked between consecutive frames using the pyramidal Lucas-Kanade method [17] as implemented in the OpenCV library [3]. We also use the reprojections of previously triangulated 3D positions (cf. Sec. 3.6) of the features as initial values for the LK tracker whenever available, which improves its accuracy and robustness, especially during rapid camera movements.
As in [34], features lost due to falling out of the view of the camera, or any other reason, are replaced by detecting new key points whenever the number of tracked features falls below a certain threshold. A minimum distance between features is also imposed in the detection phase and sub-pixel adjustment is performed on the new features.
In the case of stereo data, we detect the new features in the left camera frame, and find the matching points in the right camera frame, also using the Lucas-Kanade algorithm. This technique allows the use of raw camera images without a separate stereo rectification phase. The temporal tracking is only performed on the left camera frames, and the matches in the right frame are recomputed on each image. In addition, we reject features with incorrect stereo matches based on an epipolar constraint check.

Visual update track selection
A feature track y j with index j ∈ N is, in the monocular case, a list of pixel coordinates (y j i ∈ R 2 ), or pairs of coordinates y j,R i , y j,L i ∈ R 2 in stereo. The track is valid for a range of camera frame indices i = i j min , . . . , i j max , where i j min corresponds to the frame where the feature is first detected and i j max the last frame where it is successfully tracked. In the stereo case, both y j,R and y j,L must be continuously tracked as described in Sec. 3.3.
On camera frame i ≤ i j max , denote by b(i) the camera frame index of the last pose π (na) k(i) stored in the VIO state, and by b(i, j) = max(b(i), i j min ) the corresponding minimum valid camera frame index for track j. Unlike [34], we do not always use all key points in the track, but select the subset of indices where i ′ < i is the last frame on which feature j was used for a visual update (see Sec. 3.6). In other words, we avoid "re-using" the parts of the visual tracks that have already been fused to the filter state in previous visual updates.
Instead of using all available tracks on frame i (denoted here by U i ), we pick the indices at random from the subset . The corresponding reprojections are drawn in green for successfully visual updates and red for tracks that failed the χ 2 outlier test (cf. Sec. 3.6). The end of the track with the larger circle matches the current frame, and the long gap between the two last key points in some tracks is a consequence of Eq. (5). LK-tracked features that were not used on this frame are drawn in magenta.
corresponding to longer-than-median tracks, where the length metric is defined as In the case of stereo data, Eq. (7) is computed from the left camera features only (y j l := y j,L l ). Tracks are picked until the target number of visual updates have succeeded or the maximum number of attempts has been reached. Contrary to [18], our visual update is performed individually on each selected feature track.
The track selection logic, as well as some other aspects of our visual processing, are illustrated in Fig. 2. The essence of this process is reducing computational load by focusing on the most informative visual features. The technique is inspired by the stochastic gradient descent method, and allows us to maintain good tracking performance with a very low number of n target = 5 active features per frame (see Sec. 4 for details).

Camera model
Our method supports two different camera models: the radial-tangential distorted pinhole model and the Kannala-Brandt fisheye model [14] with four radial parameters. We assume that the calibration parameters of the camera model are known and static.
The model ϕ C : R 2 → R 3 for the camera C ∈ {L, R}, maps the pixel coordinates of a feature to the ray bearing in R 3 , and its restriction to the unit sphere S 2 is invertible, ϕ −1 C : S 2 → R 2 . We define the undistorted normalized pixel coordinates for camera C as where ρ(x, y, z) = z −1 [x, y] ⊤ is the perspective projection. When required, the extrinsic camera coordinates (as a camera-to-world transformation) are formed from the IMU coordinates (p, q) in the VIO state as where R(q) is the rotation matrix corresponding to the (world-to-IMU) quaternion q and the IMU-to-camera transformation T C →IMU ∈ R 4×4 is determined for each camera C in the calibration phase.

Visual updates
Following [34], our visual update is based on triangulation of feature tracks y j into 3D feature points p j and comparing their reprojections to the original features. In particular, this information is applied as an EKF update of the form with where denotes triangulation using the selected subset S = S(k, j) of the feature track pointsỹ j S and the corresponding poses π (S) in the VIO state. The reprojections of the triangulated point on the corresponding frames are computed as where C ⊂ {L, R} is the set of cameras (with two elements in stereo and one in mono) and projects a 3D point p * onto the normalized pixel coordinates of camera C at pose π (cf. Eq. (9)). We perform a χ 2 outlier check before applying the EKF update corresponding to Eq. (10). In case of failure, we cancel the update and proceed to the next feature track as described in Sec. 3.4. Our method and PIVO differentiate from the other MSCKF variants with Eq. (11): with tedious and repeated application of the chain rule of differentiation, one can compute the Jacobian J h with respect to the state x, which renders the linearization directly usable in an EKF update step and the null space trick introduced in [18] becomes unnecessary. We recompute the linearization for each update (x 0 = x k|k−1,j−1 ).

Triangulation
Our triangulation follows [34]: the function TRI in Eq. (12) uses a point p * 0 computed from two camera rays as an initial value and then optimizes it by minimizing the reprojection error (cf. Eq. (13)) using the Gauss-Newton algorithm. The entire optimization process needs to be differentiated with respect to the VIO state for computing the Jacobian for the EKF update (see App. A for more details). In the stereo case, p * 0 is computed using the most and least recent ray from the left camera only. If the triangulation produces an invalid 3D point, i.e., behind any of the cameras, the feature track is rejected.

Pose augmentation
As in [18], the pose trail π (·) in the VIO state is populated and updated in a process known as pose augmentation or stochastic cloning [30]: When a new camera frame is received at time t k , a copy of the current, IMU-predicted, pose π (0) is inserted into the slot π (1) , an older pose, π (d) is discarded and the remaining poses are shifted accordingly. This operation can be performed as an EKF update step as described in [34] or an EKF prediction step where n 1 := dim(x) − 7 · n a . The typical choice is always discarding the oldest pose, that is, d = n a , which makes the pose trail effectively a FIFO queue. However, by varying the discarded pose index, it is also possible to create more complex schemes that manage the pose trail as a generic n a -slot memory.
Towers-of-Hanoi scheme We use where LSB(i) denotes the least-significant zero bit index (0-based) of the integer i. This process combines a fixedsize n FIFO part with a Towers-of-Hanoi backup scheme that increases the maximum age of the stored poses by adding (exponentially) increasing strides between them. It is also possible to vary d i dynamically based on, e.g., the number of tracked features in the corresponding camera frames. In particular, if a certain frame with corresponding historical pose π (l) no longer shares any tracked feature points with the latest camera frame, we always discard it by setting d = l instead of applying Eq. (18). The more complex scheme allows reducing the dimension of the state and computational load by using less poses n a more efficiently. if KEYFRAMEDECISION(K) then 6: extend K with more key points from the image I 7: compute ORB descriptors for all kps. K cf. [31] 8: match existing map points with K 9: triangulate new map points as in [19] 10: deduplicate map points as in [19] 11: T slam ←LOCALBUNDLEADJUSTMENT(K) 12: cull map points and key frames as in [19] 13: end if 14: Tprev ← Tin stored for the next task, like T slam 15:

19:
if i = 1 (mod N ) then every N th frame 20: T vio→slam ← block on F wait for previous result 21:

. Stationarity detection
The common important special case, where the tracked device is nearly stationary, requires some special attention in MSCKF-like methods. In particular, when the device is stationary, the pose augmentation schemes in Sec. 3.8 can quickly cause the pose trail to degenerate into n a (nearly) identical copies of a single point, which can destabilize the system. This concerns especially in the monocular scenario, as the triangulation baselines consequently approach zero. We follow an approach also presented in [26], where certain frames are classified as stationary, and not stored permanently in the pose trail. To this end, we evaluate the movement of the tracked features in pixel coordinates between consecutive frames. Namely, if for a certain fixed threshold m min , we perform a pose unaugmentation operation as an EKF prediction step: where ε u ∼ N (0, σ 2 u I 7 ) with a large variance (e.g., σ u ≈ 10 6 ). This causes the previously augmented pose to be discarded (after it has been used for a visual update) and, as a result, most of the frames remain in the pose trail as long as the device remains stationary and Eq. (19) holds.

SLAM module
On a high level, our method consists of two loosely coupled modules: the filtering-based VIO module, which is described in previous sections, and an optional, optimizationbased SLAM module, which uses VIO as an input. We used OpenVSLAM [36], a re-implementation of the ORB-SLAM2 [20] method, as the basis for the implementation. Consequently, many of the details of our SLAM module coincide with ORB-SLAM2 or its predecessor, ORB-SLAM [19]. We describe these parts of the system briefly and refer the reader to the aforementioned works for details.
SLAM map structure A sparse SLAM map consists of key frames and map points, which are observed as 2D key points in one or more key frames. Equally to ORB-SLAM, our map point structure includes the viewing direction, valid distance range, and an ORB descriptor, while the key frame consists of a list of key points and a camera pose.
ORB detection and matching Unlike ORB-SLAM2, we only consider the data in the left camera frames in the stereo case for simplicity, even though our VIO module uses data from both cameras. Each key point is associated with an ORB descriptor, which, in ORB-SLAM, are computed using a multi-pyramid-level FAST detector (cf. [31]). In addition to this, we use the pixel coordinates of the Lucas-Kanade tracker features (cf. Sec. 3.3) as key points and compute their ORB descriptors on a single pyramid level.
New matches between key points are also searched from the n matching key frames spatially closest to the current key frame. As in ORB-SLAM, this is conducted both with 3D matching, where an existing map point is reprojected to the target key frame, and with 2D ORB matching, where the descriptors in two key frames are compared. The latter approach can be used to create new, previously untriangulated map points. Previously visited areas can be recognized here without a separate loop closure procedure (cf. [5], §VII) when the accumulated error is low enough. In the SLAM module, we use linear triangulation for new map points. VIO integration A high-level structure of our hybrid VIO-SLAM approach is given in Alg. 1, where lines 17-25 describe a simple and efficient parallel scheme: the VIO state is sent to the SLAM module, which outputs a VIOto-SLAM coordinate mapping. The result is read asynchronously on the next key frame candidate, which we add every N = 8 frames. The returned coordinate mapping is not required to match the latest pose and we input a fixeddelay-smoothed VIO pose π (N ) to SLAM, while outputting an undelayed pose on each input frame, using the most recent available T vio→slam .
We initialize the new key frame at a pose transformed using recent key frame and input poses as shown on 2, where MATCHGRAVITYDIR(T, T in ) ensures that T = RealSense T265 u-blox RTK-GNSS iPhone 11 Pro Huawei Mate 20 Pro rotate z (θ)T in for some θ, that is, the gravity direction in the initial key frame pose matches that of the VIO input. The KEYFRAMEDECISION passes if the distance from the previous key frame exceeds a fixed threshold (15cm), or if less 70% of the feature tracks are covisible in it.
Bundle adjustment Local bundle adjustment (cf. [19]) is performed on n BA nearest neighbor key frames (by Euclidean distance) of current key frame. In addition, we use the relative input pose changes T −1 in,i T in,i−N from VIO as extra penalty terms between consecutive key frames to limit the deviation between the SLAM and VIO trajectory shapes. Our penalty weights for both position and orientation are inversely proportional to the time interval t i −t i−N .
Post-processing As the post-processed trajectory in Sec. 4, we use the final positions of the key frames and interpolate between them using the online VIO trajectory to produce a pose estimate for each input frame.

Experiments
We compared our approach to the current-state-of-theart [5,24,28,41,42] in three academic benchmarks. Two baseline methods, OKVIS [15], for which results are reported in all of these, and PIVO [34], the most similar method, were also included. More comprehensive comparisons including older methods and visual-only approaches can be found in [5] and [41], which are also our primary sources of the results for other methods in Tables 2 and 4. Table 2 gives our results for the EuRoC MAV [4]. Similarly to [41], we clearly separate the online and post-processed cases. The former corresponds to real-time estimation of the current device pose using the data seen so far. The latter, also called mapping mode, aims to produce an accurate post-processed trajectory using all data in the sequence. In Bayesian terms, they are the filtered and smoothed solutions, respectively. Our approach yields state-of-the-art performance in all categories: monocular and stereo, as well as online and post-processed. Furthermore, we outperform BASALT [41] in the online stereo category, and consequently, report the best real-time accuracy ever published for EuRoC. The authors of ORB-SLAM3 [5], the best method in the postprocessing categories, do not report online results, but according to our experiments with the published source code (App. B.1), the online performance is not good (cf. Fig. 1). Table 3, we examine the accuracy and computational load with four different configurations detailed in Table 1. In addition, we measure the effect of the improvements presented in Sec. 3. Removing RANSAC, IMU bias random walks Eq. (4), track selection logic (Eq. (6)), stationarity detection Sec. 3.9 or Eq. (18) results in measurable reductions in accuracy. We also removed all novel features simultaneously and this configuration represents our reimplementation of the PIVO method. The average performace of the reimplementation is comparable to the numbers published in [34] and reproduced in Table 2, but the individual numbers are not identical. We presume this is mostly due to random variation and minor differences in unpublished implementation details.

Parameter variations and timing In
The computational load is evaluated on two different machines: A high-end desktop computer with an AMD Ryzen 9 3900X processor, and a Raspberry Pi 4 with an ARM Cortex-A72 processor for simulating an embedded system. Both systems run Ubuntu Linux 20.04 and the maximum RAM consumption in the EuRoC benchmark was below 500 MB. For the VIO-only (non-SLAM) variants, we measure single-core performance so that our method runs in a single thread, but auxiliary threads are used for decoding the EuRoC image data from disk, simulating a real-time   use case where this data is processed online. In the SLAM case, we use two processing threads: one for SLAM and one for VIO, as described in Alg. 1. The EuRoC camera data is recorded at 20 FPS and thus values less than 50 ms per frame correspond to real-time performance, which is achieved in all unablated (i.e. including Eq. (5)) online cases on the desktop CPU and the fast VIO configurations on the embedded processor. Even though the SLAM module increases accuracy in both monocular and stereo cases, the VIO-only mode also has very good performance compared to other approaches. In particular, by comparing the results to Table 2, we note that our VIO-only stereo method outperforms VINS-Fusion even with the fast settings. An example trajectory with this configuration is shown in Fig. 1, which also illustrates how the EKF covariance can be used for uncertainty quantification with essentially no extra computational cost.

TUM VI and SenseTime VISLAM
We also evaluate our method on the room subset of the TUM VI benchmark [32] (Table 4) and the SenseTime VISLAM Benchmark [12] (Table 5) which measures the performance of monocular VISLAM with smartphone data. Both benchmarks measure post-processed SLAM performance using the RMS-ATE-SE(3) metric. In TUM VI Room, we rank second, after ORB-SLAM3. In the SenseTime benchmark, we outperform the authors' own proprietary method, Sense-SLAM, on the average. More parameter configurations are

Commercial comparison dataset
To evaluate the performance of our method compared to (consumer-grade) commercial solutions, we collected a custom dataset using the equipment depicted in Fig. 3. Each of the devices features a commercial VISLAM algorithm, whose outputs can be recorded, together with the camera and IMU data the device observes. This allows us to compare the accuracy of our approach to the outputs of each commercial method with the same input. Fig. 4 shows the output trajectories of the experiment for two different sequences: Fig. 4a shows a vehicular test, where devices were attached to a car, exactly as shown in Fig. 3. In Fig. 4b, the same devices were rigidly attached to a short rod and carried by a walking person.
While all methods performed relatively well in the walking sequence, this is not the case in the more challenging vehicular test, which is not officially supported by any of the tested devices. However, our method (and notably, also ARKit) are able to produce stable tracking in all cases. We also clearly outperform Intel RealSense in both sequences.

Discussion and conclusions
We demonstrated how the PIVO framework could be extended to stereoscopic data and improved into a highperformance independent VIO method. Furthermore, we demonstrated a novel scheme for extending it with a parallel, loosely-coupled SLAM module. The resulting hybrid method outperforms the previous state-of-the-art in realtime stereo tracking.
The measurement of VIO-only performance is also relevant since the relative value of different VISLAM capabil-ities are dependent on the use case. For example, in vehicular setting where GNSS-VIO fusion is utilized to perform tracking during GPS breaks, e.g., in tunnels; loop closures or local visual consistency may be irrelevant compared to uncertainty quantification and long-range accuracy. In this case, we presume that a light-weight VIO solution is more suitable than full VISLAM. We also demonstrated the feasibility of our method for vehicular tracking.
With slight trade-off for accuracy, real-time performance was demonstrated on a Raspberry Pi without the use of GPU, VPU or ISP resources, which could further improve the speed and energy consumption of visual processing. The alternative approaches report similar real-time accuracy only on high-end desktop CPUs.
Note that several aspects of our VIO are simplified compared to other recent publications. In particular, the initialization presented in Sec. 3.2 is extremely simple compared to the intricate mechanisms in [5] and [6]; we do not use the First-Estimate Jacobian methodology [11], nor model orientations as probability distributions on the SO(3) manifold [10]. Implementing some of these techniques could further improve the accuracy of this approach.
Similarly to BASALT, our SLAM module lacks a separate loop closure procedure, since on the tested datasets, the low online drift could always be corrected in other SLAM steps. However, a loop closure approach similar to [19] could be valuable in challenging, large-scale benchmarks.

Quaternion update by angular velocity
If ω = (ω x , ω y , ω z ) represents a constant angular velocity, then a world-to-local quaternion q = (q w , q x , q y , q z ) ⊤ representing the orientation of a body transforms as where Note that the matrix looks different if a local-to-world quaternion representation is used (cf. [39]).

B. Additional experiments B.1. Comparison to ORB-SLAM3 and BASALT
We processed the EuRoC dataset using the publicly available source code of ORB-SLAM 1 and BASALT 2 to compare execution times and reproduce the metrics reported in [5,41]. The ORB-SLAM3 example code was modified to output its intermediary results, namely the latest key frame pose after each input frame, without any changes to the actual algorithm. The results are presented in Table B1.
All tests were performed on the same machine (the Ryzen setup described in Sec. 4.1) using two configurations: In the first, unrestricted configuration, the methods were allowed to utilize all 12 CPU cores in the system in parallel. BASALT was most efficiently parallelized and its processing time per frame was significantly lower than in the other methods. In the second configuration, we restricted the entire process (including input decoding) to use only 2 parallel CPU cores. In this mode, the processing times of the three methods were comparable.
The common results in Table B1 are similar to those reported in [5,41] (and reproduced in Table 2). None of the methods (including ours) yield strictly equal results on different machines, which can explain the small remaining discrepancies. ORB-SLAM3 outputs also varied significantly between runs, but the online instability seen in Fig. B1 was consistently observed.

B.2. Ablation studies
An ablation study equivalent to Table 3 for the TUM Room dataset is presented in Table B2. In the monocular case, the results are mixed: disabling individual novel features does not consistently improve the metrics, and the simple postprocessing actually degrades the SLAM results. However, most of our configurations, notably also Fast stereo VIO, outperform all previous methods except ORB-SLAM3 (cf .  Table 4) The SenseTime benchmark results in Table B3 are more consistent and similar to Table 3: the novel features are all beneficial and the baseline PIVO implementation is not stable. However, our simple post-processing method is not able to improve the results compared to the online case. Table B4 studies the effect of varying the parameters presented in Table 1 individually. Deviations from the selected parameters caused degraded metrics, except increasing n BA improves the baseline results. However, this larger bundle adjustment problem is too heavy for the real-time use case and therefore we only use it in the post-processed setting.  Figure B2. Vehicular experiment 2, using the setup in Fig. 3

B.3. Vehicular
This section includes additional vehicular experiments ( Fig. B2 and Fig. B3) using the setup shown in Fig. 3, as well as results from a slightly modified setup (shown in Fig. B5a), where we have added ZED 2 as a new device.
ZED 2 also has a proprietary VISLAM capability, but it did not perform well in the vehicular test cases (see Fig. B4 for an example) and we omitted it in the other sequences to avoid frame drop issues experienced when recording ZED 2 input and tracking output data simultaneously. The ZED 2 camera data was recorded at 60FPS but utilized at 30FPS.
We used the normal VIO mode (see Table 1) for all vehicular experiments. Stereo mode was used with both stereo camera devices.    Figure B4. Vehicular experiment 4. A slow drive around a parking lot, recording the setup shown in Fig. B5a. Unlike the following experiments with ZED 2, the proprietary tracking output from ZED 2 is compared to HybVIO using the same input data.
u-blox RTK-GNSS iPhone 11 Pro Zed 2 (a) Second car experiment setup: GNSS is used as ground truth.
The iPhone records ARKit and its input data simultaneously. ZED 2 records camera (stereo rolling shutter at 60FPS) and IMU data.   Figure B5. Additional vehicular experiments with higher velocities (∼ 80km/h), in which ARKit also fails. In this case, HybVIO performs better on the same data.
The visual tracking and update status are visualized on the left and right camera frames, similarly to Fig. 2 but with different colors: reprojections are in white, successfully updated tracks in black and failed tracks in blue. The lower part of the video shows the online track (x and y coordinates) in blue and ground truth in orange. Triangulated points are shown as small black circles. The online track is automatically rotated to optimally match the ground truth, after approximately 10 seconds.
(b) euroc-v1-02-medium normal-SLAM (https://youtu.be/7j1rYoD pPc) Similar to the previous video, but HybVIO is running in the normal SLAM mode on the sequence V1-02. The triangulated SLAM map points in the current local map are shown as yellow in the lower right subimage, and their reprojections as orange on the (left) camera image. The LK-tracked features (cf. Alg. 1) that correspond to SLAM map points are shown as yellow circles on the camera image.
The left part of the video shows the SLAM map. Key frame camera poses are shown in light blue. In the beginning, the triangulated points in the local map are shown in red. At time 00:27, the colors are changed to show the observation direction of the map point. The covisibility graph is shown first at 00:15, in a yellowgreen color. We consider a pair of key frames adjacent in this graph if they observe at least N neigh = 10 common map points.
(c) vehicular-experiment-6 (https://youtu.be/iVNicL S14Y) Visualizes the vehicular experiment in Fig. B5e on a map (Hyb-VIO on ZED 2). The VIO trajectory is aligned using a fixed angle and offset. The local VIO trajectory is formed using the pose trails (cf. Sec. 3.1) in the VIO state. The traffic light stops are automatically cut from the video (based on the VIO velocity estimate). Despite generally good RTK-GNSS coverage in the area, the sequence includes a GNSS outage in a tunnel, starting at time 02:32 in the video.