COVINS-G: A Generic Back-end for Collaborative Visual-Inertial SLAM

Abstract: Collaborative SLAM is at the core of perception in multi-robot systems as it enables the co-localization of the team of robots in a common reference frame, which is of vital importance for any coordination amongst them. The paradigm of a centralized architecture is well established, with the robots (i.e. agents) running Visual-Inertial Odometry (VIO) onboard while communicating relevant data, such as e.g. Keyframes (KFs), to a central back-end (i.e. server), which then merges and optimizes the joint maps of the agents. While these frameworks have proven to be successful, their capability and performance are highly dependent on the choice of the VIO front-end, thus limiting their flexibility. In this work, we present COVINS-G, a generalized back-end building upon the COVINS framework, enabling the compatibility of the server-back-end with any arbitrary VIO front-end, including, for example, off-the-shelf cameras with odometry capabilities, such as the Realsense T265. The COVINS-G back-end deploys a multi-camera relative pose estimation algorithm for computing the loop-closure constraints allowing the system to work purely on 2D image data. In the experimental evaluation, we show on-par accuracy with state-of-the-art multi-session and collaborative SLAM systems, while demonstrating the flexibility and generality of our approach by employing different front-ends onboard collaborating agents within the same mission. The COVINS-G codebase along with a generalized front-end wrapper to allow any existing VIO front-end to be readily used in combination with the proposed collaborative back-end is open-sourced. Video:this https URL

\section{Introduction}
\label{sec:introduction}
The promise of automating tedious tasks and keeping humans away from hazardous environments has driven the development of robotic systems to reach tremendous capabilities.
With the increased maturity of the single agent systems, the interest in teams of robots collaborating with each other has been steadily rising.
The use of such a team of robots collaborating towards a common goal promises to increase the efficiency and robustness of a mission by distributing the tasks among the participating agents and has been proposed for a variety of different tasks, such as in search-and-rescue missions \cite{srr}, archaeological mapping \cite{col_arch_map}, precision agriculture \cite{agri-robots}, and surveillance \cite{surveillance}.
Sharing information across the agents not only enables the robots to perform a task faster but also enables the individual agents to make better-informed decisions as they can profit from information beyond their own gathered experience, as shown in \cite{bartolomei2020multi}.

\begin{figure}
    \centering
    \includegraphics[width=0.48\textwidth]{images/expt_eurocv3.png}
    \caption{Collaborative SLAM estimate for 5 agents (EuRoC MH Dataset) running different VIO front-ends. The COVINS-G back-end does not require map points (shown here only for visualization) and thus, is compatible with any arbitrary VIO front-end.
    }
    \label{fig:irchel_5ag}
    \vspace{-20pt}
\end{figure}

However, in order for the robots to work towards any higher-level goal, they need to be aware of their surroundings and their pose in their workspace.
Moreover, for the robots to be able to collaborate, the knowledge of the pose of all other robots within the team is crucial.
The use of external sensors, such as GPS or motion capture systems can provide such a shared reference frame enabling the coordination of the robotic team, however, for many practical applications, such data is not reliable or simply not available in the first place.
For example, the uncertainty of GPS measurements can be in the order of tens of meters close to larger structures (e.g. within a city) or not available at all inside buildings or underground.
In order to remove dependencies on external sensors, research into Simultaneous Localization And Mapping (SLAM) has made significant progress.
In particular, the use of cameras and Inertial Measurement Units (IMUs) for visual-inertial SLAM has proven to provide robustness and accuracy, which led to their deployment onboard products in the market already.
With the increasing maturity of single-agent vision-based SLAM techniques, the extension towards multi-agent SLAM as a core enabler for robotic collaboration in real-world scenarios has been increasingly gaining interest, sparking a variety of works addressing multi-agent SLAM \cite{multi-uav-slam, ccm-slam, cvi-slam, door-slam, kimera-multi, covins}.
While the capabilities and robustness of the developed systems have been steadily increasing, due to their tailored architectures, their modularity is often sacrificed.
As a result, any exchange and modification of the front-end onboard such systems most often requires significant effort to adapt the back-end to the structure.
In this spirit, this work addresses this issue by proposing a generic back-end solution built on top of the architecture of \cite{covins}.
As the proposed approach requires only 2D features, it is agnostic to the front-end running onboard each agent and even allows mixing different front-end algorithms on the different agents during the same mission as illustrated in Figure \ref{fig:irchel_5ag}.
In summary, the contributions of this work are the following:
\begin{itemize}[leftmargin=10pt]
    \item a generalized collaborative SLAM back-end, which requires only 2D keypoints and a pose estimate to fuse the estimates from multiple agents, enabling the use of any arbitrary VIO and stereo front-ends onboard each agent,
    \item a publicly available codebase\footnote{\href{https://github.com/VIS4ROB-lab/covins}{https://github.com/VIS4ROB-lab/covins}}, which is integrated with the framework of \cite{covins}. Furthermore, a front-end wrapper is provided, to support any off-the-shelf front-end, and
    \item an extensive evaluation of the proposed back-end on both the EuRoC dataset \cite{euroc} as well as newly collected datasets. Our evaluation reveals the flexibility of the proposed approach, using and combining different types of front-ends onboard collaborating agents within the same mission.
\end{itemize}

\vspace{0pt}
\section{Related Work}
\label{sec:relatedwork}
The capability to process multiple trajectories sequentially ({\it{aka}} the multi-session capability) can be seen as a special case of collaborative SLAM.
Recent SLAM systems, such as ORB-SLAM3\cite{ORBSLAM3_TRO} and VINS-mono \cite{vins-mono} have such multi-session capabilities, which enables them to achieve joint pose and scene estimates similar to collaborative SLAM estimates.
While these approaches achieve greater accuracy and robustness when compared to single-agent SLAM, they are not designed to be used in real-time applications where multiple agents are operating at the same time.

In multi-agent SLAM literature, the classification of the systems is generally made into decentralized and centralized architectures.
One of the first decentralized approaches to multi-agent SLAM is DDF-SAM \cite{ddfsam}, which communicates and propagates condensed local graphs between the robots to distribute the information.
Combining efficient decentralized place recognition \cite{Cieslewski:Scaramuzza:MRS2017} and with a Gauss-Seidel based distributed pose graph optimization \cite{Coudhary:etal:ICRA2016}, in \cite{Cieslewski:etal:ICRA2018} a data-efficient and complete decentralized visual SLAM approach was proposed.
In \cite{dist-coslam}, a monocular vision-only distributed SLAM for mapping large-scale environments is presented.
The recent works \cite{door-slam, kimera-multi} both make use of distributed pose graph optimization schemes along with a robust mechanism for identifying and rejecting incorrect loop closures.
While these distributed SLAM approaches have advantages in terms of scalability, in general, as the information is also distributed the extent of collaboration is limited in order to keep the communication requirements feasible.

On the other hand, in centralized systems, all relevant information passes through a central node.
In \cite{PG-slam}, the authors present a back-end for real-time multi-robot collaborative SLAM where the server combines the local pose graphs obtained from different agents into a global pose graph.
In MOARSLAM \cite{moarslam}, each agent runs a full SLAM pipeline onboard, while the server is used to store the agents' maps and perform map merges across them.
As the agents perform all the computationally expensive tasks, in particular global optimization, this approach is not well-suited for resource-constrained platforms.
On the other end of the spectrum is  C${}^2$TAM \cite{riazuelo2014c}, which offloads all tasks onto a server platform, except pose tracking.
While this allows for very limited computational load onboard the agents, it limits the autonomy of the agents, as a loss of connection to the server eventually causes the onboard tracking to fail.
A middle ground between MOARSLAM and C${}^2$TAM was proposed in \cite{multi-uav-slam}, which introduces a system architecture enabling the agents to function autonomously, but is still able to offload heavy computations to a server and crucially, enable two-way information flow between the agents and the server.
This work was extended in CCM-SLAM \cite{ccm-slam}, shown to perform in real-time on real data, proposing redundancy detection that enabled scalability, which is key in large-scale missions.
Pushing for the incorporation of inertial cues onboard each agent, aside from the monocular cues, CVI-SLAM \cite{cvi-slam} was demonstrated to achieve higher accuracy and metrically scaled SLAM estimates aligned with the gravity direction -- which are core to robotic autonomy in reality.
Most recently, COVINS \cite{covins} was proposed, revisiting the most important components for centralized collaborative SLAM, shown to achieve significant gains in accuracy, while pushing the scalability of the architecture to up to 12 participating agents.

Despite the improved performance, however, one of the major limitations of COVINS is that this performance is highly dependent on the choice of the VIO front-end employed onboard the agents.
For example, using an ORB-SLAM3\cite{ORBSLAM3_TRO} front-end with the COVINS back-end gives an exceptional performance, but the performance drops significantly when using an alternative state-of-the-art VIO front-end, such as VINS-mono \cite{vins-mono}.
This is mainly caused due to the reliance of COVINS on large numbers of highly accurate map points for closing loops, which holds for ORB-SLAM3, for example, but not for VINS-mono.
By utilizing a generic multi-camera relative pose solver for map fusion and loop-closures we break the dependency on highly accurate map points and instead perform the operations using only 2D image observations.
This does not only enable COVINS-G to perform well with virtually any VIO frontend, but also permits the usage of more powerful image features, which in return, allows to handle drastic view-point changes where previous systems failed.

\section{Methodology}
\label{sec:methodology}
In this section, we first provide an overview of the overall architecture in Section \ref{sec:architecture}.
Since our system is closely inspired by the COVINS framework, we then focus on the individual modules of the whole architecture, which are directly impacted by the contributions of this work.

\subsection{System Overview}
\label{sec:architecture}
A summary of our system architecture can be seen in Figure \ref{fig:architecture}.
The system consists of $n$ individual agents, which all run their own local VIO independently and are able to communicate their Keyframe (KF) information to a central server.
The communication module is based on the approach of \cite{covins}, and is shown to be resilient to potential message losses as well as network delays and occasional bottlenecks in the bandwidth.
However, owing to the fact that each agent is running an independent VIO, even a complete loss in connection does not violate the autonomy of an agent.

The central server, on the other hand, is responsible for maintaining the data of the individual agents, fusing the information from the agent, and carrying out computationally expensive tasks, such as global optimization.
After decoding the received KF information, for every KF a visual descriptor based on the Bag-Of-Words (BoW) approach gets computed and added to the KF Database containing the visual descriptor for all the collected KFs.
For every incoming KF, a place-recognition query is performed to find the closest visual matches (candidate KFs).
Based on the visual similarity, it is attempted to compute the transformation between the newly received KF and the candidate KFs.
Upon successful computation of the transformation between the KFs, it is used to either perform a loop closure within a map or if it is across multiple maps, these get fused based on the computed transformation.

At the start of a mission, each of the agents is initialized separately and has a dedicated map on the server.
In our system, the map closely resembles a Pose Graph, where nodes correspond to KFs and edges are based on the relative poses from the odometry and loop closures.
As soon as a loop is found across two agents, their maps get merged together by transforming the map of one agent into the coordinate frame of the other based on the estimated loop transformation.
Upon detection of any loop, a Pose Graph Optimization (PGO) is carried out over all connected Maps.

As opposed to COVINS, which strongly relies on having a well-estimated and consistent set of map points, in our approach we only operate on 2D keypoint information.
This opens up the possibility to use and combine all sorts of front-ends, even if no map points are accessible at all (e.g. in the case of an off-the-shelf tracking sensor such as the T265).
In order to be able to compute the loop constraints without access to map points, we make use of a multi-camera relative pose estimation algorithm \cite{17PT} by treating neighboring KFs as a multi-camera system.

\begin{figure}
    \centering
    \includegraphics[width=0.47\textwidth]{images/covins_arch_v3.pdf}
    \caption{Overview of the COVINS-G system architecture.
    }
    \label{fig:architecture}
    \vspace{-19pt}
\end{figure}

\subsection{Loop Closure and Map Fusion}
\label{sec:loop_closure}
The process of closing loops consists of two main processes; first, suitable candidates for the current query KF are detected, and second, the candidates get geometrically verified by estimating the relative pose between the candidate and the query KF.
The first step is handled by the Place Recognition module, which queries the KF Database for similar KFs based on the BoW \cite{dbow2} image descriptor.

The second step, the geometrical verification, is used as an additional check beside the visual appearance  and in order to obtain a transformation between the query KF $\text{KF}_q$ and the candidate KF $\text{KF}_c$.
Hence, with $q$ being the sensor coordinate frame of the $\text{KF}_q$ and $c$ being the coordinate frame of the sensor for $\text{KF}_c$, the goal is to find the transformation $T_{cq}$, describing the transformation from frame $q$ into frame $c$.
The standard way of computing this transformation is to establish 3D-2D correspondences between the query KF and the map points associated with the candidate KF.
However, in COVINS-G the system does not have access to 3D map points, but only 2D keypoints.
Estimating the transformation between $\text{KF}_q$ and $\text{KF}_c$ using standard 2D-2D relative pose estimation algorithms like the 5-point algorithm \cite{5PT} would result in a scale ambiguity.
Instead, we propose to not only use the query and candidate KFs but also use some of their neighboring KFs and the relative odometry transformations to form two sets of cameras which can be treated as a multi-camera system as illustrated in Figure \ref{fig:17_PT}.
Using the 17-point algorithm \cite{17PT}, the transformation between two such camera systems can be estimated at metric scale.

\subsubsection{The 17-point Algorithm}
\label{sec:seventeen_pt}
The 17-point (or 17-ray) algorithm \cite{17PT} can be used to solve for the relative motion between two multi-camera systems.
A multi-camera system consists of more than one camera where the relative transformations between the cameras are known.
Using 17 2D-2D point correspondences, the relative transformation between two viewpoints of a multi-camera system can be estimated.
To do so, the generalized epipolar constraints \cite{generalized_epipolar} are used, which can be seen as an extension of the epipolar constraints to systems without a central point of projection.
Note that in our setup we only have a monocular camera, however, we leverage the fact that we have a good estimate of the relative pose between adjacent KFs provided by the VIO front-end.
Hence for a $\text{KF}_a$, we take the $m$ neighbors $\text{KF}_{a_{n_i}}, i \in [1, m]$ of $\text{KF}_a$ and use the relative transformations $T_{aa_{n_{i}}}$ to build up a conceptual multi-camera system (Figure \ref{fig:17_PT}).

\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth]{images/17PT_v5.png}
    \caption{The 17-point algorithm requires 17 2D point correspondences to estimate the relative transformation $T_{cq}$ between the multi-camera systems $S_{c}$ and $S_{q}$ (represented by blue dashed lines). }
    \label{fig:17_PT}
    \vspace{-15pt}
\end{figure}

\subsubsection{Implementation}
\label{sec:seventeen_pt_approach}
To estimate the transformation between the candidate KF ($\text{KF}_c$) and the query KF ($\text{KF}_q$), we extend the neighborhood to form two multi-camera systems $S_{q}$ and $S_{c}$ for the $\text{KF}_q$ and the $\text{KF}_c$, respectively.
In our setup the two sets comprised of one additional neighbor for $\text{KF}_q$ and two additional neighbors for $\text{KF}_c$, i.e. $S_{q} = \{ \text{KF}_q, \text{KF}_{q_{n_1}} \}$ and $S_{c} = \{ \text{KF}_c, \text{KF}_{c_{n_1}}, \text{KF}_{c_{n_2}}\}$.
In order to establish the 2D-2D candidate correspondences, we first perform a brute force descriptor matching across all pairs of the two sets $S_{q}$ and $S_{c}$, leading to 6 sets of correspondences.

Instead of using this set of correspondences directly with the 17-point algorithm inside a RANSAC framework, we first perform a pre-filtering, as a high outlier ratio would render a RANSAC with 17 correspondences infeasible.
The pre-filtering consists of performing a standard 2D-2D RANSAC on each of the 6 sets of correspondences.
To reject bad candidates early on, we require a minimum number of inliers (30) in each of the 6 sets.
On the remaining inlier correspondences, we carry out a 17-point RANSAC, where during the sampling we ensure that candidates from all sets are present in order to prevent degenerate configurations (e.g. all 17 correspondences from a single set).
For both the pre-filtering as well as the 17-point RANSAC, we utilize the OpenGV library \cite{opengv}.

For accepting a transformation we require to have at least 100 inliers after RANSAC.
To quantify the uncertainty of the computed transformation we use a sampling-based approach to compute a covariance matrix.
The samples are obtained by repeatedly selecting 17 inliers and computing the transformation using the 17-point algorithm.
We favor the sampling approach over an analytical one as it results in a more consistent covariance estimate since unmodelled effects (e.g. the relative pose uncertainty of the odometry) are better reflected in the samples.

\subsection{Pose Graph Optimization}
\label{sec:pgo}
The Pose Graph Optimization (PGO) in COVINS-G is where the information from the different agents gets fused and the drift in the corresponding trajectories can be corrected.
The PGO step is triggered every time a new constraint is added to the graph as a result of the loop detection.
The state that is optimized in the PGO consists of all KF poses that are in the corresponding graph.
In the following, we denote the pose of a KF $k$ by a rotation $q_{ws_k}$ and a translation $p_{ws_k}$, where $w$ represents the coordinate frame in which the poses are expressed and $s_k$ denotes the device coordinate frame at time-point $k$.
Hence, the state in PGO can be defined as $\mathcal{S} = \{ q_{ws_1}, p_{ws_1}, \cdots, q_{ws_n}, p_{ws_n} \}$, with $n$ being the number of keyframes in the graph.

In the actual PGO, we optimize the following objective:
\begin{equation}
\label{eq:pg}
   \mathcal{S}^{*} = \underset{\mathcal{S}}{\text{arg min}} \{ \sum_{k=1}^{n}  \sum_{l=1}^{q} \norm{e_{kk+l}}^2_{W_{kk+l}} + \sum_{i,j \in \mathcal{L}} \varrho (\norm{e_{ij}}^2_{W_{ij}}) \},
\end{equation}
where $\norm{e}^2_{W} = e^TWe$ is the squared Mahalanobis distance with the information matrix $W$, $\mathcal{L}$ denotes the set of all loop-closure edges and $\varrho(\cdot)$ denotes the use of a robust loss function, here, the Cauchy loss function.
The parameter $q$ represents the number of neighbor KFs added as odometry constraints (e.g. $q=1$ would correspond to having an edge only with the subsequent KF).
This is used to approximate correlations between poses which are generally computed in a sliding window fashion and in our implementation is set $q=4$.
The error terms $e_{ij}$ are of the form
\begin{equation}
e_{ij} = \begin{bmatrix}
\left(p_{ij} - \hat{p_{ij}} \right)^T  & 2 \cdot \text{vec}(q_{ij}^{-1} \cdot \hat{q_{ij}})^T
\end{bmatrix}^{T},
\end{equation}
where $\text{vec}(q)$ extracts the vector part of a quaternion.
The variables denoted with $\hat{\cdot}$ correspond to measurements, e.g. from a loop closure transformation or the delta pose estimated by the odometry front-end.
We use the simplified notation $p_{ij}, q_{ij}$ to denote the translation and rotation of the delta pose between the pose of KF $i$ and $j$ given by $T_{ij} = T_{ws_i}^{-1} T_{ws_j}$.
For the optimization, the Ceres's implementation of the Levenberg-Marquardt is used.
The information matrices $W$ for the loop constraints are obtained via the estimated covariance outlined in section \ref{sec:seventeen_pt_approach}.
The ones corresponding to the odometry edges are obtained using the expected accuracy of the corresponding front-end.

\section{Experiments and Discussions}
\label{sec:experiments}
In our evaluation, we demonstrate the generic nature of our collaborative back-end and its capability to use and combine all sorts of VIO front-ends.
In Sections \ref{sec:tracking_cam} and \ref{sec:feature_support} we show two applications where our back-end enables a collaborative estimation, which previous approaches cannot support.
All our experiments are performed using pre-recorded datasets which we play back in real-time.
The agents' front-ends are run on Intel NUC 7i7BNH @3.5 GHz hardware, while the server is run on a laptop with a core i7-6700HQ @2.6 GHz.
Note that in our setup the agents and the server are connected via a wireless network, in order for real communication to take place.
This setup allows us to have more comparable results over multiple runs while still making use of a real wireless network as would be the case during real-world deployment.

\subsection{Collaborative SLAM Estimation Accuracy}
\label{sec:euroc_eval}
We evaluate the accuracy of the collaborative SLAM estimate on the EuRoC Dataset\cite{euroc} using the Machine Hall (MH) and the Vicon Room 1 (V1) sequences to establish a collaborative estimation scenario with three to five participating agents.
We use various combinations of front-ends with our back-end and compare the results against COVINS \cite{covins} as well as the state-of-the-art multi-session capabilities of VINS-mono\cite{vins-mono} and  ORB-SLAM3 \cite{ORBSLAM3_TRO}.
As multi-session methods only support one agent at a time, we process the datasets sequentially with one agent at a time, in contrast to the collaborative SLAM approaches, where all agents' processes are run in parallel.

To showcase the flexibility of our approach, we perform the evaluation of our back-end with different front-end combinations.
In the first experimental setup, all agents are operated using the same front-end, which in our experiments is either the VINS-mono or the ORB-SLAM3 front-end.
In subsequent experiments, we test COVINS-G using different front-ends for the different agents, namely OpenVINS\cite{Openvins}, VINS-mono, ORB-SLAM3 and SVO-pro\cite{Forster17troSVO}.
Since COVINS performs as a post-processing step a Global Bundle Adjustment (GBA) at the end of every run, we do not directly compare against it but rather include its performance here as a reference.
For more appropriate comparisons with COVINS-G, we include the COVINS result without the additional GBA step.
The averaged results over 5 runs are summarized in Table \ref{tab:multi_ag}.

Using the ORB-SLAM3 front-end, we achieve on-par performance to the ORB-SLAM3 multi-session back-end, even though in our approach we do not perform map re-use as in ORB-SLAM3.
As ORB-SLAM3 performs a GBA upon every loop detected, it can potentially reach very high accuracy (as COVINS demonstrates with GBA enabled), however, as long as it is able to find sufficient correspondences in the re-localization, no GBA gets triggered.
Therefore, the overall accuracy is coupled also to the time that the last GBA step was performed.
Compared to COVINS without GBA, our approach reduces the error almost by a factor of two.
This can be explained by the fact that in COVINS-G we are able to close more loops, as also shown in Table \ref{tab:computation}, and also because the covariance of the loop transformations gets explicitly estimated instead of using fixed heuristics.

The comparison with the VINS-mono front-end showcases a similar outcome that COVINS-G performs similarly to the VINS-mono multi-session back-end.
The small gap in performance can be explained in that VINS-mono incorporates a larger number of loops, whereas in COVINS-G we set a minimum number of KFs between consecutive loops in order to reduce the computational burden.
Compared to COVINS with the VINS-mono frontend, COVINS-G shows a significant improvement with a factor of around 3.
This is because COVINS only detects and closes few loops because it's loop-closure pipeline is tailored to a high quality map with a large number of map points.
Due to this weak connection within and across the trajectories, even the GBA is unable to reduce the error.
With COVINS-G, even with a mix of different front-ends, we can see that the performance is in a similar range as when using the VINS-mono front-end, demonstrating the effectiveness of our generic approach.

\begin{table*}[]
\caption{Evaluation of joint-trajectory estimates for different methods on the EuRoC Dataset\cite{euroc} (lowest error in bold) reported as the average trajectory error over 5 runs each. $^\#$For the heterogenous front-end, agents 1-5 utilize OpenVINS\cite{Openvins}, VINS-mono\cite{vins-mono}, ORB-SLAM3\cite{ORBSLAM3_TRO}, SVO-Pro\cite{Forster17troSVO} and VINS-mono\cite{vins-mono} front-ends, respectively. As *COVINS\cite{covins} performs a Global Bundle Adjustment (GBA) step at the end of the run it has been included here for reference only and is excluded from our comparison.
}
\label{tab:multi_ag}
\centering
\def1.2{1.2}

\begin{tabular}{|cc|ccc|}
\hline
\multicolumn{2}{|c|}{\textbf{Method}}                                                          & \multicolumn{3}{c|}{\textbf{Translational RMSE (m)}}                                                                                \\ \hline
\multicolumn{1}{|c|}{}                                 &                                       & \multicolumn{1}{c|}{MH01-MH03}                    & \multicolumn{1}{c|}{MH01-MH05}                    & V101-V103                    \\ \cline{3-5}
\multicolumn{1}{|c|}{\multirow{-2}{*}{Front-end}}       & \multirow{-2}{*}{Back-end}             & \multicolumn{1}{c|}{(3 Agents)}                   & \multicolumn{1}{c|}{(5 Agents)}                   & (3 Agents)                   \\ \hline

\multicolumn{1}{|c|}{ORB-SLAM3\cite{ORBSLAM3_TRO}}                        & ORB-SLAM3 MS\cite{ORBSLAM3_TRO}                          & \multicolumn{1}{c|}{0.041}                        & \multicolumn{1}{c|}{0.082}                        & \textbf{0.048}               \\ \hline
\multicolumn{1}{|c|}{ORB-SLAM3\cite{ORBSLAM3_TRO}}                        & COVINS (No GBA) \cite{covins}                      & \multicolumn{1}{c|}{0.075}                        & \multicolumn{1}{c|}{0.119}                        & 0.130                        \\ \hline
\multicolumn{1}{|c|}{{\color[HTML]{666666} ORB-SLAM3\cite{ORBSLAM3_TRO}}} & {\color[HTML]{666666} COVINS (GBA) \cite{covins}*} & \multicolumn{1}{c|}{{\color[HTML]{666666} 0.024}} & \multicolumn{1}{c|}{{\color[HTML]{666666} 0.036}} & {\color[HTML]{666666} 0.042} \\ \hline
\multicolumn{1}{|c|}{ORB-SLAM3\cite{ORBSLAM3_TRO}}                        & Ours                                  & \multicolumn{1}{c|}{\textbf{0.040}}               & \multicolumn{1}{c|}{\textbf{0.064}}               & 0.067                        \\ \hline \hline

\multicolumn{1}{|c|}{VINS-mono \cite{vins-mono}}                        & VINS MS\cite{vins-mono}                               & \multicolumn{1}{c|}{\textbf{0.062}}                        & \multicolumn{1}{c|}{0.100}                        & \textbf{0.076}                        \\ \hline
\multicolumn{1}{|c|}{VINS-mono \cite{vins-mono}}                        & COVINS (No GBA) \cite{covins}                               & \multicolumn{1}{c|}{0.259}                        & \multicolumn{1}{c|}{0.305}                        & 0.183                       \\ \hline
\multicolumn{1}{|c|}{{\color[HTML]{666666} VINS-mono \cite{vins-mono}}}                        & { \color[HTML]{666666} COVINS (GBA) \cite{covins}*}                               & \multicolumn{1}{c|}{ {\color[HTML]{666666} 0.261}}                        & \multicolumn{1}{c|}{{\color[HTML]{666666} 0.321}}                        & {\color[HTML]{666666} 0.183}                       \\ \hline
\multicolumn{1}{|c|}{VINS-mono\cite{vins-mono}}                        & Ours                                  & \multicolumn{1}{c|}{0.081}                        & \multicolumn{1}{c|}{\textbf{0.095}}                        & 0.090                        \\ \hline \hline

\multicolumn{1}{|c|}{Heterogenous$^\#$}                   & Ours                                  & \multicolumn{1}{c|}{0.081}                        & \multicolumn{1}{c|}{0.090}                        & 0.088                        \\ \hline
\end{tabular}
\vspace{-10pt}
\end{table*}

\subsection{Large-Scale Outdoor Experiments}
\label{sec:outdoor_expt}
To demonstrate the applicability of our back-end to large-scale scenarios, we captured a dataset consisting of 4 sequences using a hand-held setup with an Intel Realsense D455.
The dataset was recorded by walking on the sidewalks around the campus of ETH Zurich in the center of Zurich and with a combined trajectory length of around 2400$m$ covered an area of approximately 67500$m^2$.
For this experiment, the VINS-mono front-end was used on all sequences.
As we do not have a ground truth trajectory for the dataset, we superimposed the estimated combined trajectory on a satellite image of the area as illustrated in Figure \ref{fig:outdoor_expt}.
As can be seen, the different trajectories are well aligned with the road structure.
A complete visualization of the whole experiment can be found in the supplementary video.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.48\textwidth]{images/expt_outdoor_largescalev3.png}
    \caption{Joint trajectory estimates for 4 agents superimposed on the satellite image of our large-scale outdoor experiment with a combined trajectory length of 2400 $m$.}
    \label{fig:outdoor_expt}
    \vspace{-12pt}
\end{figure}

\subsection{Tracking Camera Support}
\label{sec:tracking_cam}
This experiment aims at highlighting the generalization capabilities of our system with respect to the front-end that is used.
To do so, we captured a dataset with two different sensors, one Intel Realsense D455 camera, and one Intel Realsense T265 tracking camera inside an office floor.
For the D455 we used the VINS-mono as a front-end, whereas for the T265 we used the odometry estimate that is provided directly by the sensor.
As this sensor only provides its images and the corresponding poses but does not offer access to its internal state, we use our ROS-based front-end wrapper, which does a motion-based keyframe selection, detects feature points, and creates and communicates the KF messages to the back-end.

The outcome of the experiment is illustrated in Figure \ref{fig:tracking}, where one can see an example image for each of the sensors and the combined trajectory overlaid with the floor plan of the office (the complete experiment is visualized in the supplementary video).
As it can be seen, the estimated trajectory fits the floor plan, indicating its consistency.
In addition to the different odometry sources, the two sensors have also two rather different lenses, rendering matching between the two systems more difficult, however, our proposed back-end is able to merge the estimates nonetheless.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.48\textwidth]{images/expt_trackingv3.png}
    \caption{Joint trajectory estimate for a heterogeneous system of 2 agents superimposed on the floor plan of the
    building. Map points are shown for visualization purposes only}
    \label{fig:tracking}
    \vspace{-9pt}
\end{figure}

\subsection{Alternative Feature Descriptor support}
\label{sec:feature_support}
Visual features are designed to be tolerant to illumination changes, perspective distortions as well as viewpoint changes.
While for runtime efficiency, real-time SLAM systems are mainly restricted to highly computational efficient binary features such as ORB \cite{ORB} or BRISK \cite{Leutenegger:etal:ICCV2011}.
However, it is accepted that to date, binary features are not able to match the robustness of more expensive descriptors such as e.g. SIFT \cite{SIFT}.
As our approach is largely decoupled from the front-end, we are able to make use of more expensive features, also because we do not require them to be computed at frame rate, thus allowing us to potentially match trajectories with much larger viewpoint differences.

In this experiment, we recorded another outdoor dataset with two hand-held trajectories walking in parallel on two sides of a street.
We modified our front-end wrapper to detect SIFT instead of ORB features and run the back-end using these SIFT features as well.
While using the framework with ORB features no overlap can be detected to merge the trajectories, using the powerful SIFT features allowed us to successfully detect loops across the agents and align the trajectories.
The comparison of this experiment can be found in the accompanying video.
Such an improved tolerance to large view-point changes allows the system to be used in scenarios where larger view-point changes are inherent to the use case, for example, the collaboration between aerial and ground robots.

\subsection{Computational and Communication Requirements}
\label{sec:computation}
\begin{table}[]
\caption{Comparison of the runtime for the loop computation and the average communication traffic.}
\label{tab:computation}
\centering
\def1.2{1.2}
\begin{tabular}{|c|c|c|c|}
\hline
\multicolumn{1}{|l|}{\textbf{}} & \textbf{\begin{tabular}[c]{@{}c@{}}\# Loops\\  detected\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Computation Time \\ per loop\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Network Traffic\\  per Agent\end{tabular}} \\ \hline
Ours                            & 75                                                                    & 213 $\pm$ 171 ms                                                                        & 179.74 kB/s                                                                   \\ \hline
COVINS \cite{covins}                         & 57                                                                    & 35 $\pm$ 23 ms                                                                         & 486.41 kB/s                                                                   \\ \hline
\end{tabular}
\vspace{-15pt}
\end{table}

The statistics for communication and loop transformation computation are generated and compared with COVINS for the experiment performed on EuRoC MH dataset with 5 agents, each running an ORB-SLAM3 front-end.
The computation time for estimating the loop transformation using our approach (including feature matching, 2D-2D pre-filtering, 17-point RANSAC and covariance matrix computation) is compared against the standard approach used in COVINS (feature matching + PnP RANSAC) and summarized in Table \ref{tab:computation}.
Though the computation time for our method is around an order of magnitude higher than for COVINS, the average computation time per loop is around $213 ms$ which still makes it possible to operate in real-time for up to 5 loop computations per second.
For this experiment, a total of 75 loops were detected for a mission time of around 135 seconds, resulting in 0.55 loops per second over all agents.
The network traffic for our approach is around three times lower than that of COVINS.
This is owed to the fact that in our back-end we require only 2D keypoints and their descriptors for each KF, unlike COVINS which also requires sending the map points as well as the data required to perform the IMU pre-integration.

\section{Conclusion}
\label{sec:conclusion}
In this work, we present a front-end-agnostic back-end for collaborative visual-inertial SLAM.
By making use of a multi-camera relative pose estimator for estimating loop-closure transformations, our system is able to work using only 2D keypoint information.
This allows our collaborative back-end to be compatible with virtually any VIO front-end with minimal to no modifications to it.
In our experimental evaluation, we achieve at least on-par accuracy with state-of-the-art multi-session and collaborative SLAM systems, which use back-ends specifically designed for the respective front-end.
Owed to the decoupled nature of the data used in our back-end, we can make use of more powerful keypoint descriptors like SIFT, allowing us to close loops and merge trajectories with drastic viewpoint changes, which cannot be handled by state-of-the-art systems.
Our open-source implementation of the system enables users to unlock the capabilities of a collaborative SLAM system without the need to change their state estimation pipeline of choice.

In proceeding work, we would like to make more explicit use of the information coming from our VIO, in particular the gravity alignment, by replacing the 17-point algorithm with a minimal 4-point solver which leverages the gravity information \cite{4PT_gravity}.
This allows to speed up the relative pose estimation step significantly, as it requires drawing fewer samples within the RANSAC computation.
In the future, we would also like to deploy this system on a team of robots for multi-robot applications, such as exploration and 3D reconstruction of large areas.

\bibliographystyle{IEEEtran}

Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Multiagent Systems (cs.MA)


Computer Security Risks of Distant Relative Matching in Consumer Genetic Databases

Abstract: Consumer genetic testing has become immensely popular in recent years and has lead to the creation of large scale genetic databases containing millions of dense autosomal genotype profiles. One of the most used features offered by genetic databases is the ability to find distant relatives using a technique called relative matching (or DNA matching). Recently, novel uses of relative matching were discovered that combined matching results with genealogical information to solve criminal cold cases. New estimates suggest that relative matching, combined with simple demographic information, could be used to re-identify a significant percentage of US Caucasian individuals. In this work we attempt to systematize computer security and privacy risks from relative matching and describe new security problems that can occur if an attacker uploads manipulated or forged genetic profiles. For example, forged profiles can be used by criminals to misdirect investigations, con-artists to defraud victims, or political operatives to blackmail opponents. We discuss solutions to mitigate these threats, including existing proposals to use digital signatures, and encourage the consumer genetics community to consider the broader security implications of relative matching now that it is becoming so prominent.

\section*{Introduction}

\vspace*{-5pt}

Direct-to-consumer (DTC) genetic testing services have become immensely popular in recent years~\cite{mit-tech-year-testing}. For under \$100, DTC services will genotype customers using high-density autosomal SNP microarrays that probe between 500,000-1,000,000 markers~\cite{23andMe, ancestry, my-heritage}. These genetic profiles are analyzed to help customers better understand their ancestry and health, or for general curiosity. DTC companies also allow customers to download their raw genetic profiles so they can analyze the data themselves or upload profiles to third-party services that can aggregate profiles from different DTC companies and offer advanced features. Many third-party service providers were created for amateur or academic purposes, but recently, they have become quite large, in some cases containing close to 1 million profiles~\cite{bloomberg-tech}. We distinguish between DTC companies, which produce genetic data directly from biological material, and third-party services that only aggregate and analyze genetic data produced from other sources.

One of the most popular features offered by both DTC and third-party services is long range familial relative matching (or DNA matching). With high-density genotype data it is possible to accurately identify distant relatives, including relatives as distant as 3rd cousins or further, in large databases that include millions of profiles~\cite{mountain2012}. Relative matching algorithms work by finding large DNA segments that are shared between individuals, known as identical by descent (IBD) segments, that are indicative of shared ancestry because they have been inherited from a recent common ancestor without intervening recombination. The total quantity and distribution IBD segments between two individuals is then used to predict their familial relationship~\cite{gusev2009whole}. Analysis with MyAncestry data showed that matching relatives are quite common, with over 44\% of profiles returning a third cousin and 10\% with a second cousin or closer~\cite{Erlich350231}. We can expect that the probability of a match will only increase as these databases grow larger.

Relative matches can be profound and sometimes life changing because of the cultural and legal significance of family relationships. For example, matches routinely reunite adopted children with their birth parents and identify cases of unexpected paternity~\cite{dna-success-stories,isogg-success,parents-gift-divorce}. However, it was the high profile use of relative matching by law enforcement to identify the suspected Golden State Killer that has perhaps drawn the most attention. To solve this case, law enforcement used DNA samples obtained from old crime scenes to construct a valid DTC genetic profile; this profile was uploaded to GEDMatch, a third-party genetic ancestry service that supports relative matching~\cite{washingtonpost-gsk,nytimes-gsk}. Investigators identified third cousins of the suspect, which enabled them to narrow their search to the suspected killer~\cite{washingtonpost-gsk}. Since the Golden State Killer case, genetic genealogy approaches have been used over a dozen times to solve other cold cases and identify deceased individuals~\cite{forensic-magazine,dna-doe-project}. Business like Parabon Nanolabs labs now offer genetic genealogy services to law enforcement to help them solve cases using this technique~\cite{parabon-nanolabs}.

Relative matching in consumer genetic databases significantly enhances the capabilities of law enforcement to match distant relatives when compared to government maintained forensic databases, like the FBI maintained CODIS database. CODIS primarily maintains alleles for a small number of 13-20 core, short tandem repeat (STR) markers~\cite{budowle1998codis,fbi-faq}. Relative searching with STR markers relies on comparing markers directly, which limits the CODIS database to finding closely related family members~\cite{ge2011comparisons}. Consumer databases also contain a number of other advantages over government forensic databases. They contain profiles from a more diverse population because they include data from the general public, not just suspected or convicted criminals, and relative matching in consumer databases avoids regulations that restrict or preclude relative matching in government databases~\cite{ram2018genealogy,science-progress-transparency}. Therefore, it is no surprise to see law enforcement rapidly adopt relative matching techniques on consumer databases to solve crimes.

The growing prominence of DTC and third-party consumer genetic databases in important applications, like criminal investigations, raises significant computer security (a.k.a., cyber security) concerns. In effect, third-party genetic databases designed for amateur ancestry analysis are now used as criminal forensic databases \dash except there do not appear to be the same rigorous computer security standards that exist in government maintained forensic databases. Most concerning is that other individuals can leverage the same genetic genealogy techniques as law enforcement: malicious actors can re-identify individuals, spoof profile metadata (e.g., identity of the profile), or even make entirely fake profiles by directly manipulating the digital genotype data. As these databases continue to be used beyond their original intended purpose, criminals or other malicious actors may be motivated to compromise or manipulate databases that support relative matching. In this work we describe potential risks associated with relative matching in consumer genomic databases and discuss what services can do to enhance their safety.

\paragraph{Next steps.}
We argue that given the high profile use of these databases, now is the time to have an industry-wide conversation about the computer security risks relevant to DTC and third-party consumer genetic databases. Since the issues we raise have the potential to be industry-wide, and since many of the observations we make are natural extensions of prior observations (but with a computer security perspective), we would consider it improper to disclose our findings to only the companies and actors in this field that we can enumerate
\dash doing so would only give them and not others the ability to proactively consider mitigations. By disclosing our findings of potential risks in this work, as well as a further discussion of defenses, we hope to catalyze the incorporation of strong defenses in \emph{all} DTC and third-party consumer databases, and we hope to engender the appropriate level of caution by \emph{all} parties (such as those from law enforcement) who might seek to use present-day DTC and third-party consumer databases in their activities.

\vspace*{-5pt}

\section*{Adversarial Goals}

\vspace*{-5pt}

Support for unrestricted relative matching exposes consumer genetic databases to additional computer security and privacy related risks than what might be anticipated. Other issues, like data theft or discriminatory use of genetic data, are certainly a concern, but in this work we emphasis how relative matching, in particular, creates new problems. Consider the wide range of malicious actors, with varying capabilities, that have motivation to take advantage of relative matching services. This includes criminals with unidentified forensic DNA evidence that want to remain anonymous, con-artists that could use fabricated relatedness to gain the trust of their victims, or even sophisticated political or intelligence actors that want to blackmail or harm the reputation of their opponents (see Table~\ref{tbl:actors} for an extended list of actors and their goals). We categorize the goals of adversaries into three groups:

\vspace*{-5pt}

\begin{enumerate}
  \item \textbf{Avoid identification.} The goal is to remain anonymous or difficult to detect with relative matching queries. This is precisely the situation where a criminal perpetrated a crime and left physical DNA evidence, like the Golden State Killer, and is trying to avoid detection by law enforcement using ancestry analysis. This is also useful for individuals, like known fugitives, that have assumed a false identity to remain undetected \dash in fact, just recently, relative matching on GEDMatch was used to identify an individual that had assumed the identity of a deceased 8-year old boy for over 20 years~\cite{fugitive-wp}.
  \item \textbf{Determine a target's identity and relatives.} It has already been shown that relative matching can be used to re-identify a large number of US individuals, and just as law enforcement has used relative matching to identify unknown individuals, malicious actors could do the same~\cite{Erlich350231}. Such re-identification could threaten the privacy of anonymized research subjects or endanger foreign operatives and undercover agents that have assumed a false identity for legitimate purposes. Matching results could also be used to uncover sensitive details, like hidden children. For example, if a high profile individual, like a politician or celebrity, had an unknown or hidden child then an attacker could use this knowledge to discredit or blackmail that individual.
  \item \textbf{Remove or forge new relatives.} Criminals could remove or use forged relationships for a number of purposes: con-artists could forge relations with a target to gain their trust, perpetrators could put blame on others by forging profile data, and parents could use falsified profiles to misattribute paternity to another person.
\end{enumerate}

\begin{table}[t!]
\begin{center}
 \resizebox{\textwidth}{!}{%
 \begin{tabular}{| l | l |}
 \hline
 \textbf{Malicious Actor} & \textbf{Motivations} \\
 \hline\hline
  \multirow{2}{*}{Perpetrator of Unsolved Crime or Fugitive} & Avoid Identification \\
  & Receive Advanced Warning of Pending Discovery \\
  \hline
  Confidence Trickster & Defraud Target \\
  \hline
  Paparazzi or Stalker & Reveal Target's Ancestry \\
 \hline
  \multirow{3}{*}{Unwilling Parent} & Avoid Paternity Obligations \\
  & Imply False Paternity on a Target \\
  & Avoid Discovery of Infidelity \\
  \hline
  Genetic Surveillance & Link Unknown Genetic Profile to Identity \\
  \hline
  Curious Individual & Re-Identify Anonymous Public Research Profiles \\
  \hline
  \multirow{2}{*}{Political Operative} & Damage Opponent's Reputation \\
  & Blackmail \\
  \hline
  \multirow{2}{*}{Intelligence Agency} & Identify Foreign Operatives \\
  & Blackmail \\
  \hline
 \end{tabular}}
\end{center}
    \caption{Possible malicious actors and their motivations to attack genetic databases that support relative matching.}
\label{tbl:actors}
\end{table}

\section*{Threat Vectors}

\vspace*{-5pt}

The open nature of consumer genomics, especially third-party services, gives attackers many ways to manipulate relative searches. Below we describe different methods that a malicious actor can use to achieve different objectives: unauthorized use of DNA data or profiles, metadata spoofing, synthetic relatives, denial-of-service, and directly deleting or modifying existing genetic profiles (Table~\ref{tbl:attack-vectors}; Figure~\ref{fig:attack_vectors}).

\paragraph{Unauthorized Use of DNA Profiles.}

This is when someone fraudulently gathers or uses a target's genetic profile in a relative matching query without authorization. This would be useful to anyone wanting to determine an individual's identity or familial relationships. To use a target's genetic profile in a relative matching query, the attacker needs either access to the digital genotype data or be able to collect a DNA sample from the target to generate the profile. In some cases, the attacker will already have access to the digital profile of a target. For example, there are many public research datasets that contain high-density SNP microarray or whole genome sequencing data for many anonymous research subjects~\cite{10002015global}. This data could easily be converted into compliant DTC genetic profiles that could be uploaded to third-party databases. Moving forward, data breaches may be another source of genetic profiles. There have already been security issues at genetic testing companies. For example, in June 2018 it was discovered that over 92 million user passwords and email addresses from a major DTC company were leaked~\cite{my-heritage-incident}.

When the attacker does not have the target's genetic profile data but has physical access to them, a competent attacker can collect a physical sample from the target to construct a genetic profile. Currently, major DTC companies require saliva or buccal samples to generate profiles, which may be difficult to obtain from a target. However, there are already examples of amateurs collecting and processing DNA from heterogeneous samples. In 2013, the artist Heather Dewey-Hagborg, in her project Stranger Visions, collected random DNA around New York City from sources like cigarette butts and gum and analyzed this DNA to generate face portraits using forensic DNA phenotyping algorithms~\cite{heather-stranger-visions}. In a similar fashion, an attacker could use DIY-bio spaces, available in many cities, to extract DNA and sequence it using a third-party sequencing provider. Heather Dewey-Hagborg also demonstrated that it is easy to obtain anonymous saliva samples when she ordered some over the Internet using only an academic email and mailing address and then sent the saliva to a DTC company for processing~\cite{heather-t3511}. Some actors, like nation states, could generate profiles from DNA samples directly using their own facilities.

\begin{table}[t!]
\begin{center}
 \resizebox{\textwidth}{!}{%
 \begin{tabular}{| l | l | l |}
 \hline
 \textbf{Attack} & \textbf{Objective} & \textbf{Expertise} \\
 \hline\hline
  \multirow{2}{*}{Unauthorized Use of DNA Profiles} & Uncover Identity of Target & \multirow{2}{*}{Low} \\
  & Determine Target's Relatives & \\
  \hline
  \multirow{2}{*}{Spoof Metadata} & Hide Identity from Relative Searches & \multirow{2}{*}{Low} \\
  & Implicate Wrong Individual & \\
  \hline
  \multirow{2}{*}{Synthetic Relatives} & Misdirect Relative Searches & \multirow{2}{*}{Medium} \\
  & Create False Relatives of a Target & \\
  \hline
  Denial-of-Service & Thwart Relative Matching & Medium \\
 \hline
  \multirow{2}{*}{Compromised or Privileged Access} & Add, Delete, or Modify Relative Matches & \multirow{2}{*}{High} \\
  & Corrupt Audit Log & \\
  \hline
 \end{tabular}}
\end{center}
    \caption{Possible attacks against genetic databases with relative matching, the objective of these attacks, and necessary expertise required to execute a given attack.}
\label{tbl:attack-vectors}
\end{table}

\paragraph{Spoof Metadata.}

One of the simplest things a malicious actor can do is to falsify metadata \dash the information associated with the genetic profiles like name or email address \dash when a DNA sample or raw profile data is sent to a DTC or third-party service. In some cases, this metadata is the only information linking the genetic profile to a particular individual.

\begin{figure*}[t!]
\includegraphics[width=\linewidth]{attack_vectors.eps}
\caption{Overview of threats to 3rd-party genetic databases that support relative matching. DNA profiles can be obtained in a number of ways, including through standard DTC genetic testing, by indirectly generating them from other DNA sources, or from publicly available data. An attacker can take advantage of relative matching by uploading manipulated profile data, or if they have direct access, by manipulating the database itself.}
\label{fig:attack_vectors}
\end{figure*}

This is clearly useful to someone trying to avoid identification. For example, a criminal could upload their own profile data, which they could easily obtain from a DTC service, under a false identity. This could confuse or slow down an investigation because the criminal's profile would appear to be another individual. It is likely that in criminal investigations any suspects would be re-tested by law enforcement, which would reveal that the metadata was falsified. However, the criminal could make this more difficult by spoofing the identity of an individual that is difficult to reach, like someone that is deceased, living in another country, or is entirely fictitious. If the criminal uploaded the data of another individual under their own identity they might avoid suspicion in cases where law enforcement had more direct access to a genetic database to run queries.

\paragraph{Synthetic Relationships.}

The raw genotype data itself, including SNPs, can also be modified. This is a problem because some third-party services do not confirm that uploaded profile data was produced by a reputable DTC service and left unmodified. IBD segment sharing has been simulated in prior work to evaluate IBD detection algorithms, and we believe that similar techniques can be used to generate forged profiles that are designed to look like relatives a given individual, which we call \textit{synthetic relatives}~\cite{gusev2009whole,mountain2012}. If synthetic relative profiles were then uploaded to third-party services, those profiles could trick relative matching algorithms into making spurious relative matches.

The ability to create false relatives can lead to a number of security problems. Consider the case of a perpetrator that is trying to avoid identification. The simplest case is when the perpetrator has no identifiable relatives in a third-party database; they could upload a distant synthetic relative under a false identity to misdirect a genetic genealogy investigation. This may also give the perpetrator advanced notice of a pending investigation because the person falsely associated with the synthetic relative may be contacted for questioning about their family history and may have an existing relationship with the perpetrator.

A more challenging scenario is when the perpetrator already has distant relative profiles in third-party databases (which the perpetrator could identify using their own profile or a partially falsified one). In this case, the perpetrator could upload synthetic relatives, but with the identity of someone on a distant branch of the family tree that is not related to the perpetrator but is related to a real relative already in the database (see Figure \ref{fig:synthetic_attacks}a for a visual example). Whether this works in general will depend on how many matching relatives already exist in the database, the overall topology of the family tree, and the existence of other sources of corroborating information, like known family trees. Distant synthetic relatives can also be used to falsely implicate a particular target. For example, in some situations, two matching relatives on different branches of a family tree can narrow a search to a small number of individuals (Figure \ref{fig:synthetic_attacks}b). Any falsified, synthetic relatives would be discovered if the identity associated with them was ever closely investigated and re-tested. However, since synthetic matches are designed to appear like relatives of the perpetrator and not the perpetrator themselves, it is possible that the synthetic relatives would not be scrutinized.

It is not just perpetrators but other types of actors that could use synthetic relatives maliciously. For example, a con-artist could upload a synthetic profile designed to look like the child of a target, using the con-artist's name, to falsify a parent-child relationship. Politically motivated actors could also use synthetic children to imply a child born out of wedlock to tarnish their opponent's reputation. In these cases, the relationship might seem serendipitous and not be re-tested (or not be re-tested until after significant damage has been done).

\begin{figure*}[t!]
\includegraphics[width=\linewidth]{synthetic_attacks.eps}
\caption{Examples of attacks using synthetic relatives. \textbf{a}, A criminal suspect wants to avoid identification when their 2nd cousin is already in a database. The suspect uploads a synthetic second cousin using the identity of an individual related to the 2nd cousin but not the suspect. This makes it appear that the suspect is on the wrong branch of the family tree. \textbf{b}, The suspect uploads two synthetic relatives, on different family branches, to falsely implicate a couple as parents of the suspect.
\label{fig:synthetic_attacks}}
\end{figure*}

We propose a simple approach to generate synthetic relatives by splicing SNPs from multiple profiles together. To begin, the attacker starts with the genetic profile of the target that will be related to the synthetic relative. Then using another profile as a base \dash say from a public database or an entirely synthetic profile \dash the attacker stitches in segments from the target profile into the base profile based on the length and number of IBD segments that would be expected from the desired relationship (e.g., parent, 2nd cousin, etc.). The segment regions could be randomly selected or copied from real relative matches with the desired relationship (which are commonly published on amateur ancestry blogs).

Segments could be spliced from the target profile into the base profile by naively copying over both alleles in each segment region. However, this would make the matching segments appear to be full-IBD (i.e., matching on both chromosomes), which would only be expected for close relatives, like siblings. To construct half-IBD segments, only SNPs from one phase should be spliced into the base profile~(Figure \ref{fig:synthetic_relative_construction}). This technique could be extended to make a synthetic profile related to many individuals by splicing additional target profiles together in a similar manner. It is important to note that the synthetic relative will appear related to the base profile as well, which might cause undesirable matches if a real genetic profiles was used as the base profile. To summarize, synthetic relative construction could work as follows:

\begin{enumerate}
  \itemsep0em
  \item Determine the segment regions to copy from a an existing relative pair.
  \item Phase the genotypes of the target and base profiles.
  \item For each segment region, copy over the SNPs from one phase of the target profile and replace those SNPs in one phase of the base profile.
  \item Unphase the new modified base profile by randomly shuffling the SNPs. This modified base profile is the final synthetic relative profile.
\end{enumerate}

\begin{figure*}[t!]
\includegraphics[width=\linewidth]{synthetic_relatives.eps}
\caption{Proposed method to generate synthetic relatives.}
\label{fig:synthetic_relative_construction}
\end{figure*}

\paragraph{Denial-of-Service.}

There is no limit on the number of accounts and profiles that an attacker can create and upload if account information is not verified. This can be useful to an attacker that is trying to hide their identity because they can upload synthetic relatives under many different accounts. This will cause an excessive number of relative matches that could overwhelm any real matches with fake ones. However, this technique might arouse suspicious because the number of matches could appear unrealistic. Another approach is to spoof a person with a large family or who is a descendent of someone with a large number of children, like a sperm donor. This could make it more difficult to narrow down the search to a manageable list of suspects because the extended family would be very large.

\paragraph{Compromised or Privileged Database Access.}

The most powerful technique to manipulate relative matches is when an attacker has compromised or privileged access to a genetic database. With privileged access, an attacker is able to add, remove, or modify any profile in the database to alter relative matches or modify logs used for auditing. This threat is particularly acute when services have poor security practices. Here we note that experience in the computer security community suggests that non-commercial services, especially those run by a small number of people, can often have poor security practices; further, industries that have not yet received adversarial pressure often do not implement state-of-the-art security best practices. This threat is also relevant to actors, like governments, that are technically capable or to insiders that already have high levels of access.

\section*{Discussion}

Long range familial matching is not new, but it was not until the recent explosion of high-density consumer genetic testing that it became so effective and consequential. The ability to match distant genetic relatives at a population scale is profound and brings a new set of computer security challenges. The recent appropriation of amateur ancestry databases as an investigative tool for law enforcement significantly raises the stakes.

In this work we described a wide range of actors with varying capabilities that have interest in modifying or corrupting relative matching queries, which makes the threat space especially complex. There are a number ways to minimize the risk of relative matching attacks. A major issue is that some third-party services do not confirm that all profiles originate from legitimate DTC services. A failure to authenticate profiles  may allow attackers to upload digitally constructed or altered DTC genetic profile data files. This attack vector can be eliminated if all raw profiles are known to be generated by a reputable DTC testing service. This can be accomplished if the raw data files are digitally signed using a cryptographic key controlled by the DTC company, as was recently suggested by Erlich et.\ al.,\ or by exchanging data using APIs instead of letting users directly upload their raw genetic profiles~\cite{Erlich350231}. However, we suggest going a step further by having the genotyping instrument itself sign and attest to the data it generates, including instrument id, date, and time of data generation. As a side-effect, digitally signed genetic profiles will prevent law enforcement from uploading raw profiles in cases where profiles cannot be generated directly from a DTC service and will instead require law enforcement to work with DTC companies directly.

Digital signatures will ensure that the genotype data was created by a DTC testing service, but will not confirm that metadata associated with a profile is legitimate. To do this DTC and third-party companies should confirm the identity of their customers. It is a common practice for people to upload profiles on behalf of their relatives, so it may not be possible to verify the identity of specific profiles that are generated; however, at a minimum, the identity of the account owner should be verified. Identity verification also minimizes the risk of denial-of-service attacks because it naturally limits the number of account that can be created. Account identity could also be further tied to the profile data via digital signatures.

Finally, we have concerns about the general computer security practices of consumer genetic databases, especially third-party services that have not previously faced significant adversarial pressure. Given the sensitivity, size, and way third-party services are being used, we argue that it is important that they adopt software security best practices. This can include instituting better user authentication procedures, routine auditing and penetration testing, and frequent software updates, among others.

In the future, the efficacy of different relative matching attacks will depend largely on the population coverage in these database and the presence of other sources of information like family histories. The more dense these databases, the more difficult it will be to avoid identification. However, other attacks, like re-identification, will become correspondingly easier. In any case, whenever relative matches are used in important applications (e.g., criminal investigation, paternity, etc.) all results should be corroborated with additional testing.

\paragraph{Acknowledgements}\mbox{}\\
This research was supported in part by the University of Washington Tech Policy Lab and the Torode Family Professorship.

\paragraph{Competing Interests}\mbox{}\\
The authors declare no competing interests.

\printbibliography

\end{document}
\section*{Introduction}

In this file, we present some tips and sample mark-up to assure your
\LaTeX\ file of the smoothest possible journey from review manuscript
to published {\it Science\/} paper.  We focus here particularly on
issues related to style files, citation, and math, tables, and
figures, as those tend to be the biggest sticking points.  Please use
the source file for this document, \texttt{scifile.tex}, as a template
for your manuscript, cutting and pasting your content into the file at
the appropriate places.

{\it Science\/}'s publication workflow relies on Microsoft Word97.  To
translate \LaTeX\ files into Word97, we use an intermediate MS-DOS
routine \cite{tth} that converts the \TeX\ source into HTML\@.  The
routine is generally robust, but it works best if the source document
is clean \LaTeX\ without a significant freight of local macros or
\texttt{.sty} files.  Use of the source file \texttt{scifile.tex} as a
template, and calling {\it only\/} the \texttt{.sty} and \texttt{.bst}
files specifically mentioned here, will generate a manuscript that
should be eminently reviewable, and yet will allow your paper to
proceed quickly into our production flow upon acceptance \cite{use2e}.

\section*{Formatting Citations}

Citations can be handled in one of three ways.  The most
straightforward (albeit labor-intensive) would be to hardwire your
citations into your \LaTeX\ source, as you would if you were using an
ordinary word processor.  Thus, your code might look something like
this:

\begin{quote}
\begin{verbatim}
However, this record of the solar nebula may have been
partly erased by the complex history of the meteorite
parent bodies, which includes collision-induced shock,
thermal metamorphism, and aqueous alteration
({\it 1, 2, 5--7\/}).
\end{verbatim}
\end{quote}

\noindent Compiled, the last two lines of the code above, of course, would give notecalls in {\it Science\/} style:

\begin{quote}
\ldots thermal metamorphism, and aqueous alteration ({\it 1, 2, 5--7\/}).
\end{quote}

Under the same logic, the author could set up his or her reference list as a simple enumeration,

\begin{quote}
\begin{verbatim}
{\bf References and Notes}

\begin{enumerate}
\item G. Gamow, {\it The Constitution of Atomic Nuclei
and Radioactivity\/} (Oxford Univ. Press, New York, 1931).
\item W. Heisenberg and W. Pauli, {\it Zeitschr.\ f.\
Physik\/} {\bf 56}, 1 (1929).
\end{enumerate}
\end{verbatim}
\end{quote}

\noindent yielding

\begin{quote}
{\bf References and Notes}

\begin{enumerate}
\item G. Gamow, {\it The Constitution of Atomic Nuclei and
Radioactivity\/} (Oxford Univ. Press, New York, 1931).
\item W. Heisenberg and W. Pauli, {\it Zeitschr.\ f.\ Physik} {\bf 56},
1 (1929).
\end{enumerate}
\end{quote}

That's not a solution that's likely to appeal to everyone, however ---
especially not to users of B{\small{IB}}\TeX\ \cite{inclme}.  If you
are a B{\small{IB}}\TeX\ user, we suggest that you use the
\texttt{Science.bst} bibliography style file and the
\texttt{scicite.sty} package, both of which we are downloadable from our author help site
(http://www.sciencemag.org/about/authors/prep/TeX_help/).  You can also
generate your reference lists by using the list environment
\texttt{\{thebibliography\}} at the end of your source document; here
again, you may find the \texttt{scicite.sty} file useful.

Whether you use B{\small{IB}}\TeX\ or \texttt{\{thebibliography\}}, be
very careful about how you set up your in-text reference calls and
notecalls.  In particular, observe the following requirements:

\begin{enumerate}
\item Please follow the style for references outlined at our author
  help site and embodied in recent issues of {\it Science}.  Each
  citation number should refer to a single reference; please do not
  concatenate several references under a single number.
\item Please cite your references and notes in text {\it only\/} using
  the standard \LaTeX\ \verb+\cite+ command, not another command
  driven by outside macros.
\item Please separate multiple citations within a single \verb+\cite+
  command using commas only; there should be {\it no space\/}
  between reference keynames.  That is, if you are citing two
  papers whose bibliography keys are \texttt{keyname1} and
  \texttt{keyname2}, the in-text cite should read
  \verb+\cite{keyname1,keyname2}+, {\it not\/}
  \verb+\cite{keyname1, keyname2}+.
\end{enumerate}

\noindent Failure to follow these guidelines could lead
to the omission of the references in an accepted paper when the source
file is translated to Word97 via HTML.

\section*{Handling Math, Tables, and Figures}

Following are a few things to keep in mind in coding equations,
tables, and figures for submission to {\it Science}.

\paragraph*{In-line math.}  The utility that we use for converting
from \LaTeX\ to HTML handles in-line math relatively well.  It is best
to avoid using built-up fractions in in-line equations, and going for
the more boring ``slash'' presentation whenever possible --- that is,
for \verb+$a/b$+ (which comes out as $a/b$) rather than
\verb+$\frac{a}{b}$+ (which compiles as $\frac{a}{b}$).  Likewise,
HTML isn't tooled to handle certain overaccented special characters
in-line; for $\hat{\alpha}$ (coded \verb+$\hat{\alpha}$+), for
example, the HTML translation code will return [\^{}$(\alpha)$].
Don't drive yourself crazy --- but if it's possible to avoid such
constructs, please do so.  Please do not code arrays or matrices as
in-line math; display them instead.  And please keep your coding as
\TeX-y as possible --- avoid using specialized math macro packages
like \texttt{amstex.sty}.

\paragraph*{Displayed math.} Our HTML converter sets up \TeX\
displayed equations using nested HTML tables.  That works well for an
HTML presentation, but Word97 chokes when it comes across a nested
table in an HTML file.  We surmount that problem by simply cutting the
displayed equations out of the HTML before it's imported into Word97,
and then replacing them in the Word document using either images or
equations generated by a Word equation editor.  Strictly speaking,
this procedure doesn't bear on how you should prepare your manuscript
--- although, for reasons best consigned to a note \cite{nattex}, we'd
prefer that you use native \TeX\ commands within displayed-math
environments, rather than \LaTeX\ sub-environments.

\paragraph*{Tables.}  The HTML converter that we use seems to handle
reasonably well simple tables generated using the \LaTeX\
\texttt{\{tabular\}} environment.  For very complicated tables, you
may want to consider generating them in a word processing program and
including them as a separate file.

\paragraph*{Figures.}  Figure callouts within the text should not be
in the form of \LaTeX\ references, but should simply be typed in ---
that is, \verb+(Fig. 1)+ rather than \verb+\ref{fig1}+.  For the
figures themselves, treatment can differ depending on whether the
manuscript is an initial submission or a final revision for acceptance
and publication.  For an initial submission and review copy, you can
use the \LaTeX\ \verb+{figure}+ environment and the
\verb+\includegraphics+ command to include your PostScript figures at
the end of the compiled PostScript file.  For the final revision,
however, the \verb+{figure}+ environment should {\it not\/} be used;
instead, the figure captions themselves should be typed in as regular
text at the end of the source file (an example is included here), and
the figures should be uploaded separately according to the Art
Department's instructions.

\section*{What to Send In}

What you should send to {\it Science\/} will depend on the stage your manuscript is in:

\begin{itemize}
\item {\bf Important:} If you're sending in the initial submission of
  your manuscript (that is, the copy for evaluation and peer review),
  please send in {\it only\/} a PostScript or PDF version of the
  compiled file (including figures).  Please do not send in the \TeX\
  source, \texttt{.sty}, \texttt{.bbl}, or other associated files with
  your initial submission.  (For more information, please see the
  instructions at our Web submission site,
  http://www.submit2science.org/ .)
\item When the time comes for you to send in your revised final
  manuscript (i.e., after peer review), we require that you include
  all source files and generated files in your upload.  Thus, if the
  name of your main source document is \texttt{ltxfile.tex}, you
  need to include:
\begin{itemize}
\item \texttt{ltxfile.tex}.
\item \texttt{ltxfile.aux}, the auxilliary file generated by the
  compilation.
\item A PostScript file (compiled using \texttt{dvips} or some other
  driver) of the \texttt{.dvi} file generated from
  \texttt{ltxfile.tex}, or a PDF file distilled from that
  PostScript.  You do not need to include the actual \texttt{.dvi}
  file in your upload.
\item From B{\small{IB}}\TeX\ users, your bibliography (\texttt{.bib})
  file, {\it and\/} the generated file \texttt{ltxfile.bbl} created
  when you run B{\small{IB}}\TeX.
\item Any additional \texttt{.sty} and \texttt{.bst} files called by
  the source code (though, for reasons noted earlier, we {\it
    strongly\/} discourage the use of such files beyond those
  mentioned in this document).
\end{itemize}
\end{itemize}