Efficient Suspected Infected Crowds Detection Based on Spatio-Temporal Trajectories

Abstract: Virus transmission from person to person is an emergency event facing the global public. Early detection and isolation of potentially susceptible crowds can effectively control the epidemic of its disease. Existing metrics can not correctly address the infected rate on trajectories. To solve this problem, we propose a novel spatio-temporal infected rate (IR) measure based on human moving trajectories that can adequately describe the risk of being infected by a given query trajectory of a patient. Then, we manage source data through an efficient spatio-temporal index to make our system more scalable, and can quickly query susceptible crowds from massive trajectories. Besides, we design several pruning strategies that can effectively reduce calculations. Further, we design a spatial first time (SFT) index, which enables us to quickly query multiple trajectories without much I/O consumption and data redundancy. The performance of the solutions is demonstrated in experiments based on real and synthetic trajectory datasets that have shown the effectiveness and efficiency of our solutions.

\section{Introduction}

Human to human virus-borne infections has been a public health concern. A new coronavirus was named "SARS-CoV-2," and the disease it caused was called "Coronavirus Disease 2019" (abbreviated as "COVID-19")~\cite{zhao2020comparative}. COVID-19 pandemic is an urgent emergency facing the world. In the absence of a vaccine, early detection, early reporting, early isolation, and early treatment~\footnote{http://dwz1.cc/mAz79Dq} have proven to be the most effective measures to prevent the spread of the epidemic.

With the rapid development of mobile internet and locate service, massive spatio-temporal data have been generating from applications. Human activity trajectory is a typical spatio-temporal data, including longitude, latitude, and time. Given a trajectory $Q$ of the confirmed patient, Suspected Infected Crowds Detection (SICD) aims to detect close contacts through the spatio-temporal correlation of trajectories. As shown in Figure~\ref{fig:sic}, we search the ordinary people who have occurred within the spatio-temporal range which can be infected with the location where the patient has appeared and then determine the probability of infection rate based on their contact distance and duration. SICD helps local governments to investigate suspected people and find close contacts, isolate and protect them in time to prevent further spread of the epidemic. For improving the accuracy of SICD, it is necessary to consider the spatio-temporal correlation in each location of Q to describe the infection rate. The data volume of the underlying trajectory database used for SICD is enormous. To avoid massive memory consumption, we leverage a spatio-temporal index to manage trajectories in the NoSQL database.

In the epidemiological analysis, analyzing the relationship between people in spatial and temporal is a very standard and important analytical method. By looking at the spatial and temporal relationship, we can draw accurate close contact conclusions. In the 19th century, Snow~\cite{vinten2003cholera}, studied spatio-temporal data such as maps and found that the source of pollution in cholera cases was not air, but from public pumps on Broad Street and transmitted through contaminated drinking water. At his appeal, authorities closed and diverted pump valves to control cholera. The successful prevention of cholera is directly related to the result of spatio-temporal data analysis and is the most classic example of spatio-temporal big data analysis.

We can acquire the multiple spatio-temporal data information related to the risk of infection within the given spatio-temporal ranges through the trajectory of human activity. Moving trajectory is typical spatio-temporal data. By investigating the patient's moving trajectory, we can know who the patient is in close contact. In response to this demand, Tang discovered object groups that travel together from streaming trajectories~\cite{tang2012discovery}. In his study, in order to find travel companions, the system needs to cluster the objects of each snapshot of the query trajectory and intersect the clustering results, and retrieve the objects that move together. It is an efficient system with high precision, and it can discover crowds with a long companion. However, it can not find crowds whose local companion is long in one snapshot but total companion is short in all snapshots. Besides, many scholars have done much research on the spatial and temporal similarity between trajectories. Some of these methods mainly focus on how to extend existing trajectory similar search algorithms (e.g., ED, DTW, and Fréchet~\cite{toohey2015trajectory}). They have excellent performance for detecting duplicated or redundant trajectories in the database. However, to ensure accuracy, they require the consistency of the sampling rate of the two trajectories to be relatively high, and they lack the anti-noise ability. Some other trajectory similarity algorithms study the spatio-temporal correlation. They calculate spatial and temporal correlations separately and then combine them into overall similarity.  A liner combination method (e.g.,~\cite{shang2014personalized}) combines the spatial and temporal into a spatio-temporal similarity metric. Other existing trajectory similarities (e.g.,~\cite{bakalov2005efficient,bakalov2006continuous}) use a time interval to limit the resemblance of two trajectories. Unfortunately, they calculate similarity on the entire trajectory. However, in the actual data set, the human trajectory is not always successive and may has a larger spatial and temporal range. Therefore, in many applications, the trajectory must be segmented. However, the local segment similarity of the trajectory is not comprehensive in existing metrics. Hence, in this paper, we propose a measure that weights every segment of the trajectory, because the longer a segment of the query trajectory stays, the more possible that others will meet it. In each segment, we consider not only spatial nearing but also temporal proximity in every location, which makes sure the spatio-temporal closed location has a high infected rate.
\begin{figure}
	\centering
	\includegraphics[width=3.0in]{sic.pdf}
	\caption{An example of Suspected Infected Crowds Detection.}
	\label{fig:sic}
\end{figure}
COVID-19~\cite{zhao2020comparative} is an urgent emergency facing the world. It has an incubation period. Thus, people infected with COVID-19 do not immediately have severe symptoms after they infect the human body, with an average of 5 to 6 days and a range of 1 to 14 days. Coronavirus infection during the incubation period is also infectious. Therefore, it is necessary to confirm all GPS records of the confirmed diagnosis from the prior incubation period, which is a large amount of data with a large spatio-temporal range. However, most of the existing solutions should load all data into memory, which limits the scalability. In this paper, we first divide the long and large trajectory into several sub-segments with suitable length and spatio-temporal range. Then, we build an XZ2T~\cite{li2020just} index to manage large segments in the NoSQL database via an efficient platform JUST~\cite{li2020just}, which guarantees the scalability of our solutions. Many methods mainly focus entirely on similarity. Thus, they are slow due to the large consumption for calculating big trajectory similarity. Therefore, recent researches (e.g.,~\cite{shang2014discovery,shang2017trajectory,shang2014personalized,ta2017signature}) have focused on some pruning strategies to save time. They build indexes on trajectories to avoid computing all trajectories similarity, which greatly accelerates query speed. In this paper, we only search the smaller spatio-temporal range of each segment to reduce the range of the candidate set and propose some pruning strategies, which help reduce the computation. In most scenarios, it is always to query close contacts for multiple patients. Thus, we build an efficient SFT index to reallocate segments with the similar spatio-temporal range together, which effectively reduces the I/O cost and data redundancy.

Using our algorithm, we helped Beijing find more than 500 high-risk close contacts within 20 days. Until March 1, we assisted Suqian in discovering a quarter of newly diagnosed patients with COVID-19 in the city. Within China, 18 provinces and cities such as Guangzhou, Nanjing, and Chengdu used this algorithm as part of the high-risk population analysis system.

To sum up, the contributions of this paper are as follows.
\begin{itemize}
	\item We propose a new infection rate (IR) metric that takes into account both the spatial and temporal proximity in all segments of trajectory and is suitable for the Suspected Infected Crowds Detection (SICD). It can also use to recognize similar trajectories, detect close contacts, mine companion, and monitor high-risk groups.
	\item We store primary trajectories in the NoSQL database and only need to search a small spatio-temporal range data when it comes to a query trajectory, which reduces the memory consumption and guarantees the scalability of our solutions.
	\item We leverage some effective pruning strategies to avoid many invalid calculations.
	\item We develop an SFT index to reduce I/O communication and data redundancy.
	\item We conduct extensive experiments on trajectory sets to validate the performance of the proposed algorithms.
\end{itemize}

The rest of the paper is organized as follows. Section 2 introduces the basic definitions and trajectory infected rate. The framework of our solution is presented in Section 3. The trajectory infected rate query is described in Section 4, while the trajectory infected rate join query for multiple patients is in Section 5. The experimental results are presented in Section 6. Related work is illustrated in Section 7, and conclusions and future works are shown in Section 8.

\section{PRELIMINARIES}
In this section, we introduce the basic definitions and spatio-temporal operations in our present approaches.

\subsection{Trajectory}
The trajectory is a typical representation of a set of spatio-temporal locations for the same user. A location in the trajectory is of the form (longitude, latitude, time), and all locations of trajectory are sorted by its timestamp. A trajectory is defined as follows.
\newdef{definition}{Definition}
\begin{definition}
	A trajectory $T$ of moving object is a time-ordered locations $<l_1,l_2,...,l_n>$, where location $l_i = (p_i,t_i)$, $ i \in [1,n]$, with $p_i$ is a spatial point, $t_i$ is a timestamp, and $n$ is the location size.
\end{definition}
\subsection{Segmentation}
In order to fully depict the virus carrier's infected trajectory, we must collect all of its spatio-temporal records generated by GPS terminals from the incubation period to isolation. However, records may not be collected continuously, such as when the terminal is shut down for a while. Thus, the spatial or temporal interval between the two nearest records may be extremely large. If the entire trajectory is stored as a whole, a sizeable spatio-temporal range is required to contain this trajectory, and collected records may be intermittent that two nearest GPS records cannot be directly connected. Meanwhile, the spatio-temporal nearest locations always have similar characteristics. Storage together can improve the efficiency of the infected rate calculation because data redundancy can be avoided. Therefore, the trajectory must be segmented. In this paper, we use the stay point detection algorithm~\cite{cloudtp} to segment the trajectory. As shown in Figure~\ref{fig:segment}, we divide trajectory into four segments marked with red boxes, where the spatial distance and time interval between any two locations in any segment do not exceed fixed thresholds (e.g., 200m and 30minutes in Figure~\ref{fig:segment}), respectively.
\begin{figure}
	\centering
	\includegraphics{segment}
	\caption{A sample of segmentation.}
	\label{fig:segment}
\end{figure}

\begin{definition}
	A trajectory  $T$ can be represented by segments. $ T = <s_1,s_2,...,s_m>$, where $m$ is the segment size of this trajectory, segment $s_i = <l_{i1},l_{i2},...,l_{ik}>$ is a sub-trajectory, and segments are sorted by the timestamps of their start locations.
\end{definition}

\subsection{Spatiotemporal Operations}
In this paper, we focus on spatio-temporal operations over virus carrier's moving trajectories to link their close contact users. Thus, in this part, we discuss spatio-temporal operations of our propose approaches.

Carriers of COVID-19 affect people with whom they have close contact. Therefore, the factors of infected rate are related to the spatio-temporal distance between the ordinary person and the patient. Thus, each position where the carrier appears has an influential spatio-temporal range.
\begin{definition}
	Given a location $l$, a spatial infected range threshold $\theta_d$ and a temporal infected range threshold $\theta_t$. $STR(l,\theta_d,\theta_t)$ represents an influential spatio-temporal range of location $l$. Formally,
	$$STR(l,\theta_d,\theta_t) = \{R|\forall r \in R(|r.t - l.t| \le \theta_t \land dist(r.p,l.p) \le \theta_d)\}$$
\end{definition}

In the range $STR(l,\theta_d,\theta_t)$ of location $l$, we calculate spatio-temporal correlation of $l$ for trajectory.

\begin{definition} Given a location $l$, a trajectory $T$, a distance threshold $\theta_d$ and a time threshold $\theta_t$. The spatio-temporal correlation is defined as follows:
	\begin{equation}
	st\_cor(l,T) = \max_{v \in T \land v \in STR(l,\theta_d,\theta_t)}{st\_dist(l,v)}
	\label{st_cor},
	\end{equation}
\end{definition}
where $st\_dist(l,v)$ is the spatio-temporal correlation between $l$ and a location $v$ of trajectory $T$. Formally,
\begin{equation}
st\_dist(l,v) = \lambda e^{-\frac{dist(l.p,v.p)}{\theta_d}} + (1-\lambda)e^{-\frac{|l.t - v.t|}{\theta_t}}
\label{st_cor_s},
\end{equation}
where parameter $\lambda \in [0,1]$ controls the relative importance of the spatial and temporal correlation. $\frac{dist(l.p,v.p)}{\theta_d}$ and $\frac{|l.t - v.t|}{\theta_t}$ normalize the effects of spatial distance and time interval to the same range. Note, while trajectory $T$ does not intersect with $STR(l,\theta_d,\theta_t)$, the spatiotemporal correlation $st\_cor(l, T)$ is 0.

\subsection{Trajectory Infected Rate}
\subsubsection{Segment Infected Rate}
Given a segment $s$ of virus carrier's trajectory and a trajectory $T$, the infected rate between $s$ and $T$ is defined as follows:
\begin{equation}
IR(s, T) = \frac{\sum_{l \in s}{st\_cor(l,T)}}{|s|}
\label{IR_s},
\end{equation}
where $l$ is a location of $s$ and $|s|$ represents the number of locations that $s$ owns.
\subsubsection{Trajectory Infected Rate}
Given a virus carrier's query trajectory $Q$ and a trajectory $T$, the infected rate between $Q$ and $T$ is defined as follows:
\begin{equation}
IR(Q, T) = \sum_{i=1,s_i \in Q}^{m}{P(s_i) * IR(s_i,T)}
\label{IR_T},
\end{equation}
where $s_i$ is a segment of $Q$ and $P(s_i)$ represents the potential infected probability of each segment in $Q$. The probability $P(s_i)$ is determined by time span in segment, on account of more time the carrier stays more risk of infection the others may have. Therefore, $P(s_i)$ is defined as follows:
\begin{equation}
P(s_i) = \frac{s_i.et - s_i.st + 1}{\sum_{j=1}^{m}{(s_j.et - s_j.st + 1)}},
\label{ps}
\end{equation}
where $m$ represents the number of segments in the query trajectory $Q$, $s_i.st$ represents the start time of segment and $s_i.et$ represents the end time of segment, respectively.

\subsection{Problem Definitions}
Given a query trajectory $Q$, a set of trajectories $\mathbf{T}$ and a threshold $\theta$, the trajectory infected rate query(IRQ) finds a set of trajectories $\mathbf{T}'$ from the set whose trajectory infected rate exceeds $\theta$, i.e., $\forall T \in \mathbf{T}'(IR(Q,T) > \theta)$.

Given a set of query trajectories $\mathbf{Q}$, a set of trajectories $\mathbf{T}$ and a threshold $\theta$, the trajectory infected rate join query(IRJQ) finds a set of all trajectory pairs from the two sets whose trajectory infected rate exceeds $\theta$, i.e., $\forall (Q_i,T_j) \in \mathbf{Q \Join T}(IR(Q_i,T_j) > \theta)$.

\section{Framework}

Figure~\ref{fig:framework} depicts the architecture of the trajectory infected rate query, which consists of three processes: Data Preprocessing, Indexing and Storing, and Infectivity Query.
\begin{figure}
	\centering
	\includegraphics[width=3in]{framework}
	\caption{The framework of our solutions.}
	\label{fig:framework}
\end{figure}

\subsection{Data Preprocessing}
In many applications, trajectory preprocessing is not only necessary for filtering noise but also crucial for indexing and storing. As depicted in the bottom-most box of Figure~\ref{fig:framework}, the process of preprocessing contains two main tasks: 1) noise filtering, which eliminates outlier GPS records that may be caused by the weak signal of GPS terminals; 2) segmentation, which breaks a long trajectory into suitable short trajectories. This paper mainly focuses on the trajectory indexing and infectivity query. For more details about trajectory preprocessing, please refer to our previous work~\cite{cloudtp}.

\subsection{Indexing and Storing}
As shown in the middle box of Figure~\ref{fig:framework}, we use the XZ2T index to organize the segment and then store it as a table into the NoSQL database via JUST~\cite{li2020just}, which can efficiently and conveniently manage big spatio-temporal data.

Spatio-temporal range query is a necessary step in our algorithm. Indexing is essential for the processing of spatio-temporal query. Thus, we build an XZ2T index on the segment to effectively support the spatio-temporal range query.

XZ2T index is an extension of the XZ2 index~\cite{boxhm1999xz}, which projects a geographical polygon with a time range onto a one-dimensional value. XZ2 index is based on XZ-Ordering, a Space-Filling Curve for Spatially Extended Objects. It uses a sophisticated coding scheme for a polygon, which maps the polygon into the integer domain. As shown in Figure~\ref{fig:xz2t} (a), XZ2 index divides the root element into four sub-elements with equal size, which are numbered from 0 to 3. Then, the XZ2 index recursively numbers each sub-space until the maximum resolution is reached. Finally, we can get a sequence formed by successively traversing numbers. A polygon is represented by the most appropriate element or xelement of the xz2 index, which can completely cover the polygon. The xelement is an enlarged area of the element in xz2 index (i.e., the xelement of ``210" represents the area covered by the element ``21", and the width and height of $t_2$ are lower than ``210". Thus, instead of ``21", we can use ``210" represents $t_2$. Similarly, ``12" represents $t_1$, ``221" represents $t_3$, ``0" represents $t_4$, respectively). However, XZ2 index only supports spatial data. Therefore, considering the time dimension, the XZ2T index is designed, which allocates each disjoint period an XZ2 index, as shown in Figure~\ref{fig:xz2t} (b). Specifically, given a segment's spatio-temporal range $(mbr,st, et)$, we first calculate the period number $bin$ of $st$ according Equ (\ref{bin}), then calculate its XZ2 index number by using XZ-ordering function $XZ2(mbr)$. Finally, we combine the period number and XZ2 index to indicate the spatio-temporal range of segment. Note, our segmentation algorithm guarantees the span of $et-st$ not greater than $periodLen$ and $et$ belongs to the same $bin$ of $st$.
\begin{equation}
bin = Unit.between(t, epoch) / periodLen
\label{bin}
\end{equation}

In Equ (\ref{bin}), $epoch$ is the reference time (e.g., 1970-01-01T00:00:00), and $periodLen$ is the span of a period. $Unit$ represents the unit of time (e.g., day, week, month, year).

To support offline infectivity query and avoid all trajectories is persisted in memory because memory resources are expensive and insufficient, we store segments of trajectory to NoSQL database (e.g., HBase) via JUST. The key of our table is consisted by a shard, XZ2T index, and sid:
$$key = shard + XZ2T(s) + sid,$$
where $shard$ is a hash number to achieve load balance; $XZ2T$ encodes the segment's spatio-temporal information; $sid$ is the unique id of each segment.
\begin{figure}
	\centering
	\includegraphics[width=3.3in]{xz2t}
	\caption{The example of XZ2 and XZ2T index.}
	\label{fig:xz2t}
\end{figure}

\subsection{Infectivity Querying}
The top-most box of Figure~\ref{fig:framework} shows two tasks of infectivity query, including Infected Rate Query(IRQ) and Infected Rate Join Query(IRJQ). IRQ finds trajectories that had close contact with a virus carrier (see Section 4). IRJQ finds trajectory pairs from two sets whose infectivity exceeds $\theta$ (see Section 5).  In the following sections, we introduce the details of IRQ and IRJQ.

\section{Infection Rate Query}
\subsection{Main idea}
We propose a spatio-temporal correlation-based infectivity query. First, we break the long query trajectory $Q$ to suitable segments $S$. Second, we extract and extend the spatio-temporal range of each segment. Third, we query all infected segments covered by the spatio-temporal ranges of $S$ from database and aggregate the same trajectory's segments together (Section 4.2). Furthermore, we prune the trajectories whose infected rate can not greater than $\theta$. After pruning, we calculate the infectivity between the remain trajectories and $Q$. Finally, we filter the trajectories whose infected rate is lower than $\theta$ out and return the results (Section 4.3).
\subsection{ST Query}
\subsubsection{Main idea}
\begin{figure}
	\centering
	\includegraphics[width=3in]{base_query}
	\caption{A sample of spatio-temporal query.}
	\label{fig:base_query}
\end{figure}
In this solution, we must query segments from database with XZ2T-Index around each location of the query trajectory. However, there are many locations need to access database that cause the massive queries and data redundancy. Thus, we cluster close locations to a spatio-temporal range and use this range to query the data once from the database.

The raw long trajectory may need a large spatial and temporal range to cover it entirely. For example, a trajectory may go to many places for ten days. Hence, we break the trajectory where the spatial distance and time interval of any two nearest locations exceeds the fixed thresholds. As shown in Figure~\ref{fig:base_query}, the query trajectory $Q$ (the trajectory with blue in  Figure~\ref{fig:base_query}) is divided into two segments ($s_1$ and $s_2$). Each segment has a $mbr$ (minimum bounding rectangle, e.g., the red solid line rectangle of Figure~\ref{fig:base_query}) that covers all locations and a time range that starts from the first location's time and ends of the last location's time (e.g., the solid red lines between $q_1.t$ and $q_3.t$ in Figure~\ref{fig:base_query}). Then, we extend the $mbr$ outward $\theta_d$ (e.g., the red dotted line rectangle in Figure~\ref{fig:base_query}) and enlarger the time range of $\theta_t$ (e.g., the red dotted lines near $q_1.t$ and $q_3.t$ in Figure~\ref{fig:base_query}), where $\theta_d$ and $\theta_t$ control the infected spatio-temporal range. Later we query segments from the database via extended spatio-temporal ranges. As shown in Figure~\ref{fig:base_query}, the query trajectory $Q$ is divided into two segments $s_1$ and $s_2$. Segments of trajectory $t_1$ and $t_2$ are queried by $s_1$ and  $t_1$ and $t_3$ are queried by $s_2$, respectively. We group segments of the same trajectory together. Finally, $t_1,t_2$ and $t_3$ are the candidate trajectories of the query trajectory $Q$.

\subsubsection{Query from XZ2TTable}
We store data in the NoSQL database. Each segment is saved as the form of ``(key, value)" with file dictionary sort index, and the key consists of XZ2T index and other information. Thus, generating accurate and smaller key scan ranges for query processing can significantly reduce I/O costs.

\textbf{Key Scan Ranges Generation.}
First, we give a spatio-temporal range of the query segment, which represents as a time range and a spatial range. Second, we extract the period numbers overlapped by the time range. Third, we generate the spatial scan ranges calculated by the XZ2 index~\cite{boxhm1999xz}. Later, for each period number, we execute the scans. Finally, we refine the result to make sure that exactly in the spatio-temporal field.

The spatial scan range is generated as follows:  (1) Starting the recursive access from the root node by the breadth-first search; (2) if the current node intersects the spatial query window partly, the index value of this node is added to the scan queue and  recursive access child nodes until arriving the max resolution; (3) if the query window completely covers the current node, we put the index range represented by it and all its child nodes into the scan queue; (4) when the leaf nodes intersect, the index values of the leaf nodes are put into the queue; (5) if there is no intersecting node, the node is skipped directly. Finally, we combine the consecutive values in the scan queue to form the final scan range.

\subsection{Pruning}
The calculation of IRQ is time-consuming. Therefore, we develop several pruning strategies to avoid unnecessary calculations.
\newtheorem{theorem}{Lemma}
\begin{theorem}
	Let $s$ represents a segment of query trajectory $Q$ whose extended spatio-temporal range intersects with the candidate trajectory $T$. $T$ must satisfy Equ~(\ref{theorem1}):
	\begin{equation}\sum_{s \in Q\cap T }P(s) \ge \theta
	\label{theorem1}
	\end{equation}
	\label{Lemma1}
\end{theorem}

\begin{proof}
	Clearly $IR(s,T) \in [0,1]$. If $T$ does not intersect with $s$, $IR(s,T)$ equals 0, and the maximum value of $IR(s,T)$ is 1. Thus, based on Equ~\ref{IR_T}, we have that

	\begin{math}
	\begin{aligned}
	IR(Q, T) &= \sum_{s \in Q}{P(s) * IR(s,T)} \\
	& \leq  \sum_{s \in Q\cap T}P(s)* IR(s,T) + \sum_{s \in Q \not\cap T }P(s)* IR(s,T)\\
	& \leq \sum_{s \in Q\cap T}P(s)
	\end{aligned}
	\end{math}

	Thus, if $\sum_{s \in Q\cap T}P(s) < \theta$, the $IR(Q,T) < \theta$. Therefore, $\sum_{s \in Q\cap T}P(s)$ must be equal or greater than $\theta$.
	\label{proof1}
\end{proof}

\begin{theorem} The $IR(s, T)$ must satisfy Equ (\ref{l2}):
	\begin{equation}IR(s,T) \ge \frac{\theta -1 + P(s)}{P(s)}
	\label{l2}
	\end{equation}
	\label{theorem2}
\end{theorem}

\begin{proof}
	Let $s_i$ represents a segment of $Q$. Based on Equ~\ref{ps}, we have $\sum_{i=1}^{m}P(s_i) =1$. Then by Equ~\ref{IR_T}, we have

	\begin{math}
	\begin{aligned}
	IR(Q, T) &= \sum_{j=1}^{m}{P(s_j) * IR(s_j,T)} \\
	& \leq P(s_i)*IR(s_i,T) + \sum_{j=1, j \neq i}^{m} {P(s_j)}  \\
	& \leq P(s_i)*IR(s_i,T) + 1 - P(s_i)
	\end{aligned}
	\end{math}

	Thus, if $IR(s_i, T) < \frac{\theta -1 + P(s_i)}{P(s_i)}$, then $IR(Q,T) < \theta$, and thus trajectory $T$ can be entirely pruned.
	\label{proof2}
\end{proof}

\begin{theorem} $IR(s,T)$ must satisfy Equ (\ref{lm3}):
	\begin{equation}IR(s,T) \ge \frac{\theta -\sum_{qs \in Q\cap T,qs \neq s}{P(qs)}}{P(s)},
	\label{lm3}
	\end{equation}
	\label{theorem3}where $qs$ is the segment of $Q$.
\end{theorem}
\begin{proof}
	By combining Lemmas~\ref{Lemma1} and ~\ref{theorem2}, we have

	\begin{math}
	\begin{aligned}
	IR(Q, T) &= \sum_{s_j \in Q}{P(s_j) * IR(s_j,T)} \\
	& \leq  P(s_i)*IR(s_i,T) + \sum_{s_j \in Q\cap T, i \neq j} {P(s_j)},
	\end{aligned}
	\end{math}

	where $s_i \in Q$. Thus, if $IR(s_i, T) < \frac{\theta -\sum_{s_j \in Q\cap T, i \neq j}{P(s_j)}}{P(si)}$, then $IR(Q,T) < \theta$, namely, the trajectory $T$ can be entirely pruned.
	\label{proof3}
\end{proof}

\begin{theorem} Let $s_i$ represent a segment which intersects with the candidate trajectory $T$. Then

	\begin{equation}IR(s_i,T) \ge \frac{\theta -\sum_{j = 1}^ {i-1}{IR(s_j,T) * P(s_j)} - \sum_{j=i+1,s_j \in Q \cap T}^{m}P(s_j)}{P(s)}
	\end{equation}
	\label{theorem4}

\end{theorem}

\begin{proof}
	By combining Lemmas~\ref{Lemma1},~\ref{theorem2} and ~\ref{theorem3}, we have

	\begin{math}
	\begin{aligned}
	IR(Q, T) &= \sum_{i=1,s_i \in Q}^{m}{P(s_i) * IR(s_i,T)} \\
	& = IR(s_i,T)* P(s_i) + \sum_{j=1}^{i-1} {P(s_j)*IR(s_j,T)}  \\
	& + \sum_{j=i+1}^{m} {P(s_j)*IR(s_j,T)} \\
	& \le  IR(s_i,T)* P(s_i) + \sum_{j=1}^{i-1} {P(s_j)*IR(s_j,T)} \\
	& + \sum_{j=i+1,s_j \in Q \cap T}^{m} {P(s_j)}
	\end{aligned}
	\end{math}

	Thus, if $IR(s_i,T) < \frac{\theta -\sum_{j = 1}^ {i-1}{IR(s_j,T) * P(s_j)} - \sum_{j=i+1,s_j \in Q \cap T}^{m}P(s_j)}{P(s_i)}$,  then $IR(Q,T) < \theta$. Therefore, trajectory $T$ can be entirely pruned.
	\label{proof4}
\end{proof}
\begin{figure}
	\centering
	\includegraphics[width=3.3in]{p_fm}
	\caption{The structure of IRJQ.}
	\label{fig:p_fm}
\end{figure}
\begin{algorithm}[htb]
	\caption{ Infected Rate Query.}
	\label{alg:irq}
	\begin{algorithmic}[1]
		\Require
		The query trajectory, $Q$;
		The candidate trajectories, $\mathbf{T}$;
		A threshold, $\theta$;
		\Ensure
		Trajectories with infected rate exceed $\theta$;
		\State $result = new ~ ArrayList()$; $S = segmentation(Q);$
		\For{each $T_i\in \mathbf{T}$}
		\State //Lemma 1
		\State	$sum = 0$;
		\For{each $s$ \ in $S \cap T_i$}
		\State	$sum = sum + P(s)$;
		\EndFor
		\If{$sum < \theta$}
		continue;
		\EndIf
		\State $totalIR = 0$;  $remPS = sum$; $pruned = false$;
		\For{each $s$ \ in $S$}
		\State 	$IRP = IR(s, T) * P(s)$;
		\If{$IRP  <  {\theta - 1 + P(s)}$}//Lemma 2
		\State $pruned = false$;break;
		\EndIf

		\If{$s \cap T \neq \emptyset$}
		\If{$IRP  <  {\theta - (sum - P(s))}$}//Lemma 3
		\State $pruned = true$; break;
		\EndIf
		\State $remPS = remPS- P(s)$;
		\EndIf
		\State //Lemma 4
		\If{$IRP < \theta-totalIR - remPS $}
		\State $pruned = true$; break;
		\EndIf
		\State	$totalIR = totalIR + IRP$;
		\EndFor
		\State //filtering
		\If{$pruned = false$ and $totalIR < \theta$}
		\State $result.add(T_i)$;
		\EndIf
		\EndFor
		\State
		\Return $result$;
	\end{algorithmic}
\end{algorithm}
\subsection{Algorithm}
In Algorithm~\ref{alg:irq}, the infected rate query arguments are a query trajectory $Q$, a threshold $\theta$, and a candidate set $\mathbf{T}$, and the query result is a trajectory set of close contacts for $Q$. Initially, we let an empty ArrayList to hold the result, and $S$ is a set of segments of $Q$. Then, for each scanned trajectory $T_i$, we let $IRP = IR(s,T) * P(s)$ simplify Lemmas 2-4. The lines 5-8 for Lemma 1, lines 12-15 for Lemma 2, lines 17-19 for lemma 3, and lines 23-25 for Lemma 4. If $T_i$ does not satisfy any Lemmas 1-4, then we use the instructions (e.g., continue and break) to stop further computing $IRP$ of $T_i$ (e.g., in line 8, the sum of $P(s)$ is not greater than $\theta$, then we stop calculating $IR(Q,T_i)$ and continue to check the next trajectory $T_{i+1}$, and the $IRP$ is lower than $\theta$ in line 13, then break calculate the $IRP$ of the left segments of $Q$ and start to check next trajectory). If $T_i$ is not pruned, then it is added to the result in line 30. Then in line 33 of Algorithm~\ref{alg:irq} , we return the final result after all trajectories of $\mathbf{T}$ have been checked and calculated.

\section{Join solution}

\subsection{Basic Idea}
\begin{figure*}
	\centering
	\includegraphics[width=6.8in]{sft_index}
	\caption{A sample of SFT index.}
	\label{fig:sft_index}
\end{figure*}
To process plenty of trajectories infected rates, we develop a join query solution, named Infected Rate Join Query (IRJQ). Figure~\ref{fig:p_fm} depicts the architecture of IRJQ, which consists of three processes: Data Preprocessing, Candidates Extraction, and Infectivity Exploration.

\textbf{Data Preprocessing.} As depicted in the top-most box of Figure~\ref{fig:p_fm}, we first filter noise locations of trajectories out. After that, we extract segments of trajectories by the rules described in Figure~\ref{fig:segment}. Later, we use the XZ2T index to organize segments and then store them into the NoSQL database via JUST. The details of the steps in data preprocessing are described in Sections 3.1 and 3.2.

\textbf{Candidates Extraction.} The middlebox of Figure~\ref{fig:p_fm} shows the procedure of candidates extraction, which allocates the coarse-grained high infect rate candidates to the query set. First, we load the query set; Second, we detect segments of this set; Third, we build an SFT-Tree (Spatial First Time Tree) index for all segments, which can minimize queries and communication on a small of data redundancy. Then, we query the candidates covered by the spatio-temporal ranges of leaf nodes in the SFT-Tree. This procedure will be a detailed introduction in Section 5.2.

\textbf{Infectivity Exploration.} The bottom-most box of Figure~\ref{fig:p_fm} shows the procedure of infectivity exploration. First, we prune some candidates, which are not the close contacts; Then, we calculate the infectivity of the segment. Meanwhile, we use a map to record some trajectory that can be further pruned. Later, we group all infectivity of the same trajectory's segments, and calculate the trajectory infectivity; Last, we filter trajectories whose infectivity is lower than $\theta$. This procedure is described in detail in Section 5.3.

\subsection{Candidates Extraction}
Candidates Extraction procedure is based on the segmentation algorithm, an SFT index strategy, and spatio-temporal query. The goal of segmentation is to diminution the long and large query trajectories on several individual segments.  We built an SFT index on all query segments, where segments with similar spatio-temporal range are placed in the same SFT leaf node. Then, we extract candidate segments from the database by the extended spatio-temporal range of each leaf node of SFT. Finally, we get coarse-grained candidates from the database with low data redundancy and a small I/O consumption.

\textbf{SFT index.} SFT (Spatial first Time index) is a two layers index. The segments of the query set are allocated to the leaf node of the SFT index with a suitable spatio-temporal region. Dividing segments into different spatio-temporal ranges can decrease I/O cost and reduce the size of the candidate set when query. As shown is Figure~\ref{fig:sft_index}, the SFT index structure is established as follows. First, we divide the spatial domain into four equal-sized regions, numbered from 0 to 3. Next, each region is recursed until reaching the maximum resolution. Then we build a time tree for each leaf node of the spatial first tree. As depicted in Figure~\ref{fig:sft_index} (c), data in each spatial region is indexed in time using a one-dimensional R-Tree-like structure~\cite{guttman1984r}. The internal node of the time tree has a one-dimensional time range and the MBR of all the leaf nodes it contains. The leaf node store the segments and the spatio-temporal range of all the segments.

For each segment, there is data with spatial and temporal information. As shown in Figure~\ref{fig:sft_index} (a), we first allocate the segments whose lower points of their MBR are located in the same node of the quadtree to the leaf node. Then, for segments in the same spatial node, we construct the T-Tree according to the start times of their time ranges. The index of the time layer is shown in Figure~\ref{fig:sft_index} (c). In the actual construction process, in order to reduce the size of the T-Tree, we  improve its insertion and add the function of merging consecutive time ranges. Assuming that the time range $[t_1,t_2]$ has reached T in T-Tree, a new time range $[t_3,t_3]$ also reaches T, and the two time ranges cross, then we combine $[t1, t2]$ and $[t3, t4]$ together, delete the $[t1, t2]$ node and insert the $[t1, t4]$ node, which not only reduces the size of the T-Tree, but also facilitates merging time ranges.

When the data in a node of T-Tree reaches the threshold, it will be split, e.g., $TR_7$ and $TR_8$ are crossed in time, but their merged time range is more extensive than the threshold. Thus they are regarded as two leaves, respectively. The T-Tree split will be re-divided into two parts according to the new time range, and then the T-Tree index will be rebuilt.

\textbf{ST Query.} After building the SFT index, the segments of the query set are distributed in leaf nodes of the SFT index with individual spatio-temporal range. Therefore, we start with the root node, and when accessing the leaf node, we extract its spatio-temporal range and expand this range to the infection range, and then query candidates from the database like Section 4.2.2.
\begin{algorithm}[htb]
	\caption{ Infected Rate Join Query.}
	\label{alg:irjq}
	\begin{algorithmic}[1]
		\Require The query set $\mathbf{Q}$, A threshold $\theta$.
		\Ensure  All trajectory pairs form $\mathbf{Q}$ and database whose infectivity exceeds $\theta$
		\State $sft = new~SFT(); ~//new~ index$
		\For{each $Q \in \mathbf{Q}$}
		\State $S = segmentation(Q)$;
		\State $sft.insert(S)$
		\EndFor
		\State $removed = new~Map()$,$remain = new~Map()$;
		\State $result= new~ArrayList()$;
		\State search($sft.root$, $removed$,$remain$,$result$);
		\State $result.filter(removed, remain)$; //filtering.
		\State $finalResult =result.reduceByKey(v_1+v_2)$;
		\State \Return $finalResult.filter(v \ge \theta)$;
	\end{algorithmic}
\end{algorithm}

\subsection{Infectivity Exploration}
The infected rate join query needs to return all pairs of two sets. A trajectory $Q_i$ of the query set $\mathbf{Q}$ are divided into several segments distributed in leaf nodes of the SFT index. In Section 5.2, we have extracted candidates for each leaf node. Thus, we should first calculate the segment infected rate in each node where $Q_i$ has segment located. Then, we merge all the segment infected rates between $Q_i$ and other trajectories, respectively. However, the calculate of the infected rate is time-consuming. Therefore, we use some pruning strategies, which do not need to calculate the infected rate of all candidates of $Q_i$ to exclude those trajectories that are not close to $Q_i$ in spatial and temporal.

\textbf{Pruning.} In each leaf node with $Q_i$, we first filter segments for $Q_i$ whose spatial or temporal distance exceeds $\theta_d$ or $\theta_t$. The pruning strategy Lemma~\ref{theorem2} can also be used to the infected rate join query, but we need to use two maps to record which trajectories have been cropped and which trajectories can be used for further calculations. Then, we only need to calculate the trajectories in the remaining map for $Q_i$.

\begin{algorithm}[htb]
	\caption{Extract candidates and pruning function.}
	\label{alg:search}
	\begin{algorithmic}[1]
		\Require A node of SFT index $node$; $removed$ records the pruned trajectory pair; $remain$ records remain candidate trajectories; $result$ records the value of segments IRP.
		\Ensure
		\If{$node.type = LEAF$}
		\State $candidates = st\_query(node.mbr, node.tr, \theta_d,\theta_t)$
		\For{each $s \in node.data$}

		\For{each $c \in candidates$}
		\State // $st\_filter$ return true when the spatial or temporal distance exceeds $\theta_d$ or $\theta_t$, else return false
		\If{$st\_filter(s,c)$} 	continue;
		\EndIf
		\If{$remain \neq \emptyset$ and $ !remain.has(s.id, c.id)$}
		continue;
		\EndIf

		\If{$remove.has(s.id, c.id)$}
		continue;
		\EndIf

		\State $IRP = IR(s, c) * P(s);$
		\If{$IRP < \theta - 1 + P(s)$}  // Lemma 2
		\If{$remain \neq \emptyset$}
		\State $remain.remove(s.id, c.id)$;
		\State continue;
		\EndIf
		\State $remove.put(s.id, c.id)$
		\State continue;
		\EndIf
		\State $remain.put(s.id, c.id)$;
		\State $result.add((s.id, c.id),IRP)$;
		\EndFor
		\EndFor
		\Else{}// search the children nodes.
		\State search(node.ne);search(node.se);
		\State search(node.sw);search(node.nw);
		\EndIf
	\end{algorithmic}
\end{algorithm}
\subsection{Algorithm}
The pseudocode of IRJQ is shown in Algorithm~\ref{alg:irjq} and  Algorithm~\ref{alg:search}. The query arguments are the query set $\mathbf{Q}$ and a threshold $\theta$ in Algorithm~\ref{alg:irjq}. We first build an SFT index $sft$ for segments of the query set $\mathbf{Q}$ in lines 1-5 of  Algorithm~\ref{alg:irjq}. Then, we search from the root node of $sft$ through depth-first strategy. The search processing is shown in Algorithm~\ref{alg:search}.  In lines 1-24 of Algorithm~\ref{alg:search}  calculate the $IR$ between segments and candidates in the leaf node. If the visited node is not the leaf node, then search the child nodes while it has, as shown in lines 26-27. In line 2, we call spatio-temporal query to extract candidates from the basic database. For each segment $s$ in the leaf node, we scan candidates one by one. If the spatio-temporal of any candidate $c$ is not covered by $s$, then skip $c$. line 12, we calculate $IRP =IR(s,c) * P(s)$. In line 13, we judge $IRP$ by lemma 2. If it lower than $\theta - 1 + P(s)$, then update $remain$ and $removed$ map and continue to check the next candidate, else we put the candidate in $remain$ map and record $IRP$ for $(s.id,c.id)$ in $result$. After visiting all leaf nodes, in line 9 of Algorithm~\ref{alg:irjq}, we refine the result, where candidates in $removed$ map or not in $remain$ map will be pruned. In lines 10-11, we reduce the result by key (key = $s.id$ + $c.id$) and then filter the final results out whose value are lower than $\theta$.

\section{EXPERIMENTS}
We have implemented our algorithms and conducted extensive experiments on real and synthetic spatial data sets to verify our proposed techniques.
\setlength{\tabcolsep}{0.1em}
\begin{table}
	\centering
	\caption{Trajectory Data Sets}
	\begin{tabular}{|c|c|c|c|c|c|}
		\hline
		Attributes & MPG & MPG2 & MPG3 & MPG4 & MPG5 \\ \hline
		\# Points & 3,079,428 & 16,397,140 & 19,476,568 & ... & 25,635,424 \\ \hline
		\# Traj. & 160,840 & 33,680 &  194,520 & ... & 516,200 \\ \hline
	\end{tabular}
	\label{datasets}
\end{table}
\setlength{\tabcolsep}{0.7em}
\begin{table}
	\centering
	\caption{Default Parameters}
	\begin{tabular}{|c|c|c|c|c|c|} \hline
		$\lambda$ &	$\theta$ &$\theta_d $ &$\theta_t$ &Resolution & Query Size \\ \hline
		0.5 & 0.5 & 50m & 120s & 15 & 2,300 Traj.\\
		\hline\end{tabular}
	\label{def_param}
\end{table}
\begin{table}
	\centering
	\caption{Pruning Effectiveness \hspace{\fill} (Unit: ms)}
	\begin{tabular}{|c|c|c|c|c|} \hline
		&IRQ &	IRQ\_UP&IRJQ &IRJQ\_UP \\ \hline
		MPG & \textbf{701} & 963 & \textbf{215} & 260  \\ \hline
		MPG5 & \textbf{2647} & 3182 & \textbf{318} & 394  \\
		\hline\end{tabular}
	\label{pruning_eff}
\end{table}

\begin{figure*}[htbp]
	\centering

	\subfigure[Runtime of MPG.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{lambda-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Runtime of MPG5.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{5lambda-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Accuracy of MPG.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{lambda_acc-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Accuracy of MPG5.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{5lambda_acc-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\caption{The effect of $\lambda$.}
	\label{fig:lambda}
\end{figure*}

\begin{figure*}[htbp]
	\centering

	\subfigure[Runtime of MPG.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{theta-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Runtime of MPG5.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{5theta-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Accuracy of MPG.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{theta_acc-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Accuracy of MPG5.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{5theta_acc-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\caption{The effect of $\theta$.}
	\label{fig:theta}
\end{figure*}
\textbf{Datasets.} To evaluate the efficiency and correctness, we use the GPS records from the mobile phones, \textbf{MPG}~\footnote{http://suo.im/69LJCp} and Synthetic data sets (MPG2, MPG3, MPG4, MPG5) which are generated by copying \& offsetting one to four times of \textbf{MPG} to test the scalability of our solutions. As shown in Table~\ref{datasets}, there are 160,840 trajectories in \textbf{MPG}, with an average of 19 points. The query trajectory set$^1$ is a labeled down-sampled data set of the \textbf{MPG}.

\textbf{Setting.} All of the algorithms were implemented in Java and Scala. All the experiments were conducted on a cluster of 5 nodes, with each node equipped with CentOS 7.4, 8-core CPU, 32GB RAM, and 1T disk. In our experiments, we compare the run time and veracity of IRQ and IRJQ. The run time is the average query time. The accuracy is the portion of correctly labeled trajectories in the query result, and the recall is the number of query trajectories whose query results are not empty. We analyze the effect of preference parameter $\lambda$, precision threshold $\theta$, distance threshold $\theta_d$, time threshold $\theta_t$, query data size, and source data size for IRQ and IRJQ. We also verify the effect of the different resolution of quadtree on the IRJQ algorithm. Table~\ref{def_param} gives the default parameters.

\subsection{Pruning Effectiveness}
We first analyze the pruning effectiveness of our algorithms using the default parameters. The experimental results are shown in Table~\ref{pruning_eff}.  We can see that the IRQ and IRJQ have better performance than the unpruned algorithms IRQ\_UP and IRJQ\_UP. The join algorithms (e.g., IRJQ and IRJQ\_UP) outperform simple query algorithms (IRJQ and IRJQ\_UP) by almost an order of magnitude on MPG and MPG5. After pruning, IRQ saves 27\% and 17\%, and IRJQ improves 27\% and 29\% query time on the sets of MPG, respectively.
\begin{figure*}[htbp]
	\centering

	\subfigure[Runtime of MPG.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{theta_d-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Runtime of MPG5.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{5theta_d-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Accuracy of MPG.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{theta_d_acc-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Accuracy of MPG5.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{5theta_d_acc-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\caption{The effect of $\theta_d$.}
	\label{fig:thetaD}
\end{figure*}
\begin{figure*}[htbp]
	\centering

	\subfigure[Runtime of MPG.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{theta_t-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Runtime of MPG5.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{5theta_t-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Accuracy of MPG.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{theta_t_acc-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\subfigure[Accuracy of MPG5.]{
		\begin{minipage}[t]{0.25\linewidth}
			\centering
			\includegraphics[width=1\textwidth]{5theta_t_acc-eps-converted-to.pdf}

		\end{minipage}%
	}%
	\caption{The effect of $\theta_t$.}
	\label{fig:thetaT}
\end{figure*}
\subsection{Effect of Preference Parameter $\lambda$}
Figure~\ref{fig:lambda} shows the effect of the preference parameter $\lambda$ on efficiency and accuracy. On the dataset MPG, the runtime of IRJQ and IRJQ\_UP is around 200ms and not affected by the varying of $\lambda$. IRQ is more effective than IRQ\_UP, and saves about 200ms compare to IRQ\_UP. On the dataset MPG5, the runtime of IRJQ is around 400ms and much faster than IRQ. Then we discuss the recall and accuracy under the influence of $\lambda$. We only analyze the recall and accuracy of IRQ because IRQ and IRJQ have the same accuracy. The accuracy of IRQ varies with $\lambda$, which is lower at $\lambda=0.0$ and $\lambda=1.0$ on dataset MPG because only the side affected by spatial or temporal is considered. As the number of candidates from the spatio-temporal query on the database does not change but the final result has a little difference. Thus the recall rate remains at around 0.7 on the dataset MPG. However, on the dataset MPG5, the recall increases with $\lambda$, because more trajectories are queried in the same spatial region, which increases the number of the final results.
\subsection{Effect of Threshold $\theta$}
Figure~\ref{fig:theta} shows the effect of precision threshold $\theta$. On the datasets of MPG and MPG5, the runtime of pruned algorithms (IRQ and IRJQ) decreases with an increasing threshold $\theta$ and unpruned algorithms (IRQ\_UP and IRJQ\_UP) keep the approximately same value, which verify the efficiency of our pruning strategies. The recall decreases with the threshold because the larger $\theta$, the fewer trajectories are satisfied. Meanwhile, the accuracy rate still maintains at a relatively high value.

\subsection{Effect of Distance Threshold $\theta_d$}
Figure~\ref{fig:thetaD} shows the effect of distance threshold $\theta_d$. The running time increases as  $\theta_d$ increases because both the candidates and the computation are increased. IRJQ's runtime only increases slightly, but the increase in IRQ is pronounced. The spatio-temporal ranges infected by every location are enlarged with the increase of $\theta_d$. Thus more candidates are selected in our algorithms, which increases the recall.

\subsection{Effect of Time Threshold $\theta_t$}
Figure~\ref{fig:thetaT} shows the effect of distance threshold $\theta_t$. The running time slightly increases as $\theta_t$ increases. The temporal range expands as $\theta_t$ increases, but the data covered by each location does not increase significantly. Therefore, the running time maintains at a relatively fixed value, and the recall increases with a small numerical.

\subsection{Effect of Query Size}
Figure~\ref{fig:query_size} shows the effect of query size. The query trajectories from 460 to 2300, represented as 20\% to 100\%, respectively. Note that unlike other times, the runtime here is the total time to complete the calculation of all trajectories. We can see that as the amount of data for query trajectories doubles, the cumulative runtime of IRQ and IRQ\_UP almost doubles. Consequently, the total runtime of IRJQ and IRJQ\_UP is still below 500 seconds, because we build an efficient index on trajectories that effectively reduce the number of time to access the database and greatly avoid data redundancy.
\begin{figure}
	\centering
	\includegraphics[width=2.0in]{resolution-eps-converted-to.pdf}
	\caption{The effect of resolution.}
	\label{fig:resolution}
\end{figure}

\subsection{Effect of Resolution}
In Figure~\ref{fig:resolution}, we analyze the impact of the resolution of the SFT index. We see that the resolution equals 15 performs better than other resolutions. When the resolution is equal to 14, the spatial-temporal range distribution in the leaf nodes is more dispersed. Thus the spatial and temporal range of a single query will be relatively large, which will cause more data redundancy and some unnecessary data. When the resolution is greater than 15, although the distribution of the spatio-temporal range in the leaf node is very concentrated, it also means that it needs to query the database more times, which increases the I/O overhead.

\subsection{Scalability}
Figure~\ref{fig:data_size} shows the effect of basic dataset size. We generate five datasets with equal increments in turn, as shown in Table~\ref{datasets}. We represent MGP as 20\%, MPG2 as 40\%, MPG3 as 60\%, MPG2 as 80\%, and MPG5 as 100\%, respectively. We see that the increase of the data set has a great impact on the IRQ, which owns to spatio-temporal query gets several times of the candidate set, which increases the calculations. Although IRJQ's query time has also increased, it has not increased exponentially. It is attributed to the SFT index, which makes the close spatio-temporal ranges query only once on the database.

\begin{figure*}[htbp]
	\centering

	\begin{minipage}[b]{0.48\textwidth}
		\centering

		\subfigure[Runtime of MPG.]{
			\begin{minipage}[t]{0.5\linewidth}
				\centering
				\includegraphics[width=1\textwidth]{query_size-eps-converted-to.pdf}

			\end{minipage}%
		}%
		\subfigure[Accuracy of MPG5.]{
			\begin{minipage}[t]{0.5\linewidth}
				\centering
				\includegraphics[width=1\textwidth]{query_size_acc-eps-converted-to.pdf}

			\end{minipage}%
		}%
		\caption{The effect of query size.}
		\label{fig:query_size}
	\end{minipage}
	\begin{minipage}[b]{0.48\textwidth}
		\centering

		\subfigure[Runtime of MPG.]{
			\begin{minipage}[t]{0.5\linewidth}
				\centering
				\includegraphics[width=1\textwidth]{data_size-eps-converted-to.pdf}

			\end{minipage}%
		}%
		\subfigure[Accuracy of MPG5.]{
			\begin{minipage}[t]{0.5\linewidth}
				\centering
				\includegraphics[width=1\textwidth]{data_size_acc-eps-converted-to.pdf}

			\end{minipage}%
		}%
		\caption{The effect of data size.}
		\label{fig:data_size}
	\end{minipage}
\end{figure*}

\section{Related Work}
\subsection{Trajectory Correlation Metrics}
Many trajectory related metrics have been proposed~\cite{toohey2015trajectory,assent2009anticipatory,deng2014massjoin,yu2016string,shang2017trajectory,ding2008efficient,lin2005shapes,zheng2013towards,rong2017fast,wang2013effectiveness,magdy2015review}, which can be roughly classified as two types: (1) The point-based metrics, such as the Euclidean distance (ED)~\cite{faloutsos1994fast}, Dynamic Time Warping (DTW)~\cite{assent2009anticipatory} and  Fréchet~\cite{toohey2015trajectory}; and (2) The segment based metrics, such as the metric in ~\cite{xie2017distributed} and Longest Common Subsequence (LCSS)~\cite{zhu2017trajectory}. In general, the above methods all treat temporal attributes as simple time series. The temporal attribute is one of the important attributes. Combining the spatial and temporal attributes can be used as a judgment criterion for the correlation measure of trajectory.
\\ \textbf{Point-Based Metrics.} Correlation measurement methods based on trajectory points can be further divided into global matching methods and local matching methods. Euclidean distance~\cite{faloutsos1994fast} is mainly by calculating the Euclidean distance between corresponding points between trajectories, and then accumulating the sum as the final metric value. Euclidean distance is the simplest. It only needs to be summed up, and the time complexity is O(N). However, it also has serious limitations: (1) The sampling rate and trajectory points must be consistent; (2) The principle of monotonic continuity must be met, and local time distortion is not supported; (3) Sensitive to noise. The Fréchet distance~\cite{toohey2015trajectory} measure was proposed by Fréchet et al. It is usually described intuitively: the dog rope distance when a person walks a dog. DTW~\cite{assent2009anticipatory} locally stretches or scales the trajectories, so that trajectories of different sampling rates and different lengths can be compared. The DTW distance is the cumulative sum of the distances between all the optimal matching trajectory points.
\\ \textbf{Segment-Based Metrics.} Based on the trajectory segment similarity measures, by segmenting the trajectories and comparing the similarity of each segment separately, the time complexity is greatly reduced. However, the local information of the trajectory is not fully considered. Thus the accuracy is relatively low. Longest Common Subsequence (LCSS) mainly considers similar parts between trajectories as a measure of trajectory correlation, so it can skip some trajectory points due to matching distance exceeding the threshold, which makes it robust to noise. In~\cite{xie2017distributed}, the trajectory is divided into several segments, and then the divide-and-conquer strategy is used to calculate the discrete segment Hausdorff or discrete segment Fréchet distance. However, the distance measures they use do not adequately describe spatial and temporal proximity.

Our metric takes into account the spatial and temporal correlation of each location at the same time and uses a divide-and-conquer algorithm for the weighted trajectory segments. It is not only appropriately describes the spatio-temporal closed trajectories, but also greatly reduces the complexity.
\subsection{Trajectory Correlation Search.}
Trajectory correlation searches are widely studied~\cite{shang2017trajectory,shang2018parallel,ta2017signature,xie2017distributed,rong2017fast}. The procedure typically involves a definition step and a query processing step. First, a metric is defined to measure the spatial and temporal correlations between two trajectories. Second, an efficient strategy is developed to search spatiotemporally close trajectories for a query trajectory. For example, the BCT~\cite{zheng2017searching} algorithm proposed by Zheng uses Euclidean distance for trajectory search. Rong et al.~\cite{rong2017fast} proposed a similar measure for trajectory segments and used a distributed framework. This framework first divides trajectories into several segments and then groups nearby segments to find common trajectories, which is helpful to reduce I/O consumption. Shang et al.~\cite{shang2017trajectory} presented a two-phase divide-and-conquer trajectory similarity join framework. It first finds similar trajectories for each trajectory. Then it merges the results to the final result. Like~\cite{rong2017fast}, we proposed a clearer metric, and then grouped segments that are closed spatially and temporally with an efficient Spatial First Time (SFT) index.
Furthermore, in order to speed up query time, many frameworks have designed effective pruning strategies, which reduce the search space. Many studies~\cite{shang2017trajectory,shang2018parallel} have analyzed the low bounds of their metrics to reduce search nodes. In our algorithm, we propose four effective prune strategies to avoid unnecessary calculations.

\section{CONCLUSION}
In this paper, we studied a novel trajectory infection rate based on spatio-temporal correlation, the goal of which is to detect suspected infected crowds of COVID-19 and also targets many applications such as close contacts detection, companion mining, high-risk groups prediction and trajectory similarity recommendation. We proposed a new trajectory metric that accommodates misaligned trajectory points. We first broke down the longer and larger trajectories into several short and suitable segments and used an active spatio-temporal index (XZ2T) to manage a large number of segments in the NoSQL database, which reduce memory consumption and guarantee the scalability by avoid loading all trajectories into memory. We then developed efficient algorithms for segments with infected weight. We explored several pruning strategies for our proposed algorithms to avoid many calculations. For batch query, we designed an SFT index that groups similar segments only once to access the database to reduce I/O communication and data redundancy. We then devised experimental studies on real and synthetic datasets to verify the effectiveness and efficiency of our algorithms.

Many exciting directions for future research exist. First, it is significant to extend our algorithms for supporting top-$k$ close contacts query without a threshold $\theta$. Second, it is vital  to use some sample data to determine the parameters to be set in the algorithm.

\section{ACKNOWLEDGEMENT}
This work was supported by the National Key R\&D Program of China (2019YFB2101800).

	\balance

	\bibliographystyle{abbrv}


Virtually turning robotic manipulators into worn devices: opening new horizons for wearable assistive robotics

Abstract: Robotic sensorimotor extensions (supernumerary limbs, prosthesis, handheld tools) are worn devices used to interact with the nearby environment, whether to assist the capabilities of impaired users or to enhance the dexterity of industrial operators. Despite numerous mechanical achievements, embedding these robotics devices remains critical due to their weight and discomfort. To emancipate from these mechanical constraints, we propose a new hybrid system using a virtually worn robotic arm in augmented-reality, and a real robotic manipulator servoed on such virtual representation. We aim at bringing an illusion of wearing a robotic system while its weight is fully deported, thinking that this approach could open new horizons for the study of wearable robotics without any intrinsic impairment of the human movement abilities.

\section{Introduction}

Among wearable robotic devices, robotic sensorimotor extensions are serial augmentations of the human body aiming to increase the user's motor abilities. These devices include but does not limit to prosthesis \cite{ribeiro_analysis_2019}, supernumerary limbs \cite{yang_supernumerary_2021} or handheld robotic tools \cite{gregg-smith_design_2015}. Despite mechanical advances over the last decades, embedding these robotic devices on the human body remains critical as their weight \cite{handy2018satisfaction, prattichizzo_human_2021}, cumbersomeness and limited functionalities \cite{cordella_literature_2016} restrain their comfort and acceptance \cite{martinez2021wearable}, leading many potential users to prefer simple mechanical devices \cite{millstein_prosthetic_1986} or the use of their natural limbs \cite{guggenheim_laying_2020}.

In spite of these disadvantages, those wearable devices remain unique: as indicated by their name, they are worn by their user, meaning that, in addition to being transportable, the robot end-effector's transform relative to the world frame $T_{E_R \to W}$ can be written as a composition of:

\begin{itemize}
    \item the robotic control relative to its human attachment base $T_{E_R \to W}$;
    \item and the movement of the human body relative to the world $T_{E_H \to W}$.
\end{itemize}
We can rephrase it as: the robot and the human kinematic chains are serially connected. It writes:

\begin{equation}
    T_{E_R \to W} = T_{E_R \to E_H}  T_{E_H \to W} ~,
\end{equation}

where $T_{A\to B}$ the transform from frame $A$ to frame $B$, $E_R$ the robotic end-effector frame, $E_H$ the human effector frame to which the robot base is attached, and $W$ the world frame.

If the human plus the robot system form a kinematically redundant chain(i.e. when the dimension of joint space is greater than the dimension of end-effector space \cite{conkur_clarifying_1997}), it also means that the task completion is shared between the human motion $T_{E_H \to W}$ and the robotic motion $T_{E_R \to E_H}$ (Fig.~\ref{fig:compensations}):
\begin{itemize}
    \item $T_{E_R \to E_H}$ is usually governed by an auxiliary signal coming from a joystick-like input \cite{zahraee_toward_2010}, a sensor system measuring electrophysiological activities (such as muscles activities through EMG \cite{li_quantifying_2010} or cerebral activities through EEG \cite{vilela_applications_2020}), or an automatic task-oriented behaviour \cite{luo_modeling_2021};
    \item $T_{E_H \to W}$ depends on the attachment point of the worn device, whether the arm or forearm for an upper-limb prosthesis, the trunk or hips for a supernumerary limb, or the hand for an handheld tool, and of the movements of the user.
\end{itemize}

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.48\textwidth]{compensations.png}
    \vspace{2mm}
    \caption{A robotic sensorimotor extension in orange (prosthesis or supernumerary robotic limb) with effector $E_R$ (orange cross) is attached on an user at $E_H$ (blue cross), to perform a reaching task.}
    \label{fig:compensations}
\end{figure}

Yet, this kinematic redundancy is frequently underused, leading to undesirable postures for the user:
\begin{itemize}
    \item in prosthesis control, $T_{E_R \to E_H}$ is piloted by myoelectric activities of the stump and is often slow and non-intuitive leading to important body compensations \cite{hussaini_categorization_2017} by $T_{E_H \to W}$ over $T_{E_R \to E_H}$;

    \item oppositely, in supernumerary robotic limbs, the weight of the robotic device limits the dexterity \cite{shimobayashi_independent_2021} of $T_{E_H \to W}$ (by creating imbalance and fatigue) and imposes the use of $T_{E_R \to E_H}$ over $T_{E_H \to W}$.

\end{itemize}

Currently, the discomfort and limited performances of these devices and of their control still limit the potential of their usage, as the weight of the device is fully carried by the user.

\section{Virtually turning robotic manipulators into worn devices}
To solve the previous issue, we propose a new set-up which fully deports the weight of the robotic device to a nearby environment, while still keeping a serial connection between the user and the device: we create an augmented-reality representation of an arm, attached to the body frame $E_H$, ending with a virtual end-effector frame $E_{AR}$, while a robotic manipulator set nearby with an end-effector frame $E_R$ is servoed to follow $E_{AR}$ (see Fig.~\ref{fig:setups} and video Sec.~1):

\begin{equation}
    \begin{cases}
      T_{E_{AR} \to W} = T_{E_{AR} \to E_H}  T_{E_H \to W}\\
      T_{E_R \to W}(t) = f(T_{E_{AR} \to W}(t))
    \end{cases}
\end{equation}
with $f$ the robot dynamics and control, causing delay when visually servoing the position of robotic end-effector.

The fundamental serial connection of worn robotic devices is now purely virtual, allowing to deport the weight, whereas the real robotic manipulator, physically detached from the human body, performs the desired task with a certain delay (imposed by its dynamics or control loop limitations). By doing so, we aim at virtually creating a sensation of physical embodiment of the deported manipulator.

As said, the robotic manipulator is often slower than the human motions $T_{E_H \to W}$, creating a delay between the virtual frame $E_{AR}$ motions and the robotic effector $E_R$ motions. But, by representing the virtual arm in augmented-reality, we partially solve these dynamic issues as we delete the perception delay for the user, tending to more stable motions. This representation is however not necessary for the system to actually work and, with training, can be removed (see video Sec.~4).

Finally, $T_{E_H \to W}$ can be generated from any body part $E_H$, such as the head, trunk or arm (see video Sec.~2) depending on the user's preference and the applications, while $T_{E_{AR} \to E_H}$ can still be controlled by an auxiliary input, like a joystick (see video Sec.~3).

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.9\linewidth]{hybrid_setup2.png}
    \vspace{5mm}
    \caption{The user (grey) wears a virtual robotic arm in blue at attach point $E_H$ (blue circle on the user), with virtual end-effector $E_{AR}$ (blue circle at the end of the robot). The virtual arm made visible thanks to the AR headset in blue on the users' head. Then, the virtual end-effector servoes the effector of a deported robotic arm's (orange circle) set on a table, with a delay depending on the robot dynamics and control (symbolized by the arrow).}
    \label{fig:setups}
\end{figure}

To sum up, the proposed set-up allows to:
\begin{itemize}
    \item shift the weight and embody any existing robotic system as a virtually worn device, even heavyweight devices, offering the possibility to reach higher robotic performances and higher comfort;
    \item create mechanically unfeasible limbs, such as an arm attach to the head for disability assistive applications, or a extended third arm with wide workspace for industrial applications;
    \item instantly attach and detach any robot device to the user's body, switch systems and shift the attach point from one to another for different use-case scenarios, unlike current devices which usually require important preparation times.
\end{itemize}

\section{Practical implementation}
\subsection{Controlled Environment}
For research purposes, in controlled and room-sized environments, the easiest and most flexible implementation is to use an optical motion capture system such as the Optitrack system, and an augmented-reality headset like the Microsoft Hololens V2 for display, as seen Fig.~\ref{fig:optitrack set_up}.

The motion capture system allows to capture the body transform $T_{E_H \to W}$ while easily switching from one body attach point $E_H$ to another. By putting another marker on the robot end-effector $E_R$ it also allows to perform visual position servoing. The augmented-reality headset is then only used for display and is actually not necessary to control the robot, rather serving as a visual feedback provider which increases the easiness of use. Finally, another auxiliary control input, such as a joystick, button or electrophysiological system can be added, whether to reconfigure and control the virtual linkage $T_{E_{AR} \to E_H}$ or to control a robotic gripper.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.75\linewidth]{optitrack_setup.png}
    \vspace{5mm}
    \caption{The user frame $E_H$ (blue cube) and robot end-effector frame $E_R$ are tracked using an Optitrack system. The Microsoft Hololens V2 headset displays the virtual arm (blue), its virtual effector $E_{AR}$ (blue octagon) and current robot arm effector configuration (orange cube), represented here. The robot effector is servoed to follow the virtual effector.}
    \label{fig:optitrack set_up}
\end{figure}

\subsection{Fully wearable set-up for applied scenarios}
Optical motion capture systems cannot be worn. However, the Microsoft Hololens V2 is equipped with a visual 3D slam function which allows to detect at all time the position of the user's head and can be used to calibrate the robot with the headset. By using Inertial Measurement Units on the trunk and the arm, and a simple kinematic model of the user, one can then reconstruct the upper-body joint positions and the transform $T_{E_H \to W}$ (see video Sec.~5). Auxiliary inputs measurement can also be implemented in the headset, using its gaze tracker to control the gripper for example.

This measurement system is fully wearable and transportable (the headset being autonomous) and cheaper than the optical motion capture system, but it requires more development and time to be fully operational, as well as previous knowledge regarding the user's body part to which the virtual robotic arm should be attached.

However, as it can be seen on our setup, a critical aspect of worn robotics which was not resolved here is the transportability of the system, as the robotic manipulator is set on a table. To solve the latter, we could consider to embed the manipulator on a mobile base, wheeled or legged, which could follow the user's motions in a larger environment, or on a wheelchair for assistive applications, and use an inverse kinematic model of both the base and the manipulator to servo the system.

\section{Conclusion}
We proposed a novel approach for robotic extensions, in which the worn device is fully virtual but distantly controls a real robotic manipulator. This allows to virtually embody any high-performance robotic device without weight or embodiment issues, and explore applications that where previously unfeasible — an arm attached to the head or a very long arm fixed to the trunk — without constraining the user's body with heavy and discomfortable wearable devices. We think that this approach opens new horizons for the study or wearable robotics by fully emancipating from the mechanical constraints of the systems, allowing to explore both fundamental and applied aspects of worn devices without intrinsic impairment of the human motions.

\hypersetup{urlcolor = black}
\printbibliography

\end{document}


Learning Joint Acoustic-Phonetic Word Embeddings

Abstract: Most speech recognition tasks pertain to mapping words across two modalities: acoustic and orthographic. In this work, we suggest learning encoders that map variable-length, acoustic or phonetic, sequences that represent words into fixed-dimensional vectors in a shared latent space; such that the distance between two word vectors represents how closely the two words sound. Instead of directly learning the distances between word vectors, we employ weak supervision and model a binary classification task to predict whether two inputs, one of each modality, represent the same word given a distance threshold. We explore various deep-learning models, bimodal contrastive losses, and techniques for mining hard negative examples such as the semi-supervised technique of self-labeling. Our best model achieves an $F_1$ score of 0.95 for the binary classification task.

\section{Introduction}
The proliferation of voice-first applications and devices generated unprecedented demand to improve a plethora of speech recognition tasks. In this work, we propose a building block for speech recognition tasks like automatic speech recognition (ASR), keyword spotting, and query-by-example search. The outcome is an embedding model that maps variable-length --- acoustic or phonetic --- sequences that represent words into fixed-dimensional vectors in a shared latent space, which connects audio and phonetic modalities together, and encompasses a distributed representation of words, such that the distance between two word vectors represents how closely the two words sound: The more similar words sound, in either modality, the closer they end up in the shared vector space. Deep learning is befitting to learn pairwise relationships and joint embeddings, which have become the cornerstone of many machine learning applications~\cite{bromley1994signature, schroff2015facenet, harwath2017learning, li2018survey}. One of the applications we can reformulate is using the learned joint embeddings is ASR hypotheses reranking to reduce word error rate~\cite{bengio2014word, ma2017asr}.

The approach we took to learn distances between word embeddings in the vector space can be described as a weakly supervised task: Instead of training using ground-truth pairwise distances, we produced ground-truth data of word pairs labeled as either similar-sounding (distance of 0) or different-sounding (distance of 1) --- making it a binary classification problem. At inference time, the model predicts  real-valued distances that can be turned into labels given a distance threshold.

Despite the emergence of many successful embedding models, learning them is relatively poorly understood~\cite{histogramloss}. We experimented with various techniques to mine hard examples for training. Training using words picked at random from a corpus to be dissimilar to one another impedes learning; the model needs to be trained using hard examples of words that sound slightly different as examples of distant words~\cite{schroff2015facenet, wu2017sampling}. Another facet of weak supervision is creating ground-truth data using heuristics applied to the results of ASR systems and unreliable, non-expert transcribers.

\section{Related Work}
Thanks to recent natural language processing (NLP) research~\cite{mikolov2013word2vec, pennington2014glove, abdulkader2016deeptext}, the use of word embeddings to represent the semantic relationships between words has been prevailing. Similarly, in the field of speech recognition, the use of acoustic word embeddings has improved many tasks like keyword spotting~\cite{chen2015query}, ASR systems~\cite{bengio2014word}, query-by-example search~\cite{levin2013fixed, levin2015segmental, settle2017query}, and other word discrimination tasks~\cite{settle2016discriminative, kamper2016deep}; it also enabled attempts at unsupervised speech processing~\cite{kamper2017segmental, chung2016audio} and representing semantic relationships between words~\cite{chung2017learning, chung2018speech2vec}. The aforementioned speech recognition research learned models from the acoustic representation of words --- a single view, which requires side information to map that view into text. Infusing cross-modal (acoustic and orthographic) representations of words into the embedding model showed improvements in a plethora of applications; for example, \cite{he2016multi} jointly learned to embed acoustic sequences and their respective character sequences into a common vector space using deep bidirectional long short-term memory (LSTM) embedding models and multi-view contrastive losses. In~\cite{chung2018unsupervised}, the model separately learns acoustic and orthographic embedding spaces, and then attempts to align them using adversarial training; it employs a bidirectional LSTM encoder and a unidirectional LSTM decoder to perform the cross-modal alignment in an unsupervised fashion; the performance of such model is comparable to its supervised counterpart.

One of the limitations of mapping acoustic representations to their corresponding character sequences is confusing homographs (words that share the same spelling but not the same pronunciation) --- they may miss the subtle phonetic variations within words and/or across dialects. Multiple approaches attempted to mitigate such limitation. For example, \cite{lim2018learning} models phonetic information by training using both word- and frame- level criteria; however, it doesn't learn from phonetic labels so the learned embeddings cannot precisely represent the phonetic spelling of words. In~\cite{synnaeve2014phonetics}, the objective is to learn word embeddings from audio segments and use side information (phonetic labels) to train an acoustic model, which can be used for ASR systems --- a desideratum similar to ours; it trains the model using a Siamese neural network and explores multiple loss functions. In~\cite{chung2016audio}, the objective is to find vector representations that precisely describe represent the sequential phonetic structures of audio segments; in addition, \cite{he2016multi} considers directly training using phonetic-spelling supervision as a future direction. It's worth mentioning that we  discovered the latter after embarking on this endeavor, which validated our decision of using phonetic spelling instead of orthography.

In~\cite{bengio2014word, settle2016discriminative, he2016multi}, the models are trained using matched (similar-sounding) and mismatched (different-sounding) pairs of words; the mismatched words are drawn randomly, which makes minimizing losses like triplet and contrastive loss a challenge~\cite{wu2017sampling}. In~\cite{chung2018unsupervised}, adversarial training is used to align the acoustic and orthographic vector spaces. We use three different techniques, which we detail below, to mine for hard negative examples mining.

\section{Dataset and Representations}
The raw data consist of 25k short, single-channel recordings and their respective transcripts. The recordings were captured at a sample rate of 16kHz and encoded using pulse-code modulation with 16-bit precision. Transcripts do not include word alignments and may include errors. The dataset is proprietary and growing. To obtain the phonetic spelling of words, we used the LibriSpeech lexicon~\cite{librispeech}; it contains over 200k words and their pronunciations; out-of-vocabulary (OOV) words are excluded from the acoustic dataset as they lack ground-truth phonetic labels.

In order to obtain word alignments, we force-aligned each transcript in the raw dataset with recognition hypotheses generated by an ensemble of ASR systems. When the forced alignment failed due to an insertion or a deletion error, we considered the affected words too noisy to include. When the human-selected transcript for a word aligned successfully with an ASR hypothesis, we labeled the pair (the audio segment and its phonetic label) as similar-sounding (distance of 0). Using the set of positive pairs, we mined unique pairs of ASR hypotheses that substituted the human-selected transcript for the same audio segment and labeled them as hard negative examples (distance of 1). The restriction of starting from positive pairs reduced the overall number of examples; however, it drastically increased the quality of the labels as we minimized the number of false negatives.

We employed another technique to mine more of the hard negative examples: We started from the hard negative examples we had created above and computed their phonetic-edit distance (between the human-selected transcript and the ASR's hypothesis for the same audio segment); then, we picked ones that scored below a maximum distance threshold (0.7 was satisfactory) and grouped them by their respective human-selected transcripts; finally, we synthesized additional unique hard negative examples for ones that share the same human-selected transcript. The result of this process increased the number of negative examples from 73,845 to 439,679; it also helped balance the dataset, which included 393,623 positive examples. We removed stop words from the dataset as they are not interesting for downstream tasks; words shorter than 0.2 seconds long were also removed as manual inspection deemed them mostly mislabeled. The processed dataset consisted of 654,224 examples that were split, with class-label stratification, into train/dev/test datasets of sizes 621,905/12,692/19,627 respectively.The three sets contained 352,769/11,775/17,632 unique audio segments, respectively.

\begin{figure}[ht]
  \centering
  \begin{subfigure}{.5\textwidth}
    \centering
    \includegraphics[width=0.8\linewidth]{word_durations}
  \end{subfigure}%
  \begin{subfigure}{.5\textwidth}
    \centering
    \includegraphics[width=0.8\linewidth]{word_counts}
  \end{subfigure}
  \caption{Distributions of word durations (left) and frequencies (right) in the dataset.}
\end{figure}

The third technique we used to mine even more hard negative examples is a semi-supervised learning technique that's attributed to Scudder for his work circa 1965: self-labeling~\cite{scudder1965probability}. Every $r$ epochs during training, we allow the dataset generator to augment the training dataset with newly minted hard negative examples. We encode the phonetic spelling of the training set's lexicon and build a $k$-dimensional tree for the resultant phonetic embeddings. Then we sample, at random, from the unique audio segments in the training set such that the sample's size is proportional to the number of epochs. Constraining the contributions of the less accurate models at early epochs reduces the chances of polluting the training set with easy-to-predict negative examples and hogging the memory. For each audio segment in the sample, we compute an embedding and find the closest phonetic neighbors within a maximum distance that reflects similarity (predicted a positive label). For each neighbor, we look up the true label of the word pair (audio segment and its phonetic label); if the model made a new mistake (the true label was negative and the pair is unique), we add the example to the training set as a hard negative. We observed a healthy growth in the training set's size thanks to self-labeling; it reached a 41\% relative growth (to a total of 876,771 training examples) at one point.

We represented audio as mel-spectograms using standard parameters given a sample rate of 16KHz: 400 samples (25ms) for the length of the Fast Fourier Transform (FFT) window; a hop of 200 samples between successive frames, and 64 mel bands. The input audio signal was centered and padded with zeros (silence) to fit in a window of 2 seconds to fix the size of the model's input; the result is a tensor of size $64\times161\times1$ (single channel). Audio examples were normalized individually using cepstral mean and variance normalization~\cite{strand2004cepstral} and then normalized across examples to have zero mean and unit variance for each mel band.

We represented phonetic spelling of words as a sequence of one-hot encoded vectors; the longest word in the LibriSpeech lexicon has 20 phones and there are 69 unique phones in the ARPAbet flavor used by the lexicon. We indicate an empty phone using a sentinel value (making the length of the alphabet 70). The result is a matrix of size $70\times20$.

\section{Methods}
In this work, we follow an approach similar to~\cite{he2016multi} in the sense that we model the task at hand as a weakly supervised, bimodal binary classification task; and along the way, we learn acoustic and phonetic encoders that map words into a shared vector space such that the distance between two word vectors represents how closely the two words sound. We train a Siamese neural network~\cite{bromley1994signature} that feeds forward the acoustic and phonetic representations of a pair of words through a series of transformations to encode the two inputs into $\ell^2$-normalized vectors (embeddings), then outputs the distance between the two embeddings. Our objective is to minimize the contrastive loss~\cite{hadsell2006dimensionality}, which allows us to learn encoders that map similar inputs to nearby points and dissimilar inputs to distant points in the shared vector space. To describe the objective formally, let $(x_a,\, x_p)$ be the input word pair (acoustic and phonetic representations, respectively) and $y$ be its true binary label such that $y = 0$ when the two representations, acoustic and phonetic, are of the same word; otherwise, $y = 1$. The model learns two functions, $f(x_a)$ and $g(x_p)$, that map the inputs into $\ell^2$-normalized embeddings; for each prediction, we compute the distance $\mathcal{D}(f(x_a),\, g(x_p))$, or $\mathcal{D}$ for brevity, between the outputs of both functions using a distance function such as the Euclidean or cosine distance. Since the embeddings are $\ell^2$-normalized, we think of the two distance functions as mostly interchangeable. Given $N$ training examples, we minimize the following function:
\[
\mathcal{L} = \frac{1}{N} \sum_i^N \big[ \, (1 - y^{(i)}) \, (\mathcal{D}^{(i)})^2 + y^{(i)} \, \max(0, \, m - \mathcal{D}^{(i)})^2 \, \big]
\]

where $m > 0$ is a margin parameter that controls when dissimilar pairs contribute to the loss function: only when $\mathcal{D} < m$. Unless otherwise specified, we used $m = 1$ for the experiments below.

\section{Experiments and Results}
Evaluating embeddings depends heavily on the downstream task; in this work, we picked $F_1$ score (the harmonic mean of precision and recall) as the metric of choice for the bimodal binary classification task. The binary labels at test time were calculated at a distance threshold of $0.5$, which approximates the observed break-even point when $m = 1$ in our experiments. In order to compute the break-even point for the test set, we generated all unique pairs of acoustic and phonetic inputs in the set --- approximately, 192.6 million word pairs. The $F_1$ score of our best-performing model is $0.95$.

\begin{figure}[ht]
  \centering
  \begin{subfigure}{.5\textwidth}
    \centering
    \includegraphics[width=0.8\linewidth]{self_labeling}
    \caption{Development set loss before self-labeling to augment training data (red line) and after (gray line) — learning use to stagnate at much earlier epochs (x-axis).}
  \end{subfigure}%
  \begin{subfigure}{.5\textwidth}
    \centering
    \includegraphics[width=0.8\linewidth]{cm}
    \caption{Confusion matrix of our best-performing model.}
  \end{subfigure}
\end{figure}

Projecting a sample of embeddings derived from the test set using a t-SNE~\cite{maaten2008visualizing} model results in a reasonable clustering of words. More interestingly, the model found phonetic analogies such as "cat" to "cool" is like "pat" to "pool", etc. We also inspected a sample of 50 classification mistakes the model made. A few patterns emerged: the model failed to predict a distance that reflects similarity between the acoustic and phonetic embeddings when the audio was too noisy, too faint (far field), or the speech was accented. Cross talk and reverb were also problematic but not as common. Audio preprocessing to clean up the signal may be helpful in such cases. Also, training using n-grams and acquiring more data from speakers with accents and different acoustic environment may boost performance as we expect distributions of words to be non-IID (independent and identically distributed).

We experimented with and manually tuned various hyperparameter choices for input representations, architecture, contrastive losses, batch size, number of hidden units, number of epochs, embedding size, dimensions of filters in convolutional neural networks (CNN), single vs. bidirectional LSTM cells, Euclidean- vs. cosine- based distance functions, etc. The sparse representation of phonetic input led us to believe that the architecture for its encoder should be different from that of the acoustic one; however, empirical evidence suggests that mirroring the same architecture for both encoders, with the exception of a dropout for the acoustic input layer, yields better results. Table~\ref{table:experiments} summarizes notable experiments and their results.

\renewcommand{\arraystretch}{1.2}
\begin{table}[ht]
\caption{Summary of notable experiments and their results for training and testing, respectively.}
\label{table:experiments}
\centering
\begin{tabular}{c l c}
 \toprule
 \textbf{\#} & \textbf{Notable Experiment Details} & \textbf{$F_1$ Scores} \\ [0.5ex]
 \toprule
 1  & CNN ($3\times3\times32$); dense layer; 256-D embedding; batch size = $32$ & $0.97, 0.91$ \\ \hline
 2  & CNN ($3\times3\times32$) -> ($3\times3\times64$); dense layer; $256$-D embedding;\\
    & batch size = $32$ & $0.99, 0.93$ \\ \hline
 3  & Same as \#2 but for a dropout with a rate of $0.5$ after the first hidden layer & $0.99, 0.94$ \\ \hline
 4  & Same as \#3 but with another dropout of $0.5$ after the second hidden layer & $0.95, 0.91$ \\ \hline
 5  & Same as \#3 but with margin = the phonetic-edit distance for the pair & $0.96, 0.91$ \\ \hline
 6  & Same as \#3 but with incoming weights constrained to a maximum norm of $3$ & $0.99, 0.94$ \\ \hline
 7  & 2 unidirectional LSTM layers with $128$ hidden units; dense layer; \\
    & $256$-D embedding; batch size = $32$; $24$ epochs (in $99$ hours) & $0.91, 0.87$ \\ \hline
 8  & 2 Bidirectional LSTM layers with $512$ hidden units and a dropout of $0.4$\\
    & in between; a dropout of $0.2$ for the acoustic input; $512$-D embedding;\\
    & $28$ epochs (in $47$ hours) & $0.95, 0.91$ \\ \hline
 9  & CNN with 2 blocks [($3\times3\times64$) -> ($2\times2$) max pooling]; two dense\\
    & layers with 512 hidden units and a dropout of 0.4 in between; a dropout\\
    & of 0.2 for the acoustic input; $512$-D embedding; cosine distance;\\
    & batch size = $128$; 64 epochs (in 4.8 hours) & $0.99, \textbf{0.95}$ \\ \hline
 10 & Same as \#9 but with additional dropout of $0.4$ between convolutional\\
    & layers as well; trained for much longer (142 epochs in 19 hours) & $0.96, 0.93$ \\
 \bottomrule
\end{tabular}
\vspace{-5mm}
\end{table}

We use the $\ell^2$-normalized output of each encoder's last layer as the learned embedding hence we don't use an activation function to those layers. Unless otherwise specified, we constrained incoming weights to a maximum norm of $3$ for layers with dropout regularization to allow for large learning rates without the risk of the weights ballooning~\cite{srivastava2014dropout}. We used the Adam optimizer~\cite{kingma2014adam} with initial learning rates tuned for different architectures ($0.001$ for CNN and $0.0001$ for LSTM); we reduced the learning rate by a factor of $2$ when learning stagnates for $2$ epochs. Since the addition of self-labeling to our experiments, we observed a much steeper acceleration in learning in early epochs. To balance the learning rate decay with the addition of new data on-the-fly during training, we increase the learning rate --- by the same factor to a maximum of $0.001$ --- when new hard negative examples are mined to give the model a better chance at learning from the newly minted examples.

For LSTM, we used $tanh$ for activation; otherwise, we used rectified linear units (ReLU)~\cite{nair2010rectified}, with the exception of the model's and the encoder's output layers. Network weight were initialized using He initialization~\cite{he2015delving} when ReLU was used; otherwise, we use the Xavier initialization method~\cite{glorot2010understanding}.

\section{Conclusion and Future Work}
 We presented techniques for learning functions that map acoustic and phonetic representations of words into fixed-dimensional vectors in a shared latent space, which are extremely useful in a plethora of speech recognition tasks. We experimented with many modeling techniques, hyperparameters, and neural network architectures to learn the joint embeddings; our best model is a Siamese CNN that feeds forward acoustic and phonetic inputs and brings together, in the shared vector space, similar-sounding words while keeping apart different-sounding ones. We used binary classification as a surrogate task to learn the embeddings at the last layer of each encoder. The choice of training examples cannot be cannot be overstated: we use three different techniques, including self-labeling, to mine hard negative examples for the contrastive loss function so that it can learn the subtleties requisite to discriminate between input words.

One of the areas to explore in future work is training multiple models for multiple word duration buckets to minimize extraneous padding. We'd also like to explore other loss functions detailed in~\cite{histogramloss, he2016multi}.

\subsubsection*{Acknowledgments}
We'd like to thank Ahmad Abdulkader for his mentorship and guidance; and the authors of related work and software tools~\cite{chollet2015keras, scikit-learn, mckinney2010data, hunter2007matplotlib, kluyver2016jupyter, jones2014scipy, mcfee2019librosa, paszke2017pytortch, nltk, sox, kaldi, ffmpeg} for allowing us to build on top of what they've created.

\printbibliography

\end{document}

Machine Learning (cs.LG); Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS); Machine Learning (stat.ML)