Gated Recurrent Neural Networks with Weighted Time-Delay Feedback

Abstract: We introduce a novel gated recurrent unit (GRU) with a weighted time-delay feedback mechanism in order to improve the modeling of long-term dependencies in sequential data. This model is a discretized version of a continuous-time formulation of a recurrent unit, where the dynamics are governed by delay differential equations (DDEs). By considering a suitable time-discretization scheme, we propose $\tau$-GRU, a discrete-time gated recurrent unit with delay. We prove the existence and uniqueness of solutions for the continuous-time model, and we demonstrate that the proposed feedback mechanism can help improve the modeling of long-term dependencies. Our empirical results show that $\tau$-GRU can converge faster and generalize better than state-of-the-art recurrent units and gated recurrent architectures on a range of tasks, including time-series classification, human activity recognition, and speech recognition.

\section{Illustrations of Differences Between ODE and DDE Dynamics}
\label{sect:appA}

Of particular interest to us are the differences between ODEs and DDEs that are driven by an input.
To illustrate the differences in the context of RNNs in terms of how they map the input signal into an output signal, we consider the simple examples:
\begin{itemize}
\item[(a)] DDE based RNNs with the hidden states $h \in \RR$ satisfying $\dot{h} = -h(t-\tau) + u(t)$, with $\tau = 0.5$ and $\tau = 1$, and $h(t) = 0$ for $t \in [-\tau,0]$, and
\item[(b)] an ODE based RNN with the hidden states $h \in \RR$ satisfying $\dot{h} = -h(t) + u(t)$,
\end{itemize}
where $u(t) = \cos(t)$ is the driving input signal.

Figure \ref{fig:diff} shows the difference between the dynamics of the hidden state driven by the input signal for (a) and (b). We see that, when compared to the ODE based RNN, the introduced delay causes time lagging in the response of the RNNs to the input signal. The responses are also amplified. In particular, using $\tau = 0.5$ makes the response of the RNN closely matches the input signal. In other words, simply fine tuning the delay parameter $\tau$ in the scalar RNN model is sufficient to replicate the dynamics of the input signal.

To further illustrate the differences, we consider the following examples of RNN with a nonlinear activation:
\begin{itemize}
\item[(c)] DDE based  RNNs with the hidden states $h \in \RR$ satisfying $\dot{h} = - h + \tanh(-h(t-\tau) + s(t))$, with $\tau > 0$, and $h(t) = 0$ for $t \in [-\tau,0]$, and
\item[(d)] an ODE based  RNN with the hidden states $h \in \RR$ satisfying $\dot{h} = - h + \tanh(-h(t) + s(t))$,
\end{itemize}
where the driving input signal $s$ is taken to be the truncated Weierstrass function:
\begin{equation}
    s(t) = \sum_{n=0}^3 a^{-n} \cos(b^n \cdot \omega t ),
\end{equation}
where $a=3$, $b=4$ and $\omega=2$.

Figure \ref{fig:diff2} shows the difference between the input-driven dynamics of the  hidden states for (c) and (d). We see that, when compared to the ODE based RNN, the introduced delay causes time lagging  in the response of the RNNs to the input signal. Even though the response of both RNNs does not match the input signal precisely (since we consider RNNs with one-dimensional hidden states here and therefore their expressivity is limited), we see that using $\tau = 0.5$ produces a response that tracks the seasonality of the input signal better than the  ODE based RNN.

\begin{figure}[!t]
\centering	\includegraphics[width=0.85\textwidth]{figs/diff_plot.pdf}	\caption{Hidden state dynamics of the DDE based RNNs with $\tau=0.5$  and $\tau=1$, and the ODE based RNN ($\tau=0$). All RNNs are driven by the same cosine input signal. }	\label{fig:diff}
\end{figure}

\begin{figure}[!h]
\centering	\includegraphics[width=0.85\textwidth]{figs/simple_plot2.pdf}	\caption{Hidden state dynamics of the DDE based RNN with $\tau=0.5$ and the ODE based RNN ($\tau = 0$). All RNNs are driven by the same input signal $s(t)$. }	\label{fig:diff2}
\end{figure}

\section{Proof of Theorem \ref{thm_exist2main}}
\label{sect:appB}

In this section, we provide a proof of Theorem \ref{thm_exist2main}.
To start, note that one can view the solution of the DDE as a mapping of functions on the interval $[t -\tau,t]$ into functions on the interval $[t,t + \tau]$, i.e., as a sequence of functions defined over a set of contiguous time intervals of length $\tau$.
This perspective makes it more straightforward to prove existence and uniqueness theorems analogous to those for ODEs than by directly viewing DDEs as an evolution over the state space $\RR^d$.

The following theorem, adapted from Theorem 3.7 in \cite{smith2011introduction}, provides sufficient conditions for existence and uniqueness of solution through a point $(t_0, \phi) \in \mathbb{R} \times C$ for the IVP \eqref{eq_genddemain}. Recall that  $C := C([-\tau, 0], \mathbb{R}^d)$, the Banach space of continuous functions from $[-\tau, 0]$ into $\mathbb{R}^d$ with the topology of uniform convergence. It is equipped with the norm $\|\phi \| := \sup \{ |\phi(\theta)| : \theta \in [-\tau, 0] \}$.

\begin{theorem}[Adapted from Theorem 3.7 in \cite{smith2011introduction}]
\label{thm_existence}
Let $t_0 \in \RR$ and $\phi \in C$ be given. Assume that $f$ is continuous and satisfies the Lipschitz condition on each bounded subset of $\mathbb{R} \times C$, i.e., for all $a, b \in \mathbb{R}$, there exists a constant $K > 0$ such that
\begin{equation}
    |f(t, \phi) - f(t, \psi)| \leq K \| \phi - \psi \|, \ t \in [a,b], \ \|\phi\|, \|\psi \| \leq M,
\end{equation}
with  $K$ possibly dependent on $a, b, M$.

There exists $A > 0$, depending only on $M$ such that if $\phi \in C$ satisfies $\| \phi \| \leq M$, then there exists a unique solution $h(t) = h(t, \phi)$ of Eq. \eqref{eq_genddemain}, defined on $[t_0 - \tau, t_0 + A]$. Moreover, if $K$ is the Lipschitz constant for $f$ corresponding to $[t_0, t_0 + A]$ and M, then
\begin{equation}
    \max_{\eta \in [t_0 - \tau, t_0 + A]} |h(\eta, \phi) - h(\eta, \psi)| \leq \|\phi - \psi \| e^{KA}, \ \|\phi\|, \| \psi \| \leq M.
\end{equation}
\end{theorem}

We now provide existence and uniqueness result for the continuous-time $\tau$-GRU model, assuming that the input $x$ is continuous in $t$. As before, we define the state $h_t \in C$ as:
\begin{equation}
    h_t(\theta) := h(t+\theta), \ -\tau \leq \theta \leq 0.
\end{equation}
Then the DDE defining the model can be formulated as the following IVP for the nonautonomous system:
\begin{equation} \label{eq_gendde}
    \dot{h}(t) = -h(t) + u(t, h(t)) + a(t, h(t)) \odot z(t, h_t), \ t \geq t_0,
\end{equation}
and $h_{t_0} = \phi \in C$ for some initial time $t_0 \in \mathbb{R}$, with the dependence on $x(t)$ casted as dependence on $t$ for notational convenience.

Now we restate Theorem \ref{thm_exist2main} from the main text and provide the proof.

\begin{theorem}[Existence and uniqueness of solution for continuous-time $\tau$-GRU] \label{thm_exist2}
Let $t_0 \in \RR$ and $\phi \in C$ be given. There exists a unique solution $h(t) = h(t, \phi)$ of Eq. \eqref{eq_gendde}, defined on $[t_0 - \tau, t_0 + A]$ for any $A > 0$. In particular, the solution exists for all $t \geq t_0$, and
\begin{equation}
    \| h_t(\phi) - h_t(\psi) \| \leq \| \phi - \psi \| e^{K(t-t_0)},
\end{equation}
for all $t \geq t_0$, where $K = 1 + \|W_1\| +  \|W_2\| + \|W_4\|/4$.
\end{theorem}
\begin{proof}
We shall apply Theorem \ref{thm_existence}.
To verify the Lipschitz condition: for any $\phi, \psi \in C$,
\begin{align}
    &
    \hspace{-7mm}
    |(u(t, \phi) + a(t, \phi) \odot z(t, \phi) - \phi) - (u(t, \psi) + a(t, \psi) \odot z(t, \psi) - \psi)| \nonumber \\
    &\leq |u(t, \phi) - u(t, \psi)| + |\phi - \psi| + |(a(t,\phi) - a(t, \psi)) \odot z(t,\phi) | + | a(t, \psi) \odot (z(t, \phi) - z(t, \psi) )| \\
    &\leq \|W_1\| \cdot |\phi - \psi| + |\phi - \psi| + \frac{1}{4} \|W_4\| \cdot  |\phi - \psi| + \|W_2\| \cdot  |\phi - \psi| \\
    &=: K |\phi - \psi|,
\end{align}
where we have used the  fact that the tanh and sigmoid are Lipschitz continuous, both bounded by one in absolute value, and they have  positive derivatives of magnitude no larger than one and $1/4$ respectively  in the last inequality~above.

Therefore, we see that the right hand side function satisfies a global Lipschitz condition and so the result follows from Theorem \ref{thm_existence}.
\end{proof}

\section{Proof of Proposition \ref{prop_delaymain}} \label{sect:appC}

In this section, we restate Proposition \ref{prop_delaymain}, and  provide its proof and some remarks.

\begin{proposition} \label{prop_delay}
Consider the linear time-delayed RNN  whose hidden states are described by the update equation:
\begin{equation} \label{eq_recursion}
    h_{n+1} = Ah_n + Bh_{n-m} + Cu_n, \ n=0,1,\dots,
\end{equation}
and $h_n = 0$ for $n =-m, -m+1, \dots, 0$ with $m > 0$.
Then, assuming that $A$ and $B$ commute, we have:
\begin{equation}
    \frac{\partial h_{n+1}}{\partial u_i} = A^{n-i} C,
\end{equation}
for $n=0,1, \dots, m$, $i=0,\dots, n$, and
\begin{align}
    \frac{\partial h_{m+1+j}}{\partial u_i} &= A^{m+j-i} C + \delta_{i,j-1} BC + 2 \delta_{i,j-2} ABC \nonumber \\
    &\ \ \ \ + 3 \delta_{i,j-3} A^2 B C + \cdots + j \delta_{i,0} A^{j-1} B C,
\end{align}
for $j = 1,2,\dots, m+1$, $i=0,1,\dots, m+j$, where $\delta_{i,j}$ denotes the Kronecker delta.
\end{proposition}

\begin{proof}
Note that by definition $h_i = 0$ for $i=-m, -m+1, \dots, 0$, and, upon iterating the recursion \eqref{eq_recursion}, one~obtains:
\begin{equation}
    h_{n+1} = A^n C u_0 + A^{n-1} C u_1 + \cdots + AC u_{n-1} + C u_n,
\end{equation}
for $n = 0, 1, \dots, m$.

Now, applying the above formula for $h_1$, we obtain
\begin{align}
h_{m+2} &= A h_{m+1} + B h_1 + Cu_{m+1} \nonumber \\
&= (B + A^{m+1} ) C u_0 + A^{m} C u_1 + \cdots + A^2 C u_{m-1} + ACu_m + Cu_{m+1}.
\end{align}

Likewise, we obtain:
\begin{align}
    h_{m+3} &= A h_{m+2} + B h_2 + Cu_{m+2} \nonumber \\
    &= (BA + A^{m+2} + AB) C u_0 + (B+A^{m+1})C u_1 + A^m C u_2 + \cdots + AC u_{m+1} + C u_{m+2} \nonumber \\
    &= (2AB + A^{m+2}) C u_0 + (B+A^{m+1})C u_1 + A^m C u_2 + \cdots + AC u_{m+1} + C u_{m+2},
\end{align}
where we have used commutativity of $A$ and $B$ in the last line above.

Applying the above procedure repeatedly and using commutativity of $A$ and $B$ give:
\begin{equation} \label{eq_linear}
    h_{m+1+j} = (A^{m+j} + j A^{j-1} B) C u_0 + ( A^{m+j-1} + (j-1) A^{j-2} B) C u_1 + \cdots + ACu_{m+j-1} + C u_{m+j},
\end{equation}
for $j = 1,2, \dots, m+1$.

The formula in the proposition then follows upon taking the derivative with respect to the $u_i$ in the above formula for the hidden states $h_k$.
\end{proof}

We remark that one could also derive formula for the gradients $\frac{\partial h_{n+1+j}}{\partial u_i}$ for $n \geq 2m+1$ as well as those for our $\tau$-GRU architecture analogously, albeit the resulting expression is quite complicated.
In particular, the dependence on higher powers of $B$ for the coefficients in front of the Kronecker deltas would appear in the formula for the former case (with much more complicated expressions  without assuming commutativity of the matrices). However, we emphasize that the qualitative conclusion derived from the analysis remains the same: that introduction of delays places more emphasis on gradient information due to input elements in the past, so they act as  buffers to propagate the gradient information more effectively than the counterpart models without delays.

\section{Gradient Bounds for $\tau$-GRU}
\label{app:gradbound}

\noindent
{\bf On the exploding and vanishing gradient problem.}
For simplicity of our discussion here, we consider the loss~function:
\begin{equation}
    \mathcal{E}_n = \frac{1}{2} \| y_n - \overline{y}_n  \|^2,
\end{equation}
where $n = 1, \dots, N$ and $\overline{y}_n$ denotes the underlying growth truth. The training of $\tau$-GRU involves computing gradients of this loss function with respect to its underlying parameters $\theta \in \Theta = [W_{1,2,3,4}, U_{1,2,3,4}, V]$ at each iteration of the gradient descent algorithm. Using chain rule, we obtain \cite{pascanu2013difficulty}:
\begin{equation}
    \frac{\partial \mathcal{E}_n}{\partial \theta} = \sum_{k=1}^n \frac{\partial \mathcal{E}_n^{(k)}}{\partial \theta},
\end{equation}
where
\begin{equation}
    \frac{\partial \mathcal{E}_n^{(k)}}{\partial \theta} = \frac{\partial \mathcal{E}_n}{\partial h_n} \frac{\partial h_n}{\partial h_k} \frac{\partial^+ h_k}{\partial \theta},
\end{equation}
with $\frac{\partial^+ h_k}{\partial \theta}$ denoting  the ``immediate'' partial derivative of
the state $h_k$ with respect to $\theta$, i.e., where $h_{k-1}$ is taken as a constant with respect to $\theta$ \cite{pascanu2013difficulty}.

The partial gradient $\frac{\partial \mathcal{E}_n^{(k)}}{\partial \theta}$  measures the contribution to the hidden state gradient at step $n$ due to the information at step $k$. It can be shown that this gradient behaves as \begin{equation}
\frac{\partial \mathcal{E}_n^{(k)}}{\partial \theta} \sim \gamma^{n-k},
\end{equation}
for some $\gamma > 0$ \cite{pascanu2013difficulty}. If $\gamma > 1$, then the gradient grows exponentially in sequence length, for long-term dependencies where $k \ll n$, causing the exploding gradient problem. On the other hand, if $\gamma < 1$, then the gradient decays exponentially for $k \ll n$, causing the vanishing gradient problem.  Therefore,  we can investigate how $\tau$-GRU deals with these problems by deriving bounds on the gradients. In particular, we are interested in the behavior of the gradients for long-term dependencies, i.e., $k \ll n$, and shall show that the delay mechanism in $\tau$-GRU slows down the exponential decay rate, thereby reducing the sensivity to the vanishing gradient problem.

Recall that the update equations defining $\tau$-GRU are given by $h_n = 0$ for $n=-m, -m+1, \dots, 0$,
\begin{equation}
    h_n = (1-g(A_{n-1})) \odot h_{n-1} + g(A_{n-1}) \odot [u(B_{n-1}) + a(C_{n-1}) \odot z(D_{n-m-1})],
\end{equation}
for $n=1,2,\dots, N$, where $m := \lfloor \tau/\Delta t \rfloor \in \{1,2,\dots, N-1\}$, $A_{n-1} = W_3 h_{n-1} + U_3 x_{n-1}$, $B_{n-1} = W_1 h_{n-1} + U_1 x_{n-1}$,
$C_{n-1} = W_4 h_{n-1} + U_4 x_{n-1}$, and $D_{n-m-1} = W_2 h_{n-m-1} + U_2 x_{n-1}$.

In the sequel, we shall denote the $i$th component of a vector $v$ as $v^i$ and the $(i,j)$ entry of a matrix $A$ as $A^{ij}$.

We start with the following lemma.

\begin{lemma} \label{app_lem}
For every $i$, we have $ h_n^i = 0$, for $n = -m, -m+1, \dots, 0$, and $ |h_n^i| \leq 2$, for $n=1,2,\dots, N$.
\end{lemma}

\begin{proof}
The $i$th component of the hidden states of $\tau$-GRU are given by: $h_n^i = 0$ for $n=-m, -m+1, \dots, 0$, and
\begin{equation}
    h_n^i = (1-g(A^i_{n-1}))  h^i_{n-1} + g(A^i_{n-1})  [u(B^i_{n-1}) + a(C^i_{n-1}) z(D^i_{n-m-1})],
\end{equation}
for $n=1,2,\dots, N$.

Using the fact that $g(x), a(x) \in (0,1)$ and $u(x), z(x) \in (-1,1)$ for all $x$, we can bound the $h_n^i$ as:
\begin{align}
    h_n^i &\leq (1-g(A^i_{n-1}))  \max(h^i_{n-1},2) + g(A^i_{n-1}) \max(h^i_{n-1},2) \nonumber \\
    &\leq \max(h^i_{n-1},2),
\end{align}
for all $i$ and $n = 1,2,\dots, N$.

Similarly, we have:
\begin{align}
    h_n^i &\geq (1-g(A^i_{n-1}))  \min(-2, h^i_{n-1}) + g(A^i_{n-1}) \min(-2, h^i_{n-1}) \nonumber \\
    &\geq \min(-2, h^i_{n-1}),
\end{align}
for all $i$ and $n = 1,2,\dots, N$.

Thus,
\begin{equation}
      \min(-2, h^i_{n-1}) \leq h_n^i \leq \max(h^i_{n-1},2),
\end{equation}
for all $i$ and $n = 1,2,\dots, N$.

Now, iterating over $n$ and using $h_0^i = 0$ for all $i$, we obtain $-2 \leq h_n^i \leq 2$ for all $i$ and $n = 1,2,\dots, N$.

\end{proof}

We now provide the gradients bound for $\tau$-GRU in the following proposition and proof.

\begin{proposition} \label{app_prop}
Assume that there exists an $\epsilon > 0$ such that $\max_n g(A^i_{n-1}) \geq \epsilon$ and   $\max_n a(C^i_{n-1}) \geq \epsilon$ for all $i$. Then
\begin{equation}
    \left \| \frac{\partial h_n}{\partial h_k} \right\|_\infty \leq (1+C-\epsilon)^{n-k} + \| W_2\|_\infty \cdot \left( (1+C-\epsilon)^{n-k-2} \delta_{m,1} + \dots + (1+C-\epsilon) \delta_{m, n-k-2} + \delta_{m, n-k-1} \right),
\end{equation}
for $n=1, \dots, N$ and $k < n$, where $C = \|W_1\|_\infty +  \|W_3\|_\infty + \frac{1}{4}\|W_4\|_\infty$.
\end{proposition}
\begin{proof}
Recall $h_n = 0$ for $n=-m, -m+1, \dots, 0$, and
\begin{equation}
    h_n = (1-g(A_{n-1})) \odot h_{n-1} + g(A_{n-1}) \odot [u(B_{n-1}) + a(C_{n-1}) \odot z(D_{n-m-1})] =: F(h_{n-1}, h_{n-m-1}),
\end{equation}
for $n=1,2,\dots, N$.

Denote $q_{n,l} := \frac{\partial F}{\partial h_{n-l}}$, where $F := F(h_{n-1}, h_{n-l})$ for $l>1$.
The gradients $\frac{\partial h_n}{\partial h_k}$ can be computed recursively as~follows.

\begin{align}
    p_n^{(1)} &:= \frac{\partial h_n}{\partial h_{n-1}},  \\
    p_n^{(2)} &:= \frac{\partial h_n}{\partial h_{n-2}} = p_n^{(1)} p_{n-1}^{(1)} + q_{n,2} \delta_{m,1}, \\  p_n^{(3)} &:= \frac{\partial h_n}{\partial h_{n-3}} = p_n^{(1)} p_{n-1}^{(2)} + q_{n,3} \delta_{m,2}, \\
    \vdots \\
    p_n^{(n-k)} &:= \frac{\partial h_n}{\partial h_{k}} = p_n^{(1)} p_{n-1}^{(n-k-1)} + q_{n,n-k} \delta_{m,n-k-1}.
\end{align}
As $\|p_n^{(n-k)}\| \leq \|p_n^{(1)} \| \cdot \| p_{n-1}^{(n-k-1)}\| + \|q_{n,n-k}\| \delta_{m,n-k-1}$, it remains to upper bound the $p_n^{(1)}$ and $q_{n,n-k}$.

The $i$th component of the hidden states can be written as:
\begin{equation}
    h_n^i = (1-g(A^i_{n-1}))  h^i_{n-1} + g(A^i_{n-1})  [u(B^i_{n-1}) + a(C^i_{n-1})  z(D^i_{n-m-1})],
\end{equation}
where $A^i_{n-1} = W_3^{iq} h_{n-1}^q + U_3^{ir} x_{n-1}^{r}$, $B^i_{n-1} = W_1^{iq} h_{n-1}^q + U_1^{ir} x_{n-1}^{r}$, $C^i_{n-1} = W_4^{iq} h_{n-1}^q + U_4^{ir} x_{n-1}^{r}$, and $D^i_{n-m-1} = W_2^{iq} h_{n-m-1}^q + U_2^{ir} x_{n-1}^{r}$, using Einstein's summation notation for repeated indices.

Therefore, applying chain rule and using Einstein's summation for repeated indices in the following, we obtain:
\begin{align}
 \frac{\partial h_n^i}{\partial h_{n-1}^j} &= (1-g(A_{n-1}^i))  \frac{\partial h_{n-1}^i}{\partial h_{n-1}^j} - \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^l} \frac{\partial A_{n-1}^l}{\partial h_{n-1}^j} h_{n-1}^i + g(A_{n-1}^i) \frac{\partial u(B_{n-1}^i)}{\partial B_{n-1}^l} \frac{\partial B_{n-1}^l}{\partial h_{n-1}^j} \nonumber \\
 &\ \ \ \ \ + \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^l} \frac{\partial A_{n-1}^l}{\partial h_{n-1}^j} u(B_{n-1}^i) +    g(A_{n-1}^i) \frac{\partial a(C_{n-1}^i)}{\partial C_{n-1}^l} \frac{\partial C_{n-1}^l}{\partial h_{n-1}^j} z(D_{n-m-1}^i) \nonumber \\
 &\ \ \ \ \ + \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^l} \frac{\partial A_{n-1}^l}{\partial h_{n-1}^j} a(C_{n-1}^i) z(D_{n-m-1}^i).
\end{align}

Noting that $\frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^l} = 0$, $\frac{\partial u(B_{n-1}^i)}{\partial B_{n-1}^l} = 0$ and $\frac{\partial a(C_{n-1}^i)}{\partial C_{n-1}^l} = 0$ for $i \neq l$, we have:
\begin{align}
 \frac{\partial h_n^i}{\partial h_{n-1}^j} &= (1-g(A_{n-1}^i))  \frac{\partial h_{n-1}^i}{\partial h_{n-1}^j} - \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^i} \frac{\partial A_{n-1}^i}{\partial h_{n-1}^j} h_{n-1}^i + g(A_{n-1}^i) \frac{\partial u(B_{n-1}^i)}{\partial B_{n-1}^i} \frac{\partial B_{n-1}^i}{\partial h_{n-1}^j} \nonumber \\
 &\ \ \ \ \ + \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^i} \frac{\partial A_{n-1}^i}{\partial h_{n-1}^j} u(B_{n-1}^i) +    g(A_{n-1}^i) \frac{\partial a(C_{n-1}^i)}{\partial C_{n-1}^i} \frac{\partial C_{n-1}^i}{\partial h_{n-1}^j} z(D_{n-m-1}^i) \nonumber \\
 &\ \ \ \ \ + \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^i} \frac{\partial A_{n-1}^i}{\partial h_{n-1}^j} a(C_{n-1}^i) z(D_{n-m-1}^i) \\
 &= (1-g(A_{n-1}^i)) \delta_{i,j}  - \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^i} W_3^{ij} h_{n-1}^i + g(A_{n-1}^i) \frac{\partial u(B_{n-1}^i)}{\partial B_{n-1}^i} W_1^{ij} \nonumber \\
 &\ \ \ \ \ + \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^i} W_3^{ij} u(B_{n-1}^i) +    g(A_{n-1}^i) \frac{\partial a(C_{n-1}^i)}{\partial C_{n-1}^i} W_4^{ij}  z(D_{n-m-1}^i) \nonumber \\
 &\ \ \ \ \ + \frac{\partial g(A_{n-1}^i)}{\partial A_{n-1}^i} W_3^{ij} a(C_{n-1}^i) z(D_{n-m-1}^i).
\end{align}

Using the assumption that $\max_n g(A^i_{n-1}),  \max_n a(C^i_{n-1}) \geq \epsilon$ for all $i$, the fact that $|z(x)|, |u(x)| \leq 1$, $g(x), a(x) \in (0,1)$, $g'(x), a'(x) \leq 1/4$, $u'(x) \leq 1$ for all $x \in \RR$, and Lemma \ref{app_lem},  we obtain:
\begin{align}
 \left|\frac{\partial h_n^i}{\partial h_{n-1}^j} \right| &\leq (1-\epsilon) \delta_{i,j} + \frac{1}{4} |W_3^{ij}| \cdot |h_{n-1}^i| + |W_1^{ij}| +  \frac{1}{4} |W_3^{ij}| + \frac{1}{4} |W_4^{ij}| + \frac{1}{4} |W_3^{ij}| \\
 &\leq (1-\epsilon) \delta_{i,j} + |W_1^{ij}| + |W_3^{ij}| + \frac{1}{4} |W_4^{ij}|.
\end{align}

Therefore,
\begin{equation}
     \left \| \frac{\partial h_n}{\partial h_{n-1}} \right\|_\infty := \max_{i=1,\dots, d} \sum_{j=1}^d \left| \frac{\partial h_n^i}{\partial h_{n-1}^j } \right| \leq (1-\epsilon ) +  \|W_1\|_\infty + \|W_3 \|_\infty + \frac{1}{4} \|W_4\|_\infty =: 1-\epsilon + C.
\end{equation}

Likewise, we obtain, for $l>1$:
\begin{align}
    \frac{\partial F^i}{\partial h_{n-l}^j} &= g(A_{n-1}^i) a(C_{n-1}^i) \frac{\partial z(D_{n-l}^i)}{\partial D_{n-l}^i} \frac{\partial D_{n-l}^i}{\partial h_{n-l}^j}  \\
    &= g(A_{n-1}^i) a(C_{n-1}^i) \frac{\partial z(D_{n-l}^i)}{\partial D_{n-l}^i} W_2^{ij}.
\end{align}

Using the fact that $|g(x)|, |a(x)| \leq 1$ and $|z'(x)| \leq 1$ for all $x \in \RR$, we obtain:
\begin{align}
    \left|\frac{\partial F^i}{\partial h_{n-l}^j}\right| &\leq    |W_2^{ij}|,
\end{align}
for $l > 1$,
and thus
$\|q_{n,n-k}\|_\infty = \| \frac{\partial F}{\partial h_k} \|_\infty \leq \|W_2\|_\infty$ for $k > 1$.

The upper bound in the proposition follows by using the above bounds for $\|p_{n}^{(1)}\|_\infty$ and $\|q_{n,n-k}\|_\infty$,  and iterating the recursion $\|p_n^{(n-k)}\| \leq \|p_n^{(1)} \| \cdot \| p_{n-1}^{(n-k-1)}\| + \|q_{n,n-k}\| \delta_{m,n-k-1}$ over $k$.

\end{proof}

From Proposition \ref{app_prop}, we see that if $\epsilon > C$, then the gradient norm decays exponentially as $k$ becomes large. However,  the delay in $\tau$-GRU introduces jump-ahead connections  (buffers) to slow down the exponential decay. For instance, choosing $m=1$ for the delay, we have  $\left \| \frac{\partial h_n}{\partial h_k} \right\|  \sim (1+C-\epsilon)^{n-k-2}$ as $k \to \infty$ (instead of $\left \| \frac{\partial h_n}{\partial h_k} \right\|  \sim (1+C-\epsilon)^{n-k-1}$ as $k \to \infty$ in the case when no delay is introduced into the model). The larger the $m$ is, the more effective the delay is able to slow down the exponential decay of the gradient norm. These qualitative conclusions can already be derived by studying the linear time-delayed RNN, which we consider in the main text  for simplicity.

\section{Approximation Capability of Time-Delayed RNNs} \label{app:uat}

RNNs (without delay) have been shown to be universal approximators of a large class of open dynamical systems \cite{schafer2006recurrent}. Analogously, RNNs with delay can be shown to be universal approximators of open dynamical systems with~delay.

Let $m > 0$ (time lag) and consider the state space models (which, in this section, we shall simply refer to as delayed RNNs) of the form:
\begin{align} \label{eq_gendRNN}
    s_{n+1} &= f(As_n + B s_{n-m} + Cu_n + b), \nonumber \\
    r_n &= Ds_n,
\end{align}
and dynamical systems of the form
\begin{align} \label{ds_approx}
x_{n+1} &= g(x_n, x_{n-m}, u_n), \nonumber \\
o_n &= o(x_n),
\end{align}
for $n = 0,1,\dots, N$.
Here $u_n \in \RR^{d_u}$ is the input, $o_n \in \RR^{d_o}$ is the target output of the dynamical systems to be learned, $s_n \in \RR^d$ is the hidden state of the learning model, $r_n \in \RR^{q}$ is the model output, $f$ is the tanh function applied component-wise, the maps $g$ and $o$ are Lipschitz continuous, and the matrices $A$, $B$, $C$, $D$ and the vector $b$ are learnable parameters. For simplicity, we take the initial functions to be $s_n = y_n = 0$ for $n = -m, -m+1, \dots, 0$.

The following theorem shows that  the delayed RNNs  \eqref{eq_gendRNN} are capable of approximating a large class of time-delay dynamical systems, of the form \eqref{ds_approx}, to arbitrary accuracy.

\begin{theorem} \label{thm_uat}
Assume that there exists a constant $R > 0$ such that $\max(\|x_{n+1}\|, \|u_n\|) < R$ for $n = 0,1,\dots, N$. Then, for a given $\epsilon > 0$, there exists a delayed RNN of the form  \eqref{eq_gendRNN} such that the following holds for some $d$:
\begin{equation} \label{thm_bd}
    \|r_n - o_n \| \leq \epsilon,
\end{equation}
for $n = 0,1,\dots, N$.
\end{theorem}

\begin{proof}
The proof proceeds along the line of \cite{schafer2006recurrent, rusch2022long}, using the universal approximation theorem (UAT) for feedforward neural network maps and  with straightforward modification to deal with the extra delay variables $s_{n-n}$ and $x_{n-m}$ here.   The proof  proceeds in a similar manner as the one provided in Section E.4 in \cite{rusch2022long}.
The goal is to construct hidden states, output state, weight matrices and bias vectors such that an output of the delayed RNN approximates the dynamical system \eqref{ds_approx}.

Let $\epsilon > \epsilon^* > 0, R^* > R  \gg 1$ be parameters to be defined later. Then, using the UAT for continuous functions with neural networks with the tanh activation function \cite{barron1993universal}, we can obtain the following statements.  Given an $\epsilon^*$, there exist weight matrices $W_1$, $W_2$, $W_3$, $V_1$ and a bias vector $b_1$ of appropriate dimensions such that the neural network defined by $\mathcal{N}_1(h, \tilde{h}, u) := W_3 \tanh(W_1 h + W_2 \tilde{h} + V_1 u + b_1)$ approximates the underlying function $g$ as follows:
\begin{equation}
    \max_{\max (\|h\|, \|\tilde{h} \|, \|u\|) < R^* } \| g(h, \tilde{h}, u)  - \mathcal{N}_1(h, \tilde{h}, u) \| \leq \epsilon^*.
\end{equation}

Now, we define the dynamical system:
\begin{align}
    p_{n} &= W_3 \tanh(W_1 p_{n-1}   + W_2 p_{n-m-1} + V_1 u_{n-1} + b_1),
\end{align}
with $p_i = 0$ for $i = -m, -m+1, \dots, 0$.

Then using the above approximation bound, we obtain, for $n=1, \dots, N+1$,
\begin{align}
    \|x_{n} - p_{n} \| &= \| g(x_{n-1}, x_{n-m-1}, u_{n-1})  - p_n \| \\
    &\leq \|g(x_{n-1}, x_{n-m-1}, u_{n-1}) - W_3 \tanh(W_1 p_{n-1}   + W_2 p_{n-m-1} + V_1 u_{n-1} + b_1)\| \\
    &\leq \|g(x_{n-1}, x_{n-m-1}, u_{n-1}) - g(p_{n-1}, p_{n-m-1},  u_{n-1}) \| \nonumber \\
    &\ \ \ \ \ + \|  g(p_{n-1}, p_{n-m-1},  u_{n-1}) - W_3 \tanh(W_1 p_{n-1}   + W_2 p_{n-m-1} + V_1 u_{n-1} + b_1)\| \\
    &\leq Lip(g) (\| x_{n-1} - p_{n-1} \| + \|x_{n-m-1} - p_{n-m-1}\|) +  \epsilon^*,
\end{align}
where $Lip(g)$ is the Lipschitz constant of $g$ on the compact set $\{ (h,\tilde{h}, u): \| h\|, \|\tilde{h}\|, \|u\| < R^* \}$.

Iterating the above inequality over $n$ leads to:
\begin{equation}
    \|x_n - p_n \|\leq \epsilon^* C_1(n, m, Lip(g)),
\end{equation}
for some constant $C_1>0$ that is dependent on $n, m, Lip(g)$.

Using the Lipschitz continuity of the output function $o$, we obtain:
\begin{equation} \label{out_b}
    \| o_n - o(p_n) \| \leq \epsilon^*C_2(n, m, Lip(g), Lip(o)),
\end{equation}
for some constant $C_2$ that is dependent on $n, m, Lip(g), Lip(o)$, where $Lip(o)$ is the Lipschitz constant of $o$ on the compact set $\{ h : \| h\|  < R^* \}$.

Next we use the UAT for neural networks again to obtain the following approximation result. Given an $\overline{\epsilon}$, there exists weight matrices $W_4, W_5$ and bias vector $b_2$ of appropriate dimensions such that the tanh neural network, $\mathcal{N}_2(h) := W_5 \tanh(W_4 h + b_2)$ approximates the underlying output function $o$ as:
\begin{equation}
    \max_{\|h\| < R^*} \| o(h) - \mathcal{N}_2(h) \| \leq \overline{\epsilon}.
\end{equation}

Defining $\overline{o}_n = W_5 \tanh(W_4 p_n + b_2)$, we obtain, using the above approximation bound and the inequality \eqref{out_b}:
\begin{equation} \label{o_b_2}
    \|o_n - \overline{o}_n \| = \|o_n - o(p_n) \| + \| o(p_n) - \overline{o}_n \|    \leq \epsilon^* C_2(n, m, Lip(g), Lip(o)) + \overline{\epsilon}.
\end{equation}

Now, let us denote:
\begin{align}
    \tilde{p}_{n} &= \tanh(W_1 p_{n-1}   + W_2 p_{n-m-1} + V_1 u_{n-1} + b_1),
\end{align}
so that $p_n = W_3 \tilde{p}_n$. With this notation, we have:
\begin{equation}
\overline{o}_n = W_5 \tanh(W_4 W_3 \tanh(W_1 W_3 \tilde{p}_{n-1}   + W_2 W_3 \tilde{p}_{n-m-1} + V_1 u_{n-1} + b_1)  + b_2).
\end{equation}

Since the function $R(y) = W_5 \tanh(W_4 W_3 \tanh(W_1 W_3 y + W_2 W_3 \tilde{p}_{n-m-1} + V_1 u_{n-1} + b_1) + b_2)$ is Lipschitz continuous in $y$, we can apply the UAT again to obtain: for any $\tilde{\epsilon}$, there exists weight matrices $W_6, W_7$ and bias vector $b_3$ of appropriate dimensions such that
\begin{equation}
    \max_{\|y\| < R^*} \| R(y) - W_7 \tanh(W_6 y + b_3) \| \leq \tilde{\epsilon}.
\end{equation}
Denoting $\tilde{o}_n := W_7 \tanh(W_6 p_{n-1} + b_3)$ and using the above approximation bound, we obtain $\|\overline{o}_n - \tilde{o}_n \| \leq \tilde{\epsilon}$.

Finally, we collect all the ingredients above to construct a delayed RNN that can approximate the dynamical system \eqref{ds_approx}.
To this end, we define the hidden states (in an enlarged state space):  $s_n := (\tilde{p}_n, \hat{p}_n)$, with $\tilde{p}_n$, $\hat{p}_n$ sharing the same dimension.
These hidden states evolve according to the dynamical system:
\begin{align}
s_n &= \tanh\left(
  \left[ {\begin{array}{cc}
    W_1 W_3 & 0 \\
    W_6 W_3 & 0 \\
  \end{array} } \right] s_{n-1} +  \left[ {\begin{array}{cc}
    W_2 W_3 & 0 \\
    0 & 0 \\
  \end{array} } \right] s_{n-m-1} +  \left[ {\begin{array}{c}
    V_1 u_{n-1}  \\
    0 \\
  \end{array} } \right] +  \left[ {\begin{array}{c}
    b_1 \\
    0 \\
  \end{array} } \right]
\right).
\end{align}
Defining the output state as $r_n := [0, W_7] s_n$, with the $s_n$ satisfying the above system, we arrive at a delayed RNN that approximates the dynamical system  \eqref{ds_approx}. In fact, we can  verify that $r_n = \tilde{o}_n$.
Setting $\overline{\epsilon} < \epsilon/2$ and  $\epsilon^* < \epsilon/(2 C_2(n, m, Lip(g), Lip(o)))$ give us the  bound \eqref{thm_bd} in the theorem.
\end{proof}

\section{Additional Details and Experiments}
\label{sect:appD}

In this section, we provide additional empirical results and details to demonstrate the advantages of $\tau$-GRU when compared to other RNN architectures.

\begin{table}[!b]
	\caption{Results for Google12. Results indicated by $^*$ are produces by us, results indicated by $^+$ are from~\cite{rusch2022long}. }
	\label{tab:results_google12}
	\centering

	\scalebox{0.9}{
		\begin{tabular}{l c  c ccccccc}
			\toprule
			Model & Test Accuracy ($\%$) &  \# units & \# param \\
			\midrule
			tanh RNN~\cite{rusch2022long}$^+$ & 73.4 & 128 & 27k \\
			LSTM~\cite{rusch2022long}$^+$  & 94.9 & 128 & {107k}\\
			GRU~\cite{rusch2022long}$^+$  & 95.2 & 128 & 80k \\
			AsymRNN~\cite{chang2018antisymmetricrnn}$^+$ & 90.2 & 128 & 20k \\
			expRNN~\cite{lezcano2019cheap}$^+$ & 92.3 & 128 & 19k \\
			coRNN~\cite{rusch2021coupled}$^+$ & 94.7 & 128 & 44k \\
			Fast GRNN~\cite{kusupati2018fastgrnn}$^+$ & 94.8 & 128 & 27k \\
			LEM~\cite{rusch2022long} & 95.7 & 128 & {107k} \\
			Lipschitz RNN~\cite{erichson2020lipschitz}$^*$ & 95.6 & 128 & 34k\\
			Noisy RNN~\cite{lim2021noisy}$^*$ & 95.7 & 128 & 34k\\
			iRNN~\cite{Kag2020RNNs}$^*$ & 95.1 & - & {8.5k} \\
			TARNN~\cite{kag2021time}$^*$ & 95.9 & 128 & {107k} \\
			\midrule
			\textbf{ours} & \textbf{96.2} & 128 & {107k} \\
			\bottomrule
	\end{tabular}}
\end{table}

\subsection{Speech Recognition: Google 12}

Here, we consider the Google Speech Commands data set V2~\cite{warden2018speech} to demonstrate the performance of our model for speech recognition.
The aim of this task is to learn a model that can classify a short audio sequence, which is sampled at a rate of 16 kHz from 1 second utterances of $2,618$ speakers.
We consider the Google 12-label task (Google12) which is composed of 10 keyword classes, and in addition one class that corresponds to `silence', and a class corresponding to `unknown' keywords.
We adopt the standard train/validation/test set split for evaluating our model, and we use dropout, applied to the inputs, with rate 0.03 to reduce overfitting.

Table~\ref{tab:results_google12} presents the results for our $\tau$-GRU and a number of competitive RNN architectures.
We adopt the results for the competitive RNNs from~\cite{rusch2022long}. Our proposed $\tau$-GRU shows the best performance on this task, i.e., $\tau$-GRU is able to outperform gated and continuous-time RNNs on this task that requires an expressive recurrent unit.

\subsection{Learning the Dynamics of Mackey-Glass System}

Here, we consider the task of learning the Mackey-Glass equation, originally introduced in \cite{mackey1977oscillation} to  model the variation in the relative quantity of mature cells in the blood:
\begin{equation}
    \dot{x} = a \frac{x(t-\delta)}{1 + x^n(t-\delta)} - b x(t), \ t \geq \delta,
\end{equation}
where $\delta \geq 17$, $a, b, n > 0$, with $x$ satisfying $\dot{x} = a x(0)/(1 + x(0)^n) - b x$ for $t \in [0, \delta]$. It is a scalar equation with chaotic dynamics, with infinite-dimensional state space. Increasing the value of $\delta$ increases the dimension of the~attractor.

For data generation, we choose $a = 0.2$, $b = 0.1$, $n=10$, $\delta = 17$, $x(0) \sim \rm{Unif}(0,1)$, and use the classical Runge-Kutta  method (RK4) to integrate the system numerically from $t=0$ to $t = 1000$ with a step-size of 0.25. The training and testing samples are the time series (of length 2000) generated by the RK4 scheme on the interval $[500,1000]$ for different realizations of $x(0)$. Figure \ref{fig:data_plot} shows a realization of the trajectory produced by the Mackey-Glass system (and also the DDE based ENSO system considered in the main text).

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.44\textwidth]{figs/mg_plot.png}
	\includegraphics[width=0.44\textwidth]{figs/mz_plot.png}
	\caption{A realization of the Mackey-Glass dynamics (left) and the DDE based ENSO dynamics (right). }
	\label{fig:data_plot}
\end{figure}

Table \ref{tab:results_MG} shows that our $\tau$-GRU model (with $\alpha = \beta = 1$ and using $\tau = 10$) is more effective in learning the Mackey-Glass system when compared to other RNN architectures. We also see that the predictive performance deteriorates without making full use of the combination of the standard recurrent unit and delay recurrent unit  (setting either $\alpha$ or $\beta$ to zero). Moreover, $\tau$-GRU demonstrates improved performance when compared to the simple delay GRU model (Eq. \eqref{eq_simpleDRNN}) and the counterpart model without using the gating. Similar observation also holds for the ENSO prediction task; see Table \ref{tab:add_results_ENSO}.

Figure \ref{fig:mg_traintest} shows that our model converges much faster than other RNN models during training. In particular, our model is able to achieve both lower training and testing error (as measured by the root mean square error (RMSE)) with fewer epochs, demonstrating the effectiveness of the delay mechanism in improving the performance on the problem of long-term dependencies. This is consistent with our analysis on how the gradient
information is propagated through the delay buffers in the network (see Proposition \ref{prop_delaymain}), suggesting that the delay buffers can propagate gradient more efficiently. Similar behavior is also observed for the ENSO task; see Figure \ref{fig:mz_traintest}.

\begin{table}[!h]
	\caption{Additional results for the ENSO model prediction.}
	\label{tab:add_results_ENSO}
	\centering

	\scalebox{0.9}{
		\begin{tabular}{l c  c  ccccccc}
			\toprule
			Model & MSE ($\times 10^{-2}$) &  \# units & \# parameters \\
			\midrule

		    simple delay GRU (Eq. \eqref{eq_simpleDRNN}) & 0.2317 & 16 & 0.897k \\
			ablation (no gating) & 0.4289 & 16 & 0.929k \\
		\midrule
		\textbf{$\tau$-GRU (ours)} & \textbf{0.17} & 16 & 1.2k \\
    	\bottomrule
	\end{tabular}}
\end{table}

\begin{table}[!t]
	\caption{Results for the Mackey-Glass system prediction.}
	\label{tab:results_MG}
	\centering

	\scalebox{0.9}{
		\begin{tabular}{l c  c  ccccccc}
			\toprule
			Model & MSE ($\times 10^{-2}$) &  \# units & \# parameters \\
			\midrule
			Vanilla RNN & 0.3903 & 16 & 0.321k \\
			LSTM & 0.6679 & 16 & 1.233k \\
			GRU & 0.4351 & 16 & 0.929k \\
			Lipschitz RNN & 8.9718 & 16 & 0.561k \\
			coRNN & 1.6835 & 16 & 0.561k \\
			LEM & 0.1430 & 16 & 1.233k \\
			\midrule
			simple delay  GRU (Eq. \eqref{eq_simpleDRNN})  & 0.2772 & 16 & 0.897k \\
			ablation (no gating) & 0.2765 & 16 & 0.929k \\
			ablation ($\alpha = 0$) & 0.1553 & 16 & 0.625k \\
			ablation ($\beta = 0$) & 0.2976 & 16 & 0.929k \\
			\midrule
			\textbf{$\tau$-GRU (ours)} & \textbf{0.1358}  & 16 & 1.233k \\
			\bottomrule
	\end{tabular}}
\end{table}

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.44\textwidth]{figs/mg_trainrmse.png}
	\includegraphics[width=0.44\textwidth]{figs/mg_testrmse.png}
	\caption{Train RMSE (left) and test RMSE (right) vs. epoch for the Mackey-Glass learning task.}
	\label{fig:mg_traintest}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.44\textwidth]{figs/mz_trainrmse.png}
	\includegraphics[width=0.44\textwidth]{figs/mz_testrmse.png}
	\caption{Train RMSE (left) and test RMSE (right) vs. epoch for the ENSO  learning task.}
	\label{fig:mz_traintest}
\end{figure}

\clearpage
\section{Tuning Parameters}

To tune our $\tau$-GRU, we use a non-exhaustive random search within the following plausible ranges for $\tau={5,\dots,200}$. We used Adam as our optimization algorithm for all of the experiments. For the synthetic data sets generated by the ENSO and Mackey-Glass system, we used learning rate of 0.01. For the other experiments we considered learning rates between 0.01 and 0.0005. We used dropout for the IMDB and Google12 task to avoid overfitting.

Table~\ref{tab:tuning} is listing the tuning parameters for the different tasks that we considered in this work.

\begin{table}[!h]
	\caption{Summary of tuning parameters.}
	\label{tab:tuning}
	\centering
	\scalebox{0.85}{
		\begin{tabular}{l c c c c c c c c }
			\toprule
			Name           &  d & lr  & $\tau$ & dropout & epochs \\
      		\midrule
			Adding Task $N=2000$ & 128 & 0.0026 & 900 & - & 200 \\
            Adding Task $N=5000$ & 128 & 0.002 & 2000 & - & 200 \\
   			\midrule
			IMDB  & 128 & 00012 & 1 & 0.04 & 30 \\
			\midrule
			HAR-2  & 128 & 0.00153 & 10 & - & 100 \\
			\midrule
			sMNIST  & 128 & 0.0018 & 50 & - & 60 \\
			\midrule
			psMNIST & 128 & 0.0055 & 65 & - & 80 \\
			\midrule
			sCIFAR & 128 & 0.0035 & 30 & - & 50  \\
            \midrule
			nCIFAR & 128 & 0.0022 & 965 & - & 50 \\
			\midrule
   			Google12 & 128 & 0.00089 & 5 & 0.03  &  60 \\
            \midrule
            ENSO & 16  & 0.01  & 20 & - & 400  \\
            \midrule
            Mackey-Glass system  & 16 & 0.01  & 10 & - &  400  \\
   \bottomrule
	\end{tabular}}
\end{table}

\paragraph{Sensitivity to Random Initialization.}

We evaluate our models for each tasks using 8 seeds. The maximum, minimum, average values, and standard deviations obtained for each task are tabulated in Table~\ref{tab:minmax}.

\begin{table}[h]
	\caption{Sensitivity to random initialization evaluated over 8 runs with different seeds.}
	\label{tab:minmax}
	\centering
	\scalebox{0.91}{
		\begin{tabular}{l c c c c c c c c}
			\toprule
			  Task     & Maximum & Minimum  & Average & standard dev. & d \\
			\midrule
            IMDB  & 88.7	& 86.2	 & 87.9 & 0.82 & 128 \\
            HAR-2  & 97.4	& 96.6 & 96.9 &	0.41 & 128 \\
			sMNIST  & 99.4 & 99.1 &	99.3 & 0.08 & 128 \\
            psMNIST  & 97.2 & 96.0 & 96.8 &	0.39 & 128 \\
            sCIFAR  & 74.9 & 72.65 & 73.54 & 0.90 & 128 \\
            nCIFAR  & 62.7 & 61.7 &	62.3 & 0.32 & 128 \\
            Google12 & 96.2 & 95.7 & 95.9 &	0.17 & 128 \\

			\bottomrule
	\end{tabular}}
\end{table}

\section{Introduction}

Recurrent neural networks (RNNs) and their variants are flexible gradient-based methods specially designed to model sequential data.
Models of this type can be viewed as dynamical systems whose temporal evolution is governed by a system of differential equations driven by an external input.
Indeed, there is a long-standing tradition to formulate continuous-time variants of RNNs~\cite{pineda1988dynamics}.
In this setting, the data are formulated in continuous-time, i.e., inputs are defined by the function ${\bf x} = {\bf x}(t) \in \mathbb{R}^p$ and targets are defined as ${\bf y} = {\bf y}(t) \in \mathbb{R}^q$.
In this way, one can, for instance, employ a nonautonomous ordinary differential equation (ODE) to model the dynamics of the hidden states ${\bf h}(t)\in\mathbb{R}^d$, where $t$  denotes continuous time, as
\begin{equation*}
     \frac{d {\bf h}(t)}{d t}\,\,\, = \,\,\, f({\bf h}(t),\, {\bf x}(t);\, \boldsymbol{\theta}).
\end{equation*}
Here, $f: \mathbb{R}^d\times  \mathbb{R}^p \rightarrow  \mathbb{R}^d$ is a function which is parameterized by a neural network (NN) with the learnable weights $\boldsymbol{\theta}$.
A prototypical choice for $f$ is the $\tanh$ recurrent unit:
\begin{equation*}
    f({\bf h}(t),\, {\bf x}(t); \boldsymbol{\theta}) := \tanh({\bf W} {\bf h}(t) + {\bf U} {\bf x}(t) + {\bf b}),
\end{equation*}
where ${\bf W} \in \mathbb{R}^{d\times d}$ denotes a hidden-to-hidden weight matrix, ${\bf U}\in \mathbb{R}^{d \times p}$ an input-to-hidden weight matrix, and ${\bf b}$ a bias term.
With this continuous-time formulation in hand, one can then use tools from dynamical systems theory to study the dynamical behavior of the model as well as to motivate mechanisms that can prevent rapidly diverging or converging dynamics.
For instance,~\cite{chang2018antisymmetricrnn}  proposed a parametrization of the hidden-to-hidden matrix as an antisymmetric matrix to ensure stable hidden state dynamics, and~\cite{erichson2020lipschitz} relaxed this idea to improve model expressivity.
More recently,~\cite{rusch2022long} has proposed an RNN architecture  based on a suitable time-discretization of a set of coupled multiscale ODEs.

\begin{figure*}[!t]
	\begin{CatchyBox}{$\tau$-GRU}\vspace{+0.3cm}
		\textbf{Continuous-time formulation of $\tau$-GRU:}\\
		\begin{minipage}[h]{0.95\linewidth
			\begin{align
			\label{eq:tdRNN_de}
			\frac{d \, {\bf h}(t)}{dt}
			= \underbrace{u({\bf h}(t),\, {\bf x}(t))}_\text{instantaneous dynamics} \, + \,\,\, \underbrace{\textcolor{wine}{a({\bf h}(t), {\bf x}(t))}  \odot \textcolor{darkcyan}{z({\bf h}(t-\tau),\, {\bf x}(t))}}_\text{weighted time-delayed feedback} \,\,\, - \,\,\, {\bf h}(t)
			\end{align}

		\end{minipage}\vspace{+0.3cm}

		\textbf{Discrete-time formulation of $\tau$-GRU:}\\
		\begin{minipage}[h]{0.55\linewidth
			\begin{equation}
			{\bf h}_{n+1} = (1-{\bf g}_n) \odot {\bf h}_n + {\bf g}_n \odot ({\bf u}_n +  \textcolor{wine}{{\bf a}_n} \odot \textcolor{darkcyan}{{\bf z}_n}) \label{eq:ourdRNN}
			\end{equation}
			%
			with
			%
			\vspace{-0.3cm}
			\begin{align}
			{\bf u}_n & = u({\bf h}_n, {\bf x}_n) := \text{tanh}({\bf W}_1 {\bf h}_n + {\bf U}_1 {\bf x}_n) \label{eq:3} \\
			\textcolor{darkcyan}{{\bf z}_n} & = \textcolor{darkcyan}{z({\bf h}_{l}, {\bf x}_n)} := \text{tanh}({\bf W}_2 {\bf h}_{l} + {\bf U}_2 {\bf x}_{n}) \\
			{\bf g}_n & = g({\bf h}_n, {\bf x}_n) :=   \text{sigmoid}({\bf W}_3 {\bf h}_n + {\bf U}_3 {\bf x}_n) \\
			\textcolor{wine}{{\bf a}_n} & = \textcolor{wine}{a({\bf h}_n, {\bf x}_n)} :=\text{sigmoid}({\bf W}_4 {\bf h}_{n} + {\bf U}_4 {\bf x}_{n}) \label{eq:6}
			\end{align}
			%
			\hfill
		\end{minipage}
		%
		\hfill
		\begin{minipage}[h]{.51\linewidth}\small
			\centering
			\begin{tabular}{l|c|c}
				input & ${\bf x}$ & $\mathbb{R}^{p}$\\\hline

				time index & $t$ & $\mathbb{R}$ 	 \\\hline
				time delay & $\tau$ & $\mathbb{R}$ 	 \\\hline
				hidden state & ${\bf h}$ & $\mathbb{R}^{d}$\\\hline
				hidden-to-hidden matrix & ${\bf W}_i$ & $\mathbb{R}^{d \times d}$\\\hline
				input-to-hidden matrix & ${\bf U}_i$ & $\mathbb{R}^{d\times p}$\\\hline
				decoder matrix & ${\bf V}$ & $\mathbb{R}^{q\times d}$\\\hline
			\end{tabular} \\
			\vspace{0.2cm}
			${\bf h}_n \approx {\bf h}(t_n)$, $t_n = n \Delta t$, $n=0,1,\dots$ \\
			\vspace{0.1cm}
			$l := n - \lfloor \tau/\Delta t \rfloor$
		\end{minipage}
	\end{CatchyBox}
\end{figure*}

In this work, we consider using input-driven delay differential equations (DDEs) to model the dynamics of the hidden states:
\begin{align*}
    \frac{d {\bf h}(t)}{d t} \,\,\, = \,\,\, f({\bf h}(t),\, {\bf h}(t-\tau),\, {\bf x}(t);\, \boldsymbol{\theta}),
\end{align*}
where $\tau$ is a constant that indicates the delay (i.e., time-lag). Here, the time derivative is described by a function $f: \mathbb{R}^d\times \mathbb{R}^d\times  \mathbb{R}^p \rightarrow  \mathbb{R}^d$ that explicitly depends on states from the past.
In prior work~\cite{lin1996learning}, it has been shown that delay units can greatly improve performance on long-term dependency problems \cite{pascanu2013difficulty}, i.e., problems for which the desired model output depends on inputs presented at times far in the past.

In more detail, we propose a novel continuous-time recurrent unit, given in Eq.~\eqref{eq:tdRNN_de}, that is composed of two parts:
(i) a component $u({\bf h}(t),{\bf x}(t))$ that explicitly models instantaneous dynamics; and (ii) a component $z({\bf h}(t-\tau), {\bf x}(t))$ that provides time-delayed feedback to account for non-instantaneous dynamics.
The feedback also helps to propagate gradient information more efficiently,  thus lessening the issue of vanishing gradients.
In addition, we introduce  $a({\bf h}(t),{\bf x}(t))$ to weight the importance of the  feedback component-wise, which helps to better model different time scales.
By considering a suitable time-discretization scheme of this continuous-time setup, we obtain a gated recurrent unit (GRU), given in Eq. \eqref{eq:ourdRNN}, which we call $\tau$-GRU. The individual parts are described by Eq.~\eqref{eq:3}-\eqref{eq:6}, where ${\bf g}_n$ and ${\bf a}_n$ resemble commonly used gating functions.

\paragraph{Main Contributions.}
Here are our main contributions.
\begin{itemize}
    \item {\bf Design.}
    We introduce a novel gated recurrent unit, which we call $\tau$-GRU, that incorporates a weighted time-delay feedback mechanism to lessen the vanishing gradients issue. This model is motivated by DDEs, and it is obtained by discretizing the continuous Eq. \eqref{eq:tdRNN_de}.

    \item {\bf Theory.}
    We show that the continuous-time $\tau$-GRU model has a unique well-defined solution (see Theorem~\ref{thm_exist2main}).
    Moreover, we provide intuition and analysis  to understand how the introduction of delays in $\tau$-GRU can act as a buffer to help lessen the vanishing gradients problem, thus improving the ability to retain information far in the past.
    See Proposition \ref{prop_delaymain} for a simplified setting and Proposition \ref{app_prop} for  $\tau$-GRU.

    \item {\bf Experiments.}
    We provide empirical results to demonstrate the superior performance of $\tau$-GRU, when compared to other RNN architectures, on a variety of benchmark tasks.
    In particular, we show that $\tau$-GRU converges faster during training and can achieve improved generalization performance. Moreover, we demonstrate that it provides favorable trade-offs between effectiveness in dealing with long-term dependencies and expressivity in the considered tasks.
    See Figure \ref{fig:intro_fig} for an illustration of this.

\end{itemize}

\begin{figure}[!t]
	\centering
	\includegraphics[width=0.45\textwidth]{figs/e_vs_l_overview.pdf
	\caption{Test accuracy for nCIFAR~\cite{chang2018antisymmetricrnn} versus Google-12~\cite{warden2018speech}.  nCIFAR requires a recurrent unit with long-term dependency capabilities, while Google-12 requires a highly expressive unit.  Our proposed $\tau$-GRU is able to improve performance on both tasks, relative to existing state-of-the-art alternatives, including LEM~\cite{rusch2022long}.}
	\label{fig:intro_fig}
\end{figure}

\section{Related Work}

Here, we discuss recent RNN advances that have been demonstrated to outperform classic architectures such as Long Short Term Memory (LSTM) networks~\cite{hochreiter1997long} and Gated Recurrent Units (GRUs)~\cite{cho2014properties}.
We also briefly discuss prior works on incorporating delays into neural architectures. (We omit a detailed discussion of other recent deep learning models for sequential problems that leverage dynamical system theory:~\cite{erichson2019physics,azencot2020forecasting,gu2021combining,smith2022simplified,hasani2022liquid}.) \\

\noindent
\textbf{Unitary and orthogonal RNN.} \
The seminal work by~\cite{arjovsky2016unitary} introduced a recurrent unit where the hidden weight matrix is constructed as the product of unitary matrices.
This enforces that the eigenvalues lie on the unit circle.
This, in turn, prevents vanishing and exploding gradients, thereby  enabling the learning of long-term dependencies.
However, such unitary RNNs suffer from limited expressivity, since the construction of the hidden matrix is restrictive~\cite{azencot2021differential}.
Work by~\cite{wisdom2016full} and~\cite{vorontsov2017orthogonality} partially addressed this issue by considering the Cayley transform on skew-symmetric matrices; and
work by~\cite{lezcano2019cheap,lezcano2019trivializations} leveraged skew-Hermitian matrices to parameterize the orthogonal group to improve expressiveness.
The expressiveness of RNNs has been further improved by considering nonnormal hidden matrices~\cite{kerg2019non}. \\

\noindent
\textbf{Continuous-time RNNs.} \
The recent work on Neural ODEs~\cite{chen2018neural} and variants~\cite{kidger2020neural,queiruga2021stateful,xia2021heavy,hasani2022closed} have motivated the formulation of several modern continuous-time RNNs, which are expressive and have good long-term memory.
The work by~\cite{chang2018antisymmetricrnn} used an antisymmetric matrix to parameterize the hidden-to-hidden matrix in order to obtain stable dynamics. This was later relaxed by~\cite{erichson2020lipschitz}. In~\cite{Kag2020RNNs}, a modified differential equation was considered, which allows one to update the hidden states based on the difference between predicted and previous states. Work by \cite{rusch2021coupled} demonstrated that long-term memory can be improved by modeling the hidden dynamics by a second-order system of ODEs, which models a coupled network of controlled forced and damped nonlinear oscillators. Another approach for improving long-term memory was motivated by a time-adaptive discretization of an ODE~\cite{kag2021time}.
The expressiveness of continuous-time RNNs has been further improved by introducing a suitable time-discretization of a set of multiscale ODEs~\cite{rusch2022long}.
It has also been shown that noise-injected RNNs can be viewed as discretizations of stochastic differential equations driven by input data~\cite{lim2021noisy}.
In this case, the noise can help to stabilize the hidden dynamics during training and improve robustness to input perturbations.  \\

\noindent
\textbf{Using delays in NNs.}
The idea of introducing delays into NNs goes back to  \cite{waibel1989phoneme, lang1990time}, where a time-delay NN was proposed to tackle the phoneme recognition problem.
Several works followed:
\cite{kim1998time} considered a time-delayed RNN model that is suitable for temporal correlations and prediction of chaotic and financial time series; and
delays were also incorporated into and studied within the nonlinear autoregressive  with exogenenous inputs (NARX) RNNs \cite{lin1996learning}.
More recently, \cite{zhu2021neural} introduced delay terms in Neural ODEs and demonstrated their outstanding approximation capacities.
In particular, the model of \cite{zhu2021neural} is able to learn delayed dynamics where the trajectories in the lower-dimensional phase space could be mutually intersected, while the Neural ODEs~\cite{chen2018neural} are not able to do so.

\section{Method}

In this section, we first provide an introduction to DDEs; then, we motivate the formulation of our DDE-based models, in continuous as well as discrete time; and, finally, we propose a weighted time-delay feedback architecture.

\paragraph{Notation.} $\odot$ denotes Hadamard product, $| v |$ denotes vector norm for the vector $v$, $\| A \|$ denotes operator norm for the matrix $A$, $\sigma$ and $\hat{\sigma}$ (or sigmoid) denote the $\tanh$ and sigmoid function, respectively; and $\lceil x \rceil$ and $\lfloor x \rfloor$ denote the ceiling function and floor function in $x$, respectively.

\subsection{Background on Delay Differential Equations}

DDEs are an important class of dynamical systems that arise in natural and engineering sciences~\cite{smith2011introduction, erneux2009applied, keane2017climate}. In these systems, a feedback term is introduced to adjust the system non-instantaneously, resulting in delays in time. In mathematical terms, the derivative of the system state depends explicitly on the past value of the state variable.

Here, we focus on DDEs with a single discrete delay
\begin{equation} \label{eq_gen_dde}
\dot{h} = F(t, h(t), h(t-\tau)),
\end{equation}
with $\tau > 0$, where $F$ is a continuous function. Due to the presence of the delay term, we  need to {\it specify an initial
function} which describes the behavior of the system prior to the initial time 0. For the DDE, it would be a function $\phi$ defined on $[-\tau, 0]$.  Hence, a DDE numerical solver must save all the information needed to approximate delayed~terms.

Instead of thinking the  solution of the DDE as consisting of a sequence of values of $h$ at increasing values of $t$, as one would do for ODEs, it is more fruitful to view it as a mapping of functions on the interval $[t -\tau,t]$ into functions on the interval $[t,t + \tau]$, i.e., as a sequence of functions  defined over a set of contiguous time intervals of length $\tau$.
Since the state of the system at time $t \geq  0$ must contain all the information necessary to determine the solution for future times $s \geq  t$, it should  contain the initial condition $\phi$.

More precisely, the DDE is a functional differential equation with the state space $C := C([-\tau, 0], \mathbb{R}^d)$. This state space is the Banach space of continuous functions from $[-\tau, 0]$ into $\mathbb{R}^d$, with the topology of uniform convergence. It is equipped with the norm $\|\phi \| := \sup \{ |\phi(\theta)| : \theta \in [-\tau, 0] \}$.
In contrast to the ODEs (with $\tau = 0)$ whose state space is finite-dimensional, DDEs are generally infinite-dimensional dynamical systems.
Various aspects of DDEs have been studied, including their solution properties \cite{hale2013introduction, asl2003analysis}, dynamics \cite{lepri1994high, baldi1994delays} and stability \cite{marcus1989stability, belair1993stability, liao2002delay, yang2014exponential, park2019dynamic}.

\subsection{Formulation of Continuous-Time $\tau$-GRUs}

The basic form of a time-delayed RNN is
\begin{align}\label{eq:simple_model}
\begin{split}
    \dot{{\bf h}} =\sigma({\bf W}_1 {\bf h}(t) + {\bf W}_2 {\bf h}(t-\tau)+ {\bf U} {\bf x}(t) ) - {\bf h}(t),
\end{split}
\end{align}
for $t \geq 0$,  and ${\bf h}(t) = 0$ for $t \in [-\tau, 0]$, with the output ${\bf y}(t) = {\bf V} {\bf h}(t)$. In this expression, ${\bf h} \in \RR^d$ denotes the hidden states, $f: \RR^d \times \RR^d \times \RR^{p} \to \RR^d$ is a nonlinear function, and $\sigma: \RR \to (-1,1)$ denotes the tanh  activation function applied component-wise. The matrices ${\bf W}_1, {\bf W}_2 \in \RR^{d \times d}$, ${\bf U} \in \RR^{d \times p}$ and ${\bf V} \in \RR^{q \times d}$ are learnable parameters, and $\tau \geq 0$ denotes the discrete time-lag.
For notational brevity, we omit the bias term here, assuming it is included in ${\bf W}_1$.

It is important to equip RNNs with mechanism to better represent a large number of scales, as discussed by~ \cite{tallec2018can} and more recently by~\cite{rusch2022long}.
Therefore, we follow \cite{tallec2018can} and consider a time warping function $c:\mathbb{R}^d\rightarrow \mathbb{R}^d$ which we define to be a parametric function  $c(t)$. Using the reasoning in~\cite{tallec2018can}, and denoting $t_\tau := t-\tau$,  we can formulate the following continuous-time delay recurrent unit
\begin{align*}\label{eq:simple_model}

    \dot{{\bf h}} = \frac{d c(t)}{d t}\left[ \sigma({\bf W}_1 {\bf h}(t) + {\bf W}_2 {\bf h}(t_\tau) + {\bf U}_1 {\bf x}(t)) - {\bf h}(t)\right].
\end{align*}

Now, we need a learnable function to model $\frac{d c(t)}{d t}$. A natural choice is to consider a standard gating function, which is a universal approximator, taking the form
\begin{equation}
    \frac{d c(t)}{d t} = \hat{\sigma}({\bf W}_3 {\bf h}_t + {\bf U}_3 {\bf x}_t) =: {\bf g}(t),
\end{equation}
where ${\bf W}_3 \in \RR^{d \times d}$ and  ${\bf U}_3 \in \RR^{d \times p}$ are learnable parameters, and  where $\hat{\sigma}: \RR \to (0,1)$ is the component-wise sigmoid function.

\subsection{From Continuous to Discrete-Time $\tau$-GRUs}

To learn the weights of the recurrent unit, a numerical integration scheme can be used to discretize the continuous model. Specifically, we discretize the time as $t_n = n \Delta t$ for $n = -\lfloor \tau/\Delta t \rfloor, \dots, -1, 0, 1, \dots$, and approximate the solution $({\bf h}(t))$ to Eq. \eqref{eq:simple_model} by the sequence $({\bf h}_n = {\bf h}(t_n))$, given by ${\bf h}_n = 0$ for $n = -\lfloor \tau/\Delta t \rfloor, \dots, 0$, and
\begin{align}
    {\bf h}_{n+1} &= {\bf h}_n + \int_{t_n}^{t_n+\Delta t} f({\bf h}(s),\, {\bf h}(s-\tau), {\bf x}(s)) \,\mathrm{d}s  \\
    &\approx {\bf h}_n + \mathtt{scheme}[f,\,{\bf h}_n,\,{\bf h}_{l}, \Delta t],
\end{align}
for $n=0,1,\dots$, where the subscript $n$ denotes discrete time indices, $l := n - \lfloor \tau/\Delta t \rfloor$,  and $\Delta t$ represents the time difference between a pair of consecutive elements in the input sequence.
In addition, $\mathtt{scheme}$ refers to a numerical integration scheme whose application yields an approximate solution for the integral.
Using the explicit forward Euler scheme and choosing $\Delta t = 1$ gives:
\begin{equation} \label{eq_simpleDRNN}
   {\bf h}_{n+1} = (1-{\bf g}_n) \odot {\bf h}_n + {\bf g}_n \odot \sigma({\bf W}_1 {\bf h}_n + {\bf W}_2 {\bf h}_{l} + {\bf U} {\bf x}_n).
\end{equation}
Note that this discretization corresponds to the leaky-integrator described by~\cite{jaeger2007optimization}.

It can be shown that \eqref{eq_simpleDRNN} is a universal approximator of a large class of open dynamical systems with delay (see Theorem \ref{thm_uat}).
However, the performance of this architecture is not able to outperform state-of-the-art RNN architectures on a number of tasks. To improve the performance, we propose a modified gated recurrent unit next.

\subsection {Discrete-Time $\tau$-GRUs with a Weighted Time-Delay Feedback Architecture}

In this work, we propose to model the hidden dynamics using a mixture of a standard recurrent unit and a delay recurrent unit. To this end, we replace the $\sigma$ in Eq. \eqref{eq_simpleDRNN} by
\begin{equation} \label{eq_motivate}
    {\bf u}_n + {\bf a}_n \odot {\bf z}_n,
\end{equation}
so that we yield a new GRU that takes the form
\begin{equation}
   {\bf h}_{n+1} = (1-{\bf g}_n) \odot {\bf h}_n + {\bf g}_n \odot \left( {\bf u}_n + {\bf a}_n \odot {\bf z}_n \right).
\end{equation}
Here ${\bf u}_n$ describes the standard recurrent unit $${\bf u}_n=\text{tanh}({\bf W}_1 {\bf h}_n + {\bf U}_1 {\bf x}_n),$$ and ${\bf z}_n$ describes the delay recurrent unit $${\bf z}_n  = \text{tanh}({\bf W}_2 {\bf h}_{l} + {\bf U}_2 {\bf x}_{n}).$$
Further, the gate ${\bf g}_n$ is a learnable vector-valued coefficient
$${\bf g}_n  =\text{sigmoid}({\bf W}_3 {\bf h}_{n} + {\bf U}_3 {\bf x}_{n}).$$
The weighting term ${\bf a}_n$ is also a vector-valued coefficient
$${\bf a}_n  =\text{sigmoid}({\bf W}_4 {\bf h}_{n} + {\bf U}_4 {\bf x}_{n}),$$
with the learnable parameters ${\bf W}_4 \in \RR^{d \times d}$ and ${\bf U}_4  \in \RR^{d \times p}$ that weights the importance of the time-delay feedback component-wise for the task on hand.
From the design point of view, Eq. \eqref{eq_motivate} can be motivated by the sigmoidal coupling used in Hodgkin-Huxley type neural models (see Eq. (1)-(2) and Eq. (4) in \cite{campbell2007time}).

\section{Theory}

In this section, we define the notion of  solution for DDEs and show that the continuous-time $\tau$-GRU  has a unique solution. Moreover, we provide intuition and analysis to understand how the delay mechanism can help improving long-term dependencies.

\subsection{Existence and Uniqueness of Solution for Continuous-Time $\tau$-GRUs}

Since we must know $h(t + \theta)$, $\theta \in [-\tau, 0]$ in order to determine the solution of the DDE \eqref{eq_gen_dde} for $s > t$, we call the state of the dynamical system at time $t$ the element of $C$ which we denote as $h_t$,  defined as
$h_t(\theta) := h(t+\theta)$ for $\theta \in [-\tau, 0]$.
The  trajectory of the solution can thus be viewed as the curve $t \to h_t$ in the state space $C$. In general, DDEs can be formulated as the following initial value problem (IVP) for the nonautonomous system \cite{hale2013introduction}:
\begin{equation} \label{eq_genddemain}
    \dot{h}(t) = f(t, h_t), \ t \geq t_0,
\end{equation}
where $h_{t_0} = \phi \in C$ for some initial time $t_0 \in \mathbb{R}$, and $f: \mathbb{R} \times C \to \mathbb{R}^d$ is a given continuous function. The above equation describes a general type of systems, including ODEs ($\tau = 0$) and DDEs of the form $\dot{h}(t) = g(t, h(t), h(t-\tau))$ for some continuous function $g$.

We say that a function $h$ is a solution of Eq. (\ref{eq_genddemain}) on $[t_0 - \tau, t_0 + A]$ if there exists  $t_0 \in \mathbb{R}$ and $A > 0$ such that $h \in C([t_0 - \tau, t_0 + A), \mathbb{R}^d)$, $(t, h_t) \in \mathbb{R} \times C$ and $h(t)$
satisfies Eq. (\ref{eq_genddemain}) for $t \in [t_0, t_0 + A)$. It can be shown that (see, for instance, Lemma 1.1 in \cite{hale2013introduction}) if $f(t,\phi)$ is continuous, then finding a solution of Eq. (\ref{eq_genddemain}) is equivalent to solving the integral equation: $h_{t_0} = \phi$,
\begin{equation}
    h(t) = \phi(0) + \int_{t_0}^t f(s, h_s) \  \rm{d}s, \ t \geq t_0.
\end{equation}

We now provide existence and uniqueness result for the continuous-time $\tau$-GRU model, assuming that the input $x$ is  continuous in $t$. Defining the state $h_t \in C$ as $h_t(\theta) := h(t+\theta)$ for $\theta \in [-\tau, 0]$ as before,  the DDE describing the $\tau$-GRU model can be formulated as the following IVP:
\begin{equation} \label{eq_gendde}
    \dot{h} = - h(t) + u(t, h(t)) + a(t, h(t)) \odot z(t, h_t), \ t \geq t_0,
\end{equation}
and $h_{t_0} = \phi \in C$ for some initial time $t_0 \in \mathbb{R}$, with the dependence on $x(t)$ viewed as dependence on $t$.
By applying Theorem 3.7 in \cite{smith2011introduction}, we can obtain the following result.
See App.~\ref{sect:appB} for a proof of this theorem.

\begin{theorem}[Existence and uniqueness of solution for continuous-time $\tau$-GRU] \label{thm_exist2main}
Let $t_0 \in \RR$ and $\phi \in C$ be given. There exists a unique solution $h(t) = h(t, \phi)$ of Eq. \eqref{eq:tdRNN_de}, defined on $[t_0 - \tau, t_0 + A]$ for any $A > 0$. In particular, the solution exists for all $t \geq t_0$, and
\begin{equation}
    \| h_t(\phi) - h_t(\psi) \| \leq \| \phi - \psi \| e^{K(t-t_0)},
\end{equation}
for all $t \geq t_0$, where $K = 1 + \|W_1\| +  \|W_2\| + \|W_4\|/4$.
\end{theorem}

Theorem \ref{thm_exist2main} guarantees that the continuous-time $\tau$-GRU model, as a functional differential equation, has a well-defined unique solution that does not blow up in finite time.

\subsection{The Delay Mechanism in $\tau$-GRUs Can Help Improve Long-Term Dependencies}

RNNs suffer from the problem of vanishing and exploding gradients, leading to the problem of long-term dependencies.
While the gating mechanisms could mitigate the problem to some extent, the delays introduced in $\tau$-GRUs can further
help reduce the sensitivity to long-term dependencies.
To understand the reason for this, we consider how gradients are computed using the backpropagation through time (BPTT) algorithm~\cite{pascanu2013difficulty}.
BPTT involves the two stages of unfolding the network in time and backpropagating the training error through the unfolded network. When $\tau$-GRUs are unfolded in time, the delays in the hidden state will appear as jump-ahead connections (buffers) in the unfolded network. These buffers provide a shorter path for propagating gradient information, and therefore reducing the sensitivity of the network to long-term dependencies. Such intuition is also used to explain the behavior of the NARX RNNs in \cite{lin1996learning}.

We now make this intuition precise in the following simplified setting.
See App.~\ref{sect:appC} for a proof of this proposition.
We also provide analogous results (bounds for the gradient norm) and discussions for $\tau$-GRU (in App.~\ref{app:gradbound}, see Proposition \ref{app_prop}).

\begin{proposition} \label{prop_delaymain}
Consider the linear time-delayed RNN  whose hidden states are described by the update equation:
\begin{equation}
    h_{n+1} = Ah_n + Bh_{n-m} + Cu_n, \ n=0,1,\dots,
\end{equation}
and $h_n = 0$ for $n=-m, -m+1, \dots, 0$ with $m > 0$.
Then, assuming that $A$ and $B$ commute, we have:
\begin{equation}
    \frac{\partial h_{n+1}}{\partial u_i} = A^{n-i} C,
\end{equation}
for $n=0,1, \dots, m$, $i=0,\dots, n$, and
\begin{align}
    \frac{\partial h_{m+1+j}}{\partial u_i} &= A^{m+j-i} C + \delta_{i,j-1} BC + 2 \delta_{i,j-2} ABC \nonumber \\
    &\ \ \ \ + 3 \delta_{i,j-3} A^2 B C + \cdots + j \delta_{i,0} A^{j-1} B C,
\end{align}
for $j = 1,2,\dots, m+1$, $i=0,1,\dots, m+j$, where $\delta_{i,j}$ denotes the Kronecker delta.
\end{proposition}

We remark that the commutativity assumption is not necessary. It is used here only to simplify the expression for the gradients. Analogous formula for the gradients can be derived without such assumption, at the cost of displaying more complicated formulae.

From Proposition \ref{prop_delaymain}, we see that the presence of the delay allows the model to place more emphasis on the gradients due to input information in the past (as can be seen in Eq. \eqref{eq_linear} in the proof of the proposition, where additional terms dependent on $B$ appear in the coefficients in front of the past inputs). In particular, if $\|A\| < 1$ and $B=0$, then the gradients decay exponentially as $i$ becomes large. Introducing the delay term ($B\neq 0$) makes the rates at which this exponential decays lower by perturbing the gradients of hidden states  (dependent on the delay parameter $m$)  with respect to the past inputs with  nonzero values.

Similar qualitative conclusion can also be drawn for our $\tau$-GRU (see Proposition \ref{app_prop} and the discussions in App. \ref{app:gradbound}). Therefore, one  expects that these networks would be able to more effectively deal with long-term dependencies than the counterpart models without delays.

\section{Experimental Results}
\label{sect:exp}

In this section, we consider several benchmark datasets to demonstrate the performance of our proposed $\tau$-GRU.
We use standard protocols for training and initialization, and validation sets for parameter tuning (see App.~\ref{sect:appD} for~details).

\begin{figure}[!b]
	\centering
	\begin{subfigure}[b]{0.48\textwidth}
		\centering
		\includegraphics[width=0.99\textwidth]{figs/adding_task_2000.pdf}
		\caption{Sequence length $N=2000$.}
		\label{fig:adding_task_2000}
	\end{subfigure
 ~
	\begin{subfigure}[b]{0.48\textwidth}
		\centering
		\includegraphics[width=0.99\textwidth]{figs/adding_task_5000.pdf}
		\caption{Sequence length $N=5000$.}
		\label{fig:adding_task_5000}
	\end{subfigure}

	\caption{Results for the adding task on very long sequences.}
	\label{fig:adding_task}
\end{figure

\subsection{The Adding Task}
The adding task is a classic problem for testing the ability of models to learn long-term dependencies, originally proposed by~\cite{hochreiter1997long}.
The inputs of this problem are composed of two stacked random vectors $u$ and $v$ of length $N$. The elements of $u$ are drawn from the uniform distribution $\mathcal{U}(0,1)$. The vector $v$ has two non-zero elements (both set to 1), one placed in a random location $i$ sampled from the index set $i\in\{1,\dots,\lfloor\frac{N}{2}\rfloor\}$, and the other element is placed at location $j$ sampled from the index set $j\in\{\lceil\frac{N}{2}\rceil,\dots,N\}$. The target value for each sequence is constructed as the sum $\sum(u\odot v)$, i.e., the sum of the two elements in $u$ that correspond to the the non-negative entries $v$.
Here, we follow the work by~\cite{rusch2022long} and consider two challenging settings with very long input sequences $N=\{2000,5000\}$.

Figure~\ref{fig:adding_task} shows results for our $\tau$-GRU and several state-of-the-art RNN models which are designed to solve long-term dependencies tasks, such as LEM~\cite{rusch2022long}, coRNN~\cite{rusch2021coupled}, DTRIV$_\infty$~\cite{lezcano2019trivializations}, fastGRNN~\cite{kusupati2018fastgrnn}, and LSTM with chrono initialization~\cite{tallec2018can}. It can be seen that the DTRIV$_\infty$ and fastGRNN are performing poorly in both cases. In contrast, our $\tau$-GRU shows a favorable performance, i.e., $\tau$-GRU is converging faster, and it achieves lower mean squared errors than other methods.

\subsection{Sentiment Analysis: IMDB}
Next, we consider the IMDB dataset~\cite{maas2011learning} to study the expressiveness of our proposed model on a sentiment analysis task. The aim of this task is to predict whether a movie review is positive or negative. This dataset is composed of $50$k movie reviews, with an average length of 240 words per review. Each review is annotated by a label that indicates a positive, or negative sentiment. The dataset is split evenly into a training and test set, so that both sets contain $25$k reviews. We use 15\% of the training data for validation. Further, we use standard preprocessing schemes, following ~\cite{pennington2014glove}, to restrict the dictionary to $25$k words, and to embed the data with a pretrained GloVe model~\cite{pennington2014glove}.

Table~\ref{tab:results_imbd} shows that our $\tau$-GRU achieves a substantially higher test accuracy than LSTM, and GRU models.
Further, $\tau$-GRU is also able to outperform the continuous-time coRNN~\cite{rusch2021coupled} and the highly expressive LEM~\cite{rusch2022long}.

\begin{table}[!h]
	\caption{Results for IMDB sentiment analysis task.}
	\label{tab:results_imbd}
	\centering

	\scalebox{0.9}{
		\begin{tabular}{l c  c ccccccc}
			\toprule
			Model & Test Accuracy ($\%$) &  \# units & \# param \\
			\midrule
            LSTM~\cite{campos2018skip} & 86.8 & 128 & {220k} \\
            Skip LSTM~\cite{campos2018skip} & 86.6 & 128 & {220k} \\
            GRU~\cite{campos2018skip} & 86.2 & 128 & 164k \\
            Skip GRU~\cite{campos2018skip} & 86.6 & 128 & 164k \\
            ReLU GRU~\cite{dey2017gate} & 84.8 & 128 & 99k \\
            coRNN~\cite{rusch2021coupled} & 87.4 & 128 & 46k \\
            LEM & 88.1 & 128 & {220k} \\
			\midrule
			\textbf{$\tau$-GRU (ours)} & \textbf{88.6} & 128 & {220k} \\
			\bottomrule
	\end{tabular}}
\end{table}

\begin{table}[!b]
	\caption{Results for HAR2 task.}
	\label{tab:results_har2}
	\centering

	\scalebox{0.9}{
		\begin{tabular}{l c  c ccccccc}
			\toprule
			Model & Test Accuracy ($\%$) &  \# units & \# param \\
			\midrule
        GRU~\cite{kusupati2018fastgrnn} & 93.6 &  75 &   {19k}\\
        LSTM~\cite{Kag2020RNNs} & 93.7 &  64 &   {19k}\\
        FastRNN~\cite{kusupati2018fastgrnn} &  94.5 &  80 &  7k\\
        FastGRNN~\cite{kusupati2018fastgrnn} &  95.6 &  80 &  7k\\
        AsymRNN~\cite{Kag2020RNNs} &  93.2 &  120 & 8k\\
        iRNN~\cite{Kag2020RNNs} &  96.4 &  64 &  4k\\
        DIRNN~\cite{zhang2021deep} & 96.5 & 64 & -\\
        coRNN~\cite{rusch2021coupled} &  97.2 &  64 &  9k\\
        LipschitzRNN & 95.4 & 64 & 9k\\
        LEM & 97.1  &  64 &   {19k}\\
		\midrule
		\textbf{$\tau$-GRU (ours)} & \textbf{97.4} & 64 & {19k} \\
			\bottomrule
	\end{tabular}}
\end{table}

\begin{table*}[!t]
	\caption{Test accuracies on sMNIST, and psMNIST.}
	\label{tab:image_mnist}
	\centering
		\scalebox{0.85}{
	\begin{tabular}{lcccc|cccc}
		\toprule

		Model& sMNIST & psMNIST  &  \# units & \# parameters & sCIFAR & nCIFAR  &  \# units & \# parameters \\
		\midrule
		LSTM~\cite{kag2021time} &  97.8 & 92.6 & 128 & {68k} & 59.7 & 11.6 & 128 & 69k / 117k \\
		r-LSTM~\cite{trinh2018learning} & 98.4  & 95.2 & - & 100K & 72.2 & - & - &  101k / -\\
		chrono-LSTM~\cite{rusch2022long} & 98.9 & 94.6  & 128 & {68k} & - & 55.9 & 128 & - / 116k\\
		Antisymmetric RNN~\cite{chang2018antisymmetricrnn} &  98.0 & 95.8  & 128 & 10k & 62.2 & 54.7 & 256 & 37k / 37k\\
		Lipschitz RNN~\cite{erichson2020lipschitz} & 99.4 & 96.3 & 128 & 34k & 64.2 & 59.0 & 256 &  134k / 158k\\
		expRNN~\cite{lezcano2019cheap} &  98.4 &  96.2&  360 & {68k} & - & 49.0  & 128 & - / 47k\\
		iRNN~\cite{Kag2020RNNs} & 98.1 & 95.6 & 128 & 8k & - & 54.5 & 128 & - / 12k \\
		TARNN~\cite{kag2021time} & 98.9  & 97.1 & 128 & {68k} & - & 59.1 & 128 & - /  100K \\
		coRNN~\cite{rusch2021coupled} & 99.3 & {96.6}  & 128  & 34k & - & 59.0 & 128 & - / 46k\\
		{LEM}~\cite{rusch2022long} & \textbf{99.5} & {96.6}  & 128 & {68k}& - & 60.5 & 128 & - / 117k\\
		\midrule
		Simple delay GRU (Eq.~\eqref{eq_simpleDRNN}) & 98.7  &  94.1 & 128 & 51k & 57.2 & 59.8 & 128  & 52k / 75k  \\

		\textbf{$\tau$-GRU (ours)} & {99.4} & \textbf{97.3}  & 128 & {68k} & \textbf{74.9} & \textbf{62.2} & 128 & 69k / 117k \\
		\bottomrule
	\end{tabular}}
\end{table*}

\subsection{Human Activity Recognition: HAR-2}

Here, we study the performance of our model for human activity recognition using the HAR dataset provided by~\cite{anguita2012human}. This dataset
consists of measurements by an accelerometer and gyroscope on a Samsung Galaxy S3 smartphone which are tracking six activities for
30 volunteers within an age bracket of 19-48 years. For learning, the sequences are divided into shorter sequences of length $N=128$, and the raw measurements are summarized by 9 features per time step. Kusupati et al.~\cite{kusupati2018fastgrnn} proposed the HAR-2 dataset which groups the activities into two categories \{Sitting, Laying, Walking Upstairs\} and \{Standing, Walking, Walking Downstairs\}. We use $7,352$ sequences for training, $900$ for validation and $2,947$ for testing.

Table~\ref{tab:results_har2} shows that our $\tau$-GRU is able to outperform traditional gated architectures on this task. The most competitive model is coRNN~\cite{rusch2021coupled}, which achieves a test accuracy of $97.2\%$ with just $9$k parameters. LEM~\cite{rusch2022long} achieves $97.1\%$ with the the same number of parameters as our $\tau$-GRU.

\subsection{Sequential Image Classification}

Next, we consider four sequential variations of the MNIST~\cite{lecun1998gradient} and CIFAR-10~\cite{CIFAR10} image classification datasets. These sequence classification tasks aim to evaluate the capability of RNNs to learn long-term dependencies.

The sequential MNIST (sMNIST) task, originally proposed by~\cite{le2015simple}, is sequentially presenting the $N=784$ pixels of each thumbnail to the recurrent unit.
The final hidden state is used to predict the class membership probability of the flattened image.
A variation of the sMNIST task is the permuted sMNIST task (psMNIST), which presents the model with a fixed random permutation of the pixel-by-pixel sequence. This task removes any natural patterns in the sequence and requires that models can learn long-term dependencies between pixels that are possibly far apart.

Since the standard sMNIST task has essentially been solved by state-of-the-art RNNs, \cite{chang2018antisymmetricrnn} has proposed to consider the sequential CIFAR-10 (sCIFAR) task instead. This task is more challenging due to the increased sequence length, $N=1024$, of the flattened input images. Each element of the sequence is a 3-dimensional vector that contains the pixels for each color channel. To solve this task, models require to have long-term memory and sufficient~expressivity.

\begin{figure}[!b]
	\centering

	\includegraphics[width=0.48\textwidth]{figs/psmnist_acc.pdf}

	\caption{Test accuracy on permuted sequential MNIST as a function of the number of epochs. Our $\tau$-GRU requires substantially fewer number of epochs to reach peak performance.}
	\label{fig:psmnist_acc}
\end{figure}

Furthermore, \cite{chang2018antisymmetricrnn} has also proposed a noise-padded CIFAR-10 (nCIFAR) task, which requires that the RNN has the ability to memorize information from far in the past, and be able to suppress noisy segments that contain no relevant information. Specifically, we construct a sequence of length $N=1000$ where each element is a $96$-dimensional vector. The first $32$ two elements are the rows of an image, where the channels are stacked.
The remaining $968$ elements of the sequence are random vectors drawn from the standard normal distribution.

Table~\ref{tab:image_mnist} shows results for the four described tasks. Our $\tau$-GRU is competitive and able to outperform other models on the psMNIST, sCIFAR and nCIFAR task. The proposed weighted time-delay feedback mechanisms demonstrates a clear advantage in particular for the CIFAR tasks.

Figure~\ref{fig:psmnist_acc} shows that our $\tau$-GRU is converging faster than other recently introduced continuous-time models, such as LEM~\cite{rusch2022long}, coRNN~\cite{rusch2021coupled}, and LipschitzRNN~\cite{erichson2020lipschitz}.
This could help reduce training times when using these~methods.

\subsection{Ablation Study using psMNIST}

We use the psMNIST task to perform an ablation study. To do so, we consider the following model
$${\bf h}_{t+1} = (1-{\bf g}_t) \odot {\bf h}_t + {\bf g}_t \odot (\beta \cdot {\bf u}_t + \alpha \cdot {\bf a}_t \odot {\bf z}_t)$$
where $\alpha\in [0,1]$ and $\beta\in [0,1]$ are constants that can be used to control the effect of different components. We are interested in the cases where $\alpha$ and $\beta$ are either 0 or 1, i.e., a component is switched off or on.

Table~\ref{table:ablation} shows the results for different ablation configurations. By setting $\alpha=0$ we yield a simple gated RNN.
Second, for $\beta=0$, we yield a $\tau$-GRU without instantaneous dynamics. Third, we show how different values of $\tau$ affect the performance. Setting $\tau=0$ leads to a $\tau$-GRU without time-delay feedback.
We also show that a model without the weighting function ${\bf a}_t$ is not able to achieve peak performance.

\begin{table}[!h]

	\caption{Ablation study.}
	\label{table:ablation}
	\centering

	\scalebox{0.9}{
		\begin{tabular}{l c  c  ccccccc}
			\toprule
			Model & $\alpha$ &  $\beta$ & $\tau$ & ${\bf a}_t$ & Test Accuracy (\%) \\
			\midrule
			ablation & 0 & 1 & - & yes & 94.6 \\
			ablation & 1 & 0 & 65 & yes &94.9 \\
			\midrule
			ablation & 1 & 1 & 0 & yes & 95.1 \\
			ablation & 1 & 1 & 20 & yes & 96.4 \\
			ablation & 1 & 1 & 65 & no & 96.8 \\
			\midrule
			\textbf{$\tau$-GRU (ours)} & 1 & 1 & 65 & yes &\textbf{97.3}\\
			\bottomrule
	\end{tabular}}
\end{table}

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.48\textwidth]{figs/tau_ablation.pdf}

	\caption{Sensitivity analysis of $\tau$-GRU on psMNIST.  The light green envelopes represent $\pm 1$ standard
    deviation around the mean.}
	\label{fig:tau_ablation_main}
\end{figure}

Figure~\ref{fig:tau_ablation_main} further investigates the effect of $\tau$. The performance of $\tau$-GRU is increasing as a function of $\tau$ and peaking around $\tau=65$.
It can be seen that the performance in the range $50-150$ is relatively constant for this task. Thus, the model is relatively insensitive as long as $\tau$ is sufficiently large, but not too large. The performance is starting to decrease for $\tau>150$.  Since $\tau$ takes discrete values, tuning is easier as compared to continuous tuning parameters, for instance, parameters used by LEM~\cite{rusch2022long}, coRNN~\cite{rusch2021coupled}, or LipschitzRNN~\cite{erichson2020lipschitz}.

\subsection{Learning Climate Dynamics}

We consider  the task of learning the dynamics of the DDE model for El Ni\~{n}o Southern Oscillation (ENSO)  of \cite{falkena2019derivation} (see Eq. (46) there). It models the sea surface temperature $T$ in the eastern Pacific Ocean, and is described by:
\begin{equation}
    \dot{T} = T-T^3 - c T(t-\delta) (1-\gamma T^2(t-\delta)), \ t > \delta,
\end{equation}
where $\gamma < 1$, $c > 0$, with $T(t)$ satisfying $\dot{T} = T-T^3 - c T(0) (1-\gamma T(0)^2)$ with $T(0) \sim \rm{Unif}(0,1)$ for $t \in [0, \delta]$.   For data generation, we follow \cite{falkena2019derivation}, and choose $c = 0.93$, $\gamma = 0.49$ and $\delta = 4.8$. We use the classical Runge-Kutta  method (RK4) to numerically integrate the system from $t=0$ to $t=400$ using a step-size of 0.1. The training and testing samples are the time series (of length 2000) generated by the RK4 scheme on the interval $[200,400]$ for different realizations of $T(0)$.

Table \ref{tab:results_ENSO} shows that our model (with $\alpha = \beta = 1$ and using $\tau = 20$) is more effective in learning the ENSO dynamics when compared to other RNN architectures. We also see that the predictive performance deteriorates without using appropriate combination of standard and delay recurrent units (setting either $\alpha$ or $\beta$ to zero).

\begin{table}[!h]
	\caption{Results for the ENSO model prediction.}
	\label{tab:results_ENSO}
	\centering
	\scalebox{0.9}{
		\begin{tabular}{l c  c  ccccccc}
			\toprule
			Model & MSE ($\times 10^{-2}$) &  \# units & \# parameters \\
			\midrule
			Vanilla RNN & 0.45 & 16 & 0.3k \\
			LSTM & 0.92 & 16 & 1.2k \\
			GRU & 0.53 & 16 & 0.9k \\
			Lipschitz RNN & 10.6 & 16 & 0.6k \\
			coRNN & 4.00 & 16 & 0.6k \\
			LEM & 0.31 & 16 & 1.2k \\
			\midrule
		  ablation ($\alpha = 0$) & 0.31 & 16 & 0.6k \\
		    ablation ($\beta = 0$) & 0.38 & 16 & 0.9k \\
			\midrule
			\textbf{$\tau$-GRU (ours)} & \textbf{0.17} & 16 & 1.2k \\
    	\bottomrule
	\end{tabular}}
\end{table}

\section{Conclusion and Future Work}
Starting from a continuous-time formulation, we derive a discrete-time gated recurrent unit with delay, $\tau$-GRU. We also provide intuition and analysis to understand how the  delay can act as a buffer to improve modeling of long-term dependencies. Importantly, we demonstrate the superior performance of $\tau$-GRU in several challenging sequential learning tasks.

We now discuss some future directions. On the one hand, using multiple delays could lead to improved models, and therefore it is of interest to study the extension of $\tau$-GRU to include suitable distributed delay mechanism. On the other hand, achieving model robustness with respect to adversarial perturbations  and common corruptions is critical for sensitive applications, but it is largely unexplored in the sequential learning setting. It has been shown in \cite{lim2021noisy, lim2021nfm, erichson2022noisymix} that noise injection can be effective in improving robustness. Therefore, it is also of interest to consider and study noise-injected versions of $\tau$-GRU to improve the trade-offs between accuracy and  robustness.

\section*{Acknowledgments}
N. B. Erichson would like to acknowledge IARPA (contract
W911NF20C0035).
S. H. Lim would like to acknowledge the WINQ Fellowship, the Swedish Research Council (VR/2021-03648), and the computational resources  provided by the Swedish National Infrastructure
for Computing (SNIC)  at  Chalmers Centre for Computational Science and Engineering (C3SE)  partially funded by the Swedish Research Council through grant agreement no. 2018-05973.
M. W. Mahoney would also like to acknowledge NSF, and ONR for providing partial support of this work. Our conclusions do not necessarily
reflect the position or the policy of our sponsors, and no official endorsement should be inferred.

{\normalsize
\bibliographystyle{ieee_fullname}

Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)