On two-generator subgroups in SL_2(Z), SL_2(Q), and SL_2(R)

Abstract: We consider what some authors call 'parabolic Möbius subgroups' of matrices over Z, Q, and R and focus on the membership problem in these subgroups and complexity of relevant algorithms.

\section{Introduction: two theorems of Sanov}

Denote  $A(k) = \left(
 \begin{array}{cc} 1 & k \\ 0 & 1 \end{array} \right) , \hskip .2cm B(k) = \left(
 \begin{array}{cc} 1 & 0 \\ k & 1 \end{array} \right).$ In an old
paper \cite{Sanov}, I. N. Sanov proved two simple yet remarkable
theorems:

\begin{theorem}\label{th1}
The subgroup of $SL_2(\mathbb{Z})$ generated by $A(2)$ and $B(2)$ is free.
\end{theorem}

\begin{theorem}\label{th2}
The subgroup of $SL_2(\mathbb{Z})$ generated by $A(2)$ and $B(2)$ consists
of {\it all} matrices of the form  $\left(
\begin{array}{cc} 1+4n_1 & 2n_2 \\ 2n_3 & 1+4n_4\end{array}
\right)$ with determinant 1, where all $n_i$ are arbitrary integers.
\end{theorem}

These two theorems together yield yet another proof of the fact that
the group $SL_2(\mathbb{Z})$ is virtually free. This is because the group of
all invertible matrices of the form  $\left(
\begin{array}{cc} 1+4n_1 & 2n_2 \\ 2n_3 & 1+4n_4\end{array}
\right)$  obviously has finite index in $SL_2(\mathbb{Z})$. Thus, we have:

\begin{corollary}\label{corollary1}
The group $SL_2(\mathbb{Z})$ is virtually free.
\end{corollary}

There is another interesting corollary of Theorem \ref{th2}:

\begin{corollary}\label{corollary2}
The membership problem in the subgroup of $SL_2(\mathbb{Z})$ generated by
$A(2)$ and $B(2)$ is solvable in constant time.
\end{corollary}

We note that this is, to the best of our knowledge, the only example
of a natural (and nontrivial) algorithmic problem in group theory
solvable in constant time. In fact, even problems solvable in
sublinear time are very rare, see \cite{Sublinear}, and in those
that are, one can typically get either ``yes" or ``no" answer in
sublinear time, but not both. Complexity of an input in our case is
the ``size" of a given matrix, i.e., the sum of the absolute values
of its entries. In light of Theorem \ref{th2}, deciding whether or
not a given matrix from $SL_2(\mathbb{Z})$ belongs to the subgroup generated
by $A(2)$ and $B(2)$ boils down to looking at residues modulo 2 or 4
of the entries. The latter is decided by looking just at the last
one or two digits of each entry (assuming that the entries are given in the binary or, say, decimal form). We emphasize though that solving
this membership problem in constant time is only possible if an input
matrix is known to belong to $SL_2(\mathbb{Z})$; otherwise one would have
to check that the determinant of a given matrix is equal to 1, which
cannot be  done in constant time, although can still be done in
sublinear time with respect to the complexity $|M|$ of an input
matrix $M$, as defined in the next Section \ref{results}, see
Corollary \ref{complexity}.

\section{Our results}
\label{results}

In this paper, we show that what would be a natural generalization
of Sanov's Theorem \ref{th2} to $A(k)$ and $B(k)$, $k \in \mathbb{Z}_+$, is
not valid for $k \ge 3$ and moreover, the subgroup generated by
$A(k)$ and $B(k)$ has infinite index in $SL_2(\mathbb{Z})$ if $k \ge 3$.

\begin{theorem}\label{infinite}
The subgroup of $SL_2(\mathbb{Z})$ generated by $A(k)$ and $B(k)$, $k \in
\mathbb{Z}, ~k \ge 3$, has infinite index in the group of all matrices of
the form $\left(
\begin{array}{cc} 1+k^2m_1 & km_2 \\ km_3 & 1+k^2m_4\end{array}
\right)$ with determinant 1.
\end{theorem}

The group of  all matrices of the above form, on the other hand,
obviously has finite index in $SL_2(\mathbb{Z})$.

Our main technical result, proved in Section \ref{Peak}, is the
following

\begin{theorem} \label{peak}
Let $M = \left(
 \begin{array}{cc} m_{11} & m_{12} \\ m_{21} & m_{22} \end{array}
 \right)$ be a matrix from $SL_2(\mathbb{R})$. Call ``elementary operations"
on $M$ the following 8 operations: multiplication of $M$ by either
$A(k)^{\pm 1}$ or by  $B(k)^{\pm 1}$, on the right or on the left.
\medskip

\noindent {\bf (a)} If $k \in \mathbb{Z}$ and $M$ belongs to the subgroup of
$SL_2(\mathbb{Z})$ generated by $A(k)$ and $B(k)$, then it has the form
$\left(
\begin{array}{cc} 1+k^2n_1 & kn_2 \\ kn_3 & 1+k^2n_4\end{array}
\right)$ for some integers $n_i$.

If $k \in \mathbb{R}$ and  $M$ belongs to the subgroup of $SL_2(\mathbb{R})$
generated by $A(k)$ and $B(k)$, then it has the form $\left(
\begin{array}{cc} 1+\sum_i k^in_i & \sum_j k^jn_j \\ \sum_r k^rn_r & 1+\sum_s k^sn_s\end{array}
\right)$ where all $n_i$ are integers and all exponents on $k$ are
positive integers.
\medskip

\noindent {\bf (b)} Let $k \in \mathbb{R}, ~k \ge 2$. If $M \in SL_2(\mathbb{R})$
and there is a sequence of elementary operations that reduces
$\sum_{i,j}|m_{ij}|$, then there is a single elementary operation
that reduces $\sum_{i,j}|m_{ij}|$.

\noindent {\bf (c)} Let $k \in \mathbb{Z}, ~k \ge 2$. If $M \in SL_2(\mathbb{Z})$
and no single elementary operation reduces $\sum_{i,j}|m_{ij}|$, then
either $M$ is the identity matrix or $M$ does not belong to the
subgroup generated by $A(k)$ and $B(k)$.

\end{theorem}

We also point out a result, similar to Theorem \ref{peak}, about the
{\it monoid} generated by $A(k)$ and $B(k)$ for $k>0$. Unlike
Theorem \ref{peak} itself, this result is trivial.

\begin{proposition}
Let $M = \left(
 \begin{array}{cc} m_{11} & m_{12} \\ m_{21} & m_{22} \end{array}
 \right)$ be a matrix from $SL_2(\mathbb{R})$. Call ``elementary operations"
on $M$ the following 4 operations: multiplication of $M$ by either
$A(k)^{-1}$ or by  $B(k)^{-1}$, on the right or on the left.
\medskip

\noindent {\bf (a)} If $k \in \mathbb{Z}, ~k > 0,$ and $M$ belongs to the
{\it monoid} generated by $A(k)$ and $B(k)$, then it has the form
$\left(
\begin{array}{cc} 1+k^2n_1 & kn_2 \\ kn_3 & 1+k^2n_4\end{array}
\right)$ for some nonnegative integers $n_i$.

If $k \in \mathbb{R}, ~k > 0,$ and  $M$ belongs to the {\it monoid}
generated by $A(k)$ and $B(k)$, then it has the form $\left(
\begin{array}{cc} 1+\sum_i k^in_i & \sum_j k^jn_j \\ \sum_r k^rn_r & 1+\sum_s k^sn_s\end{array}
\right)$ where all $n_i$ are nonnegative  integers and all exponents
on $k$ are positive integers.
\medskip

\noindent {\bf (b)} Let $k \in \mathbb{Z}, ~k \ge 2$. If $M$ is a matrix
from $SL_2(\mathbb{Z})$ with nonnegative entries and no elementary operation
reduces $\sum_{i,j} m_{ij}$, then either $M$ is the identity matrix
or  $M$ does not belong to the {\it monoid}  generated by $A(k)$ and
$B(k)$.

\end{proposition}

Thus, for example, the matrix $\left(
\begin{array}{cc} 5 & 4 \\ 6 & 5 \end{array}
\right)$ does not belong to the {\it monoid}  generated by $A(2)$
and $B(2)$, although it does belong to the {\it group} generated by
$A(2)$ and $B(2)$ by Sanov's Theorem \ref{th2}.

Theorem \ref{peak} yields a simple algorithm for the membership
problem in the subgroup generated by $A(k)$ and $B(k)$ in case $k
\in \mathbb{Z}, ~k \ge 2.$ We note in passing that in general, the subgroup
membership problem for $SL_2(\mathbb{Q})$ is open, while in $SL_2(\mathbb{Z})$ it is
solvable since $SL_2(\mathbb{Z})$ is virtually free. The general solution,
based on the automatic structure of $SL_2(\mathbb{Z})$ (see \cite{Epstein}),
is not so transparent and has quadratic time complexity (with respect to
the word length of an input). For our
special subgroups we have:

\begin{corollary}\label{complexity} Let $k \in \mathbb{Z}, ~k \ge 2$, and
let the complexity $|M|$ of a matrix $M = \left(
 \begin{array}{cc} m_{11} & m_{12} \\ m_{21} & m_{22} \end{array}
 \right)$ be the sum of all $|m_{ij}|$. There is an algorithm that
decides whether or not a given matrix $M \in SL_2(\mathbb{Z})$ is in the
subgroup of $SL_2(\mathbb{Z})$ generated by $A(k)$ and $B(k)$ (and if it
does, finds a presentation of $M$ as a group word in $A(k)$ and
$B(k)$) in time $O(n \cdot \log n)$, where  $n=|M|$.

\end{corollary}

\noindent {\bf Remark.} The relation between $|M|= \sum |m_{ij}|$
and the word length of $M$ (with respect to the standard generators
$A(1)$ and $B(1)$, say) is not at all obvious and is an  interesting
problem in its own right.

\medskip

Statement similar to Corollary \ref{complexity} holds also for the
{\it monoid} generated by $A(k)$ and $B(k)$, for any $k \in \mathbb{Z}, ~k
\ge 2$.

The $O(n \cdot \log n)$ is the worst-case complexity of the
algorithm referred to in Corollary \ref{complexity}. It would be
interesting to find out what the {\it generic-case complexity} (in
the sense of \cite{KMSS}) of this algorithm is. Proposition 1 in
\cite{BSV} tacitly suggests that this complexity might be, in fact,
sublinear in $n=|M|$, which would be a really interesting result, so
we ask:

\begin{problem}\label{generic} Is the generic-case complexity of the algorithm
claimed in Corollary \ref{complexity} sublinear in $|M|$?
\end{problem}

We note that, unlike the algorithms with low generic-case complexity
considered in \cite{KMSS}, this algorithm has a good chance to have
low generic-case complexity giving both ``yes" and ``no" answers,
see our Section \ref{Corollary} for more details.

Finally, we note that if $M$ is in the subgroup generated by $A(k)$
and $B(k)$, $k \ge 2$, then the presentation of $M$ as a group word
in $A(k)$ and $B(k)$ is unique since the group generated by $A(k)$
and $B(k)$ is known to be free for any $k \in \mathbb{R}, ~k \ge 2$, see
e.g. \cite{JP}. On the other hand, the group generated by $A(1)$ and
$B(1)$ (i.e., the whole group $SL_2(\mathbb{Z})$) is not free. This implies,
in particular, that for any integer $n \ge 1$, the group generated
by $A(\frac{1}{n})$ and $B(\frac{1}{n})$ is not free because it
contains both matrices $A(1)$ and $B(1)$. Many examples of rational
$k, ~ 0 < k < 2$, for which the subgroup of $SL_2(\mathbb{Q})$ generated by
$A(k)$ and $B(k)$ is not free were found over the years, starting
with \cite{L-U}; see a recent paper \cite{Gutan} for more
references. (We can single out the paper \cite{Beardon} where the
question of non-freeness for this subgroup was reduced to
solvability of particular Diophantine equations.) In particular, it
is known that for any $k, ~ 0 < k < 2$, of the form $\frac{m}{mn+1}$
or $\frac{m+n}{mn}, ~m, n \in Z_+$, the group generated by $A(k)$
and $B(k)$ is not free. This includes $k=\frac{2}{3}, \frac{3}{2},
\frac{3}{7}$, etc. Also, if the group is not free for some $k$, then
it is not free for any $\frac{k}{n}, ~n \in Z_+$.

The following problem, however, seems to be still open:

\begin{problem} (Yu. Merzlyakov \cite{Problems}, \cite{Kourovka})
For which rational $k, ~ 0 < k < 2$, is the group generated by
$A(k)$ and $B(k)$ free? More generally, for which algebraic $k, ~ 0
< k < 2$, is this group free?
\end{problem}

To the best of our knowledge, there are no known examples of a
rational $k, ~ 0 < k < 2$, such that the group generated by $A(k)$
and $B(k)$ is free. On the other hand, since any matrix from this
group has the form $ \left(
\begin{array}{cc} p_{11}(k) & p_{12}(k)\\
  p_{21}(k) & p_{22}(k) \end{array}
\right)$ for some polynomials $p_{ij}(k)$ with integer coefficients,
this group is obviously free if $k$ is transcendental. For the same
reason, if $r$ and $s$ are algebraic numbers that are Galois
conjugate over $\mathbb{Q}$, then the group generated by $A(r)$ and $B(r)$
is free if and only if the group generated by $A(s)$ and $B(s)$ is.
For example, if $r=2-\sqrt 2$, then $A(r)$ and $B(r)$ generate a
free group because this $r$ is Galois conjugate to $s=2+\sqrt 2 >
2$. More generally, $A(r)$ and $B(r)$ generate a free group for
$r=m- n\sqrt 2$, and therefore also for $r=k \cdot (m- n\sqrt 2)$,
with arbitrary positive $k, m, n \in \mathbb{Z}$. This implies, in
particular, that the set of algebraic $r$ for which the group is
free is dense in $\mathbb{R}$ because $(m- n\sqrt 2)$ can be arbitrarily
close to 0. All these $r$ are irrational though.

\section{Peak reduction}
\label{Peak}

Here we prove Theorem \ref{peak} from Section \ref{results}. The
method we use is called {\it peak reduction} and goes back to
Whitehead \cite{Wh}, see also \cite{L-S}. The idea is as follows.
Given an algorithmic problem that has to be solved, one first
somehow defines {\it complexity} of possible inputs. Another
ingredient is a collection of {\it elementary operations} that can
be applied to inputs. Thus, we now have an action of the semigroup
of elementary operations on the set of inputs. Usually, of
particular interest are elements of minimum complexity in any given
orbit under this action. The main problem typically is to find these
elements of minimum complexity. This is where the peak reduction
method can be helpful. A crucial observation is: if there is a
sequence of elementary operations (applied to a given input) such
that at some point in this sequence the complexity goes up (or
remains unchanged) before eventually going down, then there must be
a pair of {\it subsequent} elementary operations in this sequence (a
``peak") such that  one of them increases the complexity (or leaves
it unchanged), and then the other one decreases it. Then one tries
to prove that such a peak can always be reduced, i.e., if there is
such a pair, then there is also a single elementary operation that
reduces complexity. This will then imply that there is a ``greedy"
sequence of elementary operations, i.e., one that reduces complexity
{\it at every step}. This will yield an algorithm for finding an
element of minimum complexity in a given orbit.

In our situation, inputs are matrices from $SL_2(\mathbb{R})$. For the
purposes of the proof of Theorem \ref{peak}, we define complexity of
a matrix $M = \left( \begin{array}{cc} m_{11} & m_{12} \\ m_{21} &
m_{22} \end{array} \right)$ to be the maximum of all $|m_{ij}|$.
Between two matrices with the same $\max |m_{ij}|$, the one with the
larger $\sum_{i,j}|m_{ij}|$ has higher complexity. We will see,
however, that in case of $2 \times 2$ matrices with determinant 1,
the ``greedy" sequence of elementary operations would be the same as
if we defined the complexity to be just $\sum_{i,j}|m_{ij}|$.

Elementary operations in our situation are multiplications of a
matrix by either $A(k)^{\pm 1}$ or by  $B(k)^{\pm 1}$, on the right
or on the left. They correspond to elementary row or column
operations; specifically, to operations of the form $(row_1 \pm  k
\cdot row_2)$, $(row_2 \pm  k \cdot row_1)$, $(column_1 \pm  k \cdot
column_2)$, and  $(column_2 \pm  k \cdot column_1)$.

We now get to

\noindent {\bf Proof of Theorem \ref{peak}.} Part (a) is established
by an obvious induction on the length of a group word representing a
given element of the subgroup generated by $A(k)$ and $B(k)$. Part
(c) follows from part (b). We omit the details and proceed to part
(b).

We are going to consider various pairs of subsequent elementary
operations of the following kind: the first operation increases the
maximum of $|m_{ij}|$ (or leaves it unchanged), and then the second
one reduces it. We are assuming that the second elementary operation
is not the inverse of the first one.

In each case like that, we show that either the maximum of
$|m_{ij}|$ in the given matrix could have been reduced by just a
single elementary operation (and then $\sum |m_{ij}|$ should be reduced, too, to keep the determinant unchangedk), or $\sum |m_{ij}|$ could have been
reduced by a single elementary operation leaving the maximum of
$|m_{ij}|$ unchanged. Because of a ``symmetry", it is sufficient to
consider the following cases.

First of all, we note that since the determinant of $M$ is equal to
1, there can be 0, 2, or 4  negative entries in $M$. If there are 2
negative entries, they can occur either in the same row, or in the
same column, or on the same diagonal. Because of the symmetry, we
only consider the case where two negative entries are in the first
column and the case where they are on the main diagonal. Also, cases
with 0 and 4 negative entries are symmetric, so we only consider the
case where there are no negative entries.

It is also convenient for us to single out the case where $M$ has
two zero entries, so we start with

\medskip

\noindent {\bf Case 0.} There are two zero entries in $M$. Since the
determinant of $M$ is 1, the two  nonzero entries should be on a
diagonal and their product should be $\pm 1$. If  they are not on
the main diagonal, then $M^4 = I$,  in which case $M$ cannot belong
to the subgroup generated by $A(k)$ and $B(k)$, $k \ge 2$, since
this subgroup is free.

Now suppose that the two  nonzero entries are on the main diagonal,
so $M = \left(
\begin{array}{cc} x & 0\\
 0 & \frac{1}{x} \end{array}
\right)$ for some $x \in \mathbb{R}, ~x \ne 1$. Without loss of generality,
assume $x > 0$. We are going to show, by way of contradiction, that
such a matrix is not in the subgroup generated by $A(k)$ and $B(k)$.
We have:  $MA(k)M^{-1} = \left(
\begin{array}{cc} 1 & x^2k\\
 0 & 1 \end{array}
\right),$ and for some $r \in \mathbb{Z}_+$ we have $A(k)^{-r}MA(k)M^{-1} =
\left(
\begin{array}{cc} 1 & yk\\
 0 & 1 \end{array}
\right),$ where $0 < yk \le k$. If  $yk = k$, then we have a
relation $A(k)^{-r}MA(k)M^{-1} = A(k)$, so again we have a
contradiction with the fact that the subgroup generated by $A(k)$
and $B(k)$ is free. Now let $0 < yk < k$, so $0 < y < 1$, and let $C
=  \left(\begin{array}{cc} 1 & yk\\
 0 & 1 \end{array}\right)$. We claim that $C$ does not belong to the
subgroup generated by $A(k)$ and $B(k)$. If it did, then so would
the matrix $T(m, n) = A(k)^{-m}C^n =  \left(\begin{array}{cc} 1 & (ny-m)k\\
 0 & 1 \end{array}\right)$ for any $m, n \in \mathbb{Z}_+$. Since $(ny-m)k$
 can be arbitrarily close to 0, the matrix $T(m, n)$ can be arbitrarily close to
the identity matrix, which contradicts the well-known fact (see e.g.
\cite{Jorgensen}) that the group generated by $A(k)$ and $B(k)$ is
{\it discrete} for any $k \ge 2$.

\medskip

In what follows, we assume that all matrices under consideration
have at most one zero entry. Even though we use strict inequalities
for all entries of a matrix, the reader should keep in mind that one
of the inequalities may  be not strict; this does not affect the
argument.

\smallskip

\noindent {\bf Case 1.} There are 2 negative entries, both in the
first column. Thus,  $m_{11}<0, m_{21}<0, m_{12}>0, m_{22}>0$.

\medskip

\noindent {\bf Case 1a.} Two subsequent elementary operations
reducing some entry after increasing it are both $(column_1 + k
\cdot column_2)$. If, after one operation $(column_1 + k \cdot
column_2)$, the element $m_{11}$, say, becomes positive, then
$m_{21}$ should become positive, too, for the determinant to remain
unchanged. Then, after applying $(column_1 + k \cdot column_2)$ one
more time, new $|m_{11}|$ will become greater than it was, contrary
to the assumption.

If, after one operation $(column_1 + k \cdot column_2)$, $m_{11}$
remains negative, then this operation reduces $|m_{11}|$, and this
same operation should also reduce  $|m_{21}|$  for the determinant
to remain unchanged. Indeed, the determinant is $m_{11}m_{22} -
m_{12}m_{21} = 1$. If $|m_{11}|$ decreases while $m_{11}$ remains
negative, then the value of $m_{11}m_{22}$ increases (but remains
negative). Therefore, the value of $m_{12}m_{21}$ should increase,
too, for the difference to remain unchanged. Since $m_{21}$ should
remain negative, this implies that $|m_{21}|$ should decrease, hence
$\sum |m_{ij}|$ decreases after one operation $(column_1 + k \cdot
column_2)$.

The same kind of argument works in the case where both operations
are  $(row_1 - k \cdot row_2)$,

If both operations are $(row_1 + k \cdot row_2)$, or $(column_1 - k
\cdot column_2)$, then no $|m_{ij}|$ can possibly decrease since
$m_{11}<0, m_{21}<0, m_{12}>0, m_{22}>0$.

\smallskip

\noindent {\bf Case 1b.} Two subsequent elementary operations are:
$(row_1 + k \cdot row_2)$, followed by $(column_1 + k \cdot
column_2)$. The result of applying these two operations to the
matrix $M$ is:  $ \left(
\begin{array}{cc} m_{11} + k \cdot m_{21} +k \cdot m_{12} + k^2 m_{22} & m_{12}+k \cdot m_{22}\\
  m_{21}+k \cdot m_{22} & m_{22} \end{array}
\right)$. This case is nontrivial only if the first operation
increases the absolute value of the  element in the top left corner
(or leaves it unchanged), and then the second operation reduces it.
But then the second operation should also reduce the absolute value
of $m_{21}$ for the determinant to remain unchanged. This means a
single operation  $(column_1 + k \cdot column_2)$ would reduce
$|m_{21}|$ to begin with,  and this same operation should also
reduce $|m_{11}|$ for the determinant to remain unchanged. Thus, a
single elementary operation would reduce the complexity of $M$.

The same argument takes care of any of the following pairs of
subsequent elementary operations: $(row_1 \pm k \cdot row_2)$,
followed by $(column_1 \pm k \cdot column_2)$, as well as  $(row_1
\pm k \cdot row_2)$, followed by $(column_2 \pm k \cdot column_1)$.

\smallskip

\noindent {\bf Case 1c.} Two subsequent elementary operations are:
 $(row_1 + k \cdot row_2)$, followed by $(row_2 + k \cdot row_1)$. The result of
applying these two operations to the matrix $M$ is:  $ \left(
\begin{array}{cc} m_{11}+k \cdot m_{21} & m_{12}+k \cdot m_{22}\\
   (k^2+1) m_{21}+k \cdot m_{11} & (k^2+1)m_{22}+k \cdot m_{12}\end{array}
\right)$. In this case, obviously $|(k^2+1)m_{21}+  k \cdot m_{11}|
> |m_{21}|$ and $|(k^2+1)m_{22}+  k \cdot m_{12}| > |m_{22}|$, so we do not
have a decrease, i.e., this case is moot.

\smallskip

\noindent {\bf Case 1d.} Two subsequent elementary operations are:
 $(row_1 + k \cdot row_2)$, followed by $(row_2 - k \cdot row_1)$. The result of
 applying these two operations to the matrix $M$ is:  $ \left(
\begin{array}{cc} m_{11}+k \cdot m_{21} & m_{12}+k \cdot m_{22}\\
    -(k^2-1)m_{21}- k \cdot m_{11} & -(k^2-1)m_{22}- k \cdot m_{12}\end{array}
\right)$. If $k > \sqrt{2}$, then $|-(k^2-1)m_{21}- k \cdot m_{11}|
> |m_{21}|$ and $|-(k^2-1)m_{22}- k \cdot m_{12}| > |m_{22}|$, so we
do not have a decrease, i.e., this case is moot, too.

\smallskip

\noindent {\bf Case 1e.} Two subsequent elementary operations are:
 $(row_1 - k \cdot row_2)$, followed by $(row_2 + k \cdot row_1)$. The result of
applying these two operations to the matrix $M$ is:  $ \left(
\begin{array}{cc} m_{11}-  k \cdot m_{21} & m_{12}- k \cdot m_{22}\\
    k \cdot m_{11}-(k^2-1) \cdot m_{21}  & k \cdot m_{12}-(k^2-1) \cdot m_{22} \end{array}
\right)$. If the first operation increases $|M|$ (or leaves it
unchanged) and then the second one reduces it, then the second
operation should reduce $|m_{21}|$ or  $|m_{22}|$. Assume, without
loss of generality, that $|m_{21}| \ge |m_{22}|$.

We may assume that $|m_{11}-k \cdot m_{21}| \ge |m_{11}| = -m_{11}$
and $|m_{12}-k \cdot m_{22}| \ge |m_{12}| = m_{12}$ because
otherwise, the complexity of $M$ could be reduced by a single
operation $(row_1 - k \cdot row_2)$.

Now we look at the inequality  $|k \cdot m_{11}-(k^2-1) \cdot
m_{21}| < |m_{21}|=-m_{21}$. Re-write it as follows: $|k
\cdot(m_{11}-k \cdot m_{21}) + m_{21}| < -m_{21}$.  We may assume
that  $m_{11}-k \cdot m_{21}>0$ because otherwise, a single
operation $(row_1 - k \cdot row_2)$ would reduce the complexity of
$M$. We also know that $|m_{11}-k \cdot m_{21}| \ge -m_{11}$ (see
the previous paragraph). Thus, $m_{11}-k \cdot m_{21} \ge -m_{11}$.
This inequality, together with $|k \cdot(m_{11}-k \cdot m_{21}) +
m_{21}| < -m_{21}$, yield   $|-k \cdot m_{11} + m_{21}| < -m_{21}$.
This means  a  single operation $(row_2 - k \cdot row_1)$ would
reduce $|m_{21}|$.
\smallskip

\noindent {\bf Case 1f.}  Two subsequent elementary operations are:
$(row_1 - k \cdot row_2)$, followed by $(row_2 - k \cdot row_1)$.
The result of applying these two operations to the matrix $M$ is:
$\left(
\begin{array}{cc} m_{11}-  k \cdot m_{21} & m_{12}- k \cdot m_{22}\\
   (k^2+1) \cdot m_{21} - k \cdot m_{11}  & (k^2+1) \cdot m_{22} - k \cdot m_{12} \end{array}
\right)$. If the first operation increases $|M|$ (or leaves it
unchanged) and then the second one reduces it, then the second
operation should reduce $|m_{21}|$ or  $|m_{22}|$.

We may assume that $m_{12} - k \cdot m_{22} < 0$ because otherwise,
a single operation $(row_1 - k \cdot row_2)$ would reduce $|m_{12}|$
and therefore also $|m_{11}|$ for the determinant to remain
unchanged in this case. Now look at the element in the bottom right
corner: $(k^2+1) \cdot m_{22} - k \cdot m_{12} = k \cdot (k \cdot
m_{22} -m_{12}) + m_{22}$. Since $k \cdot m_{22} -m_{12} >0$, we
have $|(k^2+1) \cdot m_{22} - k \cdot m_{12}|
> |m_{22}|$, a contradiction.

\smallskip

\noindent {\bf Case 1g.}  Two subsequent elementary operations are:
$(column_1 + k \cdot column_2)$, followed by $(column_2 + k \cdot
column_1)$. The result of applying these two operations to the
matrix $M$ is: $\left(
\begin{array}{cc} m_{11}+  k \cdot m_{12} & (k^2+1) \cdot m_{12}+ k \cdot m_{11}\\
 m_{21} + k \cdot m_{22}  & (k^2+1) \cdot m_{22} + k \cdot m_{21} \end{array}
\right)$. If the first operation increases $|M|$ (or leaves it
unchanged) and then the second one reduces it, then the second
operation should reduce $|m_{12}|$ or  $|m_{22}|$. Let us assume
here that $|m_{22}| \ge |m_{12}|$.

Now look at the element in the bottom right corner: $(k^2+1) \cdot
m_{22}  + k \cdot m_{21} = k \cdot (m_{21} + k \cdot m_{22}) +
m_{22}$. We may assume that $m_{21} + k \cdot m_{22} > 0$ because
otherwise, a single operation $(column_1 + k \cdot column_2)$ would
reduce $|m_{21}|$. In that case, however, we have $|(k^2+1) \cdot
m_{22}  + k \cdot m_{21}| = k \cdot (m_{21} + k \cdot m_{22}) +
m_{22} > m_{22} = |m_{22}|$, a contradiction.
\smallskip

\noindent {\bf Case 1h.}  Two subsequent elementary operations are:
$(column_1 + k \cdot column_2)$, followed by $(column_2 - k \cdot
column_1)$. The result of applying these two operations to the
matrix $M$ is: $\left(
\begin{array}{cc} m_{11}+  k \cdot m_{12} & (1-k^2) \cdot m_{12}- k \cdot m_{11}\\
 m_{21} + k \cdot m_{22}  & (1-k^2) \cdot m_{22} - k \cdot m_{21} \end{array}
\right)$. If the first operation increases $|M|$ (or leaves it
unchanged) and then the second one reduces it, then the second
operation should reduce $|m_{12}|$ or  $|m_{22}|$. Assume here that
$|m_{22}| \ge |m_{12}|$. Then we should have  $|(1-k^2) \cdot m_{22}
- k \cdot m_{21}| < |m_{22}| = m_{22}$.

We may assume that $m_{21} + k \cdot m_{22} > 0$ because otherwise,
a single operation $(column_1 + k \cdot column_2)$ would reduce
$|m_{21}|$ while keeping the element in this position negative. Then
this same operation should reduce $|m_{11}|$, too, for the
determinant to remain unchanged. Also, since the first operation was
supposed to increase $|M|$ (or leave it unchanged), we should have,
in particular, $|m_{21} + k \cdot m_{22}| = m_{21} + k \cdot m_{22}
\ge |m_{21}| = -m_{21}$. This, together with the inequality in the
previous paragraph, gives $|(1-k^2) \cdot m_{22} - k \cdot m_{21}| =
|m_{22} - k \cdot (m_{21} + k \cdot m_{22})| \ge |k \cdot m_{22} +
m_{21}|$. Therefore, we should have $|k \cdot m_{22} + m_{21}|  <
|m_{22}| = m_{22}$. Since $k \ge 2$, this implies $|m_{21}|  >
|m_{22}|$, contradicting the assumption of $m_{22}$ having the
maximum absolute value in the matrix $M$.

\smallskip

\noindent {\bf Case 1i.}  Two subsequent elementary operations are:
$(column_1 - k \cdot column_2)$, followed by $(column_2 + k \cdot
column_1)$. The result of applying these two operations to the
matrix $M$ is: $\left(
\begin{array}{cc} m_{11}-  k \cdot m_{12} & (1-k^2) \cdot m_{12}+ k \cdot m_{11}\\
 m_{21} - k \cdot m_{22}  & (1-k^2) \cdot m_{22} + k \cdot m_{21} \end{array}
\right)$. Since $k \ge 2$, we have $|(1-k^2) \cdot m_{22} + k \cdot
m_{21}| > |m_{22}|$, so this case is moot.
\smallskip

\noindent {\bf Case 1j.}  Two subsequent elementary operations are:
$(column_1 - k \cdot column_2)$, followed by $(column_2 - k \cdot
column_1)$. The result of applying these two operations to the
matrix $M$ is: $\left(
\begin{array}{cc} m_{11}-  k \cdot m_{12} & (1+k^2) \cdot m_{12}- k \cdot m_{11}\\
 m_{21} - k \cdot m_{22}  & (1+k^2) \cdot m_{22} - k \cdot m_{21} \end{array}
\right)$. Since $|(1+k^2) \cdot m_{22} - k \cdot m_{21}| = |m_{22} -
k \cdot (m_{21} - k \cdot m_{22})| > |m_{22}|$, this case is moot,
too.

\medskip

\noindent {\bf Case 2.} Two negative entries are on a diagonal.
Without loss of generality, we assume here that $m_{11}>0, m_{22}>0,
m_{12}<0, m_{21}<0$. Because of the ``symmetry" between row and
column  operations in this case, we can reduce the number of
subcases (compared to Case 1 above) and only consider the following.
\medskip

\noindent {\bf Case 2a.} Two subsequent elementary operations are:
$(row_1 + k \cdot row_2)$, followed by $(column_1 + k \cdot
column_2)$. The result of applying these two operations to the
matrix $M$ is:  $ \left(
\begin{array}{cc} m_{11} + k \cdot m_{21} +k \cdot m_{12} + k^2 m_{22} & m_{12}+k \cdot m_{22}\\
  m_{21}+k \cdot m_{22} & m_{22} \end{array}
\right)$. This case is nontrivial only if the first operation
increases the absolute value of the  element in the top left corner
(or leaves it unchanged), and then the second operation reduces it.
First let us look at the element $m_{21}+k \cdot m_{22}$. If
$m_{21}+k \cdot m_{22} <0$, then a single operation  $(column_1 + k
\cdot column_2)$ would reduce $m_{21}$ while keeping that element
negative. In that case, this operation would reduce $m_{11}$, too,
while keeping it positive because otherwise, the determinant would
change. Thus, a single operation  $(column_1 + k \cdot column_2)$
would reduce the complexity of $M$ in that case. The same argument
shows that $m_{12}+k \cdot m_{22} \ge 0$ because otherwise, a single
operation $(row_1 + k \cdot row_2)$ would reduce the complexity of
$M$.

If $m_{21}+k \cdot m_{22} \ge 0$  and  $m_{12}+k \cdot m_{22} \ge
0$, then $m_{11} + k \cdot m_{21} +k \cdot m_{12} + k^2 m_{22} \ge
0$ for the determinant to be equal to 1. But then the second
operation should also reduce the absolute value of $m_{21}$ for the
determinant to remain unchanged. This means a single operation
$(column_1 + k \cdot column_2)$ would reduce $|m_{21}|$ to begin
with,  and this same operation should also reduce $|m_{11}|$ for the
determinant to remain unchanged, so this single operation would
reduce the complexity of $M$.

The same argument takes care of any of the following pairs of
subsequent elementary operations: $(row_1 \pm k \cdot row_2)$,
followed by $(column_1 \pm k \cdot column_2)$, as well as  $(row_1
\pm k \cdot row_2)$, followed by $(column_2 \pm k \cdot column_1)$.

\smallskip

\noindent {\bf Case 2b.} Two subsequent elementary operations are:
$(row_1 + k \cdot row_2)$, followed by $(row_2 + k \cdot row_1)$.
The result of applying these two operations to the matrix $M$ is:
$\left(
\begin{array}{cc} m_{11}+k \cdot m_{21} & m_{12}+k \cdot m_{22}\\
   (k^2+1) m_{21}+k \cdot m_{11} & (k^2+1)m_{22}+k \cdot m_{12}\end{array}
\right)$. If the first operation increases $|M|$ (or leaves it
unchanged) and then the second one reduces it, then the second
operation should reduce $|m_{21}|$ or  $|m_{22}|$. Assume, without
loss of generality, that $|m_{21}| \ge |m_{22}|$.

Thus, we have $|(k^2+1)m_{21}+  k \cdot m_{11}| < |m_{21}| =
-m_{21}$. At the same time, we may assume that $|m_{11}+k \cdot
m_{21}| \ge |m_{11}| = m_{11}$  because otherwise, a single
operation $(row_1 + k \cdot row_2)$ would reduce the complexity of
$M$.

Then, if $m_{11}+k \cdot m_{21} >0$, then a single operation $(row_1
+ k \cdot row_2)$ would reduce $|m_{11}|$, and therefore also
$|m_{12}|$. Thus, we may assume that $m_{11}+k \cdot m_{21} <0$.

Now let us look at the inequality $|(k^2+1)m_{21}+  k \cdot m_{11}|
< -m_{21}$. We know that $m_{11}+k \cdot m_{21} <0$. Therefore, $k
\cdot m_{11}+k^2 \cdot m_{21} <0$. Now $|(k^2+1)m_{21}+  k \cdot
m_{11} = |k \cdot m_{11}+k^2 \cdot m_{21} + m_{21}| > -m_{21}$. This
contradiction completes Case 2b.

\smallskip

\noindent {\bf Case 2c.} Two subsequent elementary operations are:
 $(row_1 + k \cdot row_2)$, followed by $(row_2 - k \cdot row_1)$. The result of
 applying these two operations to the matrix $M$ is:  $ \left(
\begin{array}{cc} m_{11}+k \cdot m_{21} & m_{12}+k \cdot m_{22}\\
    -(k^2-1)m_{21}- k \cdot m_{11} & -(k^2-1)m_{22}- k \cdot m_{12}\end{array}
\right)$. The analysis here is similar to the previous Case 2b.
First we note that we may assume $m_{11}+k \cdot m_{21} <0$, so  $-k
\cdot m_{11}-k^2 \cdot m_{21} >0$. On the other hand, we may assume
that $|m_{11}+k \cdot m_{21}| = -m_{11}-k \cdot m_{21} \ge m_{11}$
because  otherwise, a single operation $(row_1 + k \cdot row_2)$
would reduce $|m_{11}|$. Therefore, $-k \cdot m_{11} - k^2 \cdot
m_{21} \ge k \cdot m_{11}$.

This, together with the inequality $|-(k^2-1)m_{21}- k \cdot m_{11}|
= |-k \cdot m_{11}-k^2 \cdot m_{21} + m_{21}| < |m_{21}| = -m_{21}$,
implies $|k \cdot m_{11} + m_{21}| < |m_{21}|$, in which case a
single operation $(row_2 + k \cdot row_1)$ would reduce $|m_{21}|$.

\smallskip

\noindent {\bf Case 2d.} Two subsequent elementary operations are:
 $(row_1 - k \cdot row_2)$, followed by $(row_2 + k \cdot row_1)$. The result of
 applying these two operations to the matrix $M$ is:  $ \left(
\begin{array}{cc} m_{11}-k \cdot m_{21} & m_{12}-k \cdot m_{22}\\
    -(k^2-1)m_{21}+ k \cdot m_{11} & -(k^2-1)m_{22}+ k \cdot m_{12}\end{array}
\right)$. If $k^2>2$, here we obviously have $|-(k^2-1)m_{22}+ k
\cdot m_{12}| > |m_{22}|$,  and $|-(k^2-1)m_{21}+ k \cdot m_{11}| >
|m_{21}|$. Thus, this case is moot.
\smallskip

\noindent {\bf Case 2e.} Two subsequent elementary operations are:
 $(row_1 - k \cdot row_2)$, followed by $(row_2 - k \cdot row_1)$.
The result of applying these two operations to the matrix $M$ is:
$\left(
\begin{array}{cc} m_{11}-k \cdot m_{21} & m_{12}-k \cdot m_{22}\\
    (k^2+1)m_{21}- k \cdot m_{11} & (k^2+1)m_{22}- k \cdot m_{12}\end{array}
\right)$. Here again we  obviously have $|(k^2+1)m_{22}- k \cdot
m_{12}| > |m_{22}|$ and $|(k^2+1)m_{21}- k \cdot m_{11}| >
|m_{21}|$, so this case is moot, too.

\medskip

\noindent {\bf Case 3.} There are no negative entries. Because of
the obvious symmetry, it is sufficient to consider the following
cases.
\medskip

\noindent {\bf Case 3a.} Two subsequent elementary operations are
both $(column_1 + k\cdot column_2)$, $(column_2 + k\cdot column_1)$,
$(row_1 + k\cdot row_2)$ or $(row_2 + k\cdot row_1)$. Since all the
entries are positive, the second operation cannot decrease the
complexity in this case.
\smallskip

\noindent {\bf Case 3b.} Two subsequent elementary operations are
$(column_1 + k \cdot column_2)$, followed by $(column_2 - k \cdot
column_1)$, give the matrix\par $\left(
\begin{array}{cc} m_{11}+  k \cdot m_{12} & (1-k^2) \cdot m_{12}- k \cdot m_{11}\\
 m_{21} + k \cdot m_{22}  & (1-k^2) \cdot m_{22} - k \cdot m_{21} \end{array}
\right)$. If the first operation increases $|M|$ (or leaves it
unchanged) and then the second operation reduces it, then the second
operation should reduce, say, $|m_{12}|=m_{12}$ (assuming that
$m_{12}\ge m_{22}$).

After the first operation we note that $|m_{11}+k\cdot
m_{12}|=m_{11}+k\cdot m_{12}\geq |m_{12}|=m_{12}$. After the second
operation, assuming that complexity was reduced, we have
$|(1-k^2)m_{12}-k\cdot m_{11}|\leq |m_{12}|=m_{12}$, which can be
rewritten as $|m_{12}-k(m_{11}+k\cdot m_{12})|\leq m_{12}$. Since
all the entries are positive, we have $m_{11}+k\cdot m_{12}\geq
m_{12}$, and hence in order for the second operation to reduce
$|m_{12}|$, the following inequality should hold:  $-m_{12}\leq
(1-k^2)m_{12}-k\cdot m_{11}\leq 0$. Subtracting $m_{12}$,
multiplying each part by $-1$ and factoring out $k$ we get
$2m_{12}\geq k(m_{11}+\cdot m_{12})\geq m_{12}$. Divide by $k$:
$\frac{2}{k}\cdot m_{12}\geq m_{11}+k\cdot m_{12}\geq
\frac{1}{k}\cdot m_{12}$. However, $k\geq 2$ implies that
$m_{11}+k\cdot m_{12}\leq m_{12}$, which brings us to a
contradiction.

\smallskip

\noindent {\bf Case 3c.} Two subsequent elementary operations are
$(column_1 - k \cdot column_2)$, followed by $(column_2 + k \cdot
column_1)$. The resulting matrix is \par $\left(
\begin{array}{cc} m_{11}-  k \cdot m_{12} & (1-k^2) \cdot m_{12}+ k \cdot m_{11}\\
 m_{21} - k \cdot m_{22}  & (1-k^2) \cdot m_{22} + k \cdot m_{21} \end{array}
\right)$. If the first operation increases $|M|$ (or leaves it
unchanged) and then the second one reduces it, then the second
operation should reduce, say, $|m_{12}|=m_{12}$ (assuming that
$m_{12}\ge m_{22}$).

Also, after the first operation we can observe that
$m_{11}\leq|m_{11}-k\cdot m_{12}|=$ and $m_{21}\leq |m_{21}-k\cdot
m_{11}|$ because otherwise,  a single operation $(column_1 - k\cdot
column_2)$ would reduce $|m_{11}|$ and $|m_{21}|$. This implies that
$m_{11}-k\cdot m_{12}\leq -m_{11}$ and $m_{21}-k\cdot m_{22}\leq
-m_{21}$.

Consider the inequality $|(1-k^2) m_{12}+k\cdot m_{11}|\leq m_{12}$.
Rewrite it in the following way: $|m_{12}-k\cdot (k\cdot
m_{12}-m_{11})|\leq m_{12}$. Combining this with the previous
inequalities we get $|m_{12}-k\cdot m_{11}|\leq |m_{12}-k\cdot
(k\cdot m_{12}-m_{11})|\leq m_{12}$. Therefore, a single operation
$(column_2 - k\cdot column_1)$ reduces $|m_{12}|$.

\smallskip

\noindent {\bf Case 3d.} Two subsequent elementary operations are
$(column_1 - k \cdot column_2)$ followed by $(column_2 - k \cdot
column_1)$.  The resulting matrix is  \par $\left(
\begin{array}{cc} m_{11}-  k \cdot m_{12} & (1-k^2) \cdot m_{12}- k \cdot m_{11}\\
 m_{21} - k \cdot m_{22}  & (1-k^2) \cdot m_{22} - k \cdot m_{21} \end{array}
\right)$. If the first operation increases $|M|$ (or leaves it
unchanged) and then the second operation reduces it, then the second
one should reduce, say, $|m_{12}|=m_{12}$ (assuming that $m_{12}\ge
m_{22}$).

We may assume that $m_{11}-k\cdot m_{12} \le 0$ because if
$m_{11}-k\cdot m_{12} >0$, then for the determinant to be unchanged
after the first operation, we would have also $m_{21}-k\cdot m_{22}
>0$, but then a single operation
$(column_1 - k\cdot column_2)$ would reduce the complexity of $M$.

Thus, $|m_{11}-k\cdot m_{12}| = k\cdot m_{12} - m_{11}$. If the
first operation $(column_1 - k \cdot column_2)$ did not reduce the
complexity of $M$, then  $m_{11}\leq k\cdot m_{12} - m_{11}$, which
implies that $m_{11}-k\cdot m_{12} \leq -m_{11}$.

Now consider the inequality $|(1-k^2) m_{12}-k\cdot m_{11}| \leq
m_{12}$. After rewriting it we get $|m_{12}-k\cdot (k\cdot
m_{12}-m_{11})|\leq m_{12}$. Combining it with the inequality in the
previous paragraph,  we get $|m_{12}-k\cdot m_{11}|\leq
|m_{12}-k\cdot (k\cdot m_{12}-m_{11})|\leq m_{12}$. Therefore, a
single elementary operation $(column_2 - k\cdot column_1)$ reduces
$|m_{12}|$.

\section{Proof of Theorem \ref{infinite}}

Let $k, m \in \mathbb{Z}, ~k \ge 3, ~m \ge 1$. Denote  $M(k, m)= \left(
\begin{array}{cc} 1-k^2m & k^2m \\ -k^2m & 1+k^2m\end{array}
\right)$. It is straightforward to check that:

\medskip

\noindent {\bf (1)} $M(k, m)$ has determinant 1;

\noindent {\bf (2)} $M(k, m)= M(k, 1)^m$;

\noindent {\bf (3)} No elementary $k$-operation reduces the absolute
value of any of the entries of $M(k, m)$.

\medskip

Since the cyclic group generated by $M(k, 1)$ is infinite, the
result follows from Theorem \ref{peak}.

\section{Proof of Corollary \ref{complexity}}
\label{Corollary}

We assume in this section that $k \ge 3$ because for $k=2$, the
membership problem in the subgroup of $SL_2(\mathbb{Z})$ generated by $A(k)$
and $B(k)$ is solvable in constant time, see Corollary
\ref{corollary2} in the Introduction.

First of all we check that a given matrix $M$ from the group
$SL_2(\mathbb{Z})$ has the form $\left(
\begin{array}{cc} 1+k^2n_1 & kn_2 \\ kn_3 & 1+k^2n_4\end{array}
\right)$ for some integers $n_i$.  Then we check that $M$ has at
most one zero entry. If there are more, then $M$ does not belong to
the subgroup in question unless $M$ is the identity matrix. We also
check that $\max |m_{ij}| > 1$. If $\max |m_{ij}| = 1$, then $M$
does not belong to the subgroup in question unless $M$ is the
identity matrix. Indeed, the only nontrivial cases here are $M=A(\pm
1)$ and $M=B(\pm 1)$. Then $M^k=A(k)^{\pm 1}$ or $M^k=B(k)^{\pm 1}$.
This would give a nontrivial relation in the group generated by
$A(k)$ and $B(k)$ contradicting the fact that this group is free.

Now let $\max |m_{ij}| > 1$. If no elementary operation either
reduces $\max |m_{ij}|$ or reduces $\sum_{i,j}|m_{ij}|$ without
increasing $\max |m_{ij}|$, then $M$ does not belong to the subgroup
generated by $A(k)$ and $B(k)$. If there is an elementary operation
that reduces  $\max |m_{ij}|$, then we apply it. For example,
suppose the elementary operation $(row_1 -k \cdot row_2)$ reduces
$|m_{11}|$. The result of this operation is the matrix $\left(
 \begin{array}{cc} m_{11}-km_{21} & m_{12}-km_{22} \\ m_{21} & m_{22} \end{array}
 \right)$. If $|m_{11}|$ here has decreased, then $|m_{12}|$ could not
increase because otherwise, the determinant of the new matrix would
not be equal to 1. Thus, the complexity of the matrix $M$ has been
reduced, and the new matrix belongs to our subgroup if and only if
the matrix $M$ does. Since there are only finitely many numbers of
the form $kn, n \in \mathbb{Z},$ with bounded absolute value, this process
should terminate either with a non-identity matrix whose complexity
cannot be reduced or with the identity matrix. In the latter case,
the given matrix was in the subgroup generated by $A(k)$ and $B(k)$;
in the former case, it was not.

To estimate the time complexity of this algorithm, we note that each
step of it (i.e., applying a single elementary operation) takes time
$O(\log m)$, where $m$ is the complexity of the matrix this
elementary operation is applied to. This is because if $k$ is an
integer, multiplication by $k$ amounts to $k-1$ additions, and each
addition of integers not exceeding $m$ takes time $O(\log m)$. Since
the complexity of a matrix is reduced at least by 1 at each step of
the algorithm, the total complexity is $O(\sum_{k=1}^n \log k) = O(n
\cdot \log n)$. This completes the proof. $\Box$

\smallskip

As for generic-case complexity of this algorithm (cf. Problem
\ref{generic} in our Section \ref{results}), we note that, speaking
very informally, a ``random" product of $A(k)^{\pm 1}$ and
$B(k)^{\pm 1}$ is ``close" to a product where $A(k)^{\pm 1}$ and
$B(k)^{\pm 1}$ alternate, in which case the complexity of the
product matrix grows exponentially in the number of factors (see
e.g. \cite[Proposition 1]{BSV}), so the number of summands in the
sum that appears in the proof of Corollary \ref{complexity} will be
logarithmic in $n$, and therefore generic-case complexity of the
algorithm should be $O(\log ^2 n)$ in case the answer is ``yes"
(i.e., an input matrix belongs to the subgroup generated by $A(k)$
and $B(k)$). Of course, a ``random" matrix from $SL_2(\mathbb{Z})$ will {\it
not} belong to the subgroup generated by $A(k)$ and $B(k)$ with
overwhelming probability. This is because if $k \ge 3$, this
subgroup has infinite index in $SL_2(\mathbb{Z})$. It is, however, not clear
how fast (generically) our algorithm will detect that; specifically,
whether it will happen in sublinear time or not.

Note that, unlike the algorithms with low generic-case complexity
considered in \cite{KMSS}, this algorithm has a good chance to have
low generic-case complexity giving both ``yes" and ``no" answers.

Finally, we note that generic-case complexity depends on how one
defines the {\it asymptotic density} of a subset of inputs in the
set of all possible inputs. This, in turn, depends on how one
defines the complexity of an input. In \cite{KMSS}, complexity of an
element of a group $G$ was defined as the minimum word length of
this element with respect to a fixed generating set of $G$. In our
situation, where inputs are matrices over $\mathbb{Z}$, it is probably more
natural to define complexity $|M|$ of a matrix $M$ as the sum of the absolute values of
the entries of $M$, like we did in this paper. Yet another natural way is to use {\it Kolmogorov complexity}, i.e., speaking informally, the minimum possible size of a description of $M$. Since Kolmogorov complexity of an integer $n$ is equivalent to $\log n$, we see that for a matrix $M \in SL_2(\mathbb{Z})$, Kolmogorov complexity is equivalent
to $\log |M|$, for $|M| = \sum |m_{ij}|$, as defined in this paper. This is not the case though if $M \in SL_2(\mathbb{Q})$ since for Kolmogorov complexity of a rational number, complexity of both the numerator and denominator matters.

\bigskip

\noindent {\it Acknowledgement.} We are grateful to Norbert A'Campo,
Ilya Kapovich, and Linda Keen for helpful comments.

\baselineskip 11 pt


Apéry Limits: Experiments and Proofs

Abstract: An important component of Apéry's proof that $\zeta (3)$ is irrational involves representing $\zeta (3)$ as the limit of the quotient of two rational solutions to a three-term recurrence. We present various approaches to such Apéry limits and highlight connections to continued fractions as well as the famous theorems of Poincaré and Perron on difference equations. In the spirit of Jon Borwein, we advertise an experimental-mathematics approach by first exploring in detail a simple but instructive motivating example. We conclude with various open problems.

\section{Introduction}

A fundamental ingredient of Ap\'ery's groundbreaking proof \cite{apery} of
the irrationality of $\zeta (3)$ is the binomial sum
\begin{equation}
  A (n) = \sum_{k = 0}^n \binom{n}{k}^2 \binom{n + k}{k}^2 \label{eq:apery3}
\end{equation}
and the fact that it satisfies the three-term recurrence
\begin{equation}
  (n + 1)^3 u_{n + 1} = (2 n + 1) (17 n^2 + 17 n + 5) u_n - n^3 u_{n - 1}
  \label{eq:apery3:rec}
\end{equation}
with initial conditions $A (0) = 1$, $A (1) = 5$ --- or, equivalently but more
naturally, $A (- 1) = 0$, $A (0) = 1$. Now let $B (n)$ be the solution to
{\eqref{eq:apery3:rec}} with $B (0) = 0$ and $B (1) = 1$. Ap\'ery showed
that
\begin{equation}
  \lim_{n \rightarrow \infty} \frac{B (n)}{A (n)} = \frac{\zeta (3)}{6}
  \label{eq:apery3:lim}
\end{equation}
and that the rational approximations resulting from the left-hand side
converge too rapidly to $\zeta (3)$ for $\zeta (3)$ itself to be rational. For
details, we recommend the engaging account \cite{alf} of Ap\'ery's proof.
In the sequel, we will not pursue questions of irrationality further. Instead,
our focus will be on limits, like {\eqref{eq:apery3:lim}}, of quotients of
solutions to linear recurrences. Such limits are often called {\emph{Ap\'ery
limits}} \cite{avz-apery-limits}, \cite{yang-apery-limits}.

Jon Borwein was a tireless advocate and champion of experimental mathematics
and applied it with fantastic success. Jon was also a pioneer of teaching
experimental mathematics, whether through numerous books, such as
\cite{bb-exp1}, or in the classroom (the second author is grateful for the
opportunity to benefit from both). Before collecting known results on
Ap\'ery limits and general principles, we therefore find it fitting to
explore in detail, in Section~\ref{sec:delannoy}, a simple but instructive
example using an experimental approach. We demonstrate how to discover the
desired Ap\'ery limit; and we show, even more importantly, how the
exploratory process naturally leads us to discover additional structure that
is helpful in understanding this and other such limits. We hope that the
detailed discussion in Section~\ref{sec:delannoy} may be particularly useful
to those seeking to integrate experimental mathematics into their own
teaching.

After suggesting further examples in Section~\ref{sec:search}, we explain the
observations made in Section~\ref{sec:delannoy} by giving in
Section~\ref{sec:diffeq} some background on difference equations, introducing
the Casoratian and the theorems of Poincar\'e and Perron. In
Section~\ref{sec:cf}, we connect with continued fractions and observe that,
accordingly translated, many of the simpler examples are instances of
classical results in the theory of continued fractions. We then outline in
Section~\ref{sec:pf} several methods used in the literature to establish
Ap\'ery limits. To illustrate the limitations of these approaches, we
conclude with several open problems in Sections~\ref{sec:franel:d} and
\ref{sec:open}.

Creative telescoping --- including, for instance, Zeilberger's algorithm and
the Wilf--Zeilberger (WZ) method --- refers to a powerful set of tools that,
among other applications, allow us to algorithmically derive the recurrence
equations, like {\eqref{eq:apery3:rec}}, that are satisfied by a given sum,
like {\eqref{eq:apery3}}. In fact, as described in \cite{alf}, Zagier's
proof of Ap\'ery's claim that the sums {\eqref{eq:apery3}} and
{\eqref{eq:apery3:2}} both satisfy the recurrence {\eqref{eq:apery3:rec}} may
be viewed as giving birth to the modern method of creative telescoping. For an
excellent introduction we refer to \cite{aeqb}. In the sequel, all claims
that certain sums satisfy a recurrence can be established using creative
telescoping.

\section{A motivating example}\label{sec:delannoy}

At the end of van der Poorten's account \cite{alf} of Ap\'ery's proof, the
reader is tasked with the exercise to consider the sequence
\begin{equation}
  A (n) = \sum_{k = 0}^n \binom{n}{k} \binom{n + k}{k} \label{eq:delannoy}
\end{equation}
and to apply to it Ap\'ery's ideas to conclude the irrationality of $\ln
(2)$. In this section, we will explore this exercise with an experimental
mindset but without using the general tools and connections described later in
the paper. In particular, we hope that the outline below could be handed out
in an undergraduate experimental-math class and that the students could (with
some help, depending on their familiarity with computer algebra systems)
reproduce the steps, feel intrigued by the observations along the way, and
then apply by themselves a similar approach to explore variations or
extensions of this exercise. Readers familiar with the topic may want to skip
ahead.

The numbers {\eqref{eq:delannoy}} are known as the central Delannoy numbers
and count lattice paths from $(0, 0)$ to $(n, n)$ using the steps $(0, 1)$,
$(1, 0)$ and $(1, 1)$. They satisfy the recurrence
\begin{equation}
  (n + 1) u_{n + 1} = 3 (2 n + 1) u_n - n u_{n - 1} \label{eq:delannoy:rec}
\end{equation}
with initial conditions $A (- 1) = 0$, $A (0) = 1$. Now let $B (n)$ be the
sequence satisfying the same recurrence with initial conditions $B (0) = 0$,
$B (1) = 1$. Numerically, we observe that the quotients $Q (n) = B (n) / A
(n)$,
\begin{equation*}
  (Q (n))_{n \geq 0} = \left(0, \frac{1}{3}, \frac{9}{26},
   \frac{131}{378}, \frac{445}{1284}, \frac{34997}{100980},
   \frac{62307}{179780}, \frac{2359979}{6809460}, \ldots \right),
\end{equation*}
appear to converge rather quickly to a limit
\begin{equation*}
  L := \lim_{n \rightarrow \infty} Q (n) = 0.34657359 \ldots
\end{equation*}
When we try to estimate the speed of convergence by computing the difference
$Q (n) - Q (n - 1)$ of consecutive terms, we find
\begin{equation*}
  (Q (n) - Q (n - 1))_{n \geq 1} = \left(\frac{1}{3}, \frac{1}{78},
   \frac{1}{2457}, \frac{1}{80892}, \frac{1}{2701215}, \frac{1}{90770922},
   \ldots \right) .
\end{equation*}
This suggests the probably-unexpected fact that these are all reciprocals of
integers. Something interesting must be going on here! However, a cursory
look-up of the denominators in the {\emph{\textit{On-Line Encyclopedia of
Integer Sequences}}} (OEIS) \cite{oeis} does not result in a match. (Were we
to investigate the factorizations of these integers, we might at this point
discover the case $x = 1$ of {\eqref{eq:delannoy:x:Qdiff}}. But we hold off on
exploring that observation and continue to focus on the speed of convergence.)
By, say, plotting the logarithm of $Q (n) - Q (n - 1)$ versus $n$, we are led
to realize that the number of digits to which $Q (n - 1)$ and $Q (n)$ agree
appears to increase (almost perfectly) linearly. This means that $Q (n)$
converges to $L$ exponentially.

\begin{exercise*}
  For a computational challenge, quantify the exponential convergence by
  conjecturing an exact value for the limit of $(Q (n + 1) - Q (n)) / (Q (n) -
  Q (n - 1))$ as $n \rightarrow \infty$. Then connect that value to the
  recurrence {\eqref{eq:delannoy:rec}}.
\end{exercise*}

At this point, we are confident that, say,
\begin{equation}
  Q (50) = 0.34657359027997265470861606072908828403775006718 \ldots
  \label{eq:delannoy:Q50}
\end{equation}
agrees with the limit $L$ to more than $75$ digits. The ability to recognize
constants from numerical data is a powerful asset in an experimental
mathematician's toolbox. Several approaches to constant recognition are
lucidly described in \cite[Section~6.3]{bb-exp1}. The crucial ingredients
are integer relation algorithms such as PSLQ or those based on lattice
reduction algorithms like LLL. Readers new to constant recognition may find
the {\emph{Inverse Symbolic Calculator}} of particular value. This web
service, created by Jon Borwein, Peter Borwein and Simon Plouffe, automates
the constant-recognition process: it asks for a numerical approximation as
input and determines, if successful, a suggested exact value. For instance,
given {\eqref{eq:delannoy:Q50}}, it suggests that
\begin{equation*}
  L = \frac{1}{2} \ln (2),
\end{equation*}
which one can then easily confirm further to any desired precision. Of course,
while this provides overwhelming evidence, it does not constitute a proof.
Given the success of our exploration, a natural next step would be to repeat
this inquiry for the sequence of polynomials
\begin{equation}
  A_x (n) = \sum_{k = 0}^n \binom{n}{k} \binom{n + k}{k} x^k,
  \label{eq:delannoy:x}
\end{equation}
which satisfies the recurrence {\eqref{eq:delannoy:rec}} with the term $3 (2 n
+ 1)$ replaced by $(2 x + 1) (2 n + 1)$. An important principle to keep in
mind here is that introducing an additional parameter, like the $x$ in
{\eqref{eq:delannoy:x}}, can make the underlying structure more apparent; and
this may be crucial both for guessing patterns and for proving our assertions.
Now define the secondary solution $B_x (n)$ satisfying the recurrence with
$B_x (0) = 0$, $B_x (1) = 1$. Then, if we compute the difference of quotients
$Q_x (n) = B_x (n) / A_x (n)$ as before, we find that
\begin{equation*}
  (Q_x (n) - Q_x (n - 1))_{n \geq 1} = \left(\frac{1}{1 + 2 x},
   \frac{1}{2 (1 + 2 x) (1 + 6 x + 6 x^2)}, \ldots \right) .
\end{equation*}
Extending our earlier observation, these now appear to be the reciprocals of
polynomials with integer coefficients. Moreover, in factored form, we are
immediately led to conjecture that
\begin{equation}
  Q_x (n) - Q_x (n - 1) = \frac{1}{n A_x (n) A_x (n - 1)} .
  \label{eq:delannoy:x:Qdiff}
\end{equation}
Note that, since $Q_x (0) = 0$, this implies
\begin{equation}
  Q_x (N) = \sum_{n = 1}^N (Q_x (n) - Q_x (n - 1)) = \sum_{n = 1}^N \frac{1}{n
  A_x (n) A_x (n - 1)} \label{eq:delannoy:x:Qsum}
\end{equation}
and hence provides another way to compute the limit $L_x = \lim_{n \rightarrow
\infty} Q_x (n)$ as
\begin{equation*}
  L_x = \sum_{n = 1}^{\infty} \frac{1}{n A_x (n) A_x (n - 1)},
\end{equation*}
which avoids reference to the secondary solution $B_x (n)$.

Can we experimentally identify the limit $L_x$? One approach could be to
select special values for $x$ and then proceed as we did for $x = 1$. For
instance, we might numerically compute and then identify the following values:
\begin{equation*}
  \renewcommand{\arraystretch}{1.3}
  \begin{array}{|l|l|l|l|}
     \hline
     & x = 1 & x = 2 & x = 3\\
     \hline
     L_x & \tfrac{1}{2} \ln (2) & \tfrac{1}{2} \ln \left(\tfrac{3}{2} \right)
     & \tfrac{1}{2} \ln \left(\tfrac{4}{3} \right)\\
     \hline
   \end{array}
\end{equation*}
We are lucky and the emerging pattern is transparent, suggesting that
\begin{equation}
  L_x = \frac{1}{2} \ln \left(1 + \frac{1}{x} \right) .
  \label{eq:delannoy:x:L}
\end{equation}
A possibly more robust approach to identifying $L_x$ empirically is to fix
some values of $n$ and then expand the $Q_x (n)$, which are rational functions
in $x$, into power series. If the initial terms of these power series appear
to converge as $n \rightarrow \infty$ to identifiable values, then it is
reasonable to expect that these values are the initial terms of the power
series for the limit $L_x$. However, expanding around $x = 0$, we quickly
realize that the power series
\begin{equation*}
  Q_x (n) = \sum_{k = 0}^{\infty} q_k^{(n)} x^k
\end{equation*}
do not stabilize as $n \rightarrow \infty$, but that the coefficients increase
in size: for instance, we find empirically that
\begin{equation*}
  q_1^{(n)} = - n (n + 1), \quad q_2^{(n)} = \frac{1}{8} n (n + 1) (5 n^2 + 5
   n + 6),
\end{equation*}
and it appears that, for $k \geq 1$, $q_k^{(n)}$ is a polynomial in $n$
of degree $2 k$. Expanding the $Q_x (n)$ instead around some nonzero value of
$x$ --- say, $x = 1$ --- is more promising. Writing
\begin{equation*}
  Q_x (n) = \sum_{k = 0}^{\infty} r_k^{(n)} (x - 1)^k,
\end{equation*}
we observe empirically that
\begin{equation*}
  \left( \lim_{n \rightarrow \infty} r_k^{(n)} \right)_{k \geq 1} = \left(-
   \frac{1}{4}, \frac{3}{16}, - \frac{7}{48}, \frac{15}{128}, \ldots \right) .
\end{equation*}
Once we realize that the denominators are multiples of $k$, it is not
difficult to conjecture that
\begin{equation}
  \lim_{n \rightarrow \infty} r_k^{(n)} = (- 1)^k \frac{2^k - 1}{k \cdot 2^{k
  + 1}} \label{eq:delannoy:x:Q:c1}
\end{equation}
for $k \geq 1$. From our initial exploration, we already know that
$\lim_{n \rightarrow \infty} r_0^{(n)} = \frac{1}{2} \ln (2)$ but we could
also have (re)guessed this value as the formal limit of the right-hand side of
{\eqref{eq:delannoy:x:Q:c1}} as $k \rightarrow 0$ (forgetting that $k$ is
really an integer). Anyway, {\eqref{eq:delannoy:x:Q:c1}} suggests that
\begin{equation*}
  L_x = L_1 + \sum_{k = 1}^{\infty} (- 1)^k \frac{2^k - 1}{k \cdot 2^{k + 1}}
   \, (x - 1)^k = \frac{1}{2} \ln (2) + \frac{1}{2} \ln \left(\frac{x + 1}{2
   x} \right),
\end{equation*}
leading again to {\eqref{eq:delannoy:x:L}}. Finally, our life is easiest if we
look at the power series of $Q_x (n)$ expanded around $x = \infty$. In that
case, we find that the power series of $Q_x (n)$ and $Q_x (n + 1)$ actually
agree through order $x^{- 2 n}$. In hindsight --- and to expand our
experimental reach, it is always a good idea to reflect on the new data in
front of us --- this is a consequence of {\eqref{eq:delannoy:x:Qdiff}} and the
fact that $A_x (n)$ has degree $n$ in $x$ (so that $A_x (n) A_x (n - 1)$ has
degree $2 n - 1$). Therefore, from just the case $n = 3$ we are confident that
\begin{equation*}
  L_x = Q_x (3) + O (x^{- 7}) = \frac{1}{2 x} - \frac{1}{4 x^2} + \frac{1}{6
   x^3} - \frac{1}{8 x^4} + \frac{1}{10 x^5} - \frac{1}{12 x^6} + O (x^{- 7})
   .
\end{equation*}
At this point the pattern is evident, and we arrive, once more, at the
conjectured formula {\eqref{eq:delannoy:x:L}} for $L_x$.

\section{Searching for Ap\'ery limits}\label{sec:search}

Inspired by the approach laid out in the previous section, one can search for
other Ap\'ery limits as follows:
\begin{enumerate}
  \item Pick a binomial sum $A (n)$ and, using creative telescoping, compute a
  recurrence satisfied by $A (n)$.

  \item Compute the initial terms of a secondary solution $B (n)$ to the
  recurrence.

  \item Try to identify $\lim_{n \rightarrow \infty} B (n) / A (n)$ (either
  numerically or as a power series in an additional parameter).
\end{enumerate}
It is important to realize, as will be indicated in Section~\ref{sec:cf}, that
if the binomial sum $A (n)$ satisfies a three-term recurrence, then the
Ap\'ery limit can be expressed as a continued fraction and compared to the
(rather staggering) body of known results \cite{wall-contfrac},
\cite{jt-contfrac}, \cite{handbook-contfrac}, \cite{bvsz-cf}.

Of course, the final step is to prove and/or generalize those discovered
results that are sufficiently appealing. One benefit of an experimental
approach is that we can discover results, connections and generalizations, as
well as discard less-fruitful avenues, before (or while!) working out a
complete proof. Ideally, the processes of discovery and proof inform each
other at every stage. For instance, experimentally finding a generalization
may well lead to a simplified proof, while understanding a small piece of a
puzzle can help set the direction of follow-up experiments. Jon Borwein's
extensive legacy is filled with such delightful examples.

Of course, one could just start with a recurrence; however, selecting a
binomial sum increases the odds that the recurrence has desirable properties
(it is a difficult open problem to ``invert creative telescoping'' in the
sense of producing a binomial sum satisfying a given recurrence). Some simple
suggestions for binomial sums, as well as the corresponding Ap\'ery limits,
are as follows (in each case, we choose the secondary solution with initial
conditions $B (0) = 0$, $B (1) = 1$).
\begin{equation*}
  \setlength{\extrarowheight}{7pt}
  \renewcommand{\arraystretch}{1.3}
  \begin{array}{|>{\displaystyle}l|>{\displaystyle}l|}
     \hline
     \sum_{k = 0}^n \binom{n}{2 k} x^k & \frac{1}{\sqrt{x}} \quad
     \text{(around $x = 1$)}\\
     \hline
     \sum_{k = 0}^n \binom{n - k}{k} x^k & \frac{2}{1 + \sqrt{1 + 4 x}} \quad
     \text{(around $x = 0$)}\\
     \hline
     \sum_{k = 0}^n \binom{n}{k} \binom{n - k}{k} x^k & \frac{\arctan \left(\sqrt{4 x - 1} \right)}{\sqrt{4 x - 1}} \quad \text{(around $x =
     \tfrac{1}{4}$)}\\
     \hline
   \end{array}
\end{equation*}
\begin{example}
  \label{eg:arctan}Setting $x = 1 / 2$ in the last instance leads to the limit
  being $\arctan (1) = \pi / 4$ and therefore to a way of computing $\pi$ as
  \begin{equation*}
    \pi = \lim_{n \rightarrow \infty} \frac{4 B (n)}{A (n)},
  \end{equation*}
  where $A (n)$ and $B (n)$ both solve the recurrence $(n + 1) u_{n + 1} = (2
  n + 1) u_n + n u_{n - 1}$ with $A (- 1) = 0$, $A (0) = 1$ and $B (0) = 0$,
  $B (1) = 1$. In an experimental-math class, this could be used to segue into
  the fascinating world of computing $\pi$, a topic to which Jon Borwein,
  sometimes admiringly referred to as Dr.~Pi, has contributed so much --- one
  example being the groundbreaking work in \cite{borwein-piagm} with his
  brother Peter. Let us note that this is not a new result. Indeed, with the
  substitution $z = \sqrt{4 x - 1}$, it follows from the discussion in
  Section~\ref{sec:cf} that the Ap\'ery limit in question is equivalent to
  the well-known continued fraction
  \begin{equation*}
    \arctan (z) = \frac{z}{1 +} \, \frac{1^2 z^2}{3 +} \, \frac{2^2 z^2}{5 +}
     \, \cdots \, \frac{n^2 z^2}{(2 n + 1) +} \, \cdots
  \end{equation*}
  \cite[p.~343, eq.~(90.3)]{wall-contfrac}. The reader finds, for instance,
  in \cite[Theorem~2]{bcp-cf-tails} that this continued fraction, as well as
  corresponding ones for the tails of $\arctan (z)$, is a special case of
  Gauss' famous continued fraction for quotients of hypergeometric functions
  ${}_2 F_1$. We hope that some readers and students, in particular, enjoy the
  fact that they are able to rediscover such results themselves.
\end{example}

\begin{example}
  For more challenging explorations, the reader is invited to consider the
  binomial sums
  \begin{equation*}
    A_x (n) = \sum_{k = 0}^n \binom{n}{k} \binom{n + k}{k}^2 x^k, \quad
     \sum_{k = 0}^n \binom{n}{k} \binom{n + k}{k}^3 x^k,
  \end{equation*}
  and to compare the findings with those by Zudilin
  \cite{zudilin-appr-polylog} who obtains simultaneous approximations to the
  logarithm, dilogarithm and trilogarithm.
\end{example}

\begin{example}
  \label{eg:cy}Increasing the level of difficulty further, one may consider,
  for instance, the binomial sum
  \begin{equation*}
    A (n) = \sum_{k = 0}^n \binom{n}{k}^2 \binom{3 k}{n},
  \end{equation*}
  which is an appealing instance, randomly selected from many others, for
  which Almkvist, van Straten and Zudilin \cite[Section~4,
  \#219]{avz-apery-limits} have numerically identified an Ap\'ery limit (in
  this case, depending on the initial conditions of the secondary solution,
  the Ap\'ery limit can be empirically expressed as a rational multiple of
  $\pi^2$ or of the $L$-function evaluation $L_{- 3} (2)$, or, in general, a
  linear combination of those). To our knowledge, proving the conjectured
  Ap\'ery limits for most cases in \cite[Section~4]{avz-apery-limits},
  including the one above, remains open. While the techniques discussed in
  Section~\ref{sec:pf} can likely be used to prove some individual limits, it
  would be of particular interest to establish all these Ap\'ery limits in a
  uniform fashion.
\end{example}

Choosing an appropriate binomial sum as a starting point, the present approach
could be used to construct challenges for students in an experimental-math
class, with varying levels of difficulty (or that students could explore
themselves with minimal guidance). As illustrated by Example~\ref{eg:arctan},
simple cases can be connected with existing classical results, and
opportunities abound to connect with other topics such as hypergeometric
functions, computer algebra, orthogonal polynomials, or Pad\'e
approximation, which we couldn't properly discuss here. However, much about
Ap\'ery limits is not well understood and we believe that more serious
investigations, possibly along the lines outlined here, can help improve our
understanding. To highlight this point, we present in
Sections~\ref{sec:franel:d} and \ref{sec:open} several specific open problems
and challenges.

\section{Difference equations}\label{sec:diffeq}

In our initial motivating example, we started with a solution $A (n)$ to the
three-term recurrence {\eqref{eq:delannoy:rec}} and considered a second,
linearly independent solution $B (n)$ of that same recurrence. We then
discovered in {\eqref{eq:delannoy:x:Qsum}} that
\begin{equation*}
  B (n) = A (n) \sum_{k = 1}^n \frac{1}{k A (k) A (k - 1)} .
\end{equation*}
That the secondary solution is expressible in terms of the primary solution is
a consequence of a general principle in the theory of difference equations,
which we outline in this section. For a gentle introduction to difference
equations, we refer to \cite{kelley-peterson-diff}.

Consider the general homogeneous linear difference equation
\begin{equation}
  u (n + d) + p_{d - 1} (n) u (n + d - 1) + \cdots + p_1 (n) u (n + 1) + p_0
  (n) u (n) = 0 \label{eq:de}
\end{equation}
of order $d$, where we normalize the leading coefficient to $1$. If $u_1 (n),
\ldots, u_d (n)$ are solutions to {\eqref{eq:de}}, then their
{\emph{Casoratian}} $w (n)$ is defined as
\begin{equation*}
  w (n) = \det \begin{bmatrix}
     u_1 (n) & u_2 (n) & \cdots & u_d (n)\\
     u_1 (n + 1) & u_2 (n + 1) & \cdots & u_d (n + 1)\\
     \vdots & \vdots & \ddots & \vdots\\
     u_1 (n + d - 1) & u_2 (n + d - 1) & \cdots & u_d (n + d - 1)
   \end{bmatrix} .
\end{equation*}
This is the discrete analog of the Wronskian that is discussed in most
introductory courses on differential equations. By applying the difference
equation {\eqref{eq:de}} to the last row in $w (n + 1)$ and then subtracting
off multiples of earlier rows, one finds that the Casoratian satisfies
\cite[Lemma~3.1]{kelley-peterson-diff}
\begin{equation*}
  w (n + 1) = (- 1)^d p_0 (n) w (n)
\end{equation*}
and hence
\begin{equation}
  w (n) = (- 1)^{d n} p_0 (0) p_0 (1) \cdots p_0 (n - 1) w (0) .
  \label{eq:casoratian:rec}
\end{equation}
In the case of second order difference equations ($d = 2$), we have
\begin{equation*}
  \frac{u_2 (n + 1)}{u_1 (n + 1)} - \frac{u_2 (n)}{u_1 (n)} = \frac{u_1 (n)
   u_2 (n + 1) - u_1 (n + 1) u_2 (n)}{u_1 (n) u_1 (n + 1)} = \frac{w (n)}{u_1
   (n) u_1 (n + 1)},
\end{equation*}
which implies that we can construct a second solution from a given solution as
follows.

\begin{lemma}
  Let $d = 2$ and suppose that $u_1 (n)$ solves {\eqref{eq:de}} and that $u_1
  (n) \neq 0$ for all $n \geq 0$. Then a second solution of
  {\eqref{eq:de}} is given by
  \begin{equation}
    u_2 (n) = u_1 (n) \sum_{k = 0}^{n - 1} \frac{w (k)}{u_1 (k) u_1 (k + 1)},
    \label{eq:u2:casoratian}
  \end{equation}
  where $w (k) = p_0 (0) p_0 (1) \cdots p_0 (k - 1)$.
\end{lemma}

Here we have normalized the solution $u_2$ by choosing $w (0) = 1$: this
entails $u_2 (0) = 0$ and $u_2 (1) = 1 / u_1 (0)$. Note also that if $p_0 (0)
\neq 0$, then $w (1) \neq 0$, which implies that the solution $u_2$ is
linearly independent from $u_1$.

\begin{example}
  For the Delannoy difference equation {\eqref{eq:delannoy:rec}} and the
  solutions $A (n)$, $B (n)$ with initial conditions as before, we have $d =
  2$ and $p_0 (n) = (n + 1) / (n + 2)$, hence $w (n) = 1 / (n + 1)$. In
  particular, equation {\eqref{eq:u2:casoratian}} is equivalent to
  {\eqref{eq:delannoy:x:Qsum}}.
\end{example}

Now suppose that $p_k (n) \rightarrow c_k$ as $n \rightarrow \infty$, for each
$k \in \{ 0, 1, \ldots, d - 1 \}$. Then the characteristic polynomial of the
recurrence {\eqref{eq:de}} is, by definition,
\begin{equation*}
  \lambda^d + c_{d - 1} \lambda^{d - 1} + \cdots + c_1 \lambda + c_0 =
   \prod_{k = 1}^d (\lambda - \lambda_k)
\end{equation*}
with characteristic roots $\lambda_1, \ldots, \lambda_d$. Poincare's famous
theorem \cite[Theorem~5.1]{kelley-peterson-diff} states, under a modest
additional hypothesis, that each nontrivial solution to {\eqref{eq:de}} has
asymptotic growth according to one of the characteristic roots.

\begin{theorem}[Poincar\'e]
  \label{thm:poincare}Suppose further that the characteristic roots have
  distinct moduli. If $u (n)$ solves the recurrence {\eqref{eq:de}}, then
  either $u (n) = 0$ for all sufficiently large $n$, or
  \begin{equation}
    \lim_{n \rightarrow \infty} \frac{u (n + 1)}{u (n)} = \lambda_k
    \label{eq:poincare}
  \end{equation}
  for some $k \in \{ 1, \ldots, d \}$.
\end{theorem}

If, in addition, $p_0 (n) \neq 0$ for all $n \geq 0$ (so that, by
{\eqref{eq:casoratian:rec}}, the Casoratian $w (n)$ is either zero for all $n$
or nonzero for all $n$), then Perron's theorem guarantees that, for each $k$,
there exists a solution such that {\eqref{eq:poincare}} holds.

\section{Continued Fractions}\label{sec:cf}

In this section, we briefly connect with (irregular) continued fractions
\begin{equation*}
  C = b_0 + \frac{a_1}{b_1 +} \, \frac{a_2}{b_2 +} \, \frac{a_3}{b_3 +}
   \ldots := b_0 + \frac{a_1}{b_1 + \frac{a_2}{b_2 + \frac{a_3}{b_3 +
   \ldots}}},
\end{equation*}
as introduced, for instance, in \cite{jt-contfrac},
\cite[Entry~12.1]{berndtII} or \cite[Chapter~9]{bvsz-cf}. The $n$-th
convergent of $C$ is
\begin{equation*}
  C_n = b_0 + \frac{a_1}{b_1 +} \, \frac{a_2}{b_2 +} \, \ldots \,
   \frac{a_n}{b_n} .
\end{equation*}
It is well known, and readily proved by induction, that $C_n = B (n) / A (n)$
where both $A (n)$ and $B (n)$ solve the second-order recurrence
\begin{equation}
  u_n = b_n u_{n - 1} + a_n u_{n - 2} \label{eq:cf:rec}
\end{equation}
with $A (- 1) = 0$, $A (0) = 1$ and $B (- 1) = 1$, $B (0) = b_0$. (Note that,
if $b_0 = 0$, then the initial conditions for $B (n)$ are equivalent to $B (0)
= 0$, $B (1) = a_1$.)

Conversely, see \cite[Theorem~9.4]{bvsz-cf}, two such solutions to a
second-order difference equation with non-vanishing Casoratian correspond to a
unique continued fraction. In particular, Ap\'ery limits $\lim_{n
\rightarrow \infty} B (n) / A (n)$ arising from second-order difference
equations can be equivalently expressed as continued fractions.

\begin{example}
  The Ap\'ery limit {\eqref{eq:delannoy:x:L}} is equivalent to the continued
  fraction
  \begin{equation*}
    \frac{1}{2} \ln \left(1 + \frac{1}{x} \right) = \frac{1}{(2 x + 1) -}
     \, \frac{1^2}{3 (2 x + 1) -} \, \frac{2^2}{5 (2 x +
     1) -} \cdots \frac{n^2}{(2 n + 1) (2 x + 1) -} \cdots
  \end{equation*}
  \cite[p.~343, eq.~(90.4)]{wall-contfrac}. To see this, it suffices to note
  that, if $A_x (n)$ and $B_x (n)$ are as in Section~\ref{sec:delannoy}, then
  $n!A_x (n)$ and $n!B_x (n)$ solve the adjusted recurrence
  \begin{equation*}
    u_{n + 1} = (2 x + 1) (2 n + 1) u_n - n^2 u_{n - 1}
  \end{equation*}
  of the form {\eqref{eq:cf:rec}}.
\end{example}

The interested reader can find a detailed discussion of the continued
fractions corresponding to Ap\'ery's limits for $\zeta (2)$ and $\zeta (3)$
in \cite[Section~9.5]{bvsz-cf}, which then proceeds to proving the
respective irrationality results.

\section{Proving Ap\'ery limits}\label{sec:pf}

In the sequel, we briefly indicate several approaches towards proving
Ap\'ery limits. In case of the Ap\'ery numbers {\eqref{eq:apery3}},
Ap\'ery established the limit {\eqref{eq:apery3:lim}} by finding the
explicit expression
\begin{equation}
  B (n) = \frac{1}{6} \sum_{k = 0}^n \binom{n}{k}^2 \binom{n + k}{k}^2 \left(\sum_{j = 1}^n \frac{1}{j^3} + \sum_{m = 1}^k \frac{(- 1)^{m - 1}}{2 m^3
  \binom{n}{m} \binom{n + m}{m}} \right) \label{eq:apery3:2}
\end{equation}
for the secondary solution $B (n)$. Observe how, indeed, the presence of the
truncated sum for $\zeta (3)$ in {\eqref{eq:apery3:2}} makes the limit
{\eqref{eq:apery3:lim}} transparent. While, nowadays, it is routine
\cite{schneider-apery} to verify that the sum {\eqref{eq:apery3:2}}
satisfies the recurrence {\eqref{eq:apery3:rec}}, it is much less clear how to
discover sums like {\eqref{eq:apery3:2}} that are suitable for proving a
desired Ap\'ery limit.

Shortly after, and inspired by, Ap\'ery's proof, Beukers
\cite{beukers-irr-leg} gave shorter and more elegant proofs of the
irrationality of $\zeta (2)$ and $\zeta (3)$ by considering double and triple
integrals that result in small linear forms in the zeta value and $1$. For
instance, for $\zeta (3)$, Beukers establishes that
\begin{align}
  & (- 1)^n \int_0^1 \int_0^1 \int_0^1 \frac{x^n (1 - x)^n y^n (1 - y)^n z^n (1
  - z)^n}{(1 - (1 - x y) z)^{n + 1}} \mathrm{d} x \mathrm{d} y \mathrm{d} z \nonumber\\ ={} & A (n) \zeta
  (3) - 6 B (n), \label{eq:linearform:beukers:zeta3}
\end{align}
where $A (n)$ and $B (n)$ are precisely the Ap\'ery numbers
{\eqref{eq:apery3}} and the corresponding secondary solution
{\eqref{eq:apery3:2}}. By bounding the integrand, it is straightforward to
show that the triple integral approaches $0$ as $n \rightarrow \infty$. From
this we directly obtain the Ap\'ery limit {\eqref{eq:apery3:lim}}, namely,
$\lim_{n \rightarrow \infty} B (n) / A (n) = \zeta (3) / 6$.

\begin{example}
  In the same spirit, the Ap\'ery limit {\eqref{eq:delannoy:x:L}} can be
  deduced from
  \begin{equation*}
    \int_0^1 \frac{t^n (1 - t)^n}{(x + t)^{n + 1}} \mathrm{d} t = A_n (x) \ln
     \left(1 + \frac{1}{x} \right) - 2 B_n (x),
  \end{equation*}
  which holds for $x > 0$ with $A_n (x)$ and $B_n (x)$ as in
  Section~\ref{sec:delannoy}. We note that this integral is a variation of the
  integral that Alladi and Robinson \cite{ar-irr} have used to prove
  explicit irrationality measures for numbers of the form $\ln (1 + \lambda)$
  for certain algebraic $\lambda$.
\end{example}

As a powerful variation of this approach, the same kind of linear forms can be
constructed through hypergeometric sums obtained from rational functions. For
instance, Zudilin \cite{zudilin-arithmetic-odd} studies a general
construction, a special case of which is the relation, originally due to
Gutnik,
\begin{equation*}
  - \frac{1}{2} \sum_{t = 1}^{\infty} R_n' (t) = A (n) \zeta (3) - 6 B (n),
   \quad \text{where } R_n (t) = \left(\frac{(t - 1) \cdots (t - n)}{t (t +
   1) \cdots (t + n)} \right)^2,
\end{equation*}
which once more equals {\eqref{eq:linearform:beukers:zeta3}}. We refer to
\cite{zudilin-arithmetic-odd}, \cite{zudilin-appr-polylog} and
\cite[Section~2.3]{avz-apery-limits} for further details and references. A
detailed discussion of the case of $\zeta (2)$ is included in
\cite[Sections~9.5 and 9.6]{bvsz-cf}.

Beukers \cite{beukers-irr} further realized that, in Ap\'ery's cases, the
differential equations associated to the recurrences have a description in
terms of modular forms. Zagier \cite{zagier4} has used such modular
parametrizations to prove Ap\'ery limits in several cases, including for the
Franel numbers, the case $d = 3$ in {\eqref{eq:franel:d}}. The limits occuring
in his cases are rational multiples of
\begin{equation*}
  \zeta (2), \quad L_{- 3} (2) = \sum_{n = 1}^{\infty} \frac{\left(\frac{-
   3}{n} \right)}{n^2}, \quad L_{- 4} (2) = \sum_{n = 1}^{\infty} \frac{\left(\frac{- 4}{n} \right)}{n^2} = \sum_{n = 0}^{\infty} \frac{(- 1)^n}{(2 n +
   1)^2},
\end{equation*}
where $\left(\frac{a}{n} \right)$ is a Legendre symbol and $L_{- 4} (2)$ is
Catalan's constant (whose irrationality remains famously unproven). A general
method for obtaining Ap\'ery limits in cases of modular origin has been
described by Yang \cite{yang-apery-limits}, who proves various Ap\'ery
limits in terms of special values of $L$-functions.

\section{Sums of powers of binomials}\label{sec:franel:d}

Let us consider the family
\begin{equation}
  A^{(d)} (n) = \sum_{k = 0}^n \binom{n}{k}^d \label{eq:franel:d}
\end{equation}
of sums of powers of binomial coefficients. It is easy to see that $A^{(1)}
(n) = 2^n$ and $A^{(2)} (n) = \binom{2 n}{n}$. The numbers $A^{(3)} (n)$ are
known as {\emph{Franel numbers}} \cite[A000172]{oeis}. Almost a century
before the availability of computer-algebra approaches like creative
telescoping, Franel \cite{franel94} obtained explicit recurrences for
$A^{(3)} (n)$ as well as, in a second paper, $A^{(4)} (n)$, and he conjectured
that $A^{(d)} (n)$ satisfies a linear recurrence of order $\lfloor (d + 1) / 2
\rfloor$ with polynomial coefficients. This conjecture was proved by Stoll in
\cite{stoll-rec-bounds}, to which we refer for details and references. It
remains an open problem to show that, in general, no recurrence of lower order
exists.

Van der Poorten \cite[p.~202]{alf} reports that the following Ap\'ery
limits in the cases $d = 3$ and $d = 4$ (in which case the binomial sums
satisfy second-order recurrences like Ap\'ery's sequences) have been
numerically observed by Tom Cusick:
\begin{equation}
  \lim_{n \rightarrow \infty} \frac{B^{(3)} (n)}{A^{(3)} (n)} =
  \frac{\pi^2}{24}, \quad \lim_{n \rightarrow \infty} \frac{B^{(4)}
  (n)}{A^{(4)} (n)} = \frac{\pi^2}{30} . \label{eq:franel:L34}
\end{equation}
In each case, $B^{(d)} (n)$ is the (unique) secondary solution with initial
conditions $B^{(d)} (0) = 0$, $B^{(d)} (1) = 1$. The case $d = 3$ was proved
by Zagier \cite{zagier4} using modular forms. Since the case $d = 4$ is
similarly connected to modular forms \cite{cooper-level10}, we expect that
it can be established using the methods in \cite{yang-apery-limits},
\cite{zagier4} as well. Based on numerical evidence, following the approach
in Section~\ref{sec:search}, we make the following general conjecture
extending {\eqref{eq:franel:L34}}:

\begin{conjecture}
  \label{conj:franel:2}For $d \geq 3$, the minimal-order recurrence
  satisfied by $A^{(d)} (n)$ has a unique solution $B^{(d)} (n)$ with $B^{(d)}
  (0) = 0$ and $B^{(d)} (1) = 1$ that also satisfies
  \begin{equation}
    \lim_{n \rightarrow \infty} \frac{B^{(d)} (n)}{A^{(d)} (n)} = \frac{\zeta
    (2)}{d + 1} . \label{eq:conj:franel:2}
  \end{equation}
\end{conjecture}

Note that for $d \geq 5$, the recurrence is of order $\geq 3$, and
so the solution $B^{(d)} (n)$ is not uniquely determined by the two initial
conditions $B^{(d)} (0) = 0$ and $B^{(d)} (1) = 1$.
Conjecture~\ref{conj:franel:2} asserts that precisely one of these solutions
satisfies {\eqref{eq:conj:franel:2}}.

Subsequent to making this conjecture, we realized that the case $d = 5$ was
already conjectured in \cite[Section~4.1]{avz-apery-limits} as sequence
\#22. We are not aware of previous conjectures for the cases $d \geq 6$.
We have numerically confirmed each of the cases $d \leq 10$ to more than
$100$ digits.

\begin{example}
  \label{eg:franel:5}For $d = 5$, the sum $A^{(5)} (n)$ satisfies a recurrence
  of order $3$, spelled out in \cite{perlstadt-franel}, of the form
  \begin{equation}
    (n + 3)^4 p (n + 1) u (n + 3) + \ldots + 32 (n + 1)^4 p (n + 2) u (n) = 0
    \label{eq:franel:5:rec}
  \end{equation}
  where $p (n) = 55 n^2 + 33 n + 6$. The solution $B^{(5)} (n)$ from
  Conjecture~\ref{conj:franel:2} is characterized by $B^{(5)} (0) = 0$ and
  $B^{(5)} (1) = 1$ and insisting that the recurrence
  {\eqref{eq:franel:5:rec}} also holds for $n = - 1$ (note that this does not
  require a value for $B^{(5)} (- 1)$ because of the term $(n + 1)^4$).
  Similarly, for $d = 6, 7, 8, 9$ the sequence $B^{(d)} (n)$ in
  Conjecture~\ref{conj:franel:2} can be characterized by enforcing the
  recurrence for small negative $n$ and by setting $B^{(d)} (n) = 0$ for $n <
  0$. By contrast, for $d = 10$, there is a two-dimensional space of sequences
  $u (n)$ solving {\eqref{eq:franel:5:rec}} for all integers $n$ with the
  constraint that $u (n) = 0$ for $n \leq 0$. Among these, $B^{(10)} (n)$
  is characterized by $B^{(10)} (1) = 1$ and $B^{(10)} (2) = 381 / 4$.
\end{example}

Now return to the case $d = 5$ and let $C^{(5)} (n)$ be the third solution to
the same recurrence with $C^{(5)} (0) = 0$, $C^{(5)} (1) = 1$, $C^{(5)} (2) =
\frac{48}{7}$. Numerical evidence suggests that we have the Ap\'ery limits
\begin{equation*}
  \lim_{n \rightarrow \infty} \frac{B^{(5)} (n)}{A^{(5)} (n)} = \frac{1}{6}
   \zeta (2), \quad \lim_{n \rightarrow \infty} \frac{C^{(5)} (n)}{A^{(5)}
   (n)} = \frac{3 \pi^4}{1120} = \frac{27}{112} \zeta (4) .
\end{equation*}
Extending this idea to $d = 5, 6, \ldots, 10$, we numerically find Ap\'ery
limits $C^{(d)} (n) / A^{(d)} (n) \rightarrow \lambda \zeta (4)$ with the
following rational values for $\lambda$:
\begin{equation*}
  \frac{27}{112}, \frac{4}{21}, \frac{37}{240}, \frac{7}{55}, \frac{47}{440},
   \frac{1}{11} .
\end{equation*}
These values suggest that $\lambda$ can be expressed as a simple rational
function of $d$:

\begin{conjecture}
  For $d \geq 5$, the minimal-order recurrence satisfied by $A^{(d)} (n)$
  has a unique solution $C^{(d)} (n)$ with $C^{(d)} (0) = 0$ and $C^{(d)} (1)
  = 1$ that also satisfies
  \begin{equation*}
    \lim_{n \rightarrow \infty} \frac{C^{(d)} (n)}{A^{(d)} (n)} = \frac{3 (5
     d + 2)}{(d + 1) (d + 2) (d + 3)} \zeta (4) .
  \end{equation*}
\end{conjecture}

More generally, we expect that, for $d \geq 2 m + 1$, there exist such
Ap\'ery limits for rational multiples of $\zeta (2), \zeta (4), \ldots,
\zeta (2 m)$. It is part of the challenge presented here to explicitly
describe all of these limits. As communicated to us by Zudilin, one could
approach the challenge, uniformly in $d$, by considering the rational
functions
\begin{equation*}
  R_n^{(d)} (t) = \left(\frac{(- 1)^t n!}{t (t + 1) \cdots (t + n)}
   \right)^d
\end{equation*}
in the spirit of \cite{zudilin-arithmetic-odd},
\cite{zudilin-appr-polylog} and \cite[Section~2.3]{avz-apery-limits}, as
indicated in Section~\ref{sec:pf}.

\section{Further challenges and open problems}\label{sec:open}

Although many things are known about Ap\'ery limits, much deserves to be
better understood. The explicit conjectures we offer in the previous section
can be supplemented with similar ones for other families of binomial sums. In
addition, many conjectural Ap\'ery limits that were discovered numerically
are listed in \cite[Section~4]{avz-apery-limits} for sequences that arise
from fourth- and fifth-order differential equations of Calabi--Yau type. As
mentioned in Example~\ref{eg:cy}, it would be of particular interest to
establish all these Ap\'ery limits in a uniform fashion.

It is natural to wonder whether a better understanding of Ap\'ery limits can
lead to new irrationality results. Despite considerable efforts and progress
(we refer the reader to \cite{zudilin-arithmetic-odd} and
\cite{brown-apery} as well as the references therein), it remains a
wide-open challenge to prove the irrationality of, say, $\zeta (5)$ or
Catalan's constant. As a recent promising construction in this direction, we
mention Brown's cellular integrals \cite{brown-apery} which are linear forms
in (multiple) zeta values that are constructed to have certain vanishing
properties that make them amenable to irrationality proofs. In particular,
Brown's general construction includes Ap\'ery's results as (unique) special
cases occuring as initial instances.

In another direction, it would be of interest to systematically study
$q$-analogs and, in particular, to generalize from difference equations to
$q$-difference equations. For instance, Amdeberhan and Zeilberger
\cite{az-qapery} offer an Ap\'ery-style proof of the irrationality of the
$q$-analog of $\ln (2)$ based on a $q$-version of the Delannoy numbers
{\eqref{eq:delannoy}}.

Perron's theorem, which we have mentioned after Poincar\'e's
Theorem~\ref{thm:poincare}, guarantees that, for each characteristic root
$\lambda$ of an appropriate difference equation, there exists a solution $u
(n)$ such that $u (n + 1) / u (n)$ approaches $\lambda$. We note that, for
instance, Ap\'ery's linear form {\eqref{eq:linearform:beukers:zeta3}} is
precisely the unique (up to a constant multiple) solution corresponding to the
$\lambda$ of smallest modulus. General tools to explicitly construct such
Perron solutions from the difference equation would be of great utility.

\begin{acknowledgment}
We are grateful to Alan Sokal for improving the
exposition by kindly sharing lots of careful suggestions and comments. We also
thank Wadim Zudilin for helpful comments, including his suggestion at the end
of Section~\ref{sec:franel:d}, and references.
\end{acknowledgment}

Apéry Limits: Experiments and Proofs