Q:

Ramanujan theta function on matrices: When does $\sum A^{n(n-1)/2} B^{n(n+1)/2}$ converge?

Assume that $AB=BA$. The infinite sum
$$\sum_{n\in\mathbb Z}A^{n(n-1)/2}B^{n(n+1)/2}$$
$$=\cdots+A^3B^1+A^1B^0+A^0B^0+A^0B^1+A^1B^3+A^3B^6+A^6B^{10}+\cdots$$
converges unconditionally if and only if
$$\sum_{n\in\mathbb Z}\lVert A^{n(n-1)/2}B^{n(n+1)/2}\rVert$$
converges (in any norm). Let's use the Frobenius norm $\lVert A\rVert^2=\text{tr}(A^TA)$, which has the property $\lVert AB\rVert\leq\lVert A\rVert\lVert B\rVert$. Applying the ratio test,
$$\frac{\lVert A^{n(n+1)/2}B^{(n+1)(n+2)/2}\rVert}{\lVert A^{n(n-1)/2}B^{n(n+1)/2}\rVert}=\frac{\lVert A^{n(n-1)/2}B^{n(n+1)/2}(AB)^nB\rVert}{\lVert A^{n(n-1)/2}B^{n(n+1)/2}\rVert}$$
$$\leq\frac{\lVert A^{n(n-1)/2}B^{n(n+1)/2}\rVert\,\lVert(AB)^nB\rVert}{\lVert A^{n(n-1)/2}B^{n(n+1)/2}\rVert}=\lVert(AB)^nB\rVert$$
so if $(AB)^n$ approaches $0$, then the sum converges. (And if ever the denominator vanishes, preventing application of the ratio test, then all later terms vanish, so the sum is finite and trivially converges.)
Is the converse true? If the sum converges, does $(AB)^n\to0$? Of course the individual terms $A^{n(n-1)/2}B^{n(n+1)/2}\to0$, but I don't see how to get from here to $(AB)^n\to0$.

A:

Negating $n$ in each term is equivalent to swapping $A$ and $B$. Convergence of the sum over all integers requires both $A^{n(n-1)/2}B^{n(n+1)/2}$ and $A^{n(n+1)/2}B^{n(n-1)/2}$ to approach $0$. So there must be some $l\in\mathbb N$ such that $\lVert A^{n(n-1)/2}B^{n(n+1)/2}\rVert<1$ for all $n\geq l$, and $m\in\mathbb N$ such that $\lVert A^{n(n+1)/2}B^{n(n-1)/2}\rVert<1$ for all $n\geq m$. In particular, for $k=\max(l,m)$,
$$\lVert(AB)^{k^2}\rVert=\lVert A^{k(k-1)/2}B^{k(k+1)/2}A^{k(k+1)/2}B^{k(k-1)/2}\rVert$$
$$\leq\lVert A^{k(k-1)/2}B^{k(k+1)/2}\rVert\lVert A^{k(k+1)/2}B^{k(k-1)/2}\rVert<1.$$
Any natural number can be written as $n=ak^2+b$ for some $a\in\mathbb N$ and $0\leq b<k^2$, and thus
$$\lVert(AB)^n\rVert\leq\lVert(AB)^{k^2}\rVert^a\lVert AB\rVert^b\leq\lVert(AB)^{k^2}\rVert^a\max(1,\lVert AB\rVert^{k^2}).$$
The factor on the right is fixed, and $\lVert(AB)^{k^2}\rVert^a\to0$ as $a\to\infty$, so we can conclude that $(AB)^n\to0$.

Convergence of the sum over the positive integers is not sufficient. Consider real numbers $AB=1,\;0<B<1$:
$$\sum A^{n(n-1)/2}B^{n(n+1)/2}=\sum(AB)^{n(n-1)/2}B^n=\sum B^n$$
converges over $\mathbb N$, but diverges over $\mathbb Z$, and $(AB)^n\not\to0$.

