If $A=\left(\begin{smallmatrix} 1 & \tan x\\ -\tan x & 1 \end{smallmatrix}\right)$ and $f(x)=\det(A^TA^{-1})$, then what is $f(f(f(f \cdots f(x))))$? 
Let $$A := \begin{bmatrix} 1 & \tan x\\ -\tan x & 1\end{bmatrix}$$ and $$f(x) := \det(A^TA^{-1})$$ Which of the following can not be the value of $f(f(f(f \cdots f(x))))$?

*

*$f^n(x)$

*$1$

*$f^{n-1}(x)$

*$nf(x)$
where $n \geq 2$.


I found $$f(x)=\frac{\det(A^T)}{\det(A)}=\frac{\det(A)}{\det(A)}=1.$$
But I could not figure out the answer. Please help me.

If we assume that the domain of $f$ is the maximal open subset of $\Bbb{R}$ on which $f$ can be defined, then the domain of $f$ is $\Bbb{R} \setminus \{ (2m+1)\pi/2 : m \in \Bbb{Z} \}$. In particular, $1$ lies in the domain of $f$ and we can legitimately talk about $f(1)$.
You have shown correctly that $f(x) = 1$ for any $x$ in the domain of $f$. Thus, $f \circ f(x) = f(1) = 1$. By induction, $(f \circ \dotsb \circ f)(x) = 1$ for arbitrary number of compositions of $f$.
Thus, it equals $f^n(x)$ as well as $1$ as well as $f^{n-1}(x)$ for any $n \geq 2$. But, it cannot equal $nf(x) = n$ for $n \geq 2$. Thus, the correct option is (4).
