Let $T : \mathbb{R}^3 \to \mathbb{R}^2$ be defined by $T(a, b, c) = (a+b, 2a-c)$. Determine $T^{-1} (1, 11)$. Let $T : R^3 \to R^2$ be defined by $T(a, b, c) = (a+b, 2a-c)$. Determine $T^{-1} (1, 11)$. (Exercise 3.3.6 from Linear Algebra by Friedberg)


The solution manual solve a system of linear equation $ a + b =1 $ and $ 2a - c = 11$, and obtain 
$T^{-1} (1, 11) = \{(a, 1-a, 2a - 11) : a \in \mathbb{R}\}$.

But, my question is whether $T$ is invertible or not. I have a theorem that $T$ is invertible iff $[T]_{\beta}^{\gamma}$ is invertible, where $\beta$ and $\gamma$ are standard ordered basis for $\mathbb{R}^3$ and $\mathbb{R}^2$. Here, $[T]_{\beta}^{\gamma}$ is $2$ by $3$ matrix, which implies $T$ is not invertible. Am I missing something or is this question wrong?
Your thinking is correct; only square matrices can have an inverse in the usual sense of one-to-one mapping. The fact that $T^{-1}(1,11)$ is an infinite set also means that $T$ is non-invertible.

You can still write $T^{-1}$, but this is the preimage function that takes a codomain vector and returns all its preimages in the domain. If $T$ is an invertible transformation, $T^{-1}$ will be a proper one-vector-to-one-vector function.
The notation $T^{-1} (1, 11)$ is not completely correct. It should read $T^{-1} \{(1, 11)\}$. This means

$$T^{-1} \{(1, 11)\}=\{(a,b,c) \in \mathbb R^3 : T(a,b,c)=(1,11)\}.$$

Hence $T^{-1} \{(1, 11)\}$ is the set of solutions of the linear system

$a+b=1$;

$2a-c=11.$