What is my error in this matrix / least squares derivation? I'm doing a simple problem in linear algebra.  It is clear that I have done something wrong, but I honestly can't see what it is.
let, 
$y = Ax$, $y_{ls} = Ax_{ls}$ where A is skinny and full rank, and $x_{ls} = (A^T A)^{-1}A^Ty$ is the standard least squares approximation.
Now, I have tried to compute $y^Ty_{ls}$.
$y^Ty_{ls} = y^TAx_{ls} = x^TA^TA(A^TA)^{-1}A^Ty$
The $A^TA(A^TA)^{-1}$ seems it should cancel to the identity leaving
$x^TA^Ty = y^Ty$.
Clearly this should not be true.
What have I done wrong here?

The issue is this: your problem is overdetermined and there is no solution vector $x$ such that
$$
\mathbf{A}x - b = 0.
$$
Instead of asking for an exact solution, we relax requirements and ask for the best solution. Picking the $2-$norm, the best solution is 
$$
x_{LS} = \left\{ x\in\mathbb{C}^{n} \colon \lVert \mathbf{A} x_{LS} - b \rVert_{2}^{2} \text{ is minimized} \right\}.
$$
Summary: 
$$
\mathbf{A} x_{LS} \ne b,
$$
instead $\mathbf{A}x_{LS}$ is as close as possible to $b$ as measured in the $2-$norm.
