How to minimize $f(x) = \|Ax-b\|$ Solve the problem of minimizing $f(x) = ||Ax-b||$. Consider all the cases and interpret geometrically.
If we write
$$\|Ax-b\| = (a_{11}x_1 + \cdots + a_{1n}x_n - b_1)^2 + \cdots +  (a_{n1}x_1 + \cdots + a_{nn}x_n - b_1)^2$$
then
$$\frac{\partial \|Ax-b\|}{\partial x_j} = 2(a_{11}x_1 + \cdots + a_{1n}x_n - b_1)a_{1j} + \cdots +  (a_{n1}x_1 + \cdots + a_{nn}x_n - b_1)a_{nj}$$
If I try to do $\frac{\partial \|Ax-b\|}{\partial x_j} = 0$ I get nothing useful. For $x$ to be a minimizer, I have to have gradient $0$ and hessian positive definite. If we do the hessian just to see:
$$\frac{\partial^2 \|Ax-b\|}{\partial x_k\partial x_j} = 2a_{1k}a_{1j} + \cdots + 2a_{nk}a_{nj}$$
I see nothing useful here. 
I think the geometric interpretation comes from the conditions for the gradient to be $0$ and the hessian to be $>0$, but I don't find these conditions useful. 
Any ideas?

Alternative:
In case you want to stick to $\lVert Ax - b \rVert = \mathrm{tr} \left[ \left(\left( Ax - b \right)^T \left( Ax - b \right) \right)^{\frac{1}{2}} \right]$, not the squared $\ell_2$ norm, i.e., $\lVert Ax - b \rVert^2$ (even though the final solution is same), here is an alternative.
We will utilize the following the identities 


*

*Trace and Frobenius product relation $$\left\langle A, B \right\rangle={\rm tr}(A^TB)$$ or $$\left\langle A^T, B \right\rangle ={\rm tr}(AB)$$

*Cyclic property of Trace/Frobenius product 
\begin{align}
\left\langle A, B C \right\rangle 
 &= \left\langle AC^T, B \right\rangle  \\
 &= \left\langle B^T A, C \right\rangle \\
 &= {\text{etc.}} \cr
\end{align}


We obtain the differential first, and then the gradient.
Moreover, you can utilize this trace differential result
\begin{align}
  d\,{\rm tr}(f(M)) 
  &= \left\langle f^\prime(M^T), \ dM \right\rangle ,\\
\end{align}
where $M = \left( Ax - b \right)^T \left( Ax - b \right)$ in the considered case and $f(\cdot) = \sqrt(\cdot)$.
So, 
\begin{align}
 d\, \lVert Ax - b \rVert &=  d\, \mathrm{tr} \left[ \left(\left( Ax - b \right)^T \left( Ax - b \right) \right)^{\frac{1}{2}} \right] \\
    &= \left\langle \frac {1} {2} \underbrace{\left[ \left(\left( Ax - b \right)^T \left( Ax - b \right) \right)^{-\frac{1}{2}} \right] }_{= \frac{1}{\lVert Ax - b \rVert}} , \ d\left[ \left(\left( Ax - b \right)^T \left( Ax - b \right) \right) \right] \right\rangle \\
    &= \left\langle \frac {1} {2 \lVert Ax - b \rVert}, \  dx^T A^T \left(Ax-b \right) + \left(Ax-b \right)^T A \ dx \right\rangle \\
    &= \left\langle \frac {1} {2 \lVert Ax - b \rVert}, \  2\left(Ax-b \right)^T A \ dx \right\rangle \\
    &= \left\langle \frac {A^T \left(Ax-b \right)} {\lVert Ax - b \rVert}, \    \ dx \right\rangle \\
\end{align}
Thus the derivative is 
$$ \eqalign {
 \frac { \partial} {\partial x}\lVert Ax - b \rVert &= \frac {A^T \left(Ax-b \right)} {\lVert Ax - b \rVert}  . \cr
} $$
Now, you can set the derivative to $0$ and the solution is the same as the above, i.e., 
\begin{align}
x = \left(A^T A \right)^{-1} A^T b \ .
\end{align}
