Does $(X'X)^{-1}$ always exist? I'm studing Machine Learning theory and I have a questions about Normal Equation. Normal Equation is:
$\Theta = (X'X)^{-1}X'Y\tag 1$
I now that ( in some cases) we can use this other equation:
$\Theta = X^{-1}Y\tag 2$
But the problem is that $X$ might not have an inverse, so it's not recommended to use $(2)$.
My question is: If $(2)$ is not usually used because $X$ might not have an inverse, does $X'X$ in $(1)$ always have an inverse?
Thank you for everyone!
$(X'X)^{-1}$ is NOT always invertible. Consider X a row vector, then $X'X$ is a matrix with rank 1.
In fact, $(X'X)^{-1}X'$ is the MP pseudo inverse of X, a generalization of inverse $X^{-1}$.
For the standard least squares regression model,
$$y=X\beta+\varepsilon, \quad y\in\mathbb{R}^n,\quad \beta\in\mathbb{R}^m$$
the optimization problem  as you may now know is
$$\hat{\beta}:=\min_{b\in \mathbb{R}^m}\|Xb -y\|_2$$
where $\|\;\|_2$ stands for the Euclidean norm own $\mathbb{R^m}$. This is equivalent to finding the best approximation to $y$ through vectors in the range of the linear operator $b\mapsto Xb$. From linear algebra it is known that such best approximation $y'\in X(\mathbb{R}^m)$ exists. In fact, it has the property that $\hat{y}\in \big((\mathbb{R}^m)\big)^\perp$, that is
$$(\hat{y}-y)\cdot Xb=0$$
for all $b\in\mathbb{R}^m$. Since the range of the liner operator described above is generated by the columns of $X$, this means that
$$ (\hat{y}-y)^\top X=0$$
Thus, there is at least one $\hat{\beta}$ such that $hat{y}=X\hat{\beta}$. When $X$ is of full rank, then $X^\top X$ is invertible and so $\hat{b}=(X^\top X)^{-1}X^\top\hat{y}$ is unique. If not, then there are infinitely many $\hat{b}$'s that satisfy $\hat{y}=X\hat{b}$.
Among all those possible $\hat{b}$'s, there is a special $\hat{\beta}$ that also has the smallest Eucliden norm own $\mathbb{R}^m$.
It turns out that such $\hat{\beta}$ is is of the form $A\hat{y}$ where $A$ is a matrix that satisfies four properties:
$$AXA=A$$
$$XAX=X$$
$$(AX)^*=AX$$
$$(XA)^*=XA$$
where for any matrix $B$, $B^*$ is the transpose conjugate of $B$. The matrix $A$ is unique and is called Moore-Pensroe's pseudo inverse of $X$.