Why $X^TX$ be the identity matrix is equvalent with saying there is no correlation between features? This was mentioned in the lecture as one of the assumptions for the problem of finding a hard/soft thresholding estimator. There was another time that my professor mentioned this assumption, so I think it is important.
But I don't know why - why the design matrix $X$ being orthonormal is equivalent with saying there is no correlation between features? Name $x_1,...,x_p$ the column of my design matrix $X$, then orthonormal means that for any $x_i, x_j$, their inner product is 0; on the other hand, no correlation is equivalent as saying the matrix cov($x_i,x_j$)=0. But the first one is purely linear algebra result, whereas the second one involves with probability of $x_i,x_j$.

$X^TX=XX^T=I$ is one definition of an orthogonal matrix. This means that each column (and equivalently, each row) is orthogonal to other columns (or rows).
As a consquence, there is no correlation between the columns (or rows) because the inner product between distinct columns (or rows) will always be $0$.
