Question about understanding Inner product Inner product is a pairing that takes two vectors $\mathbf V, \mathbf W$ and produces a real number $\langle\mathbf V; \mathbf W \rangle$. 
I have a formula $\langle Y_n -\hat{Y_n^*},X_n^j\rangle,j=1,\ldots,n$, where $X_n^j$ is the $j$th column of matrix X, $Y_n$ and $\hat{Y_n^*}$ are vectors, does it mean inner product?
If it is inner product, $\langle Y_n -\hat{Y_n^*},X_n^j \rangle=X_n^T (Y_n-\hat{Y_n^*})$ is correct or not?

Yes, that notation probably means an inner product of a vector and a matrix column. 
An inner product can be interpreted as the (matrix) product of a $1 \times n$ matrix and a $n \times 1$ matrix. This isn't quite correct because the inner product gives you a real number whereas the matrix product gives you a $ 1 \times 1$ matrix. But, close enough.
So, if you think of a vector as a column (a $n \times 1$ matrix), then
$$
\langle U, V \rangle = U^T * V =V^T * U
$$
where the "$*$" denotes an ordinary matrix product.
Your transpose formula is not quite correct. On the right-hand side, you need to use the $j$-th row of $X^T$, and you have to interpret the vector as a $n \times 1$ matrix (i.e. as a column).
