Orthogonality about Irreducible unitary representations of compact group Question: Suppose $K$ is a compact subgroup of $GL(n,C)$ and $H$ is a Hilbert space.  ($\pi$,$H$) is a unitary  representation of $K$. Let $H_1$ and $H_2$ be two irreducible invariant subspace of $H$. Then $H_1$ and $H_2$ are orthogonal if $\pi$ restrict on $H_1$ and $H_2$ are nonisomorphic. 

Here is my attempt: Let $y_i\in H_i$ fixed. Put $T: H_1\mapsto H_2; x\mapsto \int_K \langle x,\pi(\kappa)y_1\rangle\cdot\pi(\kappa)y_2d\kappa$. One can easily check $T$ is $K$-invariant. By Schur lemma, if $T$ is not isomorphism, then $T=0$. Thus $\langle Tx_1,x_2\rangle=0$ for every $x_i\in H_i$. This is equivalent to say that every matrix coefficient of $(\pi,H_1)$ is orthogonal in $L^2(K)$ to every matrix  coefficient of $(\pi,H_2)$. But how can I say $H_1$ and $H_2$ themselves are orthogonal? Or can we chose $y_i$ to make sure $T$ is not 0? If we can, Schur lemma will implies $T$ is a isomorphism. Or maybe we can use peter-weyl theorem? 

Any help or advice will be appropriated, thanks!

Your representation is probably continuous. If it is not then your integrals don't have to be well-defined. Moreover, your $T$ satisfies $g.T(x)=T(g.x)$, not only $g.T(x)\in T(H_1)$.

For $y\in H_2,x\in H_1$,
$$T_y(x)= \int_K \langle x,k.y\rangle k.ydk \in H_2$$ Since the $K$-action on $H_2$ is unitary $$T_y(g.x)=\int_K \langle g.x,k.y\rangle k.ydk=\int_K \langle x,g^{-1}k.y\rangle k.ydk=\int_K \langle x,k.y\rangle (gk).yd(gk)=g.T_y(x)$$
Since $H_1,H_2$ are irreducible, either $T_yH_1=H_2$, $\ker(T_y)=0$ and $T_y$ is an isomorphism $H_1\to H_2$, or $T_y H_1=0$.

(if $H_1$ is finite dimensional then $T_y^{-1}$ is continuous, but it is unclear what theorem we need to assume it is. If $T_y^{-1}$ is not continuous then it is sloppy to say it is an isomorphism)

Thus if $(K,H_1),(K,H_2)$ are non-isomorphic then $$0=\langle T_y(x),x\rangle=\int_K \langle x,k.y\rangle \langle k.y,x\rangle dk=\int_K |\langle x,k.y\rangle|^2 dk \implies \langle x,y\rangle=0$$
