When is $YE(Y\mid\mathcal{G}) = E(Y^2\mid\mathcal{G})$? I know this is true when $Y \in \mathcal{G}$, but is there a better (less restrictive) condition that can allow this?
In particular, I want to know when I can say that $E(YE(Y\mid\mathcal{G})) = E(E(Y^2\mid\mathcal{G}))$, which is, of course, equal to $EY^2$.

No.
If I understand correctly your notation, then $Y\in\mathcal{G}$ means that $Y$ is measurable with respect to $\mathcal{G}$. In particular you have $E(Y|\mathcal{G}),E(Y^2|\mathcal{G})\in\mathcal{G}$.
But then, from the equation $YE(Y|\mathcal{G}) = E(Y^2|\mathcal{G})$ you necessarily have that $Y\in\mathcal{G}$.
