$I = \int_{0}^\infty t^2 e^{-t^2/2} dt$ Q: Evaluate the integral $I = \int_\limits{0}^\infty t^2 e^{-t^2/2} dt$
Hint, write $I^2$ as the following iterated integral and convert to polar coordinates:
\begin{align*}
  I^2 &= \int_\limits{0}^\infty \int_\limits{0}^\infty x^2 e^{-x^2/2} \cdot y^2 e^{-y^2/2} \, dx \, dy \\
\end{align*}
I can see the final answer is $\frac{\pi}{2}$ but I don't see how to get this.
This problem is very similar to the Gaussian Integral: $I = \int_\limits{-\infty}^\infty e^{-x^2} dx = \sqrt{\pi}$. I can follow the derivation to this.
The Gaussian Integral technique of converting to polar coordinates doesn't seem to work as cleanly on this problem.
I can convert to polar coordinates:
\begin{align*}
  I^2 &= \int_\limits{0}^\infty \int_\limits{\pi/2}^\pi r^5 \sin^2 \theta \cos^2 \theta e^{-r^2/2} \, d\theta \, dr \\
\end{align*}
That doesn't look easy to evaluate.

If you want to use  your original hint, you have:  $$I^2 = \int_\limits{0}^\infty \int_\limits{\pi/2}^\pi r^5 \sin^2 \theta \cos^2 \theta e^{-r^2/2} \, d\theta \, dr $$
First, we can switch the order of integration and start with $$\int_\limits{0}^\infty r^5 e^{-r^2/2}dr=\int_\limits{0}^\infty (r^4)(r e^{-r^2/2})dr $$
That can be integrted by parts twice, using the fact that $\int re^{-r^2/2}dr=-e^{-r^2/2}$, and one final integration yields the result $$\int_\limits{0}^\infty r^5 e^{-r^2/2}dr=8$$
And our integral becomes$I^2 =\int_\limits{\pi/2}^\pi 8\sin^2 \theta \cos^2 \theta d\theta$
Using the identity $\cos^2(x)\sin^2(x)=\frac{1}{4}\sin^2(2\theta)$ we have $I^2 =\int_\limits{\pi/2}^\pi 8 \frac{1}{4}\sin^2(2\theta) d\theta=\int_\limits{\pi/2}^\pi 2\sin^2(2\theta) d\theta$ And so
$$I^2=\int_\limits{\pi/2}^\pi 2\sin^2(2\theta) d\theta=\frac{\pi}{2}$$
$$I=\sqrt\frac{\pi}{2}$$
