Transformation of xy plane to polar coordinates. (What would be the bound of polar coordinate?) I have a double integral 
$$\int_0^a \int_0^x (x^2+y^2)^{1/2} \operatorname d y \operatorname d x$$

So, I am double-integrating $r^2$ 

What would be the region of the polar coordinate..?
$$\begin{align}
\int_0^a \int_0^x (x^2+y^2)^{1/2} dy dx &amp;= \int_{0}^{\pi /4} \int_0^{a/\cos \phi} \rho^2 d\rho d\phi\\
&amp;= \frac{a^3}{3}\int_{0}^{\pi /4} \sec^3(\phi) d\phi
\end{align}$$

which can be evaluated in closed form with some effort.
Your region of integration goes from $0$ to $a$ in the horizontal direction and from $0$ to $x$ in the vertical direction, so it is an isosceles right triangle bounded by $(0,0),\ (a,0),\ (a,a)$  What is the range of $\theta$ occupied by the triangle?

A picture is below with $a \approx 4.5$.  The range of $\theta$ is $0$ to $\pi/4$  You also need to be able to see that the inner integral (over $r$) ranges from $0$ to $a/\cos \theta$