Differentiation under the integral sign - what transformations to use? Need some help with this integral 
$$I (\alpha) = \int_1^\infty  {\arctan(\alpha x) \over x^2\sqrt{x^2-1}} dx$$
Taking the first derivative with respect to $\alpha$
$$I' (\alpha) = \int_1^\infty { dx\over (1+\alpha^2 x^2) x\sqrt{x^2-1} }$$
What transformations to use in order to solve $I'(\alpha)$?

Substitute 
$$u=\sqrt{x^2-1}\implies du=\frac{x}{\sqrt{x^2-1}}dx\implies dx=\frac{\sqrt{x^2-1}}{x}du$$
Then
$$\int { dx\over (1+\alpha^2 x^2) x\sqrt{x^2-1} }=\int { du\over x^2(1+\alpha^2 x^2)}=\int { du\over (u^2+1)(a^2u^2+a^2+1) }$$
Perform partial fraction decomposition
$$\int { du\over (u^2+1)(a^2u^2+a^2+1) }=\int\frac{du}{u^2+1}-a^2\int\frac{du}{a^2u^2+a^2+1}$$
Sure you know that 
$$\int\frac{du}{u^2+1}=\arctan(u)+C$$
To solve for 
$$\int\frac{du}{a^2u^2+a^2+1}$$
Use substitution
$$v=\frac{au}{\sqrt{a^2+1}}\implies du=\frac{a^2+1}{a}$$
$$\int\frac{du}{a^2u^2+a^2+1}=\int\frac{\sqrt{a^2+1}dv}{a((a^2+1)v^2+a^2+1)}=\frac{1}{a\sqrt{a^2+1}}\int\frac{dv}{v^2+1}=\frac{\arctan(v)}{a\sqrt{a^2+1}}+C$$
Now plug in back $x$, you would get
$$\int_{1}^{\infty} { dx\over (1+\alpha^2 x^2) x\sqrt{x^2-1} }=\left[\arctan\left(\sqrt{x^2-1}\right)-\frac{a\arctan\left(\frac{a\sqrt{x^2-1}}{a^2+1}\right)}{\sqrt{a^2+1}}\right]_{1}^{\infty}$$
I think you can handle the rest of the calculation.
