A difficult integral evaluation problem How do I compute the integration for $a>0$, 
$$
\int_0^\pi \frac{x\sin x}{1-2a\cos x+a^2}dx?
$$
I want to find a complex function and integrate by the residue theorem.

Since it hasn't been specifically objected to yet, here is a solution that doesn't rely on complex variable methods.
We shall make use of the Fourier sine series,
$$\frac{a\sin x}{1-2a\cos x+a^2}=\begin{cases}
\sum_{n=1}^{\infty}a^{n}\sin{(nx)},~~~\text{for }|a|<1,\\
\sum_{n=1}^{\infty}\frac{\sin{(nx)}}{a^{n}},~~~\text{for }|a|>1.
\end{cases}$$
These series may readily be derived by taking the imaginary parts of the complex geometric series $\sum_{n=1}^{\infty}\left(ae^{ix}\right)^n$ and $\sum_{n=1}^{\infty}\left(\frac{1}{a}e^{ix}\right)^n$, respectively.
By expanding the integrand in terms of these series and then swapping the order of integration and summation, a closed form may be obtained. For the $|a|>1$ case,
$$\begin{align}I(a)&=\int_0^\pi \frac{x\sin x}{1-2a\cos x+a^2}\mathrm{d}x\\
&=\int_{0}^{\pi}x\sum_{n=1}^{\infty}\frac{\sin{(nx)}}{a^{n+1}}\mathrm{d}x\\
&=\sum_{n=1}^{\infty}\frac{1}{a^{n+1}}\int_{0}^{\pi}x\sin{(nx)}\mathrm{d}x\\
&=\sum_{n=1}^{\infty}\frac{1}{a^{n+1}}\left(\frac{\sin{(n\pi)}}{n^2}-\frac{\pi\cos{(n\pi)}}{n}\right)\\
&=\pi\sum_{n=1}^{\infty}\frac{(-1)^{n+1}}{n\,a^{n+1}}\\
&=\frac{\pi}{a}\log{\frac{1+a}{a}}.
\end{align}$$
For the $0<|a|<1$ cases, we can make use of the fact that if $0<|a|<1$, then $\frac{1}{|a|}>1$, thus allowing us to invoke the previous result. Hence,
$$\begin{align}I(a)&=\int_0^\pi \frac{x\sin x}{1-2a\cos x+a^2}\mathrm{d}x\\
&=\frac{1}{a^2}\int_0^\pi \frac{x\sin x}{a^{-2}-2a^{-1}\cos x+1}\mathrm{d}x\\
&=\frac{1}{a^2}\frac{\pi}{a^{-1}}\log{\frac{1+a^{-1}}{a^{-1}}}\\
&=\frac{\pi}{a}\log{(1+a)}.
\end{align}$$
