Finding $\int_{0}^{2\pi} \frac {(1+2\cos\theta)^n\cos(n\theta)}{3+2\cos\theta} \operatorname{d}\theta$ If $n$ is a positive integer find
$$
\int_{0}^{2\pi} \frac {(1+2\cos\theta)^n\cos(n\theta)}{3+2\cos\theta} \operatorname{d}\theta
$$

I know that I have to use contour integral with a circle of radius 1 centered at the origin, but I am having trouble converting the integral into the form $\int_{|z|}$

$$\int_{|z| = 1} \frac{(1+z+1/z)^n\cos(n\theta)}{3+(z+1/z)} \frac{1}{iz} \operatorname{d}z$$

I cant seem to find a way to expand $\cos(n\theta)$ into a function of $z$.

From the above equation, I can get that the poles of is at $z = -1.5 \pm \frac{\sqrt{5}}{2}$ and only the residual of $z = -1.5 + \frac{\sqrt{5}}{2}$ should be included.
Note that

$$\cos{n \theta} = \frac12 \left ( z^n + z^{-n}\right )$$

so upon your substitution, you get

$$-\frac{i}{2} \oint_{|z|=1} \frac{dz}{z} \frac{(1+z+z^{-1})^n (z^n+z^{-n})}{3+z+z^{-1}}$$

This can be rearranged to get

$$-\frac{i}{2} \oint_{|z|=1}  \frac{dz}{z^{2 n}} \frac{(1+z+z^2)^n (1+z^{2 n})}{1+3 z+z^2} $$

There are poles where $z^2+3 z+1=0$, or

$$z_{\pm} = \frac{-3\pm \sqrt{5}}{2}$$

i.e., $z_- = -\phi^2$, $z_+=-1/\phi^2$, where $\phi=(\sqrt{5}+1)/2$.  Thus only $z_+$ is within the unit circle and contributes to the integral.

For the pole at $z=0$, it may be easiest to find the coefficient of $z^{2 n-1}$ in the Maclurin expansion of

$$\frac{(1+z+z^2)^n (1+z^{2 n})}{1+3 z+z^2} $$

Not an easy task.  Good luck.
Because I can't submit a comment, I submit an answer. Maple calculates it for concrete values of $n$. Hope, the following would be useful: the Maple command $$[seq(int((1+2*cos(theta))^n*cos(n*theta)/(3+2*cos(theta)), theta = 0 .. 2*Pi), n = [5, 10, 15, 30])]  $$ produces $$  [-1760\,\pi +{\frac {3936}{5}}\,\sqrt {5}\pi ,-6927360\,\pi +{\frac {
15490048}{5}}\,\sqrt {5}\pi ,-27264286720\,\pi +{\frac {60964798464}{5
}}\,\sqrt {5}\pi ,-1662161745149511598080\,\pi +{\frac {
3716706651753989275648}{5}}\,\sqrt {5}\pi ].
$$