How to integrate $\int(x\pi-x^2)\cos(kx)dx$ My question is can I solve by integrating by parts if I do $u = (x\pi-x^2)$,or first I have to do $∫[x\pi \cos(kx)-x^2\cos(kx)]dx$ and then split it in two integrals $\int x\pi\cos(kx)dx-\int x^2\cos(kx)dx$ ?

You may directly integrate by parts twice,
$$
\int(\pi x-x^2) \cos (kx)\:dx=\frac1k(\pi x-x^2)\sin (k x)-\frac1k\int(\pi-2x) \sin (kx)\:dx
$$ then
$$
\begin{align}
&\int(\pi x-x^2) \cos (kx)\:dx\\&=\frac1k(\pi x-x^2)\sin (k x)-\frac1k\left((\pi-2x)(-\frac1k \cos (kx))+\int(-2) \frac1k \cos (kx)\:dx\right)
\end{align}
$$ getting

$$
\int(\pi x-x^2) \cos (kx)\:dx=\frac1{k^3}\left(2+k^2\pi  x-k^2x^2\right) \sin (kx)+\frac1{k^2}(\pi -2 x) \cos (kx)+ C
$$

