Second order ODE - undetermined coefficients for $f(x) = xe^x\cos{x}$ $$y'' + y = xe^x\cos{x}$$
So, the solution of the homogenous equation is $$y_h = C_1\cos{x} + C_2\sin{x}$$
Now, when I try to do the particular solution:
$$y_p = e^x(Ax+B)(C\cos{x}+D\sin{x})$$
Is this correct? We don't have to multiply by $x$, and because there's an $x$ in $f(x)$ we put a general polynomial of the first order, $Ax+B$. Because there's a trigonometric function, we put $C\cos{x} + D\sin{x}$, and $e^x$ is self explanatory.
However, when I find the second derivative and plug in everything into the equation to find the coefficients, I end up with a system of four equations which doesn't have a solution.
The second derivative is:
$$e^x(\sin{x} (-2acx - 2ac + 2ad - 2bc) + \cos{x} (2ac + 2adx + 2ad + 2bd))$$
I checked this on wolframalpha.
If add the $y_p$ to it and group the terms, I get
$$e^x(\sin{x} (-2acx - 2ac + 2ad - 2bc+axd+bd) + \cos{x} (2ac + 2adx + 2ad + 2bd+axc+bc))$$
Now, on the right side we have $e^xx\cos{x}$.
Let's compare the coefficients that multiply all functions containing $x\cos{x}$ with $1$ on the right side
$$2ad+ac=1$$
Now, compare all the coefficients of $\cos{x}$, $x\sin{x}$, $\sin{x}$ with 0
$$2ac+2ad+2bd+bc=0$$
$$-2ac+ad=0$$
$$-2ac+2ad-2bc+bd=0$$
This system does not have a solution, which leads me to think my $y_p$ is incorrect. However, I can't see my mistake.

Indeed, your guess for the particular solution is incomplete, or has the wrong structure. Polynomial factors on the right side imply polynomial factors for the trial solution of the same degree with all coefficients as variables. So you need a separate set of linear polynomials for the sine and the cosine term each.
More specifically, $e^x\cos x$ corresponds to two exponential terms $\frac12e^{(1+\pm i)x}$ that are independent as functions. Thus your guess will also have two (at first) independent terms. You can put them complex or real, the number of real degrees of freedom remains the same.

As the complex way was not mentioned here: One gets $y=Re(z)$ where $z$ solves the equation with a simple exponential term
$$
z''+z=xe^{(1+i)x}.
$$
As a single product polynomial times exponential, the guess has the same form $z_p=(Ax+B)e^{(1+i)x}$, where $A,B$ can and most likely will be complex numbers. Insert to get
\begin{align}
z_p'&=\Bigl(A+(1+i)(Ax+B)\Bigr)e^{(1+i)x}\\ 
z_p''&=\Bigl(2A(1+i)+2i(Ax+B)\Bigr)e^{(1+i)x}\\ 
z_p''+z_p&=\Bigl(2iAx+2A(1+i)+2iB+Ax+B\Bigr)e^{(1+i)x}
\end{align}
so that
$$
(1+2i)A=1\implies A=\frac{1}{1+2i}=\frac{1-2i}5\\
2(1+i)A+(1+2i)B=0\implies B=-\frac{2(1+i)(1-2i)^2}{25}=\frac{2(7-i)}{25}
$$
