Method of Undetermined Coeffecients - how to assume the form of third degree equation. An example differential equations questions asks me to solve;
$$y''' - 2y'' -4y'+8y = 6xe^{2x}$$ 
I begun by solving the homogeneous equation with $m^3 - 2m^2 -4m+8 =0$ and getting the answer
$$y(x) = c_1e^{2x} + c_2xe^{2x}+c_3e^{-2x} $$
The second part of the solution involves assuming a form for the solution. Because $g(x)$ is $6xe^{2x}$, I assumed the solution would be of the form $(Ax+B)e^{2x}$, however it turns out that after differentiating three times it gets extremely complicated. Is there a better way?
Also, the textbook solutions manual uses the form of $(Ax^3 + Bx^2)e^{2x}$. How did it arrive at that? (there's no accompanying explanation)

Your equation is $y'''-2y''-4y'+8y=6xe^{2x}$. Now change the $y'$ to $Dy$ form as follows. So $$y'''\to D^3y,\\ y''\to D^2y, \; \; \text{and} \;\;y'\to Dy,$$ So by new arranging respect to $D$ operator we get our equation as: $$D^3y-2D^2y-4Dy+8y=6xe^{2x}$$ or by factoring and expanding $$(D^3-2D^2-4D+8)y=(D-2)^2(D+2)y=6e^{2x}$$ which you got before. Note that considering the corresponding homogeneous equation $$(D-2)^2(D+2)y=0$$ we get $(D-2)^2=0,\;\; (D+2)=0$  which leads us to write the general solutions as $$y_c(x) = c_1e^{2x} + c_2xe^{2x}+c_3e^{-2x}$$ and you did it right above before. Now have a look at some facts:


*

*If $y=\text{constant}$ so $y'=0$ or $Dy=0$. Here, the operator $D$ annihilates $y$ which is just a constant.($Dc=0$)

*If $y=cx$ in which $c$ is a constant so $y''=0$ or $D^2y=0$. It means that the operator polynomial $P(D)=D^2$ annihilates $y=cx$. $(\text{or} \; P(D)=D^2(cx)=cD^2x=c(x'')=0)$. Generally, $D^{n+1}$ annihilates not only the function $y=cx^{n}$ but also all linear functions as $$y=c_0+c_1x+c_2x^2+...+c_nx^n$$ It means that $$P(D)y=D^{n+1}y=0$$.

*As the same the differential operator $(D-\alpha)^n$ annihilates each of the following functions and every linear combinations of them: $$e^{\alpha x},xe^{\alpha x},x^2e^{\alpha x},...,x^{n-1}e^{\alpha x}$$ Now look at the RHS of your original equation. I mean $=6xe^{2x}$. Can we guess of which proper differential polynomial annihilates it? As above it would be $(D-2)^2$. It means that $(D-2)^2 \left(6xe^{2x}\right)=0$. Don't respect to numeric coefficients like $6$ here at all.


Now we consider of what we have achieved at last: $$(D-2)^2(D+2)y=6e^{2x}$$ Put the operator $(D-2)^2$ before both sides of the above converted equation: $$(D-2)^2\left((D-2)^2(D+2)y\right)=(D-2)^26e^{2x}=0$$ or $$(D-2)^4(D+2)y=0$$ In fact, we have found a proper differential operator $P(D)=(D-2)^4(D+2)$ which if it effects to $y$, $y$ will be lost.
Now, for a while, forget our equation and look at $(D-2)^4(D+2)y=0$ and think someone gave this to us asking to guess which function $y$ may satisfy the equality above? We reply:


*

*Since we have $(D-2)$ so we have some forms as $e^{2x}$ in $y$.

*Since $(D-2)$ has a power $4$ so we have the forms $Ae^{2x},\; Bxe^{2x}, \;Cx^2e^{2x}, \text{and} \; Ex^3e^{2x}$ in $y$. Note that you multiply $e^{2x}$, in the previous line, by $A,\; Bx,\; Cx^2,\; Ex^3$. (Exactly until the power of $x$ gets $4-1=3$).

*And, since we have $(D+2)$, then $y$ has the term $Fe^{-2x}$.
So we are done. Our probable function which satisfy the original equation is $$y=Ae^{2x}+ Bxe^{2x}+Cx^2e^{2x}+Ex^3e^{2x}+Fe^{-2x}$$. Now put the terms which generate $y_c(x)$ aside and take the rest for what we have looked for. It is $$y_p=Cx^2e^{2x}+Ex^3e^{2x}$$ where $C,E$ are unknown constant. 
