find the coefficient of $x^5$ in $(1+2x-3x^2)^6$ I know that I should first factor the expression within the brackets to $-(3x+1)^6(x-1)^6$. However, after that I do not know what to do.

Actually you can do it without factorization as well. Just like binomial theorem, we have a multinomial theorem as well. For example,
$$(a+b+c)^n=\sum_{\substack{i,j,k\\i+j+k=n}}\frac{n!}{i!j!k!}a^ib^jc^k.$$
This can be generalized to higher order multinomials as well.
So in your case,
$$(1+2x-3x^2)^6=\sum_{\substack{i,j,k\\i+j+k=6}}\frac{6!}{i!j!k!}(1)^i\,(2x)^j\,(-3x^2)^k=\sum_{\substack{i,j,k\\i+j+k=6}}\frac{6!}{i!j!k!}2^j(-3)^k\,x^{\color{red}{j+2k}}.$$
For the coefficient of $x^5$, we want $j+2k=5$ and $i+j+k=6$. This means $i+(5-k)=6 \implies i-k=1.$
Thus
$$(i,j,k) \in \{(1,5,0), (2,3,1), (3,1,2)\}.$$
Thus the coefficient of $x^5$ is
$$\frac{6!}{1!5!0!}2^5(-3)^0+\frac{6!}{2!3!1!}2^3(-3)^1+\frac{6!}{3!1!2!}2^1(-3)^2=\color{blue}{-168}.$$
