ODE $(2x^6+5x^3-3x)y''(x)+(-14x^5-20x^2+6)y'(x)+30x^4y(x)=0$ $A(x)y''+B(x)y'+C(x)y=0$ represents the equation.


*

*Tried using $t=\phi(x)$, but the $\int{\sqrt{\frac{C(x)}{A(x)}}}$
gives too complicated results.

*$y=UV$ substitution didn't do anything.
Particular solutions are only thing left, combinations I've tried and that didn't work for me: $y_p=ax^2+b, y_p=\frac{a}{x^2}+b, y_p=ax^6+b$

You can simplify the equation a bit: introduce
\begin{equation}
 p(x) = 2 x^7 + 5 x^4 - 3 x^2,
\end{equation}
then the ODE has the form
\begin{equation}
\frac{p(x)}{x} y'' - \frac{p'(x)}{x} y' + 30 x^4 y = 0.
\end{equation}
You can rewrite this as
\begin{equation}
 p^2\left(\frac{y'}{p}\right)' + 30 x^5 y = 0,
\end{equation}
which is equivalent to
\begin{equation}
 p^3 \left(\frac{1}{p} \frac{\text{d}}{\text{d} x}\right)^2 y + 30 x^5 y = 0.
\end{equation}
Introducing
\begin{equation}
 \xi(x) = \int^x \!p(\bar{x})\,\text{d}\bar{x} = \frac{1}{4} x^8 + x^5 - x^3 \,\,(+ \text{constant}),
\end{equation}
you can write
\begin{equation}
 \frac{1}{p} \frac{\text{d}}{\text{d} x} = \frac{\text{d}}{\text{d} \xi},
\end{equation}
so the original ODE can be 'simplified' to
\begin{equation}
 \frac{\text{d}^2 y}{\text{d} \xi^2} + F(\xi)\, y = 0,
\end{equation}
where
\begin{equation}
 F(\xi) = \frac{30 x(\xi)^5}{p(x(\xi))^3}.
\end{equation}
Admittedly, it might be a bit cumbersome (to say the least) to invert the coordinate transformation $x \to \xi$, but it might be worth a try. 
If all else fails, you can indeed follow @Michael's suggestion and try substituting a series expansion. The singular points of the coefficient multiplying $y''$ are not easy to find however, so that approach might be of limited value -- except for solutions around $x = 0$, where the 'usual' series expansion will give you some information.
