How to solve the functional equation $ f(f(x))=ax^2+bx+c $ Find all real numbers $a,b,c\in\mathbb{R}$ for which there exists a function $f:\mathbb{R}\to\mathbb{R}$ such that:
$$
f(f(x))=ax^2+bx+c
$$
for all $x\in\mathbb{R}$.
The only thing I could deduce is:
$$
f(ax^2+bx+c)=af(x)^2+bf(x)+c
$$
Which doesn't help much. How to tackle the problem?

SHORT ANSWER: A general solution to this problem is not known in the closed form, but some special cases can be solved. Sorry.
LONG ANSWER: Notice that if a function $f$ satisfies
$$f(x)=(g^{-1} \circ h \circ g)(x)$$
for some $g,h$, then
$$f^n(x)=(g^{-1} \circ h^n \circ g)(x)$$
where the superscript denotes functional iteration rather than exponentiation.
We can solve this equation for a whole class of quadratics
$$f^2(x)=q(x)=ax^2+bx+c$$
for which
$$c=\frac{b^2-2b}{4a}$$
This is because we can rewrite
$$q(x)=ax^2+bx+\frac{b^2-2b}{4a}$$
as
$$q(x)=a\bigg(x+\frac{b}{2a}\bigg)^2-\frac{b}{2a}$$
and so, by letting $g(x)=x-\frac{b}{2a}$ and $h(x)=ax^2$,
$$q(x)=(g^{-1} \circ h \circ g)(x)$$
and thus
$$q^n(x)=(g^{-1} \circ h^n \circ g)(x)$$
and since the formula for $h^n$ is
$$h^n(x)=a^{2^n-1}x^{2^n}$$
we have
$$q^n(x)=a^{2^n-1}\bigg(x+\frac{b}{2a}\bigg)^{2^n}-\frac{b}{2a}$$
and, finally,
$$f(x)=q^{1/2}(x)=a^{\sqrt 2-1}\bigg(x+\frac{b}{2a}\bigg)^{\sqrt 2}-\frac{b}{2a}$$
So there's the solution for that special case. There is another special case when 
$$c=\frac{b^2-2b-8}{4a}$$
but the solution to that case is much longer, and so I will omit it and leave it to you for independent research.
There is another special case involving trigonometric functions. Note that if we let
$$g(x)=\arccos x$$
$$h(x)=2x$$
we have
$$(g^{-1}\circ h\circ g)(x)=\cos(2\arccos x)$$
and, using the double-angle formula,
$$(g^{-1}\circ h\circ g)(x)=\cos^2(\arccos x)-\sin^2(\arccos x)$$
$$(g^{-1}\circ h\circ g)(x)=x^2-(1-x^2)$$
$$(g^{-1}\circ h\circ g)(x)=2x^2-1$$
and so $g^{-1}\circ h\circ g$ is a quadratic. Now let
$$q(x)=(g^{-1}\circ h\circ g)(x)$$
so that
$$q^n(x)=(g^{-1}\circ h^n\circ g)(x)$$
Now, since 
$$h^n(x)=2^n x$$
we have
$$q^n(x)=\cos(2^n\arccos x)$$
and
$$f(x)=q^{1/2}(x)=\cos(\sqrt{2}\arccos x)$$
which solves yet another special case. However, notice that this is only solved on the domain $[-1,1]$, because that is where $\arccos x$ is defined.
I am sure there are other special cases involving trigonometric functions, but I will leave those for you to find.
