Is there any continuous function that satisfies the following conditions? Is there any continuous function  $y=f(x)$ on $[0,1]$ that satisfies the following conditions?
$$
f(0) = f(1) = 0,
$$
and
$$ 
f(a)^2-f(2a)f(b)+f(b)^2<0,
$$
for some $0<a<b<1$.
I tried to test with several functions (with different $a,b$) but non of them satisfied. 
Any help is appreciated. Thank you in advance.

What you want is a function that is highly peaked around $x=2a$.  And you need to enforce the conditions that $f(0)=f(1)=0$; for that you can put in a factor of $x(1-x)$. 
So try
$$
f(x) = x(1-x)e^{-20(x-\frac12)^2}
$$
with $a=\frac14$ and $b=\frac34$.  
There, 
$$f(a)^2 - f(2a)f(b) + f(b)^2 \approx -0.00766 <0$$
