How to prove that $a^2b+b^2c+c^2a \leqslant 3$, where $a,b,c >0$, and $a^ab^bc^c=1$ $a,b,c &gt;0$, and $a^ab^bc^c=1$, prove
$$a^2b+b^2c+c^2a \leqslant 3$$

I don't even know what to do with the condition $a^ab^bc^c=1$. At first I think $x^x&gt;1$, but I was wrong. This inequality is true, following by the verification from Mathematica
We employ of the rearrangement inequality.
First, since $x\mapsto x^2$ preserves the order for $x&gt;0,$ we have $$a^2b+b^2c+c^2a\le a^3+b^3+c^3.$$
Next write the condition as $$a\ln a+b\ln b+c\ln c=0.$$ Again using the rearrangement inequality, as $x\mapsto\ln x$ preserves the order for $x&gt;0,$ we have the inequalities: $$\begin{cases}a\ln b+b\ln c+c\ln a\le0\\
a\ln c+b\ln a+c\ln b\le0\\
a\ln a+b\ln b+c\ln c=0\end{cases}$$
Adding these together, we have $(a+b+c)\ln(abc)\le0,$ hence $abc\le1.$
Now let $\ln a+\ln b+\ln c=k\le0.$ Apply the Lagrange multiplier method with condition $g(a,b,c):=\ln a+\ln b+\ln c=k$ for a fixed $k\le0,$ to maximize $f(a,b,c):=a^3+b^3+c^3.$
Thus the Lagrange multiplier gives that the extreme of $f$ occurs when $$\begin{cases}3a^2-\lambda\frac{1}{a}=0\\
3b^2-\lambda\frac{1}{b}=0\\
3c^2-\lambda\frac{1}{c}=0\end{cases},$$
i.e. when $a^3=b^3=c^3=\lambda;$ then $a=b=c$ and hence $f(a,b,c)=3abc=3e^k\le3.$
$\square$  

Hope this helps.  

Edit:
As pointed out in the comment, the final part about Lagrange multipliers is incorrect; in fact, given the constraint $abc&lt;1,$ it does not follow that $a^2b+b^2c+c^2a\le3.$
Also pointed out in the comment is to use Jensen's inequality to obtain $a+b+c\le3,$ but we are not getting answers yet.
Everything we tried so far to fix this fails. We shall update if we find a way to work around.
The condition can be presented in form
$$a\ln a +b\ln b + c\ln c = 0.$$
Let
$$a\leq b\leq c,$$
then
$$\ln a \leq \ln b \leq \ln c,$$
and we can use Chebyshev sum inequality:
$$3(a\ln a +b\ln b + c\ln c) \geq (a+b+c)\ln abc,$$
$$abc\leq 1.$$
Evidently, the expression $a^2b+b^2c+c^2a$ achieves maximum when $abc=1.$

The standard way of solving the problem on a conditional extremum is the method of Lagrange multipliers, which reduces it to a system of equations.

The maximum value of function
$$f(a,b,c,λ)=a^2b+b^2c+c^2a+λ(abc-1)$$
for $a,b,c&gt;0$ on the interval
is reached or at its edges, or in one of the points with zero partial derivatives
$$f'_a=0,\quad f'_b=0,\quad f'_c=0,\quad f'_λ=0,$$
or
$$\begin{cases}
2ab+c^2+\lambda bc = 0\\
2bc+a^2+\lambda ca = 0\\
2ca+b^2+\lambda ab = 0\\
abc-1 = 0,
\end{cases}$$
then
$$\begin{cases}
a(2ab+c^2) = b(2bc+a^2)=c(2ca+b^2),\\
abc=1,
\end{cases}$$
$$\begin{cases}
2b^2c -c^2a = a^2b\\
b^2c -2c^2a = -a^2b\\
abc=1,
\end{cases}$$
$$c^2a = b^2c = a^2b,\quad abc=1,$$
$$\dfrac cb =1, \dfrac ba =1, \quad abc=1,$$
$$a=b=c=1,$$
$$f(a,b,c,\lambda)=3.$$
Note that
$$\lim_{a\to 0} a\ln a = 0,$$
so on the edge $a=0$ we have to minimize
$f(b,c,\lambda) = b^2c + \lambda(bc-1),$
and then we get the system
$$\begin{cases}
2bc+\lambda c = 0\\
b^2+\lambda b = 0\\
bc-1=0
\end{cases}$$
with solution 
$$b=2,\quad c=\dfrac12,\quad f = 2 &lt; 3.$$

That means that
$$\boxed{a^2b+b^2c+c^2a \leq 3}$$