Condition for $|AB|^2+|BC|^2+|CA|^2$ to be maximum Let $|\vec{OA}|=l,| \vec{OB}|=m,| \vec{OC}|=n$.$O$ be the origin.$A,B,C$ lie on the plane $x+2y-z=0$ and $|AB|^2+|BC|^2+|CA|^2$ is maximum,then the value of $|AB|$ is
I tried to convert the equation of plane in vector.clearly the plane passes through origin.The equation in vector form will be then $$\vec{r}.(i+2j-k)=0$$.Assuming arbitrary parameters for the given vectors I got different equations but couldn't get how to maximise $|AB|^2+|BC|^2+|CA|^2$ .
Please help me in this regard.Thanks.

Points $A,B,C$ belong resp. to spheres centered in $O$ with radii $\ell,m,n$. Thus, any plane passing through the origin intersects these spheres along a diameter, i.e., as circles, with the same radii. Let us choose one of them. All the following is a 2D problem with coordinates in this plane.
Let us give names $a,b,c$ to the polar angles of $A,B,C$ resp. 
It means that $A(\ell \cos(a),\ell \sin(a)), B(m \cos(b),m \sin(b)), C(n \cos(c),n \sin(c)),$ 
Using the law of cosines in triangles $OAB, OAC$ and $OBC$ resp.:
$$\begin{cases}
AB^2=\ell^2+m^2-2\ell m \cos(a-b)\\
AC^2=\ell^2+n^2-2\ell n \cos(a-c)\\
BC^2=m^2+n^2-2 m n \cos(b-c)
\end{cases}$$
Summing up these 3 expressions; one gets
$$AB^2+AM^2+BC^2=K-2(\ell m \cos(a-b)+\ell n \cos(a-c)+ m n \cos(b-c))$$
where $K$ is a constant. It suffices then to minimize expression:
$$\ell m \cos(a-b)+\ell n \cos(a-c)+ m n \cos(b-c)$$
As there is a rotational invariance in this problem, it is possible to assume that $c=0$. 
Thus we have only to minimize the following function of two variables:
$$f(a,b):=\ell m \cos(a-b)+\ell n \cos(a)+ m n \cos(b).$$
If this minimization occurs inside the domain, a necessary condition is that the partial derivatives of $f$ with respect to angles $a,b$ are simultaneously zero.
Can you finish from there ?
