Finding minimum distance between a circle and curve what is the minimum distance between $x^2+y^2=9$ and $2x^2+10y^2+6xy=1$ 
in Question there is a circle and a curve and we have to find the least distance between them 

Hint: These are the two ellipses:

Solution:
The red ellipse seems to be a rotated ellipse, centered around the origin. The minimal distance then is the length of the major semi-axis.
So we try to find the rotation:
We can write the second the second equation as
$u^t Q u = 1$ with $u = (x, y)^t$ and
$$
Q = 
\begin{pmatrix}
2 & 3 \\
3 & 10
\end{pmatrix}
$$
The symmetric matrix $Q$ has the characteristic polynomial
$$
\chi(\lambda) = (2-\lambda)(10-\lambda) - 9
= \lambda^2 - 12 \lambda + 11
= (\lambda - 6)^2 - 25
$$
with roots $\lambda = 6 \pm 5$.
For the eigenvalue $\lambda = 1$ we solve $Q_1 u = 0$ for 
$$
Q_1 =
\begin{pmatrix}
2-1 & 3 \\
3 & 10-1 
\end{pmatrix}
=
\begin{pmatrix}
1 & 3 \\
3 & 9 
\end{pmatrix}
$$
and get $u = \alpha (3,-1)^t$. We pick the normed vector $u_1 = (1/\sqrt{10}) (3,-1)^t$ as eigenvector.
For the eigenvalue $\lambda = 11$ we solve $Q_2 u = 0$ for 
$$
Q_2 =
\begin{pmatrix}
2-11 & 3 \\
3 & 10-11 
\end{pmatrix}
=
\begin{pmatrix}
-9 & 3 \\
3 & -1 
\end{pmatrix}
$$
and get $u = \alpha (1,3)^t$. We pick the normed vector $u_2 = (1/\sqrt{10}) (1,3)^t$ as eigenvector.
Transforming to the base of eigenvectors via 
$$
T 
= (u_1, u_2) 
=
\frac{1}{\sqrt{10}}
\begin{pmatrix}
 3 & 1 \\
-1 & 3 
\end{pmatrix}
$$ 
we get
$$
\DeclareMathOperator{diag}{diag}
T^{-1} Q T = T^{-1} (1 \cdot u_1, 11 \cdot u_2) = 1 \cdot e_1 + 11 e_2 = \diag(1, 11) \iff \\
Q = T \diag(1, 11) T^{-1}
$$
It turns out $T^{-1} = T^t$, so $T$ is orthogonal, and $\det(T) = 1$, so $T$ is the looked for rotation. This was expected because $Q$ is a symmetric ($Q = Q^t$) real valued matrix.
In the transformed system, $u' = T u$, we have the equation
\begin{align}
1 
&= (u')^t Q u' \\ 
&= (T u)^t T \diag(1, 11) T^{-1} T u \\
&= u^t \diag(1, 11) u \\ 
&= x^2 + 11 y^2 \\
&= \left( \frac{x}{1} \right)^2 + \left( \frac{y}{1/\sqrt{11}} \right)^2
\end{align}
from which we infer the ellipse parameters $a = 1$ and $b=1/\sqrt{11}$.
So the inner (red) ellipse has a circumscribed circle with $r = 1$, the sought distance is $d = 2$.

About the above image: The tiny arrows are the images of $e_1$ and $e_2$ under $T$, thus $u_1$ and $u_2$. The blue ellipse is the transformed ellipse $x^2 + 11 y^2 = 1$. The yellow circle is the unit circle.
