Confusion regarding defining an Ellipse I know a ellipse is locus of point such that the ratio of its distance from a fixed point (focus) and a fixed line (directrix) is constant and the value of which is always less than $1$ , the constant ratio is called eccentricity $e$ and $e \in( 0, 1)$ for an Ellipse.
Now let S($\alpha,\beta$) and directrix be $lx+my+n=0$.
The locus of variable point P($x,y$) will be ellipse if$\frac{SP}{PM}=e$
Therefore equation of ellipse is
$\sqrt{(x-\alpha)^2+(y-\beta)^2}=e\frac{|lx+my+n|}{\sqrt{l^2+m^2}}$
The book says that the equation will come in the form of$ax^2+by^2+2gx+2fy+c=0$
And it will represent an ellipse if $h^2-ab<0$ and $\delta$ = $abc+2hgf-af^2-bg^2-ch^2\not= 0$
I know how $\delta$ = $abc+2hgf-af^2-bg^2-ch^2\not= 0$ because if it is zero then it will represent pair of straight lines .
what is confusing me is how $h^2-ab<0$ this condition can be derived?
If I open $\sqrt{(x-\alpha)^2+(y-\beta)^2}=e\frac{|lx+my+n|}{\sqrt{l^2+m^2}}$
We get $x^2l^2(1-e^2)+y^2m^2(1-e^2)-2lme^2xy-2x(\alpha+e^2nl)-2y(\beta+mne^2)+(\alpha^2+\beta^2)-e^2n^2=0$
On comparing with standard equation
$a=l^2(1-e^2)$
, $b=m^2(1-e^2)$,
$h=lme^2$
So, now $ h^2-ab<0$
$\implies$ $l^2m^2e^4-l^2m^2(1-e^2)^2$<0
$\implies$ $2e^2<0$
$\implies$ $\frac{1}{2}<e<-\frac{1}{2}$
Which contradicts that $e\in(0,1)$
Where is my reasoning going wrong?
And how can we derive $h^2-ab<0$ for an Ellipse ?
Edit: I got that my algebra is wrong and $h^2-ab$<0 can be verified.

In the Conics classical form ( $h$ should be present! ):
$$ax^2+2 h x y +by^2+2gx+2fy+c=0$$
If $ I_2= h^2-ab <0  \text{ then it is an ellipse, if   > 0   then it is a hyperbola etc. }$ as given in Tavish's answer, not repeating.
Invariant $I_3=\delta $ is not directly relevant to this answer.
Now directly from the definition of a Conic.. continuing from your question by squaring we have
$$( x^2+y^2)(l^2+m^2)= e^2 ( l^2x^2+m^2y^2+n^2+ 2 lmxy+mny+lxn) \tag 1$$
Note the presence of non-zero $xy$ term in equation (1) which characterizes a conic when axes are not parallel to the Conic's symmetry axis.
In the given program and configuration, the directrix is in magenta, ellipse  in blue, focus F at origin.

