How can I find the integers $x,y$? $$ax+by=c$$
$$ax^2+by^2=d$$
where $ab\neq 0$ and $x,y$ are coprime?
You can solve this by talking the first equation and solving for y. Then substituting y in the second equation, as so:

$$ax+by=c$$

$$ax-c=-by$$

$$(ax-c)/(-b)=y$$

Now substitute into second equation:

$$ax^{2}+\frac{b\left(ax-c\right)^{2}}{b^{2}}=d$$

Expand:

$$ax^{2}+\frac{a^{2}x^{2}-2acx+c^{2}}{b}=d$$

Simplify:

$$\frac{bax^{2}}{b}+\frac{a^{2}x^{2}-2acx+c^{2}}{b}=d$$

$$\frac{bax^{2}+a^{2}x^{2}-2acx+c^{2}}{b}=d$$

Subtract d from both sides:

$$\frac{bax^{2}+a^{2}x^{2}-2acx+c^{2}-bd}{b}=0$$

Simplify more...

$$\frac{\left(a^{2}+ba\right)}{b}x^{2}-\frac{2ac}{b}x+\frac{\left(c^{2}-bd\right)}{b}=0$$

This is just a quadratic. You can solve it with the quadratic equation:

$$x=\frac{-g+\sqrt{g^{2}-4fh}}{2f}$$

And

$$x=\frac{-g-\sqrt{g^{2}-4fh}}{2f}$$

Where $$f=\frac{\left(a^{2}+ba\right)}{b}$$

$$g=-\frac{2ac}{b}$$

$$h=\frac{\left(c^{2}-bd\right)}{b}$$

Try it out: https://www.desmos.com/calculator/dw9xe3ppxe
Above simultaneous equation shown below:

$ax^2+by^2=d$

$ax+by=c$

Taking, $(a,b,c,d)=(4,5,22,56)$ we get:

$(x,y)=(3,2)$ &amp; $(x,y)=(17/9,26/9)$