transforming conic sections in projective geometry with 4 points Consider the following conic section
K = ${(x : y : z) \in \mathbb{R}P^2 | 16x^2+9y^2−z^2=0}$.
I have to Give the inverse of the matrix $A \in PGL(3, R)$ for a projective transformation that transforms K to a conic section through points $(1 : 0 : 0),(0 : 1 : 0),(0 : 0 : 1)$ and $(2 : −3 : 1)$ and determine the image of $K$ under the transformation.
I am not sure where i am supposed to begin, so i was wondering if anyone can give me a hint for this.

As you are asking for a methodology:
A first reaction is that we have a degree of freedom (five points are necessary to define a conic curve, instead of four as it is the case here ...)
Then, a non-projective interpretation is interesting for an intuitive interpretation of the  different constraints (see figure):

*

*passing by $(1;0;\color{red}{0})$ = passing by the point at $\color{red}{\text{infinity}}$ in the direction of $Ox$ (i.e., having a horizontal asymptote),


*passing by $(0;1;\color{red}{0})$ = passing by the point at $\color{red}{\text{infinity}}$ in the direction of $Oy$ (i.e., having a vertical asymptote),


*passing by $(0;0;\color{blue}{1})$ = passing by the $\color{blue}{\text{ordinary point "origin"}}$,


*passing by $(2;-3;\color{blue}{1})$ = passing by $\color{blue}{\text{ordinary point} (2/1,-3/1)}$.
Here is, among an infinite number of solutions, a curve that complies with these conditions:
$$Y=\dfrac{-6X}{X+2}\tag{1}$$
How can we find the matrix of a homography mapping the initial curve (an ellipse) onto the hyperbola with equation (1) ?
In a rather simple way by using the eigendecomposition of the (symmetric) matrix associated with equation (1) written under the form $-2XY-12X-4Y=0$:
$$C=\begin{pmatrix}0&-1&-6\\
     -1&0&-2\\
     -6&-2&0\end{pmatrix}=\begin{pmatrix}&&\\
     &P&\\
     &&\end{pmatrix}\begin{pmatrix}a^2&&\\
     &b^2&\\
     &&-c^2\end{pmatrix}\begin{pmatrix}&&\\
     &P^T&\\
     &&\end{pmatrix}\tag{2}$$
where $a^2>0,b^2>0$ and $-c^2<0$ are the eigenvalues of $C$.
Now let us decompose again
$$C=\underbrace{\begin{pmatrix}&&\\
     &P&\\
     &&\end{pmatrix}\begin{pmatrix}a/4&&\\
     &b/3&\\&&c\end{pmatrix}}_{Q^T}
\underbrace{\begin{pmatrix}16&&\\&9&\\&&-1\end{pmatrix}}_E
\underbrace{\begin{pmatrix}a/4&&\\&b/3&\\&&c\end{pmatrix}
\begin{pmatrix}&&\\&P^T&\\&&\end{pmatrix}}_{Q}\tag{3}$$
where $E$ is the matrix associated with the ellipse (initial curve)
(1) means that $Q$ is the matrix of the homography mapping the hyperbola onto the ellipse (beware of the direction!)
Therefore $Q^{-1}$ is the looked for homography with the inverse mapping.

The image curve with equation (1).
One can have a different matrix mapping the ellipse on the same hyperbola, here computed with exact values:
$$H=\begin{pmatrix}-8&-6&2\\
     24&6(3+\sqrt{3})&-2(3+\sqrt{3})\\
     4(1+2\sqrt{3})&3&-(1+2\sqrt{3})\end{pmatrix}\tag{4}$$
giving the transformation equations
$$X=\dfrac{-8x-6y+2}{4(1+2\sqrt{3})x+3y-(1+2\sqrt{3})}, \ \ Y=\dfrac{24x+6(3+\sqrt{3})y-2(3+\sqrt{3})}{4(1+2\sqrt{3})x+3y-(1+2\sqrt{3})}$$
(I have taken $z=1$ in order to get the ordinary points).
Remark: The computation of the equation of the image curve requires (as it is asked in your text) the inverse of $H$ (symbol $\approx$ meaning "up to a multiplicative factor"):
$$H^{-1}\approx\begin{pmatrix}3(3+\sqrt{3})&3&0\\
     4\sqrt{3} + 2& 0&4\\
     12\sqrt{3} + 42& 12&12\end{pmatrix}$$
(remember "the old coordinates expressed as functions of the new ones").
Appendix: Matlab program for the figure:

clear all;close all;a=10;hold on
A=-[0 1 6
    1 0 2
    6 2 0];
[P,D]=eig(A),E=diag(D);
[~,id]=sort(E,'descend'),
E=E(id),F=sqrt(abs(E));
P=P(:,id);
Del=diag([F(1)/4,F(2)/3,F(3)]);
M=inv(Del*P');
t=0:0.001:2*pi;
V=M*[(1/4)*cos(t);(1/3)*sin(t);ones(1,size(t,2))]
plot(V(1,:)./V(3,:),V(2,:)./V(3,:));
plot(2,-3,'or'); plot(0,0,'or');
axis(a*[-1,1,-1,1]);grid on;


