Simultaneous Diagonalization of two bilinear forms I need to diagonalize this two bilinear forms in the same basis (such that $f=I$ and $g$=diagonal matrix):
$f(x,y,z)=x^2+y^2+z^2+xy-yz $
$g(x,y,z)=y^2-4xy+8xz+4yz$
I know that it is possible because f is positive-definite, but I don't know how can I do it

I wonder if the method the OP was attempting is simply Lagrange's method from multivariable calculus. Because a quadratic form takes its extreme values on the unit circle (or any circle) at eigenvectors, and because we can diagonalize a symmetric matrix using a basis of orthonormal eigenvectors, we look for the stationary points of $g$ with the constraint $f(x,y,z)=1$ (here $f$ will be our scalar product since, as the OP mentioned, it is positive definite and this is enough to guarantee a solution). I'll represent $f$ and $g$ by matrices $A$ and $B$ respectively to match the other answers, so we have
$$A=\begin{bmatrix}1&\frac{1}{2}&0\\\frac{1}{2}&1&-\frac{1}{2}\\0&-\frac{1}{2}&1\\\end{bmatrix} \qquad B=\begin{bmatrix}0&-2&4\\-2&1&2\\4&2&0\\\end{bmatrix}$$
Lagrange's method results in the system of linear equations
$$\begin{bmatrix}-\lambda&-2-\lambda/2&4\\-2-\lambda/2&1-\lambda&2+\lambda/2\\4&2+\lambda/2&-\lambda\\\end{bmatrix}\begin{bmatrix}x\\y\\z\\\end{bmatrix}=\begin{bmatrix}0\\0\\0\\\end{bmatrix}$$
where the $(i,j)$-th entry in the leftmost matrix is $b_{ij}-\lambda a_{ij}$. The system has a nontrivial solution when that matrix's determinant vanishes, which is when $\lambda=6$ or $\lambda=4$ or $\lambda=-4$. Now, we can plug in each of these values in turn to the system to get solutions, which will be eigenvectors forming a basis $\{b_1,b_2,b_3\}$ that makes both quadratic forms diagonal. I get
$$b_1=\begin{bmatrix}-1\\2\\1\\\end{bmatrix},\quad b_2=\begin{bmatrix}1\\0\\1\\\end{bmatrix},\quad b_3=\begin{bmatrix}-1\\0\\1\\\end{bmatrix}$$
The final step is to normalize these vectors so that $A$ is not only diagonal, but the identity matrix. We need to do this because the information that $f(x,y,z)=1$ was lost when taking partial derivatives; we could have started out with $f(x,y,z)=\pi^3$ and got the same system of equations, and furthermore I arbitrarily chose $b_1,b_2,b_3$ to have third coordinate $1$ so there's no reason to think they'd be normalized. Just keep in mind that our scalar product is $f$ and not the typical dot product, so in fact every vector in the above basis has magnitude $\sqrt{2}$. If we put all this together, our change of basis matrix is
$$R=\dfrac{1}{\sqrt{2}}\begin{bmatrix}-1&1&-1\\2&0&0\\1&1&1\\\end{bmatrix}$$
and the matrices $\widetilde{A}$, $\widetilde{B}$ of the quadratic forms in the new basis are
$$\widetilde{A}=R^TAR=I \qquad \widetilde{B}=R^TBR=\begin{bmatrix}6&0&0\\0&4&0\\0&0&-4\\\end{bmatrix}$$
Two remarks: This method is described in Shilov's Linear Algebra, section 10.32. Also, I've been referring to column matrices as vectors, which if we're being pedantic is tacitly assuming that we're starting in the standard basis. However, the method works no matter what basis we start with. If it's not the standard basis, then the columns of the change of basis matrix simply represent the coordinates of the new basis vectors in terms of the starting basis, instead of the vectors themselves.
