Diagonalize a symmetric matrix let $$A = \left(\begin{array}{cccc} 1&amp;2&amp;3\\2&amp;3&amp;4\\3&amp;4&amp;5 \end{array}\right)$$

I need to find an invertible matrix $P$ such that $P^tAP$ is a diagonal matrix and it's main diagonal may have only the terms from the set $\{ 1,-1,0 \}$

I'd be glad if you could explain to me how to solve this. I haven't found the right theorem/algorithm.

Thanks.
This trick is due to Hermite. It is especially useful when you have a symmetric matrix of integers. First, we write a certain function in three variables,
$$ f(x,y,z) = x^2 + 3 y^2 + 5 z^2 + 8 yz+ 6 zx +4xy,  $$ because this is exactly the result of calculating $v^t A v,$ with
$$
v =
\left(
\begin{array}{c}
x \\
y \\
z
\end{array}
\right)
$$

In order to clear the terms with $x,$ we write
$$ (x + 2 y + 3 z)^2 =   x^2 + 4 y^2 + 9 z^2 + 12 yz+ 6 zx +4xy.  $$
So far,
$$ f(x,y,z) -  (x + 2 y + 3 z)^2 = -y^2 - 4 z^2 - 4 y z.  $$
Next we clear all $y,$ 
$$ (y + 2 z)^2 = y^2 + 4 z^2 + 4 y z,  $$ and
$$  f(x,y,z) -  (x + 2 y + 3 z)^2 +  (y + 2 z)^2 = 0,  $$
$$ \color{red}{ f(x,y,z) =  (x + 2 y + 3 z)^2 -  (y + 2 z)^2 }.  $$

The matrix multiplication that this shows is
$$
\left(
\begin{array}{ccc}
1 &amp; 0 &amp; 0 \\
2 &amp; 1 &amp; 0 \\
3 &amp; 2 &amp; 0
\end{array}
\right)
\left(
\begin{array}{ccc}
1 &amp; 0 &amp; 0 \\
0 &amp; -1 &amp; 0 \\
0 &amp; 0 &amp; 0
\end{array}
\right)
\left(
\begin{array}{ccc}
1 &amp; 2 &amp; 3 \\
0 &amp; 1 &amp; 2 \\
0 &amp; 0 &amp; 0
\end{array}
\right) =
\left(
\begin{array}{ccc}
1 &amp; 2 &amp; 3 \\
2 &amp; 3 &amp; 4 \\
3 &amp; 4 &amp; 5
\end{array}
\right)
$$
That is actually the right way to do it. 

However, I see that someone asked for invertible $P,$ despite non-full rank. Also can be done, and easily:
$$
\left(
\begin{array}{ccc}
1 &amp; 0 &amp; 0 \\
2 &amp; 1 &amp; 0 \\
3 &amp; 2 &amp; 1
\end{array}
\right)
\left(
\begin{array}{ccc}
1 &amp; 0 &amp; 0 \\
0 &amp; -1 &amp; 0 \\
0 &amp; 0 &amp; 0
\end{array}
\right)
\left(
\begin{array}{ccc}
1 &amp; 2 &amp; 3 \\
0 &amp; 1 &amp; 2 \\
0 &amp; 0 &amp; 1
\end{array}
\right) =
\left(
\begin{array}{ccc}
1 &amp; 2 &amp; 3 \\
2 &amp; 3 &amp; 4 \\
3 &amp; 4 &amp; 5
\end{array}
\right)
$$
The effect of this is to add $0$ to the function in the shape of $0z^2.$

ADDED: looking at the question again, we do need the extra $1$ in the lower right, because the matrix $P$ requested is actually the inverse of the on I display above. Life goes on.
Hint

Please follow the procedure, outlined here with an example: http://www.sosmath.com/matrix/diagonal/diagonal.html

Update your question with the results of your work and post a comment to this response. I will be happy to check your work for you.