Square root of a $3\times3$ matrix Here is  $3\times3$ matrix$$\begin{pmatrix}   0&amp;    0&amp;    1\\
    0  &amp; -1 &amp;   0\\
    1&amp;    0  &amp;  0\end{pmatrix}$$
How can I solve this  by using Cayley-Hamilton?
I know how to use Cayley-Hamilton for a $2$-dimensional matrix.
How can it help in finding the square root of a $3\times3$ matrix?
for 2 dimensional matrix we can solve this equation A^2−(trA)A+(detA)I=0 
we have A and I,
we can compute det(A^2) so we have det A,
and  we can find A.

for 2 dimensional matrix using above equation we can compute square root.
for example we have this matrix:

A^2= $$\begin{pmatrix}   4&amp;    2\\
        2&amp;    2\end{pmatrix}$$

det A^2= 4

det A=2




4 2             1 0
2 2  +6 *A+ 2*  0 1 =0


by solving above equation we can find A.
Since your matrix $M$ is symmetric you can diagonalize it. You'll get $M=U \cdot D \cdot U^\dagger$, where $D$ contains the eigenvalues and $U$ contains the eigenvectors. Take the square roots of the eigenvalues  to get $D_{\sqrt \cdot}$ and transform the resulting diagonal matrix $U^\dagger \cdot D_{\sqrt{.}}\cdot U=S$. It's obvious to see that $S^2=M$.

EDIT

And even that I don't understand your original approach, you can use it when you rewrite your matrix to 
$$
\pmatrix{0 &amp; 1 &amp; 0\\1&amp;0&amp;0\\0&amp;0&amp;-1}.
$$
Now you have one $2\times 2$ block suitable for your approach.
I solved it by Euler's method of solving for the (later named) unit-length eigenvector (a tilted 3D rotational axis), and the Euler 2D angle theta of rotation about the 3D Euler axis.

I did it computationally.

The square root matrix is:
{ { 0.5  , 0.7071, 0.5 }, {-0.7071,0,0.7071}, {0.5,-.7071,0.5}}