Use least squares to find best fit value of angle phi 
Given initial object $\begin{bmatrix}1 & 0 & 0\\0 & 0 & 2\end{bmatrix}$ and final object $\begin{bmatrix}0.9557 & 0.0965 & 0.9648\\-0.3381 & 0.0158 & 1.8977\end{bmatrix}$ after rotation with a certain angle $\phi$. There was a perturbation during the rotation so that there is no exact solution. Using least squares to find the best value of $\phi$.


So far I have
\begin{align*}
  \begin{bmatrix}
  \cos(\phi) & \sin(\phi)\\ -\sin(\phi) & \cos(\phi)
  \end{bmatrix}
  \begin{bmatrix}
  1 & 0 & 0 \\ 0 & 0 & 2
  \end{bmatrix}&=
  \begin{bmatrix}
  0.9557 & 0.0965 & 0.9648\\-0.3381 & 0.0158 & 1.8977
  \end{bmatrix}\\
  \rightarrow
  \begin{bmatrix}
  \cos(\phi) & \sin(\phi)\\ -\sin(\phi) & \cos(\phi)
  \end{bmatrix}
  \begin{bmatrix}
  1 & 0 & 0 \\ 0 & 0 & 2
  \end{bmatrix}
  \begin{bmatrix}
  1 & 0 \\ 0 & 0\\ 0 & 2
  \end{bmatrix}&=
  \begin{bmatrix}
  0.9557 & 0.0965 & 0.9648\\-0.3381 & 0.0158 & 1.8977
  \end{bmatrix}
  \begin{bmatrix}
  1 & 0 \\ 0 & 0\\ 0 & 2
  \end{bmatrix}\\
  \rightarrow
  \begin{bmatrix}
  \cos(\phi) & \sin(\phi)\\ -\sin(\phi) & \cos(\phi)
  \end{bmatrix}
  \begin{bmatrix}
  1 & 0\\ 0 & 4
  \end{bmatrix} & =
  \begin{bmatrix}
  0.9557 & 1.9296\\
  -0.3381 & 3.7954
  \end{bmatrix}\\
  \rightarrow
  \begin{bmatrix}
  \cos(\phi) & \sin(\phi)\\ -\sin(\phi) & \cos(\phi)
  \end{bmatrix}&=
  \begin{bmatrix}
  0.9557 & 0.4824\\
  -0.3381 & 0.9488
  \end{bmatrix}
 \end{align*}
At this step, I don't know how to find the angle $\phi$, can someone give me a hint or a suggestion? Thanks.

You're proceeding as if the four entry of the rotation matrix were independent quantities. Instead you need to minimize the sum of squares of the errors in the matrix entries:
$$
\pmatrix{\cos\phi&\sin\phi\\-\sin\phi&\cos\phi}\pmatrix{1&0&0\\0&0&2}=\pmatrix{\cos\phi&0&2\sin\phi\\-\sin\phi&0&2\cos\phi}\;.
$$
The errors in the middle column don't depend on $\phi$, so we can ignore them and minimize
\begin{align}
&(\cos\phi-0.9557)^2+(-\sin\phi+0.3381)^2+(2\sin\phi-0.9648)^2+(2\cos\phi-1.8977)^2\\
={}&2\left(-0.9557\cos\phi-0.3381\sin\phi-2\cdot0.9648\sin\phi-2\cdot1.8977\cos\phi\right)+\textsf{const.}
\end{align}
Setting the derivative with respect to $\phi$ to zero yields
$$
4.7511\sin\phi-2.2677\cos\phi=0\;,
$$
so $\phi=\textsf{atan2}(2.2677,4.7511)\approx0.44532$, so the best fit matrix is approximately
$$
\pmatrix{0.9025&0&0.7775\\-0.4307&0&1.8049}\;.
$$
