What transforms under SU(2) as a matrix under SO(3)? A vector $\boldsymbol{r}$ in $\mathbb{R}^3$ transforms under rotation $\boldsymbol{A}$ to $\boldsymbol{r}'=\boldsymbol{Ar}$. It is equivalent to an SU(2) "rotation" as
$$\left( \boldsymbol{r}'\cdot\boldsymbol{\sigma} \right) = \boldsymbol{h} \left( \boldsymbol{r}\cdot\boldsymbol{\sigma} \right) \boldsymbol{h}^{-1},$$
where $\boldsymbol{h}$ is the counterpart of $\boldsymbol{A}$ in SU(2) given by the homomorphism between these two groups.
Now the question is, what would be the equivalent transformation in SU(2) of the rotation of a matrix in $\mathbb{R}^3$? In other words, what is the equivalent in SU(2) of $\boldsymbol{M}'=\boldsymbol{A}\boldsymbol{M}\boldsymbol{A}^{-1}$.

Firstly, we need to map $\mathbb{R}^3$ to the representation space $V$ for $\mathrm{SU}(2)$. One possible map is given by the following formula:
$$\begin{pmatrix} x \\ y \\ z \end{pmatrix} \mapsto 
x \mathbf{I} + 
y \mathbf{J} +
z \mathbf{K}$$
where
\begin{align}
\mathbf{I} & = \begin{pmatrix} i & 0 \\ 0 & -i \end{pmatrix} &
\mathbf{J} & = \begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix} &
\mathbf{K} & = \begin{pmatrix} 0 & i \\ i & 0 \end{pmatrix}
\end{align}
$\mathrm{SU}(2)$ acts on $V$ by conjugation: so for each $X$ in $V$ and each $A$ in $\mathrm{SU}(2)$, the ordinary matrix product $A X A^{-1}$ is in $V$. This is linear in $X$ and is indeed a linear representation of $\mathrm{SU}(2)$. Indeed, if
$$A = \begin{pmatrix} r e^{i \theta} & s e^{-i \phi} \\ -s e^{i \phi} & r e^{-i \theta} \end{pmatrix}$$
where $r, s, \theta, \phi$ are real numbers and $r^2 + s^2 = 1$, then $A \in \mathrm{SU}(2)$, and
\begin{align}
A \mathbf{I} A^{-1} & = (r^2 - s^2) \mathbf{I} + 2 r s \sin (\theta - \phi) \mathbf{J} - 2 r s \cos (\theta - \phi) \mathbf{K} \\
A \mathbf{J} A^{-1} & = 2 r s \sin (\theta + \phi) \mathbf{I} + (r^2 \cos 2 \theta + s^2 \cos 2 \phi) \mathbf{J} + (r^2 \sin 2 \theta - s^2 \sin 2 \phi) \mathbf{K} \\
A \mathbf{K} A^{-1} & = 2 r s \cos (\theta + \phi) \mathbf{I} - (r^2 \sin 2 \theta + s^2 \sin 2 \phi) \mathbf{J} + (r^2 \cos 2 \theta - s^2 \cos 2 \phi) \mathbf{K}
\end{align}
Thus, the induced action of $\mathrm{SU}(2)$ on $\mathbb{R}^3$ is given by the group homomorphism below,
$$\begin{pmatrix} r e^{i \theta} & s e^{-i \phi} \\ -s e^{i \phi} & r e^{-i \theta} \end{pmatrix} \mapsto 
\begin{pmatrix} 
r^2 - s^2 & 2 r s \sin (\theta + \phi) & 2 r s \cos (\theta + \phi) \\
2 r s \sin (\theta - \phi) & r^2 \cos 2 \theta + s^2 \cos 2 \phi & -r^2 \sin 2 \theta - s^2 \sin 2 \phi \\
-2 r s \cos (\theta - \phi) & r^2 \sin 2 \theta - s^2 \sin 2 \phi & r^2 \cos 2 \theta - s^2 \cos 2 \phi
\end{pmatrix}$$
and one may verify that the RHS is a matrix in $\mathrm{SO}(3)$.
