$\operatorname{Tr}(AB)=\operatorname{Tr}(A)\operatorname{Tr}(B)$ 
Problem. Show that $\operatorname{Tr}(AB)=\operatorname{Tr}(A)\operatorname{Tr}(B)$, if $A^2+3AB+B^2=BA$ and $\det(A)=0$, where all matrices involved is real valued $2\times2$ matrices.

First I rewrote the $A^2+3AB+B^2=BA$ as 
$$ A(A+3B)=B(A-B) $$
and I took the determinant. Then I obtained that
$$ \det(B)\det(A-B)=0, $$
because $\det(A)=0$. So, $\det(B)=0$ or $\det(A-B)=0$.
Case I. Then, by Cayley-Hamiton theorem, I wrote that
$$ A^2=(\operatorname{Tr}(A))A \text{ and } B^2=(\operatorname{Tr}(B))B. $$
Multiplying the above two equations, I got $(AB)^2=(\operatorname{Tr}(A)\operatorname{Tr}(B))AB$. Another use of Cayley theorem got $(AB)^2=(\operatorname{Tr}(AB))AB$. So, $\operatorname{Tr}(A)\operatorname{Tr}(B)$ must be equal with $\operatorname{Tr}(AB)$ only if $AB=BA$. But I don't have such a hypothesis.
Case II. For the case where $\det(A-B)$ I don't know how to handle this one in order to get the answear.

Here is a bit messy solution:


Lemma. Let $F$ be a field with characteristic $\neq 2$ and $A$ and $B$ be $2\times 2$ be matrices over $F$.
  
  
*
  
*If $A$ and $B$ commute, then either $A = pB + qI$ or $B = pA + qI$ for some $p, q \in F$.
  
*If $A^2 = k(AB - BA)$, then $A^2 = 0$. In particular, if $k \neq 0$, then $AB = BA$.
  

We postponed the proof of this lemma to the end. By rearranging the equality in the assumption, we obtain $(A+B)^2 = 2[(A+B)A-A(A+B)]$. So, by Lemma, $(A+B)^2 = 0$ and $AB = BA$. Now we claim the following.

Claim. Either $A = 0$ or $B = kA$ for some $k$.

Once this is true, then the rest is straightforward: If $A = 0$, there is nothing to prove. If $B = kA$ for some $k$, then
$$ \operatorname{Tr}(AB)
= k\operatorname{Tr}(A^2)
= k\operatorname{Tr}(A)^2
= \operatorname{Tr}(A)\operatorname{Tr}(B), $$
where the second step follows from the identity $\operatorname{Tr}(A^2) = \operatorname{Tr}(A)^2$, which itself is a consequence of both Cayley-Hamilton theorem and $\det(A) = 0$. $\square$
So it remains to show the claim. Assume that $A \neq 0$. Since $\det(A) = 0$, we may utilize Lemma to write
$$A+B = aA + bI $$
for some $a, b$. Indeed, this is obvious when $B = pA + qI$. On the other hand, if $A = pB + qI$, then $p \neq 0$ by the assumption, and so, $A+B = \frac{p+1}{p}A - \frac{q}{p}I$.
Next, we consider two possibilities:


*

*If $a = 0$, then $0 = (A+B)^2 = b^2 I$, and so, $b = 0$. This implies $B = -A$.

*If $a \neq 0$, then by rearranging $0 = (A+B)^2$ we obtain $A\left(A + \frac{2b}{a}I\right) = -\frac{b^2}{a^2} I$. Since $A$ is not invertible, this must imply $b = 0$ and hence $B = (a-1)A$.
This completes the proof of Claim. $\square$

Proof of Lemma. For the first part, replacing $A$ by $A-\frac{1}{2}\operatorname{Tr}(A)$ and $B$ by $B-\frac{1}{2}\operatorname{Tr}(B)$ if needed, we may assume that $A$ and $B$ take the form
$$ A = \begin{pmatrix} a & b \\ c & -a \end{pmatrix}, \qquad B = \begin{pmatrix} p & q \\ r & -p \end{pmatrix} $$
Now write $u = (a, p)$, $v = (b, q)$, $w = (c, r)$. Then comparing both sides of $AB - BA = 0$, we find that $AB = BA$ if and only if $\det(u, v) = \det(v, w) = \det(w, u) = 0$. In particular, they are parallel to each other. This implies that $A$ and $B$ are parallel, and so, the desired conclusion follows.
For the second part, we first check $\det(A) = 0$. To this end, assume otherwise. Then
$$ 2 = \operatorname{Tr}(I) = \operatorname{Tr}(A^{-1}A^2A^{-1}) = k \operatorname{Tr}(BA^{-1} - A^{-1}B) = 0,$$
a contradiction. Then taking trace to both sides, by Cayley-Halimton theorem,
$$\operatorname{Tr}(A)^2 = \operatorname{Tr}(A^2) = k \operatorname{Tr}(AB-BA) = 0, $$
and so, $\operatorname{Tr}(A) = 0$. Therefore the desired claim follows from Cayley-Hamilton theorem. $\square$
