How to show that this group presentation is isomorphic to dihedral group? The group with the presentation $\langle a, b \mid a^2, b^2, (ab)^4\rangle$ seems like a dihedral group of order 8. But I fail to explain why. Could anyone show me how to prove it?

Yes, it is the dihedral of order eight. Just observe that
$$s=a\;,\;\;t=ab\implies sts=a(ab)a=ba=(ab)^{-1}=t^{-1} $$
and you get the usual presentation for $\;D_4$.
