Use coordinate method to solve a pretty hard geometry problem Here is a hard geometry problem for my homework.

Let $D$ be a point inside $\Delta ABC$ such that $\angle BAD=\angle BCD$ and $\angle BDC=90^\circ$. If $AB=5,BC=6$ and $M$ is the midpoint of $AC$, find the length of $DM$. 

(Taken from HK IMO Prelim 2012)
The teacher wants us to use the coordinate geometry to solve this problem. It does not appear too hard, but there is a big problem.

I set the coordinates of $A,B,C,D,M$ as in the diagram. Then, we can get the following statement:
$$\begin{cases} b^2+c^2=36 \\ x^2+\left(b+y\right)^2=25 \\ \sin\angle BAD=\sin\angle BCD \end{cases}$$
We need to find the value of $\sin\angle BAD$ and $\sin\angle BCD$
$$\sin\angle BCD=\dfrac{BD}{BC}=\dfrac{b}{6}\\ \text{The area of }\Delta ABD=\dfrac{1}{2}\left(AB\right)\left(AD\right)\sin\angle BAD \\ \dfrac{bx}{2}=\dfrac{5\sqrt{x^2+y^2}\sin\angle BAD}{2} \\ \sin\angle BAD=\dfrac{bx}{5\sqrt{x^2+y^2}}\\ \dfrac{b}{6}=\dfrac{bx}{5\sqrt{x^2+y^2}} \\ \sqrt{x^2+y^2}=\dfrac{6}{5}x \\ x^2+y^2=\dfrac{36}{25}x^2 \\ y^2=\dfrac{11}{25}x^2 \\ y=\dfrac{\sqrt{11}}{5}x$$ 
Then, when I proceed further, the expression gets very complicated. I felt puzzled, so I stopped. I found a geometric solution, which gives the answer of $\dfrac{\sqrt{11}}{2}$. 
I hope you guys can help me solve this question using coordinate method. Thank you!

Let $\angle BDA = \alpha$ and apply the sine rule to the triangle $\triangle ABD$, 
$$\frac{\sin \alpha}{\sin\angle BAD}=\frac{5}{6\sin\angle BCD}$$
Given that $\beta = \angle BAD = \angle BCD$, we immediately get
$$\sin \alpha = \frac 56$$
Now, express the coordinates of $A(-x,-y)$ as follows,
$$x=AB\sin \angle ABD = 5\sin (\alpha + \beta)$$
$$y=AB\cos \angle ABD - BD = -5\cos (\alpha + \beta) - 6 \sin\beta$$
Then, the coordinates of the point $M(x_m,y_m)$ are
$$x_m = \frac 12 (CD - x)=\frac 12 [6\cos\beta - 5\sin (\alpha + \beta)]$$
$$y_m = -\frac 12 y = \frac 12 [5\cos (\alpha + \beta) + 6 \sin\beta]$$
As a result,
$$DM^2 = x_m^2 + y_m^2 = \frac 14 (36+25-60\sin\alpha)=\frac {11}{4}$$
Note that the unknown angle $\beta$ cancels out in the process. Finally, the length of $DM$ is 
$$DM = \frac{\sqrt{11}}{2}$$
