An element of $SL(2,\mathbb{R})$ An element $A$ of $SL(2,\mathbb{R})$ is called an elliptic element if $|\text{tr}(A)|<2$.

Find the relationship between an elliptic element of $SL(2,\mathbb{R})$ and rotation.

As $|\text{tr}(A)|<2$ the characteristic equation of $A$ does not have real roots, so it has no real eigenvalue. But I am unable to go ahead.

Let $g$ be your elliptic element. Since the eigenvalues are complex conjugate, $g$ will have an eigenvector $\binom z1\in\Bbb C^2$ with the imaginary part of $z$ positive.
It can be shown that you can always find some $h\in{\rm SL}_2(\Bbb R)$ such that $h\binom i1=c\binom z1$ for some $c\in\Bbb R$.
Thus $h^{-1}gh$ will have $\binom i1$ as eigenvector.
The final step is to show that the elements of ${\rm SL}_2(\Bbb R)$ having $\binom i1$ as an eigenvector are precisely the elements in the subgroup ${\rm SO}_2(\Bbb R)$, i.e. the rotations.
I leave the details as an exercise. 
