Can an integral domain have an element that has no square root but has a square root in the field of fractions? A lemma states: 

Let $R$ be a UFD and $F=\operatorname{Frac}(R)$. Let $d\in R$, then equation $a^2=d$ has a root in $R$ iff it has a root in $F$.

So I want to ask, is there a counterexample for this if $R$ is not a UFD?  I only know some non-UFD integral domains like ring of integers for some values.

You already have a very good example, but since you mentioned ring of integers, and to put this in a more general context: 
A (full) ring of algebraic integers (a maximal order) can never work as a counter-example. The point is that those are (basically by definition) integrally closed. 
This means that every element of the fraction field of $R$ that is a root of a monic polynomial over $R$ is already in $R$. Since your equation corresponds to a particular type of monic polynomial having a root, the assertion holds for this one in particular. 
However, if you take subrings of rings of algebraic integers (non-maximal orders) you can get examples. For example, in $\mathbb{Z}[2\sqrt{2}]$ the equation $X^2 = 2$ has not solution. But $\sqrt{2}$  is of course in the quotient field. 
Let me end with an abstract argument for UFDs having the property you recalled: 
A UFD is integrally closed and thus it is also two-root closed (this is a somewhat common name for the property you give). 
Put differently, when looking for counter-examples you need to avoid integrally closed domains, since they all still have the property mentioned in your lemma. 
