What are relative open sets? I came across the following:
Deﬁnition 15. Let $X$ be a subset of $\mathbb{R}$. A subset $O \subset X$ is said to be open in $X$ (or relatively open in $X$) if for each $x \in O$, there exists $\epsilon = \epsilon(x) > 0$ such that $N_\epsilon (x) \cap X \subset O$.
What is $\epsilon$ and $N_\epsilon (x) $?  Or more general, what are relatively open sets?

Forget your definition above. The general notion is:
Let $X$ be a topological space, $A\subset X$ any subset. A set $U_A$ is relatively open in $A$ if there is an open set $U$ in $X$ such that $U_A=U\cap A$.
I think that in your definition $N_\epsilon(x)$ is meant to denote an open neighborhood of radius $\epsilon$ of $x$, ie $(x-\epsilon,\ x+\epsilon)$. As you can see, this would agree with the definition I gave you above.
