Is my assumption correct for the definition of outer measure? Royden defines outer measure as follows. Let $E$ be any subset of $\mathbb{R}$ and let $\{ I_k \}_{k=1}^{\infty}$ be a collection of open intervals. Outer measure is defined as follows: $$m^*(E) = inf\{ \sum_{k=1}^{\infty} l(I_k) : E \subseteq \cup_{k=1}^{\infty} I_k \}$$
This implies that there exists a collection $\{ I_k \}_{k=1}^{\infty}$ where $$m^*(E) \leq \sum_{k=1}^{\infty}l(I_k) < m^*(E) + \varepsilon.$$
Such a collection exists by the definition of outer measure, since $m^*(E)$ is the infimum of all the lengths of such a collection. 
This rational does not feel very rigorous, how can I show this with more rigor?

That is totally correct. If $m$ is the infimum of a set $S$, it means that $m$ is the greatest lower bound of $S$, so, if you move to the right of $m$ a distance $\epsilon$, then $m+\epsilon$ is not a lower bound of $S$, because if it was, then it would be a lower bound greater than $m$, contradicting the definition of infimum. We can say that $m+\epsilon$ is not a lower bound denying that $m+\epsilon \leq s$ for all $s \in S$, or, what is the same, saying that there exists an $s \in S$ such that $m \leq s < m + \epsilon$.

