Is O(n) a proper class or a set? Is $O(n)$ as the collection of all functions that are bounded above by $n$ a proper class or just a set?
What about $O(\infty)$?

Since $\Bbb R$ is a set, we know that $\Bbb{R\times R}$ is a set, so $\mathcal P(\mathbb{R\times R})$ is a set.
Therefore the collection of all functions from $\Bbb R$ to itself is a set. In particular any definable subcollection of a set is a set. For example, all the functions which are $O(n)$ or otherwise.
