How does ZFC solve the problem of alphabet in formal languages? (In case someone thinks this is another question about the seeming circularity in formal languages and is going to downvote because of this, it's really not; don't downvote yet, keep reading)
Perhaps the most widely known example of a formal language is the language of first-order logic, whose alphabet consists, among others, of symbols (defined as "objects meaning other objects" like ${\forall}$ or ${\implies}$. As alphabet is a set of symbols used in the language, then in the ZFC set theory$^1$, where every element of a set is another set, these symbols, since they belong to a set, should be sets too. I haven't seen any way to define them as sets (like I did in the case of natural numbers, where for example $0$ is often defined as ${\emptyset}$). Did anyone propose a way how to do such a thing?
$^1$I feel like there is a way of using ZFC even before defining a formal language: list the axioms in a natural language$^2$, define set as every object which can be built from these axioms$^3$, build a first-order logic language using this notion of a set, write down the ZFC using this first-order logic language and use this new list of axioms. Is this correct?
$^2$The meaning which ZFC tries to convey is possible to be said in a natural language.
$^3$In case somebody says that at this level the concept of a set doesn't need to be formalised and is best left to be intuitively understood, one of the reasons ZFC is so popular is that it reproduces intuitive properties of sets - so there seems to be no harm in using ZFC this way.

We just use some arbitrary set $A$ as the set of symbols. So, for example, some set plays the role of $\land$, and some other set plays the role of $\forall$. The expressions are then sequences of elements from that set $A$. This is very similar to the idea of Goedel numbering in which we use the set of natural numbers as the set of symbols. 
