Goedel's representability of simple recursive sets I'm referring to Goedel's theorem as exposed here:
https://plato.stanford.edu/entries/goedel-incompleteness/
The formal system in question is named Q and is a first order formalization of natural numbers with addition and multiplication operations. A set S of natural numbers is said to be "weakly representable" if there exists a formula A(x) in Q such that for all n in S the formula A(n) can be proved in Q. A set is "strongly representable" if both itself and its complementary set are weakly representable.
It turns out that the notions of strongly/weakly representable sets in Q is equivalent to recursive and recursive enumerable sets which. I think this is the core of Goedel's proof.
I believe that the set of factorial numbers {n | exists m: m!=n} is recursive and hence should be strongly and also weakly representable. So there must exist a formula A(n) in Q which represents the property of n being the factorial of some number m. What is such a formula? I cannot find anything easy... 
I feel there is something I'm missing in the story so far.

Thanks to the answer of @Noah Schweber and the description of the $\beta$ function in wikipedia here is a possibile way to write the requested formula:
$$
rem(a,b)=c \colon \qquad (c<b) \land \exists n\colon a=b\cdot n + c\\
\beta(a,b,i) = c\colon \qquad rem(a,1+(i+1)\cdot b) = y\\
a_i = c \colon \qquad \beta(a,b,i) = c
$$
$$
n!=k\colon \qquad
\exists a \exists b \colon (a_1 = 1) \land (a_n =k) \land \forall i\colon ((i+1\le n \land i\ge 1)\implies a_{i+1} = (i+1)\cdot a_i)    
$$
