Can I create a set of new elementary functions such that their integral is an elementary function? Many elementary integrals are well known.
For example $\int \sin(x) = \cos(x) + C$.

Then there is $\int e^{-x^2}$ that cannot be expressed in terms of elementary functions. Therefore the set of elementary functions is not closed for the operator $\int$.

Now, let's say I add a new function $\operatorname{integ}(x) = \int_{0}^{x} e^{-t^2} dt$ and define that to be a new elementary function like $\sin(x)$ and I keep doing this. Will I ever get to a complete closed set of elementary functions such that their integral is another elementary function in the new set I defined?
Sure, but it's not going to be a very simple class of functions. For the purposes of this answer, let $E$ be the set of elementary functions according to the usual definition. You could define $E$ formally by starting with a class of functions, say $F_0 = \{\sin(x), \cos(x), \exp(x), \log(x), x, c\}$, where $c$ represents all constant functions, and then taking the closure of $F_0$ with respect to addition, multiplication, and composition. Rigorously, you would define $F_{n+1} = \{f + g, f \circ g, f\cdot g \text{ | } f,g \in F_{n}\}$, then $E = \bigcup_{n\ge0} F_n$ is the set of all functions that can be built in finitely many steps. If you want to have a set $E'$ which is also closed under integration, you would just define $F_{n+1}$ to also include integrals of functions in $F_n$, and with this modified definition, the union of the $F_n$'s would also be closed under integration.
Take whatever function $f(x)$ you want (regular, good, et cetera...) and differentiate it.

Then your $f'(x)$ will be a certain weird strange function with an elementary integral.

Example:

$$f'(x) = x^{\sin \left(\log \left(x^2-x\right)\right)} \left(\frac{\sin \left(\log \left(x^2-x\right)\right)}{x}+\frac{(2 x-1) \log (x) \cos \left(\log \left(x^2-x\right)\right)}{x^2-x}\right)$$

Which gives you an elementary function, when integrated.

$$\int f(x)\ dx = x^{\sin(\ln(x^2-x))} + C$$