What is the formal definition of indefinite integrals? I know the definition of definite integral using limits (or sups and infs) in rectangles in a given partition. However I realized that I don't know the formal definition of indefinite integrals.
My question is really simple, what is the definition of indefinite integrals?

You can formally define the the concept of an anti-derivative and introduce the notation of an indefinite integral to mean the set of all anti-derivatives of a function. For the anti-derivative it would be something along the lines of:

For a function $f : [a,b] \to \mathbb{R}$, we call a differentiable function $F : [a,b] \to \mathbb{R}$ an anti-derivative of $f$ if $F'(x) = f(x)$ for all $x$ in $(a,b)$.

Then introduce the following notation for the set of all anti-derivatives of $f$:
$$\int f(x) \, \mbox{d}x$$
Or, once you have the definite integral, you can use it to introduce the indefinite integral, for example:
$$\int f(x) \, \mbox{d}x = \int_a^x f(t) \, \mbox{d}t + C $$
This relies on the fact that the definite integral in the RHS is one anti-derivative of $f$ (this is the fundamental theorem of calculus) and you use the fact that all anti-derivatives can only differ up to a constant term, so adding the arbitrary constant gives you all of them. 
Apostol does this in his Calculus, Vol. 1 where he introduces the definite integral long before the "Leibniz notation for the indefinite integral".
