Is there a formula for solving integrals of this form? I was wondering if there was a substitution formula to solve integrals of this form:
$\int f(g(x))g''(x)dx$

No, not a nice one, anyway. It is worthwhile, I think, to point out that integration rules, such as the usual substitution rule, do not always "solve" ("evaluate" is the proper term) the given integral. The usual substitution rule, for instance, only transforms the integral into another integral which may or may not be easily handled.
Of course, if the antiderivative of $f$ is known, then the usual substitution rule will allow you to evaluate integrals of the form $\int f\bigl(g(x)\bigr)g'(x)\,dx$.
I don't think a formula of the type you seek would be very useful, as it can't handle all cases when an antiderivative of the "outer function" is known: consider $\int \sin(x^2)\cdot 2\,dx$. This can't be expressed in an elementary way.
