Give a big-O estimate of $(x+1)\mathrm{log}(x^2+1) + 3x^2$ I wanted to know if the following solution demonstrates that the function $f(x) = (x+1)\mathrm{log}\, (x^2+1) + 3x^2 \in O(x^2)$, because my answer and the book's answer deviate slightly.
Clearly, 
$$3x^2 \in O(x^2) \tag{1}$$
$$x+1 \in O(x)\tag{2}$$
The following inequality is where the book and I differ, but I understand how the author obtained his big-O estimate. I said
$$\mathrm{log}(x^2+1) \in O(\mathrm{log}(x^2+1))\tag{3}$$
Therefore, the product of $(2)$ and $(3)$ renders
$$(x+1)\mathrm{log}(x^2+1) \in O(x \mathrm{log}(x^2+1)) \tag{3}$$
Finally, $(1)$ and $(3)$ gives us this big-O estimate
$$(x+1)\mathrm{log}(x^2+1) \in O(\mathrm{max}(x^2,x\mathrm{log}(x^2+1)) = O(x^2) \tag{4}$$
Any problems! Thanks!

The issue from what I see is the in the $(x+1)\mathrm{log}(x^2+1)$ portion of your proof. Here, you didn't properly derive the Big-O value to it's tightest bound. I'll give you a headstart on the problem and then I'll leave the rest for you. 
Within $\mathrm{log}(x^2+1)$ you can note that $x^2 + 1 \leq 2x^2$ whenever $x \gt  1$. This is because if $x \gt  1$ then $x^2 \gt  1$, so we would just substitute $x^2$ for $1$ to obtain $x^2 + 1 \leq 2x^2$. Now, given this information you can $\mathrm{log}$ both sides and find the correct value, which should be $O(x\mathrm{log}x)$.
In general, it is always better to have your Big-O and Big-Omega notation as tight as possible so that you can most accurately determine the growth rate.
Hope this helps.
