Rule for calculating big-O plus example I can't figure out I have the following rule:
If $f$ is $O(g)$ for ${x\to\infty}$ and $\lim_{x\to\infty}g(x) = 0$ than also $\lim_{x\to\infty}f(x) = 0$
Then my text proceeds to give an example:
$\lim_{x\to\infty}\frac{3^x+x^2+x\sqrt{x^5}-\ln(x^2+1)}{2\cdot3^{x+1}-2^x+\sqrt{1+x^6}}$
And it says: "We will use the fact that the last three terms of the numerator are $O(x^4)$, and the fact that the last two terms of the denominator are $O(2^x)$". This is what happens next:
$\lim_{x\to\infty}\frac{3^x+O(x^4)}{2\cdot3^{x+1}+O(2^x)} = \lim_{x\to\infty}\frac{1+O(\frac{x^4}{3^x})}{2\cdot3+O(\frac{2^x}{3^x})} = \frac{1}{6}$
I fail to see how the rule is used, because e.g. if $O(g)$ is $O(x^4)$, this does not equal zero. I also can't figure out what $f$ is supposed to be.

$f = O(g)$ means that $f/g$ is eventually bounded. In your case, the author writes $x^{2} + x\sqrt{x^{5}} - \log (x^{2}+1)$ as $O(x^{4}),$ for the the former divided by $x^{4}$ is eventually bounded. For the same reason, the author also writes $-2^{x} + \sqrt{1 + x^{6}}$ as $O(2^{x})$. 
Note in addition that convergence implies boundedness.
Note also that the point to use $o$ or $O$ notation is not too "calculate", but rather to preserve the essential information that matters in such a context. In your example, you find that using the $O$-notation greatly simplifies the work and helps one to catch the essentials.
