Prove that $\sum_{n\leq x}d^2(n)=O(x\log^3 x)$ 
Prove that $$\sum_{n\leq x}d^2(n)=O(x\log^3 x),\tag1$$
  where $d(n)$ is the divisor function:
  $d(n)=\sigma_0(n)=\sum_{a\mid n}1.$

I can prove that $$\sum_{n\leq x}d(n)=x\log x+(2\gamma -1)x+O(\sqrt x),\tag2$$ but how to prove $(1)$?

I can give you a slightly weaker bound easily. Note that, $$\sum_{n=1}^\infty\frac{d^2(n)}{n^s}=\prod_p\sum_{r=0}^\infty\frac{(r+1)^2}{p^{rs}}=\prod_{p}\left(1-\frac{1}{p^{2s}}\right)\left(1-\frac{1}{p^s}\right)^{-4}=\frac{\zeta^4(s)}{\zeta(2s)}.$$ 
Second equality comes from the identity,$$\sum_{n=1}^\infty(n+1)^2x^n=\frac{(1-x^2)}{(1-x)^4}.$$
Now using well-known formula for sum of coefficients of a Dirichlet series,
$$D(x):=\sum_{n\le x}d^2(n)={1\over 2\pi i}\int_{(c)}\frac{\zeta^4(s)}{\zeta(2s)}x^s\frac{\,ds}{s},$$ where $(c)=\{c+it:t\in \mathbb{R},c>1 \ \text{fixed}\}.$ 
So, $$D(x)={1\over 2\pi}\int_{-\infty}^{\infty}\frac{\zeta^4(c+it)}{\zeta(2c+2it)}\frac{x^{c+it}}{c+it}\,dt\le\frac{x^c}{2\pi}\int_{-\infty}^{\infty}\left|{\frac{\zeta^4(c+it)}{\zeta(2c+2it)}\frac{x^{it}}{c+it}}\right|\,dt=O(x^c)$$ for any $c>1$
Now for your bound, note that, $$\sum_{n=1}^\infty\frac{d_3(n)}{n^s}\sum_{n=1}^\infty\frac{|\mu(n)|}{n^s}=\zeta^3(s)\frac{\zeta(s)}{\zeta(2s)}=\sum_{n=1}^\infty\frac{d^2(n)}{n^s},$$ where $d_3(n)$ is number of ways to write $n$ as product of $3$ numbers.
So,$$d^2(n)=\sum_{r|n}d_3(r)\left|\mu\left( {n\over r}\right)\right|.$$ Hence, $$D(x):=\sum_{n\le x}d^2(n)=\sum_{n\le x}\sum_{r|n}d_3(r)\left|\mu\left( {n\over r}\right)\right|$$
$$=\sum_{ab\le x}d_3(a)|\mu(b)|\le x\sum_{a\le x}{d_3(a)\over a}=O(x\log^3x)$$.
Here, I have used $$\sum_{n\le x}d_k(x)=O(x(\log x)^{k-1}),$$ which would be easy for you to prove.
