Proof $\lim\limits_{N\to\infty}N^{-1}\sum_{k=1}^N e^{-ik^2x}=0$ for all $x\notin 2\pi\mathbb Q$ The formula has no analytic expression. Is there good way to prove that.

First let's rewrite your sum so it looks more conventional: setting $x = -2\pi\alpha$, your average is $(1/N)\sum_{k=1}^N e^{2\pi i(\alpha k^2)}$.
That this sum tends to $0$ as $N \rightarrow \infty$ is a special case of an equidistribution theorem of Weyl, who showed for any polynomial $p(t)$ with at least one irrational coefficient that the sequence $p(k) \bmod 1$ is equidistributed. Take the polynomial to be $p(t) = \alpha t^2$.  A consequence of the Weyl equidistribution theorem is that $(1/N)\sum_{k=1}^N e^{2\pi i m(\alpha k^2)} \rightarrow 0$ as $N \rightarrow \infty$ for every nonzero integer $m$.  Your average is the case $m = 1$.
This problem has been discussed already at Equidistributed problem about polynomial with irrational coefficient.
