2 Answers for Derivative of $x^2 \sin 1/x$ Derivative of $x^2\sin(1/x)$ : 
if we use the derivative rules to take the derivative of this function we conclude that It is not defined at x =0,
but if we use the derivative definition (lim x -> a), we get this:
$$x\sin(1/x)$$
which has a derivative of 1 at 0. 
Which is the answer? I mean obviously the formal defintion is right, but why does this happen?

The derivative by product rule and chain rule for $x\ne0$ gives us
$$\frac d{dx}x^2\sin(1/x)=2x\sin(1/x)-\cos(1/x)$$
While it is true that this is undefined at $x=0$, this is simply because we are not able to apply chain rule at $x=0$.  At $x=0$, we must apply the limit definition:
$$\frac d{dx}\bigg|_{x=0}x^2\sin(1/x)=\lim_{h\to0}\frac{h^2\sin(1/h)}h=\lim_{h\to0}h\sin(1/h)=0$$
which follows from the squeeze theorem:
$$-h\le h\sin(1/h)\le h$$
One may note that it is not always the case that just because a function is differentiable everywhere, then the derivative will be continuous.  Such behavior is called the smoothness of a function, and it depends on how many times you can differentiate a function and still have a continuous function.
Your specific example may be found on Wikipedia as the second example of smoothness
