Why ReLU function is not differentiable at 0? I'm kind of rusty in calculus. 
Why is the ReLU function not differentiable at $f(0)$?
$$
f(x) = 
\begin{cases}
0 & \text{if $x \leq 0$} \\ 
x & \text{if $x > 0$}.
\end{cases}
$$

Because it has a sharp corner at $0$, so it doesn't have a well defined tangent line; think about it, you can imagine many lines going through $(0, 0)$ that are tangent to the graph, so there are many possible tangent lines.
More formally, we have to investigate the limit
$$\lim_{h \to 0} \dfrac{f(0+h) - f(0)}{h}$$
This limit does not exist for the function, because if you let $h$ approach $0$ from the right, you get
$$\lim_{h \to 0^+} \dfrac {h-0}{h}=1.$$
While if you let $h$ approach $0$ from the left, that limit 
$$\lim_{h \to 0^-} \dfrac {0-0}{h}=0.$$
Therefore the limit does not exist, so the function is not differentiable at $0$.
