I'm thinking about gradient descent, but I don't get it.I understand that it can overshoot the minimum when the learning rate is too large. But I can't understand why it would diverge.Let's say we have $$J(\theta_0, \theta_1) = \frac{1}{2m}\sum_{i=1}^m (h_\theta(x^i)-y^i)^2$$$$\theta_1 := \theta_1-\alpha\frac{\partial}{\partial\theta_1}J(\theta_1)$$  When the slope is negative the cost will converge to the minimum from the left of the graph, as $\theta_1$ will increase.When the slope is positive it will converge from the right of the graph, as $\theta_1$ will decrease.Now it might overshoot when the learning rate $\alpha$ is too large.In that case it should overshoot again.But not by much, should it not circle around the minimum? Why would it diverge?  
Gradient descent overshoot - why does it diverge?
optimization;machine learning;gradient descent