For the purpose of model fitting in a large time series dataset, I am using stochastic gradient descent of the negative log likelihood. The model is nonlinear and non-convex. Is there a thumb rule for choosing a good step size? I could choose a very small step size for stable but painfully slow convergence, but I would like to be able to choose a big enough step size for faster convergence and anneal it.
How to choose a good step size for stochastic gradient descent?
optimization;stochastic