For non-convex penalized problems with a single regularization parameter \(\lambda \) , recent path-following routines apply proximal gradient descent or coordinate descent over a decreasing sequence of regularization parameters [1]}, [2]}. At each new \(\lambda \) , these routines use a warm start, initializing at the solution of the prior \(\lambda \) . The result is a sequence of estimates across a range of penalization levels, also called a regularization or solution path. Under certain conditions these `approximate path-following' approaches yield high-quality local solutions with attractive theoretical properties, even when the loss and/or regularizer are non-convex. Heuristically, many non-convex objective functions are locally convex in the neighborhood of well-behaved optima, and so incrementally optimizing over a sequence of small changes to \(\lambda \)  with warm starts ensures that each local solution remains in the convex neighborhood of the subsequent penalized function's solution.
