Stochastic Zeroth order Descent with Structured Directions
We introduce and analyze Structured Stochastic Zeroth order Descent (S-SZD),
a finite difference approach which approximates a stochastic gradient on a set
of $l\leq d$ orthogonal directions, where $d$ is the dimension of the ambient
space. These directions are randomly chosen, and may change at each step. For
smooth convex functions we prove almost sure convergence of the iterates and a
convergence rate on the function values of the form $O(d/l k^{-c})$ for every
$c<1/2$, which is arbitrarily close to the one of Stochastic Gradient Descent
(SGD) in terms of number of iterations. Our bound also shows the benefits of
using $l$ multiple directions instead of one. For non-convex functions
satisfying the Polyak-{\L}ojasiewicz condition, we establish the first
convergence rates for stochastic zeroth order algorithms under such an
assumption. We corroborate our theoretical findings in numerical simulations
where assumptions are satisfied and on the real-world problem of
hyper-parameter optimization, observing that S-SZD has very good practical
performances.