Here, \(\gamma \)  is a step parameter.
The updates can be improved to reduce convergence times by using optimizers like Adam or its variants with warm restarts using cosine annealing [1]}. The optimization loop converges to \(z^*\)  that should have a high \(D(z; \theta )\)  probability. The working intuition is that the surrogate surface is highly non-convex and optimization over diverse noise samples would lead to distinct local optima. In principle, every optimum corresponds to a unique element from the data distribution, allowing the GON model to generate a diverse set of samples. Alternatively, we could stop after each step with probability \(D(z)\)  (see Appendix ). A caveat of this approach is that sample generation could be slow as we run optimization in the input space. For some applications the overhead might be negligible or acceptable [2]}, [3]}.
