In Figure REF , we show an SGD optimizer (with momentum) and the same optimizer with SWA over a 3-dimensional loss surface with a saddle point. We observe that the original SGD reaches the desired minima, however, it almost reaches the saddle point and does a course correction and reaches minima. On the other hand, we observe that SGD with ASWA is very conservative, it repeatedly restarts and reaches the minima without reaching the saddle point. We empirically observe that this is a desired property for the stability of models over runs of the same model that differ only over random instantiations. The grey circles in Figure REF  highlight this conservative behaviour of SGD with ASWA optimizer, especially when compared to the standard SGD.
Further,  [1]} show that for convex losses, averaging SGD proposals achieves the highest possible rate of convergence for a variety of first-order SGD based algorithms.
<FIGURE>