Stochastic weight averaging (SWA) [1]} works by averaging the weights of multiple points in the trajectory of gradient descent based optimizers. The algorithm typically uses modified learning rate schedules. SWA is itself based on the idea of maintaining a running average of weights in stochastic gradient descent based optimization techniques [2]}, [3]}. The principle idea in SWA is averaging the weights that are maximally distant helps stabilize the gradient descent based optimizer trajectory and improves generalization.  [1]} use the analysis of [5]} to illustrate the stability arguments where they show that, under certain convexity assumptions, SGD iterations can be visualized as sampling from a Gaussian distribution centred at the minima of the loss function. Samples from high-dimensional Gaussians are expected to be concentrated on the surface of the ellipse and not close to the mean. Averaging iterations is shown to stabilize the trajectory and further improve the width of the solutions to be closer to the mean.
