We note that, [1]} suggest using SWA usually after 'pre-'training the model (at least until \(75\%\)  convergence) and followed by sampling weights at different steps either using large constant or cyclical learning rates. While, SWA is well defined for convex losses [2]}, [1]} connect SWA to non-convex losses by suggesting that the loss surface is approximately convex after convergence.
In our setup, we investigate the utility of averaging weights over every iteration (an iteration consists of one batch of the gradient descent).
Algorithm REF  shows the implementation pseudo-code for SWA. We note that, unlike [1]}, we average our weights at each batch update and assign the ASWA parameters to the model at the end of each epoch. That is, we replace the model's weights for the next epoch with the averaged weights.
