In this paper, we focus on the stability of deep neural models as a function of random-seeds. Our proposal is based on SWA, but we extend it to the extremes and call it Aggressive Stochastic Weight Averaging. We assume that, for small batch size, the loss surface is locally convex. We further relax the conditions for the optimizer and assume that the optimizer is based on some version of gradient descent — this means that our modification is valid even for other pseudo-first-order optimization algorithms including Adam [1]} and Adagrad [2]}.
