Owing to the non-convex nature of most loss functions, coupled with poorly understood learning dynamics, optimization of Deep Neural Networks (DNNs)
is a challenging task. Recent work can be broadly divided in two categories: (1) adaptive first order variants of SGD, such as ADAMw
[1]}, RMSprop [2]} and Adam [3]},
and (2) second order methods such as K-FAC [4]}.
These methods adapt the optimization process per parameter and obtain improved convergence and
generalization. The objective function for these frameworks is to minimize finite-sum problems:
\(\mathcal {L}(\theta ) = \frac{1}{N} \sum _i^{N} L^i(\theta )\) ,
where \(L^i(\theta )\)  denotes the loss on a single training data point.
Standard optimization frameworks assume that the \(N\)  loss functions come from the same distribution,
and share characteristics such as the curvature, Lipshitz constant, etc.
However, in practice, this is not true. Figure REF  highlights
the difference in loss function characteristics over the course of training for classes present in CIFAR100.
