The goal in this section was to study the accuracy of our proposed AUM loss function in unbalanced binary classification problems.
Our experiment used the zip.train and zip.test image classification datahttps://web.stanford.edu/~hastie/ElemStatLearn/ from [1]}.
Each input is a \(16\times 16\)  image which is represented by a vector \(x\in \mathbb {[}-1,1]^{256}\) .
There are 10 classes (one for each digit), but we only used two classes (0/1) in order to study binary classification.
For both the zip.train and zip.test files, we discarded some \(y_i=0\)  labels in order to obtain sets with equal numbers of positive and negative labels (total 2010 in train, 528 in test).
Then we generated 10 different train sets by randomly selecting 1000 examples with a class balance in \(\lbrace 1\%, 5\%, 50\%\rbrace \) .
The goal is to learn from the unbalanced train set data and then provide accurate predictions (as measured by AUC) on the balanced test set.
The learning algorithms that we considered in our comparison were all based on gradient descent with constant step size and early stopping regularization.
The step size and number of iterations hyper-parameters were chosen using grid search on a held-out validation set.
Since AUM is a full gradient method (not stochastic), we compared with several other loss functions using the full gradient method.
