This paper has introduced a novel adaptive output layer activation function, which we have named ASTra, that can either in combination with, or separately from, our previously proposed GMN loss [1]} work to facilitate the classification of minority examples for extremely high IRs. We additionally introduced a novel use of metrics based on an approximated confusion matrix for performance monitoring during training.
The proposed methods were applied to the most imbalanced datasets in a recent extensive study of ensemble classifiers [2]} and achieved performances comparable to those reported in [2]} for RUSBoost [4]}, despite our restricting this initial investigation to the use of neural networks with 3–12 neurons in a single hidden layer. We emphasise, however, that for complex problems it is essential to explore the space of model architectures, since even in the case of a high IR the main difficulties may in fact lie in an insufficient model architecture. We also recommend a progressive exploration of the methods, beginning with the GMN loss alone, then advancing to the addition of the ASTra output layer activation function, as needed.
We aim next to apply our methods to challenging real-world imbalanced datasets requiring deep neural network architectures. Additionally, especially given the nature of the derivatives involved in ASTra computations, we are interested in the use of sharpness-aware minimisation [5]} to smooth optima and improve generalisation. Finally, we note here that it is our intention to open-source our code and to ensure its integration with popular machine learning libraries (Pytorch and Tensorflow).
