Recent works [1]} on the imbalanced classification problem proposed to separate representation learning and classifier learning. Intuitively, the final classifier layer is negatively affected by highly imbalanced data distributions while the convolutional neural network backbone is not as affected. In other words, representation learning benefits from all the data regardless of their class membership. It has been shown in [1]} that weighted losses do not produce large improvement because they can negatively affect representation learning. While we need to be careful when introducing weighted losses to train the feature extractor, some previous works [3]}, [4]} showed that it can still be beneficial to carefully "fine-tune" CNNs with balancing techniques towards the end of a training cycle when the learning rate has been annealed. We propose to use recall loss as a feature learning loss to replace the standard cross entropy. We experimentally show that recall loss is a better suited loss for representation learning because it considers imbalance in datasets while dynamically adjusting the weights to not bias towards any class. To apply recall loss to classification, we introduce a Simple Decoupled Network (SDN) to decouple representation and classifier learning (fig. REF ).
