No loss is perfect and improves performance on all the datasets. There are two major scenarios when recall loss does not improve performance: 1) when a dataset is very "difficult", there are compounding factors resulting in low performance and imbalance is not the most limiting factor among them; 2) when a dataset is too "easy", the class distribution may be imbalanced but visual features are distinctive and easy to classify. For example, it is more difficult to achieve good performance on indoor semantic segmentation due to clustered environments, different orientations of objects and intra-class similarities. We tested the same DeepLab Resnet18 network on the ADE20K datasets [1]}, and no loss can perform better than the others and performance is not good across all classes. We also tested the various losses on another self-driving dataset, virtual KITTI2 [2]}. The dataset has much simpler scenes and less busy streets than Synthia. Cross-Entropy loss alone achieves over \(90\%\)  mean accuracy. Therefore, other losses including the recall loss perform similarly to the vanilla Cross-Entropy. A rule of thumb for using an imbalance-oriented loss function is when the overall accuracy is reasonably high but mean accuracy is very low. This is a very good indicator that imbalance is the most limiting factor and needs to be carefully approached.
