We further present results on the Cityscapes segmentation dataset [1]} with real images. Images in the dataset are real camera images and pixel-wise labels are manually assigned. Therefore, there exists more data noise as compared to the synthetic Synthia dataset. We show mean accuracy and mean IoU in Fig. REF  for ResNet18 and Reset101 respectively. Weighted CE and Balanced CE still yield the highest accuracy but result in significantly worse IoU than baseline Cross-Entropy. Surprisingly, SoftIoU also suffers in the presence of data noise as its IoU does not match those of the Cross-Entropy loss mostly because of lower performance on minority classes. Compared to performance on Synthia, this shows that some of the losses are not robust to label noise in a real dataset. Recall loss again outperforms other losses by improving mean accuracy and maintaining a good mean IOU.
We further provide per-class performance on selected losses in Table REF  using DeepLab-ResNet18. Recall loss improves mean accuracy from \(67.63\%\)  to \(75.06\%\)  with minimum IoU drop on minority classes compared to Cross-Entropy. In other words, recall loss improves the detection rate of small classes such as pedestrians and light poles, while maintaining a good precision. This demonstrates the effectiveness of recall loss on both synthetic and real outdoor segmentation datasets. No loss is perfect and improves performance on all the datasets. We provide further discussion on limitations in Appendix REF .
