The accepted justification of the cross-entropy loss for classification is that it is a better “surrogate” for the 0-1 classification loss than the square loss (e.g., [1]}, Section 8.1.2). There is little theoretical analysis supporting this point of view.
To the contrary, very recent theoretical works [2]}, [3]}, [4]} prove that in certain over-parameterized regimes, training using the square loss for classification is at least as good or better than using other loss functions.
Furthermore, extensive empirical evaluations conducted in [5]} show that modern neural architectures trained with the square loss slightly outperform same architectures trained with the cross-entropy loss on the majority of tasks across several application domains including Natural Language Processing, Speech Recognition and Computer Vision.
