A striking aspect of cross-entropy is that in order to achieve zero loss we need to have \(y_if({\mathbf {x}}_i) = \infty \) .
Thus, interpolation only occurs at infinity and any optimization procedure would eventually escape from a ball of any fixed radius. This presents difficulties for optimization analysis as it is typically harder to apply at infinity. Furthermore, since the norm of the solution vector is infinite,
there can be no transition to linearity on any domain that includes the whole optimization path, no matter how wide our network is and how tightly we control the Hessian norm (see Section REF ). Finally, analyses of cross-entropy in the linear case [1]} suggest that convergence is much slower than for the square loss and thus we are unlikely to approach interpolation in practice.
