Suppose I build a NN for classification. The last layer is a Dense layer with softmax activation. I have five different classes to classify. Suppose for a single training example, the true label is [1 0 0 0 0] while the predictions be [0.1 0.5 0.1 0.1 0.2]. How would I calculate the cross entropy loss for this example?
Cross-entropy loss explanation
machine learning;deep learning
Cross entropy formula given two distributions over discrete variable $x$, where $q(x)$ is the estimate for true distribution $p(x)$ is given by$$H(p,q) = -\sum_{\forall x} p(x) log(q(x))$$For a neural network, the calculation is independent of these parts:What kind of layer was used.What kind of activation you use - although many activations will not be compatible with the calculation, because it will produce nonsense values if the sum of probabilities is not equal to 1. Softmax is often used for multiclass classification because it guarantees a well-behaved probability distribution function.For a neural network, you will usually see the equation written into a form where $\mathbf{y}$ is the ground truth vector and $\mathbf{\hat{y}}$ (or some other value taken direct from the last layer output) is the estimate, and it would look like this for a single example:$$L = - \mathbf{y} \cdot log(\mathbf{\hat{y}})$$Where $\cdot$ is vector dot product.Your example ground truth $\mathbf{y}$ gives all probability to the first value, and the other values are zero, so we can ignore them, and just use the matching term from your estimates $\mathbf{\hat{y}}$$L = -(1\times log(0.1) + 0 \times log(0.5) + ...)$$L = - log(0.1) \approx 2.303$An important point from commentsThat means, the loss would be same no matter if the predictions are $[0.1, 0.5, 0.1, 0.1, 0.2]$ or $[0.1, 0.6, 0.1, 0.1, 0.1]$? Yes, this is a key feature of multiclass logloss, it rewards/penalises probabilities of correct classes only. The value is independent of how the remaining probability is split between incorrect classes.More commonly you will see this equation averaged over all examples:$$L = - \frac{1}{N}(\sum_{i=1}^{N} \mathbf{y_i} \cdot log(\mathbf{\hat{y}_i}))$$Many implementations will require your ground truth values to be one-hot encoded (with a single true class), because that allows for some extra optimisation. However, in principle the cross entropy loss can be calculated - and optimised - when this is not the case.