Cross-Entropy Loss

It’s annoying cos it’s actually so deceptively simple.

What:

In a Neural Network, you need a loss function that quantifies just how terribly your model did. A really common way for Classification problems is Cross-Entropy Loss. (Basically synonymous with Negative Log Likelihood).

How?

1. You take an example from every class $i$. You calculate the predicted confidence of it being that class ($\widehat{y_i}$).
2. You plug that into: ${\text{loss}}_i=y_i\log (\widehat{y_i})$. (where $y_i$ is the true label).
3. You sum all of the losses together (and then invert the sign).
4. That’s your total Cross Entropy Loss.

Mathematically:

$$L=-\sum _{i=1}^N{y}_i\log \widehat{y_i}$$

where:

• $\widehat{y_i}$ is the predicted probability of class $i$ (often from a softmax function in classification problems).
• $y_i$ is the actual (true) label, typically represented as a one-hot encoded value (i.e., $1$ if the sample belongs to class $i$, otherwise $0$).
• $N$ is the total number of samples in the dataset.
• The summation ${\sum }_{i=1}^N$ iterates over all samples in the dataset.
• The logarithm $\log$ ensures that incorrect predictions result in higher penalty values.
• The negative sign ensures the loss is minimised when predicted probabilities align closely with the true labels.

Why Cross Entropy and Not Sum of Squared Errors?

If we have an absolutely awful prediction, the model gets reallllyyyyy punished for it under CE. Not necessarily true for SSE:

Cross-Entropy with One-Hot Encoding:

Whenever you’re using 1 hot encodings, the summation symbol disappears. Your new loss for class $i$ has just become:

• $\text{loss}=y_i\log \widehat{y_i}$

Wait! Where'd the summation go?

When you use One-Hot Encoding as your classes in classification, the summation symbol disappears. This is because for each class’s prediction, the other classes have a $y_i$ of 0. Then, their corresponding $\log$ just turns to 0. Neat isn’t it

Binary Cross Entropy:

The original formula is the general case. But what if we just have two classes (e.g. rain / not rain )?

1. General Formula: $L=-\sum y_i\log ({\hat{y}}_i)$
2. Expanding out with 2 classes: $L=-\left(y_0\log ({\hat{y}}_0)+y_1\log ({\hat{y}}_1)\right)$
3. Subbing in for accurate $y$: $y_0=y,y_1=1-y$
4. Final BCE formula: $L=-\left(y\log (\hat{y})+(1-y)\log (1-\hat{y})\right)$
5. Note: When you’re actually filling this out in practise, you’re only ever filling out one of the sides of the formula. The other one cancels out.