Accuracy vs Precision vs Recall vs F1 vs Perplexity

Accuracy

• Simply: Out of all predictions we made, what proportion were true?
• Formula: $\text{Accuracy}=\frac{\text{True Positives}+\text{True Negatives}}{\text{Total Instances}}$
• Drawbacks: Misleading when data is imbalanced

Precision

• Simply: Out of all the times we said “positive”, how many of those were true?
• Formula: $\text{Precision}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Positives}}$
• Intuition: Focuses on how many times you cried wolf (positive) and it actually was. High precision can lead to fewer false alarms.

R ecall

• Simply: Out of all of the data points that are actually positive, how many of them did we predict to be positive.
• Formula: $\text{Recall}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Negatives}}$
• Intuition: On all of the time there was a wolf, it measures how many you correctly cried. Higher recall means you missed the wolf less.

F1 Score

• Combines precision and recall:
• Formula: $F1=2\cdot \frac{\text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}$
• If either Precision or Recall is very low, then F1 will be low.

Perplexity

• Simply: How confused is the model for a given word? Lower means better.
• Formula: $\text{Perplexity}=P(W{)}^{-\frac{1}{N}}=2^{H(W)}\text{or}{e}^{H(W)}$
  where $H(W)$ is the Cross-Entropy Loss of the language model.
• Intuition: If a model assigns high probability to correct sequences, it has low perplexity (less confused). If it spreads probability evenly across words, it has high perplexity (very unsure).
• Example: A model with perplexity = 10 is as uncertain as if it had 10 choices per word on average.