PointWise Mutual Information

What:

Describes a more nuanced way of measuring how strongly two words are associated, based on their co-occurrence.

• Raw co-occurrence says that common words appear together a lot (e.g. “the” and “and”).
• PMI asks “How surprising is it that they appear together”.

PMI Formula:

$PMI(w_1,w_2)=\log \left(\frac{P(w_1,w_2)}{P(w_1)P(w_2)}\right)$

• If $w_1$ and $w_2$ occur together more often than you’d expect by chance $\to$ PMI is positive.
• If they co-occur less often than chance $\to$ PMI is negative.
• If they are independent $\to$ PMI = 0.