Logistic Regression

What?

As opposed to Linear Regression - which predicts real values - Logistic Regression is all about predicting binary response variables. Often they predict Odds and Log Odds.

The Core Idea:

👉 Logistic regression doesn’t actually “regress” like linear regression. Instead, it models $P(y|x)$; essentially:

Given some input features $x$, what is the probability of class $y$

• Essentially, each feature $f_i(x)$ contributes a “vote” for different classes.
• The model adds up the weighted votes using the dot product (the part in the main brackets).
• Then, the Softmax Function (the $\frac{exp(\text{everything else})}{Z}$) ensures that the votes get converted into probabilities
• The part behind the highest probability wins.

Mathematically:

Choose the class that has highest probability according to

$P(y=k|x)=\frac{1}{Z}\exp \left({\sum }_i{w}_i^{(k)}{f}_i(x)\right)$

where the normalisation constant

$Z={\sum }_{k^{\prime }}\exp \left({\sum }_i{w}_i^{(k^{\prime })}{f}_i(x)\right)$

• Inside brackets is just a dot product: $s^{(k)}=w^{(k)}\cdot f(x)$. Essentially, this is the weight each feature brings to the class
• $Z$ does not depend on $k$ .
• So, we will end up choosing class $k$ for which $s^{(k)}$ is highest.
• Softmax function: exponentiation of scores $s$, followed by normalisation to turn into a distribution.

Predicting By Hand:

Take the following example:

• ${\hat{\beta }}_0=1.0$
• ${\hat{\beta }}_1=0.5$
• ${\hat{\beta }}_2=-0.5$
• ${\hat{\beta }}_3=0.1$
• $x^{(1)}$ Number of occurrences of phrase “world-beating”
• $x^{(2)}$ Number of occurrences of phrase “confidence interval”
• $x^{(3)}$ Number of occurrences of phrase “bootstrap”
• $y$ Whether the paper was rejected (1) or sent out for review (0).
• Question: Suppose a paper contains the phrase “world-beating” 5 times, and 0 occurrences of “confidence interval” or “bootstrap”. What is the predicted probability of rejection?

Solution:

To calculate the predicted probability of rejection for a paper with 5 occurrences of “world-beating” and 0 occurrences of “confidence interval” and “bootstrap”, we use the logistic regression coefficients and the following formula:

Log-odds = ${\hat{\beta }}_0+{\hat{\beta }}_1\cdot x_1+{\hat{\beta }}_2\cdot x_2+{\hat{\beta }}_3\cdot x_3$

Here, $x_1$ is the number of occurrences of “world-beating”, $x_2$ is the number of occurrences of “confidence interval”, and $x_3$ is the number of occurrences of “bootstrap”. Given the coefficients ${\hat{\beta }}_0=1.0$, ${\hat{\beta }}_1=0.5$, ${\hat{\beta }}_2=-0.5$, and ${\hat{\beta }}_3=-0.1$, and the occurrences $x_1=5$, $x_2=0$, and $x_3=0$, the log-odds are:

Log-odds = $1.0+(0.5\cdot 5)+(-0.5\cdot 0)+(-0.1\cdot 0)$
Log-odds = $3.5$

To convert the log-odds to a probability, we use the logistic function:

$P(\text{rejection})=\frac{e^{\text{Log-odds}}}{1+e^{\text{Log-odds}}}$

Plugging in the log-odds we calculated:

$P(\text{rejection})=\frac{e^{3.5}}{1+e^{3.5}}$

Training Logistic Regression:

Like any Machine Learning Model:

1. Iterate over a subset of your data.
2. Predict the classes / output of that model.
3. Use a loss function (BCE in this case) to see how incorrect you were:
4. Calculate which direction you should take to get more correct.
5. Take steps in the correct direction.

4. Calculate Which Direction You Should Take:

You calculate the direction of the steepest loss decrease. To do this, we’ll use Gradient Descent! We’ll use the following values:

• Gradient of loss function w.r.t weights is:
  $\frac{\partial L}{\partial w_i}=(\hat{y}-y)x_i$
• Gradient of loss w.r.t. the bias:
  $\frac{\partial L}{\partial b}=(\hat{y}-y)$