Logistic Regression

What?

TL;DR: It’s a giant learned weighted sum (the weights are learned). Comes in single-class and multi-class variants. The result is used for assigning probability for classification classes.

As opposed to Linear Regression - which predicts real values - Logistic Regression is all about predicting binary response variables. Often they predict Odds and Log Odds.

The Core Idea:

👉 Logistic regression doesn’t actually “regress” like linear regression. Instead, it models $P(y|x)$; essentially:

Given some input features $x$, what is the probability of class $y$

• Essentially, each feature $f_i(x)$ contributes a “vote” for different classes.
• The model adds up the weighted votes using the dot product (the part in the main brackets).
• Then, the Softmax Function (the $\frac{exp(\text{everything else})}{Z}$) ensures that the votes get converted into probabilities
• The part behind the highest probability wins.

Mathematically, Single Class:

The core idea is to model the probability that an input $x$ belongs to the positive class (class 1) using the sigmoid function:

$$P(y=1\mid x)=\sigma (w\cdot x+b)=\frac{1}{1+e^{-(w\cdot x+b)}}$$

Where:

• $x$ = input feature vector (e.g., size of house, age, etc.)
• $w$ = weight vector (learned parameters)
• $b$ = bias (intercept)
• $\sigma$ = sigmoid function
• $P(y=1\mid x)$ = probability the output is class 1 (positive)

You then classify:

$$\hat{y}=\{\begin{array}{ll}1& \text{if }P(y=1\mid x)\ge 0.5\\ 0& \text{otherwise}\end{array}$$

Funnily: This is actually one of the bases for a neuron of a neural network (and also a straight line when you simplify it)

Think 🤔: What would happen if you upped every single weight by 1? Well relative to each other, they’d still all be in the same place. But the actual probability of each would be skewed. Duh…

Mathematically, Multi-class:

Choose the class that has highest probability according to

$P(y=k|x)=\frac{1}{Z}\exp \left({\sum }_i{w}_i^{(k)}{f}_i(x)\right)$

where the normalisation constant

$Z={\sum }_{k^{\prime }}\exp \left({\sum }_i{w}_i^{(k^{\prime })}{f}_i(x)\right)$

• Inside brackets is just a dot product: $s^{(k)}=w^{(k)}\cdot f(x)$. Essentially, this is the weight each feature brings to the class
• $Z$ does not depend on $k$ .
• So, we will end up choosing class $k$ for which $s^{(k)}$ is highest.
• Softmax function: exponentiation of scores $s$, followed by normalisation to turn into a distribution.

Note: There’s no bias here. It’s removed for simplicity in derivations, but often there in practise.

Predicting By Hand:

Take the following example:

• ${\hat{\beta }}_0=1.0$
• ${\hat{\beta }}_1=0.5$
• ${\hat{\beta }}_2=-0.5$
• ${\hat{\beta }}_3=0.1$
• $x^{(1)}$ Number of occurrences of phrase “world-beating”
• $x^{(2)}$ Number of occurrences of phrase “confidence interval”
• $x^{(3)}$ Number of occurrences of phrase “bootstrap”
• $y$ Whether the paper was rejected (1) or sent out for review (0).
• Question: Suppose a paper contains the phrase “world-beating” 5 times, and 0 occurrences of “confidence interval” or “bootstrap”. What is the predicted probability of rejection?

Solution:

To calculate the predicted probability of rejection for a paper with 5 occurrences of “world-beating” and 0 occurrences of “confidence interval” and “bootstrap”, we use the logistic regression coefficients and the following formula:

Log-odds = ${\hat{\beta }}_0+{\hat{\beta }}_1\cdot x_1+{\hat{\beta }}_2\cdot x_2+{\hat{\beta }}_3\cdot x_3$

From there, we sigmoid it to get probability.

Here, $x_1$ is the number of occurrences of “world-beating”, $x_2$ is the number of occurrences of “confidence interval”, and $x_3$ is the number of occurrences of “bootstrap”. Given the coefficients ${\hat{\beta }}_0=1.0$, ${\hat{\beta }}_1=0.5$, ${\hat{\beta }}_2=-0.5$, and ${\hat{\beta }}_3=-0.1$, and the occurrences $x_1=5$, $x_2=0$, and $x_3=0$, the log-odds are:

Log-odds = $1.0+(0.5\cdot 5)+(-0.5\cdot 0)+(-0.1\cdot 0)$
Log-odds = $3.5$

To convert the log-odds to a probability, we use the logistic function:

$P(\text{rejection})=\frac{e^{\text{Log-odds}}}{1+e^{\text{Log-odds}}}$

Plugging in the log-odds we calculated:

$P(\text{rejection})=\frac{e^{3.5}}{1+e^{3.5}}$

Training Logistic Regression:

Like any Machine Learning Model:

1. Iterate over a subset of your data
2. Predict the classes / output of that model.
3. Use a loss function (BCE in this case) to see how incorrect you were:
4. Calculate which direction you should take to get more correct.
5. Take steps in the correct direction.

4. Calculate Which Direction You Should Take:

You calculate the direction of the steepest loss decrease. To do this, we’ll use Gradient Descent! We’ll use the following values:

• Gradient of loss function w.r.t weights is:
  $\frac{\partial L}{\partial w_i}=(\hat{y}-y)x_i$
• Gradient of loss w.r.t. the bias:
  $\frac{\partial L}{\partial b}=(\hat{y}-y)$