Logistic Regression

Naive Bayes classifies a document indirectly. To decide whether a review is positive or negative it estimates, for each class, the likelihood $P(d\mid c)$ of the document under that class, then multiplies by a prior and picks the class with the larger product. It models the process that generates the words. Logistic regression estimates the target quantity, $P(c\mid d)$, directly. It never models how documents are generated; it learns a boundary that separates the classes and reads off the probability of the class from which side of the boundary a document lands on.¹

That split — model the data, or model the boundary — has standard names.

In practice, the discriminative choice means logistic regression handles correlated features well. Naive Bayes assumes the features are conditionally independent given the class; add the same feature twice and it double-counts the evidence. Logistic regression, given two perfectly correlated features, simply splits the weight between them and reaches the same, calibrated answer. That robustness is why it usually wins on larger documents and larger feature sets — though naive Bayes, with no optimization step to run, remains fast, easy, and competitive on very small or very short data.²

Logistic regression is a supervised classifier: it trains on $m$ labelled pairs $({\mathbf{x}}^{(i)},y^{(i)})$, where each ${\mathbf{x}}^{(i)}$ is a feature vector and $y^{(i)}\in \{0,1\}$ is the gold label. It has four components, covered below in order:³

From a weighted sum to a probability

Represent an observation as a feature vector $\mathbf{x}=[x_1,\dots ,x_n]$. Logistic regression learns a weight $w_i$ for each feature and a single bias$b$ (also called the intercept). A weight records how strongly its feature indicates the positive class: a large positive $w_i$ pushes toward $y=1$, a large negative $w_i$ toward $y=0$. To score a test instance the classifier multiplies each feature by its weight, sums, and adds the bias, producing a single number $z$ — the weighted evidence for the class:

The dot product $\mathbf{w}\cdot \mathbf{x}$ is just that sum of products, written compactly. Nothing constrains $z$ to be a probability: the weights are real-valued, so $z$ ranges over all of $(-\infty ,\infty )$. We need to squash it into $[0,1]$. The sigmoid (or logistic) function does exactly that:⁴

The sigmoid is a good squashing function for three reasons. It is nearly linear around $z=0$, so it preserves small differences in the evidence there; it flattens toward $0$ and $1$ at the extremes, so a wildly confident outlier cannot dominate; and it is smooth and differentiable, which is what makes learning by gradient descent possible at all. Applying it to the weighted sum gives the probability of the positive class, and the negative class takes the rest:

The two probabilities sum to one by construction, so the model is a valid distribution over the two classes. A convenient identity, $1-\sigma (z)=\sigma (-z)$, lets us write $P(y=0\mid \mathbf{x})=\sigma (-(\mathbf{w}\cdot \mathbf{x}+b))$ as well.

The decision boundary

To turn a probability into a class we threshold at $0.5$: predict the positive class when it is the more probable one. The value $0.5$ is the decision boundary.

Because $\sigma (z)>0.5$ exactly when $z>0$, the boundary in feature space is the set of points where the weighted sum is zero, $\mathbf{w}\cdot \mathbf{x}+b=0$. That is a hyperplane — a flat surface, a line in two dimensions. Logistic regression is a linear classifier: it can only separate classes a straight boundary can separate. The weight vector $\mathbf{w}$ is the normal to that hyperplane, and $\mathbf{w}\cdot \mathbf{x}+b$ measures signed distance from it, which the sigmoid converts to a confidence.

A worked sentiment example

For example, take binary sentiment classification of a movie review. Following Jurafsky & Martin, represent a review by six features, hand-designed from linguistic intuition:⁵

Suppose learning has already produced the weights $\mathbf{w}=[2.5,-5.0,-1.2,0.5,2.0,0.7]$ and bias $b=0.1$. The weight $w_1=2.5$ is positive: positive- lexicon words are evidence for a positive review. The weight $w_2=-5.0$ is negative and about twice as large in magnitude: negative words are evidence against, and count for more. Plug the feature vector in:

The model calls it positive, with $0.70$ confidence. Nothing about logistic regression restricts the features to sentiment: a feature can be any real-valued property of the input. Period disambiguation (is a . the end of a sentence or part of an abbreviation?) uses features like is the current word lowercase? or is it in an abbreviation dictionary?; a feature can even combine primitives — a feature interaction such as period follows St. and the previous word is capitalized. For tasks with large vocabularies, feature templates generate one feature per observed n-gram automatically. The design of good features — by error analysis and linguistic intuition — was, before representation learning, most of the applied work; the next chapter's embeddings are the move to learning features instead of designing them.

The cross-entropy loss

We have a classifier but no way to learn its weights. Learning needs an objective: a number that measures how far the prediction $\hat{y}=\sigma (\mathbf{w}\cdot \mathbf{x}+b)$ is from the gold label $y$, small when the two agree and large when they diverge. The right objective falls out of a single principle, conditional maximum likelihood: choose the weights that make the true labels in the training data as probable as possible.⁶

Consider one example with label $y\in \{0,1\}$. Since there are exactly two outcomes, the model's probability of the observed label is a Bernoulli, which a single expression captures:

The exponents select the right case. When $y=1$ this reads ${\hat{y}}^1(1-\hat{y}{)}^0=\hat{y}$; when $y=0$ it reads ${\hat{y}}^0(1-\hat{y}{)}^1=1-\hat{y}$. Either way it is the probability the model assigned to the correct answer. Taking the log — which is monotonic, so it does not move the maximizer — turns the product into a sum:

This is a log likelihood we want to maximize. To make it a loss we want to minimize, flip the sign. The result is the cross-entropy loss:

Substituting $\hat{y}=\sigma (\mathbf{w}\cdot \mathbf{x}+b)$ gives the loss in terms of the weights:

Why is this the right objective? A perfect classifier assigns probability $1$ to the correct outcome, and $-\log 1=0$: no loss. The more probability it wastes on the wrong answer, the larger $-\log (\text{correct prob})$ grows, running to infinity as the correct probability approaches zero. On the worked example, if the gold label is positive ($y=1$) and the model said $\hat{y}=0.70$, the loss is $-\log 0.70=0.36$ — small, because the model was mostly right. If the same review were actually negative ($y=0$), the model put $0.70$ on the wrong class and the loss is $-\log (0.30)=1.2$ — larger, because it was confidently wrong. The loss is small when the model is right and large for confident mistakes.

For logistic regression this loss has a property neural networks lose: it is convex. A convex function has a single minimum and no spurious local minima, so gradient descent from any starting point is guaranteed to find the global optimum. The loss surfaces of deep networks are non-convex — riddled with the saddle points and local minima that the deep-learning optimization chapter is about — but for one linear layer, the geometry is benign.

Learning by gradient descent

The goal is the weights that minimize the average loss over the training set. Write the parameters as $\theta =(\mathbf{w},b)$; then

Gradient descent finds that minimum by repeatedly stepping downhill. At the current parameters it computes the direction in which the loss rises most steeply — the gradient — and moves the opposite way. Picture the loss over a single weight $w$ as a curve: at a point where the slope is negative, the minimum lies to the right, so we increase $w$; where the slope is positive, we decrease it. The slope's sign tells us the direction; its magnitude, scaled by a step size, tells us how far.⁷

The gradient of the loss

In $n$ dimensions the gradient ${\nabla }_{\theta }L$ is the vector of partial derivatives, one per weight (and one for the bias), each recording how much a small change in that parameter changes the loss. For the cross-entropy loss the partial derivative with respect to weight $w_j$ has a simple form:⁸

The gradient for feature $j$ is the prediction error$\hat{y}-y$ — how far the estimate overshot the truth — multiplied by the feature value $x_j$. If the model is exactly right, $\hat{y}=y$, the error is zero and the weight does not move. If it overshoots, the error is positive and the update shrinks the weights on the active features; if it undershoots, the reverse. The size of the correction scales with how active the feature was. The derivative of a sigmoid composed with a log collapses to this simple form because the sigmoid and cross-entropy are chosen together.

The collapse is worth deriving, because it is the same chain-rule pattern that scales up to backpropagation. Write $z=\mathbf{w}\cdot \mathbf{x}+b$ so $\hat{y}=\sigma (z)$, and use two facts. First, the sigmoid has an unusually clean derivative,

which follows from the quotient rule on $1/(1+e^{-z})$. Second, $\partial z/\partial w_j=x_j$, since $z$ is linear in each weight. Differentiate the loss $L_{CE}=-[y\log \hat{y}+(1-y)\log (1-\hat{y})]$ with respect to $\hat{y}$:

Now chain the three factors, $\frac{\partial L_{CE}}{\partial w_j}=\frac{\partial L_{CE}}{\partial \hat{y}}\cdot \frac{d\hat{y}}{dz}\cdot \frac{\partial z}{\partial w_j}$, and the $\hat{y}(1-\hat{y})$ from the sigmoid derivative cancels the identical factor in the denominator of $\partial L_{CE}/\partial \hat{y}$:

The cancellation is the whole point of pairing the sigmoid with cross-entropy: any other loss would leave a $\hat{y}(1-\hat{y})$ factor in the gradient, which vanishes when the model is confidently wrong ($\hat{y}$ near $0$ or $1$) and stalls learning exactly when it is most needed. Cross-entropy removes that factor, so the update is driven by the raw error even at the extremes.

The update rule and stochastic gradient descent

Each step moves the parameters against the gradient, scaled by the learning rate$\eta$ — a hyperparameter setting the step size:

Too large an $\eta$ overshoots the minimum and oscillates; too small an $\eta$ converges very slowly. Stochastic gradient descent (SGD) is the online version: it computes the gradient on one training example at a time and updates immediately, rather than waiting to average over the whole set. It processes the data example-by-example, correcting the weights after each.⁹

Termination is a judgment call: stop when the loss stops falling, when the gradient norm drops below a tolerance, or when the loss on a held-out set starts rising (a sign of overfitting). It is common to start $\eta$ high and decay it over iterations, writing ${\eta }_k$ for its value at step $k$, so early steps move fast and later ones settle.

A single step, worked out. Take one positive example ($y=1$) with two features, $x_1=3$ and $x_2=2$, and start from $\mathbf{w}=0$, $b=0$, $\eta =0.1$. With all weights zero, $\mathbf{w}\cdot \mathbf{x}+b=0$ and $\sigma (0)=0.5$, so the prediction error is $\hat{y}-y=0.5-1=-0.5$. The gradient components are $(\hat{y}-y)x_j$:

After one step the weights have moved off zero in the direction that raises the model's probability on this positive example: $w_1=0.15$, $w_2=0.10$, $b=0.05$.

Iterate the same update on the same example and it converges. Each step recomputes $z$, hence $\hat{y}$, hence the error; because this is one positive example with no opposing data, $\hat{y}$ climbs toward $1$ and the loss falls. The gradient magnitude shrinks with the error, so the steps get smaller as the fit improves — after ten steps $\hat{y}\approx 0.87$. The table traces the first three:

On a real training set the picture is the same but noisier: each example pulls the weights in its own direction, and the loss falls on average rather than monotonically, which is why SGD's stopping rule watches a held-out set instead of demanding the training loss reach zero.

Minibatch training is the middle ground between SGD's one-example updates and full-batch descent over the entire set. It averages the gradient over a group of $m$ examples — a few hundred to a thousand — cutting the noise of single-example steps while staying cheap, and the batch vectorizes cleanly onto parallel hardware. The minibatch gradient is just the average of the per-example gradients:

Regularization

A model that fits the training data too well fits its noise. If some feature happens to occur only in positive examples in the training set — a fluke of a small sample — logistic regression will assign it a huge weight to match those cases exactly, and that weight will mislead on unseen data. This is overfitting: excellent training accuracy, poor generalization.¹⁰

The fix is to add a regularization term $R(\theta )$ that penalizes large weights, so the objective trades off fitting the data against keeping the weights small:

The hyperparameter $\alpha$ sets how hard to push. A weight setting that fits the data with many small weights is now preferred over one that fits marginally better using a few enormous ones. The two standard choices differ in the norm they penalize:

Both have Bayesian readings as priors on the weights: L2 corresponds to a Gaussian prior centered at zero (weights are pulled toward zero), L1 to a Laplace prior. The choice is L2 when you want smooth shrinkage, L1 when you want feature selection built into the fit.

More than two classes: the softmax

Many tasks have more than two classes — 3-way sentiment (positive, negative, neutral), part-of-speech tags, named-entity types. Multinomial logistic regression (historically, the maxent classifier) generalizes the binary model to $K$ classes. Where the sigmoid squashed one number into $[0,1]$, the softmax squashes a vector of $K$ numbers into a probability distribution over the $K$ classes:¹¹

The numerator is exponential, so it is always positive; the denominator sums the exponentials of every class, so the outputs sum to one — a valid distribution. Like the sigmoid, the softmax sharpens differences: the largest component of $z$ is pushed toward $1$ and the rest are suppressed. Now each class $c$ has its own weight vector ${\mathbf{w}}_c$ and bias $b_c$, and the probability of class $c$ is its exponentiated score, normalized:

The loss generalizes just as cleanly. With $K$ classes the label $y$ is a one-hot vector — a $1$ in the position of the true class $k$, zeros elsewhere. The cross-entropy loss sums over the classes, but every term vanishes except the one for the true class, so it reduces to the negative log probability of the correct class:

This is the negative log likelihood loss, and its gradient with respect to the weight $w_k$ for class $k$ echoes the binary case — the difference between the true indicator for class $k$ and the model's probability for it, scaled by the input:

Classification in the transformer era

Everything above learns weights over features a human specified — counts of positive words, a no-present bit, the log document length. That design step was, for decades, most of the applied work in text classification, and it is the step modern models do away with. The output layer, though, does not change.

A transformer classifier keeps the softmax-over-logits head derived here and swaps the hand-built feature vector $\mathbf{x}$ for a learned representation of the text. The standard approach is to take a model pretrained on unlabeled text — BERT (Devlin et al., 2019, NAACL) is the canonical one¹² — run the document through it to get a contextual vector (conventionally the vector above a special [CLS] token), and put a single logistic/softmax layer on top. Fine-tuning trains that top layer, and usually the whole network, on the labeled classification data with the same cross-entropy loss and the same gradient descent this lesson built. The classifier at the top is still $\text{softmax}(\mathbf{Wh}+\mathbf{b})$; what changed is that $\mathbf{h}$ is now learned from the text rather than assembled from lexicon counts.

Two consequences follow. First, the discriminative-versus- generative distinction that opened the lesson still applies: a fine-tuned encoder with a softmax head is still a discriminative model of $P(c\mid d)$, now with a learned rather than designed feature map. Second, the very largest models can classify with no fine-tuning at all, by prompting — describe the labels in words and read off the predicted class from the model's next-token distribution (Brown et al., 2020, NeurIPS, showed this few-shot behavior).¹³ That removes even the labeled training set for the top layer, but when labeled data is available, a fine-tuned encoder with the classification head derived here remains the accurate, cheap choice for a fixed task. Logistic regression remains in use as the last layer of these models.

The bridge to neural networks

Look again at the two-class model: it takes an input vector, forms one weighted sum $\mathbf{w}\cdot \mathbf{x}+b$, and passes it through a nonlinearity $\sigma$. That is precisely a single artificial neuron — one unit, one activation. Logistic regression is a one-layer neural network, and the multinomial version, with its softmax over $K$ scores, is the output layer of a classification network. Everything assembled here reappears at scale: the softmax and cross-entropy loss are the standard output and objective of the classifiers in the deep-learning course, and the same gradient-descent machinery trains them.

Stack more layers of these units, each learning its own features from the layer below instead of relying on features a human designed, and logistic regression grows into a neural language model. The pieces do not change — a weighted sum, a nonlinearity, a cross-entropy loss, a gradient step — only their number.