Naive Bayes and Sentiment Classification

A review says awful pizza and ridiculously overpriced; you want a machine to read it and answer negative. An email says online pharmaceutical, WITHOUT ANY COST, Dear Winner; you want spam. A research paper is about epidemiology, not embryology; you want the right subject label. These are all the same task: take one document and assign it a category from a fixed set.¹ This is text classification. Its simplest and oldest solution, naive Bayes, is still a strong baseline.

Formally, supervised classification takes an input document $d$ and a fixed set of classes $C=\{c_1,\dots ,c_k\}$ and returns a predicted class $\hat{c}\in C$. Supervised means we learn the mapping from a training set of $N$ documents each hand-labeled with a class, $(d_1,c_1),\dots ,(d_N,c_N)$. A probabilistic classifier goes further and reports $P(c\mid d)$ for every class, not just the winner — useful when a later stage wants to weigh the decision rather than commit to it early.²

We focus on sentiment analysis, the extraction of the positive or negative orientation a writer expresses toward something. In its binary form the words of a review are strong cues on their own: great, richly, awesome pull positive; pathetic, awful, ridiculously pull negative. Naive Bayes formalizes this intuition.

Naive Bayes is a generative classifier: it builds a model of how each class would generate a document, then selects the class that most plausibly produced the one in hand. Its counterpart, the discriminativelogistic regression of the next lesson, skips the generative story and learns directly which features separate the classes. Discriminative models are usually more accurate, but the generative one is faster to train, needs less data, and is the right place to start.

The bag-of-words representation

Before any probability, we must decide what a document is to the classifier. Naive Bayes represents a document as a bag of words: an unordered multiset of its words, position discarded, only frequency kept. I love this movie, it's sweet and sweet, it's this movie I love are the same bag. The word love counts the same whether it is the first word or the last.

This throws away real information — the movie was not good and the movie was good, not collapse to the same bag — yet the representation works well in practice. For topic and sentiment, the presence of the right words carries most of the signal, and discarding word order costs little. The features of naive Bayes are these word identities, written $w_1,w_2,\dots ,w_n$ for the words at each position in $d$.

The naive Bayes classifier

Naive Bayes returns the class with the highest posterior probability given the document:

We cannot estimate $P(c\mid d)$ directly — there are too many possible documents — so we turn it around with Bayes' rule, $P(x\mid y)=\frac{P(y\mid x)P(x)}{P(y)}$, applied to $c$ and $d$:

The denominator $P(d)$ is the same for every class, and we are choosing the arg max over classes of the same document $d$, so it cannot change which class wins. Drop it:

The winning class maximizes the product of two things: the prior $P(c)$, how common the class is, and the likelihood $P(d\mid c)$, how well the class explains the document. This is why naive Bayes is called generative — read $P(d\mid c)P(c)$ as a recipe for producing a document: sample a class from the prior, then sample the words from that class.

The naive assumption

The likelihood $P(d\mid c)=P(w_1,w_2,\dots ,w_n\mid c)$ is still intractable: estimating the probability of every possible combination of words would need more parameters and more data than any corpus supplies. Naive Bayes makes two simplifying assumptions to escape.

The first is the bag of words already introduced: position does not matter, so love contributes the same whether it is word 1 or word 20. The second, which gives the method its name, is the naive Bayes assumption — the words are conditionally independent given the class:

The assumption is false — fun and film are not independent, and New raises the odds of York — but pretending they are collapses a combinatorial explosion into a product of $n$ easy factors, and the resulting classifier works far better than the false premise suggests. Putting the pieces together, the class naive Bayes chooses is:

Log space

That product runs over every word position, so for a document of any length it multiplies dozens or hundreds of probabilities, each well below $1$. The result underflows to zero in floating point. The fix is to work in log space: because $\log$ is monotonic, the class that maximizes the product also maximizes the log of the product, and a log turns the product into a sum:

This is numerically safe and faster. It also exposes the shape of the model: the predicted class is a linear function of the input features (each contributes an additive log-weight). Classifiers that decide by a linear combination of their inputs — naive Bayes here, logistic regression next lesson — are called linear classifiers.

Training: counting with smoothing

Where do $P(c)$ and $P(w_i\mid c)$ come from? We estimate them from the training set by maximum likelihood — just the frequencies in the data.

For the prior, ask what fraction of the training documents belong to class $c$. With $N_c$ the number of documents in class $c$ and $N_{\text{doc}}$ the total:

For the likelihood $P(w_i\mid c)$, treat a document's words as a bag drawn from class $c$. Concatenate every document of class $c$ into one large text, and estimate the probability of word $w_i$ as its fraction among all word tokens in that class:

Here $V$ is the vocabulary — the union of all word types across all classes, not just the words seen in class $c$. That detail matters for the denominator below.

The zero-probability problem

Maximum likelihood fails on unseen words. Suppose the word fantastic never appears in any positive training document (perhaps it only showed up, sarcastically, in a negative review). Then

Because naive Bayes multiplies every word's likelihood, a single zero factor drags the whole product to zero: a document containing fantastic can never be classified positive, no matter how much other evidence points that way. A single unseen word overrides all the rest.

The standard fix is add-one (Laplace) smoothing: pretend every vocabulary word was seen one extra time in each class. Add $1$ to every count and, to keep the result a probability, add $|V|$ (one for each word type) to the denominator:

No probability is ever exactly zero, so no single word can override the rest of the evidence. Two practical points on the vocabulary. Unknown words — words in the test document that never appeared in any training class — are simply dropped from the test document; naive Bayes keeps no unknown-word model. Some systems also drop stop words (very frequent function words like the and a), though in most text classification tasks a stop list does not help and the whole vocabulary is used.

The training and testing procedures

The whole training procedure is: count documents per class for the priors, count words per class for the smoothed likelihoods, over a shared vocabulary $V$.

Testing sums the log-prior and the log-likelihoods of the in-vocabulary words, then returns the highest-scoring class.

A worked example

Take a two-class sentiment problem, positive ($+$) and negative ($-$), with five tiny training reviews and one test review, simplified from real movie reviews.

Priors. Three of the five training documents are negative, two positive:

Vocabulary and counts. The vocabulary has $|V|=20$ word types across both classes. The negative class holds $14$ word tokens, the positive class $9$. The test word with never appears in training, so it is an unknown word and we drop it; the test review reduces to predictable no fun. The three surviving test words appear in the concatenated per-class text with the counts below — note that fun is unseen in the negative class and predictable and no are unseen in the positive class, exactly the zeros smoothing must repair:

Now apply add-one smoothing to the three surviving words, adding $1$ to each count and $|V|=20$ to each class-token denominator:

Scores. Multiply each prior by the product of its three word likelihoods:

The negative score is the larger, so the model predicts negative for predictable with no fun. The single word fun (twice as likely under positive) is outweighed by predictable and no leaning negative, and by the higher negative prior.

Working in log space gives the same winner and shows the additive structure directly. Taking base-$e$ logs, the negative score is $\log (3/5)+\log (2/34)+\log (2/34)+\log (1/34)=-0.51-2.83-2.83-3.53=-9.70$, while the positive score is $\log (2/5)+\log (1/29)+\log (1/29)+\log (2/29)=-0.92-3.37-3.37-2.67=-10.33$. The larger (less negative) $-9.70$ again selects negative, and each word contributes one additive term — the linear form promised earlier, now with real numbers. Exponentiating recovers the probabilities: $e^{-9.70}=6.1\times 10^{-5}$ and $e^{-10.33}=3.2\times 10^{-5}$, matching the products above.

Optimizing for sentiment

Standard naive Bayes works for sentiment, but three small changes reliably improve it.

Binary naive Bayes

For sentiment, whether a word occurs matters more than how often. A review that says great five times is not five times more positive than one that says it once. Binary naive Bayes clips each document's word counts to $1$ before training and testing: remove all duplicate words within a document, then concatenate. The counts across the whole class can still exceed $1$ — great has a class count of $2$ if it appears in two different documents — but no single document contributes more than one.

Handling negation

The bag of words is blind to negation: I really like this movie (positive) and I didn't like this movie (negative) share the token like, yet mean opposites. A simple, effective baseline runs during text normalization: prepend the prefix NOT_ to every word after a token of logical negation (n't, not, no, never) until the next punctuation mark. So

Now NOT_like and NOT_recommend accrue negative associations, while NOT_bored and NOT_dismiss accrue positive ones, and the classifier learns them like any other feature. It is crude — it ignores the true scope of negation — but works well in practice.

Sentiment lexicons

When labeled data is scarce, we can use sentiment lexicons — hand-built lists of words pre-annotated as positive or negative, such as the General Inquirer, LIWC, and the MPQA Subjectivity Lexicon. A lexicon supplies dense features: instead of a separate feature per word, add one feature counting occurrences of any positive-lexicon word and another for negative-lexicon words. With plentiful matched training data, individual-word features beat two lexicon features; but when data is sparse or unrepresentative, the dense lexicon features generalize better.

Evaluating the classifier

We now have a trained naive-Bayes classifier that labels documents. Before pushing on to a better model, we need a way to say how good any classifier is — and, when one model beats another, whether the win is real or luck. Those questions are general (they apply to every classifier in the module, not just naive Bayes), so they get their own lesson. This continues in Evaluating Classifiers, which builds the confusion matrix, defines precision, recall, and F1, and closes on the paired bootstrap test for statistical significance. The rest of this lesson steps back to place naive Bayes among the models that came after it.

Beyond bag-of-words: transformer text classification

The bag-of-words assumption that makes naive Bayes tractable is also its main limitation. Discarding word order collapses the movie was not good and the movie was good, not to the same input; conditional independence pretends New and York carry unrelated evidence. The NOT_ prefix and lexicon features are partial workarounds. The line of work that removed these assumptions entirely comes from named, public research past this chapter, and it changed both the representation and the training objective.

Contextual embeddings and fine-tuning (Devlin et al., 2019)

BERT — Bidirectional Encoder Representations from Transformers (Devlin, Chang, Lee, and Toutanova, NAACL 2019) — is a transformer encoder pretrained on a large unlabeled corpus with a masked language model objective: hide a fraction of the tokens and train the network to predict them from both left and right context. The result is a contextual embedding for every token — a vector that depends on the whole sentence, so bank in river bank and bank in savings bank receive different vectors. That alone breaks the two assumptions naive Bayes rests on: order is preserved through position information, and words condition on one another through self-attention rather than being treated as independent.

To classify a document, BERT prepends a special [CLS] token to the input, runs the sequence through the encoder, and treats the encoder's output vector at the [CLS] position as a representation of the whole document. A single linear layer — a classification head — maps that vector to class scores, and the whole network is fine-tuned end to end on the labeled sentiment data: the pretrained weights are nudged, and the head is learned from scratch, by gradient descent on the classification loss. Where naive Bayes estimates its parameters by counting once, the transformer adjusts hundreds of millions of pretrained weights against the task.

When to still reach for naive Bayes

The transformer wins on accuracy across essentially every text-classification benchmark, and it captures negation, sarcasm, and word interactions that a bag of words cannot represent. The cost is orders of magnitude more computation, a large pretrained model to host, and far less interpretability — a naive-Bayes weight is a readable log-likelihood per word, while a fine-tuned attention weight is not. When labeled data is scarce, the classes are cleanly separated by a few keywords, the model must run on modest hardware, or a human needs to audit why a document was labeled, naive Bayes remains a strong, fast baseline — which is why it is the right place to start and still a standard point of comparison. The contrast is the same one drawn between generative and discriminative models above, taken to its modern extreme: counting once versus learning a deep function, interpretable versus accurate.³