---
title: Naive Bayes and Sentiment Classification
module: Text Classification
moduleNumber: 2
lessonNumber: 1
order: 201
summary: >
  Text classification assigns a category to a document — positive or negative,
  spam or not, one topic among many. Naive Bayes is a generative solution: apply
  Bayes' rule, assume the words are conditionally independent given the class, and
  the winning class is the one maximizing the product of a prior and per-word
  likelihoods. We train it by counting with add-one smoothing, work a full
  sentiment example by hand, sharpen it for sentiment (binary counts, negation,
  lexicons), and place it among the transformer classifiers that came after.
topics: [Classification]
sources:
  - book: Jurafsky
    ref: "Ch. 4 — Naive Bayes and Sentiment Classification; §4.1 Naive Bayes Classifiers; §4.2 Training"
  - book: Jurafsky
    ref: "§4.3 Worked Example; §4.4 Optimizing for Sentiment"
---

A review says _awful pizza and ridiculously overpriced_; you want a machine to
read it and answer **negative**. An email says _online pharmaceutical, WITHOUT ANY
COST, Dear Winner_; you want **spam**. A research paper is about epidemiology, not
embryology; you want the right **subject label**. These are all the same task:
take one document and assign it a category from a fixed set.[^jm-intro] This is
**text classification**. Its simplest and oldest solution, **naive Bayes**, is still
a strong baseline.

Formally, supervised classification takes an input document $d$ and a fixed set of
classes $C = \{c_1, \ldots, c_k\}$ and returns a predicted class $\hat c \in C$.
_Supervised_ means we learn the mapping from a training set of $N$ documents each
hand-labeled with a class, $(d_1, c_1), \ldots, (d_N, c_N)$. A **probabilistic**
classifier goes further and reports $P(c \mid d)$ for every class, not just the
winner — useful when a later stage wants to weigh the decision rather than commit
to it early.[^jm-supervised]

We focus on **sentiment analysis**, the extraction of the positive or negative
orientation a writer expresses toward something. In its binary form the words of a
review are strong cues on their own: _great_, _richly_, _awesome_ pull positive;
_pathetic_, _awful_, _ridiculously_ pull negative. Naive Bayes formalizes this
intuition.

> **Definition (Text classification).** Given a document $d$ and a fixed set of
> classes $C$, return the class $\hat c \in C$ that $d$ belongs to. When $|C| = 2$
> (spam / not-spam, positive / negative) the task is **binary**; sentiment with a
> _neutral_ class, topic labeling, and language ID are **multi-class**.

Naive Bayes is a **generative** classifier: it builds a model of how each class
would _generate_ a document, then selects the class that most plausibly produced the
one in hand. Its counterpart, the **discriminative**
[logistic regression](/natural-language-processing/classification/logistic-regression)
of the next lesson, skips the generative story and learns directly which features
separate the classes. Discriminative models are usually more accurate, but the
generative one is faster to train, needs less data, and is the right place to
start.

## The bag-of-words representation

Before any probability, we must decide what a document _is_ to the classifier.
Naive Bayes represents a document as a **bag of words**: an unordered multiset of
its words, position discarded, only frequency kept. _I love this movie, it's
sweet_ and _sweet, it's this movie I love_ are the same bag. The word _love_
counts the same whether it is the first word or the last.

$$
% caption: The bag-of-words view of a movie review. Word order is discarded; only
% the count of each word type survives, so the document becomes a table of
% (word, frequency) pairs.
\begin{tikzpicture}[>=stealth, font=\small,
  doc/.style={draw, minimum width=28mm, minimum height=32mm, align=left, font=\scriptsize, text width=25mm, anchor=center},
  cnt/.style={draw, minimum width=24mm, minimum height=32mm, align=left, font=\scriptsize, text width=22mm, anchor=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[doc] (d) at (0,0) {I love this movie! It is sweet, and the dialogue is great. I would recommend it to anyone.};
  \node[draw=acc, ellipse, minimum width=42mm, minimum height=30mm, align=center, font=\scriptsize] (b) at (6.5,0) {\textbf{bag of words}\\[1mm] love \ movie \ sweet\\ dialogue \ great\\ recommend \ it \ it\\ I \ I \ the \ldots};
  \node[cnt] (c) at (12.4,0) {it \hfill 3\\[0.4mm] I \hfill 2\\[0.4mm] the \hfill 1\\[0.4mm] love \hfill 1\\[0.4mm] sweet \hfill 1\\[0.4mm] great \hfill 1\\[0.4mm] movie \hfill 1\\[0.4mm] dialogue \hfill 1};
  \draw[->, acc, thick] (d.east) -- (b.west) node[midway, above, font=\scriptsize] {discard order};
  \draw[->, acc, thick] (b.east) -- (c.west) node[midway, above, font=\scriptsize] {count types};
\end{tikzpicture}
$$

This throws away real information — _the movie was not good_ and _the movie was
good, not_ collapse to the same bag — yet the representation works well in
practice. For topic and sentiment, the presence of the right words carries
most of the signal, and discarding word order costs little. The features of naive Bayes are these word identities, written
$w_1, w_2, \ldots, w_n$ for the words at each position in $d$.

## The naive Bayes classifier

Naive Bayes returns the class with the highest posterior probability given the
document:

$$
\hat c = \argmax_{c \in C} P(c \mid d).
$$

We cannot estimate $P(c \mid d)$ directly — there are too many possible documents
— so we turn it around with **Bayes' rule**, $P(x \mid y) = \dfrac{P(y \mid x)\,P(x)}{P(y)}$,
applied to $c$ and $d$:

$$
\hat c = \argmax_{c \in C} P(c \mid d)
       = \argmax_{c \in C} \frac{P(d \mid c)\,P(c)}{P(d)}.
$$

The denominator $P(d)$ is the same for every class, and we are choosing the
_arg max_ over classes of the same document $d$, so it cannot change which class
wins. Drop it:

$$
\hat c = \argmax_{c \in C} \; \underbrace{P(d \mid c)}_{\text{likelihood}} \; \underbrace{P(c)}_{\text{prior}}.
$$

The winning class maximizes the product of two things: the **prior** $P(c)$, how
common the class is, and the **likelihood** $P(d \mid c)$, how well the class
explains the document. This is why naive Bayes is called _generative_ — read
$P(d \mid c)\,P(c)$ as a recipe for producing a document: sample a class from the
prior, then sample the words from that class.

$$
% caption: Bayes' rule turns the hard-to-estimate posterior $P(c \mid d)$ into a
% prior times a likelihood; the evidence $P(d)$ is dropped because it is constant
% across classes.
\begin{tikzpicture}[>=stealth, font=\small,
  box/.style={draw, minimum width=30mm, minimum height=12mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[box, draw=acc, text=acc] (post) at (0,0) {posterior\\P(c given d)};
  \node[box] (lik) at (5.6,1.1) {likelihood\\P(d given c)};
  \node[box] (pri) at (5.6,-1.1) {prior\\P(c)};
  \node[font=\footnotesize] (ev) at (10.4,-1.1) {drop P(d): constant};
  \draw[->, acc, thick] (post.east) to[out=20, in=180] (lik.west);
  \draw[->, acc, thick] (post.east) to[out=-20, in=180] (pri.west);
  \node[font=\small] at (3.0,0.0) {$=$};
  \draw[->, black, dashed] (pri.east) -- (ev.west);
\end{tikzpicture}
$$

### The naive assumption

The likelihood $P(d \mid c) = P(w_1, w_2, \ldots, w_n \mid c)$ is still intractable:
estimating the probability of every possible combination of words would need more
parameters and more data than any corpus supplies. Naive Bayes makes two
simplifying assumptions to escape.

The first is the bag of words already introduced: position does not matter, so
_love_ contributes the same whether it is word 1 or word 20. The second, which
gives the method its name, is the **naive Bayes assumption** — the words are
_conditionally independent_ given the class:

$$
P(w_1, w_2, \ldots, w_n \mid c) = P(w_1 \mid c) \cdot P(w_2 \mid c) \cdots P(w_n \mid c) = \prod_{i} P(w_i \mid c).
$$

The assumption is false — _fun_ and _film_ are not independent, and _New_ raises
the odds of _York_ — but pretending they are collapses a combinatorial explosion
into a product of $n$ easy factors, and the resulting classifier works far better
than the false premise suggests. Putting the pieces together, the class
naive Bayes chooses is:

$$
\hat c = \argmax_{c \in C} \; P(c) \prod_{i \in \text{positions}} P(w_i \mid c).
$$

### Log space

That product runs over every word position, so for a document of any length it
multiplies dozens or hundreds of probabilities, each well below $1$. The result
underflows to zero in floating point. The fix is to work in **log space**: because
$\log$ is monotonic, the class that maximizes the product also maximizes the log of
the product, and a log turns the product into a sum:

$$
\hat c = \argmax_{c \in C} \; \log P(c) + \sum_{i \in \text{positions}} \log P(w_i \mid c).
$$

This is numerically safe and faster. It also exposes the shape of the model: the
predicted class is a _linear_ function of the input features (each contributes an
additive log-weight). Classifiers that decide by a linear combination of their
inputs — naive Bayes here, logistic regression next lesson — are called **linear
classifiers**.

## Training: counting with smoothing

Where do $P(c)$ and $P(w_i \mid c)$ come from? We estimate them from the training
set by **maximum likelihood** — just the frequencies in the data.

For the **prior**, ask what fraction of the training documents belong to class $c$.
With $N_c$ the number of documents in class $c$ and $N_{\text{doc}}$ the total:

$$
\hat P(c) = \frac{N_c}{N_{\text{doc}}}.
$$

For the **likelihood** $P(w_i \mid c)$, treat a document's words as a bag drawn
from class $c$. Concatenate every document of class $c$ into one large text, and
estimate the probability of word $w_i$ as its fraction among all word tokens in
that class:

$$
\hat P(w_i \mid c) = \frac{\count(w_i, c)}{\sum_{w \in V} \count(w, c)}.
$$

Here $V$ is the vocabulary — the union of all word types across _all_ classes, not
just the words seen in class $c$. That detail matters for the denominator below.

### The zero-probability problem

Maximum likelihood fails on unseen words. Suppose the word _fantastic_ never appears in
any _positive_ training document (perhaps it only showed up, sarcastically, in a
negative review). Then

$$
\hat P(\texttt{fantastic} \mid \texttt{+}) = \frac{\count(\texttt{fantastic}, \texttt{+})}{\sum_{w \in V}\count(w, \texttt{+})} = 0.
$$

Because naive Bayes _multiplies_ every word's likelihood, a single zero factor
drags the whole product to zero: a document containing _fantastic_ can never be
classified positive, no matter how much other evidence points that way. A single
unseen word overrides all the rest.

The standard fix is **add-one (Laplace) smoothing**: pretend every vocabulary word
was seen one extra time in each class. Add $1$ to every count and, to keep the
result a probability, add $|V|$ (one for each word type) to the denominator:

$$
\hat P(w_i \mid c) = \frac{\count(w_i, c) + 1}{\displaystyle\sum_{w \in V}\bigl(\count(w, c) + 1\bigr)} = \frac{\count(w_i, c) + 1}{\Bigl(\displaystyle\sum_{w \in V}\count(w, c)\Bigr) + |V|}.
$$

No probability is ever exactly zero, so no single word can override the rest of the
evidence. Two
practical points on the vocabulary. **Unknown words** — words in the test document
that never appeared in _any_ training class — are simply dropped from the test
document; naive Bayes keeps no unknown-word model. Some systems also drop **stop
words** (very frequent function words like _the_ and _a_), though in most text
classification tasks a stop list does not help and the whole vocabulary is used.

### The training and testing procedures

The whole training procedure is: count documents per class for the priors, count
words per class for the smoothed likelihoods, over a shared vocabulary $V$.

```algorithm
caption: $\textsc{Train-Naive-Bayes}(D, C)$ — estimate $\log P(c)$ and $\log P(w \mid c)$
input: labeled documents $D$, classes $C$
$V \gets$ set of all word types in $D$
for each class $c \in C$ do
  $N_{\text{doc}} \gets$ number of documents in $D$
  $N_c \gets$ number of documents in $D$ with class $c$
  $\textit{logprior}[c] \gets \log \dfrac{N_c}{N_{\text{doc}}}$
  $\textit{bigdoc}[c] \gets$ concatenation of all documents in $D$ with class $c$
  for each word $w \in V$ do
    $\textit{count}(w, c) \gets$ occurrences of $w$ in $\textit{bigdoc}[c]$
    $\textit{loglikelihood}[w, c] \gets \log \dfrac{\textit{count}(w, c) + 1}{\sum_{w' \in V}\bigl(\textit{count}(w', c) + 1\bigr)}$
return $\textit{logprior}$, $\textit{loglikelihood}$, $V$
```

Testing sums the log-prior and the log-likelihoods of the in-vocabulary words, then
returns the highest-scoring class.

```algorithm
caption: $\textsc{Test-Naive-Bayes}(d, \textit{logprior}, \textit{loglikelihood}, C, V)$ — classify document $d$
for each class $c \in C$ do
  $\textit{sum}[c] \gets \textit{logprior}[c]$
  for each position $i$ in $d$ do
    $w \gets d[i]$
    if $w \in V$ then
      $\textit{sum}[c] \gets \textit{sum}[c] + \textit{loglikelihood}[w, c]$
return $\argmax_c \textit{sum}[c]$
```

## A worked example

Take a two-class sentiment problem, positive ($+$) and negative ($-$), with five
tiny training reviews and one test review, simplified from real movie reviews.

| Set | Class | Document |
| --- | --- | --- |
| Training | $-$ | just plain boring |
| Training | $-$ | entirely predictable and lacks energy |
| Training | $-$ | no surprises and very few laughs |
| Training | $+$ | very powerful |
| Training | $+$ | the most fun film of the summer |
| Test | $?$ | predictable with no fun |

**Priors.** Three of the five training documents are negative, two positive:

$$
P(-) = \frac{3}{5}, \qquad P(+) = \frac{2}{5}.
$$

**Vocabulary and counts.** The vocabulary has $|V| = 20$ word types across both
classes. The negative class holds $14$ word tokens, the positive class $9$. The
test word _with_ never appears in training, so it is an unknown word and we drop it;
the test review reduces to _predictable no fun_. The three surviving test words
appear in the concatenated per-class text with the counts below — note that _fun_ is
unseen in the negative class and _predictable_ and _no_ are unseen in the positive
class, exactly the zeros smoothing must repair:

| Word | $\count(w, -)$ | $\count(w, +)$ |
| --- | --- | --- |
| predictable | 1 | 0 |
| no | 1 | 0 |
| fun | 0 | 1 |
| class tokens $\sum_w \count(w, c)$ | 14 | 9 |

Now apply add-one smoothing to the three surviving words, adding $1$ to each count
and $|V| = 20$ to each class-token denominator:

$$
\begin{aligned}
P(\texttt{predictable} \mid -) &= \frac{1 + 1}{14 + 20}, & P(\texttt{predictable} \mid +) &= \frac{0 + 1}{9 + 20}, \\[2mm]
P(\texttt{no} \mid -) &= \frac{1 + 1}{14 + 20}, & P(\texttt{no} \mid +) &= \frac{0 + 1}{9 + 20}, \\[2mm]
P(\texttt{fun} \mid -) &= \frac{0 + 1}{14 + 20}, & P(\texttt{fun} \mid +) &= \frac{1 + 1}{9 + 20}.
\end{aligned}
$$

**Scores.** Multiply each prior by the product of its three word likelihoods:

$$
\begin{aligned}
P(-)\,P(S \mid -) &= \frac{3}{5} \times \frac{2 \cdot 2 \cdot 1}{34^{3}} = 6.1 \times 10^{-5}, \\[2mm]
P(+)\,P(S \mid +) &= \frac{2}{5} \times \frac{1 \cdot 1 \cdot 2}{29^{3}} = 3.2 \times 10^{-5}.
\end{aligned}
$$

The negative score is the larger, so the model predicts **negative** for _predictable
with no fun_. The single word _fun_ (twice as likely under positive) is outweighed
by _predictable_ and _no_ leaning negative, and by the higher negative prior.

$$
% caption: The worked naive-Bayes decision for "predictable with no fun." The
% negative class scores 6.1e-5 against the positive class's 3.2e-5, so the arg max
% picks negative. "with" is dropped as an unknown word; the two negative-leaning
% words outweigh the single positive-leaning "fun".
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \draw[->, black] (0,0) -- (0,3.0) node[above, black, font=\scriptsize] {P(c) P(S $\mid$ c)};
  % bars scaled: 6.1 -> 2.3, 3.2 -> 1.21 (x0.377)
  \fill[red!16, draw=red] (0.8,0) rectangle (2.0,2.30);
  \fill[acc!22, draw=acc] (3.2,0) rectangle (4.4,1.21);
  \node[anchor=north, font=\scriptsize, text=red] at (1.4,-0.08) {negative};
  \node[anchor=north, font=\scriptsize, text=acc] at (3.8,-0.08) {positive};
  \node[anchor=south, font=\scriptsize, text=red] at (1.4,2.30) {6.1e-5};
  \node[anchor=south, font=\scriptsize, text=acc] at (3.8,1.21) {3.2e-5};
  \node[anchor=west, font=\scriptsize] at (5.1,1.7) {arg max picks negative};
\end{tikzpicture}
$$

Working in log space gives the same winner and shows the additive structure directly.
Taking base-$e$ logs, the negative score is $\log(3/5) + \log(2/34) + \log(2/34) +
\log(1/34) = -0.51 - 2.83 - 2.83 - 3.53 = -9.70$, while the positive score is
$\log(2/5) + \log(1/29) + \log(1/29) + \log(2/29) = -0.92 - 3.37 - 3.37 - 2.67 =
-10.33$. The larger (less negative) $-9.70$ again selects negative, and each word
contributes one additive term — the linear form promised earlier, now with real
numbers. Exponentiating recovers the probabilities: $e^{-9.70} = 6.1 \times 10^{-5}$
and $e^{-10.33} = 3.2 \times 10^{-5}$, matching the products above.

## Optimizing for sentiment

Standard naive Bayes works for sentiment, but three small changes reliably improve
it.

### Binary naive Bayes

For sentiment, whether a word _occurs_ matters more than how _often_. A review that
says _great_ five times is not five times more positive than one that says it once.
**Binary naive Bayes** clips each document's word counts to $1$ before training and
testing: remove all duplicate words within a document, then concatenate. The counts
across the whole class can still exceed $1$ — _great_ has a class count of $2$ if it
appears in two different documents — but no single document contributes more than
one.

$$
% caption: Binary naive Bayes clips per-document counts to one. Within a document,
% repeated words are deduplicated before counting; the same word can still accrue
% count across several documents.
\begin{tikzpicture}[>=stealth, font=\small,
  rowbox/.style={draw, minimum width=52mm, minimum height=8mm, align=left, font=\scriptsize, text width=50mm}]
  \definecolor{acc}{HTML}{2348F2}
  \node[rowbox] (a) at (0,0.8) {great scenes great f\/ilm};
  \node[rowbox, draw=acc] (b) at (0,-0.8) {great scenes f\/ilm};
  \draw[->, acc, thick] (a) -- (b) node[midway, right, font=\scriptsize] {\hspace{2mm} dedup within doc};
  \node[font=\scriptsize, anchor=west] at (3.2,0.8) {count(great) contributes 2};
  \node[font=\scriptsize, anchor=west, text=acc] at (3.2,-0.8) {count(great) contributes 1};
\end{tikzpicture}
$$

### Handling negation

The bag of words is blind to negation: _I really like this movie_ (positive) and
_I didn't like this movie_ (negative) share the token _like_, yet mean opposites.
A simple, effective baseline runs during text normalization: prepend the prefix
`NOT_` to every word after a token of logical negation (_n't, not, no, never_)
until the next punctuation mark. So

$$
\texttt{didn't like this movie , but I} \;\longrightarrow\; \texttt{didn't NOT\_like NOT\_this NOT\_movie , but I}.
$$

Now `NOT_like` and `NOT_recommend` accrue negative associations, while `NOT_bored`
and `NOT_dismiss` accrue positive ones, and the classifier learns them like any
other feature. It is crude — it ignores the true scope of negation — but works well
in practice.

### Sentiment lexicons

When labeled data is scarce, we can use **sentiment lexicons** — hand-built
lists of words pre-annotated as positive or negative, such as the General Inquirer,
LIWC, and the MPQA Subjectivity Lexicon. A lexicon supplies dense features: instead
of a separate feature per word, add one feature counting occurrences of _any_
positive-lexicon word and another for negative-lexicon words. With plentiful
matched training data, individual-word features beat two lexicon features; but when
data is sparse or unrepresentative, the dense lexicon features generalize better.

## Evaluating the classifier

We now have a trained naive-Bayes classifier that labels documents. Before pushing on
to a better model, we need a way to say _how good_ any classifier is — and, when one
model beats another, whether the win is real or luck. Those questions are general
(they apply to every classifier in the module, not just naive Bayes), so they get
their own lesson. This continues in [Evaluating Classifiers](/natural-language-processing/classification/evaluating-classifiers),
which builds the confusion matrix, defines precision, recall, and F1, and closes on
the paired bootstrap test for statistical significance. The rest of this lesson steps
back to place naive Bayes among the models that came after it.

## Beyond bag-of-words: transformer text classification

The bag-of-words assumption that makes naive Bayes tractable is also its main
limitation.
Discarding word order collapses _the movie was not good_ and _the movie was good, not_
to the same input; conditional independence pretends _New_ and _York_ carry
unrelated evidence. The `NOT_` prefix and lexicon features are partial workarounds.
The line of work that removed these assumptions entirely comes from named, public
research past this chapter, and it changed both the representation and the training
objective.

### Contextual embeddings and fine-tuning (Devlin et al., 2019)

**BERT** — Bidirectional Encoder Representations from Transformers (Devlin, Chang,
Lee, and Toutanova, _NAACL_ 2019) — is a transformer encoder pretrained on a large
unlabeled corpus with a **masked language model** objective: hide a fraction of the
tokens and train the network to predict them from both left and right context. The
result is a **contextual embedding** for every token — a vector that depends on the
whole sentence, so _bank_ in _river bank_ and _bank_ in _savings bank_ receive
different vectors. That alone breaks the two assumptions naive Bayes rests on: order
is preserved through position information, and words condition on one another through
self-attention rather than being treated as independent.

To classify a document, BERT prepends a special `[CLS]` token to the input, runs the
sequence through the encoder, and treats the encoder's output vector at the `[CLS]`
position as a representation of the whole document. A single linear layer — a
**classification head** — maps that vector to class scores, and the whole network is
**fine-tuned** end to end on the labeled sentiment data: the pretrained weights are
nudged, and the head is learned from scratch, by gradient descent on the
classification loss. Where naive Bayes estimates its parameters by counting once,
the transformer adjusts hundreds of millions of pretrained weights against the task.

$$
% caption: Transformer text classification (BERT). A [CLS] token is prepended, the
% sentence runs through a pretrained encoder, and the encoder output at the [CLS]
% position feeds a linear classification head fine-tuned on the labels. Order and
% word interactions are preserved by self-attention, unlike the bag of words.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  tok/.style={draw, minimum width=13mm, minimum height=7mm, align=center, font=\scriptsize},
  big/.style={draw, minimum width=62mm, minimum height=9mm, align=center, font=\scriptsize}]
  \definecolor{acc}{HTML}{2348F2}
  \foreach \i/\w in {0/CLS, 1/awful, 2/but, 3/funny} \node[tok] (t\i) at (\i*1.55,0) {\w};
  \node[big, draw=acc, text=acc] (enc) at (2.325,1.5) {pretrained transformer encoder};
  \foreach \i in {0,1,2,3} \draw[->, acc] (t\i.north) -- (t\i.north |- enc.south);
  \node[tok, draw=acc, text=acc] (cls) at (2.325,3.0) {CLS vector};
  \draw[->, acc, thick] (enc.north) -- (cls.south);
  \node[big] (head) at (2.325,4.4) {linear head $\to$ positive / negative};
  \draw[->, acc, thick] (cls.north) -- (head.south);
  \node[anchor=west, font=\scriptsize, text=black] at (5.6,3.0) {contextual, order-aware};
  \node[anchor=west, font=\scriptsize, text=black] at (5.6,4.4) {f\/ine-tuned on labels};
\end{tikzpicture}
$$

### When to still reach for naive Bayes

The transformer wins on accuracy across essentially every text-classification
benchmark, and it captures negation, sarcasm, and word interactions that a bag of
words cannot represent. The cost is orders of magnitude more computation, a large
pretrained model to host, and far less interpretability — a naive-Bayes weight is a
readable log-likelihood per word, while a fine-tuned attention weight is not. When
labeled data is scarce, the classes are cleanly separated by a few keywords, the
model must run on modest hardware, or a human needs to audit _why_ a document was
labeled, naive Bayes remains a strong, fast baseline — which is why it is the right
place to start and still a standard point of comparison. The contrast is the same one
drawn between generative and discriminative models above, taken to its modern
extreme: counting once versus learning a deep function, interpretable versus
accurate.[^bert-clf]

[^jm-intro]: **Jurafsky & Martin**, _Speech and Language Processing_ (3rd ed.), Ch. 4 — Naive Bayes and Sentiment Classification: text categorization as assigning a label to a document, with sentiment analysis, spam detection, and subject/topic labeling as the running tasks (the last being the task naive Bayes was invented for in 1961).
[^jm-supervised]: **Jurafsky & Martin**, §4.1 — Naive Bayes Classifiers: supervised classification maps a document $d$ to a class $\hat c \in C$ learned from labeled examples; the multinomial naive Bayes model, Bayes' rule, the bag-of-words and conditional-independence assumptions, and the log-space linear form.
[^bert-clf]: Transformer-based text classification. **Devlin, Chang, Lee, and Toutanova**, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," _NAACL-HLT_ (2019) — a transformer encoder pretrained with a masked-language-model objective yields contextual token embeddings; a document is classified by prepending a `[CLS]` token and fine-tuning a linear head on its encoder output. This drops both the bag-of-words and conditional-independence assumptions (order and word interactions are modeled by self-attention), trading naive Bayes' single counting pass and interpretability for higher accuracy at much greater computational cost. The transformer architecture it builds on is **Vaswani et al.**, "Attention Is All You Need," _NeurIPS_ (2017).