---
title: "Bayes' Rule and Naive Bayes"
module: Uncertainty
moduleNumber: 4
lessonNumber: 2
order: 402
summary: >
  Bayes' rule inverts a causal model into a diagnostic one, turning "how a cause
  produces its symptoms" into "which cause explains what I observed." Ignoring
  the prior is the base-rate fallacy behind overconfident test results.
  Conditional independence then lets several pieces of evidence combine by
  multiplying likelihood ratios instead of building an exponential joint, giving the
  naive Bayes model and pointing directly at Bayesian networks.
topics: [Uncertainty]
sources:
  - book: AIMA
    ref: "Ch. 13 — Quantifying Uncertainty; §13.5 Bayes' Rule and Its Use"
  - book: AIMA
    ref: "§13.5.1 Applying Bayes' Rule; §13.5.2 Combining Evidence and Conditional Independence"
---

This builds on [Quantifying Uncertainty](/artificial-intelligence/uncertainty/probability-and-bayes),
which laid out degrees of belief, the full joint distribution, and plain
independence. There we left off needing a subtler tool: a way to turn a model of how
causes produce effects into an answer about which cause is present, and a way to fold
in several observations at once without the joint distribution exploding. Both come
from one equation and one refinement of independence.

## Bayes' rule

Write the product rule both ways — $P(a \land b) = P(a \mid b)P(b)$ and $P(a \land
b) = P(b \mid a)P(a)$ — set the right-hand sides equal, and divide. The result is
Bayes' rule.

> **Theorem (Bayes' rule).** $\;P(b \mid a) = \dfrac{P(a \mid b)\,P(b)}{P(a)}$, and
> in distribution form $\mathbf{P}(Y \mid X) = \dfrac{\mathbf{P}(X \mid
> Y)\,\mathbf{P}(Y)}{\mathbf{P}(X)} = \alpha\, \mathbf{P}(X \mid Y)\,\mathbf{P}(Y)$,
> with $\alpha$ the normalizer over the values of $Y$.

On its face Bayes' rule computes one conditional from three other numbers, which
looks like a step backwards. It matters in a specific, common situation:
we perceive an **effect** and want to infer its unknown **cause**. Rewriting the
rule with those roles,

$$
P(\text{cause} \mid \text{effect}) \;=\;
\frac{P(\text{effect} \mid \text{cause})\, P(\text{cause})}{P(\text{effect})}.
$$

The quantity $P(\text{effect} \mid \text{cause})$ is **causal** knowledge — how the
world works, how a cause produces its symptoms. The quantity we want,
$P(\text{cause} \mid \text{effect})$, is **diagnostic** — reasoning from symptom
back to cause. Bayes' rule turns the first into the second.

$$
% caption: Bayes' rule inverts the causal direction. A model gives $P(effect \mid
% cause)$ (how a cause produces symptoms); Bayes' rule turns it into the
% diagnostic $P(cause \mid effect)$ we need, weighted by the prior $P(cause)$.
\begin{tikzpicture}[>=stealth, font=\small,
  node/.style={draw, minimum width=22mm, minimum height=11mm, align=center, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[node, draw=acc, text=acc, thick] (cause) at (0,0)  {cause\\(meningitis)};
  \node[node] (eff) at (5.4,0) {symptom\\(neck pain)};
  % causal arrow (forward, known)
  \draw[->, black, thick] (cause) to[bend left=22]
    node[midway, above, font=\scriptsize, text=black] {causal: P(symptom given cause) = 0.7} (eff);
  % diagnostic arrow (backward, wanted)
  \draw[->, acc, thick] (eff) to[bend left=22]
    node[midway, below, font=\scriptsize, text=acc] {diagnostic: P(cause given symptom) = ?} (cause);
\end{tikzpicture}
$$

For example, take the medical version. Meningitis ($m$) causes a stiff neck
($s$) about $70\%$ of the time, so $P(s \mid m) = 0.7$ is the doctor's causal
knowledge. The priors are $P(m) = 1/50000$ and $P(s) = 0.01$. Then

$$
P(m \mid s) \;=\; \frac{P(s \mid m)\, P(m)}{P(s)}
\;=\; \frac{0.7 \times (1/50000)}{0.01} \;=\; 0.0014.
$$

Fewer than $1$ in $700$ stiff-necked patients has meningitis — even though
meningitis strongly _indicates_ a stiff neck, the disease stays rare because its
prior is tiny compared with the prior on stiff necks. The prior matters:
dropping it (a common intuitive error) inflates the estimate by orders of
magnitude.

Why prefer Bayes' rule to just storing the diagnostic number $P(m \mid s)$
directly? Because causal knowledge is more **stable** than diagnostic knowledge. If
an epidemic strikes, the prior $P(m)$ jumps, and with it the correct posterior
$P(m \mid s)$. A doctor who memorized $P(m \mid s)$ from pre-epidemic statistics has
no way to update it. A doctor who computes it from $P(s \mid m)$, $P(m)$, and $P(s)$
sees at once that it should rise in step with $P(m)$ — because the causal fact
$P(s \mid m)$, how meningitis affects the body, is unchanged by the epidemic.[^aima-diag]

If we do not have $P(s)$ we can avoid it by computing the posterior for both values
of the query and normalizing, exactly as in the joint-table example:

$$
\mathbf{P}(M \mid s) \;=\; \alpha\, \langle P(s \mid m)P(m),\; P(s \mid \lnot m)P(\lnot m)\rangle.
$$

### A worked medical test: the base-rate trap

The meningitis numbers were handed to us. In practice the causal quantities come
from a **diagnostic test** described by two rates, and the arithmetic exposes a
common error. Suppose a disease $D$ has **prevalence**
$P(d) = 0.001$ — one person in a thousand has it. A test $T$ for it has
**sensitivity** (true-positive rate) $P(t \mid d) = 0.99$: it catches $99\%$ of
sick people. Its **specificity** (true-negative rate) is $P(\lnot t \mid \lnot d)
= 0.99$, so its **false-positive rate** is $P(t \mid \lnot d) = 0.01$. You test
positive. What is the probability you are sick, $P(d \mid t)$?

The reflex answer is "about $99\%$ — the test is $99\%$ accurate." It is wrong by
almost two orders of magnitude, and Bayes' rule shows why. Compute the two
unnormalized terms:

$$
P(t \mid d)\,P(d) \;=\; 0.99 \times 0.001 \;=\; 0.00099,
$$
$$
P(t \mid \lnot d)\,P(\lnot d) \;=\; 0.01 \times 0.999 \;=\; 0.00999.
$$

Now normalize over the two values of $D$; their sum is $P(t)$:

$$
P(d \mid t) \;=\; \frac{0.00099}{0.00099 + 0.00999} \;=\; \frac{0.00099}{0.01098}
\;\approx\; 0.090.
$$

A positive result raises the probability of disease from $0.1\%$ to about $9\%$ —
higher than the prior, but nowhere near $99\%$. The reason is entirely in the base
rate. The disease is so rare that the $0.01$ false-positive rate, applied to the
huge healthy majority, produces about ten times as many false alarms as there are
true positives. Reasoning with a concrete population makes this unmistakable: out
of $100{,}000$ people, $100$ are sick and $99$ of them test positive; of the
$99{,}900$ healthy, $1\%$ — about $999$ — also test positive. So $99 + 999 = 1098$
people test positive, and only $99$ of them are sick: $99 / 1098 \approx 9\%$,
matching the Bayes computation exactly.

$$
% caption: The base-rate trap as a population of 100,000. Only 100 are sick (left),
% and 99 of them test positive; but 999 of the 99,900 healthy also test positive.
% Among the 1098 positives, only 99 are truly sick, so P(disease given positive) is
% about 9 percent, not 99 percent.
\begin{tikzpicture}[>=stealth, font=\footnotesize]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % population split
  \node[draw, thick, minimum width=20mm, minimum height=12mm, align=center] (pop) at (0,0) {100000\\people};
  \node[draw, red, minimum width=20mm, minimum height=10mm, align=center] (sick) at (4.4,1.3) {sick: 100};
  \node[draw, minimum width=22mm, minimum height=10mm, align=center] (well) at (4.4,-1.3) {healthy: 99900};
  \draw[->, red] (pop) -- (sick);
  \draw[->, black] (pop) -- (well);
  % positives
  \node[draw, red, thick, minimum width=26mm, minimum height=10mm, align=center] (tp) at (9.2,1.3) {test +: 99\\(true positive)};
  \node[draw, minimum width=26mm, minimum height=10mm, align=center] (fp) at (9.2,-1.3) {test +: 999\\(false positive)};
  \draw[->, red] (sick) -- (tp);
  \draw[->, black] (well) -- (fp);
  \node[text=acc, align=center, font=\scriptsize, anchor=north] at (9.2,-2.3)
    {P(sick given +) = 99 / (99+999) = 9 percent};
\end{tikzpicture}
$$

This is the general shape of screening: a test can be excellent (high sensitivity
and specificity) and a positive result still mean the patient probably does not
have the disease, whenever the disease is rare enough. Dropping the prior — treating
$P(d \mid t)$ as if it equalled the sensitivity $P(t \mid d)$ — is the **base-rate
fallacy**, the very mistake Bayes' rule forces you not to make: the prior $P(d)$
appears explicitly in the numerator and again inside the normalizer.
The same computation governs spam filters, fraud alarms, and any rare-event
detector: the rarer the event, the more the base rate dominates the test.

## Combining evidence: conditional independence and naive Bayes

Bayes' rule handles one piece of evidence cleanly. But real diagnosis piles up
symptoms — a toothache _and_ a catch _and_ an X-ray shadow — and the naive way of
folding them all in brings back the exponential table we were trying to escape. The
fix is the conditional independence promised at the end of the previous lesson: once
the common cause is fixed, the symptoms stop informing each other, and the evidence
combines by simple multiplication. Consider a dentist who
observes both a toothache and a catch and wants $\mathbf{P}(Cavity \mid toothache \land
catch)$. Bayes' rule reformulates it as $\alpha\, \mathbf{P}(toothache \land catch
\mid Cavity)\, \mathbf{P}(Cavity)$ — but now we need the joint likelihood of the two
symptoms for each value of $Cavity$, and with $n$ symptoms there are $2^n$ such
combinations to store. We are back to an exponential table. This is what led researchers
_away_ from probability theory in the first place: combining evidence naively
seems to require a full joint over all the evidence.

The solution is a refinement of independence. $Toothache$ and $Catch$ are _not_
independent — both are more likely when there is a cavity. But they are independent
_once we fix whether there is a cavity_. The cavity is the common cause of both; a
toothache reflects the state of the nerve, a catch reflects the geometry the probe
meets, and given the cavity neither tells us anything further about the other.

> **Definition (Conditional independence).** Variables $X$ and $Y$ are
> conditionally independent given $Z$ when, once $Z$ is known, learning $X$ says
> nothing about $Y$: $\;\mathbf{P}(X, Y \mid Z) = \mathbf{P}(X \mid Z)\,\mathbf{P}(Y
> \mid Z)$, equivalently $\mathbf{P}(X \mid Y, Z) = \mathbf{P}(X \mid Z)$. Written
> $X \perp Y \mid Z$.

$$
% caption: Conditional independence, not marginal independence. Toothache and Catch
% are correlated on their own because each signals a cavity (left); but once the
% cavity is fixed, the common cause is removed and the two become independent
% (right). Fixing Cavity blocks the only path between them.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  v/.style={draw, circle, minimum size=10mm, inner sep=0pt, font=\scriptsize},
  vg/.style={draw, circle, minimum size=10mm, inner sep=0pt, font=\scriptsize, fill=acc!18, thick, draw=acc}]
  \definecolor{acc}{HTML}{2348F2}
  % left: marginal (path open through cavity)
  \node[v] (c1) at (0,1.3) {Cav};
  \node[v] (t1) at (-1.3,-0.4) {Tch};
  \node[v] (k1) at (1.3,-0.4) {Ctch};
  \draw[->, black] (c1) -- (t1);
  \draw[->, black] (c1) -- (k1);
  \draw[<->, acc, dashed] (t1) to[bend right=30] node[below, font=\scriptsize, text=acc] {correlated} (k1);
  \node[font=\scriptsize, anchor=north] at (0,-1.6) {marginally: dependent};
  % right: cavity observed (path blocked)
  \begin{scope}[xshift=62mm]
    \node[vg] (c2) at (0,1.3) {Cav};
    \node[v] (t2) at (-1.3,-0.4) {Tch};
    \node[v] (k2) at (1.3,-0.4) {Ctch};
    \draw[->, black] (c2) -- (t2);
    \draw[->, black] (c2) -- (k2);
    \node[font=\scriptsize, anchor=north] at (0,-1.6) {given Cavity: independent};
  \end{scope}
\end{tikzpicture}
$$

With $Toothache \perp Catch \mid Cavity$, the troublesome joint likelihood factors,
$\mathbf{P}(toothache \land catch \mid Cavity) = \mathbf{P}(toothache \mid
Cavity)\,\mathbf{P}(catch \mid Cavity)$, and the query collapses to a product of
per-symptom terms:

$$
\mathbf{P}(Cavity \mid toothache \land catch) \;=\;
\alpha\, \mathbf{P}(toothache \mid Cavity)\,\mathbf{P}(catch \mid Cavity)\,\mathbf{P}(Cavity).
$$

Each symptom now contributes its own factor, and we never build the exponential
joint. This pattern — one cause directly influencing several effects that are
conditionally independent given the cause — is common enough to have a name.

> **Definition (Naive Bayes model).** When a single cause $C$ directly influences a
> number of effects $E_1, \ldots, E_n$ that are all conditionally independent given
> $C$, the full joint factors as $\;\mathbf{P}(C, E_1, \ldots, E_n) =
> \mathbf{P}(C)\prod_i \mathbf{P}(E_i \mid C)$. It is "naive" because the
> conditional-independence assumption is often only approximately true, yet the
> model works well in practice.

$$
% caption: The naive Bayes model. One cause $C$ points to each effect $E_i$; the
% effects are conditionally independent given $C$, so the joint is $P(C)$ times a
% product of the per-effect terms $P(E_i \mid C)$. The representation grows as
% $O(n)$, not $O(2^n)$.
\begin{tikzpicture}[>=stealth, font=\small,
  v/.style={draw, circle, minimum size=11mm, inner sep=0pt, font=\footnotesize}]
  \definecolor{acc}{HTML}{2348F2}
  \node[v, draw=acc, text=acc, thick] (c) at (0,1.7) {Cavity};
  \node[v] (e1) at (-3.4,-0.6) {Tooth-\\ache};
  \node[v] (e2) at (0,-0.6)    {Catch};
  \node[v] (e3) at (3.4,-0.6)  {X-ray};
  \draw[->, acc, thick] (c) -- (e1);
  \draw[->, acc, thick] (c) -- (e2);
  \draw[->, acc, thick] (c) -- (e3);
  \node[font=\scriptsize, text=black, anchor=west] at (1.2,1.7) {the common cause};
\end{tikzpicture}
$$

Carry real numbers through to see the factoring work. Suppose $P(cavity) = 0.2$,
and given a cavity the two symptoms have $P(toothache \mid cavity) = 0.6$ and
$P(catch \mid cavity) = 0.9$; without a cavity, $P(toothache \mid \lnot cavity) =
0.1$ and $P(catch \mid \lnot cavity) = 0.2$. Observing both symptoms, the two
unnormalized terms are

$$
P(cavity)\,P(t \mid cavity)\,P(c \mid cavity) \;=\; 0.2 \times 0.6 \times 0.9 \;=\; 0.108,
$$
$$
P(\lnot cavity)\,P(t \mid \lnot cavity)\,P(c \mid \lnot cavity)
\;=\; 0.8 \times 0.1 \times 0.2 \;=\; 0.016.
$$

Normalizing, $P(cavity \mid t \land c) = 0.108 / (0.108 + 0.016) \approx 0.871$.
Two mutually reinforcing symptoms push the belief from a prior of $0.2$ to about
$0.87$ — and we combined them by multiplying two one-symptom tables, never building
the joint over both. Each symptom multiplied in its own **likelihood ratio**: the
toothache contributed $0.6 / 0.1 = 6$, the catch $0.9 / 0.2 = 4.5$, so together
they shifted the prior odds $0.2 / 0.8 = 0.25$ to posterior odds $0.25 \times 6
\times 4.5 = 6.75$, i.e. a probability of $6.75 / 7.75 \approx 0.871$. The odds form
of Bayes' rule shows exactly why conditionally independent evidence combines by
_multiplying_ likelihood ratios.

$$
% caption: Odds form of Bayes' rule with conditionally independent evidence. The
% prior odds are multiplied by one likelihood ratio per symptom; for the dentistry
% numbers, 0.25 times 6 times 4.5 gives posterior odds 6.75, a probability of about
% 0.87.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, minimum width=24mm, minimum height=11mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[bx] (prior) at (0,0) {prior odds\\0.25};
  \node[bx] (l1) at (3.8,0) {x toothache\\ratio 6};
  \node[bx] (l2) at (7.4,0) {x catch\\ratio 4.5};
  \node[bx, draw=acc, text=acc, thick] (post) at (11.2,0) {posterior odds\\6.75 (p = 0.87)};
  \draw[->, black, thick] (prior) -- (l1);
  \draw[->, black, thick] (l1) -- (l2);
  \draw[->, acc, thick] (l2) -- (post);
\end{tikzpicture}
$$

The saving is substantial. For $n$ effects conditionally independent given the cause,
the representation needs one prior $\mathbf{P}(C)$ and $n$ conditional tables
$\mathbf{P}(E_i \mid C)$ — size $O(n)$ rather than $O(2^n)$. Conditional
independence, unlike absolute independence, is common: most large probabilistic
domains decompose into weakly connected pieces because each variable is directly
affected by only a few others. This decomposition of a huge joint into small,
locally connected factors is one of the most important developments in the recent
history of AI.[^aima-condind]

The naive Bayes model appears throughout applied AI. In natural-language
processing it is the classic
[naive Bayes text classifier](/natural-language-processing/classification/naive-bayes-and-sentiment):
the document's sentiment is the hidden cause, each word an effect assumed
conditionally independent of the others given the sentiment, and a document is
scored by the same product $\mathbf{P}(C)\prod_i \mathbf{P}(word_i \mid C)$. The
independence assumption is false — words are correlated — yet the classifier is hard
to beat for the effort.

## Toward Bayesian networks

The naive Bayes model is the simplest interesting case of a general idea:
represent a joint distribution not as one exponential table but as a product of
small local factors, each capturing how a variable depends on a handful of others.
Naive Bayes hard-codes a single common cause with conditionally independent
effects. Lift that restriction — let each variable depend on an arbitrary small set
of "parents," with conditional independencies read off the graph — and you get a
**Bayesian network**: a directed graph whose structure encodes the conditional
independencies, plus a small conditional table at each node.

The full joint factors along the graph, $\mathbf{P}(X_1, \ldots, X_n) = \prod_i
\mathbf{P}(X_i \mid parents(X_i))$, so a domain that would need $2^n$ numbers as a
flat table often needs only a few dozen. That is the compact, factored
representation the [next lesson](/artificial-intelligence/uncertainty/bayesian-networks)
develops in full, and the engine behind
[reasoning over time](/artificial-intelligence/uncertainty/reasoning-over-time) and
the [decision-theoretic agents](/artificial-intelligence/uncertainty/making-decisions)
that close the module.

## Probabilistic programming and Bayesian ML

Inference by enumeration is $O(2^n)$, and even the Bayesian networks of the next
lesson are worst-case intractable to solve exactly. The developments that made
probabilistic reasoning practical at scale, after _AIMA_'s third edition, share a
single move: give up on exact summation and _sample_ instead, or _approximate_ the
posterior with optimization.

**Approximate inference by sampling.** Rather than sum every entry of a joint,
draw many random possible worlds consistent with the evidence and estimate the
posterior as a frequency. **Markov-chain Monte Carlo** methods make this efficient
even in high dimensions; **Hamiltonian Monte Carlo** and its self-tuning variant
the **No-U-Turn Sampler** (Hoffman & Gelman, _JMLR_ 2014) are the samplers inside
most modern Bayesian toolkits. For posteriors where sampling is too slow, **variational
inference** turns the problem into optimization — fit a simple distribution to the
true posterior by minimizing a divergence — and stochastic variants scale it to
large data (Hoffman et al., "Stochastic Variational Inference," _JMLR_ 2013).

**Probabilistic programming.** The idea that unifies these is to treat a
probabilistic model as a _program_: you write down the generative story in a
language with random primitives, and a general-purpose inference engine computes
posteriors over the program's latent variables. **Stan** (Carpenter et al.,
_Journal of Statistical Software_ 2017) compiles a model to a tuned HMC sampler;
**Pyro** (Bingham et al., _JMLR_ 2019) builds probabilistic programs on top of a
deep-learning framework so that neural networks and probabilistic models compose.
The naive Bayes model of this lesson is one line in such a language; the same
engine then handles models far too large to enumerate by hand.

$$
% caption: Two escapes from exact inference. Exact enumeration sums all 2^n worlds
% and does not scale; sampling (MCMC) estimates the posterior from random draws,
% and variational inference fits a simple distribution q to the true posterior p by
% optimization. Both trade exactness for tractability.
\begin{tikzpicture}[>=stealth, font=\footnotesize,
  bx/.style={draw, minimum width=34mm, minimum height=11mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[bx] (exact) at (0,0) {exact enumeration\\sum all 2-to-the-n worlds};
  \node[bx, draw=acc, text=acc, thick] (mc) at (5.6,1.2) {sampling (MCMC)\\estimate from draws};
  \node[bx, draw=acc, text=acc, thick] (vi) at (5.6,-1.2) {variational\\f\/it q to p by optimizing};
  \draw[->, black, thick] (exact.east) -- ++(0.6,0) |- (mc.west);
  \draw[->, black, thick] (exact.east) -- ++(0.6,0) |- (vi.west);
  \node[font=\scriptsize, text=black, anchor=south] at (2.8,0.05) {when the table is too big};
\end{tikzpicture}
$$

**The Bayesian view in machine learning.** The same calculus reframes learning
itself. A Bayesian treats model parameters $\theta$ as a random variable with a
prior $P(\theta)$; data $\mathcal{D}$ updates it to a posterior $P(\theta \mid
\mathcal{D}) \propto P(\mathcal{D} \mid \theta)\,P(\theta)$ — Bayes' rule again,
with parameters as the "cause" and data as the "effect." Predictions average over
the posterior instead of committing to one parameter setting, which yields
calibrated uncertainty rather than a bare point estimate (MacKay's 1992 work on
Bayesian neural networks is the early reference; the idea is standard in modern
probabilistic ML). This is the direct ancestor of the
[probabilistic-learning](/artificial-intelligence/learning/probabilistic-learning)
methods later in the course, which is why the material here is
foundational.[^beyond-ppl]

Two cautions apply. Approximate methods trade a guarantee for
speed: a sampler that has not mixed, or a variational family too simple to fit the
true posterior, returns a confident but wrong answer, so diagnostics matter.
And the choice of prior is a modeling decision, not a fact read off the data — a
poorly chosen prior biases the posterior, which is a feature (it lets you inject
knowledge) and a hazard (it lets you inject error) at once.

Probability completes the account of rational agency.
Logic gave the agent a way to reason when it knew the rules exactly.
Probability gives it the calculus of rational belief when it does not — degrees of
belief that obey the axioms, updated by conditioning and Bayes' rule as evidence
arrives, factored by independence into something an agent can actually compute.

[^aima-diag]: **Russell & Norvig**, §13.5.1 — Applying Bayes' rule: the meningitis/stiff-neck example, and why causal knowledge $P(effect \mid cause)$ is more robust than memorized diagnostic knowledge under changes in the prior.
[^aima-condind]: **Russell & Norvig**, §13.5.2 — Using Bayes' rule and combining evidence; conditional independence, the naive Bayes model $\mathbf{P}(Cause)\prod_i \mathbf{P}(Effect_i \mid Cause)$, and the $O(n)$-versus-$O(2^n)$ saving from decomposition.
[^beyond-ppl]: **M. Hoffman & A. Gelman**, "The No-U-Turn Sampler," _JMLR_ 15 (2014), and **M. Hoffman et al.**, "Stochastic Variational Inference," _JMLR_ 14 (2013) — scalable approximate posterior inference by sampling and by optimization. **B. Carpenter et al.**, "Stan: A Probabilistic Programming Language," _Journal of Statistical Software_ 76 (2017), and **E. Bingham et al.**, "Pyro: Deep Universal Probabilistic Programming," _JMLR_ 20 (2019) — probabilistic programming systems with general-purpose inference engines. **D. MacKay**, "A Practical Bayesian Framework for Backpropagation Networks," _Neural Computation_ 4 (1992) — the Bayesian treatment of model parameters as a posterior updated by Bayes' rule.