---
title: Dark Energy and the Accelerating Universe
draft: false
module: Cosmic Expansion and Dynamics
moduleNumber: 11
lessonNumber: 5
order: 1105
summary: >
  In 1998 two teams found that distant Type Ia supernovae are fainter than a
  decelerating universe predicts, revealing that the expansion is accelerating and that
  a component with negative pressure dominates the energy budget. The simplest
  candidate is the cosmological constant, or vacuum energy, with an equation of state
  near minus one. It works observationally but leaves two deep puzzles: why the vacuum
  energy is a hundred and twenty orders of magnitude smaller than expected, and why it
  is comparable to the matter density just now.
topics: [Cosmology, Dark Energy]
sources:
  - book: Ryden
    ref: "Ch. 5 §5.5 The Cosmological Constant; Ch. 6 — Measuring Cosmological Parameters"
  - url: "https://arxiv.org/abs/astro-ph/9812133"
    ref: "Perlmutter et al. (1999) / Riess et al. (1998) — Type Ia supernova evidence for cosmic acceleration"
  - url: "https://arxiv.org/abs/1807.06209"
    ref: "Planck Collaboration (2018) — cosmological parameters"
---

For most of the twentieth century the central question of cosmology was assumed to be
whether the universe contained enough matter to halt its expansion and recollapse. The
expected answer was that gravity decelerates the expansion; the only question was by how
much. In 1998 two independent teams measuring distant Type Ia supernovae found the
opposite: the expansion is **accelerating**. This lesson traces that discovery through the
supernova Hubble diagram, identifies the responsible component as a **cosmological
constant** or vacuum energy with equation of state $w \approx -1$, and confronts the two
theoretical crises it created — the cosmological-constant problem, a discrepancy of some
$120$ orders of magnitude between the observed and predicted vacuum energy, and the
coincidence problem, that dark energy and matter happen to be comparable right now. It
closes with the current observational constraints on $w$ and the alternatives to a pure
constant.

## The supernova Hubble diagram

Type Ia supernovae are the workhorse of the measurement because they are
**standardizable candles**: after correcting for the empirical correlation between peak
brightness and light-curve decline rate (the Phillips relation), their peak luminosities
scatter by only about $0.15\ \text{mag}$, making them visible and calibratable to
redshifts beyond $z \approx 1$. Measuring a supernova's apparent brightness gives its
luminosity distance $d_L$; its host galaxy's spectrum gives its redshift $z$. Plotting
$d_L$ against $z$ — or equivalently the distance modulus $m - M = 5\log_{10}(d_L/10\,
\text{pc})$ against redshift — tests the expansion history, because $d_L(z)$ depends on
$\Omega_{m,0}$ and $\Omega_{\Lambda,0}$ through the Friedmann integral of the previous
lesson.

The prediction is clean. In a decelerating, matter-dominated universe the expansion was
faster in the past, so a source at given redshift is relatively **nearby** and bright. In
an accelerating universe the expansion was slower in the past, so the same redshift
corresponds to a **larger** distance and a fainter source. The 1998 data showed distant
supernovae systematically **fainter** — by about $0.25\ \text{mag}$, some 25% in flux —
than the best-fitting decelerating model, and consistent instead with an accelerating
universe. The deviation grows with redshift exactly as an accelerating $d_L(z)$ predicts.
The result required a negative deceleration parameter, $q_0 < 0$, and hence a component
with $\varepsilon + 3P < 0$: negative pressure.[^sn-discovery]

$$
% caption: Distant supernovae are fainter than a decelerating universe predicts. The
% data track the accelerating model, requiring a negative-pressure component.
\begin{tikzpicture}[scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (0,0) -- (8.6,0) node[right, black!70] {redshift};
\draw[->, black] (0,0) -- (0,4.4) node[above, black!70] {distance modulus};
% decelerating model (lower, dashed neutral)
\draw[black, very thick, densely dashed] (0.2,0.3) .. controls (2.6,2.0) and (5.0,2.9) .. (8.0,3.35);
\node[black, anchor=north west] at (5.4,2.75) {decelerating model};
% accelerating model (higher, acc solid)
\draw[acc, very thick] (0.2,0.3) .. controls (2.6,2.3) and (5.0,3.4) .. (8.0,4.1);
\node[acc, anchor=south east] at (7.4,3.95) {accelerating model};
% data points tracking the accelerating curve
\foreach \x/\y in {0.9/1.0, 1.7/1.75, 2.6/2.35, 3.5/2.85, 4.6/3.35, 5.6/3.65, 6.6/3.85, 7.4/4.0}
  \fill[black] (\x,\y) circle (0.06);
\end{tikzpicture}
$$

## The cosmological constant and vacuum energy

The simplest component with the required negative pressure is a **cosmological constant**
$\Lambda$. Einstein introduced $\Lambda$ in 1917 as a term in the field equations to
permit a static universe, then discarded it after Hubble's discovery of expansion. It
reappears now as the leading explanation for acceleration. Written as a modification of
the field equations it adds a term proportional to the metric; equivalently, it can be
moved to the matter side and interpreted as a **vacuum energy density**

$$
\varepsilon_\Lambda = \frac{\Lambda c^{4}}{8\pi G} = \text{const} ,
$$

a constant energy density filling all of space. Because the vacuum must look identical to
every inertial observer, its stress-energy is forced to be Lorentz invariant, and the only
such form has pressure equal to minus the energy density:

$$
P_\Lambda = -\varepsilon_\Lambda , \qquad w_\Lambda = -1 .
$$

This negative pressure supplies what the acceleration equation needs:
$\varepsilon_\Lambda + 3 P_\Lambda = -2\varepsilon_\Lambda < 0$, so a vacuum-dominated
universe has $\ddot a > 0$. Because $\varepsilon_\Lambda$ does not dilute while matter and
radiation do, the vacuum term is negligible early and inevitably dominant late: whatever
its value, there comes an epoch when the diluting matter falls below the constant vacuum
and acceleration begins. In the concordance model that transition happened at $z \approx
0.6$, and we live shortly after it. A cosmological constant with $\Omega_{\Lambda,0}
\approx 0.69$ fits the supernova Hubble diagram, the CMB, and the large-scale structure
data simultaneously.[^ry-lambda]

$$
% caption: Matter density dilutes as the inverse cube of the scale factor while the
% vacuum energy stays flat, so vacuum inevitably overtakes matter at late times.
\begin{tikzpicture}[scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (0,0) -- (8.6,0) node[right, black!70] {scale factor};
\draw[->, black] (0,0) -- (0,4.4) node[above, black!70] {energy density};
% matter diluting: falling curve
\draw[black, very thick, densely dashed] (0.5,4.1) .. controls (2.0,1.6) and (4.0,0.7) .. (8.0,0.25);
\node[black, anchor=north east] at (2.7,2.4) {matter};
% vacuum constant
\draw[acc, very thick] (0.4,1.1) -- (8.2,1.1);
\node[acc, anchor=south] at (6.6,1.15) {vacuum energy};
% crossover
\draw[black, densely dotted] (3.05,0) -- (3.05,1.3);
\node[black, anchor=north] at (3.05,-0.1) {acceleration begins};
\end{tikzpicture}
$$

## The cosmological-constant problem

If the vacuum carries energy, quantum field theory ought to predict how much. Every
quantum field has a zero-point energy, and summing the ground-state energies of the field
modes gives a vacuum energy density. The sum diverges and must be cut off at the scale
where the theory breaks down; taking the Planck scale as the cutoff yields an estimate

$$
\varepsilon_\Lambda^{\text{theory}} \sim \frac{E_{\text{Planck}}^{4}}{(\hbar c)^{3}}
\sim 10^{111}\ \text{J}\,\text{m}^{-3} ,
$$

while the observed value inferred from $\Omega_{\Lambda,0}$ is

$$
\varepsilon_\Lambda^{\text{obs}} \sim 10^{-9}\ \text{J}\,\text{m}^{-3} .
$$

The two differ by roughly $120$ orders of magnitude — often called the worst quantitative
prediction in physics. Even a cutoff at the far lower electroweak scale leaves a
discrepancy of some $50$ orders of magnitude. Some unknown mechanism must cancel the
enormous predicted vacuum energy almost, but not quite, to zero, leaving the tiny residue
we observe. Supersymmetry would cancel the bosonic and fermionic contributions exactly if
it were unbroken, but it is broken at accessible energies, so it cannot supply the full
cancellation. No accepted solution exists; the **cosmological-constant problem** is the
gap between the natural theoretical scale and the observed value.[^cc-problem]

$$
% caption: The vacuum energy predicted by quantum field theory exceeds the observed
% value by about one hundred twenty orders of magnitude.
\begin{tikzpicture}[scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
% vertical log axis
\draw[->, black] (0,-0.3) -- (0,4.4) node[above, black!70] {log energy density};
% predicted bar (very high)
\draw[acc, very thick] (1.6,0) -- (1.6,4.0);
\draw[acc, very thick] (1.35,4.0) -- (1.85,4.0);
\node[acc, anchor=south] at (1.6,4.05) {predicted};
% observed bar (very low)
\draw[black, very thick] (5.0,0) -- (5.0,0.4);
\draw[black, very thick] (4.75,0.4) -- (5.25,0.4);
\node[black, anchor=south] at (5.0,0.45) {observed};
% gap annotation
\draw[<->, black] (3.2,0.4) -- (3.2,4.0);
\node[black, anchor=west, align=left] at (3.35,2.2) {gap of about\\120 orders};
\end{tikzpicture}
$$

## The coincidence problem and quintessence

A second, subtler puzzle is the **coincidence problem**. The matter density scales as
$a^{-3}$ and the vacuum density is constant, so their ratio $\Omega_m/\Omega_\Lambda$
sweeps from enormous in the past to negligible in the future, passing through order unity
only during a brief cosmic window. We happen to live during exactly that window, when
$\Omega_m$ and $\Omega_\Lambda$ are comparable. With a true constant $\Lambda$ this is a
coincidence with no explanation: the epoch of comparability is not tied to anything about
observers or structure. Some regard it as a hint that dark energy is not a constant but a
**dynamical** field whose evolution tracks the matter density, making the near-equality
natural rather than accidental.

The dynamical alternative is **quintessence**: a slowly evolving scalar field $\phi$ with a
potential $V(\phi)$, whose energy density $\varepsilon_\phi = \tfrac{1}{2}\dot\phi^{2} +
V(\phi)$ and pressure $P_\phi = \tfrac{1}{2}\dot\phi^{2} - V(\phi)$ give an equation of
state

$$
w_\phi = \frac{\tfrac{1}{2}\dot\phi^{2} - V(\phi)}{\tfrac{1}{2}\dot\phi^{2} + V(\phi)} ,
$$

which approaches $-1$ when the potential dominates the kinetic energy (a slowly rolling
field) but can differ from $-1$ and vary with time. Quintessence models can be tuned so
that the field naturally comes to dominate near the present epoch, addressing the
coincidence, at the cost of introducing a new field and a finely shaped potential. The
observational signature that would distinguish quintessence from a pure constant is a value
of $w$ different from $-1$, or a $w$ that changes with redshift.[^ry-quint]

## Current constraints on the equation of state

Whether dark energy is a constant or a dynamical field is an empirical question, addressed
by combining the probes that each constrain the expansion history differently. Type Ia
supernovae fix the low-redshift $d_L(z)$; the CMB acoustic peaks fix the geometry and the
matter density at $z \approx 1100$; baryon acoustic oscillations in galaxy surveys provide
a standard ruler at intermediate redshift. Individually each probe leaves a degeneracy in
the $\Omega_m$–$\Omega_\Lambda$ plane, but the degeneracies point in different directions,
so their intersection pins both parameters. The combination converges on a flat universe
with $\Omega_{m,0} \approx 0.31$ and $\Omega_{\Lambda,0} \approx 0.69$.

Allowing $w$ to float rather than fixing it at $-1$, the same combination of data
constrains it to

$$
w = -1.03 \pm 0.03 ,
$$

fully consistent with a cosmological constant and leaving little room for a strongly
evolving quintessence field. No significant evidence for $w \neq -1$ or for time
variation has emerged, so the concordance model retains a pure $\Lambda$. The situation is
thus paradoxical: the cosmological constant fits every observation and yet is
theoretically unexplained, its magnitude a $120$-order-of-magnitude mystery and its
present dominance an unexplained coincidence. Sharpening the measurement of $w$ and its
possible evolution is the central goal of the next generation of surveys, and the nature
of dark energy stands as the outstanding open problem carried into the
[hot-Big-Bang and open-questions](/astrophysics-cosmology/the-hot-big-bang/the-thermal-history-of-the-universe)
module.[^planck-w]

$$
% caption: Supernova, CMB, and BAO constraints intersect in the density plane; jointly
% they select a flat, accelerating universe with the vacuum term dominant.
\begin{tikzpicture}[scale=1.0, font=\footnotesize]
\definecolor{acc}{HTML}{4A6FA5}
\draw[->, black] (0,0) -- (5.8,0) node[right, black!70] {matter density};
\draw[->, black] (0,0) -- (0,5.0) node[above, black!70] {vacuum density};
% flat line: Omega_m + Omega_Lambda = 1
\draw[black, densely dashed] (0,4.6) -- (4.6,0);
\node[black, anchor=west, rotate=-45] at (2.7,2.5) {f\/lat};
% SN contour (elongated diagonal band), acc outline
\draw[acc, very thick, rotate around={40:(1.5,3.0)}] (1.5,3.0) ellipse (1.7 and 0.45);
\node[acc, anchor=east] at (0.7,4.2) {SNe};
% CMB contour (perpendicular band), neutral
\draw[black, very thick, densely dashed, rotate around={-50:(1.5,3.0)}] (1.5,3.0) ellipse (1.9 and 0.4);
\node[black, anchor=west] at (3.0,4.0) {CMB};
% BAO contour, dotted
\draw[black, very thick, densely dotted, rotate around={10:(1.5,3.0)}] (1.5,3.0) ellipse (1.5 and 0.5);
\node[black, anchor=north] at (3.3,2.1) {BAO};
% intersection marker
\fill[acc] (1.5,3.0) circle (0.1);
\node[black!70, anchor=west] at (1.65,3.25) {joint};
\end{tikzpicture}
$$

## Summary

The 1998 Type Ia supernova Hubble diagram showed distant supernovae fainter than any
decelerating model, demonstrating that the expansion accelerates and requiring a component
with negative pressure, $\varepsilon + 3P < 0$. The leading candidate is the cosmological
constant, or vacuum energy, with constant density and equation of state $w = -1$, which is
subdominant early and inevitably dominant late; with $\Omega_{\Lambda,0} \approx 0.69$ it
fits supernovae, the CMB, and large-scale structure together. But it carries two crises:
the cosmological-constant problem, a $\sim 120$-order-of-magnitude gap between the vacuum
energy quantum field theory predicts and the value observed, and the coincidence problem,
the unexplained near-equality of matter and vacuum densities today. Dynamical
alternatives such as quintessence — a slowly rolling scalar field with $w$ near but not
exactly $-1$ — could ease the coincidence, but current data give $w = -1.03 \pm 0.03$,
consistent with a pure constant and offering no evidence for evolution. Dark energy thus
remains the deepest open problem in cosmology: observationally pinned, theoretically
unexplained.

[^sn-discovery]: Perlmutter et al. (1999); Riess et al. (1998) — Type Ia supernova evidence for an accelerating universe, https://arxiv.org/abs/astro-ph/9812133.
[^ry-lambda]: Ryden, *Introduction to Cosmology*, Ch. 5 §5.5 — the cosmological constant, vacuum energy, and $w = -1$.
[^cc-problem]: Ryden, Ch. 5 — the cosmological-constant problem and the vacuum-energy discrepancy.
[^ry-quint]: Ryden, Ch. 5–6 — dynamical dark energy (quintessence) and the coincidence problem.
[^planck-w]: Planck Collaboration (2018) — constraints on $\Omega_m$, $\Omega_\Lambda$, and $w$, https://arxiv.org/abs/1807.06209.