Dark Energy and the Accelerating Universe

For most of the twentieth century the central question of cosmology was assumed to be whether the universe contained enough matter to halt its expansion and recollapse. The expected answer was that gravity decelerates the expansion; the only question was by how much. In 1998 two independent teams measuring distant Type Ia supernovae found the opposite: the expansion is accelerating. This lesson traces that discovery through the supernova Hubble diagram, identifies the responsible component as a cosmological constant or vacuum energy with equation of state $w\approx -1$, and confronts the two theoretical crises it created — the cosmological-constant problem, a discrepancy of some $120$ orders of magnitude between the observed and predicted vacuum energy, and the coincidence problem, that dark energy and matter happen to be comparable right now. It closes with the current observational constraints on $w$ and the alternatives to a pure constant.

The supernova Hubble diagram

Type Ia supernovae are the workhorse of the measurement because they are standardizable candles: after correcting for the empirical correlation between peak brightness and light-curve decline rate (the Phillips relation), their peak luminosities scatter by only about $0.15\text{mag}$, making them visible and calibratable to redshifts beyond $z\approx 1$. Measuring a supernova's apparent brightness gives its luminosity distance $d_L$; its host galaxy's spectrum gives its redshift $z$. Plotting $d_L$ against $z$ — or equivalently the distance modulus $m-M=5{\log }_{10}(d_L/10\text{pc})$ against redshift — tests the expansion history, because $d_L(z)$ depends on ${\Omega }_{m,0}$ and ${\Omega }_{\Lambda ,0}$ through the Friedmann integral of the previous lesson.

The prediction is clean. In a decelerating, matter-dominated universe the expansion was faster in the past, so a source at given redshift is relatively nearby and bright. In an accelerating universe the expansion was slower in the past, so the same redshift corresponds to a larger distance and a fainter source. The 1998 data showed distant supernovae systematically fainter — by about $0.25\text{mag}$, some 25% in flux — than the best-fitting decelerating model, and consistent instead with an accelerating universe. The deviation grows with redshift exactly as an accelerating $d_L(z)$ predicts. The result required a negative deceleration parameter, $q_0<0$, and hence a component with $\epsilon +3P<0$: negative pressure.¹

The cosmological constant and vacuum energy

The simplest component with the required negative pressure is a cosmological constant$\Lambda$. Einstein introduced $\Lambda$ in 1917 as a term in the field equations to permit a static universe, then discarded it after Hubble's discovery of expansion. It reappears now as the leading explanation for acceleration. Written as a modification of the field equations it adds a term proportional to the metric; equivalently, it can be moved to the matter side and interpreted as a vacuum energy density

a constant energy density filling all of space. Because the vacuum must look identical to every inertial observer, its stress-energy is forced to be Lorentz invariant, and the only such form has pressure equal to minus the energy density:

This negative pressure supplies what the acceleration equation needs: ${\epsilon }_{\Lambda }+3P_{\Lambda }=-2{\epsilon }_{\Lambda }<0$, so a vacuum-dominated universe has $\ddot{a}>0$. Because ${\epsilon }_{\Lambda }$ does not dilute while matter and radiation do, the vacuum term is negligible early and inevitably dominant late: whatever its value, there comes an epoch when the diluting matter falls below the constant vacuum and acceleration begins. In the concordance model that transition happened at $z\approx 0.6$, and we live shortly after it. A cosmological constant with ${\Omega }_{\Lambda ,0}\approx 0.69$ fits the supernova Hubble diagram, the CMB, and the large-scale structure data simultaneously.²

The cosmological-constant problem

If the vacuum carries energy, quantum field theory ought to predict how much. Every quantum field has a zero-point energy, and summing the ground-state energies of the field modes gives a vacuum energy density. The sum diverges and must be cut off at the scale where the theory breaks down; taking the Planck scale as the cutoff yields an estimate

while the observed value inferred from ${\Omega }_{\Lambda ,0}$ is

The two differ by roughly $120$ orders of magnitude — often called the worst quantitative prediction in physics. Even a cutoff at the far lower electroweak scale leaves a discrepancy of some $50$ orders of magnitude. Some unknown mechanism must cancel the enormous predicted vacuum energy almost, but not quite, to zero, leaving the tiny residue we observe. Supersymmetry would cancel the bosonic and fermionic contributions exactly if it were unbroken, but it is broken at accessible energies, so it cannot supply the full cancellation. No accepted solution exists; the cosmological-constant problem is the gap between the natural theoretical scale and the observed value.³

The coincidence problem and quintessence

A second, subtler puzzle is the coincidence problem. The matter density scales as $a^{-3}$ and the vacuum density is constant, so their ratio ${\Omega }_m/{\Omega }_{\Lambda }$ sweeps from enormous in the past to negligible in the future, passing through order unity only during a brief cosmic window. We happen to live during exactly that window, when ${\Omega }_m$ and ${\Omega }_{\Lambda }$ are comparable. With a true constant $\Lambda$ this is a coincidence with no explanation: the epoch of comparability is not tied to anything about observers or structure. Some regard it as a hint that dark energy is not a constant but a dynamical field whose evolution tracks the matter density, making the near-equality natural rather than accidental.

The dynamical alternative is quintessence: a slowly evolving scalar field $\varphi$ with a potential $V(\varphi )$, whose energy density ${\epsilon }_{\varphi }=\frac{1}{2}{\dot{\varphi }}^2+V(\varphi )$ and pressure $P_{\varphi }=\frac{1}{2}{\dot{\varphi }}^2-V(\varphi )$ give an equation of state

which approaches $-1$ when the potential dominates the kinetic energy (a slowly rolling field) but can differ from $-1$ and vary with time. Quintessence models can be tuned so that the field naturally comes to dominate near the present epoch, addressing the coincidence, at the cost of introducing a new field and a finely shaped potential. The observational signature that would distinguish quintessence from a pure constant is a value of $w$ different from $-1$, or a $w$ that changes with redshift.⁴

Current constraints on the equation of state

Whether dark energy is a constant or a dynamical field is an empirical question, addressed by combining the probes that each constrain the expansion history differently. Type Ia supernovae fix the low-redshift $d_L(z)$; the CMB acoustic peaks fix the geometry and the matter density at $z\approx 1100$; baryon acoustic oscillations in galaxy surveys provide a standard ruler at intermediate redshift. Individually each probe leaves a degeneracy in the ${\Omega }_m$–${\Omega }_{\Lambda }$ plane, but the degeneracies point in different directions, so their intersection pins both parameters. The combination converges on a flat universe with ${\Omega }_{m,0}\approx 0.31$ and ${\Omega }_{\Lambda ,0}\approx 0.69$.

Allowing $w$ to float rather than fixing it at $-1$, the same combination of data constrains it to

fully consistent with a cosmological constant and leaving little room for a strongly evolving quintessence field. No significant evidence for $w\ne -1$ or for time variation has emerged, so the concordance model retains a pure $\Lambda$. The situation is thus paradoxical: the cosmological constant fits every observation and yet is theoretically unexplained, its magnitude a $120$-order-of-magnitude mystery and its present dominance an unexplained coincidence. Sharpening the measurement of $w$ and its possible evolution is the central goal of the next generation of surveys, and the nature of dark energy stands as the outstanding open problem carried into the hot-Big-Bang and open-questions module.⁵

Summary

The 1998 Type Ia supernova Hubble diagram showed distant supernovae fainter than any decelerating model, demonstrating that the expansion accelerates and requiring a component with negative pressure, $\epsilon +3P<0$. The leading candidate is the cosmological constant, or vacuum energy, with constant density and equation of state $w=-1$, which is subdominant early and inevitably dominant late; with ${\Omega }_{\Lambda ,0}\approx 0.69$ it fits supernovae, the CMB, and large-scale structure together. But it carries two crises: the cosmological-constant problem, a $\sim 120$-order-of-magnitude gap between the vacuum energy quantum field theory predicts and the value observed, and the coincidence problem, the unexplained near-equality of matter and vacuum densities today. Dynamical alternatives such as quintessence — a slowly rolling scalar field with $w$ near but not exactly $-1$ — could ease the coincidence, but current data give $w=-1.03\pm 0.03$, consistent with a pure constant and offering no evidence for evolution. Dark energy thus remains the deepest open problem in cosmology: observationally pinned, theoretically unexplained.