Storage Technologies and the Latency Gap

A programmer would like one memory that is enormous, instant, and free. Physics allows any two of those at once and never all three: the cell that answers in a fraction of a nanosecond is large in area and dear per bit, and the cell that is small and cheap is slow. This module is built entirely on that tension. Before we can stack storage into a hierarchy, we need to know the materials: what a machine can store a bit in, and how the choices rank on speed, density, and cost. And we need to see the problem the hierarchy exists to solve: the processor has been pulling away from memory for fifty years.

Two kinds of RAM

The previous module built both volatile memory cells from transistors. SRAM stores a bit in a six-transistor cross-coupled latch: it holds its value as long as power is applied, needs no refresh, and switches in well under a nanosecond, but six transistors per bit is expensive in silicon area. DRAM stores a bit as charge on a single tiny capacitor behind one access transistor: one device per bit makes it dense and cheap, but the capacitor leaks, so every cell must be refreshed every few milliseconds, and a read is destructive and must be restored. That single trade, transistor count per bit, sets everything downstream.

So SRAM is the technology of choice where speed dominates and the byte count is small — the register file and the on-chip caches — while DRAM, dense and cheap, holds the gigabytes of main memory. Neither is persistent: cut the power and both forget. For storage that survives a reboot we need a different cell entirely.

Inside a DRAM chip

DRAM is slow hides structure worth knowing, because the structure explains why some DRAM accesses are far cheaper than others. The cells in a DRAM chip are not a flat line of bytes; they form a two-dimensional grid of supercells, $r$ rows by $c$ columns, each supercell holding a few bits (commonly 8). Addressing a supercell takes two steps. The memory controller first sends a row address; the chip's row decoder activates that row, and the contents of the entire row are copied into an internal row buffer. The controller then sends a column address, and the chip selects the requested supercell out of the buffer and drives it onto the data pins. Splitting the address in two halves lets the row and column share one set of address pins, which keeps the package small and cheap, the same economy that gave DRAM one transistor per bit.

The row buffer is the detail that matters for performance. Activating a row is the slow part of the access: it involves sensing the tiny capacitor charges, which destroys them, and restoring them afterward. But once a row sits in the buffer, further accesses to the same row are answered straight out of the buffer at a fraction of the cost. A random access that lands in a fresh row pays full price; a sequential run of accesses marches through one row after another, paying the activation once per row and reading many supercells from each. Real DRAM chips push this further. Banks are several independent grids, each with its own row buffer, so the controller can activate a row in one bank while reading from another; burst transfers stream a run of consecutive columns per request. All of these favor the same pattern: neighboring addresses, accessed together.

Rotating disks

A rotating disk stores bits as magnetized regions on one or more spinning platters. Each platter surface is divided into thousands of concentric rings called tracks, and each track into sectors of (typically) 512 bytes, the smallest unit a disk reads or writes. A read/write head, one per surface, rides on an arm that swings all heads in unison to a given track.

Getting to a byte is a mechanical process with three parts. Total access time is their sum:

Take $R=7,200$ RPM, an average seek of 4 ms, and $N=500$ sectors per track. A revolution takes $\frac{60}{7200}=8.33$ ms, so $T_{\text{rot}}\approx 4.17$ ms and $T_{\text{xfer}}=8.33/500\approx 0.017$ ms $=17\mu \text{s}$. Hence

Read the proportions, not just the sum: the 512-byte payload is $0.017/8.2\approx$0.2 % of the time; positioning the head is 99.8 %, and positioning is paid once per location, not once per byte. After one sector, the next arrives $17\mu \text{s}$ later for free. Bandwidth is therefore a strong function of access pattern:

Reading a 250 KB track (500 sectors) sequentially costs one seek plus one revolution; reading the same bytes as 500 scattered sectors costs $500\times 8.2$ ms $\approx 4.1$ s. Same disk, same bytes, a 500x gap: $B_{\text{seq}}/B_{\text{rand}}\approx 500$.

Solid-state drives

A solid-state drive (SSD) replaces the platter with flash memory: floating-gate transistors that trap charge and hold it without power. There are no moving parts, so a read takes tens of microseconds, between DRAM and disk. But flash has a granularity constraint that shapes how SSDs behave. Data is read and written in pages of a few kilobytes, and pages are grouped into blocks of roughly 32 to 128 pages, and a page can only be written after its entire block has been erased. Erasing a block is slow, around a millisecond, and a block survives only a finite number of erase cycles (on the order of tens of thousands) before it wears out.

This is why SSD writes are more expensive than reads. Overwriting a page in place would mean erasing and rewriting its whole block, so the drive's controller (the flash translation layer) does something else: it writes the new data to a fresh, already-erased page somewhere else, remaps the logical address to it, and marks the old page stale. Stale pages accumulate until the controller must copy a block's live pages elsewhere and erase it: background garbage collection that can turn one logical write into several physical ones. The controller also spreads writes evenly across all blocks (wear leveling) so no single block burns through its erase budget while others sit idle; done well, wear leveling turns each block dies after 100,000 erases into a drive lifetime measured in years of continuous writing.

The one logical write, several physical writes overhead can be quantified. To write 4 KB of new data the drive must first relocate a block's worth of live pages. Take a 256 KB block that is 75 % live: garbage-collecting it copies $0.75\times 256=192$ KB of survivors before the erase, then writes the 4 KB. The drive moved $192+4=196$ KB of flash to satisfy $4$ KB from the host, a write amplification factor

Real drives keep WAF in the single digits by holding spare erased blocks in reserve (over-provisioning) and garbage-collecting the emptiest blocks first, but the arithmetic explains two facts every SSD exhibits: writes to a nearly-full drive slow sharply (fewer empty blocks, more relocation per write), and endurance is the raw flash endurance divided by WAF, $\text{host bytes}=\text{flash endurance}/\text{WAF}$.

Sequential beats random, everywhere

The same pattern holds across all three technologies. DRAM serves a second access to an open row straight from the row buffer, so sequential runs are several times cheaper than row-hopping randoms. Disk pays seek and rotation once per location, so sequential transfer is hundreds of times cheaper than scattered sectors. Flash reads whole pages and its controller streams consecutive pages across internal channels in parallel. At every level of the hierarchy, contiguous access is cheaper than scattered access. That shared physics is why the hierarchy moves data in blocks rather than bytes, and it is half of why locality (the subject of the next lesson) is so effective.

Ranking the technologies

Lay the technologies on a single scale and a clean ordering appears: as you trade speed for cost per bit, capacity grows by orders of magnitude at each step. The numbers below are rough and date quickly, but the ratios — and the empty decades of latency between DRAM and disk — are the durable lesson.

Stretched on a logarithmic axis, the access times span roughly seven decades from a register to a disk seek, a gap too wide for any linear plot. The jump from the SRAM caches to DRAM is large; the jump from DRAM to disk is far larger, which is why software works hard to avoid going to disk.

The pyramid compresses to one sentence: everything above is faster, smaller, and more expensive per bit than everything below. The processor accesses only the very top; each level acts as a cache, a fast staging area, for the larger, slower level beneath it.

The processor–memory gap

None of this would force a hierarchy if processors and memory had improved in lockstep. They did not. For decades processor speed roughly doubled every couple of years, while DRAM latency improved glacially. DRAM got far denser and far cheaper (capacity per dollar improved by factors in the tens of thousands), but the time to fetch a random word barely moved. Rough milestones:

In 1985 a memory access cost about one instruction's worth of time; the processor and memory were peers. Thirty-five years later the same access costs the time of hundreds of instructions, and a superscalar core that issues several instructions per cycle loses even more.

A modern core issues several instructions per nanosecond; a miss all the way to DRAM costs on the order of $100$ ns, hundreds of instruction slots. The stall cost of a reference stream with miss rate $m$ and miss penalty $T_{\text{miss}}$ is $T_{\text{avg}}=T_{\text{hit}}+m{T}_{\text{miss}}$; at $m=1$ the machine spends almost all its time waiting and faster clocks buy almost nothing. The hierarchy is the answer to the gap: keep the working data in fast storage near the top, driving $m$ toward zero, and pay DRAM or disk latency only on the rare access that falls through.

The gap did not stop widening

CS:APP's storage chapter describes the classical pyramid — SRAM, DRAM, flash, disk. The rungs have since multiplied, and each addition is a direct response to a gap the pyramid still had.

The interface caught up to the medium. Early SSDs were bolted onto the SATA interface designed for disks, which capped them near 550 MB/s and added a thick software stack per request. NVMe (Non-Volatile Memory Express, 2011) replaced it with a protocol native to flash: the drive sits directly on PCI Express, and instead of one command queue it exposes thousands of deep queues that a multicore host can fill in parallel.¹ The medium had been fast for a decade; NVMe let software actually reach it, moving consumer drives from hundreds of MB/s to several GB/s with far lower per-request overhead.

A rung appeared between DRAM and flash. The DRAM-to-flash step is still a 100-fold latency jump, and two technologies tried to fill it. Persistent memory — Intel's Optane, built on 3D XPoint — was byte-addressable like DRAM but non-volatile like flash, with latency a few times DRAM's rather than flash's thousands (Izraelevitz et al. measured it at scale, 2019).² It let a program treat persistent data as ordinary memory, no block I/O, and forced a rethink of what durable means at the load/store level. The product line was discontinued in 2022, but it proved out the architectural niche, and the research on programming a persistent memory tier outlived the hardware.

A rung appeared above DRAM. At the top, the processor-memory bandwidth gap (distinct from the latency gap this lesson traced) drove High-Bandwidth Memory (HBM): DRAM dies stacked vertically and connected by through-silicon vias, sitting in the same package as the processor on a silicon interposer. Stacking trades a little capacity for an enormous, wide bus — terabytes per second — which is why every serious GPU and AI accelerator now ships with HBM rather than the commodity DIMMs a CPU uses. The hierarchy has not settled into four fixed rungs; new ones appear wherever a latency or bandwidth gap is wide enough to justify the silicon.

A hierarchy only pays off if accesses cluster — if the next reference is likely near a recent one. That property has a name and a structure, and it is the subject of the next lesson.