--- title: Storage Technologies and the Latency Gap module: The Memory Hierarchy moduleNumber: 6 lessonNumber: 1 order: 601 summary: > No single memory is both fast and large and cheap. We survey the technologies a machine can store bits in — SRAM, DRAM, flash, and rotating disk — open up a DRAM chip to find the row buffer, work a disk access down to the millisecond, and rank everything by speed, density, and cost per bit. Then we watch the processor outrun memory decade after decade. That widening gap is the whole reason a machine stacks fast small storage on top of slow large storage into a hierarchy. topics: [The Memory Hierarchy] sources: - book: Bryant & O'Hallaron ref: "CS:APP — §6 The Memory Hierarchy" - book: Bistriceanu ref: "Computer Architecture Notes — §7–9 The Memory Hierarchy / The Cache / Main Memory" --- A programmer would like one memory that is enormous, instant, and free. Physics allows any two of those at once and never all three: the cell that answers in a fraction of a nanosecond is large in area and dear per bit, and the cell that is small and cheap is slow. This module is built entirely on that tension. Before we can stack storage into a **hierarchy**, we need to know the materials: what a machine can store a bit in, and how the choices rank on speed, density, and cost. And we need to see the problem the hierarchy exists to solve: the processor has been pulling away from memory for fifty years. ## Two kinds of RAM The [previous module](/computer-architecture/digital-logic/register-files-and-random-access-memory) built both volatile memory cells from transistors. **SRAM** stores a bit in a six-transistor cross-coupled latch: it holds its value as long as power is applied, needs no refresh, and switches in well under a nanosecond, but six transistors per bit is expensive in silicon area. **DRAM** stores a bit as charge on a single tiny capacitor behind one access transistor: one device per bit makes it dense and cheap, but the capacitor leaks, so every cell must be **refreshed** every few milliseconds, and a read is **destructive** and must be restored. That single trade, transistor count per bit, sets everything downstream. | Property | SRAM | DRAM | | --- | --- | --- | | Transistors / bit | 6 | 1 (+ capacitor) | | Relative speed | ~10x faster | baseline | | Relative cost / bit | ~100x more | baseline | | Needs refresh? | no | yes | | Typical use | registers, caches | main memory | So SRAM is the technology of choice where speed dominates and the byte count is small — the register file and the on-chip caches — while DRAM, dense and cheap, holds the gigabytes of main memory. Neither is persistent: cut the power and both forget. For storage that survives a reboot we need a different cell entirely. > **Definition (Volatile vs non-volatile).** A **volatile** memory loses its > contents when power is removed (SRAM, DRAM). A **non-volatile** memory retains > them (flash, magnetic disk, ROM). Main memory is volatile; the file system lives > on non-volatile storage. ## Inside a DRAM chip "DRAM is slow" hides structure worth knowing, because the structure explains why some DRAM accesses are far cheaper than others. The cells in a DRAM chip are not a flat line of bytes; they form a two-dimensional grid of **supercells**, $r$ rows by $c$ columns, each supercell holding a few bits (commonly 8). Addressing a supercell takes **two** steps. The memory controller first sends a **row address**; the chip's row decoder activates that row, and the contents of the _entire row_ are copied into an internal **row buffer**. The controller then sends a **column address**, and the chip selects the requested supercell out of the buffer and drives it onto the data pins. Splitting the address in two halves lets the row and column share one set of address pins, which keeps the package small and cheap, the same economy that gave DRAM one transistor per bit. $$ % caption: Anatomy of a DRAM chip. The row address activates one row of the supercell % caption: grid, and the whole row is copied into the internal row buffer; the column % caption: address then picks one supercell out of the buffer. A second access to the % caption: same row skips the expensive row activation entirely. \begin{tikzpicture}[font=\footnotesize,>=stealth] \definecolor{acc}{HTML}{2348F2} % 4x4 supercell grid; bottom row (r=3) highlighted as the activated row \foreach \r in {0,1,2,3} \foreach \c in {0,1,2,3} { \ifnum\r=3 \draw[fill=acc!8] (\c*0.9,-\r*0.9) rectangle ++(0.7,0.7); \else \draw (\c*0.9,-\r*0.9) rectangle ++(0.7,0.7); \fi } \node[anchor=south,black] at (1.7,0.85) {supercell grid (r rows x c cols)}; % row decoder at left, one select line per row \node[draw, minimum width=11mm, minimum height=32mm, align=center] (rdec) at (-1.7,-1.0) {row\\dec}; \foreach \r in {0,1,2} \draw[black] (-1.15,-\r*0.9+0.35) -- (-0.1,-\r*0.9+0.35); \draw[->,acc,thick] (-1.15,-2.35) -- (-0.1,-2.35); \draw[->] (-4.1,-1.0) -- (rdec.west) node[midway,anchor=south] {row addr}; % row buf\/fer below the grid, fed by the activated row \node[draw, fill=acc!8, minimum width=36mm, minimum height=6.5mm] (rbuf) at (1.7,-3.75) {row buf\/fer}; \draw[->,acc,thick] (1.7,-2.75) -- (rbuf.north); % column select to the right of the buf\/fer \node[draw, minimum width=17mm, minimum height=6.5mm, align=center] (csel) at (5.9,-3.75) {column select}; \draw[->] (rbuf.east) -- (csel.west); \draw[->] (5.9,-5.0) -- (csel.south) node[midway,anchor=west] {col addr}; \draw[->] (csel.east) -- ++(1.0,0) node[anchor=west,text=acc] {data}; \end{tikzpicture} $$ The row buffer is the detail that matters for performance. Activating a row is the slow part of the access: it involves sensing the tiny capacitor charges, which destroys them, and restoring them afterward. But once a row sits in the buffer, further accesses to _the same row_ are answered straight out of the buffer at a fraction of the cost. A random access that lands in a fresh row pays full price; a sequential run of accesses marches through one row after another, paying the activation once per row and reading many supercells from each. Real DRAM chips push this further. **Banks** are several independent grids, each with its own row buffer, so the controller can activate a row in one bank while reading from another; **burst transfers** stream a run of consecutive columns per request. All of these favor the same pattern: neighboring addresses, accessed together. ## Rotating disks A **rotating disk** stores bits as magnetized regions on one or more spinning **platters**. Each platter surface is divided into thousands of concentric rings called **tracks**, and each track into **sectors** of (typically) 512 bytes, the smallest unit a disk reads or writes. A read/write **head**, one per surface, rides on an **arm** that swings all heads in unison to a given track. $$ % caption: Disk geometry. Bits live on concentric tracks divided into sectors; the % caption: arm swings the head to a track (the seek) and the platter's rotation brings % caption: the target sector under the head (rotational latency). \begin{tikzpicture}[font=\footnotesize,>=stealth] \definecolor{acc}{HTML}{2348F2} % platter and tracks \draw[thick] (0,0) circle (2.3); \draw[black] (0,0) circle (0.8); \draw[black] (0,0) circle (1.3); \draw[acc,thick] (0,0) circle (1.8); \fill[black] (0,0) circle (0.16); % one sector on the highlighted track: a wedge between two radii \fill[acc!25] (25:1.55) arc (25:52:1.55) -- (52:2.05) arc (52:25:2.05) -- cycle; \draw[acc] (25:1.55) arc (25:52:1.55) -- (52:2.05) arc (52:25:2.05) -- cycle; % rotation arrow near the top, outside the platter \draw[->,black] (118:2.55) arc (118:152:2.55); \node[black,anchor=south east] at (140:2.7) {rotation}; % arm from a pivot at lower right to the head on the highlighted track \fill[black] (3.7,-2.1) circle (0.14); \draw[line width=1.6pt,black] (3.7,-2.1) -- (-15:1.86); \fill[acc] (-15:1.8) circle (0.10); % labels with leader lines, all outside the platter \node[anchor=west] (tlab) at (3.1,1.4) {track}; \draw[black] (tlab.west) -- (38:2.32); \node[anchor=west,text=acc] (slab) at (3.1,0.6) {sector}; \draw[black] (slab.west) -- (36:1.85); \node[anchor=north] at (3.7,-2.35) {arm}; \node[anchor=north west] at (0.3,-2.42) {head}; \draw[black] (0.62,-2.35) -- (-15:1.88); \node[anchor=east] (spin) at (-2.9,-0.9) {spindle}; \draw[black] (spin.east) -- (-0.12,-0.12); \end{tikzpicture} $$ Getting to a byte is a mechanical process with three parts. Total access time is their sum: $$ T_{\text{access}} = T_{\text{seek}} + T_{\text{rot}} + T_{\text{xfer}}. $$ > **Definition (Seek time, rotational latency, transfer time).** **Seek time** > $T_{\text{seek}}$ swings the arm to the target track, a few milliseconds on > average. **Rotational latency** $T_{\text{rot}}$ waits for the platter to spin > the target sector under the head; for a disk at $R$ revolutions per minute, > $T_{\text{rot}} \approx \tfrac{1}{2}\cdot\tfrac{60}{R}$ (half a revolution on > average). **Transfer time** $T_{\text{xfer}} = \tfrac{60}{R}\cdot\tfrac{1}{N}$ is > the time for one of $N$ sectors on a track to pass under the head. Take $R = 7{,}200$ RPM, an average seek of 4 ms, and $N = 500$ sectors per track. A revolution takes $\tfrac{60}{7200} = 8.33$ ms, so $T_{\text{rot}} \approx 4.17$ ms and $T_{\text{xfer}} = 8.33 / 500 \approx 0.017$ ms $= 17\,\mu\text{s}$. Hence $$ T_{\text{access}} \approx 4 + 4.17 + 0.017 \approx 8.2\ \text{ms}. $$ Read the proportions, not just the sum: the 512-byte payload is $0.017/8.2 \approx$ **0.2 %** of the time; positioning the head is 99.8 %, and positioning is paid once per _location_, not once per byte. After one sector, the next arrives $17\,\mu\text{s}$ later for free. Bandwidth is therefore a strong function of access pattern: $$ B = \frac{\text{bytes}}{T_{\text{access}}}, \qquad B_{\text{seq}} = \frac{250\,\text{KB}}{8.33\,\text{ms}} \approx 30\ \text{MB/s}, \qquad B_{\text{rand}} = \frac{512\,\text{B}}{8.2\,\text{ms}} \approx 60\ \text{KB/s}. $$ Reading a 250 KB track (500 sectors) sequentially costs one seek plus one revolution; reading the same bytes as 500 scattered sectors costs $500 \times 8.2$ ms $\approx 4.1$ s. Same disk, same bytes, a **500x** gap: $B_{\text{seq}} / B_{\text{rand}} \approx 500$. $$ % caption: Where a random disk access spends its time: seek plus rotational latency % caption: dwarf the transfer (the transfer sliver is drawn wider than scale). A % caption: sequential access to the next sector pays only the transfer. \begin{tikzpicture}[font=\footnotesize,>=stealth] \definecolor{acc}{HTML}{2348F2} % random access bar \node[anchor=east] at (-0.25,0.55) {random access}; \draw[fill=acc!8] (0,0.25) rectangle (3.7,0.85); \node at (1.85,0.55) {seek 4 ms}; \draw[fill=acc!20] (3.7,0.25) rectangle (7.55,0.85); \node at (5.6,0.55) {rotation 4.2 ms}; \draw[fill=acc!50] (7.55,0.25) rectangle (7.75,0.85); \node[anchor=west,text=acc] (tr) at (8.3,1.35) {transfer 17 us}; \draw[->,black] (tr.west) -- (7.68,0.9); % sequential access bar: transfer only \node[anchor=east] at (-0.25,-0.75) {next sector, sequential}; \draw[fill=acc!50] (0,-1.05) rectangle (0.2,-0.45); \node[anchor=west,text=acc] at (0.5,-0.75) {transfer only}; % time axis \draw[->] (0,-1.7) -- (8.6,-1.7) node[anchor=north east] {time}; \end{tikzpicture} $$ ## Solid-state drives A **solid-state drive (SSD)** replaces the platter with **flash memory**: floating-gate transistors that trap charge and hold it without power. There are no moving parts, so a read takes tens of microseconds, between DRAM and disk. But flash has a granularity constraint that shapes how SSDs behave. Data is read and written in **pages** of a few kilobytes, and pages are grouped into **blocks** of roughly 32 to 128 pages, and a page can only be written after its entire block has been **erased**. Erasing a block is slow, around a millisecond, and a block survives only a finite number of erase cycles (on the order of tens of thousands) before it wears out. This is why SSD writes are more expensive than reads. Overwriting a page in place would mean erasing and rewriting its whole block, so the drive's controller (the **flash translation layer**) does something else: it writes the new data to a fresh, already-erased page somewhere else, remaps the logical address to it, and marks the old page stale. Stale pages accumulate until the controller must copy a block's live pages elsewhere and erase it: background garbage collection that can turn one logical write into several physical ones. The controller also spreads writes evenly across all blocks (**wear leveling**) so no single block burns through its erase budget while others sit idle; done well, wear leveling turns "each block dies after 100,000 erases" into a drive lifetime measured in years of continuous writing. The "one logical write, several physical writes" overhead can be quantified. To write 4 KB of new data the drive must first relocate a block's worth of live pages. Take a 256 KB block that is 75 % live: garbage-collecting it copies $0.75 \times 256 = 192$ KB of survivors before the erase, then writes the 4 KB. The drive moved $192 + 4 = 196$ KB of flash to satisfy $4$ KB from the host, a **write amplification factor** $$ \text{WAF} = \frac{\text{physical bytes written}}{\text{logical bytes requested}} = \frac{196}{4} \approx 49. $$ Real drives keep WAF in the single digits by holding spare erased blocks in reserve (**over-provisioning**) and garbage-collecting the _emptiest_ blocks first, but the arithmetic explains two facts every SSD exhibits: writes to a nearly-full drive slow sharply (fewer empty blocks, more relocation per write), and endurance is the raw flash endurance divided by WAF, $\text{host bytes} = \text{flash endurance} / \text{WAF}$. > **Definition (Write amplification).** $\text{WAF} = (\text{physical bytes written > to flash}) / (\text{logical bytes requested by the host})$. Erase-before-write > granularity and garbage collection push it above 1; over-provisioning and workload > locality pull it back down. It multiplies wear and divides endurance. ## Sequential beats random, everywhere The same pattern holds across all three technologies. DRAM serves a second access to an open row straight from the row buffer, so sequential runs are several times cheaper than row-hopping randoms. Disk pays seek and rotation once per location, so sequential transfer is hundreds of times cheaper than scattered sectors. Flash reads whole pages and its controller streams consecutive pages across internal channels in parallel. **At every level of the hierarchy, contiguous access is cheaper than scattered access.** That shared physics is why the hierarchy moves data in blocks rather than bytes, and it is half of why locality (the subject of the [next lesson](/computer-architecture/memory-hierarchy/locality)) is so effective. > **Definition (Latency vs throughput).** **Latency** is the time to satisfy one > access — what you wait for a single byte. **Throughput** (bandwidth) is bytes > per second under a stream of accesses. A disk has terrible latency (a seek of > milliseconds) but respectable throughput once the head is in place, because the > hierarchy moves data in big contiguous chunks. ## Ranking the technologies Lay the technologies on a single scale and a clean ordering appears: as you trade speed for cost per bit, capacity grows by orders of magnitude at each step. The numbers below are rough and date quickly, but the **ratios** — and the empty decades of latency between DRAM and disk — are the durable lesson. | Level | Technology | Typical latency | Typical capacity | | --- | --- | --- | --- | | Registers | SRAM (flip-flops) | sub-ns | ~1 KB | | L1 cache | SRAM | ~1 ns | ~32 KB | | L2 cache | SRAM | ~4 ns | ~256 KB–1 MB | | L3 cache | SRAM | ~10–20 ns | a few MB–tens of MB | | Main memory | DRAM | ~50–100 ns | GBs | | SSD | flash | ~50 µs | TBs | | Disk | magnetic | ~5–10 ms | TBs | Stretched on a logarithmic axis, the access times span roughly **seven decades** from a register to a disk seek, a gap too wide for any linear plot. The jump from the SRAM caches to DRAM is large; the jump from DRAM to disk is far larger, which is why software works hard to avoid going to disk. $$ % caption: Access time on a log scale, sub-nanosecond to milliseconds. Each step down % caption: the hierarchy is roughly an order of magnitude slower; the leap from DRAM to % caption: disk is the widest, about a million-fold. Labels sit at the bar ends. \begin{tikzpicture}[font=\footnotesize,>=stealth] \definecolor{acc}{HTML}{2348F2} % log axis: x position ~ log10(seconds), bars grow rightward from a common left edge % registers 1e-10, L1 1e-9, L3 1e-8, DRAM 1e-7, SSD 5e-5, disk 5e-3 \draw[->] (0,-0.3) -- (0,4.6); \draw[->] (0,-0.3) -- (9.2,-0.3); \node[anchor=north] at (4.4,-0.95) {latency (log scale)}; \foreach \y/\name/\len in { 4.0/registers/1.1, 3.2/{L1 cache}/2.0, 2.4/{L3 cache}/3.1, 1.6/{DRAM}/4.0, 0.8/{SSD}/6.4, 0.0/{disk}/8.6} { \fill[acc!20] (0,\y-0.18) rectangle (\len,\y+0.18); \draw (0,\y-0.18) rectangle (\len,\y+0.18); \node[anchor=west] at (\len+0.15,\y) {\name}; } % decade ticks \foreach \x/\lab in {0.0/{1 ns},2.0/{10 ns},4.0/{100 ns},6.4/{50 us},8.6/{5 ms}} \node[anchor=north,black] at (\x,-0.35) {\lab}; \end{tikzpicture} $$ $$ % caption: The memory hierarchy as a pyramid: small, fast, costly storage on top; % caption: large, slow, cheap storage below. Speed and cost per bit fall as you % caption: descend; capacity grows. Size/latency annotations sit to the side, never % caption: crossing the pyramid. \begin{tikzpicture}[font=\footnotesize,>=stealth] \definecolor{acc}{HTML}{2348F2} % pyramid: f\/ive trapezoidal levels stacked, widening downward % level i spans x in [-w_i, w_i] at vertical band [y_i, y_i+h] \def\h{0.92} \foreach \i/\wt/\wb/\name in { 0/0.55/1.25/{registers}, 1/1.25/2.35/{caches (SRAM)}, 2/2.35/3.45/{main memory (DRAM)}, 3/3.45/4.55/{SSD (f\/lash)}, 4/4.55/5.65/{disk (magnetic)}} { \pgfmathsetmacro{\yb}{-\i*\h} \pgfmathsetmacro{\yt}{\yb+\h} \ifnum\i=0 \fill[acc!8] (-\wt,\yt) -- (\wt,\yt) -- (\wb,\yb) -- (-\wb,\yb) -- cycle; \draw (-\wt,\yt) -- (\wt,\yt) -- (\wb,\yb) -- (-\wb,\yb) -- cycle; \else \draw (-\wt,\yt) -- (\wt,\yt) -- (\wb,\yb) -- (-\wb,\yb) -- cycle; \fi \node at (0,\yb+0.5*\h) {\name}; } % side annotations: faster/smaller at top, slower/larger at bottom \draw[->,acc] (6.6,0.46) -- (6.6,-3.22); \node[anchor=west,text=acc] at (6.75,0.46) {larger}; \node[anchor=west,text=acc] at (6.75,-3.22) {(GB-TB)}; \draw[->,acc] (-6.6,-3.22) -- (-6.6,0.46); \node[anchor=east,text=acc] at (-6.75,0.46) {faster}; \node[anchor=east,text=acc] at (-6.75,-3.22) {(ms)}; \end{tikzpicture} $$ The pyramid compresses to one sentence: **everything above is faster, smaller, and more expensive per bit than everything below.** The processor accesses only the very top; each level acts as a cache, a fast staging area, for the larger, slower level beneath it. ## The processor–memory gap None of this would force a hierarchy if processors and memory had improved in lockstep. They did not. For decades processor speed roughly doubled every couple of years, while DRAM **latency** improved glacially. DRAM got far denser and far cheaper (capacity per dollar improved by factors in the tens of thousands), but the time to fetch a random word barely moved. Rough milestones: | Year | Processor cycle (effective) | DRAM access | Cycles per DRAM access | | --- | --- | --- | --- | | 1985 | ~150 ns | ~200 ns | ~1 | | 2000 | ~1 ns | ~60 ns | ~60 | | 2020 | ~0.3 ns | ~50 ns | ~150+ | In 1985 a memory access cost about one instruction's worth of time; the processor and memory were peers. Thirty-five years later the same access costs the time of hundreds of instructions, and a superscalar core that issues several instructions per cycle loses even more. $$ % caption: The processor–memory performance gap (log scale, schematic). Processor % caption: speed climbs steeply while DRAM latency improves slowly, so the vertical % caption: distance between the two lines — the gap — widens every year. Labels sit % caption: beside the endpoints, off the curves. \begin{tikzpicture}[font=\footnotesize,>=stealth] \definecolor{acc}{HTML}{2348F2} % axes \draw[->] (0,0) -- (8.4,0) node[anchor=north east] {year}; \draw[->] (0,0) -- (0,4.7) node[anchor=south east] {relative speed (log)}; \foreach \x/\yr in {0.4/1985, 3.6/2000, 7.2/2020} \node[anchor=north] at (\x,-0.05) {\yr}; % processor line: steep \draw[acc,very thick] (0.4,0.7) -- (3.6,2.6) -- (7.2,4.3); % memory line: shallow \draw[black,very thick] (0.4,0.45) -- (3.6,0.95) -- (7.2,1.45); % endpoint labels, off the curves \node[text=acc,anchor=south west] at (7.2,4.3) {processor}; \node[black,anchor=north west] at (7.2,1.45) {DRAM}; % the gap, annotated to the side \draw[<->,dashed] (6.4,1.30) -- (6.4,3.92); \node[anchor=east,align=right] at (6.25,2.6) {the\\gap}; \end{tikzpicture} $$ A modern core issues several instructions per nanosecond; a miss all the way to DRAM costs on the order of $100$ ns, hundreds of instruction slots. The stall cost of a reference stream with miss rate $m$ and miss penalty $T_{\text{miss}}$ is $T_{\text{avg}} = T_{\text{hit}} + m\,T_{\text{miss}}$; at $m = 1$ the machine spends almost all its time waiting and faster clocks buy almost nothing. **The hierarchy is the answer to the gap:** keep the working data in fast storage near the top, driving $m$ toward zero, and pay DRAM or disk latency only on the rare access that falls through. ## The gap did not stop widening CS:APP's storage chapter describes the classical pyramid — SRAM, DRAM, flash, disk. The rungs have since multiplied, and each addition is a direct response to a gap the pyramid still had. **The interface caught up to the medium.** Early SSDs were bolted onto the SATA interface designed for disks, which capped them near 550 MB/s and added a thick software stack per request. **NVMe** (Non-Volatile Memory Express, 2011) replaced it with a protocol native to flash: the drive sits directly on PCI Express, and instead of one command queue it exposes thousands of deep queues that a multicore host can fill in parallel.[^nvme] The medium had been fast for a decade; NVMe let software actually reach it, moving consumer drives from hundreds of MB/s to several GB/s with far lower per-request overhead. **A rung appeared between DRAM and flash.** The DRAM-to-flash step is still a 100-fold latency jump, and two technologies tried to fill it. **Persistent memory** — Intel's Optane, built on 3D XPoint — was byte-addressable like DRAM but non-volatile like flash, with latency a few times DRAM's rather than flash's thousands (Izraelevitz et al. measured it at scale, 2019).[^optane] It let a program treat persistent data as ordinary memory, no block I/O, and forced a rethink of what "durable" means at the load/store level. The product line was discontinued in 2022, but it proved out the architectural niche, and the research on programming a persistent memory tier outlived the hardware. **A rung appeared above DRAM.** At the top, the processor-memory _bandwidth_ gap (distinct from the latency gap this lesson traced) drove **High-Bandwidth Memory (HBM)**: DRAM dies stacked vertically and connected by through-silicon vias, sitting in the same package as the processor on a silicon interposer. Stacking trades a little capacity for an enormous, wide bus — terabytes per second — which is why every serious GPU and AI accelerator now ships with HBM rather than the commodity DIMMs a CPU uses. The hierarchy has not settled into four fixed rungs; new ones appear wherever a latency or bandwidth gap is wide enough to justify the silicon. > **Takeaway.** Storage technologies trade speed against cost per bit: SRAM is > fast, small, and dear; DRAM is denser and cheaper but leaks and is slower; flash > and disk are non-volatile and vast but slower still — disk by a factor of a > million. Inside each technology, sequential access is cheaper: the DRAM row > buffer, the once-per-location seek, the flash page all favor neighboring > accesses. Across > decades the processor pulled hundreds of cycles ahead of DRAM latency, and that > gap is precisely what the memory hierarchy exists to hide. A hierarchy only pays off if accesses cluster — if the next reference is likely near a recent one. That property has a name and a structure, and it is the subject of the [next lesson](/computer-architecture/memory-hierarchy/locality). [^nvme]: **NVM Express base specification** (NVMe 1.0, 2011) — a host-controller interface for non-volatile storage over PCIe, with up to 64K deep parallel command queues, replacing the disk-era AHCI/SATA stack. [^optane]: **J. Izraelevitz et al.**, "Basic Performance Measurements of the Intel Optane DC Persistent Memory Module," arXiv:1903.05714, 2019 — the first thorough public measurements of byte-addressable persistent memory, latency a few times DRAM's, sitting between DRAM and flash.