Hardware Multithreading

The previous lesson ended with a machine that cannot clock its one instruction stream any faster. Before spending transistors on whole extra cores, there is a cheaper question: is the core we already have even busy? Mostly, no. A modern pipeline issues several instructions per cycle in bursts, then sits: an L2 miss from the memory hierarchy stalls a load for two hundred cycles, a mispredicted branch throws away everything in flight, a chain of dependent instructions serializes the issue slots. Averaged over real programs, most issue slots go empty. Hardware multithreading addresses the waste directly: hold the state of several threads in the core at once, and when one thread cannot use the hardware this cycle, hand it to another.

What a thread context costs

Holding an extra thread is affordable because of an asymmetry the first lesson set up: a thread's private state is small. To hold an extra thread a core must duplicate only the architectural state — the program counter, the general-purpose and condition-code registers, and in an out-of-order machine the register alias table (RAT) that maps those architectural names onto the big physical register file (register renaming is sketched in the capstone's honesty section). Everything expensive stays single: the fetch unit and decoders, the branch predictor, the schedulers, the ALUs and load/store units, and the whole cache hierarchy. The duplicated state is a few kilobits; the shared machinery is almost the entire core.

What a second hardware thread duplicates versus shares. Each context owns a PC, architectural registers, and a rename table; both feed one shared pool of schedulers, execution units, and caches.

The payoff structure follows from the split. Duplicating contexts is cheap, so vendors do it: Intel's hyperthreading runs two contexts per core. Sharing the execution machinery means the two threads are not independent the way two cores are: they compete for every shared resource, and that competition determines when multithreading helps and when it hurts.

Coarse-grained multithreading

The simplest policy switches threads only on a long stall. Thread A runs alone at full speed; when it misses in the last-level cache and faces a two-hundred-cycle DRAM access, the core switches to thread B and runs it until B stalls in turn. The switch is not free — the pipeline holding A's in-flight instructions must drain or be squashed before B's can enter, costing a few cycles — but a few cycles against a two-hundred-cycle miss is a fine trade.

Coarse-grained multithreading. Thread A runs until an LLC miss, the pipeline switches (a few dead cycles), and thread B covers the two hundred cycles A spends waiting on DRAM.

To quantify this, let $W$ be useful cycles between last-level misses, $M$ the miss penalty, and $s$ the switch cost. Single-thread utilization is $U_{1} = \frac{W}{W + M} .$ With $W = 100$ , $M = 200$ , $s = 10$ : $U_{1} = 100/300 \approx 33 %$ . Add thread B and switch to it on A's miss. B's work covers the stall, less two switches (out to B, back to A), so a window of $W + M$ cycles delivers $W + (M - 2 s)$ useful cycles: $U_{2} = \frac{W + ( M - 2 s )}{W + M} = \frac{100 + 180}{300} \approx 93 %.$ One extra context lifts throughput $U_{2} / U_{1} \approx 2.8 \times$ , at the cost of the duplicated architectural state alone. The larger the miss penalty $M$ , the more a second thread is worth.

Coarse-grained multithreading hides only the big latencies. The short stalls — a 3-cycle load-use bubble, a branch flush — stay unhidden, because a thread switch costs about as much as the stall it would cover. Single-thread speed is untouched (a thread that never misses runs exactly as before), which is the policy's main virtue.

Fine-grained multithreading

The opposite extreme switches threads every cycle, round-robin among the ready ones, with zero switch cost: the fetch stage just reads a different PC each cycle. Now even tiny stalls are covered: if thread A has a load-use bubble this cycle, threads B, C, D fill the slot before A's turn comes around again. Barrel processors and GPU warp schedulers push this to dozens of contexts.

The cost is single-thread latency. With $k$ contexts interleaving round-robin, each thread advances at most once per $k$ cycles, so a program running alone on a fine-grained machine runs at $1/ k$ speed while the other slots issue nothing. Fine-grained designs suit workloads with throughput to burn and no latency-critical single thread, which describes a GPU's situation and almost never a desktop's.

Fine-grained multithreading rotates threads every cycle. Short stalls in any one thread vanish into the rotation, but a single thread now advances at most once per round.

Simultaneous multithreading

Both schemes above share the machine in time: each cycle belongs to one thread. A wide superscalar core wastes issue bandwidth that way, because the issue stage can start four or more instructions per cycle and a single thread rarely has four independent instructions ready. Simultaneous multithreading (SMT) shares the machine in space as well: in one cycle, the scheduler fills issue slots with instructions from any ready thread, mixing them freely. Thread A's two ready instructions and thread B's two ready instructions issue together, in the same cycle, to different execution units.

Out-of-order machinery makes SMT nearly free to add. The core already renames architectural registers to physical ones, so instructions in the shared scheduler carry no thread identity that the ALUs care about; tagging each in-flight instruction with a context ID, doubling the rename table, and partitioning a few queues is most of the work. That is why hyperthreading costs a few percent of die area against roughly 15–30 % more throughput on mixed workloads.

The three disciplines differ only in whether cycles are shared in time or space, and at what switch granularity:

discipline	switch granularity	switch cost	slots/cycle	hides	single-thread speed
coarse-grained	on long stall	few cycles	one thread	long latencies only	unchanged
fine-grained	every cycle	zero	one thread	all stalls	$1/ k$
SMT	within a cycle	zero	multiple threads	stalls + issue-width waste	shared, degrades under contention

Issue slots (4 wide) cycle by cycle. Left: one thread leaves slots empty wherever it lacks independent work. Right: 2-way SMT fills the same slots with the second thread's ready instructions.

When SMT helps, and when it hurts

SMT helps only when the two threads' demands interleave rather than collide. Let a core issue $W$ instructions per cycle at peak and a lone thread sustain IPC $I_{1}$ ; utilization is $I_{1} / W$ . With two threads sustaining $I_{2}$ together, the SMT gain is $G = \frac{I _{2}}{I _{1}}, I_{2} \leq W .$ The gain tracks the empty slots a single thread leaves. Good case: $W = 4$ , $I_{1} = 1.6$ (utilization $40 %$ ), a second thread whose stalls fall in different cycles lifts the pair to $I_{2} = 2.9$ , so $G = 2.9/1.6 \approx 1.8 \times$ — most of a second core's worth, for a few percent of the die.

The bad case runs the same formula backward. If each thread already sustains $3.5$ of the $4$ slots (dense arithmetic loops), there is no empty space: the pair hits the ceiling $I_{2} = W = 4$ , so $G = 4/3.5 \approx 1.14 \times$ , or $0.57 \times$ per thread. Each thread runs at barely over half speed, and if they also collide in the cache the pair can fall below back-to-back execution.

It helps on memory-bound mixes. A thread stalled on a DRAM miss uses no issue slots for two hundred cycles; its sibling inherits essentially the whole core for the duration. Two threads that miss often but at different times can nearly double a core's throughput, which is why server workloads (many independent requests, each chasing pointers through big data structures) are SMT's best case. The same logic applies to the latency costs from the memory-hierarchy module: multithreading is latency tolerance, where caches are latency reduction.

It hurts when the bottleneck is already shared. Two compute-bound threads saturating the same ALUs split one core's throughput two ways and gain nothing. Worse is the cache. The L1 and L2 from the cache lessons do not grow when a second thread arrives; two threads whose working sets each almost fit now share a cache that fits neither, and hit rates collapse for both. A thread that ran clean alone can start missing constantly with a sibling, and the pair runs slower than the two would have run back-to-back. High-performance computing shops and some databases disable SMT for exactly this reason; the effect is workload-dependent enough that it must be measured case by case.

Measuring the effect is straightforward and worth doing once on any machine you care about. Run the workload with its threads pinned to distinct physical cores, then again pinned to sibling contexts of the same cores (taskset on Linux chooses the logical CPUs), and compare. The ratio is the SMT gain for that workload: near 1.0 for two ALU-saturating threads, 1.2–1.4 for typical mixes, occasionally below 1.0 when the working sets collide. Datasheets cannot substitute for the two runs, because the result depends on the workload's miss pattern, not on the core.

There is also a sharper concern: two threads timing each other's effects on a shared cache can leak information across protection boundaries, which is why several side-channel mitigations amount to do not schedule mutually distrusting code on sibling hyperthreads.

The OS view: logical CPUs

To the operating system, every hardware thread context is a schedulable processor. A machine with 4 cores and 2-way SMT advertises 8 logical CPUs: the scheduler sees eight run queues, and /proc/cpuinfo on Linux lists eight entries. The topology still matters, and the OS knows it: siblings share a core, so a sane scheduler spreads runnable threads across physical cores first and doubles up on siblings only when every core already has work. Getting this wrong is a real performance bug: two hot threads packed onto sibling contexts of core 0 while cores 1–3 idle run at roughly half speed for no reason.

Four physical cores with 2-way SMT present eight logical CPUs to the OS. Siblings (same column pair) share one core's execution units and caches; the scheduler f/ills separate cores before doubling up.

Consider the 8-logical-CPU machine with two runnable threads. A topology-blind scheduler placing both on logical CPUs 0 and 4 — siblings of core 0 — leaves cores 1–3 idle; the pair shares one core's ALUs and caches at the SMT rate, perhaps $1.4 \times$ combined. A topology-aware scheduler places them on logical CPUs 0 and 1 — distinct physical cores — for the full $2 \times$ with no contention. Same threads, same hardware, a factor $2/1.4 \approx 1.4$ apart purely from run-queue choice. Hence Linux models the CPU topology explicitly (lscpu prints the sibling map) and spreads runnable work across physical cores first, doubling on siblings only when every core already has a thread. The rule inverts under a throughput-over-latency policy — packing siblings first frees whole cores to sleep — which the scheduler must be told, since the hardware cannot know whether you want speed or battery.

The distinction matters when counting processors. 8 CPUs on a 4-core laptop does not promise 8 cores' worth of arithmetic; it promises 4 cores that each keep two contexts warm. Amdahl-style scaling estimates from the previous lesson should use physical cores as $N$ , with SMT as a bonus of tens of percent, not a doubling.

The ideas and where they run

The three disciplines in this lesson each have a clear origin, and the modern hardware descends from them directly.

Fine-grained multithreading is the oldest idea here. The Denelcor HEP (Burton Smith, 1978) built a pipeline that switched thread every cycle specifically so it never had to detect or forward hazards — with enough threads in flight, a thread's next instruction is always far enough behind its last that the pipeline is naturally hazard-free. Smith carried the idea forward into the Tera MTA, which hid all memory latency behind up to 128 hardware threads and dispensed with data caches entirely. That lineage is alive today in the GPU: an NVIDIA streaming multiprocessor holds dozens of warps and issues from a ready one each cycle, which is fine-grained multithreading at large scale, and the reason a GPU tolerates the hundreds-of-cycles latency of its own memory.

Simultaneous multithreading was named and analyzed by Tullsen, Eggers, and Levy (1995, ISCA), who showed that a wide out-of-order core wastes most of its issue bandwidth on any single thread and that interleaving several threads in the same cycle recovers it cheaply. Intel shipped the idea as hyperthreading in the Pentium 4 (2002) and has carried it through the Core line since; IBM's POWER series pushes it to 4- and 8-way SMT per core, betting on the many-independent- request server workloads where the extra contexts pay best.

The security caveat matters in modern practice. Because SMT siblings share the L1 cache and other microarchitectural state, one thread can infer the other's memory-access pattern from timing — the basis of cache-timing side channels and, more sharply, of the microarchitectural data sampling (MDS) and L1TF disclosures around 2018–2019 that let one sibling read data in the other's in-flight buffers. The mitigation of last resort, core scheduling, is now in Linux: the kernel refuses to run mutually distrusting tasks on sibling contexts of one core, trading some SMT throughput for isolation. Several cloud providers and HPC sites disable SMT outright for the same reason, which turns the always measure advice above into an operational default.

Multithreading squeezes more from one core; the next step is to build many. Two cores mean two L1 caches, and two caches mean two copies of the same variable that can disagree. Keeping them consistent is the cache-coherence problem, next.