---
title: Hardware Multithreading
module: Multithreading & Multicore
moduleNumber: 9
lessonNumber: 2
order: 902
summary: >
  A pipeline spends much of its life waiting — on cache misses, on dependences,
  on branches. Hardware multithreading fills the dead cycles with instructions
  from another thread. We compare coarse-grained switching (change threads on a
  long stall), fine-grained interleaving (change every cycle), and simultaneous
  multithreading (mix threads inside a single cycle), work out exactly which
  hardware a second thread context duplicates and which it shares, and weigh
  when SMT pays off and when two threads just fight over one cache.
topics: [Multithreading & Multicore]
sources:
  - book: Bryant & O'Hallaron
    ref: "CS:APP — §1.9.1 Thread-Level Concurrency; §6.4 Cache Memories"
---

The [previous lesson](/computer-architecture/multithreading-and-multicore/processes-threads-and-parallelism)
ended with a machine that cannot clock its one instruction stream any faster.
Before spending transistors on whole extra cores, there is a cheaper question:
is the core we already have even busy? Mostly, no. A modern pipeline issues
several instructions per cycle in bursts, then sits: an L2 miss from the
[memory hierarchy](/computer-architecture/memory-hierarchy/storage-technologies-and-the-latency-gap)
stalls a load for two hundred cycles, a
[mispredicted branch](/computer-architecture/pipelining/control-hazards-and-branch-prediction)
throws away everything in flight, a chain of
[dependent instructions](/computer-architecture/pipelining/data-hazards-stalling-and-forwarding)
serializes the issue slots. Averaged over real programs, most issue slots go
empty. **Hardware multithreading** addresses the waste directly: hold the state of
_several_ threads in the core at once, and when one thread cannot use the
hardware this cycle, hand it to another.

## What a thread context costs

Holding an extra thread is affordable because of an asymmetry the
[first lesson](/computer-architecture/multithreading-and-multicore/processes-threads-and-parallelism)
set up: a thread's private state is small. To hold an extra thread a core must
duplicate only the **architectural state** — the program counter, the
general-purpose and condition-code registers, and in an out-of-order machine the
**register alias table** (RAT) that maps those architectural names onto the big
physical register file (register renaming is sketched in the
[capstone's honesty section](/computer-architecture/capstone/the-whole-machine)).
Everything expensive stays single: the fetch unit and
decoders, the branch predictor, the schedulers, the ALUs and load/store units,
and the whole cache hierarchy. The duplicated state is a few kilobits; the
shared machinery is almost the entire core.

> **Definition (Hardware thread).** A per-thread copy of the architectural
> state — PC, architectural registers, condition codes, rename table — held
> inside one physical core, letting that core issue instructions from the
> thread without any OS intervention. The execution units, predictors, and
> caches are shared among the core's hardware threads.

$$
% caption: What a second hardware thread duplicates versus shares. Each context
% caption: owns a PC, architectural registers, and a rename table; both feed one
% caption: shared pool of schedulers, execution units, and caches.
\begin{tikzpicture}[font=\footnotesize,>=stealth,
  dup/.style={draw, minimum width=22mm, minimum height=6.5mm, inner sep=2pt, align=center},
  sh/.style={draw, fill=acc!8, minimum width=52mm, minimum height=7mm, inner sep=2pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  % duplicated contexts
  \node at (-2.1,3.4) {context 0};
  \node[dup] (p0) at (-2.1,2.7) {PC, registers};
  \node[dup] (r0) at (-2.1,1.9) {rename table};
  \node at (2.1,3.4) {context 1};
  \node[dup] (p1) at (2.1,2.7) {PC, registers};
  \node[dup] (r1) at (2.1,1.9) {rename table};
  % shared pool
  \node[sh] (sched) at (0,0.7) {shared schedulers};
  \node[sh] (exec) at (0,-0.15) {ALUs, load/store units};
  \node[sh] (cache) at (0,-1.0) {L1/L2 caches, predictor, fetch};
  \draw[->, acc, thick] (r0.south) -- (sched.north west);
  \draw[->, acc, thick] (r1.south) -- (sched.north east);
\end{tikzpicture}
$$

The payoff structure follows from the split. Duplicating contexts is cheap, so
vendors do it: Intel's **hyperthreading** runs two contexts per core.
Sharing the execution machinery means the two threads are not independent the
way two cores are: they compete for every shared resource, and that competition
determines when multithreading helps and when it hurts.

## Coarse-grained multithreading

The simplest policy switches threads only on a **long stall**. Thread A runs
alone at full speed; when it misses in the last-level cache and faces a
two-hundred-cycle DRAM access, the core switches to thread B and runs _it_
until B stalls in turn. The switch is not free — the pipeline holding A's
in-flight instructions must drain or be squashed before B's can enter, costing
a few cycles — but a few cycles against a two-hundred-cycle miss is a fine
trade.

$$
% caption: Coarse-grained multithreading. Thread A runs until an LLC miss, the
% caption: pipeline switches (a few dead cycles), and thread B covers the two
% caption: hundred cycles A spends waiting on DRAM.
\begin{tikzpicture}[font=\footnotesize,>=stealth,
  sl/.style={draw, minimum width=8mm, minimum height=6mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \node[anchor=east] at (-0.6,0) {issue};
  \foreach \i in {0,...,3} \node[sl, fill=acc!8]  at (\i*0.9,0) {A};
  \foreach \i in {4,5}     \node[sl, fill=black!8] at (\i*0.9,0) {};
  \foreach \i in {6,...,11} \node[sl, fill=acc!20] at (\i*0.9,0) {B};
  \draw[->, black] (3.15,1.25) -- (3.15,0.45);
  \node[anchor=south] at (3.15,1.3) {\scriptsize A misses in LLC};
  \node[anchor=north] at (4.05,-0.45) {\scriptsize drain, then swap};
  \draw[->, black] (-0.3,-1.1) -- (10.3,-1.1) node[anchor=west] {cycles};
\end{tikzpicture}
$$

To quantify this, let $W$ be useful cycles between last-level misses, $M$ the
miss penalty, and $s$ the switch cost. Single-thread utilization is
$$U_1 = \frac{W}{W + M}.$$
With $W = 100$, $M = 200$, $s = 10$: $U_1 = 100/300 \approx 33\,\%$. Add thread
B and switch to it on A's miss. B's work covers the stall, less two switches (out
to B, back to A), so a window of $W + M$ cycles delivers $W + (M - 2s)$ useful
cycles:
$$U_2 = \frac{W + (M - 2s)}{W + M} = \frac{100 + 180}{300} \approx 93\,\%.$$
One extra context lifts throughput $U_2/U_1 \approx 2.8\times$, at the cost of
the duplicated architectural state alone. The larger the miss penalty $M$, the
more a second thread is worth.

Coarse-grained multithreading hides only the big latencies. The short stalls —
a 3-cycle load-use bubble, a branch flush — stay unhidden, because a thread
switch costs about as much as the stall it would cover. Single-thread speed is
untouched (a thread that never misses runs exactly as before), which is the
policy's main virtue.

## Fine-grained multithreading

The opposite extreme switches threads **every cycle**, round-robin among the
ready ones, with zero switch cost: the fetch stage just reads a different PC
each cycle. Now even tiny stalls are covered: if thread A has a load-use bubble
this cycle, threads B, C, D fill the slot before A's turn comes around again.
Barrel processors and GPU warp schedulers push this to dozens of contexts.

The cost is single-thread latency. With $k$ contexts interleaving round-robin,
each thread advances at most once per $k$ cycles, so a program running alone on a
fine-grained machine runs at $1/k$ speed while the other slots issue nothing.
Fine-grained designs suit workloads with throughput to burn and no
latency-critical single thread, which describes a GPU's situation and almost never a
desktop's.

$$
% caption: Fine-grained multithreading rotates threads every cycle. Short stalls
% caption: in any one thread vanish into the rotation, but a single thread now
% caption: advances at most once per round.
\begin{tikzpicture}[font=\footnotesize,>=stealth,
  sl/.style={draw, minimum width=8mm, minimum height=6mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  \node[anchor=east] at (-0.6,0) {issue};
  \foreach \i/\l in {0/A, 1/B, 2/C, 3/D, 4/A, 5/B, 6/C, 7/D, 8/A, 9/B}
    \node[sl, fill=acc!8] at (\i*0.9,0) {\l};
  \draw[->, black] (-0.3,-0.75) -- (9.4,-0.75) node[anchor=west] {cycles};
  \node[anchor=north] at (1.8,-1.1) {\scriptsize each thread issues once per round};
\end{tikzpicture}
$$

## Simultaneous multithreading

Both schemes above share the machine in _time_: each cycle belongs to one
thread. A wide superscalar core wastes issue bandwidth that way, because the
issue stage can start four or more instructions per cycle and a single thread
rarely has four independent instructions ready. **Simultaneous multithreading
(SMT)** shares the machine in _space_ as well: in one cycle, the scheduler
fills issue slots with instructions from **any** ready thread, mixing them
freely. Thread A's two ready instructions and thread B's two ready
instructions issue together, in the same cycle, to different execution units.

> **Definition (Simultaneous multithreading).** A multithreading discipline for
> superscalar cores in which instructions from multiple hardware threads issue
> in the _same_ cycle, filling issue slots any single thread would leave empty.
> Intel's hyperthreading is 2-way SMT: two contexts per physical core.

Out-of-order machinery makes SMT nearly free to add. The core already renames
architectural registers to physical ones, so instructions in the shared
scheduler carry no thread identity that the ALUs care about; tagging each
in-flight instruction with a context ID, doubling the rename table, and
partitioning a few queues is most of the work. That is why hyperthreading
costs a few percent of die area against roughly 15–30 % more throughput on
mixed workloads.

The three disciplines differ only in whether cycles are shared in time or space,
and at what switch granularity:

| discipline | switch granularity | switch cost | slots/cycle | hides | single-thread speed |
|:---|:---|:---:|:---:|:---|:---:|
| coarse-grained | on long stall | few cycles | one thread | long latencies only | unchanged |
| fine-grained | every cycle | zero | one thread | all stalls | $1/k$ |
| SMT | within a cycle | zero | multiple threads | stalls + issue-width waste | shared, degrades under contention |

$$
% caption: Issue slots (4 wide) cycle by cycle. Left: one thread leaves slots
% caption: empty wherever it lacks independent work. Right: 2-way SMT fills the
% caption: same slots with the second thread's ready instructions.
\begin{tikzpicture}[font=\footnotesize,
  sl/.style={draw, minimum width=6.5mm, minimum height=6mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % single thread grid: rows = cycles (top to bottom), cols = slots
  \node at (1.5,2.85) {one thread};
  \foreach \r/\y in {0/2.2, 1/1.5, 2/0.8, 3/0.1} {
    \node[anchor=east] at (-0.35,\y) {\scriptsize cyc \r};
  }
  \foreach \c/\r in {0/0, 1/0, 0/1, 0/2, 1/2, 2/2, 0/3}
    \node[sl, fill=acc!8] at (\c*0.75,{2.2-\r*0.7}) {A};
  \foreach \c/\r in {2/0, 3/0, 1/1, 2/1, 3/1, 3/2, 1/3, 2/3, 3/3}
    \node[sl, fill=black!6] at (\c*0.75,{2.2-\r*0.7}) {};
  % SMT grid
  \node at (7.0,2.85) {2-way SMT};
  \foreach \c/\r in {0/0, 1/0, 0/1, 0/2, 1/2, 2/2, 0/3}
    \node[sl, fill=acc!8] at (5.5+\c*0.75,{2.2-\r*0.7}) {A};
  \foreach \c/\r in {2/0, 3/0, 1/1, 2/1, 3/2, 1/3, 2/3}
    \node[sl, fill=acc!25] at (5.5+\c*0.75,{2.2-\r*0.7}) {B};
  \foreach \c/\r in {3/1, 3/3}
    \node[sl, fill=black!6] at (5.5+\c*0.75,{2.2-\r*0.7}) {};
\end{tikzpicture}
$$

## When SMT helps, and when it hurts

SMT helps only when the two threads' demands interleave rather than collide.
Let a core issue $W$ instructions per cycle at peak and
a lone thread sustain IPC $I_1$; utilization is $I_1/W$. With two threads
sustaining $I_2$ together, the SMT gain is
$$G = \frac{I_2}{I_1}, \qquad I_2 \le W.$$
The gain tracks the empty slots a single thread leaves. Good case:
$W = 4$, $I_1 = 1.6$ (utilization $40\,\%$), a second thread whose stalls fall in
_different_ cycles lifts the pair to $I_2 = 2.9$, so $G = 2.9/1.6 \approx
1.8\times$ — most of a second core's worth, for a few percent of the die.

The bad case runs the same formula backward. If each thread already sustains
$3.5$ of the $4$ slots (dense arithmetic loops), there is no empty space: the
pair hits the ceiling $I_2 = W = 4$, so $G = 4/3.5 \approx 1.14\times$, or
$0.57\times$ per thread. Each thread runs at barely over half speed, and if they
also collide in the cache the pair can fall _below_ back-to-back execution.

**It helps on memory-bound mixes.** A thread stalled on a DRAM miss uses no
issue slots for two hundred cycles; its sibling inherits essentially the whole
core for the duration. Two threads that miss often but at _different_ times
can nearly double a core's throughput, which is why server workloads (many
independent requests, each chasing pointers through big data structures) are
SMT's best case. The same logic applies to the latency costs from the
[memory-hierarchy module](/computer-architecture/memory-hierarchy/cache-performance-and-cache-friendly-code):
multithreading is latency _tolerance_, where caches are
latency _reduction_.

**It hurts when the bottleneck is already shared.** Two compute-bound threads
saturating the same ALUs split one core's throughput two ways and gain nothing.
Worse is the cache. The L1 and L2 from the
[cache lessons](/computer-architecture/memory-hierarchy/cache-memories-direct-mapped)
do not grow when a second thread arrives; two threads whose working sets each
_almost_ fit now share a cache that fits neither, and hit rates collapse for
both. A thread that ran clean alone can start missing constantly with a
sibling, and the pair runs _slower_ than the two would have run back-to-back.
High-performance computing shops and some databases disable SMT for exactly
this reason; the effect is workload-dependent enough that it must be measured
case by case.

Measuring the effect is straightforward and worth doing once on any machine
you care about. Run the workload with its threads pinned to distinct physical
cores, then again pinned to sibling contexts of the same cores (`taskset` on
Linux chooses the logical CPUs), and compare. The ratio is the SMT gain for
_that_ workload: near 1.0 for two ALU-saturating threads, 1.2–1.4 for typical
mixes, occasionally below 1.0 when the working sets collide. Datasheets cannot
substitute for the two runs, because the result depends on the workload's miss
pattern, not on the core.

There is also a sharper concern: two threads timing each other's effects on a
shared cache can leak information across protection boundaries, which is why
several side-channel mitigations amount to "do not schedule mutually
distrusting code on sibling hyperthreads."

## The OS view: logical CPUs

To the operating system, every hardware thread context is a schedulable
processor. A machine with 4 cores and 2-way SMT advertises **8 logical CPUs**:
the scheduler sees eight run queues, and `/proc/cpuinfo` on Linux lists eight
entries. The topology still matters, and the OS knows it: siblings share a
core, so a sane scheduler spreads runnable threads across _physical_ cores
first and doubles up on siblings only when every core already has work.
Getting this wrong is a real performance bug: two hot threads packed onto
sibling contexts of core 0 while cores 1–3 idle run at roughly half speed for
no reason.

$$
% caption: Four physical cores with 2-way SMT present eight logical CPUs to the
% caption: OS. Siblings (same column pair) share one core's execution units and
% caption: caches; the scheduler f\/ills separate cores before doubling up.
\begin{tikzpicture}[font=\footnotesize,
  lc/.style={draw, minimum width=10mm, minimum height=6mm, inner sep=0pt},
  co/.style={draw, minimum width=24mm, minimum height=18mm}]
  \definecolor{acc}{HTML}{2348F2}
  \foreach \c in {0,...,3} {
    \node[co] at (\c*3.0,0) {};
    \node at (\c*3.0,-0.55) {core \c};
    \node[lc, fill=acc!8]  at (\c*3.0-0.6,0.35) {\scriptsize \c};
    \pgfmathtruncatemacro{\s}{\c+4}
    \node[lc, fill=acc!20] at (\c*3.0+0.6,0.35) {\scriptsize \s};
  }
  \node at (4.5,1.6) {logical CPUs 0-7, siblings share a core};
\end{tikzpicture}
$$

Consider the 8-logical-CPU machine with two
runnable threads. A topology-blind scheduler placing both on logical CPUs 0 and
4 — siblings of core 0 — leaves cores 1–3 idle; the pair shares one core's ALUs
and caches at the SMT rate, perhaps $1.4\times$ combined. A topology-aware
scheduler places them on logical CPUs 0 and 1 — distinct physical cores — for the
full $2\times$ with no contention. Same threads, same hardware, a factor
$2/1.4 \approx 1.4$ apart purely from run-queue choice. Hence Linux models the
CPU topology explicitly (`lscpu` prints the sibling map) and spreads runnable
work across physical cores first, doubling on siblings only when every core
already has a thread. The rule inverts under a throughput-over-latency policy —
packing siblings first frees whole cores to sleep — which the scheduler must be
told, since the hardware cannot know whether you want speed or battery.

The distinction matters when counting processors. "8 CPUs" on a 4-core laptop does
not promise 8 cores' worth of arithmetic; it promises 4 cores that each keep
two contexts warm. Amdahl-style scaling estimates from the
[previous lesson](/computer-architecture/multithreading-and-multicore/processes-threads-and-parallelism)
should use physical cores as $N$, with SMT as a bonus of tens of percent, not a
doubling.

## The ideas and where they run

The three disciplines in this lesson each have a clear origin, and the modern
hardware descends from them directly.

**Fine-grained multithreading** is the oldest idea here. The Denelcor **HEP**
(Burton Smith, 1978) built a pipeline that switched thread every cycle
specifically so it never had to detect or forward hazards — with enough threads
in flight, a thread's next instruction is always far enough behind its last that
the pipeline is naturally hazard-free. Smith carried the idea forward into the
**Tera MTA**, which hid _all_ memory latency behind up to 128 hardware threads
and dispensed with data caches entirely. That lineage is alive today in the
**GPU**: an NVIDIA streaming multiprocessor holds dozens of warps and issues
from a ready one each cycle, which is fine-grained multithreading at large scale,
and the reason a GPU tolerates the hundreds-of-cycles latency of its own memory.

**Simultaneous multithreading** was named and analyzed by Tullsen, Eggers, and
Levy (1995, ISCA), who showed that a wide out-of-order core wastes most of its
issue bandwidth on any single thread and that interleaving several threads in the
same cycle recovers it cheaply. Intel shipped the idea as **hyperthreading** in
the Pentium 4 (2002) and has carried it through the Core line since; IBM's POWER
series pushes it to 4- and 8-way SMT per core, betting on the many-independent-
request server workloads where the extra contexts pay best.

**The security caveat** matters in modern practice. Because SMT
siblings share the L1 cache and other microarchitectural state, one thread can
infer the other's memory-access pattern from timing — the basis of cache-timing
side channels and, more sharply, of the **microarchitectural data sampling**
(MDS) and **L1TF** disclosures around 2018–2019 that let one sibling read data
in the other's in-flight buffers. The mitigation of last resort, **core
scheduling**, is now in Linux: the kernel refuses to run mutually distrusting
tasks on sibling contexts of one core, trading some SMT throughput for isolation.
Several cloud providers and HPC sites disable SMT outright for the same reason,
which turns the "always measure" advice above into an operational default.

> **Takeaway.** Multithreading keeps a starved pipeline fed. Coarse-grained
> switching hides only long stalls and preserves single-thread speed;
> fine-grained rotation hides everything and sacrifices it; SMT fills a wide
> core's empty issue slots with a second thread in the _same_ cycle for a few
> percent of area. A hardware thread duplicates only PC, registers, and rename
> state — execution units and caches stay shared, so SMT helps most on
> complementary, memory-bound mixes and hurts when siblings contend for one
> cache. The OS sees each context as a logical CPU and fills real cores first.

Multithreading squeezes more from one core; the next step is to build many.
Two cores mean two L1 caches, and two caches mean two copies of
the same variable that can disagree. Keeping them consistent is the
[cache-coherence problem](/computer-architecture/multithreading-and-multicore/cache-coherence),
next.