---
title: Multicore Organization
module: Multithreading & Multicore
moduleNumber: 9
lessonNumber: 5
order: 905
summary: >
  Where everything sits on the die. A modern die gives each core private L1 and L2
  caches, spreads a shared last-level cache across slices, and wires it all
  together with a ring or mesh; multi-socket servers add NUMA, where memory is
  local to one socket and every remote access pays a latency penalty. We walk the
  floorplan, put numbers on local versus remote latency, meet thread affinity,
  and account for the two shared resources — coherence traffic and LLC
  capacity — that decide how far a parallel program scales.
topics: [Multithreading & Multicore]
sources:
  - book: Bryant & O'Hallaron
    ref: "CS:APP — §6.2–6.4 Storage Technologies, Locality, Cache Memories; §1.9.1 Thread-Level Concurrency"
  - book: Bistriceanu
    ref: "Computer Architecture Notes — §7 The Memory Hierarchy; §9 Main Memory"
---

The module so far treated "two cores with caches on a bus" as an abstract
picture. This closing lesson makes the picture physical: what a multicore die
actually looks like, how the caches from the
[memory-hierarchy module](/computer-architecture/memory-hierarchy/storage-technologies-and-the-latency-gap)
are distributed across it, what replaces the bus when eight cores would
saturate one, and what happens when even the _memory controller_ stops being a
single shared point. The die's geometry determines latency, and latency
determines how far a parallel program scales.

## The die: private levels, shared floor

A representative desktop part (four cores, in the style of every Intel or AMD
design since about 2008) organizes its cache levels by a simple rule: **the
levels that must be fast are private, the level that must be big is shared**.

Each core owns its **L1** (split instruction/data, ~32 KB each, 4-cycle hit)
and its **L2** (~256 KB–1 MB, ~12-cycle hit). Private levels answer at core
speed with no arbitration, and they are where
[SMT siblings](/computer-architecture/multithreading-and-multicore/hardware-multithreading)
collide. The whole hierarchy this module inherits from the
[memory-hierarchy lessons](/computer-architecture/memory-hierarchy/storage-technologies-and-the-latency-gap)
now lays out on the die by a single trade of speed against size and sharing:

| level | typical size | hit latency | private or shared |
|:---|:---|:---:|:---:|
| L1 (I + D) | 32 KB each | ~4 cycles | private per core |
| L2 | 256 KB – 1 MB | ~12 cycles | private per core |
| L3 (LLC) | 8 – 32 MB | ~40 cycles | shared, sliced |
| local DRAM | GBs | ~200 cycles | shared |
| remote DRAM (NUMA) | GBs | ~300+ cycles | shared, another node |

The order-of-magnitude jumps between rows are the whole reason for the layout:
you keep the small, hot levels private so no other core can slow them, and share
only the large, already-slow last level where the extra tens of cycles of
arbitration barely register against a 40-cycle base. The **last-level cache** (LLC, usually L3: 8–32 MB, ~40-cycle hit) is
shared by every core, for two reasons that outweigh the contention it invites:
a core that suddenly needs capacity can use all of it, and a line two cores
both read exists once instead of four times. Physically the LLC is not one
block but **slices**, one per core stop, with each address hashed to a fixed
slice so lookups never search.

$$
% caption: A four-core die. Per-core L1 and L2 are private; the last-level cache
% caption: is sliced along a ring interconnect that also joins the memory
% caption: controller. An address hashes to exactly one slice.
\begin{tikzpicture}[font=\footnotesize,>=stealth,
  core/.style={draw, minimum width=19mm, minimum height=7mm, align=center},
  pl/.style={draw, fill=acc!8, minimum width=19mm, minimum height=6mm, align=center},
  llc/.style={draw, fill=acc!20, minimum width=19mm, minimum height=6.5mm, align=center},
  mc/.style={draw, minimum width=26mm, minimum height=6.5mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \foreach \i in {0,...,3} {
    \node[core] at (\i*2.5,3.3) {core \i};
    \node[pl] at (\i*2.5,2.55) {L1};
    \node[pl] at (\i*2.5,1.85) {L2};
    \draw[thick] (\i*2.5,1.5) -- (\i*2.5,1.05);
    \node[llc] at (\i*2.5,0.7) {LLC slice \i};
  }
  % ring
  \draw[acc, thick] (-1.4,0.05) -- (8.9,0.05);
  \node[anchor=west, text=acc] at (9.0,0.05) {ring};
  \foreach \i in {0,...,3} \draw[acc, thick] (\i*2.5,0.35) -- (\i*2.5,0.05);
  \node[mc] (mc) at (3.75,-1.0) {memory controller};
  \draw[acc, thick] (3.75,0.05) -- (mc.north);
  \draw[->, thick] (mc.south) -- +(0,-0.55) node[anchor=north] {DRAM};
\end{tikzpicture}
$$

The **interconnect** joins the cores, slices, and controllers. A shared bus stops scaling
around four to eight agents: one transaction at a time, and
[snooping](/computer-architecture/multithreading-and-multicore/cache-coherence)
means everyone processes it. A **ring** (Intel's choice for client parts)
gives each core, slice, and controller a stop; packets hop stop to stop, so
several transfers proceed at once and latency grows gently with distance.
The reason for the ring-to-mesh switch is a latency calculation. On a ring of
$n$ stops a packet hops on average $\bar h_\text{ring} = n/4$ stops (up to $n/2$
worst-case), so latency grows _linearly_ with core count. A 2-D **mesh** of $n$
nodes on a $\sqrt{n} \times \sqrt{n}$ grid has $\bar h_\text{mesh} \approx
\sqrt{n}$:
$$\bar h_\text{ring} = \frac{n}{4}, \qquad \bar h_\text{mesh} \approx \sqrt{n}.$$
Going $16 \to 64$ cores thus multiplies ring hop count by $4$ but mesh hop count
only by $2$. Below a dozen cores the ring's simplicity wins; past that the
$\sqrt{n}$ scaling of the mesh pays for its extra routers, which is why
client parts stayed on rings and server parts moved to meshes. On a mesh, cores
sit on a grid, each talking to four neighbors, and bandwidth grows with the
perimeter you cross rather than being one shared wire.

| interconnect | concurrent transfers | avg latency vs $n$ | scales to |
|:---|:---|:---:|:---:|
| bus | one | flat, one conversation | 4–8 agents |
| ring | several | $O(n)$ | ~12 cores |
| mesh | many (grid bisection) | $O(\sqrt{n})$ | tens+ |

Coherence adapts alongside: broadcast
snooping gives way to **directory** protocols, where a home slice keeps a
sharer list per line and forwards invalidations to exactly the caches that
hold a copy, the same MESI states delivered point-to-point instead of
broadcast.

$$
% caption: Interconnect shapes. A bus is one shared wire and one conversation; a
% caption: ring gives every agent a stop and pipelines hops; a mesh gives a grid
% caption: of stops and scales bandwidth with the cut you cross.
\begin{tikzpicture}[font=\footnotesize,
  nd/.style={draw, fill=acc!8, minimum size=5.5mm, inner sep=0pt}]
  \definecolor{acc}{HTML}{2348F2}
  % bus
  \node at (1.05,2.1) {bus};
  \draw[thick, black] (-0.3,0.1) -- (2.4,0.1);
  \foreach \i in {0,...,3} {
    \node[nd] at (\i*0.8-0.15,0.9) {};
    \draw (\i*0.8-0.15,0.62) -- (\i*0.8-0.15,0.1);
  }
  % ring
  \node at (5.55,2.1) {ring};
  \draw[thick, acc] (4.3,0.5) rectangle (6.8,1.3);
  \foreach \x in {4.3,6.8} \foreach \y in {0.5,1.3} \node[nd] at (\x,\y) {};
  \foreach \x in {5.13,5.96} \foreach \y in {0.5,1.3} \node[nd] at (\x,\y) {};
  % mesh
  \node at (10.4,2.1) {mesh};
  \foreach \x in {0,1,2} \foreach \y in {0,1} {
    \node[nd] at (9.4+\x,0.35+\y*0.95) {};
  }
  \foreach \x in {0,1,2} \draw[acc, thick] (9.4+\x,0.62) -- (9.4+\x,1.03);
  \foreach \y in {0.35,1.3} \draw[acc, thick] (9.67,\y) -- (10.13,\y);
  \foreach \y in {0.35,1.3} \draw[acc, thick] (10.67,\y) -- (11.13,\y);
\end{tikzpicture}
$$

One more design choice ties the levels together:
whether the LLC is **inclusive** of the private caches. An inclusive LLC
guarantees that every line in any L1 or L2 also has a copy (or at least a tag)
in the LLC, which makes it a natural **snoop filter**: a request that misses
in the LLC provably misses in every private cache too, so no core need be
disturbed. The cost is capacity, since the LLC spends space duplicating what
the private levels already hold; as private L2s grew, several server designs
moved to non-inclusive LLCs and keep a separate directory of tags to preserve
the filtering. Either way, the goal is the same: answer "does anyone have this
line?" without broadcasting the question.

## NUMA: when memory takes sides

One socket has one set of memory controllers. Servers bolt two or more sockets
together with a cache-coherent socket-to-socket link (Intel UPI, AMD Infinity
Fabric), and coherence still holds across the whole machine: a
line dirty in socket 0's cache is found and flushed even when socket 1 asks.
What stops being uniform is **latency**. Each socket's DRAM hangs off its own
controllers; a core reaching its own socket's DRAM pays about **80–100 ns**,
while the same load to the _other_ socket's DRAM crosses the link and pays
**130–200 ns**, plus it consumes cross-link bandwidth that is far scarcer than
local bandwidth. Memory access is now **non-uniform**: NUMA.

> **Definition (NUMA).** Non-uniform memory access: a shared-memory machine in
> which every core can address all memory, but latency and bandwidth depend on
> which node (socket) the memory is attached to. Coherence still spans the
> machine; only the cost varies.

$$
% caption: Two NUMA nodes. Every core reaches all DRAM, but a local access costs
% caption: about 90 ns while a remote one crosses the socket link for 150 ns or
% caption: more, and shares that link with all other cross-node traf\/f\/ic.
\begin{tikzpicture}[font=\footnotesize,>=stealth,
  sock/.style={draw, minimum width=34mm, minimum height=16mm, align=center},
  dram/.style={draw, fill=acc!8, minimum width=30mm, minimum height=7mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[sock] (s0) at (0,0) {socket 0\\4 cores + LLC};
  \node[sock] (s1) at (6.8,0) {socket 1\\4 cores + LLC};
  \node[dram] (d0) at (0,-2.1) {DRAM node 0};
  \node[dram] (d1) at (6.8,-2.1) {DRAM node 1};
  \draw[<->, thick] (s0.south) -- node[left=2pt] {\scriptsize local, 90 ns} (d0.north);
  \draw[<->, thick] (s1.south) -- node[right=2pt] {\scriptsize local, 90 ns} (d1.north);
  \draw[<->, acc, thick] (s0.east) -- node[above=2pt] {\scriptsize socket link} (s1.west);
  \draw[->, acc, thick, dashed] (s0.south east) to[out=-25, in=155]
    node[below=3pt, pos=0.35] {\scriptsize remote, 150 ns or more} (d1.north west);
\end{tikzpicture}
$$

The OS default — allocate a page on the node whose core **first touches**
it — works until threads migrate. A thread that faults its working set in on
socket 0, then gets rescheduled to socket 1, drags nothing with it: every one
of its pages is now remote, and it runs 50–100 % slower with no code change at
all. Hence **thread affinity**: pinning a thread to a core or node
(`sched_setaffinity`, `numactl`) so the scheduler cannot separate it from its
memory, plus allocating memory _from_ the node where the consuming thread
lives. NUMA-aware programs place threads first and memory second, in that
order, deliberately.

To quantify the first-touch mistake, take a two-socket machine, local DRAM at $t_\text{loc} = 90$ ns, remote at
$t_\text{rem} = 150$ ns, and a memory-bound parallel loop — every element is one
DRAM access. If thread 0 zeroes the whole array first (the innocent `memset`),
first-touch places _every_ page on node 0. Then eight threads, four per socket,
sweep the array: the four on socket 0 pay $t_\text{loc}$, the four on socket 1
pay $t_\text{rem}$ and pile onto the single cross-socket link. Average access is
$$\bar t = \tfrac{1}{2}(t_\text{loc} + t_\text{rem}) = 120\text{ ns},$$
so the run is $\bar t / t_\text{loc} = 120/90 \approx 1.33\times$ slower than it
should be — worse once link contention is added. Now have _each_ thread
initialize its own slab: first-touch places every thread's pages locally, every
access is $t_\text{loc}$, the link carries nothing, and the same code runs a
third faster. The two versions differ only in _who runs the initialization
loop_ — identical arithmetic, identical results, a stable 1.33x that no profiler
pointed at the loop will ever explain.

In practice: decide the parallel decomposition — which
thread owns which slab of data — before anything runs. Pin each thread to a
core on the node where its slab will live. Then have **each thread initialize
its own slab**, so first-touch places every page locally, rather than letting
thread 0 zero the whole array from node 0 and leave half the machine's
accesses remote for the run's lifetime. That single initialization mistake is the most
common NUMA bug in numerical code, it is invisible in the source (the
initialization looks like an innocent loop), and it costs a stable
1.5x for the rest of the run. The diagnosis, as always, is measurement: per-node memory
traffic counters show one controller saturated and the other idle.

## What actually limits scaling

The [first lesson](/computer-architecture/multithreading-and-multicore/processes-threads-and-parallelism)
priced scaling with Amdahl's law: the serial fraction sets the ceiling. The
floorplan adds two more costs, and both grow _with_ core count.

**Coherence traffic.** Every write to a line another core holds costs an
invalidation and later a miss, the
[fourth C](/computer-architecture/multithreading-and-multicore/cache-coherence).
With more cores, more sharers exist per hot line, each invalidation reaches
further across the ring or mesh, and true sharing (one lock, one counter, one
queue head touched by everyone) turns into serialization delivered by
hardware: the line's ownership hops core to core, one writer at a time, no
matter how many cores wait. A single hot cache line can flatten a 32-core
scaling curve by itself, which is why scalable designs shard hot state
per-core and combine late, exactly as the
[false-sharing fix](/computer-architecture/multithreading-and-multicore/cache-coherence)
did.

**Shared-LLC and bandwidth contention.** Each added thread brings its working
set, but the LLC does not grow: at some thread count the union stops fitting,
the LLC miss rate climbs for _everyone_, and demand spills into a DRAM
bandwidth that is also fixed. A program whose per-thread working set is small
scales past this; one that streams memory saturates the controllers at four
or six cores and flatlines, with additional cores merely queueing at the
memory wall. The [locality lessons](/computer-architecture/memory-hierarchy/locality)
compound in value on multicore: bytes you never fetch are bandwidth someone
else gets to use.

$$
% caption: Measured-style scaling. Ideal is the diagonal; a CPU-limited program
% caption: with a small hot-line tax bends away slowly; a DRAM-limited one
% caption: saturates the memory controllers early and goes flat.
\begin{tikzpicture}[font=\footnotesize,>=stealth]
  \definecolor{acc}{HTML}{2348F2}
  \draw[->, black] (0,0) -- (8.6,0) node[anchor=north west] {threads};
  \draw[->, black] (0,0) -- (0,5.4) node[anchor=south] {throughput};
  \foreach \x/\l in {0.25/1, 2/8, 4/16, 8/32}
    \node[anchor=north] at (\x,-0.05) {\scriptsize \l};
  % ideal diagonal: linear in thread count
  \draw[black, dashed] (0.25,0.16) -- (8,5.0);
  \node[anchor=east, text=black] at (6.75,4.75) {\scriptsize ideal};
  % CPU-limited: bends away gently
  \draw[acc, very thick]
    plot[smooth] coordinates {(0.25,0.16) (1,0.58) (2,1.1) (4,2.0) (6,2.7) (8,3.2)};
  \node[anchor=west, text=acc] at (5.9,2.15) {\scriptsize CPU limited};
  % DRAM-limited: saturates early
  \draw[black, thick]
    plot[smooth] coordinates {(0.25,0.16) (1,0.55) (2,0.95) (3,1.15) (4,1.2) (8,1.3)};
  \node[anchor=west, text=black] at (5.4,0.85) {\scriptsize DRAM limited};
\end{tikzpicture}
$$

Work one diagnosis end to end. You parallelize a program and measure: 1 thread
does 10 units/s, 4 threads 34 (a solid $3.4\times$), 8 threads 40, 16 threads 41.
The curve bends hard between 4 and 8 and flatlines after. The flatline at ~40
units/s says one shared resource is pinned; the counters say which. The curve's
shape identifies the bottleneck, and two counters — per-thread IPC and LLC miss
rate, plus measured memory latency — distinguish the cases:

| symptom | per-thread IPC | LLC miss rate | memory latency | wall | fix |
|:---|:---:|:---:|:---:|:---|:---|
| bends, then tracks | steady | steady | steady | fixed serial overhead | Amdahl ceiling |
| bends ever harder | falls | steady | steady | coherence | shard hot line per-core |
| bends ever harder | steady | climbs | steady | capacity | shrink footprint, block |
| hard flatline | steady | steady | climbs | bandwidth | locality, not threads |

Past the bandwidth wall, each added thread does not merely waste a core — it
lengthens the DRAM queue and slows the threads you already had.

## The machine, whole

This module completes the stack the course has been building. A core
is the [pipelined datapath](/computer-architecture/pipelining/the-complete-pipe-processor)
this course assembled, kept busy by
[SMT](/computer-architecture/multithreading-and-multicore/hardware-multithreading)
when one stream stalls. Cores multiply because
[Dennard scaling ended](/computer-architecture/multithreading-and-multicore/processes-threads-and-parallelism);
their private caches stay coherent through
[MESI](/computer-architecture/multithreading-and-multicore/cache-coherence);
their store buffers relax ordering until
[fences and atomics](/computer-architecture/multithreading-and-multicore/memory-consistency-and-synchronization)
restore it; and the whole system shares a sliced LLC, a ring or mesh, and,
past one socket, NUMA memory with non-uniform costs. Every layer is
one you have now built or reasoned through, and the
[capstone](/computer-architecture/capstone/the-whole-machine) walks a single
line of C down through all of them at once.

## Interconnects, NUMA, and the roofline

The floorplan of a modern die is the subject of a large public literature, and a
few landmarks orient the rest.

The **on-die interconnect** moved from bus to ring to mesh in production over
about a decade. Intel's ring bus debuted in Sandy Bridge (2011) and served
client parts for years; the switch to a 2-D **mesh** for the server line came
with Skylake-SP (2017), documented in Intel's own architecture disclosures,
because a ring's latency grows linearly with stops and a mesh's grows with the
square root. The academic groundwork is older: Dally and Towles' _Principles and
Practices of Interconnection Networks_ (2004) is the standard text, and the
network-on-chip research program it seeded (Dally and Towles, 2001, DAC) is why
a chip's cores now talk over a packet-routed fabric rather than shared wires.

**NUMA** is not new either — Stanford's DASH multiprocessor (Lenoski et al.,
1992, ISCA) built directory-based cache-coherent NUMA and measured exactly the
local-versus-remote penalty this lesson quotes. What changed is that NUMA is now
inside a single two-socket box on any server, so the first-touch and affinity
discipline that used to be supercomputer lore is everyday performance
engineering, exposed through `numactl` and `libnuma` on Linux.

The scaling curves at the end of the lesson have a clean formal companion in the
**roofline model** (Williams, Waterman, and Patterson, 2009, _CACM_), which plots
attainable performance against a kernel's arithmetic intensity — flops per byte
of memory traffic — and shows a program pinned either under a sloped
memory-bandwidth roof or a flat compute roof. The DRAM-limited flatline in this
lesson's figure is the roofline's bandwidth roof seen from the thread-count axis:
below a threshold intensity, adding cores cannot help, because the shared
memory system, not the cores, is the binding constraint.

> **Takeaway.** Fast levels private, big level shared: per-core L1/L2, a sliced
> LLC on a ring or mesh, directories replacing snoop broadcast as agents
> multiply. Multiple sockets make memory non-uniform (all of it reachable,
> the local part 1.5–2x cheaper), so threads get pinned and memory allocated
> to match. Scaling is bounded by Amdahl's serial fraction, then by coherence
> traffic on hot lines, then by shared LLC capacity and DRAM bandwidth; the
> fixes are sharding, padding, locality, and affinity, in roughly that order.

That closes the module. The machine on your desk is no longer a diagram of
somebody else's design: it is SEQ, pipelined, cached, virtualized, replicated,
and wired together — and the [capstone](/computer-architecture/capstone/the-whole-machine)
is where all of it runs one program, end to end.