Multicore Organization

The module so far treated two cores with caches on a bus as an abstract picture. This closing lesson makes the picture physical: what a multicore die actually looks like, how the caches from the memory-hierarchy module are distributed across it, what replaces the bus when eight cores would saturate one, and what happens when even the memory controller stops being a single shared point. The die's geometry determines latency, and latency determines how far a parallel program scales.

The die: private levels, shared floor

A representative desktop part (four cores, in the style of every Intel or AMD design since about 2008) organizes its cache levels by a simple rule: the levels that must be fast are private, the level that must be big is shared.

Each core owns its L1 (split instruction/data, ~32 KB each, 4-cycle hit) and its L2 (~256 KB–1 MB, ~12-cycle hit). Private levels answer at core speed with no arbitration, and they are where SMT siblings collide. The whole hierarchy this module inherits from the memory-hierarchy lessons now lays out on the die by a single trade of speed against size and sharing:

The order-of-magnitude jumps between rows are the whole reason for the layout: you keep the small, hot levels private so no other core can slow them, and share only the large, already-slow last level where the extra tens of cycles of arbitration barely register against a 40-cycle base. The last-level cache (LLC, usually L3: 8–32 MB, ~40-cycle hit) is shared by every core, for two reasons that outweigh the contention it invites: a core that suddenly needs capacity can use all of it, and a line two cores both read exists once instead of four times. Physically the LLC is not one block but slices, one per core stop, with each address hashed to a fixed slice so lookups never search.

The interconnect joins the cores, slices, and controllers. A shared bus stops scaling around four to eight agents: one transaction at a time, and snooping means everyone processes it. A ring (Intel's choice for client parts) gives each core, slice, and controller a stop; packets hop stop to stop, so several transfers proceed at once and latency grows gently with distance. The reason for the ring-to-mesh switch is a latency calculation. On a ring of $n$ stops a packet hops on average ${\bar{h}}_{\text{ring}}=n/4$ stops (up to $n/2$ worst-case), so latency grows linearly with core count. A 2-D mesh of $n$ nodes on a $\sqrt{n}\times \sqrt{n}$ grid has ${\bar{h}}_{\text{mesh}}\approx \sqrt{n}$: ${\bar{h}}_{\text{ring}}=\frac{n}{4},{\bar{h}}_{\text{mesh}}\approx \sqrt{n}.$ Going $16\to 64$ cores thus multiplies ring hop count by $4$ but mesh hop count only by $2$. Below a dozen cores the ring's simplicity wins; past that the $\sqrt{n}$ scaling of the mesh pays for its extra routers, which is why client parts stayed on rings and server parts moved to meshes. On a mesh, cores sit on a grid, each talking to four neighbors, and bandwidth grows with the perimeter you cross rather than being one shared wire.

Coherence adapts alongside: broadcast snooping gives way to directory protocols, where a home slice keeps a sharer list per line and forwards invalidations to exactly the caches that hold a copy, the same MESI states delivered point-to-point instead of broadcast.

One more design choice ties the levels together: whether the LLC is inclusive of the private caches. An inclusive LLC guarantees that every line in any L1 or L2 also has a copy (or at least a tag) in the LLC, which makes it a natural snoop filter: a request that misses in the LLC provably misses in every private cache too, so no core need be disturbed. The cost is capacity, since the LLC spends space duplicating what the private levels already hold; as private L2s grew, several server designs moved to non-inclusive LLCs and keep a separate directory of tags to preserve the filtering. Either way, the goal is the same: answer does anyone have this line? without broadcasting the question.

NUMA: when memory takes sides

One socket has one set of memory controllers. Servers bolt two or more sockets together with a cache-coherent socket-to-socket link (Intel UPI, AMD Infinity Fabric), and coherence still holds across the whole machine: a line dirty in socket 0's cache is found and flushed even when socket 1 asks. What stops being uniform is latency. Each socket's DRAM hangs off its own controllers; a core reaching its own socket's DRAM pays about 80–100 ns, while the same load to the other socket's DRAM crosses the link and pays 130–200 ns, plus it consumes cross-link bandwidth that is far scarcer than local bandwidth. Memory access is now non-uniform: NUMA.

The OS default — allocate a page on the node whose core first touches it — works until threads migrate. A thread that faults its working set in on socket 0, then gets rescheduled to socket 1, drags nothing with it: every one of its pages is now remote, and it runs 50–100 % slower with no code change at all. Hence thread affinity: pinning a thread to a core or node (sched_setaffinity, numactl) so the scheduler cannot separate it from its memory, plus allocating memory from the node where the consuming thread lives. NUMA-aware programs place threads first and memory second, in that order, deliberately.

To quantify the first-touch mistake, take a two-socket machine, local DRAM at $t_{\text{loc}}=90$ ns, remote at $t_{\text{rem}}=150$ ns, and a memory-bound parallel loop — every element is one DRAM access. If thread 0 zeroes the whole array first (the innocent memset), first-touch places every page on node 0. Then eight threads, four per socket, sweep the array: the four on socket 0 pay $t_{\text{loc}}$, the four on socket 1 pay $t_{\text{rem}}$ and pile onto the single cross-socket link. Average access is $\bar{t}=\frac{1}{2}(t_{\text{loc}}+t_{\text{rem}})=120\text{ ns},$ so the run is $\bar{t}/t_{\text{loc}}=120/90\approx 1.33\times$ slower than it should be — worse once link contention is added. Now have each thread initialize its own slab: first-touch places every thread's pages locally, every access is $t_{\text{loc}}$, the link carries nothing, and the same code runs a third faster. The two versions differ only in who runs the initialization loop — identical arithmetic, identical results, a stable 1.33x that no profiler pointed at the loop will ever explain.

In practice: decide the parallel decomposition — which thread owns which slab of data — before anything runs. Pin each thread to a core on the node where its slab will live. Then have each thread initialize its own slab, so first-touch places every page locally, rather than letting thread 0 zero the whole array from node 0 and leave half the machine's accesses remote for the run's lifetime. That single initialization mistake is the most common NUMA bug in numerical code, it is invisible in the source (the initialization looks like an innocent loop), and it costs a stable 1.5x for the rest of the run. The diagnosis, as always, is measurement: per-node memory traffic counters show one controller saturated and the other idle.

What actually limits scaling

The first lesson priced scaling with Amdahl's law: the serial fraction sets the ceiling. The floorplan adds two more costs, and both grow with core count.

Coherence traffic. Every write to a line another core holds costs an invalidation and later a miss, the fourth C. With more cores, more sharers exist per hot line, each invalidation reaches further across the ring or mesh, and true sharing (one lock, one counter, one queue head touched by everyone) turns into serialization delivered by hardware: the line's ownership hops core to core, one writer at a time, no matter how many cores wait. A single hot cache line can flatten a 32-core scaling curve by itself, which is why scalable designs shard hot state per-core and combine late, exactly as the false-sharing fix did.

Shared-LLC and bandwidth contention. Each added thread brings its working set, but the LLC does not grow: at some thread count the union stops fitting, the LLC miss rate climbs for everyone, and demand spills into a DRAM bandwidth that is also fixed. A program whose per-thread working set is small scales past this; one that streams memory saturates the controllers at four or six cores and flatlines, with additional cores merely queueing at the memory wall. The locality lessons compound in value on multicore: bytes you never fetch are bandwidth someone else gets to use.

Work one diagnosis end to end. You parallelize a program and measure: 1 thread does 10 units/s, 4 threads 34 (a solid $3.4\times$), 8 threads 40, 16 threads 41. The curve bends hard between 4 and 8 and flatlines after. The flatline at ~40 units/s says one shared resource is pinned; the counters say which. The curve's shape identifies the bottleneck, and two counters — per-thread IPC and LLC miss rate, plus measured memory latency — distinguish the cases:

Past the bandwidth wall, each added thread does not merely waste a core — it lengthens the DRAM queue and slows the threads you already had.

The machine, whole

This module completes the stack the course has been building. A core is the pipelined datapath this course assembled, kept busy by SMT when one stream stalls. Cores multiply because Dennard scaling ended; their private caches stay coherent through MESI; their store buffers relax ordering until fences and atomics restore it; and the whole system shares a sliced LLC, a ring or mesh, and, past one socket, NUMA memory with non-uniform costs. Every layer is one you have now built or reasoned through, and the capstone walks a single line of C down through all of them at once.

Interconnects, NUMA, and the roofline

The floorplan of a modern die is the subject of a large public literature, and a few landmarks orient the rest.

The on-die interconnect moved from bus to ring to mesh in production over about a decade. Intel's ring bus debuted in Sandy Bridge (2011) and served client parts for years; the switch to a 2-D mesh for the server line came with Skylake-SP (2017), documented in Intel's own architecture disclosures, because a ring's latency grows linearly with stops and a mesh's grows with the square root. The academic groundwork is older: Dally and Towles' Principles and Practices of Interconnection Networks (2004) is the standard text, and the network-on-chip research program it seeded (Dally and Towles, 2001, DAC) is why a chip's cores now talk over a packet-routed fabric rather than shared wires.

NUMA is not new either — Stanford's DASH multiprocessor (Lenoski et al., 1992, ISCA) built directory-based cache-coherent NUMA and measured exactly the local-versus-remote penalty this lesson quotes. What changed is that NUMA is now inside a single two-socket box on any server, so the first-touch and affinity discipline that used to be supercomputer lore is everyday performance engineering, exposed through numactl and libnuma on Linux.

The scaling curves at the end of the lesson have a clean formal companion in the roofline model (Williams, Waterman, and Patterson, 2009, CACM), which plots attainable performance against a kernel's arithmetic intensity — flops per byte of memory traffic — and shows a program pinned either under a sloped memory-bandwidth roof or a flat compute roof. The DRAM-limited flatline in this lesson's figure is the roofline's bandwidth roof seen from the thread-count axis: below a threshold intensity, adding cores cannot help, because the shared memory system, not the cores, is the binding constraint.

That closes the module. The machine on your desk is no longer a diagram of somebody else's design: it is SEQ, pipelined, cached, virtualized, replicated, and wired together — and the capstone is where all of it runs one program, end to end.