Memory Layout and Buffer Overflows

Every fact from this module — the stack growing down, the return address call pushes, arrays as bare base-plus-offset with no bounds check — converges on one of the most consequential bugs in computing: the buffer overflow. This closing lesson lays out how a process arranges its memory, shows how writing past a stack array can overwrite the return address and redirect ret, and covers the standard defenses. The framing throughout is defensive: the goal is to understand the mechanism so you can recognize and prevent it.

The process address space

A running process sees its virtual memory partitioned into regions, each holding a different kind of data and growing in a fixed direction. From the lowest addresses upward: the text segment of machine code, the data segments of globals, the heap that grows up as malloc hands out memory, and at the top the stack that grows down.¹

The stack and heap grow toward each other into the same gap, which is why their opposing directions matter: each has room to expand without a fixed boundary between them. Local arrays live in the stack region, right alongside the saved return addresses from procedures — the adjacency that makes the overflow possible.

How an overflow overwrites the return address

A stack-allocated array sits in the current frame at addresses below the saved return address, and the array grows toward higher addresses as it fills, toward the return address. C array writes carry no bounds check: a copy that writes more bytes than the array holds keeps going into whatever lies above it — and what lies above it is the return address.

void echo(void) {
    char buf[8];       // 8 bytes on the stack, below the return address
    gets(buf);         // gets() writes input with NO length limit
    puts(buf);
}

The vulnerability comes down to layout. The buffer is allocated low in the frame, and the return address that call pushed sits just above it. Input fills the buffer upward toward higher addresses, straight at the return address. Nothing in hardware separates the two; they are adjacent bytes in the same frame.

gets writes characters until a newline with no notion of buf's size, so an input longer than 8 bytes spills past buf's end. The figure shows the frame before and after such an overrun.

The distance is countable, which is what makes the bug mechanical. Let the buffer have size $b$ bytes at %rsp, with $g$ bytes of other locals and saved registers above it; on x86-64 the return address is 8 bytes at

In this minimal frame $b=8$, $g=0$, so the return address sits at 8(%rsp). An input of exactly 8 bytes fills buf and stops at the return address's doorstep; input bytes $9$ through $16$ land on the 8-byte saved return address, one input byte per address byte. To redirect ret to 0x401156 an attacker supplies 8 filler bytes followed by 0x401156 little-endian (56 11 40 00 00 00 00 00). No guessing about layout is needed once $b+g$ is known; the overflow is a straight byte-for-byte overwrite.

When echo finishes, its ret pops the now-corrupted bytes into %rip and the processor continues executing wherever those bytes point. The payload structure that achieves this is fixed once the offset is known:

In the benign case an arbitrary overwrite is a crash; in the exploit case the overwritten return address points back into the buffer, where the same input placed bytes meant to run as instructions. That is stack smashing in one sentence: input that is both data overflowing the buffer and a redirected return address pointing at attacker-chosen code. We keep the sketch conceptual deliberately; the point is the mechanism and its prevention, not a working exploit.

The fix at the source

The root fault is the unbounded read, and the first defense is to never write one. gets cannot be used safely: it takes no buffer size, so it will always overflow a buffer given long enough input, which is why it was removed from the C standard entirely. Its bounded replacement is fgets, which takes the destination size and stops one byte short of it to leave room for the terminator.

void echo(void) {
    char buf[8];
    fgets(buf, sizeof(buf), stdin);   // reads at most 7 bytes, then '\0'
    puts(buf);
}

The same discipline applies across the unsafe/safe pairs: strcpy → strncpy, sprintf → snprintf, strcat → strncat. Each safe form takes a length and refuses to write past it. The runtime defenses below exist because not all code can be rewritten, and even careful code has bugs, but a bounded read is the cleanest place to stop an overflow: it never happens.

Three runtime protections

The classic attack is a chain of three steps: overflow the buffer, reach and overwrite the return address, then have ret jump into attacker code on the stack. Each defense snaps one link, and breaking any one link breaks the chain.

Modern systems make this attack far harder with three layers of defense, each attacking a different link in the chain.²

Stack canaries. The compiler places a random canary value between the local buffers and the saved return address, and checks it just before ret. An overflow that reaches the return address must pass through the canary first, corrupting it; the mismatch is detected and the program aborts before the poisoned ret runs.

The compiled code makes the check visible. With -fstack-protector, gcc wraps the body in a prologue that stashes the canary and an epilogue that verifies it. The canary itself comes from %fs:40, a per-thread value the operating system seeds with random bits at thread start.

echo:
        subq    $24, %rsp          # allocate the frame
        movq    %fs:40, %rax       # load the per-thread canary
        movq    %rax, 8(%rsp)      # place it just below the return address
        xorl    %eax, %eax         # scrub the copy from the register
        # ... buf lives at (%rsp); the vulnerable read runs here ...
        movq    8(%rsp), %rax      # reload the canary
        subq    %fs:40, %rax       # compare with the original
        jne     .Lfail             # any difference: it was overwritten
        addq    $24, %rsp
        ret
.Lfail:
        call    __stack_chk_fail   # abort; never returns

An overrun large enough to reach the return address must cross the canary slot at 8(%rsp) and change it, so the subq %fs:40, %rax leaves a non-zero result and jne diverts to __stack_chk_fail. The program dies with a diagnostic instead of returning through a corrupted address.

Non-executable stack (NX). The hardware can mark the stack region as non-executable, so even if ret lands there, attempting to run bytes from the stack faults. This breaks the execute code I placed in the buffer half of the classic attack outright.

Address-space layout randomization (ASLR). The loader places the stack, heap, and libraries at randomized addresses on each run, so an attacker cannot reliably predict the address to point the corrupted return at. The hard-coded target that a fixed layout would permit is no longer knowable in advance.

None is a complete defense alone — canaries can be leaked, NX is countered by reuse of existing code, ASLR by information disclosure — but together they raise the cost of an attack enormously, which is why all three are on by default in current toolchains and operating systems.

The arms race after NX

The phrase NX is countered by reuse of existing code names an entire technique the textbook only alludes to: return-oriented programming (ROP). Once a non-executable stack forbids running attacker-supplied bytes, the attacker stops injecting code and instead chains together short snippets — gadgets — that already exist in the program's executable text, each ending in a ret. By overwriting the stack with a sequence of gadget addresses, the corrupted ret jumps to the first gadget, whose own ret pops the next address off the still-attacker-controlled stack, and so on, stitching existing instruction fragments into a new program without writing a single executable byte. Shacham's 2007 paper showed the available gadgets in a normal C library are Turing-complete, which is why NX alone does not end the game.³

The hardware answer, now shipping, is control-flow integrity enforced in silicon. Intel's CET (Control-flow Enforcement Technology) and ARM's equivalent add a shadow stack: a second, protected stack that holds a duplicate copy of every return address. call pushes to both stacks; ret checks that the return address on the ordinary stack still matches the shadow copy, and faults if an overflow has changed one but not the other. Because the shadow stack lives in memory the ordinary overflow cannot reach, it detects the return-address corruption that canaries only probabilistically catch. CET's companion feature, indirect-branch tracking, similarly constrains indirect jumps to legitimate targets, cutting off the jump-oriented cousin of ROP. Together they represent the current frontier of the same battle this lesson opened with: the return address is the crown jewel, and each defensive generation guards it more tightly than the last.⁴