---
title: Memory Layout and Buffer Overflows
module: Machine-Level Programming
moduleNumber: 1
lessonNumber: 7
order: 107
summary: >
  The process address space — text, data, heap, and stack — and the classic
  vulnerability it enables. A stack buffer that is written past its end can
  overwrite the saved return address and redirect ret, so we sketch the mechanism
  defensively and then the three standard protections: stack canaries, a
  non-executable stack, and address-space layout randomization.
topics: [Machine-Level Programming]
sources:
  - book: Bryant & O'Hallaron
    ref: "CS:APP — §3.10.1 Understanding Pointers; §3.10.3 Out-of-Bounds Memory References and Buffer Overflow; §3.10.4 Thwarting Buffer Overflow Attacks"
---

Every fact from this module — the stack growing down, the return address `call`
pushes, arrays as bare base-plus-offset with no bounds check — converges on one of
the most consequential bugs in computing: the **buffer overflow**. This closing
lesson lays out how a process arranges its memory, shows how writing past a stack
array can overwrite the return address and redirect `ret`, and covers the standard
defenses. The framing throughout is defensive: the goal is to understand the
mechanism so you can recognize and prevent it.

## The process address space

A running process sees its virtual memory partitioned into regions, each holding a
different kind of data and growing in a fixed direction. From the lowest addresses
upward: the **text** segment of machine code, the **data** segments of globals, the
**heap** that grows up as `malloc` hands out memory, and at the top the **stack**
that grows down.[^layout]

| Region | Address range | Grows | Contents | Permissions |
| --- | --- | --- | --- | --- |
| Stack | high | down | frames, locals, return addresses | RW, NX |
| Heap | low → up | up | `malloc` allocations | RW, NX |
| Data | low | fixed | globals, statics | RW |
| Text | lowest | fixed | machine code | R, X |

$$
% caption: A process address space (high addresses at top). Read-only code and
% caption: globals sit low; the heap grows upward and the stack grows downward,
% caption: leaving a large unused gap between them.
\begin{tikzpicture}[font=\footnotesize, >=stealth,
  seg/.style={draw, minimum width=34mm, minimum height=8mm, inner sep=1pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[seg, draw=acc, fill=acc!8] (stk) at (0,3.0) {stack};
  \node[seg, minimum height=12mm] (gap) at (0,1.9) {unused gap};
  \node[seg] (heap) at (0,0.6) {heap};
  \node[seg] (data) at (0,-0.2) {data (globals)};
  \node[seg] (text) at (0,-1.0) {text (code)};
  % address labels on the right
  \node[anchor=west, font=\scriptsize] at (2.1,3.0) {high addr};
  \node[anchor=west, font=\scriptsize] at (2.1,-1.0) {low addr};
  % growth arrows, kept clear of the boxes
  \draw[->, thick, acc] (-2.5,3.4) -- (-2.5,2.4)
    node[midway, left, font=\scriptsize, align=center] {stack\\grows down};
  \draw[->, thick] (-2.5,0.2) -- (-2.5,1.2)
    node[midway, left, font=\scriptsize, align=center] {heap\\grows up};
\end{tikzpicture}
$$

The stack and heap grow toward each other into the same gap, which is why their
opposing directions matter: each has room to expand without a fixed boundary
between them. Local arrays live in the stack region, right alongside the saved
return addresses from
[procedures](/computer-architecture/machine-level-x86-64/procedures) — the
adjacency that makes the overflow possible.

## How an overflow overwrites the return address

A stack-allocated array sits in the current frame at addresses **below** the saved
return address, and the array grows toward higher addresses as it fills, toward the
return address. C array writes carry **no bounds check**: a copy that writes more
bytes than the array holds keeps going into whatever lies above it — and what lies
above it is the return address.

```c [echo.c]
void echo(void) {
    char buf[8];       // 8 bytes on the stack, below the return address
    gets(buf);         // gets() writes input with NO length limit
    puts(buf);
}
```

The vulnerability comes down to layout. The buffer is allocated low in the frame, and the
return address that `call` pushed sits just above it. Input fills the buffer
upward toward higher addresses, straight at the return address. Nothing in
hardware separates the two; they are adjacent bytes in the same frame.

$$
% caption: The fatal adjacency. buf is allocated low in the frame; the return
% caption: address pushed by call sits just above it. Input fills buf upward, toward
% caption: the return address, with no boundary between them.
\begin{tikzpicture}[font=\footnotesize, >=stealth,
  cell/.style={draw, minimum width=28mm, minimum height=7mm, inner sep=1pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[cell, draw=acc, fill=acc!8] (ret) at (0,1.5) {return address};
  \node[cell] (sav) at (0,0.8) {saved registers};
  \node[cell] (buf) at (0,0.1) {\texttt{buf[8]}};
  \node[anchor=west, font=\scriptsize] at (1.7,1.5) {high addr};
  \node[anchor=west, font=\scriptsize] at (1.7,0.1) {low addr};
  % input fills upward toward the return address
  \draw[->, thick, acc] (-2.0,0.1) -- (-2.0,1.5)
    node[midway, left, font=\scriptsize, align=center] {input\\f\/ills up};
\end{tikzpicture}
$$

`gets` writes characters until a newline with no notion of `buf`'s size, so an
input longer than 8 bytes spills past `buf`'s end. The figure shows the frame
before and after such an overrun.

$$
% caption: A stack frame before and after an overrun. Writing past buf's 8 bytes
% caption: continues upward into the saved return address; ret then jumps wherever
% caption: those overwritten bytes point.
\begin{tikzpicture}[font=\footnotesize, >=stealth,
  cell/.style={draw, minimum width=26mm, minimum height=7mm, inner sep=1pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  % ---- before ----
  \node[font=\scriptsize] at (0,2.6) {before};
  \node[cell, draw=acc, fill=acc!8] (rb) at (0,1.8) {return address};
  \node[cell] (mb) at (0,1.1) {saved regs};
  \node[cell] (bb) at (0,0.4) {\texttt{buf[8]}};
  \node[anchor=west, font=\scriptsize] at (1.5,0.4) {bottom of frame};
  % ---- after ----
  \node[font=\scriptsize] at (6,2.6) {after overrun};
  \node[cell, draw=red, text=red, thick] (ra) at (6,1.8) {\textbf{overwritten}};
  \node[cell, draw=red] (ma) at (6,1.1) {clobbered};
  \node[cell, draw=red] (ba) at (6,0.4) {input bytes};
  \node[anchor=west, text=red, font=\footnotesize] at (7.5,1.8) {\texttt{ret} jumps here};
  % overrun direction arrow to the left of the after-column, clear of the boxes
  \draw[->, thick, red] (4.4,0.1) -- (4.4,1.9);
  \node[text=red, font=\scriptsize, align=center] at (4.4,-0.35) {write\\past end};
\end{tikzpicture}
$$

The distance is countable, which is what makes the bug mechanical. Let the buffer
have size $b$ bytes at `%rsp`, with $g$ bytes of other locals and saved registers
above it; on x86-64 the return address is 8 bytes at

$$
\text{ret\_addr} = \mathtt{\%rsp} + (b + g), \qquad
\text{overwrite starts at input byte } b + g + 1.
$$

In this minimal frame $b = 8$, $g = 0$, so the return address sits at `8(%rsp)`. An
input of exactly 8 bytes fills `buf` and stops at the return address's doorstep;
input bytes $9$ through $16$ land on the 8-byte saved return address, one input byte
per address byte. To redirect `ret` to `0x401156` an attacker supplies 8 filler bytes
followed by `0x401156` little-endian (`56 11 40 00 00 00 00 00`). No guessing about
layout is needed once $b + g$ is known; the overflow is a straight byte-for-byte
overwrite.

$$
% caption: Byte offsets in echo's frame. buf occupies input bytes 1-8; bytes 9-16
% caption: overwrite the 8-byte return address one-for-one. Supplying a target
% caption: address as input bytes 9-16 redirects where ret jumps.
\begin{tikzpicture}[font=\footnotesize, >=stealth,
  cell/.style={draw, minimum width=44mm, minimum height=6.5mm, inner sep=2pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[cell, draw=red, text=red] (ret) at (0,0.7) {return address (input bytes 9 to 16)};
  \node[cell] (buf) at (0,0.0) {\texttt{buf[8]} (input bytes 1 to 8)};
  \node[anchor=west, font=\scriptsize] at (2.5,0.7) {at $\mathtt{8(\%rsp)}$};
  \node[anchor=west, font=\scriptsize] at (2.5,0.0) {at $\mathtt{(\%rsp)}$};
  \draw[->, thick, red] (-2.6,0.0) -- (-2.6,0.7)
    node[midway, left, font=\scriptsize, align=center] {input\\f\/ills up};
\end{tikzpicture}
$$

When `echo` finishes, its `ret` pops the now-corrupted bytes into `%rip` and the
processor continues executing wherever those bytes point. The payload structure that
achieves this is fixed once the offset is known:

```algorithm
payload := shellcode                          // bytes to run, land in buf
payload += filler up to length (b + g)        // pad to the return-address slot
payload += little_endian(&buf)                // overwrite ret_addr, point back at buf
```

In the benign case an arbitrary overwrite is a crash; in the **exploit** case the
overwritten return address points back into the buffer, where the same input placed
bytes meant to run as instructions. That is stack smashing in one sentence: input
that is both data overflowing the buffer and a redirected return address pointing at
attacker-chosen code. We keep the sketch conceptual deliberately; the point is the
mechanism and its prevention, not a working exploit.

## The fix at the source

The root fault is the unbounded read, and the first defense is to never write one.
`gets` cannot be used safely: it takes no buffer size, so it will always overflow a
buffer given long enough input, which is why it was removed from the C standard
entirely. Its bounded replacement is `fgets`, which takes the destination size and
stops one byte short of it to leave room for the terminator.

```c [echo-fixed.c]
void echo(void) {
    char buf[8];
    fgets(buf, sizeof(buf), stdin);   // reads at most 7 bytes, then '\0'
    puts(buf);
}
```

The same discipline applies across the unsafe/safe pairs: `strcpy` → `strncpy`,
`sprintf` → `snprintf`, `strcat` → `strncat`. Each safe form takes a length and
refuses to write past it. The runtime defenses below exist because not all code can
be rewritten, and even careful code has bugs, but a bounded read is the cleanest
place to stop an overflow: it never happens.

## Three runtime protections

The classic attack is a chain of three steps: overflow the buffer, reach and
overwrite the return address, then have `ret` jump into attacker code on the
stack. Each defense snaps one link, and breaking any one link breaks the chain.

$$
% caption: The attack as a three-link chain and the defense that breaks each link.
% caption: A canary guards the path to the return address, NX stops execution on the
% caption: stack, and ASLR hides the address to aim the overwrite at.
\begin{tikzpicture}[font=\footnotesize, >=stealth,
  lnk/.style={draw, minimum width=26mm, minimum height=10mm, align=center},
  def/.style={minimum width=26mm, minimum height=8mm, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \node[lnk] (s1) at (0,0)   {overf\/low\\the buf\/fer};
  \node[lnk] (s2) at (3.6,0) {overwrite\\return address};
  \node[lnk] (s3) at (7.2,0) {\texttt{ret} into\\stack code};
  \draw[->, thick] (s1.east) -- (s2.west);
  \draw[->, thick] (s2.east) -- (s3.west);
  \node[def, draw=acc, fill=acc!8] (d1) at (3.6,-1.8) {canary\\guards the path};
  \node[def, draw=acc, fill=acc!8] (d2) at (7.2,-1.8) {NX + ASLR\\break the jump};
  \draw[->, thick, acc] (d1.north) -- (s2.south);
  \draw[->, thick, acc] (d2.north) -- (s3.south);
\end{tikzpicture}
$$

Modern systems make this attack far harder with three layers of defense, each
attacking a different link in the chain.[^defend]

**Stack canaries.** The compiler places a random **canary** value between the local
buffers and the saved return address, and checks it just before `ret`. An overflow
that reaches the return address must pass through the canary first, corrupting it;
the mismatch is detected and the program aborts before the poisoned `ret` runs.

$$
% caption: The stack-canary mechanism. A random value sits between the buffer and
% caption: the return address; an overflow reaching the return address must first
% caption: alter the canary, and the pre-ret check catches the mismatch.
\begin{tikzpicture}[font=\footnotesize, >=stealth,
  cell/.style={draw, minimum width=28mm, minimum height=7mm, inner sep=1pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  \definecolor{red}{HTML}{C0392B}
  \node[cell] (ret) at (0,1.6) {return address};
  \node[cell, draw=acc, fill=acc!8, thick] (can) at (0,0.9) {canary (random)};
  \node[cell] (buf) at (0,0.2) {\texttt{buf[\,]}};
  % overf\/low arrow must pass through canary
  \draw[->, thick, red] (2.0,0.2) .. controls (3.0,0.9) .. (2.0,1.6)
    node[midway, right, font=\scriptsize, text=red, align=left] {overrun\\hits canary};
  % check annotation
  \node[anchor=east, text=acc, font=\scriptsize, align=right] at (-1.7,0.9)
    {checked\\before \texttt{ret}};
  \draw[->, acc] (-1.65,0.9) -- (can.west);
\end{tikzpicture}
$$

The compiled code makes the check visible. With `-fstack-protector`, gcc wraps the
body in a prologue that stashes the canary and an epilogue that verifies it. The
canary itself comes from `%fs:40`, a per-thread value the operating system seeds
with random bits at thread start.

```asm [canary.s]
echo:
        subq    $24, %rsp          # allocate the frame
        movq    %fs:40, %rax       # load the per-thread canary
        movq    %rax, 8(%rsp)      # place it just below the return address
        xorl    %eax, %eax         # scrub the copy from the register
        # ... buf lives at (%rsp); the vulnerable read runs here ...
        movq    8(%rsp), %rax      # reload the canary
        subq    %fs:40, %rax       # compare with the original
        jne     .Lfail             # any difference: it was overwritten
        addq    $24, %rsp
        ret
.Lfail:
        call    __stack_chk_fail   # abort; never returns
```

An overrun large enough to reach the return address must cross the canary slot at
`8(%rsp)` and change it, so the `subq %fs:40, %rax` leaves a non-zero result and
`jne` diverts to `__stack_chk_fail`. The program dies with a diagnostic instead of
returning through a corrupted address.

**Non-executable stack (NX).** The hardware can mark the stack region as
non-executable, so even if `ret` lands there, attempting to run bytes from the
stack faults. This breaks the "execute code I placed in the buffer" half of the
classic attack outright.

**Address-space layout randomization (ASLR).** The loader places the stack, heap,
and libraries at randomized addresses on each run, so an attacker cannot reliably
predict the address to point the corrupted return at. The hard-coded target that a
fixed layout would permit is no longer knowable in advance.

$$
% caption: ASLR across two runs of the same program. The stack, heap, and library
% caption: bases are shifted by different random offsets each time, so an address
% caption: hard-coded from one run misses in the next.
\begin{tikzpicture}[font=\footnotesize, >=stealth,
  seg/.style={draw, minimum width=20mm, minimum height=6mm, inner sep=1pt, align=center}]
  \definecolor{acc}{HTML}{2348F2}
  % run 1
  \node[font=\scriptsize] at (0,2.5) {run 1};
  \node[seg, draw=acc, fill=acc!8] (s1) at (0,1.9) {stack};
  \node[seg] (l1) at (0,0.7) {libraries};
  \node[seg] (h1) at (0,-0.1) {heap};
  % run 2, same segments at different heights
  \node[font=\scriptsize] at (4,2.5) {run 2};
  \node[seg, draw=acc, fill=acc!8] (s2) at (4,2.1) {stack};
  \node[seg] (l2) at (4,0.4) {libraries};
  \node[seg] (h2) at (4,-0.5) {heap};
  % a hard-coded target from run 1 lands in the wrong place in run 2
  \node[anchor=west, text=acc, font=\scriptsize] at (5.3,1.9) {address from run 1};
  \draw[->, acc] (1.05,1.9) -- (5.2,1.9);
  \node[anchor=west, font=\scriptsize, align=left] at (5.3,0.75) {misses the shifted\\layout of run 2};
  \draw[->, thick, black] (5.25,1.7) .. controls (5.9,1.2) .. (5.05,2.1);
\end{tikzpicture}
$$

| Protection | What it attacks | Effect |
| --- | --- | --- |
| Stack canary | reaching the return address | overflow corrupts canary, caught pre-`ret` |
| Non-executable stack | running code in the buffer | execution on the stack faults |
| ASLR | predicting the target address | layout differs each run, no fixed target |

None is a complete defense alone — canaries can be leaked, NX is countered by reuse
of existing code, ASLR by information disclosure — but together they raise the cost
of an attack enormously, which is why all three are on by default in current
toolchains and operating systems.

## The arms race after NX

The phrase "NX is countered by reuse of existing code" names an entire technique the
textbook only alludes to: **return-oriented programming** (ROP). Once a
non-executable stack forbids running attacker-supplied bytes, the attacker stops
injecting code and instead chains together short snippets — **gadgets** — that already
exist in the program's executable text, each ending in a `ret`. By overwriting the
stack with a sequence of gadget addresses, the corrupted `ret` jumps to the first
gadget, whose own `ret` pops the next address off the still-attacker-controlled stack,
and so on, stitching existing instruction fragments into a new program without writing
a single executable byte. Shacham's 2007 paper showed the available gadgets in a
normal C library are Turing-complete, which is why NX alone does not end the game.[^rop]

The hardware answer, now shipping, is **control-flow integrity** enforced in silicon.
Intel's CET (Control-flow Enforcement Technology) and ARM's equivalent add a
**shadow stack**: a second, protected stack that holds a duplicate copy of every
return address. `call` pushes to both stacks; `ret` checks that the return address on
the ordinary stack still matches the shadow copy, and faults if an overflow has
changed one but not the other. Because the shadow stack lives in memory the ordinary
overflow cannot reach, it detects the return-address corruption that canaries only
probabilistically catch. CET's companion feature, indirect-branch tracking, similarly
constrains indirect jumps to legitimate targets, cutting off the jump-oriented cousin
of ROP. Together they represent the current frontier of the same battle this lesson
opened with: the return address is the crown jewel, and each defensive generation
guards it more tightly than the last.[^cet]

> **Takeaway.** A process lays out **text**, **data**, **heap** (growing up), and
> **stack** (growing down). Because a local array sits just below the saved return
> address and C array writes are unchecked, an overrunning write can overwrite the
> return address and redirect `ret`. The standard defenses are **stack canaries**
> (a guard value checked before `ret`), a **non-executable stack** (NX), and
> **ASLR** (randomized region addresses) — layered, since each closes a different
> link in the chain.

[^layout]: **Bryant & O'Hallaron**, _CS:APP_, §3.10.1 — the run-time memory image: read-only code, initialized and uninitialized data, the upward-growing heap, and the downward-growing stack.
[^defend]: **Bryant & O'Hallaron**, _CS:APP_, §3.10.4 — Thwarting Buffer Overflow Attacks: stack canaries (stack protector), non-executable memory regions, and address-space layout randomization.
[^rop]: **Shacham**, "The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86)," _ACM CCS_ (2007) — return-oriented programming, chaining existing `ret`-terminated gadgets into arbitrary computation despite a non-executable stack.
[^cet]: **Intel**, _Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 1_ (2023), §17–§18 — Control-flow Enforcement Technology: the shadow stack that mirrors return addresses and faults on mismatch, plus indirect-branch tracking.