Procedures

A procedure call must do several things at once: transfer control to the callee and back, pass arguments in, return a result out, and let both functions use the registers without trampling each other's data. x86-64 splits this work between two instructions — call and ret — and a body of conventions, the System V AMD64 ABI, that the hardware does not enforce but every compiler obeys. This lesson assembles the full picture of a function call.

The run-time stack

Each active function call needs private storage: space for local variables that do not fit in registers, for saved registers, and for arguments beyond the sixth. That storage is a stack frame, and frames are stacked on the same downward- growing run-time stack from data movement. When a function is called its frame is pushed; when it returns its frame is popped. %rsp marks the current top.

call and ret

The control transfer is the job of two instructions, and both use %rsp and the stack.

• call Label pushes the address of the instruction immediately after the call (the return address) onto the stack, then sets %rip to Label. It is pushq of the return address plus a jmp.
• ret pops the return address off the stack into %rip, resuming the caller at the instruction after its call.

Seen as control flow rather than stack edits, the pair is a round trip: call remembers where it was and jumps away; ret reads that remembered address and jumps back. The saved return address is what lets control resume in the caller.

Because the return address sits in ordinary stack memory, anything that overwrites it redirects where ret goes — the mechanism behind the buffer overflows of the final lesson.

Passing arguments and returning a value

The System V convention routes the first six integer or pointer arguments through registers, in a fixed order, and any further arguments on the stack. The result comes back in %rax.

long add3(long a, long b, long c) { return a + b + c; }

add3:
        leaq    (%rdi,%rsi), %rax  # rax = a + b   (args in %rdi, %rsi)
        addq    %rdx, %rax         # rax = a + b + c   (third arg in %rdx)
        ret                        # return value already in %rax

When there are more than six arguments

The first six arguments ride in registers; a seventh and beyond travel on the stack, placed there by the caller just below the return address. A function of eight arguments makes the boundary concrete: a..f arrive in the six registers, and g and h sit on the stack, which the callee reads as displacements off %rsp.

long add8(long a, long b, long c, long d,
          long e, long f, long g, long h) {
    return a + b + c + d + e + f + g + h;   // g, h passed on the stack
}

add8:
        addq    %rsi, %rdi         # a + b
        addq    %rdx, %rdi         # + c
        addq    %rcx, %rdi         # + d
        addq    %r8, %rdi          # + e
        addq    %r9, %rdi          # + f      (last register arg)
        addq    8(%rsp), %rdi      # + g      (7th arg, on the stack)
        addq    16(%rsp), %rdi     # + h      (8th arg, next stack slot)
        movq    %rdi, %rax         # return the running sum
        ret

The register arguments come in the fixed order %rdi, %rsi, %rdx, %rcx, %r8, %r9, exactly the sequence the figure above lists. The seventh and eighth are the interesting part: at function entry (%rsp) holds the return address call just pushed, so the caller placed g at 8(%rsp) and h at 16(%rsp), right above it. Reading 8(%rsp) and 16(%rsp) recovers them in order. This is the mechanism to watch for whenever a function takes many arguments — the register slots fill first, then the overflow spills onto the caller's stack.

Caller-saved versus callee-saved

The registers are shared between caller and callee, so the convention partitions them by who is responsible for a register's value surviving a call.¹

This convention is why the first lesson'smultstore wrapped its body in pushq %rbx / popq %rbx: it needed a register that survived the inner call mult2, chose the callee-saved %rbx, and therefore took on the obligation to preserve the caller's %rbx by saving it.

A stack frame through a two-deep call

The conventions become concrete when traced through an actual call chain. Let main call p, and p call q. The interesting function is p, because it holds a value across a call:

long q(long y) { return y + 1; }

long p(long x) {
    long t = q(x);        // q is free to clobber the caller-saved registers
    return x + t;         // but x must survive the call to q
}

p needs x both before and after call q, and q may overwrite every caller-saved register, including the argument register %rdi that carried x in. So p copies x into a callee-saved register, %rbx, and takes on the matching obligation: save the caller's %rbx on entry, restore it on exit.

p:
        pushq   %rbx              # save caller's %rbx (obligation for using it)
        movq    %rdi, %rbx        # keep x in callee-saved %rbx across the call
        call    q                 # t = q(x), result in %rax
        addq    %rbx, %rax        # rax = x + t
        popq    %rbx              # restore caller's %rbx
        ret
q:
        leaq    1(%rdi), %rax     # return y + 1
        ret

Watch the stack at three moments. After p's prologue, its frame holds the return address into main and the saved %rbx. When p executes call q, a second return address (into p) is pushed, growing the stack to its deepest point. When q returns, that address is popped and the stack shrinks back to exactly the shape it had before the call.

The call/ret pair moves %rsp by exactly 8 in each direction, so the frame is back to its (A) shape the instant q returns. p's own popq %rbx then undoes the prologue, and p's ret returns the stack to main untouched. Every function that plays by the rules leaves the stack exactly as it found it.

A recursive call traced

Recursion needs nothing special: each call gets its own frame, so each invocation's locals and saved state are independent. Here is recursive factorial and its frames.

long rfact(long n) {
    if (n <= 1) return 1;
    return n * rfact(n - 1);
}

rfact:
        cmpq    $1, %rdi          # compare n, 1
        jg      .Lrec             # if n > 1, recurse
        movl    $1, %eax          # base case: return 1
        ret
.Lrec:
        pushq   %rbx              # save callee-saved %rbx
        movq    %rdi, %rbx        # keep n across the recursive call
        leaq    -1(%rdi), %rdi    # arg = n - 1
        call    rfact             # rax = rfact(n - 1)
        imulq   %rbx, %rax        # rax = n * rfact(n - 1)
        popq    %rbx              # restore %rbx
        ret

The function stashes n in callee-saved %rbx because it must survive the recursive call, which is free to clobber the caller-saved argument registers. As the recursion descends, each level pushes its own return address and %rbx, so the stack carries one frame per pending call.

Unwinding multiplies on the way back: rfact(1) returns 1, rfact(2) computes $2\cdot 1$, rfact(3) computes $3\cdot 2$, each ret popping a frame and each restored %rbx supplying that level's n.

Follow rfact(3) value by value to see the role of the saved %rbx. The outer call saves the caller's %rbx, sets %rbx = 3, and calls rfact(2). That call saves the %rbx holding 3, sets %rbx = 2, and calls rfact(1). The innermost call takes the base case, returning %rax = 1 without touching %rbx at all. Now the returns fire in reverse: back in rfact(2), %rbx was restored to 2 by nothing yet — it is still 2 because this frame set it — so imulq %rbx, %rax gives $2\cdot 1=2$; its popq %rbx then restores the caller's %rbx to 3. Back in rfact(3), %rbx is 3, so imulq gives $3\cdot 2=6$, and its popq restores the original caller's %rbx. The result $6=3!$ emerges in %rax. Each frame's private %rbx held that level's n across the deeper call, which is the entire reason recursion needs no special machinery: the stack gives every invocation its own copy.

Tail calls and unwinding the stack

Two topics the wider literature treats that CS:APP touches only lightly are worth knowing because they change what the stack does. The first is the tail call. When the very last action of a function is to call another and return its result — like return g(x); with nothing after — the current frame is no longer needed once g starts, so a compiler can reuse it: instead of call g; ret, it emits a plain jmp g, letting g's own ret return directly to the original caller. This tail-call optimization turns a chain of recursive tail calls into a loop that uses constant stack space instead of one frame per call, which is why a tail-recursive loop does not overflow the stack the way naive rfact eventually would. The optimization is central to how functional-language compilers target the same machine, and gcc and clang both perform it at -O2.²

The second is stack unwinding: given a %rsp deep in a call chain, how does a debugger or a C++ exception reconstruct the sequence of pending calls? When the compiler omits the frame pointer (the common case now that %rbp is a general register), the saved return addresses are still on the stack, but their exact offsets vary per function. The answer is out-of-band metadata: the compiler emits DWARF call-frame information — a table, keyed by instruction address, describing where each function saved the return address and callee-saved registers. A debugger walks the stack by consulting this table frame by frame, and the same information drives C++ exception propagation. It is the reason an optimized, frame-pointer-free binary can still produce a correct backtrace.³