--- title: Floating Point module: Foundations moduleNumber: 0 lessonNumber: 4 order: 4 summary: > IEEE-754 trades the exactness of integers for enormous range by storing numbers as sign, exponent, and fraction — scientific notation in binary. We lay out the single and double formats, the bias that encodes the exponent, the three regimes (normalized, denormalized, special), a worked encode/decode, the four rounding modes and round-to-even at the bit level, why addition is not associative, the pitfalls of float-int conversion, and why 0.1 has no exact binary representation. topics: [Foundations] sources: - book: Bryant & O'Hallaron ref: "CS:APP — §2.4 Floating Point" --- Integers are exact but cramped: a 32-bit `int` cannot reach a billionth or a trillion. **Floating point** gains range by devoting bits to an _exponent_, storing a number as a sign, a fractional significand, and a scale — binary scientific notation. In exchange, most values are stored _approximately_, and the rules governing that approximation are subtle. This lesson lays out the IEEE-754 formats, decodes and encodes a concrete value, and explains the rounding behavior that makes `0.1` inexact. ## The IEEE-754 form Every IEEE-754 number is built from three fields — a sign bit $s$, an exponent field $\mathit{exp}$ of $e$ bits, and a fraction field $\mathit{frac}$ of $f$ bits — packed sign-first, exponent next, fraction last. The two standard widths fix $e$ and $f$: | Format | Total | Sign | Exponent $e$ | Fraction $f$ | Bias | | --- | --- | --- | --- | --- | --- | | Single (`float`) | 32 | 1 | 8 | 23 | 127 | | Double (`double`) | 64 | 1 | 11 | 52 | 1023 | $$ % caption: The 32-bit single-precision layout: one sign bit, an 8-bit exponent % caption: field, and a 23-bit fraction field, packed from the most significant % caption: bit downward. \begin{tikzpicture}[font=\footnotesize] \definecolor{acc}{HTML}{2348F2} % sign (1), exp (8), frac (23), widths to scale \draw[fill=acc!8] (0,0) rectangle (0.7,0.8); \node at (0.35,0.4) {$s$}; \draw[fill=acc!8] (0.7,0) rectangle (3.5,0.8); \node at (2.1,0.4) {$\mathit{exp}$}; \draw[fill=acc!8] (3.5,0) rectangle (11.5,0.8); \node at (7.5,0.4) {$\mathit{frac}$}; % bit-width brackets \node[text=acc] at (0.35,1.15) {1}; \node[text=acc] at (2.1,1.15) {8}; \node[text=acc] at (7.5,1.15) {23}; % bit indices \node[black] at (0.35,-0.35) {31}; \node[black] at (3.0,-0.35) {30..23}; \node[black] at (7.5,-0.35) {22..0}; \end{tikzpicture} $$ The general value, when the number is **normalized**, is $$ V = (-1)^s \times M \times 2^E, \qquad M = 1 + \mathit{frac}, \qquad E = \mathit{exp} - \text{Bias}. $$ Two design choices make this compact. The significand $M = 1.\!f$ has an **implied leading 1** that is never stored, gaining a free bit of precision. And the exponent is stored **biased** — as $E + \text{Bias}$, an unsigned field — so that the bit patterns sort in the same order as the numbers, letting integer comparison hardware compare floats. > **Definition (Bias).** The exponent is stored as the unsigned value > $\mathit{exp} = E + \text{Bias}$ with $\text{Bias} = 2^{e-1} - 1$. This is $127$ > for single precision and $1023$ for double, chosen so the representable exponents > straddle zero and the encoding is monotonic. ## Three regimes: normalized, denormalized, special The exponent field does double duty as a _selector_ among three encodings. The all-zeros and all-ones patterns are reserved; everything between is normalized. - **Normalized** ($\mathit{exp}$ neither all-0 nor all-1): the usual case above, $M = 1.\!f$ with implied leading one, $E = \mathit{exp} - \text{Bias}$. - **Denormalized** ($\mathit{exp} = 0$): the leading digit becomes $0$, so $M = 0.\!f$, and the exponent is fixed at $E = 1 - \text{Bias}$. This represents zero (when $\mathit{frac} = 0$, with a signed $\pm 0$) and the tiny values that fill the **gap between zero and the smallest normalized number**, giving _gradual underflow_. - **Special** ($\mathit{exp}$ all-1): $\mathit{frac} = 0$ encodes $\pm\infty$ (from overflow or division by zero); $\mathit{frac} \ne 0$ encodes **NaN**, the result of an undefined operation like $0/0$ or $\sqrt{-1}$. $$ % caption: The value line for non-negative single precision. Denormalized values % caption: pack densely just above zero, normalized values cover the wide middle, % caption: and the all-ones exponent is reserved for infinity and NaN. \begin{tikzpicture}[font=\footnotesize, >=stealth] \definecolor{acc}{HTML}{2348F2} \draw[->, thick] (-0.3,0) -- (12,0) node[right] {$V$}; % regime boundaries \draw[acc, thick] (0,0.18) -- (0,-0.18); \node[anchor=north] at (0,-0.25) {$0$}; \draw[acc, thick] (2.4,0.18) -- (2.4,-0.18); \node[anchor=north, text=acc, align=center] at (2.4,-0.3) {smallest\\normal}; \draw[acc, thick] (9.6,0.18) -- (9.6,-0.18); \node[anchor=north, text=acc, align=center] at (9.6,-0.3) {largest\\normal}; % regime labels above \node[align=center] at (1.2,0.75) {denormalized\\(gradual underf\/low)}; \node[align=center] at (6.0,0.75) {normalized}; \node[align=center, text=acc] at (10.9,0.75) {special:\\inf, NaN}; % shading bands \fill[acc!8] (0,0.05) rectangle (2.4,-0.05); \fill[acc!18] (2.4,0.05) rectangle (9.6,-0.05); \end{tikzpicture} $$ The denormalized region's purpose is continuity: without it there would be a sudden jump from the smallest normalized number straight to zero, and many small differences would flush to zero. Denormals fill that gap with evenly spaced values so $x - y = 0$ if and only if $x = y$. The numbers make the design visible in single precision. The smallest normalized value is $1.0 \times 2^{-126} \approx 1.18 \times 10^{-38}$; the denormals below it march down in uniform steps of $2^{-149}$, from $(1 - 2^{-23}) \times 2^{-126}$ just under the boundary to the smallest positive value of all, $2^{-149} \approx 1.4 \times 10^{-45}$. As a value decreases through this band it loses fraction bits to leading zeros, trading precision for range — **gradual underflow**, and the reason the newer IEEE vocabulary calls these values _subnormal_. ## Density: precision is relative, not absolute Fixed-point spacing is what integers have: neighbors sit exactly $1$ apart everywhere. Floating point instead keeps _relative_ spacing constant. Within one binade $[2^E, 2^{E+1})$ the $2^f$ fraction patterns are evenly spaced $2^{E-f}$ apart, and stepping into the next binade doubles the gap. A toy format with three fraction bits shows the shape: $$ % caption: A toy format with three fraction bits. Inside each binade the eight % caption: significands are evenly spaced; every new binade doubles the spacing. % caption: Values near 1 sit 1/8 apart, values near 8 a full 1/2 apart. \begin{tikzpicture}[font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \draw[->, thick] (0.9,0) -- (12.0,0); \foreach \i in {1,...,7} \draw[acc] ({1.4+\i*0.175},0.14) -- ({1.4+\i*0.175},-0.14); \foreach \i in {1,...,7} \draw[acc] ({2.8+\i*0.35},0.14) -- ({2.8+\i*0.35},-0.14); \foreach \i in {1,...,7} \draw[acc] ({5.6+\i*0.7},0.14) -- ({5.6+\i*0.7},-0.14); \foreach \v/\x in {1/1.4, 2/2.8, 4/5.6, 8/11.2} { \draw[thick] (\x,0.2) -- (\x,-0.2); \node[below] at (\x,-0.28) {\v}; } \node[text=acc, anchor=south] at (2.1,0.32) {step 1/8}; \node[text=acc, anchor=south] at (4.2,0.32) {step 1/4}; \node[text=acc, anchor=south] at (8.4,0.32) {step 1/2}; \end{tikzpicture} $$ The same geometry at full width explains a fact worth remembering: single precision represents every integer up to $2^{24}$ exactly, then starts skipping. At $2^{24}$ the gap between neighbors reaches $2$, so $16{,}777{,}217$ has no `float`. A `double`, with $52$ fraction bits, holds every integer through $2^{53}$ and then develops the same holes. Precision is about one part in $2^{f+1}$ of the magnitude, wherever on the line the value lives. ## A worked decode and encode Take the single-precision value $V = 6.5$. To **encode** it, first write it in binary: $6.5 = 110.1_2 = 1.101_2 \times 2^2$. Read off the three fields. The sign is $0$ (positive). The exponent is $E = 2$, so the stored field is $\mathit{exp} = E + 127 = 129 = 1000\,0001_2$. The fraction is the bits after the implied leading one: $\mathit{frac} = 101\,0000\dots_2$ (23 bits, trailing zeros). $$ % caption: Encoding 6.5 = 1.101 x 2^2 into single precision: sign 0, exponent % caption: field 2 + 127 = 129, fraction 101 padded to 23 bits. Decoding reverses % caption: it: 1.625 x 2^(129-127) = 6.5. The packed word is 0x40d00000. \begin{tikzpicture}[font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \draw[fill=acc!8] (0,0) rectangle (0.7,0.8); \node at (0.35,0.4) {$0$}; \draw[fill=acc!8] (0.7,0) rectangle (3.5,0.8); \node[text=acc] at (2.1,0.4) {$1000\,0001$}; \draw[fill=acc!8] (3.5,0) rectangle (11.5,0.8); \node[text=acc] at (7.5,0.4) {$101\,0000\,0000\,0000\,0000\,0000$}; \node at (0.35,1.15) {sign}; \node at (2.1,1.15) {$\mathit{exp}=129$}; \node at (7.5,1.15) {$\mathit{frac}$}; \end{tikzpicture} $$ To **decode** in reverse, read $s = 0$, $\mathit{exp} = 129$ so $E = 129 - 127 = 2$, and $M = 1.101_2 = 1 + \tfrac12 + \tfrac18 = 1.625$. Then $V = 1.625 \times 2^2 = 6.5$, recovering the original. The packed 32-bit word is `0x40D00000`. A tiny 8-bit toy format makes the three regimes visible in numbers small enough to check by hand. Give it one sign bit, a 4-bit exponent (bias $2^{4-1} - 1 = 7$), and a 3-bit fraction. Three encodings from this format, one per regime, show the selector at work. | Bits ($s\ \mathit{exp}\ \mathit{frac}$) | Regime | $E$ | $M$ | Value | | --- | --- | --- | --- | --- | | `0 0110 100` | normalized | $6-7=-1$ | $1.100_2 = 1.5$ | $1.5 \times 2^{-1} = 0.75$ | | `0 0000 100` | denormalized | $1-7=-6$ | $0.100_2 = 0.5$ | $0.5 \times 2^{-6} = 0.0078125$ | | `0 1111 000` | special | — | — | $+\infty$ | The denormal row is the instructive one: because $\mathit{exp} = 0$ pins the exponent at $E = 1 - \text{Bias}$ (not $0 - \text{Bias}$) and drops the implied leading one, the values slide continuously down toward zero instead of leaving a gap. Its neighbor `0 0000 001` decodes to $0.001_2 \times 2^{-6} = 2^{-3} \times 2^{-6} = 2^{-9}$, the smallest positive value this toy format can represent, and `0 0000 000` is $+0$. Line them up and the denormals fill the interval below the smallest normal $2^{-6} = 0.015625$ with evenly spaced steps of $2^{-9}$. ## Rounding: four modes, one default Most reals cannot be hit exactly, so every operation ends by **rounding** its true result to a representable neighbor. IEEE-754 defines four ways to choose the neighbor: - **Round-to-nearest-even** (the default): pick the closer neighbor; on an exact tie, pick the one whose last fraction bit is $0$. - **Round-toward-zero**: truncate, dropping the excess bits. This is the mode C uses for `float`-to-`int` casts. - **Round-down** (toward $-\infty$) and **round-up** (toward $+\infty$): the directed pair, useful for bracketing a true result between guaranteed bounds. Rounding the decimal values $1.4$, $1.6$, $1.5$, $2.5$, and $-1.5$ to integers separates the four: | Mode | $1.4$ | $1.6$ | $1.5$ | $2.5$ | $-1.5$ | | --- | --- | --- | --- | --- | --- | | Nearest even | $1$ | $2$ | $2$ | $2$ | $-2$ | | Toward zero | $1$ | $1$ | $1$ | $2$ | $-1$ | | Down | $1$ | $1$ | $1$ | $2$ | $-2$ | | Up | $2$ | $2$ | $2$ | $3$ | $-1$ | At the bit level, round-to-even looks at the **tail**, the bits below the last position kept, and compares it against one half of the last kept unit, the pattern $100\ldots$ Rounding to two fraction bits (nearest quarter): | Value | Tail against half | Result | | --- | --- | --- | | $10.00\underline{011}_2 = 2\tfrac{3}{32}$ | $011 < 100$, below half | $10.00_2 = 2$ | | $10.00\underline{110}_2 = 2\tfrac{3}{16}$ | $110 > 100$, above half | $10.01_2 = 2\tfrac14$ | | $10.11\underline{100}_2 = 2\tfrac78$ | exact tie, kept LSB $1$ is odd | $11.00_2 = 3$ | | $10.10\underline{100}_2 = 2\tfrac58$ | exact tie, kept LSB $0$ is even | $10.10_2 = 2\tfrac12$ | $$ % caption: Round-to-nearest-even as a decision procedure. Only an exact tie % caption: consults the kept LSB, and it always leaves that bit 0: ties round to % caption: the even neighbor. \begin{tikzpicture}[font=\footnotesize, >=stealth, box/.style={draw, minimum height=7mm, inner sep=3pt, align=center}] \definecolor{acc}{HTML}{2348F2} \node[box] (root) at (0,2.3) {tail bits below the kept LSB}; \node[box] (lt) at (-4.6,0.9) {tail $<$ 100...}; \node[box] (eq) at (0,0.9) {tail = 100... (tie)}; \node[box] (gt) at (4.6,0.9) {tail $>$ 100...}; \node[box, fill=acc!8] (down) at (-4.6,-0.7) {round down\\(truncate)}; \node[box, fill=acc!8] (keep) at (-1.7,-0.7) {kept LSB 0:\\stay (even)}; \node[box, fill=acc!8] (bump) at (1.7,-0.7) {kept LSB 1:\\round up (even)}; \node[box, fill=acc!8] (up) at (4.6,-0.7) {round up\\(add 1 at LSB)}; \draw[->, black] (root.south) -- (lt.north); \draw[->, black] (root.south) -- (eq.north); \draw[->, black] (root.south) -- (gt.north); \draw[->, black] (lt.south) -- (down.north); \draw[->, acc] (eq.south) -- (keep.north); \draw[->, acc] (eq.south) -- (bump.north); \draw[->, black] (gt.south) -- (up.north); \end{tikzpicture} $$ Why bother with the even rule? Always rounding ties the same direction (say, up) introduces a systematic drift across a long sum; ties to even go up half the time and down half the time, and the bias cancels. The same principle in decimal is the familiar "round half to even" (banker's rounding): $0.5$ rounds to $0$, but $1.5$ and $2.5$ both round to $2$.[^round] ## Rounding twice is not rounding once A subtle corollary: rounding to an intermediate precision and then to the final one can give a different answer than rounding directly, even when every step is correctly rounded. Take $1.0101_2 = 1.3125$ and round it to one fraction bit. Directly, the tail $101 > 100$ is above half, so the result is $1.1_2 = 1.5$. Now go through two fraction bits first: the tail $01$ rounds $1.0101_2$ down to $1.01_2 = 1.25$, and $1.01_2$ is an exact tie at one fraction bit, so the even rule picks $1.0_2 = 1.0$. Direct rounding gives $1.5$; **double rounding** gives $1.0$. The first rounding discarded the information that the value sat _above_ the halfway point. This occurs in real hardware. The x87 floating-point unit that predates x86-64's vector registers computes everything in 80-bit extended precision and rounds again when a result spills to a 64-bit `double`, so the same C expression could yield different bits depending on when the compiler spilled. The same effect occurs in `float f = d1 + d2;`, which rounds the sum to `double` and then rounds that to `float`.[^fpc] ## Why 0.1 is not exact A binary fraction can represent exactly only those values whose denominator is a power of two. The decimal $0.1 = 1/10$ has a factor of $5$ in its denominator, so in binary it is a _repeating_ expansion, just as $1/3 = 0.333\dots$ repeats in decimal: $$ 0.1_{10} = 0.0\overline{0011}_2 = 0.0001100110011001100\ldots_2. $$ Stored in a 23- or 52-bit fraction, this infinite tail is truncated and rounded, so the value held is _near_ $0.1$ but not equal to it. The stored `double` is actually $0.1000000000000000055511\ldots$, which is why the classic test fails. ```c [point_one.c] #include int main(void) { double sum = 0.1 + 0.2; printf("%.17f\n", sum); /* 0.30000000000000004 */ printf("%d\n", sum == 0.3); /* prints 0: NOT equal */ return 0; } ``` Neither $0.1$, $0.2$, nor $0.3$ is exactly representable, and the rounding errors in $0.1$ and $0.2$ do not cancel against the rounding error in the stored $0.3$. This is not a bug in any one machine; it is inherent to base-2 fractions, and the practical rule is to never compare floats with `==`, testing instead whether the difference is below a small tolerance.[^cmp] ## Addition that refuses to associate Because every operation rounds, algebraic identities that hold for reals fail for floats. The most consequential loss is associativity. Floating-point addition is still commutative, and it is monotonic, but the grouping of a sum changes its value: ```c [assoc.c] float a = (3.14f + 1e10f) - 1e10f; /* 0.0f */ float b = 3.14f + (1e10f - 1e10f); /* 3.14f */ ``` The mechanism is **absorption**, and it is visible at the bit level. To add two floats the hardware must first align their radix points, shifting the smaller operand's significand right by the difference in exponents. Here $10^{10} \approx 2^{33.2}$ and $3.14 \approx 2^{1.65}$, an exponent gap of about $32$; shifting a 24-bit significand right by $32$ slides every bit past the edge of the fraction, so the addition contributes nothing and the parenthesized sum is exactly $10^{10}$. Subtracting $10^{10}$ then leaves $0$. Group the other way and $10^{10} - 10^{10} = 0$ first, preserving $3.14$ intact. $$ % caption: Absorption during alignment. Adding 3.14 to 10^10 shifts the smaller % caption: significand right by the exponent gap of 32 bits; all 24 of its bits % caption: slide past the fraction's edge and are rounded away. \begin{tikzpicture}[font=\footnotesize, >=stealth] \definecolor{acc}{HTML}{2348F2} \draw[fill=acc!8] (0,1.3) rectangle (5.2,2.0); \node at (2.6,1.65) {24 bits of 1e10}; \draw[dashed, black] (5.2,-0.9) -- (5.2,2.35); \node[anchor=west, black] at (5.35,2.2) {edge of the kept fraction}; \draw[draw=acc, fill=acc!8] (7.0,0.1) rectangle (9.6,0.8); \node[text=acc] at (8.3,0.45) {24 bits of 3.14}; \draw[->, acc] (2.6,0.45) -- (6.85,0.45); \node[anchor=south, text=acc] at (3.55,0.55) {shifted right 32}; \node[anchor=west, black] at (5.35,-0.55) {everything past the edge is dropped}; \end{tikzpicture} $$ The practical consequence: the order in which a program sums a list changes the answer, so compilers do not reorder floating-point arithmetic unless explicitly told correctness does not matter, and numerical code that adds values of wildly different magnitudes must choose the order that loses less.[^ops] ## Crossing between int and float C converts freely between the integer and floating families, and each direction has a failure mode worth knowing cold. - **`int` to `float` rounds.** A `float` fraction holds $23$ bits, so integers above $2^{24}$ are not all representable: `16777217`, which is $2^{24} + 1$, rounds to the even neighbor $16777216.0$. Conversion to `double` is exact for every `int` ($52 > 31$) but rounds for large `long` values. - **`float` to `int` truncates**, rounding toward zero: `3.9` becomes `3`, `-3.9` becomes `-3`. - **Out-of-range conversions have no answer.** Casting `1e10f` or a NaN to `int` is undefined; x86-64 hardware returns the "integer indefinite" pattern `0x80000000`, which is $TMin$, regardless of sign. ```c [convert.c] int big = 16777217; /* 2^24 + 1 */ float f = big; /* rounds to 16777216.0f */ int back = (int) f; /* 16777216: the round trip lost the 1 */ int t = (int) -3.9f; /* -3: truncation toward zero */ int boom = (int) 1e10f; /* undefined; x86-64 yields INT_MIN */ ``` The round trip is the case that surfaces in real systems: an integer stored in a floating type comes back changed once it crosses the fraction width, $2^{24}$ for `float` and $2^{53}$ for `double` — the latter being why integer identifiers silently lose precision past $2^{53}$ in every language whose only number type is a double, with no warning at either conversion.[^fpc2] ## The low-precision formats of machine learning CS:APP covers the two IEEE-754 widths that existed when it was written, single and double. Machine learning has since driven a proliferation of _narrower_ formats, and they are a direct application of everything above: each is the same sign-exponent-fraction layout, trading fraction bits (precision) against exponent bits (range) for a specific workload. The IEEE **half** precision (`fp16`, 1/5/10) has a 5-bit exponent, so its range tops out near $65{,}504$ — small enough that gradients in a deep network overflow to infinity. Google's **bfloat16** (1/8/7) keeps `float`'s _full_ 8-bit exponent and its range, spending only 7 bits on the fraction; it deliberately sacrifices precision to keep the dynamic range — the trade training tolerates and overflow does not.[^bf16] Training is far more sensitive to _range_ (not overflowing) than to the last few bits of _precision_, so truncating a `float` to its top 16 bits — which is all bfloat16 is — costs almost nothing. | Format | Bits | Exp | Frac | Max finite | Use | | --- | --- | --- | --- | --- | --- | | double | 64 | 11 | 52 | $\sim 1.8 \times 10^{308}$ | general | | float | 32 | 8 | 23 | $\sim 3.4 \times 10^{38}$ | general | | bfloat16 | 16 | 8 | 7 | $\sim 3.4 \times 10^{38}$ | ML training | | fp16 | 16 | 5 | 10 | $65{,}504$ | ML inference | | fp8 (E4M3) | 8 | 4 | 3 | $448$ | ML inference | The trend has run all the way down to 8-bit floats, standardized by the OCP in 2023 in two flavors — E4M3 (more precision) and E5M2 (more range) — for the weights and activations of large models, where storing a number in one byte instead of four quarters the memory traffic that dominates inference cost.[^fp8] Each new format is the sign-exponent-fraction layout of this lesson, re-partitioned. Reading the low-precision literature comes down to two questions about any format: how many exponent bits (what is its range and where does it overflow), and how many fraction bits (how coarse is its spacing) — the same two knobs that separate `float` from `double`. > **Takeaway.** IEEE-754 stores a number as $(-1)^s \times M \times 2^E$ across a > sign bit, an $e$-bit biased exponent ($\text{Bias} = 2^{e-1}-1$, so $127$/$1023$), > and an $f$-bit fraction (single $1/8/23$, double $1/11/52$). The exponent field > selects **normalized** ($M = 1.\!f$), **denormalized** ($\mathit{exp}=0$, $M=0.\!f$, > gradual underflow), or **special** (all-1s: $\pm\infty$, NaN). Spacing doubles > each binade, so precision is relative and integers past $2^{f+1}$ develop holes. > Every operation rounds (to even by default), which is why $0.1$ is inexact, why > addition is not associative, why rounding twice differs from rounding once, and > why an `int` does not survive a round trip through `float` past $2^{24}$. Floating point and integers cover the numbers; the final foundations lesson returns to the bits themselves as logical objects, in [boolean algebra and bit manipulation](/computer-architecture/foundations/boolean-algebra-and-bit-manipulation). [^round]: **Bryant & O'Hallaron**, _CS:APP_, §2.4.4 — Rounding: the four modes, round-to-nearest-even as the IEEE default, and why it avoids the statistical bias of always rounding ties one way. [^cmp]: **Bryant & O'Hallaron**, _CS:APP_, §2.4.2–2.4.6 — Floating-Point Representation and Operations: rounding error makes most decimal fractions inexact in binary, so equality tests on floats are unreliable. [^fpc]: **Bryant & O'Hallaron**, _CS:APP_, §2.4.6 — Floating Point in C: the x87 extended-precision registers and the double-rounding hazards of computing wider than the stored type. [^ops]: **Bryant & O'Hallaron**, _CS:APP_, §2.4.5 — Floating-Point Operations: addition is commutative and monotonic but not associative, and multiplication does not distribute over addition. [^fpc2]: **Bryant & O'Hallaron**, _CS:APP_, §2.4.6 — Floating Point in C: conversion rules among `int`, `float`, and `double`, including rounding on int-to-float and truncation with undefined out-of-range behavior on float-to-int. [^bf16]: **bfloat16** originates in Google's TPU work; the rationale that neural-network training needs `float`'s dynamic range far more than its precision, so the format keeps the 8-bit exponent and truncates the fraction to 7 bits, is documented in the TPU and TensorFlow numerics literature (e.g. Wang & Kanwar, "BFloat16: The secret to high performance on Cloud TPUs," 2019). [^fp8]: **Micikevicius et al.**, "FP8 Formats for Deep Learning" (2022), and the **Open Compute Project** OCP 8-bit floating-point specification (2023): the two 8-bit formats E4M3 and E5M2 that trade one exponent bit for one fraction bit, adopted for the weights and activations of large models.