Numerical Analysis

If you work on building computer systems, chances are you'll need to understand how floating-point arithmetic works. Surprisingly, there aren't many clear and detailed resources out there on the topic. One of the few books that covered it in depth - Floating-Point Computation by Pat H. Sterbenz - is difficult to acquire and tedious to read.

In this part of the site, I'm building a practical introduction to the numerical analysis behind floating-point arithmetic for system builders.

This is a work in progress, and will probably remain so for quite some time - I am still stuck on just making mathematical symbols appear properly here.

Theorem 1 - Relative error

Theorem 2 - Subtraction with guard digit

Theorem 11 - Subtraction error

Theorem 15 - Uniquely recoverable floating point numbers

Rounding errors

Different rounding strategies affect the results of basic math operations like addition, subtraction, multiplication, and division. There are two key ways to measure rounding error: ulps (units in the last place) and relative error.

Exactness

In floating-point, exactness is rare due to rounding. But knowing when you can subtract without error is useful for designing robust numerical algorithms—like square root calculations, computing differences, or detecting small perturbations.