đ Quality of Guarantees
What if the rock-solid foundations your system relies on⌠arenât?
đ§ Foundations Built on Sand?
Many software engineers like to think they work in a very âpureâ field - much like in math, where 1 + 1 always (okay, normally) equals 2, we assume a given operation with the same inputs always yields the same output. But does it? In reality, software is built atop countless conventions and guarantees - some strong, some⌠not so much.
đ Conventions
There are a lot of things we take for granted: Unix time starts on Jan 1st 1970; a zero exit code means success; index.html is the default page for a website. These are conventions. Theyâre not universal truths - they exist because we, as a software community, decided to standardise them.
Some conventions have sensible origins. Zero as false comes from checking if the electrical charge is greater than the required threshold. \n terminates a line, and Windowsâ \r returns the cursor to the start of the line thanks to the mechanical process of typewriters. Arrays start at 0 due to pointer arithmetic - see Dijkstraâs explanation. Others are more arbitrary: HTTP uses port 80 because Tim Berners-Lee picked it.
And yet, these conventions underpin everything from our shell scripts to our distributed services.
đĄď¸ Guarantees
Beyond conventions, we rely heavily on guarantees. Memory we set stays set - unless a cosmic ray flips a bit. UUIDv4 wonât collide - probably. Disk writes are durable after fsync() - usually.
Some guarantees sound strong but arenât:
- âąď¸ Time always moves forward? NTP corrections can rewind the clock.
- đ The internet always routes traffic correctly? BGP misconfigurations can misroute entire countries.
- â Floating point math is consistent? IEEE 754 compliance varies - and optimisations may reorder operations, making
a + b + cnot equal tob + a + cdue to non-associativity in floating point numbers.
đ An Issue of Scale
Yes, the internet is unreliable. Any request can fail. Networks partition. Clocks drift. But once you scale to thousands of requests per second, those rare edge cases become frequent visitors.
That shiny âfive ninesâ SLA (99.999% availability) sounds robust - but that still allows over 5 minutes of downtime per year. In practice, issues tend to cluster rather than distribute evenly - we notice some and fix them, but a steady trickle of subtle, overlooked blips still slip through. These unnoticed glitches can become landmines when they collide with systems that assumed certain behaviours were guaranteed.
Once you start hitting these edges, the cracks in your assumptions widen. What you thought was âguaranteedâ becomes âlikelyâ. What was ârareâ becomes âfrequentâ.
𧨠When Guarantees Arenât
Some guarantees look solid until you realise theyâre balanced on assumptions that arenât guaranteed at all. For example, a pattern I see gaining popularity is the Event Outbox pattern - designed to solve the issue of writing data and emitting an event in a single atomic operation.
database.update("user1234", "some_data")
messageQueue.push("Hey, I updated user1234")
Looks good? Not quite. If the system crashes in between those lines - or the coroutine thread is torn down - youâll lose the message. Runtimes like Node.js or collaborative concurrency libraries like asyncio in Python are particularly exposed here, but no environment is immune. A SIGKILL or OOM can cut off your process between those two lines.
Enter the Event Outbox: write the message to a table in the same transaction as the data update. A background service reads from this table and pushes to the queue, marking it as sent only once acknowledged.
This solves the crash-in-the-middle issue, but it doesnât fix more critical vulnerabilities:
- â What if a developer forgets to add the message to the outbox?
- đ What if a migration script corrects malformed data but skips messaging?
- đ§âđť What about manual DB edits? (Yes, Iâve seen cultures where engineers run direct SQL against prod to âfixâ bugs.)
Iâve worked with companies where entire teams - as much as 10% of the engineering org - were dedicated solely to repairing data inconsistencies: patching over bugs, correcting silent human mistakes, and recovering from missing or misfired events.
I brought this up in a recent interview. The tech director dismissed the problem with a shrug - âyou canât solve everything.â And sure, not everything needs solving. But here, you actually can harden a fragile, human-enforced guarantee into a deterministic computing one.
Rather than watching the outbox table with a CDC tool and hoping developers remember to populate it, monitor all database changes directly. Let the event stream be derived from the source of truth itself - the write-ahead log or a logical replication stream. With that in place, no manual edit, silent script, or forgotten migration can alter data without triggering an event.
â Wrapping Up
Most of what we rely on in software isnât as stable as we think. Some things are conventions with strong cultural momentum, others are guarantees with asterisks and fine print. Sometimes we build systems on rock; other times, weâre skating over ice and hoping it doesnât crack.
Knowing the difference - and designing with eyes open - is often what separates a system that wakes up the on-call team at 3 a.m. from one that keeps on ticking. The job isnât to eliminate every risk. Itâs to understand which ones youâve taken - and to recognise when itâs time to reassess, adapt, or reinforce those assumptions before they turn brittle.