A Blind Write Is a Race You'll Lose

The read-modify-write is the most natural thing in the world: fetch a value, compute a new one from it, write it back. It’s also a lost-update race waiting to happen, and the ugly part is that it works perfectly every time you test it, because you test it alone. Put two of them in flight at once — two requests, two workers, a background job racing a live user — and one silently overwrites the other’s change. No error. No conflict. Just a write that quietly won a race it shouldn’t have been allowed to run.

The lost update, in slow motion

Two actors both read the value 10. The first adds 5 and writes 15. The second, working from the 10 it read a moment ago, adds 3 and writes 13. The first actor’s update is gone — not merged, not rejected, just erased — and nothing anywhere reports a problem. Each write was individually valid. The database did exactly what it was told. The bug is that neither write knew the other existed.

The dangerous concurrency bug isn’t a crash. It’s two correct-looking writes where the second quietly deletes the first, and no log line ever mentions it.

This is invisible in every test with one caller, which is why it survives to production and then only shows up under load — intermittent, unreproducible, and infuriating, exactly the signature of a concurrency bug.

Make the write conditional on what you read

The fix is to stop writing blindly and start writing conditionally: attach to your write the value you originally read, and tell the system to only apply the change if the value is still that. “Set this column to the new value only if it currently equals the old one I based my change on.” If a concurrent writer got there first, the current value no longer matches, the condition fails, and your write is rejected instead of clobbering. Now the race is safe: at most one writer wins, and the loser finds out.

Databases expose this directly — a conditional update, a compare-and-set, a lightweight transaction. I leaned on exactly this doing a data migration against a table that live writers were still touching: every rewrite was conditional on the value I’d read, so if a real user changed a row mid-migration, my write failed and skipped that row rather than stomping their change. The migration could run against a moving target precisely because it refused to overwrite anything it hadn’t seen.

The pattern has many names but one idea

Compare-and-set shows up everywhere under different labels, and they’re all the same move — guard the write with the state you assumed:

• Optimistic locking — a version column that increments on each write; your update includes the version you read, and mismatches are rejected.
• HTTP conditional requests — ETag plus If-Match, so a PUT fails with a conflict if the resource changed since you fetched it.
• Atomic compare-and-swap — the CPU instruction the whole idea is named after; swap only if the memory still holds the expected value.
• Conditional writes / lightweight transactions — the database-level version I used.

Different layers, identical logic: I believe the value is X; make my change only if that’s still true. The value you read becomes a precondition, so a stale assumption fails instead of silently winning.

Handle the failure, because now there is one

The trade you’re making is that writes can now fail, and that’s the point — but it means you have to handle the rejection. Usually that’s a small retry loop: re-read the current value, recompute from it, try the conditional write again; repeat until it sticks or you give up. That’s optimistic concurrency in a sentence — assume you’ll win, verify you did, retry if you didn’t. It’s cheap when contention is rare (the common case) and it degrades honestly when contention is high, instead of degrading into silent data loss.

The mindset shift is small and load-bearing: a write is not just “put this value there,” it’s “put this value there given what I assumed.” Once you write down the assumption and let the system enforce it, the lost-update race stops being a race at all — it becomes a failed precondition you can see and retry. If you’ve hunted down a lost-update bug that only showed up under real traffic, I’d like to hear how it surfaced.