Cheap Writes Have a Bill, and It's Called Compaction

A whole family of modern databases — the ones built on log-structured merge trees, like Cassandra, RocksDB, and their relatives — advertise blazing write performance, and they deliver it. What the headline number leaves out is how they’re fast: they make writes cheap by not doing the expensive part yet. The expensive part is real, it’s deferred, not deleted, and it comes back as a background process called compaction. Understand that and you understand why an LSM-backed system can be a dream under normal load and fall over under sustained writes.

Fast writes by deferral, not by magic

An LSM engine takes a write by appending it to an in-memory buffer and, eventually, flushing that to a new immutable file on disk. It doesn’t go find the old value and update it in place — it just writes the new version and moves on. That’s why writes are so cheap: they’re basically sequential appends, and the engine never blocks a write to reconcile it with what’s already there. Beautiful, until you notice what’s accumulating.

Because nothing is updated in place, the same key can now live in several files, with older versions and deletions scattered across them. Reads have to reconcile all of that, and the files pile up. Something has to periodically merge them back down — throw away superseded values, drop deleted rows, produce fewer, cleaner files. That something is compaction, and it’s the bill for all those cheap writes.

An LSM tree doesn’t make the write work disappear. It puts it on a tab, and compaction is when the tab comes due.

Compaction competes with the work you actually care about

Here’s the tension: compaction runs in the background, but it isn’t free background magic. It reads and rewrites large amounts of data, so it hammers the same disk I/O and CPU that your live reads and writes need. Under gentle load, it keeps up quietly and you never think about it. Under sustained heavy writes, it can fall behind — new files arrive faster than it can merge old ones — and now you’re in trouble on several fronts at once:

• Read amplification. More un-merged files means a read has to check more places to answer a query. Read latency climbs precisely when the system is busiest.
• Space amplification. Superseded and deleted data hangs around waiting to be compacted, so disk usage balloons well past the size of the live dataset. A disk that “should” be half full fills up.
• I/O contention spikes. When compaction finally does run hard to catch up, it steals I/O from live traffic, and latency lurches. The system gets slower right when it was already struggling.

The cruel part is timing: the moment you most need the database — peak write load — is exactly when compaction is most likely to fall behind and make everything worse.

Read, write, and space amplification are a pick-two

LSM engines expose a genuine three-way tradeoff, and it’s worth knowing before you trust a benchmark. Compact aggressively and you keep reads fast and space tight, but you spend more I/O on compaction (hurting write throughput). Compact lazily and writes fly, but reads and disk usage suffer. Different compaction strategies just move where you sit on that triangle — optimized for write-heavy, read-heavy, or space-constrained workloads. There’s no setting that wins all three; there’s only the one that matches your workload. A write-optimized engine tuned like a read-optimized one will disappoint you, and the disappointment shows up under load, not in the demo.

Budget the headroom the deferral requires

Once you see compaction as deferred write cost, operating these systems gets clearer:

• Leave I/O and CPU headroom for compaction, don’t size the box for the write load alone. Compaction needs room to run, and a node pinned at 100% serving traffic has nothing left to pay down its tab.
• Keep real disk slack for space amplification — well beyond your live data size — or a burst of writes plus lagging compaction fills the disk and takes the node down.
• Watch compaction as a first-class metric. Pending compaction backlog growing over time is an early warning that the tab is outpacing your ability to pay it. It’s quiet until it isn’t.
• Match the compaction strategy to the workload, on purpose, instead of trusting the default to fit.

Know your engine’s happy-path lie

The transferable lesson reaches past any one database: every storage engine has a trick that makes its headline number look great, and a corresponding cost hiding somewhere less visible. LSM trees trade cheap writes for background merge work and space overhead. B-trees trade read-friendly layout for write amplification of their own. Knowing which corner your engine cuts, and where the bill lands, is the difference between a system that scales and one that surprises you. It’s the same “understand the failure, not the spec sheet” instinct as choosing storage by how it fails, and compaction is a close cousin of the other quiet, scheduled upkeep in eventual consistency has homework — invisible work that your guarantees silently depend on. Cheap writes are wonderful. Just remember you’re running a tab, and plan for the day it’s called. If you’ve watched compaction fall behind under load, I’d like to hear how it went.