Your Client Decides Which Server Gets Hammered

You point your application at a cluster of, say, five database nodes, and you assume the work spreads across all five. Sometimes it does. Often it doesn’t — and the thing that decides isn’t the cluster, it’s your client. The driver’s load-balancing policy, the contact points you configured, whether it’s aware of which node owns which data — those client-side choices determine which server actually eats your traffic. I learned this the way most people do: by watching one node in a “balanced” cluster pinned at capacity while the others sat nearly idle, and realizing the imbalance was coming from my side of the wire.

The cluster doesn’t route your requests; your driver does

It’s tempting to think of a cluster as a single smart entity that receives your requests and distributes them internally. But you connect with a client library, and that library makes the routing decision before the request leaves your process — which node to send this query to. Different policies produce wildly different distributions:

• Round-robin spreads requests evenly across nodes regardless of what the request is about.
• Token-aware / data-aware routing sends each request to the node that actually owns the relevant data, avoiding an extra internal hop.
• A single contact point — the naive setup — can funnel a disproportionate share through one coordinator node, especially if the client never learned about the others.

Same cluster, same query, three completely different pictures of which node is busy. The load distribution is a property of your client’s configuration at least as much as the cluster’s.

A cluster is only as balanced as the client talking to it. The nodes don’t reach out and grab work — the driver decides where it lands.

A “hot node” is often a client-routing story

This flips how you diagnose a hotspot. When one node is running hot and the rest are lazy, the first instinct is “something’s wrong with that node.” Sometimes. But just as often the node is fine and your client is aiming at it — a single contact point, a policy that isn’t data-aware, a connection pool that latched onto one coordinator. The node isn’t slow; it’s oversubscribed by a client that decided to send it everything. You can spend a long time “fixing” a server that was only ever guilty of receiving the traffic your driver pointed at it.

Your benchmark might be measuring the driver

This bites hardest in benchmarking, where it quietly poisons your numbers. If your load generator’s client concentrates requests on one node, you’re not measuring “the cluster’s throughput” — you’re measuring one node’s throughput, plus your client’s routing bias. Scale the cluster and the number won’t move, because the bottleneck was never the cluster; it was the single node your client kept talking to. That’s a close cousin of the lesson that the load generator is part of the experiment: the client isn’t a neutral observer, it’s an active participant whose configuration shapes the result. A benchmark that ignores the driver’s routing is measuring the wrong system.

Know your client’s policy, and configure it on purpose

The fix is unglamorous: treat the client’s load-balancing policy as a real, deliberate configuration decision, not a default you inherited. Give it the full set of nodes to discover, not one contact point it clings to. Use data-aware routing if the system supports it and your access pattern benefits. And when you measure or debug distribution, look at the client’s config first, because that’s where the routing decision is actually made.

The broader principle reaches past databases: in a lot of “distributed” systems, the intelligence about where load goes lives in the client — DNS-based load balancing, service-mesh sidecars, connection pools, SDK retry-and-route logic. If you only look at the servers, you’re looking downstream of the decision. The thing that chose the victim is upstream, in the caller. So when the distribution looks wrong, I’ve learned to turn around and check the client before I go blaming a node. If you’ve chased a “hot node” that turned out to be your own driver’s routing, I’d like to hear how you caught it.