What Actually Happens in a TLS Handshake

Every secure connection you make — loading a web page, an API call, a database driver connecting over TLS — starts with a handshake that happens in a few milliseconds and that almost nobody watches. It’s worth watching, because the handshake is where nearly every TLS problem actually lives. “Certificate error,” “handshake failure,” “connection reset during TLS” — these are all specific steps of a well-defined negotiation, and once you know the steps, the error tells you where you are instead of just that you’re stuck.

The job of the handshake

Before any application data flows, the client and server have to accomplish three things, in order and quickly:

1. Agree on how to talk — which TLS version and which cipher suite (the bundle of algorithms they’ll use).
2. Prove who the server is — the server presents a certificate chain, and the client verifies it actually vouches for the name it asked for.
3. Agree on secret keys — establish shared session keys that only the two of them know, without ever sending those keys across the wire.

Only after all three succeed does the encrypted application data start. Miss any one and you never send a single byte of real payload — which is why a TLS failure is total, not partial.

Step by step (the modern flow)

Here’s the sequence, in the shape today’s TLS uses:

• ClientHello. The client opens with its supported TLS versions, a list of cipher suites it can do, a random value, and — crucially — extensions. Two matter constantly: SNI (Server Name Indication), which tells the server which hostname you’re reaching so it can present the right certificate, and (in the current version) the client’s key-exchange material offered up front to save a round trip.
• ServerHello + certificate. The server picks a version and cipher suite from what the client offered, sends its own random value, and presents its certificate chain — the leaf cert for its identity plus the intermediates linking it toward a trusted root.
• Verification. The client checks that chain: does it build to a root the client trusts, is it in its validity window, and — the step people forget — does the name on the cert actually match the name the client asked for? A perfect certificate for the wrong hostname still fails here.
• Key agreement. Both sides use their exchanged material to independently derive the same session keys via a key-exchange algorithm. The shared secret is computed on each end, never transmitted, so someone recording the traffic can’t recover it.
• Finished. Each side sends a message proving it derived the same keys, and from that point the connection is encrypted. Application data begins.

The thing to internalize: identity (the certificate) and secrecy (the key exchange) are separate concerns handled in the same handshake. A cert proves who you’re talking to; the key exchange keeps the conversation private. Different failures, different fixes.

Where handshakes actually break

Map real errors onto the steps and they stop being generic:

• No shared version or cipher. ClientHello offers a set the server won’t accept (or vice versa) — common when one end is old and the other has dropped legacy protocols. Fails at the very first step, before any certificate.
• Certificate verification fails. Untrusted issuer, expired, name mismatch, broken chain — all the ways a cert can be wrong while looking fine, which is a whole post of its own: expiry is the least of your certificate problems.
• Wrong certificate presented. Usually a missing or wrong SNI — the server didn’t know which host you wanted and served a default cert for the wrong name.
• Reset mid-handshake. Often a middlebox, a firewall, or a protocol-version intolerance killing the connection partway.

Because it’s a negotiation, the failure names the step, and the step names the fix.

Mutual TLS: now the client proves itself too

The handshake above authenticates the server to the client. Mutual TLS (mTLS) adds the reverse: the server also requires the client to present a certificate and verifies it against a trusted issuer. Now both ends prove their identity with certificates, and the connection only completes if both chains check out. This is the backbone of a lot of service-to-service and machine-to-machine auth, and it’s why mTLS setups have twice the certificate surface to get right — two chains, two trust stores, two ways to be misconfigured.

Watch one, and it demystifies

The best way to make this concrete is to watch a handshake. A single verbose client connection against a live endpoint prints the negotiated version and cipher, the chain the server actually presented, the names on it, and the verify result — the whole negotiation, laid out. Do that once and TLS stops being a black box that either works or throws a scary error; it becomes a short, legible sequence where any failure points at a specific step. That’s the real payoff of knowing the handshake: it’s why a TLS failure is a trust problem you can locate rather than a wall you bounce off. If you’ve debugged a handshake down to one stubborn step, tell me which one.