Contents

Global Request Routing for a Hosted-Services Platform

Routing at this scale is a state-distribution problem wearing a proxy costume. The proxying itself — terminate TLS, pick a backend, forward bytes — is well-trodden. The hard part is that 4 million hostnames, 1 million certificates, and a mapping table that churns tens of thousands of times a minute all have to be near every edge request, correct within seconds, and available even when the database that owns them is down. So most of this design is about how route state moves, and the proxy is deliberately boring.

My core position, stated up front so you can push on it: the request path never talks to the control-plane database. Routers serve entirely from local state — an in-memory route table backed by an embedded key-value store (RocksDB) on disk — fed by a durable, partitioned event log (Kafka) with periodic snapshots in object storage (S3), plus a lazy pull path for misses. Everything else falls out of that decision: cold start, DB outages, and propagation latency are all answered by the same machinery.

Assumptions

The prompt fixes the big numbers; here's what I'm adding.

Requirements, prioritized

  1. Correctness under churn. A request for myapp.platform.app reaches a healthy instance of myapp, even mid-deploy. Routing to the wrong service is a security incident, not a bug — tenant A's traffic must never land on tenant B.
  2. Availability over freshness. When the control plane is degraded, serve stale routes rather than fail. A route that's 10 minutes old is almost always still right; a 502 is always wrong.
  3. Fast propagation. New custom domain live in <60s end to end; deploy/scale events visible at every edge in single-digit seconds (target p99 < 5s).
  4. Low added latency. The routing layer should add ~1–2ms of processing on top of network time. At 5M RPS, per-request DB lookups are off the table anyway.
  5. Operable. Routers deploy daily without dropping WebSockets; a router restarts without downloading the world.

Non-goals for the hour: DDoS scrubbing, WAF, caching/CDN semantics, billing metering.

Estimates that shape the design

Rough numbers, chosen to be conservative:

Architecture

Four planes. The names matter less than the arrows.

flowchart TB
    subgraph Edge PoP
        L4[L4 balancer\nECMP + Maglev hashing]
        R1[L7 router\nin-mem routes + RocksDB]
        R2[L7 router]
        LS[PoP lookup service\nmiss path, cert cache]
        L4 --> R1
        L4 --> R2
        R1 -. miss / SNI miss .-> LS
    end

    subgraph State distribution
        K[(Event log\nKafka: route-deltas)]
        S3[(Object storage: S3\nhourly snapshots)]
        K --> R1
        K --> R2
        K --> LS
        S3 -. boot hydrate .-> R1
    end

    subgraph Control plane
        API[Platform API]
        PG[(Postgres\nsource of truth)]
        CM[Cert manager\nACME client]
        ORCH[Deploy / autoscale\norchestrator]
        API --> PG
        ORCH --> PG
        CM --> PG
        PG -- CDC --> K
        SNAP[Snapshotter] --> S3
        K --> SNAP
    end

    subgraph Region: compute
        HC[Regional health checker]
        I1[Service instances]
        HC --> I1
        HC --> K
        R1 == proxied requests ==> I1
    end

    Client((Client)) -->|anycast| L4
    LS -->|cache fill| PG

Request path, end to end

  1. Client resolves myapp.platform.app (or a custom domain CNAMEd to us) to an anycast IP; BGP delivers it to the nearest PoP.
  2. An L4 balancer layer — stateless boxes doing consistent hashing on the 5-tuple (Maglev-style, via ECMP from the PoP routers) — picks an L7 router. Consistent hashing here is what later makes router drains clean: a router leaving the pool only disturbs its own flows.
  3. The L7 router terminates TLS. On ClientHello it looks up the SNI name in its cert cache; miss → fetch from the PoP lookup service (~1 RTT inside the PoP, then cached).
  4. It looks up the Host header in its in-memory route table: hostname → service → home region + healthy endpoint set. It verifies Host matches the SNI name (blocks domain-fronting one tenant behind another's cert).
  5. In-region request: pick an endpoint (P2C — power of two choices — on locally tracked in-flight counts), proxy, done. Out-of-region: forward over pooled, mTLS-authenticated inter-PoP connections to a router in the service's home region, which does the final hop. Two lookups, one design.

Added latency budget: TLS-resumed requests should see well under a millisecond of routing logic; the dominant cost is the hop to the service's region, which no router design can remove.

The proxy itself

I'd build the L7 router as a custom proxy in Go (or Rust; Go's crypto/tls GetCertificate hook and mature HTTP stack make it the pragmatic pick). I rejected two obvious alternatives:

The router process holds the route table as an immutable in-memory structure swapped atomically on update (readers never lock), mirrored to RocksDB on local disk keyed by hostname, with the last-applied log offset stored alongside. RocksDB isn't there for query speed — it's the restart story.

Data model and APIs

Source of truth in Postgres. Core tables (abridged):

services(id, owner_id, name, home_region, plan, created_at)
instances(id, service_id, region, host_ip, port, state, deploy_id, updated_at)
  -- state: starting | healthy | draining | dead
domains(id, hostname UNIQUE, service_id, kind,            -- platform | custom
        status,                                           -- pending_dns | verifying | active | error
        verify_token, created_at, activated_at)
certificates(id, hostname, status,                        -- issuing | active | renewing | failed
        not_before, not_after, chain_pem,
        key_ciphertext, key_kms_ref)                      -- private key envelope-encrypted, never plaintext at rest
acme_challenges(token, key_auth, hostname, expires_at)    -- shared store the edge answers from

Every mutation to routable state produces an event via change-data-capture from Postgres into Kafka (CDC via the write-ahead log — Debezium-style — so the DB commit and the event can't diverge). One logical topic, route-deltas, partitioned by hostname/service id, with per-key monotonically increasing versions:

{ "seq": 918273645, "kind": "endpoint_set",
  "service": "svc_abc", "region": "oregon", "version": 42,
  "endpoints": [{"ip":"10.2.3.4","port":10001,"state":"healthy"}] }

{ "seq": 918273646, "kind": "domain_map",
  "hostname": "shop.example.com", "service": "svc_abc", "version": 7 }

Deltas carry the full new value for their key, not diffs — so applying the latest version per key is always safe regardless of what you missed. The log is compacted by key, which is what makes "catch up from the log" bounded.

Public API surface (the piece customers touch):

POST /v1/services/{id}/domains        {"hostname": "shop.example.com"}
  → 201 {"status":"pending_dns", "verify": {"cname":"shop.example.com → svc-abc.platform.app",
                                            "txt":"_platform-verify.shop.example.com = tok_9f2..."}}
GET  /v1/services/{id}/domains/{hostname}   → status, cert expiry, last error
DELETE ...

Internal lookup API (PoP lookup service): GET /route?host=... and GET /cert?sni=..., mTLS-only, backed by its own log-fed copy of state, with Postgres as a last-resort fill.

Distributing route state: push, pull, and the layers in between

Push alone or pull alone both fail here. Pure pull (routers query a central store per miss, cache with TTL) gives you either stale routes (long TTL) or a thundering herd on a hot store (short TTL), and propagation is bounded by TTL — you can't hit 5-second deploys with a 60-second TTL. Pure push (every router holds everything, always) is actually almost affordable because state is only ~3GB — but it makes cold start heavy and turns every new node into a full sync before it can serve. So, both:

Push for freshness. Every router and every PoP lookup service is a Kafka consumer of route-deltas. At 1K msgs/sec this is a rounding error. A change committed in Postgres is at every edge in roughly: CDC latency (100ms–1s) + log fan-out (<1s) + apply (<10ms). Call it p50 ~1s, p99 ~5s globally. That's the deploy-propagation answer.

Pull for misses. If a hostname isn't in the local table (new router still warming, or a domain created 200ms ago), the router asks the PoP lookup service (one intra-PoP RTT); if it misses, it fills from Postgres and caches. Misses are rare in steady state, so the DB sees miss-fill traffic, not request traffic.

Snapshots for bootstrap. A snapshotter consumes the compacted log and writes a full-state snapshot (RocksDB SST files, a few GB) to S3 hourly, tagged with its log offset. Rehydration = download snapshot + replay the log from that offset. Never a DB dump.

Router cold start — three tiers

The prompt's constraint — restart must not require a full copy of the mapping database — is really three scenarios:

  1. Process restart on an existing node (the common case: deploys). RocksDB is already on disk with the last-applied offset. Open it, resume the log from that offset, serving within seconds with state that's stale only by the downtime. No bulk transfer at all.
  2. Fresh node, warm PoP. Start with an empty table and serve immediately via the pull path — every request is a miss against the PoP lookup service, which is fine at intra-PoP latency — while a background job hydrates from the latest S3 snapshot (or, cheaper, streams SSTs from a peer router in the same PoP) and then tails the log. The node is degraded-but-correct for the few minutes hydration takes.
  3. Fresh PoP. Lookup service hydrates from S3 first, routers then hydrate from it. Only this case pulls gigabytes across the backbone, and it's a provisioning event, not a restart.

The principle threaded through all three: correct when cold, fast when warm — the pull path is the correctness floor, push and hydration are the performance ceiling.

Consistency and invalidation

Custom domains: verification, TLS, and the 60-second clock

Flow when a customer adds shop.example.com:

  1. API call creates the domains row (pending_dns) and returns instructions: CNAME shop.example.com → svc-abc.platform.app (or A/ALIAS to our anycast IPs for apex), plus a TXT token for pre-pointing verification.
  2. Verification. A verifier polls DNS (with jittered backoff, seconds apart initially). Ownership is proven by either the TXT token or the CNAME itself resolving to us. This step matters for security: without proof of DNS control, tenant B could claim tenant A's lapsed domain, or claim a hostname that was never theirs, and we'd happily mint them a cert for it. Verification also pins the hostname to this service — a second tenant claiming the same hostname is rejected at the unique constraint.
  3. Route propagation. On active, the domain_map event flows to every edge — ~1–5s, per above.
  4. Cert issuance. The cert manager runs an ACME HTTP-01 flow: it writes the challenge token to the acme_challenges store (replicated to PoP lookup services via the same log), then tells the CA to validate. The CA's probe hits our anycast edge; every router special-cases /.well-known/acme-challenge/* before route lookup and answers from the challenge store — this works even though the service's route just appeared. Issuance at the CA takes ~5–30s. DNS-01 is the fallback for customers who can't point DNS yet or need wildcards, via a delegated _acme-challenge CNAME to a DNS zone we control.
  5. Cert distribution. Cert stored in Postgres (key envelope-encrypted under a KMS-managed key), cert_ready event published. Edges don't preload it — first ClientHello for that SNI pulls it through the PoP lookup service and caches it in memory and on local disk (key still encrypted at rest on the router; decrypted per-process via KMS grant at load).

Clock check: verification (DNS already set) ~5–15s + route propagation ~1–5s (parallel) + ACME ~5–30s + first-request cert pull ~50ms. Comfortably under 60s in the good case; the failure mode is customer DNS not actually pointing at us, which we surface as status, not silence. Until the cert exists, we can serve HTTP and answer HTTPS with a cert-pending error page rather than a handshake failure.

1M certs at the edge is why certs are pull-not-push: pushing 4GB of secrets to every router multiplies key exposure and boot cost for material that's mostly cold at any given PoP. A per-PoP cert cache (lookup service, disk-backed) plus per-router in-memory LRU means a hot domain costs one fetch per router per restart. Renewals happen 30 days before expiry with jittered scheduling, so a CA outage of even a week strands nothing; a cert_rotated event proactively invalidates edge caches so rotation isn't hostage to cache TTLs. CA rate limits are managed by spreading across multiple ACME accounts and by never retrying failed issuance without backoff — LE's per-account and per-registered-domain limits are exactly the kind of external constraint that turns a retry loop into a self-inflicted outage.

Health: who decides an endpoint is alive

Two mechanisms with different jobs, because one can't do both.

Deploys use the same vocabulary: orchestrator starts new instances → health checker marks them healthy → event adds them at the edge → orchestrator marks old ones draining (event removes them from selection; in-flight requests finish) → kill. The edge never needs to know what a "deploy" is.

Failure modes

Control-plane Postgres down. The request path doesn't reference it, so existing traffic is untouched — routers keep serving from local state, certs keep serving until not_after. What degrades: no new domains, no deploys, no new instances registering, and the miss-fill path loses its backstop (the PoP lookup services still answer from their own log-fed state, so only never-seen hostnames fail — which, with the DB down, can't be created anyway). Kafka retains the log independently, so routers can still restart, hydrate, and catch up during the outage. The stated stance: routes have no TTL — stale state serves indefinitely, and we alarm on staleness (log lag age is a first-class metric per router) rather than fail closed. The risk of serving a 30-minute-old route (traffic to a drained instance → connection refused → passive ejection retries elsewhere) is strictly smaller than the risk of refusing traffic.

Kafka down. Push stops; pull still works; edges serve stale plus miss-fill from the lookup services and Postgres. Propagation SLO is violated, availability isn't. Two independent legs is the point of having both.

PoP failure. Withdraw its BGP announcements; anycast reconverges clients to the next PoP in seconds to minutes. Long-lived connections through that PoP die and reconnect — unavoidable with anycast, and why client SDKs and our WebSocket guidance mandate reconnect-with-backoff.

Compute-region failure. Edge PoPs elsewhere are fine, but a single-region service has no healthy endpoints anywhere — the router's region-forwarding hop fails, and we serve a platform 503. For services with replicas in multiple regions, the route entry carries the full region set with weights; the router fails over to the next region when the home region's endpoint set is empty or the inter-region path is dark (circuit breaker on the forwarding pool). We don't try to make single-region services survive region loss — that's a compute-tier product feature (standby replicas), and pretending routing can solve it just hides the gap. Split-brain guard: region failover at the edge is driven by endpoint-set emptiness and connectivity, not by any global "region is down" flag that itself needs a quorum.

Bad state, not absent state. The scarier failure is the control plane publishing wrong data — a bug that emits empty endpoint sets for everyone. Mitigations: the snapshotter validates invariants before publishing (e.g., total healthy endpoints can't drop >50% between snapshots without a human ack), routers rate-limit mass-deletion applies, and the log gives us replay — roll the fleet back to offset N and re-apply. An event-sourced edge is rewindable; a "just sync the DB" edge isn't.

Router deploys and long-lived connections

WebSockets are the reason routers can't just be killed and replaced. The mechanism, in order:

  1. L4 stickiness. The Maglev/ECMP layer hashes on 5-tuple, so an established flow keeps hitting the same router box even as the pool changes. Consistent hashing means adding routers doesn't reshuffle existing flows.
  2. In-place binary handoff. Deploying a router upgrades the process on the same box: new binary starts, inherits the listening sockets via SO_REUSEPORT (or explicit fd passing over a Unix socket, SCM_RIGHTS-style), and takes all new connections. The old process stops accepting and enters drain.
  3. Tiered drain. Plain HTTP requests finish in seconds. HTTP/2 gets GOAWAY so clients migrate at their next request. WebSockets drain naturally as clients disconnect; we hold the old process for a generous window — hours, not minutes (say 6–24h, configurable) — because process residency is cheap and forced disconnects aren't. Connections that outlive the window get a WebSocket Close 1012 ("service restart"); clients reconnect and land on the new process. A max-connection-lifetime (e.g., 24h, jittered) enforced even outside deploys keeps the tail bounded and — more importantly — keeps clients honest about reconnecting, so a rare forced drain isn't the first time their reconnect path runs.
  4. Host replacement (kernel upgrades, hardware): remove the box from L4 hashing so it gets no new flows, drain in place on the same schedule, then retire. Consistent hashing again limits blast radius to that box's own flows.

I rejected connection handoff between processes (serializing TLS + WebSocket state across a process boundary): it's been done, but the complexity is enormous and the drain-window approach achieves the same user-visible result — routine deploys drop no connections — with machinery a new on-call engineer can understand.

Note the layering dividend: backend deploys don't touch router connections at all. The WebSocket terminates on the router; when the service instance drains, the router can only pass the close through — surviving backend deploys is the service's reconnect contract, and we make that explicit in the platform docs rather than promising magic.

Security notes

Tradeoffs I'm consciously making

Evolution

First year of follow-ups, in likely order: TCP/UDP passthrough via SNI-less L4 routing (needs port-based or PROXY-protocol service identification); per-service edge policies (rate limits, IP allowlists) riding the same delta log as just another versioned key; regional PoP-to-PoP request hedging for tail latency; and, at ~10× scale, the lookup-service sharding move above. None of these change the skeleton — which is the real test of the skeleton.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): You prioritize tenant isolation yet serve routes and certificates stale forever: after a custom domain is removed from tenant A and reassigned to tenant B, what prevents a partitioned edge from continuing to terminate TLS and send B's traffic to A, and what consistency or revocation guarantee resolves that fundamental availability-versus-isolation conflict?

This lands, and the fix is to stop treating all staleness as one class. Endpoint churn can be stale forever — a dead instance is a retry, not a breach. Domain ownership changes are different, and I revise the design in three ways. First, reassignment becomes two-phase: publish a tombstone for the hostname (edge serves 404), dwell 15 minutes, then publish the new mapping — B must pass DNS re-verification before activation, which already covers the common case (lapsed domain re-registered by a stranger). Second, the dwell buys a concrete guarantee: any edge whose log lag is under 15 minutes never crosses an ownership boundary. Third, an edge that knows its lag exceeds the dwell (lag age is already a first-class metric) flags custom-domain lookups for revalidation through the pull path before serving; in a partial partition (edge lost Kafka, control plane alive — the common case) that closes the hole entirely.

The residual risk is the double failure — edge partitioned from both Kafka and the pull path for >15 minutes while a transfer completes — and there I initially chose availability, because the attack also requires tenant A to keep a backend accepting a hostname they just lost, mid-transfer, during our outage. (Round 2 pushed on exactly this; see Q7 for the stronger mechanism.) Also worth stating: the cert private key never left our infrastructure in either tenancy — the exposure is misdirected plaintext at A's backend, not key compromise.

Design change: staleness classes (endpoint churn: unbounded; ownership: bounded); domain reassignment is tombstone → 15-min dwell → re-verified activation; edges with log lag over the dwell revalidate custom-domain lookups via the pull path before serving.

Q2 (Codex): Health checkers publish endpoint sets directly while Postgres CDC also publishes routable state; what mechanism allocates a single monotonic per-service version across those independent writers, and how does a snapshot record a replayable consistent cut when a partitioned Kafka topic has a vector of offsets rather than one global offset?

The version question lands because my diagram shows two writers into the same log; the fix is a rule the doc should have stated: single writer per key. endpoint_set(service, region) is owned solely by that region's health-checker shard; domain_map and service config keys are owned solely by Postgres CDC. No key ever has two writers, so no cross-writer coordination exists to get wrong. Health-checker shard failover is fenced with an epoch from the shard lease: version = (epoch, counter), compared lexicographically, so a zombie old shard's writes lose. The orchestrator's deploy sequencing goes through Postgres (instances table → CDC), and the health checker's liveness transitions ride its own keys — they meet only at the router's union of the two, which is per-key LWW either way.

On the snapshot: the premise assumes I need a consistent cut, and I don't. Deltas carry the full value per key and application is per-key LWW on versions, so the snapshot only needs two properties: some value per key, and a recorded offset vector (one offset per partition) such that the state includes every event before those offsets. Replay from the vector re-delivers a suffix per partition; duplicates and cross-partition reordering are absorbed by the version check. The doc's "tagged with its log offset" should read "tagged with its consumer offset vector" — a wording fix, not a mechanism change. Consistent cuts are what you need when you replay operations; I replay states.

Design change: explicit single-writer-per-key ownership rule; health-checker versions are (lease epoch, counter) fenced tuples; snapshots and RocksDB store a per-partition offset vector, not a scalar offset.

Q3 (Codex): At 300–1,000 updates per second, how is an "immutable" 2–3 GB route table atomically swapped without repeatedly copying it, blocking readers, or retaining multiple gigabytes of obsolete versions, and what crash-ordering protocol keeps the in-memory state, RocksDB contents, and committed Kafka offsets consistent?

"Immutable, swapped atomically" was shorthand for structural sharing, not whole-table copies. Concretely: the table is a persistent map (HAMT-style, or 4096-shard map where an update copy-on-writes one shard's bucket path), so an update touches O(log n) nodes — a few KB — and readers traverse old versions lock-free until epoch-based reclamation frees them. At 1K updates/sec the garbage rate is a few MB/sec, reclaimed within milliseconds of the last reader leaving; there is never more than one full logical version resident plus in-flight deltas.

Crash ordering has one source of truth: RocksDB. Each delta is applied as an atomic write batch containing both the key's new value and the last-applied offset vector; memory is updated after the batch commits; Kafka consumer-group offsets are not used at all — on restart the router seeks explicitly to the vector stored in RocksDB. Any crash between batch and memory update just means replaying a suffix, and per-key version checks make reapplication idempotent. There is no window where memory is ahead of what a restart can reconstruct.

Q4 (Codex): During a simultaneous fleet restart or loss of a PoP's local disks, every request becomes a lookup-service miss while that service itself must download the full snapshot; how does the PoP remain available at peak traffic without requiring a full mapping copy to boot, overwhelming the lookup service, or returning stale values that race snapshot hydration?

Concede the doc oversold "fresh node serves immediately via the pull path" — that only works when the pull path is warm. The design change is readiness gating: a router doesn't enter the L4 pool until its table is hydrated or the PoP lookup service reports warm, and the PoP announces its anycast prefixes only when it can actually serve. Total cold start of a PoP (disks gone, lookup service gone) is handled as PoP failure: BGP stays withdrawn, anycast sends the traffic to neighboring PoPs — which is exactly what would happen if the PoP were down, because functionally it is — while it hydrates from S3 with zero request pressure. Neighbor PoPs are capacity-planned for one-PoP-out (they must be, for the PoP-failure case anyway).

Within a warm PoP, the miss path is shaped: singleflight coalescing per hostname at both router and lookup service (a thousand concurrent requests for one cold hostname cost one upstream fetch), and hydration is applied through the same per-key version filter as pull fills, so a stale pull result can never overwrite a newer hydrated value or vice versa — the race the question worries about is resolved by the versions, not by timing.

Design change: readiness gating — routers join the L4 pool only when hydrated or when the PoP lookup service is warm; a fully cold PoP stays BGP-withdrawn until hydrated; singleflight coalescing on the miss path.

Q5 (Codex): Because TLS certificate retrieval and KMS decryption occur during ClientHello, how do you prevent high-cardinality SNI traffic from exhausting lookup, cache, and KMS capacity, and how can you claim to show an HTTPS "certificate pending" page when no certificate valid for that hostname exists to complete the handshake?

Two different points. On capacity: junk SNI never reaches the lookup service on a warm router, because the hostname is checked against the fully local route table first — an unknown name is rejected from memory with a TLS unrecognized_name alert, no network. Cert pulls happen only for hostnames that exist, are coalesced by singleflight, and KMS is nowhere near per-handshake: one decrypt per (router, cert) cache fill, amortized over the LRU lifetime; a KMS outage strands new fills but everything cached keeps serving. Cold routers lean on the 30s negative cache plus singleflight, and readiness gating (Q4) means cold routers aren't taking peak traffic anyway.

On the pending page: you're right and I withdraw it. You cannot present an HTTPS page for a hostname you hold no valid cert for without a browser warning — serving a self-signed placeholder is worse than failing. Revised behavior: until the cert exists, HTTPS gets a clean handshake rejection (unrecognized_name), port-80 HTTP serves the holding/redirect page, and "cert pending" is surfaced where it can actually be seen — the API status endpoint and dashboard.

Design change: the HTTPS "cert-pending error page" claim is withdrawn; pre-cert HTTPS fails the handshake with unrecognized_name, HTTP and the API carry the pending status.

Q6 (Codex): A stateless Maglev/ECMP layer does not preserve an established flow when its selected router is removed — the flow is rehashed and its TCP/TLS state is lost — so what exact connection-tracking or two-phase membership mechanism makes router replacement and long-lived WebSocket draining actually lossless?

Correct — consistent hashing alone minimizes disruption, it doesn't eliminate it, and my "stateless boxes" line was wrong as stated. Revised mechanism, which is the standard Maglev/Katran/GLB shape: each L4 box keeps a local connection-tracking table (5-tuple → router) and uses the Maglev hash only for packets with no conntrack entry. Router drain is then two-phase: remove the router from the new-flow hash table while conntrack keeps steering established flows to it — the box keeps its address and stays reachable — until the drain window closes. Established WebSockets never rehash.

The remaining hole is losing an L4 box itself: its conntrack dies with it and flows rehash to a sibling with an empty table. Maglev's property saves the common case — the hash is deterministic and identical across L4 boxes, so the sibling computes the same router for the 5-tuple, and the flow survives statelessly provided the router set hasn't changed since the flow began. For flows that began before a router-set change, add GLB-style second chance: the lookup table stores (current, previous) router per bucket, and a router receiving a mid-flow packet it has no TCP state for forwards it to the previous owner. That covers the double event (L4 box loss during a drain) without any shared state store.

Design change: L4 boxes keep per-box conntrack with Maglev as the miss-path only; router drains remove a box from the new-flow table while conntrack pins established flows; bucket history plus second-chance forwarding covers L4-box loss during membership changes.

Round 2 — pushback

Q7 (Codex): You explicitly rank cross-tenant routing as a security incident, yet Q1 knowingly sends a reassigned domain's traffic to the former tenant during a dual outage. Why is that acceptable, and what fail-closed mechanism prevents the breach without trusting the old tenant's backend behavior?

Fair — "rare and requires a malicious A" is a risk argument, not a guarantee, and I said misrouting is a security incident. So make the transfer fail closed instead of the traffic: activation of B's mapping is gated on tombstone acknowledgment from every currently-announced PoP. Each PoP's lookup service already reports its last-applied sequence over the mTLS control channel (a heartbeat independent of Kafka, so a Kafka partition doesn't blind us); the transfer coordinator proceeds only when every announced PoP has confirmed the tombstone applied. A PoP that can't confirm within the dwell is de-announced — BGP withdrawal is enforced upstream via the edge-network interface, not by the partitioned PoP's own cooperation — before activation continues. That yields a real invariant: B's mapping is never active anywhere while any PoP still able to receive traffic could route the hostname to A. Nothing trusts A's backend.

The cost profile is right: the fail-closed unit is one unhealthy PoP during the rare intersection of an ownership transfer and a partition, and de-announcing it moves its users to the next PoP — a latency bump, not an outage. In the full dual outage (control plane and Kafka both down), transfers simply don't complete: the tombstone can't propagate, so B never activates, and the hostname keeps serving A as A's still-legitimate domain, because the transfer hasn't happened yet. The window I conceded in Q1 closes: either the transfer completes with proof no stale edge remains reachable, or it doesn't complete. Same machinery handles the adjacent case that actually keeps me up at night — emergency takedown of a compromised domain — where "tombstone ACK from all announced PoPs, de-announce the deaf ones" is exactly the hammer you want.

Design change: ownership transfers (and takedowns) are gated on tombstone ACKs from all announced PoPs over a Kafka-independent control channel; non-ACKing PoPs are de-announced upstream before the new mapping activates.

Q8 (Codex): Q6's (current, previous) bucket history survives only one router-set change, but a WebSocket can span several deploys or scaling events. When its owner is multiple generations old, how do you preserve the flow without unbounded membership history, shared connection state, or disconnecting it?

The premise conflates deploys with membership changes, and the design's whole point is that they're decoupled: a router deploy is an in-place binary handoff on the same box — same IP, same L4 bucket owner — so the router set does not change and buckets never move. A WebSocket spanning ten deploys has experienced zero membership changes. Bucket history only matters for genuine set changes: host adds, host retirements, host failures. Those are infrequent — planned ones we schedule, and we can measure the rate.

For those, bounded depth is enough; unbounded history is solving a problem the flow-lifetime cap already bounds. Keep depth-k history per bucket (k router IDs × 65K buckets — kilobytes) with recursive second-chance forwarding up to k hops; each hop is one intra-PoP forward. With max connection lifetime at 24h (jittered — already in the design for exactly this reason) and planned membership changes batched to at most one per PoP per day, k=3 covers every flow that can exist, deliberately, not probabilistically. What I won't chase: flows on a router box that itself dies — no L4 scheme saves a connection whose TCP state lived on the dead box — and flows caught by an unplanned membership change beyond depth k during a simultaneous L4-box loss, which get a RST and a reconnect. The promise stays what the original doc stated: routine deploys drop no connections. Compound hardware failures were never lossless, and pretending otherwise is how you end up with shared conntrack state — the complexity I rejected for process handoff, rejected again here for the same reason.

Design change: depth-k (k=3) bucket history with recursive second-chance forwarding; planned membership changes batched to ≤1 per PoP per day so k provably covers the 24h max flow lifetime.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The design's central bet — full route state replicated to every edge box, fed by a log, request path never touches the database — is Cloudflare's Quicksilver, almost exactly. Quicksilver v1 gave every server a complete LMDB copy of the config database, replicated via monotonic sequence numbers in a transaction log, propagating changes to 200+ cities in seconds — the same shape as my Kafka-fed RocksDB mirrors with per-key versions. The instructive part is what happened next: by 2025 the full-copy model stopped paying, and Quicksilver v2 split the fleet into replicas (full dataset) and proxies (persistent caches), after measuring that big data centers touch ~20% of the keyspace and small ones ~1%. That's my stated 10× evolution path — routers go pull-heavy against a per-PoP lookup service — executed for real, and their working-set numbers also back the certs-pulled-not-pushed split.

Fly.io solved the same state-distribution problem and landed somewhere I deliberately didn't: no central log at all. A Foolish Consistency walks through why strongly consistent Consul buckled under their service discovery load (10 GB/sec of long-poll traffic from an N² wakeup bug), and Corrosion is the replacement — SWIM gossip plus CRDTs over SQLite, with fly-proxy building routing tables from the gossip stream. Their "workers own their own state, so updates almost never conflict" is my single-writer-per-key rule (Q2) arrived at independently. But the same post documents a contagious deadlock that took their network down in September 2024, which is a fair price tag on gossip: I'd still take Kafka's boring, replayable, rewindable log for a greenfield build, and the Corrosion incident is the argument.

The L4 mechanics I converged on in Q6/Q8 are published designs, not inventions. GitHub's GLB director is the origin of second-chance forwarding: stateless directors, rendezvous hashing to a (primary, secondary) pair per bucket, and a proxy that doesn't recognize a flow forwards it to the previous owner — plus the active/draining/filling state machine my two-phase drain mirrors. Cloudflare's Unimog explicitly builds on GLB's daisy-chaining, runs the L4 balancer on the same general-purpose servers as everything else, and keeps connections alive for days — confirmation that "long-lived flows survive membership churn without shared conntrack state" is the industry answer, not a corner I cut. My depth-3 bucket history is a small extension of GLB's depth-2, bought by capping flow lifetime at 24h.

On certificates, Caddy's on-demand TLS is the reference implementation of issue-at-first-handshake for SaaS custom domains, gated by an "ask" endpoint that checks the domain against your tenant database. I diverged on purpose: issuing at first handshake makes the first visitor eat ACME latency and makes issuance load traffic-driven; issuing eagerly at domain-add time (with pull-on-first-ClientHello for distribution) is what the 60-second SLO forces. The eager model at real scale is Shopify — Let's Encrypt serves their 4.5M domains, and their pre-LE estimate of 100+ days to reissue everything versus hours now is the case for treating the CA as an automated, rate-limited utility rather than a ceremony.

The Envoy rejection also checks out against Envoy's own docs: the VHDS page opens by conceding that default RDS ships every route to every proxy and doesn't scale, and its fix — pause the request, fetch the virtual host on demand over delta-xDS — is my pull path rebuilt inside a config protocol, with a mid-request stall where my design has an intra-PoP RTT. If you already run Envoy, VHDS plus SDS is the sanctioned way to do millions of hosts; greenfield, it's the same control plane you'd write anyway plus Envoy.

Updates from post-training information

Two things moved after my training data ends, both in the cert section's blast radius. First, the CA/Browser Forum's ballot SC-081v3 — maximum TLS certificate validity stepping down from 398 days to 47 days, phased from March 2026 through March 2029 — has now taken its first step: the initial reduction landed in March 2026, so shrinking lifetimes are enforced reality, not a roadmap. My renewal math (1M certs ≈ 17K issuances/day on ~60-day cycles) roughly quadruples by 2029; the architecture holds, but ACME throughput, rate-limit budgets across accounts, and renewal jitter stop being footnotes and become capacity planning. Second, Let's Encrypt published a March 2026 post on ACME Renewal Information (ARI, RFC 9773) describing Shopify's adoption: instead of a hard-coded "renew 30 days before expiry" — exactly what my design specifies — the client asks the CA for a recommended renewal window, which lets the CA spread load, adapt to shrinking lifetimes, and trigger early renewal on revocation events. I'd revise the cert manager to be ARI-driven with the 30-days-with-jitter rule demoted to fallback.

Further reading