Contents

Container Scheduler and Autoscaling Control Plane

Scheduling 3 million containers on 50,000 hosts is not one hard problem, it's two: making a good placement decision quickly, and keeping reality converged with intent while hosts die, deploys churn 100,000 containers an hour, and free-tier services blink in and out of existence. My design separates those concerns aggressively: a globally replicated desired-state store that is the single source of intent, regional control planes sharded into cells that do all the real-time work, and hosts that are autonomous enough to keep customer traffic flowing when the control plane is down.

I'll state my assumptions, do the math that shapes the architecture, then walk the components, the placement algorithm, the reconciliation loops, and the hard cases: persistent disks, scale-to-zero, noisy neighbors, evacuation, and control-plane failure.

Assumptions

Beyond what the prompt gives me:

The math that shapes the design

Placement rate. 100k containers/hour of deploy churn is ~28 placements/sec sustained. Add autoscaling churn (guess: comparable) and failure-driven rescheduling: 500 host failures/day × 60 containers = 30k placements/day of failure noise — negligible on average, but a single dead 60-container host is a burst, and a rack failure (say 20 hosts) is a 1,200-placement burst that should clear in seconds. I budget the scheduler for 1,000 placements/sec burst per region with p99 decision latency under 100 ms. That's modest — the reason schedulers fall over is not decision rate, it's state synchronization, which is why sharding state matters more than a clever algorithm.

State size. 3M container records at ~2 KB each is ~6 GB of hot observed state, plus 50k host records. Too big and too write-hot for one coordination store, comfortable when sharded 25–50 ways.

Heartbeat load. 50k hosts reporting every 5 s is 10k msgs/sec globally, 2k/sec per region. Trivial if agents send deltas (what changed) rather than full container inventories; a full-state resync only on agent restart or checksum mismatch.

Cells. These numbers drive the core structural decision: shard each region into cells of 2,000 hosts (120k containers). Each cell has its own scheduler, its own coordination store, its own failure domain. 2,000 hosts is small enough that one scheduler can hold the entire cell's host state in memory and score placements without RPCs, and small enough that losing a cell's control plane strands only 4% of a region's capacity. A service's replicas can span cells; the cell is a control-plane shard, not a placement boundary the customer sees.

Architecture

Three layers, with a strict rule about who owns which state:

  1. Global layer — the customer API, the desired-state store, the deploy orchestrator, quota service, and the global edge/routing layer. Owns intent: what should exist, where, at what scale. Storage is a multi-region, strongly consistent replicated SQL database (CockroachDB): survives a region loss, gives me transactions for quota accounting and deploy state machines, and its throughput needs are low — intent changes at human/deploy speed, tens of writes/sec, not container speed.
  2. Cell layer (per ~2,000 hosts) — a scheduler, a reconciler, and a strongly consistent coordination store (etcd) holding the cell's bindings (container → host assignments), host records, and leases. Etcd fits because the per-cell working set is ~120k small, watch-friendly records and I need cheap leader election and watches; I rejected putting bindings in the global SQL store because binding churn (thousands/sec fleet-wide during deploy peaks) is exactly the write pattern that would make the global store the bottleneck.
  3. Host layer — an agent on every host that watches its own assignment set, drives the container runtime, enforces isolation, and reports observed state. The agent journals to a local embedded store (SQLite) so it can restart hosts' containers after a reboot without asking anyone.

Connecting them: an event bus — a partitioned durable log (Kafka) — carries state-change events (container started/died, deploy progressed, scale decisions) to the deploy orchestrator, autoscaler, billing, and observability. The bus is for telling people what happened, never for commanding; commands flow through stores that can be re-read, so a lost message never loses intent.

flowchart TB
    subgraph Global["Global layer (multi-region)"]
        API[Customer API]
        DS[(Desired-state store\nCockroachDB, multi-region)]
        DO[Deploy orchestrator]
        QS[Quota service]
        EDGE[Global edge / L7 routing]
        ACT[Activator\nscale-to-zero wakes]
    end

    subgraph Region["Region (10k hosts)"]
        subgraph Cell1["Cell A (~2,000 hosts)"]
            SCH[Scheduler\nleader-elected]
            REC[Reconciler]
            CS[(Cell store: etcd\nbindings, hosts, leases)]
            subgraph Host["Host x2000"]
                AG[Agent + SQLite journal]
                FC[Firecracker microVMs]
                VOL[Local NVMe volumes]
            end
        end
        CellN[Cells B..E]
        CAP[Capacity manager\nheadroom, defrag]
        IMG[Image cache + P2P distribution]
        MET[Metrics pipeline]
        AS[Autoscaler controllers]
    end

    API --> DS
    API --> QS
    DO --> DS
    DS -- watch intent --> SCH
    SCH --> CS
    CS -- watch assignments --> AG
    AG -- delta status, heartbeats --> CS
    REC --> CS
    REC -- diff --> SCH
    AG --> FC
    AG --> VOL
    AG -- events --> MET
    MET --> AS
    AS -- desired replicas --> DS
    EDGE -- requests --> FC
    EDGE -- zero-scaled? --> ACT
    ACT -- wake --> SCH
    CAP --> SCH
    IMG --> AG

Desired vs. observed state — who owns what

This split is the backbone, so I'll be precise:

Reconciliation is then mechanical: the reconciler diffs desired (projected into the cell) against observed, and emits work — "place 3 replicas," "kill this orphan," "replica unhealthy, replace." Every loop is level-triggered: it acts on current state, not on edges, so missed events are self-healing on the next pass (full re-diff every 30 s, watch-driven in between).

Data model (core records)

Service:    tenant_id, service_id, plan, image_ref, resources{cpu, mem, disk},
            scaling{min, max, target_metric}, placement{region, spread_policy},
            volumes[], isolation_class
Revision:   service_id, rev_id, image_digest, config_hash          # immutable
Deploy:     service_id, from_rev, to_rev, strategy, state machine, wave cursor
ReplicaSet: service_id, rev_id, desired_count                      # autoscaler writes count
Binding:    replica_id, host_id, resources_reserved, lease, generation   # cell store
Host:       host_id, cell, shape, labels, allocatable{}, taints, health, agent_lease
Volume:     volume_id, service_id, host_id (pin), replica_host_id?, size, snapshot_policy
Quota:      tenant_id, cpu/mem/replica/disk caps, current usage    # transactional

generation on bindings matters: every command an agent receives carries the binding generation, and agents reject stale generations. That's the idempotency/fencing primitive that makes scheduler failover and retries safe.

APIs

Customer API (the platform's public surface): CreateService, Deploy(image), Scale(min,max), AttachVolume, EvacuatePreference. All writes go through quota admission and land as desired-state transactions.

Agent API (host ↔ cell), the interface the prompt lets me define against the runtime:

Scheduler API (internal): Place(replica_spec, constraints) → binding, Evict(binding, reason, deadline), ReserveCapacity(shape, count, ttl) — the last one is what deploys and the activator use to pre-book slots.

Placement: filter, score, commit

The scheduler is per-cell, leader-elected via the cell store, and keeps the whole cell in memory (2,000 host records with bitmap-ish free-resource vectors — a few MB). Placement is the classic two phases, and the interesting part is the scoring weights and the concurrency control:

Filter (hard constraints): resource fit against allocatable (host capacity minus system reserve minus headroom reserve), isolation class compatibility, volume pinning (must-run-on host H), taints (draining, cordoned), spread rules (not two replicas of the same service on one host; for paid plans, spread across racks/AZs), and per-tenant anti-affinity when a tenant has been flagged abusive.

Score (soft optimization). Online bin packing is the game, and the failure mode to avoid is resource stranding: a host with 40 free vCPUs and 200 MB free RAM is useless. So the primary score is dominant-resource-aware best fit: prefer the host where, after placement, the CPU:memory free ratio stays closest to the fleet's demand ratio, and prefer fuller hosts over emptier ones (best-fit-decreasing behavior, which for our skewed size distribution keeps waste in the 10–15% range versus ~30%+ for naive spreading — an estimate, but the direction is well established). Secondary scores: image locality (layers already cached → faster start), volume-replica locality, and a noisy-neighbor budget score that avoids stacking multiple historically-bursty tenants on one box.

Two deliberate asymmetries by tier: pack free-tier tightly, spread paid tiers. Free-tier replicas score toward consolidation (they're capped hard anyway, and density is the margin on a $0 plan); paid replicas take a spread bonus across failure domains because their SLA is the product.

Commit: the scheduler writes the binding to etcd with a compare-and-swap on the host's resource vector version. With one scheduler leader per cell there's no optimistic-concurrency fight in steady state; the CAS exists so a deposed leader's in-flight writes fail cleanly. I rejected Omega-style multiple optimistic schedulers per cell — our per-cell decision rate (tens/sec) doesn't need it, and one leader is far easier to reason about.

I also rejected solving placement as a global optimization (MIP/ILP): it gives maybe 5–10% better packing in exchange for seconds-to-minutes solve times and an unexplainable system. Instead, a background defragmenter (part of the capacity manager) gets the same benefit asynchronously: it watches for stranded hosts and migrates stateless containers (start new, drain old — never live-migrate as the default) at a strictly limited rate, say 1% of cell containers per hour, pausing entirely during deploys or elevated failure rates.

Capacity management and headroom

Bin packing without headroom management is how you end up unable to schedule during the exact incident that requires scheduling. Three mechanisms:

  1. Reserved headroom per cell: the capacity manager keeps N% of each resource unallocatable to normal placement — my starting point is enough to absorb the largest single failure domain in the cell (a rack, ~1% of hosts) plus one deploy wave, roughly 10–15% CPU / 15–20% memory. Failure-driven rescheduling and cold-start wakes are allowed to dip into headroom; new deploys and scale-ups are not. That priority ordering is the whole point of headroom.
  2. Fleet autoscaling with lead time: cloud host provisioning takes minutes, hardware weeks. The capacity manager forecasts per-shape demand (simple: trailing 2-week peak + growth trend, plus scheduled-deploy hints) and orders hosts to keep headroom at target. When a cell's headroom breaches the floor, it sheds by pushing new placements to sibling cells before it starts rejecting.
  3. Admission control as the last resort: when a region is genuinely out, scale-ups queue with per-tenant fairness (below), free tier queues first, and paid-tier failover placements preempt free-tier burst capacity — free-tier containers above their guaranteed floor are the designated shock absorber, and that's disclosed in the plan.

Reconciliation loops

Everything that keeps the fleet converged is a level-triggered loop with a small, single job:

Autoscaling and scale-to-zero

Signals. Agents push per-container usage and the edge pushes per-service request concurrency into a regional streaming metrics pipeline (a Prometheus-compatible TSDB fed by a lightweight aggregation tier). For request-driven services I scale on concurrency (in-flight requests per replica), Knative-style, because it reacts in seconds and doesn't need CPU profiles per app; CPU/memory-based scaling is the fallback for non-HTTP workloads. The autoscaler controller per service computes desired = ceil(observed_concurrency / target_per_replica), smoothed (fast up: 15 s window; slow down: 5 min window, because flapping costs more than brief overprovisioning), clamps to the customer's min/max and the tenant quota, and writes ReplicaSet.desired_count to the global store. The scheduler never sees metrics; it sees a number. That keeps the autoscaler independently testable and swappable.

Scale-to-zero and the 5-second cold start. After an idle window (no requests for 15 min), the last replica is snapshotted and stopped, and the edge marks the service zero-scaled. The wake path:

  1. Request hits the edge; service is at zero. The edge holds the request (buffering it, not erroring) and pings the activator.
  2. The activator calls the scheduler's Place with a wake-priority flag. Target: <100 ms, achievable because the scheduler pre-maintains reserved wake slots per cell — capacity already earmarked for the common free-tier shapes, so placement is a lookup, not a search.
  3. The agent restores the container. Here's where the isolation choice pays twice: every customer workload runs in microVM isolation (Firecracker), and Firecracker supports memory snapshot restore — resuming a booted, warmed app from a snapshot in the low hundreds of milliseconds instead of cold-booting the runtime and app. The snapshot (taken at scale-to-zero time, or at first successful boot for a new revision) lives in cell-local blob cache, a few hundred MB compressed.
  4. Edge releases the held request. Budget: hold + wake 100 ms, place 100 ms, snapshot fetch (cache-local) 500 ms–1.5 s, restore 300 ms, app resume ~0. Comfortably under 5 s; p99 with a cold snapshot cache is the risk, so snapshots for recently-active services stay pinned in cell cache.
  5. Fallback when no snapshot exists (first deploy, cache eviction): full boot from a pre-pulled image — which is why image pre-warming (next section) is load-bearing for the SLO, not just for deploy speed. A full boot of a typical free-tier app is 2–4 s; a JVM monolith won't make 5 s cold, and that's fine — scale-to-zero is a free/starter-tier feature and we say so.

Wake storms (a burst of requests to a zero-scaled service) collapse into one wake via a per-service singleflight in the activator; subsequent requests queue at the edge behind the same pending replica.

Image distribution and pre-warming

A 100k-container/hour deploy peak against a naive registry is a self-inflicted DDoS. The registry is external (per prompt) but content-addressed; my distribution layer on top:

Noisy-neighbor isolation

Untrusted multi-tenant code means isolation is a security boundary first and a performance question second.

Security boundary: every customer container runs in its own Firecracker microVM. I rejected bare Linux containers (shared-kernel escape risk is not acceptable for arbitrary tenant code) and rejected syscall-emulation sandboxes (gVisor) as the default because microVMs give a cleaner hardware-virtualization boundary and the snapshot-restore primitive the cold-start SLO wants. Cost: ~5 MB overhead per microVM and a small I/O tax — acceptable at our density.

Per-resource performance isolation, enforced by the agent around each microVM using the kernel's unified resource-control hierarchy (cgroups v2):

Beyond static limits, the agent runs a noisy-neighbor detector: when a host's steal time, PSI (pressure stall) metrics, or device latency degrade, it identifies the top burster, clamps them to their guaranteed floor, and emits an event the scheduler's noisy-neighbor score consumes — repeat offenders get spread out at placement time. Detection-plus-placement is the belt and suspenders; static caps alone waste capacity, dynamic response alone reacts too late.

I rejected fully separate host pools for free vs. paid: it simplifies isolation but strands capacity in both pools and forfeits the elasticity trick of using free-tier burst as paid-tier failover headroom. Mixed hosts with strict QoS classes get better utilization; the exception is the top enterprise tier, which can buy dedicated hosts.

Persistent disks: the pinning problem

A volume on host H means the service must run on H — which converts every host failure from a 30-second reschedule into a data problem. Two disk products, priced accordingly:

Fencing is the part that keeps this from eating data. Before promoting a replica or restoring a snapshot, the control plane must guarantee the old primary can't write: the FenceVolume call is acknowledged by the old host's agent, or — if the host is unreachable — enforced by cutting the host's network at the switch/SDN layer and waiting out a fencing timeout. Only after fencing does the new binding get written. This is exactly the flow the two-phase suspect→dead host logic feeds; stateful workloads take the slow careful path while stateless ones reschedule fast.

Planned drains (maintenance, defrag) for disk-pinned services never involve failover: the agent background-copies the volume to the destination host while the service runs, then does a brief freeze-cutover (sub-second for the final delta) — the same primitive evacuation uses at scale.

Per-tenant quotas and fairness

Quotas live in the quota service backed by the global store, checked transactionally at admission: API writes, autoscaler scale-ups, and deploy surge all pass through it. Dimensions: replicas, vCPU, memory, disk GB, concurrent deploys, and — easy to forget — control-plane operation rate. A tenant scripting 10k deploys/hour is a control-plane DoS regardless of their compute quota, so the scheduler's work queue is drained via per-tenant weighted fair queuing (token buckets keyed by tenant, weighted by tier). Failure-recovery work bypasses tenant queues entirely; it's the platform's fault, not the tenant's demand.

Regional evacuation

Desired state is global and multi-region-replicated, so evacuation is a desired-state edit plus an orderly migration, not a bespoke fire drill. An evacuation coordinator runs the state machine:

Planned (region maintenance, weeks of notice down to hours): (1) capacity check — target regions must absorb the load; the capacity manager pre-orders hosts and pre-warms images for the top services in target cells. (2) Stateless services: surge replicas in target regions, shift global edge weights gradually, drain source — literally the deploy machinery pointed across regions, tenant by tenant, rate-limited by target-region headroom. (3) Volume-backed services: background-copy volumes cross-region (or promote cross-region replicas for HA volumes that opted into it), then freeze-cutover each service — seconds of write pause per service, scheduled in tenant-notified windows for the big ones. (4) DNS/anycast never changes per-customer: the global edge owns routing, which is why the prompt's "without customer action" requirement holds.

Forced (region is gone): the edge fails traffic over immediately to surviving regions. Reconcilers in surviving regions see desired state whose region constraint is overridden by the coordinator's emergency policy and mass-place — this is where the headroom policy meets its sizing question. Holding 20%+ everywhere for a full region's load is unaffordable; instead: paid tiers get placed first into headroom plus capacity reclaimed by pausing free-tier services above their floor (free tier degrades to queued wakes), and the capacity manager slams emergency host orders. Stateless paid services should be re-serving in minutes. Standard volumes restore from their latest cross-region snapshot copy (RPO minutes — every snapshot is async-replicated out of region for exactly this day); HA volumes without cross-region replicas are the honest gap, and the product line says so.

When the scheduler itself is down or partitioned

Design principle, stated once and applied everywhere: the data plane must not need the control plane to keep serving.

Security

Beyond microVM isolation: per-tenant network segmentation (each service gets an identity-addressed overlay; default-deny east-west, tenant services reach each other only via declared private networking), mTLS with per-host SVID-style identities between agent and control plane, short-lived scoped tokens for image pulls, encryption at rest on volumes with per-tenant keys, and audit events for every desired-state mutation (the event bus doubles as the audit feed). The control plane treats agents as semi-trusted: a compromised host can lie about its own observed state but can't write another host's records or read another tenant's secrets, because secrets are delivered per-binding, sealed to the specific host identity, at start time.

Tradeoffs I'd defend, and what I rejected

Evolution

First year: two regions, cells of 1,000 hosts, standard volumes only, snapshot-restore wakes for one runtime family, manual evacuation playbook. The seams that matter are laid from day one — desired/observed split, generation fencing, level-triggered loops — because those are unretrofittable. Then: HA volumes, the defragmenter, predictive pre-pull, the noisy-neighbor detector feeding placement, and finally the forced-evacuation automation, which you only trust after game-daying it quarterly against a real (drained) region. The system's north star doesn't change as it grows: intent in one place, autonomy at the edge, and every loop safe to run twice.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): During a forced evacuation caused by a network partition rather than proven power loss, how does the surviving control plane fence a local-NVMe primary that intentionally keeps running while disconnected, given that binding generations exist only in per-cell etcd and are not enforced on the disk write path?

This lands, and the fix is to stop pretending remote fencing is sufficient. You cannot remotely fence a host you can't reach; the SDN cut only works when the partition is between the host and the control plane but not between the control plane and the switch fabric. So I add local self-fencing for volume-pinned workloads only: the agent holds a per-volume write lease against the cell store, and the virtio-block backend on the host — which the agent controls, outside the tenant VM — suspends volume writes when the lease can't be renewed for T seconds. The VM keeps running, its disk I/O stalls. The promotion side (HA replica promote, or snapshot restore) waits out lease TTL plus a clock-skew margin before acting. This is a deliberate carve-out from my "agents never act on control-plane silence" rule: stateless workloads keep serving through partitions; stateful writes self-fence. Same trade I already chose — stateful partition means unavailability, never split-brain — but now enforced on the write path, not just in etcd.

Generation enforcement on the disk path comes with it: the volume attach epoch is stamped into the block backend at attach time, and a fenced volume can't be re-armed without a new epoch from the cell store. When a partitioned host comes back after we've restored its volume from snapshot elsewhere, its copy is marked diverged and quarantined for operator/customer recovery, never merged. The forked-timeline data (writes between last snapshot and the fence) is a real RPO loss on standard volumes and the product docs say so.

Design change: agent-enforced per-volume write leases with host-local I/O fencing at the virtio-block layer; volume attach epochs checked on the disk path; diverged copies quarantined, never merged.

Q2 (Codex): A service's replicas may span cells, but ReplicaSet.desired_count is global and every cell reconciles desired versus observed state; which component assigns replica identities and count budgets to cells, and how does it prevent over- or under-creation during concurrent autoscaling, deploys, rebalancing, and cell failure?

Concede: the design said "desired projected into the cell" and never defined the projector. That's a missing component and it's load-bearing. The fix: replicas become named slots (service/rev/index 0..N-1), and a leader-elected regional assigner is the single writer of the slot→cell mapping. Cells reconcile only slots assigned to them, so no two cells can create the same replica, and under-creation is visible as an unassigned or unsatisfied slot rather than silent. The write rate is fine for a small regional store (or the global store): assignments change on scale events and rebalances, not on deploy churn — a deploy replaces the revision inside a slot, which is cell-internal.

Slot moves (rebalance, cell drain, cell death) are a two-phase handoff with a generation on the slot record: mark moving, new cell places, old cell drains, CAS to commit. If a cell's control plane dies, the assigner reassigns its slots only after the cell is confirmed down through the same suspect→dead discipline hosts get, and slot generations make a resurrected cell drop stale slot ownership — the same fencing primitive I already use for bindings, one level up. Concurrent autoscaler writes are safe because the autoscaler only writes desired_count; the assigner is the only component that turns count changes into slot creates/deletes, in order, per service.

Design change: add a regional replica assigner owning named replica slots and the slot→cell mapping, with generation-fenced two-phase slot handoff.

Q3 (Codex): What protocol preserves global quota, maxSurge, maxUnavailable, and rollout health invariants when intent is committed in CockroachDB, capacity is reserved in independent etcd stores, routing changes elsewhere, and the orchestrator can crash between those operations?

There is deliberately no cross-store transaction; the protocol is single-writer state machine plus idempotent, generation-fenced effects, with the invariants enforced at specific single points. Quota is enforced only at intent commit in CockroachDB — etcd ReserveCapacity reservations are TTL'd hints, never quota, so an expired or burned reservation can't corrupt accounting. Rollout invariants are enforced by the deploy state machine, whose cursor lives in CockroachDB and advances only in transactions that read back observed state: count of new-revision replicas health-gated (with a freshness bound — status older than the bound counts as unhealthy), count of old-revision replicas still serving. Orchestrator crash = new leader re-reads cursor and observed state and re-issues effects; every effect (place, weight-shift, drain) is idempotent and fenced, so the failure mode is a stalled wave, not an over-kill. maxUnavailable specifically cannot be violated by a crash because drains are only issued after the health-gate transaction commits, and re-issuing a drain is a no-op.

Ordering with the edge: weight-shift precedes drain, and the drain step is gated on an edge ack recorded in the deploy record — the orchestrator never removes the last serving replica on the strength of an unacknowledged routing change. The honest gap: maxSurge is enforced optimistically across cells, so during a cell-store partition the orchestrator can briefly over-surge (bounded by the per-cell reservation counts it recorded in the deploy record before the partition). Bounded overshoot of surge costs money; overshoot of unavailability costs customers — the protocol is built so only the first is possible.

Q4 (Codex): How can the platform guarantee the mandatory sub-five-second cold start when snapshots may be absent, evicted, stale, or incompatible with a destination host's CPU, kernel, or Firecracker version, and the design explicitly concedes that some applications cannot boot within five seconds?

Two real problems here and one framing dispute. Snapshot portability is real: Firecracker snapshots don't survive arbitrary CPU/kernel/VMM drift. Fix: fixed CPU templates (masked CPUID) per host-shape family, snapshots keyed by (cpu-template, kernel ABI, FC version) compatibility class, and the wake path filters placement on compatibility class — incompatible means fall back to boot, never restore-and-pray. Fleet OS/VMM rolls invalidate snapshot classes cell by cell, and re-baking rides the roll.

Staleness across deploys is handled by making snapshots per-revision and adding a step I didn't have: a snapshot bake in the deploy pipeline — for zero-scaled services, the pipeline boots the new revision once off the request path, health-checks it, snapshots it, and only then marks the revision wake-ready. That kills the "first wake after deploy eats a full boot" hole, and it gives me measured boot time per revision for free. Which answers the framing dispute: I never claimed a guarantee for arbitrary apps — the prompt says free-tier cold start under 5 s, and my design already concedes a JVM monolith won't make it. With bake-measured boot times I can enforce that honestly at deploy time: if measured boot exceeds the budget and no snapshot path exists, the platform tells the customer scale-to-zero is off (or offers paid keep-warm) instead of missing the SLO at 3 a.m. The SLO is a p99 backed by snapshot pinning for recently-active services, not a promise physics can't keep.

Design change: CPU-template-keyed snapshot compatibility classes; a deploy-time snapshot-bake step producing per-revision snapshots and measured boot times; scale-to-zero eligibility gated on measured boot time.

Q5 (Codex): With five similarly sized regions, evacuating one region adds roughly 25% load to each surviving fleet in aggregate, while the proposed headroom is only 10–20% and is fragmented by resource shapes, anti-affinity, volume locality, and cell boundaries; what concrete capacity model makes forced evacuation schedulable?

Concede the arithmetic: with 5 regions, survivors absorb +25% each in aggregate, and 10–20% fragmented headroom doesn't cover that. The design gestured at the levers; it needs the model. The invariant I'd actually run: the platform is N+1 for paid load only. Per survivor region, continuously: headroom + reclaimable(free-tier above floor, paused) + preemptible(batch/starter burst) + emergency-provisionable(within RTO) ≥ displaced_paid(largest region) / (N-1). The capacity manager evaluates this solvency check continuously against live allocation data and it gates growth — when a region's paid load rises enough to make some peer's loss unschedulable, that's an alarm and a host-ordering trigger, not a surprise during the incident. Whether the check closes depends on the paid/free resource split, which is a business input I'd measure, not invent; the design decision is that the check exists, runs on real numbers, and is allowed to say "buy hardware or stop selling in this region."

Fragmentation gets three specific answers. Shape mix: the capacity manager orders the same shape mix everywhere, sized so the displaced demand distribution fits the survivor supply distribution — and holds explicit large-shape headroom, because the 64 GB monoliths are the pinch point that percentage headroom hides. Degraded placement mode: during forced evacuation, paid services land at min replicas (not autoscale max), spread constraints relax from rack-level to host-level, and autoscale-up freezes except on SLO breach — evacuation places a floor, not a peak. Cell boundaries don't fragment much because the regional assigner (Q2) places slots across all cells; volume locality is the one constraint that genuinely doesn't relax, and those services ride the snapshot-restore path whose RPO/RTO is already priced into the product.

Design change: an explicit, continuously-evaluated N+1-for-paid solvency invariant in the capacity manager; per-shape (not just percentage) headroom; a defined degraded placement mode for forced evacuation (min replicas, relaxed spread, frozen scale-up).

Q6 (Codex): How do cgroup limits and a reactive noisy-neighbor detector guarantee that paying workloads are not degraded on mixed-tier hosts when tenants still share unpartitioned resources such as memory bandwidth, CPU caches, kernel paths, NIC queues, and NVMe tail latency?

Correct, and I'll retract the implied strength of "guarantee." What I can enforce: hardware partitioning where silicon supports it — RDT/AMD-QoS cache allocation and memory-bandwidth throttling to cap free-tier classes; no SMT sibling ever spans tenants (scheduling domains pin tenant vCPUs to whole physical cores, which also kills a class of side channels); paid tiers on dedicated physical cores as designed. NIC: SR-IOV virtual functions with hardware rate limiters for paid tiers so their queues aren't behind free-tier softirq load. NVMe: io.max ceilings plus device-level separation — paid volumes and free-tier scratch on different namespaces or different devices where the host shape allows, because io.cost weighting shares a device's tail, it doesn't isolate it. Kernel paths are the one you can't partition; microVMs already move most tenant syscall pressure into guest kernels, which is a real (if partial) answer — the host kernel sees virtio, not the tenant's syscall storm.

What remains is statistical, and the design should say so plainly: the layered defense is hardware caps for the worst offenders, placement that spreads historically bursty tenants, a detector that clamps and then evicts — and the eviction targets the burster, moving it to a packed free-tier host, not the victim. The product truth: mixed-tier hosts get an interference SLO backed by per-VM PSI/steal/latency telemetry and that enforcement ladder; a physics-grade guarantee is only available as dedicated hosts, which is exactly what the enterprise tier sells. I'd rather state that boundary than defend "guarantee" for shared uncore.

Design change: RDT/MBA class caps for free tier, tenant-exclusive physical cores (no cross-tenant SMT), SR-IOV with hardware rate limits for paid NICs, device/namespace-level NVMe separation by tier; "guarantee" downgraded to a monitored interference SLO with dedicated hosts as the hard-isolation product.

Round 2 — pushback

Q7 (Codex): Your rollout transaction is in CockroachDB, but health and bindings live in independent cell etcd stores; exactly where do you atomically reserve the maxUnavailable budget so two concurrent waves, stale health snapshots, or a failover cannot both drain replicas and exceed it?

The budget lives in exactly one place: a per-service disruption ledger in CockroachDB, and no component may drain a replica without first transactionally acquiring a drain token from it. "Two concurrent waves" for one service can't exist by construction — the deploy orchestrator is a per-service singleton state machine, and a new deploy adopts or cancels the old one inside the same transaction domain before it may act. But the sharper version of the question is that deploys aren't the only voluntary drainer: defrag migrations and planned host drains also kill replicas. So the ledger is the unifying fix — every voluntary disruption (deploy drain, defrag move, maintenance drain) draws from the same per-service token pool; involuntary failures don't ask permission, but observed-unhealthy replicas reduce the tokens available to voluntary actions. This is a disruption budget with teeth: acquisition is a CockroachDB transaction, so it is atomic, and the token names the specific replica slot and carries a generation.

The token's lifecycle handles the failure cases. A drain command delivered to the agent is fenced by token id + generation, so re-delivery after orchestrator failover is a no-op, and a failed-over leader reads the ledger and counts every outstanding token as unavailable — regardless of what stale etcd health snapshots claim. Stale health can only bias the system toward conservatism: the budget check computes unavailable as desired − provably-healthy-and-fresh, so a replica whose status is older than the freshness bound counts against the budget, never for it. Tokens are returned when the drain completes and the replacement health-gates (observed via status read-back), or when the token expires and the cell confirms the drain was never applied (agent nacked the generation). The worst reachable state is over-counting unavailability and stalling the wave — never exceeding it.

Design change: a per-service disruption-budget ledger in the global store; all voluntary drains (deploy, defrag, maintenance) must transactionally acquire generation-fenced drain tokens from it; token acquisition is the single atomic enforcement point for maxUnavailable.

Q8 (Codex): You replaced "no degradation" with a statistical SLO, while SR-IOV queues, RDT classes, and physical NVMe devices are far fewer than the roughly 60 microVMs per host; what concrete admission rule prevents paying tenants from sharing exhausted hardware-isolation domains with noisy free-tier workloads?

The admission rule is: isolation resources are first-class countable resources in the scheduler's filter phase, exactly like CPU and memory. Each host's allocatable vector includes exclusive-core slots, SR-IOV VFs, RDT class occupancy, and per-tier NVMe lanes; a paid placement that requires an isolation slot fails filter on a host that has none free — it is never placed with a softened fallback. That converts "exhausted isolation domain" from a runtime surprise into an ordinary capacity signal the capacity manager already handles (headroom per shape, order more paid-optimized hosts). Concretely, mixed hosts carry a hard cap: max paid replicas per host = min(free exclusive cores, free VFs, free paid-I/O lanes), and the scheduler's tier asymmetry does the rest — 60-VM-dense hosts are free-tier-packed hosts by construction, while paid replicas land where the isolation slot math still closes.

The domain counts work because isolation is class-based against the free tier, not per-VM. All free-tier VMs on a host share one capped RDT class (capping the aggregate is the point — I need to bound free-tier's total cache/bandwidth draw, not partition it internally), share virtio networking behind the host's shaper, and share weighted I/O on their own lanes; they consume zero VFs, zero exclusive cores, zero paid RDT classes. Paid VMs are the only consumers of the scarce domains, and per host there are few paid VMs: modern NICs expose 64+ VFs and hosts have 16–128 physical cores, so cores — not VFs or CLOS ids — are the binding constraint the packing already respects. Where it still doesn't close (a paid tenant needing exclusive NVMe on a host without a free device), filter fails and the replica lands elsewhere. Paid-vs-paid interference inside the protected class remains statistical — that's the residual the interference SLO and the bursty-tenant spread score cover, and the dedicated-host tier prices away.

Design change: isolation slots (exclusive cores, VFs, RDT class occupancy, per-tier NVMe lanes) become countable scheduler resources with hard filter-phase admission; mixed hosts get an explicit max-paid-density cap derived from them.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The cell structure is Borg with the dial turned down. Google's Borg paper runs cells up to tens of thousands of machines and gets its utilization from exactly the combination I leaned on: admission control, tight task-packing, overcommitment, and machine sharing with per-process isolation. My 2,000-host cells are deliberately smaller because my tenants are hostile and my blast-radius tolerance is lower, but the shape — one scheduler owning one cell's state in memory, packing against a skewed size distribution — is the Borg lesson, not an invention. The filter/score placement split is likewise the industry-standard shape; the Kubernetes scheduling framework formalizes it as Filter → Score → Reserve → Bind with the same serial-decide, concurrent-bind structure my commit path uses. And the claim that manual resource limits waste capacity is measured, not folklore: Google's Autopilot paper reports autopiloted jobs run at 23% slack versus 46% for manually-set limits — which is the quantitative case for letting the platform's autoscaler, not the customer, own the replica count and the limits.

The honest divergence is Fly.io, who solved almost this exact problem and went the other way. In Carving the Scheduler Out of Our Orchestrator they abandon Nomad and central scheduling entirely: flyd workers each hold their own source of truth in a local append-only store, a broker matches placement requests against worker capacity market-style, and placements either succeed synchronously or fail — no pending state, no consensus. They argue bin packing itself is wrong for a platform of many small apps. I still think my per-cell leader is the right call at 3M containers with cross-cutting invariants (disruption budgets, evacuation solvency) that a marketplace can't see — but their design is proof the "no central scheduler" corner of the space is habitable in production, and my host-autonomy rules (agents keep serving on control-plane silence, SQLite journal as local truth) borrow from the same instinct.

On scale-to-zero, the snapshot-restore bet is thoroughly validated. The Firecracker paper documents microVMs carrying millions of Lambda/Fargate workloads and trillions of requests a month, which settles "is microVM-per-tenant viable at density." Marc Brooker's SnapStart post describes the same restore-instead-of-boot move at Lambda scale, and it surfaces a gap my design has: restored clones share CPU registers, memory, and therefore PRNG state — AWS had to work with OpenSSL, Linux, and the JVM to make entropy reseed after restore. My snapshot-bake step needs a reseed hook (VM genid / /dev/urandom reinjection) or every wake of the same revision starts with identical randomness; that's a security bug, not a nit. Fly.io's suspend/resume docs show the production envelope: resume in a few hundred milliseconds versus ~2 s cold, snapshots capped at 2 GB machines and discarded on every deploy — which independently arrives at two of my calls, per-revision snapshots and snapshot-restore as a small-service feature with a boot fallback, never a durability promise.

Render themselves made the opposite build-vs-buy call, and published the bill. How Render Scaled Knative to 100k+ Web Apps describes running free-tier scale-to-zero on Kubernetes plus Knative — and spending an engineering effort stripping 2N+1 Kubernetes Services per app because Calico and kube-proxy were burning hundreds of CPU-seconds churning through them. Their fix (route all free-tier traffic through the Knative activator, which both wakes pods and reverse-proxies to them) is the same edge-holds-request-then-activator pattern in my wake path. I read their post as evidence for both sides: you can ship this on Kubernetes, and the control plane fights you the whole way once free-tier density is the business.

Two more of the hard parts have close production analogues. Volume pinning: Fly.io's Making Machines Move migrates NVMe-backed machines with dm-clone — the destination lazily hydrates blocks over iSCSI from the source while the workload runs, background threads copy the rest — which is a real implementation of the background-copy-then-cutover primitive my planned drains and evacuations assume (they also tried nbd first and hit kernel hangs; the details are why the post is worth reading). Noisy-neighbor I/O: Meta's IOCost paper is the production evaluation behind my io.cost choice — proportional, work-conserving I/O control that held latency SLOs with a deliberately abusive neighbor colocated, deployed across Meta's fleet. For image fan-out, Spegel is the stateless P2P layer-mirror pattern I specified, shipping embedded in K3s. And the design's loudest principle — the data plane must not need the control plane — has a 37-hour natural experiment: Cloudflare's November 2023 post-mortem, where the entire control plane was down for a day and a half while the global data plane kept serving, and the stated remediation was to push even more control-plane function out of core facilities.

Further reading