Generative Video Platform Design

The design in one sentence: a thin, boring submission path writes jobs to a regional relational database (Postgres) that is the source of truth; a custom two-level scheduler (global router + per-region gang scheduler) allocates a permanently oversubscribed GPU fleet across tiers with preemption and checkpoints; workers stream progress events through a durable, partitioned event log (Kafka) to an SSE fanout and upload results straight to object storage behind a CDN.

The number that shapes everything comes first, because it changes what kind of problem this is.

The fleet is oversubscribed, and that's the design

Back-of-envelope demand (all estimates):

10M requests/day = ~116 submissions/sec average, ~2,300/sec at the 20x peak.
Assume the median job uses 4 GPUs for 4 minutes (the prompt gives 1–16 GPUs and 30s–20min; a 4×4 midpoint is a reasonable planning number). That's 10M × 4 GPUs × 240s ≈ 9.6B GPU-seconds/day of demand.
Supply: 50,000 GPUs × 86,400s = 4.3B GPU-seconds/day, and realistically 70–80% of that after fragmentation, node failures, and model loading.

Demand is roughly 2–3x supply even before the peak multiplier. This isn't a "scale the workers" problem — you cannot buy your way out of it on the timescale of a request. The system's real job is deciding who waits, who gets preempted, and who gets rejected, while keeping the paid and safety tiers inside their guarantees. Every major decision below follows from that.

Storage: 10M/day × 250MB avg = 2.5 PB/day, ~75 PB at 30-day retention. At roughly $20/TB-month for standard object storage that's on the order of $1.5M/month, so lifecycle policy and tiering are a real requirement, not hygiene.

Metadata is tiny by comparison: 300M job rows in the 30-day window, a few KB each — under 1 TB. Job metadata is not a scaling problem; job scheduling is.

Requirements I'm prioritizing

Functional:

Submit a generation request (text + reference images), get an ID back immediately.
Observe progress, cancel, receive the finished video, list past generations for 30 days.
Four workload classes — free, paid, internal, safety-review — share the fleet with different capacity and latency guarantees.

Non-functional, in priority order:

Never lose an accepted job. A user who waited 15 minutes and lost the result is worse than a user who was told "queue is full."
Tier guarantees hold under overload. Concretely, my SLO targets: safety-review p99 queue wait < 60s (it gates other launches), internal p95 < 5 min, paid p95 queue wait < 2 min, free is explicitly best-effort with honest wait estimates and admission control.
GPU utilization above ~80%. Idle GPUs at this fleet size are the single largest cost.
Submission-path availability 99.95%; the GPU plane can degrade (queues grow) without the control plane going down.

Explicit non-goals for the hour: the generation model itself (black box per the prompt), billing, recommendation/feed features, editing tools.

Additional assumptions I'm making: the black-box model exposes three things — a progress callback or pollable step counter, a checkpoint/restore mechanism (diffusion-style models checkpoint naturally at denoising steps), and a process-level cancel. If checkpointing didn't exist I'd push hard to add it, because preemption and failure recovery both depend on it; the fallback (restart from zero) makes everything below work but wastes GPU-time on every preemption. I also assume a user's requests can be served from any region — model weights are replicated everywhere — with a preference for the region nearest the user.

Architecture

flowchart TB
    subgraph Client
        U[User / App]
    end

    subgraph ControlPlane["Control plane (regional, stateless services)"]
        GW[API Gateway<br/>auth, rate limits, idempotency]
        SUB[Submission Service<br/>validate, quota check, safety pre-screen]
        QRY[Query Service<br/>status, history]
        FAN[Event Fanout<br/>SSE / WebSocket]
        ROUTER[Global Admission Router<br/>pick region, reject on overload]
    end

    subgraph RegionA["Region A (repeated per region)"]
        PG[(Postgres<br/>jobs = source of truth)]
        SCHED[Regional Scheduler<br/>gang scheduling, tiers, preemption]
        subgraph Fleet["GPU fleet, pooled by GPU class"]
            AG1[Node Agent + Model Runner<br/>1-16 GPUs per job]
        end
        K[(Kafka<br/>job-events)]
    end

    subgraph Storage["Storage & delivery"]
        S3[(Object storage<br/>videos, ref images, checkpoints)]
        PP[Post-processing<br/>transcode, thumbnail, safety scan]
        CDN[CDN<br/>signed URLs]
    end

    U -->|POST /generations| GW --> SUB --> ROUTER
    ROUTER -->|INSERT job| PG
    U -->|upload ref images<br/>presigned PUT| S3
    SCHED <-->|lease jobs, heartbeats| PG
    SCHED -->|place gang| AG1
    AG1 -->|progress, state| K
    AG1 -->|multipart upload| S3
    K --> FAN --> U
    K -->|state transitions| PG
    S3 --> PP --> S3
    PP -->|approve / block| PG
    U -->|GET video| CDN --> S3
    U -->|status, history| QRY --> PG

Why this shape

Postgres as the job source of truth, not a message queue. The tempting default is "put jobs on Kafka or SQS, workers consume." It doesn't survive contact with the requirements. Cancellation needs random access to a specific queued job; priority needs reordering; preemption needs to re-enqueue a running job ahead of newer ones; tier guarantees need per-tier visibility into queue depth and age. A log gives you none of that. A database-backed queue at 2,300 writes/sec peak is trivial for Postgres, and every state transition (QUEUED → SCHEDULED → RUNNING → …) becomes a transactional compare-and-swap, which kills a whole class of double-execution bugs. I keep Kafka, but only for what logs are good at: the high-volume, append-only progress event stream (a job emitting progress every 2s at ~50k concurrent jobs is ~25k events/sec — that load stays off Postgres entirely).

Rejected: a managed NoSQL key-value store (DynamoDB) for job state. It scales further than we need, and conditional-write state machines are workable, but "give me the 50 oldest queued paid-tier jobs needing H100s in this region, skip locked rows" is one indexed SELECT ... FOR UPDATE SKIP LOCKED in Postgres and an awkward GSI-scan dance in Dynamo. Metadata volume (< 1 TB hot) doesn't justify the ergonomic cost. One regional Postgres cluster per region, jobs pinned to the region that runs them, job IDs carry the region so lookup needs no global directory.

A custom scheduler, not the container orchestrator's (Kubernetes) and not Slurm. The default K8s scheduler has no gang scheduling — a 16-GPU job would grab 11 GPUs, deadlock waiting for 5 more, and starve everyone. Volcano/Kueue add gangs but their preemption and multi-tenant fairness knobs don't express "free tier may be checkpoint-preempted by paid within 10 seconds, and safety-review preempts anyone." Slurm has the semantics but is built for long batch jobs and an HPC operational model, not 2,300 submissions/sec with per-job cancel APIs. Our workload is actually simple — one process per job, no DAGs, no inter-job communication — so a purpose-built scheduler is a few thousand lines, and we own the one component the entire product economics hinge on. Kubernetes still runs the node agents and control-plane services; it just doesn't make placement decisions for generation jobs.

Push placement, not worker pull. Workers pulling from a queue is simpler and self-load-balancing, and I'd use it if every job were 1 GPU. It can't do gang scheduling: a 16-GPU job spanning two 8-GPU nodes needs a coordinator that reserves both nodes atomically. So the scheduler holds the fleet inventory and pushes placements to node agents; agents are dumb executors that heartbeat and obey.

Data model

-- Regional Postgres. Partitioned by created_at (daily), dropped after ~35 days.
CREATE TABLE jobs (
    job_id          TEXT PRIMARY KEY,      -- ULID with region prefix: "use1-01J8..."
    user_id         BIGINT NOT NULL,
    tier            SMALLINT NOT NULL,     -- 0=free 1=paid 2=internal 3=safety
    state           SMALLINT NOT NULL,     -- state machine below
    prompt          TEXT NOT NULL,
    ref_assets      TEXT[],                -- asset IDs, uploaded beforehand
    model_version   TEXT NOT NULL,
    gpu_class       TEXT NOT NULL,         -- e.g. "h100"; derived from model+params
    gpu_count       SMALLINT NOT NULL,     -- 1..16
    est_runtime_s   INT,                   -- from a predictor, drives scheduling & ETA
    priority        INT NOT NULL,          -- tier base + boosts (age, retry)
    progress_pct    SMALLINT DEFAULT 0,
    checkpoint_uri  TEXT,                  -- object storage; set on preempt/periodic
    attempt         SMALLINT DEFAULT 0,
    lease_expires   TIMESTAMPTZ,           -- worker heartbeat lease
    output_asset    TEXT,
    error_code      TEXT,
    idempotency_key TEXT,                  -- unique per user
    created_at / scheduled_at / started_at / finished_at TIMESTAMPTZ
);
CREATE UNIQUE INDEX ON jobs (user_id, idempotency_key);
CREATE INDEX ON jobs (state, gpu_class, priority DESC, created_at)  -- scheduler scan
    WHERE state IN (0 /*QUEUED*/, 1 /*PREEMPTED*/);
CREATE INDEX ON jobs (user_id, created_at DESC);                    -- history listing

CREATE TABLE assets (
    asset_id    TEXT PRIMARY KEY,
    user_id     BIGINT,
    kind        SMALLINT,          -- ref_image | video_master | video_rendition | thumb | checkpoint
    storage_uri TEXT,
    bytes       BIGINT,
    safety      SMALLINT,          -- pending | approved | blocked
    expires_at  TIMESTAMPTZ        -- 30d for videos
);

State machine (every transition is a conditional UPDATE, invalid transitions fail):

QUEUED → SCHEDULED → RUNNING → UPLOADING → POST_PROCESSING → COMPLETED
   |          |          |                        |
   |          |          +→ PREEMPTED → QUEUED    +→ BLOCKED_SAFETY
   +----------+----------+→ CANCELED
                         +→ FAILED (attempt < 3 → QUEUED, else terminal)

job_events (progress ticks, transitions) lives in Kafka with a 7-day retention and a compacted "latest state" topic; Postgres only stores the current row plus coarse transition timestamps. Reference images and outputs never touch the database — object storage only, with the asset row as the pointer.

APIs

POST /v1/assets                      -> { asset_id, presigned_put_url }   # ref images, ≤20MB each
POST /v1/generations                 -> 202 { job_id, state, queue_position, eta_s }
     Idempotency-Key header required; retries return the original job.
     Overload response: 429 { retry_after_s } (free tier only — see admission control)
GET  /v1/generations/{job_id}        -> full job state + progress + signed URLs when done
GET  /v1/generations?cursor=...      -> user history, newest first
POST /v1/generations/{job_id}/cancel -> 202 (idempotent; races with completion resolve
                                        in the state machine — whoever CASes first wins)
GET  /v1/generations/{job_id}/events -> SSE stream: state changes, progress %, preview frames
GET  /v1/videos/{asset_id}           -> 302 to CDN signed URL (15-min expiry, owner-scoped)

SSE over WebSockets because the flow is strictly server-to-client, SSE reconnects natively with Last-Event-ID, and it traverses proxies as plain HTTP. Clients that can't hold a connection poll GET /generations/{id} — the fanout service serves both from the same Kafka consumer plus an in-memory cache (Redis) of latest-progress-per-job, so reconnects and polls never hit Postgres for progress.

Submission path: gateway authenticates (JWT) and rate-limits per user; submission service validates, checks the user's quota, runs the synchronous prompt safety classifier (a fast text model, ~50ms; clearly abusive prompts are rejected before they consume GPU); the router picks a region (user proximity, then queue depth per GPU class); one INSERT; return. The whole path is stateless services in front of Postgres — at 2,300/sec peak this is unremarkable, which is the point. The write path must stay boring because it's the availability SLO.

Scheduling — the hard part

Per region, per GPU class, the scheduler runs a loop (in-memory state, rebuilt from Postgres on failover, active/standby with a lease):

1. Tier capacity as weighted shares with borrowing, not static partitions. Example split of a region's H100 pool: safety-review 5% reserved (never borrowed — it gates everything else and its p99 must hold at peak), internal 15%, paid 55%, free 25%. Any tier may borrow idle capacity from another, but borrowed capacity is reclaimable by preemption: when paid demand returns, the scheduler checkpoints and evicts free-tier jobs (borrower first, youngest first, jobs nearest completion last — evicting a job at 95% wastes the most work). Static partitions were the rejected alternative: they make guarantees trivial but strand capacity, and at 2–3x oversubscription stranded GPU-seconds are the most expensive thing in the system.

2. Gang scheduling with backfill. Multi-GPU jobs reserve all slots atomically. To stop a 16-GPU job from idling nodes while it waits for a full gang, the scheduler plans the gang (reserves slots with a future start estimate) and backfills the reserved-but-idle slots with small jobs whose est_runtime_s fits the gap — this is the classic Slurm backfill trick and it's worth 10–15 points of utilization on mixed fleets (estimate, but consistent with published HPC experience). The runtime predictor is a simple model on (prompt length, resolution, duration, model version); it also powers the user-facing ETA. It'll be wrong sometimes; backfill overruns get checkpoint-preempted like anything else.

3. Bin packing by GPU class. Nodes are pooled by class; jobs declare a class. Score placements to keep whole nodes free (a fleet fragmented into 3-GPU holes can't place 8-GPU jobs). Prefer packing multi-GPU jobs onto single nodes for NVLink; only span nodes for 16-GPU jobs on 8-GPU hosts.

4. Admission control is a feature, not a failure. At peak (20x average, demand > 2x supply) free-tier queues will grow without bound unless we stop them. Per-user daily quotas cap total free demand; beyond that, when free-tier predicted wait exceeds ~30 min, reject at submission with a 429 and an honest retry time. A rejected request costs nothing; an accepted-then-4-hours-late request costs trust and support load. Paid tier never gets a queue-full rejection under normal operation — its share plus preemption of free is sized to absorb paid peaks; if paid alone ever exceeds total fleet, that's a capacity-planning failure the system surfaces loudly, not something scheduling can hide.

5. Anti-starvation. Priority within a tier is age-boosted (priority grows with queue time), so a 16-GPU free job eventually outranks fresher 1-GPU jobs and gang-reservation guarantees it lands.

Cancellation: API does a CAS to CANCELED; if the job was queued, done. If running, the scheduler (watching transitions via Kafka) sends kill to the node agent; agent SIGTERMs the runner, frees GPUs within seconds. Races (cancel vs. finish) resolve at the database — exactly one terminal state wins.

Execution, progress, and the result path

Node agent per host: receives a placement, pulls prompt + reference images (from object storage, cached per node), launches the model runner pinned to its GPUs, renews the job's lease_expires every 10s, forwards runner progress to Kafka every ~2s, and writes a checkpoint to object storage every 60s and on SIGTERM.

When generation finishes, the runner uploads the video directly to regional object storage (S3 multipart, ~100MB parts; a 2GB output is 20 parts uploaded in parallel from a machine with fat pipes). The video never transits the control plane. Then UPLOADING → POST_PROCESSING: a CPU fleet (this part is elastic and cheap — keep it off GPUs) transcodes renditions for streaming (HLS ladder), extracts a thumbnail, and runs the output safety scan — sampled-frame classification plus known-content hash matching. Pass → COMPLETED, user notified via SSE and push; fail → BLOCKED_SAFETY and the artifact routes to the safety-review queue (human review runs as the safety-review workload tier on the same fleet where it needs model access, e.g. re-generation probes).

Delivery: CDN in front of object storage, signed URLs scoped to the owner, 15-minute expiry, issued by the query service on each fetch. Renditions serve playback; the 250MB–2GB master is download-on-request. Lifecycle: masters to infrequent-access at day 7 (most views happen in the first days — assumption to verify with real data), everything deleted at day 30 except safety-flagged content, which is legal-held separately.

Failure handling

Worker/node death: lease expires (no heartbeat for 30s) → scheduler CASes RUNNING → QUEUED (attempt+1), rescheduled from the last checkpoint. Multi-GPU gangs: one member dying kills the gang; restart whole gang from checkpoint. Three failed attempts → terminal FAILED, refund the user's quota. Deterministic output keys ({job_id}/attempt-{n}/master.mp4) make replays idempotent.
Zombie worker: a worker that lost connectivity but kept running discovers on its next lease renewal that it no longer owns the job, and kills itself. Two attempts racing to upload is harmless — the state machine records exactly one completion.
Scheduler death: standby takes the lease, rebuilds in-memory state from Postgres in seconds. Running jobs are untouched (agents execute independently); worst case is a brief placement stall.
Postgres regional failover: synchronous replica in-region, async cross-region. Submission in that region fails for the seconds of failover; other regions unaffected; running jobs keep running (agents buffer transitions to Kafka and reconcile).
Region loss: the router stops sending new work there. Queued jobs in the dead region's DB are re-submitted to healthy regions from the async replica (accepting that a few in-flight jobs restart from zero — the "never lose an accepted job" guarantee is durability of the request, not of GPU progress). Completed videos: replicate paid-tier masters cross-region; free tier accepts that a region loss may force a re-generate. Replicating 2.5PB/day for everyone doubles storage cost for a rare event; tier it.
Poison jobs: a prompt that reliably crashes the runner would burn 3 attempts × 16 GPUs. Attempt cap plus a circuit breaker on crash-signature clustering (same fault at same step across attempts → fail fast, flag for engineering).
Peak overload: absorbed by design — queues, borrowing, preemption, then free-tier 429s, in that order. The control plane never sheds paid or safety traffic.

Security and safety

AuthN via OIDC/JWT; per-user and per-IP rate limits at the gateway; idempotency keys prevent retry-storms from double-billing quota.
Reference-image uploads go through presigned URLs into a quarantine prefix; a scan (malware, CSAM hash-match against NCMEC lists, policy classifier) must pass before the scheduler will start the job.
Three safety gates total: prompt classifier at submission (cheap, synchronous), input-image scan (async, pre-schedule), output frame scan (post-generation). Defense in depth because each layer misses different things.
Job isolation on GPU nodes: one container per job, no shared GPU between tenants (MIG or whole-GPU allocation), egress from runner containers restricted to object storage and the agent socket — a compromised runner shouldn't reach the control plane.
All artifacts encrypted at rest; signed CDN URLs are the only read path; safety reviewers' access to user content is role-gated and audit-logged, with the audit log immutable (reviewers see user content, which is exactly the access that needs a paper trail).

Evolution

Day one, I'd ship: one region, the Postgres job store and state machine, a scheduler with tiers and preemption but without backfill or the runtime predictor (FIFO-within-tier is fine at low volume), SSE progress, direct-to-S3 upload, all three safety gates. That's a correct product. Then, in the order the pain will arrive: the runtime predictor + backfill (utilization pressure comes first), multi-region routing, checkpoint tuning (checkpoint interval trades GPU overhead against preemption waste — measure it), paid-tier cross-region video replication, and spot/opportunistic capacity for the free tier once preemption is battle-tested — free tier on preemptible capacity is the natural end state, because the scheduler already treats it as preemptible.

The two decisions I'd defend hardest if pushed: Postgres-as-queue over a log-based queue, because cancel/priority/preempt are random-access operations and the write volume is small; and owning the scheduler, because when demand is structurally 2–3x supply, the scheduler is the business — it decides which tier's promise gets kept, and that's not logic to rent from a general-purpose orchestrator's defaults.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): Your paid and safety SLOs rest on an assumed median workload and arbitrary capacity shares, but you never specify tier mix, service-time tails, peak duration, or GPU-class skew; what admission and capacity model proves those p95/p99 queue bounds, and what precisely happens when paid demand alone exceeds available compatible GPUs?

Fair hit: the 55/25/15/5 split and the 4×4 median are planning numbers, not a proof, and I shouldn't present p95 targets as if they follow from them. You can't prove queue bounds without the tier mix and service-time distribution, which I don't have on day one. What I can commit to is the mechanism that makes the bounds hold once measured: a per-region, per-GPU-class admission model in GPU-seconds, not request counts. The router maintains, per (region, class, tier), admitted-but-unfinished GPU-seconds (sum of est_runtime_s × gpu_count for queued+running jobs). Paid p95 wait ≤ 2 min holds iff paid's admitted backlog per class stays under (paid share + reclaimable free borrow) × effective throughput × 120s — a Little's-law style invariant the admission layer enforces continuously, with shares recomputed from the measured mix rather than fixed at 55/25. The predictor's error distribution gets folded in as a multiplicative safety margin on est_runtime_s (start at p90 of observed error).

When paid demand alone exceeds compatible supply in a class, the ladder is explicit: (1) route paid jobs to other regions with headroom in that class, (2) reclaim everything free/internal borrowed, (3) offer degraded options at submission (lower resolution or a smaller GPU class where the model supports it), (4) queue paid with honest ETAs and page capacity planning, and (5) if predicted paid wait exceeds ~15 min, 429 paid too. My original "paid never gets queue-full" was overclaimed — no scheduler can conjure GPU-seconds. The honest guarantee is: paid is the last tier to degrade and the system says so loudly and early, at submission, not after a silent 4-hour queue.

Design change: admission control tracks per-(region, class, tier) admitted GPU-seconds against effective capacity as a hard invariant; tier shares become measured/recomputed values, not constants; paid gets an explicit degradation ladder ending in a 429, replacing "paid never rejected."

Q2 (Codex): How can you claim "never lose an accepted job" when regional Postgres replicates asynchronously across regions—so acknowledged inserts can disappear during region loss—and idempotency keys exist only in the now-unavailable regional database; what protocol prevents both lost jobs and duplicate execution when clients retry an ambiguous submission elsewhere?

This is a real hole in what I wrote. With async cross-region replication, a region loss loses the replication-lag window of accepted jobs — seconds of submissions at 2,300/sec is potentially thousands of acknowledged-then-vanished jobs, and the idempotency key vanished with them, so a client retry in another region creates a second job with no memory of the first. Either I weaken the guarantee to "durable within region" or I make acceptance durable across regions. The record is tiny, so I'll do the latter.

Design change: split acceptance durability from scheduling state. Before returning 202, the submission service writes a small intake record — keyed (user_id, idempotency_key), holding job_id, tier, target region, and the request itself — to a cross-region quorum KV (DynamoDB global table with strong consistency on the key, or a 3-region Raft KV; the write is conditional-on-absence). Regional Postgres stays the scheduling source of truth; the intake log is the durability and idempotency root. Cost: one conditional cross-region write per submission, ~2,300/sec peak, adds roughly 30–70ms to the submit path — acceptable on an async endpoint. On region loss, a reconciler scans intake records pointing at the dead region and lacking a terminal state, and re-drives them into healthy regions; client retries anywhere hit the same intake key and get the original job_id back, so no duplicate execution. The 202 now means what I claimed it meant. (Q7 below forced me to sharpen what the intake record must contain.)

Q3 (Codex): What fencing mechanism makes placement safe during scheduler lease expiry, Postgres failover, or a network partition, given that an old scheduler can still push work to node agents while its replacement reallocates the same GPUs, and a conditional update on the job row does not atomically reserve resources across multiple nodes?

The design said "active/standby with a lease" and left the fencing implicit; here it is explicitly, because a deposed scheduler that keeps pushing placements is exactly how you double-allocate GPUs. Every scheduler leadership acquisition increments a monotonically increasing epoch stored in the same Postgres row as the lease (CAS on acquire). Every placement message carries (epoch, job_id, attempt). Node agents remember the highest epoch they've seen and reject anything lower — a deposed scheduler's pushes bounce. The QUEUED→SCHEDULED transition in Postgres also carries the epoch (CAS: only the current epoch may transition), so even if an agent somehow accepted a stale placement, the stale scheduler's DB write fails and the job is never acknowledged as scheduled by a dead leader.

On "a conditional update doesn't atomically reserve multi-node resources": correct, and I don't try to make it atomic across nodes. Placements are offers, and agents are the authority on their own occupancy — an agent NACKs a placement if its GPUs are busy, and the scheduler treats NACK as a retry with updated inventory. A new leader rebuilds inventory not from Postgres alone but from agent state reports (agents report what they're actually running on reconnect), so the recovering scheduler's view converges to physical truth rather than to a possibly-stale plan. The invariant that keeps gangs safe: a gang starts only when every member agent has ACKed, and any NACK releases all the gang's reservations. Worst case during failover is a rejected offer and a few seconds of placement stall, never two jobs on one GPU. (Q8 below caught the partition case where highest-epoch-seen isn't enough; the commit protocol there closes it.)

Design change: explicit epoch fencing — monotonic scheduler epoch in Postgres, carried on every placement and on the SCHEDULED CAS; agents reject lower epochs, NACK on occupancy conflicts, and report running state on reconnect; gang start requires all-member ACK.

Q4 (Codex): You assume a black-box model can produce globally consistent checkpoints for 1–16-GPU jobs every 60 seconds; what are the checkpoint size, pause time, storage bandwidth, and restore-time budgets at fleet scale, and how do preemption and recovery work if checkpoint creation partially succeeds or takes longer than the promised ten-second eviction?

The 60-second periodic checkpoint doesn't survive the math, so let me do it. A diffusion checkpoint is latents + RNG state + step counter — not weights — call it 0.5–2 GB for a long high-res job (estimate; it scales with latent volume). Concurrency: ~40k busy GPUs / 4-GPU median = ~10k concurrent jobs. At 60s intervals and 1 GB each that's on the order of 170 GB/s sustained to object storage, an absurd bill for checkpoints that are almost never read. So: periodic checkpoints drop to every 5 minutes (they only serve crash recovery, and losing up to 5 min of a 20-min job on a rare node death is fine), and preemption checkpoints happen on demand — the agent delivers a preempt signal, the runner finishes its current denoising step (steps are the natural barrier; for multi-GPU jobs the process group hits the barrier together, which is what makes the checkpoint globally consistent — one writer, one manifest), serializes, and uploads. 1–2 GB over a node's 25–100 Gbps pipe is a few seconds of transfer; the realistic end-to-end preempt budget is step time + serialize + upload, so I'm revising "within 10 seconds" to a 30-second eviction SLA, which the paid-tier admission math absorbs as reclaim latency.

Partial or slow checkpoints: a checkpoint is written to a temp key and committed by atomically writing a manifest pointer; a partial write is invisible and restore falls back to the previous manifest or to zero. If the runner blows the 30s budget, the agent SIGKILLs anyway — the job restarts from its last committed manifest and the overrun is wasted work, bounded by the periodic interval. And if the black box turns out not to checkpoint at all, preemption degrades to kill-and-restart, which changes victim selection: prefer preempting short/young/small jobs where restart waste is minimal, and never preempt jobs over 8 GPUs mid-flight.

Design change: periodic checkpoints at 5 min (not 60s); on-demand checkpoint on preempt with a 30s eviction SLA (not 10s); checkpoint commit via temp-key + atomic manifest pointer; multi-GPU consistency via step-barrier coordination in the runner process group.

Q5 (Codex): Why are racing attempts "harmless" merely because one database transition wins when stale attempts can still upload outputs, trigger post-processing, emit progress, or overwrite asset metadata; where are attempt-scoped fencing tokens enforced across workers, Kafka consumers, object storage, and post-processing, and how are losing artifacts reclaimed?

The design had the right primitive — attempt-scoped output keys — but I only enforced the token at the state machine, and the question is right that stale attempts can still make noise elsewhere. The fix is to make attempt a fencing token checked at every consumer, not just at the jobs row. Concretely: every event a worker emits to Kafka carries (job_id, attempt); the fanout service and the Postgres transition consumer drop events where attempt < the current attempt on the row (fanout reads it from its Redis cache, refreshed on transition events), so a zombie can't move progress bars backwards. The RUNNING→UPLOADING and UPLOADING→POST_PROCESSING CASes include attempt in the WHERE clause, so only the owning attempt's completion triggers post-processing, and post-processing reads the output key recorded in that winning transition ({job_id}/attempt-{n}/master.mp4), never "latest object under the job prefix." The asset row is inserted in the same transaction as the winning transition, so metadata can't be overwritten by a loser — the loser's INSERT hits a conflict and aborts.

Losing artifacts: a stale attempt uploads only under its own attempt-{n} prefix, so it collides with nothing. Reclamation is a GC sweep on terminal states — delete all attempt prefixes except the winning one — with an object-storage lifecycle rule on the job prefix as the backstop for GC bugs. Checkpoint manifests are attempt-scoped the same way.

Design change: attempt number enforced as a fencing token at all consumers — Kafka event filtering in fanout and the transition consumer, attempt-qualified CAS on every post-RUNNING transition, asset-row insert transactional with the winning transition — plus a terminal-state GC for losing attempt prefixes.

Q6 (Codex): What prevents a reference-image time-of-check/time-of-use attack when presigned uploads, asynchronous scanning, and generation submission are separate operations: can an asset be submitted before approval or overwritten after scanning, and where is the immutable content digest and explicit WAITING_FOR_ASSETS state that closes those races?

Legitimate gap — as written, nothing stops a client from submitting a job whose asset is still pending, or from re-PUTting a new object under the same presigned URL after the scan passed. Both races get closed the same way: bind approval to content, not to the key. The scanner computes the sha256 of the quarantine object, runs its checks, and on pass copies the bytes to an immutable content-addressed key (approved/{sha256}) in a prefix that denies overwrite (object lock; no PUT permission except the scanner's copy role), then marks the asset row approved with the digest. The model runner pulls reference images only from approved/{digest} — never from quarantine — so re-uploading over the quarantine key after the scan changes nothing the job can see. Presigned PUTs are single-key, content-length-bounded, short-expiry, and never re-issued for the same asset_id.

On the submission side, I'm adding the state the question asked for: jobs with pending assets enter WAITING_FOR_ASSETS and the scheduler ignores them; the asset-approval event CASes the job to QUEUED (queue age counts from approval, so slow scans don't jump the line); asset blocked → job FAILED with a clear error. The scheduler-side check ("scan must pass before start") stays as defense in depth, but now it's verifying a digest match, not a mutable flag on a mutable object.

Design change: new WAITING_FOR_ASSETS job state ahead of QUEUED; approved reference images stored at immutable content-addressed keys with overwrite denied; runners read only digest-addressed approved objects; single-use, bounded presigned PUTs.

Round 2 — pushback

Q7 (Codex): Your quorum intake stores only a prompt hash, not the prompt, parameters, or reference-asset IDs: if the target region dies immediately after the 202, from what durable record can the reconciler reconstruct and execute the accepted request?

Caught — a hash lets you verify a request but not re-run one, so my Q2 fix as stated recovers job identity and loses job content, which fails the actual guarantee. The fix is cheap because the request is small: the intake record stores the full submission payload — prompt text (a few KB), model_version, generation parameters, tier, and the ref-asset digests from Q6 — not a hash of it. A well-formed request stays under a few tens of KB, comfortably inside a KV item limit at 2,300 writes/sec; the hash was false economy.

Reference-asset content is the harder half, and the honest answer has a seam: the intake record can only carry digests, so the bytes must survive the region too. Since approved assets are already immutable and content-addressed (Q6), replicate the approved/ prefix cross-region at approval time — assets are ≤20MB, most requests carry zero or few, and at average load this is hundreds of MB/s of replication, not the 2.5 PB/day video problem. That leaves one window: region dies after the job's 202 but before its assets finish replicating. The reconciler detects it (digest in intake, object absent in surviving regions) and fails that job with an explicit "re-upload reference images" error plus a quota refund — a bounded, rare, honestly-reported loss of convenience, not a silent loss of the request. I'd rather ship that seam than put 20MB blobs on the synchronous submit path.

Design change: intake record carries the full request payload (prompt, params, ref-asset digests), not a prompt hash; approved reference assets replicate cross-region on approval; reconciler fails jobs whose assets didn't survive with an explicit re-upload error and quota refund.

Q8 (Codex): During a partition, the old scheduler and isolated agents know only epoch E, so they can ACK and start a gang before learning that epoch E+1 exists—even if the old scheduler's Postgres CAS fails; what concrete commit protocol or agent-verifiable token prevents that stale leader from starting work?

Right — "agents reject lower epochs" only works if some message carrying E+1 reaches them, and in a partition none does. Gossiped fencing can't save a fully stale partition; the token has to be verified against the authority at the moment work starts. So the gang commit becomes an explicit two-phase protocol with Postgres as the single commit point, and agents fail closed. Phase 1: the scheduler sends reserve(epoch, job_id, attempt, gpu_set) to each member agent; agents hold the GPUs idle under a short TTL (~10s) and ACK — reservation burns no GPU-seconds. Phase 2: the scheduler does the epoch-carrying QUEUED→SCHEDULED CAS; only if that commits does it send start. A stale leader at E can collect ACKs all day; its CAS fails (the epoch row lives in the same Postgres, which knows E+1), it can never legitimately issue start, and the reservations TTL out.

The remaining case is a buggy or malicious stale leader that sends start without a committed CAS — so start alone can't be sufficient. The agent-verifiable token the question asks for is the agent's own lease claim: before launching the runner, the node agent (gang leader agent, for multi-node jobs) performs one conditional write to Postgres — claim the lease WHERE job_id matches, state = SCHEDULED, scheduler_epoch = E, attempt matches. No successful claim, no GPU work, no exceptions. An agent partitioned from Postgres can't claim, so it can't start — fail closed — which is the right default because an agent that can't reach Postgres also can't heartbeat, meaning the new leader already counts its GPUs as lost. Cost: one conditional write per job start, noise at ~10k concurrent jobs. The invariant in one line: reservations are cheap and revocable; execution requires proof of current epoch from the database itself.

Design change: gang placement is two-phase — TTL'd reserve/ACK, then epoch-CAS in Postgres, then start — and node agents fail closed: launching a runner requires the agent's own conditional lease-claim write against (state = SCHEDULED, epoch, attempt); no claim, no launch.

What changed, summarized

Admission control is now stated as a per-(region, GPU-class, tier) invariant on admitted GPU-seconds vs. effective capacity, with tier shares recomputed from measured mix; "paid never rejected" replaced by an explicit degradation ladder (cross-region routing → full reclaim → degraded options → honest queue → paid 429).
New cross-region quorum intake log written before the 202: keyed (user_id, idempotency_key), carrying the full request payload; it's the durability and idempotency root, and a reconciler re-drives orphaned jobs after region loss. Approved reference assets replicate cross-region on approval; unrecoverable-asset jobs fail with an explicit re-upload error.
Scheduler fencing made concrete: monotonic epoch in Postgres, carried on every placement and on the SCHEDULED CAS; agents NACK on occupancy and report running state on leader failover.
Gang starts are two-phase (TTL'd reserve → epoch-CAS commit → start), and node agents fail closed: no launch without the agent's own conditional lease-claim write verifying state/epoch/attempt in Postgres.
Checkpointing rebudgeted: periodic every 5 min instead of 60s (the 60s cadence implied ~170 GB/s of checkpoint traffic), on-demand checkpoint on preempt, eviction SLA relaxed 10s → 30s, temp-key + atomic manifest commit, step-barrier consistency for multi-GPU jobs.
attempt enforced as a fencing token at every consumer (Kafka event filtering, attempt-qualified CASes, transactional asset-row insert), plus a terminal-state GC for losing attempt prefixes.
New WAITING_FOR_ASSETS state; reference images become immutable content-addressed objects after scanning, runners read only digest-addressed approved keys, presigned PUTs single-use and bounded.

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The design's founding claim — that GPU utilization, not throughput, is the business — is what the serverless GPU vendors say about their own economics. Modal defines the metric as allocation utilization, GPU-seconds running application code over GPU-seconds paid for, and reports that real organizations land at 10–20% against a nominal 70% (How we achieved truly serverless GPUs). Their answer to spiky demand is the same shape as mine — a warm buffer of allocated GPUs plus fast job start — just tuned for sub-minute inference instead of 20-minute generations. fal.ai's infrastructure makes the direct-upload decision I made for the result path: outputs upload asynchronously off the GPU's critical path so "the only GPU time charged is for actual inference," with saturated 10Gb links doing the transfer (Tigris case study on fal.ai). Same reason I keep the video out of the control plane.

On build-vs-buy for the scheduler, my dismissal of Kueue was thinner than the current reality deserves. Kueue's model — nominal quota per ClusterQueue, borrowing within a cohort, reclaimWithinCohort preemption when the lender's demand returns, and fair sharing that preempts from the queue with the highest share first (Kueue preemption docs) — is nearly isomorphic to my tier-shares-with-borrowing design. Safety-review-preempts-anyone is priority preemption; free-borrows-and-gets-reclaimed is reclaimWithinCohort. What Kueue still doesn't give me: checkpoint-then-evict (it evicts pods, it doesn't ask the workload to save 15 minutes of denoising first), admission in GPU-seconds against a latency invariant, and runtime-estimate backfill. Those three carry the product economics, so I'd still own the scheduler — but the honest argument is "the last mile is the business," not "the tools can't express fairness."

The backfill design is lifted from Slurm, and Slurm's own docs state both the payoff and the dependency I flagged: backfill starts lower-priority jobs only when they won't delay any higher-priority job's expected start, and "reasonably accurate time limits are important for backfill scheduling to work well" (Slurm scheduling configuration). My runtime predictor is exactly the substitute for user-supplied time limits — Slurm gets estimates from users and punishes overruns; I get them from a model and checkpoint-preempt overruns.

Preemption-by-checkpoint is not a research bet; two production systems prove it at different layers. Microsoft's Singularity makes every job in a global fleet "preemptible, migratable, and dynamically resizable by default" via transparent checkpointing that needs no cooperation from the job's code — the strongest version of my Q4 assumption, applied fleet-wide (Singularity paper, arXiv 2202.07848). Modal ships the driver-level flavor in production: NVIDIA's driver checkpoints device memory into host memory, restore skips CUDA graph and Torch compilation, and they report ~15 million GPU snapshot restorations in three months (Modal on GPU snapshots, with the container-side CRIU/gVisor mechanics in their memory-snapshots post). And on the failure-handling side, ByteDance's MegaScale treats checkpoint-restore as the normal recovery path for 10,000+ GPU jobs, paired with deep diagnostics to find the faulty node fast — the same detect-evict-resume loop as my lease-expiry path, at larger gang sizes than mine (MegaScale, arXiv 2402.15627).

Postgres-as-queue was the decision I said I'd defend hardest, and the published ceiling is comfortably above my load. DBOS runs Postgres-backed queues at 30,000 dequeues/sec using the same three moves in my schema — FOR UPDATE SKIP LOCKED, READ COMMITTED on the dequeue path, and partial indexes covering only enqueued rows (Making Postgres queues scale). My peak is 2,300 submissions/sec with ~10k concurrent jobs, an order of magnitude under their measured throughput. The partial index in my DDL (WHERE state IN (QUEUED, PREEMPTED)) turns out to be the load-bearing trick in their writeup too: full indexes on a high-churn jobs table are what melt autovacuum, not the row count.

Updates from post-training information

Three things moved since this design was written, and two of them soften claims I made.

First, the "Volcano/Kueue can't express our fairness" argument is now overstated. Kueue's fair sharing, cohort borrowing, and reclaim preemption (hierarchical cohorts became compatible with fair sharing in v0.11) express the tier-share-with-borrowing model almost directly. The custom-scheduler decision survives on the narrower grounds above — checkpoint-aware eviction, GPU-seconds admission, backfill — but I'd rewrite that paragraph to concede the fairness half.

Second, Kubernetes Dynamic Resource Allocation graduated to GA in v1.34 (September 2025) and is on by default: device allocation is now a first-class scheduler concern, with structured device requests and prioritized alternatives like "one big GPU or two mid-tier" (Kubernetes v1.34 DRA announcement). It still has no gang semantics, so it doesn't displace my scheduler, but the node-agent layer's GPU claiming and the "degraded options" rung of the paid ladder (Q1) get cheaper to build on DRA than on device-plugin-era plumbing.

Third, my Q4 fallback — "if the black box won't checkpoint, preemption degrades to kill-and-restart" — is too pessimistic in 2026. NVIDIA driver-level GPU memory checkpointing works on unmodified processes and is running in production at Modal at the scale of millions of restores. The realistic worst case is now a transparent driver-level checkpoint (bigger and slower than a cooperative latents-only checkpoint, since it captures all device memory), not a restart from zero. That changes the victim-selection math: even uncooperative jobs can be evicted without losing their work, at the cost of moving more bytes.