AI Playground — System Design

Assumptions

I'll state these up front rather than ask:

"Model runs" means a single request to a model: one prompt (possibly multi-turn history) in, one streamed completion out. A conversation is a sequence of runs.
Inference is an external service (or several providers) we call over HTTP with streaming responses, per-provider quotas, and per-token billing. We don't run GPUs.
Average run: ~4 KB of input (prompt + history reference), ~2 KB of output text. p95 maybe 50 KB. Attachments (images, files) exist but are a minority; I'll design for them without optimizing for them.
A run's latency is dominated by the model: 300 ms–2 s to first token, 5–60 s to completion.
Read traffic (viewing history, loading projects) is roughly 5x write traffic.
"Share with teammates" means link- and role-based sharing inside an org, not public internet sharing (that can come later).
The playground reuses the platform's existing auth (API keys, OAuth, org membership). I'll design authz for playground objects, not identity from scratch.

The user experience, in one screen

Before the architecture, the thing we're building:

+--------------------------------------------------------------------------+
| [Project: checkout-copilot v]        [Share] [History] [</> Get code]    |
+---------------------+----------------------------------------------------+
| SIDEBAR             |  CONVERSATION                    | SETTINGS        |
|  Prompts            |  ┌ system ─────────────────────┐ | Model: gpt-5 v  |
|   - extract-v3 *    |  │ You are a checkout assistant│ | Temp:   [0.7]   |
|   - extract-v2      |  └─────────────────────────────┘ | Max tok: [1024] |
|  Conversations      |  ┌ user ───────────────────────┐ | Top-p:  [1.0]   |
|   - "refund flow"   |  │ Customer says: {{input}}    │ | Tools:  [edit]  |
|  Compare runs       |  └─────────────────────────────┘ | Stop:   [...]   |
|                     |  ┌ assistant ── streaming... ──┐ |                 |
|                     |  │ Based on the order histo▌   │ | [Run]  [Compare]|
+---------------------+----------------------------------+-----------------+

The core loop: edit a prompt, pick a model and settings, hit Run, watch tokens stream in, tweak, run again. Everything else — versioning, history, compare, share, "get code" — hangs off that loop. The design should make the core loop feel instant and make everything else durable.

"Get code" deserves a note because it constrains the data model: the playground state must be losslessly convertible to an API request. That means we store runs as the exact request we sent (model, messages, parameters, tool definitions), not as some UI-shaped document. If we store the real request, "copy as curl/Python/TypeScript" is a template render, not a translation layer that drifts.

Requirements

Functional, in priority order:

Run a prompt against a chosen model with settings; stream the response token-by-token.
Multi-turn conversations (append turns, edit earlier turns and fork).
Persist everything: prompts, versions, conversations, runs, for one year.
Compare N runs side by side (same prompt across models, or prompt variants).
Share projects within an org with viewer/editor roles.
Export any run as an equivalent API request.
Model catalog: expose dozens of models with their capabilities (context window, modalities, tool support) and limits.

Non-functional:

Streaming first token in the browser within ~100 ms of the provider producing it — our overhead must be small relative to model latency.
Durability: a run the user saw must survive. Losing run history is the cardinal sin for an experimentation tool.
Sensitive data: encryption at rest and in transit, org-scoped isolation, audit logging, strict retention (delete at one year, provably).
Availability target 99.9% for the playground itself; we can't promise more than the model providers give us, but a provider outage should degrade (that model unavailable), not take down the app.

Out of scope for the hour: fine-tuning UIs, evals/batch jobs, public sharing, billing UI (I'll cover usage metering since it feeds billing).

Scale estimates

These numbers drive the architecture, so let's do them honestly.

100 M runs/day = 1,160 runs/s average, **11,600 runs/s peak** (10x).
200 K concurrent streams. At one open HTTP response each, that's 200 K concurrent connections. A Go/Node service handles 10–20 K idle-ish streaming connections per node comfortably, so ~20–40 streaming nodes — small. The connections are cheap; the fan-out plumbing is the design problem.
Token throughput: if a stream emits ~30 tokens/s and we forward ~10 events/s per stream (coalescing tokens), that's 2 M events/s at peak across the fleet. Fine for a durable, partitioned event log (Kafka) or direct forwarding; rules out anything per-event hitting a database.
Storage: 100 M runs/day × ~~6 KB (request + response + metadata) ≈ 600 GB/day, ~220 TB/year at the retention limit. That number kills "put run content in Postgres." Metadata (~~500 bytes/run) is ~50 GB/day of hot rows — still too much for one Postgres box over a year, so metadata gets partitioned by time and org, and content goes to object storage.
DAU 1 M, say 100 K concurrent at peak. Read QPS for browsing history/projects: a few thousand/s. Easy with caching.

The shape that falls out: a modest stateless serving fleet, a serious streaming/durability pipeline, and tiered storage.

Architecture

flowchart LR
    B[Browser SPA] -->|HTTPS / SSE| GW[API Gateway<br/>authn, rate limit]
    GW --> PS[Playground API<br/>projects, prompts,<br/>versions, sharing]
    GW --> RS[Run Service<br/>create run, SSE stream]

    PS --> PG[(Postgres<br/>metadata, partitioned)]
    PS --> RC[(Redis<br/>cache, sessions)]

    RS --> MG[Model Gateway<br/>routing, quotas,<br/>retries, metering]
    MG -->|streaming HTTP| P1[Provider A]
    MG -->|streaming HTTP| P2[Provider B]
    MG --> P3[Internal models]

    RS -->|token chunks| SB[(Redis Streams<br/>per-run buffer, TTL 1h)]
    SB -.->|resume /replay| RS
    RS -->|run events| K[Kafka]

    K --> PW[Persist workers]
    PW --> S3[(S3: run content<br/>encrypted, lifecycle 1yr)]
    PW --> PG
    K --> CH[(ClickHouse<br/>usage & analytics)]
    K --> AU[(Audit log<br/>append-only)]

    PS --> CAT[Model Catalog<br/>capabilities & limits]
    RS --> CAT

Six components matter; the rest is plumbing.

1. Playground API (CRUD plane)

Stateless service behind the gateway handling projects, prompts, prompt versions, conversations, sharing, and the model catalog read path. Boring on purpose: a relational store (Postgres) + an in-memory Redis cache, no surprises. Language choice is whatever the org runs; I'd pick Go for the streaming services (below) and would happily keep this one in the same stack.

2. Run Service (the hot path)

Owns the lifecycle of a run:

POST /v1/runs arrives with the full request payload (messages, model, params) or a conversation_id + new turn.
Validate against the model catalog (context window, allowed params), check org quota, write a runs row in state pending (this is the durability anchor — the run exists before we call the provider).
Call the Model Gateway, stream chunks back to the client over SSE, and simultaneously append chunks to a Redis Stream keyed by run_id.
On completion (or error/cancel), emit a run.completed event to Kafka with the full request/response payload; persist workers write content to object storage (S3) and finalize the Postgres row.

Why SSE and not WebSockets. The traffic is strictly server-to-client after an initial POST; SSE is plain HTTP, so it traverses proxies and corporate middleboxes that mangle WebSocket upgrades, works with standard HTTP load balancers and HTTP/2 multiplexing, and gives us Last-Event-ID resume semantics for free. WebSockets buy bidirectionality we don't need (cancel is just POST /runs/{id}/cancel) at the cost of stickier infrastructure. I'd revisit if we add live multi-user cursors, not before.

Why the Redis Stream buffer. Two reasons. First, resumability: laptops sleep, WiFi drops, users refresh mid-generation. On reconnect the client sends Last-Event-ID; we replay from the Redis Stream and reattach to the live tail. Without this, a 60-second generation lost at second 55 is money burned and a user swearing at us. Second, decoupling: the provider keeps streaming even if the client is slow or gone, so the run completes and is persisted regardless. A run the provider finished is never lost just because the browser left. Streams get a 1-hour TTL; Redis holds only in-flight and recently finished runs — at 200 K concurrent × ~50 KB, that's ~10 GB, one small Redis Cluster.

I rejected "stream straight from provider to browser with no buffer" (simplest, but drops = lost tokens and unpersisted runs) and "Kafka as the live buffer" (per-run topics or partition-key replay for interactive reads is awkward; Kafka's strength is the firehose, not random-access replay of 200 K tiny streams). Redis Streams do exactly this: append, tail, replay from offset, expire.

3. Model Gateway

One service fronts every provider. It owns:

Routing: model id → provider adapter, translating our canonical request format to each provider's API.
Quota and admission control: providers give us org-level rate limits (RPM/TPM). We track spend against them in Redis (sliding-window token buckets per provider-model). When a model is saturated, we reject fast with a clear "model at capacity, retry in Ns" rather than queueing — an interactive user won't wait in a queue, and hiding a 30 s queue behind a spinner is worse than an honest error. A small, bounded (~2 s) wait for a token is fine; beyond that, fail fast.
Retries: only for idempotent failures before first token (connection refused, 429 with retry-after honored, 5xx before streaming starts). Never retry mid-stream — you'd bill the user twice and splice two generations.
Metering: every run emits {org, user, model, input_tokens, output_tokens, latency, ttft, status} to Kafka → a columnar OLAP store (ClickHouse). This feeds billing, per-org budgets, and abuse detection.
Circuit breaking per provider-model: error-rate-based breakers so a dying provider degrades to "this model is unavailable" in the UI instead of tying up connections.

I rejected putting quota logic in the Run Service: dozens of models × several providers × changing limits is a domain of its own, and the API-platform side of the business will want the same gateway. One place to update when a provider changes its API.

4. Storage tiers

Postgres (partitioned) — metadata. Orgs, users, projects, prompts, versions, conversations, run metadata, sharing grants. Partition runs by month (retention = drop old partitions, which also makes the one-year delete cheap and provable) and shard by org_id if a single cluster strains — org is a natural shard key since no query crosses orgs except internal admin. I rejected DynamoDB here: the access patterns are relational (project → prompts → versions → runs, sharing joins, "compare these 4 runs"), the write rate to metadata (~12 K rows/s peak) is well within a sharded Postgres, and transactional versioning (below) is much nicer with real transactions.

S3 — run content. Full request/response JSON per run, zstd-compressed, keyed org/{org_id}/runs/{yyyy-mm}/{run_id}.json.zst, SSE-KMS with per-org KMS keys (envelope encryption). Lifecycle: Standard → Infrequent Access at 30 days → delete at 365. Loading a historical run is one GET, ~50 ms — fine for a history view. Recent runs (last ~24 h) are also cached in Redis so the common "rerun what I just did" path never touches S3.

Redis — cache + live streams. Session-ish state, hot run cache, stream buffers, quota counters. All reconstructible or expendable; Redis loss degrades (streams can't resume, caches cold) but loses no source of truth.

ClickHouse — usage and analytics. Token counts, latencies, per-model error rates, org dashboards ("what did we spend on gpt-x this week"). Kafka → ClickHouse via materialized views. Rejected: doing this in Postgres (100 M rows/day of append-only analytics is exactly what Postgres is bad at and ClickHouse is built for).

5. Model catalog

A small config service (backed by Postgres, aggressively cached, pushed to clients on load): for each model — context window, max output, supported modalities, tool support, parameter ranges, price, current availability (fed by the gateway's circuit breakers). The UI renders settings panels from this, so launching a new model is a config change, not a frontend deploy. Validation uses the same source, so the client and server can't disagree about what max_tokens is allowed to be.

6. Persist workers & audit

Kafka consumers that write run content to S3, finalize Postgres rows, and append to an immutable audit log (who ran/viewed/shared/exported what, when — required once "sensitive customer data" is in scope). Kafka gives us replayable durability between "run finished" and "run persisted": if S3 or Postgres hiccups, events wait in the log instead of being dropped on the floor.

Data model

orgs(id, name, kms_key_arn, retention_days=365, ...)
users(id, email, ...)
memberships(org_id, user_id, role)                    -- role: admin|member

projects(id, org_id, name, created_by, visibility)    -- visibility: private|org
project_grants(project_id, principal_id, principal_type, role)
                                                      -- principal: user|group, role: viewer|editor

prompts(id, project_id, name, head_version_id)
prompt_versions(id, prompt_id, parent_version_id, seq, content_ref,
                params jsonb, model_id, created_by, created_at, label)
                                                      -- immutable; content_ref → S3 or inline if small

conversations(id, project_id, title, created_by, forked_from_run_id NULL)

runs(id, org_id, project_id, conversation_id NULL, prompt_version_id NULL,
     model_id, params jsonb, status, input_tokens, output_tokens,
     ttft_ms, total_ms, error_code NULL, content_ref, created_by, created_at)
     -- PARTITION BY RANGE (created_at), monthly

run_groups(id, project_id, created_by)                -- a "compare" is a group
run_group_members(group_id, run_id, slot)

Decisions worth defending:

Prompt versions are immutable snapshots, not diffs. Every explicit save creates a prompt_versions row; the editor autosaves a draft separately (Redis + periodic flush) so versions stay meaningful, not one-per-keystroke. Immutability means a run's prompt_version_id pins exactly what produced it — reproducibility is the whole point of a playground. Diff storage would save bytes we don't need to save (prompts are KBs) at the cost of reconstruction bugs.
Editing an earlier turn forks the conversation. A conversation is append-only; "edit turn 3 and rerun" creates a new conversation with forked_from_run_id set. This keeps history honest — you can always see what actually happened — and it's how users think about experiments anyway ("that branch where I tried the stricter system prompt").
A run stores the request verbatim. content_ref points at the canonical request + response. "Get code" and "rerun" both read the same object, so they can't diverge.
Compare is just a group of runs. No special "comparison" execution engine: the client (or server, one call) creates N runs with a shared run_group_id, streams each independently, renders side by side. N independent SSE streams over one HTTP/2 connection — no new infrastructure for the feature.

APIs

External surface (the playground's own API; the "copy code" output targets the inference API, which is separate):

POST /v1/runs
  { project_id, conversation_id?, prompt_version_id?,
    model, params:{temperature,...}, messages:[...], run_group_id?, stream:true }
  → 200, text/event-stream
     event: chunk  id: 41  data: {"delta":"Based on"}
     event: chunk  id: 42  data: {"delta":" the order"}
     event: done          data: {"run_id":"...","usage":{...},"finish_reason":"stop"}
     event: error         data: {"code":"provider_overloaded","retryable":true}

GET  /v1/runs/{id}/stream          # resume: honors Last-Event-ID, replays from Redis
POST /v1/runs/{id}/cancel
GET  /v1/runs/{id}                 # metadata + content (S3-backed)
GET  /v1/runs?project_id=&cursor=  # history, keyset-paginated

POST /v1/projects | GET/PATCH /v1/projects/{id}
POST /v1/projects/{id}/grants      # share: {principal, role}
POST /v1/prompts/{id}/versions     # save version
GET  /v1/prompts/{id}/versions
POST /v1/conversations/{id}/fork   # {at_run_id}
GET  /v1/models                    # catalog with capabilities/limits/availability
GET  /v1/runs/{id}/export?lang=python|ts|curl

Notes: run creation carries an Idempotency-Key header (double-clicks on Run shouldn't double-bill); cancel is best-effort — we stop forwarding immediately, propagate cancellation to the provider if its API supports it, and record whatever tokens were billed. All list endpoints are keyset-paginated because offset pagination over month-partitioned run tables is a trap.

Hard parts, walked through

Streaming at 200 K concurrent

The Run Service holds two connections per active run (client-side SSE, provider-side HTTP). 400 K sockets across ~30 nodes is unremarkable for an event-loop or goroutine-per-conn server; the real risks are elsewhere:

Load balancer idle timeouts. A model can "think" for 30+ s before the first token. We send SSE keepalive comments every 15 s and set LB idle timeouts to 5 min. This is the kind of thing that works in dev and pages you in prod.
Slow clients. A phone on bad WiFi can't backpressure the provider. The Redis buffer absorbs the gap; if a client's SSE write blocks past a threshold we drop the connection and let it resume — the run keeps going.
Deploys. Draining a node with 10 K live streams: stop accepting new runs, let streams finish (most complete within 60 s), force-migrate stragglers by closing with a retry hint — the client resumes on another node via the Redis buffer. Resumability turns deploys from scary to routine.
Node death mid-run. Client resumes elsewhere and replays what Redis has, but the provider-side connection died with the node, so the tail of that generation is lost. We mark the run interrupted and offer one-click retry. Handoff of the provider connection isn't possible; pretending otherwise adds complexity for a case retry handles. This is the honest gap in the design and I'm choosing to accept it — it needs a node crash to coincide with an active generation, and the blast radius is "press Run again."

Provider quotas at 11,600 runs/s peak

Admission control is a distributed counter problem: many gateway nodes sharing per-provider-model token budgets. Redis-based sliding-window counters with local in-process sub-allocation (each node leases a slice of the budget for ~1 s) keep the hot path off Redis for most requests. When a provider 429s despite our accounting (their view is authoritative), the breaker tightens our local budget — treat provider 429s as the ground truth signal, our counters as a first approximation. Per-org fairness matters too: one org running a batch script through the playground UI shouldn't starve everyone on a popular model, so budgets are hierarchical — provider-model total, then per-org caps within it.

Grants live at the project level (project_grants), checked in the Playground API on every read/write, with org membership as the outer wall. Two subtleties:

10 K-member orgs mean per-user grants don't scale for "share with the team" — grants accept groups, and group membership resolves through a cached (Redis, 60 s TTL) membership service. Revocation within a minute is acceptable for this product; if a customer demands instant revocation we invalidate on write.
Sensitive data raises the stakes on read paths. Every run/prompt read re-checks the grant — no signed URLs handed to the browser for S3 content. Content is served through the API, which also lets us write the audit log entry (user X viewed run Y) at the only place that can.

Retention and deletion

One-year retention with sensitive data means deletion must actually happen. Three mechanisms, layered: monthly Postgres partition drops (metadata), S3 lifecycle rules (content), and — the backstop — per-org KMS keys, so an org offboarding or a legal deletion request is a key deletion that renders every remaining ciphertext unreadable even if a stray copy survives in a backup. Backups inherit the same encryption, so we don't need to rewrite year-old backup sets to honor deletion.

"Copy as code" fidelity

Because runs store the canonical request, export is: load content, render through per-language templates (curl, Python SDK, TS SDK), inject a placeholder for the API key. The one real risk is drift between playground capabilities and SDK capabilities — mitigated by generating the templates from the same OpenAPI spec the SDKs are generated from, and a CI test that executes each exported snippet against a mock and asserts the request matches the stored one byte-for-byte (modulo auth).

Failure modes

Failure	Blast radius	Behavior
Provider outage	One provider's models	Breaker opens in seconds; catalog marks models unavailable; UI grays them out and suggests alternatives. Playground itself unaffected.
Redis (streams) down	In-flight runs lose resume; caches cold	Runs still complete and persist via Kafka path; new runs stream without resume capability. Degraded, not down.
Kafka down	Persistence delayed	Run Service spools completed-run events to local disk and replays when Kafka returns; streams unaffected. Alert loudly — spool is a bounded buffer.
Postgres primary failover	~30 s of failed writes	Run creation errors briefly (client retries with idempotency key); reads served by replicas.
S3 regional issue	Historical run content unreadable	Recent runs served from Redis cache; metadata (Postgres) still lists history. Cross-region replication if the business justifies it — for a dev tool, probably wait.
Run Service node crash	Streams on that node	Clients auto-resume elsewhere; provider-side tails lost → runs marked `interrupted`, one-click retry.
Quota accounting drift	429s from provider	Provider 429 is ground truth; breaker tightens local budgets; users see "at capacity" not stack traces.

Cross-cutting: every write that must not be lost goes through either a Postgres transaction or Kafka; nothing durable depends on Redis; and the client treats run_id + idempotency key as the recovery handle for every "did that go through?" ambiguity.

Security

TLS 1.3 everywhere; mTLS service-to-service.
At rest: SSE-KMS with per-org keys for S3 content; Postgres encrypted volumes; field-level encryption is overkill here since access is already mediated by one API layer.
Contractual + technical guarantee that playground traffic to providers is excluded from training (zero-data-retention endpoints where offered) — for a tool holding "sensitive customer data," this is a top-3 customer question, not a footnote.
Audit log: append-only (Kafka → object storage, WORM), covering run creation, reads, shares, exports, deletions. Queryable by org admins.
Secrets: provider API keys live in the Model Gateway only, from a secrets manager, never in run content or logs. Log scrubbing on the streaming path — the easiest way to leak a customer's prompt is a debug log line.
Standard rate limiting and per-org spend caps at the gateway double as abuse control (someone will try to use the playground as a free inference API via scripted browsers; spend caps and bot signals contain it).

Evolution

Week 1 version (and roughly the MVP cut): one Postgres, one Redis, Run Service streaming provider→client with the Redis buffer, S3 for content, no Kafka (synchronous persist on completion), no ClickHouse (usage rows in Postgres). That serves the first ~1 M runs/day fine, and every later step is additive: introduce Kafka when synchronous persistence starts adding tail latency; introduce ClickHouse when usage queries hurt; partition, then shard Postgres by org when a partition-pruned query plan stops saving us.

Later: public sharing (adds an anonymized snapshot object — never a live pointer into org data), prompt-diff views (client-side over immutable versions), evals/batch runs (a different execution plane on the same run schema — batch wants queues and cheap off-peak capacity, exactly what we refused to build into the interactive path), and live multiplayer editing (the one thing that would finally justify WebSockets/CRDTs; deliberately not paid for today).

The design bet, in one line: treat the model as an unreliable expensive dependency and the user's run history as sacred — everything else (SSE, Redis buffers, immutable versions, verbatim stored requests) follows from those two commitments.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): You call run history "sacred," yet chunks exist only in Redis until completion and Kafka receives the full payload only afterward; if a Run Service dies after showing tokens but before emitting run.completed, what durable record reconstructs those observed tokens, and how do you reconcile that gap with your durability guarantee?

You're right, and this is a real hole, not an edge case I can wave at. I wrote "a run the user saw must survive" and then built a pipeline where the only pre-completion copies live in Redis (non-durable) and on the dying node. If the Run Service crashes at second 50 of a 60-second generation, the user watched 50 seconds of output that now exists nowhere durable. "Marked interrupted, press Run again" contradicts my own cardinal-sin framing — the tokens were seen, so they must survive.

The fix is cheap because the pipeline is already sized for it: the Run Service dual-writes coalesced chunk batches to Kafka (topic run-chunks, keyed by run_id, batched to ~1 event/s per stream, so ~200K msg/s at peak — well within what I already budgeted for event throughput). A finalizer job watches for runs stuck in pending/streaming past a heartbeat window, reconstructs partial content from run-chunks, writes it to S3, and marks the run interrupted with its partial output attached. Now "interrupted" means "here's everything you saw, plus one-click retry" instead of "gone." Redis Streams stay as the low-latency resume path; Kafka becomes the durability path for in-flight output, not just completed runs.

Design change: Run Service appends coalesced chunk batches to Kafka run-chunks during streaming (not only run.completed at the end); a finalizer reconstructs orphaned runs from that topic; interrupted runs retain partial output.

Q2 (Codex): At 100 million runs/day, your design creates roughly 36.5 billion S3 objects and 18 TB/year of Postgres metadata before indexes and replicas; what concrete partitioning, sharding, object-compaction, and query strategy prevents lifecycle processing, object-request costs, hot organizations, and cross-partition project-history queries from breaking your scale claims?

The object count is the real problem and I under-thought it. One object per run means ~100M PUTs/day; at S3's ~$0.005/1K PUTs that's ~$500/day, and lifecycle transitions on 36.5B objects/year cost more than the storage they're managing (transition requests bill per object). The fix follows from the architecture I already have: persist workers consume from Kafka in batches, so instead of one PUT per run they pack runs into packfiles — per-org, flushed every few minutes or at 64 MB — and content_ref becomes s3://bucket/org/{org}/packs/{yyyy-mm-dd}/{pack_id}#offset:length. Reads are ranged GETs, still one round-trip. Object count drops three orders of magnitude, lifecycle and transition costs become rounding errors, and per-run deletes are handled by the crypto-erasure mechanism in Q6, not by S3 object deletion.

Postgres: 500 bytes/run × 36.5B runs is ~18 TB before indexes, call it 40–50 TB with indexes and replicas. One cluster doesn't hold that; I said "shard by org_id if it strains" — it strains, so shard from the start: 32–64 shards hashed on org_id, monthly range partitions within each shard. Every product query is org-scoped (project history, conversation lists, compare groups), so routing is a hash lookup and no query fans out across shards. Cross-partition project history is fine because it's always "newest first, keyset-paginated": index (project_id, created_at DESC, id), the planner prunes to the partitions the cursor touches, and pagination walks backward through months one partition at a time. Hot orgs: per-org run caps bound any single org's row rate, and an org that outgrows its neighbors gets moved to a dedicated shard — org_id sharding makes that a copy-and-cutover, not a re-key.

Design change: run content is packed into batched S3 objects with byte-range content_refs; Postgres is sharded by org_id (32–64 shards) from the start rather than "when it strains."

Q3 (Codex): POST /v1/runs streams immediately, but the client receives run_id only in the final done event; if the connection drops earlier, how does it identify the run for GET /runs/{id}/stream, cancellation, or idempotent recovery without accidentally starting and billing a second generation?

That's a bug in my API sketch, plainly. The recovery handle has to exist before anything can go wrong. Two changes: first, the SSE response's first event — emitted immediately after the pending row is written, before we touch the provider — is event: created, data: {"run_id": "..."}. The client has the id before the first token, usually within tens of milliseconds. Second, the Idempotency-Key isn't just dedup decoration: it's stored on the runs row with a unique index, so if the connection dies before even the created event arrives, the client retries the POST with the same key, hits the unique index, and gets back the existing run's id and an attach to its live stream instead of a second billed generation. The key is the recovery handle for the window before run_id exists; run_id takes over after.

Design change: add a created first event carrying run_id; runs.idempotency_key gets a unique index and replayed POSTs attach to the existing run's stream.

Q4 (Codex): How does admission control reserve provider TPM and enforce real-time spend caps when output-token usage is unknown until generation finishes: do you reserve max_tokens and waste capacity, reserve an estimate and risk oversubscription, or terminate streams mid-generation when the budget is exhausted?

I reserve an estimate, and I'm explicit about which risks that trades. Input tokens are countable at admission. Output is reserved as min(max_tokens, p90 of observed output length for that model), trued-up when the run finishes — reserve-then-reconcile, not reserve-max. Reserving max_tokens wastes most of the budget (users leave max_tokens at 1024+ and median outputs are a few hundred tokens), which turns into artificial 429s for our own users; that's a worse steady-state than occasional oversubscription. Oversubscription is survivable precisely because I already treat provider 429s as ground truth: when our estimate runs hot and the provider pushes back, the breaker tightens the local budget and admission gets stricter. Provider TPM is a rate limit, not a hard wall — brief overshoot means throttling, not an outage.

Spend caps are a different animal because they're a budget, not a rate. I enforce them at admission against the same estimate, and I never kill a stream mid-generation for a small overshoot — an org near its cap gets in-flight runs completed. The cap is a cost control, not a security boundary. The one case that terminates mid-stream is runaway abuse (spend rate wildly beyond cap, bot signals), and that's a kill switch, not the accounting path. (Round 2 sharpens the spend-cap invariant — my "overshoot bounded by one run" claim here doesn't survive concurrency; see Q8.)

No design change to mechanism — but the doc should have said "reserve estimate, reconcile on completion" explicitly instead of leaving reservation semantics implicit.

Q5 (Codex): Your schema has no conversation-turn table, stable turn ordering, or explicit parent-run edge; how do concurrent appends, replay, editing an earlier message, and branching produce a deterministic conversation without repeatedly downloading and interpreting full request blobs from S3?

Correct — I leaned on "a conversation is a sequence of runs" and never gave turns first-class rows, which means ordering lives implicitly in run timestamps and full message lists live only in S3 blobs. That fails exactly where you point: concurrent appends have no serialization point, and rendering a conversation would mean fetching and diffing request blobs. Concede and fix.

Add: turns(id, conversation_id, seq, role, content, content_ref NULL, run_id NULL, created_at) with a unique constraint on (conversation_id, seq). Appends carry the client's expected_last_seq; the insert takes seq = expected_last_seq + 1 and the unique constraint turns a concurrent append into a clean 409 (client refetches and retries or forks). That's optimistic concurrency with the database as the arbiter — no locks held across the provider call. Most turn content is small enough to inline in the row (my own estimate was ~KB-scale messages); big/multimodal content overflows to content_ref. Rendering a conversation is now one indexed Postgres query, no S3 involved. The verbatim S3 request blob per run stays, but its job narrows to what it's good at: export fidelity and exact rerun. Forks copy turn rows up to the fork point — cheap, since content is either small or a shared immutable ref. Editing turn 3 = fork with turns 1–2 copied and a new turn 3, exactly the semantics I described but now with a table that actually implements them.

Design change: add the turns table with (conversation_id, seq) uniqueness and expected_last_seq optimistic appends; conversation rendering reads Postgres, not S3.

Q6 (Codex): How do you prove deletion of one expired run or one user's data across S3 versions and replicas, Kafka retention, Redis caches, ClickHouse, audit records, and backups when the proposed cryptographic-erasure mechanism uses one KMS key for the entire organization and therefore cannot selectively erase that data?

The premise is right: an org-level key erases an org, and I sold it as a backstop for finer-grained deletion, which it can't do. Let me separate the two deletion problems, because they have different mechanisms. Retention expiry (the one-year rule) doesn't need crypto erasure: partition drops delete metadata, pack lifecycle deletes content (versioning disabled on this bucket, so no shadow copies; replication propagates deletes), Kafka retention is 7 days, Redis TTLs are minutes-to-hours, and ClickHouse never holds prompt/response text at all — metering fields only (tokens, latency, model, status), which is worth stating as a hard rule because it takes the biggest analytics store out of the deletion problem entirely. Audit logs store ids and actions, not content, and legitimately survive deletion (that's their job).

Selective deletion — one run, one user's data, a GDPR request — needs finer keys. Envelope encryption already gives every packed object segment a data key; the change is where wrapped keys live: a data_keys table in Postgres (run_id → data key wrapped by the org KEK), and that table gets short backup retention, ~30 days, versus the standard cycle for everything else. Hard-deleting a run = delete its key row. After 30 days no backup anywhere holds the wrapped key, so even a stray ciphertext copy is unreadable — crypto erasure at run granularity, with a bounded 30-day window that goes in the DPA rather than a false "instant everywhere" claim. "Proof" is a deletion certificate: the deletion job records what it deleted (key row, S3 versions if any, ClickHouse mutation ids) into the audit log, queryable by org admins.

Design change: ClickHouse is contractually content-free (metering fields only); add a per-run data_keys table wrapped by the org KEK with 30-day backup retention; deletion jobs emit signed deletion certificates to the audit log. Org KMS key deletion remains the org-offboarding backstop, no longer oversold as per-run.

Round 2 — pushback

Q7 (Codex): Your Kafka fix still loses displayed tokens unless each append is acknowledged before the SSE write; but waiting puts Kafka latency and outages directly on 200,000 live streams. What exact ordering and backpressure protocol preserves the <100 ms latency target when Kafka slows or fails?

The interviewer is right that I can't have both, so I'm choosing: display is never gated on Kafka. The ordering is display-first — SSE write goes out immediately, the same chunk is appended to Redis (sub-millisecond, on the path anyway for resume), and Kafka gets asynchronous batched appends with acks=all, flushed every ~1 s or 16 KB per run. That preserves the latency target unconditionally, and it means the strict guarantee "every displayed token survives" is really "every displayed token survives except the last unflushed batch, and only if the node dies inside that window." I'd rather state that honestly as an RPO than pretend to zero: RPO for in-flight output is ~1 s of tokens, and it takes a node crash landing inside that exact second to realize it. Completed runs stay RPO-zero in effect — run.completed carries the full payload, and a run isn't the user's problem anymore once the finalizer or persist worker has it.

When Kafka slows or fails, the protocol never backpressures the stream: the producer's in-memory buffer is bounded per node, and on overflow or broker unavailability, batches spill to the local disk spool — the same spool the original design already uses for run.completed during Kafka outages — and a replayer drains it when brokers recover. The degradation ladder is explicit: healthy = Kafka within ~1 s; Kafka slow = disk spool, durability now survives process restart but not disk loss; Kafka down and disk full = alert and shed durability for in-flight chunks only, streams keep flowing, completed runs still spool with priority over chunk batches. Each rung trades a little durability for zero latency impact, and the monitoring story is one number: end-to-end chunk-to-broker lag.

Design change: the chunk durability guarantee is restated as an explicit ~1 s RPO for in-flight output (display → Redis → async batched Kafka with acks=all); local disk spool extends to run-chunks batches with run.completed events prioritized over chunk batches when spooling.

Q8 (Codex): A p90 reservation provides no bound under 200,000 correlated long-output runs, and the claim that spend-cap overshoot is limited to one run is false under concurrency. What atomic admission invariant actually guarantees provider TPM and org spend limits in the worst case?

Conceded on the concurrency math: N in-flight runs admitted near the cap can each overshoot, so "bounded by one run" was wrong — the true exposure under estimate-based admission is N × (max_tokens − estimate), which for a big org is not small. The fix is to stop using one mechanism for two limits with different semantics. Org spend caps become a hard invariant with atomic worst-case reservation: admission runs a single Redis Lua script against the org's budget key — admit only if spent + reserved_outstanding + worst_case_cost(this run) ≤ cap, where worst_case_cost uses full max_tokens, reserve atomically, release the unused portion when the run completes with actual usage. The invariant is that the sum of actual spend plus outstanding worst-case reservations never exceeds the cap, so overshoot is exactly zero regardless of concurrency. The cost is transient utilization: an org can be refused admission while headroom is "reserved but won't be used." That's fine for a budget — reservations release within a run's duration (seconds to a minute), and the failure mode is "you hit your cap a few minutes early," which is what a cap is for. The org key is a single Redis key, but it's per-org, so there's no global hot key; the largest org's run rate is thousands per second at absolute worst, well within one Redis shard's Lua throughput.

Provider TPM stays estimate-based, deliberately, because the worst case there is categorically different: it isn't our promise, the provider enforces it independently, and the penalty for oversubscription is throttling, not money or a broken contract. Under 200 K correlated long-output runs the p90 estimate runs hot, the provider 429s, the breaker tightens our admission budget, and new runs see "model at capacity" — degraded throughput, no violated invariant, no killed streams. Reserving max_tokens against TPM would permanently sacrifice the majority of real provider throughput to guard against a failure mode that self-corrects in seconds. So the split is: hard atomic reservation where the limit is a promise we make (spend caps), feedback-controlled estimation where the limit is a rate someone else enforces (provider TPM).

Design change: org spend caps move to atomic worst-case (max_tokens-priced) reserve-and-release via a per-org Redis Lua script — overshoot becomes zero by invariant; provider TPM admission stays estimate-based with 429-driven tightening.

What changed, summarized

In-flight output is now durable: coalesced chunk batches stream to Kafka (run-chunks) during generation, a finalizer reconstructs orphaned runs, and interrupted runs keep partial output. Stated honestly as ~1 s RPO, display-first, with disk-spool fallback when Kafka degrades (Q1, Q7).
Run content is packed into batched per-org S3 objects with byte-range content_refs instead of one object per run; Postgres is sharded by org_id (32–64 shards) from day one (Q2).
The SSE stream opens with a created event carrying run_id, and idempotency keys are unique-indexed so a retried POST attaches to the existing run instead of double-billing (Q3).
Conversations get a real turns table with (conversation_id, seq) uniqueness and optimistic expected_last_seq appends; rendering reads Postgres, not S3 blobs (Q5).
Deletion is split by granularity: ClickHouse is contractually content-free, per-run data keys (wrapped by the org KEK, 30-day backup retention) give run-level crypto erasure, and deletion jobs emit certificates to the audit log; the org KMS key is only the org-offboarding backstop (Q6).
Org spend caps are enforced by atomic worst-case reserve-and-release (zero overshoot by invariant); provider TPM stays estimate-based with provider 429s as the feedback signal (Q4, Q8).

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The resumable-stream design — buffer chunks server-side, hand the client a cursor, replay on reconnect — is now the published pattern at both ends of the stack. OpenAI's own API does exactly this: background mode stamps every streaming event with a sequence_number and lets a dropped client resume with GET /v1/responses/{id}?stream=true&starting_after=42. That's the same shape as my created event plus Last-Event-ID replay, just with the cursor in a query param instead of a header. On the app side, the Vercel AI SDK's resume-streams feature is literally the Q1 mechanism: a Redis buffer keyed by stream id, a GET endpoint that reattaches, and the producer kept alive after the client disconnects. Ably's analysis of that pattern names the same costs I conceded in Q7 — write amplification (every token hits Redis even when nobody disconnects) and cancellation ambiguity (a dropped socket looks identical to a deliberate stop) — and argues for durable per-session state when streams run long or span devices.

The Model Gateway maps onto what LiteLLM ships. Their "life of a request" doc draws my exact storage split: Postgres for keys, teams, and spend; Redis for rate-limit counters shared across proxy pods; and — the part I'd underline — spend logging runs as an async background task with no database write in the request path, which is the same display-first, account-later ordering I argued for in Q7. Their budget enforcement is the Q8 mechanism verbatim: estimate the maximum request cost up front, reserve it against the applicable budgets, reject if the reservation would blow the cap, replace the reservation with actual cost when the response lands. Seeing reserve-then-reconcile as the shipped default in the most-deployed open-source gateway makes me more confident that the spend-cap/TPM split is the right cut, not an interview-only construction.

Where a real gateway diverges from my sketch is failure detection. OpenRouter's gateway writeup lists failure modes my error-rate circuit breakers would miss entirely: providers returning consistent HTTP 200s with truncated or structurally wrong responses, providers at 10x normal latency, providers healthy in us-east-1 and degraded everywhere else. They monitor throughput, time-to-first-token, and output-quality signals over a rolling 5-minute window instead of counting errors. My breaker design should absorb that — TTFT and truncation-rate signals per provider-model-region, not just 5xx counts.

Cloudflare made the opposite transport call, and their reasoning is worth stating fairly. AI Gateway added WebSockets on Durable Objects because their clients multiplex many concurrent inference requests over one persistent connection, correlating parallel streamed responses by an eventId field. That's my "compare = N runs" case solved at the connection layer instead of by HTTP/2 multiplexing of N SSE streams. I'd still take SSE for a browser playground — the middlebox and LB arguments hold — but the Durable Object per session is also the "durable session" answer to node-death-mid-run: the provider-side connection lives in an addressable object that survives client reconnects, which closes the one gap I explicitly accepted in the original design.

The metering pipeline (Kafka → ClickHouse, content-free) is the standard build. ClickHouse's LLM-observability writeup documents Langfuse, Helicone (billions of logs), and LangSmith — which moved off Postgres for exactly the append-only-analytics reason I cited — all landing token counts, latencies, and traces in ClickHouse. And one layer down, the reason I treat provider TPM as "a rate limit, not a hard wall" has a mechanism behind it: continuous batching means providers admit new sequences into a running batch iteration-by-iteration, so brief oversubscription degrades into queueing and throttling rather than failure — and it's the serving architecture we'd want for the "internal models" box if that path ever gets built.

Updates from post-training information

Two things I'd adjust, both from OpenAI's post-2024 API direction rather than anything that breaks the architecture.

First, the "store the request verbatim = lossless export" invariant weakens against stateful provider APIs. OpenAI's Responses API (March 2025, now the recommended surface for new integrations) keeps reasoning state server-side and chains turns by previous_response_id — so for reasoning models, the verbatim request alone no longer reproduces the run, and "get code" for a multi-turn conversation should either render the full stateless message history or store the provider's response-id linkage alongside the canonical request. The design change is small (an extra field on runs and a branch in the export templates) but the invariant needs the asterisk.

Second, a footnote on my zero-data-retention security claim: OpenAI's background mode docs state that even with store=false in ZDR projects, response data is held on disk for roughly 10 minutes to enable async polling. "Zero-data-retention endpoints where offered" is still the right posture, but the DPA language should say "provider-side retention per provider documentation," not "zero," because the providers themselves don't promise literal zero on every path.