Contents

AI Evaluation and Improvement Platform

Assumptions I'm making up front

The prompt leaves a few things open. Here's where I'm landing, so the rest of the design has a floor to stand on:

Scale estimates that shape the architecture

Trace ingest. 1B traces/day = ~11,600/sec average; assume a 3× peak → ~35K/sec. At 50 KB average that's ~580 MB/s average, ~1.7 GB/s peak. Raw volume is 50 TB/day, 4.5 PB for 90 days. That one number decides the storage design: 4.5 PB does not go in a search cluster or an OLTP database. Payloads go to object storage; only extracted fields get indexed.

Search index. If we extract ~2 KB of searchable fields per trace (app, timestamps, model, latency, cost, outcome, error class, first N chars of input/output), that's 2 TB/day raw, roughly 15–25 TB for 90 days after columnar compression. That fits a modest columnar OLAP cluster (ClickHouse).

Eval execution. 10,000 proposed changes/month is only ~14/hour on average, but a full run is 100K model calls. If 20% of runs are full-dataset and the rest are ~1K-call smoke runs, that's roughly 208M eval calls/month, ~80/sec sustained — except it won't be sustained, it'll be spiky (everyone evaluates before the Thursday release cut). The runner is a throughput problem with hard per-provider rate limits, not a latency problem. At even $0.005/call blended, worst case is seven figures per month in model spend, so cost estimation, caching, and budgets are first-class features, not ops afterthoughts.

Curated cases. 1M cases at ~100 KB per snapshot (case includes copied context) is ~100 GB. Tiny. Curation is a correctness and workflow problem, not a scale problem — which is exactly why it deserves the careful data model below.

The reviewer experience first

Three screens, because there are three jobs.

1. Trace explorer (support staff and engineers). Left: filter bar — app, time range, model, outcome, error class, free-text over indexed fields. Center: trace list with outcome badges. Click a trace → a waterfall of spans (model call → retrieval → tool call), each expandable to full prompt/context/output. One button matters: "Flag", which opens a small form: reason (taxonomy: hallucination, bad retrieval, tone, refusal, tool misuse, other), free-text note, suggested expected behavior. Flagging is deliberately one click plus one sentence — support staff won't do more mid-ticket.

2. Case & dataset workbench (engineers, domain reviewers). A triage queue of flags. Selecting a flag shows the frozen trace and a case editor: pick which span is the unit under test, edit/redact inputs, write the expected behavior or a rubric, attach scorers, assign to datasets. A dataset page looks like a git repo: current version, commit history ("v14: +32 refund-dispute cases, removed 3 duplicates — maria, Aug 12"), diff between versions, and per-case review status (draft → reviewed → approved).

3. Experiment comparison (the decision screen). Top: candidate vs. baseline config diff (prompt text diff, model name, retrieval params). Middle: scorecard per criterion — baseline score, candidate score, delta with a confidence interval, cost and latency deltas. Bottom: the case-level table, sortable by "regressed" — because the first thing a good engineer does is read the five cases that got worse, not admire the aggregate. Buttons: request changes, approve, approve-with-canary. Approval requires a second reviewer for regulated apps.

Architecture

flowchart TB
    subgraph Apps["500 production AI apps"]
        SDK["Tracing SDK (OTel-compatible)"]
    end

    SDK -->|"~35K traces/s peak"| GW["Ingest gateway (stateless, auth + validate)"]
    GW --> K["Kafka (partitioned by app_id)"]

    K --> ARC["Archiver → S3: Parquet, partitioned app/day (4.5 PB / 90d)"]
    K --> IDX["Indexer → ClickHouse: extracted fields (~20 TB)"]

    subgraph ControlPlane["Control plane — Postgres"]
        FLAGS[Flags]
        CASES["Cases (immutable versions)"]
        DS["Dataset manifests (versioned)"]
        SC["Scorers (versioned)"]
        RUNS["Runs & results metadata"]
        AUD["Audit log (append-only, hash-chained)"]
    end

    UI["Web app / API"] --> ControlPlane
    UI --> IDX
    UI --> ARC
    FLAGS -->|"promote to case: copy payload"| CS["Case snapshot store (S3, versioned, no TTL)"]

    subgraph Eval["Eval runner"]
        TEMP["Temporal: run orchestration"]
        POOL["Worker pool"]
        RL["Per-provider rate limiter + budget guard"]
        CACHE["Result cache: hash(input, config) → output"]
    end

    RUNS --> TEMP
    TEMP --> POOL --> RL --> PROV["Model providers"]
    POOL <--> CACHE
    POOL -->|scores + outputs| RESULTS["Run results (Postgres + S3 for payloads)"]

    RESULTS --> CMP["Comparison & stats service"]
    CMP --> UI
    UI -->|approve| REG["Config registry (versioned prompts/models)"]
    REG -->|fetch config / webhook| Apps
    UI --> AUD
    REG --> AUD

Trace ingestion path

SDK → stateless ingest gateway (auth, schema validation, size caps) → a durable event log (Kafka), partitioned by app_id so one noisy app can't starve others and per-app ordering is preserved. From Kafka, two consumers:

Why Kafka and not Kinesis or direct-to-S3: I want replayability (re-run the indexer after a schema change over days of history), consumer independence, and ~1.7 GB/s peak with room to grow. Kinesis works but shard management at this size is worse than operating MSK/Confluent. Direct-to-S3 without a log loses replay and forces the gateway to do batching, buffering, and retry itself.

Why ClickHouse and not Elasticsearch: this is analytical filtering and aggregation over append-only columnar data, at 2 TB/day of index. Elasticsearch at that volume is a much larger, more expensive cluster, and we don't need relevance-ranked full-text search — we need fast filters and group-bys, plus prefix/token search over short text fields, which ClickHouse handles with bloom-filter indexes. I rejected "just query Parquet with Athena/Trino" for the interactive path: fine for offline analytics, but 2–20s query latency kills a triage UI.

Ingestion SLO: 99.9% of traces durable in Kafka within seconds; indexing lag under 2 minutes. Traces are fire-and-forget from the app's perspective — the SDK buffers locally and drops on sustained backpressure rather than ever blocking production inference.

The curation data model — the part that makes evals trustworthy

The core problem: eval results are meaningless unless every run is reproducible, and reproducibility dies in three places — traces expire at 90 days, datasets drift while experiments run, and scorers get edited after the fact. The design rule, applied everywhere: anything a run depends on is immutable and versioned; mutation creates a new version.

Control plane is a relational OLTP database — Postgres (Aurora). Everything here is small — millions of rows, not billions — and it needs transactions, foreign keys, and row-level multi-tenancy. No exotic choice to defend.

flags(id, app_id, trace_id, span_id, reason, note, created_by, created_at, status)

-- Promoting a flag COPIES the trace payload out of the 90-day store
-- into a no-TTL S3 bucket. Cases must outlive their source traces.
cases(id, app_id, created_from_flag_id, created_at)
case_versions(case_id, version, snapshot_uri,     -- inputs/context, frozen
              expected_uri,                        -- rubric or reference output
              redactions_json, review_status, edited_by, created_at)

datasets(id, app_id, name)
dataset_versions(dataset_id, version, manifest_uri, parent_version,
                 message, created_by, created_at)
-- manifest = ordered list of (case_id, case_version). Content-addressed,
-- so identical manifests dedupe and "did the dataset change?" is a hash compare.

scorers(id, app_id, name, type)   -- type: code | llm_judge | human
scorer_versions(scorer_id, version, spec_uri, created_by, created_at)
-- code: container image digest + entrypoint
-- llm_judge: judge prompt + judge model + judge model params, all pinned
-- human: rubric text + queue config

runs(id, app_id, dataset_id, dataset_version, baseline_config_version,
     candidate_config_version, scorer_set_json,   -- [(scorer_id, version), ...]
     status, budget_usd, actual_cost_usd, seed, created_by, created_at)

results(run_id, case_id, case_version, arm,       -- baseline | candidate
        output_uri, scorer_id, scorer_version, score, score_detail_uri,
        model_usage_json, latency_ms, error, attempt)

audit_events(id, app_id, actor, action, entity_type, entity_id,
             before_hash, after_hash, at, prev_event_hash)

Two decisions worth calling out:

Datasets are git-style manifests, not row memberships. A dataset version is an immutable pointer list. Adding a case creates version N+1; a run pins (dataset_id, version) and is forever re-interpretable. Diff between versions is a set difference over the manifest. I rejected mutable membership tables with added_at/removed_at timestamps — temporal queries ("what was the dataset when run 831 started?") are exactly the kind of thing people get subtly wrong, and an experiment comparing against a dataset that shifted mid-run is a silently corrupted conclusion.

Case snapshots are copies, not references. Copying the payload at promotion time costs ~100 GB total across 1M cases — nothing — and buys us cases that survive the 90-day trace TTL and survive later redaction of the source trace. Redaction of a case (PII discovered later) creates a new case version and tombstones the old snapshot object; the audit log records the tombstone, so history shows redaction happened without retaining the payload. This matters for regulated apps where "we can reproduce run 831" and "we deleted that user's data" must both be true.

Scorers

Three types, one interface: score(case, output) → {value, detail}.

The eval runner — the hard operational part

A run is: dataset version × two config versions (baseline, candidate) × scorer set → up to 200K model calls (100K cases × 2 arms) plus judge calls, against rate-limited flaky providers, with a dollar budget.

I'd build this on a durable workflow engine (Temporal). Each run is a workflow; each case-arm execution is an activity with retry policy. Temporal gives durable state for a job that may take hours, per-activity retries with backoff, and clean resume after worker crashes — writing that checkpointing logic by hand on Kafka or SQS is exactly the wheel I don't want to reinvent. I rejected Airflow (batch/cron-shaped, poor fit for 100K dynamic fan-out with per-task retries) and a homegrown Postgres job queue (fine at 10× smaller scale, and honestly a defensible v1, but recovery semantics are where homegrown queues go to die). One caveat: 200K activities in one workflow blows past Temporal's event-history limits, so the run workflow shards into ~100 child workflows of ~2K cases each.

Around the workers, four mechanisms:

Partial failure policy: a case-arm that fails after retries is recorded as error, and the comparison excludes that case from both arms (paired analysis requires pairs). A run completing with >2% errored cases is marked degraded and can't be used for approval on regulated apps.

Comparison, decision, rollout

The comparison service computes per-scorer deltas with proper paired statistics — McNemar's test for binary pass/fail, paired bootstrap CIs for continuous scores — because with 500 cases a 2-point aggregate delta is often noise, and the UI should say so instead of letting a green number ship a regression. It also always shows the regressions list: cases where baseline passed and candidate failed. Aggregate-win-with-scary-regressions is the most common real-world outcome and the UI must make it impossible to miss.

Approval writes to the config registry: versioned prompts/model configs/retrieval params that apps fetch at startup and on change notification (or via webhook into the team's own deploy pipeline). Approval requires: a completed non-degraded run on an approved dataset version, and for regulated apps a second approver who isn't the author. Approve-with-canary flips the pointer for X% of traffic; the trace pipeline already captures outcomes, so the canary dashboard is just a filtered view of production traces by config version — the loop closes with the same data it started from.

Every state transition — flag, case edit, dataset commit, scorer change, run launch, approval, rollout — lands in audit_events, append-only, each event carrying the hash of the previous event (hash chain), with periodic anchoring of the chain head to WORM S3 (Object Lock). That's tamper-evident, which is what auditors actually ask for; I rejected heavier options (QLDB, blockchain-flavored anything) as complexity without an additional requirement to justify it.

Concurrent reviewers

Several people edit the same dataset or review the same run. I'm deliberately not building CRDTs or Google-Docs-style merge — the entities are coarse (a case, a manifest) and edits are short. Instead: optimistic concurrency with version tokens on every mutable draft (If-Match: version, 409 on conflict with a "reload and reapply" UX), plus soft advisory locks ("maria is editing this case") via a Redis presence heartbeat to make conflicts rare rather than merely survivable. Dataset commits are transactional in Postgres, so two simultaneous commits serialize into v15 and v16 instead of clobbering. Human-scoring queues use atomic claim-with-lease so two reviewers never score the same item in the same run.

APIs (the shape, not the full surface)

POST /v1/traces                          # SDK batch ingest (async, 202)
POST /v1/apps/{app}/flags                # flag a trace/span
POST /v1/flags/{id}/promote              # -> case draft (copies snapshot)
PUT  /v1/cases/{id}                      # If-Match: version -> new case_version
POST /v1/datasets/{id}/commits           # manifest changes -> new dataset_version
POST /v1/scorers/{id}/versions
POST /v1/runs                            # {dataset@v, baseline@v, candidate@v,
                                         #  scorers[@v], budget_usd} -> cost estimate + run_id
GET  /v1/runs/{id}                       # status, progress, spend
GET  /v1/runs/{id}/comparison            # scorecard, CIs, regressions
POST /v1/runs/{id}/approve               # 2nd approver enforced for regulated apps
GET  /v1/apps/{app}/config               # what production fetches
GET  /v1/apps/{app}/audit?entity=...     # audit trail

Auth: OIDC for humans, scoped API keys for SDKs (write-only to /traces). Tenancy enforced with Postgres row-level security keyed on app_id grants, and per-app S3 prefixes with per-tenant KMS keys — support staff scoping isn't a UI filter, it's enforced at the row and object layer. PII: field-level redaction rules at ingest (app-configured), plus the case-tombstone path above for late discoveries.

Failure modes

Tradeoffs I'd defend

Evolution

v1 (one quarter, ~5 engineers): ingest → S3 + ClickHouse, flags → cases → versioned datasets, code + LLM-judge scorers, runner with cache and budgets, comparison UI, manual approval, audit log. v2: human-scoring queues with blind A/B, canary rollout wired to the config registry, judge calibration dashboards, early-stop. Later: automatic candidate mining (cluster negative-outcome traces to propose eval cases, human-approved before entering datasets — suggestion, not auto-curation, or the datasets rot), scheduled regression runs on every prompt registry change, and cross-app scorer libraries.

The design rule to leave the room with: the trace side is a firehose you index cheaply and mostly never read; the eval side is small data where every byte is versioned, because a reproducible "no" is worth more than a fast "maybe."

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): You claim runs are reproducible, but provider-hosted model behavior, retrieval indexes, tool responses, and sampled outputs can drift even when names and parameters are pinned; worse, your cache can compare an old baseline execution against a candidate executed today. What exactly constitutes a reproducible run, and under what conditions is cached reuse scientifically valid?

Fair hit — I used "reproducible" loosely and the design conflates two guarantees. What I can actually promise is re-interpretability: every run permanently pins and stores its inputs, configs, outputs, and scores, so you can always re-read exactly what was executed and how it was judged. Bit-identical re-execution against a hosted model is not a promise anyone can make, and I shouldn't imply it. The dangerous version of this isn't philosophical, it's the cache: comparing a baseline executed in June against a candidate executed in August silently measures provider drift, not your change.

Design change: cache entries and results get executed_at plus a provider fingerprint (the provider-reported model/system version where available, e.g. OpenAI's system_fingerprint; otherwise our own "model epoch" counter). Cache reuse is valid only within the same model epoch and within a configurable freshness window (default 14 days). To detect silent drift inside an epoch, each pinned model gets a daily sentinel probe: a fixed 50-case set run at temperature 0; output distribution shift beyond a threshold bumps the epoch and invalidates cached baselines. The comparison UI always shows the execution-time gap between arms, and approval on regulated apps requires both arms executed within the same epoch — which usually means re-running the baseline, and that's the correct spend.

Q2 (Codex): With 10,000 proposals repeatedly evaluated against the same curated dataset, optional sequential stopping, many scorers, and approval based on confidence intervals, how do you prevent test-set overfitting, multiple-comparison false wins, and invalid confidence bounds caused by adaptive stopping?

This is a real hole — the design polices per-run statistics and ignores the meta-level: 10,000 proposals per month against 1M mostly-static cases is a textbook garden of forking paths. Three mechanisms, honest about their limits.

First, holdout partitions: a dataset version can mark a fraction of cases (say 20%) as holdout — during iteration, runs report only aggregate holdout scores, never case-level holdout results, so engineers can't tune against them; the approval-gating run is scored on the holdout. Second, exposure accounting: every dataset version carries a run counter, and the UI shows "this dataset version has been evaluated against 340 times" with a staleness warning past a threshold — you can't prevent overfitting from the platform side, but you can make the exposure visible and nag teams to mine fresh cases (the candidate-mining feature in Evolution feeds this). Third, fix the stats: sequential early-stop as I described it (peek at 5%, then decide) invalidates naive CIs. Replace it with an alpha-spending group-sequential design or e-value-based always-valid inference, so early stopping is statistically licensed rather than a bug. And runs must pre-register one primary scorer; the rest are reported as descriptive, not approval-gating, which kills the "12 scorers, one went green" false-win pattern.

Design change: dataset versions support holdout partitions with case-level results suppressed; runs pre-register a single primary scorer; early-stop uses alpha-spending sequential tests instead of naive peeking; dataset versions carry an exposure counter with a staleness warning.

Q3 (Codex): The problem includes classifier and retrieval changes, but your rollout mechanism only flips a configuration pointer. How do you version and execute code artifacts, retrieval corpora/indexes, embedding models, and tool dependencies during evaluation, then atomically canary or roll back that entire dependency graph in production?

Pushback on half of this, concession on the other half. The concession: "config version" as written undersells what a candidate is. The unit of evaluation and rollout becomes an artifact bundle — a manifest that pins prompt text, model+params, classifier artifact (image digest or model file hash), embedding model version, and a retrieval corpus snapshot ID. Runs reference bundles, not prompt strings. For execution there are two modes: platform-executed (pure prompt/model changes — we render and call providers directly, the v1 path) and customer-hosted (the app exposes an eval endpoint that takes (case_input, bundle_ref) and runs its real pipeline — retrieval against the pinned corpus snapshot, its classifiers, its tools — returning the output plus sub-span telemetry). Retrieval changes need customer-hosted mode because replicating 500 teams' retrieval stacks inside my platform is a non-starter.

The pushback: "atomically canary or roll back that entire dependency graph in production" is not a thing my platform can or should promise. I can't atomically swap a customer's embedding index. What I do promise: the bundle manifest makes the dependency graph explicit and versioned; approval gates on the whole bundle; rollout is either a registry pointer flip (for the parts apps fetch from us) or a webhook into the team's deploy system carrying the bundle ID; and the canary dashboard keys traces by bundle ID, so a partially-rolled-out bundle is visible as mixed versions in production traces rather than invisible. Atomicity of their infra is their deploy system's job — my job is making sure nobody evaluates bundle A and ships bundle A'.

Design change: config versions become artifact bundles (prompt, model+params, classifier digest, embedding model version, corpus snapshot ID); the eval runner gains a customer-hosted execution mode; canary dashboards and traces key on bundle ID.

Q4 (Codex): Kafka partitioning by app_id gives a high-volume application one hot partition and allows applications hashing to the same partition to interfere, contradicting both your 1.7 GB/s peak target and noisy-neighbor isolation claim. What concrete partitioning, ordering, admission-control, and replay scheme actually scales a single application horizontally?

Correct, and my ordering claim was actually wrong on its own terms: nothing in this system needs per-app ordering. What it needs is per-trace locality — spans of one trace should land together so the archiver and indexer can assemble the tree without a shuffle. Partitioning by hash(trace_id) spreads any single app across all partitions (a 200-partition topic gives a big app 200-way parallelism instead of one hot partition) and still keeps each trace's spans in one partition. Noisy-neighbor isolation moves to where it belongs: per-app token-bucket quotas at the ingest gateway (reject with 429 above the app's provisioned rate, before Kafka sees it), plus Kafka client quotas per principal as a backstop. Replay for one app is no longer "read one partition" — it's a filtered read of all partitions — but replay is a batch operation where scanning is fine, and the S3 archive is partitioned by app_id/date anyway, so app-scoped reprocessing usually replays from Parquet, not Kafka.

Design change: Kafka partitions by hash(trace_id) instead of app_id; noisy-neighbor isolation via per-app token buckets at the gateway plus Kafka client quotas; app-scoped replay runs from the S3 archive.

Q5 (Codex): A trace stored inside batched Parquet cannot be fetched by trace ID with a byte-range request unless you maintain an exact file, row-group, and row locator; compaction, retries, late events, and incomplete files can invalidate that locator. How does your write and indexing pipeline provide low-latency, exactly-once trace lookup without creating billions of tiny objects?

Real gap — I gestured at "byte-range read into a row group" without owning the locator problem. Concrete design: the archiver writes batch objects of 2,000 traces (100 MB before compression) with a per-object footer index of trace_id → (offset, length) for the trace's serialized blob. Honestly, at that point the natural format is a simple length-prefixed blob container with a footer — Parquet was the wrong tool for the payload copy, since two S3 copies would double cost; v1 keeps one copy in the blob-container format and does analytics via the ClickHouse envelope. Lookup: when the archiver finalizes an object (S3 multipart complete), it emits locator rows (trace_id, object_uri, offset, length) into ClickHouse — the same 1B rows/day scale the indexer already writes, a few percent overhead. The locator is written only after successful commit, so the index never points at an incomplete file. Trace-open is then a ClickHouse point lookup plus one S3 byte-range GET — I'd estimate 50–200ms warm. The hot last-48h tier keeps recently-written traces in a per-trace KV so support staff opening a trace minutes after it happened don't wait on archiver batching. Exactly-once: archiver output objects are named deterministically by (partition, first_offset, last_offset), so replays overwrite idempotently rather than duplicate; if a trace lands in two objects across a rebalance edge, the locator table keeps the latest commit and readers dedupe by trace_id. Compaction of these blob containers is simply not done — they're immutable until the 90-day lifecycle delete, which removes the invalidation problem entirely.

Design change: archive format switches from Parquet to immutable length-prefixed blob containers (~2K traces each) with footer indexes; the archiver writes (trace_id, object_uri, offset, length) locator rows to ClickHouse after object commit; objects are deterministically named by offset range for idempotent replay; no compaction.

Q6 (Codex): You promise both reproducible regulated audits and deletion by tombstoning old case snapshots, yet deleting the inputs makes historical runs unreproducible while retained outputs, scorer details, caches, and audit metadata may still contain the same personal data. What are the precise retention and erasure semantics, and how do you prove both deletion completeness and audit integrity?

The tension is real and the resolution is to be precise about what each guarantee covers. The audit guarantee is completeness of events and hashes: audit_events stores actor, action, entity IDs, and content hashes — never payloads — so erasing payload bytes cannot break the chain; the chain proves "case v3 existed, had hash H, was redacted by X on date D," which is what auditors need. The reproducibility guarantee after erasure is explicitly degraded: a run touching a purged case becomes "reproducible minus N redacted cases," and the run page says so.

The mechanism I hand-waved: a deletion index and purge fan-out. At ingest, apps can tag traces with a subject ID; the indexer maintains subject_id → trace_ids, and the control plane maintains case → {snapshot objects, run output objects, score_detail objects, cache entries (cache values carry back-refs to the case IDs that produced them)}. An erasure request resolves the full closure — source traces, case snapshots across all versions, both arms' outputs and score details in every run that used the case, cache entries — and purges each, writing one deletion manifest (object keys + content hashes destroyed) as an audit event. Completeness is verified, not assumed: a weekly scrubber job scans the closure indexes for dangling references to purged hashes and alerts on any hit. Aggregate run statistics are retained — they're derived numbers — and for regulated apps we recompute the affected run's aggregates excluding the purged cases so the stored scorecard never embeds data we claim to have deleted. What I won't promise: erasure from provider-side logs of past model calls — that's governed by the provider DPA, and the design's honest statement is that erasure covers every byte we store.

Design change: subject-ID tagging at ingest with a subject → trace index; cache entries carry case back-refs; erasure runs a purge fan-out over the full closure (traces, snapshots, outputs, score details, cache) and writes a deletion manifest to the audit log; affected run aggregates are recomputed; a scrubber job verifies no dangling references to purged hashes.

Round 2 — pushback

Q7 (Codex): Your "holdout" still returns an aggregate after every iteration, so 10,000 adaptive proposals can overfit it through leaderboard feedback without ever seeing case-level results; what mechanism prevents repeated querying, and what uncontaminated evidence ultimately gates approval once that holdout is burned?

Right — aggregate feedback leaks bits per query, and the Kaggle-leaderboard literature says a determined (or merely iterative) team overfits a static holdout through aggregates alone. So I'll tighten it in two ways and then concede the deeper point. Tighten: iteration runs don't score the holdout at all — they run on the dev split only; the holdout is scored exactly once, on the approval-gating run. Between approvals the holdout gives zero feedback, so the query budget per approved change is one. For teams that iterate through many approval attempts, the holdout carries an explicit query counter, and past a budget (say 10 scored attempts on the same holdout partition) it's marked burned and the dataset needs a re-split with fresh cases before it can gate another approval. I'd also quantize the reported holdout aggregate (report deltas only above a minimum step, Ladder-style) so each query leaks fewer bits.

The concession: any static holdout has a finite lifetime, and the honest answer to "what gates approval once it's burned" is that this platform's structural advantage is a continuous supply of uncontaminated evidence. Two sources. First, fresh-slice gating: approval on regulated apps can require the candidate to be evaluated on cases promoted from production after the candidate was created — timestamp ordering makes contamination impossible, and at ~100K flags/day platform-wide there's a steady stream. Second, the canary stage is the true holdout: live traffic can't be overfit in advance, and the design already closes that loop — canary outcomes are just production traces filtered by bundle ID. So the gate is layered: dev split for iteration, one-shot holdout for the approval run, fresh slice for regulated apps, canary before full rollout. A team that games all four has shipped something that works in production, which is the point.

Design change: holdout is scored only on approval-gating runs (never during iteration), carries a query budget and a burned state requiring re-split; holdout aggregates are quantized; regulated approvals additionally require a fresh-slice run on cases promoted after candidate creation.

Q8 (Codex): You claim verified erasure completeness, yet your purge closure omits Kafka retention, ClickHouse fields, the hot KV tier, S3 versions, replicas, backups, and untagged traces — and a weekly dangling-reference scan cannot find data absent from its indexes; how do you prove, within a bounded deletion SLA, that every subject-linked byte across every copy is actually gone?

The list is correct and I'll answer it store by store, because "prove every byte is gone" decomposes into per-store mechanisms with different characters — some are active deletes, some are bounded retention, and pretending they're all the same is how deletion programs fail audits. Kafka: 24–48h retention means subject bytes age out mechanically well inside any deletion SLA; that's handled by configuration, documented as such, not scanned. ClickHouse: the indexed envelope does carry PII (first-N-chars fields), so the purge fan-out issues delete mutations by trace_id there and in the hot KV tier (which also has a 48h TTL as a backstop). S3 versions: payload buckets run with lifecycle rules that expire noncurrent versions immediately, and purge deletes all versions, not just the current one — this is a config-audit item, checked continuously. Backups: active deletion inside backups is fiction, so the semantics are bounded retention — backups expire at 35 days, and the deletion SLA is stated as purge-now-plus-backup-expiry; any restore procedure replays all deletion manifests newer than the backup before the restored data serves traffic. If a contract demands physical erasure faster than the backup tail, the upgrade is per-app envelope keys and crypto-shredding — priced, not default.

Untagged traces are the honest limit, and I won't pretend the scanner covers them: the subject index only knows what apps tag, and a "scan 4.5 PB for a name" service is a different (and mostly futile) product. The contract states it plainly: subject-scoped erasure guarantees apply to subject-tagged traffic; for untagged data we offer best-effort discovery by running the request's identifiers through the ClickHouse index. And the proof itself is a process artifact, not a metaphysical claim: each erasure produces a manifest with per-store deletion receipts (S3 delete markers across versions, ClickHouse mutation IDs, KV delete confirmations), plus the retention configs for the stores handled by expiry — that bundle, anchored in the hash-chained audit log, is what an auditor verifies. The scrubber's job is narrower than I implied and I'll restate it: it verifies that nothing reachable through our indexes still references a purged hash — a regression tripwire for the fan-out logic, not a proof of universal absence. SLA: 30 days for all online stores, backup-expiry-bounded (or crypto-shred) for cold copies.

Design change: the purge fan-out explicitly covers ClickHouse mutations, the hot KV tier, and all S3 object versions; payload buckets expire noncurrent versions immediately (config-audited); backup restores must replay deletion manifests before serving; erasure guarantees are contractually scoped to subject-tagged traffic; erasure produces per-store deletion receipts anchored in the audit log; the scrubber is scoped as an index-reachability tripwire.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The ClickHouse call is the one this design shares most directly with a shipped system. LangChain built LangSmith on Postgres first, hit exactly the wall my estimates section predicts — high-throughput trace ingest plus low-latency analytical filtering don't coexist in an OLTP store — and moved to ClickHouse, rejecting Druid and Pinot because both needed dedicated ingestion services wired to Kafka (How LangChain chose ClickHouse). Their trace-ID lookup is worth stealing: a materialized view sorted by trace ID that acts, in their words, "almost as an inverted index." That's the same shape as my Q5 locator rows, solved inside ClickHouse instead of an S3 footer index — a reasonable v1 simplification if payloads are small enough to live in ClickHouse, which at 50 KB average and 90 days they are not, so the split stands.

The "OTel-compatible SDK" assumption matches where the industry landed, with one caveat that validates a hedge I made implicitly. There are two overlapping conventions: the official OpenTelemetry GenAI semantic conventions (spans, metrics, and events for inference, agents, tool calls, and MCP — still in active development, now in a dedicated repo) and Arize's OpenInference, an OTel-complementary spec for LLM, retrieval, and tool spans with 40-plus instrumentation packages across Python, JS, Java, and Go. Neither is stable. The design indexes its own extracted envelope rather than binding the ClickHouse schema to a convention, and while the standards churn, that's the right call — accept convention-shaped spans at the gateway, normalize into your own envelope.

The flag → case → versioned-dataset loop is now standard product surface, not a novel idea. LangSmith converts production traces to dataset examples, creates automatic dataset versions on every edit, and runs annotation queues with rubrics and pairwise comparison (evaluation concepts) — essentially my three screens. Braintrust's framing is "every production failure is a candidate for your eval suite," with evals run on every commit as CI gates and hill climbing against a diffed baseline (Evals are the new PRD). Their regression view — which cases got worse, side by side — is the same bet as my regressions-first comparison table. Where we diverge: Braintrust-style gating lives in the customer's CI; my design gates approval inside the platform and offers the webhook for teams who deploy themselves. Both exist in the market, and the interview's Q3 answer (artifact bundles + customer-hosted execution) is what you need once changes stop being pure prompt edits.

On judges, Anthropic's agent-evals guidance uses the same three-grader split as my scorers section — code-based, model-based, human — and states the calibration requirement plainly: LLM judges "should be closely calibrated with human experts." My calibration-set-plus-kappa mechanism is the concrete version of that sentence. The bias literature justifies the paranoia: a systematic study across 15 judge models and ~150K evaluation instances found position bias is systematic, not random, and varies by judge and task, and the LLM-as-a-judge survey catalogs position, verbosity, and self-enhancement bias with consistency checks and calibration as the main mitigations. One thing Anthropic's post has that my design lacks: pass@k and pass^k as first-class metrics for non-deterministic systems. My cache already stores N samples per case; the scorecard should report pass^k for reliability-critical apps instead of collapsing samples into a mean.

The statistics I cited from memory in Q2 and Q7 check out as real, deployed machinery. Johari et al.'s always-valid inference was built for exactly the peek-and-stop problem — p-values and CIs that stay valid under continuous monitoring — and ran in production on a commercial A/B platform across hundreds of thousands of experiments. The Ladder is the source for the holdout-aggregate quantization in Q7: it survives fully adaptive resubmission against a static holdout, validated on real Kaggle data, and has a parameter-free variant. Neither is exotic; both are older than this product category.

Updates from post-training information

Further reading