Contents

Agent Observability for a Developer Platform

The problem and my plan

Customers run agents on our platform and can't see inside them. When a run fails, the developer's questions are concrete: what did this run do, step by step? Why did it fail? What did it cost? Which model and prompt version served it? Is quality getting better or worse? We need a backend that answers all five, at 200M runs a day, without ever mixing tenants' data.

Two properties of this workload drive the whole design, so I'll state them up front.

First, the data splits cleanly into two kinds with opposite requirements. Span metadata (timing, hierarchy, model name, token counts, status) is small, uniform, and queried constantly — it must be complete and fast. Span payloads (prompts, completions, tool inputs/outputs) are kilobytes to megabytes, sensitive, subject to per-customer retention and redaction, and read rarely — only when a human opens a specific trace. Storing them together would let the rare, heavy, sensitive thing wreck the common, light, fast thing. So the first decision is: split metadata from payloads at the front door, and never rejoin them except at read time.

Second, one stream of data is money. Token counts feed customer bills, so that path gets stronger guarantees (dedup, exactly-once aggregation, reconciliation) than the debugging paths, and it is extracted before any sampling can touch it. Billing is never sampled; debugging payloads are. I'll repeat that where it applies, because it's the load-bearing principle.

Requirements

Functional, in priority order:

  1. Full trace of a run: nested spans for model calls, tool calls, retrievals, sub-agents, linked to the app's logs and metrics.
  2. Live tail: spans visible within 2 seconds of being emitted.
  3. Usage and cost per run, per customer, per model, per provider — accurate enough to invoice from.
  4. Search and analytics: error rates, latency percentiles, cost trends, routing behavior across providers, filterable by app, model, prompt version, time.
  5. Quality: eval scores and human/LLM-judge feedback attached to traces, trendable over time.
  6. Per-customer retention and redaction policies on payloads, including "never store content."

Non-functional: hard tenant isolation (this is a multi-tenant platform product — one leak ends it), ingest availability above query availability (data you never captured is gone forever; a query you can't run for ten minutes is an annoyance), and cost control, because naively storing every prompt is a petabyte-a-month habit.

Explicitly out of scope in 60 minutes: the SDKs (given), the model gateway itself (I'll assume one exists and emits telemetry), alerting/notification plumbing, and the billing invoice system downstream of the usage ledger.

Assumptions and scale math

All estimates below are assumptions for sizing, not measurements.

Architecture

The shape: a stateless ingest tier authenticates, redacts, and splits each span into a small metadata envelope and a payload blob. Everything flows through a durable, partitioned event log (Kafka) — that's the spine. Three consumers hang off it: a live-tail router for the 2-second path, a loader into a columnar OLAP store (ClickHouse) for search and analytics, and a stateful stream processor (Flink) that produces the billing-grade usage ledger. Payloads go to an object store (S3) and are fetched only when someone opens a trace.

flowchart LR
    SDK[Customer apps<br/>OTLP + GenAI semconv] --> GW[Ingest gateway<br/>authn, rate limits,<br/>redaction, split]
    MG[Platform model gateway<br/>server-side spans] --> GW

    GW -->|payload blobs| S3[(Object store: S3<br/>per-tenant prefix + KMS)]
    GW -->|metadata + payload refs| K[/Kafka<br/>spans topic, keyed by trace_id/]
    GW -->|usage events| KU[/Kafka<br/>usage topic/]

    K --> LT[Live-tail router] --> WS[WebSocket gateways] --> DEV[Developer<br/>watching a run]
    K --> CHL[Batch loader] --> CH[(ClickHouse<br/>spans, runs, scores,<br/>routing rollups)]
    KU --> FL[Flink<br/>dedup + windowed aggregation] --> PG[(Postgres<br/>usage ledger)]
    K --> TS[Tail sampler<br/>payload keep/drop] --> S3

    CH --> API[Query API / UI]
    S3 --> API
    PG --> API
    EV[Eval runners<br/>online + offline] --> CH
    K --> EV

Why these picks, and what I rejected:

Trace data model

A run is a trace. The root span is the agent invocation; everything nests under it via parent_span_id. Ten levels of sub-agents is just ten levels of parentage — OTel handles arbitrary depth natively, which is exactly why we take the OTel GenAI semantic conventions as our schema rather than inventing one. Instrumented apps interoperate with the whole OTel ecosystem, and we don't maintain a bespoke SDK contract.

Mapping to the GenAI conventions:

Two structural cases need more than parent/child:

The ClickHouse spans table, abbreviated:

CREATE TABLE spans (
  tenant_id      UUID,
  app_id         UUID,
  trace_id       FixedString(16),
  span_id        FixedString(8),
  parent_span_id FixedString(8),
  start_time     DateTime64(6),
  duration_ns    UInt64,
  operation      LowCardinality(String),   -- invoke_agent | chat | execute_tool | ...
  status         LowCardinality(String),
  provider       LowCardinality(String),
  request_model  LowCardinality(String),
  response_model LowCardinality(String),
  input_tokens   UInt64,
  output_tokens  UInt64,
  cost_micros    UInt64,                   -- computed at ingest from price book
  prompt_name    LowCardinality(String),
  prompt_version LowCardinality(String),
  payload_ref    String,                   -- '' if sampled out or policy says no-store
  attrs          Map(String, String),
  retention_ttl  DateTime,                 -- computed at insert from tenant policy
  ingest_time    DateTime64(6)
) ENGINE = ReplacingMergeTree(ingest_time)
ORDER BY (tenant_id, app_id, start_time, trace_id, span_id)
PARTITION BY toDate(start_time)
TTL retention_ttl

tenant_id leads every sort key in every table — that's isolation policy expressed as physical layout. A runs table (one row per root span, denormalized with total tokens, cost, status, depth, span count) is maintained by the loader so run-list screens don't aggregate 4B rows; it's the same trick as a covering index.

Payloads: separate, redacted, encrypted, deletable

The gateway splits every span. Metadata (~1KB) goes to Kafka. Content goes to S3 as one object per span, keyed tenant_id/app_id/date/trace_id/span_id, and the span carries only the payload_ref.

Order of operations at the gateway matters:

  1. Authenticate the ingest token; resolve tenant_id server-side. Tenant identity is never read from the payload — a bug or a malicious client can't write into another tenant.
  2. Redact, per the tenant's policy, before anything is durably written. You cannot unwrite a social security number from a Kafka topic with 7-day retention. Policies range from "store everything" through field- and pattern-level redaction (regex + lightweight PII detectors for emails, card numbers, keys) to "no-content mode," where content is dropped at the gateway and only metadata survives. Policies are versioned; each payload records which policy version scrubbed it.
  3. Split and write. Payload to S3, envelope to Kafka, usage event to the usage topic.

Encryption is envelope-style: per-tenant data keys in a KMS, rotated, wrapping per-object keys. That buys crypto-erasure — "delete this tenant" or "retention expired for this date-partition" is a key deletion plus an async S3 lifecycle sweep, not a synchronous scan of 600TB. Retention itself is boring on purpose: S3 lifecycle rules per tenant prefix for payloads, the retention_ttl column for metadata rows (customers can keep metadata 90 days and payloads 7 — a common and sensible split). Targeted GDPR deletions (one end-user's data) run as a manifest-driven job: find trace_ids via ClickHouse, delete objects, issue a ClickHouse mutation. Slow is fine; provable is required, so the job writes an audit record.

Why not store payloads inline in ClickHouse? Megabyte values poison merge performance, make every retention change a table rewrite, and force the analytics store to inherit the strictest compliance posture of any byte in it. Rejected early and firmly.

The live path: 2 seconds

Here's where a single-path design dies. ClickHouse wants big batched inserts — loaders buffer 5–15 seconds for insert efficiency — so "poll the database" blows the 2-second budget before the query even runs. The live path therefore reads Kafka directly and never touches the OLAP store.

A live-tail router (a small consumer-group service) reads the spans topic continuously. It keeps the set of active subscriptions — (tenant, app, optional trace_id filter) — in an in-memory cache backed by a fast shared store (Redis), updated when a developer opens or closes a live view. For each span it checks the subscription set (a hash lookup; the firehose is ~200K spans/sec at peak but active subscriptions number in the thousands, so almost everything is dropped immediately) and forwards matches to the WebSocket gateway holding that session. Payloads aren't pushed; the UI shows the span skeleton instantly and lazily fetches content from S3 when the developer clicks a span — content is usually already written by then, and a 300ms lazy fetch of a prompt is imperceptible next to a 2-second span-visibility SLO.

Latency budget, roughly: SDK batch/export ~500ms (SDK exporters default to batching; we recommend a 200–500ms schedule for instrumented agents), gateway + Kafka ~100ms, router ~50ms, WebSocket push ~50ms. Comfortably under 2s with room for a slow network. The SLO's biggest enemy is the customer's own SDK batching config, so the docs say so.

The UI assembles the tree client-side from parent_span_id as spans stream in, tolerating out-of-order arrival (children before parents render as pending). This is also the honest answer to "incomplete traces" in the live view: you're watching a run that hasn't finished; the tree fills in.

I rejected a "stream everything into the browser's region of interest via ClickHouse live views" design — it couples the SLO to insert batching — and rejected per-session Kafka consumers (thousands of consumers re-reading the firehose is quadratic waste; one router tier that filters once is linear).

The analytical path

Loaders consume the spans topic and batch-insert into ClickHouse. Materialized views maintain the rollups that power dashboards:

Replays and redeliveries make duplicate rows inevitable; ReplacingMergeTree keyed on the sort key plus FINAL-free query patterns (dedup-tolerant aggregates, or argMax by ingest_time) keep the analytics honest enough. "Honest enough" is fine here precisely because billing doesn't come from this store.

Trace-by-ID reads (the trace detail page) use a bloom-filter skip index on trace_id. It's a scan-reduction trick, not a B-tree, and at one date-partition granularity it's plenty: open-trace latency of a few hundred ms.

Billing-grade usage accounting

This is the path an interviewer should push hardest on, because "accurate enough to bill from" and "distributed telemetry pipeline" are natural enemies. Telemetry is at-least-once, late, and duplicated; invoices must be exactly-once and final.

Design principles:

Bill on spans, not traces. Each model-call span carries its own token usage. A run whose root span never arrives (crashed agent, dropped export) still has every completed model call accounted for — the money doesn't depend on trace completeness. Incomplete traces are a UX problem, not a revenue problem, and that's by construction.

Extract before sampling. The gateway emits a compact usage event — (tenant, app, trace_id, span_id, response_model, provider, input_tokens, output_tokens, gen_ai.response.id, end_time) — onto its own Kafka topic for every model-call span, before any retention or sampling decision. Never sampled; billing is never sampled.

Dedup twice. Flink keys state by (tenant, trace_id, span_id) and drops duplicates from SDK retries and gateway redelivery, with state TTL of ~48h. Second layer: gen_ai.response.id (the provider's response ID) catches the nastier case of double instrumentation — two spans, different span_ids, same underlying API call. Same response ID within a window → count once, flag the app for a "you're double-counting" diagnostic.

Windows with declared lateness. Usage aggregates per (tenant, model, provider, hour) close with 48h allowed lateness — agents can run long, exporters can retry through outages. On window close, Flink writes idempotently (upsert keyed by window + dimensions) to the Postgres ledger. Spans arriving after 48h don't vanish: they land in a corrections stream, applied to the ledger as dated adjustment rows before the monthly invoice cut. An invoice line is a sum over ledger rows, so late data becomes an explicit, auditable adjustment rather than a silently shifting number.

Reconcile independently. Nightly, a ClickHouse job recounts usage from raw spans (dedup via argMax) and diffs against the ledger. Drift beyond a small threshold (I'd start at 0.1%) pages someone. Two pipelines computing money from the same source and disagreeing is exactly the alarm you want before a customer's finance team finds it. Where model traffic goes through the platform's own model gateway, the gateway's server-side usage records are a third, ground-truth check — the gateway saw the actual provider response, no SDK involved.

Price at ingest, reprice at invoice. cost_micros on spans uses the price book current at ingest — good for dashboards. The invoice recomputes from token counts and the price book effective at usage time, so a price-book correction doesn't require touching 4B rows.

Sampling: what we keep, what we drop

At this scale the question isn't whether to sample, it's what's sacred. The sacred list:

Everything else — payload content for healthy, unremarkable runs — is where the 20TB/day gets cut. Mechanism: a tail sampler consumes the spans topic and decides per trace, not per span (a half-payloaded trace is worse than none). Decisions need the trace to end, so payloads are staged in a short-TTL tier (S3 staging prefix, or an S3 bucket with 24h lifecycle) and either promoted to retained storage or aged out. Rules, per tenant with defaults: keep if error anywhere in trace, keep if duration > p99 for that app, keep if flagged/scored, else keep with probability p (default 10%) by consistent hash of trace_id — consistent hashing means the keep decision is deterministic and trace-scoped, so cross-service spans of one trace agree without coordination. Customers can turn p to 100% and pay for it, or 0% and keep only the sacred list; the knob is theirs because the payloads and the compliance exposure are theirs.

Head sampling (decide at span 1) was rejected as the primary mechanism because the interesting traces — errors, slow ones — aren't identifiable at span 1. It survives only as the consistent-hash fallback rule, which happens to be head-decidable.

Model-routing visibility

Customers route across providers with fallbacks, and they need to see it. The data model already carries it: every model-call span has provider, requested vs served model, latency, error.type. Fallbacks are modeled per the GenAI conventions' spirit: a logical route_model parent span with one child span per attempt, each attempt marked retry.attempt=n and its outcome. The dashboard is then just queries on routing_stats: error rate by provider over time, fallback frequency, p95 by provider and model, "provider X started 429ing at 14:02 and traffic shifted to Y at 14:03."

Two sources of this data, and they disagree in a useful way. SDK-emitted spans show what the app experienced (including its own timeout policy); the platform model gateway (where customers use it) emits server-side spans showing what the provider actually did. Both are ingested; the gateway's are marked as such. When a customer says "your gateway is slow," the pair settles it.

Tenant isolation

Layers, because any single mechanism fails eventually:

  1. Identity at the edge. tenant_id comes from the authenticated ingest token or the session, resolved server-side. No client-supplied tenant field is ever trusted, on write or read.
  2. Physical layout. tenant_id is the first sort-key column in every ClickHouse table and the first path segment of every S3 key. Cross-tenant reads aren't just forbidden — they're not how the data is arranged.
  3. Query mediation. No customer query touches ClickHouse directly. The query API compiles a restricted filter grammar into SQL and injects the tenant predicate; ClickHouse row policies enforce the same predicate as defense in depth, per-service credentials scoped so even an API bug can't select across tenants.
  4. Crypto separation. Per-tenant KMS keys on payloads. A leaked object is ciphertext without the tenant's key grant.
  5. Noisy neighbors. Per-tenant ingest rate limits and quotas at the gateway (with clear 429s, not silent drops), and query resource groups in ClickHouse so one tenant's monster scan can't starve the rest. The largest customers can be pinned to dedicated ClickHouse shards — same schema, separate blast radius — which is also the answer when a regulated customer demands physical separation and will pay for it.

Quality: evals and scores on traces

Scores are their own records, not span mutations — spans are immutable facts about what happened; opinions about quality arrive later, from multiple sources, and change.

CREATE TABLE scores (
  tenant_id UUID, app_id UUID,
  trace_id FixedString(16), span_id FixedString(8),  -- span_id optional
  name LowCardinality(String),       -- 'helpfulness', 'thumbs', 'groundedness'
  value Float64, label String,
  source LowCardinality(String),     -- human | llm_judge | code | end_user
  eval_run_id UUID, scored_at DateTime64(3)
) ENGINE = MergeTree ORDER BY (tenant_id, app_id, scored_at, trace_id)

Three producers: online scorers (customer-configured LLM-judge or code checks consuming the spans topic, scoring a sample of fresh runs — their payload reads hit the staging tier before sampling ages it out, which is another reason staging holds 24h); offline eval runs over historical retained traces, each batch stamped with eval_run_id so "prompt v12 vs v13" is a two-run comparison; and the POST /v1/scores API for human feedback and end-user thumbs, correlated by trace_id the customer's app already holds. Quality trends are then ordinary ClickHouse group-bys — mean helpfulness by prompt_version by week — and score values feed back into sampling (keep payloads for low-scoring traces, per rule above). The link between "quality dipped" and "here are the actual bad transcripts" is the product moment this whole design exists for.

API surface

Failure modes

Evolution and honest tradeoffs

Day one I'd ship: gateway with redaction and the metadata/payload split, Kafka, ClickHouse with the spans/runs/scores tables, S3 payloads, live-tail router, and the Flink usage pipeline with reconciliation. That's already a real product.

What I'd add as it grows: cold-tier span metadata to S3-backed ClickHouse storage past 30 days; cross-region query federation when customers deploy multi-region; per-tenant BYOK for payloads (the enterprise ask that closes deals); streaming-token spans for live "the model is typing" views, which the GenAI conventions are still settling on.

Tradeoffs I'm consciously making: two stores for money (Flink→Postgres and ClickHouse) is more machinery than one, and the reconciliation job is a standing cost — I take it because a single path has no way to notice it's wrong. 100% metadata retention is a bet that its compressed cost stays trivial relative to its debugging value; if span counts grow 10×, the pressure valve is per-tenant metadata sampling of successful runs, and the sacred list above already defines what never goes. Tail sampling with a 24h staging tier doubles some payload writes; the alternative — head sampling — throws away exactly the traces people need. And betting on OTel GenAI conventions while they're still stabilizing means occasional schema churn absorbed in the gateway's normalization layer; the alternative, a proprietary format, trades that churn for a lifetime of SDK lock-in maintenance, and it's not close.

The product's one job, restated: when an agent misbehaves at 2 a.m., the developer opens the run, sees every step, reads the exact prompt that went wrong, sees what it cost and which model served it — and the customer's finance team and their compliance team never had a reason to call. Billing is never sampled, payloads never outlive their policy, and tenants never see each other. Everything above is in service of those three sentences.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): How can SDK-reported token counts be billing-grade when customers can omit, duplicate, forge, or incorrectly estimate them, provider response IDs may be absent, and only some traffic crosses your model gateway—what authoritative record determines the invoice during disagreement?

This lands, and the fix is to stop pretending all usage records are equal. Revision: every usage event gets a provenance class. Class A is gateway-witnessed — traffic through the platform model gateway, where we saw the provider's actual response including its usage block; that record is authoritative and is the only class an invoice line can sum. Class B is SDK-attested — token counts reported by customer instrumentation for traffic that went direct to a provider. Class B never reaches an invoice as a pass-through charge, because we were never in that money path: the customer pays the provider directly, and we'd be billing on numbers a customer can forge or fat-finger. Class B shows on dashboards and cost reports with an "attested" badge, and it's still useful — it's how the customer sees their own direct spend next to gateway spend.

During disagreement, the authority order is: gateway server-side record > provider response usage echoed with a gen_ai.response.id we can tie to a gateway record > bare SDK claim. Where a customer disputes a gateway-witnessed line, the gateway record includes the provider request id, so we can reconcile against the provider's own invoice. The design already had the gateway as a "third check"; the change is promoting it from check to sole billing source, and demoting SDK-only usage from billable to attested-display.

Design change: usage events carry a provenance class (gateway-witnessed vs SDK-attested); invoices sum only gateway-witnessed rows; SDK-attested usage is display-only and badged.

Q2 (Codex): The gateway independently writes a payload to S3, metadata to one Kafka topic, and usage to another; what concrete atomicity or recovery protocol prevents every partial outcome, especially "provider call occurred but usage event was lost," when Kafka is unavailable and the gateway's finite local disk or process fails?

Fair — the design listed partial outcomes but not the protocol. Here it is. The invariant: the gateway does not ack the OTLP export until the span's metadata and its usage event are durably written (Kafka acks=all to both topics, or, when Kafka is degraded, appended to the gateway's local WAL with fsync). No ack means the SDK's OTLP exporter retries with backoff — that's the standard exporter contract — so any pre-ack crash resolves as at-least-once redelivery, which the downstream dedup (Flink keyed state, ClickHouse insert dedup) already absorbs. The WAL is bounded; when it fills, the gateway returns 429/503 and sheds in the stated priority order. So the failure mode when Kafka is down and disk is full is "customer SDK buffers and retries," not "money silently lost." (Round 2 pushes on the WAL's single-host durability — see Q7, which revises the billing half of this.)

The S3 payload write is deliberately outside the ack path — asynchronous with a redrive queue — because every partial state involving it is recoverable: payload-without-metadata is an orphan object that the 24h staging TTL deletes; metadata-without-payload is a payload_ref that 404s, which the UI already handles. And the specific case named — provider call happened, usage event lost — has a second net beyond the ack protocol: token counts live in the span metadata too, so the nightly ClickHouse recount recomputes usage from spans and emits ledger adjustment rows for anything the usage topic missed. The usage topic is a low-latency projection of the spans topic, not the only place the money exists.

Design change: OTLP ack gated on durable write of metadata + usage (Kafka acks=all or fsync'd local WAL); S3 payload write explicitly async and outside the ack; enumerated partial states with their recovery path.

Q3 (Codex): How do your ClickHouse rollups remain correct under replay when ReplacingMergeTree deduplicates only during asynchronous merges and materialized views have already irreversibly double-counted requests, errors, tokens, and latency samples?

Conceded outright — this is a real bug in the design as written. Incremental MVs fire on insert, before any merge-time dedup, so a replayed block double-counts every rollup even though the base table eventually converges. Two-part fix. First, make loader inserts idempotent at the block level: the loader forms insert blocks deterministically from Kafka (topic, partition, offset-range) and sets insert_deduplication_token from that range, so a redelivered block is dropped by ClickHouse's insert dedup and the MVs never see it twice. That covers the common replay case — consumer restart, redelivery — for free.

Second, stop trusting incremental MVs for anything that counts. routing_stats and cost_daily become periodically refreshed rollups: a scheduled job recomputes them from the deduped base table (argMax by ingest_time) for the trailing window, and any operational replay that bypasses block identity (manual offset reset, disaster recovery) triggers a rebuild of the touched partitions as part of the runbook. Dashboards tolerate a rollup that's minutes stale; they don't tolerate one that's silently 2x. Billing was never in this store, so the blast radius was always dashboards — but "the error dashboard shows double the errors after every replay" is still a product bug, and now it's closed.

Design change: deterministic insert blocks + insert_deduplication_token keyed on Kafka offsets; routing_stats/cost_daily converted from incremental MVs to scheduled recomputation over the deduped table, with partition rebuild in the replay runbook.

Q4 (Codex): Once asynchronous sub-agents become separate traces connected only by span links, what stable logical-run identifier and completion protocol let you assemble the full run, prevent cycles or double attribution, and charge all descendant model calls to the originating run?

The design leaned on span links for structure and quietly left "the run" without an identity once work crossed a trace boundary. Fix: mint an explicit run_id. The root agent invocation creates it; it propagates via W3C baggage to every descendant, including async work spawned into new traces; every span carries it as an attribute. Attribution then stops being a graph problem: charging descendant model calls to the originating run is a group-by on run_id, not a traversal over links. Cycles can't corrupt attribution because run_id is inherited at spawn time, not computed by walking the link graph — a span has exactly one run_id stamped before it's ever stored. Links remain what they were: UI navigation between the traces of one run. A depth counter rides along in baggage; past a sane bound (say 50 spawn hops) the gateway flags the run as runaway rather than refusing data.

Completion protocol: there deliberately isn't one. Runs with humans in the loop stay open for hours, so nothing in the system waits for "run complete." The runs rollup row updates as spans arrive and the UI shows an activity-based idle state ("no spans for 15m") — that's presentation, not a state machine. Billing never needed completion because it bills on spans; now cost-per-run doesn't need it either, because the run's cost is the running sum over its run_id, correct at every moment for the spans that have arrived.

Design change: explicit run_id minted at root, propagated via baggage across trace boundaries, stamped on every span; runs table keyed by run_id; per-run cost = group-by run_id; links demoted to navigation only.

Q5 (Codex): How does tail sampling make a final trace-wide keep decision when spans can arrive hours late or never arrive, while live-tail pins and eval scores may arrive after deletion—and how do you "promote" tens of terabytes of staged S3 objects without copy amplification or lifecycle-rule explosion across 50,000 tenants?

Three sub-questions, three answers. Timing: the sampler decides per trace at an idle timeout (15 minutes without a new span), and persists the decision record. A late span that arrives after a "drop" decision re-opens it only if it changes the answer (an error). If that late error arrives inside the staging window, the payloads get promoted; if it arrives after staging TTL — concede — the payloads are gone, and I'm taking that loss explicitly: metadata is 100% retained, so the failure is still diagnosable (which step, which model, what tokens, what error type), just without transcript content. Traces whose first error signal arrives more than 24 hours late are a corner I'll trade against holding 20TB/day hot for longer. Pins and eval scores don't hit this in practice by construction: a live-tail pin happens while a human is watching (inside the window by definition), online scorers read within the 24h staging window (stated in the design), and offline evals run only over already-retained traces. A pin request against an aged-out trace returns "content not retained" — honest, and rare.

Promotion mechanics: correct that copying tens of TB or minting per-tenant lifecycle rules both fail — S3 caps lifecycle rules per bucket around a thousand, so 50K per-tenant rules were never going to work and I withdraw that mechanism. Revision: payloads are written once; "promote" is a PUT-object-tagging call, no data movement, and the bucket has a small global rule set. Per-tenant retention (the 7-day vs 90-day policies) is enforced not by lifecycle rules but by a sweeper: a daily job walks the S3 Inventory manifest, joins object age against the tenant policy table, and issues batched deletes — plus per-tenant crypto-erasure for the whole-tenant case, which was already in the design. Slow is fine; provable is required, and the sweeper writes audit records the same way the GDPR job does. (Round 2 pushes on the tag-filter semantics and the deadline guarantee — see Q8, which sharpens this.)

Design change: persisted per-trace decision records with error-triggered reopen inside staging TTL; promotion via object tagging (no copy); per-tenant S3 lifecycle rules withdrawn, replaced by an inventory-driven retention sweeper + crypto-erasure.

Q6 (Codex): What database-enforced mechanism actually isolates tenants when the query service uses shared ClickHouse credentials, given that leading with tenant_id in the sort key is only a performance layout and a row policy cannot derive the current tenant from an arbitrary request parameter?

Conceded as underspecified: as written, both "layers" — the API injecting a predicate and a row policy checking a value the API supplies — trust the same component, so a bug in the query compiler defeats both. The premise of the question is right and the fix is to split the trust. Revision: a small connection broker, separate from the query-compiling API, owns all ClickHouse credentials. The API never holds them. Per request, the broker validates the customer's session token itself (it does not take the API's word for tenant identity), opens a ClickHouse session, binds SET custom_tenant_id = '<uuid>', applies a settings profile whose constraints mark that setting read-only for the session, and only then hands the connection to the query layer. Row policies filter every table on tenant_id = getSetting('custom_tenant_id'). Now a compiler bug that emits WHERE 1=1 or forgets the predicate still returns only the bound tenant's rows, because the enforcement lives in a component that never parses customer queries — a few hundred lines that do nothing else, auditable in an afternoon.

That's the database-enforced layer for the long tail. On top of it, the pieces already in the design still count: dedicated shards for the largest tenants make cross-tenant reads physically impossible for the data that matters most commercially, and per-tenant KMS keys mean a metadata-layer breach still yields no payload content. Isolation isn't one mechanism; but the design previously had two mechanisms wearing one trust domain, and now it has two trust domains.

Design change: connection broker (independent trust domain, validates session tokens itself) binds tenant_id into the ClickHouse session as a constrained read-only setting; row policies enforce getSetting('custom_tenant_id'); query API never holds DB credentials.

Round 2 — pushback

Q7 (Codex): You ACK after an fsync to a gateway's local WAL during Kafka failure—but if that host dies before replay, both the span and usage event are permanently lost and ClickHouse has nothing to reconcile. What replicated durability mechanism makes that ACK billing-grade?

Right — an acked-then-lost event is unrecoverable by any downstream reconciliation, because the SDK won't resend and no store ever saw it. Single-host fsync was never billing-grade and I withdraw the claim that it was. But the Q1 revision changes what has to be billing-grade: invoices now sum only gateway-witnessed usage, and the model gateway is on the serving path — so the billing record should be written there, synchronously with proxying the provider call, to replicated storage (a transactional outbox in the gateway's regional Postgres, or Kafka with acks=all and no local-WAL fallback for this write class). If the model gateway cannot durably record usage, it degrades per the tenant's configured preference — fail the model call, or serve it and log a flagged best-effort record — but the default is that a served call has a replicated usage record before the response streams back. The telemetry pipeline's usage topic is then a projection for dashboards and reconciliation, and losing it loses no money.

For the telemetry path itself, the WAL fallback survives with an honest downgrade: it protects spans, where single-host loss during a simultaneous Kafka-outage-plus-host-death is an acceptable observability gap (bounded to that host's WAL, minutes of one gateway's traffic). If we ever need stronger-than-that telemetry durability in degraded mode, the mechanism is peer-replication — the gateway forwards WAL appends to one peer in another AZ and acks on 2-of-2 — but I wouldn't build that day one; multi-AZ Kafka with acks=all already makes full unavailability the rare case, and the money no longer rides on it.

Design change: billable usage recording moves onto the model gateway's serving path with replicated durability (transactional outbox, or acks=all Kafka with no WAL fallback for that write class); the local-WAL ack is downgraded to telemetry-only, its loss window stated and accepted.

Q8 (Codex): Your retention scheme depends on expiring objects "without" a retain tag, which S3 lifecycle filters cannot express, while a daily Inventory-driven sweeper is delayed and non-atomic. How do you guarantee each tenant's deletion deadline across billions of objects, especially after a retention-policy change?

Two concessions and a reframe. The tag mechanics as I stated them were backwards: S3 lifecycle filters match objects that have a tag, not objects that lack one. Invert it — every payload is written with class=staged, one global lifecycle rule expires class=staged objects after 1 day, and promotion rewrites the tag set to class=retained, which removes the object from the expiry rule's filter. Same one-rule, no-copy design, with filter semantics S3 actually implements.

The reframe: the deletion deadline is never guaranteed by object deletion at all — at billions of objects, no enumerate-and-delete process is atomic or on-time, and pretending otherwise is how you fail an audit. The guarantee is crypto-erasure, sharpened from per-tenant keys to per-(tenant, day) data keys: payloads for tenant T written on day D are encrypted under K(T,D), a wrapped data key in a KMS-backed key table — 50K tenants × 90 days is ~4.5M key rows, trivial. The retention deadline for (T,D) is enforced by deleting K(T,D) at D + retention(T): one atomic, auditable, precisely scheduled operation per tenant-day, independent of object count. A retention-policy change is an update to the scheduled deletion dates on existing key rows — thousands of rows, not billions of objects — and shortening retention takes effect at the next key-deletion tick, with the audit record saying so. The Inventory-driven sweeper is thereby demoted from compliance mechanism to storage reclamation: it lazily deletes ciphertext that is already unreadable, and its being a day late costs dollars, not compliance. Deletion-deadline SLA lives in the key scheduler, which is small enough to test exhaustively and alarms if any scheduled deletion misses its tick.

Design change: tag polarity inverted (class=staged written at ingest, promotion re-tags to class=retained, one global expiry rule); per-(tenant, day) data keys with scheduled key deletion as the compliance mechanism for retention deadlines; sweeper demoted to cost reclamation.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The ClickHouse bet is the industry default, and the migration stories all rhyme. LangSmith started on Postgres and moved to ClickHouse when production tracing volume arrived; Langfuse did the same move for v3, and their infrastructure evolution post describes almost exactly this design's write path: events persisted to S3 first, references queued (Redis in their case, Kafka here), async workers batch-inserting into ReplacingMergeTree tables — updates turned into inserts because "every row update in ClickHouse is an immensely expensive operation." Their ordering key leads with project + time, the same isolation-as-physical-layout trick as tenant_id first in the sort key. Uber's log analytics platform is the non-LLM precedent at bigger scale: they left Elasticsearch because 80% of queries were aggregations, and a single ClickHouse node ingested ~300K logs/sec, about 10x an ES node. ClickHouse's own OTel traces guide reports 9–10x compression on span data — the same order-of-magnitude shrink my 4TB-to-400GB/day estimate leaned on — and notably concludes that a trace_id lookup materialized view is "an unnecessary optimization in most cases," which supports settling for a bloom-filter skip index.

One real divergence: this design keeps payloads out of the OLAP store entirely, while Langfuse stores multi-megabyte input/output in ClickHouse columns and relies on columnar layout to keep them off disk until queried, per their ClickHouse at agent scale post — plus a materialized view of truncated versions for dashboards. Their approach costs less machinery; mine buys per-tenant crypto-erasure and retention that never touches the analytics store. The same post is candid about the price of ClickHouse: substring search over payloads is a full-table scan, deletes are background mutations, and merge tuning needs active monitoring — all consistent with why payload retention here lives in S3 tags and key deletion, not table mutations.

The Kafka spine is also well-trodden, including the failure that motivates it. Helicone added Kafka after their process-each-log-inline design lost data during service interruptions and couldn't reprocess — exactly the "welds ingest availability to query-store availability" trap — and landed on batch consumers with idempotent Postgres upserts and ClickHouse versioned-merge dedup, a smaller cousin of the Flink/insert_deduplication_token treatment here. Honeycomb's Retriever, a custom Scuba-inspired columnar store fed from Kafka, is the proof that the shape (durable log → columnar consumers) predates LLM observability by a decade.

On tail sampling, the OTel Collector's tail sampling processor makes the same calls this design made under interview pressure, with different constants: a decision_wait timer instead of my 15-minute idle timeout, policies for status code/latency/probabilistic (my sacred list plus consistent-hash fallback), a decision cache so late spans inherit the verdict (my persisted decision records from Q5), and drop-beats-keep on policy conflict. Its hard deployment constraint — all spans of a trace MUST reach the same collector instance — is the thing the Kafka-keyed-by-trace_id design gets structurally for free. On redaction, the Collector's redaction processor (see Dash0's guide) encodes the same order-of-operations principle as my gateway step 2: sanitize in transit, before the backend stores anything, with allowlist-by-default as the fail-closed posture.

The most interesting recent datapoint pushes past this design. LangChain concluded ClickHouse itself wasn't the end state for agent traces and built SmithDB (May 2026): Rust on Apache DataFusion and the Vortex file format, trace data on object storage, a small Postgres metastore, and stateless ingest/query/compaction. Their stated reasons are the exact pressures the interview rounds surfaced — spans that stay open for hours, megabyte deeply-nested payloads, and update semantics that fight immutable-row stores; SmithDB models a run as "a sequence of events" rather than a finished row, which is the same conclusion as my Q4 answer that there deliberately is no completion protocol. I'd still ship ClickHouse day one (so did they, for three years), but SmithDB is a credible sketch of what this system's v3 looks like when agent traces stop resembling request/response traces at all.

Updates from post-training information

Further reading