Contents

Multi-Tenant Log Ingestion and Search

The short version: put a durable, partitioned event log (Kafka) between everything that produces logs and everything that consumes them, key it by stream so per-stream order is free, and fan out to three independent consumers — live tail, a columnar OLAP store (ClickHouse) for search with S3-backed cold storage, and a per-tenant forwarder. Every component that can drop a log line emits an auditable drop record into the same pipeline, so "were my logs dropped?" is a query, not a shrug. The rest of this document is the reasoning and the hard parts.

Assumptions

Stated up front since I can't ask:

Requirements, prioritized

  1. Never leak logs across tenants. This is the one non-negotiable.
  2. Degrade predictably under overload, with customer-visible drop accounting. The prompt calls this out and it drives more of the design than the happy path does.
  3. Live tail within 2s; search over 7–90 days by time and text.
  4. Per-plan retention, enforced.
  5. Forwarding to Datadog, S3, syslog.
  6. Cost sanity at 10–50 GB/s, because at this volume the storage bill is the design.

Estimates that shape the architecture

Architecture

flowchart LR
    subgraph Host
        A[Log agent<br/>per host, external]
    end
    subgraph Ingest
        GW[Ingest gateways<br/>stateless, mTLS, rate limits]
    end
    subgraph Buffer
        K[(Kafka<br/>logs topic, keyed by stream_id)]
        AU[(Kafka<br/>ingest-audit topic: drop records)]
    end
    subgraph Consumers
        T[Tail service<br/>partition-aligned nodes]
        W[Storage writers]
        F[Forwarders<br/>Datadog / S3 / syslog]
    end
    subgraph Storage
        CH[(ClickHouse<br/>hot: NVMe ~48h)]
        S3[(Object storage S3<br/>cold: up to 90d)]
    end
    subgraph Serving
        Q[Query service<br/>tenant scoping + quotas]
        D[Dashboard / API]
    end

    A -->|gRPC batches, per-stream seq| GW
    GW --> K
    GW -->|drops| AU
    A -->|agent drop counts| GW
    K --> T
    K --> W
    K --> F
    F -->|delivery failures| AU
    AU --> W
    W --> CH
    CH <--> S3
    T -->|WebSocket| D
    Q --> CH
    D --> Q

Ingest path

Agent contract. The host agent is external, but I get to define its output: gRPC batches of {instance_id, stream (stdout|stderr), records: [{seq, ts, line}], drops: [{window, count, reason}]}. Two details matter. First, seq is a per-stream monotonically increasing counter assigned by the agent — this is what makes ordering and loss detection possible everywhere downstream. Second, the agent reports its own drops in-band: it buffers to host disk (bounded, say 1 GB per host, a few minutes of typical output), and when that fills it drops oldest-first and increments a counter it ships with the next batch. The pipeline can't count drops it never saw; the sequence numbers and self-reported counts close that hole.

Trust. The agent runs on our hosts, so identity comes from host mTLS certificates issued by the platform, and the gateway maps instance_id → tenant_id/service_id via a control-plane lookup (cached, pushed on deploy). The agent never asserts its own tenant. This is the first isolation boundary: a compromised container can spam its own stream, but it can't claim to be someone else's, because tenant attribution happens outside the container's reach.

Gateway. Stateless, behind a regional load balancer, autoscaled. It authenticates, attributes tenant/service, enforces per-stream and per-tenant token buckets (below), batches, and produces to Kafka with key = stream_id. Its in-memory buffer is small and bounded — order of 256 MB per node — because the gateway's job is to push backpressure outward, not to hide it. If Kafka is slow, the gateway returns 429 with Retry-After and the agent falls back to its disk buffer. Buffering belongs in exactly two places, both explicitly bounded: host disk at the edge, Kafka in the core. Everything else stays thin so that failure means backpressure, not silent accumulation in some queue nobody sized.

The event log. Kafka, acks=all, replication factor 3, min.insync.replicas=2 — once the gateway acks a batch, it survives a broker loss. Sizing at peak: 50 GB/s × 3 replicas = 150 GB/s of disk write. At 300 MB/s sustained per NVMe broker with headroom, that's ~500 brokers — a big cluster but a boring one, and it's the component I most want boring. Retention on the logs topic is the system's shock absorber and its bound: size it for ~4 hours of average traffic (145 TB per replica). That number is a contract: any consumer that lags more than 4 hours starts losing data, so 4 hours is the repair budget for ClickHouse or the forwarders, and we alarm at 25% of it.

Partitions: capping a partition at ~15 MB/s peak gives ~3,500 partitions minimum; provision 8,192 for headroom and to keep the stream→partition hash stable. stream_id hashing spreads a tenant's streams across partitions, which is what we want — a tenant's aggregate volume is diffuse, while each stream stays totally ordered on one partition.

Why Kafka and not alternatives: a managed queue (Kinesis) has per-shard throughput quotas and pricing that get ugly at 50 GB/s; Pulsar's tiered storage is attractive but I'll take the ecosystem and operational muscle memory of Kafka; an object-storage-backed log (WarpStream-style) would cut the broker bill dramatically but adds ~500ms–1s of produce latency, which spends a quarter of the 2-second tail budget before we've done anything — noted as a cost evolution, rejected for v1.

Ordering

The guarantee, stated exactly: total order within a single stream, best-effort merge across streams. One instance's stdout is ordered by agent seq — same stream_id → same partition → same consumer, and the storage sort key includes seq as a tiebreaker. Across streams of one service we merge by timestamp at query time, which is honest about clock skew rather than pretending a distributed total order exists. Customers tailing one crashing instance get exact order, which is the case that matters; nobody can meaningfully order two instances' interleaved lines at millisecond granularity anyway, and we document that.

At-least-once means duplicates on gateway retry. We don't chase exactly-once: the writer dedupes best-effort on (stream_id, seq) within its insert batches, and a rare surviving duplicate log line is a cosmetic bug, not a correctness one.

Backpressure and drop accounting

This is where "degrade predictably" gets teeth. The chain, edge to core:

  1. Per-stream token bucket at the gateway, plan-based — say 2 MB/s sustained / 10 MB/s burst on paid, 500 KB/s / 2 MB/s on free. The 100 MB/s abuser hits this wall immediately and only its own stream is cut; nothing shared is consumed. Overage is dropped at the gateway (not queued — queuing an abuser just moves the pain), and counted.
  2. Per-tenant aggregate bucket above that, so a tenant with 10,000 instances can't multiply the per-stream limit into a platform problem.
  3. Gateway → Kafka backpressure: bounded gateway buffer, then 429 to the agent.
  4. Agent disk buffer, bounded, then oldest-first drops, self-reported.
  5. Global shedding, for the case where legitimate aggregate demand exceeds Kafka capacity. Defined levels, published in the docs so "predictable" is literal:
    • L1 (>70% of provisioned produce capacity): burst allowances tighten to sustained rates.
    • L2 (>85%): free-tier streams sampled at 50%, marked as sampled.
    • L3 (>95%): all tenants hard-capped at plan sustained rate. Paid ingestion at its committed rate is the last thing touched, and drop records are never shed — they're a few bytes per stream per window on a separate topic and they're the trust budget.

Accounting. Every dropping component — agent, gateway limiter, global shedder, forwarder — emits a drop record {tenant, service, stream, window_start, window_end, count, reason, component} to a small high-priority ingest-audit topic. The storage writer additionally detects seq gaps it can't attribute (a lost batch, a bug) and emits reason=unaccounted — the sequence numbers make the audit end-to-end, not just a sum of self-reports. Drop records land in a metering table and surface three ways: a synthetic marker line inline in tail and search results ("⚠ 12,431 lines dropped 14:02–14:03, rate limit"), a GET /v1/logs/gaps API, and a banner on the dashboard. A customer should never have to wonder whether an absent log line means the event didn't happen or we lost it.

Choice. ClickHouse, sharded, with its native tiered disks: recent ~48 hours on local NVMe, older parts moved to S3. Sort key (tenant_id, service_id, toStartOfHour(ts), ts, seq), partitioned by day, compressed with zstd.

What I rejected and why:

Indexing strategy. No inverted index anywhere. The sort key makes every query a contiguous range read for one tenant/service/time window. On top of that, two data-skipping indexes on the message column: a token bloom filter (tokenbf_v1) for word matches and an ngram bloom for substring matches. Blooms are a few KB per granule regardless of content cardinality, so a tenant logging millions of unique request IDs makes their queries selective, not our index big — this is the property that makes high-cardinality content a non-event on the write path. Typical query: prune partitions by day, prune granules by sort key, prune again by bloom, then brute-force scan what remains with SIMD. Hot-tier searches come off NVMe; a 90-day needle-in-haystack query reads cold parts from S3 and is allowed to take tens of seconds — we set that expectation in the UI rather than paying to make the rare case fast.

High-cardinality abuse, structured edition. If we later index JSON attributes as columns, a tenant emitting unbounded attribute keys (user_12345_latency) can explode the schema. Cap indexed keys per tenant (~128, first-seen wins); overflow keys stay inside the raw message — still findable by text search, just not as typed fields. Cardinality of values is already handled by the bloom design; cardinality of keys is handled by the cap. Stream churn (millions of short-lived instance IDs) is also fine: instance_id is a plain column, not part of any index structure that grows per distinct value.

Schema sketch:

CREATE TABLE logs (
    tenant_id       UInt64,
    service_id      UInt64,
    instance_id     String,
    stream          Enum('stdout','stderr'),
    ts              DateTime64(3),
    seq             UInt64,
    level           LowCardinality(String),
    message         String CODEC(ZSTD(3)),
    retention_days  UInt16,
    INDEX msg_tokens message TYPE tokenbf_v1(30720, 3, 0) GRANULARITY 4,
    INDEX msg_ngrams message TYPE ngrambf_v1(4, 30720, 3, 0) GRANULARITY 4
) ENGINE = MergeTree
PARTITION BY toDate(ts)
ORDER BY (tenant_id, service_id, toStartOfHour(ts), ts, seq)
TTL ts + toIntervalDay(retention_days);

Retention per plan. retention_days is stamped on every row at ingest from the tenant's plan, and the row-level TTL enforces it — no per-tenant tables, no cron jobs walking tenants. Two edges: TTL merges are lazy, so the query service also clamps every query's time range to the tenant's current plan window — expired-but-unpurged rows are invisible immediately, and a downgraded tenant loses access to day 8+ the moment the plan changes, ahead of physical deletion. Upgrades apply going forward; we can't resurrect purged data and say so. Physical deletes are cheap because day partitions age out whole.

Hot/cold mechanics. A TTL ts + INTERVAL 48 HOUR TO VOLUME 's3' move policy keeps NVMe holding ~170 TB compressed (2 days × 85 TB) across the cluster — with 2× replication, roughly 100 shards of 4 TB NVMe each, which is a manageable fleet. Cold parts on S3 are single-copy plus S3's own durability.

Live tail

Requirement: line visible in the dashboard within 2s. Budget: agent flush ≤500ms (flush on 500ms timer or 1 MB, whichever first) + gateway batch ≤200ms + Kafka produce/replicate ~50ms + tail consume and push ~100ms. Comfortable.

Implementation: tail nodes are partition-aligned — each node consumes a static slice of the 8,192 partitions from the head (no history). Because stream→partition is a deterministic hash, a router can compute exactly which partitions carry a given service's streams: the client opens a WebSocket to the router, the router looks up the service's live instances from the control plane, hashes their stream_ids, and bridges to the handful of tail nodes owning those partitions. Each tail node keeps a subscription map {stream_id → sessions} and filters as it consumes; unsubscribed lines cost one hash lookup. No node ever scans traffic it doesn't own, so the tail fleet scales linearly with ingest, and a tenant's tail session touches only their own decoded lines — filtering happens server-side on trusted stream identity, never in the client.

A session is capped at ~1,000 lines/s; past that we sample and show "tail is sampling — full logs in search." A tail node crash means affected sessions reconnect and resume from the partition head; the missed seconds are already in search. Tail is deliberately the least durable consumer, and that's fine because the event log made durability someone else's job.

Query API and tenant isolation on the read path

POST /v1/logs/search   {service_ids, start, end, text?, level?, stream?, limit, cursor}
GET  /v1/logs/tail     (WebSocket; service_id, filters)
GET  /v1/logs/gaps     ?start=&end=          → drop records for your streams
PUT  /v1/log-forwarding/{destination_id}     → Datadog | S3 | syslog config

The query service is the only thing holding ClickHouse credentials. tenant_id comes from the auth token and is injected server-side as the leading predicate — it is not a request parameter, so there is nothing to forget or spoof. ClickHouse row policies enforce the same predicate as a second, independent layer; a query-service bug should hit a wall, not a data set. We fuzz this boundary with canary tenants in CI.

Capacity isolation on reads mirrors the write side: per-tenant ClickHouse quotas — max memory per query, max execution time, max 2–3 concurrent searches — so one tenant's 90-day cold scan queues behind their own quota, not in front of everyone's dashboard. Cold-tier scans run in a separate, lower-priority workload class.

Forwarding

Forwarders are their own consumer group on the same logs topic, so a slow or dead Datadog endpoint can never back up ingest, tail, or search — it only grows forwarder lag, bounded by the same 4-hour Kafka retention. Per-destination delivery workers with retries and exponential backoff; a destination failing hard trips a circuit breaker, and once its lag budget is exhausted we drop for that destination only and emit drop records with reason=forwarding_destination_unavailable — the same visible accounting as everywhere else, plus an email. S3 forwarding writes batched, compressed objects to the customer's bucket; syslog gets TLS with a bounded in-flight window. We explicitly do not promise the forwarded copy is loss-free when the customer's endpoint isn't; we promise they'll know.

Failure modes

Cost, because 50 GB/s is a budget line

Rough monthly shape (estimates, not quotes): ~500 Kafka brokers plus ~200 ClickHouse and tail/gateway/forwarder nodes — order of 700–1,000 machines; ~2.5–3 PB in S3 at ~$21/TB-month ≈ $55–65k/month for cold storage, which is pleasantly small next to the compute. The levers, in order of impact: compression (the assumed 10× is the single biggest cost decision in the system — we'd validate it in week one); no inverted index (blooms are ~0.1% of data size); tiered storage (NVMe holds 2 days, not 90); plan-based ingest caps with paid overage rather than unlimited free volume; and free-tier sampling under overload. The obvious next lever — moving the Kafka layer itself onto object storage — is the one we deliberately deferred for tail latency, and it's where I'd look first if the broker fleet's bill outgrows its value.

Evolution

Week one is a thin slice: agent → gateway → Kafka → ClickHouse hot tier only, tail reading Kafka directly, drop accounting from day one (retrofitting trust is much harder than retrofitting features). Then S3 tiering and forwarding. Later: structured field extraction behind the key cap, dedicated ClickHouse subclusters for whale tenants who want them (isolation by physics), metrics-from-logs, and the object-storage-backed event log when the economics tip. The principle that shouldn't change as it evolves: bounded buffers in exactly two places, and every drop, anywhere, leaves a record the customer can see.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): Your "no silent loss" guarantee relies on a later sequence number revealing a gap, but how do you determine the exact loss if Kafka expires unread offsets and the stream never emits again, the host dies before reporting local drops, or the audit-topic write fails during gateway shedding?

The question lands: seq-gap detection needs a later record, and three of the ways we lose data don't produce one. Fixes, per case. Dead-end streams: the agent emits a per-stream checkpoint record {stream_id, max_seq} every 30s and a final {stream_id, closed_at_seq} on stream close; the gateway independently publishes its per-stream high-water mark to the audit topic on the same cadence. The storage writer reconciles "highest seq the gateway saw" against "highest seq that landed in ClickHouse" — a gap at end-of-stream is now detectable with no later data record. Kafka expiring unread offsets: the writer knows the broker's log-start-offset per partition; when it falls off the head it emits a per-partition drop record for the exact expired offset range and time window, and every stream hashing to that partition gets a window-level "possible loss 14:02–14:50" marker. We can't give per-stream counts for that case, so the gaps API returns three grades: counted (self-reported drops), bounded (offset-range expiry), unknown.

Host dies before flushing its buffer: I concede this one outright — lines the host never persisted are indistinguishable from lines never written, and no pipeline can count them. What we can do is emit an honest marker: the control plane knows the instance terminated uncleanly, so we publish "instance died at T; logs after last checkpoint (seq N, T−30s) may be missing." Audit-write failure at the gateway: drop records are tiny, so the gateway spools them to local disk and retries until acked; if the gateway itself dies with an unflushed spool, the seq/checkpoint reconciliation still detects the missing data as unaccounted — the audit record is attribution, the reconciliation is detection, and only attribution is lost.

Design change: periodic per-stream checkpoints from agent and gateway, unclean-termination markers, per-partition expiry records, gateway disk-spooling of audit records, and a three-grade (counted/bounded/unknown) gaps API.

Q2 (Codex): Where does a forwarder commit its Kafka offset when one partition contains logs for many destinations: before delivery, risking loss on worker failure, or after delivery, letting one dead customer endpoint block unrelated destinations—and what durable per-destination spool resolves this within the four-hour retention bound?

Concede — "forwarders are their own consumer group" as written has exactly the dilemma you describe: commit-after-delivery lets one dead Datadog endpoint block offset commits for every destination sharing the partition. The fix is two stages with a durable spool between them. Stage 1, a demux consumer group reads the logs topic, filters to streams with forwarding configured (a minority of volume — forwarding is opt-in), and produces to a forwarding topic keyed by destination_id; it commits the main-topic offset after the spool produce is acked, so the main topic's consumer lag never depends on any customer endpoint. Stage 2, per-destination delivery workers consume the spool and commit only after delivery confirmation — at-least-once to the destination, and a dead endpoint stalls only its own key's progress.

Head-of-line within one destination is the destination's own problem and I accept it. Across destinations sharing a spool partition there is still coupling; we bound it by circuit-breaking: records for a circuit-broken destination are counted as dropped (with the existing forwarding_destination_unavailable audit record) and the offset advances. That keeps the spool's 4-hour retention as the lag budget for slow-but-alive endpoints and turns dead endpoints into accounted drops instead of blocked partitions. A single hot destination exceeding one partition's throughput gets its key sharded by stream_id.

Design change: forwarding split into demux → per-destination spool topic (keyed by destination_id) → delivery workers; main-topic commit decoupled from delivery; circuit-broken destinations drop-and-account rather than block.

Q3 (Codex): How can ClickHouse row policies independently enforce tenant isolation when the query service uses a shared credential—what database-authenticated, immutable tenant identity drives the policy without trusting the same query construction or client-supplied setting whose bug the policy is supposed to contain?

Correct, and I concede the mechanism as written. ClickHouse row policies bind to a DB user; with one shared query-service credential, the policy can only key off something the query itself supplies — the same trust domain whose bug it was supposed to contain. Per-tenant DB users don't scale to the tenant count. The revision: make the second layer a different component, not a different DB feature fed by the same input. First, the DB grant for the query-service user is restricted to a parameterized view — SELECT ... FROM logs WHERE tenant_id = {tenant:UInt64} — with no direct SELECT on the underlying table. A query that forgets the tenant predicate is now impossible to execute, not merely forgettable. Second, token→tenant resolution moves out of the query builder into a small, separately deployed auth proxy whose only job is mapping the customer's auth token to a tenant_id and stamping it on the internal request; the query builder never parses tokens. Third, detection: an async auditor samples the ClickHouse query log and verifies every executed query's tenant parameter matches the session's authenticated tenant, alarming on any mismatch.

The honest residual: if the token→tenant mapping itself is wrong, every layer downstream is wrong together — that mapping is the real root of trust, and it's where the canary-tenant fuzzing and property tests concentrate. (Round 2 pushes on this answer; the stronger mechanism is under Q8.)

Design change: drop "row policies as independent layer"; replace with grant-restricted parameterized view (no raw table access), a separate token→tenant auth proxy, and query-log auditing.

Q4 (Codex): How is Kafka a four-hour repair buffer when it is sized for four hours at 10 GB/s but holds only about 48 minutes at 50 GB/s, and where is the capacity calculation for replication traffic plus three full-rate consumer groups, broker recovery, rebalancing, and operational headroom?

Concede the framing: I sized retention in bytes at average traffic and then advertised it as a time contract, and those diverge exactly when it matters — during a sustained peak, the "4-hour repair budget" is 48 minutes. Two changes. Retention gets sized in bytes for 4 hours at peak: 50 GB/s × 4h ≈ 720 TB per replica, ~2.2 PB across the cluster — about 4.3 TB of log segments per broker at 500 brokers, well within an NVMe broker's disk. And the alarm changes from "lag > 25% of retention" to projected time-to-loss: remaining retained bytes ahead of the slowest consumer divided by current ingest rate, which is the number the contract actually depends on.

On the capacity math the doc waved past, at peak per broker (500 brokers, leaders spread evenly): ~100 MB/s producer in, ~200 MB/s replication out to followers and ~200 MB/s replication in as a follower, so ~300 MB/s of disk append — that's where the 300 MB/s/broker figure came from, and it's disk-write-bound, not NIC-bound. The three consumer groups add 3 × 50 GB/s = 150 GB/s of cluster-wide reads, ~300 MB/s out per broker — but tail, writers, and forwarders all read at the head, which is page cache, not disk. The case that breaks this is a lagging consumer doing cold catch-up reads plus a broker re-replicating 4.3 TB after loss; both get throttled quotas (catch-up and re-replication each capped, re-replication at a rate that finishes in low hours), and the headroom for them is why brokers run at ~60% of NIC (aggregate ~800 MB/s against 25 Gbps) rather than sized to the happy path.

Design change: logs-topic retention resized to 4h of peak (~720 TB/replica); lag alarm becomes projected time-to-loss; explicit throttles on catch-up reads and re-replication.

Q5 (Codex): How does per-row TTL remain economical when daily partitions mix tenants with 7–90-day retention, preventing whole-partition deletion until the longest-lived rows expire and forcing repeated merges of enormous cold parts; and how are downgraded tenants' previously stamped 90-day rows physically removed?

Concede: with PARTITION BY toDate(ts) and per-row TTL, a day's partition can't drop until its longest-retention rows expire, so 7-day free-tier data is physically removed by TTL delete-merges — rewriting parts that have already moved to S3, which means reading and re-writing multi-TB cold parts to delete rows. That's the expensive way. The fix is to put the retention class in the partition key: PARTITION BY (retention_days, toDate(ts)). Retention classes are plan values — a handful (7, 30, 90, maybe two more) — so partition count multiplies by ~5, which MergeTree handles fine. Now every partition expires as a unit: retention enforcement is DROP PARTITION, a metadata operation, and TTL delete-merges disappear from the steady state entirely.

Downgrades: rows already stamped 90 days sit in the 90-day partitions. The query-time clamp hides them the moment the plan changes (that part stands from the original design), and for physical removal we run a per-tenant ALTER TABLE DELETE mutation on the affected partitions as a scheduled off-peak job — downgrades are rare events, so paying a targeted mutation per downgrade is cheap, unlike paying TTL merges continuously for everyone. If a contract requires purge-within-N-days, that job is the mechanism that satisfies it.

Design change: partition key becomes (retention_days, toDate(ts)); retention enforcement moves from row TTL to whole-partition drops; downgrade purge is a targeted per-tenant delete mutation.

Q6 (Codex): What false-positive rate do the proposed token and n-gram Bloom indexes have for a granule full of high-cardinality messages, and when they saturate, what prevents an adversarial 90-day substring query from scanning a tenant's entire cold corpus and exhausting shared S3 bandwidth or caches despite per-query quotas?

The FPR math first, because the question deserves the number: tokenbf_v1(30720, 3, 0) is 245,760 bits with 3 hash functions over an index granule of GRANULARITY 4 × 8,192 = 32,768 rows. At ~15 tokens/line that's ~500K token insertions; even at 50% distinct, n ≈ 250K into m = 245K bits saturates the filter — FPR effectively 1, zero pruning. The 4-gram filter saturates faster (a 250-byte line contributes ~247 4-grams). So for high-cardinality tenants — exactly the request-ID case I claimed was a non-event — the blooms as parameterized do nothing and the claim doesn't hold. Revision: GRANULARITY 1 (8,192 rows per filter) and larger filters sized from measured token distributions, accepting ~1% of raw size for the index instead of 0.1%; that targets n/m ratios giving FPR in the few-percent range for token search. The ngram filter is the one that can't be saved for adversarial content — substring search over high-entropy lines degrades to a time-range scan, and we say so rather than pretend.

Which is why the second half of the question needs a non-bloom answer: pruning is an optimization, never the resource bound. Three enforced caps, independent of filter effectiveness: max_bytes_to_read per query (a few hundred GB compressed), long ranges executed as cursor-driven day-chunks so no single query can touch 90 days of parts at once, and cold reads running in a low-priority workload class with a cluster-wide S3 bandwidth pool — an adversarial tenant saturates their own 2–3 concurrent query slots inside a bounded byte budget and queues behind themselves. Worst case for them is a slow, paginated scan of their own data; worst case for everyone else is unchanged.

Design change: bloom indexes resized (GRANULARITY 1, filters sized from measured token cardinality, ~1% storage budget); per-query max_bytes_to_read; long-range queries forced through day-chunk pagination; cluster-wide S3 read bandwidth pool for the cold workload class.

Round 2 — pushback

Q7 (Codex): Your forwarding spool is still Kafka, where offsets are committed per partition—not per key—so one slow destination still blocks every destination sharing that partition; how does a "per-destination skip set" preserve at-least-once delivery while advancing past undelivered records, rather than merely reintroducing loss or head-of-line blocking?

Fair — my round-1 answer blurred two states that need different treatment. The commit rule, stated exactly: a delivery worker's committed offset for a spool partition is the low-water mark — the lowest offset not yet either (a) delivery-confirmed or (b) converted into an accounted drop record. A slow-but-alive destination may hold that low-water mark back, but only within its lag budget; when the budget is exhausted, the circuit breaker trips and every held-back record for that destination is converted to a drop record before the offset advances past it. So the invariant is preserved in a form I can defend: every record is either delivered at least once or becomes a customer-visible drop record — nothing is silently skipped, because "skip" is defined as the atomic act of emitting the drop record. The pain that remains is honest HOL: a slow destination can delay partition-mates' commit progress (not their delivery — workers keep delivering ahead of the committed offset from an in-memory in-flight window) by up to its lag budget.

The crash case is where that in-flight window would turn into pain: restart from a held-back committed offset would re-send everything partition-mates already received. Fix: workers record per-destination delivered offsets to a small compacted forwarding-acks topic (keyed by destination_id); on restart, redelivered records already covered by the ack ledger are filtered instead of re-sent. That converts crash-redelivery from duplicate sends into cheap local filtering, and it means the lag budget bounds wasted reads, not wasted deliveries. Plus placement hygiene: hot destinations get dedicated spool partitions (sharded by stream_id when one partition isn't enough), so the destinations that share partitions are the small ones whose budgets are short.

Design change: low-water-mark commits with drop-conversion before offset advance; a compacted forwarding-acks ledger so crash redelivery filters instead of re-sends; hot destinations isolated onto dedicated spool partitions.

Q8 (Codex): Your parameterized view still lets the shared query-service credential supply any tenant_id, so a compromised or buggy query builder can request another tenant's rows and the auditor only notices afterward; what independent enforcement mechanism actually prevents that leak at query time?

Right — the view constrains query shape, not the tenant value, and an after-the-fact auditor is not enforcement. The at-query-time mechanism is to make the component that authenticates the customer the same component that binds the tenant value to the database session, and strip the query builder of both the credential and the ability to set it. Concretely: the auth proxy (from Q3) is the only holder of ClickHouse credentials. It terminates the customer token, resolves tenant_id, opens the DB session, and sets SETTINGS custom_tenant = <id> under a settings profile that declares the setting read-only for the rest of the session. The row policy on the logs table reads getSetting('custom_tenant'). The query builder receives the open session and submits query text over it, but it cannot change the setting (constraint-enforced by ClickHouse), holds no credential to open its own session, and its grant reaches only the policy-guarded table. A fully compromised query builder can now submit arbitrary SQL and still only sees rows for the tenant whose authenticated token opened the session — the tenant value was never in its hands.

What I'm claiming and not claiming: this is genuine at-query-time enforcement against a buggy or compromised query builder, which is the failure the second layer exists for. It is not enforcement against a compromised auth proxy — that component is the root of trust by construction, which is why it stays tiny (token in, tenant-bound session out, no query logic), separately deployed, and heavily tested. And for the tenants where blast radius matters most, the evolution path already had the physical answer: whale tenants on dedicated subclusters, where the isolation is machines, not predicates.

Design change: ClickHouse credentials move to the auth proxy, which binds tenant_id as a session-read-only custom setting at session open; row policies key off that setting; the query builder holds no DB credential. The auditor stays, demoted from enforcement to detection.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The Kafka-in-the-middle shape is the industry default, and the published numbers back the sizing debate from Q4. Cloudflare's internal pipeline — about a million log lines/s — puts Kafka between syslog-ng receivers and every consumer, partitions by host and service name to keep per-source ordering (same move as my stream_id keying), and sizes retention so "we can tolerate up to eight hours of total outage for our consumers without losing any data" (An overview of Cloudflare's logging pipeline, Jan 2024). Retention stated as consumer-outage-tolerance time is exactly the contract Codex forced me to fix in Q4; Cloudflare just published theirs as 8 hours where I landed on 4-at-peak.

ClickHouse-for-log-search at scale is well attested. Uber replaced its ELK platform with ClickHouse fed from Kafka and reported roughly 300K logs/s ingested per node — about 10x their Elasticsearch nodes — with hardware cost down more than half (Fast and reliable schema-agnostic log analytics platform), the same write-amplification argument I used to reject ES. ClickHouse's own LogHouse runs the pattern at 100+ PB uncompressed and 37M lines/s, and their lesson transfers: they burned 800+ CPU cores on OTel-collector transformations for 2M logs/s before replacing the generic pipeline with a purpose-built one (Scaling our observability platform beyond 100 petabytes, Jun 2025) — a concrete argument for the thin gateway that transforms nothing.

Datadog's Husky is the strongest counter-design worth knowing. It goes further than my tiered ClickHouse: compute fully separated from storage, all data on S3, metadata in FoundationDB, stateless writers/compactors/readers scaled independently (Introducing Husky). Two divergences matter. Husky's shard router keys Kafka by tenant+timestamp to create per-tenant locality, where I hash by stream to diffuse a tenant across partitions — they optimize read locality, I optimize hot-tenant blast radius, and their Autosharder is the machinery you need once you pick locality (Husky: exactly-once ingestion and multi-tenancy at scale). And Husky pays for exactly-once via event-ID dedup in FoundationDB transactions, where I settled for at-least-once plus best-effort dedup; that's the buy-vs-build line between a v1 platform and Datadog's third-generation one. My "custom Parquet-on-S3 in five years, maybe" is roughly Husky.

On live tail, Cloudflare's Instant Logs validates both halves of my tail design: a per-session budget with sampling when it's exceeded, and drop accounting attached to the data. Their target is under 3 seconds edge-to-client (mine: 2s), and under load they reservoir-sample and attach a sample interval to every surviving line so the client can still compute correct totals (How we built Instant Logs). That per-line sample interval is more useful than my "tail is sampling" banner — the same idea, but quantified per record; I'd steal it.

The Loki rejection needs a correction. I wrote that Loki has "no per-content pruning," and that's no longer quite true: Loki now builds bloom filters over structured metadata (trace IDs, customer IDs) so needle-in-haystack queries can skip whole streams and chunks — precisely the request-ID-in-90-days case I used against it. It's still experimental, gated to very large deployments, and scoped to structured metadata rather than free text (Loki bloom filters), so the conclusion survives, but the stated reason was stale. Meanwhile my own bloom troubles from Q6 turn out to be an industry-wide fork in the road — see the update below. And the WarpStream rejection holds on their own published numbers: ~400ms P99 produce and ~1s end-to-end producer-to-consumer, against 5–10x lower cost than self-hosted Kafka (Kafka is dead, long live Kafka) — exactly the latency-for-money trade I deferred, with the interzone-networking line item ($641/day vs <$15/day at 1 GiB/s) quantifying what the deferral costs.

Updates from post-training information

ClickHouse shipped general-availability full-text search in March 2026: a native inverted text index, tokenized at insert, mapping tokens to rows (Announcing general availability of ClickHouse full-text search, Mar 2026). Their benchmark shows the gap against exactly the mechanism I chose — 0.4s with the text index vs 143s with bloom-filter skip indexes on the same query. This lands directly on my Q6 concession: the bloom saturation problem on high-cardinality tokens now has a first-party alternative, and "no inverted index anywhere" is no longer obviously right. I wouldn't flip the whole design — an insert-time inverted index re-introduces write-path cost at 200M lines/s peak, which is why I rejected ES — but I'd benchmark the text index on the token-search path (keeping ngram blooms or plain scans for substrings) before committing to resized blooms as the v1 answer.

Further reading