Usage Metering and Billing for a Developer Platform

Metering is a distributed counting problem where the counts turn into money. That framing decides most of the design: every component below exists either to count correctly despite hours-late, duplicated data, or to prove afterward that we counted correctly. My recommendation up front: one immutable raw event log as the single source of truth, two read paths built from it — a streaming path for dashboards and spend caps (fast, allowed to undercount) and a batch path for invoices (slow, complete, reproducible) — and a rating engine that prices hourly usage buckets against a versioned price book so proration falls out of the data model instead of being special-cased.

Assumptions I'm making

The prompt fixes the big numbers; here's what I'm adding.

Events are cumulative-friendly deltas: a collector reports "resource R used 60 compute-seconds in minute-window W," not a raw stream of syscalls. Collectors already do first-level aggregation to one record per resource per meter per minute.
Collectors spool to local disk and retry with the same record identity. If a host dies with an unsent spool, that usage is lost — we detect it, we don't bill it. Undercount is the tolerated direction.
Billing is monthly per customer, in UTC calendar months. A customer owns many resources; a resource belongs to exactly one customer at a time (transfers are a plan-history event like any other).
"Days-late" means up to ~3 days is normal-rare; beyond that is an incident, not a data path.
Payment collection (cards, dunning, tax) is out of scope — I'd hand a finished invoice to a payment processor (Stripe) and keep everything upstream of the invoice in-house, because metering accuracy is where the margin and the trust live. Buying a usage-billing vendor (Metronome, Orb) is a legitimate alternative for a smaller company; at 3M resources with a spend-cap SLO measured in minutes, I want to own the pipeline.

Requirements, ranked

Invoice correctness, asymmetric. Never bill for usage we can't trace to raw events; never bill past a spend cap; never double-count. Undercounting costs known margin; overcounting costs trust and support load, so every ambiguous decision in the pipeline resolves toward the lower number.
Spend caps enforced within minutes of true usage crossing the cap — this is a safety feature, and it forces a streaming path into a system that would otherwise be happy as nightly batch.
Auditability: invoice line → rated items → hourly aggregates → raw events, every hop reproducible.
Near-real-time dashboard (~1 minute lag), explicitly allowed to differ from the invoice — it will say so.
13 months of per-resource-per-hour queryability.
Then the usual: availability of ingest (collectors buffer, so ingest downtime is survivable but eats into cap-enforcement latency), cost, operability.

Estimates that shape the architecture

All back-of-envelope, stated so we can check them against reality later.

Ingest: 1M events/min average ≈ 17K/s; I'll design for 5× peak, ~85K/s. At ~400 bytes/event that's ~35 MB/s peak — small for a partitioned log, so ingest throughput is not the hard part. Lateness and duplication are.
Raw volume: ~1.44B events/day, ~600 GB/day uncompressed; columnar compression should bring that to roughly 60–120 GB/day, call it ~25–50 TB for 13 months. Cheap in object storage, ruinous in an OLTP database — which settles where raw data lives.
Hourly aggregates: 3M resources × ~3 active meters × 24 hours × ~400 days ≈ 86B rows over the retention window, growing ~216M rows/day. That's squarely columnar-store territory and rules out row stores for the serving layer.
Dedup state: at 1M events/min, a 7-day dedup horizon means ~10B keys of hot state. That number picks the stream processor: we need one with large, checkpointed keyed state.

Architecture

flowchart LR
    subgraph hosts["50K hosts"]
        C[Collector agent<br/>disk spool, seq numbers,<br/>interval checkpoints]
    end
    C -->|mTLS, batched| G[Ingest gateway<br/>stateless, validates schema]
    G --> K[(Durable partitioned event log<br/>Kafka, keyed by resource_id,<br/>7-day retention)]

    K --> A[Archiver]
    A --> S3[(Immutable raw store<br/>S3 + Parquet, object lock,<br/>13+ months — SOURCE OF TRUTH)]

    K --> F[Stream processor - Flink<br/>dedup 7d TTL → hourly upserts<br/>+ per-customer MTD spend]
    F --> CH[(Columnar serving store<br/>ClickHouse: hourly usage,<br/>13 months)]
    F --> R[(In-memory spend counters<br/>Redis: customer MTD $)]
    R --> CAP[Cap engine] -->|suspend cmd, idempotent| CP[Control plane]
    CAP --> N[Alerts 50/75/90/100%]

    S3 --> B[Batch recompute + rating engine<br/>Spark: dedup whole month,<br/>rate vs versioned price book]
    PB[(Price book + plan history<br/>Postgres, append-only)] --> B
    B --> INV[(Billing DB - Postgres:<br/>rated items, invoices,<br/>adjustment ledger)]
    B -->|nightly repair| CH

    CH --> DASH[Dashboard API]
    INV --> FIN[Finance / disputes]

    C -.->|checkpoint records| K
    CP -.->|resource inventory| REC[Completeness checker]
    K -.-> REC

The shape is deliberately Lambda-style: streaming for latency, batch from the same raw log for truth, and a nightly repair job that overwrites streaming output with batch output so the two paths can't drift apart silently. I considered the Kappa alternative — streaming as the only truth, invoices read from Flink's output — and rejected it. "The invoice is whatever the stream job's state said" is a hard sentence to say to an auditor or in a dispute; "the invoice is a deterministic function of these immutable files, and here's the rerun" is easy. The cost is running two computations of the same aggregation, which at ~35 MB/s peak is cheap insurance.

Component choices, and what I rejected

Durable partitioned event log: Kafka, partitioned by hash(resource_id) so all events for a resource land in order on one partition, replication factor 3, acks=all, 7-day retention. Rejected Kinesis (24h–7d retention with weaker replay ergonomics and shard limits that fight the 5× peak) and Pulsar (fine technically, thinner operational bench). Kafka's retention is a recovery buffer, not storage — truth lives in S3.

Immutable raw store: object storage (S3) holding Parquet files, partitioned by event_time hour and sub-partitioned by arrival date, with S3 Object Lock in compliance mode for 13+ months. The archiver writes files named by Kafka partition and offset range, so any aggregate can name the exact files it came from. This is the audit backbone.

Stream processor: Flink. The deciding requirement is the ~10B-key dedup state with event-time semantics and checkpointed recovery. Kafka Streams struggles operationally at that state size; Spark Structured Streaming's micro-batches make the cap path lumpier and its late-data model is weaker. Flink's RocksDB state backend and event-time machinery are built for exactly this.

Serving store for hourly usage: ClickHouse (a columnar OLAP store), table ordered by (customer_id, resource_id, meter, hour), partitioned by month, replicated. Dashboard queries are "sum this customer's bandwidth by hour for 30 days" — columnar scans, not point lookups. Rejected Cassandra/DynamoDB (great point reads, painful ad-hoc aggregation, and the dashboard is nothing but ad-hoc aggregation), TimescaleDB (fine to ~1–10B rows; 86B is asking for trouble), and BigQuery/Snowflake for serving (per-query latency and cost profile wrong for an interactive dashboard, though fine as the batch engine).

Spend counters: Redis — one month-to-date dollar counter per customer, written by Flink, read by the cap engine. Rejected "query ClickHouse per evaluation": 100K+ customers evaluated continuously is a point-read workload, exactly what ClickHouse is bad at and Redis is trivial at. Redis here is a disposable cache: lose it, rebuild every counter from ClickHouse in minutes, caps degrade to slightly-stale during rebuild.

Batch/rating engine: Spark jobs (deterministic, versioned code) reading Parquet. Billing/price data: Postgres — invoices, price book, adjustment ledger are low-volume, relational, transactional. No debate needed there.

Event model and dedup: exactly-once where it counts

I'm not going to chase exactly-once delivery across five systems; I'm going to make every write idempotent and dedup at two places with different horizons.

Raw event:

event_id        uuid      -- v7, assigned by collector at record creation,
                          -- persisted in the disk spool BEFORE first send;
                          -- every retry reuses it
collector_id    string    -- host identity from mTLS cert
resource_id     string
customer_id     string    -- stamped at ingest from control-plane ownership map
meter           enum      -- compute_seconds | egress_bytes | build_minutes | addon_unit
window_start    timestamp -- minute the usage occurred (event time)
quantity        decimal
seq             bigint    -- per (collector, meter stream), monotonic, gapless
schema_version  int

The dedup key is event_id, and the reason it works is the spool discipline: the id is written to disk before the first network attempt, so a retry after a partition or a crash reuses it byte-for-byte. Content-hashing as the identity is the fallback if a collector loses its spool, but it's fragile (any serialization drift forks the identity), so it's the backup, not the plan.

Dedup happens twice, on purpose:

Streaming (Flink): keyed state on event_id with a 7-day TTL. A duplicate arriving 8 days late can slip past this and briefly inflate a dashboard number. Accepted — the dashboard is labeled provisional.
Batch (rating time): DISTINCT ON event_id across the entire month plus grace window, no TTL, over the Parquet files. This is the dedup that touches money, and it has no horizon problem because it sees the whole period at once. Tie-break rule if two records share an event_id with different payloads (should be impossible, happens anyway): keep the lower quantity. That's the asymmetric accuracy requirement expressed as one line of code.

So the honest exactly-once claim is: at-least-once delivery, idempotent everywhere, exactly-once-effective on the invoice path. I rejected leaning on Kafka transactions end-to-end: they cover Kafka-to-Kafka, and this pipeline crosses into S3, ClickHouse, Redis, and Postgres, so idempotent sinks were required anyway — at which point transactions add complexity without adding a guarantee.

Windows, watermarks, and when a month actually closes

Classic closing windows are the wrong tool when days-late is normal — any allowed-lateness you pick either drops money or holds state forever. So the streaming layer never closes windows. It runs keyed incremental aggregation: each event updates its (resource_id, meter, hour) bucket and re-emits the bucket as an upsert into ClickHouse (a ReplacingMergeTree — last write per key wins, so re-emits are harmless). An event three days late just re-emits a three-day-old bucket. The "window" is a row that can always be rewritten.

Watermarks still exist, but demoted to freshness metadata: the dashboard shows "complete through 14:32 UTC" computed from per-partition event-time progress, and the completeness checker alarms when any collector's watermark stalls. Watermarks inform humans; they never drop data.

A billing period closes on a business clock, not a stream watermark. Month M closes at M+1, day 3, 00:00 UTC — a 72-hour grace window, chosen because the prompt says hours-late is normal and days-late happens; in production I'd set it from the measured lateness CDF (cover ≥99.9% of event mass) and I'd expect 72h to be roughly right, but that's a target to validate, not a measurement I have. At close:

month M ──────────────┤ M ends
                      ├── 72h grace: late events keep landing in M's buckets
close ────────────────┤ rating run R over raw S3 data, full-month dedup
                      ├── validation: diff vs streaming totals, anomaly checks,
                      │   sample lineage audits
issue ────────────────┤ invoice issued; M's rated data is now immutable
after ────────────────┤ late events → adjustment ledger, never mutate M

Events for M that arrive after close go to an adjustment ledger, and policy is asymmetric by construction: below a threshold (say $1 per customer per month — a product decision, not mine to invent precisely) we write it off; above it, it appears on the next invoice as a labeled prior-period line, within a contractual lookback (e.g. 60 days). Adjustments in the customer's favor (we over-billed, discovered via dispute or reconciliation) have no threshold and no lookback limit — credits always flow. The issued invoice itself is never edited. That immutability is what lets finance close the books: "closed" means "this rating run's output will never change," and corrections are new ledger entries, exactly like accounting has always done it.

"Hold on," you might say — "if you recompute the whole month from raw at close, why bother maintaining hourly aggregates in the streaming path at all?" Because the cap has to fire in minutes and the dashboard in seconds, and neither can wait for a batch. The two paths answer two different questions: what's happening (streaming) and what happened (batch). One truth, two projections of it, and the nightly repair job keeps the fast projection honest: it recomputes buckets older than 48 hours from S3 and overwrites ClickHouse, and it alerts if the diff exceeds ~0.1% — a drift alarm on the pipeline itself.

Rating: usage × price, both versioned over time

Rating is deliberately dumb: a deterministic join. All the intelligence is in keeping both inputs versioned.

Price book (Postgres, append-only — rows are never updated, only end-dated):

price_version(version_id, sku, unit_price, currency, effective_at, effective_until)
plan_history(resource_id, plan_id, effective_at, effective_until)   -- from control plane
cap_history(customer_id, cap_amount, effective_at, effective_until)

Changing a price means inserting a new version with a future effective_at, behind a dual-approval workflow with an audit log — this table is the one place a bad write silently misprices every customer, so it gets change control like production config.

Rating for month M, run R:

rated_item(rating_run_id, customer_id, resource_id, meter, hour,
           quantity, price_version_id, amount,
           lineage_ref)   -- pointer into the run manifest: which S3 files/offset
                          -- ranges produced this bucket

Each hourly bucket joins to the price version and plan effective at that hour. A price change on the 17th means hours 1–16×24 rate at v1 and the rest at v2 — no proration formula, no special case; proration is the join. The one genuinely fiddly case is a plan migration mid-hour. Since raw events are minute-grained, aggregation splits that single hour's bucket at the migration timestamp into two rows keyed by plan context; it's rare enough (one hour per migration) that the extra rows are noise. I rejected the alternative — monthly-total proration formulas ("charged 13/30ths of plan A") — because every formula is a new thing to test, dispute, and get wrong, and because hourly rating also gives finance revenue recognition by day for free.

Rating runs are reproducible: rating_run(run_id, period, code_version, price_book_snapshot_id, input_manifest, output_hash). Rerun run R's code version on R's manifest and you must get R's output hash. That property is the dispute and audit story in one sentence.

Invoice = grouped rollup of rated items into invoice_line(invoice_id, description, quantity, amount, rated_item_range). Every line traces to rated items, every rated item to an hourly bucket, every bucket to named Parquet files, every file to signed collector batches. When a customer disputes a bandwidth charge, support drills from the line to per-hour usage in the dashboard UI; if it escalates, an engineer replays the exact raw slice. Distrust never has to travel more than one hop.

Spend caps: minutes, not batch runs

The cap path is the latency-critical slice through the same pipeline:

Flink, keyed by customer_id, rates usage on the fly at current prices (approximate is fine here — this is protection, not billing) and increments the customer's month-to-date counter in Redis.
The cap engine evaluates counters against cap_history on every update, emits notifications at 50/75/90/100%, and at 100% publishes an idempotent suspend(customer, cap_epoch) command to a Kafka topic the control plane consumes. Suspension fans out to the customer's resources; commands are idempotent so retries and duplicate crossings are harmless.

Latency budget: collector flush (≤30s) + ingest and stream (single-digit seconds) + control-plane suspension (seconds to tens of seconds) ≈ well under 2 minutes in the healthy case. The unavoidable caveat: a partitioned host's usage is invisible until it reconnects, so a customer can be truly over cap while our counter reads under. You cannot enforce on data you haven't received.

Which is where caps meet late data, and where the asymmetry does real work: the cap is a billing ceiling, not just a kill switch. At invoice time, rated charges for capped meters are clamped to the cap. Late events that push a suspended customer's true usage past the cap are written off — we served the traffic, we eat the cost, the customer pays the number they agreed to. The reverse policy (billing overage that our own enforcement lag allowed) is exactly the trust-destroying move the requirements forbid. This also resolves the ugly interaction case — customer near cap, three-hours-late data arrives, suspension fires "late" — without any cleverness: suspend as soon as we know, never bill past the line. One more subtlety worth saying out loud in the design review: suspension stops future compute but can't unsend egress bytes already served, so the write-off is a real, bounded cost of the feature.

Failure posture: Redis loss → rebuild counters from ClickHouse month-to-date sums (minutes of stale caps, alarmed). Flink restart → checkpoint restore replays a few minutes of the log into idempotent sinks. Cap engine down → alert page; this is the one component whose downtime has a direct dollar meter attached, so it's small, replicated, and boring.

Reconciliation: missing hosts, not just missing events

Dedup handles data that arrives twice. The harder problem is data that never arrives, because absence is silent. Two mechanisms, attacking it from opposite ends:

Checkpoint records (bottom-up). Every collector flush includes a checkpoint: (collector_id, interval, events_sent, seq_min, seq_max), and seqs are gapless per stream. The completeness checker continuously verifies two invariants: every expected interval from every known collector has a checkpoint (a silent host is detected by a missing checkpoint within ~5 minutes, not by someone noticing low revenue), and observed seq ranges have no gaps (a lost batch from a live host is detected the same way). A dead host with an unflushed spool shows up as a terminal gap — flagged, quantified from its last checkpoint rate, written off. Undercount, but known and measured undercount, which is a different animal from silent loss.

Inventory audit (top-down). The control plane knows which resources are supposed to exist and run; a resource in running state should produce compute-seconds every interval. A continuous job diffs the control-plane inventory against observed usage and flags expected-but-silent resources. This catches the failure class checkpoints can't: a collector that's alive and checkpointing but has a broken meter — cgroup accounting bug, a service the agent never discovered. Missing events have a shape; missing meters only show up when you know what should be emitting.

Both feed a daily completeness report — estimated unbilled dollars by cause — which is a finance artifact, not just an ops one: known undercount is a margin line item leadership can decide about, and it's also the canary for pipeline regressions.

APIs (external and internal surface)

POST /v1/meter/batch                     collector → gateway; mTLS; body: events[] + checkpoint
GET  /v1/usage?resource_id=&meter=&granularity=hour&from=&to=     (13-month range, ClickHouse)
GET  /v1/customers/{id}/spend/current    MTD estimate + freshness watermark + "provisional" flag
PUT  /v1/customers/{id}/caps             {amount, behavior: suspend|alert_only}
GET  /v1/invoices/{id}                   issued invoice, immutable
GET  /v1/invoices/{id}/lines/{n}/lineage rated items → bucket refs → raw file manifest (support/audit)

The usage API serves both the customer dashboard and internal tools, tenant-scoped by customer_id claims in the auth token — every table carries customer_id precisely so isolation is a WHERE clause the API layer enforces, not a convention.

Security and audit posture

Collectors authenticate with per-host mTLS certificates issued by the fleet PKI; collector_id comes from the cert, not the payload, so a compromised host can lie about its own usage but can't inject usage for other hosts — and its lies are bounded by anomaly detection on its own historical rates. Raw Parquet is under S3 Object Lock (compliance mode, 13+ months): nobody, including us, edits history. Price book and cap changes are dual-approved and logged. Rating code is versioned and its runs record the version. The overall claim to an auditor: every dollar on every invoice is a deterministic function of WORM-stored inputs and version-pinned code, and we can rerun any of it.

Failure modes, walked

Ingest gateway down / region blip: collectors spool and retry — the system's normal mode, just more of it. Cost is cap-enforcement blindness for the duration; alarm on gateway availability with that framing.
Kafka broker loss: RF=3, acks=all; a partition leader failover is seconds of latency, no loss.
Flink failure: restore from checkpoint, replay from Kafka offsets; sinks are idempotent upserts, so the replayed minutes are harmless. Worst case (checkpoint corruption): cold-rebuild streaming state from S3 for the open month — dashboards stale for hours, invoices untouched.
ClickHouse node loss: replicated tables; nightly repair re-heals any gap from S3.
Archiver lag: the scary one, because S3 is truth. Kafka's 7-day retention is the safety margin; alert at 12 hours of archiver lag, page at 24, and nothing in Kafka is eligible for retention-deletion until the archiver's committed offset passes it.
Bad deploy of aggregation logic: the nightly batch-vs-streaming diff alarm catches it within a day; the month's invoice is safe regardless because rating reruns from raw with pinned code.
Poison events / schema drift: gateway validates against schema_version; rejects go to a dead-letter topic with a queue-depth alarm, because a systematic reject pattern is an undercount in progress.

Tradeoffs I'm consciously accepting

Two computations of one aggregation (Lambda over Kappa). Extra infra and a class of drift bugs, bought back by the drift alarm and by an invoice path that's trivially explainable. For a billing system I take that trade every time.
Dashboard ≠ invoice, briefly. Provisional numbers can move as late data lands. Mitigated by labeling freshness in the UI rather than pretending; the alternative — delaying the dashboard until data is final — contradicts the near-real-time requirement outright.
Known write-offs: post-close small late usage, past-cap usage, dead-spool losses. All are bounded, measured on the completeness report, and all bleed in the direction the business chose to bleed.
7-day streaming dedup TTL: duplicates older than that can wobble a dashboard number until nightly repair. Invoices are immune (full-month batch dedup).
Minute-granularity collector pre-aggregation: we can't reconstruct sub-minute bursts. Acceptable because nothing prices at sub-minute granularity; if a future product does, the collector interval is a config change and the pipeline is already keyed by window_start.

Evolution

First version, honestly: Kafka → S3, hourly Spark aggregation into ClickHouse, monthly rating — no Flink. The spend-cap SLO is the one requirement that forces streaming from day one, so the cap path (Flink → Redis → suspend) ships first as a narrow slice while dashboards briefly tolerate hourly lag. From there: multi-region ingest with region-tagged partitions (rating already merges by resource, so it's additive); tier aggregate months older than 90 days to S3-backed ClickHouse storage; per-customer ML anomaly detection on the completeness report once there's a year of lateness and loss data to train against. The core contract — immutable raw log, versioned prices, reproducible rating runs — doesn't change as any of that lands, and that's the property I'd defend hardest in this room: everything else in the system is replaceable because the truth isn't.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): Flink checkpoints do not atomically cover Redis increments, ClickHouse upserts, and suspension commands; after a crash between those writes and checkpoint completion, how do you prevent replay from double-incrementing spend, firing a cap early, or publishing inconsistent totals across sinks?

This lands, and the specific bug is the Redis increment. INCRBY is not idempotent: crash after the increment but before the checkpoint, replay increments again, and a cap can fire early on a phantom double-count. The fix is to stop incrementing. Flink already holds per-customer keyed state; make the month-to-date dollar sum part of that state and write the absolute MTD value to Redis with SET, keyed by (customer, month). Replay re-emits the same absolute number — idempotent by construction — and it simplifies the Redis-loss story too: Flink state is the source, ClickHouse the fallback.

ClickHouse was already safe (we re-emit whole bucket values, not deltas), and suspension was already safe (idempotent per (customer, cap_epoch); a duplicate suspend of a suspended customer is a no-op). The residual anomaly: after a restore, Redis can briefly hold a value ahead of Flink's restored state until replay catches up — the counter regresses, then recovers. That can delay a cap by the replay time, never fire it early, and delay is the direction we already tolerate.

Design change: Redis writes become absolute MTD snapshots from Flink keyed state; increments are banned on the cap path.

Q2 (Codex): The gateway stamps customer_id using the ownership map at arrival time, but events may arrive days after a resource transfer, deletion, or plan migration; how will you perform an event-time ownership and pricing join with an authoritative, versioned history while handling races at exact transition boundaries?

Partially concede. The ingest stamp is wrong as a billing input — an event arriving two days after a resource transfer gets stamped with the new owner for usage the old owner incurred. So: demote it to a routing hint, renamed customer_id_at_ingest. It exists so the streaming path can key by customer without a lookup, and dashboards tolerate its brief wrongness around a transfer. The batch rating path never trusts it: rating joins resource_id against an append-only ownership_history (same table family as plan_history, fed by the control plane) at window_start, with half-open intervals [effective_at, effective_until) so a boundary minute belongs to exactly one owner.

Two supporting rules. The control plane commits the history row before the transfer takes effect — publish-before-effective — so rating can never see an event in a gap. And if history is genuinely missing for an event's time (deleted resource, corrupted record), we bill nobody and write it off onto the completeness report, per the asymmetry rule.

Design change: rating derives the customer from an event-time ownership join; the ingest stamp is provisional and never touches an invoice. (Round 2 tightens the streaming side of this — see Q8.)

Q3 (Codex): After an invoice closes, how does the adjustment pipeline distinguish genuinely new late usage from a duplicate of an event already included in the closed rating run, without retaining and querying a permanent dedup index for billions of historical event_ids?

No permanent online index, because adjustments are batch too. At close, rating run R already records its input manifest; additionally, it writes one cheap artifact per closed month: sorted event_id sidecar files — ids only, columnar, tens of GB per month before compression. A late event for M arriving after close lands in Parquet partitioned by event-time M / arrival date. The adjustment run (daily or weekly within the 60-day lookback) anti-joins new arrivals' ids against M's sidecar: a sorted-merge, no random reads, bounded by the size of new arrivals, which post-close is tiny by assumption. Survivors are genuinely new usage and flow to the adjustment ledger under the existing threshold policy; the rest are duplicates and are dropped. The 60-day contractual lookback is what keeps this finite — we only anti-join against months still inside their adjustment window.

Design change: rating close emits a sorted event-id sidecar per month; the adjustment pipeline is an anti-join against it. (Round 2 fixes a real gap in this answer — see Q7.)

Q4 (Codex): Both Flink and nightly repair can write the same ClickHouse bucket, while even later events can cause Flink to write it again; what revision or fencing protocol guarantees that an older or incomplete computation cannot overwrite a newer authoritative value, given that ReplacingMergeTree deduplication is asynchronous?

Concede. The original text had Flink writing arbitrarily-old buckets ("a three-day-late event just re-emits a three-day-old bucket") while nightly repair overwrites buckets older than 48h — with ReplacingMergeTree's asynchronous merges, that's a race with no defined winner. The fix is disjoint write ownership by bucket age instead of a fencing protocol: Flink writes only buckets younger than 72 hours (matching the grace window); repair owns everything older. An event later than 72h still reaches Kafka, S3, and the Redis spend counter — caps still see it — but its ClickHouse bucket updates only via the next repair run, and the dashboard's freshness label covers the gap.

Within the young band Flink is the only writer, so ReplacingMergeTree with an explicit version column (Flink processing time) converges correctly even with async merges; dashboard queries use FINAL/argMax to tolerate unmerged parts. Repair runs hold a lease — one run at a time — and stamp their rows with a run-scoped version strictly above any streaming version for the buckets they own.

Design change: 72h streaming write horizon, single writer per age band, explicit version column, leased repair runs.

Q5 (Codex): Your recovery claims require rebuilding seven days of roughly 10 billion dedup keys, or an open month of billions of hourly rows and spend totals, while caps remain enforceable within minutes; what are the concrete state size, checkpoint duration, restore throughput, and degraded-mode bounds that make those claims credible?

Fair demand; here are the estimates, labeled as estimates. Dedup state: ~10B keys at roughly 60–100 bytes each in RocksDB (16-byte id plus overhead) is ~0.6–1 TB. Over ~128 task slots that's 5–8 GB per slot — large but ordinary for RocksDB. Checkpoints are incremental: steady-state new-key rate is ~17K/s × ~100B ≈ 2 MB/s of delta, so checkpoint intervals stay in seconds; we never snapshot the full TB. Restore: ~1 TB from S3 across 128 workers at ~100–200 MB/s each is a few minutes of I/O; the honest end-to-end bound including scheduling and catch-up replay is 5–15 minutes. During that window caps don't go dark — they enforce on Redis, which is independent of Flink and merely goes stale; exposure is (customer burn rate × outage minutes), and because the cap is a billing ceiling, that exposure is a bounded write-off, not a bill. Cold ClickHouse rebuild of an open month from S3 (single-digit TB compressed) is a Spark scan measured in tens of minutes to an hour — dashboards degraded, invoices untouched.

Doing this arithmetic exposed dead weight: the 7-day streaming dedup TTL bought nothing the batch dedup doesn't already guarantee, and 72h — matching the grace window — cuts hot state to ~300–450 GB.

Design change: streaming dedup TTL drops from 7 days to 72h; a 15-minute restore SLO is pinned with a tested runbook, because "minutes" was doing unexamined work in the original.

Q6 (Codex): How are stateful add-ons such as custom domains metered and prorated when lifecycle events are duplicated, reordered, or missing, given that the proposed minute-delta event model, inventory silence checks, and hourly usage aggregation only compose naturally for continuously measured resources?

Full concession — custom domains aren't a flow, they're a state, and pushing state through a duplicated/reordered lifecycle-event pipeline is borrowing trouble. Split meters into two classes. Measured meters (compute, egress, builds) keep the collector path. Declared meters (add-ons) never ride the event pipeline: the control plane already maintains transactional attach/detach records, so we extend the history-table family with entitlement_history(customer_id, addon_sku, effective_at, effective_until), and a deterministic generator job emits synthetic hourly usage rows (quantity = 1 per active hour) straight from those intervals into the same hourly-bucket shape. No dedup problem — the rows are a pure function of versioned state, regenerable at any time, which is a stronger property than the event path has. Duplicated or reordered lifecycle events can't corrupt it because the source of truth is the transactional interval record, not the events.

Proration then falls out of the existing machinery unchanged: hourly rows join the price book at each hour, and an add-on attached on the 17th rates exactly like a price change on the 17th. Inventory audit gets simpler for this class too: expected rows are computable exactly, so any diff is a generator bug, not a lost packet.

Design change: meters split into measured (collector path) and declared (generated from entitlement_history); add-ons move to the declared path.

Round 2 — pushback

Q7 (Codex): Your closed-month sidecar contains only IDs seen at close; when a genuinely late event is accepted in adjustment run 1 and retried before run 2, what prevents run 2 from charging it again without a durable index of previously adjusted IDs?

Caught — the round-1 answer had a hole exactly there. A late event accepted by adjustment run 1, then retried by its collector (which is normal behavior, not an edge case), survives run 2's anti-join against the close-time sidecar and gets charged twice. So yes, we need durable memory of adjusted IDs, and I concede the "no index" framing. But it's the same sidecar mechanism, made cumulative, not a new online index: each adjustment run appends the sorted IDs of every event it consumed — accepted or written off — to the month's sidecar, and run k anti-joins against the union of the close-time files plus runs 1..k-1. Still sorted-merge over append-only files, still no random reads. Growth is bounded by post-close arrivals within the 60-day lookback, which is small by the lateness assumption; when the lookback expires, the month's sidecar is frozen and eventually tiered away with the rest of the audit trail. Adjustment runs for a month are serialized under a lease (they were implicitly; now it's a stated invariant), so no two runs race on the same sidecar, and each run records a manifest exactly like rating runs do — the adjustment path gets the same reproducibility contract as the close.

Design change: the per-month event-id sidecar becomes cumulative — every adjustment run appends its consumed IDs — and adjustment runs for a month are serialized under a lease, each with its own recorded manifest.

Q8 (Codex): You admit late pre-transfer usage is routed into the new owner's streaming spend counter; why is suspending an innocent new owner "tolerable," and how will you correct both customers' counters within the cap SLO using event-time ownership?

I'll push back on scale before conceding mechanism: to suspend the new owner, misattributed late usage from one transferred resource has to close the new owner's entire remaining cap headroom, and transfers of resources with hours of buffered unsent usage are rare-times-rare. But "rare" is not a mechanism, suspension is customer-visible, and the fix is cheap — so concede the design point. The streaming path does the same event-time ownership join batch does: ownership_history is small (transfers are infrequent; the table is a fraction of the price book's change rate), so Flink consumes it as broadcast state, and each event is attributed by comparing window_start against the ownership intervals — a late pre-transfer event lands in the old owner's MTD state even in streaming. customer_id_at_ingest drops out of spend attribution entirely; both customers' counters are correct within normal pipeline latency, which is seconds, well inside the cap SLO.

The residual race is the broadcast propagation window: an event processed in the seconds before the transfer record reaches the operator gets the old interval. That misattribution is bounded by propagation lag and self-corrects — the state is re-keyed when the record lands, and because Redis writes are absolute MTD snapshots (Q1), the corrected values overwrite cleanly. Which exposes the one missing verb: counters can now decrease, so the cap engine needs a resume path. If a re-emitted MTD drops a suspended customer back under their cap, the engine publishes an idempotent resume(customer, cap_epoch) the control plane consumes, symmetric with suspend.

Design change: streaming spend attribution uses an event-time ownership join via broadcast state (ingest stamp removed from that path), and the cap engine gains an idempotent resume command for counter regressions.

What changed, summarized

Redis spend counters: absolute per-(customer, month) MTD snapshots written with SET from Flink keyed state; increments banned on the cap path.
Customer attribution: event-time join against append-only ownership_history in batch rating and (round 2) in streaming via Flink broadcast state; the ingest-time stamp is demoted to customer_id_at_ingest, a routing hint that never touches an invoice or a spend counter.
Post-close dedup: each closed month gets a sorted event-id sidecar; adjustment runs anti-join against it, append their own consumed IDs to it (round 2), and are serialized under a lease with recorded manifests.
ClickHouse write ownership: Flink writes only buckets younger than 72h, repair owns older ones; explicit version column on ReplacingMergeTree; repair runs leased.
Streaming dedup TTL cut from 7 days to 72h (~300–450 GB hot state instead of ~1 TB); 15-minute Flink restore SLO pinned with a tested runbook.
Meters split into measured (collector events) and declared (generated hourly rows from entitlement_history); add-ons like custom domains move to the declared path.
Cap engine gains an idempotent resume(customer, cap_epoch) command, symmetric with suspend, for when corrected counters drop a customer back under cap.

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The two-path split — fast provisional numbers, slow authoritative ones — is exactly what Stripe built. Their usage-based billing writeup describes a dual-path Flink aggregation: a fast path on 30-second in-memory tumbling windows for real-time alerts, a slow path on 5-minute disk-backed windows that absorbs late and out-of-order events, with P95 under 30 seconds for alerts and ~5 minutes to invoicing. Same shape as my streaming/batch split, compressed — both their paths are streaming, and their tolerated lateness is minutes where this prompt's is days. The 72-hour grace window and the batch-recompute-from-raw layer here are what "days-late is normal" forces that Stripe's workload doesn't.

The stronger endorsement of raw-events-as-truth comes from Orb, who push it further than I did: events are stored immutably and never aggregated, and every invoice is computed by querying event history at billing time — late events, backfills, and backdated contract changes are just re-runs of the query. That's my rating-run reproducibility contract taken to its limit, with no streaming aggregates at all. I kept hourly buckets because 86B rows of 13-month dashboard queries and a minutes-level cap SLO need a serving layer; Orb's model shows which half of the design is the load-bearing half. The invoice-as-deterministic-function-of-immutable-inputs idea is the industry consensus, not a novelty.

On exactly-once, the field split the same way I did. Flink's own end-to-end exactly-once post is explicit that the two-phase-commit sink only works when every external system supports coordinated transactions — which S3, ClickHouse, Redis, and Postgres in this design don't, in the shape required. So practitioners converge on at-least-once plus idempotent sinks: OpenMeter abandoned Kafka Connect for a custom Go consumer that routes each idempotency key to one partition, checks a Redis seen-set, batch-inserts into ClickHouse, and reprocesses whole batches on failure — the same "idempotent everywhere, dedup where it counts" stance as my Q1 answer, including the ban on non-idempotent increments. Uber's ad-event billing pipeline mixes both: Kafka transactions with read_committed inside the streaming layer, then record UUIDs and Pinot upserts once data crosses a system boundary. Transactions where the boundary supports them, idempotency keys where it doesn't.

ClickHouse as the metering serving store is now the boring choice, which is the best kind. Lago moved raw billable events out of Postgres into ClickHouse when millions of events per minute started blocking their transactional queue — the same OLTP-vs-OLAP line my estimates drew — and by March 2026 they were quoting ingestion approaching a million events per second on ClickHouse Cloud, querying raw events with no pre-aggregation. OpenMeter went the other way within the same database: pre-aggregate into one-minute tumbling windows with AggregatingMergeTree and materialized views to cut query cost. My design pre-aggregates at the collector and again into hourly buckets, so it's closer to OpenMeter's economics; Lago's raw-query model is viable but pays for it in query-time compute, which their managed-cloud pricing absorbs.

Spend caps are where the big clouds validate the framing by their absence. AWS still offers budgets and alerts, not hard stops. Google shipped spend cap budgets in Preview on July 28, 2026, and their announcement concedes the core problem in one line: "traditional billing data can sometimes take hours to reconcile, but Spend Caps for AI services trigger within minutes" — a dedicated fast path bolted beside the billing pipeline, which is precisely why the cap engine here rides Flink-to-Redis instead of ClickHouse. Their enforcement is non-destructive suspension with one-click resume, matching the suspend/resume pair from Q8. Vercel's hard caps pause all projects at the cap and eat any overrun — the cap-as-billing-ceiling policy, deployed: nobody who ships a hard cap bills past it.

Updates from post-training information

Two of the sources above postdate my training data, both checked directly. Google Cloud's spend cap budgets entered Preview on July 28, 2026 (Gemini API, Agent Platform, Cloud Run, Cloud Run Functions), with minutes-level enforcement and 50/80/100% alert thresholds — as of my cutoff, GCP had budgets and programmatic-disable recipes, not a native hard cap. And ClickHouse published the Lago case study on March 5, 2026, with the ~1M events/second billing-ingestion figure and Lago's stated plan to retire Postgres as their event store entirely. Neither changes a decision in this design; the Google launch strengthens the claim that minutes-level cap enforcement requires a purpose-built fast path, because that's what they built.