Contents

Webhook Delivery Platform

My recommendation up front: an ingest API that dual-writes every event to a durable, partitioned event log — Kafka — in two regions before acking, a fan-out service that matches events against an in-memory copy of the subscription table, a delivery tier built around per-endpoint isolation (bounded queues, circuit breakers, tiered retry topics), attempt history in a columnar OLAP store (ClickHouse) with a 30-day TTL, and payloads over a size threshold in object storage (S3). Control-plane data lives in a relational database (Postgres). Delivery semantics are at-least-once with best-effort ordering, and I'll defend that choice below because it drives most of the architecture.

Assumptions

The prompt leaves a few things open. I'm assuming:

Scale estimates (the ones that matter)

Two numbers shape everything: 175k deliveries/s peak means the delivery tier is a large async-IO fleet with careful backpressure, and 45 TB of hot queryable history rules out Postgres for attempts.

Requirements I'm prioritizing

  1. Durability of accepted events (regional failure survival) — this is the contract with producers.
  2. Isolation — one broken endpoint, or one huge customer, can't degrade anyone else. This is the hard part of any webhook system.
  3. Observability for customers — the UI/API over delivery attempts is half the product; a webhook platform customers can't debug generates support tickets instead of value.
  4. Throughput and latency come after those. Nobody notices a webhook arriving in 800 ms instead of 400 ms; everybody notices lost events.

Architecture

flowchart LR
    P[Internal producers] -->|POST /v1/events| ING[Ingest API]
    ING -->|write| KA[(Kafka: events\nregion A)]
    ING -->|sync mirror write| KB[(Kafka: events-mirror\nregion B)]
    ING -.->|idempotency check| RD[(Redis)]
    ING -->|payloads > 64 KB| S3[(S3, cross-region\nreplicated)]

    KA --> FO[Fan-out / matcher]
    PG[(Postgres:\nendpoints, subscriptions)] -->|CDC cache refresh| FO
    FO -->|task per endpoint,\nkeyed by endpoint_id| DT[(Kafka: deliveries)]

    DT --> DW[Delivery workers\nper-endpoint queues,\ncircuit breakers,\nrate limits]
    DW -->|HTTPS + HMAC,\negress proxy| EP[Customer endpoints]
    DW -->|on failure| RT[(Retry topics:\n30s 2m 10m 30m\n2h 6h 12h 24h)]
    RT --> DW
    DW -->|attempt records| AT[(Kafka: attempts)]
    AT --> CH[(ClickHouse\n30-day TTL)]
    CH --> API[Customer API + UI]
    PG --> API

Ingest and the regional-durability decision

POST /v1/events hits a stateless ingest tier. It validates the event, checks an idempotency key against an in-memory key-value store (Redis; 24 h TTL — producers retry, and we shouldn't fan out twice), assigns a ULID event ID, and persists before returning 202.

"Persists" is where the regional requirement bites. Kafka with RF=3 across three AZs survives an AZ loss but not a regional outage. Async cross-region replication (MirrorMaker) has an RPO of seconds — events acked but not yet mirrored die with the region, which violates the requirement as I've read it. So the ingest service writes synchronously to Kafka in the local region and to a mirror topic in a second region, and acks only when both succeed. Cost: one cross-region round trip, ~30–70 ms added to ingest latency. For asynchronous event submission from internal services, that's a fine price for RPO = 0.

Degraded mode matters: if region B is unreachable, we don't stop accepting events — we ack after the local write, tag the event single_region_durability, and page. The alternative (refusing writes) turns a partial outage into a full one. This is a deliberate tradeoff: during a cross-region partition, the durability guarantee temporarily weakens rather than availability going to zero. I'd put that in the SLA in writing.

I considered a stretch/quorum log across three regions (Confluent multi-region clusters, or an S3-backed log like WarpStream). Rejected the stretch cluster for operational complexity and the blast radius of one cluster spanning regions; rejected S3-backed logs because per-event latency and the "what if S3 in-region blips" story were weaker than two independent Kafka clusters I can reason about separately. Dual-write is dumb and legible, and legible wins for the durability path.

Payloads ≤ 64 KB ride inline in the Kafka message. Larger ones (up to 256 KB) go to S3 with cross-region replication, and the event carries a pointer — this keeps Kafka message sizes predictable and lets the history UI fetch bodies lazily.

Fan-out / matching

A consumer group reads the events topic and answers "which endpoints want this?" The subscription dataset is tiny: 100k customers × ~5 endpoints × a set of event types is a few million rows — it fits in memory on every matcher node. Matchers hold a full copy, kept fresh by Postgres CDC (Debezium) with a periodic full reload as a safety net. No per-event database lookup at 116k events/s; that would be the first thing to fall over.

For each match, the matcher emits a delivery task — (delivery_id, event_ref, endpoint_id, attempt=0) where delivery_id = hash(event_id, endpoint_id) so retries and failovers stay idempotent — onto the deliveries topic, partitioned by endpoint_id. Partitioning by endpoint gives best-effort ordering and makes per-endpoint rate limiting tractable, at the cost of hot partitions for huge endpoints. A customer doing 20k deliveries/s to one endpoint gets salted across N partitions, explicitly giving up ordering — you can have extreme throughput or ordering hints, not both, and I'd make that a documented per-endpoint setting.

Delivery workers — where webhook platforms live or die

The naive design — consume the deliveries topic, make the HTTP call, commit offset — has a fatal flaw: head-of-line blocking. One endpoint timing out at 30 s stalls every other endpoint sharing its partition. With "some endpoints are permanently broken" in the prompt, this isn't an edge case; it's Tuesday.

So workers decouple consumption from delivery:

Retries: exponential backoff with full jitter — roughly 30 s, 2 m, 10 m, 30 m, 2 h, 6 h, 12 h, 24 h — implemented as tiered Kafka retry topics, one per delay bucket. A failed attempt is produced to the bucket matching its next retry time with a not_before timestamp; that bucket's consumer pauses until the head message is due (safe because everything in one topic shares the same delay). I chose this over a Redis sorted-set scheduler because the retry backlog can get big — a popular endpoint down for 12 hours can park tens of millions of tasks — and Kafka handles a 100 GB backlog without me thinking about it, while a Redis timer wheel at that size becomes its own incident. The cost is coarse retry granularity, which is fine: backoff schedules don't need precision.

After 24 hours of failures the delivery is marked failed (dead-lettered — the task record persists in history, and the payload is still in S3 for manual redelivery within the 30-day window). Endpoints that fail everything for several consecutive days get auto-disabled with customer notification, same as Stripe does — otherwise the retry tier fills with permanently broken endpoints forever.

Thundering herd on recovery: when a big endpoint comes back, jittered backoff plus the endpoint's own rate limit means we drain its backlog at its configured ceiling rather than instantly firing every parked delivery at a service that just barely recovered.

Delivery history

Every attempt emits a record to an attempts Kafka topic, batch-inserted into ClickHouse: (customer_id, endpoint_id, delivery_id, event_id, event_type, attempt_no, ts, status, http_status, latency_ms, error, response_snippet ≤ 4 KB), ordered by (customer_id, endpoint_id, ts), partitioned by day, TTL ts + 30 days. 3B rows/day is comfortable for a modest ClickHouse cluster, compression brings 45 TB down to single-digit TB, and the customer-facing queries — filter by endpoint, status, event type, time range; count failures per hour — are exactly what a columnar store is good at.

Rejected alternatives: Postgres (45 TB of hot rows plus this write rate is not its job), Cassandra/DynamoDB (handles the writes fine, but the UI wants flexible filtering and aggregation, which means either many hand-built index tables or expensive scans), Elasticsearch (works, costs roughly 3–5x more to run at this volume for no query we actually need). The tradeoff I'm accepting: history is eventually consistent by a few seconds behind the Kafka consumer. For a debugging UI, nobody cares.

Data model and APIs

Postgres (control plane):

endpoints(id, customer_id, url, secret_kms_ref, status,        -- active | disabled
          rate_limit, ordered boolean, created_at)
subscriptions(endpoint_id, event_type)                          -- supports wildcards like "invoice.*"

Signing secrets are generated per endpoint, stored encrypted under KMS, never displayed after creation, and rotatable with an overlap window (both old and new signatures sent during rotation).

Producer API (internal):

POST /v1/events
  { idempotency_key, customer_id, event_type, payload }
  → 202 { event_id }

Customer API:

POST   /v1/endpoints                      create (returns secret once)
POST   /v1/endpoints/{id}/rotate_secret
GET    /v1/deliveries?endpoint_id=&status=&event_type=&from=&to=&cursor=
GET    /v1/deliveries/{id}                all attempts, payload, response snippets
POST   /v1/deliveries/{id}/redeliver      manual redelivery (works for 30 days)
GET    /v1/endpoints/{id}/stats           success rate, latency, backlog depth

Cursor pagination on ClickHouse's sort key; the UI is a thin client over this API.

Delivery request to the customer:

POST {endpoint.url}
webhook-id: dl_9f3k...            (delivery_id — customers dedupe on this)
webhook-timestamp: 1755600000
webhook-signature: v1,base64(HMAC-SHA256(secret, id + "." + ts + "." + body))

Timestamp in the signed string gives replay protection (reject if older than 5 minutes). 2xx within 10 s counts as delivered; anything else — including 3xx, which we do not follow — is a failure.

Security

The nastiest risk in this system is that it's an SSRF machine by design: customers hand us URLs and we make requests to them from inside our network. Mitigations, all mandatory:

Plus the ordinary hygiene: HMAC signatures as above, HTTPS-only endpoints with certificate validation, per-tenant rate limits at ingest, and tenant scoping enforced in the history API at the query layer (every ClickHouse query is keyed by the authenticated customer_id — it's the first column of the sort key precisely so tenant isolation is structural, not a WHERE clause someone can forget... though it is also a WHERE clause).

Failure modes

Tradeoffs I made on purpose

Evolution

Day one I'd actually build a smaller version of exactly this shape: one region (accepting async replication and an honest RPO-seconds SLA until the dual-write lands), one retry topic with a not_before re-enqueue loop, Postgres for the first month of history. Every component above is the scale-up of that skeleton, not a rewrite.

Later: payload filtering/transformation at fan-out (customers subscribing to invoice.* but only fields they need), batched deliveries for high-volume endpoints, and a pull option — letting a big customer consume from a dedicated Kafka topic or Kinesis stream instead of receiving pushes. Past a certain volume per customer, push-over-HTTP is the wrong transport, and the fan-out/history layers here don't care which transport sits at the end.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): Your RPO=0 claim relies on non-transactional writes to two independent Kafka clusters and asynchronous S3 cross-region replication; what exact protocol prevents an acknowledged event—or its large payload—from existing in only the failed region, and how do you reconcile that guarantee with your explicit single-region degraded acknowledgments?

The dual-write doesn't need to be transactional because the 202 is the commit point: ingest writes both clusters in parallel and acks only after both succeed. A crash before the ack leaves at most an orphan in one region; the producer retries, and the orphan becomes a duplicate, which the delivery-side dedupe contract absorbs. But the question does expose a real bug in my dedupe chain: idempotency keys live in regional Redis and event_id is a fresh ULID, so if region A dies after acking and the producer retries against region B, Redis there has no key, we mint a new event_id, delivery_id changes, and the customer sees a true duplicate with a different webhook-id — the one identifier I told them to dedupe on. Fix: derive event_id deterministically — keep the ULID timestamp prefix but fill the random bits with hash(producer_id, idempotency_key) — so the same submission yields the same event_id and the same delivery_ids no matter which region accepts it. Redis becomes a fast-path optimization instead of a correctness dependency.

The S3 point I concede outright: async cross-region replication breaks RPO=0 for >64 KB payloads — an acked event whose body exists only in region A's bucket is exactly the loss I claimed to prevent. Fix: for large payloads, ingest does synchronous PUTs to buckets in both regions, in parallel with the two Kafka writes, and acks only when all four succeed. Affordable because >64 KB is rare tail traffic, and the PUTs run concurrently with the cross-region Kafka write so the latency adder is roughly zero.

Degraded mode isn't a contradiction of RPO=0; it's a carve-out I'd write into the SLA: the guarantee requires two healthy regions, and while B is down, new acks are explicitly single-region-durable. Actual loss then requires a second, correlated failure — region A dying inside the same window. I'd surface the degradation on the status page, not just page on-call, so producers can decide whether to keep submitting.

Design change: event_id derived deterministically from (producer_id, idempotency_key); large payloads dual-PUT synchronously to both regions before ack.

Q2 (Codex): After regional failover, region B replays raw events through a matcher using the current subscription cache, not the subscription state at original acceptance; how do you prevent deleted endpoints from losing already-owed deliveries, newly created endpoints from receiving historical events, and partially completed fan-outs from producing a different delivery set?

First a mechanical sharpening the doc glossed over: region B can't discover region A's consumer offsets after A is gone, so the standby matcher runs hot — continuously consuming the mirror topic and committing offsets locally on cluster B — with delivery emission gated off. Promotion rewinds a safety window (a few minutes) and opens the gate. Because delivery_id = hash(event_id, endpoint_id) is deterministic, re-matching the overlap produces the same delivery set for the same subscription state: duplicates, not divergence.

For subscription drift inside that window, the contract I'd document is match-time semantics: fan-out reflects the subscription table when the event is matched, not when it was accepted. Under that contract, an endpoint deleted during the window losing its pending deliveries is correct — deletion means stop sending. The genuinely bad case is the inverse: a just-created endpoint receiving events from before it existed. That gets a structural rule, not a hope: event ULIDs carry the accept timestamp, and the matcher delivers only if event_ts >= endpoint.created_at. One comparison per match, and replays can never leak history into new endpoints. Region B's matcher reads a lagging Postgres replica, but that's the same bounded staleness the CDC cache already has in steady state, with the same full-reload safety net.

Design change: standby matcher in region B runs hot with emission gated; matcher enforces event_ts >= endpoint.created_at.

Q3 (Codex): Walk through every crash boundary among the outbound HTTP call, publishing the attempt record, publishing a retry task, and committing the source offset: which operations are atomic, and how do you prove that every accepted delivery is either completed or retryable without silently losing it when Kafka produces or commits fail independently?

Nothing here is atomic, and the design doesn't need atomicity — it needs one invariant: a partition offset is committed only when every task at or below it is resolved, where resolved means delivered (2xx) or parked (retry-produce confirmed with acks=all). Because per-endpoint queues deliver out of order relative to partition offsets, the commit has to be watermark-based — each worker tracks, per partition, the low watermark of unresolved tasks and commits that. The doc said "offsets commit once a task is either delivered or durably parked" but didn't name the watermark mechanism; it's load-bearing, so consider it named now.

The proof of no-loss is then a case walk. Crash before the HTTP call: task unresolved, offset not committed, replay. Crash after HTTP success but before resolving: replay, duplicate delivery, covered by the webhook-id contract. Crash after retry-produce but before commit: replay produces a second parked copy of the same delivery_id — a duplicate attempt later, again by contract, at the cost of a slightly inflated attempt count. Retry-produce fails outright (retry cluster unhealthy): the task stays unresolved, the watermark stalls, consumption backpressures into the deliveries topic, and Kafka absorbs the backlog — degraded latency, zero loss. Attempt records are deliberately off this critical path: a crash can lose an attempt row, which costs a blip in the debugging UI, not a delivery. History is observability, not the source of truth; the deliveries and retry topics are.

Design change: the offset-commit rule is now explicit: per-partition low-watermark commit over resolved (delivered-or-parked) tasks.

Q4 (Codex): Your retry consumer pauses when the head record's not_before is in the future, but jitter and differing enqueue times mean later records in that partition may already be due; how do you avoid head-of-line blocking across unrelated endpoints while preserving bounded retry timing and per-endpoint isolation?

You've caught an inconsistency in the doc as written: full jitter on the backoff delay makes not_before non-monotonic within a partition, and then "pause until the head is due" either blocks due messages behind not-yet-due heads or fires early. The fix is to move the jitter. Each retry topic's delay is fixed: not_before = enqueue_ts + D, nothing else. Kafka preserves enqueue order per partition, so not_before is monotonic and waiting on the head is exact — a later message can't be due before the head. Jitter migrates to the release side: the retry consumer adds a small random hold (0–10% of D) as it hands tasks to the delivery pool, and per-endpoint token buckets spread any residual burst. Jitter's job is to decorrelate attempts, and it does that job equally well applied at release.

Per-endpoint isolation inside a retry partition follows the same deliver-or-park rule as the main path: when a due task is released and its endpoint is circuit-open, over its concurrency cap, or over its rate limit, it's immediately re-parked into the next backoff bucket rather than held. A retry partition therefore only ever waits on the clock, never on an endpoint. Bounded timing still holds: a task's actual retry time is its scheduled time plus consumer lag plus the small release jitter, and consumer lag is monitorable and scalable per bucket since each bucket is just another consumer group.

Design change: fixed per-topic delay with monotonic not_before; jitter applied at release, not enqueue; deliver-or-park applies to retry consumers too.

Q5 (Codex): You provision for roughly two attempts per delivery, yet your schedule permits many attempts and a widespread endpoint outage can turn peak ingress into sustained retry amplification; what workload bound keeps the delivery fleet, retry topics, and 24-hour deadline stable when arrival rate exceeds the aggregate rate limits of affected endpoints?

The 2-attempts figure sizes the steady-state fleet; it was never the bound. The actual bound is the retry schedule itself: at most 9 HTTP attempts per delivery in 24 hours. Worst case — every endpoint failing all day at average ingest — is 1.5B deliveries x 9 / 86,400 s, roughly 156k attempts/s sustained, inside the 350k/s burst budget I already provisioned. And the realistic worst case is much cheaper than that ceiling, because the two failure classes are both cheap: permanently-dead endpoints go circuit-open, and an open circuit means parked tasks re-park without an HTTP call — tens of millions of tasks behind one dead endpoint cost about one probe request per cooldown, not one call per task. Up-but-erroring endpoints burn at most their own token bucket.

Two safety valves sit above the arithmetic. First, first attempts and retries run in separately provisioned worker pools, with the retry pool globally capped (around 1.5x average delivery rate): a mass outage stretches retry drain toward the 24 h deadline but cannot starve fresh deliveries. Second, retry topics carry pointer-sized tasks (~200 bytes), so even the absurd case — 12B parked tasks — is ~2.5 TB of Kafka; storage never becomes the binding constraint. There is one honestly accepted failure mode: if an endpoint's backlog exceeds rate_limit x time-to-deadline, its tail deliveries expire at 24 h — that's what the deadline means. I'd make it visible rather than surprising: the endpoint stats API already reports backlog depth, and it should also report projected drain time so the customer sees the expiry coming and can raise their rate limit or fix their endpoint.

Design change: separate first-attempt and retry worker pools with a global cap on retry throughput; stats API adds projected drain time.

Q6 (Codex): Manual redelivery and payload inspection are promised for 30 days, but only payloads over 64 KB go to S3 and no 30-day retention is specified for inline Kafka payloads; where are smaller payloads durably retained, and how are their retention, deletion, encryption, and event references kept consistent with ClickHouse history?

This is a contradiction in the doc, plainly: I promised redelivery and payload inspection for 30 days, said "the payload is still in S3," and only sent >64 KB payloads to S3. Inline payloads live in Kafka, and nobody runs a 30-day-retention events topic at 4 TB/day x RF3 as their payload store. The fix is a payload archive that covers everything: an archiver consumer group reads the events topic and packs payloads into batched segment objects in S3 — append-log style — because the naive alternative, 1B individual 4 KB PUTs/day, is roughly $5k/day in request costs alone before you talk about GET amplification. Every attempt row carries a payload_ref resolving into those segments (mechanics sharpened in Q8 below), the UI and the redeliver endpoint do S3 range-GETs, and the archiver runs in both regions against their local copy of the log, so the archive survives a region like everything else.

Retention and deletion: 30-day lifecycle expiry on the archive bucket, matching the ClickHouse TTL, with SSE-KMS on the segments. Segments are mixed-tenant, so a customer deletion request is handled by immediately tombstoning the ClickHouse rows — the payload becomes unreachable through every API the same day — with physical bytes gone at segment expiry, at most 30 days later. If a compliance regime demands faster physical erasure, the upgrade path is per-customer envelope keys and crypto-shredding, which I'd only pay for when a contract actually requires it.

Design change: all payloads archived to S3 in batched segment objects with a 30-day lifecycle; attempt rows carry a payload_ref into the archive; redelivery reads from the archive, not Kafka.

Round 2 — pushback

Q7 (Codex): Your standby has already committed old events with emission gated, yet promotion rewinds only minutes—when Region A dies, how do you recover a retry for an event accepted 20 hours ago that exists only in A's retry topic instead of silently abandoning its 24-hour delivery window?

You're right, and my Q2 answer made this worse, not better. The original doc claimed parked retries "replay from the mirror as fresh deliveries" — that only works if failover replays the whole 24-hour mirror. The hot-standby-with-minutes-rewind I introduced in Q2 commits past those events without ever emitting anything for them in B, so a delivery parked 20 hours deep in A's retry topics gets silently abandoned. That's not a duplicate problem; it's a lost delivery obligation, and "accepted events survive a regional failure" has to cover the retry state, not just the raw event.

The fix is to make parked state cross-region the same way accepted events are: when a delivery worker parks a task, it produces it to the local retry bucket and to a single pending-mirror topic on cluster B (one flat topic, carrying the task plus its next not_before — B doesn't need the bucket structure until promotion). Parking isn't latency-sensitive, so the cross-region produce with acks is an acceptable cost, and tasks are ~200-byte pointers, so the volume is noise next to the event mirror we already pay for. Workers also emit best-effort resolved(delivery_id) tombstones to the same topic on success. On promotion, a seeder drains pending-mirror, drops tombstoned delivery_ids, and re-buckets the remainder into B's retry topics with their original deadlines intact. Tombstones are best-effort, so some deliveries that succeeded in A's final seconds replay — duplicates, covered by the webhook-id contract, and bounded to the tombstone lag rather than 24 hours of state. The events-mirror rewind stays at minutes; the retry mirror carries everything older.

Design change: parked retry tasks are dual-written to a pending-mirror topic in region B with best-effort resolution tombstones; failover seeds region B's retry buckets from it, preserving original 24 h deadlines. The original claim that A's retry state is "acceptable loss" is withdrawn.

Q8 (Codex): Your independent archiver determines (segment_key, offset, length) only while batching objects, so how can the concurrently running matcher stamp that reference without synchronously joining on an archive result—and what prevents delivery history from pointing to no payload if either side crashes?

Fair catch — as I stated it in Q6, the matcher stamps a byte address that doesn't exist yet. The fix is to make the reference deterministic instead of assigned: payload_ref is (kafka_partition, kafka_offset), which the matcher knows for free the moment it consumes the event — no join, no coordination. The archiver's segment keys encode the partition and starting offset (events/{partition}/{base_offset}.seg), and each segment carries a footer index mapping offset to byte position. Resolving a ref is: derive the segment key from the offset range, range-GET the footer, range-GET the payload. This is just tiering the Kafka log into S3 — the log's own addressing scheme is the reference, so history and archive can't disagree about where a payload lives.

Crash consistency falls out of commit ordering: the archiver commits its consumer offset only after the segment PUT succeeds, so the invariant is "everything at or below the archiver's committed offset is durably in S3." A crash mid-segment re-uploads the same deterministic key with the same content — idempotent overwrite. The remaining gap is young events: an attempt row can land in ClickHouse minutes before its segment does. The read path handles that with a watermark check — refs above the archiver's committed watermark are served straight from the events topic (retention of a few days dwarfs archiver lag measured in minutes), refs below it from S3. A ref can therefore be transiently served from Kafka but never permanently dangle. One honest limitation: refs are region-scoped (B's mirror has different offsets than A's log), which is fine because history serving fails over as a unit — post-failover, B's pipeline stamps B-scoped refs into B's ClickHouse, and A's rows died with A's region just like the rest of A's serving stack.

Design change: payload_ref becomes (partition, kafka_offset), resolved via deterministically named archive segments with offset-indexed footers; archiver commits offsets only after segment upload; reads above the archive watermark fall back to the events topic.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The delivery contract in this design — webhook-id / webhook-timestamp / webhook-signature, with the signature computed as v1,base64(HMAC-SHA256(secret, id.ts.body)) — is not just Stripe-flavored convention anymore; it is byte-for-byte the Standard Webhooks specification, the cross-vendor spec Svix and others published. The spec pins down the details this design also landed on: the id stays constant across retries and serves as the consumer's idempotency key, the timestamp is signed so replay rejection can't be stripped, and rotation works by sending multiple space-delimited signatures during an overlap window. Stripe's own docs use a different header (Stripe-Signature) but the same construction — HMAC-SHA256 over timestamp.payload, a 5-minute default replay tolerance, and secret rolls that keep the old secret live for up to 24 hours. The design's 5-minute tolerance and dual-signature rotation window are the industry numbers, not inventions.

The at-least-once decision is universal, and the reasoning is published. Hookdeck's delivery-guarantees guide walks the four ambiguous-failure cases (delivered-but-ack-lost vs. actually lost) and lands where this design does: what vendors market as exactly-once is at-least-once plus an idempotent consumer keyed on the event id. Stripe tells consumers to log event IDs and skip duplicates, retries with exponential backoff for up to three days, and explicitly does not guarantee event ordering. The retry-window spread across providers is wider than I expected when writing the design: Stripe retries for 3 days, Svix's schedule runs up to 2 days, and GitHub doesn't automatically retry at all — it records the failure (anything over 10 seconds counts, same timeout this design chose) and gives you a 30-day manual redelivery window. This design's 24-hour automatic window plus 30-day manual redelivery sits between Stripe and GitHub, which I'm comfortable with.

On ordering, Svix wrote up exactly the tradeoff the design's ordered flag encodes. Their FIFO webhooks post quantifies why strict ordering isn't the default anywhere: one failed delivery blocks the endpoint's whole stream, and even a healthy serialized stream caps out around 20 messages/second on network latency alone. Their answer — best-effort by default, opt-in FIFO endpoints for customers who accept the throughput cost — is the same per-endpoint setting this design documents. Their architecture post also validates the broader shape: a persistent queue between API and dispatch workers, customer-configurable rate limits, honoring 429s from endpoints as a smoothing signal (a refinement worth adopting — this design's token buckets are provider-set, and letting the endpoint push back via 429 is cheap to add), and running the whole webhook system in its own VPC with its own database, citing GitHub's history of webhook-load incidents as the reason.

The SSRF section matches what senders who've audited this in production publish. PlanetScale's webhook security post layers URL validation (HTTPS-only, private/loopback IP blocks, internal-domain blocklists) under an egress proxy that can only reach external networks, on the stated principle that "no matter how rigorous your URL validations are, you cannot fully trust any URL provided by a user." The resolve-then-pin rule in this design — validate the resolved IP and connect to that exact IP — is the specific fix for the DNS rebinding attack Ameya Lokare documented after finding PagerDuty's webhook sender vulnerable to it: the attacker's DNS returns a public IP for the validation lookup and a private IP for the client library's second lookup. Checking at resolve time without pinning is precisely the broken pattern.

One place the design is deliberately heavier than most published practice: the dual-region synchronous write for RPO=0. None of the provider write-ups above claim that guarantee — Svix, Stripe, and GitHub publish retry and dedupe contracts, not cross-region durability protocols. That asymmetry is worth knowing: the regional-survival requirement in this prompt is the part you won't find a reference architecture for, which is consistent with it having consumed most of the interview follow-ups.

Updates from post-training information

Apache Kafka 4.2.0 shipped on February 17, 2026, and its release announcement declares share groups (KIP-932, "Queues for Kafka") production-ready: multiple consumers processing the same partition concurrently, with per-record acknowledgment and broker-tracked delivery counts. That lands directly on this design's most intricate piece of worker machinery. The per-partition low-watermark commit from Q3 exists because consumer-group offsets are a single high-water line over out-of-order per-endpoint completion; share groups replace that with per-record acks the broker tracks, and the delivery count gives poison-task detection for free. If I were building the delivery tier on Kafka 4.2, I'd seriously consider share groups for the deliveries topic and delete the watermark logic.

Two caveats before treating that as a rewrite. Gunnar Morling's KIP-932 walkthrough (written against the 4.0 early-access version) flags that share groups have no delayed-redelivery support — a released record comes back immediately — so the tiered retry topics with fixed per-bucket delays survive untouched; share groups don't schedule, they ack. And share groups give up per-partition ordering, so endpoints with the ordered flag would stay on a classic consumer group. The honest framing: this is a post-training simplification of the Q3 machinery, not a correction of it — the watermark design remains what you'd build on any Kafka before 4.2, which is most clusters running today.

Further reading