Environment Configuration and Deployment Notifications

Design for the config + notifications subsystem of a developer platform, as I'd walk through it in a 60-minute interview.

My recommendation up front: model config as immutable versions in a relational OLTP store (PostgreSQL), materialize a content-addressed snapshot per deploy so a deploy sees exactly one config version, encrypt secret values with envelope encryption backed by a managed key service (AWS KMS), and drive both the shared-group fan-out and all customer notifications off a durable, partitioned event log (Kafka) fed by a transactional outbox. Webhook delivery gets its own delivery ledger in Postgres — not raw Kafka consumers — because per-endpoint retry schedules and customer-visible replay need random access to individual deliveries, and a log doesn't give you that.

Assumptions

The prompt fixes most of the scale; here's what I'm adding:

Deploy frequency: ~2 deploys per service per day on average → ~1M deploys/day, ~12/s average, call it 100/s at peak. Config reads are deploy-driven, so this is the read rate that matters.
Config writes are human-paced: maybe 200k config changes/day platform-wide, ~5/s peak. Writes are never the bottleneck; fan-out is.
The largest shared environment group is linked to ~5,000 services (a big org's shared datastore credentials). That's the blast-radius case I design for.
Events: 5M/day ≈ 58/s average. Deploys cluster around business hours and incident storms, so I size delivery for 10× average: ~600 events/s, ~1,200 deliveries/s with 2 subscriptions each. This is small — the hard part of notifications is reliability and observability, not throughput.
Values are small: config entries ≤ 32KB, a full resolved config ≤ 1MB.
Single-region control plane to start, with deploy runtimes in multiple regions pulling from it. I'll note the multi-region path at the end.

Requirements, prioritized

Functional: versioned config with per-service entries and shared groups; atomic snapshot per deploy; one-action rollback; secrets encrypted, permission-gated, never logged; full audit of reads and changes; lifecycle events to webhooks/Slack/email with visible, replayable deliveries; the platform's own automation consuming the same events.

Non-functional, in the order I'd defend them:

Correctness of the snapshot. A deploy that sees a half-applied group change can take down a service. This is the one property I won't trade.
Secret confidentiality. A leak here is an existential incident for a platform whose whole pitch is "trust us with your prod credentials."
Blast-radius control on fan-out. Restarting 5,000 services because someone fixed a typo in a group is how you cause the outage you exist to prevent.
At-least-once event delivery with dedup handles. Exactly-once to an arbitrary HTTP endpoint is a myth; at-least-once plus an idempotency key is the honest contract.
Availability of the control plane (99.95%): a config-service outage blocks deploys, but running services keep running — the data plane must never depend on the control plane at runtime.

Explicitly out of scope: the deploy pipeline itself, service health checking (I consume its events), billing.

Architecture

flowchart LR
    subgraph Control plane
        API[Config API] --> PG[(PostgreSQL\nversions, snapshots,\naudit, deliveries)]
        API --> SEC[Secrets service]
        SEC --> KMS[KMS]
        PG -- outbox --> KAFKA[(Kafka\nlifecycle + config events)]
        KAFKA --> FAN[Fan-out coordinator]
        FAN --> ORCH[Deploy orchestrator]
        KAFKA --> SUB[Subscription matcher]
        SUB --> PG
        DW[Delivery workers] --> PG
        DW --> WH[Customer webhooks]
        DW --> SLACK[Slack / email adapters]
        KAFKA --> AUTO[Platform automation]
    end
    subgraph Data plane
        ORCH --> RT[Deploy runtime]
        RT -- fetch snapshot by id --> API
        RT -- decrypt via --> SEC
    end
    UI[Dashboard] --> API

Everything customer-visible flows through one Config API; the secrets service is a separately deployed, separately credentialed component so that a compromise of the general API tier doesn't yield plaintext secrets. Kafka carries two kinds of traffic on separate topics: config-changes (drives fan-out) and lifecycle-events (drives notifications and platform automation). Same log, same guarantees, different consumers.

Config data model: immutable versions

Nothing is ever edited in place. Every change creates a new version row; "current" is a pointer.

env_group(id, org_id, name, current_version)
env_group_version(id, group_id, version_num, parent_version_id,
                  created_by, created_at, change_note)
env_entry(id, group_version_id NULLABLE, service_version_id NULLABLE,
          key, is_secret,
          value_plain NULLABLE,          -- non-secrets only
          value_ciphertext NULLABLE,     -- secrets: AES-256-GCM
          dek_id, value_sha256)

service_config(service_id, current_version)
service_config_version(id, service_id, version_num, parent_version_id,
                       created_by, created_at)
service_group_link(service_version_id, group_id, precedence)

config_snapshot(id,             -- sha256 of canonical resolved content
                service_id, service_version_id,
                resolved_entries JSONB,   -- ciphertext for secrets, never plaintext
                group_versions JSONB,     -- {group_id: version_id} actually used
                created_at)

deploy(id, service_id, snapshot_id, image_ref, status, ...)

Two things worth defending. First, entries are stored per-version (copy-on-write of the changed rows, shared rows referenced from the parent in practice — I'd start with full copies because 30 entries × a few hundred versions per service is nothing: even 500k services × 300 lifetime versions × 30 entries is ~4.5B small rows, which partitioned Postgres handles, and full copies make reads a single indexed scan with no version-resolution logic). Simplicity wins until storage says otherwise.

Second, services link to groups unpinned by default — the link says "group X," and resolution to a concrete env_group_version happens at snapshot time. That's what makes a group change propagate. A customer who wants stability can pin a link to a version; then group changes don't touch them until they re-pin.

value_sha256 exists so the diff view and audit log can say "SECRET_KEY changed" and detect no-op writes without ever holding plaintext.

Versioning model summary: groups and per-service config version independently; a snapshot records exactly which version of each it resolved; the version graph is a chain with parent_version_id, so history, diffs, and rollback are all walks of that chain.

Atomic snapshots: a deploy sees exactly one config

At deploy trigger, the API resolves a snapshot in one Postgres transaction at REPEATABLE READ: read the service's current version, each linked group's current version, merge entries by precedence (service-level entries beat groups; groups ordered by explicit precedence; duplicate keys at equal precedence are a validation error at link time, not a silent override), canonicalize, hash, and insert the config_snapshot row. The transaction guarantees no torn read of a group mid-change — a concurrent group commit either fully lands before the snapshot's MVCC view or fully after.

The deploy record carries snapshot_id and nothing else config-related. The runtime fetches the snapshot by id — an immutable, cacheable-forever object — and asks the secrets service to decrypt its ciphertexts. If a group changes one millisecond after the snapshot is cut, that deploy is untouched; the change produces a new fan-out intent (below). Restarts without a new image reuse the identical mechanism: cut a snapshot, restart against it. There is exactly one code path that binds config to running processes, and it always goes through a snapshot id.

Content-addressing (snapshot id = hash of resolved content) gives free dedup — a "restart to pick up a change" that resolves to identical content is a detectable no-op and gets skipped — and makes "what exact config is this instance running?" answerable by comparing two ids.

Validation before apply

Writes are two-phase at the API level: propose, then commit.

POST /groups/:id/versions with the entry changes creates a draft version and runs validation: key-name rules, size limits, type checks, and — the important one — link-impact analysis: for every linked service, dry-run the merge and report new duplicate-key conflicts, and report the fan-out size ("this will restart 4,812 services").
The response is a diff plus the impact report. POST /versions/:id/commit flips current_version and emits the change event. The dashboard makes step 2 an explicit confirm when fan-out exceeds a threshold.

Customers can attach a per-service required-key manifest ("this service needs DATABASE_URL") checked at snapshot time; a snapshot that fails it fails the deploy before anything restarts, with a precise error. I rejected arbitrary customer validation webhooks in v1 — a synchronous callout in the write path couples config commits to customer uptime — but the draft/commit split leaves room to add them as an async check on drafts later.

Secrets

Encryption. Envelope encryption: each org gets a data-encryption key (DEK); secret values are AES-256-GCM-encrypted with the DEK; the DEK is stored only wrapped by the org's key in KMS. Postgres holds ciphertext and wrapped DEKs; KMS holds the root keys and never sees data. Per-org DEKs (not per-value) keep KMS call volume sane — the secrets service caches unwrapped DEKs in memory for 5 minutes, so a KMS blip degrades to "decryption keeps working from cache, new orgs' first decrypt fails" rather than "all deploys fail." Deploys fail closed if decryption is truly unavailable: launching with missing secrets is worse than not launching.

Rotation. Three layers, three answers. KMS root key rotation is KMS-native and transparent (old ciphertext stays decryptable). DEK rotation is a background job: mint a new DEK, re-encrypt rows org by org, tracked by dek_id per entry — no downtime, no version change, because the plaintext didn't change. Rotating the secret value itself is a customer action and is just a normal config change: new version, new snapshot, fan-out — which means secret rotation inherits audit, diff, and rollback for free.

Never logged. Policy plus mechanics, because policy alone fails:

Plaintext exists only inside the secrets service and the deploy runtime's process env. The general API tier, workers, and dashboard backend never hold it.
API responses return secrets as {key, is_secret: true, value_sha256} — write-only by default. The only path to plaintext is POST /services/:id/secrets/:key/reveal, which requires a distinct permission and writes an audit event in the same transaction as the read — no audit row, no reveal.
In the secrets service codebase, plaintext lives in a wrapper type whose String()/serialization yields [REDACTED], so a stray log statement or error dump can't leak it. Logging middleware additionally redacts by field name as a second net.
Event payloads, webhooks, audit rows, and snapshots-at-rest carry key names and hashes only. Snapshot resolved_entries stores ciphertext for secret keys.

Access control. RBAC with the crucial split: config:read (see keys and non-secret values), config:write (change anything — you can set a secret without being able to read one), secret:reveal (see plaintext, granted narrowly, every use audited), plus group:link gated on both sides so a random service can't attach itself to a group and siphon its secrets. Deploy runtimes authenticate with short-lived per-service identities (SPIFFE-style workload certs) and can decrypt only snapshots belonging to their service — the secrets service checks the snapshot→service binding, not just "is this a runtime."

Audit

An append-only audit_log table: (id, org_id, actor, actor_type, action, target, key_name, value_sha256_before/after, request_id, at). Config commits, link changes, secret reveals, subscription changes, and rollbacks all write it transactionally with the action itself, so audit can't lag or drop the write it describes. Rows also flow to Kafka for customer SIEM export and get archived to an object store (S3) with per-day manifest hashes for tamper evidence. I skipped full hash-chaining of individual rows — the threat model (compliance attestation, not Byzantine insiders) doesn't justify the write-path cost; daily manifest hashes over immutable archives cover "prove the history wasn't edited later."

"Every config read attributable" needs care at deploy volume: human reads and reveals are audited individually; runtime snapshot fetches are audited as one row per (deploy, snapshot) rather than per key, which is the semantically meaningful unit and keeps volume at ~1M rows/day instead of 30M.

Fan-out: when a shared group changes

The commit of a group version emits one group-changed event through a transactional outbox (an outbox table written in the commit transaction, relayed to Kafka by a poller — this closes the "committed to Postgres but never told Kafka" gap). The fan-out coordinator consumes it and expands it to per-service restart intents in a pending_apply table:

pending_apply(service_id PRIMARY KEY, group_version_id, enqueued_at, wave, state)

service_id as primary key is the load-bearing choice: fan-out is level-triggered, not edge-triggered. If a group changes three times in a minute — someone fixing a typo of a typo — a service that hasn't restarted yet gets its pending row updated, not tripled. The reconciler converges each service to the latest desired config exactly once. This coalescing is what turns "thrashing customer" from an incident into a non-event.

The deploy orchestrator drains pending_apply under blast-radius controls:

Waves with a canary. Restart 1% (min 5 services) first, wait for health checks, then proceed in exponentially growing waves. A wave failure rate above threshold pauses the fan-out and pages the customer — the group version stays current, remaining services stay on the old snapshot (which is safe: they're running exactly what they ran before), and the customer resolves it with a rollback or a forced continue.
Per-customer and per-cluster concurrency caps enforced via a shared counter store (Redis), so one org's 5,000-service fan-out can't starve the deploy fleet or overwhelm a shared cluster's scheduler. At a cap of 50 concurrent restarts and ~30s per restart, 5,000 services drain in ~50 minutes — deliberate, and adjustable per org.
Customer policy per link: auto-restart (default), manual (mark stale, show a banner, customer triggers), or next-deploy (pick up the change whenever the next deploy happens). Plus maintenance windows for the cautious.

Each restart is an ordinary snapshot-cut-then-restart, so ordering, atomicity, and audit need no special cases here.

Notifications

Producing. Deploy and health subsystems write lifecycle events through the same transactional outbox pattern into the lifecycle-events Kafka topic, partitioned by service_id so events for one service are ordered in the log. Each event: event_id (UUIDv7 — time-ordered, globally unique), service_id, sequence (per-service monotonic counter), type, occurred_at, payload with resource ids and — for deploy events — the snapshot_id, tying the two halves of this design together. Platform automation (auto-rollback on failed health checks, the fan-out coordinator itself) consumes these topics directly as ordinary Kafka consumer groups; customers and automation see the same events, which is the point.

Matching and the delivery ledger. A subscription matcher consumes the topic, looks up matching subscriptions (indexed by org + event type + optional service filter, cached with pub/sub invalidation), and inserts one row per (event, subscription) into a delivery table with UNIQUE(event_id, subscription_id) — so a matcher crash and Kafka redelivery can't create duplicate deliveries. This table is the customer-visible ledger: state (pending/succeeded/failed/dead), attempt count, next retry time, last response code and truncated body, latency.

Why a Postgres ledger instead of driving delivery straight off Kafka? Because retries need per-delivery timers (retry delivery X in 4 minutes without blocking the partition behind it) and the product needs random access (show me this delivery, replay it) — both awkward on a log, natural on a table. Kafka gives ordering and fan-in; Postgres gives per-item state. At ~10M delivery rows/day, partition by day and retain 30 days hot, older in S3.

Delivering. Workers claim due rows with FOR UPDATE SKIP LOCKED, then per attempt:

Sign the body with a per-subscription HMAC secret (X-Signature, timestamped to block replay), include X-Event-Id and X-Delivery-Id so consumers can dedupe under our at-least-once contract.
10s timeout; only a 2xx counts. Retry on failure with exponential backoff + jitter: 30s, 2m, 10m, 30m, then hourly to 24h (~30 attempts), then dead.
Per-endpoint circuit breaker: an endpoint failing hard gets its pending deliveries deferred as a group rather than each burning attempts — routine endpoint failures are the norm, per the prompt, so the system treats a down endpoint as one fact, not a thousand independent ones. Auto-disable endpoints dead for 7 days, with an email.
Egress through dedicated NAT IPs (published, so customers can allowlist) and an SSRF guard blocking internal address space — customers control the target URL, so the delivery worker is an attacker-reachable HTTP client and must be treated as one.

Ordering. In the log, per-service order is exact. Over HTTP, retries make strict delivery order impossible without head-of-line blocking — if deploy-started is retrying for 30 minutes, holding deploy-succeeded hostage behind it is strictly worse. So the contract is: deliveries are attempted in order per service, may arrive out of order under retry, and every payload carries sequence and occurred_at so a consumer can reorder or drop stale events. Slack and email adapters are just alternative senders off the same ledger with provider-specific rate limiting and message formatting; a Slack outage queues in exactly the machinery webhooks use.

Replay. POST /deliveries/:id/replay clones the row with a new delivery_id and the original event_id — the consumer's dedup key still works. Bulk replay ("everything to this endpoint since 14:00") is a filtered scan of the ledger. Since the ledger is the source of truth for the customer-facing history, replay needs no Kafka rewind at all.

APIs (the load-bearing subset)

POST /groups/{id}/versions              # draft: changes → diff + impact report
POST /group-versions/{id}/commit        # flips current, emits group-changed
POST /group-versions/{id}/rollback      # one action, see below
GET  /services/{id}/config              # keys + values (secrets masked)
POST /services/{id}/secrets/{key}/reveal
POST /services/{id}/snapshots           # cut snapshot (deploy pipeline calls this)
GET  /snapshots/{id}                    # immutable, runtime fetch
GET  /services/{id}/config/history      # version chain + diffs + who/when
POST /subscriptions                     # {event_types, filter, channel, url|slack|email}
GET  /deliveries?subscription&status&since
POST /deliveries/{id}/replay

One-action rollback

POST /group-versions/{id}/rollback (and the service-level twin) creates a new version whose content equals the version before the bad one — roll-forward-as-revert, like git revert. History stays append-only and honest ("v14: revert of v13, by alice, 14:02"), the audit trail never rewrites, and the change propagates through the exact fan-out machinery as any commit — canaries and all, though rollbacks default to an accelerated wave schedule since the old config is known-good. One HTTP call, one dashboard button.

Deploy-level rollback is even cheaper: redeploy the previous deploy record, which pins both the old image and the old snapshot_id — image and config roll back as a unit, which matters because a config change often ships to match a code change.

I rejected "flip current_version back to the old row" — it's tempting because it's one UPDATE, but it makes history non-linear (what was current when?) and breaks the invariant that every state the system was ever in is a version on the chain.

Failure modes

Postgres primary loss. Synchronous-replica failover, ~30s of failed config writes and snapshot cuts. Running services unaffected — the data plane holds its config in process env and never reads the control plane at runtime. This isolation is the design's most important availability property.
KMS outage. DEK cache carries existing orgs for its TTL; beyond that, deploys fail closed with a clear error. Non-secret-only deploys could proceed, but I'd keep the simpler invariant: no snapshot decrypts partially.
Kafka down. Outbox rows accumulate in Postgres (bounded by disk, alert at depth); on recovery the relay drains in order. Events delayed, never lost. Fan-out and notifications both stall gracefully.
Fan-out gone wrong. Canary wave catches config that kills services; pause-and-page bounds the damage to the wave size. The failure the canary can't catch — config that passes health checks but corrupts behavior — is what one-action rollback is for.
Poison event / poison endpoint. A payload that crashes the matcher goes to a dead-letter topic after N attempts rather than wedging the partition; endpoint failures are absorbed by the ledger and breaker, never back-pressuring Kafka consumption.
Duplicate everything. Outbox relay, matcher, and workers are all at-least-once; dedup lives at three seams: Kafka producer idempotence, the UNIQUE(event_id, subscription_id) constraint, and the customer-side X-Event-Id contract.

What I rejected, and why

Git as the config store. Free versioning and diffs, but a terrible secrets story (ciphertext blobs in git make rotation and access control miserable), no transactional multi-entity snapshot, and poor latency for a synchronous deploy path.
A dedicated secret manager (Vault) as the primary store. Excellent secret handling, but I'd still need versions, groups, snapshots, and links in Postgres, leaving Vault as a second source of truth to keep consistent. Envelope encryption inside my own model gives the properties I need with one source of truth. Vault-style dynamic credentials are a plausible v3 feature, not a foundation.
A distributed KV store (DynamoDB/etcd) for config. The snapshot cut is a multi-row transactional read and the impact analysis is a relational query; giving up transactions and joins to gain write scale I don't need (5 writes/s!) is backwards.
Pure-Kafka webhook delivery (consumer per endpoint, or delay-topic retry tiers). Topic explosion, coarse retry granularity, and no story for "show me and replay this one delivery." The ledger costs a table; it buys the product.
Push config to running services without restart. Genuinely useful (feature flags live there) but it breaks the snapshot invariant — a process would run a config no deploy ever pinned — and doubles the runtime contract. It's the natural evolution once the versioned core exists: a dynamic: true entry class delivered via a watch channel, still snapshot-versioned, opt-in per key.

Evolution

Multi-region: the control plane stays single-writer; regional read replicas + a snapshot cache (snapshots are immutable, so caching is trivial and safe) let deploys in any region proceed through a control-plane partition of the read path. Later: config-as-code (declare groups/links in a repo, reconciled through the same draft/commit API — the API was designed so a file-based frontend bolts on), policy checks on drafts (OPA-style rules like "no plaintext entries matching key pattern *_KEY"), and customer-visible delivery SLOs computed straight off the delivery ledger, which already holds every attempt with timestamps.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): How can snapshot_id be a hash of resolved content when secrets use randomized AES-GCM ciphertext and DEK rotation re-encrypts them; and if identical hashes deduplicate across services, how does the secrets service preserve an unambiguous snapshot-to-service authorization binding?

This lands, twice. First: hashing ciphertext is broken — AES-GCM is randomized, so the same plaintext re-encrypted (or DEK-rotated) yields a different hash and destroys both dedup and "same id = same config." The fix is to hash canonical plaintext-derived content: for non-secrets, the value; for secrets, the stored value fingerprint. And that fingerprint can't stay a bare SHA-256 of the plaintext — low-entropy secrets are dictionary-attackable offline — so value_sha256 becomes an HMAC-SHA256 with a per-org fingerprint key held by the secrets service. The snapshot hash is then stable under DEK rotation (rotation touches ciphertext, not fingerprints), and diff/no-op detection still works.

Second: cross-service dedup was a mistake I shouldn't have implied. Snapshot rows are per-service — service_id goes into the hashed content, so the hash is unique per service and the secrets service's snapshot-to-service authorization check stays a plain column comparison. The dedup I actually want ("restart resolves to identical config, skip it") is within one service: compare the new content hash to the running snapshot's. Nothing is lost by not sharing rows across services; storage was never the constraint.

Design change: snapshot content hash computed over (service_id + canonical entries using per-org HMAC fingerprints for secrets), not ciphertext; value_sha256 becomes a keyed HMAC; dedup is per-service hash comparison, never cross-service row sharing. (Revised again in Q7: identity and hash split into two fields.)

Q2 (Codex): With pending_apply(service_id PRIMARY KEY), what prevents a group-change upsert arriving during an in-flight restart from being overwritten when that worker marks or deletes the row, thereby permanently losing the newer desired state—especially when several linked groups change concurrently?

Correct — a worker that reads the row, restarts the service, then deletes the row can erase a newer intent that arrived mid-restart. Two changes fix it, and they simplify the table. First, drop group_version_id from the row entirely: the target of a reconcile is never "apply group version X," it's "cut a fresh snapshot of whatever is current now." Storing a specific version was both the race and a design smell — with multiple linked groups changing concurrently it was ambiguous anyway. The row becomes a dirty flag: pending_apply(service_id PK, generation BIGINT, enqueued_at, wave, state). Second, generation is bumped on every upsert, the worker records the generation it claimed, and completion is a compare-and-swap: DELETE WHERE service_id = ? AND generation = ?. If a change landed mid-restart the delete no-ops, the row stays due, and the reconciler goes around again — converging on the latest config, which is the level-triggered behavior I claimed but hadn't actually mechanized.

Design change: pending_apply loses group_version_id, gains generation; completion is CAS-guarded on generation; the snapshot cut always resolves current versions, never a version captured at enqueue time.

Q3 (Codex): Your impact analysis runs at draft creation, but service configs, group links, precedence, and other group versions can change before commit; what commit-time concurrency control prevents approving a stale analysis and making the newly current group version unresolvable for thousands of services?

The draft report is advisory; the commit has to re-earn it. Two mechanisms. First, optimistic concurrency on the chain: a draft records its parent_version_id, and commit fails with 409 if current_version != parent — two concurrent drafts can't both land; the loser rebases. Second, commit revalidates inside the commit transaction: take SELECT ... FOR UPDATE on the group row, re-run the structural checks (duplicate-key conflicts against the currently linked services, fan-out count). This is affordable because it's the same relational query the draft ran, writes are ~5/s platform-wide, and even the 5,000-service case is an indexed join, not a scan. If the fresh result differs materially from what the user approved — new conflicts, fan-out grown past the confirm threshold — commit returns the fresh report and requires re-confirm.

And "unresolvable for thousands of services" has a backstop the design already carries: the merge is validated again at every snapshot cut, and a conflicting snapshot fails the deploy before anything restarts. Worst case is a precise deploy error, never a service running torn config.

Design change: commit requires parent-version match (409 on mismatch) and re-runs validation under a group-row lock in the commit transaction; material drift from the approved report forces re-confirm. (Tightened in Q8: link writes must take the same lock.)

Q4 (Codex): Deploy and health events come from independent subsystems, yet you require one gap-free monotonic sequence per service and claim exact Kafka order; where is that sequence serialized transactionally without creating a single availability bottleneck, and what happens when events are published late or out of order?

I over-claimed. A single gap-free counter per service, shared by deploy and health subsystems that own their own stores, would need a serialization point I don't want to build — a cross-subsystem counter service on every event's write path is exactly the availability bottleneck you're naming. So I'm cutting the claim to what the consumer actually needs, which is reorder-and-drop-stale, not gap detection.

Revised contract: sequence is scoped per (service, source stream) — deploy events carry a sequence assigned in the deploy orchestrator's own outbox transaction (its Postgres, its counter row, no new bottleneck), health events likewise in the health subsystem. Within a stream, ordering is exact: the sequence is assigned in the same transaction as the outbox row and the relay drains in order, so late or out-of-order publishing within a stream can't happen by construction. Across streams, ordering is occurred_at + UUIDv7, best effort — and that's not a cop-out, it's the truth of independent producers: no scheme short of a global sequencer gives more, and consumers correlate "deploy succeeded" with "service unhealthy" causally, by the deploy id and snapshot id in the payload, not by a global counter. Kafka's ordering claim narrows the same way: exact per partition per topic, i.e., per (service, stream).

Design change: per-service sequence replaced by per-(service, source-stream) sequences assigned transactionally in each producer's outbox; cross-stream ordering documented as timestamp-based best effort with causal correlation via deploy/snapshot ids in payloads.

Q5 (Codex): Your rollback copies the version preceding the bad version, which is not equivalent to git revert: if valid changes were committed afterward, it silently removes them too; how would one-action rollback invert only the targeted change while detecting conflicts with subsequent edits?

Right — "copy the version before the bad one" is git reset wearing a git revert costume, and it's only correct in the special case where the bad version is still current. I'm keeping that fast path (it is the overwhelmingly common case: you notice a bad change minutes after making it) and fixing the general case. Rollback of version N when current is M > N computes N's inverse diff — the set of keys N touched, restored to their pre-N values or deletions — and applies it to M as a new version. If any version after N also touched a key in that set, the rollback blocks and returns a per-key conflict list; the user resolves explicitly (keep current / restore old) and commits the resolution — same shape as a git revert conflict. Still one action when there's no conflict; when there is one, silently picking a winner would be worse than asking. The audit note becomes precise too: "v14: revert of v13," not "v14: copy of v12."

Design change: rollback is inverse-diff application with per-key conflict detection; whole-version copy survives only as the degenerate case where the reverted version is still current.

Q6 (Codex): Do delivery workers hold a Postgres row lock and transaction open across a potentially 10-second HTTP call, or release it and risk concurrent duplicate attempts; and if replay retains the original event_id, how can an idempotent customer distinguish an intentional replay from a duplicate and actually process it again?

On the lock: no, and I should have spelled the mechanism out. FOR UPDATE SKIP LOCKED covers only the claim query. Claiming is a short transaction: set state = in_flight, lease_expires_at = now() + attempt timeout + slack, commit, release the lock. The HTTP attempt runs with no database transaction open; a second transaction records the outcome. A worker that dies mid-attempt leaves a lease that expires, and a reaper returns the row to pending — which can produce one duplicate attempt, and that's fine: duplicate attempts are inside the at-least-once contract the customer already holds. Holding a row lock across a 10-second call at 1,200 deliveries/s would mean ~12,000 open transactions; the lease costs one column.

On replay: the premise is half wrong, because the design already carries both keys, but the contract needs to be explicit. Keeping the original event_id on replay is deliberate — the dominant replay case is "endpoint was down, deliveries went dead, replay them," where the customer never processed the event and their X-Event-Id dedup correctly lets it through. For the other case — "I processed it but lost the result, force reprocessing" — the customer keys on X-Delivery-Id, which is fresh on every replay. So the documented contract is: dedupe on X-Event-Id to suppress everything including replays; dedupe on X-Delivery-Id to let deliberate replays through. I'll also add an explicit X-Replayed: true header so a consumer can branch on intent directly.

Design change: delivery claim uses a lease (in_flight + lease_expires_at, reaper on expiry) instead of a transaction held across the HTTP call; replay semantics documented as the two-key contract, plus an X-Replayed header.

Round 2 — pushback

Q7 (Codex): Your snapshot ID hashes only resolved content, so two different version combinations—or a revert—can produce the same ID while requiring different group_versions provenance. How can one immutable snapshot row truthfully audit both deploys without separating snapshot identity from the content hash?

You can't, and I concede the split. A revert is the everyday counterexample: v12 and v14 (revert of v13) resolve to byte-identical content but were cut from different version combinations at different times by different actors — one row can't hold both provenances, and content-as-identity forces exactly that. So: snapshot_id becomes a surrogate UUID, one row per snapshot cut, carrying its own provenance (group_versions, service_version_id, created_at, and the triggering actor); content_hash becomes a separate column. Everything I wanted from content-addressing survives in the hash column — no-op restart detection ("new cut's hash equals the running snapshot's hash, skip"), and "are these two instances running the same config?" is a hash comparison. What identity-as-UUID buys is that deploy.snapshot_id now joins to exactly one truthful provenance record, which is what the audit requirement actually demands. Content-addressing was an aesthetic I let overreach into the audit domain, where identity has to mean "this cut," not "this content."

Design change: config_snapshot splits identity from content: snapshot_id is a per-cut UUID, content_hash a separate column; dedup and comparability use the hash, deploys and audit use the id.

Q8 (Codex): You lock only the group row during commit, but concurrent service–group link changes can still alter the impacted-service set while validation runs. What exact locking or isolation protocol prevents commit from approving a stale impact analysis?

The exact protocol is a lock-ordering rule: every write that can change a group's impacted set takes FOR UPDATE on that group's row first. That means link create/remove and precedence changes lock the group row they touch, not just the service side. Commit takes the same lock, so while its validation query runs, the set of links into that group is frozen — a concurrent link change blocks until commit finishes, then proceeds and gets validated against the newly current version. Operations touching multiple groups (linking a service to three groups atomically) acquire group locks in group-id order, which rules out deadlock. The cost is a per-group mutex on human-paced writes — at ~5 writes/s platform-wide, contention on any single group is negligible, and the lock is held for one validation query, not the draft's lifetime.

And I'll restate where correctness actually lives, because the locking is UX, not the safety net: the invariant "no service ever runs torn or conflicting config" is enforced at snapshot cut, which re-validates the merge inside its own REPEATABLE READ transaction and fails the deploy before anything restarts. A link that lands one millisecond after commit is not a stale-analysis bug — it's a newer change, validated on its own write path against the now-current group version. Commit-time validation guarantees the impact report was true at the moment of commit; snapshot-time validation guarantees nothing wrong ever reaches a running process. Two layers, and the second one is the load-bearing one.

Design change: lock-ordering protocol — link and precedence writes take FOR UPDATE on the affected group row(s), acquired in group-id order; commit's validation therefore runs against a frozen link set.

What changed, summarized

Snapshot identity split from content: snapshot_id is a per-cut UUID with full provenance; content_hash is a separate column computed over plaintext-derived canonical content including service_id; dedup is per-service hash comparison, never cross-service row sharing.
Secret fingerprints: value_sha256 replaced by HMAC-SHA256 with a per-org key held by the secrets service, closing the offline-dictionary hole and making hashes stable under DEK rotation.
pending_apply: dropped group_version_id, added generation; completion is a CAS delete on (service_id, generation); reconcile target is always "current config," never a captured version.
Group commit: requires parent-version match (409 otherwise), revalidates under the group-row lock, forces re-confirm on material drift from the approved impact report.
Lock-ordering rule: link and precedence writes lock the affected group row(s) in group-id order, so commit validation sees a frozen link set.
Event ordering contract narrowed: per-(service, source-stream) sequences assigned in each producer's outbox transaction; cross-stream ordering is timestamp best-effort with causal ids in payloads.
Rollback generalized: inverse-diff with per-key conflict detection when later versions exist; whole-version copy only when the reverted version is still current.
Delivery workers: lease-based claim (in_flight + lease_expires_at + reaper) instead of a lock held across the HTTP attempt.
Replay contract made explicit: X-Event-Id dedup suppresses replays, X-Delivery-Id dedup admits them; added X-Replayed: true header.

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

Start with the product this prompt is modeled on. Render's own environment variables docs describe exactly the shape I designed: environment groups linked to any number of services, service-level values always beating group values, and "if you make changes to an environment group (including deleting it), Render kicks off a new deploy for every linked service that has autodeploys enabled" — that's my fan-out, with auto-restart vs next-deploy as the policy knob. One divergence I'd defend: for the same key defined in two linked groups, Render's docs say the most recently created group wins and warn "this behavior might change in the future without notice." My design makes that a link-time validation error instead, and I think the warning in their docs is the argument for my choice. Render's environment-scoped groups (Feb 2024) — restricting a group to one project environment so staging services can't link prod config — is the same instinct as my two-sided group:link permission.

The canonical paper here is Facebook's Holistic Configuration Management at Facebook (SOSP 2015). Their defenses against config errors are the same ones I reached for, in the same order: compiler-run validators on every change, config changes treated as code changes with mandatory review (my draft/commit split with an impact report), and an automated canary tool that "rolls out a config change to production in a staged fashion, monitors the system health, and rolls back automatically in case of problems" — my waves with pause-and-page, except they close the loop automatically. The big divergence: their entire paper is about live runtime config updated "multiple times a day, without application redeployment or restart," which is precisely the capability I rejected for v1 to protect the snapshot invariant. At Facebook's scale restart-to-apply is untenable; at this platform's scale it's an honest contract. LaunchDarkly's polling-to-streaming evolution shows what the road I deferred looks like fully built — SSE streams, a relay proxy, per-user evaluation at the edge. That's the mature form of my "dynamic: true entries via a watch channel" evolution note, and it's a whole company, which is why it's an evolution note.

On secrets, AWS Secrets Manager's encryption doc is the published reference for the envelope pattern I used: KMS generates a 256-bit AES data key, the plaintext key encrypts the value outside KMS and is dropped from memory, the wrapped key is stored alongside the ciphertext. They mint a fresh data key per secret value change where I chose per-org DEKs — they can afford a KMS call per write because KMS is in-house; I amortized to keep call volume and outage blast radius sane, and the tradeoff is a wider blast radius per key. The mechanism of theirs I'd actually steal: encryption context. KMS cryptographically binds SecretARN + SecretVersionId into every encrypt/decrypt, so the "this ciphertext belongs to this secret" check is enforced by the key service, not by a column comparison in my code — my snapshot-to-service binding should ride the same rails. And Vault's architecture doc states the stance I borrowed without adopting Vault: the storage backend sits outside the security barrier and is untrusted, so only ciphertext ever reaches it. Reading it is also the best defense of my "no Vault as primary store" call — Vault's value is the barrier and key hierarchy, and envelope encryption gives me both without a second source of truth to reconcile.

On webhooks, the design converged on what the payments industry ships. Stripe's webhook docs retry "for up to three days with an exponential back off," sign with timestamped HMAC-SHA256 to block replay, tell consumers to dedupe by logging event IDs — and state flatly that "Stripe doesn't guarantee the delivery of events in the order that they're generated." That's the same at-least-once, unordered, consumer-dedupes contract I defended in Q4 and the ordering section. Svix's idempotency docs describe a webhook-id that is "unique per message but is reused across retries of the same message" — exactly the two-key split from Q6, where the stable event ID suppresses duplicates and the fresh delivery ID admits deliberate replays. The Standard Webhooks spec (v1.0.0, from the Svix folks and a community committee) codifies all of it: webhook-id/webhook-timestamp/webhook-signature headers, HMAC-SHA256 over msg_id.timestamp.payload, timestamp tolerance for replay protection, multi-day retry schedules. I'd rename my X-Event-Id/X-Signature headers to the spec's — there's no reason to make consumers learn a dialect. On the producing side, Convoy's outbox write-up walks through the same dual-write reasoning that put a transactional outbox between my Postgres and Kafka.

For audit tamper evidence, CloudTrail's log file integrity validation is the grown-up version of my daily manifest hashes: hourly digest files carrying a SHA-256 per log file, each digest signed with RSA and containing the previous digest's signature — a chain of digests, not of rows. That's the middle ground I'd adopt if a compliance regime pushed past my daily manifests: chaining and signing the manifests costs almost nothing on the write path (it's off the transaction entirely) and buys "prove no window was silently dropped," which a bare per-day hash doesn't.