Contents

Build-and-Deploy Platform (Heroku-style)

What we're building, in one sentence

On every Git push, turn source into a container image, optionally run the customer's migrations, and roll the new version out behind their URL without dropping a request — at 2M deploys/day, with live logs, cancel, and rollback.

Assumptions I'm making up front

Requirements

Functional: push → build → (migrate) → deploy → live at URL; live log streaming for build and deploy; cancel in-flight; roll back to any deploy from the last 30 days; deploy history retained 30 days.

Non-functional, in priority order:

  1. Safety. A failed deploy leaves the previous version serving. This beats speed every time — the worst outcome for this product is the platform taking down a healthy service.
  2. Zero-downtime rollouts for healthy services.
  3. Correctness under concurrency. Several pushes in quick succession must converge on the latest commit, never interleave, never deploy an older commit over a newer one.
  4. Latency. Median push-to-live should stay near the 90-second median build; queue wait should be seconds, not minutes, even at peak.
  5. Isolation. A build runs arbitrary customer code. It must not be able to touch another customer's build, cache, or secrets.

Estimates that shape the architecture

Architecture overview

The design splits into a control plane (source of truth, orchestration), a build plane (isolated, elastic, cache-hungry), and a data plane (registry, runtime cells, edge) — with one principle repeated everywhere: running services must never depend on the control plane being up. The edge and the runtime cells serve from last-known-good state; a control-plane outage pauses deploys, nothing else.

flowchart LR
  GH[Git provider] -->|webhook| WH[Webhook ingest]
  WH --> CP

  subgraph CP[Control plane - primary region]
    API[Public API / SSE]
    PG[(Relational system of record - Postgres)]
    WF[Durable workflow engine - Temporal]
    SCHED[Build scheduler]
    API --- PG
    WF --- PG
    WF --- SCHED
  end

  subgraph BP[Build plane - per region]
    POOL[Warm pool of microVMs - Firecracker]
    BK[Build toolkit in VM - BuildKit]
    POOL --- BK
  end

  SCHED --> POOL
  BK -->|push image| REG[(OCI registry on object storage - S3)]
  BK -->|log stream| LOG[Log pipeline - Kafka -> S3 + live fanout]
  API -->|SSE tail| LOG

  subgraph DP[Runtime region - cells]
    CELL1[Runtime cell 1]
    CELL2[Runtime cell N]
    MIG[One-off migration job]
  end

  WF -->|rollout commands| CELL1
  WF --> MIG
  REG -->|pull-through cache| CELL1
  CELL1 --> EDGE[Programmable edge LB]
  EDGE --> USERS[Customer traffic]

Component choices and why

System of record: a relational database (Postgres). Services, deploys, config snapshots, and the deploy state machine live here. State transitions are transactional with optimistic version checks — "move deploy 123 from BUILDING to MIGRATING iff it's still BUILDING and not superseded" is one UPDATE. I rejected a distributed KV store (DynamoDB): write volume is modest (~500/s), and the transactional guards are exactly what prevent the concurrency bugs this system is prone to.

Orchestration: a durable workflow engine (Temporal). A deploy is a long-running, multi-step process — up to 30 minutes of build, then migration, then a gated rollout — that needs retries, timers, cancellation signals, and per-service serialization. That is the exact feature list of a workflow engine. I tried the alternative on paper first: a hand-rolled state machine on queues (SQS + worker pollers + a deploys.state column). It works until you need "cancel this build that's 20 minutes in," "time out the rollout after 15 minutes," and "when this finishes, start the newest queued push, skipping the three in between" — at which point you've rebuilt Temporal's timer and signal machinery, badly, inside your workers. The workflow engine is the single most load-bearing choice in this design.

Build isolation: single-use microVMs (Firecracker). A build executes arbitrary customer code — npm install runs whatever the repo says. Container isolation on a shared kernel isn't enough when a kernel exploit means reading another customer's build secrets, and a userspace-kernel sandbox (gVisor) still shares more surface than I want and complicates running BuildKit itself. One VM per build, booted from a golden snapshot in well under a second, destroyed after. Nothing is reused across customers except the read-only base image.

Build engine: a container build toolkit (BuildKit) inside the VM, driving either the customer's Dockerfile or a buildpack path (Nixpacks) for repos without one. Chosen over Kaniko for two features the requirements directly need: first-class remote cache import/export, and secret mounts that never enter an image layer.

Artifacts: an OCI registry backed by object storage (S3), fronted by per-region pull-through caches. Content addressing gives layer dedup for free — a 10GB ML image whose code layer changed pushes megabytes, not gigabytes. Runtime regions pull from a local cache so a rollout doesn't cross an ocean for blobs.

Runtime: cell-based container orchestration (Kubernetes cells). 500K services don't fit in one cluster and shouldn't: a cell is ~5K services, a bad cell upgrade or a runaway controller hits 1% of customers, not all of them. The workflow engine talks to a thin per-cell "rollout agent," so the control plane never holds cluster credentials for everything at once. I'd consider a custom bin-packing scheduler later for density (this is where the money goes at scale), but not in v1 — cells of a boring orchestrator are the version two people can operate.

Logs: a durable, partitioned event log (Kafka) feeding object storage, plus live fanout. Details below.

Data model

services(id, customer_id, repo, branch, region, cell_id,
         build_config, health_check_config, instance_count, ...)

config_snapshots(id, service_id, env_version, secrets_version, runtime_spec)
  -- immutable; a deploy pins one, so rollback restores code AND config together

deploys(id, service_id, seq,               -- seq: monotonic per service
        trigger,                            -- push | manual | rollback
        commit_sha, image_digest,
        config_snapshot_id,
        state, error, superseded_by,
        migration_ran boolean,
        created_at, started_at, finished_at)
  -- partitioned by day; partitions dropped after 30 days = retention for free

deploy_events(deploy_id, ts, phase, detail)          -- the timeline the UI shows
log_segments(deploy_id, stream, seq_from, seq_to, s3_key, bytes)
releases(service_id, live_deploy_id, edge_generation) -- what's serving right now

deploys.seq is the concurrency backbone: the platform never lets a deploy with a lower seq go live after a higher one has.

API sketch

POST /v1/webhooks/git              (HMAC-verified; idempotent on delivery id)
POST /v1/services/{id}/deploys     {commit_sha? | image_digest?}   -- manual deploy
GET  /v1/services/{id}/deploys     ?limit&cursor
GET  /v1/deploys/{id}              -- state, timeline, commit, digest
POST /v1/deploys/{id}/cancel
POST /v1/services/{id}/rollback    {to_deploy_id, run_migrations: false}
GET  /v1/deploys/{id}/logs         ?follow=1&from_seq=N   (SSE)

Internal interfaces I get to define on the external systems:

The deploy state machine

stateDiagram-v2
    [*] --> QUEUED
    QUEUED --> BUILDING : builder VM assigned
    QUEUED --> SUPERSEDED : newer push coalesced over it
    BUILDING --> BUILD_FAILED
    BUILDING --> CANCELED : user cancel / supersede policy
    BUILDING --> MIGRATING : image pushed, migrations configured
    BUILDING --> ROLLING_OUT : image pushed, no migrations
    MIGRATING --> MIGRATION_FAILED : job exit != 0 (old version untouched)
    MIGRATING --> ROLLING_OUT : job exit 0
    ROLLING_OUT --> LIVE : new instances healthy, edge shifted, old drained
    ROLLING_OUT --> ROLLOUT_FAILED : health gate / timeout -> auto-revert edge
    ROLLING_OUT --> CANCELED : cancel -> revert to previous
    LIVE --> RETIRED : a later deploy went LIVE

Every transition is a guarded transactional update in Postgres, driven by the workflow. Terminal failure states always leave releases.live_deploy_id pointing at the previous version — that invariant is checked, not assumed: the rollout phase never modifies the edge until the new version passes health, and any failure path re-asserts the previous edge generation.

Walking the pipeline

1. Ingest and coalescing — the rapid-pushes problem

Webhook lands, we verify the HMAC, dedupe on delivery ID, resolve the service, and signal a long-lived per-service workflow (workflow ID = svc-{id}, so the engine itself guarantees there's exactly one). That workflow is a serialization point and a coalescer:

The intuitive design here is a plain FIFO queue per service. It's wrong twice: it deploys stale commits one after another (wasted builds, and each rollout is a customer-visible restart), and it makes "cancel everything except the newest" a queue-surgery operation. Coalescing in a stateful workflow makes both trivial.

The per-service workflow also enforces the invariant that matters most: at most one deploy per service past the BUILDING state, and seq only moves forward at the edge. Even if a bug let two pipelines run, the edge update is guarded by generation, so an older deploy physically can't overwrite a newer one's traffic config.

Webhooks get a belt-and-suspenders poller: every few minutes, compare each active repo's head against the last seen commit, and synthesize a push if the webhook was lost.

2. Build — capacity, caching, secrets

Scheduling. The build scheduler assigns each build to a Firecracker VM from a regional warm pool, sized by time-of-day forecast plus queue-depth autoscaling; the 10x diurnal peak is predictable, so most capacity is pre-warmed rather than reactive. Two queues — interactive (default) and long-tail (builds that historically run >10 min) — so a 90-second build never waits behind a 30-minute one. Long-tail capacity can run on spot/preemptible instances; a preempted build retries, which is annoying but cheap.

Caching is what keeps the median at 90 seconds. Three layers:

  1. Cache-affinity scheduling: prefer placing a service's build on a host that built it recently, so BuildKit's local layer cache is warm on disk. (The VM is destroyed; the cache volume, encrypted per-service, is attached read-write to that service's next build only.)
  2. Remote layer cache: BuildKit exports cache manifests to the registry (--cache-to=type=registry) keyed per service+branch, imported on cache-miss hosts. Correctness never depends on cache — a total cache wipe means slow builds, not wrong builds.
  3. Dependency caches (node_modules, ~/.m2, pip) keyed by lockfile hash, on the same per-service volume — this is where most of the 90-seconds-vs-10-minutes difference lives for interpreted stacks.

Caches are strictly per-service. A shared cross-customer cache would be a cache-poisoning vector (customer A poisons a layer customer B imports); I rejected it even though it would cut cost.

Secrets during builds. Build-time secrets are envelope-encrypted in Postgres under a key-management service (KMS), decrypted only by the build scheduler at assignment time, and delivered to the VM over the authenticated placement channel into tmpfs. Inside the build they're exposed via BuildKit secret mounts — available to RUN steps, never written to a layer, never part of a cache key, gone with the VM. Additional guards: the VM has no cloud instance role and its egress blocks the metadata endpoint; the registry push credential is a per-build token scoped to that one service's repository with a 45-minute TTL; and the log pipeline scrubs exact matches of known secret values before storage (best-effort — we say so in the docs rather than pretend it's a guarantee).

Output: image pushed by digest, deploy row updated with image_digest, VM destroyed. Pushing by digest makes the step idempotent — a retried push of identical content is a no-op.

3. Migrations

If the service configures a migrate command, the workflow runs it as a one-off job in the service's runtime cell using the new image, before any traffic shift, holding a per-service lock so two migration jobs can never overlap. Exit 0 → proceed; nonzero → MIGRATION_FAILED, new instances never start, old version keeps serving untouched.

The honest tension: during the migration and the rollout, old code runs against the new schema. Zero-downtime makes that unavoidable — there is no instant where nothing is serving. So the platform's contract is expand/contract migrations (add the column now, drop the old one a deploy later), documented loudly, with migration_ran recorded on every deploy so rollback can reason about it. We do not run down-migrations automatically — reversing a schema change against live data is a decision a human makes.

The migration job is also the one step cancellation treats specially: killing a half-applied migration is worse than letting it finish. Cancel during MIGRATING waits for the job (bounded by its timeout), then stops.

4. Rollout — health checks and zero downtime

Create-before-destroy, gate-before-shift:

  1. Cell agent starts the new deploy's instances alongside the old (temporary 2x capacity for that service; at ~1% of services deploying concurrently even at peak, the fleet-wide surge is small).
  2. Instances warm up (image mostly served from the regional pull-through cache), then must pass the readiness gate: the customer's health check (HTTP path or TCP, configurable interval/timeout), 3 consecutive passes per instance, all instances ready, within a rollout deadline (default 10 minutes). No pass → ROLLOUT_FAILED, new instances torn down, edge never touched. The edge not being touched is the "failed deploy leaves previous version serving" guarantee — it's structural, not a recovery path.
  3. Edge shift: one SetBackends call moves 100% of traffic to the new instance set (optional canary weights for customers who want them), with a bumped generation number.
  4. Drain: old instances stop receiving new connections, in-flight requests get a grace period (default 30s, configurable up to the customer's long-request tolerance), then SIGTERM → SIGKILL. No dropped requests for well-behaved services.
  5. Bake: for 2 minutes post-shift the workflow watches crash-loops and health flaps on the new version; regression → automatic revert of the edge to the previous generation (old instances are only retired after the bake passes). Then LIVE, previous deploy RETIRED.

I chose this blue-green-per-service shape over in-place rolling updates because it makes both failure and rollback a single edge operation with the old set still intact — rolling updates save transient capacity but smear the two versions across the fleet and make "put it back" a second rolling update.

5. Rollback

Rollback is a deploy that skips the expensive parts. POST /rollback {to_deploy_id} creates a new deploy row (new seq — history is append-only, the timeline never rewrites) pinned to the old deploy's image_digest and config_snapshot_id. No build, no migration by default, same health gates, same edge shift. Since the image is retained (see GC) and usually still cached in-region, rollback lands in tens of seconds. If the deploy being rolled back ran a migration, the API response and UI carry a warning: the schema is newer than the code you're restoring — expand/contract is what makes that safe, and run_migrations: true exists for customers who ship reversible migrations.

30-day retention means any of the last 30 days of deploys is a valid rollback target, which pins those images against GC.

6. Cancellation

Cancel sends a workflow cancellation signal; what happens depends on phase:

Every path converges on the same invariant: the previous version is serving.

Live log streaming

Builders and cell agents emit logs as (deploy_id, stream, seq, line) over gRPC into a durable, partitioned event log (Kafka), partitioned by deploy ID so a deploy's lines stay ordered. Two consumers:

A client opening logs mid-build does backfill-then-follow: serve stored segments from from_seq, then splice into the live stream at the exact next sequence number — the per-deploy seq is what makes the splice gapless and dupe-free. Kafka rather than a bare pub/sub (Redis/NATS) because the event log absorbs archiver lag and lets it replay after a crash; the lighter option loses lines exactly when things are on fire, which is when customers are watching logs. At ~2TB/day this is a small Kafka cluster.

Failure modes

Builder host dies mid-build. Heartbeat lapses → workflow retries the build activity on another host. Cache import makes the retry cheap. Idempotent because nothing external happened until the digest push.

Registry push succeeds, workflow crashes before recording it. Activity retries; push by digest is a no-op the second time. Every external side effect in the pipeline is idempotent or generation-guarded for exactly this reason.

Cell agent partitioned from control plane. Rollout activity times out → ROLLOUT_FAILED → revert. Running services in the cell keep serving from local state; the edge keeps its last generation. Deploys into that cell pause; nothing serving is affected.

Thundering herd after a Git-provider outage. GitHub comes back and fires a day of webhooks at once. Coalescing absorbs most of it (each service collapses to its newest commit); the build queue absorbs the rest with per-customer concurrency caps so one monorepo org can't starve everyone.

Regional failures — three distinct cases:

  1. A runtime region fails. Services pinned there are down; that's the customer's blast radius in a single-region product, and our job is to not make it worse: freeze deploys targeting the region, keep status honest, resume cleanly. (Multi-region services with edge failover are the evolution path, and this architecture supports it — the edge API already takes weighted backend sets.)
  2. A build region fails. Builds are region-agnostic: the scheduler routes to another region, remote cache imports cross-region (slower — cache is a locality optimization, never a correctness dependency), registry blobs replicate.
  3. The control-plane region fails. The important one. By the data-plane-independence principle, every running service keeps serving — edges and cells operate on cached last-known-good state. What stops is new deploys, cancels, and log tailing. Recovery: Postgres has a synchronous in-region standby (HA without data loss) and an async cross-region replica; Temporal runs multi-cluster async replication to the same standby region. Failover promotes both, workers reconnect, workflows resume from replicated history; deploys in the async-replication gap may replay a step — safe, because every step is idempotent. Target: deploy capability restored in ~15 minutes, zero impact on serving traffic. I rejected an active-active control plane: consensus on seq assignment across regions buys little when the data plane already survives the outage, and it doubles the hardest operational surface.

Registry GC

Mark-and-sweep weekly: roots are (a) every live deploy's digest, (b) every deploy in the 30-day window, (c) cache manifests younger than 14 days. Everything else unreferenced gets deleted, blobs refcounted across manifests. Without this, at petabytes/month of growth, the registry quietly becomes the largest line item in the company.

Tradeoffs I'd defend, and what I'd do next

Next after v1, in order: canary analysis on the bake window (auto-rollback on error-rate regression, not just crash loops), multi-region services over the same edge API, preview environments per pull request (same pipeline, ephemeral service rows), and a build-cache CDN if cache-affinity hit rates disappoint.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): PostgreSQL and Temporal replicate asynchronously and independently: after control-plane failover, how do you establish a consistent recovery point, fence the old primary, and prevent a replayed workflow from assigning a conflicting sequence or issuing an edge update that supersedes a newer release?

This lands. The design says "every step is idempotent or generation-guarded," but a bare monotonic generation allocated from an async replica is not safe: the promoted Postgres can be behind and re-allocate a generation the old primary already used, and the old primary can still be issuing commands during the failover window. Two concrete changes. First, a control-plane epoch: a small strongly-consistent counter (its own quorum store; the promotion runbook bumps it) that increments on every promotion. Every command to the edge and cell agents carries the epoch; generations become (epoch, counter) ordered lexicographically. Anything the old primary emits carries the old epoch and is rejected at the data plane — the fence is enforced where the damage would land, not by hoping the old primary is really dead. Second, we hard-fence the old primary anyway (cloud API stop/demote) before promoting.

For the Postgres/Temporal consistency point: I don't try to align their replication streams. Postgres is the sole authority — seq, generations, and state transitions are allocated and guarded there; Temporal is only the driver. A replayed workflow's first move in any activity is a guarded UPDATE ("iff still in state X and not superseded"); if Postgres already moved on, the guard fails and that path terminates. If Temporal is behind Postgres (deploy row exists, no workflow), a post-failover reconciliation pass marks orphaned non-terminal rows FAILED — safe, the old version is still serving. If Temporal is ahead, its next guard fails. The recovery point is "wherever the promoted Postgres is," the only definition that doesn't require cross-system transactions. (Q7 pushes on the case where the data plane is ahead of the promoted Postgres — see below.)

Design change: epoch-prefixed edge generations with epoch checks enforced by edge and cell agents; hard-fence before promote; post-failover reconciliation pass that resolves Postgres/Temporal disagreement in Postgres's favor.

Q2 (Codex): RunJob cannot be exactly-once: if a migration commits but its success response is lost, retrying may execute it twice; if it fails halfway, the old application may face a partially changed schema. What concrete protocol handles ambiguous completion, non-idempotent migrations, timeouts, and safe recovery?

Correct — "exit 0 → proceed" glossed over the ambiguous cases. The protocol: RunJob is keyed by (service_id, deploy_id) and the cell agent makes it idempotent — a resubmit returns the existing job's status, never starts a second run (concretely: a Kubernetes Job with a deterministic name derived from the deploy id, so even an agent restart can't double-launch). The workflow's retry loop is poll, don't re-run: a lost success response is resolved by GetJob(service, deploy_id), so the migration executes at most once per deploy no matter how many times the activity retries.

The genuinely ambiguous case — the cell lost the job record mid-run, or the job timed out and won't confirm it stopped — gets a new terminal state, MIGRATION_UNKNOWN. We do not guess. The old version keeps serving (nothing after MIGRATING has touched instances or edge), the deploy halts, and the customer is told: your migration may be partially applied, inspect your database. Retrying a non-idempotent migration on the platform's initiative is the one move that can destroy customer data, so the platform never makes it. Half-applied schema against old code is bounded by the expand/contract contract already in the doc, and most migration tools (Rails, Flyway, Alembic) keep their own applied-versions ledger, which makes a customer-initiated retry safe in practice — we document that but don't depend on it.

Design change: RunJob idempotent on deploy id with poll-based completion; new MIGRATION_UNKNOWN terminal state requiring explicit customer action instead of any automatic retry.

Q3 (Codex): How would you support a coordinated pipeline spanning several services — ordered builds and migrations, compatibility gates, and an all-or-nothing traffic cutover — when orchestration, coalescing, sequence numbers, and edge generations are defined only per service? What happens when the rollout fails halfway?

Partly concede, partly push back. The concession: the design has no multi-service story and needs one. The pushback: "all-or-nothing traffic cutover" across N hostnames is a promise I won't make — the edge applies per-hostname atomically, and N per-hostname updates can't become one atomic operation without building a distributed transaction into the serving path. The honest contract is a bounded mixed-version window plus cheap group rollback.

The mechanism composes from existing primitives: a deploy group is a parent workflow above the per-service workflows, with stages — build all members first (no rollout starts until every build has a digest), run migrations in declared order, then shift services in declared order. Each per-service deploy keeps its own seq and edge generation; the group is a coordinator, not a new consistency domain. Two additions make halfway failure cheap: members hold their old instance sets until the whole group passes bake (extending the per-service retire rule), so when service 3 of 5 fails its health gate, the group reverts services 1–2 with single edge calls to still-warm old sets. Coalescing also moves up a level for grouped services: the group workflow picks one head commit per burst, so members never mix commits within a group deploy. The residual exposure — service A new, service B old for the seconds-to-minutes between shifts — is the same compatibility contract we already impose on schemas, applied to APIs between the customer's own services, stated rather than hidden.

Design change: deploy-group parent workflow (staged build → migrate → shift), group-scoped coalescing, and old instance sets retained until group-wide bake passes so mid-group failure reverts with per-service edge calls.

Q4 (Codex): For a monorepo connected to many services, how do you atomically map one commit to affected services, avoid cloning and rebuilding shared code repeatedly, preserve path-filter correctness across renames and dependency changes, and prevent each service's independent latest-wins coalescer from producing an incompatible mixture of commits?

Concede: the per-service webhook path quietly assumes repo≈service, and for the biggest customers that's false. The fix is a repo-level ingest workflow (workflow id repo-{id}) between the webhook and the per-service workflows. It does three jobs. Clone once: on a new head it produces one source snapshot (tarball in object storage keyed by commit sha); the N affected services' builders fetch the snapshot instead of cloning N times. Change detection: each service declares path filters plus dependency globs (its own dir, shared libs it consumes); the repo workflow diffs from each service's last-built sha to the new head — not just the pushed range — which keeps filters correct when commits were coalesced and across renames (the diff between the two shas sees the rename). Ambiguity resolves toward building: a change to the filter config itself, or a diff we can't classify, triggers the build. A wasted build is cheap; a missed one ships stale code.

Coalescing moves to the repo workflow for monorepo services, which answers the incompatible-mixture problem directly: one head sha per burst, every affected service builds from that same sha. Their rollouts still land at different times because builds finish at different times — if the customer needs them to shift together, that's a deploy group (Q3), and the repo workflow can feed one. On rebuild sharing: registry layers already dedupe by content address, and I'd relax the strictly-per-service cache rule to per-repo for monorepos — services in one repo are one trust domain, so the cache-poisoning argument that justified per-service isolation doesn't apply between them, and shared-library layers cache once instead of N times.

Design change: repo-level ingest workflow owning clone, per-service last-built-sha diffing, and repo-scoped coalescing; source snapshots by sha in object storage; build-cache scope widened from per-service to per-repo.

Q5 (Codex): Your per-service registry credentials, cache isolation, deploy records, and GC roots assume artifacts belong to one service. How can one verified build artifact be promoted safely across services, environments, regions, and customers without rebuilding, while preserving provenance, authorization, secret isolation, rollback retention, and garbage-collection correctness?

Push back on one axis, concede the rest. Cross-customer artifact reuse stays off the table — the isolation stance is deliberate, and content-addressed storage already dedupes identical blobs across customers at the storage layer without granting anyone access to anyone else's manifests. That's the only cross-customer sharing I'll do.

Within a customer, build-once-promote-everywhere is a real product need (staging → prod is the obvious case) and the current model can't express it, because image_digest lives only on the deploy row of the service that built it. The change: split artifact out as a first-class row — artifacts(id, customer_id, repo, commit_sha, image_digest, provenance, attestation) — written at build time, with the digest signed by the build plane (provenance records builder identity, source sha, and cache inputs; SLSA-style attestation, verified at deploy time). A deploy references an artifact, and any service under the same customer/project may reference it, subject to an authorization check at deploy creation; the registry grants the target service's pull token read on that manifest. Secrets don't travel with it — build secrets exist only as BuildKit secret mounts and never enter a layer, so promotion can't leak them; runtime config comes from the target service's config snapshot, which is the point of separating code from config. GC roots change from "digests referenced by this service's 30-day deploys" to "artifacts referenced by any deploy in any service's 30-day window" — the mark-and-sweep structure already supports the refcount; the roots enumerate deploys→artifacts instead of deploys→digest. Regions are already solved: blobs replicate; promotion is a metadata write.

Design change: first-class artifacts table with signed provenance verified at deploy time; deploys reference artifacts; same-customer cross-service promotion with per-manifest ACL grants; GC roots become artifact-scoped.

Q6 (Codex): How does infrastructure-as-code fit into this system: what is the desired-state model and plan/apply protocol for atomically creating, renaming, updating, or deleting multiple services and their dependencies, and how do you reconcile drift or concurrent dashboard, Git-push, and IaC operations without deploying against half-applied infrastructure?

Concede: it's absent, and a platform at this scale without it forces every serious customer to script the dashboard API. The model: a versioned project spec (a platform.yaml in the repo, or applied via API) declaring services, env groups, and managed databases. Apply is plan/apply: compute the diff against current project state in Postgres, emit a plan (create/update/delete per resource, ordered by declared dependencies), execute it as a Temporal workflow holding a per-project lock — the same serialization trick the design already uses per service, one level up.

Two hard cases the question names. Concurrency and drift: every mutation — dashboard click, direct API call, IaC apply — goes through the same API and bumps a monotonic project version; apply is compare-and-set on the version it planned against, so a concurrent dashboard change fails the apply with "replan" rather than silently merging. There is no second write path to drift against, which is most of what makes Terraform-style drift painful. Atomicity: I won't fake it. Infra creation isn't atomic anywhere; the plan executes in dependency order (database before the service that references it), records each completed step, and on failure halts resumable rather than auto-rolling-back — automatically deleting a half-created database is strictly worse than pausing with a clear state. Deploying against half-applied infra can't happen because spec-triggered deploys are the plan's last steps, gated on every resource they reference reporting ready, and a deploy created against a not-ready resource fails validation at creation time. Config changes to running services don't get a side door: they become config snapshots flowing through the existing deploy pipeline, health gates and all.

Design change: project spec + plan/apply workflow with per-project lock, CAS on a project version bumped by every mutation path, dependency-ordered execution that halts resumable on failure, and spec-triggered deploys gated on resource readiness.

Round 2 — pushback

Q7 (Codex): After async failover, the promoted Postgres may be missing a newer deploy that is already live at the edge; why doesn't assigning its stale workflow a higher epoch cause the edge to accept an older release as newer, and what authoritative reconciliation prevents that rollback?

Real gap in my Q1 answer, and the epoch alone makes it worse: a stale workflow resumed under the new epoch would carry a generation that outranks everything pre-failover, so the edge would obediently roll a customer back. The fix is that the direction of truth inverts during recovery. Normally Postgres tells the data plane what should be serving; during the failover window the data plane provably knows a fact the promoted Postgres may have lost — what is serving — so recovery reads it back before issuing anything. The promotion runbook becomes: fence old primary → promote → bump epoch → freeze all outbound edge/cell commands → reconcile → unfreeze. Reconciliation queries every edge for its current per-hostname state and every cell for running deploy ids, and adopts that into Postgres: upsert releases, and where the edge is serving a deploy the promoted Postgres never heard of, synthesize the deploy row and mark it LIVE.

For that synthesis to be possible, the edge has to store enough to reconstruct the fact — so SetBackends now carries (service_id, seq, deploy_id, image_digest) alongside the generation, and the edge persists the tuple it's currently applying. That's the authoritative record: not a log we hope replicated, but the actual serving configuration read from the component that enforces it. After adoption, the stale workflow is harmless by the existing mechanism — its guarded UPDATE sees a higher seq live in releases and fails, and it never gets an edge command out anyway because commands are frozen until adoption completes. The freeze is cheap: it extends the deploy-capability outage by however long it takes to scan the edges — seconds to low minutes — against a stated 15-minute recovery target, and it buys the invariant that the control plane never contradicts a data-plane fact it hasn't read.

Design change: SetBackends payload extended with (service_id, seq, deploy_id, image_digest), persisted at the edge; failover runbook gains a freeze → read-back → adopt → unfreeze phase in which data-plane state is authoritative for what's live, before any post-failover command is issued.

Q8 (Codex): Your deploy-group coordinator sits "above" independently serialized per-service workflows, but what exact locking and fencing protocol prevents a normal push to one member from deploying mid-group — or deadlocking with the group — while still allowing cancellation and recovery after coordinator failure?

The group doesn't bypass the per-service workflow — it goes through it, which is why no new lock is needed for the push case. The per-service workflow stays the sole serialization point: the group coordinator signals it StartMemberDeploy(group_id, artifact, lease), and that member deploy is the service's in-flight deploy. A normal push arriving mid-group hits the existing coalescing rule — recorded as pending-latest, deployed after the group completes — because from the per-service workflow's view a group deploy is indistinguishable from any other in-flight deploy. No new mechanism, no second lock to get out of sync with the first.

Deadlock is prevented by refusing to wait rather than by ordering waits. Group admission writes one row per member into service_group_membership(service_id UNIQUE, group_id, lease_expiry) in a single Postgres transaction — all members or none. A second group overlapping any member fails admission immediately; there is no queue of groups holding some members while waiting on others, and no hold-and-wait means no deadlock. Coordinator failure splits into two cases. Ordinary crash: the coordinator is a Temporal workflow, it resumes from history like everything else. Genuinely stuck or gone: the membership row carries a lease the coordinator must renew (~every 30s); on lease expiry the per-service workflow aborts its member deploy through the normal failure path — old version still serving, by the standing invariant — deletes its membership row, and resumes normal push processing. Cancellation is the group workflow running its compensation path: signal un-shifted members to cancel (existing per-phase semantics), revert already-shifted members to their previous edge generation, which works because group members retain old instance sets until group-wide bake. Fencing of a zombie coordinator is the lease again: its signals carry group_id, and a member whose membership row is gone or expired ignores them.

Design change: group admission as an all-or-nothing transactional insert with a uniqueness constraint (fail-fast on overlap, no group queuing); coordinator lease with per-member expiry-abort; member deploys routed through the existing per-service workflow so pushes coalesce mid-group with no additional locking.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The coalescing design ships in production at Render, almost knob for knob. Their deploy docs describe two overlapping-deploy policies: "Wait" queues a new deploy and skips the intermediate ones — latest-wins coalescing — and "Override" immediately cancels the in-flight deploy, which is my cancel_in_flight_on_push under a different name. Their zero-downtime sequence is the same shape too: new instances start while old ones keep serving, optional health checks gate the switch, old instances drain on a grace period. Migrations run as a pre-deploy command whose failure cancels the deploy — my MIGRATING phase, though their public docs don't surface the ambiguous-completion case that Q2 forced me to handle with MIGRATION_UNKNOWN.

The Temporal bet — the single most load-bearing choice in the design — has strong company. Netflix re-platformed Spinnaker's cloud operations onto Temporal and reports deploy failures from transient errors dropping from 4% to 0.0001%, and the discipline they had to adopt is exactly Q1/Q2's: deterministic workflows, idempotent activities, because Temporal retries by default. Closer to this exact product, WunderGraph built a commit-to-production pipeline on Firecracker microVMs (via Fly Machines) orchestrated by Temporal — Temporal owns build queueing and cancel-on-new-commit, and each project gets a dedicated machine with a persistent volume as its build cache. That's my cache-affinity scheduling and my coalescer, running in a real system. Fly's own Machines post covers the substrate: Firecracker VMs booting in ~300ms from locally cached images, which is what makes VM-per-build economics work at all.

On build caching, Depot is the pure-play version of my build plane: BuildKit builders with a persistent NVMe cache attached per project, claiming order-of-magnitude speedups. One divergence worth noting: Depot shares the cache across a whole team and its CI, where I started at strictly per-service and only relaxed to per-repo under Q4's pressure — Depot's project scope and my per-repo scope land in about the same place, one trust domain per cache.

One choice in the design needs correcting: I named Nixpacks for the no-Dockerfile path, and Railway — who created Nixpacks — retired it in March 2025 for Railpack, a builder that emits BuildKit LLB directly, citing exactly the problems my caching section cares about: Nix's single giant /nix/store layer defeated layer caching, and version pinning was unmaintainable. Heroku made the parallel move, replacing slugs with Cloud Native Buildpacks producing OCI images in its Fir generation. Revised design: the buildpack path is Railpack or CNB, still driven by BuildKit.

Two more decisions map cleanly. Rollback-as-a-new-deploy is Heroku's decades-old model: releases are an immutable append-only ledger of artifact + config vars + add-ons, and a rollback appends a new release pointing at old content — my new-seq deploy pinned to an old image_digest and config_snapshot_id, including the code/config separation. And the expand/contract contract I impose on customer migrations is written up canonically by PlanetScale: six phases, expand → migrate → contract, with old code running safely against the new schema at every step — the exact property my rollout window depends on.

The cell-sizing argument also has a real-world data point. Render's post on scaling Knative to 100k+ apps describes the free-tier surge quadrupling app count and the per-cluster networking layer buckling — Calico and kube-proxy burning CPU on every node, network-programming latency causing routing failures — fixed by surgery that removed Kubernetes Services per app, with an acknowledgment they'd eventually replace the layer entirely. That's what pushing one cluster toward six figures of services looks like, and it's why the design caps a cell at ~5K services and treats the runtime as swappable under the cell-agent API.

Updates from post-training information

Further reading