Contents

Multi-Tenant CI/CD Platform

The shape of the problem

This is really three systems wearing one trench coat: a metadata/orchestration system (workflows, jobs, state machines — small data, strong consistency), a compute scheduling system (get 1M concurrent jobs onto isolated machines, fairly, fast), and a high-volume data pipeline (logs and artifacts — big data, weak consistency is fine). The most common design mistake is letting one of these leak into another: putting logs in the metadata database, or putting scheduling state in the log pipeline. I'll keep them separate and let each pick its own storage and consistency model.

The genuinely hard parts, in order: running untrusted user code at 1M concurrency without tenants escaping into each other, fair scheduling under 20x bursts so one org's 50,000-job matrix doesn't starve everyone, and log ingest at multi-GB/s with live tailing. Everything else is fairly standard CRUD-plus-queues.

Assumptions

Stating these rather than asking, as instructed:

Requirements

Functional, in priority order:

  1. Push → webhook → parse workflow → execute job DAG on isolated runners.
  2. Real-time status and log streaming to browsers; cancel; re-run failed jobs.
  3. Artifact upload/download, up to 20 GB.
  4. Per-org plans: concurrency quotas, runner sizes, self-hosted runners, minute metering for billing.

Non-functional:

Out of scope in 60 minutes: the caching subsystem (I'll gesture at it), a marketplace of reusable actions, IDE integrations.

Scale estimates (the ones that drive decisions)

Architecture

flowchart TB
    subgraph external [External]
        GF[GitForge webhooks + repo API]
        Browser[Browser / CLI]
    end

    subgraph control [Control plane - sharded by org]
        GW[API gateway<br/>authn, rate limits]
        WH[Webhook ingestor<br/>verify, dedupe, enqueue]
        K[(Kafka<br/>event backbone)]
        ORCH[Workflow orchestrator<br/>DAG state machine]
        PG[(Postgres shards<br/>workflows, jobs, quotas)]
        SCHED[Job scheduler<br/>per-org fair queues, quotas]
    end

    subgraph data [Data plane]
        DISP[Dispatchers - gRPC]
        subgraph cell1 [Runner cell 1..N]
            RA[Runner agents on<br/>Firecracker microVMs]
        end
        SH[Self-hosted runners<br/>long-poll]
    end

    subgraph pipeline [Logs / artifacts / realtime]
        LOGI[Log ingest]
        RT[(Redis - live tail ring buffers)]
        S3[(Object storage<br/>logs + artifacts)]
        FAN[Realtime fan-out<br/>SSE/WebSocket]
        ART[Artifact service<br/>presigned URLs]
    end

    GF -->|push event| WH --> K --> ORCH
    ORCH <--> PG
    ORCH -->|runnable jobs| SCHED
    SCHED --> DISP --> RA
    SCHED --> SH
    RA -->|status, heartbeats| ORCH
    RA -->|log frames| LOGI
    LOGI --> RT --> FAN --> Browser
    LOGI -->|chunks| S3
    RA <-->|presigned up/down| S3
    Browser --> GW --> ORCH
    Browser --> GW --> ART --> S3
    RA -->|fetch source| GF

Ingestion: never lose a push

The webhook ingestor verifies GitForge's HMAC signature, dedupes on delivery ID, writes a tiny receipt row, and publishes to a durable, partitioned event log — Kafka. That's all it does — parsing, quota checks, and DAG planning happen downstream. This is the "accept work even when degraded" principle: the ingestor's dependencies are one Kafka topic and one small table, so it stays up when the orchestrator is having a bad day. Kafka (over SQS or Rabbit) because we want replay: after an orchestrator bug, re-consuming an hour of pushes is a rewind, not a data-recovery incident.

Orchestrator: the workflow state machine

The orchestrator consumes push events, fetches the workflow YAML from GitForge at the pushed commit, expands matrices, and writes the workflow + job rows in one transaction on the org's Postgres shard. It then advances the DAG: when a job reaches a terminal state, mark dependents runnable; when all jobs are terminal, the workflow is terminal.

Why hand-rolled state machine on Postgres rather than Temporal. I considered Temporal seriously — durable execution is exactly this shape. Two reasons I rejected it: at 50M jobs/day the Temporal cluster becomes its own large distributed system to operate, roughly as complex as what it replaces; and our state machine is shallow (workflow → jobs → steps, a handful of states) so we'd be paying Temporal's generality for a DAG walk. The classic failure mode of the hand-rolled approach — stuck states after a crash — we handle with two boring mechanisms: every state transition is an idempotent conditional update (UPDATE jobs SET state='running' WHERE id=? AND state='scheduled'), and a sweeper per shard re-drives anything sitting in a non-terminal state without a fresh lease. Correctness comes from the database, liveness from the sweeper.

Sharding by org (an org's data lives wholly on one shard, e.g. 64 shards via Vitess-style routing or app-level shard map) keeps every workflow's transitions on a single Postgres — no cross-shard transactions anywhere in the hot path. The cost: a whale org can hotspot its shard. Acceptable, because per-org quotas cap how hot one org can run, and whales can be moved to dedicated shards.

Scheduling: fairness is the product

At peak we have ~12k jobs/s arriving and 1M slots to fill. A single global FIFO fails immediately — one org pushing a 10,000-job matrix build starves everyone behind them, and "my 30-second test run waited behind someone else's fleet build" is the complaint that churns customers.

Design: per-org logical queues (sorted sets in an in-memory store — Redis — or an in-memory scheduler service checkpointed to its shard, keyed by org × runner-pool), drained by a weighted deficit-round-robin scheduler per runner pool. Weights come from plan tier; each org also has a hard concurrency cap (its quota). Within an org, FIFO by workflow with a small priority bump for re-runs and jobs unblocking a nearly-done workflow. Admission control happens at enqueue: over quota → job sits in queued visibly, with the reason attached, rather than being rejected — users forgive waiting, they don't forgive mystery.

The scheduler is sharded by runner pool (os × size × region × self-hosted-label), so pools scale independently and a bad pool (say, macOS capacity crunch) can't back up Linux dispatch.

Dispatch model: push over gRPC to our own fleet (dispatcher holds streams to warm runner agents — lowest latency), long-poll for self-hosted runners (they're behind NAT; they call us, matched by labels; we never connect inbound to customer networks).

Execution: Firecracker microVMs, cell architecture

Each job runs in a fresh hardware-virtualized microVM (Firecracker) that is destroyed afterward. This is the load-bearing decision, so the alternatives I rejected:

Firecracker gives a hardware-virtualization boundary with ~125ms boots and small overhead — it's what AWS Lambda and (per their public writing) Fly.io run untrusted code on. We keep a warm pool of booted, pre-imaged microVMs per pool type sized by short-horizon arrival forecasting; hitting the warm pool makes time-to-start ≈ dispatch + source fetch. Warm pool misses fall back to cold boots, which is a latency degradation, not an outage.

Runners are grouped into cells of ~10-20k microVMs, each cell with its own dispatchers and log-ingest edge. Cells are the blast-radius unit: a poisoned image, a bad agent rollout, or a zonal failure takes out one cell's capacity, and the scheduler just stops routing to it. 1M concurrent ≈ 50-100 cells.

Inside the microVM, a runner agent (ours, trusted, runs outside the user's step processes) fetches source from GitForge with a job-scoped token, executes steps, streams logs, heartbeats every ~10s carrying a lease, uploads artifacts, and reports the terminal state. Lease expiry (say 60s) is how the orchestrator detects dead runners: sweeper sees an expired lease on a running job → mark infra-failed → auto-requeue once (infra failure, not user failure — doesn't count against the user) → fail visibly the second time.

Cancellation rides the same channel: the orchestrator flips the job row to cancelling; the dispatcher signals the agent over the gRPC stream (or the next long-poll for self-hosted); the agent SIGTERMs the step, waits a grace period, SIGKILLs, reports cancelled. If the agent is unreachable, lease expiry converges it anyway — cancellation is eventually correct even for zombie runners.

Logs: two paths, one write

The agent batches log lines into frames (≤64 KB or 200ms, sequence-numbered per step) and ships them to the cell's log-ingest service. Ingest does one fan-out:

  1. Live path: append to a Redis ring buffer per step (capped, ~5 MB) and publish to the step's channel. The realtime fan-out service holds browser SSE connections; "watch a job" = replay ring buffer from your cursor, then stream. If the buffer's been overwritten, the UI backfills from the durable path. Best-effort by design.
  2. Durable path: accumulate frames into ~8 MB compressed chunks, write to object storage as logs/{org}/{job}/{step}/{seq}.zst, index chunk offsets in the metadata shard. Ack to the agent only after the durable write — the agent retries unacked frames, and sequence numbers make retries idempotent. Finished logs are served by stitching chunks through a CDN.

Why not Kafka as the log transport? 8 GB/s through Kafka is doable but buys us nothing here — there's exactly one consumer (the chunk writer), no replay requirement (the agent is the replay buffer, it retries until acked), and Kafka retention at this volume is real money. Kafka stays for control-plane events where replay matters; logs go direct.

I'd also skip full-text log search at launch (grep-in-browser over fetched chunks covers most usage) and later offer indexed search as a paid feature via a columnar analytics store (ClickHouse or similar) over the recent window — indexing 2 PB nobody searches is the classic money bonfire.

SSE fan-out at scale: watching is sparse — maybe 1-5% of running jobs have a viewer. Even at 1M concurrent that's ~50k live streams across a stateless fan-out tier subscribed to Redis pub/sub; tens of nodes.

Artifacts: the platform never touches the bytes

Agent asks the artifact service for an upload slot → gets object-storage (S3) multipart presigned URLs scoped to artifacts/{org}/{workflow}/{name} → uploads directly (multipart is mandatory at 20 GB: parallelism plus per-part retry) → commits the manifest (parts, sizes, SHA-256) to the artifact service, which finalizes the metadata row. Downloads are presigned GETs behind an authz check. Retention is an S3 lifecycle rule per prefix plus a metadata sweeper — the 90-day delete is enforced in both layers so a sweeper bug can't silently retain compliance-scoped data.

Secrets

Secrets are stored envelope-encrypted (per-org data key wrapping, KMS root). The agent receives only the secrets the workflow references, injected at job start over the dispatch channel, held in memory inside the microVM, never written to the job row or logs. Ingest scrubs known secret values from log frames (imperfect — users are told so). Fork-PR workflows get no secrets by default: running a stranger's PR code with your deploy keys is the classic CI self-own, and the default has to be the safe one.

Data model (metadata shards, Postgres)

orgs        (org_id PK, plan, concurrency_limit, shard_id, ...)
repos       (repo_id PK, org_id, gitforge_ref, ...)
workflow_runs (run_id PK, repo_id, org_id, commit_sha, definition_snapshot JSONB,
               state, trigger, created_at, started_at, finished_at, attempt)
jobs        (job_id PK, run_id, org_id, name, state, runner_pool,
             needs JSONB, attempt, lease_expires_at, cell_id, exit_code,
             queued_at, started_at, finished_at)
steps       (step_id PK, job_id, idx, name, state, log_chunk_index JSONB, timings)
artifacts   (artifact_id PK, run_id, org_id, name, size_bytes, sha256,
             storage_key, expires_at, state)
usage_events (org_id, job_id, pool, seconds, ts)  -- append-only, feeds billing

Key points: definition_snapshot freezes the parsed workflow at trigger time so a later push editing the YAML can't change what a re-run means. attempt on runs and jobs makes re-run a new attempt, immutable history preserved. Job states: queued → scheduled → running → {succeeded, failed, infra_failed, cancelled} plus cancelling; every transition is a guarded conditional update. Time-ordered partitioning on workflow_runs/jobs by month makes 90-day retention a partition drop, not a DELETE storm.

API sketch

External (REST + SSE, org-scoped tokens):

POST /repos/{repo}/dispatches                  # manual trigger
GET  /runs/{id}          GET /runs/{id}/jobs
POST /runs/{id}/cancel   POST /runs/{id}/rerun-failed
GET  /jobs/{id}/logs?step=n                    # finished: CDN redirect
GET  /jobs/{id}/logs/stream                    # SSE live tail, cursor resume
GET  /runs/{id}/artifacts   GET /artifacts/{id}/download   # presigned redirect
POST /orgs/{org}/runners/registration-token    # self-hosted enrollment

Internal agent ↔ platform (gRPC, mTLS, job-scoped token minted at dispatch):

AcquireJob / stream JobEvents (dispatch, cancel)   Heartbeat(lease)
AppendLogFrames(step, seq, frames)                 ReportStepState / ReportJobResult
RequestArtifactSlot -> presigned parts             CommitArtifact(manifest)

The job-scoped token is the security keystone: it authorizes exactly this job's log streams, this run's artifact prefix, this commit's source read — so a compromised runner (assume it will be compromised; it runs attacker code) can vandalize its own job and nothing else.

Failure modes, walked through

Multi-tenancy and quotas

Quota enforcement lives in the scheduler (concurrency caps per org × pool) and the metering pipeline (usage_events → aggregation → billing; also drives "you've used 80% of your minutes" warnings). Plans also gate runner sizes and features (larger microVM shapes, GPU pools, longer retention). Noisy-neighbor defense is layered: fair queues at dispatch, hard VM-level CPU/memory/disk limits at execution, per-step rate limits at log ingest, per-org rate limits at the API. Self-hosted runners bypass our compute quotas (their hardware) but still meter through the same pipeline for visibility.

Tradeoffs I'm consciously making

Evolution

First year, roughly in order: build caching (keyed content-addressed cache over the same artifact machinery — it's the #1 speed feature users ask for), merge/deploy queues and environment approvals (this is what forces the orchestration re-evaluation), GPU and macOS pools (macOS breaks the Firecracker assumption — that pool runs on real Macs with VM-per-job via Virtualization.framework, its own cell type), regional cells near customer source control for clone latency, and OIDC federation so jobs can assume cloud roles without long-lived secrets — which quietly removes the scariest secret-storage burden from the platform entirely.

The invariant that shouldn't change as it grows: control plane decides, data plane executes, and bytes that are big (logs, artifacts, source) flow between agents and object storage without passing through the services that think.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): When a runner's lease expires and its job is requeued, what fencing mechanism prevents the original runner from continuing to upload artifacts, report success, or execute irreversible deployment side effects after the replacement starts — and how does your concurrency-group primitive enforce that fence against external systems?

You're right that the design detects the zombie but never fences it — that's a real gap. The fix is to make attempt a fencing token. The job token minted at dispatch embeds (job_id, attempt), and every internal write path — log ingest, ReportJobResult, CommitArtifact — validates the token's attempt against the current attempt on the job row. The requeue transaction increments attempt, so the old token is dead the moment the replacement exists; the check rides the guarded conditional updates already in the design.

Presigned S3 URLs can't be revoked, so artifact keys get the attempt baked in: artifacts/{org}/{run}/{attempt}/{name}. A zombie can pour bytes into its own dead prefix, but CommitArtifact is the linearization point — the manifest commit is a conditional update that fails on a stale attempt, so zombie bytes never become a visible artifact, and orphaned parts are cleaned by an AbortIncompleteMultipartUpload lifecycle rule.

External side effects can't be fenced — nothing I do can un-run half a deploy against someone's Kubernetes cluster. So the concurrency-group behavior changes: a job holding a concurrency group is never auto-requeued on lease expiry; it fails visibly, and the group lease (a row keyed on group, held by (job_id, attempt), acquired via conditional update) releases only after a grace timeout. The replacement can't start while a zombie might still be mid-deploy. That's the honest version of at-least-once: retries for builds, mutual exclusion plus human-visible failure for deploys.

Design change: attempt-scoped tokens validated on every internal write; attempt-prefixed artifact keys with a commit-time attempt check; no auto-requeue for concurrency-group holders.

Q2 (Codex): You dual-write at several correctness boundaries: webhook receipt row → Kafka, Postgres job transition → scheduler queue, and terminal job update → dependent-job enqueue. Walk through every crash window and show the transactional outbox, deduplication key, and recovery rule that guarantees neither lost nor permanently duplicated work.

The unifying rule: Postgres is the truth, every queue is a hint, and hints may be lost or duplicated. Boundary by boundary.

Receipt row → Kafka: the receipt row is a transactional outbox. It's written state=pending; the publisher marks it published after the Kafka ack; a relay republishes any pending row older than a few seconds. Crash before publish → relay republishes. Crash after publish, before marking → duplicate publish, killed downstream: the orchestrator's run-creation insert carries a unique index on (repo_id, delivery_id), so the second consume is a no-op. Dedupe key is GitForge's delivery ID end to end.

Postgres transition → scheduler enqueue: the enqueue is post-commit and allowed to fail either direction. Lost: the per-shard sweeper scans for jobs in state='queued' with no live scheduler lease and re-enqueues them — the same sweeper that already re-drives stalled states. Duplicated: dispatch commits via UPDATE jobs SET state='scheduled' WHERE id=? AND state='queued'; the second dispatcher matches zero rows and drops the job.

Terminal update → dependent enqueue: not a dual write at all — this is why I sharded by org. The terminal transition and marking dependents runnable are one transaction on one shard, since the whole run lives there. The post-commit enqueue of the newly runnable jobs is the previous case again. Every crash window resolves to "late, sweeper re-drives" or "duplicate, guarded update kills it." No mechanism change; the revision is stating explicitly that the receipt row is an outbox with a relay, which the original text implied but didn't say.

Q3 (Codex): Scheduling is sharded by runner pool, while concurrency quotas and fairness are defined per organization across pools. How do independent pool schedulers atomically enforce a global org limit without a central bottleneck, oversubscription during failover, or stranded capacity — and where is authoritative slot ownership stored?

Fair push — I wrote "per-org cap" and "per-pool schedulers" without saying who owns the count. The authority is the org's Postgres shard, and the enforcement point is the queued→scheduled conditional update that already exists: the dispatch transaction updates an org_slots counter row on the same shard with a WHERE running < concurrency_limit guard, atomically with the job's state transition. If the guard fails, the job stays queued and the scheduler moves to the next org in its round-robin pass.

Pool schedulers become advisory: they pick candidates using a cached (slightly stale) view of org usage, and the shard commit is the atomic check. No new bottleneck, because an org's job transitions already serialize on its shard — a whale running 50k concurrent jobs is a few hundred transitions/s on one shard, well within Postgres range. Cross-pool fairness stays where it was: DRR weights within each pool; the global cap is the only cross-pool invariant, and it's enforced at commit.

Failure cases: the counter is derived state, recomputable from job rows — on shard failover we recompute it during promotion, and any oversubscription from in-flight dispatches during the window is bounded by dispatch concurrency and reconciled by the sweeper. Stranded capacity doesn't happen because a failed cap check costs a candidate-selection retry, not a held slot.

Design change: org concurrency enforced at the shard commit inside the dispatch transaction; scheduler-side caps demoted to advisory hints for candidate selection.

Q4 (Codex): You call the runner agent trusted, but place it inside the same microVM where attacker-controlled code may run with privileges sufficient for Docker-in-Docker. What boundary prevents that code from reading job tokens and secrets, tampering with heartbeats/results, impersonating log or artifact uploads, or attacking the agent itself?

The strongest question of the six, and the design as written is wrong: "the agent is trusted" and "assume the runner will be compromised" can't both live inside one VM, especially when jobs legitimately run as root for Docker-in-Docker. The fix is to split the agent across the isolation boundary.

A host-side job supervisor — one per microVM, running on the cell host, outside the VM — holds the job token, owns the gRPC stream to the dispatcher, and sends heartbeats. Inside the VM runs an executor that is untrusted, same as the user code. They talk over vsock through a narrow API: append log frames, report step state, request artifact upload. The supervisor enforces job scoping on everything that crosses; the token never enters the VM.

What an in-VM attacker retains: they can forge their own log content (they already write it), upload garbage as their own artifact (they already produce it), and report success falsely (they could just exit 0). Everything spoofable is confined to the attacker's own job — that's the actual invariant, and it now holds mechanically rather than by hoping the agent survives contact with root-level attacker code. Heartbeats can't be spoofed to keep a wedged VM alive forever, because the supervisor also enforces the job's wall-clock timeout from outside. Secrets are the irreducible residue: user code must be able to use the secrets the workflow references, so code in that VM can read those secrets. No architecture removes that. It's why fork PRs get no secrets by default and why OIDC federation is on the evolution list — short-lived scoped cloud roles instead of long-lived stored secrets shrinks what a compromised job can exfiltrate.

Design change: the runner agent splits into a host-side supervisor (token holder, heartbeats, wall-clock enforcement, log/artifact proxy over vsock) and an untrusted in-VM executor.

Q5 (Codex): "Hard delete after 90 days" is not established by an asynchronous object-store lifecycle rule plus metadata partition drops. How do you prove deletion from multipart remnants, object versions, replicas, backups, CDN caches, Redis buffers, Kafka events, and disaster-recovery copies within a defined deadline?

The lifecycle-rule answer was too thin for a compliance requirement — lifecycle rules are asynchronous, versioning and replicas leak, and backups outlive everything. Revised mechanism, crypto first: log and artifact objects are envelope-encrypted with data keys scoped per org × time bucket. At day 90 a key-destruction job destroys the bucket's keys. Every copy — replicas, DR copies, backups of the object store, anything a lifecycle rule missed — becomes unreadable at key destruction, without having to enumerate the copies. That's the primary guarantee.

Mechanical deletion continues as the second layer: versioning off for these prefixes (or noncurrent-version expiration at 1 day), AbortIncompleteMultipartUpload at 7 days for orphaned parts, lifecycle deletes run independently per region rather than trusting delete-marker replication, CDN uses short TTLs plus signed URLs and gets a purge on delete, Redis ring buffers are already 5 MB caps with TTLs measured in hours, and Kafka control-plane topics get 30-day retention — they carry metadata rather than log content, but run metadata is in scope too.

Postgres backups: retention capped at 35 days, so a partition dropped at day 90 has left all backups by day 125. The compliance SLA is stated in two stages: unreadable at day 90 (key destruction), physically gone from all media by day 125 (backup expiry). And verification, because compliance wants proof: a weekly audit job samples deleted run IDs and attempts every read path — API, CDN, direct object GET — and alarms on any success.

Design change: per-org time-bucketed envelope keys destroyed at retention; two-stage deletion SLA (unreadable at 90, gone by 125); deletion audit sampler. (Key granularity gets tightened further in Q8.)

Q6 (Codex): At peak, your durable log path already accepts roughly 8 GB/s, yet overload handling merely disables live delivery while sending the same bytes to durable storage. What are the concrete backpressure and buffering limits per runner and per cell, and what happens to logs when object storage slows long enough for a two-hour, 10 MB/min stream to exhaust local storage?

Concrete numbers. Per runner: the agent spools unacked frames to a local disk buffer capped at 2 GB. The absolute worst case a single job can produce is 2h × 10 MB/min = 1.2 GB, so one runner can ride out object storage being down for its entire job. Backpressure is ack-driven and needs no separate protocol: when S3 slows, cell ingest delays acks, and the spool absorbs the difference.

Per cell: 20k VMs × ~0.5 MB/min average ≈ 170 MB/s of ingest per cell edge. A 30-minute S3 outage parks ~300 GB — spread across 20k per-VM spools that's 15 MB each, trivial. The ingest service itself holds only in-flight chunks (bounded to a few MB per active stream) precisely because the agent is the durable retry buffer — that was the argument for skipping Kafka on this path, and it's also what makes ingest cheap to keep stateless.

The piece the question rightly exposes as missing is an unconditional cap. "Durable once acked" with no ceiling is an unbounded liability against a hostile tenant running 10 MB/min × 2h across thousands of jobs. So: a per-step log limit (512 MB default, plan-adjustable), enforced at the agent; past it, logs truncate with an explicit marker in the stream. And if a spool ever fills anyway — which given the cap means local disk failure, not backlog — the agent fails the job as infra_failed rather than silently dropping data: acked-log durability stays inviolate, and unacked logs on a failed runner were never promised.

Design change: 2 GB agent disk spool stated explicitly; per-step log cap (512 MB default) with visible truncation; spool-full fails the job as infra_failed instead of dropping logs.

Round 2 — pushback

Q7 (Codex): Your concurrency-group "grace timeout" is not fencing: if a partitioned zombie keeps deploying after the timeout, the replacement starts concurrently. What concrete mechanism proves the old attempt is dead — or makes the deployment target reject its writes — before you release the group?

Correct — a timeout is a bet, not a proof, and I shouldn't have called it a fence. Two mechanisms replace it, split by who owns the hardware.

On our fleet, the Q4 split gives us the proof: the host-side supervisor is our code on our host, outside the VM, and it enforces "lease not renewed → destroy the microVM" locally, on its own clock, with no dependency on reaching the control plane. A partition that isolates the host doesn't save the zombie — the supervisor kills the VM when the local lease timer expires. So the contract is: the VM is provably destroyed within T_kill = lease TTL + bounded local-clock skew of the last successful renewal, and the sweeper releases the concurrency group only after T_kill has elapsed since the lease was last renewed. That's the same shape as a lease-plus-local-watchdog fence: the kill decision is made by the entity that can actually enforce it. If the supervisor host itself dies, the VM dies with it — the failure modes where the zombie survives are gone because the zombie's life support is the thing enforcing the fence.

Self-hosted runners are the honest exception: I can't kill hardware I don't own, so proof-of-death doesn't exist there. The fence has to move to the deployment target's side: job credentials for deploy jobs are attempt-scoped OIDC tokens with TTL ≤ the lease TTL, so by the time the group is eligible for release, the zombie's credentials have already expired and the cloud target rejects its writes — the target enforces the fence even though we can't reach the runner. For targets that can't validate short-lived credentials (a long-lived SSH key stored as a secret), no fence is possible, and the platform shouldn't pretend: a concurrency-group deploy job on a self-hosted runner that infra-fails defaults to requiring manual group release, with the UI stating why.

Design change: group release on our fleet gated on the supervisor's local kill bound (T_kill), not a soft grace timeout; deploy credentials become attempt-scoped OIDC tokens with TTL ≤ lease TTL so targets reject expired attempts; self-hosted concurrency-group jobs default to manual release after infra-failure.

Q8 (Codex): A single org-month key cannot enforce 90-day deletion: destroying it 90 days after month-end retains early-month data for up to 31 extra days, while destroying it sooner deletes late-month data early. What exact key granularity, destruction schedule, and key-backup policy let you prove every object is unreadable by its individual 90-day deadline?

The arithmetic is right and month granularity was sloppy. Revised scheme, exact:

Granularity: one data key per org per day. Data written on day D is encrypted under k(org, D) — a random 256-bit key we generate, not a KMS key (100k orgs × 365 days is ~36M keys/year; KMS-managed keys at that count are absurd, wrapped data keys in a small key store are a few GB). Destruction schedule: k(org, D) is destroyed at exactly D+90, so every object meets its individual deadline — nothing retained late, nothing deleted early, and "available for 90 days" stays true too.

The key-backup question is the sharp part: if the key store sits in a database with 35-day backups, deleting a row leaves recoverable copies for 35 days and the whole scheme leaks. Answer: double-wrap. k(org, D) is wrapped under the org's KMS key and under a global per-day KMS wrapping key K_D (365 real KMS keys per year — a few dollars a month, fine). Destruction of day D is two independent acts: delete the wrapped rows from the key store, and schedule K_D for KMS deletion timed to complete at D+90 (KMS deletion has a pending window, so schedule at D+83 with a 7-day window). Now a backup of the key store is harmless — the copies it holds are ciphertext under a KMS key that provably no longer exists, and KMS key destruction is exactly the thing cloud providers give you an audit trail for (CloudTrail entry, no export path, no backup of the key material). That audit log plus the weekly read-path sampler from Q5 is the proof pack: for any object, its write date D, the destruction record for K_D at D+90, and a failed-read sample.

Design change: key granularity is org × day; destruction at exactly D+90; data keys double-wrapped under a global per-day KMS key whose scheduled deletion lands at D+90, making key-store backups unrecoverable by construction; compliance evidence = KMS deletion audit trail + read-path sampler.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The microVM bet is the industry's bet. The Firecracker paper (NSDI '20) is AWS explaining why Lambda and Fargate run every customer workload in its own VMM — "strong security and performance isolation" with minimal overhead, at millions of production workloads and trillions of requests a month. Fly.io's sandboxing and workload isolation post walks the same rejection ladder this design does — containers, hardened containers, gVisor, then lightweight VMs — and lands on Firecracker for concrete reasons the design only gestured at: a small memory-safe Rust VMM (the block device is ~1,400 lines including tests), self-restricted to ~40 syscalls, run under an external jailer. One correction to my own framing: the ~125ms figure is VMM boot; a CI runner VM with a full rootfs, Docker, and an agent takes longer, which is why the warm pool, not raw boot speed, carries the time-to-start budget. Depot, for what it's worth, skipped microVMs entirely and runs each job on its own EC2 instance backed by a standby pool — single-tenancy by machine rather than by VMM, same isolation goal, more hardware.

GitHub has published enough about Actions' orchestration internals to score the Temporal-vs-hand-rolled call. Their 2022 post on tripling max concurrent jobs describes the failure mode: the orchestrator kept each workflow's execution state as one event blob, read-modify-written on every transition and replayed from the start for long workflows. Moving to incremental per-event rows tripled concurrent jobs on GHES (2,200 to 7,000) and halved orchestration CPU. That's the design's "guarded row updates, not replayed history" argument, learned in production. And the axis they hit at real scale is the one the estimates here predicted: per their December 2025 retrospective, they rebuilt the Actions backend starting in early 2024, finished migrating in August 2025 at 71M jobs/day (3x growth from 23M), and the headline win is enterprises starting 7x more jobs per minute — job-start throughput, the same bottleneck that forced shard-parallel dispatch in this design.

Buildkite runs the split this design gives self-hosted runners as its entire product. Their architecture docs describe a SaaS control plane for orchestration, scheduling, and UI, with agents on customer infrastructure polling outbound over HTTPS — no inbound connections, matching the long-poll choice here — and a stronger privacy line than this design's hosted path: "sensitive data, such as source code and secrets, remain within your environment." The OIDC item on my evolution list is the road toward that posture, and GitHub's OIDC hardening docs show the mature mechanism: a per-job JWT carrying repo, branch, and environment claims, exchanged at the cloud provider for a short-lived credential, no stored secret anywhere.

Caching — which I deferred to the evolution section — is where an entire cohort of companies built their business, and their published numbers sharpen what "build caching first" has to mean. Depot reverse-engineered the Actions cache and found the default remote cache moves 100-150 MB/s; a small Go proxy on each runner redirecting cache traffic to nearby S3, with tuned parallelism (4 upload / 8 download streams), got them ~1 GB/s. Blacksmith did the same trick with an NGINX proxy inside the VM rewriting Azure-blob URLs to a co-located MinIO cluster: 49.8 MB/s to 327.5 MB/s. The lesson for this design: the content-addressed cache can reuse the artifact machinery's control plane, but the bytes must live inside the cell, next to the runners. Co-location is the feature.

The incident record backs the two security positions that survived interview pressure. The March 2025 tj-actions/changed-files compromise (Wiz's writeup, CVE-2025-30066) had a malicious action dump the runner worker's memory and print workflow secrets into logs — public on public repos — leaking AWS keys, GitHub PATs, npm tokens, and RSA keys. That is exactly the Q4 boundary: anything inside the job's trust domain is readable by job code, no agent split fixes it, and the real mitigations are the ones already in this design — short-lived OIDC credentials over stored secrets, and no secrets for fork PRs. CircleCI's January 2023 incident report is the control-plane version of the same lesson: malware on one engineer's laptop stole a session cookie, impersonation reached production, and customer env vars, tokens, and keys were exfiltrated — encryption at rest didn't help because "the third party extracted encryption keys from a running process." Their remediation list (OIDC adoption, automatic token rotation and expiry, drastically restricted production access) reads like this design's secrets section written after the breach instead of before it.

Updates from post-training information

Two 2026 developments I'd fold into the design:

Further reading