Durable Background-Job and Workflow Platform

What we're building, and the assumptions I'll run with

Customers of a developer platform want to say "run this function later, retry it if it fails, chain it into a workflow, and don't lose it" — and they want the platform to run the code, not their own fleet. We're building the queueing, orchestration, and execution layers as a product.

Assumptions I'm making beyond the prompt:

Customers package task code as container images (they already deploy services this way on the platform). A deploy registers named task handlers with the platform.
The SDK model is function-level: a handler receives a JSON-ish payload and returns a result or throws. Workflow code is also customer code, written against our SDK's step primitives.
Payloads are small most of the time — I'll assume p50 around 1 KB, capped at 256 KB inline, with larger blobs passed by reference.
"Never silently lost" means: once we ack an enqueue, the task reaches a terminal state (succeeded, failed-after-retries, canceled, or expired) and that state is visible to the customer. It does not mean exactly-once execution — that's impossible with arbitrary side effects, so we give at-least-once plus tools to control duplicates.
Regional product initially. Multi-zone durability is required; multi-region is evolution, not v1.

Requirements

Functional: enqueue with delay or cron; automatic retries with configurable policy; multi-step workflows with fan-out/fan-in that survive restarts and code deploys; status/history/failure inspection; cancellation; per-plan quotas.

Non-functional, in priority order:

Durability. An acked task survives worker crashes, deploys, and the loss of a zone.
Correctness of state. Task state transitions are the product; they must never go backwards or fork.
Fairness. One tenant enqueueing 5M tasks must not starve the other 99,999 apps.
Latency. Enqueue ack p99 under ~50 ms; schedule-to-start p50 under ~1 s for an unloaded queue. Explicitly not a real-time system — we trade dispatch latency for durability and fairness.
Throughput and elasticity to the numbers below.

Scale estimates

500M executions/day ≈ 5,800/s average. Daily peaks maybe 3–4x, plus single-tenant bursts on top; design the ingest path for ~50k enqueues/s sustained.
Each execution generates roughly 4–6 durable state writes (enqueued, leased, heartbeats folded in, completed, plus timer bookkeeping) → ~300k writes/s at peak. That's beyond one database; it's comfortable for ~128 shards at ~2–3k writes/s each.
Metadata: ~~2 KB/task including attempt records → ~1 TB/day. Keep 30 days hot (~~30 TB across shards), archive the rest to object storage.
Concurrency: if mean task duration is ~2 s, average concurrent running tasks ≈ 500M × 2s / 86,400s ≈ 12k; long-tail 24-hour tasks and bursts push the worker plane to plan for ~100k concurrent task slots.
Workflows: thousands of steps over weeks means a workflow's history can hit tens of MB. History must be paged and offloadable, never assumed to fit in one row.

These numbers drive the two central choices: the source of truth must be sharded, and dispatch must be pull-based with per-tenant flow control, because push-based dispatch collapses exactly when a tenant bursts.

Architecture

The heart of the system is a horizontally sharded transactional store (Postgres, ~128 shards, each a 3-AZ synchronous-replication cluster) holding task state, timers, and workflow histories, fronted by per-shard dispatcher processes that lease work to a pull-based worker plane running customer containers.

Why a sharded relational store as the spine, rather than the obvious alternatives:

A durable, partitioned event log (Kafka) as the queue itself — rejected as the source of truth. Kafka gives cheap durable ingest but no per-message ack, no delayed delivery, no per-tenant selective consumption, and head-of-line blocking inside a partition. A 24-hour task at the head of a partition is fatal. We still use Kafka, but only for the observability firehose where those properties don't matter.
A wide-column / KV store (Cassandra, DynamoDB) — rejected for the core because every state transition here is a multi-row transaction (flip task state + write attempt record + insert/delete a timer, atomically) and we need rich secondary access paths (by run_at, by state, by tenant+queue). Shard-local ACID transactions in Postgres give us both without distributed transactions, because we route all rows for a given task or workflow to the same shard.
Adopting a durable workflow engine (Temporal) wholesale — the strongest alternative, and we borrow its ideas (task queues, leases via timeouts, event-sourced progress). Rejected as the implementation because our execution model is bring-your-own-container, not SDK-managed deterministic workers: Temporal's replay model requires deterministic workflow code, which we can't enforce on arbitrary customer containers, and its namespace model doesn't map cleanly onto 100k tenants with per-plan fairness we control. We take the pattern, own the engine, and pick a simpler checkpointing model (below).

flowchart LR
  subgraph Client["Customer app"]
    SDK[SDK / REST]
  end

  subgraph ControlPlane["Control plane"]
    GW[API gateway<br/>authn, idempotency, admission]
    QS[Quota service<br/>token buckets]
    CRON[Cron service<br/>materializes next firing]
  end

  subgraph Core["Core (per shard x128)"]
    PG[(Task store<br/>Postgres, 3-AZ sync repl)]
    DISP[Dispatcher<br/>lease, timers, fairness]
  end

  subgraph WorkerPlane["Worker plane"]
    POLL[Dispatch API<br/>long-poll]
    W1[Tenant A microVMs]
    W2[Tenant B microVMs]
    AS[Autoscaler]
  end

  subgraph Obs["Observability"]
    K[Event log Kafka]
    CH[(Analytics store<br/>ClickHouse)]
    S3[(Blob/archive S3)]
  end

  SDK --> GW --> PG
  GW <--> QS
  CRON --> PG
  DISP <--> PG
  W1 & W2 <--> POLL <--> DISP
  AS -.scales.-> W1 & W2
  DISP --> K --> CH
  PG -. payloads, archived history .-> S3

Data flow, happy path. Enqueue hits the gateway; after auth and an admission check against the quota service, the gateway writes the task row (state ENQUEUED, plus a timer row if delayed) to the owning shard and acks. The shard's dispatcher notices ready work (LISTEN/NOTIFY plus a polling fallback), applies fairness, and hands the task to a long-polling worker with a lease. The worker runs the handler, heartbeats, and reports completion; the dispatcher commits the terminal state and emits an event to the firehose. Enqueue ack means "committed to a 3-AZ quorum" — durability is settled at step one, everything after is liveness.

Sharding

Route by hash(task_id) for standalone tasks and hash(workflow_id) for workflows and all their steps, so every transaction stays shard-local. I deliberately do not shard by tenant: a tenant bursting 5M tasks then spreads across all 128 shards instead of melting one. The costs are (a) cross-shard tenant-wide queries — served by the analytics store, not the OLTP path — and (b) per-tenant concurrency limits need coordination across shards, which the quota service provides. Shard map lives in a small strongly-consistent metadata store (etcd); resharding is by splitting hash ranges, virtual-node style.

Data model

Per shard, the load-bearing tables:

tasks(
  task_id uuid PK, tenant_id, app_id, queue,
  task_type,                    -- registered handler name
  state,                        -- ENQUEUED|LEASED|RETRY_WAIT|SUCCEEDED|FAILED|CANCELED|EXPIRED
  attempt int, max_attempts int,
  payload bytea | payload_ref,  -- inline <=256KB else S3 pointer
  retry_policy jsonb, timeouts jsonb,
  idempotency_key, run_at, expires_at,
  workflow_id nullable, parent_step_id nullable,
  lease_id, lease_expires_at, worker_id,
  created_at, updated_at
)
task_attempts(task_id, attempt, worker_id, started_at, finished_at,
              outcome, error_class, error_msg, stack_ref, result_ref)
timers(timer_id PK, fire_at, kind,        -- delay|retry|lease_expiry|timeout|sleep
       task_id/workflow_id, payload) INDEX (fire_at)
workflow_steps(workflow_id, step_id,      -- deterministic: hash(name + sequence)
       state, result_ref, children_total, children_done, updated_at)
schedules(schedule_id, tenant_id, cron_expr, task_template,
          next_fire_at, overlap_policy, last_fired_at)   -- on a control shard

Idempotency keys get a unique index on (tenant_id, app_id, idempotency_key) with a 24-hour default retention window, so a client retrying an enqueue after a lost ack gets the original task_id back instead of a duplicate. This is the first of three idempotency layers; the other two are below.

APIs

Customer-facing (REST plus SDK sugar):

POST /v1/tasks            {task_type, payload, queue?, delay?|run_at?,
                           idempotency_key?, retry?, timeouts?}  -> {task_id}
POST /v1/tasks:batch      up to 1,000 per call — the answer to million-task bursts
GET  /v1/tasks/{id}       state, attempts, timings, error, result
GET  /v1/tasks?state=&queue=&since=      (served from analytics store)
POST /v1/tasks/{id}/cancel
POST /v1/schedules        {cron, task_template, overlap_policy}
POST /v1/workflows        {workflow_type, input, ...}   -- same shape, orchestrated

Worker protocol (internal, spoken by the runner shim we inject into customer containers):

POST /worker/poll       {app_id, queues[], capacity} -> [{task_id, lease_id, payload, attempt, deadline}]
POST /worker/heartbeat  {lease_id, progress?}        -> {action: CONTINUE | CANCEL}
POST /worker/complete   {lease_id, outcome, result | error}

Pull, not push: the worker declares capacity and the dispatcher chooses what to hand it. That single decision is what makes quotas, fairness, and burst absorption tractable — backpressure is the default, not a bolt-on.

SDK sketch:

@task(retries=RetryPolicy(max_attempts=5, backoff="exp", base=2s, jitter=True),
      timeout="10m")
def resize_image(ctx, payload): ...

@workflow
def onboard(ctx, user):
    acct = ctx.step("create_account", create_account, user)      # memoized
    results = ctx.parallel("welcome", [(send_email, u) for u in user.contacts])
    ctx.sleep("cool_off", days=1)                                 # durable timer
    ctx.step("activate", activate, acct)

The task lifecycle: leases, heartbeats, timeouts, retries

Leases. A dispatch is a lease: lease_id (fenced, monotonic per task), lease_expires_at = now + 30s by default. Workers heartbeat every 10s to extend. Lease expiry is detected by the shard's timer scan; the task flips to RETRY_WAIT (attempt++) and any late complete call from the old lease is rejected by the fencing token — this is what prevents a GC-paused zombie worker from double-completing or resurrecting a task. A crashed worker therefore costs at most one lease interval of delay, and nothing is lost. For 24-hour tasks the lease stays short; it's the heartbeat that's long-lived, and a missed-heartbeat requeue of an 8-hour task is exactly why long tasks should checkpoint (below).

Timeouts. Three, all enforced server-side by timers, never trusted to the worker: schedule_to_start (staleness guard — an email task that waited 6 hours may be worse than dropped, so tenants can expire it), start_to_close per attempt (dispatcher sends CANCEL on heartbeat, then instructs the compute layer to kill the container 30s later), and schedule_to_close total, which caps retry loops in wall-clock time.

Retries. Default: exponential backoff with full jitter (1s base, 2x, cap 1h), max 5 attempts. Customers tune per task type and per error class — the SDK distinguishes RetryableError from FatalError (bad payload, 4xx from a downstream) so a validation failure doesn't burn five attempts. Jitter is non-negotiable at this scale: a downstream outage failing 100k tasks simultaneously must not produce a synchronized retry stampede. Exhausted tasks land in FAILED with the payload, final error, and stack retained — a dead-letter state, queryable and bulk-retryable from the dashboard (POST /tasks:retry?filter=...), which is how customers recover after they fix a bug. A poison task that crashes its container repeatedly is contained by the same mechanism plus a circuit breaker per task type: repeated container crashes trip the breaker and pause dispatch for that type, alerting the customer, instead of grinding their whole fleet.

Idempotency and at-least-once, stated honestly. Lease expiry means a task can run twice — the worker may have finished the side effect and died before complete landed. We can't fix that for arbitrary side effects, so the contract is at-least-once execution with three duplicate controls: (1) enqueue-side idempotency keys, above; (2) every delivery carries task_id and attempt, and the docs push customers hard to key external writes on task_id (charge key, upsert key); (3) inside workflows, step memoization means a completed step is never re-executed even when the surrounding code re-runs. Customers who need stronger guarantees put the side effect behind an idempotent API of their own; we say this in the docs rather than pretending exactly-once.

Scheduling: delays and cron

Delayed tasks are just rows with run_at in the future; each shard's dispatcher runs a timer loop — SELECT ... WHERE fire_at <= now ORDER BY fire_at LIMIT k FOR UPDATE SKIP LOCKED — every ~250 ms. SKIP LOCKED lets multiple dispatcher threads drain the same shard's timer table without contention. One mechanism serves five masters: delays, retry backoff, lease expiry, timeouts, and workflow sleeps are all timer rows. That uniformity is deliberate; timers are the mechanism everything else leans on, so there is exactly one of them to make correct.

Cron lives in a separate cron service because it's a different shape of problem: ~low volume of schedule definitions, but firing must be exactly-once-ish and drift-free. It materializes only the next occurrence per schedule as a timer (never the infinite future), and on firing it enqueues the concrete task with idempotency key schedule_id + scheduled_time — so a crashed-and-recovered cron service that fires twice dedupes at the enqueue layer. Overlap policy per schedule (allow | skip | cancel_previous) handles the job that sometimes runs longer than its interval. Misfires after downtime follow a catch-up policy: fire once for the missed window, don't replay every missed tick.

Workflows: checkpointing, fan-out/fan-in

The model is durable step memoization (the Inngest/Restate pattern) rather than Temporal-style full deterministic replay. Workflow code runs in the customer's container like any task, but every ctx.step(...) call is a checkpoint: the runner asks the shard "has step create_account#1 completed?"; if yes, it returns the stored result without executing; if no, it executes and commits the result before proceeding. When a worker dies mid-workflow — or the customer deploys new code — the workflow function simply re-runs from the top, fast-forwards through memoized steps, and resumes at the first incomplete one.

Why this over replay: replay demands deterministic workflow code (no wall clocks, no random, version-pinned logic), which we cannot enforce on arbitrary containers, and non-determinism bugs under replay are the worst debugging experience in the Temporal world. The costs we accept: re-running the function's glue code from the top (cheap — steps dominate), results must be serializable, and code between steps re-executes on resume, so the rule we teach is "side effects go inside steps." Deploy safety follows the same rule: steps are addressed by name, so inserting new steps after completed ones is safe, and we detect a renamed/removed completed step and fail loudly with a versioning error instead of silently misbehaving.

Long workflows don't hold a worker. ctx.sleep("cool_off", days=1) commits a timer and releases the container; the workflow has no live process for that day. Same for ctx.wait_for_event(...). A thousand-step, three-week workflow is thus mostly rows, occasionally a process — which is the only way weeks-long workflows are affordable.

Fan-out/fan-in is a barrier in the store. ctx.parallel(...) enqueues N child tasks in batched shard-local transactions (children hash to the parent's shard) and writes children_total = N on the step. Each child completion decrements the barrier in the same transaction that commits the child's terminal state; the decrement that reaches zero enqueues a "resume parent" task. No process waits on N children — a 10,000-branch fan-out costs 10,001 rows, not a blocked worker. Partial failure policy is per-step: fail_fast (cancel siblings on first fatal child) or collect (parent receives a results array containing errors to handle in code).

Individual long tasks (not workflows) get a smaller tool: ctx.checkpoint(cursor) piggybacked on heartbeats, persisted with the lease, and delivered back on the next attempt — so the 24-hour backfill that dies at hour 8 restarts at record 3.2M, not record 0.

Quotas and fairness

Two enforcement points, because throughput and concurrency are different problems.

Admission (throughput): the gateway checks a per-tenant token bucket — plan-sized enqueue rate, generous burst allowance — in a replicated in-memory counter store (Redis, Lua-scripted buckets). Over the burst allowance, the API returns 429 with Retry-After. Millions-in-a-burst is a supported pattern via the batch API: 1,000 tasks per call means a 5M-task burst is 5,000 requests, and the queue absorbs it — the quota that actually matters downstream is dispatch, not enqueue. Redis here is a soft dependency: on unavailability we fail open with conservative static limits, because refusing all enqueues over a rate-limiter outage violates the durability promise for no good reason.

Dispatch (concurrency + fairness): each tenant has a plan-level max_concurrent (say 50 / 500 / 5,000), which the quota service partitions into per-shard sub-budgets, refreshed every few seconds as usage skews — slightly stale, never wrong by much, and crucially requiring no cross-shard transaction on the dispatch path. Within a shard, the dispatcher runs weighted round-robin over tenants with ready work (weights by plan), not FIFO over arrival time. That's the fairness core: tenant A's 5M queued tasks and tenant B's 10 queued tasks both get their next task dispatched on the next scheduling round; A's backlog costs A latency, not B. Within a tenant, customers get named queues with relative priorities so their own batch backfill doesn't starve their own user-facing jobs.

The worker plane

The compute layer is external per the prompt; the interface I need from it: create_sandbox(image, cpu, mem, env) -> sandbox_id, kill(sandbox_id, grace), list/health, and a log/stdio tap. I'd specify sandboxes as microVM-isolated containers (Firecracker): tenants run arbitrary code, so container-only isolation (shared kernel) is not an acceptable boundary between tenants — kernel exploits from a hostile tenant are in-scope. Isolation policy: sandboxes are strictly single-tenant, warm-pooled and reused across tasks of the same app version by default (amortizes cold start), with a per-task-fresh-sandbox opt-in for tenants who need it (e.g., running semi-trusted third-party payloads). Sandboxes get a locked-down egress profile — customer-configured allowlists, no platform-internal network access except the dispatch API — and per-tenant envelope-encrypted payloads (KMS) so a compromised platform host doesn't read every tenant's data.

Inside each sandbox, our runner shim (injected at image build or as the entrypoint) speaks the worker protocol: polls, invokes the registered handler in-process, heartbeats, streams logs, reports completion. Customer code never holds credentials to the core.

Autoscaling is per tenant-app, driven by the store's own signals: target concurrency = min(quota, f(ready_backlog, arrival rate, observed task duration)), scale up aggressively on backlog growth (seconds matter), scale down lazily via idle warm pools (cold starts cost more than idle memory). Long tasks pin sandboxes for hours, so the scaler tracks slots, not sandbox counts. Deploys of customer code drain gracefully: old-version sandboxes stop polling, finish leased short tasks, and long tasks either run to completion on the old version (default, up to a cap) or get requeued — workflows don't care either way, because resume-by-memoization is version-tolerant by construction.

Cancellation is cooperative first, forceful second. POST /cancel sets a flag on the task row; on ENQUEUED/RETRY_WAIT tasks it's an immediate transition to CANCELED (the timer row is deleted in the same transaction). On a LEASED task, the next heartbeat returns action: CANCEL; the SDK surfaces ctx.cancelled() for graceful cleanup, and if the worker hasn't reported terminal state within the grace window (30s default) the dispatcher tells the compute layer to kill the sandbox and marks the task CANCELED. Workflow cancellation cascades: the engine cancels outstanding children and pending timers, and runs customer-registered compensation steps if defined. The honest caveat, documented: cancellation cannot un-send an email — it stops future work, it doesn't undo side effects.

Observability

Every state transition the dispatcher commits is also emitted (transactional outbox on the shard, tailed and published — never a dual-write) to a durable, partitioned event log (Kafka), feeding a columnar analytics store (ClickHouse) for the dashboard: per-queue depth and age, throughput, error rates, p50/p99 durations, retry heatmaps. The OLTP shards answer "what is task X doing right now"; ClickHouse answers "show me all failures of resize_image since the 14:00 deploy" — mixing those workloads on the task store is how the dispatch path dies, so they're separated by construction.

Per-task inspection shows the full attempt timeline: each attempt's worker, timings, error class, stack trace, captured stdout/stderr (tapped from the sandbox, stored in object storage (S3), linked from the attempt row), and the exact payload — with payload visibility per-tenant configurable since payloads carry PII. Workflows render as a step graph with per-step status and results. Webhooks and an event-subscription API push terminal-state notifications so customers don't poll. After 30 days, task rows archive to S3 (Parquet) and remain queryable through ClickHouse; the OLTP shards stay lean.

Platform-side, the golden signals are schedule-to-start latency per shard (the single best health metric — it degrades first under almost every failure), timer-loop lag, lease-expiry rate (spikes mean a sick worker pool or a poison task type), and per-tenant dispatch share (fairness regression detector).

Failure modes

Worker/sandbox crash: lease expires, task requeues, attempt++. Bounded delay, no loss. Covered above.
Dispatcher crash: dispatchers are stateless; shard ownership is held via leases in etcd, and a standby takes over in seconds. In-flight leases keep ticking in the store, so takeover loses no work — at worst, dispatch pauses briefly on that shard.
Shard primary failure: synchronous replication means failover (Patroni-style) loses zero acked writes; the shard is unavailable for ~10–30s. Enqueues route by hash, so a down shard rejects ~1/128th of new enqueues; the gateway retries a fresh task_id onto a healthy shard (safe — the write never acked), while tasks resident on the failing shard wait out the failover. This is the "correct when degraded" posture: we accept brief per-shard latency, never loss.
Zone failure: every shard has replicas in 3 AZs, dispatchers and workers are zone-spread, Kafka/Redis/etcd are zone-redundant. Losing a zone costs capacity (~1/3), triggers shard failovers, and the autoscaler rebuilds worker capacity in surviving zones. No data loss; elevated schedule-to-start latency until capacity recovers.
Quota service (Redis) down: fail open with static conservative limits (enqueue keeps working); dispatch falls back to last-known per-shard budgets. Fairness degrades before availability does.
Retry stampede / downstream outage: jitter plus per-task-type circuit breakers plus tenant concurrency caps bound the blast radius; the backlog is the buffer, which is the point of a queue.
Clock skew: all timer decisions use the shard database's clock, not worker clocks, so skew shows up as slight lateness, never as incorrect ordering of state transitions.

Tradeoffs I'm consciously making

At-least-once with duplicate controls, not exactly-once. Anyone selling exactly-once side effects to arbitrary containers is selling something else. We spend our effort making duplicates rare (fenced leases, generous heartbeats) and controllable (keys, memoized steps).
Step memoization over deterministic replay: better DX and deploy-safety for BYO containers, at the cost of re-running glue code and requiring serializable step results.
Sharded Postgres over a distributed KV store: shard-local transactions and secondary indexes are exactly the workload; the price is owning a shard map, resharding tooling, and per-shard failover machinery. That's real operational cost and I'd staff it — it's still cheaper than building transactional semantics on top of a KV store.
Pull-based dispatch over push: adds a long-poll hop of latency; buys backpressure, fairness, and burst tolerance. For a system whose stated floor is ~1 s schedule-to-start, that trade is easy.
Hash-sharding over tenant-sharding: spreads bursts, complicates tenant-wide queries and quotas; both complications land in components (analytics store, quota service) that tolerate slight staleness.

Evolution

v1 ships single-region, three-AZ, with tasks, retries, delays, cron, and linear workflows; fan-out and wait_for_event follow once the barrier machinery is proven under fault injection (we'd test lease expiry and shard failover with induced faults before GA, since those paths never run in the demo and always run in production). Later: multi-region — first as isolated regional deployments with per-app home regions (simple, honest), then cross-region task-metadata replication for regional-failover of scheduling state, which is a substantial project on its own. Also on the roadmap: priority aging within tenant queues, per-step compute sizing for workflows, and result streaming for tasks that produce large outputs.

The promise the whole design serves, and the line I'd repeat to the team building it: an acked task always reaches a terminal state the customer can see. Every component above either makes that true or makes it observable.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): A shard may commit an enqueue and fail before the client receives the acknowledgment. How can the gateway safely retry with a fresh task_id on another shard when idempotency uniqueness is shard-local and routing is by hash(task_id)—what prevents two durable tasks from being created?

This lands. As written, the idempotency unique index lives on the shard chosen by hash(task_id), and a gateway retry mints a fresh task_id, routes to a different shard, and the index never fires. Two fixes. First: when an idempotency key is present, shard routing is hash(tenant_id, idempotency_key), not hash(task_id) — the retry deterministically lands on the shard that may hold the original row, and the unique index does its job, returning the original task_id. The consequence I accept: a keyed enqueue cannot fail over to a healthy shard; it waits out that shard's 10–30s failover, because "retry elsewhere" and "dedupe here" are incompatible. Second: the SDK auto-generates an idempotency key per logical enqueue call (UUID minted client-side, carried across transport retries), so the keyed path is the default path.

What remains: raw REST callers who omit the key get at-least-once enqueue across an ack-lost failover — documented, not silent. The original text's "gateway retries a fresh task_id onto a healthy shard (safe)" was wrong for the committed-but-unacked case and I'm striking it.

Design change: route keyed enqueues by hash(tenant_id, idempotency_key); SDK auto-generates idempotency keys; keyed enqueues never retry onto a different shard; keyless enqueues documented as at-least-once across failover.

Q2 (Codex): Two workflow runners can overlap after a lease expires: both can observe an incomplete ctx.step, execute its arbitrary side effect, and then race to checkpoint it. What exact transactional claim/fencing protocol prevents concurrent step execution, and what duplicate-control guarantee remains if either runner crashes after the side effect but before the checkpoint?

Partially covered, partially conceded. What the design already has: the workflow task itself is leased with a fenced, monotonic lease_id. What it didn't say and now does: every step-check and step-checkpoint RPC carries that lease_id, and the shard rejects any RPC bearing a stale token, the same way /worker/complete is fenced. The zombie runner's checkpoint commit loses the race by construction — history never forks, which is the correctness-of-state promise.

What fencing cannot give: the zombie may have executed the side effect before its checkpoint bounced, and separately, any runner can crash after the side effect but before the checkpoint commit. Both re-execute the step. That is the at-least-once contract applied to steps, stated plainly: memoization guarantees a step whose result is committed never re-executes; it guarantees nothing about a step that ran but didn't commit. The added mechanism to shrink the window: before executing a non-memoized step, the runner writes a fenced claim row (step state RUNNING, lease_id) — a new runner arriving after lease takeover sees the claim from a dead lease, knows the step may have partially run, and the SDK surfaces that as attempt>1 to the step function so customers can key their side effect. Duplicate side effects inside steps remain possible and remain the customer's idempotency-key problem, exactly like standalone tasks.

Design change: step-check and step-checkpoint RPCs are fenced by the workflow's lease_id; add fenced per-step claim rows, with step attempt numbers exposed to customer code.

Q3 (Codex): Tasks store only a handler name, not an immutable image or workflow-code version. After a deploy removes a handler, changes payload/result schemas, or changes control flow between memoized steps, which code version resumes queued and weeks-old workflows, and how is that version retained without making deployments unsafe or storage unbounded?

Real gap — the design said "insert-safe, fail loudly on rename" and dodged the version question. Concretely: a workflow records app_version (the deployed image digest) at start. Resume policy is per workflow type: latest (default) or pinned. Latest re-runs the function on the current deploy under the step-compatibility check (tightened in Q8 below). Pinned resumes on the recorded image digest — we're the deploy platform, the image is already in our registry, so pinning is a registry retention policy: any digest refcounted by a live pinned workflow is retained, and the refcount releases at workflow terminal state. Storage is bounded by live pinned workflows, not by deploy history.

For plain tasks, dispatch always uses the current active version — deliberate, because it's what makes fix-the-bug-then-bulk-retry from the dead-letter state work. Deploys get a preflight: the deploy API diffs registered handler names against queued tasks and live workflows and warns "this deploy removes handler X referenced by N queued tasks"; if the customer proceeds, those tasks move to FAILED (dead-letter) at dispatch time rather than sitting in limbo. Payload schema drift between enqueue and dispatch is the customer's contract with themselves; we surface attempt errors, we don't validate their schemas.

Design change: workflows record start-time image digest; per-type resume policy latest|pinned; refcounted registry retention for pinned digests; deploy preflight against queued work, with removed-handler tasks dead-lettered.

Q4 (Codex): Every fan-out child is placed on the workflow's single shard, and every completion updates the same barrier row. How does a 10,000-way fan-in avoid serial lock contention and a hot shard, especially when many large workflows complete concurrently or fail_fast must cancel thousands of siblings?

Concede both halves. The single barrier row is a real serialization point: every child completion takes a row lock on it, low-thousands of updates/s best case, and a few large workflows completing concurrently can stall the shard. Fix one, contention: stripe the barrier into K sub-counters (child assigned by sequence mod K, K sized to the fan-out); a child decrements its stripe, and only a stripe hitting zero touches the top-level barrier — the hot row sees K writes, not N. Fix two, the hot shard: shard-local children stay right for small fan-outs (one transaction, no coordination) and wrong at 10k, so fan-outs above a threshold (~1,000 children) spill: children route by hash(child_task_id) across shards, and completion sends an idempotent "child done" message (keyed on child_id, deduped by a done-marker row) to the parent's shard via the internal enqueue path — at-least-once delivery plus an idempotent decrement is exactly the machinery the platform already sells.

Cost: cross-shard fan-in loses single-transaction atomicity and gains a small completion-visibility lag; the barrier fires within internal delivery latency, fine for a system whose floor is ~1s. fail_fast at 10k stops being a synchronous cancel: it flips a "canceling" flag on the step and cancellation propagates asynchronously per stripe — first-fatal-child to full sibling cancel is seconds, documented.

Design change: striped barrier counters; fan-outs over ~1k children spill across shards with idempotent cross-shard completion messages; fail_fast becomes an async flag-then-propagate cancel at scale.

Q5 (Codex): Per-tenant concurrency is described as a hard plan quota, yet 128 dispatchers use dynamically repartitioned, stale sub-budgets and continue from cached budgets during quota-service failure. What allocation-transfer protocol prevents double-spending during rebalance or failover, and what is the actual worst-case quota overshoot?

The design said "refreshed every few seconds, slightly stale, never wrong by much" — that's a vibe, not a protocol. The protocol: sub-budgets are leased grants — the quota service issues each shard {slots, epoch, TTL≈10s}; the dispatcher treats the grant as a ceiling on new dispatches (running tasks count against it; a grant shrunk below current usage just stops new dispatch until usage drains). Rebalancing is shrink-before-grow: the service issues the shrink and issues the corresponding grow only after the shrunken shard acks the new epoch or its old grant expires. A partitioned dispatcher stops dispatching new work on grant expiry.

On quota-service outage, shards freeze on last-acked grants, which by the invariant still sum to the cap — fairness rebalancing stops, the cap holds. The residual is undershoot, not overshoot: a busy shard can't borrow an idle shard's slack for up to one TTL. (My original claim of zero worst-case overshoot was too strong — Q7 below corrects it for the expired-grant-with-running-tasks case.)

Design change: sub-budgets become TTL'd, epoch-fenced leased grants with shrink-before-grow ordering; dispatch stops on grant expiry.

Q6 (Codex): The runner shim and hostile customer code execute inside the same customer sandbox, yet the shim can poll, heartbeat, complete tasks, and access payloads while "customer code never holds credentials to the core." What isolation and attestation mechanism prevents customer code from stealing those credentials, forging completions, fetching another task, or extending leases indefinitely?

Full concession on the stated architecture — "customer code never holds credentials to the core" is false if the shim holding those credentials runs in the same microVM as the customer's process. The fix moves the trust boundary to where the isolation already is: split the shim. A host agent runs on the Firecracker host, outside every guest, holds the real dispatch-API credentials (per-host mTLS from the platform CA), and is bound at create_sandbox time to exactly one sandbox_id→(tenant, app, version). The in-guest stub talks only to its host agent over vsock and holds zero credentials — it receives a payload, invokes the handler, reports an outcome. Poll, heartbeat, and complete are host-agent calls; lease_ids never enter the guest. So a hostile guest cannot forge a completion for a task it wasn't handed, cannot fetch another tenant's task (the dispatch API authorizes by the agent's registered sandbox identity, not anything the guest asserts), and cannot extend its lease — heartbeats are agent-driven, and the server-side start_to_close timeout already caps total runtime no matter how healthy the heartbeats look.

What the guest can still do is lie about the outcome of its own task. That's not a vulnerability; the customer running the code can always sabotage their own tasks. The boundary we defend is between tenants and between tenants and the platform, and after this change the guest holds nothing that crosses it.

Design change: split the runner shim into a credential-holding host agent (outside the microVM, mTLS, sandbox-identity-bound) and a zero-credential in-guest stub speaking vsock; lease_ids never enter the guest.

Round 2 — pushback

Q7 (Codex): Your zero-overshoot claim is false: if shard A's grant expires while its 5,000 long-running tasks continue, and those slots are reassigned to shard B, what concrete protocol prevents both sets from exceeding max_concurrent without killing healthy tasks?

Correct — the claim was false. Grant expiry stops new dispatches, but running tasks keep running (killing healthy work would invert the durability priority), so TTL-based reclamation alone double-spends: 5,000 still running on A plus 5,000 newly granted to B. The fix is that reclamation is drain-based, not time-based. Slots have two counts: granted and occupied. Occupancy is not the dispatcher's claim — it's derivable from the store of record, because lease creation and terminal-state commits write per-tenant running counts in the shard DB transactionally. When shard A's grant expires, the quota service reclaims only grant − occupied as read from A's database (the dispatcher being partitioned doesn't make the DB unreadable — they fail independently); the occupied remainder returns to the pool incrementally as the reconciler tails A's completions and lease expiries. If the shard DB itself is unreachable, its last-known occupancy is frozen as still-occupied — pessimistic, so the failure mode is underutilization, never double-spend.

The honest worst-case after this fix: overshoot zero, at the price of reclamation latency bounded by actual task drain (a 24-hour task holds its slot for 24 hours — as it should; it is genuinely consuming the tenant's concurrency), plus undershoot up to one lease-expiry interval for tasks that died but whose leases haven't expired yet.

Design change: grant reclamation is drain-based — the quota service reads occupancy from the shard store of record and reclaims only unoccupied slots on expiry; unreachable shards freeze as fully occupied.

Q8 (Codex): If "latest" workflow code deletes or conditionally skips a completed step and proceeds to a new side effect, how can the engine "fail loudly" before executing it when steps are discovered dynamically and the absence of a ctx.step call is unobservable?

Right — the absence of a call is unobservable, so name-only matching can't deliver the "fail loudly" I promised. What is observable is positional divergence, and the schema already carries it: step_id is hash(name + sequence). So the check becomes positional log matching: the engine keeps the workflow's committed step log ordered by sequence; on resume, the Nth ctx.step call must match the name at position N in the log. The check runs when the call arrives, before executing anything. If new code deleted a completed step, the next step call it makes sits at the wrong position — mismatch, loud versioning error, no new side effect executed. This doesn't break legitimate dynamic workflows: conditionals and loops that branch on memoized step results are deterministic on resume, because those results come from the log, not from re-execution. The non-determinism that can still shift positions is glue-code inputs outside steps (wall clock, random) — and positional matching converts exactly that from silent misbehavior into a loud error, which is the point of the existing "decisions go inside steps" rule.

Two honest limits. First, a side effect the new code performs outside any ctx.step runs before any check can fire — glue code is arbitrary, and no engine that runs customer containers can see it; that's the documented contract, unchanged. Second, positional strictness makes intentional migrations trip the error, so it needs an escape hatch: a ctx.version("name") marker API (the Temporal patch pattern) that records a branch point in the log, letting customers change control flow for in-flight workflows deliberately instead of by exception.

Design change: step compatibility checking becomes positional log matching (checked before execution at each step call); add a ctx.version() marker API for intentional control-flow migrations.

What changed, summarized

Keyed enqueues route by hash(tenant_id, idempotency_key); SDK auto-generates keys; keyed retries stay on the owning shard through failover; the "retry a fresh task_id onto a healthy shard" claim is struck for the committed-but-unacked case.
Workflow step-check/checkpoint RPCs are fenced by the workflow lease_id; fenced per-step claim rows added; step attempt numbers exposed to customer code.
Workflows record their start-time image digest; per-type resume policy latest|pinned; refcounted registry retention; deploy preflight dead-letters tasks whose handlers a deploy removes.
Fan-in barriers are striped into sub-counters; fan-outs over ~1k children spill across shards with idempotent cross-shard completion messages; large fail_fast cancels are async.
Per-shard concurrency sub-budgets become TTL'd, epoch-fenced leased grants with shrink-before-grow; reclamation is drain-based against store-of-record occupancy, so quota-service failures and partitions cause undershoot, never overshoot.
The runner shim splits into a credential-holding host agent outside the microVM and a zero-credential in-guest stub over vsock; lease_ids never enter the guest.
Step compatibility checking moves from name matching to positional log matching, with a ctx.version() marker API for intentional migrations.

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The core shape — hash a workflow id to a shard, keep every state transition a shard-local transaction, and use a transactional outbox to feed downstream consumers — is exactly how Temporal builds its history service. Their architecture doc describes history shards that own the full lifecycle of their workflows, atomically committing history events, mutable state, and transfer/timer tasks in one transaction, with the transfer-task processor named outright as the transactional outbox pattern. My timers table doing quintuple duty (delays, retries, lease expiry, timeouts, sleeps) mirrors their timer task queue. The scale ceiling is proven too: Uber's Cadence 1.0 announcement reports 12 billion executions and 270 billion actions per month across 1,000+ services — well past this design's 500M/day — on the same sharded history architecture, and lists hot-shard detection and noisy-neighbor prevention as first-class features, which matches my fairness-as-priority-3 call.

On memoization versus deterministic replay, the design sits where Inngest and Restate landed and Temporal didn't. Inngest's execution docs describe the identical mechanism: re-invoke the function, skip memoized steps, inject stored results. One divergence worth owning: their versioning docs choose "graceful determinism" — step reordering warns but completes, removed completed steps are silently ignored — where my Q8 answer chose strict positional matching that fails loudly. Theirs is friendlier; mine catches the deleted-step-then-new-side-effect case Codex raised, which graceful mode lets through. Restate's first-principles engine post is the strongest validation of the fencing story: they use epoch-bumped leadership where appends from a deposed leader are ignored — structurally the same move as my fenced lease_ids on checkpoint RPCs and epoch-fenced quota grants. They built a bespoke replicated log (Bifrost) where I chose sharded Postgres; at their measured ~94k actions/s that's defensible, and at my ~300k writes/s peak it's the road I said I'd rather not build in v1.

The lease/heartbeat/at-least-once contract is the industry's settled answer, not a compromise. SQS's visibility timeout docs are the canonical statement: receive hides but doesn't delete, heartbeat by extending the timeout, and — stated plainly by AWS — at-least-once means a message can be delivered twice even inside the visibility window. SQS caps extensions at 12 hours and tells you to reach for Step Functions past that, which is the same conclusion as my 24-hour-task rule: short lease, long-lived heartbeat, checkpoint. And the Step Functions workflow-type docs show even AWS splits the guarantee: Standard is "exactly-once" for state transitions (the orchestrator's own bookkeeping — my "correctness of state" promise), while Express is at-least-once and the docs tell you to keep actions idempotent. Exactly-once state, at-least-once side effects is precisely the line this design draws.

The Postgres-queue mechanics have public scar tissue that matches my numbers. DBOS's Making Postgres Queues Scale walks the exact ladder: FOR UPDATE SKIP LOCKED to get past ~100/s of worker contention, serialization failures past ~1,000 dequeues/s (fixed by dropping to READ COMMITTED where flow control allows), partial indexes to tame autovacuum — reaching 30k executions/s. My 2–3k writes/s per shard sits comfortably under what they wrung from single instances. For fairness, Hatchet's multi-tenant queues post solves my exact tenant-A-buries-tenant-B problem, with one refinement I'd steal: they compute round-robin ordering at write time by assigning tasks into per-group ID blocks, because read-time window functions collapse under a deep backlog. My dispatcher does weighted round-robin at dispatch time; at millions queued per tenant, write-time sequencing is the cheaper invariant.

Render itself shipped this product while I was designing it on paper. Render Workflows launches tasks as SDK-wrapped functions (TypeScript and Python), each task in its own container with independent retries, timeouts, and compute, with chaining and parallel fan-out — the same task-as-function, platform-runs-the-compute model assumed here. Two divergences: they run every task in a fresh per-task instance (my design warm-pools and reuses sandboxes per app version, taking the cold-start trade the other way), and their v1 shipped without cron or state checkpointing — both named as roadmap, both in my v1 cut. That's a real data point that my v1 scope was drawn generously; their sequencing (tasks and chaining first, durable timers and checkpoints later) is the more shippable order.

Updates from post-training information

Render Workflows moved to public beta on April 7, 2026 — after my training cutoff — with TypeScript and Python SDKs, automatic queuing/retries/state management, and parallel execution. The announcement confirms the beta still lacks cron triggers, workflow pause/resume, and state checkpointing (all listed as planned), so the checkpoint-and-resume machinery this design spends its hardest sections on is exactly the part Render hasn't shipped yet.