IAM for a Developer Platform: Humans, Workloads, and Agents

The design in one paragraph

Every request carries a short-lived signed token (a JWT, ES256, 10-minute TTL) that the API gateway verifies locally with cached public keys — no network hop for authentication. Long-lived secrets never travel on requests: dashboard sessions, legacy API keys, workload identities, and agent grants are all exchange credentials that a token service (an STS, in the AWS Security Token Service sense) converts into those short-lived tokens. Authorization is a Zanzibar-style relationship-based authorization store (SpiceDB) — because "member of team X, which has deploy rights on project Y, which contains service Z" is a graph question, not a role lookup — with regional read replicas and a decision cache at the gateway to stay inside the 10 ms budget. Every issuance and every check outcome flows into an append-only audit pipeline built on a distributed log (Kafka). Agents never hold standing keys: they hold a token bound to a human-approved grant, with the human's identity carried in the token's actor chain, revocable by killing the grant.

I'll state assumptions, size the problem, then walk the architecture, the token and authorization decisions in depth, the migration and failure stories, and what I'd punt on.

Assumptions

~2M human users across 200K orgs (most orgs are small; the top 1% have 1,000+ members and nested teams). Maybe 5M platform-managed resources (services, datastores, cron jobs, environments) organized as org → project → environment → resource.
50K req/s peak on the platform API. Auth-plane traffic (logins, token exchanges, SCIM syncs) is a small fraction of that — I'll estimate ~2K/s peak for token exchange if tokens live 10 minutes and there are on the order of a million active credential holders.
Permission changes are rare relative to checks: thousands of writes per second worst case (a big SCIM sync), tens per second typically. Checks outnumber writes by 4–5 orders of magnitude. This asymmetry drives the whole authorization design: optimize reads, tolerate slower writes.
Customer workloads run on infrastructure we control (we're the platform), so we can inject identity into them rather than asking customers to manage secrets.
Two API regions initially (us-east, eu-west), more later. Tokens issued in one region must verify in all.
"Well under 10 ms" for authz means p99, not p50. I'll budget p50 < 1 ms, p99 < 5 ms.

Requirements, prioritized

Authn + authz on every one of 50K req/s without becoming the platform's latency floor or its single point of failure. This is the constraint that shapes everything.
Short-lived, scoped credentials for all three principal types, with a no-breakage migration path off long-lived API keys.
Fine-grained, delegatable authorization — org/team/project/resource hierarchy, enterprise role mapping.
Enterprise identity: OIDC and SAML SSO, SCIM provisioning, role mapping.
Complete audit: every issuance, every check outcome, queryable.
Agent credentials scoped to exactly the human approval, with the human visible in the audit trail.

Explicitly out of scope in 60 minutes: end-user auth for customers' own apps (we're doing platform IAM, not an auth product), secrets management beyond credentials, and network-layer policy (mTLS between customer services — I'll gesture at it under workload identity).

Scale math that matters

50K req/s of authn+authz. If verification required a network call to a central service, that service would need to absorb 50K req/s at sub-millisecond latency with perfect availability. That's the wrong shape. So: verify locally, check authz against something regional and cached.
Audit volume: 50K decisions/s × ~500 bytes ≈ 25 MB/s, ~2 TB/day raw. Trivial for Kafka, meaningful for storage — so hot store for 90 days, object storage (Parquet on S3) beyond that.
Relationship data: 2M users × ~10 memberships + 5M resources × a few tuples each ≈ low hundreds of millions of tuples. Tens of GB. Fits comfortably in one Postgres-backed SpiceDB cluster per region; sharding is a someday problem.
Token exchange at ~2K/s is an ordinary stateless service. The signing key is the only hot secret.

Architecture

flowchart LR
    subgraph Clients
        H[Human: dashboard / CLI]
        W[Customer workload]
        A[AI agent / MCP client]
    end

    subgraph Edge["API Gateway (per region)"]
        V[Token verifier<br/>cached JWKS + revocation bloom filter]
        AZ[Authz client<br/>decision cache]
    end

    subgraph AuthPlane["Auth control plane"]
        IDP[Identity service<br/>users, orgs, sessions<br/>OIDC/SAML federation, SCIM]
        STS[Token service STS<br/>exchange: session / API key /<br/>workload identity / agent grant → JWT]
        GR[Grant service<br/>agent approvals, scopes]
    end

    SDB[(SpiceDB<br/>relationship store<br/>regional replicas)]
    PG[(Postgres<br/>principals, keys, grants)]
    K[[Kafka audit log]]
    WH[(ClickHouse 90d hot +<br/>S3/Parquet archive)]

    H -->|session cookie| STS
    W -->|platform-injected identity| STS
    A -->|grant-bound refresh token| STS
    H & W & A -->|short-lived JWT| V
    V --> AZ
    AZ -->|check + cache| SDB
    STS --> PG
    IDP --> PG
    GR --> PG
    IDP -->|writes relationships| SDB
    STS & AZ & IDP --> K
    K --> WH

The load-bearing separation: the data plane (verify token, check permission) runs at the edge against local/regional state and must survive control-plane outages. The control plane (login, token exchange, SCIM, grant approval) is centralized-ish and can afford to be briefly down — a 10-minute token keeps working while the STS restarts.

Principals and enterprise identity

One principals table for all three kinds — humans, service accounts (workloads), agents — because everything downstream (tokens, authz tuples, audit) wants a single principal ID format. prn_h_..., prn_w_..., prn_a_... prefixes so a human reading a log knows what they're looking at.

SSO. We are an OIDC relying party and a SAML service provider; per-org IdP connections (Okta, Entra, Google) live in config. Rather than hand-implementing SAML — which is where signature-wrapping vulnerabilities go to breed — I'd run a federation broker (Dex, or Auth0/WorkOS if we're willing to buy): it normalizes SAML and OIDC into one OIDC flow our identity service consumes. I rejected building SAML in-house; it's undifferentiated and dangerous. Enforcement details that enterprises will actually test in the sales cycle: domain-capture (anyone with @bigco.com must go through BigCo's IdP), session lifetime caps per org, and re-auth for sensitive operations.

SCIM. A standard /scim/v2 endpoint per org connection. SCIM group → platform team mapping is configured by the org admin ("IdP group eng-payments → team payments with role developer"). SCIM writes go to Postgres first, then produce relationship tuples in SpiceDB. Deprovisioning is the case that matters: a SCIM deactivate must revoke the user's sessions and grants immediately, not at next token expiry — this feeds the revocation channel described below.

Credentials: the token decision

Opaque tokens or JWTs? Walk through it, because the intuitive answer is opaque.

Opaque tokens (random strings, validated by DB/cache lookup) have the property security people love: revocation is instant and the token carries nothing. But every verification is a network call. At 50K req/s that lookup service becomes a Tier-0 dependency with a sub-millisecond SLO, and in multi-region either every region calls home (latency, availability coupling) or you replicate the token store everywhere (now revocation isn't instant anyway — you've rebuilt the JWT propagation problem with more moving parts).

Stateless signed tokens (JWTs) verify locally in microseconds — an ES256 signature check is pure CPU. The classic objection is revocation: a stolen JWT is valid until expiry. The answer is to make expiry short enough that the revocation window is tolerable, then close the remaining gap with a small denylist.

Decision: JWTs, ES256, 10-minute TTL, ~3 KB max, plus a pushed revocation set.

Claims: sub (principal), org, sid (session/grant ID — the revocation handle), scp (scope narrowing, see agents), act (actor chain for on-behalf-of), iat/exp, iss (issuing region), jti.
Deliberately not in the token: permissions. Fat tokens that embed roles or resource lists go stale, bloat past header limits for a user with 500 projects, and turn every permission change into a token-revocation event. The token answers "who is this?"; SpiceDB answers "can they do X?". Keeping those separate is the design's spine.
Revocation: revoking a session, key, or grant writes a tombstone on its sid and publishes it on a stream the gateways subscribe to; each gateway holds the live tombstone set (only tombstones younger than max token TTL — so it stays small, thousands of entries) in memory. Propagation target: under 5 seconds globally. Belt and suspenders: even if the push fails, the token dies within 10 minutes.
Keys: per-region signing keypairs, all public keys in a merged JWKS document that every gateway caches (and persists to disk, so a cold start during a JWKS outage still verifies). Rotate signing keys weekly by overlap: publish the new public key, wait a TTL, start signing with it. Any region verifies any region's tokens with zero cross-region calls — this is the whole multi-region verification story, and it's boring by design.

Dashboard browser sessions stay opaque (httpOnly cookie, server-side session row) — browsers are where instant logout matters most and where the session store lookup is affordable. The dashboard's backend exchanges the session for a JWT when it calls the platform API. CLI login uses the OAuth device flow and holds a refresh token, exchanging as needed.

The STS: one exchange endpoint, four credential types

POST /v1/token is the narrow waist. Everything long-lived is only good for calling it:

You present	You are	You get
Session cookie	Human via dashboard	10-min JWT, full user identity
Refresh token (device flow)	Human via CLI	10-min JWT
Legacy/scoped API key	Human or automation	10-min JWT, scoped to the key's scope
Workload identity assertion	Customer workload	10-min JWT as the service's principal
Agent grant refresh token	Agent	10-min JWT with `act` chain and grant scopes

This buys three things at once. Revocation has one shape (kill the long-lived thing; outstanding JWTs age out in ≤10 min). Audit has one shape (every issuance is one event type). And migration off legacy keys becomes an implementation detail of the exchange, not a customer-visible flag day.

Workload identity. Customer services shouldn't hold secrets at all. We run their workloads, so we can do what cloud providers do: each container gets a link-local metadata endpoint (the AWS IMDS pattern) serving a workload identity document — a signed assertion of "this is service svc_abc in env production," in the shape of a SPIFFE-style workload identity (SPIRE is the reference implementation; I'd start with a simpler homegrown attestor that signs assertions based on which container is asking, and adopt SPIRE when we need cross-platform federation). The SDK exchanges it at the STS automatically. No env-var secrets, rotation is invisible, and a leaked workload token is worth 10 minutes. Same identity doc later bootstraps mTLS between customer services if we build that.

Agents: grants, not keys

The failure mode to design against is the obvious one: a developer pastes their personal API key into an MCP config, and now a coding agent has standing superuser rights and the audit log says the human did everything.

Instead, an agent connects via the OAuth authorization-code flow (or device flow for headless), and the human sees a consent screen: "Claude Code wants: read services in project checkout, view logs, trigger deploys to staging. For 8 hours." Approval creates a grant — a first-class record: {grant_id, human principal, agent client, scope set, resource pattern, expiry, status}. The agent receives a refresh token bound to that grant and exchanges it for 10-minute JWTs.

Three properties make this work:

The token carries the chain. Following the OAuth token-exchange actor pattern (RFC 8693's act claim): sub is the agent principal, act.sub is the approving human, chains nest if an agent spawns a sub-agent. Every audit event therefore answers "which agent, on whose behalf."
Authorization is the intersection, not the union: the agent may do X only if the grant's scopes allow X and the human currently may do X. Enforced at check time — the gateway asks SpiceDB about the human (via a caveat on the grant relationship, below), so if the human loses access or leaves the org, the agent's access dies with them, mid-grant. A grant is never an escalation channel.
Revocation is one click. Kill the grant → refresh token dead, sid tombstone pushed → agent locked out inside seconds. Grants max out at hours-to-days, never "until revoked."

Scope escalation is a fresh consent screen ("the agent requests: delete datastores"), never a silent widening. That friction is the feature.

Authorization: why relationship-based, and how it makes 10 ms

Start with the intuitive answer: RBAC. Roles per org (admin, developer, viewer), role checks in the API handlers. Every platform starts here and it maps cleanly to what enterprises say they want ("map IdP group to role"). It breaks on this problem in two places. First, the resource hierarchy: "developer on team payments, which owns project checkout, which contains service api-server" means a permission check must walk a graph — org-level roles are too coarse (thousands-member orgs will not give everyone org-wide deploy), and per-resource ACLs without inheritance turn every project creation into an ACL fan-out. Second, agents need per-grant, per-resource-subtree scoping that no static role enumerates.

Pure policy-as-code engines (OPA) evaluate rules fast but leave "fetch the relationship data" as your problem — and the data is the problem here, not the rule logic.

Decision: a Zanzibar-style relationship-based store (SpiceDB), Postgres-backed, one write master with regional read replicas. The model is tuples like:

org:acme          #member         @user:alice
team:payments     #member         @user:alice
project:checkout  #maintainer_team @team:payments#member
project:checkout  #parent          @org:acme
service:api       #parent          @project:checkout
service:api       #deployer        = maintainer of parent project ∪ admin of org

Roles don't disappear — RBAC becomes a pattern inside the relationship schema (a role is a relation; IdP role mapping writes membership tuples), so enterprises still see roles while the engine answers graph questions. Agent grants also live here as caveated tuples: service:api#deployer @agent:claude[grant:g123] where the caveat checks grant validity and, critically, references the granting human's own permission — that's the intersection rule from the agents section, enforced in one place.

I rejected building a bespoke authz service on Postgres joins: we'd spend a year re-deriving Zanzibar's hard parts (recursive expansion with caching, consistency tokens) and get a worse version.

The latency budget. Target p50 < 1 ms, p99 < 5 ms added:

JWT verify: local CPU, microseconds. Effectively free.
Decision cache at the gateway: an in-process cache keyed on (principal, permission, resource, snapshot) with 30-second TTL. API traffic is heavily repetitive (a CI pipeline hammering one service, a dashboard polling one project), so I'd expect a high hit rate — cache hits cost nothing measurable. This is a target to validate with real traffic, not a measurement I have.
Cache miss: gRPC to the same-region SpiceDB replica, which has its own internal caching of subgraph expansions. Single-digit milliseconds is the realistic budget for an intra-region check; if p99 misses, the levers are more replica cache memory and precomputed denormalized relations for the hottest permission shapes.
What never happens on the request path: a cross-region call, or a call to Postgres.

The consistency trap (the "new enemy" problem). Caches and replicas mean a check can see stale data: revoke Bob's access, and a cached "allow" lets Bob in for another 30 seconds. Zanzibar's answer is consistency tokens (SpiceDB's ZedTokens): a check can demand "at least as fresh as this write." Evaluating everything at full consistency would forfeit the caches, so we tier it. Ordinary API reads accept bounded staleness (≤ 30 s). Sensitive operations — credential issuance, member/role changes, resource deletion, secret reads — check at full consistency and eat the extra milliseconds; they're a sliver of traffic. And explicit revocations don't rely on cache expiry at all: a revocation event fans out on the same push channel as token tombstones and evicts matching decision-cache entries within seconds. Staleness you chose is a tradeoff; staleness during an incident response is a hole, so revocation gets the fast path.

Data model and APIs

Postgres (control plane, source of truth for principals and credentials):

principals(id, kind[human|workload|agent_client], org_id, display, status, created_at)
users(principal_id, email, idp_connection_id, idp_subject, scim_external_id)
orgs(id, name, sso_config, scim_config, session_policy)
api_keys(id, org_id, principal_id, hash, prefix, scopes, resource_pattern,
         expires_at, last_used_at, created_at, revoked_at)   -- hash only, never plaintext
grants(id, human_principal, agent_client, scopes, resource_pattern,
       expires_at, status, approved_at, revoked_at, revoked_by)
sessions(id, principal_id, idp_session_ref, expires_at, revoked_at)
revocations(sid, reason, created_at)          -- feeds the push stream

SpiceDB: the relationship tuples above; schema versioned in the repo like code, migrated like code.

External APIs (the interesting ones):

POST /v1/token                 grant_type ∈ {session, refresh_token, api_key,
                               workload_identity, agent_grant} → {jwt, expires_in}
POST /v1/token/revoke          by sid / key id / grant id
GET  /v1/introspect            for internal services that receive tokens indirectly
POST /v1/keys                  create scoped API key (scopes + resource pattern + expiry required)
GET/DELETE /v1/keys/:id
POST /v1/grants                start agent consent flow;  GET /v1/grants  (list "what did I approve")
DELETE /v1/grants/:id
/scim/v2/*                     per-org SCIM
GET  /v1/audit/events?actor=&resource=&from=&to=      org-scoped audit query

Internal: gateways call SpiceDB's CheckPermission directly (with the decision cache in the gateway's authz client library). Internal services beyond the gateway trust the gateway-forwarded JWT and re-verify it locally — verification is cheap enough that "verify everywhere, trust no hop" costs nothing.

Migrating off long-lived API keys without breaking anyone

The keys exist, they're in CI configs and cron jobs across 200K orgs, and a flag-day breaks customers. The migration is a ratchet — each step shrinks the blast radius and none breaks a working key:

Re-plumb, change nothing visible. Legacy keys stop being verified by API handlers and become STS exchange credentials: gateway sees Authorization: Bearer rnd_live_..., exchanges (and caches the resulting JWT ~5 min) at the STS. Customers notice nothing; we gain one enforcement point, last_used_at telemetry, and instant revocation for every legacy key.
New keys are born scoped. Key creation now requires scopes, a resource pattern, and an expiry (default 90 days, max 1 year). The word "admin key" leaves the UI.
Shrink the legacy pool with data, not deadlines. last_used_at tells us which keys are dead — notify, then disable after 90 days idle (revival is one support click, so the failure mode is an email, not an outage). For live keys, dashboards nag with the exact replacement: "this key only ever calls GET /services on project checkout from GitHub Actions — here's a scoped key / here's OIDC federation for your CI."
CI federation kills the biggest key population. GitHub Actions and GitLab already mint OIDC ID tokens per job; we accept those at the STS (grant_type=workload_identity with an external issuer allowlisted per org) — CI needs no stored secret at all. This is the single highest-leverage move, because CI configs are where keys leak from.
Expiry becomes universal for new and rotated keys. Old unscoped keys are grandfathered until orgs opt in — enterprises get an org policy toggle ("no unscoped keys, max TTL 90d"), and their security teams will turn it on for us.

Never breaking a working credential is the constraint; everything above respects it.

Leak detection and response

Detectable format: keys look like rnd_live_<base62> with an embedded checksum. This is what makes scanning work at all — GitHub's secret-scanning partner program will send us candidate matches from public commits; we verify the checksum, auto-revoke (per org policy: revoke vs. alert-only), and notify with the commit URL. Also run our own scanner over customers' connected repos and build logs.
Honeytokens: offer decoy keys that alert on first use. Cheap, and enterprises love them.
Anomaly detection on the audit stream: a consumer flags a key suddenly used from a new ASN/country, a token calling endpoints outside its historical set, impossible-travel between uses. Start with these coarse rules; a leaked CI key used from a residential IP is not subtle.
Response is the same one-click path every time: revoke the sid/key/grant → tombstone pushed → dead everywhere in seconds, plus outstanding JWTs ≤ 10 min. The audit store answers the incident question — "everything this credential did, ever" — with one indexed query on credential ID. That query is the reason the audit pipeline logs allows, not just denials.
The structural fix outranks all detection: after the migration above, most leaked things are 10-minute tokens or scoped 90-day keys, and the leak story changes from "rotate everything, assume org compromise" to "narrow blast radius, exact activity list."

Audit pipeline

Two event families, one pipeline: issuance events (every login, exchange, key creation, grant approval/revocation — low volume, high value) and decision events (every authz check: principal, actor chain, permission, resource, decision, consistency level, gateway, latency — 50K/s at peak).

Gateways emit decisions to a local buffering agent that batches into Kafka (audit must not add request latency, so it's async; the buffer survives short broker outages, and gateway crash loss of a few in-flight seconds is an accepted, stated risk — issuance events, the high-value family, are written through the control plane and don't share it). Consumers: a columnar analytics store (ClickHouse) for the 90-day hot window powering the customer-facing audit API and anomaly detection; Parquet on S3 with 7-year retention for compliance; per-day hash chaining over the archive so tampering is provable, which auditors ask about. Issuance events additionally land in Postgres because they're part of credential state, not just history.

One honest wrinkle: decision events at the gateway know the token, not the story. The act chain in the token is what keeps "agent did X for Alice" reconstructible from a single event without joins — that's why the chain lives in the token rather than only in the grants table.

When the auth system itself is down

Fail-open vs. fail-closed isn't one decision; it's a matrix, and the biggest wins come from making the question not arise.

Token verification never goes down in any meaningful sense: it's local CPU against a JWKS cached in memory and on disk. An expired-but-unverifiable token is impossible by construction. This is most of why stateless tokens won the earlier decision.

STS down: existing tokens work for up to 10 minutes; new exchanges fail. Mitigations: the STS is stateless behind its Postgres (replicated, regional failover), deployed per region; gateways serve slightly-early refreshes (SDKs refresh at 80% of TTL) so a brief blip is invisible. A >10-minute regional STS outage degrades the platform's control plane — but here's the part that matters for a platform like this: customer workloads keep serving their own traffic, because their runtime doesn't depend on our IAM; only calls to our platform API degrade.

SpiceDB replica down / slow — the per-endpoint-class policy:

Endpoint class	On authz unavailability	Why
Reads of non-secret resources (list services, view metrics/logs)	Fail-static: serve from decision cache, honoring entries up to 5 min stale	Availability worth a bounded staleness window; revocation push still evicts
Mutations (deploy, scale, edit config)	Fail-closed, fast 503 + retry guidance	Wrong writes are worse than delayed writes
Credential issuance, IAM changes, secret reads, deletes	Hard fail-closed, no cache fallback ever	The attack an outage would enable is exactly here

"Fail-static" — keep honoring what you last knew, never guess about what you don't — is the principle; it's how the gateway degrades, and it's why the decision cache exists as a resilience layer and not just a latency layer. Blanket fail-open is indefensible for an IAM system (an outage becomes an authorization bypass, and outages are attacker-observable). Blanket fail-closed turns every authz blip into a full platform outage. The matrix is the defensible middle, and it's written down per route, not decided during an incident.

Kafka down: buffers absorb minutes; beyond that, decision logging degrades to sampled-plus-all-denials before we'd ever block requests on audit. Issuance events are never dropped — they're on the control-plane write path.

Multi-region

Verification: solved by construction — merged JWKS everywhere, any region verifies any token. Nothing to do at request time.
Authorization reads: regional SpiceDB replicas; checks never cross regions.
Writes: single-master relationship writes (login region or home region), replicated out. Cross-region staleness falls under the same bounded-staleness + revocation-push regime as caching — one mechanism, not two. If write latency from far regions ever matters, SpiceDB's CockroachDB-backed multi-region deployment is the known upgrade; I wouldn't start there, because one more distributed database on day one is cost without a proven need.
Region isolation: a region runs its own gateways, SpiceDB replica, STS, and Kafka. A full region loss means clients re-exchange in another region and lose nothing but in-flight sessions. EU data residency (audit events, user records pinned to EU) is a real enterprise requirement I'd scope as a fast-follow with the archive partitioned per region.

Tradeoffs I'm accepting, and evolution

A revocation window exists: ≤ 10 min on token expiry, ~seconds via push. I'm trading a hard guarantee for removing a 50K req/s Tier-0 lookup service. For the operations where the window is unacceptable, full-consistency checks close it. If a regulator ever demands zero-window revocation platform-wide, TTLs drop to 1–2 minutes and exchange QPS rises 5–10× — the architecture holds, the STS fleet grows.
SpiceDB is a serious operational dependency and Zanzibar-style modeling is a skill the team must build; schema mistakes are subtle (a mis-defined arrow can over-grant quietly). Mitigations: schema changes reviewed like security-sensitive code, with an assertion suite (SpiceDB supports declared expected-allow/deny test cases) in CI. I still take this over growing a bespoke engine, which is the same risk plus we wrote the bugs ourselves.
The decision cache trades freshness for latency, patched by revocation push. Honest residual: a narrowing change that isn't an explicit revocation (e.g., a team membership edit) can take up to the staleness bound to apply on cached read paths. Stated, bounded, and monitored beats implicit.
Deliberately deferred: per-request policy conditions beyond caveats (time-of-day, IP allowlists — SpiceDB caveats cover the near-term asks), customer-authored authorization policies, and cross-org resource sharing (modelable as tuples later; consciously excluded from v1 schema).

Evolution order once v1 holds: CI OIDC federation (biggest leak-surface kill for the least work), then org-level credential policies for enterprises, then SPIRE-based workload identity with cross-platform federation, then mTLS between customer services riding the same identity documents. The end state the whole design bends toward: the only long-lived secret in the system is the STS signing key — everything else is minutes old, scoped, attributed, and revocable in one click.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): SpiceDB caveats cannot invoke another CheckPermission; what concrete schema and check sequence computes agent grant scope ∩ resource pattern ∩ human's current permission, including nested agents, without multiple inconsistent reads or blowing the 5 ms p99 budget?

Conceded: the design as written overloads caveats. A SpiceDB caveat is a CEL expression over static context — it can check grant expiry, scope strings, and resource patterns, but it cannot recursively ask "does Alice currently have deploy on this service?" That claim was wrong as a mechanism.

The revised mechanism moves the intersection into the gateway's authz client, pinned to one snapshot. The token already carries every principal involved: sub (agent) and the full act chain (approving human, and any intermediate agents). On a request, the authz client issues parallel CheckPermission calls — one per principal in the chain — against the same-region replica, all pinned to the same ZedToken, and allows only if every check allows. The agent's own check goes through the grant tuple (service:api#deployer_grant @agent:claude[grant:g123]), whose caveat validates only the static parts: grant not expired, scope covers this permission, resource matches the pattern. The human's check is an ordinary service:api#deploy @user:alice evaluation. Same snapshot means no inconsistent reads between the legs; parallel means latency is the max, not the sum, of two independently cacheable checks. Nested chains add one parallel check per hop; I'd cap chain depth at 3 in policy (the consent UX collapses beyond that anyway).

Design change: intersection enforcement moves from "caveat references the human's permission" (not expressible) to the gateway authz client fanning out snapshot-pinned parallel checks over the token's full actor chain; caveats validate only static grant facts (expiry, scopes, resource pattern).

Q2 (Codex): During a SCIM deactivation or team-membership removal, Postgres, SpiceDB replicas, gateway caches, and revocation streams can disagree; what exact write ordering, consistency token, and recovery protocol guarantees access cannot be re-authorized after revocation amid partial failures?

The correctness anchor is that revocation never depends on the relationship plane. Ordering: (1) one Postgres transaction marks the user inactive, marks all their sessions/refresh tokens/grants revoked, inserts revocation rows, and writes outbox events — atomically. From this instant the STS will not issue anything new for this principal: issuance always reads credential status from Postgres, the system of record, fail-closed. (2) An outbox relay (at-least-once, idempotent by revocation id) publishes sid tombstones to the push stream and applies tuple deletions to SpiceDB, capturing the ZedToken of the delete. (3) Gateways receiving a tombstone add the sids to the denylist, evict decision-cache entries for that principal, and set a per-principal consistency floor: until the local replica reports at least the captured ZedToken, checks for that principal run at-least-as-fresh-as that token (or go to the primary).

Partial-failure analysis: if the outbox relay dies, revocation is delayed, never lost — retried until acked, and the hard backstop is token expiry: outstanding JWTs die within 10 minutes regardless, and no new ones can be minted because step (1) already committed. If SpiceDB tuple deletion lags the tombstone, the tombstone alone kills every outstanding token for that principal, so stale tuples are unreachable — a check only ever runs behind a valid token. Re-authorization after revocation would require the STS to issue against a revoked Postgres row, which is the one read that is always strongly consistent and fail-closed.

Design change: SCIM and revocation writes go through a transactional outbox; tombstone events carry the ZedToken of the corresponding tuple deletion, and gateways enforce a per-principal minimum-freshness floor until replicas catch up.

Q3 (Codex): The requirement says every permission outcome must be auditable, yet the design accepts crash loss and switches to sampling when Kafka is unavailable; how would you provide a complete, tamper-evident record without making audit availability part of the synchronous request path?

Partly conceded. Issuance events were already synchronous on the control-plane write path — those are complete, full stop. For decision events I gave away too much with "sampled-plus-all-denials." Revised: (a) the gateway's buffering agent writes decision batches to a local disk WAL (segment files, group-committed every ~50 ms) before acking to the emitter; a process crash loses nothing, a host loss loses one group-commit window — the residual shrinks from "any restart" to "host-level failure." (b) Every gateway stamps decisions with a per-gateway monotonic sequence number; downstream consumers detect gaps, so the record is either complete or provably incomplete over a marked window — which is what an auditor actually needs "auditable" to mean. The per-day hash chain incorporates the sequence numbers, so the archive is tamper-evident and gap-evident. (c) Kafka outage: spill to local disk sized for hours, then ship segments directly to S3 if the outage outlives disk. Sampling is removed from the plan entirely. Requests are never blocked on audit; durability comes from the local WAL, not from synchronizing with Kafka.

Design change: local disk WAL ahead of Kafka, per-gateway sequence numbers with gap detection folded into the hash chain, and the "degrade to sampling" mode is deleted — replaced by disk spill and direct-to-S3 shipping.

Q4 (Codex): What evidence supports the claimed sub-5 ms p99 when cache misses can require recursive graph evaluation over hundreds of millions of tuples, and how many replicas and how much headroom are needed during cold-cache traffic, hot-resource skew, and a large SCIM update?

The design flags this as a target to validate, not a measurement, and I'll keep that honesty — but the shape argument is defensible. Check cost in a Zanzibar-style engine scales with the subproblem count of the evaluated permission, not the total tuple count: hundreds of millions of tuples is a storage number. The subproblem count here is bounded by the hierarchy — org → project → environment → resource is depth 4, team nesting capped in product (I'd enforce ≤8), unions of a few relations per level — so a miss is tens of dispatches, most served by SpiceDB's internal dispatch cache because subgraphs (org membership, project parentage) are shared across millions of checks. Skew helps rather than hurts: a hot resource concentrates cache hits at both layers.

Sizing, stated as assumptions to load-test before cutover, not measurements: a 90% gateway decision-cache hit rate leaves ~5K checks/s per region; a SpiceDB node in this shape handles low thousands of checks/s, so 4–6 replica nodes per region gives 3–4× headroom. Cold cache is a deploy-time event — stagger gateway rollouts so the fleet never starts cold together. A large SCIM sync churns tuples and invalidates dispatch-cache subgraphs; two containments: SCIM-driven writes are rate-limited (they're batch by nature; no one needs a 50K-user sync to land in one second), and bounded-staleness reads mean replicas keep serving the pre-sync snapshot rather than stampeding the primary. If p99 still misses after tuning, the named levers are flattened membership relations for the hottest permission shapes and a longer gateway cache TTL on read-only endpoints. What I won't do is quote a p99 I haven't measured.

Q5 (Codex): What cryptographically binds a workload identity assertion to the intended container and prevents a compromised neighboring workload, SSRF vulnerability, replayed assertion, or malicious tenant process from querying the metadata endpoint and impersonating another service?

The binding is kernel-derived caller identity, not caller-supplied claims. The metadata endpoint is served by a per-node agent; when a container connects, the agent identifies it from the socket itself — SO_PEERCRED/cgroup on a unix socket, or, for the link-local IP flavor, a CNI configuration where the source IP maps one-to-one to a pod and the vswitch drops spoofed source addresses. The caller never names itself; the agent answers "who is asking" from the kernel's answer, so a neighboring container asking for someone else's identity is not a request the API can express. The node agent signs assertions with a per-node key enrolled at node bootstrap (TPM-backed where hardware allows, provisioning-credential otherwise), and the STS cross-checks "this assertion for svc_abc came from a node that actually schedules svc_abc" against the scheduler's placement record — so a compromised node can only mint identities for workloads placed on it, which is the blast radius a compromised node already has.

SSRF: adopt the IMDSv2 lesson — session-token handshake (PUT then use), required custom header, TTL-1 on the link-local hop — which kills the "GET-only proxy reaches the metadata service" class. Honest residual: an attacker with full request-forging ability inside your own service gets your own service's identity. That's the floor every cloud provider lands on, and it's the right floor — that identity is scoped to what the one service is allowed to do. Replay: assertions are audience-bound to our STS, 60-second validity, nonce'd, single-use (the STS tracks assertion jti within the validity window). A stolen assertion is worth one exchange at the one endpoint that logs every issuance.

Design change: the assertion spec is tightened to kernel-attested caller identity, per-node signing keys cross-checked against scheduler placement, an IMDSv2-style handshake, and single-use 60-second assertions.

Q6 (Codex): For unchanged clients that send a legacy API key on every request, does the gateway synchronously exchange it or cache a derived JWT; in either case, how do scope narrowing, revocation, STS outages, and replay behave without either adding a new per-request dependency or extending a leaked key's validity?

Cached, with a synchronous exchange only on cache miss — this was in the design (migration step 1: "exchanges and caches the resulting JWT ~5 min"), but the four behaviors deserve the precision. Per-request cost: a local lookup keyed by key hash. Cache miss (first sight of a key, or TTL expiry): one intra-region STS call, singleflighted per key so a hot CI key doesn't stampede — roughly once per 5 minutes per key per gateway. Scope narrowing: editing a key's scopes revokes its current sid — the tombstone both kills outstanding JWTs and evicts the key-hash cache entry, so narrowing bites in seconds, not at cache expiry. Revocation: same path, dead in seconds — strictly better than the status quo, where a legacy key was valid until a handler rechecked a DB row, which was never. STS outage: cached entries keep serving (fail-static, bounded by the JWT's remaining TTL ≤10 min); uncached keys fail — and legacy-key traffic is dominated by CI/cron, the most retry-tolerant clients on the platform.

Replay and leak surface: the key transits TLS exactly as today, so no new exposure; the derived JWT never leaves the gateway (attached internally, stripped from anything client-visible), so there is no new replayable artifact. The cache cannot extend a leaked key's life: the key was standing-valid before this change; the cache only bounds how long revocation takes to bite (seconds via tombstone, ≤5 min worst case via TTL). The one real cost is the STS becoming load-bearing for cold caches — which is why it's per-region, stateless, and sized against exchange spikes with the gateway singleflight in front.

Round 2 — pushback

Q7 (Codex): Your audit path still admits losing up to 50 ms of decisions on host failure, and sequence numbers merely prove records are missing — so what exact durable acknowledgment must occur before an API response is returned, and what happens when that durability cannot be achieved?

For the bulk of traffic: no durable acknowledgment before the response, deliberately, and I'll defend that rather than pretend it away. Making a synchronous replicated audit write a precondition for every one of 50K req/s re-creates exactly the Tier-0 coupling the whole design exists to avoid — audit infrastructure would become a platform-wide availability dependency, and the failure mode of "audit cluster slow" becomes "platform down." A ~50 ms gap-evident window on host failure for read-path decision events is a residual I'll state in the compliance story, not engineer away at that price.

But the question exposes a tier I was treating uniformly and shouldn't: the sensitive classes — mutations, credential issuance, IAM changes, secret reads, deletes — are precisely the events an incident responder cannot tolerate missing, and they already pay for full-consistency authz checks because they're a sliver of traffic. So split the guarantee along the existing endpoint-class matrix: for those classes, the decision event gets a durable acknowledgment before the response — fsync to the local WAL plus a synchronous replicate to a second host (or, for issuance and IAM changes, the existing control-plane Postgres write they already do). If that durability cannot be achieved, the request fails — fail-closed already governs these routes for authz, and audit joins the same contract: no un-audited mutation. Reads never block on audit; mutations never complete without it.

Design change: the fail-static/fail-closed endpoint matrix gains an audit column — sensitive-class requests require durable (fsync + second-host) decision-event acknowledgment before responding and fail closed without it; read-path events keep the async WAL with the gap-evident residual stated.

Q8 (Codex): Your latency claim rests on an assumed 90% cache-hit rate and unmeasured "low thousands" of checks per node; during a cold-cache regional failover, agent requests can multiply 50K req/s into several SpiceDB checks each — what concrete capacity model keeps authz under 10 ms p99 in that scenario?

Model the worst instant honestly: one region absorbs all 50K req/s with cold gateway caches, and actor-chain fan-out multiplies checks — agents are a minority of traffic, so average chain length is maybe 1.3, but take 2× as the planning number: 100K raw check demands/s at hit-rate zero. No replica fleet I'd provision serves that naively, and autoscaling can't act in seconds. The load-bearing mechanism is that raw demand is not unique demand: gateways singleflight identical (principal, permission, resource, snapshot) checks, and API traffic is dominated by repetitive pollers — CI loops, dashboards, SDK retries — so the coalesced unique-check rate in any 30-second window is bounded by distinct (principal, resource) pairs actually active, orders of magnitude below 100K/s. Chain checks coalesce too: a busy human's own check is shared across all their agents' requests. And the cold period is self-limiting — with a 30-second TTL and repetitive traffic, the gateway cache is near steady-state within one TTL.

The capacity model, stated as sizing assumptions to be validated, not measurements: provision each region's replica fleet for the coalesced failover estimate with 2–3× headroom — my planning number is ~10 replica nodes per region against a coalesced unique-check rate in the low tens of thousands per second for the first TTL window — and bound the transient with admission control: each gateway caps concurrent outstanding SpiceDB misses; beyond the cap, requests fall into the existing endpoint-class matrix (reads queue briefly or fail-static, mutations get fast 503s) instead of stampeding the replicas into collapse. Two things make this a model rather than a hope: the unique-check rate and per-node check capacity are measurable today from production traffic traces before any cutover, and regional failover with cold caches becomes a scheduled game-day drill with a pass/fail p99 gate — if the drill fails, the levers are pre-warming the standby region's gateway caches from the live region's decision stream and more replica memory. I'll commit to the mechanism and the drill; I won't commit to a p99 number I haven't measured under real traffic shape.

Design change: per-gateway admission control caps concurrent SpiceDB misses with overflow routed through the endpoint-class degradation matrix; replica fleets are sized to the coalesced (singleflighted) unique-check rate for full-platform failover, verified by scheduled cold-cache failover drills; cross-region decision-stream cache pre-warming is the named lever if drills miss the p99 gate.

What changed, summarized

Agent/human intersection is no longer claimed via SpiceDB caveats (not expressible); the gateway authz client fans out parallel, snapshot-pinned (same ZedToken) checks for every principal in the token's act chain, with chain depth capped at 3. Caveats validate only static grant facts.
SCIM/revocation writes gain a transactional outbox; tombstones carry the ZedToken of the matching tuple deletion, and gateways hold a per-principal freshness floor until their replica catches up.
Audit: local disk WAL ahead of Kafka, per-gateway sequence numbers folded into the hash chain (gap-evident), "degrade to sampling" deleted in favor of disk spill and direct-to-S3 shipping.
Sensitive-class requests (mutations, issuance, IAM changes, secret reads, deletes) now require a durable decision-event acknowledgment before responding and fail closed without it; read paths stay async with a stated, bounded residual.
Workload identity tightened: kernel-attested caller identity (SO_PEERCRED/CNI source binding), per-node signing keys cross-checked against scheduler placement, IMDSv2-style handshake, single-use 60-second audience-bound assertions.
Capacity/failover: per-gateway admission control on SpiceDB misses, replica sizing against the coalesced unique-check rate for full-platform cold-cache failover, and scheduled failover drills with a p99 gate; cache pre-warming from the peer region's decision stream is the named fallback lever.

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The authorization layer follows published practice closely. The Zanzibar paper (USENIX ATC '19, annotated here by AuthZed) is the source for almost everything the design claims about relationship-based authz: relation tuples, zookies as consistency tokens, and the "new enemy" problem the design tiers its consistency around. Google's production numbers put the latency budget in context — p95 under 10 ms across trillions of tuples and millions of checks/s — so the design's 5 ms p99 target is aggressive but the same order as the reference system. One thing I claimed as a "lever if p99 misses" turns out to be load-bearing at Google: flattened membership for hot shapes is Zanzibar's Leopard index, a whole secondary system that denormalizes group membership because recursive expansion alone didn't hold up. AuthZed's own ZedTokens post documents the per-request consistency dial (bounded staleness vs. at-least-as-fresh) that the interview's Q1/Q2 answers lean on — the snapshot-pinned parallel checks are the intended use of that API, not an improvisation.

On token format, the design picked the contested side of a real debate. Thomas Ptacek's API Tokens: A Tedious Survey on the Fly.io blog lays out the case against JWT — algorithm-confusion attacks, alg=none, "design-by-committee cryptographic kitchen sink" — and Fly.io went the other way, to macaroons, largely for offline attenuation: a holder can narrow a macaroon without calling the issuer. This design gets attenuation server-side instead (scoped keys, grants at the STS), which costs a round trip but keeps one revocation shape; the JWT risk it accepts is contained the standard way, by pinning a single algorithm (ES256) at the verifier. Both are defensible; the survey is the honest map of what's being traded. The actor chain in the agent tokens is not improvised either — it's RFC 8693's nested act claim verbatim, including the "outermost act is the current actor, nested acts are prior actors" semantics the Q1 fan-out iterates over.

Workload identity is the part where the design deliberately re-derives an existing standard. SPIFFE already specifies the whole shape: node plus workload attestation, short-lived SVIDs (X.509 or JWT) rotated automatically, and the observation that short lifetimes make CRLs mostly unnecessary — which is this design's revocation story too. The "homegrown attestor first, SPIRE later" call is a sequencing bet, not a disagreement. The metadata-endpoint hardening in Q5 is taken from AWS's own IMDSv2 writeup: the PUT-then-token session handshake and the TTL-1 hop limit are their published answers to SSRF and misconfigured proxies, and the design copies them on purpose. The migration plan's highest-leverage step — CI federation — is exactly what GitHub's OIDC docs describe from the issuer side: per-job ID tokens with claim-based trust conditions so CI stores no long-lived secret at all. And the leak-detection section's dependency is real and documented: the secret scanning partner program sends matches from public repos to a provider-run verify endpoint, which is precisely why the design gives keys a detectable prefix and checksum.

The agent story now has a spec to compare against. The MCP authorization spec (2025-11-25) makes the MCP server an OAuth 2.1 resource server, requires PKCE and RFC 8707 resource indicators (audience-binding tokens to one server), and explicitly forbids token passthrough — the same "no standing key pasted into an MCP config" failure mode this design's grants exist to kill. Aaron Parecki's walkthrough of that revision covers the two additions that matter for a platform like this: Client ID Metadata Documents replacing dynamic client registration, and enterprise "cross app access" that routes agent authorization through the customer's IdP. That second one is a real divergence to watch — this design keeps grants platform-local with the org admin as the policy surface, while the spec's enterprise extension pulls the IdP into the loop. For enterprise customers who already run everything through Okta, the IdP-in-the-loop model is likely where the sales pressure lands.

Updates from post-training information

Uber published Solving the Identity Crisis for AI Agents in May 2026, after my training data ends, and it's the closest production validation of this design's agent plane I've seen: an internal STS minting short-lived JWTs for agents, the full human-through-agents actor chain embedded in the token for audit, SPIRE attesting the agent's compute before any token is issued, and an MCP gateway as the enforcement point. Two differences worth stealing from. First, their tokens are single-hop and audience-bound — each hop re-exchanges at the STS for a token scoped to exactly the next destination, instead of one token carrying a deep chain that the gateway fans out over (my Q1 answer). That moves work from check time to issuance time and makes a stolen token worth even less; the cost is STS traffic per hop, which is why their published p99 for token exchange is under 40 ms — a useful real-world calibration point against my ~2K/s, "ordinary stateless service" sizing. Second, they treat agent identity as attested workload identity plus delegation, not a new principal type with its own credential — consistent with this design's single principals table, and evidence that the "grants, not keys" direction is where production systems actually landed.

The MCP authorization spec also moved on 2025-11-25 (weeks before my cutoff, so partially known to me): if this design were built today, the grant consent flow should implement Client ID Metadata Documents and OpenID Connect discovery per that revision, not the 2025-06-18 dynamic-registration flow I had in mind while writing it.