S3-Compatible Object Storage for a Developer Platform

The shape of the answer, up front

Build three things and buy everything else. The three things we build: a stateless S3 API gateway, a metadata layer on top of a strongly consistent, ordered key-value store (FoundationDB), and a deliberately dumb blob layer of append-only extent servers with background erasure coding. Everything else — the event pipeline, key management, the CDN, monitoring — we run off the shelf or consume as an external service.

The reasoning behind that split drives the whole design, so I'll state it now: the metadata layer is where every hard product requirement lives — strong read-after-write, prefix listing, multipart atomicity, metering, delete compliance — and it's the layer we must be able to read and debug. The blob layer, by contrast, can be made almost trivial if we refuse to put any cleverness in it: immutable segments, no in-place updates, no consensus in the data path. A small team can operate a dumb component. What a small team cannot operate is Ceph.

Assumptions I'm adding

Single region at launch, three availability zones. 99.95% availability (~4.4 hours/yr) is achievable multi-AZ; multi-region is an evolution, not a launch requirement.
Average object size ~1 MB (median 200 KB, mean dragged up by big objects), so 10 PB ≈ 10 billion objects. I'll sanity-check metadata sizing against this.
Peak 50k req/s splits ~45k reads / 5k writes. Assuming reads skew small (the median object), aggregate egress at the gateway is in the several-GB/s range before CDN offload; ingest is a similar order. These are planning numbers, not measurements.
No object versioning or object lock at launch (S3-compatible subset, not the full API). Overwrite is last-writer-wins, like S3.
Compliance delete window: 30 days from DELETE to physical irrecoverability. I'll actually do much better than that via crypto-shredding, below.
We run on the platform's own hosts and disks (their interface: attach disks, they fail at some rate; I'll assume 2% annualized disk failure rate for durability math and flag it as an assumption to validate against the fleet's real numbers).

Estimates that shape the architecture

Three numbers matter more than the rest.

10 PB with 11 nines forces erasure coding. At 3× replication that's 30 PB raw; at 8+4 Reed-Solomon it's 15 PB. On ~20 TB disks that's the difference between ~1,500 and ~750 spindles. We own the economics, so EC is not optional — the design question is how to get EC's storage cost without paying its latency and small-object penalty on the write path.

Median 200 KB forces packing. Erasure-coding a 200 KB object directly produces twelve ~17 KB fragments — an IOPS disaster and a metadata explosion. Small objects must be packed into large shared segments, which means deletes can't reclaim space directly; they create garbage inside shared segments, and GC becomes a compaction problem. The compliance-delete requirement collides with this head-on, and resolving that collision (crypto-shredding plus deadline-driven compaction) is one of the load-bearing decisions.

10 billion objects forces a real metadata store. At ~300 bytes of metadata per object plus manifests, that's 3–5 TB of hot, strongly consistent, range-scannable state doing ~45k point reads/s at peak. That's squarely inside what a modest FoundationDB cluster handles, and far inside what Apple and Snowflake run it at. It is not inside what a single Postgres box handles, and it's a bad fit for an eventually consistent store like Cassandra, where "strong read-after-write" and "ordered prefix scans" both become application-level projects.

Architecture

flowchart TB
    subgraph clients [Clients]
        SDK[S3 SDKs / presigned URLs]
        RT[Platform runtime services]
        CDN[CDN edge - external]
    end

    LB[Load balancer]
    subgraph gw [Stateless S3 gateways]
        G1[Gateway: SigV4 auth, routing,\nrate limits, usage events]
    end

    subgraph meta [Metadata layer]
        FDB[(FoundationDB\nordered KV, ACID txns:\nobjects, manifests, buckets,\nuploads, segments, counters,\ndelete queue)]
    end

    subgraph blob [Blob layer]
        OS[Open segments\n3x replicated appends\nacross 3 AZs]
        EC[Sealed segments\n8+4 Reed-Solomon\nfragments across 3 AZs]
    end

    subgraph bg [Background services]
        SEAL[Sealer: replicate to EC]
        REP[Repair: rebuild lost fragments]
        GC[Compactor: reclaim deleted bytes,\nenforce compliance deadline]
        SCRUB[Scrubber: full read + checksum\nevery segment ~2 weeks]
        METER[Metering aggregator]
    end

    KAFKA[Durable partitioned event log\n Kafka]
    BILL[(Billing DB)]
    KMS[Key management service\nenvelope encryption]

    SDK --> LB
    RT --> LB
    CDN -->|origin pulls, signed| LB
    LB --> G1
    G1 <--> FDB
    G1 -->|writes| OS
    G1 -->|reads| OS
    G1 -->|reads| EC
    G1 --> KAFKA
    OS --> SEAL --> EC
    REP --> EC
    GC --> EC
    SCRUB --> EC
    KAFKA --> METER --> BILL
    KAFKA -->|purge events| CDN
    G1 <--> KMS

Four moving parts in the request path (LB, gateway, FoundationDB, blob nodes), and everything else is an offline loop that can be paused without user-visible impact. That's the operational-simplicity budget spent deliberately: background complexity is cheap, request-path complexity is expensive.

Gateway

Stateless HTTP servers implementing the S3 subset: PUT/GET/HEAD/DELETE object, ListObjectsV2, the five multipart calls, and SigV4 verification (which gives us presigned URLs for free — more below). At 50k req/s, a few dozen nodes behind the load balancer, trivially horizontally scaled, deploys are rolling restarts. The gateway also enforces per-tenant rate limits and emits usage events. It holds no durable state; any gateway can serve any request.

I'd write this ourselves rather than adapt MinIO's or Ceph RGW's frontend. The S3 wire protocol subset we need is well-documented and small, and the gateway is where tenancy, metering, and platform integration live — exactly the code we want to own.

Metadata: FoundationDB

Why FDB specifically, versus what I rejected:

CockroachDB / TiDB: strongly consistent and range-partitioned, would work. But we need a key-value store with transactions, not SQL — the relational layer adds latency and operational surface we won't use. FDB's core is smaller and its failure behavior is famously well-tested (deterministic simulation).
Cassandra / Scylla: wrong consistency model. Read-after-write and multi-key transactions (multipart commit) would be rebuilt on top with lightweight transactions, which is where Cassandra deployments go to suffer.
DynamoDB or a cloud database: if the platform runs on a cloud provider this is a legitimate small-team answer, and I'd take it seriously — but a developer platform at this scale is usually on its own metal, and per-request pricing on 45k reads/s of someone else's database is exactly the economics we're told we own.
Sharded Postgres: the team would spend its life on resharding and failover tooling. Rejected on the operational-simplicity constraint, ironically.

FDB gives us an ordered keyspace (prefix listing is a range scan), strict serializable transactions (multipart completion and counter updates are single atomic commits), automatic range splitting under load, and a three-AZ deployment that survives an AZ loss. Its limits — 100 KB values, 10 MB / 5-second transactions — are fine for metadata and we design around them (large manifests split across rows).

Key layout (logical; FDB keys are byte strings, tuples shown for clarity):

("obj",  bucket_id, object_key)          -> {version, size, etag, content_type,
                                             created_at, wrapped_dek,
                                             chunks: [(segment_id, offset, len)] | manifest_ptr}
("man",  bucket_id, object_key, ver, n)  -> manifest page n (for huge objects)
("up",   bucket_id, object_key, upload_id)            -> {initiated_at, owner}
("part", bucket_id, object_key, upload_id, part_no)   -> {etag, size, chunks}
("seg",  segment_id)                     -> {state: open|sealed|ec, placement,
                                             live_bytes, total_bytes, created_at}
("bkt",  bucket_id)                      -> {owner, region, public?, policy, quota}
("key",  access_key_id)                  -> {encrypted_secret, owner, status}
                                            // SigV4 needs the secret for HMAC, so it's
                                            // stored encrypted (KMS-wrapped), not hashed
("cnt",  bucket_id, shard_n)             -> {bytes, object_count}   // sharded counters
("del",  due_ts, deletion_id)            -> {wrapped_dek, chunks}   // delete queue

A 5 TB object at 64 MB chunks is ~~80k manifest entries (~~2–3 MB), split across ~30 manifest rows. The median object's chunk list is one entry, inlined in the object row.

Blob layer: append-only extent servers

Each storage node exposes four verbs: append(segment, data) -> offset, read(segment, offset, len), seal(segment), delete(segment). Segments are ~1 GB, immutable once sealed, checksummed per 64 KB block. No node talks to another node; placement and membership live in the metadata layer. This is the Azure Storage stream layer / Facebook f4 shape, and it's the simplest distributed component I know how to specify — which is the point.

Write path (small and medium objects). The gateway appends the object's bytes to an open segment — three replicas in three different AZs, written in a chain (gateway → replica A → B → C, ack flows back). Only after all three acks does the gateway commit the metadata row. Open segments seal at ~1 GB or after a few minutes; a background sealer then erasure-codes the sealed segment into 12 fragments (8 data + 4 parity, Reed-Solomon), places 4 fragments per AZ, verifies, and releases the three replicas.

Why replicate-then-EC instead of EC-on-write? Latency and IOPS. A 200 KB PUT becomes three sequential appends instead of twelve tiny fragment writes, and the client acks fast. The cost is a transient 3× tail: with ~5k writes/s at ~1 MB average, a few minutes of unsealed data is tens of GB replicated at 3× — noise against 15 PB. Steady state overhead is ~1.5×, which is the economics we wanted.

For large objects (multipart parts, big PUTs), the gateway streams into 64 MB chunks; each chunk goes through the same open-segment path. A 5 TB upload is just 80k chunk appends spread across many segments and nodes — no single node ever holds a whole object.

Read path. Metadata lookup → for each chunk, read from a surviving replica (young data) or from data fragments directly (sealed data; a normal read touches only the fragments covering the requested range, no decode needed unless a fragment is missing). Range GETs map byte ranges to chunk ranges to fragment ranges. Chunks are immutable, so there is no read-side coordination at all.

Why not run Ceph or MinIO for this layer? This is the build-vs-buy question that deserves the most honesty. Ceph RGW would hand us the entire product — S3 API, EC, multipart — on day one. I rejected it for three reasons. First, operations: Ceph's peering, CRUSH rebalancing, and recovery storms are a full-time specialty, and the prompt makes operational simplicity a product constraint; "we run Ceph with a team of five" is a sentence with a body count behind it. Second, RGW's bucket index has known pain at large listing scale, and fixing it means patching Ceph — now we're a Ceph shop. Third, metering, tenancy, compliance deletes, and platform integration would be bolted onto someone else's abstractions instead of designed into ours. MinIO is lighter but couples metadata to its own layout and its per-object EC pays the small-object penalty I called out above. The honest counterargument — "you're building a distributed storage system with a small team, that's the risky part" — is real, and my mitigation is scope: the extent layer as specified has no consensus, no rebalancing protocol, no in-place mutation, and a four-verb API. If we can't build that, we can't operate Ceph either.

API semantics and the hard parts

Strong read-after-write

Mechanics, precisely: a PUT commits the metadata row only after every byte is durable on three replicas in three AZs. A GET starts with a linearizable FDB read of the object row, then reads immutable chunks named by that row. There is no metadata cache in the read path at launch — 45k point reads/s is affordable for FDB, and no cache means no invalidation bug can violate the guarantee. Overwrites commit a new row version pointing at new chunks (old chunks go to the delete queue), so a reader either sees the old complete object or the new complete one, never a mix. Same argument covers read-after-delete: the DELETE transaction removes the row; the next GET's linearizable read returns 404.

If FDB read load ever becomes the constraint, the escape hatch is a cache keyed by (bucket, key) with invalidation through FDB watches — but that's an optimization we earn our way into with measurements, not a launch feature.

Listing with prefix at scale

ListObjectsV2(bucket, prefix, delimiter) is a range scan over ("obj", bucket_id, prefix)…. The subtle part is the delimiter: when the scan hits photos/2024/… and the delimiter is /, we must emit one CommonPrefix and skip the possibly-millions of keys under it. FDB's key selectors make this a skip scan: emit photos/2024/, then seek directly to the successor of photos/2024/\xff. A directory listing over a bucket with a billion objects costs O(entries returned), not O(objects skipped). Pagination tokens are just the last key returned — stateless, resumable, no server-side cursors.

Hot buckets with sequential key writes (timestamped uploads) concentrate load on one FDB range; FDB detects and splits hot ranges automatically. If a single tenant's listing patterns hurt neighbors, the gateway's per-tenant limits treat LIST as a more expensive operation class than GET — listing is the op most worth rate-limiting separately.

Multipart upload state

All multipart state is rows, and completion is a transaction. CreateMultipartUpload inserts an ("up", …) row and returns the upload_id. Each UploadPart streams the part through the normal chunk write path and inserts a ("part", …) row with the part's ETag and chunk list — parts are re-uploadable (last write wins, replaced part's chunks queued for GC). CompleteMultipartUpload is one FDB transaction: read the part rows, validate the client's part list and ETags, assemble the manifest, insert the object row, delete the upload/part rows. Atomic, so the object appears exactly once, fully formed — read-after-write holds for multipart the same as for simple PUT. The ETag is the S3-compatible md5-of-md5s with a -N suffix.

Abandoned uploads: a sweeper deletes uploads older than 7 days (configurable per bucket, matching S3 lifecycle behavior) and reclaims their chunks. Since parts live in ordinary segments, reclamation is the same GC path as everything else.

Presigned URLs

Presigned URLs are SigV4 with the signature in the query string, so we get them for free from correct SigV4 support — no server-side URL state at all. The gateway recomputes the signature from the canonical request using the customer's secret key, checks the expiry embedded in the signed string, and checks the access key's status. Two operational details matter: secret keys are cached in gateway memory with a short TTL (say 60 seconds), so key revocation propagates within a minute — we document that window; and expiry is bounded (7 days max, like S3), so a leaked presigned URL has a bounded blast radius. Presigned PUTs go through the same path with the same quota checks as authenticated PUTs.

Durability: the math and the real threats

Independent-failure math first, with stated assumptions: 2% disk AFR (≈2.3×10⁻⁶ per disk-hour), fragments of one segment on 12 distinct disks, and a repair budget that rebuilds a lost fragment within 6 hours. An 8+4 segment dies only if 5 of its 12 fragments are lost inside one repair window: roughly C(12,5)·(1.4×10⁻⁵)⁵ ≈ 4×10⁻²² per window, ~6×10⁻¹⁹ per segment-year. Across ~20 million sealed segments that's an expected fleet-wide loss rate around 10⁻¹¹ events/year — orders of magnitude inside 11 nines for any single object.

Which tells you the real story: independent failures are not what kills 11 nines. Correlated failures and software bugs are. The mitigations are the design, not an afterthought:

Placement: 4 fragments per AZ means a whole-AZ loss costs 4 of 12 — still readable, still repairable. No two fragments on one host.
End-to-end checksums: the gateway computes a checksum on ingest, it's stored in metadata, verified on every read and every repair. Catches bit rot, torn writes, and bugs in our own EC code.
Scrubbing: every segment fully read and verified on a ~2-week cycle. At 15 PB that's ~12 GB/s of background read — ~12 MB/s per disk across ~1,000 disks, affordable.
Staggered rollouts: storage node software and disk firmware never updated across AZs simultaneously. Most real correlated-loss incidents in the industry are self-inflicted.
Delete quarantine (below) doubles as protection against the scariest bug class: our own code deleting the wrong thing.
Metadata durability: losing metadata is losing data. FDB is triple-replicated across AZs, plus continuous backup shipped off-cluster (to a separate cluster or an external store — not to ourselves).

The repair story concretely. A dead 20 TB disk holds ~20 TB of fragments; RS(8,4) repair reads 8 surviving fragments to rebuild each lost one, so ~160 TB of reads, spread across the whole fleet (every segment's survivors live on different disks). At a 6-hour target that's ~7.5 GB/s aggregate — a few MB/s per disk, throttled below scrub-plus-serving headroom. With ~1,000 disks at 2% AFR we expect a disk failure roughly every 2–3 weeks: routine, automated, paged only if the repair queue ages past threshold. The known cost of plain RS is that 8× read amplification; if repair traffic ever becomes the constraint as we grow, the evolution is local reconstruction codes (as in Azure LRC), which cut repair reads several-fold for a small parity overhead — but LRC at launch is complexity we don't need yet.

Deletes, GC, and the compliance window

Deletes are the place where "small objects packed in shared segments" bites, so this is a two-mechanism design.

Mechanism 1 — crypto-shredding for fast, provable erasure. Every object is encrypted at the gateway with its own data-encryption key (DEK); the DEK is wrapped by a per-bucket key held in a key-management service (Vault or the cloud KMS) and stored in the object's metadata row. DELETE removes the object row transactionally (instant 404, S3 semantics) and moves the wrapped DEK to the delete queue. After a 24-hour quarantine — insurance against a runaway bug or a compromised credential mass-deleting data, since inside the window we can restore rows — the DEK is destroyed. From that moment the ciphertext in the segments is unreadable regardless of when compaction physically reclaims it. The compliance answer is key destruction, which is a small, auditable metadata operation, not a race against a petabyte-scale compactor.

One trap to close: FDB backups also contain wrapped DEKs, so a restore could resurrect "destroyed" keys. Two rules fix it: backups are retained less than the compliance window, and the delete queue (retained as an append-only audit log) is replayed against any restored cluster before it serves traffic.

Mechanism 2 — compaction for space and belt-and-suspenders physical erasure. Each segment row tracks live vs. total bytes (decremented transactionally as chunks are freed). The compactor rewrites a segment's surviving chunks into new segments and destroys the old one when either (a) garbage exceeds ~50% — the space-economics trigger — or (b) the segment contains any deleted chunk older than 25 days — the compliance deadline trigger, comfortably inside the 30-day window. Trigger (b) puts a hard bound on physical residency even for segments that are barely garbage; the worst-case cost is rewriting a nearly-full 1 GB segment to erase one 200 KB object, which is exactly the cost crypto-shredding lets us not care about being slow.

Metering and billing

Three meters, two mechanisms.

Requests and egress: the gateway emits one usage event per request — tenant, bucket, operation class, bytes in/out, whether it arrived via CDN origin pull — into a durable, partitioned event log (Kafka; Redpanda if we want fewer moving parts, same protocol). Events carry a (gateway_id, sequence) idempotency key; an aggregator consumes, deduplicates, and rolls up per customer per hour into the billing database. At-least- once delivery plus dedup means we never double-bill; a crashed gateway can lose at most its unflushed buffer (seconds), which under-bills trivially — the right direction to err. CDN egress to end users is metered from CDN logs (its interface), reconciled against origin-pull events.

Storage: sharded per-bucket counters (("cnt", bucket, shard)) updated in the same FDB transaction that commits or deletes an object — so quota enforcement is exact and transactional, no counter drift under concurrency because each gateway increments a random shard. A weekly reconciliation scan (range scan per bucket, off-peak) corrects any drift from bugs and doubles as an invariant check. Billing samples the counters hourly for GB-hour pricing.

CDN integration and runtime access

Public buckets are served through the platform CDN with the gateway as origin. Origin pulls authenticate with a signed internal header (short-lived token, not a customer credential), so the bucket can stay private to the world while public through the CDN. Object metadata carries Cache-Control; overwrites and deletes of public objects emit purge events through the same Kafka log to the CDN's purge API. We document the edge semantics honestly: read-after-write is a property of the storage API; the CDN edge is eventually consistent within the purge SLA (seconds to minutes), same as S3+CloudFront.

Hosted services get low latency by construction: gateways run in the same region and network fabric as the runtime, so an in-region GET is one FDB point read plus one blob read — single-digit to low-tens of milliseconds for the median object. If runtime read latency needs to drop further later, the clean lever is an immutable-content read cache keyed by (object version/ETag): because chunks and versions never mutate, such a cache needs no invalidation and cannot violate read-after-write — the metadata read still comes from FDB.

Tenant isolation

Logical, not physical — with teeth. AuthZ at the gateway (per-tenant access keys, bucket ownership, bucket policies for public read). Noisy-neighbor control is per-tenant token buckets with separate budgets per operation class (GET cheap, LIST and PUT dearer), plus per-tenant concurrency caps so one customer's 5 TB multipart can't occupy every gateway stream. Storage is shared — segments interleave tenants — and per-object encryption is what makes that safe: no bug that returns the wrong bytes returns plaintext of the wrong tenant, because the DEK lookup is bound to the metadata row. FDB hot-range splitting handles skewed tenants at the metadata layer; quotas (enforced transactionally via the counters) cap storage. Physical isolation (dedicated cells) is an evolution for a future compliance tier, not launch.

Failure modes, walked through

Gateway dies mid-PUT: no metadata row was committed, so nothing is visible; orphaned chunks (appended but never referenced) are found by the reconciliation scan and GC'd. Client retries per SDK behavior.
Storage node down: open-segment appends re-route to a new replica set immediately (the gateway just picks a different open segment); sealed-segment reads reconstruct from any 8 of 12 fragments with one extra round trip; repair rebuilds in the background. No user-visible failure.
AZ loss: gateways and FDB continue in two AZs; open segments lose one of three replicas (re-replicate); sealed segments lose 4 of 12 fragments (read-degraded, repair when the AZ returns rather than rebuilding 5 PB — repair policy distinguishes "disk is dead" from "AZ is probably coming back").
FDB unavailable: full outage of the API — this is the availability keystone, which is why it gets three AZs, the most conservative change management, and the off-cluster backup. Budgeted within 99.95%.
Repair storm / correlated disk batch failure: repair is throttled with priority by how many fragments a segment has lost; a segment at 8-of-12 repairs before a hundred segments at 11-of-12.
Clock skew and presigned expiry: gateways run NTP-disciplined clocks; we honor S3's ±15-minute request-time window.
Our own worst enemy: a bad deploy that corrupts or mass-deletes. Defenses, in order: staggered rollouts, end-to-end checksums on read, the 24-hour delete quarantine, the append-only delete audit log, and off-cluster metadata backups.

Evolution

In rough order of when the pressure arrives: lifecycle policies and object versioning (pure metadata features — the immutable-chunk substrate already supports them); a metadata read cache when FDB point-read cost is measured to matter; LRC codes when repair bandwidth is; a cold tier (wider codes, denser disks, relaxed latency) when storage economics demand it; multi-region buckets (async replication driven off a change feed from the metadata layer — the Kafka pipeline already exists) when customers demand it. None of these require reworking the three core layers, which is the test I'd apply to any launch architecture: the v1 should be small because the team is small, but nothing in it should have to be torn out to grow 50% a year.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): Your delete queue and append-only audit log retain the wrapped per-object DEK, while the per-bucket wrapping key remains usable; how is that crypto-shredding at all, and how do you prevent FDB backups, KMS backups, or audit-log copies from decrypting a deleted object after the 30-day deadline?

Two separate concerns, one fine and one that lands. The per-bucket wrapping key staying alive is fine: what crypto-shredding destroys is the wrapped-DEK ciphertext itself. Without that ciphertext there is nothing for the bucket key to unwrap — the bucket key alone recovers nothing. KMS backups hold bucket keys, not DEKs, so they resurrect nothing by themselves. The FDB-backup hole is already closed in the design: backup retention shorter than the compliance window, plus replay of the delete queue against any restored cluster before it serves traffic.

Where the question lands is the audit log. As written — "the delete queue (retained as an append-only audit log)" — the queue rows, wrapped DEK included, outlive key destruction. That breaks the scheme outright.

Design change: the audit log stores deletion_id, bucket/key, timestamps, and a hash of the wrapped DEK — never key material. Destruction means deleting the only copy of the wrapped DEK (the queue row itself); the hash lets an auditor verify a restored backup contains no key that should be gone. Backup retention pinned at 7 days against the 30-day window, so expiry plus delete-queue replay always completes inside the deadline.

Q2 (Codex): Completing, overwriting, or deleting a 5 TB object can involve roughly 80,000 chunk references, yet FoundationDB limits transactions to 10 MB and five seconds; what concrete staged-manifest protocol makes each operation atomic while preventing GETs and GC from observing partially published or partially retired manifests?

The design says manifests split across rows but left the protocol implied; here it is. Manifests are version-addressed: upload start mints a version UUID, and manifest pages are keyed ("man", bucket, key, version, page). Pages are written in ordinary batched transactions before the commit; each batch re-checks that the upload row still exists, fencing against a concurrent abort. The commit is then one small transaction: re-check the upload row, write the object row containing just a pointer (version, page_count, etag, size), delete the upload row, and enqueue the previous version (if overwriting) to the delete queue. S3 caps multipart at 10,000 parts, so part validation stays a few MB — and it happens in the staged phase, not the commit.

GETs follow the pointer in the object row, so pages of an uncommitted version are unreachable — no reader can observe a partial manifest. On retirement, DELETE enqueues the version pointer and GC walks that version's manifest pages incrementally, freeing chunks page by page. GC treats manifest pages as live iff referenced by an object row or by an upload row younger than the 7-day sweep horizon. No operation anywhere needs a transaction proportional to object size.

Design change: version-addressed manifest pages written pre-commit, a pointer-flip commit transaction, and lazy part-row cleanup by the sweeper instead of deletion inside the commit.

Q3 (Codex): Multiple stateless gateways concurrently append to the same three-replica open segment: who assigns canonical offsets, orders writes, resolves partial chain acknowledgments, fences a failed writer, and prevents sealing from racing with an append—without the consensus, leases, or primary state you claim the blob layer does not need?

I concede the phrasing oversold it. The honest claim is "no consensus among blob nodes" — coordination exists, and it lives in FDB, which we already run. The mechanism that keeps the rest simple: open segments have exactly one writer. Each gateway creates and exclusively owns its own open segments; gateways never share one. At ~5k writes/s over a few dozen gateways that's ~100–150 appends/s per gateway, and a 1 GB segment still fills in minutes. Single writer means the gateway assigns offsets and orders appends trivially; replicas just append what the chain hands them.

Failure handling: the metadata commit requires all three chain acks, so a partially replicated append is never referenced — a torn tail past the sealed length is garbage by construction. Sealing is epoch-fenced: the segment row in FDB carries an epoch; sealing (by the owner, or by a janitor after missed heartbeats) first CASes the epoch in FDB, then issues seal(segment, epoch+1) to the replicas, which thereafter reject appends and seals bearing stale epochs. The janitor seals at the minimum fully-replicated length and records it in the segment row. A zombie gateway's in-flight append fails at the replicas, its metadata commit fails the epoch check in the transaction, and the client gets a retryable error.

Design change: single-writer-per-open-segment and epoch-fenced sealing made explicit; the "no consensus" claim narrowed to "no consensus among blob nodes — FDB is the sole coordination point."

Q4 (Codex): After a gateway appends data but crashes before committing metadata, how can reconciliation distinguish an orphan from an in-flight PUT without scanning billions of object manifests or racing ahead and deleting bytes that a still-running gateway is about to publish?

Conceded — "found by the reconciliation scan" was a hand-wave, and a global manifest scan is not an answer at 10B objects. The fix is a reverse index written in the same commit transaction that makes a chunk live: ("segidx", segment_id, offset) -> (bucket, key, version), one small row per chunk. The median object is one chunk, so one extra ~60-byte row per PUT; part commits write it too. Because it's in the commit transaction, it is exactly consistent with reachability at creation time.

Orphan rule: a byte range in a segment sealed more than 24 hours ago with no segidx entry is garbage — an O(one segment) range scan. The race can't happen: a live gateway commits seconds after its append, its segment can't be sealed out from under it without the epoch fence from Q3, and once fenced its pending commits fail — so "sealed + 24 h + unreferenced" never describes a byte still about to be published. The same index is what the compactor needs anyway to answer "which chunks in this segment are live," which the original design required and didn't specify.

Design change: segidx reverse-index rows written in the commit transaction; the vague reconciliation scan demoted to a periodic invariant check.

Q5 (Codex): With exactly four EC fragments per AZ, losing one AZ leaves the object at the minimum eight fragments, so one additional disk failure or latent corrupt fragment makes it unreadable; how does the policy of waiting for the AZ to return satisfy 99.95% availability and 11-nine durability, especially with a two-week scrub interval and correlated failures excluded from your calculation?

The sharpest question, and partially conceded. Separating the claims: an AZ outage does not destroy media — the 4 fragments inside it still exist on disks, so durability is only at risk in the compound case (AZ permanently destroyed plus additional failures), which my independent-failure math did not cover. The availability exposure during an outage is real and quantifiable: 8 survivors, zero slack, one more disk death makes that segment unreadable (7 < 8) until the AZ returns. With ~667 surviving disks at 2% AFR over a 4-hour outage, that's ~7×10⁻⁵ per segment — roughly 1,500 of 20M segments (≈1.5 TB) unreadable in a bad 4-hour AZ outage. Fleet-weighted that fits a 99.95% budget, but for the affected objects it's a total outage, and I said "no user-visible failure" where I should have quantified.

Design change (superseded by a stronger one in Q8): (1) never carry latent zero-slack into an outage — any segment with a scrub-detected bad fragment repairs at top priority; (2) an AZ-outage re-protection mode: once an AZ is down longer than ~1 hour, background workers read each segment's 8 survivors and regenerate missing fragments into the surviving AZs, hottest and oldest first, throttled — restoring slack incrementally rather than choosing between "rebuild 5 PB now" and "wait." Round 2 pushed on the math here and forced a bigger revision — see Q8.

Q6 (Codex): A PUT that increments one randomly selected counter shard cannot atomically enforce a bucket-wide quota unless it reads all shards, which recreates contention and still races with concurrent PUTs; what exact admission protocol prevents aggregate usage from exceeding quota, and why is weekly reconciliation needed if the counters are truly exact?

The premise is right and I overclaimed. "Exact and transactional" is true of accounting — increments are FDB atomic ADDs in the commit transaction, conflict-free, so committed counters are exactly correct. It is not true of bucket-wide admission: reading all shards inside the PUT transaction would put a read conflict range on every shard, and concurrent ADDs would abort it constantly — recreating the contention the sharding existed to avoid.

Design change: two-mode admission with bounded overshoot instead of a false exactness claim. Normal mode: PUT does the atomic ADD; admission checks a near-real-time per-bucket sum that a small aggregator folds from the shards into a ("usage", bucket) row every few seconds, cached at gateways. Overshoot is bounded by bucket write bandwidth × refresh interval — a bucket writing 100 MB/s with a 5 s refresh overshoots at most ~500 MB — and the soft admission threshold sits below quota by that bound. Strict mode: past ~95% of quota the bucket flips to a single counter row with transactional read-check-add; contention is now acceptable because throttling that bucket's writes is the point. Weekly reconciliation stays for the reason the design gave — an invariant check against our own bugs, not a correctness dependency. Billing reads the committed counters, which are exact.

Round 2 — pushback

Q7 (Codex): Your segidx is "exactly consistent" only on creation: how do overwrite and DELETE atomically remove up to 80,000 reverse-index rows within FDB's 10 MB transaction limit—without stale rows blocking reclamation or asynchronous cleanup racing compaction?

They don't get removed atomically, and they don't need to be — the invariant is weaker and still sufficient. The only atomic step in a DELETE or overwrite is the pointer flip: remove/replace the object row, enqueue the old version to the delete queue. segidx rows for the old version become stale at that instant but are removed incrementally by the GC worker that already walks the dead version's manifest pages: each batch transaction frees a page's chunks, deletes their segidx rows, and decrements segment live_bytes — 80k rows is ~80 batches, done in minutes. Stale rows don't block reclamation, they delay it per-chunk, which is exactly the 24-hour quarantine behavior the design wants anyway; the hard compliance bound never depends on this path, because it's the DEK destruction at 24 hours and the 25-day compaction deadline that enforce it, both of which run off the delete queue, not segidx.

The compaction race resolves with per-chunk CAS, no big transactions anywhere. The compactor moves a live chunk by reading its segidx entry, copying the bytes, then in one small transaction: verify the manifest page entry still points at the old location, update it to the new location, move the segidx row. If GC killed the version concurrently, the verify fails and the compactor drops the chunk as garbage. Both walkers make monotonic per-chunk progress; neither can resurrect or double-free. The one-sentence restatement of the invariant: segidx may over-approximate liveness for minutes, never under-approximate it — over-approximation delays space reclaim, under-approximation would lose data, so the asymmetry is chosen deliberately.

Design change: segidx consistency contract stated as create-atomic, retire-lazy; compactor chunk moves are per-chunk verify-and-swap transactions against the manifest page.

Q8 (Codex): Your launch design still has zero redundancy after an AZ loss, while "re-protection" requires reconstructing multiple petabytes; what concrete bandwidth and completion-time calculation shows you can restore slack before the next disk failure and still honestly claim 11-nines durability and 99.95% availability?

I ran the numbers and the re-protection answer doesn't survive them, so the design changes. Full re-protection after AZ loss means reading 8 fragments and writing 2 per segment: ~20 PB read across ~667 surviving disks. At a 50 MB/s per-disk background budget that's ~33 GB/s aggregate — about 7 days. Expected survivor-disk failures during those 7 days: 667 × 2% × 7/365 ≈ 0.26, so we probably finish first, but "probably" is not an 11-nines argument, and for the common transient outage (hours) re-protection barely starts before the AZ returns. Worse, in the permanent-destruction case a segment that loses one survivor before re-protection reaches it is at 7 < 8: data gone. 8+4 at 4-per-AZ cannot honestly claim 11 nines against permanent AZ loss, full stop.

Design change: launch with RS(9,6) instead of 8+4 — 15 fragments, 5 per AZ, 1.67× overhead versus 1.5×. That's ~1.7 PB more raw at launch scale, roughly 85 more 20 TB spindles: a priced, deliberate durability purchase. What it buys is qualitative, not incremental: after an AZ loss, 10 fragments survive against k=9, so repair still works during the outage — a lost survivor is regenerated from the remaining 9 into the surviving AZs within the normal 6-hour repair window, instead of being unrecoverable until the AZ returns. Unreadability now requires two survivor losses inside one repair window: ~C(10,2)×(1.4×10⁻⁵)² ≈ 9×10⁻⁹ per segment-window, ~180 expected unreadable segments per window across 20M — versus ~1,500 with zero slack and no repair path, and the durability math holds even if the AZ never comes back. The AZ-outage re-protection mode from Q5 stays, but demoted to what it honestly is: the recovery procedure for declared permanent AZ loss, not the thing the durability claim leans on. Availability budget is then dominated by FDB and gateways again, where it belongs.

What changed, summarized

Audit log stores a hash of the wrapped DEK, never key material; the delete-queue row is the only copy, and destroying it is the shred. Backup retention pinned at 7 days.
Manifests are version-addressed pages written before the commit; commit is a small pointer-flip transaction; part rows and dead-version pages retire lazily. No transaction scales with object size.
Open segments are single-writer (the creating gateway), with epoch-fenced sealing CASed through FDB; "no consensus" narrowed to "no consensus among blob nodes."
New segidx reverse-index rows written in the commit transaction: orphan detection and compaction liveness become O(segment); contract is create-atomic, retire-lazy; compactor moves chunks via per-chunk verify-and-swap.
Erasure code changed from RS(8,4) to RS(9,6) at launch (1.67× vs 1.5×, ~85 extra spindles) so one-fragment slack survives an AZ loss and repair keeps working during the outage; re-protection mode retained only as the permanent-AZ-loss recovery procedure.
Quota admission is two-mode: conflict-free atomic ADDs plus a near-real-time aggregated sum with bounded overshoot, flipping to a strict single-row transactional check near the limit. "Exact" now claimed only for accounting/billing, not admission.

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The three-layer split — stateless gateway, strongly consistent metadata store, dumb blob layer — is the shape shipping systems converge on, not an invention of this design. Cloudflare's How R2 works documents exactly this decomposition: a gateway on Workers, a metadata service built on Durable Objects, and separate distributed storage for encrypted object data — and the same commit ordering I used, metadata committing only after the data write succeeds, which is where their strong consistency comes from. Tigris went further and built its S3-compatible store's metadata layer directly on FoundationDB; their key layout post — tuple-encoded keys with prefixes per record type, integer-compressed identifiers — reads like a production version of my ("obj", …) / ("seg", …) keyspace. So "FDB under an object store" is a proven pattern, not a bet.

On FDB itself, the SIGMOD '21 paper is the primary source for the two properties this design leans on: strict serializable transactions (OCC + MVCC, an unbundled transaction/storage architecture) and the deterministic simulation framework that justifies trusting its failure behavior — with production deployment at Apple and Snowflake stated in the paper, whose author list spans both companies. Snowflake's own account describes running FDB as their metadata store since 2014 for exactly our workload profile: very high frequency of tiny reads and writes at sub-millisecond latency, triple-replicated across availability zones. My 3–5 TB / 45k reads/s estimate sits well inside that operating envelope.

The blob layer's lineage is real too. The Windows Azure Storage SOSP 2011 paper is the canonical description of the append-only-extent stream layer under a separate partition (index) layer, delivering strong consistency and availability together. Replicate-then-EC as an economics move is Facebook's f4 paper (OSDI '14): hot blobs stay triple-replicated in Haystack, warm blobs move to RS(10,4) cells, cutting effective replication from 3.6× to 2.8× and then 2.1× across 65 PB. My open-segment 3× → sealed RS(9,6) at 1.67× is the same trade compressed from a tiering decision into a sealing pipeline. On geometry: Backblaze Vaults run 17+3 across 20 storage pods — ~1.18× overhead, much cheaper than my 1.67× — but the shards live in one facility; f4 pays extra XOR parity across datacenters for the same reason I pay RS(9,6) across AZs. Wider-and-cheaper versus survive-a-site is the actual industry axis, and where you land depends on what a "zone" means for you. When repair reads become the constraint, the published answer is Local Reconstruction Codes (USENIX ATC '12), whose stated motivation is cutting reconstruction I/O while keeping overhead low — which is precisely the evolution slot this design reserved for LRC rather than launching with it.

Two consistency data points frame the strong read-after-write decision. S3 itself was eventually consistent for 14 years; the December 2020 retrofit to strong consistency for all GET/PUT/LIST — no performance or cost change — is the industry conceding that eventual consistency at the storage API was a mistake customers paid for (EMRFS Consistent View, S3Guard). R2 launched strongly consistent from day one. And the sharpest support for my "no metadata cache in the read path at launch" call arrived after this design was written — see the updates below.

Finally, on what actually threatens durability: Andy Warfield's Building and operating a pretty big storage system called S3 spends its durability section not on independent-failure math but on process — human "durability reviews" modeled on threat modeling, and lightweight formal methods (ShardStore) against their own bugs — plus heat management across millions of spindles. That matches this design's claim that correlated failures and software bugs, not disk AFR arithmetic, are what kill 11 nines; the scrubber, staggered rollouts, delete quarantine, and end-to-end checksums are the budget spent accordingly.

Updates from post-training information

Backblaze Drive Stats for 2025 (published 2026-02-12): fleet-wide AFR for 2025 was 1.36%, lifetime 1.30%, across 344,196 drives. My durability math assumed 2% AFR, so the assumption is conservative by roughly 1.5× — the margin widens, no revision needed. Two secondary signals: 20 TB+ drives are now ~23% of their fleet and the first 26 TB models are in service, so my "20 TB spindles" sizing is already on the small side; bigger disks mean more data behind each failure and longer rebuilds, which moves the LRC evolution point closer.
Antithesis report: Tigris Data (published 2026-04-21): Tigris ran their full FDB-backed S3 system under deterministic-simulation fault injection — 20.3 million unique states explored, July 2025 to March 2026. Every key bug found shared one root cause: the coherence window between a metadata operation committing in FDB and their edge cache reflecting it, including a delete-then-read race that served deleted objects. That is the exact bug class this design avoided by refusing a metadata cache at launch and restricting any future cache to immutable content keyed by version. I made that call defensively; the Tigris report is evidence it's the bug you actually get.