Contents

Global live streaming with on-demand replay

What we're building, from the user's side

A creator opens the app, taps "Go live," and the phone starts pushing video to the nearest ingest point. Within a few seconds the broadcast appears on their channel. Viewers find it through search or a home feed, tap it, and playback starts at a quality matched to their connection, staying within about five seconds of real time. A chat pane scrolls next to the video. When the creator ends the broadcast, the recording is watchable almost immediately — no upload step, no processing wait — and the creator can see how many people watched, for how long, from where. They can also delete the recording, and deletion has to actually mean deletion.

Two experiences, one system: the live path and the replay path. The design below is built around a decision that makes both cheap at the same time: the recording is a byproduct of the live pipeline, not a separate product.

Assumptions I'm adding

The prompt gives scale; I'll add the assumptions that shape the architecture:

Requirements, prioritized

  1. Live playback works at 25M concurrent viewers, within ~5s of live. This is the whole product. Everything else degrades before this does.
  2. A 1M-viewer join spike in one minute doesn't take down the stream — or any other stream.
  3. Replay is available right after the broadcast ends, and recordings are retained indefinitely unless deleted.
  4. Chat works in rooms from 3 viewers to 3 million, degrading gracefully at the top end.
  5. Creator stats are accurate enough (concurrent viewers within a few percent, watched minutes within ~1%) — not billing-grade.
  6. Deletion propagates: CDN, storage, search, within a bounded time (say, minutes for reachability, days for physical erasure).

One principle runs through the whole design and I'll repeat it where it applies: the media plane must keep working when the control plane is down. A database outage may block starting new broadcasts; it must never stop 25M people from watching the ones already running.

The numbers that force the architecture

Rough estimates, but each one eliminates a class of design:

The core decision: chunked CMAF over HTTP, not WebRTC

Delivery protocol is the decision everything else hangs off, so I'll make it first.

The intuitive pull is toward WebRTC: it's the low-latency video protocol, sub-second, built into every browser. And for a 50-person video call it's right. But WebRTC delivery means a stateful, per-viewer session terminated on a server we run. At 25M viewers that's 25M concurrent sessions of un-cacheable traffic — we'd be building a 75 Tbps real-time relay network ourselves, and the 1M-joins-in-a-minute spike becomes 1M new stateful session negotiations against our own fleet. Commercial CDNs can't help because there's nothing to cache.

So: LL-HLS with CMAF segments (plus LL-DASH from the same media, since CMAF lets one set of files serve both). Video is encoded into 2-second segments, each delivered in ~500ms chunks via HTTP chunked transfer. Every viewer of a given rendition requests the same URLs, so the CDN edge serves almost everyone from cache, and edge request-coalescing means the origin sees roughly one request per segment per rendition per CDN — independent of viewer count. That property is what makes the flash crowd survivable: viewer number 1,000,000 costs the origin nothing more than viewer number 3.

Latency budget, roughly: encode + package ~1.5–2.5s, CDN and network ~0.5s, player buffer ~2–3s. That lands at 3–5 seconds glass-to-glass, meeting the target. Plain HLS with 6-second segments would sit at 15–30s — fails the requirement, rejected. WebRTC delivery — rejected above, though I'd keep WHIP (WebRTC ingest) on the roadmap for creators, where per-session state is only 50k sessions, not 25M.

Architecture

flowchart LR
    subgraph FirstMile["First mile"]
        C[Creator app / OBS] -->|RTMPS or SRT| IG[Ingest PoP\n~30 sites, anycast]
    end

    subgraph Region["Media region (per continent)"]
        IG -->|internal backbone| TC[Transcode fleet\nGPU jobs, 1 per stream]
        TC --> PK[Packager\nLL-HLS / DASH, CMAF]
        PK --> OS[(Object storage\nsegments + manifests)]
        PK --> HOT[Hot origin cache\nin-memory, last ~60s]
    end

    subgraph Delivery["Delivery"]
        HOT --> SH[Origin shield]
        OS --> SH
        SH --> CDN1[CDN A edge]
        SH --> CDN2[CDN B edge]
        CDN1 --> V[Viewers]
        CDN2 --> V
        STEER[CDN steering svc\nRUM-driven] -.picks CDN per session.-> V
    end

    subgraph Control["Control plane"]
        API[API gateway] --> BSVC[Broadcast svc]
        API --> USVC[User/channel svc]
        BSVC --> PG[(Postgres\nmetadata)]
        BSVC -->|schedule job| TC
        API --> TOK[Playback token svc]
    end

    subgraph Realtime["Chat + analytics"]
        V -->|WebSocket| CG[Chat gateways]
        CG <--> RB[Room brokers\nfanout tree]
        V -->|heartbeats| BE[Beacon endpoint] --> K[Kafka] --> F[Flink] --> CH[(ClickHouse)]
        F --> RC[(Redis\nlive counts)]
    end

    OS --> VOD[VOD lifecycle svc\ntiering, deletes]

Five planes, deliberately decoupled: first mile, media processing, delivery, control, and realtime (chat/analytics). The media plane — ingest through CDN — runs on pre-issued credentials and cached configuration, so it keeps working when the control plane is down.

First mile: ingest PoPs

Creators push RTMPS (universal: OBS, every mobile SDK) or SRT (better loss recovery on bad networks; we prefer it where the client supports it). I rejected ingesting straight into a central cloud region: a creator on mobile in Jakarta pushing to Virginia will drop frames on the long lossy path, and no amount of server cleverness recovers video that never arrived. Instead, ~30 ingest PoPs worldwide, reached by anycast/GeoDNS, terminate the connection close to the creator and forward over our provisioned backbone (or cloud backbone — AWS Global Accelerator-style — before we own fiber).

The ingest PoP authenticates the stream key, stamps the broadcast ID, buffers the last ~30 seconds of source on local disk (this buffer matters for failure recovery below), and forwards to the assigned media region.

Transcode: lazy ladders

One job per live stream, containerized, scheduled onto a GPU fleet (Kubernetes-style scheduler with bin-packing by codec/resolution). GPU (NVENC), not CPU x264 — at 50k concurrent streams the power and fleet-size math for software encoding doesn't close; the quality gap at live bitrates is acceptable. Custom transcode ASICs are the 10× answer, not the day-one answer (YouTube's VCU and Meta's MSVP exist precisely because this fleet dominates cost at their scale).

The skew assumption pays off here. A stream with 4 viewers doesn't need six renditions. So: every stream gets the top rendition transcoded (that's also the recording), plus one low rung for compatibility. When concurrent viewers cross a threshold (say 50), the control plane promotes the stream and the job spins up the full ladder — 1080p60, 720p60, 720p30, 480p, 360p, audio-only. The promotion takes a few seconds and viewers on the partial ladder never notice. Estimated saving: most of the fleet, since most of the 50k streams are small. The tradeoff — a small stream's viewers get fewer quality choices — is one I'll take.

The transcoder writes CMAF chunks to the packager, which maintains LL-HLS and DASH manifests and does two writes per segment: to an in-memory hot origin (the last ~60 seconds, serving live edge requests) and to regional object storage (S3), which is simultaneously the live origin for older segments, the DVR seek buffer, and the recording. That's the "recording is a byproduct" decision: when the broadcast ends, we write a final VOD manifest over segments that are already durably stored. Replay is available in seconds because nothing needs to move.

Delivery: multi-CDN and the flash crowd

Day one, we buy delivery: two or three commercial CDNs (say CloudFront + Fastly + Akamai) behind a steering service that picks a CDN per playback session using real-user measurements and cost, and can shift traffic away from a failing CDN in about a minute by changing what the playback API hands out. I rejected single-CDN (one partner's bad day is our outage; no pricing leverage at 75 Tbps) and rejected building our own edge first (it's the right long-term move — see the 10× section — but it's a multi-year buildout and the product can't wait).

The 1M-viewers-in-a-minute spike, walked through end to end, because it's the scenario most likely to kill a naive design:

Playback tokens are short-lived signed URLs validated at the CDN edge (all major CDNs support edge signature checks), so stolen manifest URLs die in minutes without any origin involvement.

Storage lifecycle: "indefinitely" is a cost curve, not a feature flag

Two exabytes a year, growing 10×, retained forever. If we keep the full ladder for every recording forever, storage becomes the company's largest line item behind egress. The observation that saves us: replay viewing decays fast — most recordings get nearly all their views in the first days.

So a tiered lifecycle, run by the VOD lifecycle service:

Erasure coding (built into S3-class storage) rather than 3× replication for durability. Popular archival content gets pinned back to warm tiers by the same watch-rate signals.

Deletion: creator deletes → metadata tombstone (immediate, hides it from every API and search index) → CDN purge of manifests (minutes; segments without a manifest and token are unreachable and age out) → hard delete from object storage via an async job with verification (hours to days). Tombstone first is what makes "deleted" true from the user's perspective immediately, and it's also the GDPR story.

Chat: a fanout tree with a throttle at the root

Viewers hold a WebSocket to a chat gateway (each gateway comfortably holds tens of thousands of connections). Gateways subscribe to per-room brokers. Small rooms are trivial. The problem is the megaroom, and the honest answer is that a 1M-viewer chat is not a conversation — it's a firehose — so we design for perceived liveness, not completeness:

Delivery is at-most-once over the socket. Accepted messages also land in a durable, partitioned event log (Kafka) for moderation and for the replay-chat track (chat replayed alongside the recording, aligned by broadcast timestamp — cheap, since it's just a time-indexed read from a columnar analytics store (ClickHouse) or an object-store log). I rejected guaranteed-delivery chat (per-viewer acks and resend state at 25M connections buys nothing the product needs) and rejected building chat on Kafka consumer-groups-per-room (partition churn with 50k rooms appearing and vanishing hourly).

Analytics and view counts

Clients beacon heartbeats (join, progress every 30s, quality switches, rebuffer events) to a beacon endpoint → Kafka (~830k events/sec peak) → Flink. Flink maintains per-broadcast concurrent-viewer counts with HyperLogLog sketches (approximate, tiny state, fine for a number displayed as "1.2M watching") pushed to an in-memory cache (Redis) for the live UI, and aggregates watched minutes, geo splits, and rendition mix into ClickHouse for the creator dashboard. ClickHouse over Druid mostly for operational simplicity and because our queries are per-channel time-series rollups, which it eats. Beacons are fire-and-forget; losing 1% of heartbeats moves creator stats by about 1%, which is inside the accuracy target.

Data model and APIs

Metadata lives in Postgres (Aurora-style, partitioned by channel), because it's low-volume, relational, and transactional — the write rate here is thousands per second, not millions. The high-volume data (segments, beacons, chat) never touches it.

users(user_id, handle, region, created_at, ...)
channels(channel_id, owner_user_id, stream_key_hash, settings_json)
broadcasts(broadcast_id, channel_id, state ENUM[live,ended,deleted],
           started_at, ended_at, ingest_pop, media_region,
           ladder ENUM[minimal,full], title, ...)
recordings(broadcast_id PK, manifest_url, renditions_json,
           storage_tier ENUM[hot,warm,cold], visibility, deleted_at NULL)

Segments themselves are keyed in object storage as /{broadcast_id}/{rendition}/{seq}.m4s — the key scheme is the index; no database row per segment (at ~2s segments, that would be ~2 billion rows a day of pure liability).

APIs, the load-bearing ones:

POST /v1/broadcasts                  -> {broadcast_id, ingest_url, stream_key}   (creator)
POST /v1/broadcasts/{id}/stop
GET  /v1/broadcasts/{id}/playback    -> {manifest_url, playback_token, cdn}      (viewer; token TTL ~5 min, refreshed in-band)
GET  /v1/channels/{id}/stats?window= -> concurrents, watched_min, geo, renditions
DELETE /v1/recordings/{broadcast_id} -> 202 (tombstone now, purge async)
WS   /v1/chat/{broadcast_id}         (send/receive; join carries auth token)

Uploaded (non-live) videos enter through a resumable upload API and join the same transcode → package → object-store pipeline as a broadcast with no live viewers; the 1M hours/day figure covers both, and sharing the pipeline is free.

Failure modes, walked through

Security and abuse

Getting to 10×

At 10× — 250M concurrent viewers, ~750 Tbps, 500k concurrent transcodes, 20+ EB/year — three things change in kind, not just in count:

  1. Own the edge. At 750 Tbps, commercial CDN pricing is the biggest cost in the company. The move is Netflix Open Connect's: our own caching appliances embedded inside ISPs, with commercial CDNs kept for overflow and long-tail geographies. The multi-CDN steering service we built on day one is exactly the control point that makes this migration incremental.
  2. Transcode silicon. GPU encoding at 500k concurrent streams is a power-and-fleet problem; dedicated VPU/ASIC encoders (the VCU/MSVP path) cut cost-per-stream several-fold. Also: per-title/per-scene encoding and AV1 for the biggest streams — at this egress volume, a 20–30% bitrate saving (typical claims for AV1 over H.264; I'd validate on our corpus) is worth real money per day.
  3. Storage discipline becomes existential. The tiering policy tightens (earlier down-laddering, JIT everything cold), and we'd revisit "keep the top rendition forever" versus "keep source + re-encode on demand" with actual watch-decay data.

The things that don't change are the point of the design: HTTP-cacheable media, lazy ladders, recording-as-byproduct, and a media plane that ignores control-plane weather. Those were chosen so that 10× is a procurement and buildout problem, not a re-architecture.

Tradeoffs I'm consciously making

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): During a one-million-viewer flash crowd, how do LL-HLS blocking playlist reloads, 500 ms partial segments, per-session authorization, and multi-CDN URL signing preserve a shared cache key and request collapsing—and what is the quantified shield/origin load across thousands of cold edge caches?

The seam is real if the token rides in the URL path: per-session signed URLs would make every viewer's cache key unique and destroy collapsing. So the rule is: the cache key is the bare object path, and auth travels out-of-band — a signed cookie or an Authorization header (or a query param the CDN config strips from the cache key before lookup), validated by edge logic (CloudFront signed cookies, Fastly VCL, Akamai token auth all support this). Every viewer of a rendition then hits the same cache entry. LL-HLS blocking reloads (_HLS_msn/_HLS_part query params) do vary the URL, but at the live edge all players ask for the same next msn/part, so there are only a handful of distinct blocking URLs in flight per rendition, and the CDNs hold-and-collapse those requests until the origin releases the part.

Quantified, per promoted stream with a 6-rung ladder and 500ms parts: each rendition generates ~2 part fetches/s, ~2 blocking playlist responses/s, ~0.5 full-segment fetches/s — call it ~5 object-completions/s per rendition, ~30/s per stream. Each CDN's shield collapses its edges down to one origin fetch per object, so the origin sees ~30 req/s per stream per CDN, ~90/s across three CDNs — independent of whether there are 50 viewers or 5 million. The cold-edge fanout lands on the shield, not the origin: 1,000 cold edges each fetching ~30 small objects/s for the megastream is ~30k req/s at the shield, served from memory; that's a provisioning line item, not a scaling cliff. One thing the doc glossed: the 1s-TTL figure was for classic manifest polling; with blocking reloads what matters is collapse-and-hold, and we must verify each CDN partner actually supports request collapsing on chunked-transfer responses — a procurement checklist item, because a CDN that can't do it turns the megastream into an origin DDoS.

Design change: playback tokens move out of the cache key by contract — signed cookie / stripped query param, edge-validated — and CDN request-collapsing on blocking LL-HLS requests becomes an explicit acceptance criterion for any CDN we contract.

Q2 (Codex): If a transcoder or media region fails while the creator's connection to the ingest PoP remains healthy, what mechanism reassigns the stream, transfers the PoP's local buffer, fences the old writer, and preserves timestamps, segment sequence numbers, and atomic manifests without gaps or conflicting objects?

The original text said "resumes at the next segment boundary" and hand-waved how both sides agree on the boundary. Concretely: segment numbering is derived deterministically from source PTS — seq = floor(source_PTS / 2s), stamped at the ingest PoP, which is the single source of truth for the stream's timeline. Any transcoder, old or new, computes the same seq for the same source frames, so a replacement job doesn't need to inherit encoder state to continue the numbering; it pulls from the PoP's 30s disk buffer (the buffer isn't transferred — the PoP holds it and the new job reads it over the backbone, addressed by broadcast_id) and starts encoding at the next seq boundary it can fully cover.

Fencing: each transcode placement gets a session epoch from the scheduler, and the packager for a broadcast holds a lease (etcd-style). Segment writes are conditional PUTs (If-None-Match: * — first writer per key wins), so a zombie writer can't overwrite a segment the new writer already produced; and because two encoders produce different bytes for the same seq, first-writer-wins is what keeps the object store consistent. Manifest updates go through the lease-holding packager only, as atomic versioned puts, so there's never a torn manifest. What I won't promise: bitstream continuity across the failover. The new encoder starts fresh at an IDR on a segment boundary, so the player sees at worst a brief stall if the gap exceeded its buffer; if the region died and a few seconds of source outran the PoP buffer, those seconds are a hole in the recording, marked with EXT-X-GAP in the VOD manifest. That's the async-replication tradeoff already priced in.

(Round 2 pushes on the fencing half of this — see Q7, where the PUT-race scheme gets replaced.)

Design change: seq assignment moves from the transcoder to the ingest PoP (deterministic from source PTS); transcode placements carry an epoch, manifests are written only by a lease-holding packager, and segment PUTs are conditional first-writer-wins.

Q3 (Codex): When a stream crosses the promotion threshold, how do existing players—many of which fetch the master playlist only once—discover the new renditions, and how are GOP boundaries, codecs, timestamps, and manifests aligned so ABR switching and the final VOD remain valid across ladder changes and restarts?

Two honest parts. First, who needs the new renditions: promotion fires at ~50 viewers, so at most ~50 sessions hold the stale master; everyone joining during the growth that triggered promotion fetches a fresh master and sees the full ladder. For the stale few, we control the player SDK on our own apps, and it refetches the multivariant playlist every 60s (a cached, cheap request); third-party HLS players that never refetch just stay on the two launch renditions, which is the same experience they had before promotion — degraded choice, not breakage. I won't pretend mid-session discovery works for players we don't control; it doesn't, and it doesn't need to.

Second, alignment and the VOD. All renditions are encoded from the same source with IDR-aligned 2s GOPs and seq derived from source PTS (same mechanism as Q2), so a rendition that begins at seq N starts cleanly on a segment boundary with consistent timestamps — ABR switching between rungs works because switch points are always aligned IDRs. The final VOD is where mid-stream rendition starts bite: a variant playlist that only covers the last 80% of the timeline makes ABR ugly in replay. So the VOD manifest includes only renditions that cover the full timeline — the top rung and the low rung always do — and for promoted (i.e., popular) broadcasts, a post-broadcast backfill job transcodes the missing early segments of the mid-ladder rungs from the stored top rendition, then swaps in the complete variants. Backfill cost is proportional to popularity, which is exactly where we want to spend transcode.

Design change: player SDK refetches the master playlist periodically; VOD manifests admit only full-timeline renditions, with an async backfill job completing mid-ladder rungs for promoted broadcasts.

Q4 (Codex): How can existing playback survive a control-plane outage when five-minute playback tokens require refresh, while still supporting rapid key rotation, access revocation, creator deletion, and the moderation kill switch without either failing closed on viewers or failing open on revoked content?

The question exposes a misclassification in my diagram: I drew the token service inside the control plane while claiming the media plane survives control-plane death — and with 5-minute TTLs, playback dies within 5 minutes of the token service dying. The fix is to make the token service genuinely media-plane: it's a stateless signer that needs only its private key and a cached policy blob, no database on the hot path, so it deploys per-region next to delivery, on the same isolation footing as the packager. A Postgres or broadcast-service outage then never touches token refresh.

Which forces the second half of the question: if we lean toward failing open on playback, revocation can't depend on token expiry. It doesn't. The kill switch and deletion are enforced by making content unreachable, not by starving tokens: kill = tear down ingest + stop the packager + purge manifests at the CDN (all media-plane actions, seconds to minutes); delete = tombstone + manifest purge as designed. A valid token pointing at a purged manifest fetches nothing. Token TTL is the defense against casual link-sharing and hotlinking, and that's all it needs to be. Key rotation composes fine: edges hold an overlapping key set, new tokens sign with the new key, old keys expire after the longest outstanding TTL.

(Round 2 pushes on the "what if every signer is down" corner — see Q8.)

Design change: the playback token service is reclassified into the media plane (per-region stateless signer, no DB dependency); revocation/kill semantics are explicitly enforced by manifest purge and ingest teardown, never by token expiry.

Q5 (Codex): What is the complete deletion protocol across regional replicas, object versions, cold-storage archives, CDN caches, asynchronously generated renditions, backups, chat replay, Kafka, and ClickHouse, and how do you prove completion when tombstones or purge jobs fail partway through?

The doc gave the shape (tombstone → purge → hard delete) but not the machinery. Deletion runs as a durable workflow (Temporal-style) with one idempotent, independently verified step per sink, and the broadcast_id-prefix key scheme is what makes verification tractable — every media artifact for a broadcast lives under /{broadcast_id}/..., including backfilled renditions and the chat replay log. The checklist: metadata tombstone (synchronous, hides content from every API and search — this is the user-visible "deleted," in seconds); CDN purge of manifests plus a hard cap of 24h on any segment TTL so unreferenced segments age out even where purge-by-prefix isn't supported; object-store prefix delete in the home region and every async replica, deleting all versions where buckets are versioned; cold-tier delete via the archive API; ClickHouse delete-by-broadcast_id via async mutation; Kafka needs no per-record delete — retention is bounded at days, so it ages out inside the erasure window. Media has no separate backup system to chase: durability comes from erasure coding plus the cross-region replicas, and the replicas are in the checklist. Database backups contain metadata only (titles, IDs), which the tombstone covers for access and normal backup expiry covers for erasure.

Proving completion: each step's verifier is a read-after-delete (LIST on the prefix returns empty, archive inventory shows absence, mutation reported applied), and the workflow marks the deletion complete only when every verifier passes, emitting an auditable deletion record with timestamps. Steps retry indefinitely with escalation to a human on stall. And because workflows can be wrong or sinks can regress, a reconciliation sweep runs periodically: scan storage inventories (S3 Inventory-class listings) for prefixes whose broadcast_id carries a deleted tombstone, and re-fire the workflow for anything found. Target bound: reachability gone in minutes, physical erasure verified within 30 days — which is the GDPR commitment we'd publish.

Design change: deletion becomes a durable workflow with per-sink verification and an auditable completion record, a 24h max segment TTL at the CDN, and a periodic inventory-based reconciliation sweep keyed on tombstoned broadcast_ids.

Q6 (Codex): HyperLogLog cannot remove expired viewers, so how does your 30-second-heartbeat pipeline compute current concurrency within a few percent and watched minutes within 1% while handling missing heartbeats, duplicate retries, delayed events, reconnects, and clients that disappear without a leave event?

Right that a single ever-growing HLL can't do concurrency — that's why the sketches are windowed, which I should have said. Flink keeps one HLL per broadcast per 30s bucket, keyed by session_id, and "current concurrency" is the union of the last two buckets (union is the operation HLLs are actually good at). A viewer who disappears without a leave event simply stops appearing in new buckets and falls out of the number within ≤60s; duplicate beacons from client retries are free because HLL insertion is idempotent; one dropped heartbeat is absorbed by the two-bucket window. The displayed count lags reality by up to a minute and carries HLL's ~1–2% error — fine for "1.2M watching," and the promotion trigger doesn't wait on this pipeline anyway (it keys off join-event rate, which is instant).

Watched minutes need dedup, not distinct-count, so they take a different path: each heartbeat carries (session_id, seq), Flink dedups by that pair in keyed state with a short TTL (25M active sessions × a few tens of bytes is a couple of GB, sharded — unremarkable), and each accepted heartbeat credits 30s to the session. Missing heartbeats undercount by exactly the loss rate, duplicates are removed exactly, and late events inside the watermark's allowed lateness (a few minutes) land in the right window; later than that, they're dropped and eat into the same ~1% budget as beacon loss. Reconnects are just a new session_id whose minutes sum with the old one under the same viewer — correct by construction for watched minutes, and for concurrency the brief double-count of one viewer across two session_ids during a reconnect is noise at these magnitudes.

Design change: none in mechanism, but the doc's one-line HLL mention is upgraded to the real spec: windowed per-bucket HLLs unioned for concurrency, and (session_id, seq)-deduped heartbeat credits for watched minutes.

Round 2 — pushback

Q7 (Codex): Your "first writer wins" PUT is not fencing: after an epoch change, a zombie encoder can still win new segment keys. What epoch-aware commit protocol prevents stale output from becoming canonical and guarantees correct init-segment/discontinuity handling during overlapping failover?

Conceded — first-writer-wins protects existing keys, but a zombie that's ahead of the new writer races it for future seq keys and can win them, and nothing in my Q2 answer stops that. The fix is to stop treating an object's existence as canonicality. Canonical = referenced by the manifest chain, and the manifest pointer is the only thing that's fenced. Concretely: segments are written under an epoch-qualified prefix, /{broadcast_id}/{epoch}/{rendition}/{seq}.m4s, and each writer only ever writes inside its own epoch. The current epoch is a compare-and-swap'd record in a store that actually supports CAS (etcd / a conditional-write row), owned by the lease-holding packager; failover increments it atomically. A zombie keeps writing into its stale epoch prefix, wins every PUT it attempts, and it doesn't matter — nothing references that prefix, and GC deletes it. No epoch check needs to exist inside S3, which is good, because S3 can't enforce one.

This also cleans up the init-segment story. An epoch boundary is exactly where encoder state resets, so each epoch gets its own init segment, and the manifest marks the transition with EXT-X-DISCONTINUITY plus a new EXT-X-MAP pointing at the new epoch's init — which is the standard HLS mechanism for a mid-stream encoder change, so compliant players handle it natively. Players never construct segment URLs themselves; they follow the manifest, so the epoch in the path costs nothing in cacheability (each URL is still one shared cache key). The final VOD manifest stitches epochs the same way: per-epoch segment runs joined by discontinuity tags, with EXT-X-GAP where source outran the PoP buffer. The residual race — two packagers both believing they hold the lease during a partition — is closed by the CAS on the epoch record: only one of them can have performed the increment, and manifest publication requires writing through that same conditional record, so the loser's manifest update fails atomically rather than tearing.

Design change: segment keys become epoch-prefixed; canonicality is defined by manifest reference, with the current epoch held in a CAS store written only by the lease-holding packager; zombie output lands in unreferenced prefixes and is garbage-collected; epoch boundaries carry EXT-X-DISCONTINUITY + new EXT-X-MAP in both live and VOD manifests. First-writer-wins PUTs are demoted from fencing mechanism to belt-and-suspenders.

Q8 (Codex): If every token service is down, what actually signs your "static fallback" tokens—and what edge-enforced mechanism lets legitimate viewers refresh while immediately blocking a deleted, banned, or private stream whose manifest and predictable segment URLs were already fetched?

Fair on the first half: "static fallback issues tokens with degraded checks" was sloppy, because if every signer is down there is nothing to sign with. The honest mechanism is a break-glass grace mode at the edge, not a phantom signer: edge validation logic (CloudFront Functions / Fastly compute, which already validate signatures) reads a config flag we can flip — operator-initiated or automated on signer health — that tells edges to accept tokens expired by up to N minutes (say 30). Viewers already watching hold real, once-valid tokens; grace mode keeps them watching without anyone signing anything new. Viewers with no token at all still can't join — that's the residual failure, and I accept it: an all-region simultaneous signer outage for a stateless service with no dependencies is a correlated-change event (bad config push, bad key distribution), defended primarily by staged rollout of exactly those two things. Grace mode bounds the blast radius; it doesn't pretend the outage away.

The second half exposes a real gap I hadn't closed: my segment URLs are predictable (/{broadcast_id}/{epoch}/{rendition}/{seq}.m4s), so a client that has seen the pattern can keep fetching future seqs with a still-valid (or grace-mode) token, even after we purge the manifest. Manifest purge alone doesn't block it. Two mechanisms close it. For a banned live stream, the kill switch tears down ingest and the packager, so future segments are never produced — a 404 is the strongest possible enforcement and needs no edge logic. For deleted or private recorded content, where the segments exist, the edge gets a denylist: on tombstone or ban, the broadcast_id is pushed to an edge KV store (CloudFront KeyValueStore, Fastly KV — built for exactly this), and the edge auth function, which already parses the URL to validate the token, rejects any path whose broadcast_id is listed — cache hit or not. Propagation is seconds; the list stays small because entries only need to live until hard-delete finishes and the 24h segment TTL expires. Private streams also get the cheaper static check: tokens are scoped to a broadcast_id claim, and the edge matches claim to path, so a token for stream A never fetches stream B — and grace mode doesn't weaken either check, since an expired-but-accepted token still carries its scope and the denylist still applies.

Design change: the token-service failure story is restated as edge grace mode (bounded acceptance of expired tokens, flag-flipped) with no fallback signer claimed; playback tokens gain a broadcast_id scope claim matched against the URL path at the edge; and a tombstone/ban-fed edge KV denylist keyed on broadcast_id closes the predictable-URL gap that manifest purge leaves open.

What changed, summarized

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The core bet of this design — cacheable HTTP objects plus request coalescing, so viewer one million costs the origin nothing — is exactly how Cloudflare built Stream Live. Their behind-the-scenes post describes anycast RTMPS/SRT ingest (same shape as my ingest PoPs, including the awkward problem of a broadcaster reconnecting to a different server than the one holding their state — they solve it with volatile/committed state in Durable Objects where I used the PoP disk buffer plus PTS-derived seq numbers) and request coalescing so that any number of viewers triggers one encode and one fetch. Their LL-HLS post moves the pipeline from keyframe-interval blocks to individual frames and lands under 10 seconds glass-to-glass — and their beta requires a 2-second keyframe interval, the same GOP choice this design makes. The protocol mechanics the Q1 answer leans on (blocking playlist requests, preload hints, and why HTTP/2 PUSH was dropped from the spec) are laid out in Mux's Low Latency HLS 2: Judgment Day, which is also honest about what LL-HLS demands from a CDN — supporting my "collapse-and-hold is a contractual acceptance criterion" conclusion.

On transcoding, Twitch's FFmpeg vs TwitchTranscoder post is the published version of the Q3 alignment problem: they rebuilt their transcoder partly because independent FFmpeg instances produce misaligned IDR frames across renditions, which breaks ABR switching — the exact reason this design derives seq from source PTS and requires IDR-aligned GOPs across the ladder. One divergence worth owning: Twitch's stack is CPU software encoding (C/C++/Go, shared decode across variants), and Netflix's live pipeline also chose software encoding for flexibility, so my GPU-by-default call is a bet on 2026 hardware economics at 50k concurrent channels, not settled industry consensus. The lazy-ladder idea has an even more aggressive published cousin: Cloudflare encodes a rendition only when a viewer actually requests that quality level — demand-driven rather than threshold-promoted.

Netflix's live buildout validates the recording-as-byproduct and origin-design instincts from a different angle. As covered in InfoQ's summary of their Behind the Streams series (the primary posts are on the Medium-hosted Netflix TechBlog, which blocks automated fetching — search "Behind the Streams" there), they built a custom live origin that prepares segments, manages encryption, and generates manifests while absorbing millions of concurrent reads, runs redundant pipelines across regions, and delivers through their own Open Connect CDN. And when it went wrong, it went wrong the way this design predicts: Kentik's traffic analysis of the Tyson–Paul fight shows ~70% of delivery on embedded caches, transit at 3.3%, and buffering concentrated where saturated single-provider paths left no alternatives — the strongest public evidence I found for buying multi-CDN on day one and treating own-the-edge as the 10× move, not the day-one move. The menu of steering mechanisms the design's steering service would pick from (DNS, manifest rewriting, server-side content steering, client-side switching) is cataloged in the SVTA's multi-CDN investigation.

The flash-crowd numbers in this design are not hypothetical — cricket has run them. The Pragmatic Engineer's interview with JioCinema's chief architect on the 32M-concurrent IPL 2023 final confirms two of my calls and corrects my emphasis on a third: reactive autoscaling is too slow for live spikes (they scale on concurrency as the metric, which matches my join-rate promotion trigger), and CDN capacity is negotiated as a procurement problem. What they add that my design under-weights: capacity planning started a year before the event, with pre-warmed fleets and game-day load simulations. For scheduled megastreams, "elastic" is marketing; you pre-provision. Notably they ran 4–6 second segments — accepting more latency than my 3–5s target in exchange for calmer CDN behavior, a knob this design would also turn for its biggest events.

Chat: Twitch's engineering overview describes the same shape as my broker → relay → gateway tree — an Edge tier speaking IRC over raw TCP and WebSockets, with an internal Pubsub tier forming "a hierarchical message distribution system which executes massive fanout," delivering billions of messages a day. The design's accept-side throttling (slow mode, per-user cooldowns) is lifted from what Twitch ships as product features, which is the tell that the megaroom problem gets solved at the acceptance edge, not the delivery edge.

Updates from post-training information

One real update. On March 5, 2026, the India–England T20 World Cup semi-final on JioHotstar peaked at 65.2 million concurrent viewers — a new world record, per the ICC's release, with 619M total views and 23B+ watch minutes. Two consequences for this design. First, the prompt's 25M concurrent peak has already been exceeded 2.6× by a single event on a single platform, so the "Getting to 10×" section is closer to a two-year roadmap than a hedge — I'd treat ~65M single-event concurrency as the validation target, not the growth ceiling. Second, it sharpens the pre-warming lesson above: records like this happen on scheduled events, which means the flash-crowd design must be paired with an event calendar and pre-provisioned transcode/CDN capacity, something the original design mentions only implicitly via the promotion path. (A widely shared Medium post claims the final hit "821 million concurrent" — that conflates total views with concurrency; the ICC's own numbers are the ones above.)

Further reading