Slack-Like Team Messaging: MVP in Two Weeks, Then to 10B Messages/Day

Recommendation up front

Ship the MVP as a single Elixir/Phoenix monolith on Postgres, with WebSockets terminated in the app itself — no separate realtime infrastructure at all. That gets channels, DMs, history, live delivery, and presence in two weeks with a team of three or four. The evolution path then extracts exactly three things as load demands it: a stateful WebSocket gateway tier, a channel-server tier that sequences and fans out messages (consistent-hashed by channel ID), and a wide-column message store (ScyllaDB) partitioned by channel. Metadata stays in relational storage, sharded by workspace. Everything else — search, push, archival — hangs off a change stream on a durable, partitioned event log (Kafka) and never touches the hot path.

The one principle that survives from day one to target scale: the channel is the unit of everything — ordering, storage partitioning, fan-out routing, and caching. Get that boundary right in week one and the rewrites later are extractions, not redesigns.

Assumptions

Stating these rather than asking, since I can't:

Text-first product. Files are a URL-upload to object storage (S3 presigned PUTs) even in the MVP, but no previews, no image processing. No voice/video.
Clients: web app first; desktop is the web app wrapped in Electron; native mobile is post-MVP (which also defers push notifications).
Median message is small — assume ~100 bytes of text, ~1 KB stored with metadata and indexes.
A message's audience is its channel. Average channel is small (most channels have < 50 members); assume ~10 online recipients per message on average, which is the number that drives fan-out.
"Reach online recipients within 500 ms" means p99 server-side, sender's gateway to recipient's gateway, same or nearby region.
Delivery semantics: at-least-once with client-side dedup. Exactly-once on the wire isn't worth the cost; exactly-once appearance in the UI is achievable and is what users perceive.
Compliance (SSO, DLP, e-discovery) is out of scope for both MVP and this design, though the retention hook is in the schema from day one because the prompt requires workspace deletion policies.

The primary user experience

Three panes. This layout is load-bearing for the architecture, not just the UI: the left sidebar defines what the client must know at boot (channel list, unread badges, DM presence), and the message pane defines the two read patterns (tail of one channel live, plus paginated history upward).

+----------------+---------------------------------------------+
| acme-corp   v  |  # backend                        3 members |
|----------------|---------------------------------------------|
| CHANNELS       |  ana  10:02  deploy is out                  |
|  # general     |  bo   10:03  seeing 500s on /billing        |
|  # backend  3  |  ana  10:04  rolling back now               |
|  # design      |   ...                                       |
|                |  (scroll up = older history, paginated)     |
| DIRECT MSGS    |---------------------------------------------|
|  o ana         |  [ Message #backend...            ] [Send]  |
|  * bo (away)   |                                             |
+----------------+---------------------------------------------+
   o = online     "# backend 3" = 3 unread

MVP feature list — what's in and what I deliberately cut:

In: email/password auth, workspace create/join by invite link, public and private channels, DMs (modeled as channels), send/edit/delete messages, infinite-scroll history, live delivery, typing indicator, online/away presence, per-channel unread badges, S3 file links.

Out, and why: threads (doubles the read-path complexity for a feature you can fake with replies), reactions (cheap but not essential; first fast-follow), search (Postgres tsvector is a weekend add later; not needed to demo the core loop), mobile apps and push (a platform each), message formatting beyond basic markdown. Cutting threads is the call I'd defend hardest: every messaging MVP that includes them ships late.

Requirements and the numbers that shape the design

Functional requirements are the feature list above. Non-functional, in priority order: (1) no acknowledged message is ever lost, (2) per-channel order is consistent for all viewers, (3) p99 online delivery < 500 ms, (4) history reads stay fast regardless of channel age.

Back-of-envelope at target scale (all estimates, not measurements):

Quantity	Math	Result
Message writes, average	10B / 86,400 s	~116 K msg/s
Message writes, peak	× 10	~1.16 M msg/s
Delivery events, peak	1.16 M × ~10 online recipients	~12 M events/s across the gateway fleet
Concurrent WebSockets	given	5 M
Gateway fleet	64 K conns/node target, 2× headroom	~150 nodes
Raw message storage	10B × 1 KB/day	~10 TB/day, ~3.7 PB/yr pre-replication
On disk (RF=3, ~3× compression)		~3.7 PB/yr
History fetches	100 M DAU × ~30 channel opens × 1 page	~35 K reads/s avg, each ~50 rows

Three conclusions fall out of this table. First, 1.16 M writes/s with indefinite retention rules out a relational store for message bodies at target scale — this is an LSM-tree workload. Second, 12 M deliveries/s rules out anything that does per-recipient work on the write path; fan-out must be to subscriptions (connected clients watching a channel), not to members. Third, multiple petabytes a year means storage tiering is a requirement, not an optimization.

The two-week MVP

One Phoenix (Elixir) application, one Postgres, one S3 bucket. Deployed as 2 app instances behind a load balancer so a deploy isn't an outage.

Why Elixir/Phoenix specifically: WebSocket fan-out and presence are the hard 20% of this product, and Phoenix ships both — Channels gives per-topic pub/sub across nodes, and Phoenix Presence is a CRDT-based presence tracker that survives node failures without a central store. That's one to two weeks of infrastructure work the framework already did. What I rejected for the MVP:

Rails or Django + a WebSocket sidecar (AnyCable, Channels): the sidecar is a second moving part and the presence story is DIY. If the team is deep in Ruby/Python and has never seen Elixir, this beats fighting an unfamiliar stack — team familiarity legitimately outranks my default here.
Node + Socket.IO + Redis adapter: perfectly workable, second choice. You hand-roll presence and reconnect semantics that Phoenix gives you.
Firebase/Firestore as the whole backend: fastest week one, then it fights you on history pagination, per-workspace authz, and export, and the per-read pricing at anything like this scale is a trap. Rejected for lock-in, not capability.
Postgres over anything NoSQL for the MVP: not a close call. One database, transactions across users/channels/messages, LISTEN/NOTIFY as a backstop. A single well-provisioned Postgres handles thousands of messages/s and a few years of a successful-but-normal startup's data.

MVP schema (Postgres)

workspaces (id, name, retention_days NULL, created_at)
users      (id, email UNIQUE, password_hash, display_name, created_at)
workspace_members (workspace_id, user_id, role, PRIMARY KEY (workspace_id, user_id))

channels   (id, workspace_id, name, kind,            -- 'public'|'private'|'dm'
            created_by, created_at,
            UNIQUE (workspace_id, name) WHERE kind <> 'dm')

channel_members (channel_id, user_id,
                 last_read_msg_id BIGINT DEFAULT 0,   -- unread = latest > this
                 PRIMARY KEY (channel_id, user_id))

messages   (channel_id BIGINT,
            id BIGINT,                                -- per-channel monotonic
            sender_id, body TEXT, client_msg_id UUID,
            created_at, edited_at, deleted_at,
            PRIMARY KEY (channel_id, id),
            UNIQUE (channel_id, client_msg_id))       -- retry dedup

Two decisions here are the seeds of the target design. messages is keyed (channel_id, id) with a per-channel monotonic id — in the MVP it's assigned inside the insert transaction; later a channel server assigns it, but the contract ("ids in a channel are dense-ish and ordered; sync = give me everything after id X") never changes, so clients never change. And unreads are a cursor, not a counter: last_read_msg_id per member, badge computed as latest_msg_id > last_read_msg_id. Counters break at scale (more below); cursors don't.

DMs are channels of kind dm with two members. One message path, one history path, one delivery path for everything — this is the single biggest simplification in the design.

MVP API

REST for anything request/response, WebSocket for events. Sends go over REST, not the socket: you get retries, idempotency, and load-balancing for free, and the socket stays a one-way event stream (plus typing/read-cursor pings, where loss is acceptable).

POST /v1/auth/login | /v1/workspaces | /v1/workspaces/:id/invites
GET  /v1/workspaces/:id/boot                 -- channels, members, unread cursors
POST /v1/channels                            -- also creates DMs (kind=dm)
POST /v1/channels/:id/messages   {client_msg_id, body}   -- idempotent
GET  /v1/channels/:id/messages   ?before_id=&limit=50    -- history page
PATCH/DELETE /v1/channels/:id/messages/:msg_id
POST /v1/channels/:id/read       {last_read_msg_id}
POST /v1/files/presign

WS   /v1/socket?token=...
  server→client: message.new|edited|deleted, channel.created,
                 presence.diff, typing, read.updated
  client→server: subscribe {channel_ids}, typing, ping

Send path in the MVP: POST → auth + membership check → transaction (bump channel's latest_msg_id, insert message) → ack sender → broadcast on Phoenix Channel topic channel:{id} → subscribed sockets on any node deliver. On reconnect the client sends its highest seen id per channel and gets the gap from the history endpoint. That's at-least-once with dedup on client_msg_id — the same semantics we'll keep at 10B/day.

This MVP comfortably serves ~5–10 K concurrent users and tens of messages/s on two app nodes and one database. Two weeks is realistic because the framework carries realtime and presence, and everything else is CRUD.

Target architecture

flowchart LR
  subgraph Clients
    C1["Web / Desktop"]
    C2["Mobile"]
  end

  subgraph Edge["Edge (many regions)"]
    LB["L4 load balancer"]
    GW["WebSocket gateway fleet<br/>~150 nodes, 64K conns each<br/>stateless-ish: holds conns + subscriptions"]
    BOOT["Boot cache<br/>workspace snapshot for reconnect storms"]
  end

  subgraph Cell["Workspace home region (cell)"]
    API["API service (REST: sends, history, CRUD)"]
    CS["Channel servers<br/>consistent-hash by channel_id<br/>sequence, persist, fan out"]
    PRES["Presence service<br/>Redis cluster, TTL'd keys"]
    META[("Metadata DB<br/>Postgres sharded by workspace<br/>(Vitess-style)")]
    MSGDB[("ScyllaDB<br/>messages, hot ~90 days<br/>PK (channel_id, bucket)")]
    RC[("Redis<br/>recent-tail cache per channel")]
    K["Kafka<br/>message change stream"]
  end

  subgraph Async["Off the hot path"]
    SRCH["Indexer → OpenSearch"]
    PUSH["Push service → APNs / FCM"]
    ARCH["Archiver → S3 (Parquet, cold tier)"]
    RET["Retention reaper (workspace policies)"]
  end

  C1 --> LB
  C2 --> LB
  LB --> GW
  GW --> BOOT
  GW -->|"subscribe channel:{id}"| CS
  GW --> PRES
  C1 -->|HTTPS| API
  C2 -->|HTTPS| API
  API --> META
  API --> RC
  API --> MSGDB
  API -->|"send → route by channel_id"| CS
  CS --> MSGDB
  CS --> RC
  CS --> K
  K --> SRCH
  K --> PUSH
  K --> ARCH
  K --> RET

The send path, end to end

Client POSTs the message with a client_msg_id. API authenticates, checks channel membership against the metadata shard (cached), and forwards to the channel server that owns hash(channel_id) on a consistent-hash ring.
The channel server is the single writer for that channel. It assigns the next per-channel sequence id, writes to ScyllaDB at quorum (LOCAL_QUORUM, RF=3), appends to the channel's in-memory tail cache (Redis), and acks. Durable-then-ack — the "never lose an acked message" requirement is enforced at exactly one point in the system.
In parallel with the ack, it pushes the event to every gateway that holds a subscription for that channel (gateways subscribe once per channel per gateway, not per client — a channel with 3,000 viewers on 150 gateways costs the channel server at most 150 pushes, and each gateway fans out locally over its own sockets).
It emits the message to Kafka, where search indexing, mobile push for offline members, archival, and analytics consume it. None of these can add latency to step 3.

Latency budget for the 500 ms p99: client→API ~50 ms, route+sequence ~5 ms, Scylla quorum write ~10–20 ms p99, channel server→gateway ~5 ms, gateway→client ~50 ms, plus queueing slack. Comfortably inside budget with room for a cross-region hop from an edge gateway to the workspace's home cell.

Why a channel-server tier instead of off-the-shelf pub/sub — the two obvious candidates fail for stated reasons:

Kafka on the hot path: millions of active channels don't map to topics/partitions; per-channel ordering with shared partitions means head-of-line blocking across channels, and consumer-group rebalances blow the tail latency budget. Kafka is exactly right off the hot path, so that's where it lives.
Redis cluster pub/sub: this is the correct intermediate step (stage 2 below) and carries the product a long way. It falls down at target scale on delivery guarantees (fire-and-forget — a blip silently drops messages, forcing aggressive client resync) and on giving us a place to put per-channel logic (sequencing, dedup, large-channel throttling). The channel server is where that logic wants to live anyway, so we build the tier. This is essentially the architecture Slack has described publicly (channel servers consistent-hashed by channel, edge gateways in front), which I'll take as evidence the shape survives contact with reality.

Storage: two databases on purpose

Messages → ScyllaDB. Partition key (channel_id, bucket) where a bucket is ~10 days or ~50 K messages (whichever first — quiet channels roll by time, firehose channels roll by count, so no partition goes multi-GB). Clustering key id DESC, so "latest page" and "everything after id X" are single-partition, sequential reads. 1.16 M writes/s peak is bread-and-butter for an LSM store at maybe 150–200 nodes; Discord runs this exact shape (they've written publicly about trillions of messages on Cassandra, then ScyllaDB — the migration was driven by Cassandra's JVM GC pauses on hot partitions, which is also why I pick Scylla over Cassandra). Rejected: sharded Postgres for messages (B-tree write amplification and vacuum at this write rate; painful resharding of an ever-growing table) and DynamoDB (a managed wide-column/KV store — fits technically, but at on-demand pricing 10B 1 KB writes/day is roughly $12–13 K/day for writes alone before storage — an estimate from list prices, not a benchmark — and at that spend self-managed wins).

Everything else → Postgres, sharded by workspace. Users, workspaces, channels, memberships are relational, transactional, and comparatively small. Workspace is a natural shard key because almost every query is workspace-scoped, and it makes tenant isolation and workspace-level deletion tractable. Vitess-over-MySQL is the battle-tested version of this (Slack's own metadata path); Citus gets Postgres there too. A directory service maps workspace_id → shard, which later doubles as the cell-routing map.

Tiering, because retention is indefinite: Scylla holds the hot ~90 days. An archiver compacts older buckets into compressed columnar objects in S3 (with a per-channel manifest so a bucket is one GET). History pagination past the hot window reads S3 — slower (hundreds of ms), and that's fine: scrolling to last year is rare and visibly "deep history." Cuts steady-state cluster size by an order of magnitude versus keeping petabytes in Scylla. The retention reaper enforces workspace policies by deleting Scylla partitions and S3 objects, and writes a tombstone audit record.

Caches: Redis holds each active channel's last ~100 messages (serves most history opens without touching Scylla) and presence. Metadata reads (membership checks on every send) are cached in-process in the API tier with short TTLs and event-driven invalidation off Kafka.

Presence at 5 M connections

Naive presence melts before anything else does: 5 M heartbeats × broadcast-to-everyone is quadratic. Three rules keep it boring:

Gateways own liveness. A client's gateway marks it online in Redis with a 60 s TTL and refreshes in batches per gateway (one pipelined write per node per interval, not 5 M individual writes).
Presence is pull + subscribe, not broadcast: a client asks for presence only for users currently on its screen (sidebar DMs, visible member list) and subscribes to diffs for that set only. Nobody ever receives presence for a whole workspace.
Presence is allowed to be seconds stale and lossy. It's the one part of the system where dropping data is a feature — never let it share a queue with messages.

The hard part: the 1 M-member workspace

Everything that iterates members breaks at 1 M. Each fix is the same move — replace per-member work with per-subscription or lazy work:

Fan-out already doesn't iterate members: a message to a 1 M-member channel costs one push per gateway with a subscriber (≤ 150), and each gateway fans out to its own connected viewers. Cost scales with people actually looking, not membership.
Unread counters: eagerly incrementing 1 M rows per message would be ~10^12 counter writes/day for one chatty channel. The cursor model makes unreads O(1) per user per read instead: badge = latest_msg_id > last_read_msg_id, both already at hand. Numeric counts show "99+" beyond a page. Mentions are the exception — an @mention writes an explicit per-user mention row (bounded, small) so it can badge and push reliably even in huge channels.
Boot: the client never downloads the member list. Boot payload = your channels + cursors + your DM roster; member lists are server-side searched and paged. This caps boot at KBs instead of tens of MBs, which matters most during reconnect storms.
@channel in #all-hands: an explicit product limit (confirm dialog, rate limit, push fan-out via Kafka consumer over minutes, not milliseconds). Some problems are best solved by not doing the thing.

Ordering and delivery semantics, precisely

Order within a channel is the sequence the channel server assigned — single writer per channel, so it's total and cheap. No cross-channel ordering promises (nobody needs them).
At-least-once from server to client: gateways push, clients ack cumulatively per channel; on reconnect the client presents its cursor and fetches the gap over REST. Duplicates are dropped by id; sender retries are dropped by client_msg_id. Result: exactly-once appearance.
A channel-server crash between Scylla write and gateway push is the interesting window: the message is durable but possibly undelivered. Recovery is client-driven — the new owner of the channel announces itself, gateways resubscribe, clients gap-fetch. Correct when degraded, fast when healthy.

Failure modes

Gateway node dies (33 K clients): clients reconnect elsewhere with jittered exponential backoff and a resume cursor. The dangerous part isn't the sockets, it's the reconnect storm hitting boot: hence the edge boot cache serving workspace snapshots without touching the metadata DB (same problem Slack built Flannel for). Gateways drain on deploy; we deploy this tier constantly, so this path is exercised daily, not just in disasters.
Channel server dies: its ring segment reassigns in seconds (ring membership in a strongly consistent coordination store — etcd/ZooKeeper); gateways resubscribe; per-channel sequence resumes from Scylla's max id. Brief delivery pause on affected channels, no loss.
Scylla degraded: quorum writes ride out one replica per range; a whole-cluster incident means sends fail fast with client retry — visible sending pause, still no acked-message loss. Reads fall back tail-cache-first.
Kafka down: messages still deliver in real time (hot path doesn't touch it); search/push/archival lag and catch up from the stream. This decoupling is most of why the async tier exists.
Region loss: workspaces are homed to cells; each cell's stores replicate async to a standby region. Failover per cell: flip the directory mapping, accept seconds of RPO on the async replica, gateways (multi-region already) re-route. Rejected active-active multi-master for message writes — reconciling divergent per-channel sequences across regions buys enormous complexity for a failure mode that's better handled by fast failover. Cross-region latency is instead handled by edge gateways + boot caches near users, backhauling to the home cell.
Overload/abuse: per-user and per-workspace token-bucket rate limits at the API; channel servers shed typing indicators and presence first, messages last.

Security

TLS everywhere; WebSocket auth via short-lived signed tokens (~10 min) minted from refresh-token sessions, so a leaked socket URL goes stale fast.
Authorization is workspace- and channel-membership-checked on the server for every read and send — the cached membership set makes this cheap, and no client-supplied workspace ID is ever trusted. Sharding by workspace adds a structural backstop: a bug in one query can't easily cross tenants that don't share a shard.
Encryption at rest on all stores; S3 file access only via expiring presigned URLs scoped to workspace membership.
Workspace deletion and retention policies actually delete (Scylla partitions, S3 objects, search index docs), with an audit trail. Passwords argon2; invite links single-use with expiry; audit log of admin actions from the MVP onward — it's one Kafka consumer and saves you retrofitting it for the first enterprise deal.

Evolution: MVP → target, in stages

Each stage is triggered by a measurement, not a date. Boundaries are rough thresholds where the previous stage's bottleneck bites.

Weeks 0–2 — the monolith (to ~10 K concurrent, tens of msg/s). Phoenix + Postgres + S3, 2 app nodes. Ship it.
Extract the socket tier (~10 K → 200 K concurrent). Move WebSocket termination to dedicated gateway nodes (same Elixir code, separate deploy) so app deploys stop dropping sockets; Redis cluster pub/sub between API and gateways; Postgres read replicas; Redis tail cache. The monolith's code barely changes — the seams were in the message flow from day one.
Split the stores (~200 K → 1 M concurrent, ~5–20 K msg/s). Messages move to ScyllaDB (dual-write, backfill, cut reads over — the (channel_id, id) contract makes this invisible to clients); metadata Postgres shards by workspace behind the directory service; a full-text search index (OpenSearch) fed via Kafka CDC; native mobile + push land here.
Channel servers and cells (toward 5 M concurrent, 100 K+ msg/s). Replace Redis pub/sub with the channel-server ring; add edge gateways + boot cache in multiple regions; home workspaces to cells; storage tiering to S3; the giant-workspace work (lazy rosters, mention rows, @channel limits).

The stage-2/3 rewrites are the ones teams usually botch by doing them too early. Redis pub/sub plus one Scylla cluster serves a product doing hundreds of millions of messages a day; build the channel-server tier when fan-out tail latency or Redis's delivery guarantees actually hurt, and not before.

Tradeoffs I'd flag in the last five minutes

Two databases is real operational cost. One team now runs Postgres and Scylla. The alternative — everything in one store — either caps message scale (all-relational) or forces awkward relational modeling in a KV store (all-NoSQL). The split follows the workload split, and I'd pay it, but not before stage 3.
Sends over REST, not the socket costs a little latency (HTTP overhead vs. an open socket) and buys idempotent retries, sane load-balancing, and a socket protocol that stays trivially simple. For a 500 ms budget that trade is easy; a product chasing 100 ms would revisit it.
Single-writer-per-channel is a deliberate serialization point. It's what makes ordering, dedup, and sequence assignment simple, and its ceiling — one channel's throughput bounded by one core — is far above what a human channel produces. The pathological case is a bot flooding one channel; rate-limit it rather than distribute the writer.
Cursors over counters for unreads gives up cheap exact counts in huge channels ("99+"). Nobody has ever cared.
Home-region cells over active-active gives up write availability during a regional failover window (seconds to minutes, per cell) to avoid multi-master reconciliation forever. For team messaging, a brief visible "reconnecting…" beats ever showing two users different channel orders.

What I'd build next after target scale: threads and reactions (the data model extends: threads are a parent_msg_id, reactions are a small per-message set — both fan out on the existing path), and the enterprise wedge (SSO/SCIM, e-discovery export off the Kafka archive), because that's where this product makes money.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): During channel-server ring changes or network partitions, how do you prevent the old and new owners from concurrently assigning the same sequence IDs? Specify the lease/fencing mechanism, where its authority lives, and what happens to writes when that authority is unreachable.

The design as written has a hole here — "ring membership in etcd" names a component but not a fencing mechanism, and consistent hashing alone permits two owners during a partition. The fix is epoch-fenced sequence ids. A channel server acquires ownership of a ring segment by taking an etcd lease and getting a monotonically increasing epoch number for that segment (etcd's lease plus a fenced counter). The per-channel sequence id becomes epoch-in-high-bits, counter-in-low-bits (say 16/48). Now a deposed owner and a new owner can never assign the same id, and ordering stays monotonic across ownership changes because a newer epoch always sorts higher. The deposed owner can still commit a few writes under its old epoch during the lease-TTL window — those land with old-epoch ids, which sort before the new owner's, so the total order stays consistent for all readers; worst case is a message appearing "slightly in the past" for a sub-second window, which I'll take.

When etcd is unreachable, a server keeps serving under its unexpired lease; when the lease expires and can't be renewed, it stops accepting writes and sends fail fast with client retry — a CP choice, consistent with "never lose or duplicate an ack." Residual: client_msg_id dedup is normally enforced in the single writer's memory; across an epoch boundary the new owner warms by reading the channel tail from Scylla (last few minutes of client_msg_ids) before serving, and clients also dedup on client_msg_id in the UI as the last line.

Design change: sequence ids gain an epoch prefix allocated per ring segment from etcd (lease + fenced counter); channel servers stop accepting writes when their lease expires unrenewed.

Q2 (Codex): You require that no acknowledged message is ever lost, yet regional failover uses asynchronous replication with "seconds of RPO." If the home region dies immediately after a local-quorum acknowledgment, do you lose the message or refuse failover until it is recovered, and how do you preserve the channel sequence either way?

You've caught the two claims contradicting each other, and I won't pretend they don't. As written, "no acked message is ever lost" holds for every failure except permanent destruction of a region, where async replication means seconds of acked writes can vanish. I'll revise the guarantee rather than the architecture: the durability SLA becomes "no acked message lost, except on unrecoverable regional destruction, where RPO ≤ ~5 s" — and most "region loss" is transient, in which case the old region's Scylla still has the tail and we reconcile on recovery: divergent tail messages are re-sequenced under a new epoch and re-delivered late rather than dropped.

Two mitigations shrink the real exposure. Senders whose ack was in flight retry against the new region (the client outbox holds a message until acked), so the lost set is only acked-but-unreplicated writes. And per-channel sequence continuity in the new region starts from the replica's max id under a fresh epoch, so ordering never forks — some ids are simply absent, same as a retention delete. For workspaces that genuinely can't accept that carve-out, offer synchronous cross-region quorum as a per-workspace (enterprise) option — the ~30–60 ms cross-region write penalty fits inside the 500 ms budget; I don't want to pay it for all 10 M workspaces, but it's a per-cell config, not a redesign. Going fully sync for everyone, or multi-master, buys reconciliation complexity forever to cover minutes-per-decade events.

Design change: the durability guarantee is restated with an explicit carve-out (RPO ≤ ~5 s on unrecoverable regional destruction); transient region loss reconciles by re-sequencing the divergent tail under a new epoch; synchronous cross-region replication becomes a per-workspace opt-in.

Q3 (Codex): A message is committed to Scylla before a separate Kafka publish, so a crash between those operations can permanently omit it from archival, push, search, and retention processing. What durable outbox, CDC, or reconciliation mechanism closes that gap while preserving per-channel order and deduplicating replays?

Real gap — a crash between the Scylla write and the Kafka publish silently drops the message from search, push, archival, retention. The fix uses Scylla itself as the outbox, since the message is already durably there in per-channel sequence order: each channel server persists, per channel (batched, every ~1 s or N messages), a "published-to-Kafka watermark" — the highest sequence id confirmed by Kafka. On ownership change (crash or rebalance), the new owner reads the watermark, reads Scylla for everything above it, and republishes in id order before serving new sends' Kafka emissions.

Kafka partitioning is by channel_id, so per-channel order is preserved; replays produce duplicates, so every downstream consumer is idempotent on (channel_id, id) — the indexer upserts by id, the push service dedups by id, the archiver's bucket compaction is naturally idempotent. Watermark lag of ~1 s means at most ~1 s of messages replayed per channel on a crash — cheap. This costs one extra small batched write per channel per second on the owner, not per message.

Design change: add a per-channel published-to-Kafka watermark with replay-on-recovery; all Kafka consumers become idempotent on (channel_id, id).

Q4 (Codex): Which channels does each client actually subscribe to? Subscribing only to the open channel prevents real-time sidebar unread and mention updates, while subscribing to every joined channel makes a million-member channel fan out to every online member; quantify the gateway CPU and outbound bandwidth and explain where backpressure is enforced.

The design was vague and the vagueness hides a real cost, so here's the concrete model: two subscription tiers. The client's gateway holds a full subscription (complete message bodies) only for channels in an open pane — typically 1–3. Every other joined channel gets a digest subscription: the gateway receives the message stream (still one push per gateway per channel — the ≤150 bound is unchanged) but forwards only coalesced {channel_id, latest_msg_id} bumps per socket, at most one per channel per second. That's exactly what the sidebar needs, since the unread badge is latest > cursor, and it collapses a 10-msg/s firehose channel to one tiny event per second per viewer. Mentions bypass this via the per-user mention row, which emits a dedicated per-user event, so an @mention in a background channel badges and pushes immediately regardless of tier.

Numbers for the worst case — 1 M-member channel, ~300 K online members, 150 gateways ⇒ ~2 K subscribed sockets per gateway: at 10 msg/s, full fan-out would be 20 K socket writes/s and ~20 MB/s per gateway just for that channel; with digests it's ~2 K writes/s of ~50-byte events, ~100 KB/s — noise. Backpressure lives at the gateway's per-socket bounded send queue: on overflow the gateway drops that socket's queue for the affected channel, sends a resync {channel_id} marker, and the client gap-fetches over REST — degradation is explicit, never a silent gap.

Design change: subscriptions become two-tier (full for open panes, coalesced digest for background channels), with per-socket bounded queues and an explicit resync marker on overflow.

Q5 (Codex): Gateways do not persist events, yet clients only gap-fetch on reconnect. If the final message in a channel is dropped while the socket remains connected, no later sequence reveals the gap — so what triggers retransmission or anti-entropy, and how does cumulative acknowledgment produce at-least-once delivery within the latency objective?

Two hops can drop: channel-server→gateway (a blip on that internal connection) and gateway→client (only via a deliberate queue shed, since a live WebSocket is TCP — it doesn't silently drop, it stalls or dies). The design change: the channel-server→gateway protocol becomes sequenced-and-acked per (channel, gateway) — the gateway acks contiguous prefixes, the channel server retransmits from its in-memory recent buffer on nack or timeout. That closes the common loss case in milliseconds, inside the latency budget. Gateway→client sheds are already explicit (resync marker, Q4).

The backstop for anything neither catches — the "last message ever, then silence" case — is piggybacked anti-entropy: the socket heartbeat runs every ~30 s anyway, and the pong carries latest_msg_id for the client's full-subscription channels (the gateway already knows them from its digest state, so this touches no database); a mismatch triggers a gap-fetch. So: acked internal hop = fast-path correctness, heartbeat audit = bounded ~30 s worst-case staleness for a message that hit a shed and whose channel then went silent. I'll take a 30 s bound on that compound corner; the 500 ms objective is for the healthy path, and this keeps it honest without per-message end-to-end acks from every client.

Design change: channel-server→gateway pushes become sequenced and acked with retransmit; heartbeat pongs carry per-channel latest ids as anti-entropy.

Q6 (Codex): How do edits, user deletes, and retention deletion work after messages have been compacted into immutable S3/Parquet objects while copies also exist in Scylla, Redis, OpenSearch, Kafka consumers, replicas, and backups? Define the authoritative version/tombstone protocol that prevents stale content from reappearing during reads or replays.

Rule one: mutations are events with the message's (channel_id, id) plus a monotonic version — an edit is version+1 with a new body, a delete is a version with a tombstone flag. Authority is tiered: the Scylla row for hot-window messages, and for archived ones, the per-channel manifest + amendment overlay. Parquet objects stay immutable; a post-archive edit/delete writes a small amendment row keyed (channel_id, bucket, id, version) into a compact overlay store, and the cold read path merges base object + overlay (overlays are tiny — edits of months-old messages are rare). When a bucket's overlay crosses a size threshold, a compactor rewrites the Parquet object and clears the overlay — LSM logic at the archive tier.

Retention and workspace deletion go through the manifest: the reaper first removes bucket entries from the manifest (readers never fetch an object the manifest doesn't list, so nothing resurrects mid-delete), then deletes objects and Scylla partitions, issues delete-by-id to OpenSearch, and writes the audit tombstone. Stale copies elsewhere are handled by scope: Redis tail-cache entries are overwritten by the edit/delete event on the hot path and TTL out in minutes; Kafka topics get finite retention (~7 days) so content ages out, and any replayer is idempotent-by-version (a consumer applying version 3 ignores a replayed version 2); backups get a documented maximum age (e.g., 35 days) after which purged content is provably gone everywhere — that's the line I'd put in the retention policy rather than pretending backups can be surgically edited.

Design change: message mutations become versioned events; the archive tier gains a manifest + amendment-overlay protocol with threshold compaction; Kafka retention is capped at ~7 days and backup age at ~35 days as stated policy.

Round 2 — pushback

Q7 (Codex): Your epoch IDs prevent collisions but do not fence stale writes: if a client has seen an epoch-8 message, then the old owner ACKs an epoch-7 write, that message sorts behind the client's cursor and is permanently missed by after_id sync — what storage-level mechanism prevents that stale ACK?

Right — epochs stop id collisions, but a stale-acked epoch-7 write is durable, acked, and invisible to every cursor already past epoch 8. Scylla won't check fencing tokens per write without LWT, and LWT on 1.16 M writes/s is off the table, so I close it with three layers instead of one storage check. First, lease-epsilon self-fencing: the old owner rechecks its lease on a local monotonic clock immediately after the Scylla write and before sending the ack, and the new owner may not serve until lease TTL + ε has elapsed (etcd enforces the wait). That kills the common case; the residual is a process pause landing exactly between the recheck and the ack byte. Second, a sealed-bucket takeover: the new owner's first act is to write a seal marker at the old bucket's max id and open a fresh bucket for its epoch. Any epoch-7 straggler physically lands in the sealed bucket, above the seal point — a distinguishable anomaly, not a needle in a token range. The new owner rescans the sealed bucket once more at takeover + 60 s and re-emits anything found under a new epoch-8 id (so it sorts above every cursor), deduped by client_msg_id.

Third — and this is why the 60 s rescan actually terminates the problem — look at the client whose ack was that stale: for the ack to arrive after the rescan window, the old owner must have been paused longer than any client retry timeout. The sender's client gave up waiting at ~5–10 s and retried through the new owner, getting an epoch-8 id; when the epoch-7 straggler surfaces at rescan, client_msg_id dedup marks it superseded. So every path ends with the message visible exactly once under an id above the cursors: fast path via epsilon fencing, slow path via rescan re-emission, pathological path via the client's own retry. What I give up is a pure storage-level guarantee — I'm buying it with a bounded-clock-rate assumption plus a reconciliation scan, and I'd say that out loud rather than claim Scylla is doing something it can't.

Design change: lease-epsilon ack fencing (recheck after write, before ack; new owner waits TTL + ε); takeover seals the old epoch's bucket and rescans it at +60 s, re-emitting stragglers under new-epoch ids with client_msg_id dedup.

Q8 (Codex): After a channel server crashes, how does its replacement discover every channel with messages above the Kafka watermark — including inactive channels that receive no future traffic — without scanning its entire Scylla token range, and what bounds the delay before those messages reach archival and retention consumers?

The watermark answer assumed the new owner knows which channels to check, and for a channel that goes silent forever after the crash, nothing ever prompts the check. The fix is a per-segment pending-channel registry: each ring segment owns one small partition listing channels that currently have messages above their Kafka watermark. A channel is added on the dirty transition (first unpublished message — one batched write, not one per message; a continuously active channel stays listed with zero further writes) and lazily removed ~30 s after its watermark catches up. On takeover, the new owner reads exactly one partition — typically a few hundred to a few thousand channels, the ones active in the last ~30 s on that segment — checks each watermark against max id, and republishes the tails. Because removal is lazy, the persisted set is a strict superset of the truly dirty channels; a crash can never hide one. Delay bound: ring reassignment in seconds, one-partition scan plus replay in seconds — call it under ~30 s from crash to downstream consumers being whole, and only for that segment's channels.

But the sharper move the question pushed me to: archival and retention shouldn't ride on Kafka completeness at all. The archiver already reads whole Scylla buckets to compact them — make Scylla, not the Kafka stream, its source of truth (Kafka just nudges it about bucket activity), and have the retention reaper enumerate from the metadata DB's channel list, not from stream events. Then a Kafka gap can never lose archival or retention work even in principle, and the pending-registry replay only has to make search and push whole — consumers where a ~30 s recovery lag is invisible. Correctness anchored to the store, latency served by the stream.

Design change: per-segment pending-channel registry (dirty-transition insert, lazy removal) scanned on takeover; archiver and retention reaper re-anchored to read Scylla and the metadata DB directly, with Kafka reduced to a latency hint for them.

What changed, summarized

Sequence ids are now epoch-fenced: etcd lease + per-segment fenced epoch counter in the id's high bits; servers stop writing when their lease can't renew (Q1).
Durability guarantee restated honestly: no acked loss except unrecoverable regional destruction (RPO ≤ ~5 s); transient region loss reconciles by re-sequencing the divergent tail; synchronous cross-region replication offered per-workspace (Q2).
Scylla-as-outbox for the Kafka stream: per-channel published watermark, replay on ownership change, all consumers idempotent on (channel_id, id) (Q3).
Two-tier subscriptions: full bodies for open panes, coalesced {channel_id, latest_msg_id} digests for background channels; per-socket bounded queues with an explicit resync marker on overflow (Q4).
Channel-server→gateway hop becomes sequenced-and-acked with retransmit; heartbeat pongs carry per-channel latest ids as a ~30 s anti-entropy backstop (Q5).
Versioned mutation events end to end; archive tier gets manifest + amendment overlay with threshold compaction; Kafka retention ~7 days and backup age ~35 days become stated policy (Q6).
Stale-ack fencing: lease-epsilon recheck before every ack, sealed-bucket takeover with a +60 s straggler rescan re-emitting under new-epoch ids, client retry as the terminal safety net (Q7).
Per-segment pending-channel registry for crash recovery of unpublished tails; archiver and retention reaper read Scylla/metadata directly instead of trusting Kafka completeness (Q8).

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The channel-server shape isn't speculative — it's Slack's production architecture, described in their Real-time Messaging post (2023). Their Channel Servers are stateful and in-memory, consistent-hashed by channel ID (~16 million channels per host at peak), with Gateway Servers at the edge holding the websocket subscriptions and doing local fan-out — the same CS/GW split as this design's channel servers and gateways. Two details line up almost exactly: Slack states messages are delivered "across the world in 500 ms," the same number this prompt sets as the SLO, and their Consistent Hash Ring Managers replace an unhealthy channel server in under 20 seconds, which is the ballpark I claimed for ring reassignment in the failure-modes section.

The boot cache I hand-waved as "same problem Slack built Flannel for" is worth reading in full: Flannel is an edge cache that exists because clients used to download the whole team at boot, and reconnect storms turned that into a self-inflicted DDoS. Flannel's fix is the one in the Q4/boot section — slim boot payload, lazy queries, predictive push — and their numbers give it teeth: 4 million simultaneous connections, 600 K queries/s, boot payloads 44× smaller for 32 K-user teams. My gateway estimate also looks conservative against published single-node results: Phoenix demonstrated 2 million websocket connections on one 40-core box back in 2015 (the broadcast-sharding fix in that post is the same local-fan-out trick the gateways use), and WhatsApp ran ~1–2 million connections per Erlang/FreeBSD server while serving 465 M users with about ten Erlang engineers — they deliberately backed off from 2 M to 1 M per server for headroom, the same instinct as my 64 K-per-node-at-2×-headroom sizing. The Elixir/Erlang MVP pick leans on exactly this lineage.

The message store follows Discord's published evolution step for step. Their 2017 post How Discord Stores Billions of Messages (MongoDB→Cassandra) lands on partition key (channel_id, bucket) with ~10-day buckets and a snowflake clustering key — the schema in this design's storage section, nearly column for column, and for the same reason: unbounded channels make unbounded partitions. Their 2023 follow-up How Discord Stores Trillions of Messages documents the Cassandra→ScyllaDB move I cited from memory: hot partitions cascading latency cluster-wide, JVM GC pauses long enough to require manual reboots, and a 177→72 node shrink with p99 reads going from 40–125 ms to 15 ms. One thing Discord built that this design gets implicitly: a Rust data-service layer in front of the store doing request coalescing on hot channels — in this design the single-writer channel server plus the Redis tail cache play that role.

For the 1 M-member channel, Discord's Maxjourney post (2023) is the closest published analogue, and it validates the Q4 answer's core move. Their per-guild Elixir process scaled quadratically with guild size; the fix that bought the most was "passive sessions" — members not actively viewing the guild stop receiving full fan-out, cutting fan-out work ~90%. That is the two-tier subscription model (full bodies for open panes, digests for background channels) under a different name. Their "relays" tier — processes that each own ~15 K sessions and do permission-checked fan-out so the guild process pushes once per relay — is structurally the gateways-fan-out-locally bound. Result: Midjourney's server with 1 M+ concurrently online. On unreads, Discord's Read States service (2020) confirms the cursor model is what survives at scale — one read state per user per channel, billions of them — and adds a warning this design should inherit: read states sit on the hot path (touched on every connect, send, and read), and at that rate the service itself becomes a latency-sensitive cache tier. Discord's every-2-minutes Go GC spikes there drove their first Rust rewrite.

Where reality diverges from this design, it diverges against my metadata plan, and it's worth stating plainly. Scaling Datastores at Slack with Vitess says Slack started where this design sits — MySQL sharded by workspace — and migrated off it over three years because giant customers made hot shards that couldn't be subdivided, and cross-workspace features (Enterprise Grid, Slack Connect) didn't fit workspace sharding at all; on Vitess they reshard per-keyspace, and notably moved message metadata to channel-sharding, reaching 2.3 M QPS. So workspace sharding is the right stage-3 answer and the wrong final answer; the design's own 1 M-member workspace is precisely the hot shard that breaks it, and the escape hatch is flexible per-table shard keys, which is an argument for Vitess/Citus over hand-rolled sharding from the start. Similarly, Slack's Migration to a Cellular Architecture (2023) uses "cell" to mean an availability-zone-scoped silo inside one region — Envoy weighted clusters draining an AZ at 1% granularity in under 5 minutes — aimed at gray failures, not region loss. My region-homed workspace cells solve a different problem (data residency and region failover); a production system would want both layers, and I'd conceded only one.

Updates from post-training information

Two things surfaced that the design as written doesn't account for.

ScyllaDB is no longer open source. In December 2024 ScyllaDB ended its AGPL open-source edition at 6.2 and moved to a source-available Enterprise license, with a free tier capped at 50 vCPUs and 10 TB per organization. The design's 150–200-node message cluster blows through that cap by two orders of magnitude, so the "DynamoDB costs ~$12–13 K/day, therefore self-managed wins" comparison now has a ScyllaDB license line item on the self-managed side that I priced at zero. The architecture doesn't change — a wide-column LSM store partitioned by channel is still right — but the vendor math needs redoing at stage 3: ScyllaDB Enterprise license vs. DynamoDB vs. Cassandra 5.x (still Apache 2.0, and its newer trie-based memtables and unified compaction close some of the gap that drove Discord's migration).

Fan-out observability got a published playbook. In March 2026, InfoQ covered Discord's distributed tracing across Elixir actors: an envelope primitive carrying trace context through GenServer call/cast, with sampling scaled to fan-out size — 100% for single-recipient sends down to 0.1% at 10 K+ recipients — so tracing a million-user fan-out doesn't melt the telemetry pipeline. This design specifies delivery mechanics in detail (Q5's acked hops, Q7's fencing) but says nothing about observing them; I'd bake trace propagation into the channel-server→gateway protocol from stage 2, with exactly that fan-out-proportional sampling, because debugging the Q5/Q7 corner cases without per-message traces means debugging from logs alone.