Permission-Aware Knowledge Platform

The hard problem here isn't search — it's that every result is a potential data leak. A knowledge platform that surfaces one Slack DM to the wrong person is dead, so the design below treats permission enforcement as the load-bearing wall and builds ingestion, indexing, and retrieval around it. My core decision, stated up front: filter coarsely in the index, check authoritatively at read time. Everything else follows from that.

Assumptions

Stating these rather than asking, as instructed:

"Take effect within one minute" for revocations means one minute from when we learn of the change. If a source only exposes a 15-minute polling API, no design can beat that; I'll cover how we minimize detection lag per source. Revocations made through our platform's view of identity (user offboarded in the IdP, removed from a group) take effect in under a minute end-to-end.
Average source item produces ~~10 chunks (100M items → 1B chunks checks out). Average chunk is ~300 tokens (~~1.2 KB of text).
Of the 10M daily queries, roughly 70% are keyword-ish searches and 30% are natural-language questions that need an LLM answer. That's an estimate that shapes GPU/LLM capacity, so I'll flag it as tunable.
Each org brings its own OAuth credentials per source; we're multi-tenant SaaS with hard tenant isolation as a requirement, not a nice-to-have.
Organizations have an IdP (Okta, Entra, Google Workspace) we can sync via SCIM. Identity mapping across sources (Slack user U123 = jane@acme.com = GitHub jdoe) is our job.
AI agents are first-class callers but always act on behalf of a human principal or an explicitly-scoped service principal. An agent never gets a superset of any user's permissions.

Requirements and priorities

Functional: connect sources (Drive, Slack, Notion, Confluence, Zendesk, GitHub, websites, uploads); keyword and semantic search; question answering with cited sources; admin console for connectors; API for agents.

Non-functional, in priority order:

Correctness of permissions. Zero tolerance for over-exposure. We fail closed: if we can't verify access, we don't show the result.
Freshness: content searchable ≤5 min after change; revocations effective ≤1 min after detection.
Latency: search p95 under ~500 ms; answer first-token under ~2 s.
Tenant isolation and 1,000× size variance — a 50-person startup and a 200,000-person bank on the same platform.
Availability ~99.9% for query path. Ingestion can degrade (staleness) without taking queries down.

Scale estimates

These drive real architecture choices, so let me do them properly.

Queries: 10M/day ≈ 116 QPS average, 20× peak ≈ 2,300 QPS. Of that, ~700 QPS peak needs LLM generation (30% assumption).
Ingest: 5M items/day → ~50M chunk writes/day ≈ 580 chunk updates/s average; budget 3–5× for bursts (someone connects a new 10M-document Drive) → design the pipeline for ~3,000 chunks/s sustained.
Embedding compute: 580 chunks/s × 300 tokens ≈ 175K tokens/s average. A single modern GPU embeds a small model (~100–300M params) at very roughly 50–100K tokens/s, so this is an estimated fleet of 5–20 GPUs for steady state, autoscaled for backfills. Small enough to self-host; also small enough that a hosted embedding API is viable. I'd self-host (Qwen or E5-class open model, 768-dim) because backfilling a new tenant's 10M documents through a metered API is slow and expensive, and because model version pinning matters — you cannot mix embedding versions in one index.
Index size: 1B chunks. Vectors at 768-dim int8 = 768 GB; text + inverted index + ACL fields + metadata ≈ 3–5 TB more. Call it ~5 TB of hot index — comfortably a 30–60 node cluster of a Lucene-based search engine (OpenSearch), not a "build something exotic" scale.
Permission tuples: 100M items × maybe 5 ACL entries average = ~500M tuples, plus group membership. Billions of rows at the high end. This is exactly the shape Zanzibar was built for.

Conclusion from the numbers: nothing here demands custom infrastructure. The difficulty is semantics (permissions, freshness, identity), not raw scale.

Architecture

flowchart TB
    subgraph Sources
        GD[Google Drive]
        SL[Slack]
        GH[GitHub / Notion / Confluence / Zendesk / uploads]
    end

    subgraph Ingestion
        CF[Connector framework<br/>webhooks + cursors + polling]
        K1[(Kafka: raw-changes)]
        NORM[Normalizer<br/>parse, extract, canonical doc]
        K2[(Kafka: doc-updates / acl-updates)]
        CHUNK[Chunk + embed workers<br/>GPU fleet]
    end

    subgraph Storage
        S3[(S3: raw blobs +<br/>canonical docs)]
        PG[(Postgres: doc metadata,<br/>connector state, tenants)]
        OS[(OpenSearch:<br/>hybrid BM25 + HNSW,<br/>ACL tokens per chunk)]
        SDB[(SpiceDB:<br/>relation tuples)]
    end

    subgraph IdentityPlane
        SCIM[IdP sync SCIM/OIDC]
        IDMAP[Identity resolver<br/>source identity → principal]
    end

    subgraph QueryPath
        GW[API gateway<br/>authn, rate limits, tenant routing]
        QS[Query service]
        RANK[Reranker<br/>cross-encoder]
        ANS[Answer service<br/>LLM + citations]
        PC{{Read-time<br/>permission check}}
    end

    Sources --> CF --> K1 --> NORM
    NORM --> S3
    NORM --> K2
    K2 --> CHUNK --> OS
    K2 -->|ACL fast path| SDB
    NORM --> PG
    SCIM --> IDMAP --> SDB

    User([User / AI agent]) --> GW --> QS
    QS -->|principal → group tokens| SDB
    QS --> OS --> RANK --> PC --> ANS --> User
    PC -->|check top-k| SDB

Two planes: an ingestion plane that's eventually consistent and can lag, and a query plane that must be fast and must never show an unauthorized result. The seam between them is the permission model, so I'll start there.

The permission model — the hard part

There are three classic strategies, and the first two fail alone:

Late binding only (check every candidate at query time against the source system or a permission service): correct, but a semantic search that retrieves 1,000 candidates and post-filters can return zero results for a locked-down user, and recall craters. You end up re-querying with bigger and bigger k. At 2,300 QPS this is also a brutal fan-out.

Early binding only (bake the resolved set of allowed users into each indexed chunk): fast filtering, but a group membership change would require re-indexing every document that group can see. One removal from eng-all at a 100K-person org could touch millions of chunks. You cannot meet a 1-minute revocation SLO this way. This intuitively feels fine until you work out the write amplification, and then it doesn't.

What I'd build: early binding of principal tokens, late binding of membership, plus an authoritative read-time check.

Each chunk carries an acl_tokens keyword field in the index: the direct grants on the source item, expressed as opaque tokens — user:jane@acme, group:drive/eng-leads, channel:slack/C123, org:acme:public, link:anyone-in-domain. These change only when the document's ACL changes, which the connector observes as a change event like any other. No fan-out on group membership changes.
At query time, the query service expands the caller into their token set: their user token, all group/channel tokens they belong to, org-public. This comes from a relationship-based authorization store, SpiceDB (backed by the SCIM sync and connector-observed memberships) with a cache TTL of 30 seconds — half the revocation budget. The search query gets a filter clause: acl_tokens ∩ caller_tokens ≠ ∅. Removing someone from a group takes effect within 30 s, at query expansion, with zero re-indexing. A typical user might carry 50–500 tokens; OpenSearch terms filters handle that fine, and for the pathological 5,000-group user we precompute a bloom-style rollup.
Before any snippet is rendered or fed to the LLM, the top-k results (k ≈ 20–50, not 1,000) get an authoritative Check against SpiceDB — no cache for negative-sensitive paths, or a ≤10 s cache. This is the backstop that makes the 1-minute revocation SLO real: when a connector sees an ACL change, it writes the tuple to SpiceDB immediately (a fast, tiny write on a separate topic, acl-updates, in a durable partitioned event log — Kafka — with priority over content re-indexing). The search index's acl_tokens field may lag minutes; the read-time check doesn't. Stale index tokens cause at worst a false candidate that gets filtered — never a leak.

So: the index filter is a recall/performance optimization that's allowed to be stale; SpiceDB is the truth that's required to be fresh. That separation is the whole design. I'll repeat it because it's the principle everything hangs off: the index may be stale, the check may not.

Why SpiceDB (a Zanzibar implementation) and not rolling our own ACL tables in Postgres? Because the sources have genuinely different permission semantics — Drive has nested folders with inheritance and "anyone with link", Slack has channel membership, GitHub has org/team/repo roles, Confluence has space + page restrictions — and Zanzibar's relation-tuple + userset-rewrite model expresses all of these declaratively. Writing inheritance resolution ourselves per source is exactly the code that ships a leak. I rejected OPA (policy engine, wrong shape for relationship data at this row count) and I rejected checking against the source systems live (rate limits make 2,300 QPS × k checks impossible, and a source outage would take down search).

Fail closed, concretely: if SpiceDB is unreachable, the query path returns an error, not unfiltered results. SpiceDB runs multi-AZ with read replicas; it's on the critical path and gets treated like it.

Identity resolution

Every source has its own identity space. The identity resolver maintains (*source*, source_user_id) → platform_principal, seeded from the IdP (email as join key) and enriched per connector (Slack profile email, GitHub verified emails via the org API). Unmappable identities get a synthetic principal that matches no one — fail closed again. This table is small (millions of rows, in a relational store — Postgres) but it's a correctness keystone: a wrong mapping is a cross-user leak, so mappings require either IdP confirmation or verified email, never fuzzy name matching.

AI agents

Agents authenticate with OAuth tokens carrying an on_behalf_of principal, or as a service principal an admin explicitly granted specific group tokens. The permission machinery is identical — an agent is just a caller with a token set. The rule we enforce structurally: an agent session's answers are computed only from chunks its effective principal can read, so an agent can't aggregate-and-launder content across users. Agent traffic gets separate rate limits and a per-call audit log (who asked, on behalf of whom, which documents were retrieved and which were filtered), because "why did the bot say that" is a question security teams will ask.

Ingestion plane

Connector framework. One framework, per-source adapters, three acquisition modes ranked by preference: webhooks (Slack Events, Drive push notifications), change cursors (Drive Changes API, Zendesk incremental export), and scheduled polling (Confluence, generic websites — crawl with sitemap + conditional GETs). Every adapter emits the same envelope onto Kafka raw-changes: (tenant, source, item_id, change_type: content|acl|delete, cursor, payload_ref). Connector state (cursors, OAuth tokens, backfill progress) lives in Postgres. Webhooks are treated as hints, not truth — they get confirmed by a cursor fetch, because every webhook system drops events. A slow reconciliation crawl (full listing weekly, per tenant) catches anything both missed; this is the only defense against silent divergence and it's non-negotiable.

Why Kafka and not SQS: we need per-tenant ordering for cursor semantics (partition by tenant:source), replay for reprocessing after a parser bug, and two priority lanes. acl-updates is its own topic with its own consumer fleet so a 10M-document backfill can never queue behind a permission revocation.

Normalizer. Parses source payloads into a canonical document: extracted text (Tika for PDFs/Office, a vision model only for image-heavy docs where OCR fails — that's expensive, so it's opt-in per tenant), structure (headings, thread → messages, table cells), source ACL, timestamps, URL. Raw blob and canonical doc go to durable object storage (S3); metadata to Postgres; doc-updates and acl-updates to Kafka.

Chunker + embedder. Structure-aware chunking: ~200–400 tokens, split on headings/messages, tables kept row-grouped with header context, threads chunked per few messages with the parent message prepended. Each chunk gets the doc's acl_tokens, tenant id, source, timestamps, and its embedding, then upserts to OpenSearch. Idempotent by (doc_id, chunk_seq, content_hash) — reprocessing is safe, which matters because we will reprocess.

The 5-minute freshness budget: detection (webhook: seconds; cursor poll: ≤60 s) → normalize (~seconds; a 500-page PDF is the tail, so big documents publish extracted text incrementally) → embed (batched, seconds at our QPS) → OpenSearch refresh (5 s interval). Comfortable at p50, and the p99 offender is parsing, not queueing — so parsing gets its own autoscaled pool and a 2-minute timeout that falls back to plain-text extraction.

Backfill vs steady state. New tenant connects a source with 10M items: that's 100× daily change volume in one burst. Backfills run on a separate consumer group at bounded concurrency against the source API's rate limits, with per-tenant quotas so one onboarding can't starve another tenant's freshness. Order of operations on backfill matters: ACLs and metadata first (so the permission model is complete), content second.

Storage and index choices

OpenSearch as the single hybrid index — BM25 and HNSW vectors in one engine, one filter execution, one operational surface. The alternative I seriously considered is Elasticsearch + a dedicated vector DB (Qdrant/Turbopuffer): better pure-vector economics, but now every ACL filter, delete, and freshness guarantee must be implemented twice and kept consistent, and ACL-filtered HNSW search is exactly where separate systems diverge (filtered ANN with high-selectivity filters is a known hard case; you want the engine that can fall back to exact search over the filtered set, which OpenSearch/Lucene does). Vespa would also be a fine answer with stronger filtered-ANN behavior; I pick OpenSearch for team familiarity and ecosystem, and I'd revisit at 10B chunks.

Tenant layout in the index — this is where the 1,000× size variance bites. Three tiers:

Small orgs (the long tail of the 2,000): shared indices, ~50 tenants per index group, routed by tenant_id, every query hard-filtered by tenant. A noisy small tenant is contained by routing.
Large orgs: dedicated index (or index-per-source), own shard count, own refresh tuning.
Regulated/paranoid orgs: dedicated cluster, priced accordingly.

Tier migration is an online reindex — the canonical data is in S3/Postgres, the index is always rebuildable. Treating the search index as a derived, disposable view is what makes embedding-model upgrades and disaster recovery sane: blue/green reindex from canon, flip an alias.

Postgres (Aurora or similar) for tenants, connector state, doc metadata, identity map, audit log — relational, transactional, modest size. S3 for blobs and canonical docs, lifecycle-tiered. SpiceDB on its own Postgres/CockroachDB backend. Redis for query-path caches (token-set expansion, rate limits).

Data model (abridged)

documents:  tenant_id, doc_id, source, source_item_id, url, title,
            content_hash, acl_hash, version, updated_at, deleted_at
chunks (in OpenSearch):
            tenant_id, doc_id, chunk_seq, text, embedding[768],
            acl_tokens[], source, updated_at, lang
tuples (SpiceDB):
            document:d1#viewer@group:drive/eng-leads
            group:drive/eng-leads#member@user:jane
            document:d1#parent@folder:f9        (inheritance via rewrite rules)
identity_map: source, source_user_id, principal_id, verified_by
connector_state: tenant_id, source, cursor, oauth_ref, backfill_status

acl_hash on documents lets the connector skip ACL writes when only content changed — most changes — keeping the acl-updates lane quiet and fast.

APIs

POST /v1/search        { query, filters?, page }         → ranked results + snippets
POST /v1/ask           { question, conversation_id? }    → SSE stream: answer tokens + citations
GET  /v1/documents/:id                                   → metadata + deep link (permission-checked)
POST /v1/admin/connectors           # create/configure, OAuth dance
GET  /v1/admin/connectors/:id/health
POST /v1/agents/tokens              # scoped agent credentials, on_behalf_of

/v1/ask streams over SSE because first-token latency is the perceived latency. Citations are emitted as structured events referencing chunk ids, and the UI deep-links to the source system — we show snippets, but "open in Drive" re-encounters the source's own auth, a nice second net.

Query path, end to end

Gateway authenticates (OIDC session or agent token), resolves tenant, rate-limits.
Query service expands principal → token set (Redis, 30 s TTL, SpiceDB on miss).
Hybrid retrieval: BM25 and kNN (query embedded by the same model family), both with tenant_id + acl_tokens filters pushed into the engine, ~top-100 each, fused with reciprocal rank fusion.
Cross-encoder reranker (small model, GPU, ~30–50 ms for 100 pairs) → top-20.
Authoritative SpiceDB check on the top-20. Anything failing is dropped and backfilled from the next candidates.
Search: return results. Ask: assemble ~6–8 chunks into context, call the LLM with an instruction to answer only from provided context and cite chunk ids, stream out, validate that every cited id was actually in the permitted context (a citation to anything else is a bug we hard-fail on).

Latency budget at p95: expansion 5 ms (cached) + retrieval 150 ms + rerank 50 ms + checks 20 ms (batched) ≈ 250 ms for search. Ask adds LLM time; first token ~1–2 s.

LLM choice: a hosted frontier-class model behind an abstraction, with a self-hosted open-weight fallback for tenants that require no-egress. Grounded-answer quality is the product; embedding and reranking are where self-hosting pays, generation is where it usually doesn't yet at 700 QPS peak — rented capacity with a second provider configured for failover.

Failure modes

SpiceDB down: queries fail closed (error, not open results). This is the one component where we accept availability loss for correctness.
LLM provider down: /v1/ask degrades to search results with an apology banner; /v1/search unaffected. Secondary provider failover before that.
Connector/source outage or OAuth expiry: content goes stale, which is visible, not dangerous — except stale ACLs. Per-connector staleness is tracked; past a tenant-configurable threshold (default: 24 h without ACL confirmation for a source), that source's results get a staleness warning, and paranoid tenants can opt into auto-hiding it. Admins get alerted on day one, not when a leak happens.
Kafka lag / backfill flood: the acl-updates lane is isolated, so revocations stay fast even when content indexing is hours behind. Freshness SLO burns are per-tenant-visible in the admin console.
Poisoned content / prompt injection: retrieved chunks are data, never instructions — the answer prompt is structured accordingly, retrieved text is delimited, and tool-use is disabled during answer generation. Not a solved problem; it gets a red-team suite in CI.
Embedding model upgrade: never in place. Blue/green: reindex from canonical docs into a new index with the new model, shadow-query for quality, flip alias per tenant.
The leak drill: if over-exposure is ever detected, the audit log (every query, every returned chunk, every check result) answers "who saw what" in minutes. That log is append-only and is as much a part of the security story as the checks themselves.

Tradeoffs I'm consciously making

Read-time checks cost latency on every query (~20 ms, plus SpiceDB as a hard dependency) to buy revocation correctness. I'll pay that forever; it's the product's premise.
Single hybrid engine over best-of-breed vector DB: somewhat worse vector economics for one consistency model and one ACL-filter implementation. Revisit at 10× scale.
Copy-and-index over federated live search: we replicate content (a real security liability — we're now a high-value target and must encrypt per-tenant, audit, and honor deletion in ≤ hours) because federated query-time search against eight source APIs can't hit 300 ms or survive rate limits. Deletion propagation gets the same priority lane as ACLs.
30 s membership cache: a tunable dial between SpiceDB load and revocation lag; regulated tenants can set it to zero and pay the latency.
Eventual consistency for content, near-real-time for permissions: users forgive a 5-minute-stale document; nobody forgives a leak. The asymmetry is deliberate and it shows up as two Kafka lanes, two SLOs, two dashboards.

Evolution

First year, roughly in order: activity-signal ranking (clicks, source popularity — search quality is won here, not in the embedding model); an MCP-style tool interface so agents can retrieve with scoped credentials instead of scraping the UI; cross-source entity resolution ("this Jira ticket, this PR, and this Slack thread are one incident"); per-tenant fine-tuned rerankers for the largest customers. The permission architecture shouldn't need to change for any of these — if it does, I got the seam wrong, and the seam is the design: the index may be stale, the check may not.

Interview follow-ups (Codex as interviewer)

The original design above is unchanged. Codex read the prompt and the design, then asked the follow-ups below; answers and any design revisions follow each question.

Round 1

Q1 (Codex): You redefine the one-minute revocation SLO as starting "when we learn" of a change, yet continue serving results when a source or connector may be stale for 24 hours. If a user loses Drive or Slack access while that connector is down, how does your system uphold the stated "must never see" requirement rather than serving content for hours?

You've found an inconsistency, and I'll own it: I applied fail-closed to SpiceDB but not to connectors. Two revocation classes behave differently here. Identity-plane revocations — user offboarded in the IdP, removed from a group — flow through SCIM, not the connector, so they keep working during a connector outage and stay under a minute; that covers the most common case (someone leaves the company or a team). Source-native ACL changes (someone removes Jane from a specific Drive folder) are genuinely undetectable while that connector is down, and my 24-hour default with opt-in hiding quietly traded correctness for availability. That's the wrong default for this product.

Design change: ACL-lane liveness becomes its own tracked heartbeat per (tenant, source) — a lightweight cursor poll that only confirms the ACL stream, cheap enough to run every minute. Warn admins at 15 minutes without ACL confirmation; auto-hide that source's results at 60 minutes, default-on (tenants can tighten to zero or loosen with an explicit signed-off risk acceptance). The cost is availability for that source during an outage — which is exactly the fail-closed premise I claimed as priority #1, now applied consistently. The one-minute SLO still starts at detection; what changes is that "we can't detect" now has a bounded, enforced consequence instead of a banner.

Q2 (Codex): Your index admits candidates using a simple acl_tokens ∩ caller_tokens ≠ ∅, while SpiceDB models inheritance, nested groups, conjunctions, exceptions, and source-specific rules. What exact tokenization preserves recall for every policy SpiceDB can authorize—without materializing all effective grants—and how do you handle policies that cannot be represented as an OR of tokens?

The question assumes the tokens must express the policy. They don't — they only need to over-approximate it. The invariant, stated in the error-direction the original doc left implicit: the set of principals reachable from a chunk's acl_tokens must be a superset of the true viewer set, never a subset. Under-approximation is the only unrecoverable failure (an authorized doc becomes invisible forever); over-approximation just produces false candidates that the SpiceDB check removes. Given that, an OR-of-tokens superset always exists — the degenerate case is the tenant token itself.

So the connector's token compiler works per policy shape. Pure grants (Drive share, Slack channel): index the grants directly — exact. Conjunctions (Confluence page = space permission AND page restriction): index the most selective conjunct — the page-restriction principals when present, the space token otherwise. Users holding one conjunct but not the other become false candidates; SpiceDB, which holds the full conjunctive policy, filters them. Denials and exceptions: index positive grants only, denials live only in SpiceDB — again over-admits, again filtered. Anything a source invents that we can't compile: coarse fallback token (repo, space, org) and a metric that tells us precision is degrading. The cost of every fallback is precision, not correctness, and the precision cost shows up as check-time drop rate — which is Q4's problem and is instrumented as such.

Design change: the token compiler's contract is now explicit — indexed acl_tokens must always expand to a superset of true viewers; conjunctive policies index their most selective conjunct; denials never appear in the index.

Q3 (Codex): You claim SpiceDB is authoritative immediately after an ACL update, but its replicas, tuple-ingestion pipeline, identity sync, and membership cache are all independently asynchronous. What consistency token or watermark does a query carry to prove its authorization check is at least as fresh as the revocation, especially during replica lag or failover?

Fair hit — "authoritative" was doing unexamined work. SpiceDB already has the right primitive: ZedTokens with at_least_as_fresh consistency.

Design change: when the acl-updates consumer writes a revocation tuple, SpiceDB returns a ZedToken; the consumer advances a per-tenant high-watermark in Redis (monotonic, replicated). Every read-time Check on the query path runs at_least_as_fresh(tenant_watermark) — a lagging replica either serves at or beyond that snapshot or the query routes to one that can. That makes the revocation bound equal to tuple-write latency plus nothing, independent of replica lag.

Two clarifications this forces. First, the 30-second membership-expansion cache only feeds candidate generation — the final Check re-derives group membership inside SpiceDB at the watermark, so a stale expansion can only cost recall (missing candidates), never grant access. Second, failover: if the watermark is lost (Redis failover), checks fall back to fully_consistent until the watermark repopulates — slower, correct, and self-healing. IdP-driven membership writes advance the same watermark, so offboarding gets the same guarantee.

Q4 (Codex): You retrieve roughly 200 candidates, rerank 20, then permission-check and "backfill," but users with sparse access—or stale, overly broad index ACLs—may require examining thousands of candidates. What algorithm bounds latency and preserves top-k recall without leaking relevance signals, and what happens when authorized results lie beyond the ANN candidate set?

The loop is bounded because the index filter already did the permission work — the SpiceDB check is a backstop, not the filter. False candidates come from exactly two places: index staleness (a minutes-wide window) and deliberate over-approximation from Q2's compiler. In steady state I'd expect the top-20 drop rate well under a few percent — an estimate, and it's instrumented per tenant/source so we know when a token compiler is over-approximating too coarsely.

Design change: the backfill is now explicitly bounded — round one checks the top-20; past a drop threshold, exactly one retry at k×5 with checked ids excluded; after that, return what passed, flagged internally as partial. Never an unbounded loop. The sparse-access user is actually the good case, not the bad one: their acl_tokens filter is highly selective, and Lucene's kNN falls back to exact search over the filtered set when the filter matches few docs — perfect recall over their permitted universe, cheap because the universe is small. The risky band is medium selectivity, where ANN with a restrictive filter can under-recall; that's handled by scaling ef_search with estimated filter selectivity. On leaking relevance signals: users never see filtered counts or placeholders — drops are visible only in the audit log.

Q5 (Codex): You attach one document ACL to every chunk, but several sources contain compound objects whose subparts can have different visibility, and edits can change chunk boundaries. How do you define the source-native authorization unit and atomically replace all old chunks so restricted messages, rows, comments, or orphaned prior-version chunks cannot remain searchable?

Right that "document" was too coarse.

Design change, two parts. First, the authorization unit becomes the protected object: the finest granularity at which the source assigns visibility — a Zendesk ticket splits into public-comment and internal-note objects; a Slack channel's messages are channel-scoped but a DM thread is its own object; a Drive file is one object. The chunker gets a hard rule: no chunk spans a protected-object boundary, and the index's doc_id becomes the protected-object id, which is also the SpiceDB object the read-time check targets. So a restricted internal note is never inside a chunk whose tokens came from the public part.

Second, orphans. Chunks carry a version; reindex is a two-phase swap — write all version-N chunks, then delete-by-query (object_id, version < N), and the ingestion task doesn't ack until the delete is confirmed, so a crash re-runs the whole swap idempotently. The residual risk — content that moved to a more restricted object while stale chunks retain the old object id — needed more than this, and round 2's Q8 forced the full protocol (below). The weekly reconciliation crawl diffs the index's chunk inventory against canonical docs in S3 as the last net, and a stale chunk still gets checked against its object's current ACL at read time, so an orphan has to slip every one of those layers before it can leak.

Q6 (Codex): The capacity estimate counts mostly raw int8 vectors and asserts that 1 billion filtered-HNSW chunks fit comfortably on 30–60 OpenSearch nodes. After HNSW graphs, inverted indexes, stored text, replicas, segment-merge headroom, tenant skew, and 500–5,000-term ACL filters at 2,300 peak QPS, what measured per-node memory, disk, and latency assumptions make that claim hold?

No measured numbers — I won't pretend to have them — and rebuilt with the overheads you list, my 30–60 was thin. Redoing the arithmetic: 768 GB raw int8 vectors; HNSW graph links at M=16 add roughly 100–130 GB; stored text ~1.2 TB; inverted index on that text plus keyword fields, call it another ~1–1.2 TB; doc values and ACL token fields ~200 GB. That's ~3.5 TB primary, ~7–8 TB with one replica, before segment-merge headroom (Lucene wants ~30–50% free disk). The binding constraint isn't disk, it's page cache: filtered kNN wants vectors plus graph hot, ~1.8 TB across replicas. On 128 GB-RAM NVMe nodes with ~60–80 GB usable cache each, that's cache pressure math pointing at 60–100 data nodes, not 30–60.

Design change: capacity estimate revised to ~7–8 TB with replicas on 60–100 data nodes, and GA is gated behind a benchmark: fixed 1B-chunk synthetic corpus, worst-case 5,000-term filters, measured p95 at 20× projected per-cluster peak. The QPS side survives for a structural reason: tenant tiering means a query fans out to one tenant's index group, not the whole 1B-chunk cluster, so per-shard QPS is peak-per-tenant, not 2,300 global. The 5,000-term ACL filter is pre-resolved to a bitset per segment once per query — tens of ms worst case, which is why the pathological user gets the precomputed rollup token. If the benchmark says 150 nodes, the answer is 150 nodes, not a redesign.

Round 2 — pushback

Q7 (Codex): Your "fail-closed" correction still serves results for 60 minutes after losing ACL confirmation; what prevents a source-native revocation made at the start of that outage from leaking during that window, and how is this compatible with "must never see" and the one-minute revocation SLO?

Nothing prevents it, and I won't pretend otherwise: during an ACL-blind window, a source-native revocation is served until the hide threshold fires. But notice this window isn't created by the outage — it exists in healthy steady state too. A source that only exposes 15-minute polling serves a between-polls revocation for up to 15 minutes with every connector green. Any copy-and-index design has this property, and the literal reading of "must never see" is unattainable by any architecture that can't observe the source — including live federation, which during a source outage can't serve at all (which is just fail-closed with a worse product). The prompt's own SLO — "revocations take effect within one minute" — presupposes a detection event to count from. So the honest contract has two clauses, not one: within one minute of detection, always; and when detection is impossible, serving stops within a bounded exposure window T.

Design change: T becomes an explicit, per-tenant, per-source contract term instead of an internal default. Strict mode sets T to two missed ACL heartbeats (~2–3 minutes) — a Drive folder revocation made the instant the connector dies is served for at most that long before the source's results vanish. The 60-minute default stays for tenants that prefer availability, but it's now a documented exposure bound they chose, surfaced in the admin console next to the SLO, not a buried platform behavior. And the SLO document itself gets rewritten with the conditional structure above — selling "never" while running a poller would be dishonest, and the fix is to sell the real guarantee: one minute from detection, bounded blindness, and you pick the bound.

Q8 (Codex): Deleting (object_id, version < N) cannot remove stale chunks when content moves to a new protected-object ID, so the old permissive object may still authorize sensitive content; what stable-lineage or tombstone protocol revokes every old chunk before the replacement becomes searchable, including under crashes and out-of-order events?

Correct — version-scoped delete only covers same-object rewrites; a subpart that migrates to a new object id (public comment made internal, message moved to a restricted thread) leaves chunks under the old, still-permissive id. The fix is to anchor lineage one level up, at the source item, which is stable.

Design change: the canonical layer keeps, per (tenant, source, source_item_id), the current set of protected-object ids derived from it (object ids are deterministic: hash of tenant, source, source_item_id, subpart key). Every normalizer run over a source item produces S_new; the diff against the stored S_old drives the protocol, in this order: (1) tombstone every object in S_old \ S_new in Postgres, (2) delete-by-query all chunks for tombstoned objects and all version < N chunks for surviving objects, with an index refresh, (3) only then upsert the new version-N chunks, (4) commit S_new and ack the Kafka offset. Crashes replay the whole unit idempotently — steps 1–2 re-run harmlessly, and nothing new became searchable before the deletes were confirmed. Out-of-order events can't regress it: Kafka partitioning by tenant:source keeps per-item order, and every write and delete is guarded by the source cursor version, so a late old event is rejected by version comparison rather than resurrecting dead chunks. Tombstones are permanent rows, which gives the weekly reconciliation a checkable assertion — no chunk may exist for a tombstoned object id — instead of a fuzzy diff. So the interval where restricted content is searchable under the old id is the normalizer's processing time for that one item, the deletes land strictly before the replacement is visible, and two independent nets (version guards, tombstone reconciliation) cover the crash and reorder cases.

What changed, summarized

ACL-staleness handling inverted: per-(tenant, source) ACL heartbeat; warn at 15 min, auto-hide the source's results at a bounded window T — default 60 min, default-on, strict mode ~2–3 min. The revocation SLO is restated as a two-clause contract: one minute from detection, plus an explicit per-tenant exposure bound when detection is impossible.
Token compiler contract made explicit: indexed acl_tokens must over-approximate the viewer set, never under-approximate; conjunctions index their most selective conjunct; denials live only in SpiceDB.
Freshness proof added to the check: per-tenant ZedToken watermark advanced on every ACL write; all read-time Checks run at_least_as_fresh(watermark), with fully_consistent fallback if the watermark is lost.
Backfill loop bounded: top-20 check, at most one retry at k×5, then partial results — a hard two-round cap.
Authorization unit refined to the protected object: finest source-native visibility granularity; chunks never span object boundaries; index doc_id and SpiceDB object are the protected object.
Lineage + tombstone protocol for chunk replacement: per-source-item diff of protected-object sets, tombstone and delete before new chunks become searchable, cursor-version guards against reordering, tombstones checked by weekly reconciliation.
Capacity estimate revised: ~7–8 TB with replicas on 60–100 OpenSearch data nodes (was 30–60), with a 1B-chunk worst-case-filter benchmark gating GA.

Industry practice and further reading

Added after the interview rounds: how real systems handle the hard parts above, with verified sources (checked 2026-08-19).

How industry does it

The design's central move — mirror source permissions into your own model, filter coarsely in the index, check at read time — is what the two biggest shipping systems in this space actually do. Glean maps every connector's users, groups, and memberships into a unified identity schema refreshed by periodic identity crawls, which is my identity resolver plus token expansion under a different name; their post is mostly about making those crawls fast (they cut crawl time up to 98% for some customers by splitting API-bound work from internal group resolution), which confirms that permission sync, not query-time checking, is where the operational pain concentrates. Microsoft 365 Copilot's semantic index is the read-time half: a tenant-wide vector index over Graph content that is permission-trimmed at query time against the same RBAC used everywhere else in M365 — "the grounding process only accesses content that the current user is authorized to access." Neither company trusts index-time filtering alone, and neither re-checks against the original source at query time. Same seam.

The SpiceDB/ZedToken machinery from Q3 isn't speculative — it's the published mechanism. The Zanzibar paper exists precisely because Google needed causally consistent ACL evaluation across Drive, Calendar, and YouTube, and its zookie protocol is the ancestor of the per-tenant watermark I adopted; AuthZed's ZedTokens post walks through the same "new enemy problem" (a revoked viewer seeing content because the check ran against a stale replica) and the at_least_as_fresh semantics this design leans on. My Q3 fix is a direct application of that post, not an invention.

Where I diverge from AuthZed's own guidance is candidate filtering. They publish two patterns: LookupResources, which computes the caller's full accessible-resource set at query time (pure late binding — exactly what I rejected at 2,300 QPS with millions of resources per user), and Materialize, which continuously precomputes per-user permission sets and streams them into a secondary index like Elasticsearch — pure early binding of effective grants, the approach I rejected for write amplification, made viable by turning the fan-out into a continuously maintained materialized view. My acl_tokens middle path (index direct grants, expand membership at query time) trades Materialize's index freshness problem for a 30-second expansion cache, and keeps the fan-out out of the index entirely. Both are defensible; the token approach costs less when group churn is high, which enterprise group churn is.

On filtered ANN, the design's two specific claims check out against engine documentation. OpenSearch's efficient k-NN filtering (since 2.9) does exactly what Q4 relies on: it applies the filter during graph traversal and switches to exact search when the filtered set is small, with published recall above 0.99 — the sparse-access user really is the good case. And the reason the design carries its own acl_tokens field instead of using the engine's built-in document-level security is visible in that doc: DLS attaches queries to roles, and per-user ACLs across eight sources don't compile to a manageable role set. The dedicated-engine world made the same call from the other direction: turbopuffer's native filtering (the engine behind Notion's and Cursor's search) couples attribute bitmaps into the clustering index itself because, as they put it, the pre-filter and post-filter plans are both bad — >90% recall at 25 ms under complex filters. Everyone who ships this converges on filter-during-traversal plus an exact-search escape hatch.

Updates from post-training information

One mechanism in the Q4 answer is now dated. I handled the medium-selectivity band — where ANN with a restrictive filter under-recalls — by "scaling ef_search with estimated filter selectivity." Lucene merged a variant of ACORN-1 in February 2025, which handles filtered traversal by exploring only filter-passing nodes (extending to neighbors-of-neighbors where the graph gets sparse), and Elasticsearch 9.1 turned it on by default in July 2025 with ~5× speedups on filtered vector search, alongside BBQ quantization as the default for 384+-dim vectors. Two adjustments: the GA benchmark from Q6 should compare OpenSearch's efficient filtering against Elasticsearch's ACORN path on our worst-case 5,000-term ACL filters before the engine choice is final — this materially strengthens the Elasticsearch side of a choice I made on team familiarity — and BBQ-style quantization changes the Q6 page-cache math (the ~1.8 TB of hot vectors could shrink well under 100 GB), which could pull the 60–100 node estimate back down. The permission architecture is unaffected; this is a retrieval-engine update only.