Automated Customer Support Agent

This is my walkthrough of a 60-minute design for a multi-tenant support agent platform, written the way I'd present it: requirements first, then the estimates that actually shape the architecture, then the design, then failure modes and tradeoffs. It's one reasonable answer, not the canonical one.

Requirements I'm prioritizing

Functional, in order of what I'd build first:

1. Customer chats over web; the agent answers from the business's knowledge base and the customer's account data.
2. The agent asks follow-up questions when it lacks information.
3. The agent performs a small set of actions (check order, cancel order) against each business's existing systems.
4. Escalation to a human, and human takeover at any moment — including mid-agent-turn.
5. Businesses manage their KB, tools, and policies; updates go live in ≤5 minutes.

Non-functional, the ones that drive decisions:

• Tenant isolation. 10,000 businesses share the platform. Leaking one tenant's KB chunk or one customer's order into another conversation is the worst failure this system can have — worse than downtime. Every design choice below gets checked against this first.
• Correct actions. A wrong answer is embarrassing; a wrong cancellation is money. Actions need authorization independent of the model, idempotency, and an audit trail.
• Latency. First streamed token in ~1–2s, full answer in ~5–10s. Chat users tolerate a "typing" indicator; they don't tolerate silence.
• Availability of the takeover path. The LLM path can degrade; the "talk to a human" path must not. I'd target 99.9% for messaging and design so model-provider outages degrade to human handoff rather than to errors.
• Retention: conversation history queryable for one year.

Assumptions I'm adding: customers are authenticated by the business's own site (we receive a signed identity token, we don't run login); order/customer systems are reachable via per-business connectors (REST APIs the business exposes or pre-built integrations like Shopify); the action set is small and schema-defined per business, not open-ended; text only.

Estimates that shape the design

• Messages: 2M conversations/day × 10 messages = 20M messages/day ≈ 230/s average, ~4,600/s at 20× peak. Small for a message store; any partitioned database handles this.
• Agent turns: roughly half the messages are from customers, so 115 turns/s average, **2,300 turns/s at peak**. Each turn is 1–3 model calls (tool loop), so peak model traffic is thousands of concurrent LLM requests. This — not the database — is the bottleneck and the cost center.
• Tokens: at ~3K input tokens per call (system prompt + history + retrieved chunks + tool schemas), 10M turns/day × ~2 calls ≈ 60B input tokens/day. This number is why the design needs prompt caching, context truncation, and a cheap model for routing. Model spend will dwarf all infrastructure spend combined.
• Connections: 100K concurrent conversations ⇒ ~100–150K open WebSocket connections (customers plus human agents). ~30 gateway nodes at 5K connections each. Routine.
• Vector index: 50M chunks. At 1,024 dims float32 that's ~200GB of vectors plus graph overhead; with int8 quantization ~50–70GB. Fits on a sharded cluster of a handful of nodes — big enough to shard, small enough to keep in memory.
• Storage: 20M messages/day × ~1KB ≈ 20GB/day ≈ 7TB/year of conversation history, plus agent traces (prompts, tool calls) which are several times larger. Hot store for recent, object storage for the rest.

The takeaway from the numbers: this is a moderate-scale distributed system attached to a very large LLM workload. I'll spend my complexity budget on the agent loop, retrieval freshness, and tenant/action safety, and keep the messaging plumbing boring.

Architecture

flowchart TB
    subgraph Clients
        W[Customer web chat widget]
        H[Human agent console]
        B[Business admin / KB sources]
    end

    subgraph Edge
        GW[API gateway<br/>auth, rate limits]
        WS[WebSocket gateway<br/>~100K conns]
    end

    subgraph Core
        CS[Conversation service<br/>state machine, message append]
        Q[(Message queue<br/>partitioned by conversation)]
        ORCH[Agent orchestrator<br/>tool loop, streaming]
        MG[Model gateway<br/>multi-provider, budgets, failover]
        ACT[Actions service<br/>tool registry, authz, idempotency]
        RET[Retrieval service<br/>hybrid search, tenant filter]
        ESC[Escalation service<br/>routing, agent queues]
    end

    subgraph Data
        MDB[(Message store<br/>partition: conversation_id)]
        RDS[(Config DB<br/>tenants, tools, policies)]
        RED[(Redis<br/>active conv state, presence)]
        VIX[(Vector + keyword index<br/>50M chunks, sharded)]
        S3[(Object storage<br/>archive, traces)]
        AUD[(Audit log)]
    end

    subgraph Ingestion
        ING[KB ingestion pipeline<br/>chunk, embed, upsert ≤5 min]
    end

    subgraph External
        LLM1[Provider A]
        LLM2[Provider B]
        BIZ[Business order/customer APIs]
    end

    W --> GW --> WS
    H --> GW
    WS <--> CS
    CS --> MDB
    CS --> RED
    CS --> Q --> ORCH
    ORCH --> MG
    MG --> LLM1
    MG --> LLM2
    ORCH --> RET --> VIX
    ORCH --> ACT --> BIZ
    ACT --> AUD
    ORCH -->|stream tokens| CS
    ORCH --> ESC
    ESC --> H
    B --> ING --> VIX
    CS --> S3
    RDS --- ORCH
    RDS --- ACT

Data flow: one customer message

1. Customer sends a message over WebSocket. The gateway authenticates the session (a JWT the business's site minted, carrying tenant_id and customer_id) and forwards to the conversation service.
2. The conversation service appends the message to the message store with a per-conversation sequence number, acks the client, and publishes a turn_requested event to a queue partitioned by conversation_id — so turns for one conversation are processed in order, and one slow conversation can't block others.
3. An orchestrator worker claims the event and checks the conversation's control state in Redis. If a human has taken over, it drops the turn — the human replies instead.
4. The orchestrator builds context: business system prompt and policies from config, recent messages (older history compressed into a running summary), customer profile fetched through the actions layer, and KB chunks from retrieval — hybrid keyword + vector search, always filtered by tenant_id, top ~10 chunks after reranking.
5. It calls the model gateway. The model either answers, asks a follow-up, requests a tool call, or requests escalation (escalation is itself a tool). Tool calls go to the actions service; results feed back into the loop, capped at ~5 iterations and ~30s.
6. Tokens stream back through the conversation service to the customer's socket. Before committing the final message (and before executing any mutating tool), the orchestrator re-checks the control flag — this is the gate that makes mid-turn human takeover clean.

The pieces that deserve their design time

Conversation service and takeover

Each conversation is a small state machine: agent_active → human_active → resolved/expired, with a control field (agent | human) that a human can flip at any time from the console. Takeover does three things: sets the flag in Redis and the durable store, cancels any in-flight orchestrator turn for that conversation (the worker checks the flag at every commit point — before each tool execution and before sending), and pushes the full transcript plus an auto-generated summary to the human's console. Escalation is the same mechanism initiated by the agent: it enqueues the conversation into the business's agent queue (routed by skill/language), posts a "connecting you to a person" message, and the customer keeps the same chat session — the handoff is invisible at the transport layer.

Why a control flag checked at commit points, rather than trying to kill the LLM call instantly? Because the race is unavoidable — the model may already be mid-generation when the human clicks takeover — and what actually matters is that no agent message lands and no action executes after takeover. The flag check at the two commit points guarantees exactly that, cheaply.

Model gateway

One internal service fronts all external providers. It holds provider keys, enforces per-tenant token budgets and rate limits, meters usage for billing, and handles retries and failover. Routing policy: a small/cheap model for intent classification and the "does this even need retrieval?" decision; the strong model for answer generation and tool use. Prompt structure puts the static parts first (platform system prompt, then per-business instructions and tool schemas, then history) so provider-side prompt caching hits — with 60B input tokens/day, cache hit rate is a first-order cost lever, plausibly a 50–80% reduction on input cost.

Failover ladder when a provider degrades: retry once → switch provider (we keep prompts provider-portable and pre-provision capacity on at least two) → if both are down, the agent stops pretending: the conversation service sends a templated "we're connecting you with a person / we'll email you" message and escalates. The chat, takeover, and human paths share none of the LLM dependency chain, so a total provider outage turns the product into a plain human-support chat instead of an error page.

Retrieval and the 5-minute freshness requirement

Ingestion: businesses connect sources (help-center crawls, uploaded docs, APIs). A pipeline normalizes → chunks (~500 tokens, overlapping, with heading metadata) → embeds → upserts into the index, with tombstones for deleted docs. The 5-minute SLA is what forces this to be a streaming upsert pipeline rather than a nightly index rebuild: change events flow through a queue, and embedding a changed document is seconds of work. The SLA is met per-document; a business re-uploading its entire 100K-chunk KB at once gets eventual catch-up, not 5 minutes, and I'd state that openly.

Index layout: one shared index, sharded by tenant_id, with the tenant filter applied inside the engine (filtered HNSW / partition routing), not as a post-filter. Per-tenant indexes would give hard isolation but 10,000 mostly-tiny indexes are an operational mess; a shared post-filtered index risks cross-tenant leakage if a filter is ever dropped. Sharding by tenant gives physical locality, and I'd add a belt-and-suspenders check: the retrieval service verifies tenant_id on every returned chunk before handing it to the orchestrator. The few whale tenants get dedicated shards.

Search is hybrid — BM25 plus vectors, fused, then a lightweight reranker — because support queries are full of exact strings (SKUs, error codes, plan names) that pure embedding search misses.

Actions service: the model proposes, this service disposes

The security stance: the model is untrusted input handling untrusted input. KB content and customer messages can both contain prompt-injection attempts, so nothing the model says is treated as authorization. The actions service independently enforces:

• Scope. Tool credentials are per-tenant; every call is bound to the customer_id from the session JWT. The model physically cannot request order #123 for a different customer — the customer ID comes from the session, not from model output.
• Policy. Per-business rules in config: which tools are enabled, caps (e.g., auto-cancel only within 1 hour of order placement, refunds under $50), and which actions require an explicit customer confirmation message before execution. Anything outside policy returns "escalate" to the model.
• Idempotency. Mutating calls carry an idempotency key (conversation_id:turn:action_seq), so a retried turn can't cancel an order twice.
• Audit. Every tool invocation — who, what, arguments, result, which model turn requested it — goes to an append-only audit log. When a business asks "why did the bot cancel this order," the answer is a query, not archaeology.

Connectors are the multi-tenant reality tax: pre-built adapters for the big platforms (Shopify, Stripe, Zendesk) plus a generic "bring your OpenAPI spec

• credentials" path where the business maps our canonical tools (get_order, cancel_order, get_customer) onto their endpoints.

APIs (external, abbreviated)

POST /v1/conversations                     → {conversation_id, ws_url}   # customer JWT
WS   /v1/ws?token=…                        # bidi: send message, receive tokens/events
GET  /v1/conversations/{id}/messages?after=seq   # history / reconnect backfill

POST /v1/conversations/{id}/takeover       # human console
POST /v1/conversations/{id}/messages       # human reply
POST /v1/conversations/{id}/release        # hand back to agent

PUT  /v1/kb/sources/{id}                   # register/update a KB source
POST /v1/kb/documents                      # push a document (upsert)
PUT  /v1/tools/{name}                      # tool config: schema, mapping, policy
GET  /v1/analytics/…                       # deflection rate, CSAT, escalations

WebSocket for the chat because it's bidirectional (typing indicators, human takeover events pushed to the client), with SSE + polling fallback for restrictive networks. Reconnects resume via after=seq — sequence numbers make the client idempotent to redelivery.

Storage

Consistency: the message append is the source of truth and is strongly consistent per conversation (single partition). Everything downstream — queue, orchestrator, analytics — is at-least-once, made safe by sequence numbers and idempotency keys. I'd rather re-process a turn than lose one.

Scaling and failure handling

• Stateless where it counts. Gateways, conversation service, and orchestrator workers are all stateless and horizontally scaled; state lives in Redis and the message store. The 20× peak is absorbed by autoscaling workers against queue depth.
• Backpressure. If the queue backs up (peak + a slow provider), the system degrades in order: shed the cheap-model preprocessing, shrink retrieval depth, then show "the assistant is busy, a person will follow up" and enqueue for humans/async email. Latency degrades before correctness does.
• Poison turns. A turn that fails 3 times goes to a dead-letter queue and auto-escalates the conversation — a customer must never be stuck talking to a crashed worker.
• Noisy tenants. Per-tenant rate limits and token budgets at the gateway and model gateway; one business's product launch can't starve the other 9,999.
• WebSocket node loss. Clients reconnect to any node and backfill from seq; conversation state is in Redis/DB, not on the socket node.
• Region failure. Active-passive per region to start: async-replicated message store, Redis rebuilt from the durable store on failover, vector index rebuilt from the ingestion source of truth (document store), RTO in minutes. Active-active is a later investment; support chat tolerates a brief failover better than it tolerates the complexity of multi-master message ordering.
• Business API down. Tool calls time out at ~5s with a circuit breaker per connector; the model is told the tool failed and instructed to offer escalation instead of guessing order status.

Security and safety

• Tenant isolation enforced at every layer: tenant_id in every partition key, every retrieval filter, every credential lookup — and asserted again at the retrieval and actions boundaries, so a single dropped filter is a detected error, not a breach.
• Customer identity comes only from the business-signed JWT; model output is never an authorization input (see actions service above).
• Prompt injection: retrieved chunks and tool results are wrapped as clearly-delimited untrusted content; the blast radius of a successful injection is bounded by the tool policy layer, which is the real defense.
• PII: encrypted at rest and in transit, per-tenant encryption keys, deletion API for GDPR-style requests, hard delete at 1-year retention. Contracts with model providers must include no-training-on-data terms; optionally redact obvious PII (card numbers) before prompts leave the platform.
• Every human takeover, agent action, and admin config change is audited.

Tradeoffs I'd call out explicitly

• Async turn processing vs. request/response. The queue between message ingestion and the orchestrator adds ~tens of ms and some machinery, but buys ordered turns, retry/DLQ semantics, clean takeover cancellation, and backpressure. For a 5-second LLM turn, the added latency is noise. I'd take this trade every time.
• Shared filtered index vs. per-tenant indexes. I chose shared-sharded with defense-in-depth filtering. The honest cost: isolation is enforced by software, not physics. Businesses with regulatory needs get the dedicated- shard tier.
• Two providers vs. one. Provider portability costs real engineering (prompts tuned per model, eval suites run per provider) and dilutes prompt-cache efficiency. I still take it, because the estimates say model availability is product availability, and 2,300 turns/s of peak capacity is not something you want to source from a single vendor on outage day.
• Aggressive automation vs. escalation. I'd tune the agent to escalate early on low confidence and on any policy-boundary action. Deflection rate is the metric businesses buy on, but one confidently wrong cancellation costs more trust than ten escalations. Start conservative, loosen per tenant as eval data accumulates.
• What I skipped for the hour: voice/email channels, fine-tuning vs. prompting, the eval/offline-testing pipeline (in production this deserves a whole design of its own — you can't safely change a prompt serving 10,000 businesses without one), and billing.