2,200 live chats on 138 GPUs instead of ~400.

Characters with long persona prompts, multi-turn conversations, four languages. Each request carries the character card and the conversation so far: about 5.5K tokens on average, ~9K at p95, with replies of up to 300 tokens. At peak 2,200 requests are in flight at once. The app’s first priority is cost per conversation; its latency target is a median reply under 3.8 s and p95 under 7 s.

On a 32 GB card the FP8 model weights take 26.2 GB, which leaves about 0.6 GB for conversation state: room for roughly three conversations. Past two in flight, the stock engine starts evicting conversations and recomputing them from scratch, its cache hit rate falls to zero, and throughput stops growing. So each GPU holds about two chats, and 2,200 chats need about 1,100 GPUs. The bottleneck was memory, not compute.

1. 01
   Make room for conversations

   Capped how many requests run at once, served the text-only model without its unused vision tower, stored the shared embedding table once in FP8, and retuned the memory split between the model’s sliding-window and full-attention layers.

   conversations per batch 17 → 29
2. 02
   Keep each conversation warm in host memory

   A 12 GB tier of pinned host memory per GPU holds conversation state the card can’t, so a returning turn reloads it (an estimated ~5 ms) instead of recomputing ~4K tokens (~300 ms). It needed two fixes before it was safe under load.

   chats per GPU 24 → 28 at p95 ≤ 10 s
3. 03
   Kernels for the batch sizes chat actually makes

   Decode runs at small batch sizes, where general-purpose kernels waste most of the memory bandwidth. We wrote FP8 matrix kernels and a router kernel for small batches and tuned the mixture-of-experts tiles for this card.

   one decode step 9.13 → 5.53 ms; +16–22% requests/s on the full stack
4. 04
   Schedule for reuse

   Requests that share the longest cached prefix go first, and long prompts are prefilled in 2K-token chunks so they don’t stall replies already streaming.

   p90 latency −7% at 32 chats in flight

Fleet size is the peak divided by the chats one GPU holds inside the target. The binding constraint is the median: 300 tokens of decoding alone takes about 3 s, so median headroom runs out before p95 does.

Every change had to pass a gate against stock FP8 serving of the same model before it was kept: on GSM8K (all 1,319 questions) and 40 tool-call JSON cases, the paired 95% confidence bound could drop at most 1 point; on role-play, the model’s likelihood of stock’s own replies could move by at most 0.02, with language, termination and looping counted separately. The final stack scored GSM8K 96.59%, tool-call JSON 40/40, and role-play NLL within noise of stock FP8.

• Two GPUs per replica: these cards have no peer-to-peer link, so a 4K prefill ran 3.3× slower; one GPU per replica served more per GPU.
• Speculative decoding: at the app’s sampling temperature a draft-and-verify round cost 2.1–2.2× a plain step, so replies got slower.
• One shared memory pool for all layers: it crashed the scheduler once prefix caching was on.
• Releasing the slid-out attention window: it freed memory the very next turn needed, and capacity fell.

One RTX 5090 replaying role-play traffic, in closed loop at fixed concurrency and in open loop with random arrivals, at increasing load until the target broke. Stock and optimized runs used the same card model, checkpoint and traffic, in separate sessions. Measured 2026-10-21.

Limits: turns ran back to back, so every conversation’s prefix stayed hot. How well the host-memory tier holds up across real users’ pauses between turns was not measured, and the fleet figure assumes traffic spreads evenly across replicas.