What tuning does for chat, voice and your bill.
Inside makes open models feel native to consumer apps: they stay in character, answer before a pause feels long, and cost less to run. Here is what we have measured on each, and how.
- Same harness
The baseline is the stock engine on the same card, checkpoint and traffic.
- Real traffic
Replays of the conversation lengths and arrival patterns customers send.
- Quality gates
A change is kept only if it holds quality against the reference.
- Rounded down
Ratios are computed from raw measurements and rounded toward zero.
Stays in character.
Users push characters to break: they argue, flatter, change the subject. A tuned model holds the persona where a frontier API with its best prompt gives in.
- 5.7%replies that break character, tuned Qwen3.8 27Bfrontier API 58.9% · same model untuned 71.4%
- $0.049cost per chatfrontier API $0.344, 7× more
How it was measured
- Test
- 8 unseen characters × 3 pressure scenarios, 192 five-turn windows, English
- Judge
- the frontier model grades its own replies; a second, independent judge rates both lower (28.6% vs 2.1%)
- Serving
- 2× RTX 5090; API cost at uncached list price, GPUs at $0.70 per hour
- Measured
- 2026-10-16
Answers before the pause feels long.
In a voice call every turn re-sends the whole conversation. We cut the time to the first word on long calls, and on one B200 a customer’s own load test came in under their target.
- 62 msto first token, 30 calls at once on one B200server side · the customer’s target was 192 ms
- 2.5×faster first token on a 40K-token call vs stock145.4 → 56.3 ms, Gemma 4 31B
- 23 msto first audio, speech-to-speech modelfrom 213 ms on stock, Qwen3 Omni 30B
How it was measured
- B200 test
- the customer’s client-side TTFT p50 was 197 ms (10 calls × 5K tokens), 248 ms (30 calls × 5K tokens), 210 ms (19 calls × 12K tokens), 238 ms (4 calls × 40K tokens), including a ~130–140 ms network round trip
- Stock vs ours
- trace replay of voice calls on 1× RTX 5090 (145.4 → 56.3 ms) and 1× H100 (205.1 → 86.9 ms), same checkpoint and flags, GSM8K-200 within tier of stock
- Speech
- 1× RTX 5090, one request at a time, word error rate 1.59% → 1.01%
- Measured
- 2026-10-06 to 2026-11-03
Costs less to run.
The same model, served for less: fewer GPUs for the same peak, less GPU time per request, and a lower price per request than the APIs you would otherwise pay.
- −65%GPUs for 2,200 live role-play chats~400 → 138, same quality · read the case study →
- −35%cost per request vs OpenRouterDeepSeek V3.2, character chat, reserved node
- −51%GPU time per request vs stock25.57 → 12.33 GPU-seconds, Qwen3.8 27B
How it was measured
- Fleet
- Gemma 4 26B A4B (FP8) on RTX 5090, at the app’s target (p50 ≤ 3.8 s, p95 ≤ 7 s); role-play NLL within noise of stock FP8. Details in the case study
- Price
- DeepSeek V3.2 (NVFP4) on 8× B200 at $41/h and 70% use; −26% vs DeepSeek’s list price. At spot rates the node costs more than list
- GPU time
- 1× RTX 5090, 173 long-context task requests, 8 in flight, against stock SGLang with its published RTX 5090 NVFP4 recipe; task quality within 2 points of the FP8 two-GPU reference
- Measured
- 2026-10-08 to 2026-10-22
Measured on RTX 5090, H100, H200, B200 and MI350X. Results depend on the model, the traffic and the hardware; we measure on yours before you change anything.
Want these numbers on your traffic?
Send us your model, traffic shape and latency target. We measure on your workload, with your quality bar, before you change anything in production.
Talk to us