Preprint: agent-aware serving cut P95 time to first token up to 14x versus Nvidia Dynamo
Researchers report in a preprint that Evolving Infra moves nodes between prefill and decode and keeps KV caches agents will reuse. On up to 256 GB300 GPUs, it cut mean latency up to 4.1x and P95 TTFT up to 14x versus Dynamo.
Why it matters: If you serve your own models for agents, check whether your stack evicts context the agent will need again; the authors argue generic serving signals miss how agents reuse it.
From Drop #087 · Sun, Nov 15, 2026 · Morning