Evolving Infra:可隨代理工作負載自適應的大規模 LLM 服務架構
Evolving Infra: Agent-Adaptive LLM Serving at Scale
研究速讀
這篇論文提出 Evolving Infra,一套能依代理工作負載在執行時調整的 LLM 服務框架。作者指出,現有的預填解碼分離式服務系統在運算分配與 KV 快取管理上缺乏彈性,且所用訊號過於粗略,導致預填與解碼節點間負載不均,並使代理之後會重用的 KV 快取被逐出。此框架將運算分配與 KV 快取管理解耦,讓節點能在預填與解碼間有效率地切換,並以線上規劃器與代理感知的分散式 KV 管理器進行調度。
值得留意
- 作者主張,現有 PD 分離式服務系統對執行時調整運算分配與 KV 快取管理的彈性有限,且依賴的粗略訊號只能部分反映多樣且動態的代理工作負載。
- Evolving Infra 將運算分配與 KV 快取管理解耦,使節點能有效率地在預填與解碼之間切換,並以線上規劃器依工作負載組成調整 PD 分割比例。
- 作者報告,在程式設計、數學與電腦操作任務上,使用 GLM-5.2 與 Kimi K2.6、最多 256 張 GB300 GPU,相較於 NVIDIA 的 Dynamo,平均端到端延遲與 P95 TTFT 分別最多降低 4.1 倍與 14 倍。
閱讀限制
僅閱讀標題與摘要,無法確認實驗設定、基準比較細節、統計方法或是否有同儕審查,上述效能數字均為作者自述,未經獨立驗證。
原始摘要 [EN]
Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload. These limitations lead to load imbalance between prefill and decode nodes and eviction of KV caches that agents will reuse. We present Evolving Infra, an agent-adaptive serving framework that adapts compute allocation to agent workloads and KV cache management to agent context management at runtime. We decouple compute allocation from key--value (KV) cache management, allowing nodes to switch efficiently between prefill and decode. Building on this design, an online planner matches the PD split to workload composition, while an agent-aware distributed KV manager guides cache eviction and transfer with signals from agent context management. Across coding, math, and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, Evolving Infra reduces mean end-to-end request latency and P95 TTFT by up to 4.1x and 14x over Dynamo, NVIDIA's production framework.
AI 摘要僅根據公開標題與摘要整理;研究結論來自作者,請以原文為準。