Self-hosted multi-GPU LLM serving: scheduling bottlenecks and cost
What happened
On October 16, Computing Life published Yage's hands-on test of a self-hosted multi-GPU LLM service: a host with three RTX 5090 cards, each running a 4-bit quantized Qwen3.8-27B. The test found that cache-priority routing pushes agent branches onto the same GPU. The report says NInfer cannot give the scheduler real cache and in-flight load information, which limits coordination between cache reuse and load balancing. Low utilization also keeps the daily cost of this self-hosted setup above using an API.
Written by AI from the coverage · updated 3 hours ago
Coverage
Follow the reports to see the story from different sides.
- Computing Life · Share · Yage自建大模型：多卡为什么卡，以及为什么还是比 API 贵
鸭哥实测自建三卡大模型服务发现，缓存优先路由会让智能体分支挤到同一张卡，低利用率也使日常成本高于 API。主机用 3 张 5090 分别运行 4-bit 量化的 Qwen3.8-27B，NInfer 无法向调度器提供真实缓存与在途负载，限制了缓存复用和负载均衡的协调。
Heat over time
Not enough continuous observations to draw a trend yet.