Skip to content
Hacker News front page

GLM built its own inference infra on 100k+ Chinese accelerators, tripling throughput in under two weeks

GLM Built Its Own Inference Infrastructure

Zhipu AI disclosed how GLM-5.3-Flash inference was built from scratch on a cluster of over 100,000 Chinese-made AI accelerators. The team faced limited chip memory, low bandwidth, and an immature software ecosystem. Instead of relying solely on human engineers, they deployed an Infra Agent powered by GLM-5.3 that turned sparse end-to-end metrics into fine-grained, attributable feedback—kernel-level correctness checks, microbenchmarks, and execution traces—so the agent could pinpoint bottlenecks. Combined with tensor parallelism, W8A8 quantization, mixed-precision KV cache, and an Encode-Prefill-Decode disaggregated architecture, end-to-end throughput improved roughly 3× over the initial baseline, with per-token cost reaching parity with mainstream NVIDIA GPUs. Within a week of launch under the anonymous name Ox-Alpha, the model processed over 62 trillion tokens and became the most-used model on both OpenCode and OpenRouter.

Why it matters: Zhipu built GLM-5.3-Flash's production inference service from scratch on over 100,000 domestic accelerators, with much of the debugging done by a GLM-driven Infra Agent.

Read the original ↗Export Markdown