r/LocalLLaMAJun 9, 2026, 08:28 UTC+8

New MLX LM Server From Apple

A Reddit post says Apple’s MLX LM Server uses continuous batching for concurrent sub-agent requests and supports distributed inference across multiple Macs via Thunderbolt RDMA.

Why it matters: HKR-H/K/R all pass, but the item is based on a Reddit summary and lacks throughput, latency, model-size, or release details. Treat it as a mid-weight Apple/MLX inference update, just above the featured threshold.

Read the original ↗Export Markdown

Take

Apple MLX LM Server should be treated as a high-risk but important signal: the title claims continuous batching and Thunderbolt RDMA, while the captured body is just a Reddit 403. No version, throughput, latency, or model size is disclosed. If the claim holds, Apple is not merely copying vLLM. It is filling the missing server layer for local agent workloads, where concurrent sub-agent calls punish naive single-request serving.

I still have doubts about the multi-Mac story. Thunderbolt bandwidth and topology can eat the gains fast, especially once KV cache traffic and 70B-class models enter the path. NVIDIA plus vLLM still owns the serious serving stack. But if Apple ships this inside MLX, Mac Studio and Mac mini clusters stop being hobby demos and become a plausible CUDA-free local inference lane.

Source: r/LocalLLaMA · reddit.com
Published: Jun 9, 2026, 08:28 UTC+8 · Jun 9, 2026