Skip to content
Hacker News front page

Mesh LLM pools GPUs across machines into one OpenAI-compatible inference mesh

Mesh LLM: distributed AI computing on iroh

The n0 team open-sourced Mesh LLM, which pools GPUs from multiple ordinary machines to run large models and exposes an OpenAI-compatible API at localhost:9337/v1. Requests can run locally, route to a peer with the model loaded, or split a model across machines in a layer-wise pipeline. Underneath, iroh handles NAT traversal and direct QUIC connections with no central server. Over 40 models are included, from 0.5B to 235B MoE. The post doesn't disclose inference latency or throughput numbers, so I'd hold off on production assumptions.

Why it matters: The n0 team's open-source Mesh LLM pools GPUs and memory across ordinary machines to run large models, serving an OpenAI-compatible endpoint locally with requests run locally, forwarded, or split layer by layer into a pipeline.

Read the original ↗Export Markdown