Skip to content
r/LocalLLaMA

Truespar open-sources Paddock, a Rust/C++ inference engine with custom CUDA kernels under MIT/Apache-2.0

We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)

Truespar open-sourced Paddock, its in-house inference engine written in Rust and C++ with custom CUDA kernels. One binary serves both OpenAI and Anthropic-style APIs, loading GGUF and safetensors. The team runs ~300B tokens/year through it. On an RTX PRO 6000 with Qwen3.8-27B FP8, it beats vLLM by 2–19%, mostly beats SGLang but loses in two cells, and claims 1.5–37x over llama.cpp Q8_0—though a commenter flagged that the llama.cpp benchmark used an unusual KV allocation, so I'd discount the 37x figure. CUDA-only on Windows and Linux, validated on Blackwell and Ampere; Ada kernels exist but lack a validation board, so the engine refuses to start without an env var. No Mac, ROCm, Vulkan, or tensor parallelism—one model per GPU.

Why it matters: Truespar open-sourced its in-house inference engine Paddock under MIT or Apache-2.0, CUDA kernels included, with published benchmarks ahead of vLLM and llama.cpp in some cases.

Read the original ↗Export Markdown