Hacker News front pageSep 26, 2026, 02:33 UTC+8

Ollaya runs open-source decision models locally with single-pass, sub-10ms latency

Ollaya – Ollama for open-source, Jev-style decision models

Ollaya is a local runtime for open-source decision models—think Ollama but for classification and scoring. It produces answers in a single forward pass with no token-by-token generation. A five-question request to the Laya model on an RTX 4090 takes about 8–10 ms end-to-end. Weights are pulled directly from Hugging Face, pinned to a commit and sha256-checked; the runtime uses ONNX Runtime and listens on 127.0.0.1 by default. It speaks TypeSafe's /v1/systemone API, so the TypeSafe Python SDK 0.7.1 works unchanged against a local server. Four model families are available: Laya, decider, nli, and gliclass, ranging from 322M to 1.9B parameters, covering English and 100+ languages. The post does not disclose training data sources or fine-tuning details. Desktop apps cover macOS, Windows, and Linux; a Docker image is also provided. GPU acceleration requires NVIDIA driver R580 or newer.

Why it matters: Ollaya packages classification/scoring models as an Ollama-style local tool — clean concept, solid latency data (8-10ms vs 200ms+ for hosted APIs). But it's early beta with no model ecosystem or real-world deployment stories yet, so it lands at the 72 featured threshold.

Read the original ↗Export Markdown

Take

This caught my eye because it turns decision models into a local runtime you can pull and run like Ollama. You feed it text and a set of typed questions—classification, scoring, yes/no—and it answers in a single forward pass, no token generation. On an RTX 4090, a five-question request to Laya takes 8–10 ms end-to-end, roughly an order of magnitude faster than TypeSafe's hosted Jev API.

Four model families ship out of the box: Laya (322M–421M, English and 100+ languages), decider (Qwen3.5-based, 0.75B–1.9B), nli (zero-shot classifiers, 396M–435M), and gliclass (439M). Weights come straight from Hugging Face, pinned to a commit and sha256-checked. The runtime uses ONNX Runtime and listens on 127.0.0.1 by default.

I'd discount it a bit until we see more. The post doesn't disclose training data or fine-tuning details, and there are no public benchmark numbers to compare against. The decider models run at 155–190 ms on the same 4090—fine for some tasks, but a big gap from Laya's 8 ms. The API compatibility is with TypeSafe's /v1/systemone endpoint, so the ecosystem is narrow for now.

If these models hit your accuracy bar on the classification tasks you actually care about, the real win is keeping sensitive data local and skipping per-call API fees. But you'll need to test precision and generalization yourself.

Source: Hacker News front page · ollaya.dev
Published: Sep 26, 2026, 02:33 UTC+8 · 4 days ago