Skip to content
AI HOT (Curated Pool)

Hugging Face transformers now runs GGUF quantized models directly

transformers 支持直接加载 GGUF 量化模型,本地推理性能接近 llama.cpp

transformers now loads GGUF files natively, with local inference speed close to llama.cpp. You can use from_pretrained to load a GGUF checkpoint and run models like Qwen3.5 on a Mac. It reuses llama.cpp's ggml kernels under the hood, with initial optimization targeting Apple Silicon. Only the Qwen3.5 architecture is supported for now; more models and features are coming.

Why it matters: Hugging Face added GGUF support to transformers, so users can pick a checkpoint from the Hub and load it with from_pretrained, with local inference close to llama.cpp.

Read the original ↗Export Markdown