Skip to content

#Hugging Face

3 today

May 11Monday

r/LocalLLaMA

MTP benchmark results: task type determines speculative inference speedups or slowdowns

A Reddit LocalLLaMA user ran 300+ tests on Qwen 3.6 27B MTP quants, finding coding draft acceptance at 79-89% and F16 coding speed up 171%, while Q4_K_M creative writing slowed down 9%.

Why it matters: More than 300 runs on a quantized Qwen 3.6 27B show speculative decoding's speedup depends on the task, not the setup.

May 10Sunday

r/LocalLLaMA

NVIDIA AI Releases Star Elastic: One Checkpoint Contains 30B, 23B, and 12B Reasoning Models

NVIDIA AI released Star Elastic, a single checkpoint that can zero-shot slice 30B, 23B, and 12B reasoning models in BF16, FP8, and NVFP4; when the 23B submodel handles thinking and the 30B model handles final answers, reported accuracy rises 16% and latency drops 1.9× on AIME-2025, GPQA, LiveCodeBench v5, and MMLU-Pro.

Why it matters: NVIDIA's Star Elastic slices 30B, 23B and 12B reasoning models from one checkpoint with no extra training, the 23B thinking and the 30B answering.

May 9Saturday

AI HOT (Curated Pool)

EMO: Expert Mixture Models for Emergent Modular Pretraining

AllenAI introduced EMO, a mixture-of-experts model with 14B total parameters and 1B active parameters, trained on 1 trillion tokens and able to use only 12.5% of its experts for specific tasks while retaining near-full-model performance.

Why it matters: EMO lets experts emerge from 1 trillion tokens instead of assigning them by subject, and activates only 12.5% of them at run time while nearing full-model quality.

May 8Friday

r/LocalLLaMA

WARNING: Open-OSS/privacy-filter Malware

A Reddit user says Hugging Face repo Open-OSS/privacy-filter is an infostealer. It mimics OpenAI's privacy filter, uses loader.py to fetch PowerShell, then downloads an EXE and runs it via Task Scheduler. The author says they reported it to Microsoft and Hugging Face; the post says Linux is unaffected.

Why it matters: A Hugging Face repo named Open-OSS/privacy-filter poses as an OpenAI privacy filter but uses loader.py to pull a PowerShell script and run a downloaded EXE, a supply chain risk for anyone running local models.

May 7Thursday

r/LocalLLaMA

Exaggerated PCI-E Bandwidth Concerns?

Reddit user ziphnor tested 2x RTX 5060 Ti 16GB with vLLM TP=2 and 32k-context prefill. PCIe peaked at 3–4 GB/s, about 40–50% of a PCIe 4.0 x4 link. Prefill reached ~840–850, 1500, and 1600–1700 t/s; the post does not disclose decode bandwidth.

Why it matters: User ziphnor ran vLLM tensor parallelism on two RTX 5060 Ti cards and measured PCIe peak bandwidth of only 3-4 GB/s during 32k-context prefill, 40-50% of the PCIe 4.0 x4 link.

May 6Wednesday

r/LocalLLaMA

2.5x Faster Inference with Qwen 3.6 27B Using MTP on 48GB

A llama.cpp PR adds MTP support for Qwen 3.6 27B, with a reported 2.5x inference speedup. The author measured 28 tok/s on a Mac M2 Max 96GB and shared GGUF builds, compile steps, and a 262144-context server command. The key detail is turbo4 4.25-bit KV cache: a 48GB Mac runs Q5_K_M at 262K context.

Why it matters: A llama.cpp PR adds MTP support for Qwen 3.6 27B, measured at 2.5x faster inference with 262k context on 48GB, a usable setup for local agentic coding.

r/LocalLLaMA

Gemma 4 MTP Released

Google released Gemma 4 MTP drafters with 4 Hugging Face checkpoints listed. MTP uses a smaller draft model to predict multiple tokens, then the target model verifies them in parallel, giving up to 2x decoding speedups with identical output quality.

Why it matters: Google posted four Gemma 4 MTP checkpoints on Hugging Face, where a small model guesses ahead and the main model verifies in parallel, doubling decode speed with unchanged output quality.

May 5Tuesday

r/LocalLLaMA

SenseNova-U1-8B-MoT open-source multimodal architecture draws LocalLLaMA discussion

SenseNova open-sourced SenseNova-U1-8B-MoT, an 8B native multimodal understanding and image-generation model. Its Hugging Face text says NEO-Unify removes VE and VAE, supports interleaved image-text generation, and high-density rendering; the post does not disclose test scores. The key question is whether the monolithic design yields reproducible gains.

Why it matters: SenseTime's open 8B multimodal model drops the vision encoder and VAE, putting image understanding and generation in one model, a different trade-off in multimodal architecture.

r/LocalLLaMA

Interactive Guide from Hugging Face Comparing RL Environments Across Frameworks

Hugging Face’s post-training team published an interactive guide comparing RL environment frameworks. The team spent one month building environments in verifiers, OpenEnv, Nemo-Gym, OpenRewards, and others, then trained models to study scaling. The post does not disclose benchmark scores, model sizes, or training costs.

Why it matters: Hugging Face's post-training team spent a month building the same RL environment in verifiers, OpenEnv, Nemo-Gym and other frameworks, then trained models to compare how each scales.

Apr 30Thursday

r/LocalLLaMA

Qwen-Scope: Official Sparse Autoencoders (SAEs) for Qwen 3.5 models

Qwen Team released Qwen-Scope, SAEs for Qwen 3.5 models from 2B to 35B MoE. It maps residual-stream features across all layers, including Feature #6159 for Chinese activation. The key point is feature-level debugging and steering; the license discourages removing safety filters.

Why it matters: Qwen open-sourced Qwen-Scope, sparse autoencoders spanning 2B to 35B MoE models with full residual streams, splitting tangled activations into separate features people can inspect and debug.

r/LocalLLaMA

inclusionAI/Ling-2.6-1T · Hugging Face

inclusionAI open-sourced Ling-2.6-1T on Hugging Face, with 1 trillion parameters. It uses MLA plus Linear Attention and Contextual Process Redundancy Suppression to reduce CoT overhead. The post cites AIME26 and SWE-bench Verified but does not disclose scores.

Why it matters: inclusionAI open-sourced the trillion-parameter Ling-2.6-1T on Hugging Face, using MLA and linear attention plus context process redundancy suppression, an architecture reference for anyone working on large-model inference optimization.

Apr 29Wednesday

r/LocalLLaMA

mistralai/Mistral-Medium-3.5-128B · Hugging Face

Mistral AI released Mistral Medium 3.5 128B on Hugging Face, with 128B dense parameters and a 256k context window. It supports text and image input, function calls, JSON output, and a Modified MIT License with exceptions for high-revenue firms. Reasoning effort is configurable as none or high per request.

Why it matters: Mistral AI put Mistral Medium 3.5, a 128B dense model, on Hugging Face with image and text input, function calling, adjustable reasoning effort and a 256K-token context window.

r/LocalLLaMA

XiaomiMiMo MiMo-V2.5: Sparse MoE with 310B total and 15B activated parameters

XiaomiMiMo shared MiMo-V2.5 with 310B total parameters and 15B activated parameters. The post only links Hugging Face and says it runs on more “human” configs than its larger sibling. It does not disclose VRAM needs, quantization, or benchmarks.

Why it matters: Xiaomi released the non-Pro MiMo-V2.5 on Hugging Face, a sparse MoE with 310B total parameters and 15B activated per inference, calling on large capacity at lower compute.

NVIDIA Blog

NVIDIA Launches Nemotron 3 Nano Omni for Vision, Audio, and Language Agents

NVIDIA launched Nemotron 3 Nano Omni, claiming up to 9x higher throughput at the same interactivity. It uses a 30B-A3B hybrid MoE with Conv3D, EVS, and 256K context, taking text, images, audio, video, documents, charts, and GUIs as input. Open weights, datasets, and training methods arrive April 28, 2026 on Hugging Face, OpenRouter, build.nvidia.com, and 25+ platforms.

Why it matters: NVIDIA open-sourced Nemotron 3 Nano Omni, one model handling image, text, audio and video with 3B active parameters and 256K context, giving teams building multimodal agents an open path.

Apr 24Friday

r/LocalLLaMA

DeepSeek releases V4: 1.6T Pro, 284B Flash, MIT license, 1M context

DeepSeek released two open-weight V4 models: Pro at 1.6T total with 49B active, and Flash at 284B total with 13B active; both use an MIT license and support 1M context. The RSS snippet points to a Hugging Face collection and a tech report, but the post does not disclose benchmark scores, pricing, training data size, or real inference throughput. The key thing to watch is the 1M context plus low active-parameter ratio; if evals hold, self-hosted long-context and routing economics change materially.

Why it matters: DeepSeek open-sources V4 in Pro and Flash versions under the MIT license, both with a 1M-token context window and only 49B and 13B active parameters, setting a new reference point for self-hosting long-context models.

Hugging Face Blog

DeepSeek-V4: a million-token context that agents can actually use

DeepSeek released V4 with two MoE checkpoints, Pro and Flash, both supporting a 1M-token context. Pro has 1.6T total and 49B active parameters; Flash has 284B total and 13B active. The key detail is KV cost: Pro uses 27% of V3.2 single-token FLOPs and 10% of its KV cache; Flash uses 10% and 7%.

Why it matters: DeepSeek-V4 comes in Pro and Flash, both with a million-token context, Pro at 1.6 trillion total parameters and 49 billion active, Flash at 284 billion total and 13 billion active.

Apr 23Thursday

QbitAI · WeChat

Qwen3.6-27B open-weights, beats its 397B flagship predecessor on agentic coding

Qwen released Qwen3.6-27B and says it beats Qwen3.5-397B on 4 agentic coding benchmarks with about 1/15 the parameters. The post cites SkillsBench rising from 30.0 to 48.2, GPQA Diamond at 87.8, and AIME26 at 94.1; it uses a dense architecture, Thinking Preservation, and Gated DeltaNet, with weights on Hugging Face and ModelScope.

Why it matters: Qwen released Qwen3.6-27B, a 27B dense model it says beats its own 397B previous flagship on four agentic coding benchmarks.

Hugging Face Blog

How to Use Transformers.js in a Chrome Extension

Hugging Face published a guide for a Transformers.js Chrome extension using Gemma 4 E2B. It defines three MV3 entry points: background service worker, side panel, and content script. The key design keeps local inference in the background and uses messaging plus a tool loop.

Why it matters: Hugging Face walks through the engineering trade-offs of running Gemma 4 E2B locally in a Chrome MV3 service worker.

Apr 22Wednesday

r/LocalLLaMA

ServiceNow-AI/SuperApriel-15B-Instruct · Hugging Face

ServiceNow released SuperApriel-15B-Instruct, a single-checkpoint 15B model with 8 deployment presets spanning 1.0× to 10.7× decode throughput at 32K sequence length. It has 48 decoder layers with 4 mixer variants per layer and up to 262K context positions depending on runtime; the key point is that speed-quality tradeoffs and speculative decoding are exposed from the same weights.

Why it matters: ServiceNow released SuperApriel-15B-Instruct, a 15B instruction model whose single weight set offers eight speed tiers, up to 10x faster decode throughput, so latency and quality trade off inside one model.

Hacker News front page

Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

Qwen released the open-weight 27B dense model Qwen3.6-27B and made it available in Qwen Studio. It scores 77.2 on SWE-bench Verified vs. 76.2 for Qwen3.5-397B-A17B, and 59.3 on Terminal-Bench 2.0 under a 256K context and 3-hour timeout. The real takeaway is deployment: this is not a larger MoE, but a denser 27B model with stronger coding results.

Why it matters: Qwen open-sourced Qwen3.6-27B, a 27B dense coding model that scores 77.2 on SWE-bench Verified, beating the previous 397B-parameter MoE flagship, making the dense-versus-MoE comparison the draw.

Apr 19Sunday

Synced · WeChat

MIA, a next-generation memory agent framework, aims to end agents' "amnesiac" workflows

A Shanghai Institute for Advanced Learning and ECNU team released MIA, a memory agent framework, and said it achieved the best results on 7 datasets. MIA uses a Manager-Planner-Executor design, dual parametric and non-parametric memory, alternating RL, and test-time continual learning; the post does not disclose exact benchmark scores. The key point is memory as capability internalization, not just retrieval, for open-world agents.

Why it matters: A Shanghai research team's MIA memory framework splits agent memory into parametric and non-parametric types and updates them with alternating reinforcement learning, offering a transferable way to keep evolving on open tasks.

Apr 16Thursday

r/LocalLLaMA

Qwen3.6-35B-A3B released

Qwen released Qwen3.6-35B-A3B as open source under Apache 2.0; it is a sparse MoE with 35B total parameters and 3B active. The post also claims agentic coding, strong multimodal perception and reasoning, plus thinking and non-thinking modes; the post does not disclose benchmarks, context length, or latency.

Why it matters: Qwen releases Qwen3.6-35B-A3B under Apache 2.0 for commercial use, with 35B total parameters and 3B active per inference, making the sparse MoE efficiency route the main point.

Apr 7Tuesday

Latent Space

[AINews] Gemma 4 crosses 2 million downloads

Google’s Gemma 4 reached about 2 million downloads in its first week. The post compares that with Gemma 3 at 6.7 million over the past year, Gemma 2 at 1.4 million since June 2024, and Qwen 3.5 at about 27 million in roughly 1.5 months. The signal for practitioners is local deployment: one iPhone 17 Pro demo ran Gemma 4 E2B at about 40 tok/s via MLX, with support across Hugging Face, vLLM, llama.cpp, Ollama, and NVIDIA.

Why it matters: Gemma 4 hit about 2 million downloads in its first week, against 6.7 million for Gemma 3 over a year and about 27 million for Qwen 3.5 in six weeks.

Mar 24Tuesday

Mistral AI

Mistral AI releases Voxtral TTS, a 4B-parameter speech model

Mistral AI released Voxtral TTS, its first text-to-speech model. It has 4B parameters and supports nine languages: English, French, German, Spanish, Dutch, Portuguese, Italian, Hindi and Arabic. It handles emotional expression and zero-shot cross-lingual voice adaptation.

Why it matters: A 4B open TTS model covering 9 languages, with a head-to-head against ElevenLabs as a reference point.

Mar 12Thursday

NVIDIA Blog

NVIDIA Nemotron 3 Super delivers 5x higher throughput for agentic AI

NVIDIA launched Nemotron 3 Super, a 120B open model with 12B active parameters, and says it delivers up to 5x higher throughput for agentic AI. It has a 1M-token context window and uses hybrid MoE, latent MoE, and multi-token prediction; the post says Blackwell NVFP4 gives up to 4x faster inference than Hopper FP8, with over 10T training tokens disclosed. What matters is that NVIDIA is releasing open weights, training recipes, and RL environments for reproduction and fine-tuning.

Why it matters: NVIDIA open-sourced Nemotron 3 Super: 120B total parameters, 12B active per inference, 1M token context, and 5x throughput on multi-step agent tasks.

Feb 20Friday

Hugging Face Blog

GGML and llama.cpp join Hugging Face to support the long-term progress of Local AI

Hugging Face said the GGML and llama.cpp team is joining the company, while Georgi Gerganov’s team will still spend 100% of its time maintaining llama.cpp. The post says the project remains 100% open source and community driven, with full technical and community autonomy. The key angle is tighter delivery from transformers model definitions into llama.cpp, aiming for near “single-click” shipping; the post does not disclose timeline, team size, or deal terms.

Why it matters: The GGML and llama.cpp team joined Hugging Face while staying full-time on llama.cpp, keeping technical direction and community management independent, so the core local-inference engineering effort gains long-term backing.

Feb 5Thursday

Mistral AI

Mistral releases Voxtral Transcribe 2 speech-to-text model family

Mistral released Voxtral Transcribe 2, a family of two speech-to-text models: Voxtral Mini Transcribe V2 for batch transcription and Voxtral Realtime for live use.

Why it matters: An Apache 2.0 realtime transcription model with latency configurable under 200ms and support for 13 languages.

Jan 6Tuesday

NVIDIA Blog

NVIDIA DGX Spark and DGX Station power the latest open-source and frontier models from the desktop

NVIDIA showed at CES that DGX Spark and DGX Station can run 100B to 1T-parameter models locally on deskside systems. The post cites a 35% average llama.cpp speedup, up to 70% NVFP4 compression, 775GB coherent memory on DGX Station, and a 250,000 token/sec pretraining demo. The real signal is the local dev loop: fine-tuning, inference, RAG, coding assistants, and robotics demos all target replacing some cloud iteration with deskside compute.

Why it matters: DGX Spark and DGX Station show what desktop hardware is needed to run 100B to 1T parameter open models locally.

NVIDIA Blog

NVIDIA unveils new open models, data and tools across agents, robotics, AVs and biomedicine

NVIDIA released open models, datasets and training tools spanning Nemotron, Cosmos, Alpamayo, Isaac GR00T and Clara, plus 10T language tokens, 500K robotics trajectories, 455K protein structures and 100TB of vehicle sensor data. Newly disclosed items include Nemotron Speech/RAG/Safety, Cosmos Reason 2, Transfer 2.5, Predict 2.5, GR00T N1.6 and Alpamayo 1; the key signal is that NVIDIA is opening the data stack across agents, physical AI, AVs and biomedicine.

Why it matters: NVIDIA's open release spans agents, robotics, autonomous driving and biotech, with new Nemotron models for speech, RAG and safety alignment.

Oct 22, 2025Wednesday

Hugging Face Blog

Hugging Face and VirusTotal collaborate to strengthen AI security

Hugging Face said on Oct. 22, 2025 it is continuously scanning more than 2.2 million public model and dataset repositories on the Hub through a VirusTotal collaboration. The Hub checks file hashes against VirusTotal and returns status, detection counts, and threat intel without sending raw file contents. The key point is earlier supply-chain visibility before download; the post does not disclose false-positive rates, scan latency, or remediation flow.

Why it matters: Hugging Face and VirusTotal now continuously scan over 2.2 million public model and dataset repos on the Hub, showing users flagged files and engine counts.

Oct 21, 2025Tuesday

Hugging Face Blog

Unlock the power of images with AI Sheets

Hugging Face added vision support to its open-source AI Sheets, letting users analyze images, extract data, generate visuals, and edit images inside a spreadsheet. The post says AI Sheets uses Inference Providers to access thousands of open models, and manual edits plus thumbs-up feedback become few-shot examples; outputs can be exported as CSV or Parquet. What matters is the unified data workflow, not a standalone demo.

Why it matters: Hugging Face added vision to its open-source spreadsheet tool AI Sheets, so users can drop in receipts, documents or product images and extract or edit with natural language.

Jun 10, 2025Tuesday

Mistral AI

Mistral AI releases its first reasoning model, Magistral, in open and enterprise versions

Mistral AI released Magistral, its first reasoning model, in two versions: the 24B open-source Magistral Small and the enterprise Magistral Medium.

Why it matters: Magistral ships as a 24B open model and an enterprise version, with AIME2024 scores for both.