Skip to content

Alibaba's Qwen family: open releases and iterations, from flagship models to small on-device ones.

Latest picks

161–180 of 215

May 8Friday

AI HOT (Curated Pool)

Readable behavioral signals remain in frozen LLM hidden states, Cygnus boosts accuracy

Proprioceptive AI says Cygnus adds adapters to frozen LLMs and raises Qwen-32B on ARC-Challenge from 82.2% to 94.97%. It projects hidden states into a gl(4,R) Lie-algebra space to isolate “dark modes.” Watch replication; the post does not disclose full eval sets or controls.

Why it matters: Proprioceptive AI attaches a self-awareness adapter to a frozen model, reading behavior signals from hidden states without retraining, lifting Qwen-32B's ARC score from 82% to nearly 95%.

May 7Thursday

r/LocalLLaMA

Qwen/WebWorld 32B/14B/8B (Qwen3 finetune)

Qwen released WebWorld 32B/14B/8B, Qwen3 finetunes for training and evaluating web agents. It uses 1M+ real web trajectories and supports 30+ step simulation plus A11y Tree, HTML, XML, Markdown, and natural-language states. Agents trained on its synthetic trajectories gain 9.9% on MiniWob++ and 10.9% on WebArena.

Why it matters: Qwen fine-tuned three WebWorld models of different sizes on Qwen3, training web agents on over 1 million real web interaction traces, with support for 30-plus consecutive steps and five kinds of page-state input.

r/LocalLLaMA

Running Qwen3.5/Qwen3.6 with NextN MTP in llama.cpp on one RTX 3090 Ti

A Reddit user posted a llama.cpp guide for Qwen3.5/3.6 with NextN MTP on one RTX 3090 Ti. It requires two unmerged PRs, #22400 and #22673; Qwen3.6-35B-A3B-MTP reaches 157 tok/s at 350W, 1700MHz, with q8 KV. The key reproducible detail is nextn=q8_0 quant override; missing it yields “////” output.

Why it matters: A Reddit user posts a hands-on llama.cpp guide for running Qwen3.5 and 3.6 on a single RTX 3090 Ti, built on two unmerged PRs and covering NextN MTP speculative decoding setup.

r/LocalLLaMA

GB10 inference engine Atlas is open source, with Qwen3.6-35B-FP8 over 100 tok/s

Avarok open-sourced Atlas, an inference engine running Qwen3.5-35B at ~111 tok/s sustained on one DGX Spark. It uses Rust+CUDA, a ~2.5GB image, and sub-2-minute cold start; the author claims 3.0–3.3x vLLM in tests. The key details are Blackwell SM120/121 kernels, NVFP4/FP8, and MTP decoding.

Why it matters: Avarok open-sourced Atlas, a Rust and CUDA inference engine, hitting about 111 tok/s for Qwen3.5-35B on a DGX Spark, which the author puts at 3.0 to 3.3 times vLLM.

May 6Wednesday

r/LocalLLaMA

Qwen3.6 27B NVFP4 + MTP on a Single RTX 5090: 200k Context in vLLM

A Reddit user ran Qwen3.6 27B NVFP4 on one RTX 5090 32GB and validated 200k context in vLLM. The setup used fp8_e4m3 KV cache, FlashInfer, and MTP with 3 speculative tokens; a 10-run 200k pass completed with 73.6 tok/s mean generation and 70.2s TTFT. The key constraint is 32GB VRAM: logs showed 8.3GiB KV cache and about 30478MiB total GPU use.

Why it matters: A Reddit user ran Qwen3.6 27B NVFP4 on a single RTX 5090 in vLLM, verified 200k context, and posted the fp8_e4m3 KV cache and MTP speculative decoding config.

QbitAI · WeChat

Claude Team Tests New Training Method on Qwen

Anthropic proposed MSM training between pretraining and alignment fine-tuning. Tests on Qwen2.5-32B and Qwen3-32B cut misalignment from 68% and 54% to 5% and 7%. The key point is MSM complements AFT rather than replacing it.

Why it matters: Anthropic tested an MSM stage between pretraining and alignment fine-tuning on Qwen2.5-32B and Qwen3-32B, cutting misalignment from 68% and 54% to 5% and 7%.

r/LocalLLaMA

2.5x Faster Inference with Qwen 3.6 27B Using MTP on 48GB

A llama.cpp PR adds MTP support for Qwen 3.6 27B, with a reported 2.5x inference speedup. The author measured 28 tok/s on a Mac M2 Max 96GB and shared GGUF builds, compile steps, and a 262144-context server command. The key detail is turbo4 4.25-bit KV cache: a 48GB Mac runs Q5_K_M at 262K context.

Why it matters: A llama.cpp PR adds MTP support for Qwen 3.6 27B, measured at 2.5x faster inference with 262k context on 48GB, a usable setup for local agentic coding.

Synced · WeChat

Alibaba open-sources PromptEcho for T2I rewards using frozen VLMs

Alibaba open-sourced PromptEcho, which uses one frozen Qwen3-VL-32B forward pass to score T2I training rewards. It computes token-level cross-entropy for the original prompt under teacher forcing, then uses the negative value as a continuous reward. In 5,000 poster tests, text accuracy rose from 68% to 75%.

Why it matters: Alibaba open-sources PromptEcho, which uses a frozen Qwen3-VL-32B to generate continuous reward signals without training a separate scorer, lifting text accuracy from 68% to 75% on 5,000 posters.

Synced · WeChat

Two Chinese open-source projects turn Mac into a private AI workstation

Mininglamp open-sourced Cider and Mano-P 1.0 for Apple Silicon local inference and GUI agents. Cider speeds Qwen3-VL-2B prefill by 57%–61% on M5 Pro; Mano-P 1.0-72B scores 58.2% on OSWorld. The key constraint is W8A8 memory: on 16GB devices accuracy falls from 58.0% to 54.0%, so 32GB+ is recommended.

Why it matters: Minglue open-sources Cider and Mano-P 1.0, with Cider cutting Qwen3-VL-2B first-token latency about 60% on M5 Pro and the 72B Mano-P scoring 58.2 on OSWorld.

r/LocalLLaMA

DeepSeek V4 at 17x lower cost prompted a local-vs-cloud coding workflow test

Reddit user spencer_kw logged a 10-day coding workflow and retested 150 tasks on local Qwen 3.6 27B versus cloud models. Local was equivalent for 65% of tasks, acceptable for 20%, and cloud was needed for 15%; the API bill fell from $85/month to about $22. The useful signal is task-based routing, not headline model pricing alone.

Why it matters: A concrete split of which coding tasks run on a local Qwen model and which still need the cloud, with the API bill to match.

May 5Tuesday

r/LocalLLaMA

Heretic 1.3 Released: Reproducible Models, Integrated Benchmarks, Lower Peak VRAM

Heretic 1.3 adds reproducible runs, integrated benchmarks, lower peak VRAM, and broader model support. The project claims 20,000 GitHub stars and 13 million model downloads. Reproduce directories capture PyTorch, GPU, driver, and accelerator details; benchmarks use lm-evaluation-harness for MMLU, EQ-Bench, GSM8K, and HellaSwag. The post names Qwen3.5 and Gemma 4 support, but does not disclose VRAM reduction figures.

Why it matters: Heretic 1.3 records the PyTorch, GPU and driver environment so others can reproduce results, and adds built-in benchmarking with lower peak VRAM use.

r/LocalLLaMA

Prompt injection benchmark: delimiter and strict prompt took Gemma 4 from 21% to 100% defense rate

A Reddit user posted a prompt-injection benchmark covering 15 models, 7 attack types, and 6,100+ cases. The setup wraps untrusted documents in long random delimiters; Gemma 4 E4B rose from 21.6% to 100% defense. The key detail is the reproducible metric: blocked/(blocked+failed).

Why it matters: A Reddit user ran over 6,100 prompt injection tests across 15 models and 7 attack styles, and a long random delimiter plus strict instructions lifted Gemma 4 E4B's defense rate from 21.6% to 100%.

r/LocalLLaMA

MTPLX: 2.24x Faster TPS Native MTP Inference Engine for Apple Silicon

MTPLX raises Qwen3.6-27B on a MacBook Pro M5 Max from 28 to 63 tok/s. The test used 4-bit MLX, temperature 0.6, top_p 0.95, top_k 20, with D3 as the best depth. The key detail is native MTP heads: no external drafter and no second-model memory.

Why it matters: MTPLX on a MacBook Pro M5 Max raised Qwen3.6-27B generation speed from 28 to 63 tokens per second, a 2.24x gain from built-in MTP heads with no separate drafter model to load.

r/LocalLLaMA

Benching Local Qwen as a Codex Validator, Co-agent, and Challenger

robert896r1 tested Qwen3.6 27B GGUF beside Codex as a coding validator and released a reproducible eval suite. The runs covered Bartowski, Unsloth, 65k/128k context, and q8/f16 KV cache; three 128k profiles tied for best, with no measured q8 KV accuracy loss in this suite. The useful signal is the sidecar eval: missed directives, overbuilding, UI judgment, and long-context misses, not a universal leaderboard.

Why it matters: The author runs a GGUF-quantized Qwen3.6 27B beside Codex as a code validator and published a reproducible benchmark covering two quantization sources, 65k and 128k context, and q8 and f16 KV cache precision.

May 4Monday

r/LocalLLaMA

Deep research report with Hermes Agent and qwen3.6-35b-a3b Q6_K

A Reddit user used Hermes Agent and qwen3.6-35b-a3b Q6_K to produce a 21-page research report. The run took 6 loops and over 5 hours on an RTX 4060, at about 28 tokens/s. The repo includes prompts, scripts, intermediate artifacts, and the final report.

Why it matters: A Reddit user ran Hermes Agent with a quantized qwen3.6-35b on one RTX 4060 through six loops to produce a 21-page report, with prompts and scripts in the repo.

r/LocalLLaMA

Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB

A Reddit user benchmarked Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB. llama.cpp tensor split raised Mistral tg128 from 10.37 to 21.59 t/s, but Qwen MoE fell from 60.08 to 53.49 t/s. vLLM served Qwen GPTQ-Int4 at 187.04 tok/s; the key signal is MoE sensitivity to parallel strategy.

Why it matters: A Reddit user ran two large models on 4 RTX 3080 20GB cards, and Mistral Medium 3.5 128B hit 21.59 tokens per second with tensor split, up from 10.37.

r/LocalLLaMA

Pushing a 5-Year-Old 6GB VRAM Laptop to Its Limits: Qwen3.6-35B-A3B

Reddit user abhinand05 ran Qwen3.6-35B-A3B on a 5-year-old Asus ROG Zephyrus G14, reaching about 23 t/s plugged in and 10+ t/s unplugged. The setup uses RTX 2060 Max-Q 6GB, 24GB DDR4, Ryzen 7, plus llama-server configs for 64k and 128k context. The key detail is the mix of CPU MoE, KV-cache quantization, and ngram speculative decoding.

Why it matters: A Reddit user got a 35B model running on a 5-year-old laptop with 6GB of VRAM, about 23 tokens per second plugged in, showing what CPU hybrid inference can do under tight memory.

May 3Sunday

r/LocalLLaMA

Local LLM Benchmark for Backend Generation via Function Calling: GLM vs Qwen vs DeepSeek

AutoBe posted a controlled backend-generation benchmark and says qwen3.5-35b-a3b matches gpt-5.4 on DB/API design. One shopping-mall run uses 200–300M tokens, costing $1,000–$1,500 per model at GPT 5.5 pricing. The key caveat is n=4 projects and self-scoring harness bias.

Why it matters: AutoBe tests database and API design through function calling, and qwen3.5-35b-a3b matches gpt-5.4 on 4 projects, with one mall project burning 200 to 300 million tokens.

r/LocalLLaMA

Paper on Hummingbird+: low-cost FPGAs for LLM inference

A Hummingbird+ paper claims low-cost FPGAs run Qwen3-30B-A3B Q4 at 18 t/s generation. The title lists 24GB memory and an expected $150 mass-production cost; the post does not disclose FPGA model, power, or test conditions.

Why it matters: The Hummingbird+ paper runs a Q4 Qwen3-30B-A3B on a low-cost FPGA at 18 tokens per second with 24GB on board, and the authors expect roughly $150 at mass production.

r/LocalLLaMA

Local image generation on Mac: 10 models compared

A Reddit user tested 10 image models on an M1 Max with 64GB RAM. Qwen-Image Lightning’s 8-step distillation beat the full model at 10 minutes versus 93. Flux dev led local photorealism but showed English-centric bias; Gemini handled kanji and context better but is cloud-only.

Why it matters: A Reddit user benchmarked 10 image models on an M1 Max, and Qwen-Image Lightning used 8-step distillation to cut generation from 93 minutes to 10, giving local deployment a concrete speedup case.