Skip to content

Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

Latest picks

221–240 of 592

Jun 7Sunday

QbitAI · WeChat

Kuaishou Kling Proposes VLM-as-Teacher for Test-Time Video Reasoning Optimization

City University and Kuaishou Kling proposed VLM-as-Teacher, which uses VLM feedback to optimize VGM LoRA at test time, reporting a 16.7-point average gain and raising VBVR-Bench from 0.666 to 0.781.

Why it matters: Kuaishou Kling and CityU use a vision-language model as a test-time teacher, feeding LoRA updates to a video generator so it follows text rules while it generates.

QbitAI · WeChat

Chinese open-source framework targets stable 5-minute AI long-video generation

JD open-sourced JoyAI-Echo, a long audio-video generation framework for 5-minute consistent videos, using cross-modal memory, DMD post-training for about 7.5x faster inference, and real-time upscaling from 720P to 1K or 2K output.

Why it matters: JD's open-source JoyAI-Echo uses a cross-modal memory module to keep characters and scenes consistent across 5-minute videos, with roughly 7.5x faster inference.

Jun 6Saturday

AI HOT (Curated Pool)

OpenCV 5 Released with New DNN Engine and Native LLM Support

OpenCV 5 introduces a graph-based DNN engine, raising ONNX operator coverage from under 23% in 4.x to over 80%, with native support for Transformer, VLM, and LLM workloads.

Why it matters: OpenCV 5 moves the DNN engine to a compute graph with operator fusion, lifting ONNX operator coverage from under 23% in 4.x to over 80%, so models that previously would not import now run directly.

r/LocalLLaMA

Big week for open AI, with 25+ notable open-weight drops across every modality

Victor M summarized 25+ open-weight model releases in one week, including NVIDIA Nemotron 3 Ultra, a 550B hybrid Mamba-MoE with 55B active parameters and a 1M-token context window.

Why it matters: More than 25 open-weight models shipped in a week across text, image and speech, with NVIDIA's Nemotron 3 Ultra at 550B parameters and a 1M token context.

Synced · WeChat

Daxiao Robotics and NTU Release PhysX-Omni for Simulation-Ready Physical 3D Generation

PhysX-Omni models rigid, deformable, and articulated objects in one simulation-ready 3D generation framework, while PhysXVerse contains over 8.7K physical 3D assets across more than 2.9K categories.

Why it matters: Daxiao Robotics and NTU's PhysX-Omni generates 3D objects with physics properties that run directly in simulators, backed by 8,700+ assets in PhysXVerse.

Synced · WeChat

Video AI Moves to 5 Minutes: Fully Open Source, One-Pass Generation, No Blind-Box Sampling

JD open-sourced JoyAI-Echo, a long audio-video generation framework that supports up to 5 minutes of cross-shot audiovisual consistency, local repainting, 8-step DMD distillation, and output up to 1472×2560 resolution.

Why it matters: JD open-sourced JoyAI-Echo, which generates video with audio up to five minutes long and keeps visuals and sound consistent across shots, giving long-video generation a reusable open path.

AI HOT (Curated Pool)

Google AI weekly product updates: Nano Banana 2, Co-Scientist, dreambeans, Gemma 4, and more

Google AI announced six updates: Nano Banana 2 is generally available, Gemma 4 12B can run fully offline on laptops, and Magenta RealTime 2 is open source.

Why it matters: Google shipped six AI updates in one week, including Nano Banana 2 and Pro plus the multi-agent research assistant Co-Scientist, covering multimodal generation and research hypothesis optimization.

AI HOT (Curated Pool)

Gemini Live supports real-time image creation and editing

Gemini App adds real-time image creation and editing inside Live; users must open Live, share the camera, and tell Gemini what they want to see.

Why it matters: Live mode in the Gemini app now generates and edits images in real time, so users can point a camera, describe a change and see it applied on the spot, from wall colors to memes.

Hacker News front page

Launch HN: General Instinct (YC P26) – Frontier Models on Edge Devices

General Instinct open-sourced InstinctRazor, compressing Qwen3.5-122B-A10B from a roughly 245GB BF16 MoE model into a 48GiB GGUF, with a small-GPU mode that streams experts from system RAM and uses about 7.6–8GB peak VRAM at an 8k context window.

Why it matters: General Instinct open-sourced InstinctRazor, a compression method that shrinks the 245GB Qwen3.5-122B-A10B mixture-of-experts model to 48GB for edge devices, a path worth studying for teams running large models locally.

Jun 5Friday

AI HOT (Curated Pool)

Meta Smart Glasses App Contains Face Recognition Code, NameTag Pushed to Over 50 Million Devices

Meta pushed face-recognition code named NameTag into its smart-glasses companion app, which has more than 50 million downloads; the feature uses three AI models to convert faces into local face templates and match them against a phone database.

Why it matters: A researcher pulled unused face-recognition code, NameTag, from Meta's smart glasses app, and its core components already shipped silently to devices with over 50 million downloads.

r/LocalLLaMA

Microsoft released MAI models instead of something like Qwen3.6-27B or Gemma-4-31B

Microsoft AI released seven MAI models, with MAI-Thinking-1 listed as 1T A35B with a 256K context window and MAI-Code-1-Flash listed as 137B A5B with a 256K context window.

Why it matters: Microsoft released seven MAI models at once, including a 1T A35B reasoning version and a 137B A5B code version, both with 256K context, against community hopes for mid-size open models.

Synced · WeChat

MetaFine proposes a diagnostic meta-evaluation framework for fine-grained robot manipulation

Southeast University and Peking University researchers introduced MetaFine, a diagnostic meta-evaluation framework that tests fine-grained robot manipulation across understanding, perception, and behavior, and the article says traditional binary success metrics can overestimate fine-manipulation capability by up to 70%.

Why it matters: Southeast University and Peking University researchers propose MetaFine, a meta-evaluation framework that diagnoses robot fine manipulation through understanding, perception and action, and finds success-rate-only evaluation overstates ability by up to 70%.

QbitAI · WeChat

Yao Shunyu Responds to Whether Tencent Is Behind in AI

Yao Shunyu said at Tencent Cloud’s AI industry application conference that Hunyuan 3 rebuilt pretraining and reinforcement-learning infrastructure, changed data and evaluation, and assigned its strongest post-training staff to improve Yuanbao first; he named coding agents, multimodality, and embodied AI as Tencent’s next focus areas.

Why it matters: Tencent's response to claims it lags in AI, plus how Hunyuan 3 rebuilt pretraining and RL and where post-training staff were reassigned.

AI HOT (Curated Pool)

Google Magenta RealTime 2 (MRT2) real-time music model released

Google AI for Developers released the open-weight Magenta RealTime 2 music model, supporting MIDI, live text prompts, and gestures, with native MacBook latency under 200 ms.

Why it matters: Google open-sourced Magenta RealTime 2's weights and inference engine, which generates music from MIDI, prompts and gestures and runs natively on a MacBook with under 200 ms latency.

AI HOT (Curated Pool)

Boson AI and LMSYS Release Higgs Audio v3 TTS End-to-End Service Based on SGLang-Omni

Boson AI and LMSYS released the Higgs Audio v3 TTS service with about 4B parameters, a Qwen3-4B backbone, support for 100 languages, streaming synthesis, and text tags for controlling 20+ emotions plus style, rhythm, and sound effects.

Why it matters: Boson AI and LMSYS put the 4B-parameter Higgs Audio v3 TTS into SGLang-Omni for end-to-end serving, with 100 languages and tuning for streaming voice assistants that generate speech as they go.

Jun 4Thursday

AI HOT (Curated Pool)

Nex-N2-Pro launches as a 397B MoE reasoning model based on Qwen3.5

neolab released Nex-N2-Pro, a 397B-parameter MoE reasoning model based on Qwen3.5-397B-A17B, with 262K context, VLM support, claimed GPT-5.5 and Claude Opus 4.7-level performance, 30–50% fewer thinking tokens, SOTA results on Terminal Bench 2.1, GDPVal, and SWE-Verified, plus free access for the first two weeks via SiliconFlow.

Why it matters: neolab released a 397B-parameter MoE reasoning model built on Qwen3.5, with 260k-token context and image recognition, claiming benchmark scores on par with GPT-5.5.

Xinzhiyuan · WeChat

MoleculeMind releases MMDesign, claims over 90% target hit rate

MoleculeMind released MMDesign, an AI platform for de novo biologics design. In tests across 12 therapeutic targets, it validated specific binding on 11 targets, sending only 14 to 50 molecules per target into wet-lab assays and reporting a target success rate above 90%.

Why it matters: MoleculeMind's MMDesign platform ran wet-lab tests on 14 to 50 molecules per target across 12 therapeutic targets, with specific binding measured on 11 of them.

Xinzhiyuan · WeChat

Silicon Valley CEO backs MiniMax M3 as it tops open-source rankings amid Chinese community debate

MiniMax M3 ranks first among open-source models on Artificial Analysis, and the article says it supports a 1M-token context window, used 100T-scale pretraining, and will open-source its weights and full technical report within 10 days.

Why it matters: MiniMax M3 tops Artificial Analysis's open-source ranking with a 1M token context and hundreds of trillions of pretraining tokens, and the weights and tech report are promised within 10 days.

Synced · WeChat

Google releases Gemma 4 12B for 16GB laptops

Google released Gemma 4 12B, a medium-size model that runs locally with 16GB VRAM or unified memory. It uses an encoder-free multimodal architecture, supports native audio input, ships under Apache 2.0, and includes an MTP draft model for lower latency.

Why it matters: Gemma 4's new 12B member runs locally on a 16GB laptop and takes audio input natively, giving developers a fresh open option for on-device deployment.

QbitAI · WeChat

CVPR 2026: NVIDIA, Tesla, and Waymo hear Xpeng present physical AI

Xpeng presented its world-model stack at CVPR 2026, covering X-World, X-Foresight, and X-Cache; the article says X-Cache cuts about 70% of repeated computation, the second-generation VLA used over 4 trillion training tokens, and the in-car stack reduced inference latency to 80 ms.

Why it matters: XPeng's CVPR 2026 world-model stack cuts redundant compute by about 70% with X-Cache and trains its second-gen VLA on over 4 trillion tokens.