Skip to content

#多模态

4 today

May 29Friday

r/LocalLLaMA

StepFun 3.7 Flash

StepFun released Step 3.7 Flash with 196B total parameters, 11B active MoE, a built-in 1.8B ViT, and local execution on 128GB RAM.

Why it matters: StepFun released Step 3.7 Flash, a 196B-parameter model that activates only 11B, plus a 1.8B vision module, and says it runs on a local machine with 128GB of memory.

AI HOT (Curated Pool)

StepFun Releases Step 3.7 Flash, Focused on Agent Efficiency

StepFun released the open-source Step 3.7 Flash model with a 198B-parameter MoE architecture, about 11B active parameters, a 256K context window, and a 67.1 score on ClawEval-1.1.

Why it matters: StepFun open-sourced Step 3.7 Flash, a 198B-parameter MoE model that activates about 11B parameters at runtime and supports 256K context and tool calling, giving agent workflows a lighter deployment option.

AI HOT (Curated Pool)

Nano Banana Pro and Nano Banana 2 officially released

Google AI Developers released Nano Banana Pro and Nano Banana 2, two image models available for production use through the Gemini API; the post names gemini-3-pro-image and gemini-3.1-flash-image but does not disclose pricing, benchmarks, or rate limits.

Why it matters: Google added the Nano Banana Pro and Nano Banana 2 image models to the Gemini API, so developers can call them now and use them in production.

The Verge · AI

A $2,000 AI-generated film will debut at Tribeca

Tribeca Festival will premiere the 75-minute AI-generated film Dreams of Violets, which cost $2,000 to make and uses people and images fully created by AI.

Why it matters: A 75-minute narrative feature with every on-screen character generated by AI will premiere at Tribeca, made for $2,000 from news reports and witness accounts.

May 28Thursday

NVIDIA Blog

NVIDIA Research Advances Robotics From Simulation to the Real World

NVIDIA Research presented 8 ICRA papers on sim-to-real robotics: ScheduleStream delivered a 3x speedup for multi-arm planning, COMPASS reached about 80% success across 20 real-world navigation trials, and Grasp-MPC achieved about 75% real-robot grasping success.

Why it matters: NVIDIA's COMPASS navigates real robots after simulation-only training, hitting roughly 80% success across 20 trials, 4.5x a pure imitation baseline.

r/LocalLLaMA

Nvidia LocateAnything: Fast Vision-Language Grounding with Parallel Box Decoding

The title says Nvidia LocateAnything-3B performs vision-language grounding with parallel box decoding and runs 10x faster than Qwen3-VL; the post body only provides Hugging Face, GitHub, demo, and project links, and does not disclose benchmark setup or accuracy numbers.

Why it matters: Nvidia open-sourced LocateAnything, a 3B vision grounding model that uses parallel box decoding instead of serial generation and claims 10x the speed of Qwen3-VL.

Synced · WeChat

ICML 2026: AutoMoT reaches SOTA on Bench2Drive and nuScenes

NTU AutoMan Lab, Harvard, and Xiaomi Auto proposed AutoMoT, a unified VLA driving model using a 4B Qwen3-VL Understanding Expert and a 1.6B Action Expert with asynchronous inference, reaching 89.42 DS and 74.09% SR on Bench2Drive with AutoMoT+.

Why it matters: AutoMoT splits a 4B vision-language model and a 1.6B action expert into asynchronous inference, taking first place on both Bench2Drive and nuScenes, a split-model idea for end-to-end driving.

Synced · WeChat

Chinese pretrained embodied model Wall-OSS-0.5 is open sourced

X Square Robot open sourced Wall-OSS-0.5, a VLA model whose 400k pretraining checkpoint scored above 80 on 4 of 17 real-robot zero-shot tasks, with weights, code, training recipe, ablations, and a DMuon optimizer implementation released.

Why it matters: X Square Robot open-sources Wall-OSS-0.5, a vision-language-action model with a 400k-step pretrained checkpoint, weights, code and training recipe; 4 of 17 real-robot zero-shot tasks top 80% success.

AI HOT (Curated Pool)

Open-source FastVideo Dreamverse real-time video generation tool

Hao AI Lab open-sourced FastVideo Dreamverse, a real-time video generation tool that generates a 30-second 1080p video in 7 seconds under the stated setup of one NVIDIA B200 GPU and LTX-2.

Why it matters: Hao AI Lab open-sourced FastVideo Dreamverse, which pairs LTX-2 with a single NVIDIA B200 to generate 30 seconds of 1080p video in 7 seconds, code and blog included.

May 27Wednesday

AI HOT (Curated Pool)

Runway launches Model Context Protocol server

Runway launched an MCP server that lets compatible agents such as Claude, ChatGPT, and Cursor generate images and videos inside chat interfaces, with access to Gen-4.5, Seedance 2.0, GPT Image 2, Kling 3.0, and Nano Banana Pro.

Why it matters: Runway put its image generation and video editing inside Claude, ChatGPT and Cursor chat boxes, so users send a link, reference image or text prompt and get a finished result without switching apps.

TechCrunch · AI

YouTube will now automatically label AI videos

YouTube will automatically label videos using significant photorealistic AI, no longer relying only on creator self-disclosure. The RSS snippet says AI labels will become more prominent, but the post does not disclose rollout timing, detection thresholds, appeal rules, or whether the system covers shorts and livestreams.

Why it matters: YouTube will use internal systems to detect and label videos with substantial realistic AI imagery, with labels more prominent on long videos and Shorts, shifting AI labeling from creator disclosure to active detection.

QbitAI · WeChat

7B Medical AI Agent Beats o3 and GPT-5 by Learning Where and How to Look

Shanghai Innovation Institute’s LeapQuest and three universities released Ophiuchus and MedScope, applying Think with Images and Think with Videos to medical AI; Ophiuchus-7B scored 68.0 on eight VQA benchmarks, above OpenAI-o3 at 62.2, Gemini 2.5 Pro at 61.8, and GPT-5 at 59.9.

Why it matters: Ophiuchus-7B scores 68.0 across 8 medical visual QA benchmarks, beating o3 and GPT-5, a sample of a small model catching general models through method design.

r/LocalLLaMA

PrismML Released Binary and Ternary Bonsai Image 4B

PrismML released Binary and Ternary Bonsai Image 4B, 1-bit and ternary text-to-image diffusion transformers around 3GB, compared with FLUX.2 Klein 4B at about 16GB, with browser-local WebGPU demo links and an Apache-2.0 license disclosed in the Reddit snippet.

Why it matters: PrismML compressed Bonsai Image 4B's text-to-image diffusion transformer weights to 1-bit or ternary, about 3GB against FLUX.2 Klein 4B's roughly 16GB, and it runs locally in a browser on WebGPU.

May 26Tuesday

AI HOT (Curated Pool)

SenseNova-U1 full training code open-sourced for multimodal multitask training

OpenSenseNova released the full SenseNova-U1 training code on GitHub under Apache-2.0, supporting an 8B dense model, an A3B MoE architecture, and multimodal tasks such as text-to-image generation, image editing, interleaved generation, and text-visual understanding.

Why it matters: SenseTime open-sourced SenseNova-U1's full training code on GitHub under Apache-2.0, covering 8B dense and A3B MoE variants, with one framework handling text-to-image, image editing, interleaved generation and understanding.

AI HOT (Curated Pool)

Project Luxo: Crossing the Uncanny Valley of AI Media

Runway released Project Luxo, showing AI shorts and ad samples including The Rogue; each work was made by a single-person team, with production times ranging from three weeks to four hours.

Why it matters: Runway shows three short films and an ad spot made by one person, cutting production from three weeks to four hours, with feedback from producers, actors and union members.

Alibaba Technology · WeChat

Nearly 9x training speedup: residual streams in DiT are becoming a convergence bottleneck

Nanjing University LAMDA and Alibaba Intelligent Engine proposed DAR, a timestep-aware cross-layer routing method that replaces fixed residual accumulation in DiT; on ImageNet 256x256, it reduced SiT-XL/2 FID from 9.67 to 7.56 and reached baseline convergence quality with 8.75x fewer training iterations.

Why it matters: Nanjing University's LAMDA lab and Alibaba replace DiT's fixed residual accumulation with per-timestep cross-layer routing, reaching equal convergence on ImageNet in about one-ninth the training steps.

QbitAI · WeChat

Zhejiang University and Alibaba Make AI Think Before Drawing Sudoku or Burning Candles | ACL 2026

Zhejiang University and Alibaba introduced Unified Thinker, an independent planning module trained with 40,000 HieraReason-40K samples and a two-stage GRPO reinforcement-learning setup that turns structured reasoning traces into executable visual instructions for image generation and editing.

Why it matters: Zhejiang University and Alibaba propose Unified Thinker, a planning module for image generation that separates thinking from execution, producing structured reasoning steps before converting them into visual instructions.

AI HOT (Curated Pool)

Grok Build Beta Opens to SuperGrok Users

xAI opened Grok Build Beta to all SuperGrok and X Premium+ users, with Plan Mode, Imagine-based image and video creation, and a CLI for automation or orchestrator workflows at x.ai/cli.

Why it matters: xAI opens its Grok Build beta to SuperGrok and X Premium+ users, with a Plan Mode that breaks tasks into steps, an Imagine module for images and video, and a command-line tool for automation.

May 25Monday

r/LocalLLaMA

NuExtract3 released: open-weight 4B VLM for Markdown, OCR and structured extraction

Numind released NuExtract3, a 4B open-weight VLM based on Qwen3.5-4B under Apache-2.0, supporting image and text to Markdown, OCR, and JSON-template extraction, with self-hosting from 4GB VRAM and weights in Safetensors, GGUF, and MLX formats.

Why it matters: Numind open-sourced NuExtract3, a 4B vision model that turns image text into Markdown, does OCR and pulls structured data into JSON templates, running locally on as little as 4GB of VRAM.

Synced · WeChat

A 1B-Gaussian 3D World Runs in the Browser, Outperforming Fei-Fei Li’s Spark

Manycore Tech open-sourced Aholo Viewer, a browser-based 3D Gaussian Splatting viewer that used half the memory of Spark 2.0 in a 300M-Gaussian test, loaded 2x faster, rendered 3x faster, and supports scenes with up to 1B Gaussian points.

Why it matters: A browser-side 3D Gaussian splatting viewer with concrete memory, load and render numbers against Spark 2.0, plus a 1-billion-point claim.

May 24Sunday

Synced · WeChat

ICML 2026: First Parallel Thinking Framework for Vision-Language Models

Visual Para-Thinker introduces a parallel thinking framework for vision-language models, using Pa-Attention and LPRoPE to isolate four visual reasoning paths and training on 163,000 question-answer pairs.

Why it matters: Visual Para-Thinker, accepted at ICML 2026, splits visual reasoning into four parallel paths with Pa-Attention and LPRoPE, trained on 163,000 question-answer pairs.

Xinzhiyuan · WeChat

Anthropic’s Three Cards Surface: Mythos 1 Appears, Opus 4.8 Spotted

Xinzhiyuan says Anthropic’s claude-opus-4.8 appeared in Google Vertex AI, while a 59.8MB Claude Code source-map leak with 512,000 TypeScript lines exposed Sonnet 4.8 references and Mythos 1 clues tied to Claude Code and Claude Security.

Why it matters: An unreleased Anthropic model surfaced on Google Vertex AI, and a leaked Claude Code source map points to more names in the pipeline.

r/LocalLLaMA

Vision-capable LLMs vs. OCR for long-document QA with charts, images, and tables

The author tested Claude Sonnet 4.5 on 171 questions from 30 image-heavy MMLongBench-Doc PDFs, comparing native PDF vision use with OCR pipelines. Native PDF ranked fifth of six at 52.0% accuracy and cost $0.2552 per query, while LlamaCloud premium with full context reached 59.6% at $0.1885 per query.

Why it matters: A hands-on test compares reading chart-heavy PDFs directly against OCR-then-QA on 30 documents, with accuracy and per-query cost reported.

May 23Saturday

Synced · WeChat

FlashAR speeds up pretrained autoregressive image models by 22.9x using 0.05% data

Zhejiang University and the University of Adelaide introduced FlashAR, using 0.05% of the original training data to reduce Emu3.5-Image-34B 512×512 generation latency from 130.10 seconds to 5.68 seconds, while GenEval changed from 80.48 to 80.29.

Why it matters: Zhejiang University and the University of Adelaide propose FlashAR, which fine-tunes on just 0.05% of the original training data and cuts Emu3.5-Image-34B 512x512 generation from 130.10 seconds to 5.68 seconds.

The Verge · AI

Google’s New Anything-to-Anything AI Model Is Wild

The Verge tried Google’s new Gemini anything-to-anything model for a stuffed-deer deepfake video, but the RSS snippet discloses only one example and does not disclose model parameters, pricing, release timing, or safety controls.

Why it matters: A Verge writer used a photo of a plush deer to try Gemini's any-to-any model, and it produced realistic video with almost no technical skill, showing how fast the bar for generation tools is dropping.

r/LocalLLaMA

meituan-longcat/LongCat-Video-Avatar-1.5 on Hugging Face

Meituan LongCat released LongCat-Video-Avatar-1.5 on Hugging Face, supporting AT2V, ATI2V, and video continuation while replacing Wav2Vec2 with Whisper-Large and using DMD2 distillation to reduce inference to 8 NFE; the model weights are released under the MIT License.

Why it matters: Meituan's LongCat open-sourced version 1.5 of its video digital human model under the MIT license, swapping the speech encoder from Wav2Vec2 to Whisper-Large and cutting inference to 8 steps.

AI HOT (Curated Pool)

Gemini update: over 900 million users and new agent features

Google announced that the Gemini app has surpassed 900 million monthly active users and introduced two agent features: Daily Brief for personalized daily summaries and Gemini Spark, a 24/7 personal agent that manages tasks under user authorization.

Why it matters: Google reached 900 million monthly Gemini app users and shipped two agent features, Daily Brief and Gemini Spark, the latter running tasks continuously with user permission.

May 22Friday

TechCrunch · AI

We tried Google’s AI glasses and they’re almost there

Google demonstrated prototype Android XR glasses that overlay Gemini-powered translation, navigation, and other information into the user’s field of view; the post does not disclose pricing, launch timing, battery life, or hardware specifications.

Why it matters: Google showed short hands-on demos of its display-equipped Android XR glasses at I/O, which overlay navigation and translation and work with both iPhone and Android phones.

AI HOT (Curated Pool)

Project Genie and Google Maps Street View launch interactive worlds

Project Genie partnered with Google Maps Street View to turn real U.S. locations into interactive worlds; the post does not disclose supported cities, generation mechanics, pricing, or access scope.

Why it matters: Project Genie links into Google Street View to turn real US locations into walkable interactive worlds, skipping manual modeling and giving simulation and world models a ready source of environments.

AI HOT (Curated Pool)

NetEase Youdao Open-Sources Ziyue 4 Multimodal and Text-to-Speech Models

NetEase Youdao open-sourced its Ziyue 4.0 multimodal and text-to-speech models, with the 27B multimodal model reporting 81.4% accuracy on Chinese math reasoning tasks and the speech model supporting 14 languages.

Why it matters: NetEase Youdao open-sources a 27B multimodal model and a speech synthesis model, with the former scoring 81.4% on Chinese math and science problems.

Hacker News front page

Deepfakes Tore a High School Apart

404 Media reports that five girls at Radnor Township High School were targeted with AI-generated CSAM, and a freshman allegedly spent $250 on a Movely subscription from Apple’s App Store; the visible article does not disclose the police outcome.

Why it matters: A Radnor High School freshman paid $250 for a subscription app and pasted five female classmates' faces onto nude bodies to make AI child sexual abuse material, and the fallout kept spreading through the school.

Synced · WeChat

CVPR 2026 | HiF-VLA: A Motion-Centric World Action Model

Westlake University and collaborators introduced HiF-VLA, a motion-centric VLA framework that extracts compact Motion vectors with codecs such as H.264 and uses a joint expert to predict future visual motion and generate action sequences, reporting 31.4GB peak memory and 117.7ms latency under the cited history-window setting.

Why it matters: A Westlake University team proposes HiF-VLA, replacing per-frame visual encoding with H.264 motion vectors and predicting visual dynamics and actions together, at 31.4GB peak memory and 117.7ms latency.

Synced · WeChat

Meta Chinese Researcher Releases ATLAS for Generalizable Visual Reasoning with One Word

Meta AI and the Chinese University of Hong Kong proposed ATLAS, a visual reasoning method that uses one Functional Token to connect Agentic and Latent Visual Reasoning, with ATLAS-178K, a two-stage SFT+RL pipeline, and LA-GRPO to train sparse visual-operation tokens.

Why it matters: Meta AI and CUHK propose ATLAS, adding function words to a vision-language model so one inference pass can switch between explicit tool calls and internal latent computation.

AI HOT (Curated Pool)

Plastic Interfaces: The Future Shape of AI-Driven Software

Salesforce has adopted a headless architecture that lets salespeople update data through AI; the post says MCPs, HTML, audio, and web interfaces can be generated dynamically by context, but it does not disclose implementation metrics or adoption numbers.

Why it matters: Salesforce has reps update deal records through AI instead of logging into the site, which the author uses to argue for malleable interfaces generated on the fly for the current situation.

AI HOT (Curated Pool)

Aleph 2.0 and Edit Studio

Runway released Aleph 2.0 and Edit Studio, combining generation, editing, and post-production into one platform; the post does not disclose pricing, technical parameters, or rollout scope.

Why it matters: Runway folds its new Aleph 2.0 model into an editing tool that previews natural-language video edits before you commit, bundling generation, compositing, upscaling, motion capture and object removal.

May 21Thursday

TechCrunch · AI

Hark raises $700M Series A for its secretive ‘universal’ AI interface

Hark raised a $700 million Series A and plans to release its first multimodal models this summer; the post does not disclose investors, valuation, model specifications, or a hardware launch schedule.

Why it matters: Hark raised a $700M Series A at a $6B valuation for a universal AI interface, starting with text, image and voice models this summer and hardware later.

Xinzhiyuan · WeChat

USTC Papers Study Lifelong Learning for LMMs via Multimodal Knowledge Injection

USTC researchers released MMEVOKE and KORE: MMEVOKE contains 9,422 samples across 159 subcategories, while KORE uses knowledge-tree augmentation and null-space constrained fine-tuning to reduce catastrophic forgetting during multimodal knowledge injection.

Why it matters: A USTC team's MMEVOKE benchmark and KORE fine-tuning method offer reusable evaluation and training ideas for forgetting in continual learning of multimodal models.

Synced · WeChat

VAST and Tsinghua propose density-controlled 3D Gaussian generation for SIGGRAPH 2026

VAST and Tsinghua propose DeG, a 3D Gaussian generation method that samples Gaussian centers from a learned density distribution and trains density control with a render loss contribution gradient; in some settings, it reaches TRELLIS-like visual quality with less than half the Gaussian count.

Why it matters: VAST and Tsinghua propose DeG, which allocates Gaussian points by object detail density and matches TRELLIS-level quality with under half the points.

Synced · WeChat

Xie Saining’s Team Releases Second-Generation Representation Autoencoder RAEv2

Xie Saining’s team, Adobe Research, and the Australian National University released RAEv2, which reaches gFID 1.06 after 80 epochs on ImageNet-256 and reduces EPFID@2 from 177 epochs to 35 epochs while keeping compute at 189 GFLOPs.

Why it matters: Xie Saining's team with Adobe Research and ANU released RAEv2, cutting ImageNet-256 convergence from 177 epochs to 35 while holding compute at 189 GFLOPs.

AI HOT (Curated Pool)

Tencent Launches OS-Level AI Assistant Mavis on Windows, Mac, and Android

Tencent launched the OS-level AI assistant Mavis on May 21 across Windows, Mac, and Android, with document parsing, image recognition, system maintenance, partial offline use, model dispatching, and desktop control of mobile apps listed as supported functions.

Why it matters: Tencent puts its AI assistant into the OS layer across Windows, Mac and Android, letting a desktop agent read files, images and control phone apps.