Skip to content

Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

Latest picks

301–320 of 592

May 26Tuesday

Alibaba Technology · WeChat

Nearly 9x training speedup: residual streams in DiT are becoming a convergence bottleneck

Nanjing University LAMDA and Alibaba Intelligent Engine proposed DAR, a timestep-aware cross-layer routing method that replaces fixed residual accumulation in DiT; on ImageNet 256x256, it reduced SiT-XL/2 FID from 9.67 to 7.56 and reached baseline convergence quality with 8.75x fewer training iterations.

Why it matters: Nanjing University's LAMDA lab and Alibaba replace DiT's fixed residual accumulation with per-timestep cross-layer routing, reaching equal convergence on ImageNet in about one-ninth the training steps.

QbitAI · WeChat

Zhejiang University and Alibaba Make AI Think Before Drawing Sudoku or Burning Candles | ACL 2026

Zhejiang University and Alibaba introduced Unified Thinker, an independent planning module trained with 40,000 HieraReason-40K samples and a two-stage GRPO reinforcement-learning setup that turns structured reasoning traces into executable visual instructions for image generation and editing.

Why it matters: Zhejiang University and Alibaba propose Unified Thinker, a planning module for image generation that separates thinking from execution, producing structured reasoning steps before converting them into visual instructions.

AI HOT (Curated Pool)

Grok Build Beta Opens to SuperGrok Users

xAI opened Grok Build Beta to all SuperGrok and X Premium+ users, with Plan Mode, Imagine-based image and video creation, and a CLI for automation or orchestrator workflows at x.ai/cli.

Why it matters: xAI opens its Grok Build beta to SuperGrok and X Premium+ users, with a Plan Mode that breaks tasks into steps, an Imagine module for images and video, and a command-line tool for automation.

May 25Monday

r/LocalLLaMA

NuExtract3 released: open-weight 4B VLM for Markdown, OCR and structured extraction

Numind released NuExtract3, a 4B open-weight VLM based on Qwen3.5-4B under Apache-2.0, supporting image and text to Markdown, OCR, and JSON-template extraction, with self-hosting from 4GB VRAM and weights in Safetensors, GGUF, and MLX formats.

Why it matters: Numind open-sourced NuExtract3, a 4B vision model that turns image text into Markdown, does OCR and pulls structured data into JSON templates, running locally on as little as 4GB of VRAM.

Synced · WeChat

A 1B-Gaussian 3D World Runs in the Browser, Outperforming Fei-Fei Li’s Spark

Manycore Tech open-sourced Aholo Viewer, a browser-based 3D Gaussian Splatting viewer that used half the memory of Spark 2.0 in a 300M-Gaussian test, loaded 2x faster, rendered 3x faster, and supports scenes with up to 1B Gaussian points.

Why it matters: A browser-side 3D Gaussian splatting viewer with concrete memory, load and render numbers against Spark 2.0, plus a 1-billion-point claim.

May 24Sunday

Synced · WeChat

ICML 2026: First Parallel Thinking Framework for Vision-Language Models

Visual Para-Thinker introduces a parallel thinking framework for vision-language models, using Pa-Attention and LPRoPE to isolate four visual reasoning paths and training on 163,000 question-answer pairs.

Why it matters: Visual Para-Thinker, accepted at ICML 2026, splits visual reasoning into four parallel paths with Pa-Attention and LPRoPE, trained on 163,000 question-answer pairs.

Xinzhiyuan · WeChat

Anthropic’s Three Cards Surface: Mythos 1 Appears, Opus 4.8 Spotted

Xinzhiyuan says Anthropic’s claude-opus-4.8 appeared in Google Vertex AI, while a 59.8MB Claude Code source-map leak with 512,000 TypeScript lines exposed Sonnet 4.8 references and Mythos 1 clues tied to Claude Code and Claude Security.

Why it matters: An unreleased Anthropic model surfaced on Google Vertex AI, and a leaked Claude Code source map points to more names in the pipeline.

r/LocalLLaMA

Vision-capable LLMs vs. OCR for long-document QA with charts, images, and tables

The author tested Claude Sonnet 4.5 on 171 questions from 30 image-heavy MMLongBench-Doc PDFs, comparing native PDF vision use with OCR pipelines. Native PDF ranked fifth of six at 52.0% accuracy and cost $0.2552 per query, while LlamaCloud premium with full context reached 59.6% at $0.1885 per query.

Why it matters: A hands-on test compares reading chart-heavy PDFs directly against OCR-then-QA on 30 documents, with accuracy and per-query cost reported.

May 23Saturday

Synced · WeChat

FlashAR speeds up pretrained autoregressive image models by 22.9x using 0.05% data

Zhejiang University and the University of Adelaide introduced FlashAR, using 0.05% of the original training data to reduce Emu3.5-Image-34B 512×512 generation latency from 130.10 seconds to 5.68 seconds, while GenEval changed from 80.48 to 80.29.

Why it matters: Zhejiang University and the University of Adelaide propose FlashAR, which fine-tunes on just 0.05% of the original training data and cuts Emu3.5-Image-34B 512x512 generation from 130.10 seconds to 5.68 seconds.

The Verge · AI

Google’s New Anything-to-Anything AI Model Is Wild

The Verge tried Google’s new Gemini anything-to-anything model for a stuffed-deer deepfake video, but the RSS snippet discloses only one example and does not disclose model parameters, pricing, release timing, or safety controls.

Why it matters: A Verge writer used a photo of a plush deer to try Gemini's any-to-any model, and it produced realistic video with almost no technical skill, showing how fast the bar for generation tools is dropping.

r/LocalLLaMA

meituan-longcat/LongCat-Video-Avatar-1.5 on Hugging Face

Meituan LongCat released LongCat-Video-Avatar-1.5 on Hugging Face, supporting AT2V, ATI2V, and video continuation while replacing Wav2Vec2 with Whisper-Large and using DMD2 distillation to reduce inference to 8 NFE; the model weights are released under the MIT License.

Why it matters: Meituan's LongCat open-sourced version 1.5 of its video digital human model under the MIT license, swapping the speech encoder from Wav2Vec2 to Whisper-Large and cutting inference to 8 steps.

AI HOT (Curated Pool)

Gemini update: over 900 million users and new agent features

Google announced that the Gemini app has surpassed 900 million monthly active users and introduced two agent features: Daily Brief for personalized daily summaries and Gemini Spark, a 24/7 personal agent that manages tasks under user authorization.

Why it matters: Google reached 900 million monthly Gemini app users and shipped two agent features, Daily Brief and Gemini Spark, the latter running tasks continuously with user permission.

May 22Friday

TechCrunch · AI

We tried Google’s AI glasses and they’re almost there

Google demonstrated prototype Android XR glasses that overlay Gemini-powered translation, navigation, and other information into the user’s field of view; the post does not disclose pricing, launch timing, battery life, or hardware specifications.

Why it matters: Google showed short hands-on demos of its display-equipped Android XR glasses at I/O, which overlay navigation and translation and work with both iPhone and Android phones.

AI HOT (Curated Pool)

Project Genie and Google Maps Street View launch interactive worlds

Project Genie partnered with Google Maps Street View to turn real U.S. locations into interactive worlds; the post does not disclose supported cities, generation mechanics, pricing, or access scope.

Why it matters: Project Genie links into Google Street View to turn real US locations into walkable interactive worlds, skipping manual modeling and giving simulation and world models a ready source of environments.

AI HOT (Curated Pool)

NetEase Youdao Open-Sources Ziyue 4 Multimodal and Text-to-Speech Models

NetEase Youdao open-sourced its Ziyue 4.0 multimodal and text-to-speech models, with the 27B multimodal model reporting 81.4% accuracy on Chinese math reasoning tasks and the speech model supporting 14 languages.

Why it matters: NetEase Youdao open-sources a 27B multimodal model and a speech synthesis model, with the former scoring 81.4% on Chinese math and science problems.

Hacker News front page

Deepfakes Tore a High School Apart

404 Media reports that five girls at Radnor Township High School were targeted with AI-generated CSAM, and a freshman allegedly spent $250 on a Movely subscription from Apple’s App Store; the visible article does not disclose the police outcome.

Why it matters: A Radnor High School freshman paid $250 for a subscription app and pasted five female classmates' faces onto nude bodies to make AI child sexual abuse material, and the fallout kept spreading through the school.

Synced · WeChat

CVPR 2026 | HiF-VLA: A Motion-Centric World Action Model

Westlake University and collaborators introduced HiF-VLA, a motion-centric VLA framework that extracts compact Motion vectors with codecs such as H.264 and uses a joint expert to predict future visual motion and generate action sequences, reporting 31.4GB peak memory and 117.7ms latency under the cited history-window setting.

Why it matters: A Westlake University team proposes HiF-VLA, replacing per-frame visual encoding with H.264 motion vectors and predicting visual dynamics and actions together, at 31.4GB peak memory and 117.7ms latency.

Synced · WeChat

Meta Chinese Researcher Releases ATLAS for Generalizable Visual Reasoning with One Word

Meta AI and the Chinese University of Hong Kong proposed ATLAS, a visual reasoning method that uses one Functional Token to connect Agentic and Latent Visual Reasoning, with ATLAS-178K, a two-stage SFT+RL pipeline, and LA-GRPO to train sparse visual-operation tokens.

Why it matters: Meta AI and CUHK propose ATLAS, adding function words to a vision-language model so one inference pass can switch between explicit tool calls and internal latent computation.

AI HOT (Curated Pool)

Plastic Interfaces: The Future Shape of AI-Driven Software

Salesforce has adopted a headless architecture that lets salespeople update data through AI; the post says MCPs, HTML, audio, and web interfaces can be generated dynamically by context, but it does not disclose implementation metrics or adoption numbers.

Why it matters: Salesforce has reps update deal records through AI instead of logging into the site, which the author uses to argue for malleable interfaces generated on the fly for the current situation.

AI HOT (Curated Pool)

Aleph 2.0 and Edit Studio

Runway released Aleph 2.0 and Edit Studio, combining generation, editing, and post-production into one platform; the post does not disclose pricing, technical parameters, or rollout scope.

Why it matters: Runway folds its new Aleph 2.0 model into an editing tool that previews natural-language video edits before you commit, bundling generation, compositing, upscaling, motion capture and object removal.