Skip to content

Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

Latest picks

401–420 of 592

May 12Tuesday

AI HOT (Curated Pool)

Thinking Machines Releases Native Multimodal Interaction Model for Real-Time Human-AI Collaboration

Thinking Machines released an interaction model that natively receives audio, video, and text input, processes foreground interaction at 200-millisecond intervals, and uses a background reasoning model for long-horizon planning and tool calls.

Why it matters: Thinking Machines splits real-time multimodal work into a fast 200ms front end and a slower planning back end, an architecture pattern worth studying.

The Verge · AI

Here’s What Mira Murati’s AI Company Is Up To

Thinking Machines announced work on “interaction models” that continuously take in audio, video, and text and respond or act in real time; the post does not disclose model size, release timing, pricing, or the final product format.

Why it matters: Thinking Machines says it is building an interactive model that takes in audio, video and text at once and responds in real time, instead of waiting for the user to finish typing.

AI HOT (Curated Pool)

The Evolution of Human-Computer Interfaces: From Text to Interactive Neural Video

Karpathy argues that LLM output is moving from Markdown toward richer HTML, while interactive neural video still has an open problem: how to combine neural generation with precise traditional software.

Why it matters: Karpathy suggests large models can output HTML with layout and interaction directly, and imagines diffusion models generating operable screens in real time, an output-first angle on interface design.

May 11Monday

AI HOT (Curated Pool)

Qwen-Image-2.0 Technical Report

Qwen-Image-2.0 uses a Qwen3-VL condition encoder and multimodal diffusion transformer for image generation and precise editing, with instruction inputs up to 1K tokens and reported gains in multilingual text rendering, layout quality, and human-rated generation and editing tasks.

Why it matters: Qwen-Image-2.0 wires Qwen3-VL in as a conditioning encoder for a multimodal diffusion model, merging generation and precise editing with prompts up to 1,000 tokens.

May 10Sunday

Synced · WeChat

Ted Xiao Reviews Three Eras of Robot Learning, from RT-1/RT-2 to Scaling

Ted Xiao divides nearly a decade of robot learning into three eras: Google’s team trained RT-1 on 87,000 teleoperation trajectories, then adapted 5B to 55B VLMs into VLA policies for RT-2.

Why it matters: Ted Xiao walks through Google's robot learning path, from RT-1 trained on 87,000 teleoperation trajectories to RT-2 turning 5B-55B vision-language models into action policies.

r/LocalLLaMA

BeeLlama.cpp: DFlash and TurboQuant with reasoning and vision support

Anbeeld released BeeLlama.cpp, a llama.cpp fork that runs Qwen 3.6 27B Q5 with 200k context and vision on a single RTX 3090 or 4090; the title claims 2–3x faster than baseline and a 135 tps peak.

Why it matters: Anbeeld's llama.cpp fork BeeLlama.cpp adds DFlash memory-saving attention and TurboQuant quantization, letting Qwen 3.6 27B run 200k context on a single 3090 at 2 to 3 times baseline speed.

May 9Saturday

AI HOT (Curated Pool)

Tesla Uses Vision AI to Anticipate Collisions and Reduce Injury Risk

Tesla combined vision systems with crash sensors to trigger airbags and seatbelt pretensioners earlier, using real fleet crash data and simulation replay with human-body force measurements; the post does not disclose supported vehicle models or quantified injury-risk reductions for the OTA update.

Why it matters: Tesla pairs camera feeds with crash sensors so seatbelts and airbags fire before the sensors confirm impact, and uses real crash data plus simulation replays to assess forces on the body.

AI HOT (Curated Pool)

Peekaboo 3.0 Launches With Action-First macOS Control and UI Detection

Peekaboo 3.0 is now live with action-first macOS control, unified screenshots and UI detection, cleaner JSON exchange between CLI and MCP, and improved snapshots; the post does not disclose pricing, model choices, or release timeline beyond the 3.0 launch.

Why it matters: Peekaboo 3.0 folds screenshots and UI detection into one feature, letting models run Mac tasks directly and standardizing JSON between CLI and MCP.

QbitAI · WeChat

Qwen AI Glasses S1 Adds Spatial 3D Display, Proactive Reminders, and Daily AI Features

Qwen AI Glasses S1 added spatial 3D display and proactive services, with ride-hailing, instant shopping, and photo-based homework help scheduled for this month; Wellsenn XR says Qwen AI Glasses hold 53% of China’s online AI glasses sales since March 8.

Why it matters: Qwen's AI glasses add spatial 3D display and proactive reminders, and Wellsenn XR data puts them at 53% of China's online AI glasses sales.

Synced · WeChat

StarVLA Open-Sources a Unified VLA Framework from HKUST and the Community

HKUST and the open-source community released StarVLA, a unified Vision-Language-Action framework that integrates backbones, action heads, training strategies, and evaluation interfaces; the repository has 2.2k GitHub stars and supports benchmarks including LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, and BEHAVIOR-1K.

Why it matters: HKUST and the community open-sourced StarVLA, making VLA backbones, action heads and training interfaces pluggable instead of rebuilt per paper.

May 8Friday

Synced · WeChat

ICLR 2026: NVIDIA and Purdue Use an Agentic Loop for Text-to-3D Scene Generation

NVIDIA Cosmos Lab and Purdue University proposed Scenethesis, a language-and-vision agentic framework for text-to-3D scene generation that uses visual grounding, SDF-based physical constraints, and a judge module; experiments report about 72% first-pass success, 91% after self-checking, and collision rate reduction from 6.1% to 0.8%.

Why it matters: A concrete agent pipeline for 3D scene generation, with SDF physics constraints and a judge module that checks and corrects its own output.

AI HOT (Curated Pool)

Apple's First AI Wearable: Camera-Equipped AirPods Enter DVT Stage

Apple’s camera-equipped AirPods have entered DVT, with launch possible in September. Each earbud uses a low-res camera for visual Q&A with the upgraded Siri. The post cites Google Gemini support and a data-upload indicator light.

Why it matters: Apple's camera-equipped AirPods have reached design validation testing, with the shape and features largely settled; the camera lets Siri identify objects in front of the wearer and answer questions about them.

The Verge · AI

Apple’s AirPods with cameras for AI are reportedly close to production

Mark Gurman says Apple’s camera-equipped AirPods are in DVT, one step before PVT. Testers are using prototypes; the cameras capture low-resolution visual input, not photos or video, for Siri queries like ingredient prompts.

Why it matters: Apple's camera-equipped AirPods have reached design validation testing, with the camera capturing only low-resolution images to help Siri understand visual questions.

Bloomberg Technology

Apple’s Camera-Equipped AirPods Reach Late Testing in AI Device Push

Apple moved camera-equipped AirPods into late-stage development. The RSS snippet says they may be Apple’s first wearable built for the AI era; the post does not disclose camera specs, mechanisms, or launch timing.

Why it matters: Apple's camera-equipped AirPods have reached late development, a possible first wearable built for AI use, worth watching for anyone tracking where AI hardware goes.

May 7Thursday

AI HOT (Curated Pool)

SenseNova-U1 Open-Sources 8-Step Distilled LoRA, Speeds Diffusion Inference by 11x

SenseNova-U1 open-sourced an 8-step distilled LoRA that cuts diffusion generation from 100 steps to 8. GPU inference time drops from 23 seconds to 2 seconds, with ComfyUI workflows for text-to-image, image editing, and interleaved generation. The key signal is distillation for latency, not parameter scale.

Why it matters: An open 8-step LoRA cuts diffusion sampling from 100 steps to 8, dropping GPU inference from 23 seconds to 2, with a ComfyUI workflow included.

QbitAI · WeChat

Zhejiang University and Alibaba MetaCompress reaches 90% token compression for multi-turn VQA

Zhejiang University and Alibaba proposed MetaCompress, a learned token-compression framework that generates a compression mapping from the input image alone for multi-turn VQA. The article says it can remove 90% of visual tokens while preserving accuracy, and reports only 1.71% overlap between optimally retained tokens and high-attention tokens.

Why it matters: Zhejiang University and Alibaba let a model decide which visual tokens to keep from the input image alone, cutting 90% of tokens in multi-turn VQA with no accuracy loss.

QbitAI · WeChat

Vidu Claw Generates Ad Videos From One Prompt and a Hundred-Yuan Budget

Shengshu Technology opened Vidu Claw, which generates ad scripts, voiceover, music, editing, and final videos from one prompt; its Video Plan includes up to 40 minutes of daily generation across video, image, and audio.

Why it matters: Shengshu's Vidu Claw turns one prompt into a finished video with script, voiceover, music and editing, and its lobster ad shows the whole chain.

Xinzhiyuan · WeChat

Zhejiang University and Harvard open-source UniGeo for geometry-guided camera-controllable editing

Zhejiang University and Harvard released UniGeo with code, a report, a project page, and an HF Space. UniGeo injects geometry guidance into representation, architecture, and loss layers; it reports SOTA on DL3DV, RE10K, and Tanks against five methods. The key is video priors plus geometry-anchor attention, not just using a video model.

Why it matters: Zhejiang University and Harvard open-sourced UniGeo, which bakes geometric constraints into the model structure, representation layer and loss function to reduce shape and position distortion when generating from new viewpoints.

May 6Wednesday

r/LocalLLaMA

2.5x Faster Inference with Qwen 3.6 27B Using MTP on 48GB

A llama.cpp PR adds MTP support for Qwen 3.6 27B, with a reported 2.5x inference speedup. The author measured 28 tok/s on a Mac M2 Max 96GB and shared GGUF builds, compile steps, and a 262144-context server command. The key detail is turbo4 4.25-bit KV cache: a 48GB Mac runs Q5_K_M at 262K context.

Why it matters: A llama.cpp PR adds MTP support for Qwen 3.6 27B, measured at 2.5x faster inference with 262k context on 48GB, a usable setup for local agentic coding.

Synced · WeChat

Alibaba open-sources PromptEcho for T2I rewards using frozen VLMs

Alibaba open-sourced PromptEcho, which uses one frozen Qwen3-VL-32B forward pass to score T2I training rewards. It computes token-level cross-entropy for the original prompt under teacher forcing, then uses the negative value as a continuous reward. In 5,000 poster tests, text accuracy rose from 68% to 75%.

Why it matters: Alibaba open-sources PromptEcho, which uses a frozen Qwen3-VL-32B to generate continuous reward signals without training a separate scorer, lifting text accuracy from 68% to 75% on 5,000 posters.