Skip to content

Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

Latest picks

321–340 of 514

May 12Tuesday

Latent Space

Thinking Machines' Native Interaction Models: TML-Interaction-Small 276B-A12B Advances Realtime Voice

Thinking Machines released TML-Interaction-Small, a 276B-parameter MoE model with 12B active parameters, and the post says it advances realtime voice through 200ms time-aligned microturns, encoder-free early fusion for audio and images under 200ms, and benchmark wins over GPT-Realtime-2 and Gemini 3.1-Flash.

Why it matters: HKR-H/K/R all pass: TML-Interaction-Small gives architecture, active parameters, 200ms interaction, and named rivals. Benchmarks still need replication, but a real-time voice SOTA claim is same-day material.

QbitAI · WeChat

OpenClaw quietly updates with Peekaboo v3 for Mac computer use

OpenClaw-related Peekaboo v3 adds Mac agent capabilities for pixel-level screenshots, UI position reading, clicks, text input, hotkeys, scrolling, and drag-and-drop, with MCP server integration for Cursor, Claude Code, and Codex.

Why it matters: HKR-H/K/R all pass: Peekaboo v3 adds Mac GUI perception and action primitives plus MCP access for Cursor, Claude Code, and Codex. This is a useful open-source agent-tooling update, not a model-level event, so it sits in the low featured band.

AI HOT (Curated Pool)

Thinking Machines Releases Native Multimodal Interaction Model for Real-Time Human-AI Collaboration

Thinking Machines released an interaction model that natively receives audio, video, and text input, processes foreground interaction at 200-millisecond intervals, and uses a background reasoning model for long-horizon planning and tool calls.

Why it matters: HKR-H/K/R all pass: this is more than a model notice, with a two-layer foreground/background interaction design. Pricing, access scope, and benchmarks are missing, so it sits at the lower end of 85-94.

The Verge · AI

Here’s What Mira Murati’s AI Company Is Up To

Thinking Machines announced work on “interaction models” that continuously take in audio, video, and text and respond or act in real time; the post does not disclose model size, release timing, pricing, or the final product format.

Why it matters: HKR-H/K/R all pass, but the body lacks parameters, launch timing, and product form. This is a high-interest startup direction reveal, not a usable model release, so it stays at the top of the 72–77 band.

AI HOT (Curated Pool)

The Evolution of Human-Computer Interfaces: From Text to Interactive Neural Video

Karpathy argues that LLM output is moving from Markdown toward richer HTML, while interactive neural video still has an open problem: how to combine neural generation with precise traditional software.

Why it matters: HKR-H/K/R pass: Karpathy gives a fresh UI frame, a concrete Markdown→HTML→neural-video path, and a builder-facing product question. Single X post with no data keeps it at the featured floor.

May 11Monday

AI HOT (Curated Pool)

Qwen-Image-2.0 Technical Report

Qwen-Image-2.0 uses a Qwen3-VL condition encoder and multimodal diffusion transformer for image generation and precise editing, with instruction inputs up to 1K tokens and reported gains in multilingual text rendering, layout quality, and human-rated generation and editing tasks.

Why it matters: HKR-H/K/R all pass: Qwen’s flagship image model report gives concrete architecture, 1K-token instruction input, and editing claims. The domestic flagship-model signal lifts it into the must-write band.

May 10Sunday

Synced · WeChat

Ted Xiao Reviews Three Eras of Robot Learning, from RT-1/RT-2 to Scaling

Ted Xiao divides nearly a decade of robot learning into three eras: Google’s team trained RT-1 on 87,000 teleoperation trajectories, then adapted 5B to 55B VLMs into VLA policies for RT-2.

Why it matters: HKR-H/K/R all pass: a named Google robotics insider, concrete RT-1/RT-2 numbers, and strong embodied-AI resonance. It is retrospective commentary, not a launch, so it stays in the 72–77 featured band.

r/LocalLLaMA

BeeLlama.cpp: DFlash and TurboQuant with reasoning and vision support

Anbeeld released BeeLlama.cpp, a llama.cpp fork that runs Qwen 3.6 27B Q5 with 200k context and vision on a single RTX 3090 or 4090; the title claims 2–3x faster than baseline and a 135 tps peak.

Why it matters: HKR-H/K/R all pass, but the claims come from a Reddit title and summary without independent reproduction. Treat as a mid-weight open-source inference update, so it lands in the low featured band.

May 9Saturday

AI HOT (Curated Pool)

Tesla Uses Vision AI to Anticipate Collisions and Reduce Injury Risk

Tesla combined vision systems with crash sensors to trigger airbags and seatbelt pretensioners earlier, using real fleet crash data and simulation replay with human-body force measurements; the post does not disclose supported vehicle models or quantified injury-risk reductions for the OTA update.

Why it matters: HKR-H/K/R all pass, but the facts come from a single Musk post; OTA coverage, injury reduction, and validation method are not disclosed. This fits a mid-weight product update, not a must-write release.

AI HOT (Curated Pool)

Peekaboo 3.0 Launches With Action-First macOS Control and UI Detection

Peekaboo 3.0 is now live with action-first macOS control, unified screenshots and UI detection, cleaner JSON exchange between CLI and MCP, and improved snapshots; the post does not disclose pricing, model choices, or release timeline beyond the 3.0 launch.

Why it matters: HKR-H/K/R all pass for a concrete desktop-agent tooling update. Score stays at the featured floor because pricing, model details, and adoption data are not disclosed.

QbitAI · WeChat

Qwen AI Glasses S1 Adds Spatial 3D Display, Proactive Reminders, and Daily AI Features

Qwen AI Glasses S1 added spatial 3D display and proactive services, with ride-hailing, instant shopping, and photo-based homework help scheduled for this month; Wellsenn XR says Qwen AI Glasses hold 53% of China’s online AI glasses sales since March 8.

Why it matters: HKR-H/K/R all pass, but this is an AI-glasses feature update rather than a model or platform release. The 53% online-sales share and this-month feature list justify low featured range.

Synced · WeChat

StarVLA Open-Sources a Unified VLA Framework from HKUST and the Community

HKUST and the open-source community released StarVLA, a unified Vision-Language-Action framework that integrates backbones, action heads, training strategies, and evaluation interfaces; the repository has 2.2k GitHub stars and supports benchmarks including LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, and BEHAVIOR-1K.

Why it matters: HKR-H/K/R all pass: StarVLA ships a concrete open-source VLA framework with unified interfaces, 2.2k stars, and named robotics benchmarks. The robotics scope keeps it in the 78–84 band, below model-release weight.

May 8Friday

Synced · WeChat

ICLR 2026: NVIDIA and Purdue Use an Agentic Loop for Text-to-3D Scene Generation

NVIDIA Cosmos Lab and Purdue University proposed Scenethesis, a language-and-vision agentic framework for text-to-3D scene generation that uses visual grounding, SDF-based physical constraints, and a judge module; experiments report about 72% first-pass success, 91% after self-checking, and collision rate reduction from 6.1% to 0.8%.

Why it matters: HKR-H/K/R all pass: NVIDIA/Purdue plus an agent loop is clickable, and the post gives SDF constraints, a judge module, and 72%→91% results. Strong research signal, but not a product release, so it stays in 78–84.

AI HOT (Curated Pool)

Apple's First AI Wearable: Camera-Equipped AirPods Enter DVT Stage

Apple’s camera-equipped AirPods have entered DVT, with launch possible in September. Each earbud uses a low-res camera for visual Q&A with the upgraded Siri. The post cites Google Gemini support and a data-upload indicator light.

Why it matters: HKR-H/K/R all pass, but this is an unconfirmed hardware rumor, not an Apple launch. DVT status, camera design, and Gemini dependency keep it in the low featured band.

The Verge · AI

Apple’s AirPods with cameras for AI are reportedly close to production

Mark Gurman says Apple’s camera-equipped AirPods are in DVT, one step before PVT. Testers are using prototypes; the cameras capture low-resolution visual input, not photos or video, for Siri queries like ingredient prompts.

Why it matters: HKR-H/K/R all pass: Gurman/The Verge provides a concrete DVT-stage Apple AI hardware update. It is still pre-production, not a launch, so it stays in the 72–77 band.

Bloomberg Technology

Apple’s Camera-Equipped AirPods Reach Late Testing in AI Device Push

Apple moved camera-equipped AirPods into late-stage development. The RSS snippet says they may be Apple’s first wearable built for the AI era; the post does not disclose camera specs, mechanisms, or launch timing.

Why it matters: Bloomberg sourcing and camera-equipped AirPods give HKR-H/K/R. The report stays in the 72–77 band because it discloses late testing only, not specs, AI workflow, or launch timing.

May 7Thursday

AI HOT (Curated Pool)

SenseNova-U1 Open-Sources 8-Step Distilled LoRA, Speeds Diffusion Inference by 11x

SenseNova-U1 open-sourced an 8-step distilled LoRA that cuts diffusion generation from 100 steps to 8. GPU inference time drops from 23 seconds to 2 seconds, with ComfyUI workflows for text-to-image, image editing, and interleaved generation. The key signal is distillation for latency, not parameter scale.

Why it matters: HKR-H/K/R all pass: the 11x speedup hooks attention, the post gives step and latency numbers, and open LoRA affects diffusion deployment cost. Scope stays within image generation, so this is featured, not P1.

QbitAI · WeChat

Zhejiang University and Alibaba MetaCompress reaches 90% token compression for multi-turn VQA

Zhejiang University and Alibaba proposed MetaCompress, a learned token-compression framework that generates a compression mapping from the input image alone for multi-turn VQA. The article says it can remove 90% of visual tokens while preserving accuracy, and reports only 1.71% overlap between optimally retained tokens and high-attention tokens.

Why it matters: HKR-H/K/R all pass: 90% visual-token compression, no accuracy loss, and image-conditioned mapping give builders a testable cost-cutting mechanism. Zhejiang/Alibaba plus CVPR 2026 is strong research signal, not a platform-level product release.

QbitAI · WeChat

Vidu Claw Generates Ad Videos From One Prompt and a Hundred-Yuan Budget

Shengshu Technology opened Vidu Claw, which generates ad scripts, voiceover, music, editing, and final videos from one prompt; its Video Plan includes up to 40 minutes of daily generation across video, image, and audio.

Why it matters: HKR-H has a concrete ad-test hook, HKR-K adds the 40-minute daily quota and one-prompt workflow, and HKR-R hits production-cost pressure. No benchmark or pricing detail, so this stays at the featured threshold.

Xinzhiyuan · WeChat

Zhejiang University and Harvard open-source UniGeo for geometry-guided camera-controllable editing

Zhejiang University and Harvard released UniGeo with code, a report, a project page, and an HF Space. UniGeo injects geometry guidance into representation, architecture, and loss layers; it reports SOTA on DL3DV, RE10K, and Tanks against five methods. The key is video priors plus geometry-anchor attention, not just using a video model.

Why it matters: HKR-H and HKR-K pass: open code, HF Space, and three geometry-guidance layers make it testable. HKR-R is weak because it is specialized vision-generation research, so this sits near the featured floor.