Skip to content

Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

Latest picks

281–300 of 592

May 30Saturday

The Verge · AI

Tech companies desperately want to film you doing chores

Shift said it would clean New Yorkers’ homes for free if it can film cleaners doing chores such as washing dishes, wiping counters, dusting tables, and mopping floors, creating domestic robot training data; the snippet does not disclose consent terms, data retention, pricing, or expansion timing for cities such as London.

Why it matters: Shift trades free house cleaning for video of New York households washing dishes and mopping floors, footage meant to train home robots, with plans to expand to London and other cities.

May 29Friday

The Verge · AI

This AI startup will clean your home for free to train future robots

Shift offers free home cleaning and records cleaners scrubbing, vacuuming, dusting, tidying, and washing to collect robot training footage; the RSS snippet does not disclose service cities, privacy terms, consent mechanics, or dataset scale.

Why it matters: Shift trades free house cleaning for footage of the work, turning ordinary chores into a robot-training data pipeline whose data value covers the service cost.

The Verge · AI

Adobe’s Conversational AI Agent Is a Mediocre Design Intern

The Verge tested Adobe Firefly AI Assistant in beta. It can operate Adobe design apps as a conversational middleman, rather than only generating images or video. The post says it explains edit steps clearly, but the results were not impressive. The RSS snippet does not disclose pricing, release timing, or the full list of supported apps.

Why it matters: A Verge editor tries Adobe's Firefly AI assistant beta, which drives design software through conversation instead of generating images, and finds the output mediocre.

AI HOT (Curated Pool)

Xiaomi Open-Sources Controllable Video Foley Model ControlFoley

Xiaomi’s large model application team open-sourced ControlFoley, a controllable video Foley model supporting three tasks: text-guided video dubbing, text-controlled video dubbing, and reference-audio-controlled video dubbing, with code, model weights, and an online demo released.

Why it matters: Xiaomi open-sources ControlFoley, letting video sound effects be specified by text or reference audio instead of guessed by the model, while staying aligned to on-screen motion.

AI HOT (Curated Pool)

Google DeepMind CEO Demis Hassabis Says AGI Could Arrive Within Three Years

Demis Hassabis predicts AGI could arrive around 2029 to 2030, with mature multimodal capabilities and autonomous decision-making as key conditions, while warning that society remains underprepared and needs rules and safeguards before deployment.

Why it matters: Hassabis moves his AGI timeline up to 2029-2030 and calls independently working agents its preview, a notably faster estimate than before.

r/LocalLLaMA

StepFun 3.7 Flash

StepFun released Step 3.7 Flash with 196B total parameters, 11B active MoE, a built-in 1.8B ViT, and local execution on 128GB RAM.

Why it matters: StepFun released Step 3.7 Flash, a 196B-parameter model that activates only 11B, plus a 1.8B vision module, and says it runs on a local machine with 128GB of memory.

AI HOT (Curated Pool)

StepFun Releases Step 3.7 Flash, Focused on Agent Efficiency

StepFun released the open-source Step 3.7 Flash model with a 198B-parameter MoE architecture, about 11B active parameters, a 256K context window, and a 67.1 score on ClawEval-1.1.

Why it matters: StepFun open-sourced Step 3.7 Flash, a 198B-parameter MoE model that activates about 11B parameters at runtime and supports 256K context and tool calling, giving agent workflows a lighter deployment option.

AI HOT (Curated Pool)

Nano Banana Pro and Nano Banana 2 officially released

Google AI Developers released Nano Banana Pro and Nano Banana 2, two image models available for production use through the Gemini API; the post names gemini-3-pro-image and gemini-3.1-flash-image but does not disclose pricing, benchmarks, or rate limits.

Why it matters: Google added the Nano Banana Pro and Nano Banana 2 image models to the Gemini API, so developers can call them now and use them in production.

The Verge · AI

A $2,000 AI-generated film will debut at Tribeca

Tribeca Festival will premiere the 75-minute AI-generated film Dreams of Violets, which cost $2,000 to make and uses people and images fully created by AI.

Why it matters: A 75-minute narrative feature with every on-screen character generated by AI will premiere at Tribeca, made for $2,000 from news reports and witness accounts.

May 28Thursday

NVIDIA Blog

NVIDIA Research Advances Robotics From Simulation to the Real World

NVIDIA Research presented 8 ICRA papers on sim-to-real robotics: ScheduleStream delivered a 3x speedup for multi-arm planning, COMPASS reached about 80% success across 20 real-world navigation trials, and Grasp-MPC achieved about 75% real-robot grasping success.

Why it matters: NVIDIA's COMPASS navigates real robots after simulation-only training, hitting roughly 80% success across 20 trials, 4.5x a pure imitation baseline.

r/LocalLLaMA

Nvidia LocateAnything: Fast Vision-Language Grounding with Parallel Box Decoding

The title says Nvidia LocateAnything-3B performs vision-language grounding with parallel box decoding and runs 10x faster than Qwen3-VL; the post body only provides Hugging Face, GitHub, demo, and project links, and does not disclose benchmark setup or accuracy numbers.

Why it matters: Nvidia open-sourced LocateAnything, a 3B vision grounding model that uses parallel box decoding instead of serial generation and claims 10x the speed of Qwen3-VL.

Synced · WeChat

ICML 2026: AutoMoT reaches SOTA on Bench2Drive and nuScenes

NTU AutoMan Lab, Harvard, and Xiaomi Auto proposed AutoMoT, a unified VLA driving model using a 4B Qwen3-VL Understanding Expert and a 1.6B Action Expert with asynchronous inference, reaching 89.42 DS and 74.09% SR on Bench2Drive with AutoMoT+.

Why it matters: AutoMoT splits a 4B vision-language model and a 1.6B action expert into asynchronous inference, taking first place on both Bench2Drive and nuScenes, a split-model idea for end-to-end driving.

Synced · WeChat

Chinese pretrained embodied model Wall-OSS-0.5 is open sourced

X Square Robot open sourced Wall-OSS-0.5, a VLA model whose 400k pretraining checkpoint scored above 80 on 4 of 17 real-robot zero-shot tasks, with weights, code, training recipe, ablations, and a DMuon optimizer implementation released.

Why it matters: X Square Robot open-sources Wall-OSS-0.5, a vision-language-action model with a 400k-step pretrained checkpoint, weights, code and training recipe; 4 of 17 real-robot zero-shot tasks top 80% success.

AI HOT (Curated Pool)

Open-source FastVideo Dreamverse real-time video generation tool

Hao AI Lab open-sourced FastVideo Dreamverse, a real-time video generation tool that generates a 30-second 1080p video in 7 seconds under the stated setup of one NVIDIA B200 GPU and LTX-2.

Why it matters: Hao AI Lab open-sourced FastVideo Dreamverse, which pairs LTX-2 with a single NVIDIA B200 to generate 30 seconds of 1080p video in 7 seconds, code and blog included.

May 27Wednesday

AI HOT (Curated Pool)

Runway launches Model Context Protocol server

Runway launched an MCP server that lets compatible agents such as Claude, ChatGPT, and Cursor generate images and videos inside chat interfaces, with access to Gen-4.5, Seedance 2.0, GPT Image 2, Kling 3.0, and Nano Banana Pro.

Why it matters: Runway put its image generation and video editing inside Claude, ChatGPT and Cursor chat boxes, so users send a link, reference image or text prompt and get a finished result without switching apps.

TechCrunch · AI

YouTube will now automatically label AI videos

YouTube will automatically label videos using significant photorealistic AI, no longer relying only on creator self-disclosure. The RSS snippet says AI labels will become more prominent, but the post does not disclose rollout timing, detection thresholds, appeal rules, or whether the system covers shorts and livestreams.

Why it matters: YouTube will use internal systems to detect and label videos with substantial realistic AI imagery, with labels more prominent on long videos and Shorts, shifting AI labeling from creator disclosure to active detection.

QbitAI · WeChat

7B Medical AI Agent Beats o3 and GPT-5 by Learning Where and How to Look

Shanghai Innovation Institute’s LeapQuest and three universities released Ophiuchus and MedScope, applying Think with Images and Think with Videos to medical AI; Ophiuchus-7B scored 68.0 on eight VQA benchmarks, above OpenAI-o3 at 62.2, Gemini 2.5 Pro at 61.8, and GPT-5 at 59.9.

Why it matters: Ophiuchus-7B scores 68.0 across 8 medical visual QA benchmarks, beating o3 and GPT-5, a sample of a small model catching general models through method design.

r/LocalLLaMA

PrismML Released Binary and Ternary Bonsai Image 4B

PrismML released Binary and Ternary Bonsai Image 4B, 1-bit and ternary text-to-image diffusion transformers around 3GB, compared with FLUX.2 Klein 4B at about 16GB, with browser-local WebGPU demo links and an Apache-2.0 license disclosed in the Reddit snippet.

Why it matters: PrismML compressed Bonsai Image 4B's text-to-image diffusion transformer weights to 1-bit or ternary, about 3GB against FLUX.2 Klein 4B's roughly 16GB, and it runs locally in a browser on WebGPU.

May 26Tuesday

AI HOT (Curated Pool)

SenseNova-U1 full training code open-sourced for multimodal multitask training

OpenSenseNova released the full SenseNova-U1 training code on GitHub under Apache-2.0, supporting an 8B dense model, an A3B MoE architecture, and multimodal tasks such as text-to-image generation, image editing, interleaved generation, and text-visual understanding.

Why it matters: SenseTime open-sourced SenseNova-U1's full training code on GitHub under Apache-2.0, covering 8B dense and A3B MoE variants, with one framework handling text-to-image, image editing, interleaved generation and understanding.

AI HOT (Curated Pool)

Project Luxo: Crossing the Uncanny Valley of AI Media

Runway released Project Luxo, showing AI shorts and ad samples including The Rogue; each work was made by a single-person team, with production times ranging from three weeks to four hours.

Why it matters: Runway shows three short films and an ad spot made by one person, cutting production from three weeks to four hours, with feedback from producers, actors and union members.