Skip to content

Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

Latest picks

441–460 of 592

Apr 30Thursday

Hacker News front page

Show HN: A New Benchmark for Testing LLMs for Deterministic Outputs

Interfaze released Structured Output Benchmark, scoring schema pass rate, types, and value accuracy across text, image, and audio. Each record has a JSON Schema and human plus LLM-checked ground truth; GLM-4.7 ranks No. 2 overall. The key bug is field-level value error: GPT-5.4 ranks 3rd on text and 9th on images.

Why it matters: Interfaze released the SOB benchmark, testing structured output across 5,000 text items, 209 images and 115 audio clips, checking whether field values are correct rather than just whether the JSON parses.

Apr 29Wednesday

r/LocalLLaMA

mistralai/Mistral-Medium-3.5-128B · Hugging Face

Mistral AI released Mistral Medium 3.5 128B on Hugging Face, with 128B dense parameters and a 256k context window. It supports text and image input, function calls, JSON output, and a Modified MIT License with exceptions for high-revenue firms. Reasoning effort is configurable as none or high per request.

Why it matters: Mistral AI put Mistral Medium 3.5, a 128B dense model, on Hugging Face with image and text input, function calling, adjustable reasoning effort and a 256K-token context window.

X · @op7418

Deepseek’s multimodal model is fully rolled out

Deepseek fully rolled out a multimodal model, available via the web image-recognition mode. The post says it looks like a separate model; it does not disclose name, size, pricing, or API timing.

Why it matters: DeepSeek rolled out its multimodal model to all users, and image recognition is now live on the web version; hands-on use suggests a standalone multimodal model rather than an image-to-text pipeline.

Xinzhiyuan · WeChat

Google Translate Turns 20 as Pichai Highlights Four AI Generations

Google Translate turned 20 on April 28, and Pichai said it now has 1B monthly users. The post traces four AI phases: SMT, GNMT, PaLM 2, and Gemini 2.5 Flash Native Audio, including 110 languages added in 2024. The key shift is native speech-to-speech translation that preserves intonation, pacing, and pitch.

Why it matters: Twenty years of Google Translate, traced by Sundar Pichai from statistical machine translation to Gemini 2.5 Flash native audio translation that handles sound directly to keep tone and rhythm.

Xinzhiyuan · WeChat

MotuBrain Tops WorldArena and RoboTwin2.0 Rankings

Shengshu MotuBrain scored 63.77 EWM on WorldArena and 95.8/96.1 on RoboTwin2.0 Clean/Randomized. The post says it extends Motus with video-action modeling, Latent Action VAE, MoT, and UniDiffuser for cross-embodiment long tasks. Track reproducibility: it does not disclose training scale, submission details, or real-robot success rates.

Why it matters: Shengshu Technology's MotuBrain topped both WorldArena and RoboTwin2.0 at once, scoring 63.77 EWM on WorldArena and 95.8 and 96.1 on RoboTwin2.0's clean and randomized tests.

QbitAI · WeChat

Avenir-Web Open-Sources Web Agent Harness With 53.7% on ONLINE-MIND2WEB

UCL, Princeton, and Edinburgh open-sourced Avenir-Web, reaching 53.7% success on ONLINE-MIND2WEB. The training-free harness uses EIP, MoGE, checklists, and adaptive memory across 136 sites and 300 live tasks. The key signal: with Gemini 3 Pro, it beats Claude Computer Use 3.7 at 47.3%.

Why it matters: Avenir-Web from UCL, Princeton and Edinburgh plugs into models without extra training and hits 53.7% success on ONLINE-MIND2WEB, 136 sites and 300 real tasks.

QbitAI · WeChat

DeepSeek’s multimodal AI has entered testing

DeepSeek researchers confirmed V4 vision mode is in gray testing, with an image-recognition mode on the homepage. A screenshot shows it identified drinks and cup types in a non-text-heavy image after 4 seconds. The post does not disclose rollout scope, API access, or pricing.

Why it matters: DeepSeek researchers confirm the V4 vision version is in gray-scale testing, with an image recognition entry on the homepage and a screenshot showing drink and cup types identified in 4 seconds.

QbitAI · WeChat

ShengShu Technology Claims MotuBrain, a Dual-Benchmark Robot Brain for Long-Horizon Tasks

ShengShu Technology claimed MotuBrain on April 29 after it topped WorldArena and RoboTwin2.0 in mid-April. It scored 95.8 and 96.1 in RoboTwin2.0 Clean and Randomized settings, and a demo used 3 humanoid robots across 5 tasks. The key detail is its World Action Model: a video-action-language MoT design for cross-embodiment tasks beyond 10 atomic actions.

Why it matters: Shengshu Technology claims MotuBrain, which topped WorldArena and RoboTwin2.0, and shows three robots handing off a long-horizon task, a concrete look at a video company moving into embodied AI.

r/LocalLLaMA

SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-Unify Architecture

SenseNova released SenseNova-U1 with 4 MoT multimodal models. The post lists 8B and A3B variants with GitHub and HuggingFace weight links. The key claim is a monolithic architecture instead of adapters; benchmarks are not disclosed.

Why it matters: SenseNova-U1 handles text and images inside the model with no external image adapter, and releases 8B and A3B MoT weights, with A3B smaller and suited to local runs.

Bloomberg Technology

Apple Readies Photo-Editing Overhaul With New AI Tools in iOS 27

Apple plans to overhaul built-in photo editing for iPhone, iPad, and Mac in iOS 27 with AI tools. The RSS snippet says it targets Android competition; the post does not disclose features, models, timing, or supported devices.

Why it matters: Bloomberg reports Apple's planned AI photo tools for iOS 27 and macOS 27, with expand, enhance and recompose pointing to where system-level editing is headed.

NVIDIA Blog

NVIDIA Launches Nemotron 3 Nano Omni for Vision, Audio, and Language Agents

NVIDIA launched Nemotron 3 Nano Omni, claiming up to 9x higher throughput at the same interactivity. It uses a 30B-A3B hybrid MoE with Conv3D, EVS, and 256K context, taking text, images, audio, video, documents, charts, and GUIs as input. Open weights, datasets, and training methods arrive April 28, 2026 on Hugging Face, OpenRouter, build.nvidia.com, and 25+ platforms.

Why it matters: NVIDIA open-sourced Nemotron 3 Nano Omni, one model handling image, text, audio and video with 3B active parameters and 256K context, giving teams building multimodal agents an open path.

Apr 28Tuesday

QbitAI · WeChat

ModelBest Releases MiniCPM-o 4.5 Technical Report for Consumer-GPU Deployment

ModelBest, OpenBMB, Tsinghua THUNLP and THUMAI released the MiniCPM-o 4.5 technical report, covering a roughly 9B-parameter model. It supports video, audio and text streams; a 12GB RTX 5070 runs full-duplex mode at RTF 0.4. The key mechanism is Omni-Flow: a unified timeline with time-division multiplexing, without external VAD.

Why it matters: ModelBest with Tsinghua and others published the MiniCPM-o 4.5 technical report, a roughly 9B-parameter model using Omni-Flow to time-share audio, video and text on one timeline, running full-duplex on a 12GB consumer GPU.

QbitAI · WeChat

Open-source SenseNova-U1 unifies image understanding and generation

SenseTime open-sourced two SenseNova-U1 models: an 8B version and a 38B-total MoE version using NEO-unify. The architecture removes VE and VAE, processes pixels directly, and generates 2048×2048 images in about 9 seconds on one H100/H200 node. The key item is interleaved text-image reasoning; 32K context, long-text rendering, and beta interleaved creation remain limits.

Why it matters: SenseTime open-sourced two SenseNova-U1 models using a NEO-unify architecture that drops the vision encoder and VAE to work on pixels directly, generating a 2048x2048 image in about 9 seconds on one H100 or H200.

Synced · WeChat

Open-source medical video understanding system uAI-NEXUS-MedVLM released

United Imaging Intelligence released uAI-NEXUS-MedVLM for medical video understanding, with a CVPR 2026 paper. MedVidBench has 532k video-instruction pairs across 8 medical sources and 8 tasks. Qwen2.5-VL-7B SFT reached 89.4% CVS accuracy; GPT-5.4 scored 16.4%.

Why it matters: United Imaging Intelligence open-sourced uAI-NEXUS-MedVLM, a large model for ultrasound, endoscopy and other medical video, plus MedVidBench, an evaluation set of 532,000 video instruction pairs.

Xinzhiyuan · WeChat

NUS and NTU Release Pask with Streaming Intent Detection and Persistent Memory

NUS and NTU released Pask, with paper arXiv:2604.08000. Pask uses DD, MM, and PAS, with IntentFlow detecting intent in 1.5 seconds. The key bet is real-time intent detection, not longer execution chains.

Why it matters: Pask, from NUS and NTU, uses its IntentFlow component to detect user intent in real time within 1.5 seconds and adds permanent memory, offering proactive AI a path based on real-time intent recognition.

r/LocalLLaMA

Microsoft Presents TRELLIS.2: Open-Source 4B Image-to-3D Model

Microsoft’s title says TRELLIS.2 is an open-source 4B image-to-3D model. The title lists 1536³ PBR assets, native 3D VAEs, and 16× spatial compression; the Reddit body is blocked by 403 and discloses no license or benchmarks.

Why it matters: Microsoft open-sourced TRELLIS.2, a 4B-parameter image-to-3D model that outputs assets up to 1536³ with PBR materials, worth a look for anyone tracking open 3D generation.

Apr 27Monday

The Verge · AI

Canva apologizes after its AI tool replaces ‘Palestine’ in designs

Canva said Magic Layers replaced “Palestine” with “Ukraine” in designs. The tool should split flat images into editable layers, not alter visible content; X user @ros_ie9 said “Gaza” was unaffected. Canva says it fixed the issue; the post does not disclose the trigger mechanism.

Why it matters: Canva's Magic Layers swapped the word Palestine for Ukraine in design text; Canva confirmed a fix but did not explain what triggered it, a concrete case of how AI editing tools handle political terms.

Xinzhiyuan · WeChat

Five Months After Altman’s Code Red, GPT Image 2 Tops Arena Image Rankings

GPT Image 2 topped three Arena image charts within 12 hours, scoring 1512 in text-to-image and beating Nano Banana 2 by 241 points. Arena calls it the largest Image Arena gap, with 93% blind-test wins and a 316-point text-rendering gain. The key shift is native thinking: planning, self-checking, web search, and 8 coherent images per run.

Why it matters: Within half a day of launch, GPT Image 2 took three first places on the Arena image leaderboard, scoring 1512 in text-to-image, 241 points above second, with a 93 percent blind-test win rate.

QbitAI · WeChat

Meshy tops 10M users and moves into 3D printing as ARR rises 14x

Meshy says it passed 10M registered users, reached $40M ARR, and grew 2025 revenue 14x year over year. Meshy Creative Lab supports keychain, magnet, and keycap design; physical ordering is not live yet. The key signal is print fit: 97% slice-pass rate in Bambu Studio across 75 tested models.

Why it matters: Meshy passed 10 million registered users and $40 million ARR, and its new Creative Lab generates models for small objects like keychains and fridge magnets, showing AI 3D content moving into physical consumer goods.

Synced · WeChat

Apple paper asks: What do your logits know?

Apple researchers posted an arXiv paper testing whether VLM top-k logits leak image details. Using CLEVR, MSCOCO, and probes, 30–80 logits recover noise, target traits, and some background attributes. The key risk is gray-box APIs exposing top-k log probabilities.

Why it matters: Apple's experiments on CLEVR and MSCOCO found that the top 30 to 80 logits alone can recover image noise, object features and some background attributes, giving quantifiable evidence of output-interface risk in vision-language models.