Skip to content

Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

Latest picks

501–520 of 592

Apr 17Friday

X · @op7418

Seedance 2.0 API is now fully open

Volcano Engine has opened the Seedance 2.0 API to domestic users, while BytePlus serves overseas access; the API currently accepts 4 input modalities: text, image, audio, and video. The post also confirms face registration, portrait authorization, and preset virtual avatars, but does not disclose pricing, rate limits, model variants, or regional availability. The real watchpoint is whether video-agent workflows can be wired through Skills and MCP, not the ecosystem rhetoric.

Why it matters: Volcano Engine fully opened the Seedance 2.0 API, which takes text, image, audio and video inputs and adds face registration and portrait authorization, giving video generation a compliant integration path.

Latent Space

[AINews] Anthropic Claude Opus 4.7 - one step better than 4.6 in every dimension

Anthropic launched Claude Opus 4.7 at the same $5/$25 per million input/output tokens; the post says 4.7-low through 4.7-high each outperform the matching higher 4.6 tiers. Reported changes include a new xhigh reasoning tier, Claude Code defaulting to xhigh, an 11-point gain on SWE-Bench Pro, and image input up to 2,576 px on the long edge (~3.75 MP). Do not overread the tokenizer change: the same input can use up to 35% more tokens, but the post says total token use still falls by up to 50% from prior equivalents.

Why it matters: Claude Opus 4.7 costs the same as 4.6, yet its low tier now beats 4.6's mid tier, so every tier effectively moves up one notch.

Hacker News front page

Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7

Simon Willison ran a 20.9GB quantized Qwen3.6-35B-A3B on a MacBook Pro M5 and judged its SVG pelican output better than Claude Opus 4.7. He used LM Studio with an Unsloth Q4_K_S GGUF, then repeated the test with “a flamingo riding a unicycle” and again scored Qwen higher. This is not a general capability result; the author says this joke benchmark no longer tracks overall model usefulness in this comparison.

Why it matters: Simon Willison ran a 20.9GB quantized Qwen3.6-35B-A3B on an M5 MacBook Pro and asked it and Claude Opus 4.7 for the same pelican SVG; Qwen got the bike frame right and Opus missed it twice.

X · @dotey

browser-use open-sources video-use, a Claude Code skill that turns raw camera footage into edited videos

browser-use released video-use, a Claude Code skill that turns raw footage into a final.mp4 automatically. It converts footage into ElevenLabs word-level timestamp transcripts, shrinking one asset to about 12KB; the post says feeding frames directly would cost about 45 million tokens. The key detail is the structured editing pipeline: the model mostly reads text, uses timeline images only at uncertain cuts, and runs up to 3 self-check repair passes after rendering.

Why it matters: browser-use open-sourced video-use as a Claude Code skill that takes raw footage and handles filler-word removal, color grading, subtitles and animation overlays to produce a finished cut.

Apr 16Thursday

X · @dotey

Anthropic officially releases Claude Opus 4.7 at unchanged pricing

Anthropic released Claude Opus 4.7 at unchanged pricing: $5 per million input tokens and $25 per million output tokens; the API name is claude-opus-4-7, now live across Claude, Amazon Bedrock, Google Cloud Vertex AI, and Microsoft Foundry. The post gives two concrete changes: vision input now supports up to 2576 pixels on the long edge, and the new tokenizer can raise token usage to 1.0-1.35x for the same text. Watch migration cost, not list price; higher reasoning settings and multi-turn runs can increase output length and bills.

Why it matters: Anthropic's Claude Opus 4.7 keeps Opus 4.6 pricing, adds autonomous coding and 2576px vision, and its new tokenizer may raise token use per task.

X · @op7418

Anthropic releases Claude Opus 4.7 with the following main updates

Anthropic has rolled out Claude Opus 4.7 across all Claude products and the API, with pricing unchanged from Opus 4.6. The post lists better long-horizon task handling, more precise instruction following, self-verification before reporting, vision support up to 2,576-pixel long-edge images, plus Claude Code Ultra Review, an xhigh thinking level, and auto-approval for Max users.

Why it matters: Anthropic shipped Opus 4.7 across all products and the API at Opus 4.6 pricing, with longer task handling and self-checking before output.

Hacker News front page

Introducing Claude Opus 4.7

Anthropic released Claude Opus 4.7 on Apr. 16 at the same price as Opus 4.6: $5 per million input tokens and $25 per million output tokens. The post says it improves on Opus 4.6 in advanced software engineering, long-running tasks, and higher-resolution vision, and ships across Claude, the API, Amazon Bedrock, Vertex AI, and Microsoft Foundry. The key detail is the first deployment of Anthropic’s cyber request blocking on a less capable model; the post cites benchmark gains but does not fully disclose every score in text.

Why it matters: Anthropic launched Claude Opus 4.7 at the same price as the prior generation, focused on stability in complex software engineering and long workflows, with cyber-security blocks deployed on weaker models.

Hacker News front page

Qwen3.6-35B-A3B: Agentic coding power, now open to all

Qwen released Qwen3.6-35B-A3B as open weights, with 35B total parameters and 3B active parameters. The post reports 73.4 on SWE-bench Verified, 51.5 on Terminal-Bench 2.0, and 92.0 on RefCOCO. The key point is agentic coding and multimodal performance at a 3B active-parameter budget, with weights, Qwen Studio, and API access available.

Why it matters: Qwen open-sources Qwen3.6-35B-A3B, a coding-agent model with 35B total parameters that activates only 3B per inference, so the trade-off is deployment cost.

r/LocalLLaMA

Qwen3.6-35B-A3B released

Qwen released Qwen3.6-35B-A3B as open source under Apache 2.0; it is a sparse MoE with 35B total parameters and 3B active. The post also claims agentic coding, strong multimodal perception and reasoning, plus thinking and non-thinking modes; the post does not disclose benchmarks, context length, or latency.

Why it matters: Qwen releases Qwen3.6-35B-A3B under Apache 2.0 for commercial use, with 35B total parameters and 3B active per inference, making the sparse MoE efficiency route the main point.

Hacker News front page

Darkbloom – Private inference on idle Macs

Eigen Labs launched Darkbloom, linking 100M+ Apple Silicon Macs into a decentralized inference network. It offers an OpenAI-compatible API, claims end-to-end encryption plus hardware attestation, and lists prices up to 70% below OpenRouter comps. The real point is the trust model: hardware keys, hardened runtime, and signed outputs are disclosed, but enterprise audit scope still needs the paper.

Why it matters: Eigen Labs pools idle compute from over 100 million Apple-silicon Macs into a decentralized inference network with an OpenAI-compatible API, end-to-end encryption and token pricing 50% below OpenRouter.

TechCrunch · AI

Google rolls out a native Gemini app for Mac

Google launched a native Gemini app for Mac on April 15 for all users worldwide on macOS 15 and later, with Option + Space as the summon shortcut. Users can share their screen or local files with Gemini, and the app also supports image generation with Nano Banana and video generation with Veo. The key shift is desktop access plus live context sharing, not just another client.

Why it matters: Google launched a native Gemini app for Mac that opens anywhere with Option plus Space and can take screen content or local files as input.

Apr 14Tuesday

最佳拍档 (BestPartners)

Global GPU shortage worsens: H100 rental prices rose nearly 40% in five months

SemiAnalysis says Nvidia H100 one-year rental pricing rose from $1.70 to $2.35 per GPU-hour between Oct 2025 and Mar 2026, up nearly 40% in five months. The post attributes this to Anthropic-driven demand, multi-agent and media generation workloads, and memory cost spikes, with LPDDR5 and DDR5 contract prices up about 4x and 5x year over year; much new capacity is already prebooked. The key variable is the supply gap, not Blackwell refreshes alone.

Why it matters: SemiAnalysis reports H100 one-year rental prices rose from $1.70 to $2.35 per hour in five months, with spot clusters of 64 GPUs now hard to rent, putting compute tightness into concrete numbers.

Apr 11Saturday

QbitAI · WeChat

OpenClaw-style methods reach multimodal generation, with a 6B model beating Nano Banana 2 on some tasks

A team led by Shanghai AI Laboratory introduced GEMS, adding Agent Loop, Memory, and Skills to multimodal generation, and reports that 6B Z-Image-Turbo beats Nano Banana 2 on some tasks. The post reports +14.22 average gains on 5 mainstream tasks and +8.92 over the best baseline on 4 downstream tasks; the paper and code are public, but the post does not disclose Nano Banana 2's full setup.

Why it matters: Shanghai AI Lab brings agent loops, memory and skill modules into multimodal generation, lifting a 6B model by 14.22 points on average across 5 tasks.

QbitAI · WeChat

Liu Zhuang and Danqi Chen team open-source Vero, a general visual reasoning RL framework, reaching SOTA with zero thinking data

Princeton researchers including Liu Zhuang and Danqi Chen open-sourced Vero, an RL framework for visual reasoning, and report beating Qwen3-VL-8B-Thinking on 23 of 30 benchmarks. The post says Vero uses 600K samples filtered from 59 datasets, task-routed rewards, and single-stage RL across six task groups. The key point is the mechanism mix: no private thinking data, but the post does not disclose training cost or base model configuration.

Why it matters: Princeton's Vero, open sourced by Zang Liu and Danqi Chen's teams, filters 600,000 samples from 59 datasets and beats Qwen3-VL-8B-Thinking on 23 of 30 benchmarks after single-stage RL.

QbitAI · WeChat

A Chinese embodied model reached global No.1 as a 100,000-hour human dataset for robots was released

Psibot says it released a 100,889-hour human-plus-robot manipulation dataset, and that Psi-R2 ranked first on AllenAI’s MolmoSpace benchmark. The post lists 95,472 hours of human data, 5,417 hours of robot data, 1,000 open-sourced hours, 294 scenes, 4,821 tasks, and 1,382 objects; Psi-W0 adds 30% failure samples, and Psi-R2 latency drops from 2.2s to under 100ms. The key point is the data loop and benchmark framing: the post claims nearly 10x higher success, but does not disclose task setup, full baselines, or statistics.

Why it matters: Psi-R2 tops AllenAI's MolmoSpace benchmark and ships a 100,889-hour manipulation dataset, 95,472 hours of it human, with 1,000 hours open sourced so far.

Apr 10Friday

QbitAI · WeChat

Tencent open-sources 3B SVG model HiVG to make tokens geometry-aware

Tencent Hunyuan open-sourced the 3B-parameter HiVG, claiming 62.7%-63.8% shorter SVG sequences via hierarchical tokenization and better SVG generation metrics than GPT-5.2, Claude-4.5-Sonnet, and some 8B open models. The post reports 0.896 SSIM, 0.114 LPIPS, and 0.957 CLIP-S on Image-to-SVG; the core method packs drawing commands plus coordinates into segment tokens and uses HMN to initialize coordinate embeddings. The part to watch is token design, not parameter count; paper, code, and project page are public.

Why it matters: Tencent Hunyuan open-sources HiVG, a 3B-parameter model that packs drawing commands and coordinates into segment tokens with initialized coordinate encoding, producing SVG sequences about 60% shorter than conventional methods.

Apr 9Thursday

QbitAI · WeChat

Beyond MoE, Tencent introduces MoT: a 2B embodied model ranks first in 16 of 22 evaluations

Tencent Hunyuan and Robotics X released HY-Embodied-0.5; its MoT-2B uses 4B total params with 2B active and ranks first in 16 of 22 embodied evaluations. The post says it uses 100M+ embodied data, 600B+ pretraining tokens, 30M+ mid-training samples, plus visual latent tokens, bidirectional attention, RFT, RL, and online distillation. The key point is a rebuilt edge-oriented embodied stack, not a simple VLM fine-tune.

Why it matters: Tencent's HY-Embodied-0.5 reports 16 first-place finishes across 22 embodied AI benchmarks with a 4B-parameter model that activates 2B, plus disclosed training data and architecture.

X · @op7418

Meta releases Muse Spark model

Meta released the Muse Spark model with native multimodal reasoning, tool use, visual chain-of-thought, and multi-agent orchestration, but it is only available in the Meta AI app and is not open source for now. The snippet says its Contemplating mode coordinates multiple parallel agents for reasoning, and its Artificial Analysis score is below Gemini 3.1 Pro, GPT-5.4, and Claude Opus 4.6. The post does not disclose model size, pricing, or rollout timing.

Why it matters: Meta's Muse Spark, built by its recruited team, ships native multimodal reasoning and parallel agents, with a thinking mode that coordinates several agents on one problem.

Apr 8Wednesday

QbitAI · WeChat

Xiaomi unveils two AI audio frameworks: Any2Speech and Midasheng-audio-generate

Xiaomi's large-model application team introduced Xiaomi Any2Speech and Midasheng-audio-generate. Any2Speech generates up to about 10 minutes per inference, while the other model turns one text prompt into mixed audio with speech, music, and ambient sound. The post names GST labeling, dual-path planning with dimension dropout, Flow Matching, and five-field structured labels; benchmark scores, training scale, and commercial terms are not disclosed.

Why it matters: Xiaomi's team published two speech frameworks, one generating roughly 10 minutes of audio per inference, the other mixing voice, music and ambient sound from text.

QbitAI · WeChat

After a late-night update, DeepSeek reportedly said: I am V4?

DeepSeek added Fast and Expert modes on its web app and started gray-testing a Vision model; the claim that Expert mode is V4 comes only from user probes and the model’s own replies. The post gives one concrete detail: Expert mode focuses on code, web, and harder generation tasks, is supply-limited, does not support multimodal or file upload, and one user reported a length cap at about 133K tokens. What matters is the official model ID and context spec; the post does not disclose them, pricing, or a release timeline.

Why it matters: DeepSeek's web version now splits into fast and expert modes, and users measured the expert mode's context ceiling at about 133,000 tokens.