Multimodal

Beyond text: vision, mixed image-text, audio and video input and output in models and products.

514 picksRelated topicsImage generation AI video Voice & audio

Latest picks

221–240 of 514

May 26Tuesday

AI HOT (Curated Pool)May 26, 22:58

SenseNova-U1 full training code open-sourced for multimodal multitask training

OpenSenseNova released the full SenseNova-U1 training code on GitHub under Apache-2.0, supporting an 8B dense model, an A3B MoE architecture, and multimodal tasks such as text-to-image generation, image editing, interleaved generation, and text-visual understanding.

Why it matters: HKR-H/K/R all pass, but the source is a short official post with no dataset, training budget, or eval results disclosed. The practical value of full training code puts it in the featured band.

Take

SenseTime is betting on reproducible training plumbing, not a SenseNova-U1 leaderboard story. The hard details are useful: an 8B dense model, an A3B MoE path, Apache-2.0, and one framework covering text-to-image, image editing, interleaved generation, and vision-language understanding. For practitioners, mixed parallelism and a resumable streaming data pipeline matter more than another polished demo, because multimodal training usually breaks on data restarts, config drift, and cluster scaling.

I still have doubts about the “large-scale training” claim. The snippet says it runs from 1×8 GPUs to multi-node clusters, but gives no throughput, memory curve, recovery trace, weights, or data recipe. Compared with Qwen or Llama-style open weights, SenseNova-U1 reads more like SenseTime exposing its internal training scaffold first. Apache-2.0 helps, but the value lives in real launch scripts and failure handling, not a pretty README.

AI HOT (Curated Pool)May 26, 22:34

Project Luxo: Crossing the Uncanny Valley of AI Media

Runway released Project Luxo, showing AI shorts and ad samples including The Rogue; each work was made by a single-person team, with production times ranging from three weeks to four hours.

Why it matters: HKR-H/K/R all pass, but this is a Runway research showcase with samples, not a new model or shipped product capability. It lands at the lower end of the good-quality band.

Take

Runway’s strongest claim is not “the uncanny valley is crossed.” It is the production math: one person made The Rogue in three weeks, Last Night in seven hours, and Pigeons in Time in four hours. That is brutal pressure on spec ads, pitch films, and previsualization teams.

I don’t buy the validation layer. Runway says producers, actors, guild members, studios, press, and other participants all agreed “the films worked,” but gives no sample size, blind test, retention data, survey design, or control group. Sora and Veo have produced impressive clips for a year; the hard part has been sustained performance, shot continuity, and cheap iteration under creative direction. Project Luxo reads like a strong enterprise sales deck, not a clean proof that AI video has graduated from the uncanny valley.

Alibaba Technology · WeChatMay 26, 18:50

Nearly 9x training speedup: residual streams in DiT are becoming a convergence bottleneck

Nanjing University LAMDA and Alibaba Intelligent Engine proposed DAR, a timestep-aware cross-layer routing method that replaces fixed residual accumulation in DiT; on ImageNet 256x256, it reduced SiT-XL/2 FID from 9.67 to 7.56 and reached baseline convergence quality with 8.75x fewer training iterations.

Why it matters: HKR-H/K/R all pass, but the topic is a narrow DiT training method rather than a broad model or product launch. Concrete ImageNet metrics and the Alibaba/LAMDA mechanism clear the featured bar, not the 78+ band.

QbitAI · WeChatMay 26, 13:30

Zhejiang University and Alibaba Make AI Think Before Drawing Sudoku or Burning Candles | ACL 2026

Zhejiang University and Alibaba introduced Unified Thinker, an independent planning module trained with 40,000 HieraReason-40K samples and a two-stage GRPO reinforcement-learning setup that turns structured reasoning traces into executable visual instructions for image generation and editing.

Why it matters: HKR-H/K/R all pass: the paper has a concrete visual-failure hook, a 40k-sample planning/RL mechanism, and relevance to multimodal-agent reliability. It remains a paper-level advance, not a product or flagship model release.

Take

Unified Thinker hits the open image-model problem cleanly: image quality is no longer the only bottleneck; executable planning is. The concrete hook is 40K HieraReason-40K samples, an independent Thinker module, and two-stage GRPO where generated image quality rewards the reasoning trace. That is a more credible path than bolting a general LLM onto a diffusion model and hoping the prompt survives translation.

The best detail is the “do not describe unchanged regions” rule for editing. It sounds low-tech, but it targets semantic drift, the thing that makes diffusion edits quietly ruin the rest of the image. I’m less sold on the “comparable to closed models” claim: the article names RISEBench and WiseBench but gives no scores. Nano Banana and GPT-Image win with planner quality, private data, and generator scale.

AI HOT (Curated Pool)May 26, 02:09

Grok Build Beta Opens to SuperGrok Users

xAI opened Grok Build Beta to all SuperGrok and X Premium+ users, with Plan Mode, Imagine-based image and video creation, and a CLI for automation or orchestrator workflows at x.ai/cli.

Why it matters: HKR-H/K/R all pass: xAI opened a paid beta with named workflow features. The score stays at the featured floor because the post lacks capability limits, pricing detail, and test results.

May 25Monday

r/LocalLLaMAMay 25, 21:14

NuExtract3 released: open-weight 4B VLM for Markdown, OCR and structured extraction

Numind released NuExtract3, a 4B open-weight VLM based on Qwen3.5-4B under Apache-2.0, supporting image and text to Markdown, OCR, and JSON-template extraction, with self-hosting from 4GB VRAM and weights in Safetensors, GGUF, and MLX formats.

Why it matters: HKR-H/K/R all pass: NuExtract3 packages OCR, Markdown, and structured extraction into a 4B open-weight VLM with a 4GB self-hosting condition. Source and lab reach keep it in the low featured band.

Synced · WeChatMay 25, 14:32

A 1B-Gaussian 3D World Runs in the Browser, Outperforming Fei-Fei Li’s Spark

Manycore Tech open-sourced Aholo Viewer, a browser-based 3D Gaussian Splatting viewer that used half the memory of Spark 2.0 in a 300M-Gaussian test, loaded 2x faster, rendered 3x faster, and supports scenes with up to 1B Gaussian points.

Why it matters: HKR-H/K/R all pass: the hook is vivid, the post gives 300M-point benchmarks and a 1B-point ceiling, and browser-side 3D deployment matters to practitioners. Score stays at 80 because this is a strong tool release, not a foundation-model event.

May 24Sunday

Synced · WeChatMay 24, 15:00

ICML 2026: First Parallel Thinking Framework for Vision-Language Models

Visual Para-Thinker introduces a parallel thinking framework for vision-language models, using Pa-Attention and LPRoPE to isolate four visual reasoning paths and training on 163,000 question-answer pairs.

Why it matters: HKR-H/K/R pass: the ICML 2026 paper offers a concrete parallel-thinking mechanism, four isolated paths, and 163K training pairs. It remains a single research release without broad replication or product impact, so it fits 78–84.

Take

Visual Para-Thinker’s useful part is the mechanism, not the “parallel thinking” branding. It isolates four visual reasoning paths with Pa-Attention, keeps shared position ranges unbiased, then adds LPRoPE so paths stay distinguishable. The training set is also concrete: 163,000 QA pairs distilled mainly from Qwen3-VL-235B-A22B-Instruct.

That targets a real VLM failure mode. Long CoT often dilutes attention over visual tokens, which shows up as hallucination rather than better reasoning. The reported gains are nontrivial: +12.6 / +6.3 on V* for 3B / 7B, and +6.1 / +5.0 on HallusionBench. I don’t buy the “first framework” framing, since K2.5, Step3-VL, and LongCat-Flash-Thinking already explored reasoning width. This reads more like a clean VLM-specific patch; the open question is whether it holds outside curated perception benchmarks.

Xinzhiyuan · WeChatMay 24, 12:09

Anthropic’s Three Cards Surface: Mythos 1 Appears, Opus 4.8 Spotted

Xinzhiyuan says Anthropic’s claude-opus-4.8 appeared in Google Vertex AI, while a 59.8MB Claude Code source-map leak with 512,000 TypeScript lines exposed Sonnet 4.8 references and Mythos 1 clues tied to Claude Code and Claude Security.

Why it matters: HKR-H/K/R all pass, but this is a leak plus Vertex listing, not an Anthropic launch. No capability numbers, pricing, context window, or reproducible evals, so it stays in the 78–84 band.

Take

Anthropic’s signal here looks like an engineering leak, not a model reveal. The article body is just a WeChat verification page, so the usable facts come from the summary: claude-opus-4.8 appeared on Google Vertex AI, and a 59.8MB Claude Code source-map leak exposed 512,000 TypeScript lines with Sonnet 4.8 and Mythos 1 references. That is concrete enough to take seriously, but pricing, context window, benchmarks, and launch timing are missing.

I would not auto-file Mythos 1 as a frontier model. The clues tie it to Claude Code and Claude Security, which sounds more like product packaging or a security layer than a clean model-family launch. Anthropic has spent the last year turning coding agents into distribution. This leak has weight because of where the names surfaced, not because it proves a capability jump.

r/LocalLLaMAMay 24, 11:05

Vision-capable LLMs vs. OCR for long-document QA with charts, images, and tables

The author tested Claude Sonnet 4.5 on 171 questions from 30 image-heavy MMLongBench-Doc PDFs, comparing native PDF vision use with OCR pipelines. Native PDF ranked fifth of six at 52.0% accuracy and cost $0.2552 per query, while LlamaCloud premium with full context reached 59.6% at $0.1885 per query.

Why it matters: HKR-H/K/R pass: the post gives 30 PDFs, 171 questions, accuracy, and per-question cost for long-document QA. Limited sample and Reddit sourcing keep it in the featured-threshold band.

May 23Saturday

Synced · WeChatMay 23, 21:58

FlashAR speeds up pretrained autoregressive image models by 22.9x using 0.05% data

Zhejiang University and the University of Adelaide introduced FlashAR, using 0.05% of the original training data to reduce Emu3.5-Image-34B 512×512 generation latency from 130.10 seconds to 5.68 seconds, while GenEval changed from 80.48 to 80.29.

Why it matters: HKR-H/K/R all pass: FlashAR gives speedup, data ratio, latency, and GenEval deltas for AR image inference. It is a strong research item, but not a top-lab model release, so 80 featured rather than P1.

The Verge · AIMay 23, 19:00

Google’s New Anything-to-Anything AI Model Is Wild

The Verge tried Google’s new Gemini anything-to-anything model for a stuffed-deer deepfake video, but the RSS snippet discloses only one example and does not disclose model parameters, pricing, release timing, or safety controls.

Why it matters: HKR-H/R pass: a Google/Gemini multimodal hands-on has a strong deepfake hook and safety resonance. HKR-K fails because the feed discloses one example only, with no params, pricing, or launch timing.

r/LocalLLaMAMay 23, 11:27

meituan-longcat/LongCat-Video-Avatar-1.5 on Hugging Face

Meituan LongCat released LongCat-Video-Avatar-1.5 on Hugging Face, supporting AT2V, ATI2V, and video continuation while replacing Wav2Vec2 with Whisper-Large and using DMD2 distillation to reduce inference to 8 NFE; the model weights are released under the MIT License.

Why it matters: HKR-H/K/R all pass: open MIT video-avatar weights plus 8 NFE inference give local multimodal builders real signal. This is a mid-weight open-source model update, not an 85+ same-day industry event.

AI HOT (Curated Pool)May 23, 07:59

Gemini update: over 900 million users and new agent features

Google announced that the Gemini app has surpassed 900 million monthly active users and introduced two agent features: Daily Brief for personalized daily summaries and Gemini Spark, a 24/7 personal agent that manages tasks under user authorization.

Why it matters: HKR-H/K/R all pass: Google gives a 900M MAU number and two agent features for Gemini. This is an entry-point product update with competitive weight, not a routine small feature.

Take

Google is pushing Gemini Spark into a 900M-MAU surface, so this is a distribution bet first. Daily Brief is a summary product; Spark touches task management and “digital life,” which is where the liability sits. The snippet names Gemini 3.5 Flash, Gemini Omni video, and a “Neural Expressive” design layer, but gives no permission model, audit log, rollback path, or Gmail / Calendar / Android action boundary.

I don’t buy the “24/7 personal agent” framing yet. OpenAI and Anthropic have both been moving agents into browsers, computer control, and enterprise workflows, but consumer agents fail on trust before they fail on benchmarks. Google’s edge is real: Android plus Workspace gives it surfaces most labs lack. If the consent layer is sloppy, 900M MAU turns from distribution into blast radius.

May 22Friday

TechCrunch · AIMay 22, 23:37

We tried Google’s AI glasses and they’re almost there

Google demonstrated prototype Android XR glasses that overlay Gemini-powered translation, navigation, and other information into the user’s field of view; the post does not disclose pricing, launch timing, battery life, or hardware specifications.

Why it matters: HKR-H/K/R all pass: TechCrunch tested Google’s Android XR glasses and identified Gemini overlays for translation and navigation. Price, launch timing, and battery life are not disclosed, keeping it in the lower featured band.

AI HOT (Curated Pool)May 22, 23:12

Project Genie and Google Maps Street View launch interactive worlds

Project Genie partnered with Google Maps Street View to turn real U.S. locations into interactive worlds; the post does not disclose supported cities, generation mechanics, pricing, or access scope.

Why it matters: Google DeepMind’s official post says Genie × Street View turns real US locations into interactive worlds, so HKR-H and HKR-R pass. HKR-K fails because cities, generation method, and access are not disclosed.

AI HOT (Curated Pool)May 22, 17:45

NetEase Youdao Open-Sources Ziyue 4 Multimodal and Text-to-Speech Models

NetEase Youdao open-sourced its Ziyue 4.0 multimodal and text-to-speech models, with the 27B multimodal model reporting 81.4% accuracy on Chinese math reasoning tasks and the speech model supporting 14 languages.

Why it matters: HKR-H/K/R pass: the story has a concrete open-source hook, specific model numbers, and practitioner relevance. NetEase Youdao is not a frontier lab, so it stays below the 78+ good-quality band.

Hacker News front pageMay 22, 15:52

Deepfakes Tore a High School Apart

404 Media reports that five girls at Radnor Township High School were targeted with AI-generated CSAM, and a freshman allegedly spent $250 on a Movely subscription from Apple’s App Store; the visible article does not disclose the police outcome.

Why it matters: HKR-H/K/R all pass: 404 Media reports a concrete AI CSAM school incident with victim count and tool cost. It is strong safety-policy signal, not a model or platform launch, so it stays in the 78–84 band.

Take

The ugly lesson here is distribution, not model capability. Five Radnor High girls were targeted with AI-generated sexual images, and the freshman allegedly paid $250 for Movely through Apple’s App Store. Radnor has just over 1,000 students, and Pennsylvania criminalized malicious deepfakes in 2024. The law, school policies, and platform gatekeeping all existed before the incident.

I don’t buy the soft framing that schools and police are still learning how to handle AI. 404 Media describes conflicting accounts between parents, mandated reporters, and administrators; the visible article does not give the police outcome. That gap matters. App Store review, school discipline, and CSAM enforcement each own a slice, so the victim gets a process maze while the tool ships as a subscription product.

Synced · WeChatMay 22, 10:00

CVPR 2026 | HiF-VLA: A Motion-Centric World Action Model

Westlake University and collaborators introduced HiF-VLA, a motion-centric VLA framework that extracts compact Motion vectors with codecs such as H.264 and uses a joint expert to predict future visual motion and generate action sequences, reporting 31.4GB peak memory and 117.7ms latency under the cited history-window setting.

Why it matters: HKR-H/K/R all pass: the H.264-motion angle, concrete VRAM/latency numbers, and robotics deployment pressure are clear. It remains a single research item without adoption or cross-source heat, so it sits in the lower featured band.

Take

HiF-VLA’s useful move is not the WAM branding; it is turning codec motion vectors into cheap temporal memory for VLA control. Under the cited history-window setting, it reports 31.4GB peak memory and 117.7ms latency. Naive history-frame stacking hits 63.6GB and 229.5ms. That gap matters because long-horizon robotics usually dies on latency and memory before it dies on slogans.

I don’t buy the “understands the physical world” framing yet. CALVIN and LIBERO-LONG support a claim about motion-action alignment on benchmarks, not learned physical causality. Compared with RT-2 and OpenVLA-style image-language-action pipelines, HiF-VLA makes a cleaner engineering bet: feed less pixel history, feed more motion structure. If the paper lacks strong real-robot failures across objects, cameras, and lighting, the WAM label is still oversized.

Synced · WeChatMay 22, 10:00

Meta Chinese Researcher Releases ATLAS for Generalizable Visual Reasoning with One Word

Meta AI and the Chinese University of Hong Kong proposed ATLAS, a visual reasoning method that uses one Functional Token to connect Agentic and Latent Visual Reasoning, with ATLAS-178K, a two-stage SFT+RL pipeline, and LA-GRPO to train sparse visual-operation tokens.

Why it matters: HKR-H/K/R pass: the one-token angle is clickable, and the post gives dataset and training details. As a Meta AI/CUHK research release rather than a flagship model or product launch, it fits the 78–84 band.

Take

ATLAS’s useful contribution is not the “one word” branding; it is LA-GRPO rescuing sparse visual-operation tokens from reward noise. The article gives two concrete hooks: ATLAS-178K spans 40-plus visual reasoning tasks, and training uses SFT plus RL. Functional Tokens occupy only a few positions, so vanilla sequence-level GRPO dilutes their learning signal across ordinary text tokens. That is a real problem in multimodal CoT, especially for line drawing, labeling, region selection, and counting.

I don’t buy the “new paradigm” pitch. Tool-using VLMs and latent reasoning papers have been compressing intermediate visual steps for a while. ATLAS looks stronger as a clean interface: no external executor, no generated intermediate image, but the model still learns when to emit <|Line|>, <|Shape|>, and <|Text|>.