Skip to content

#多模态

0 today

Nov 11Wednesday

Nov 8Sunday

The Decoder

AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test

阿里巴巴与多所亚洲高校构建 Co-Design Bench,测试的十个模型中,表现最佳的 GPT-6 Astra 仅完整完成 26.4% 的任务。该模型平均满足约 70% 的需求,测试要求智能体根据真实用户请求修改现有网页应用,并由两名 AI 裁判检查功能是否实际可用。任务混合多种修改类型时所有模型表现均下降,同一模型在不同执行环境中的表现也有明显差异,研究团队建议同时报告模型与执行环境。

Nov 7Saturday

OpenAI News

Measuring how AI agents handle everyday wet-lab science

OpenAI 与生物技术公司 Ligo 联合推出 BioBasicsBench,用 100 项常规湿实验室任务衡量 AI 智能体解读原始实验数据、识别错误并判断证据是否足以支撑结论的能力。

Nov 4Wednesday

OpenAI News

OpenAI uses AI to speed design, simulation and manufacturing of physical products

OpenAI published research on models helping engineer physical products, covering mechanical, electrical and systems design, and announced a research partnership with Mazda. Two team members used Astra to design a device that talks to Codex in under a week of spare time, and a manufacturer built a small prototype batch in under a month. Multiple review rounds and iteration are still needed before manufacturing to handle production limits, tolerances and assembly details.

Why it matters: OpenAI used Astra to link a device's mechanical, circuit and software design, but manufacturing still needs review rounds for production limits, tolerances and assembly details.

Nov 1Sunday

The Decoder

Adobe is testing how well AI agents can use editing tools like a Photoshop professional would

Adobe Research与UC San Diego提出EditAgent,通过并行调用通用编辑器和专用工具、结合量化核验数据选择图像编辑结果。在600项编辑测试中,获得核验数据的Qwen3.8-27B选中可用结果的比例为81.3%,高于EditScore-72B的74.8%。EditAgent在三个基准上改善了开源编辑器的表现,但所需计算量约为单个编辑器的十倍,性能也取决于接入的工具。

The Decoder

Google wants video AI to understand what people need as they work

Google 牵头的研究团队提出 StreamVibe,让视频智能体识别明确、隐含及变化中的用户意图,自行判断何时提供帮助。团队基于超过 183 小时的录像构建约 4,100 个样例,并训练 StreamVibe-VL,结合强化学习和独立记忆系统减少打断、跟踪任务进度。

Oct 30Friday

AI HOT · Products

腾讯混元发布 Hy-Nomas 技术报告并开源 Hy-Nomas-Bench

腾讯混元发布交互式视频世界模型 Hy-Nomas 的技术报告(arXiv:2610.36729),并开源 Hy-Nomas-Bench 数据集用于可复现研究。官方用 Hy-Nomas 重现 Genie 4 示例并发布对比视频,演示无需排队即可试用,地址为 https://hy-nomas.hy.tencent.com/。

Oct 29Thursday

Hacker News front page

Scaling Hierarchical Networks

Cartesia公布改进版层次网络H-Net的扩展实验,通过端到端动态分块替代固定分词,并报告语言建模效率提升。严格匹配的一阶段H-Net在验证损失上的计算效率为BPE Transformer的1.28x;两阶段H-Net约用~1T tokens达到SmolLM3约~11T tokens时的最终平均评测分数,两者每字节计算量差异在2%以内,但参数量和训练阶段设置不同。

Jun 7Sunday

QbitAI · WeChat

Kuaishou Kling Proposes VLM-as-Teacher for Test-Time Video Reasoning Optimization

City University and Kuaishou Kling proposed VLM-as-Teacher, which uses VLM feedback to optimize VGM LoRA at test time, reporting a 16.7-point average gain and raising VBVR-Bench from 0.666 to 0.781.

Why it matters: Kuaishou Kling and CityU use a vision-language model as a test-time teacher, feeding LoRA updates to a video generator so it follows text rules while it generates.

Jun 6Saturday

Synced · WeChat

Daxiao Robotics and NTU Release PhysX-Omni for Simulation-Ready Physical 3D Generation

PhysX-Omni models rigid, deformable, and articulated objects in one simulation-ready 3D generation framework, while PhysXVerse contains over 8.7K physical 3D assets across more than 2.9K categories.

Why it matters: Daxiao Robotics and NTU's PhysX-Omni generates 3D objects with physics properties that run directly in simulators, backed by 8,700+ assets in PhysXVerse.

Jun 5Friday

Synced · WeChat

MetaFine proposes a diagnostic meta-evaluation framework for fine-grained robot manipulation

Southeast University and Peking University researchers introduced MetaFine, a diagnostic meta-evaluation framework that tests fine-grained robot manipulation across understanding, perception, and behavior, and the article says traditional binary success metrics can overestimate fine-manipulation capability by up to 70%.

Why it matters: Southeast University and Peking University researchers propose MetaFine, a meta-evaluation framework that diagnoses robot fine manipulation through understanding, perception and action, and finds success-rate-only evaluation overstates ability by up to 70%.

Jun 4Thursday

QbitAI · WeChat

CVPR 2026: NVIDIA, Tesla, and Waymo hear Xpeng present physical AI

Xpeng presented its world-model stack at CVPR 2026, covering X-World, X-Foresight, and X-Cache; the article says X-Cache cuts about 70% of repeated computation, the second-generation VLA used over 4 trillion training tokens, and the in-car stack reduced inference latency to 80 ms.

Why it matters: XPeng's CVPR 2026 world-model stack cuts redundant compute by about 70% with X-Cache and trains its second-gen VLA on over 4 trillion tokens.

Jun 3Wednesday

Synced · WeChat

RSS 2026: Ant Lingbo Proposes Autoregressive Causal World Model for Robot Manipulation with 50 Demos

Ant Lingbo and HKUST introduced LingBot-VA, an autoregressive video-action world model that unifies visual dynamics prediction and action inference, and the paper reports fine-tuning with 50 real-world demonstrations per task plus 92.0% and 91.1% success on RoboTwin 2.0 Easy and Hard settings.

Why it matters: Ant Group's Lingbo and HKUST fold frame prediction and action decisions into one autoregressive model, fine-tuned on just 50 real demos per task.

QbitAI · WeChat

Daxiao Robot and NTU Release PhysX-Omni for Unified Physical 3D Generation

Daxiao Robot and NTU introduced PhysX-Omni, a unified simulation-ready physical 3D generation framework for rigid, deformable, and articulated objects, with PhysXVerse covering 8.7K assets across 2.9K categories and PhysX-Bench evaluating six dimensions including geometry, scale, material, affordance, kinematics, and description.

Why it matters: One pipeline generates rigid, soft and articulated 3D assets, so teams skip separate processing and manual mesh fixes.

May 30Saturday

Synced · WeChat

Apple Uses AI to Rework Image Compression: Same Visual Quality at One-Third the File Size

Apple’s team published PICO, a perceptual image codec that uses 57%-70% fewer bits than AV1, VVC, and JPEG AI at the same subjective visual quality, while encoding a 12MP photo in 230 ms and decoding it in 150 ms on an iPhone 17 Pro Max.

Why it matters: Apple's paper claims 57% to 70% bitrate savings over AV1, VVC and JPEG AI at similar subjective quality, with on-device timings on iPhone 17 Pro Max.

Synced · WeChat

NVIDIA and Tsinghua Team's Gamma-World Tops Hugging Face Daily Chart

NVIDIA, Tsinghua, University of Toronto, and Vector Institute released Gamma-World, a multi-agent world model using simplex-based positional encoding and hub tokens to cut interaction cost from quadratic to linear, with 8-player latency dropping from 17.6 ms to 4.5 ms.

Why it matters: NVIDIA and Tsinghua researchers propose the multi-agent world model γ-World, using simplex positional encoding and hub tokens to cut inter-agent interaction cost from quadratic to linear.

May 28Thursday

NVIDIA Blog

NVIDIA Research Advances Robotics From Simulation to the Real World

NVIDIA Research presented 8 ICRA papers on sim-to-real robotics: ScheduleStream delivered a 3x speedup for multi-arm planning, COMPASS reached about 80% success across 20 real-world navigation trials, and Grasp-MPC achieved about 75% real-robot grasping success.

Why it matters: NVIDIA's COMPASS navigates real robots after simulation-only training, hitting roughly 80% success across 20 trials, 4.5x a pure imitation baseline.

r/LocalLLaMA

Nvidia LocateAnything: Fast Vision-Language Grounding with Parallel Box Decoding

The title says Nvidia LocateAnything-3B performs vision-language grounding with parallel box decoding and runs 10x faster than Qwen3-VL; the post body only provides Hugging Face, GitHub, demo, and project links, and does not disclose benchmark setup or accuracy numbers.

Why it matters: Nvidia open-sourced LocateAnything, a 3B vision grounding model that uses parallel box decoding instead of serial generation and claims 10x the speed of Qwen3-VL.

Synced · WeChat

ICML 2026: AutoMoT reaches SOTA on Bench2Drive and nuScenes

NTU AutoMan Lab, Harvard, and Xiaomi Auto proposed AutoMoT, a unified VLA driving model using a 4B Qwen3-VL Understanding Expert and a 1.6B Action Expert with asynchronous inference, reaching 89.42 DS and 74.09% SR on Bench2Drive with AutoMoT+.

Why it matters: AutoMoT splits a 4B vision-language model and a 1.6B action expert into asynchronous inference, taking first place on both Bench2Drive and nuScenes, a split-model idea for end-to-end driving.

May 27Wednesday

QbitAI · WeChat

7B Medical AI Agent Beats o3 and GPT-5 by Learning Where and How to Look

Shanghai Innovation Institute’s LeapQuest and three universities released Ophiuchus and MedScope, applying Think with Images and Think with Videos to medical AI; Ophiuchus-7B scored 68.0 on eight VQA benchmarks, above OpenAI-o3 at 62.2, Gemini 2.5 Pro at 61.8, and GPT-5 at 59.9.

Why it matters: Ophiuchus-7B scores 68.0 across 8 medical visual QA benchmarks, beating o3 and GPT-5, a sample of a small model catching general models through method design.

May 26Tuesday

AI HOT (Curated Pool)

Project Luxo: Crossing the Uncanny Valley of AI Media

Runway released Project Luxo, showing AI shorts and ad samples including The Rogue; each work was made by a single-person team, with production times ranging from three weeks to four hours.

Why it matters: Runway shows three short films and an ad spot made by one person, cutting production from three weeks to four hours, with feedback from producers, actors and union members.

Alibaba Technology · WeChat

Nearly 9x training speedup: residual streams in DiT are becoming a convergence bottleneck

Nanjing University LAMDA and Alibaba Intelligent Engine proposed DAR, a timestep-aware cross-layer routing method that replaces fixed residual accumulation in DiT; on ImageNet 256x256, it reduced SiT-XL/2 FID from 9.67 to 7.56 and reached baseline convergence quality with 8.75x fewer training iterations.

Why it matters: Nanjing University's LAMDA lab and Alibaba replace DiT's fixed residual accumulation with per-timestep cross-layer routing, reaching equal convergence on ImageNet in about one-ninth the training steps.

QbitAI · WeChat

Zhejiang University and Alibaba Make AI Think Before Drawing Sudoku or Burning Candles | ACL 2026

Zhejiang University and Alibaba introduced Unified Thinker, an independent planning module trained with 40,000 HieraReason-40K samples and a two-stage GRPO reinforcement-learning setup that turns structured reasoning traces into executable visual instructions for image generation and editing.

Why it matters: Zhejiang University and Alibaba propose Unified Thinker, a planning module for image generation that separates thinking from execution, producing structured reasoning steps before converting them into visual instructions.

May 24Sunday

Synced · WeChat

ICML 2026: First Parallel Thinking Framework for Vision-Language Models

Visual Para-Thinker introduces a parallel thinking framework for vision-language models, using Pa-Attention and LPRoPE to isolate four visual reasoning paths and training on 163,000 question-answer pairs.

Why it matters: Visual Para-Thinker, accepted at ICML 2026, splits visual reasoning into four parallel paths with Pa-Attention and LPRoPE, trained on 163,000 question-answer pairs.

May 23Saturday

Synced · WeChat

FlashAR speeds up pretrained autoregressive image models by 22.9x using 0.05% data

Zhejiang University and the University of Adelaide introduced FlashAR, using 0.05% of the original training data to reduce Emu3.5-Image-34B 512×512 generation latency from 130.10 seconds to 5.68 seconds, while GenEval changed from 80.48 to 80.29.

Why it matters: Zhejiang University and the University of Adelaide propose FlashAR, which fine-tunes on just 0.05% of the original training data and cuts Emu3.5-Image-34B 512x512 generation from 130.10 seconds to 5.68 seconds.

May 22Friday

Synced · WeChat

CVPR 2026 | HiF-VLA: A Motion-Centric World Action Model

Westlake University and collaborators introduced HiF-VLA, a motion-centric VLA framework that extracts compact Motion vectors with codecs such as H.264 and uses a joint expert to predict future visual motion and generate action sequences, reporting 31.4GB peak memory and 117.7ms latency under the cited history-window setting.

Why it matters: A Westlake University team proposes HiF-VLA, replacing per-frame visual encoding with H.264 motion vectors and predicting visual dynamics and actions together, at 31.4GB peak memory and 117.7ms latency.

Synced · WeChat

Meta Chinese Researcher Releases ATLAS for Generalizable Visual Reasoning with One Word

Meta AI and the Chinese University of Hong Kong proposed ATLAS, a visual reasoning method that uses one Functional Token to connect Agentic and Latent Visual Reasoning, with ATLAS-178K, a two-stage SFT+RL pipeline, and LA-GRPO to train sparse visual-operation tokens.

Why it matters: Meta AI and CUHK propose ATLAS, adding function words to a vision-language model so one inference pass can switch between explicit tool calls and internal latent computation.

May 21Thursday

Xinzhiyuan · WeChat

USTC Papers Study Lifelong Learning for LMMs via Multimodal Knowledge Injection

USTC researchers released MMEVOKE and KORE: MMEVOKE contains 9,422 samples across 159 subcategories, while KORE uses knowledge-tree augmentation and null-space constrained fine-tuning to reduce catastrophic forgetting during multimodal knowledge injection.

Why it matters: A USTC team's MMEVOKE benchmark and KORE fine-tuning method offer reusable evaluation and training ideas for forgetting in continual learning of multimodal models.

Synced · WeChat

VAST and Tsinghua propose density-controlled 3D Gaussian generation for SIGGRAPH 2026

VAST and Tsinghua propose DeG, a 3D Gaussian generation method that samples Gaussian centers from a learned density distribution and trains density control with a render loss contribution gradient; in some settings, it reaches TRELLIS-like visual quality with less than half the Gaussian count.

Why it matters: VAST and Tsinghua propose DeG, which allocates Gaussian points by object detail density and matches TRELLIS-level quality with under half the points.

Synced · WeChat

Xie Saining’s Team Releases Second-Generation Representation Autoencoder RAEv2

Xie Saining’s team, Adobe Research, and the Australian National University released RAEv2, which reaches gFID 1.06 after 80 epochs on ImageNet-256 and reduces EPFID@2 from 177 epochs to 35 epochs while keeping compute at 189 GFLOPs.

Why it matters: Xie Saining's team with Adobe Research and ANU released RAEv2, cutting ImageNet-256 convergence from 177 epochs to 35 while holding compute at 189 GFLOPs.

May 20Wednesday

Hacker News front page

Show HN: Lance – Image/video generation and understanding in one model

ByteDance released Lance as a research project for image and video generation and understanding in one model; the RSS snippet states 3B active parameters, fewer than 128 GPUs used for training, and links to a homepage, arXiv paper, and Hugging Face model, while the post does not disclose benchmark results or licensing terms.

Why it matters: ByteDance open-sourced Lance, one model for image and video understanding, generation and editing, with 3B active parameters, under 128 GPUs to train, and code, paper and weights all public.

r/LocalLLaMA

Nemotron-Labs-Diffusion from NVIDIA

NVIDIA released the Nemotron-Labs-Diffusion 3B, 8B, and 14B dense model family with AR decoding, diffusion parallel decoding, and self-speculation; the 8B model reaches 850 tok/s on GB200 at concurrency 1, compared with 253 tok/s for AR and 360 tok/s for Eagle3.

Why it matters: NVIDIA's Nemotron-Labs-Diffusion ships three dense models from 3B to 14B that decode both autoregressively and by diffusion, plus a self-speculation speedup.

May 17Sunday

QbitAI · WeChat

TGO Aligns Visual Generative Models with Scalar Feedback Without Preference Pairs | ICML 2026

NUS proposed Threshold-Guided Optimization, which converts scalar feedback into positive or negative updates through a score-distribution threshold and was accepted by ICML 2026; experiments cover Stable Diffusion v1.5, FLUX, Wan 1.3B, and Meissonic across image and video generation settings.

Why it matters: NUS's TGO, accepted at ICML 2026, skips preference-pair construction and aligns visual generative models directly from scalar scores like aesthetics or image-text matching by thresholding.

May 16Saturday

Synced · WeChat

Why Robots Need World Models: Top Institutions Release Joint Survey

NTU MARS Lab and collaborators released a 43-page survey on robot world models, covering definitions, architectures, applications, benchmarks, and challenges around action-conditioned consistency, inference efficiency, and physical grounding.

Why it matters: NTU MARS Lab and partner institutions release a 43-page survey on robot world models, covering how they let robots predict physical outcomes before acting, as background on simulation training for embodied AI.

QbitAI · WeChat

Zhejiang University and Microsoft use 3,000 text prompts to improve video 3D consistency with World-R1

Zhejiang University and Microsoft introduced World-R1, training Wan 2.1 with about 3,000 text-only prompts, Flow-GRPO, and a four-part reward; the 1.3B version improves PSNR over the baseline by 10.23 dB.

Why it matters: World-R1 from Zhejiang University and Microsoft aligns Wan 2.1 video generation using about 3,000 text-only prompts and no 3D annotations, a cheap route for teams without labeled data.

May 15Friday

Xinzhiyuan · WeChat

Hassabis Praises Google DeepMind's AI-enabled Pointer Powered by Gemini

Google DeepMind released a Gemini-powered AI-enabled pointer and opened two demos in Google AI Studio: image editing and place finding on maps, while the post says Chrome pointer selection and a Googlebook Magic Pointer are planned product paths.

Why it matters: Google DeepMind put Gemini on the mouse pointer, so pointing at something lets the AI read the context and edit an image or find a place, with two demos live in AI Studio.

May 14Thursday

Synced · WeChat

China in Focus: PsiBot Uses 100,000 Hours of Human Data for Embodied AI

PsiBot says it uses 100,000 hours of human operation data to train robot policies, with the W0 world model acting only as a training-time transfer module while deployment runs R2 alone.

Why it matters: Lingchu claims 100,000 hours of human operation data trained its robot policy, with the W0 world model used only in training; the data collection and task results can't be confirmed.

May 13Wednesday

QbitAI · WeChat

ByteDance Proposes Generative Refinement Networks as a Third Route for Visual Generation

ByteDance’s commercial technology team proposed GRN, a visual generation architecture using HBQ, global refinement, and complexity-aware sampling to address quantization loss, error accumulation, and fixed-step inference; on a 130M model, adaptive sampling reduced inference from 50 steps to an average of 24, while gFID changed from 3.56 to 3.79.

Why it matters: ByteDance's GRN drafts a coarse image then refines it, with HBQ layered binary quantization cutting loss in discrete codes, a concrete third path beyond diffusion and autoregression.

AI HOT (Curated Pool)

SenseNova-U1 Technical Report Released: Guide to Native Multimodal Model Building

SenseTime released the SenseNova-U1 technical report, covering six-stage training, RL post-training, and distillation; the open-source SenseNova-U1-A3B-MoT uses an MoE architecture and activates only 3 billion parameters.

Why it matters: SenseTime's SenseNova-U1 technical report lays out the six-step pipeline for a native multimodal model, from unified modeling to RL post-training and distillation, and open-sources a 3B-active-parameter MoE version.