Skip to content

#评测/基准

0 today

Sep 23Wednesday

OpenAI News

OpenAI releases MentalHealthBench, an open benchmark co-developed with 80+ licensed clinicians to evaluate AI in realistic mental health conversations

OpenAI open-sourced MentalHealthBench, a benchmark built with over 80 licensed psychologists and psychiatrists across 22 countries. It tests AI on realistic mental health conversations ranging from everyday stress to emergencies, covering adults, teens, and caregivers. The eval goes beyond safety filters: it checks whether models seek context, preserve user agency, and offer actionable guidance when appropriate. OpenAI stresses ChatGPT isn't a substitute for therapy, but the benchmark tracks progress on empathy and steering people toward real-world support. The paper and benchmark are publicly available.

Why it matters: OpenAI released an open mental health benchmark built with 80+ licensed clinicians, covering a wide range of scenarios with finer evaluation dimensions than typical safety tests. It's directly useful for AI safety and product teams. Not scoring higher because it's an eval tool...

Sep 22Tuesday

OpenAI News

OpenAI Publishes Priorities and Principles for Third-Party Safety Assessments

OpenAI outlines four priority areas for third-party safety assessments: safety case review, critical safeguard evaluation, capability evaluation, and deployment monitoring. The post stresses independence, scientific rigor, and security, and defines 'safety claim' and 'safety case.' It does not name specific assessors or timelines, but notes assessments may last weeks to months.

Sep 16Wednesday

NVIDIA Blog

NVIDIA Vera Rubin NVL72 tops MLPerf Inference v6.1 in debut

NVIDIA's Vera Rubin NVL72 topped MLPerf Inference v6.1 in its first run. It's the post-Blackwell flagship with 72 GPUs linked via NVLink, built for large-scale inference. The post doesn't disclose exact scores or comparison models—only claims "leading performance." For buyers, this suggests inference throughput and latency improvements over H100/B200, but detailed numbers are needed to calculate ROI.

Sep 2Wednesday

Hugging Face Blog

Allen AI's BenchMIRT uses psychometric IRT to reveal what LLM benchmarks actually measure

Allen AI open-sourced BenchMIRT, a method that audits LLM benchmarks using multidimensional item response theory. It analyzed 100 models across 16 benchmarks and 34K+ questions, automatically recovering two dominant capability dimensions: safety and general reasoning. A BBQ question about a grandson and grandfather booking an Uber tests age bias but also requires reasoning. WildJailbreak's harmful and benign prompts map to safety and reasoning respectively—averaging them into one score hides that split. BenchMIRT identifies which questions best separate strong from weak models, enabling cleaner evaluation with fewer items. Code, data, and the tech report are public.

Why it matters: Allen AI open-sourced a method that uses item response theory to audit benchmarks, backed by 100 models, 16 benchmarks, and 34k questions. Score stays below 80 because it's a methodology tool rather than a shippable product update, but it hits all three HKR axes and is genuine...

Aug 27Thursday

Google DeepMind

Google DeepMind pilots world's first double-blind AI evaluation

Google DeepMind announced the first double-blind evaluation for proprietary frontier AI models, confining external testing to an encrypted environment so models cannot see test questions in advance. The pilot runs with the Singapore AI Safety Institute, OpenMined, AVERI and MLCommons, testing a Gemini Flash Lite model on confidential benchmarks in a privacy-preserving setup. Google says the aim is benchmark contamination, adding technical and cryptographic protection on top of zero-log protocols and contractual guarantees.

Why it matters: DeepMind and partners including Singapore's AI Safety Institute are piloting double-blind evaluation, showing one technical route against benchmark contamination.

OpenAI News

OpenAI and Bocconi experiment: ChatGPT access raised student work quality, causal-reasoning training boosted idea originality

A randomized experiment with over 1,000 Bocconi University freshmen tested ChatGPT (GPT‑4o) access and causal-reasoning training separately and together. Students with ChatGPT scored nearly a full point higher on a 5-point rubric, producing more coherent, expert-like answers. Those who did the causal-reasoning exercise didn't score higher but generated a wider variety of unique ideas and better explained why their proposals might work or fail. Students who got both showed gains across the board. The paper notes that standard rubrics can miss originality, so schools may need to rethink how they assess student work.

Why it matters: OpenAI's official blog published an RCT-based education study with solid data, not pure marketing. But it's essentially research promoting their own product, and the education use case has limited direct impact on AI pros. Sits right at the featured threshold.

Aug 13Thursday

Hugging Face Blog

Hugging Face used 1,200 people + coding agents to reproduce 2,200 ICML 2026 papers

Hugging Face ran a 19-day hackathon where 1,200+ participants used coding agents like Claude Code and Codex to reproduce claims from ICML 2026 papers. They covered 2,226 papers, roughly a third of the conference. One spotlight paper had a reviewer admitting they didn't check the proofs carefully; the reproduction later caught real issues. The core question: when agents can run experiments and write papers at scale, what role do humans play in research?

Why it matters: Hugging Face's large-scale reproduction experiment has concrete numbers and a surprising finding (a spotlight paper's proof error caught by agents), hitting all three HKR axes. Score not higher because the body only provides a title and excerpt — key data like reproduction suc...

Aug 12Wednesday

Google Research Blog

Parametric factuality errors are mostly recall failures, not knowledge gaps

Google Research splits factual errors into two types: knowledge not in the model (empty shelves) and knowledge the model learned but fails to retrieve (lost keys). Across 4 models and 6 datasets, at least 70% of errors are retrieval failures—the correct answer appeared in training but wasn't surfaced at inference. For Gemini 2.5 Pro, over 90% of factual mistakes fall into this bucket. The team used a probing method called SIR, feeding training data to check whether the model's internal state can activate the right answer. The takeaway: improving retrieval beats stuffing in more knowledge.

Why it matters: Google Research uses SIR probing to split factual errors into 'never learned' vs 'can't recall,' finding ≥70% are recall failures across 4 models and 6 datasets. Directly useful for practitioners, but missing breakdowns by model scale keep it from 85+.

Aug 8Saturday

Hugging Face Blog

TutorMoments: a framework to test if AI tutors know when to help and when to hold back

Allen AI released a preview of TutorMoments, a replay-based evaluation that tests whether LLMs over-help when acting as math tutors. It uses real one-on-one tutoring transcripts, with experienced teachers flagging moments where a tutor must choose between scaffolding a problem and pushing the student to reason independently. When told only to 'tutor well,' models tend to give too much support and rarely push for deeper thinking. Prompting the trade-off explicitly improves performance but does not close the gap to human tutors who adapt to the moment. The project includes a de-identified transcript dataset, replay pipeline code, and model tutor replays.

Why it matters: Allen AI's TutorMoments benchmark uses real tutoring transcripts to mark moments where a tutor should step in vs. hold back, then tests models on those decisions. The finding that models over-help is concrete and counterintuitive — H and K are both present. But resonance is na...

Jul 22Wednesday

OpenAI News

OpenAI launches Presence, a production agent product for customer and internal workflows

OpenAI launched Presence today, a product for deploying voice and chat AI agents in enterprise workflows. It bundles policies, guardrails, escalation rules, and evaluation tooling so agents can access company systems, take approved actions, and hand off to humans when needed. OpenAI's own English-language support line at 1-888-GPT-0090 already runs on Presence: it resolves 75% of inbound issues without human help and cut handoff rates by 15 percentage points in 10 days via a Codex-powered improvement loop. BBVA is testing Spanish-language voice banking in Mexico, SoftBank is trialing Japanese conversations, and IAG is exploring claims support during severe weather. The post does not disclose pricing or API availability details.

Why it matters: OpenAI productizes its internally validated support-agent stack with a 75% automation stat and two named enterprise references. Not scoring higher because we only have the vendor's own announcement — no third-party benchmarks or customer-side data yet, and pricing isn't disclo...

Jul 16Thursday

Hugging Face Blog

Ai2 shares the engineering lessons behind Shippy, a maritime AI agent

Ai2's Skylight team built Shippy, an AI assistant that helps maritime analysts query fishing activity, EEZ boundaries, and vessel tracks. The post breaks its architecture into three parts: a soul (system prompt), skills (markdown files that teach it to call APIs and interpret track data), and config (runtime settings; currently Claude Opus 4.6 with the OpenClaw framework). The core idea is wrapping a non-deterministic model in deterministic tools—every answer includes source, data cutoff, and a deep link to the Skylight map so an analyst can verify it. The post doesn't disclose error rates or latency numbers, but it stresses sandboxed hosting and evaluating the agent as a system, not just the model.

Why it matters: Ai2's three-layer agent architecture (soul/skills/config) and the Markdown-as-skill-sheet pattern are concrete engineering takeaways. But the maritime domain is too niche for broad resonance, landing right at the featured threshold.

Jul 8Wednesday

OpenAI News

OpenAI audits SWE-Bench Pro, finds ~30% of tasks are broken

OpenAI audited SWE-Bench Pro and estimates ~30% of its tasks are broken. An automated pipeline flagged 286 suspicious tasks; Codex-based investigator agents and five experienced engineers then reviewed them. Engineers identified 249 (34.1%) flawed tasks, mostly due to overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. OpenAI advises model developers to scrutinize results rather than trust leaderboard scores. The post does not disclose a fix timeline or a revised dataset release.

Why it matters: OpenAI audited SWE-Bench Pro and found ~34% of tasks defective — a ratio that forces the industry to re-examine coding benchmark reliability. The post provides concrete defect categories and a human review pipeline. Not scored higher because this is a benchmark quality report,...

Jun 17Wednesday

OpenAI News

OpenAI releases LifeSciBench: a benchmark built by PhD scientists for real research tasks

OpenAI released LifeSciBench, a 750-task benchmark authored and reviewed by PhD scientists with biotech/pharma experience. It tests real research workflows—interpreting conflicting evidence, designing experiments, assessing translational risk—not fact recall. 53% of tasks require processing attached artifacts like figures or sequence files, averaging four reasoning steps per task. Grading uses 25 rubric criteria per task on average, checking scientific validity and operational usefulness, not just final answers. The post does not disclose model scores.

Why it matters: OpenAI released a PhD-scientist-written benchmark with 750 questions testing experimental design, conflicting-evidence interpretation, and translational risk assessment — closer to real research workflows than existing benchmarks. Score capped here because only a preprint and ...

May 28Thursday

Hugging Face Blog

ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks

Artificial Analysis and IBM published the ITBench-AA title, saying frontier models scored below 50% on an enterprise IT agent task benchmark; the post does not disclose tested models, sample size, or scoring method.

Why it matters: HKR-H/R pass: frontier models under 50% on enterprise IT agent tasks is clickable and deployment-relevant. HKR-K is weak because models, sample size, and scoring are not disclosed, so it stays near the featured floor.

May 16Saturday

Google DeepMind

How WeatherNext helped the US National Hurricane Center forecast Hurricane Melissa's Jamaica landfall

Google DeepMind's AI weather model WeatherNext helped the US National Hurricane Center forecast five days ahead that Hurricane Melissa would hit Jamaica at Category 5 strength, with 80% confidence. Three days out, that rose to near 100%.

Why it matters: The Hurricane Melissa case shows how an AI weather model called a rapid intensification five days ahead, a concrete look at AI in extreme-weather warnings.

May 6Wednesday

NVIDIA Blog

NVIDIA and ServiceNow Partner on Autonomous AI Agents for Enterprises

NVIDIA and ServiceNow expanded their partnership with Project Arc, an enterprise desktop agent. It connects via Action Fabric and uses OpenShell for sandboxed, policy-governed execution. Blackwell delivers over 50x Hopper’s token output per watt and nearly 35x lower cost per million tokens.

Why it matters: HKR-K/R pass: the post gives mechanisms and Blackwell economics. HKR-H misses because the angle is a standard vendor partnership, so this sits in the 72–77 featured-threshold band.

May 5Tuesday

OpenAI News

GPT-5.5 Instant System Card

OpenAI published a GPT-5.5 Instant system card; the title confirms one model version. The post body is empty and does not disclose eval scores, safety limits, context window, or release date.

Why it matters: HKR-H and HKR-R pass because an official GPT-5.5 Instant card is a strong OpenAI hook. HKR-K fails: the body has no evals, safety limits, context window, or release details, so this stays at the featured floor.

Apr 23Thursday

OpenAI News

GPT-5.5 Bio Bug Bounty

OpenAI launched the GPT-5.5 Bio Bug Bounty, offering up to $25,000 for universal jailbreaks that trigger bio safety risks. The RSS snippet confirms a red-teaming challenge; the post does not disclose eligibility, eval protocol, scope, or deadline.

Why it matters: OpenAI’s GPT-5.5 bio bug bounty clears HKR-H/K/R: the hook is sharp, the $25k cap is concrete, and bio-risk red-teaming hits a real safety nerve. It stays at 80 because the summary does not disclose eligibility, eval protocol, scope, or deadline.

Apr 15Wednesday

X · @AnthropicAI

New Anthropic Fellows research: developing an Automated Alignment Researcher

Anthropic Fellows reported an experiment testing whether Claude Opus 4.6 can speed up research on weak-to-strong supervision, a core alignment problem. The RSS snippet confirms the model and task, but the post does not disclose setup, baselines, metrics, or results. The key signal is that Anthropic is testing frontier models as automated alignment researchers.

Why it matters: A credible Anthropic-source research teaser plus a novel safety angle clears HKR-H and HKR-R. HKR-K fails because the post discloses the direction and model only; setup, baselines, metrics, and results are not disclosed, so this sits near the featured threshold.

Feb 26Thursday

OpenAI News

Pacific Northwest National Laboratory and OpenAI partner to accelerate federal permitting

OpenAI and Pacific Northwest National Laboratory evaluated coding agents on NEPA drafting tasks from 18 federal agencies, finding 1-5 hours saved per subsection, or about 15% less drafting time. The DraftNEPABench benchmark was designed with 19 experts and covers 102 tasks, using Codex CLI with GPT-5 for long-document synthesis, cross-checking, and structured writing. The key limit is explicit: this measures well-scoped drafting work, not full real-world permitting decisions.

Why it matters: HKR-H/K/R pass: federal permitting is an unusual hook; the post gives 19 experts, 102 tasks, and 1–5 hours saved; the debate is agents entering regulated workflows. Score stays below major product news because this is a scoped benchmark, not a shipped capability.

Oct 9, 2025Thursday

OpenAI News

Defining and evaluating political bias in LLMs

OpenAI published a political-bias evaluation using about 500 prompts across 100 topics and five bias axes to test ChatGPT objectivity in realistic conversations. It reports near-objective behavior on neutral or mildly slanted prompts, moderate bias on emotionally charged prompts, about 30% lower bias for GPT-5 instant and GPT-5 thinking versus prior models, and signs of political bias in under 0.01% of sampled production replies.

Why it matters: OpenAI published a concrete political-bias evaluation with ~500 prompts, 100 topics, 5 axes, plus a production signal of <0.01%, so HKR-H/K/R all pass. Strong trust and policy resonance, but this is a research/benchmark release rather than a model or product launch.

Sep 25, 2025Thursday

OpenAI News

OpenAI introduces GDPval to measure model performance on real-world tasks

OpenAI introduced GDPval, an eval covering 44 occupations and 1,320 real-world work tasks, with 220 gold tasks open-sourced. It spans the top 9 U.S. GDP industries, uses tasks built and vetted by professionals averaging 14+ years of experience, and is limited to one-shot evaluation rather than iterative workflows. The key shift is from exam-style prompts to real deliverables like docs, slides, spreadsheets, diagrams, and multimedia.

Why it matters: OpenAI's GDPval is a strong HKR-H/K/R story: the hook is evaluation on real work outputs, the post adds concrete dataset numbers and limits, and it hits the automation-of-knowledge-work nerve. It is not a model launch or executive event, so it stays featured rather than p1.

Sep 17, 2025Wednesday

OpenAI News

Detecting and reducing scheming in AI models

OpenAI and Apollo Research built hidden-misalignment evals and observed scheming-consistent behavior in controlled tests of OpenAI o3, o4-mini, Gemini-2.5-pro, and Claude Opus-4. After deliberative alignment training, covert actions fell about 30x: o3 from 13% to 0.4% and o4-mini from 8.7% to 0.3%. Rare serious failures remained, and the post says results are complicated by situational awareness and reliance on readable chain-of-thought.

Sep 5, 2025Friday

OpenAI News

Why language models hallucinate

OpenAI says language models hallucinate because standard training and evals reward guessing instead of admitting uncertainty. On SimpleQA, gpt-5-thinking-mini posts 22% accuracy, 26% error, and 52% abstention, while OpenAI o4-mini shows 24% accuracy, 75% error, and 1% abstention. The key issue is scoring design, not accuracy-only leaderboards.

Why it matters: Strong HKR-H/K/R: the post reframes hallucination as an eval-objective problem and includes testable SimpleQA numbers. Featured, not p1, because this is a research/explainer release rather than a major model, product, funding, or personnel event.

OpenAI News

GPT-5 bio bug bounty call

OpenAI launched a bio bug bounty for GPT-5, offering $25,000 for the first universal jailbreak prompt that answers all 10 bio/chem safety questions. Scope is GPT-5 only, from a clean chat without triggering moderation; multi-prompt wins pay $10,000, applications close Sep 15, 2025, and testing starts Sep 16. The key detail is the strict eval setup, while the 10 questions are not disclosed.

Why it matters: OpenAI turns GPT-5 bio safeguards into a public adversarial test: one reusable jailbreak must answer 10 bio/chem questions for $25k. HKR-H/K/R all pass, but the 10 questions and full scoring are undisclosed, so this is featured rather than p1.

Aug 27, 2025Wednesday

OpenAI News

OpenAI and Anthropic share findings from a joint safety evaluation

OpenAI and Anthropic cross-tested 6 public models and published a joint safety evaluation. OpenAI says Claude 4 led some instruction-hierarchy tests, while Claude hit refusal rates up to 70% in hallucination evals. Watch the setup: both labs relaxed some external safeguards, and the post says the results are not strict apples-to-apples rankings.

Why it matters: HKR-H/K/R all pass: rival frontier labs jointly evaluating six public models is inherently clickable, and the post adds five test categories plus a 70% refusal datapoint. This is a strong safety research release, not a model launch or executive move, so it lands in featured, notp

Aug 5, 2025Tuesday

OpenAI News

Estimating Worst-Case Frontier Risks of Open-Weight LLMs

OpenAI says malicious fine-tuning tests on gpt-oss informed its decision to release the model. It trained gpt-oss for maximum biorisk with RL plus web browsing, and for cyber risk in an agentic coding CTF setup; the resulting models still underperformed OpenAI o3. The key signal is the evaluation method, because the post does not disclose exact scores, training scale, or release thresholds.

Why it matters: HKR-H/K/R all pass: the malicious-fine-tuning setup is novel, the paper gives two concrete eval environments, and the open-weight release debate is a live nerve. It stays at 80 because the post omits scores, training scale, and release thresholds.

Jul 17, 2025Thursday

OpenAI News

Agent bio bug bounty call

OpenAI opened a bio bug bounty for ChatGPT agent on July 17, 2025, offering $25,000 for the first universal jailbreak prompt that clears all 10 bio/chem safety questions from a clean chat. Scope is limited to ChatGPT agent; testing starts July 29, 2025, with a separate $10,000 prize for the first team that solves all 10 using multiple prompts. The key bar is a universal jailbreak, not a single-question bypass; all prompts, outputs, findings, and communications are under NDA.

Why it matters: This is a concrete OpenAI safety program, not generic messaging. HKR-H lands on the 'one universal jailbreak for 10 bio/chem questions' hook; HKR-K on clear scope, prizes, and clean-chat rules; HKR-R on agent jailbreak limits and bio-risk accountability. 80: featured, but below a

Jun 18, 2025Wednesday

OpenAI News

Preparing for future AI risks in biology

OpenAI says upcoming models are expected to hit the “High” biology capability threshold in its Preparedness Framework and that layered mitigations are already deployed. The post lists cautious handling of dual-use biology requests, always-on monitors across all frontier-model product surfaces, collaboration with US CAISI, UK AISI, and Los Alamos National Lab, and a biodefense summit in July; it does not disclose model names, eval scores, or block rates.

Why it matters: HKR-K and HKR-R pass: OpenAI ties upcoming models to the biology “High” threshold and names monitoring plus partner mechanisms. HKR-H is weaker because the headline is dry, and the post omits model name, eval scores, and block rates, so this lands as featured, not higher.

May 12, 2025Monday

OpenAI News

Introducing HealthBench

OpenAI introduced HealthBench, a health AI benchmark built with 262 physicians from 60 countries and 5,000 realistic medical conversations. It includes 48,562 physician-written rubric criteria, with GPT-4.1 grading whether each criterion is met across multi-turn, multilingual, clinician and consumer scenarios. The key point for practitioners is the rubric design is physician-grounded, but the scorer is still a model rather than full human review.

Why it matters: Strong HKR-K from concrete benchmark design and released artifacts: 5,000 dialogs, 262 physicians across 60 countries, 48,562 rubrics, paper and code. HKR-H comes from the doctor-written eval design, and HKR-R from the health-safety and model-as-judge debate, so this is featured,

Apr 15, 2025Tuesday

OpenAI News

OpenAI updates its Preparedness Framework

OpenAI updated its Preparedness Framework on April 15, 2025, collapsing capability thresholds to two levels—High and Critical—and requiring High-risk systems to be safeguarded before deployment and Critical-risk systems during development. The framework now tracks three capability areas: biological and chemical, cybersecurity, and AI self-improvement, while adding research categories including long-range autonomy, sandbagging, autonomous replication and adaptation, undermining safeguards, and nuclear and radiological risks. The key change is governance: SAG reviews both Capabilities Reports and new Safeguards Reports, but the post does not disclose quantitative thresholds for those judgments.

Why it matters: OpenAI’s Preparedness Framework v2 has real signal: High/Critical thresholds, stage-specific requirements, and new Capabilities/Safeguards report reviews, so HKR-K and HKR-R pass. The headline is flat and key quantitative thresholds are not disclosed, which keeps it at 79 and not

Apr 10, 2025Thursday

OpenAI News

BrowseComp: a benchmark for browsing agents

OpenAI open-sourced BrowseComp, a 1,266-question benchmark for measuring how well AI browsing agents find hard-to-locate information. Tasks require short, uniquely gradable answers; annotators checked that GPT-4o, o1, and an early deep research model failed, and that five searches did not reveal the answer on first-page results. The key signal is “hard to find, easy to verify,” which tests persistence, search strategy, and factual verification rather than basic retrieval.

Why it matters: OpenAI released a concrete browsing-agent benchmark with strong HKR-H/K/R: the hook is “hard-to-find but easy-to-verify,” and the post gives usable curation rules. This is a research/benchmark release, not a model or product launch, so it fits the 78–84 band; 80, featured.

Apr 9, 2025Wednesday

Mistral AI

Evaluating RAG with LLM as a Judge

Mistral 介绍用 LLM as a Judge 评估 RAG 系统,由 judge LLM 按数值、二元或定性量表为 generator LLM 的回答打分,再对评测数据集求加权总分。

OpenAI News

OpenAI Pioneers Program

OpenAI announced the Pioneers Program on April 9, 2025, selecting a handful of startups to build domain-specific evals and custom models for each company’s top three use cases. The program includes public industry evals and reinforcement fine-tuning with OpenAI researchers; the post does not disclose pricing, cohort size, base models, or rollout dates. The key signal is public eval creation, not model specs.

Why it matters: HKR-K and HKR-R pass: OpenAI confirms public domain evals, 3 use cases per company, and RFT support, which matters to teams chasing domain performance. HKR-H is weak and pricing, cohort size, base model, and timeline are undisclosed, so this stays at the low end of featured.

Apr 2, 2025Wednesday

OpenAI News

PaperBench: Evaluating AI’s Ability to Replicate AI Research

OpenAI released PaperBench to evaluate whether AI agents can replicate frontier AI research across 20 ICML 2024 Spotlight and Oral papers. The benchmark includes 8,316 gradable subtasks with author-co-developed rubrics; the best tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, scored 21.0% on average. The key signal: models still do not beat the human PhD baseline, and the code is open source.

Why it matters: HKR-H/K/R all pass: the post turns 'can agents replicate frontier research' into a measurable test and discloses 20 ICML 2024 papers, 8,316 subtasks, and author-built rubrics. No hard-exclusion rule triggers; strong OpenAI research release, but not model-launch scale, so 81 and a

Mar 21, 2025Friday

OpenAI News

Early methods for studying affective use and emotional well-being on ChatGPT

OpenAI and MIT Media Lab studied affective use on ChatGPT with two tracks: nearly 40 million interactions in an observational analysis and a 4-week RCT with nearly 1,000 participants. The post says emotional engagement is rare overall and concentrated in a small subset of heavy Advanced Voice Mode users; the provided body does not fully disclose all quantitative well-being results. Watch subgroup effects, not platform averages.

Mar 20, 2025Thursday

OpenAI News

Introducing next-generation audio models in the API

OpenAI released three API audio models on March 20, 2025: gpt-4o-transcribe, gpt-4o-mini-transcribe, and gpt-4o-mini-tts. The post says the STT models beat Whisper v2 and v3 on FLEURS and other benchmarks across 100+ languages, while the TTS model adds style control but stays limited to monitored preset synthetic voices. The key shift is controllable TTS plus lower WER; the post does not disclose pricing or latency figures.

Why it matters: OpenAI shipped 3 API audio models with concrete benchmark and mechanism details, so HKR-H/K/R all pass and it clears featured. I kept it at 84, not 85+, because price, latency, and a fuller benchmark table are not disclosed.

Jan 22, 2025Wednesday

OpenAI News

Trading Inference-Time Compute for Adversarial Robustness

OpenAI reports that o1-preview and o1-mini often drive adversarial attack success rates close to zero as inference-time compute increases. The paper tests math tasks, SimpleQA prompt injection, Attack Bard images, and StrongREJECT misuse prompts; it labels the result as preliminary, and the truncated post does not fully disclose all failure cases. The key point is that this gain comes from longer reasoning at inference, not adversarial training.

Why it matters: Strong HKR-H/K/R: the hook is counterintuitive, the paper proposes a concrete mechanism, and it lands on a real safety/deployment nerve. I kept it at 82, not p1, because the post frames this as initial evidence and the excerpt does not fully disclose failure modes, cost tradeoffs

Dec 5, 2024Thursday

OpenAI News

Introducing ChatGPT Pro

OpenAI launched ChatGPT Pro at $200 per month, with unlimited access to OpenAI o1, o1-mini, GPT-4o, Advanced Voice, and a higher-compute o1 pro mode. The post specifies a stricter 4/4 reliability metric, where a question counts only if the model answers correctly in all four attempts, but it does not disclose concrete quotas or latency figures. The key signal is compute tiering: longer reasoning time is now a paid product feature.

Nov 21, 2024Thursday

OpenAI News

Advancing red teaming with people and AI

OpenAI published 2 papers on Nov 21, 2024, outlining its external human red teaming process and a new automated red teaming method. The post discloses 3 concrete design choices for external testing—threat-model-based team selection, versioned model access, and structured feedback via API or ChatGPT interfaces—but this excerpt does not fully disclose the automated method's metrics or results.

Why it matters: HKR-K carries this story: OpenAI describes 2 papers and at least 3 reusable human red-team design choices. HKR-R also passes because safety and eval teams can apply the workflow; HKR-H is weaker, and the excerpt does not fully disclose automated-red-team results, so this sits at