Skip to content

#评测/基准

1 today

Jul 17, 2025Thursday

OpenAI News

Agent bio bug bounty call

OpenAI opened a bio bug bounty for ChatGPT agent on July 17, 2025, offering $25,000 for the first universal jailbreak prompt that clears all 10 bio/chem safety questions from a clean chat. Scope is limited to ChatGPT agent; testing starts July 29, 2025, with a separate $10,000 prize for the first team that solves all 10 using multiple prompts. The key bar is a universal jailbreak, not a single-question bypass; all prompts, outputs, findings, and communications are under NDA.

Why it matters: Puts a cash prize on proving ChatGPT agent can answer all ten biosecurity questions from a blank chat with one prompt.

Jun 18, 2025Wednesday

OpenAI News

Preparing for future AI risks in biology

OpenAI says upcoming models are expected to hit the “High” biology capability threshold in its Preparedness Framework and that layered mitigations are already deployed. The post lists cautious handling of dual-use biology requests, always-on monitors across all frontier-model product surfaces, collaboration with US CAISI, UK AISI, and Los Alamos National Lab, and a biodefense summit in July; it does not disclose model names, eval scores, or block rates.

Why it matters: OpenAI expects its next models to reach the high-risk threshold for biological capability in its own Preparedness Framework, and lays out how it handles bioweapon-related and dual-use requests today.

May 12, 2025Monday

OpenAI News

Introducing HealthBench

OpenAI introduced HealthBench, a health AI benchmark built with 262 physicians from 60 countries and 5,000 realistic medical conversations. It includes 48,562 physician-written rubric criteria, with GPT-4.1 grading whether each criterion is met across multi-turn, multilingual, clinician and consumer scenarios. The key point for practitioners is the rubric design is physician-grounded, but the scorer is still a model rather than full human review.

Why it matters: 5000 multi-turn medical dialogues and 48562 hand-written rubric items give medical AI evaluation a public, reusable benchmark.

Apr 15, 2025Tuesday

OpenAI News

OpenAI updates its Preparedness Framework

OpenAI updated its Preparedness Framework on April 15, 2025, collapsing capability thresholds to two levels—High and Critical—and requiring High-risk systems to be safeguarded before deployment and Critical-risk systems during development. The framework now tracks three capability areas: biological and chemical, cybersecurity, and AI self-improvement, while adding research categories including long-range autonomy, sandbagging, autonomous replication and adaptation, undermining safeguards, and nuclear and radiological risks. The key change is governance: SAG reviews both Capabilities Reports and new Safeguards Reports, but the post does not disclose quantitative thresholds for those judgments.

Why it matters: OpenAI cut its capability risk tiers to High and Critical, tying High to pre-deployment safeguards and Critical to the development stage.

Apr 10, 2025Thursday

OpenAI News

BrowseComp: a benchmark for browsing agents

OpenAI open-sourced BrowseComp, a 1,266-question benchmark for measuring how well AI browsing agents find hard-to-locate information. Tasks require short, uniquely gradable answers; annotators checked that GPT-4o, o1, and an early deep research model failed, and that five searches did not reveal the answer on first-page results. The key signal is “hard to find, easy to verify,” which tests persistence, search strategy, and factual verification rather than basic retrieval.

Why it matters: BrowseComp tests browsing agents on 1,266 questions built to be hard to find and easy to verify.

Apr 9, 2025Wednesday

OpenAI News

OpenAI Pioneers Program

OpenAI announced the Pioneers Program on April 9, 2025, selecting a handful of startups to build domain-specific evals and custom models for each company’s top three use cases. The program includes public industry evals and reinforcement fine-tuning with OpenAI researchers; the post does not disclose pricing, cohort size, base models, or rollout dates. The key signal is public eval creation, not model specs.

Why it matters: OpenAI's Pioneers program builds and publishes domain benchmarks with law, finance and healthcare firms, giving industry teams a view of how models are judged in their fields.

Apr 2, 2025Wednesday

OpenAI News

PaperBench: Evaluating AI’s Ability to Replicate AI Research

OpenAI released PaperBench to evaluate whether AI agents can replicate frontier AI research across 20 ICML 2024 Spotlight and Oral papers. The benchmark includes 8,316 gradable subtasks with author-co-developed rubrics; the best tested agent, Claude 3.5 Sonnet (New) with open-source scaffolding, scored 21.0% on average. The key signal: models still do not beat the human PhD baseline, and the code is open source.

Why it matters: PaperBench scores AI agents on replicating 20 ICML 2024 papers, with rubrics written with the original authors and 8,316 gradable subtasks.

Mar 21, 2025Friday

OpenAI News

Early methods for studying affective use and emotional well-being on ChatGPT

OpenAI and MIT Media Lab studied affective use on ChatGPT with two tracks: nearly 40 million interactions in an observational analysis and a 4-week RCT with nearly 1,000 participants. The post says emotional engagement is rare overall and concentrated in a small subset of heavy Advanced Voice Mode users; the provided body does not fully disclose all quantitative well-being results. Watch subgroup effects, not platform averages.

Why it matters: OpenAI and MIT paired observational analysis of nearly 40 million conversations with a four-week randomized trial of almost 1,000 people to study ChatGPT's emotional use and mental health.

Mar 20, 2025Thursday

OpenAI News

Introducing next-generation audio models in the API

OpenAI released three API audio models on March 20, 2025: gpt-4o-transcribe, gpt-4o-mini-transcribe, and gpt-4o-mini-tts. The post says the STT models beat Whisper v2 and v3 on FLEURS and other benchmarks across 100+ languages, while the TTS model adds style control but stays limited to monitored preset synthetic voices. The key shift is controllable TTS plus lower WER; the post does not disclose pricing or latency figures.

Why it matters: OpenAI shipped three audio models in the API, two for speech-to-text and one for text-to-speech that learns tone, drawing finer lines around the voice stack.

Jan 22, 2025Wednesday

OpenAI News

Trading Inference-Time Compute for Adversarial Robustness

OpenAI reports that o1-preview and o1-mini often drive adversarial attack success rates close to zero as inference-time compute increases. The paper tests math tasks, SimpleQA prompt injection, Attack Bard images, and StrongREJECT misuse prompts; it labels the result as preliminary, and the truncated post does not fully disclose all failure cases. The key point is that this gain comes from longer reasoning at inference, not adversarial training.

Why it matters: OpenAI tested o1-preview and o1-mini on math problems, prompt injection, adversarial images and jailbreak prompts, and attack success rates dropped sharply as inference-time compute rose.

Dec 5, 2024Thursday

OpenAI News

Introducing ChatGPT Pro

OpenAI launched ChatGPT Pro at $200 per month, with unlimited access to OpenAI o1, o1-mini, GPT-4o, Advanced Voice, and a higher-compute o1 pro mode. The post specifies a stricter 4/4 reliability metric, where a question counts only if the model answers correctly in all four attempts, but it does not disclose concrete quotas or latency figures. The key signal is compute tiering: longer reasoning time is now a paid product feature.

Why it matters: OpenAI launched ChatGPT Pro at $200 a month with unlimited o1, o1-mini, GPT-4o and Advanced Voice Mode, where o1 pro mode trades more compute for longer thinking on data science, coding and case analysis.

Nov 21, 2024Thursday

OpenAI News

Advancing red teaming with people and AI

OpenAI published 2 papers on Nov 21, 2024, outlining its external human red teaming process and a new automated red teaming method. The post discloses 3 concrete design choices for external testing—threat-model-based team selection, versioned model access, and structured feedback via API or ChatGPT interfaces—but this excerpt does not fully disclose the automated method's metrics or results.

Why it matters: OpenAI published two red-teaming papers, one on picking external human red teamers by threat model, one on using AI to find flaws automatically, useful for how safety teams staff and feed back.

Oct 30, 2024Wednesday

OpenAI News

Introducing SimpleQA

OpenAI open-sourced SimpleQA, a 4,326-question benchmark for factual short-answer QA and model calibration. Two independent AI trainers verified each item; a 1,000-question audit showed 94.4% agreement and an estimated inherent error rate near 3%. The key signal: it is built to challenge frontier models, and the post says GPT-4o scores below 40%.

Why it matters: OpenAI released SimpleQA, a benchmark of 4,326 fact questions cross-verified by two trainers with 1,000 spot-checked and 94.4% agreement, for measuring how accurately models answer short factual questions.

Oct 23, 2024Wednesday

OpenAI News

Simplifying, stabilizing, and scaling continuous-time consistency models

OpenAI introduced sCM and scaled continuous-time consistency models to 1.5B parameters on ImageNet at 512×512. The post says sCM reaches sample quality comparable to leading diffusion models in 2 sampling steps, with about 50x wall-clock speedup. Its largest model generates one sample in 0.11s on a single A100 at batch size 1 without inference optimization.

Why it matters: OpenAI scales continuous-time consistency models to 1.5B parameters, matching diffusion quality on ImageNet 512×512 in two sampling steps.

Oct 15, 2024Tuesday

OpenAI News

Evaluating fairness in ChatGPT

OpenAI analyzed millions of ChatGPT requests to test whether user names trigger harmful stereotypes, finding an overall rate of about 0.1%. The study used GPT-4o as a privacy-preserving evaluator; its gender-related judgments matched human raters over 90% of the time, while race and ethnicity agreement was lower. The key signal is model drift across versions: GPT-3.5 Turbo showed the highest task-level bias.

Why it matters: OpenAI audits ChatGPT fairness across millions of real conversations, finding answer quality holds across name-implied gender and race, with about 0.1% harmful stereotypes.

Oct 10, 2024Thursday

OpenAI News

MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

OpenAI released MLE-bench, a benchmark built from 75 Kaggle competitions to measure ML engineering ability in AI agents. The best setup, o1-preview with AIDE scaffolding, reached at least Kaggle bronze-medal level on 16.9% of tasks; the benchmark code is open-source.

Why it matters: OpenAI builds MLE-bench from 75 real Kaggle competitions so agents run the full pipeline, with the best setup reaching bronze-medal level in 16.9% of contests.

Oct 1, 2024Tuesday

OpenAI News

Model Distillation in the API

OpenAI launched an API distillation workflow on October 1, 2024, letting developers use outputs from GPT-4o and o1-preview to fine-tune cheaper models such as GPT-4o mini. The suite includes Stored Completions, Evals in beta, and fine-tuning; setting store:true auto-saves input-output pairs with no added latency, per the post. Pricing includes 2M free GPT-4o mini training tokens per day and 1M for GPT-4o through October 31; Evals are free up to 7 runs per week through year-end if shared with OpenAI.

Why it matters: Turns distillation into a built-in API step instead of a pipeline teams build themselves.

Sep 12, 2024Thursday

OpenAI News

Learning to reason with LLMs

OpenAI released o1-preview and reported 74% single-sample accuracy on AIME 2024, versus 12% for GPT-4o. The post says o1 reached the 89th percentile on Codeforces and exceeded human PhD experts on GPQA Diamond; it attributes this to large-scale RL and gains from both train-time and test-time compute. The key signal is scaling reasoning with compute, not just pretraining a larger base model.

Why it matters: OpenAI's o1-preview scores 74% on AIME 2024 math problems against GPT-4o's 12%, rising to 83% with repeated sampling.

Aug 20, 2024Tuesday

OpenAI News

Fine-tuning now available for GPT-4o

OpenAI has opened GPT-4o fine-tuning to developers on all paid tiers, with 1M free training tokens per org per day through September 23. Training costs $25 per 1M tokens, and inference costs $3.75 per 1M input tokens and $15 per 1M output tokens on gpt-4o-2024-08-06. The signal for practitioners: partners reported 43.8% on SWE-bench Verified and 71.83% on BIRD-SQL with fine-tuned GPT-4o.

Why it matters: OpenAI opens GPT-4o fine-tuning to all paying developers at $25 per million training tokens, with 1 million free tokens per organization per day until September 23.

Aug 13, 2024Tuesday

OpenAI News

Introducing SWE-bench Verified

OpenAI released SWE-bench Verified, a human-validated subset built with the benchmark’s authors to assess real software issue resolution more reliably. The post names 3 failure modes in SWE-bench: overly narrow tests, underspecified issue statements, and unreliable environment setup; as of Aug. 5, 2024, top agents scored about 20% on SWE-bench and 43% on SWE-bench Lite. The key point is that the original benchmark can systematically underestimate coding-agent ability.

Why it matters: OpenAI and the original SWE-bench authors built a human-verified subset, arguing the original systematically underestimates real coding ability due to rigid unit tests, vague descriptions and environment issues.

Jul 17, 2024Wednesday

OpenAI News

Prover-Verifier Games improve legibility of language model outputs

OpenAI trained GPT-4-family prover-verifier games so stronger models write solutions weaker models can verify; under time-limited human review, correctness-only optimization led to nearly 2x more evaluation errors. The post says the large and small models differ by about 3 orders of magnitude in pretraining compute, and checkability training recovers about half the performance gain of correctness-only optimization; the full experimental numbers are not fully disclosed in the provided text.

Why it matters: Training a large prover against a small verifier cut human evaluation error from double to half, offering a training pattern that transfers to other scalable oversight problems.