#评测/基准

0 today

Oct 30, 2024Wednesday

OpenAI NewsOct 30, 18:00

Introducing SimpleQA

OpenAI open-sourced SimpleQA, a 4,326-question benchmark for factual short-answer QA and model calibration. Two independent AI trainers verified each item; a 1,000-question audit showed 94.4% agreement and an estimated inherent error rate near 3%. The key signal: it is built to challenge frontier models, and the post says GPT-4o scores below 40%.

Why it matters: This is not a routine paper post. HKR-H comes from the inversion that a 'simple' benchmark stumps frontier models; HKR-K comes from the dataset size, agreement rate, and irreducible-error estimate; HKR-R comes from the ongoing industry fixation on hallucination and calibration,so

Oct 23, 2024Wednesday

OpenAI NewsOct 23, 18:00

Simplifying, stabilizing, and scaling continuous-time consistency models

OpenAI introduced sCM and scaled continuous-time consistency models to 1.5B parameters on ImageNet at 512×512. The post says sCM reaches sample quality comparable to leading diffusion models in 2 sampling steps, with about 50x wall-clock speedup. Its largest model generates one sample in 0.11s on a single A100 at batch size 1 without inference optimization.

Why it matters: This clears HKR-H/K/R: the hook is 2-step sampling with diffusion-like quality, and the paper gives concrete numbers—1.5B params, ImageNet 512x512, ~50x wall-clock speed, and 0.11s per sample on one A100. Strong research release, but not a shipped product, so featured fits better

Take

OpenAI’s hard move here is scaling consistency models past toy-demo territory, not posting another FID chart. sCM reaches 1.5B parameters on ImageNet 512×512, uses 2 sampling steps, and produces one sample in 0.11s on a single A100 at batch size 1 without inference optimization. The post claims about 50x wall-clock speedup and under 10% effective sampling compute versus leading diffusion baselines. That is the kind of number product teams care about.

I’m less sold on the extrapolation. ImageNet 512×512 is a controlled benchmark, not a text-to-image or video workload with prompt following, temporal consistency, and safety filters in the loop. If this survives audio or video generation, the cost model behind Sora-like systems changes. If it stays inside ImageNet, it is a strong sampling paper, not a deployment story.

Oct 15, 2024Tuesday

OpenAI NewsOct 15, 18:00

Evaluating fairness in ChatGPT

OpenAI analyzed millions of ChatGPT requests to test whether user names trigger harmful stereotypes, finding an overall rate of about 0.1%. The study used GPT-4o as a privacy-preserving evaluator; its gender-related judgments matched human raters over 90% of the time, while race and ethnicity agreement was lower. The key signal is model drift across versions: GPT-3.5 Turbo showed the highest task-level bias.

Why it matters: OpenAI provides a rare production-scale fairness audit with concrete rates, evaluator agreement, and a model-comparison result, so HKR-K is strong and HKR-R clears on trust and safety. This is a substantive research release, not a model launch or major product shift, so it lands

Oct 10, 2024Thursday

OpenAI NewsOct 10, 18:00

MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

OpenAI released MLE-bench, a benchmark built from 75 Kaggle competitions to measure ML engineering ability in AI agents. The best setup, o1-preview with AIDE scaffolding, reached at least Kaggle bronze-medal level on 16.9% of tasks; the benchmark code is open-source.

Why it matters: Strong HKR-H/K/R: OpenAI moves evaluation from exam-style tasks to real ML engineering, anchored by 75 Kaggle competitions and a 16.9% bronze-level result. Important as a benchmark release with concrete numbers, but still research rather than a major product launch, so featured,

Oct 1, 2024Tuesday

OpenAI NewsOct 1, 18:02

Model Distillation in the API

OpenAI launched an API distillation workflow on October 1, 2024, letting developers use outputs from GPT-4o and o1-preview to fine-tune cheaper models such as GPT-4o mini. The suite includes Stored Completions, Evals in beta, and fine-tuning; setting store:true auto-saves input-output pairs with no added latency, per the post. Pricing includes 2M free GPT-4o mini training tokens per day and 1M for GPT-4o through October 31; Evals are free up to 7 runs per week through year-end if shared with OpenAI.

Sep 12, 2024Thursday

OpenAI NewsSep 12, 18:02

Learning to reason with LLMs

OpenAI released o1-preview and reported 74% single-sample accuracy on AIME 2024, versus 12% for GPT-4o. The post says o1 reached the 89th percentile on Codeforces and exceeded human PhD experts on GPQA Diamond; it attributes this to large-scale RL and gains from both train-time and test-time compute. The key signal is scaling reasoning with compute, not just pretraining a larger base model.

Why it matters: This is a substantive OpenAI research release with product implications. HKR-H lands on the new reasoning line, HKR-K on the disclosed benchmark jumps and compute-scaling mechanism, and HKR-R on the direct impact to model strategy and inference economics; strong 90s, not 95+.

Take

o1-preview’s sharp signal is not just 74% single-sample accuracy on AIME 2024. OpenAI shows reasoning quality rising smoothly with test-time compute. GPT-4o sits at 12% on the same AIME setup, while o1 reaches the 89th percentile on Codeforces and beats human PhD experts on GPQA Diamond. That explains the limited launch: ChatGPT plus trusted API users, not a broad default model swap.

I buy the “test-time compute as a new control knob” thesis. I don’t buy a clean benchmark victory lap. The post says the evals used maximal test-time compute, and the charts also show 64-sample majority-vote regions. That turns capability into a pricing and latency problem. Claude and Gemini do not just need prettier reasoning scores; they need APIs where developers can choose thinking budget, delay, and failure mode without guessing the bill.

Aug 20, 2024Tuesday

OpenAI NewsAug 20, 18:00

Fine-tuning now available for GPT-4o

OpenAI has opened GPT-4o fine-tuning to developers on all paid tiers, with 1M free training tokens per org per day through September 23. Training costs $25 per 1M tokens, and inference costs $3.75 per 1M input tokens and $15 per 1M output tokens on gpt-4o-2024-08-06. The signal for practitioners: partners reported 43.8% on SWE-bench Verified and 71.83% on BIRD-SQL with fine-tuned GPT-4o.

Why it matters: This is a substantive OpenAI developer release with concrete details: temporary free training quota, train/inference prices, base model version, and two benchmark datapoints. HKR-H/K/R all pass, but this is an API capability expansion, not a new frontier-model launch or platform-

Take

GPT-4o fine-tuning is a pricing move before it is a capability story. Training costs $25 per million tokens, with inference at $3.75 input and $15 output. That forces application teams to price their private data, output format constraints, and domain behavior against a higher serving bill.

The benchmark hooks are strong: Cosine reports 43.8% on SWE-bench Verified, and Distyl reports 71.83% execution accuracy on BIRD-SQL. But both are partner cases, not a general transfer curve. Fine-tuning shines when the target is narrow: patch formatting, SQL generation, intent routing, self-correction loops. The article does not give ablations for broad agent behavior. I’d treat the SOTA claims as proof that curated task data still matters, not evidence that every GPT-4o wrapper should fine-tune next week.

Aug 13, 2024Tuesday

OpenAI NewsAug 13, 18:00

Introducing SWE-bench Verified

OpenAI released SWE-bench Verified, a human-validated subset built with the benchmark’s authors to assess real software issue resolution more reliably. The post names 3 failure modes in SWE-bench: overly narrow tests, underspecified issue statements, and unreliable environment setup; as of Aug. 5, 2024, top agents scored about 20% on SWE-bench and 43% on SWE-bench Lite. The key point is that the original benchmark can systematically underestimate coding-agent ability.

Why it matters: This is a strong benchmark release, not a routine post: OpenAI re-audited SWE-bench with the original authors, named 3 defect classes, and reported new score ceilings of 20% and 43%. HKR-H/K/R all pass because it changes how builders read code-agent leaderboards.

Take

OpenAI is pulling SWE-bench back from leaderboard theater into risk evaluation. The original set spans 12 Python repos, and the top agents sat around 20% on SWE-bench and 43% on Lite as of Aug. 5, 2024. That reads like weak coding agents. The post names three benchmark defects: narrow tests, underspecified issues, and unreliable environments. Those defects reject valid patches and mislabel harness failure as model failure.

I buy this move more than another model claiming a few extra points. SWE-bench Verified is a human-validated subset built with the benchmark authors, and it fits OpenAI’s Preparedness Framework for Model Autonomy. Cursor, Devin, and Claude Code live on long-loop debugging and environment recovery, not clean single-shot pass rates. The old SWE-bench was partly blind to that.

Jul 17, 2024Wednesday

OpenAI NewsJul 17, 18:00

Prover-Verifier Games improve legibility of language model outputs

OpenAI trained GPT-4-family prover-verifier games so stronger models write solutions weaker models can verify; under time-limited human review, correctness-only optimization led to nearly 2x more evaluation errors. The post says the large and small models differ by about 3 orders of magnitude in pretraining compute, and checkability training recovers about half the performance gain of correctness-only optimization; the full experimental numbers are not fully disclosed in the provided text.

Why it matters: This is a substantive OpenAI research release with HKR-H/K/R all present: novel setup, clear mechanism, and strong relevance to scalable oversight. The excerpt confirms the method and the human-evaluation effect, but not the full experimental tables, so it fits the 78–84 band, نه

Take

OpenAI’s useful admission here is that optimizing for correct answers can make reasoning hostile to review. In their GPT-4-family prover setup, correctness-only optimization made time-limited human evaluators commit nearly twice as many errors. Training the prover to write chains a weaker verifier can check recovered about half the performance gain from accuracy-only tuning.

I buy the direction because legibility becomes a training target, not a post-hoc “please explain” wrapper. The setup also has the right asymmetry: the large and small models differ by roughly three orders of magnitude in pretraining compute, which resembles strong systems writing for weaker auditors. The caveat is ugly: the provided article cuts off before the full experimental tables, so that “half the gain” claim is not yet enough for product decisions.