Skip to content
Trending storyDeveloping

OpenAI proposes perplexity as a progress metric and a cap on development

1 report1 sourceupdated 55 minutes ago

What happened

AI digest

On November 22, OpenAI published its internal pretraining perplexity measurements and proposed perplexity as a metric for comparing model progress across developers and for setting shared safety standards and development limits. On its main evaluation dataset, GPT-3.5 scored about 0.99 log perplexity and GPT-4 about 0.91. The post says models start producing breakthrough results on open math problems once they drop to roughly 0.76. Only the measurements and the proposal are disclosed; no shared safety standard or development cap has been adopted.

Written by AI from the coverage · updated 53 minutes ago

Coverage

Follow the reports to see the story from different sides.

Nov 23
  1. OpenAI News
    Perplexity and Intelligence

    OpenAI 披露内部预训练困惑度测量结果,提议将困惑度作为跨开发者比较模型进展、制定共同安全标准和开发上限的指标。其主要评估数据集上,GPT‑3.5 的对数困惑度约为 0.99,GPT‑4 约为 0.91;文章称模型降至约 0.76 后开始在数学开放问题上产出突破性结果。

Heat over time

Not enough continuous observations to draw a trend yet.