OpenAI proposes perplexity as a progress metric and a cap on development
What happened
On November 22, OpenAI published its internal pretraining perplexity measurements and proposed perplexity as a metric for comparing model progress across developers and for setting shared safety standards and development limits. On its main evaluation dataset, GPT-3.5 scored about 0.99 log perplexity and GPT-4 about 0.91. The post says models start producing breakthrough results on open math problems once they drop to roughly 0.76. Only the measurements and the proposal are disclosed; no shared safety standard or development cap has been adopted.
Written by AI from the coverage · updated 53 minutes ago
Coverage
Follow the reports to see the story from different sides.
- OpenAI NewsPerplexity and Intelligence
OpenAI 披露内部预训练困惑度测量结果,提议将困惑度作为跨开发者比较模型进展、制定共同安全标准和开发上限的指标。其主要评估数据集上,GPT‑3.5 的对数困惑度约为 0.99,GPT‑4 约为 0.91;文章称模型降至约 0.76 后开始在数学开放问题上产出突破性结果。
Heat over time
Not enough continuous observations to draw a trend yet.