Scaling Law's three corrections in five years: from bigger models to smaller models with more data

Scaling Law 塌房了吗？三次修正的真实故事，和越来越小的模型

Scaling law is an empirically fitted curve, not a physical law. OpenAI's 2020 Kaplan paper concluded 'prioritize parameters' due to experimental biases, shaping GPT-3. DeepMind's 2022 Chinchilla corrected the ratio to 20:1, showing smaller models with more data outperform. Two 2024 replication studies confirmed that fixing Kaplan's setup reproduces Chinchilla's result—no fraud, just calibration. Since 2023, Meta and others deliberately deviate from Chinchilla: Llama 3 8B was trained on 15T tokens because the optimization target shifted from training cost to total cost of training plus inference. Tsinghua's Densing Law shows the parameter count needed for equal capability halves roughly every 3.5 months, but there is a floor: each parameter stores only ~2 bits of knowledge. The viral 'collapse' article cited a blog comment posted the same day as if it were peer-reviewed research; the post does not provide a paper source for that claim.

Why it matters: A high-quality explainer and fact-check on scaling laws, debunking a recent viral post with specific numbers and paper citations while tracing three key revisions over five years. Hits all three HKR axes, but as commentary/education rather than a first-party product release, i...