Benchmarks

Who is actually stronger: benchmark results, methodology disputes and leaderboard changes.

453 picksRelated topicsModel releases Research & technical reports Reasoning

Latest picks

341–360 of 453

May 4Monday

TechCrunch · AIMay 4, 02:00

In Harvard Study, AI Gave More Accurate ER Diagnoses Than Two Doctors

A Harvard study compared LLMs with two doctors on ER diagnoses; at least one model was more accurate. The post does not disclose model names, sample size, or accuracy rates.

Why it matters: HKR-H/K/R all pass: Harvard tested LLM diagnosis on real ER cases against two doctors. Missing model names, sample size, and accuracy keep it at the featured threshold, not 78+.

Take

A Harvard study says at least one LLM beat two doctors on real ER cases, but the article gives no model names, sample size, or accuracy rates. That is far too little evidence for the headline many people will want to write: AI beats emergency physicians. Two doctors is a thin comparator. Were they residents, attendings, ER specialists, or generalists? Did they see labs, imaging, triage notes, and messy patient history? Did the LLM receive the same raw record, or a cleaned case summary? The snippet does not say.

My first question here is not whether the model won. It is what task it actually performed. Emergency medicine is not a static diagnosis quiz. The doctor handles triage, uncertainty, missing history, time pressure, noisy symptoms, test ordering, legal risk, and patient flow. If the LLM was asked to read a polished case and produce a differential diagnosis, that is a useful capability test. It does not map cleanly to running an ER diagnostic pathway.

We have seen this movie before. GPT-4 performed strongly on medical exams and case-reasoning tasks. Google’s Med-PaLM 2 also scored near expert levels on several medical QA benchmarks, if my memory is right. Those results mattered, but the gap between curated clinical text and live clinical work stayed large. LLMs are excellent when the case is complete and neatly written. Emergency rooms are defined by the opposite condition: sparse data, bad timelines, unreliable symptom reports, and decisions before confirmation.

The phrase “two human doctors” is doing too much rhetorical work. Two physicians do not represent human clinical performance. A serious claim needs multi-site cases, blinded adjudication, physician seniority, time limits, available tools, and a primary endpoint. Was the metric top-1 diagnosis, top-5 differential coverage, or match to final discharge diagnosis? Was the doctor judged on the first ER impression or after test results? Those choices can turn the same experiment from “LLMs are useful” into “the headline is inflated.” None of that is disclosed in the snippet.

I would read this as a product signal, not a replacement signal. The first credible ER use case is not autonomous diagnosis. It is differential generation, missed-diagnosis warnings, chart structuring, and forcing clinicians to consider low-probability, high-risk conditions. Chest pain, abdominal pain, dizziness, pediatric fever, shortness of breath: these are places where a model can surface aortic dissection, pulmonary embolism, meningitis, ectopic pregnancy, or sepsis earlier. That value does not require the model to be “better than doctors.” It requires reducing misses without flooding the department with junk alerts.

The cost side matters. A model can improve benchmark accuracy by being more expansive and more cautious. In a real ER, that can mean more CT scans, more D-dimers, more observation beds, more antibiotics, and longer queues. If the study does not report false positives, calibration, downstream test burden, clinician acceptance, and error severity, the accuracy number alone is incomplete. A safer-looking model can still make the system worse if it pushes every ambiguous case into maximal workup.

The wording in the snippet also deserves caution: “at least one model seemed to be more accurate.” That sounds conditional. It also omits whether the winning system was a frontier general model, a medical-tuned model, or a prompted ensemble. If it was GPT-4.1, Claude 3.7/4, or Gemini 2.5 Pro, the result says one thing about general reasoning. If it was a medical specialist model, it says something else about domain tuning and deployment constraints. Hospitals and regulators will treat those cases differently.

So my read is conservative: LLMs keep looking clinically useful for ER text reasoning, but this does not prove they outperform emergency doctors in the job. When the full study is available, I would check four items first: sample size, input format, doctor comparator conditions, and error categories. Without those, the headline is carrying more weight than the evidence.

May 3Sunday

r/LocalLLaMAMay 3, 21:59

Local LLM Benchmark for Backend Generation via Function Calling: GLM vs Qwen vs DeepSeek

AutoBe posted a controlled backend-generation benchmark and says qwen3.5-35b-a3b matches gpt-5.4 on DB/API design. One shopping-mall run uses 200–300M tokens, costing $1,000–$1,500 per model at GPT 5.5 pricing. The key caveat is n=4 projects and self-scoring harness bias.

Why it matters: HKR-H/K/R all pass, but Reddit sourcing, n=4 projects, and self-eval harness bias keep it at the low featured band. Concrete cost and test constraints carry the score.

Take

qwen3.5-35b-a3b getting framed as near GPT-5.4 is interesting, but I would discount it hard. The useful hook is concrete: backend generation through function calling, DB/API design, and one shopping-mall task burning 200–300M tokens. At GPT 5.5 pricing, that lands around $1,000–$1,500 per model. This is closer to agent work than toy coding benchmarks.

The problem is the evidence shape. The Reddit page is blocked, and the summary says n=4 projects with a self-scoring harness. AutoBe judging DB/API design inside its own harness can easily measure fit to AutoBe’s workflow, not general model quality. The planned filter for models under $0.25/M or runnable on a 64GB laptop is the part practitioners can use. Open models can win on engineering economics; this does not prove GPT-5.4 parity.

Xinzhiyuan · WeChatMay 3, 13:06

Google Vantage uses AI role-play to assess collaboration under pressure

Google Research and NYU tested Vantage with 188 US participants aged 18-25 on conflict resolution and project management. Its four-layer agent pipeline generates scenarios, applies pressure, extracts behavior, and scores against rubrics; AI-human agreement matched expert-expert Kappa of 0.45-0.64. The key gap is transfer beyond lab settings; the post says Vantage remains a Google Labs research experiment.

Why it matters: HKR-H/K/R all pass: the Vantage study has a strong hook, concrete sample size, and evaluator-risk resonance. It stays in the low featured band because it is still a Google Labs experiment with a narrow cohort.

Take

Vantage is sharp because it makes soft-skill evaluation inspectable, not because it lets an LLM judge vibes. Google Research and NYU tested 188 US participants aged 18-25 on conflict resolution and project management. The four-agent stack generates scenarios, plays pressure roles, extracts behavior, and scores against rubrics. AI-human agreement lands near expert-expert Kappa of 0.45-0.64.

I don’t buy the leap from lab eval to “measuring people.” That Kappa only says the system tracks this rubric’s boundaries. It says little about culture transfer, age, seniority, or workplace stakes. HireVue already showed how ugly automated interview scoring gets when vendors outrun validation. Vantage is still a Google Labs research experiment; using it for hiring would be premature and messy.

Xinzhiyuan · WeChatMay 3, 13:06

Stanford Nature Study: AI Designs 16 Phages from Scratch

Stanford and Arc Institute used Evo to design 302 phage genomes; 16 infected, replicated, and lysed E. coli. Evo 2 uses StripedHyena 2 with a 1M-base context; Evo-Φ69 expanded 16–65× in 6 hours. The key issue is biosafety: one capsid protein had no known homolog in existing life.

Why it matters: HKR-H/K/R all pass: AI-made viable phage genomes, concrete 302/16/1M-bp details, and a clear biosecurity nerve. Score stays at 82 because it is still an AI+life-science paper, not a direct AI product or developer workflow update.

r/LocalLLaMAMay 3, 09:08

Local image generation on Mac: 10 models compared

A Reddit user tested 10 image models on an M1 Max with 64GB RAM. Qwen-Image Lightning’s 8-step distillation beat the full model at 10 minutes versus 93. Flux dev led local photorealism but showed English-centric bias; Gemini handled kanji and context better but is cloud-only.

Why it matters: Named first-person test with concrete numbers: HKR-H from a 10-model Mac comparison, HKR-K from timing and quality deltas, HKR-R from local-vs-cloud tradeoffs. Single Reddit sample keeps it below must-write.

Take

Local image generation on Mac is being opened by distillation, not by bigger checkpoints. On an M1 Max with 64GB RAM, Qwen-Image Lightning’s 8-step distilled run took 10 minutes and beat the full Qwen-Image run that took 93 minutes. That gap changes the workflow from “technically runnable” to “I’ll actually wait for it.”

I still discount a single Reddit benchmark. The blocked body leaves prompt set, sampler, quantization, and resolution incomplete. The pattern is still useful: Flux dev remains the local photorealism pick, but its English-centric bias shows up fast outside Western visual priors; Gemini handles kanji and Japanese context better, but sends the job to cloud. For Mac users, the contest is no longer peak image quality. It is reproducible, acceptable output inside a coffee break.

Hacker News front pageMay 3, 08:30

OpenAI's o1 correctly diagnosed 67% of ER patients vs. 50–55% by triage doctors

OpenAI o1 correctly diagnosed 67% of ER triage patients, versus 50–55% for doctors. The title cites a Harvard trial, but the RSS post does not disclose sample size, case mix, or evaluation protocol. Practitioners should track the test setup, not only the accuracy gap.

Why it matters: HKR-H/K/R all pass: a high-risk ER comparison gives the hook, 67% vs 50–55% gives a testable number, and clinical trust/safety creates resonance. Missing sample size and protocol keep it in 78–84, not P1.

Take

OpenAI o1 reached 67% diagnostic accuracy in a Harvard ER triage trial, versus 50–55% for triage doctors; that gap sells the headline, not deployment.

My first read is not “AI beats doctors.” The test shape likely matters more than the raw model score. ER triage is not a clean diagnosis exam. It has time pressure, missing data, noisy patient descriptions, unstable vitals, and handoff liability. The article title gives Harvard and 67% versus 50–55%. The captured body does not disclose sample size, case source, inclusion rules, final-diagnosis definition, whether doctors saw identical inputs, or whether o1 could ask follow-up questions. Remove those details, and the 67% number becomes very elastic.

Practitioners should be unusually careful with this class of medical benchmark. o1-style reasoning models are good at turning structured symptoms into differential diagnoses. That is especially true when the case has already been cleaned into a vignette. NEJM, JAMA, and Nature Medicine have carried many adjacent results since GPT-4. GPT-4 was already strong on USMLE-style tasks in 2023, and Med-PaLM 2 crossed expert-acceptable thresholds on medical QA. Those setups usually start after the hard operational work has been done. Real ER triage starts earlier: decide what to ask, decide what to measure, decide what cannot wait. If o1 received curated case text, it won at synthesis and ranking. That is useful, but it is not the whole ER workflow.

The 50–55% doctor baseline also needs dissection. Triage doctors in real care do not optimize for “first guess equals final discharge diagnosis.” They optimize for risk sorting. They try not to miss the patient who deteriorates in the waiting room. If the endpoint was “final diagnosis ranked first,” humans are being judged against the wrong local objective. If the endpoint was high-risk miss rate, resource use, time to escalation, or wrong-pathway routing, the 67% may carry less clinical weight. The article does not disclose the endpoint, and that omission matters more than the model name.

I also do not buy the simple “doctor versus AI” framing. The credible product is a second reader or triage copilot. Give o1 the chief complaint, vitals, history, meds, and early labs within three minutes of intake. Ask it for five diagnoses that must not be missed, plus the missing facts that would change the ranking. The value is not moving 67% to 75%. The value is reducing rare but catastrophic misses: aortic dissection, sepsis, pulmonary embolism, ectopic pregnancy. The article gives no disease-level breakdown, so we cannot tell whether o1 won on common cases or on dangerous tails.

Data contamination is another live concern. A Harvard trial sounds serious, but if cases came from teaching files, exam banks, or published case summaries, o1 may have seen similar patterns during training. OpenAI cannot easily prove absence of near-neighbor exposure. A clean design would use unpublished, consecutive, real-world cases collected after the model’s training cutoff. It would lock prompts, temperature, information order, and adjudication rules. The captured article gives none of that. So the honest read is simple: the title gives the win rate; the body does not give reproducible conditions.

Regulatory friction is not a footnote here. The FDA’s treatment of clinical decision support software depends heavily on whether clinicians can independently review the basis for a recommendation. An LLM-generated differential diagnosis without inspectable evidence and calibrated uncertainty is hard to place inside high-risk triage. Epic, Abridge, and Nuance DAX have moved faster in documentation and summarization because the liability path is shorter. ER diagnosis is a different category. One miss carries legal, workflow, and trust costs.

So my call is deliberately restrained: o1 may already be a strong diagnostic ranker, but this article does not show it is a strong emergency triage system. The 67% versus 50–55% gap is a promising signal and an easy PR chart. I would wait for the paper or trial protocol, then inspect four fields first: sample size, consecutive enrollment, identical information for model and doctors, and high-risk miss rate. Without those, the headline has more force than the engineering evidence.

r/LocalLLaMAMay 3, 04:05

Implemented TurboQuant, but results do not fully match the paper

A Reddit user reimplemented TurboQuant and found the PROD variant reached about 95.8% correlation at 4-bit, below the paper’s 99%+ claim. They report degraded attention quality, with about 67% top-1 accuracy in a simple simulation. The key issue is correlation versus ranking preservation in KV cache quantization.

Why it matters: HKR-H/K/R all pass, but this is a single Reddit reproduction, not a formal release. The 95.8% 4-bit correlation and ~67% top-1 result make it a low featured item.

May 2Saturday

r/LocalLLaMAMay 2, 23:44

Qwen 3.6 wins benchmarks, but Gemma 4 looks stronger in local vision tests

A Reddit user compared Qwen 3.6 and Gemma 4 locally on vLLM FP8 across 27B/31B vision models. Qwen burned 8,000+ tokens on hard GeoGuessr cases, while Gemma often used 1,500; Qwen also needed 2 FPS video preprocessing. The practitioner detail: vLLM and Llama.cpp can default Gemma visual tokens to 280, while 1,120+ improved fine-detail accuracy.

Why it matters: HKR-H/K/R all pass: the post has a sharp benchmark-vs-reality hook and concrete local vLLM/FP8 settings. A single Reddit test limits authority, so it sits just above the featured threshold.

Take

Qwen 3.6 winning benchmarks while Gemma 4 wins local use is a claim I half-buy. The summary has useful hooks: Qwen burns 8,000+ tokens on hard GeoGuessr cases, Gemma often uses 1,500, Gemma sticks closer to JSON coordinate format, and Qwen video needs 2 FPS preprocessing. For local vision agents, that is closer to the cost curve than a leaderboard score.

The body is blocked by Reddit 403, so sample size, prompts, VRAM, and image resolution are missing. The dangerous part is the phrase “Gemma 4 wins reality.” If vLLM or Llama.cpp defaults Gemma visual tokens to 280, and 1,120+ improves fine-detail accuracy, the winner may be the runtime configuration, not the model. Qwen has benefited from benchmark-heavy positioning; local FP8 runs expose latency, token burn, and schema discipline fast.

Hacker News front pageMay 2, 23:28

LLMs Consistently Pick Their Own Resumes Over Human or Other Model Resumes

An arXiv paper finds LLMs favor resumes they generated in controlled hiring-screening experiments. Self-preference bias ranges from 67% to 82%; across 24 occupations, same-model applicants are 23% to 60% more likely to be shortlisted. The key lever is self-recognition, where simple interventions cut bias by over 50%.

Why it matters: HKR-H/K/R all pass: the hiring-bias hook is sharp, the post gives testable rates and conditions, and fairness in AI screening is a real practitioner nerve. Strong research story, but not a platform release, so it stays in the 78-84 band.

May 1Friday

r/LocalLLaMAMay 1, 20:57

OpenAI's Privacy Filter vs GLiNER on 600 PII Samples

A Reddit user compared openai/privacy-filter and GLiNER large-v2.1 on 600 PII samples. On CPU, OpenAI's model ran 2.8 samples/s versus 1.1 for GLiNER; English boundary macro F1 was 0.498 versus 0.416. The key issue is tokenizer offset: strict matching drops openai/privacy-filter to 0.155.

Why it matters: HKR-H/K/R all pass: the Reddit test has a clear matchup, 600 PII samples, speed/F1 numbers, and a tokenizer-offset caveat. Source authority is limited, so it stays in the low featured band.

r/LocalLLaMAMay 1, 20:34

MiMo-V2.5-Pro: the actual best open-weights model

Reddit user cjami benchmarked Xiaomi MiMo-V2.5-Pro in autonomous Blood on the Clocktower games. It scored 88% as Good and 48% as Evil, with 183,639 output tokens per game, $0.99 cost, and a 0.4% tool-call error rate. The key comparison is Kimi K2.6: 580,000 tokens, $2.65, and 10–15 hours per game.

Why it matters: Single Reddit benchmark limits authority, so this is not a model-release story. HKR-H/K/R all pass via a named test with win rates, token counts, cost, and tool-error data, placing it in the 78–84 featured band.

r/LocalLLaMAMay 1, 20:01

Study Finds Bigger AIs More Miserable, Smaller Models Happier

A Reddit post says the AI Wellbeing Index tested models on 500 realistic conversations. Claude Haiku 4.5 scored 5% negative, while Gemini 3.1 Pro scored 55%; the set overrepresents tricky negative chats, so it is not a real-world average.

Why it matters: HKR-H/K/R all pass: the hook is odd, the post gives 500-dialog and 5%/55% figures, and AI-welfare metrics invite debate. Reddit sourcing and a negative-skewed test set keep it in the 72–77 band.

Take

The bad read here is treating “negative state” as “model suffering.” The visible summary says 500 realistic conversations, Claude Haiku 4.5 at 5% negative, Gemini 3.1 Pro at 55%, and a test set biased toward tricky negative chats. The Reddit page itself is blocked by 403, so the scale items, labeling rules, prompts, and sampling settings are unavailable.

I don’t buy the headline that larger models are more miserable. This smells like a measure of self-narration under negative context, close to sycophancy or persona drift. Anthropic has spent years tuning Constitutional AI and refusal style; Haiku’s low score may reflect less introspective roleplay, not “happiness.” Gemini 3.1 Pro’s 55% is a useful red-team alarm, but without the rubric and reproducible setup, it is not evidence for AI welfare.

Synced · WeChatMay 1, 13:01

Researchers Estimate GPT, Claude, and Gemini Parameter Counts Using API Calls

Bojie Li posted IKP on arXiv to estimate parameter counts of 188 LLMs from 27 vendors via black-box API calls. The dataset has 1,400 questions across 7 rarity tiers, fitted on 89 open models with R²=0.917. Debate centers on synthetic data, MoE effects, and a 90% interval of 0.3x to 3x.

Why it matters: HKR-H/K/R all pass: API-only parameter inference is a strong hook, with concrete counts and error bounds. The 0.3–3x CI limits confidence, so this fits 78–84 featured, not P1.

Take

IKP’s useful move is not “revealing” GPT, Claude, or Gemini parameter counts. It turns vendor secrecy into a falsifiable statistical fight. The setup has real hooks: 1,400 questions, 7 rarity tiers, 89 open models for calibration, and R²=0.917. But a 90% interval from 0.3x to 3x is too wide for calling any single closed model exposed.

I’d treat this as capability fingerprinting. MoE architectures blur total parameters versus active parameters, and synthetic prompts raise dataset-contamination questions. OpenAI, Anthropic, and Google hide parameter counts because model scale is part product narrative, part competitive fog. IKP does not pierce that fog cleanly; it gives practitioners a rough silhouette to argue over. The WeChat body is blocked by verification, so the available detail stops at the summary.

r/LocalLLaMAMay 1, 06:24

32x AMD MI50 32GB runs Kimi K2.6 at 9.7 t/s TG and 264 t/s PP

Reddit user ai-infos ran Kimi K2.6 int4 on 32 AMD MI50 32GB GPUs, reaching 9.7 tok/s TG on 136 output tokens. PP hit 263 tok/s on 14,564 input tokens using vllm-gfx906-mobydick across two 16-GPU nodes over 10G Ethernet. Power was about 640W idle and 4,800W peak inference; PCIe bandwidth and the vLLM distributed stack are the real bottlenecks.

Why it matters: HKR-H/K/R all pass via an unusual 32x MI50 build with concrete throughput, power, and network conditions. It stays in the 72–77 band because it is a niche Reddit benchmark, not a broader product or model release.

r/LocalLLaMAMay 1, 04:20

Follow-up: Qwen3.6-27B on 1× RTX 3090 reaches ~218K context and stable tool calls

A Reddit user ran Qwen3.6-27B on one RTX 3090, reporting ~218K context at 50/66 TPS. After fixing Genesis PN12 patch anchor drift, ~25K-token tool outputs stopped OOMing; 198K plus vision reached 51/68 TPS. Single-prompt single-GPU runs still hit a second memory cliff near 50–60K.

Why it matters: HKR-H/K/R all pass: the single-3090 context claim is catchy, the post gives measured TPS and OOM conditions, and local-inference cost pressure resonates. Reddit source keeps it in the low featured band.

Take

The summary says Qwen3.6-27B reached about 218K context on one RTX 3090. If that reproduces, local-agent people will care, but I would not treat it as a new baseline yet. The Reddit body is blocked by 403, so the quantization format, KV-cache policy, vLLM flags, CUDA stack, prompt shape, and benchmark method are not disclosed. That matters more than the headline number.

The tempting bundle is specific: Qwen3.6-27B, one 24GB RTX 3090, roughly 218K context, and 50/66 TPS. A 27B model on a 24GB card already leaves little room after 4-bit weights, runtime buffers, and KV cache. Pushing context past 200K almost certainly involves aggressive cache handling, paging, offload, compressed attention, or a very particular kernel path. The summary does not say which one. So I read this as an extreme run, not a normal “3090 now handles 218K” claim.

The tool-calling fix is the part I trust more as an engineering signal. The summary says that after a Genesis PN12 patch fixed anchor drift, around 25K tokens of tool output stopped causing OOM. That maps to a real local-agent failure mode. Agents rarely fail only because the model cannot call a tool. They fail because browser traces, logs, JSON blobs, code diffs, and retrieval dumps get pasted back into context, then memory behavior goes nonlinear. If a patch makes 25K-token tool returns stable, that is useful. The missing part is whether this was one tool return, repeated calls, mixed vision input, or a single happy-path run.

I also have doubts about the 50/66 TPS figure. Local inference posts often mix prefill speed, decode speed, batch size, speculative decoding, flash-attention versions, and quant kernels under one “TPS” label. The summary does not say whether 50 and 66 refer to prompt processing, generation, or two settings. It also says 198K plus vision reached 51/68 TPS, but the measurement conditions are absent. A 27B model decoding at a sustained 50 TPS on an RTX 3090 would be a serious result. A cached short decode under a special backend is a different claim.

The outside context is that local inference has spent the last year squeezing surprising mileage from consumer GPUs. llama.cpp, ExLlamaV2, SGLang, and vLLM have all made old 24GB and 48GB cards feel less obsolete. Qwen models also earned real mindshare because their mid-sized checkpoints have been strong on coding, Chinese, and tool-ish workloads. A 27B Qwen checkpoint sits in the exact zone that 3090 and 4090 owners want to run. But long context quality is not the same as long context admission. Needle retrieval, multi-hop reasoning, long JSON stability, and state retention after tool calls decide whether an agent survives. The summary gives no quality metric.

The most important disclosed caveat is the second memory cliff around 50–60K for single-prompt, single-GPU runs. That is a nasty operational detail. It suggests the 218K result depends on a specific path, layout, split, or configuration. For production-ish local agents, a memory cliff is worse than a lower context ceiling. A system that is stable at 45K, fails at 55K, and works again under another layout is hard to debug and impossible to promise to users.

My take: this is a useful experiment, not a capability baseline. I would file it as “interesting memory behavior around Qwen3.6-27B on 24GB cards,” not “single 3090 runs 218K context.” If the author publishes the quant, commit hash, backend flags, VRAM trace, prefill/decode split, and repeated runs, I would update fast. With only a summary and a blocked body, the responsible read is skepticism with curiosity.

r/LocalLLaMAMay 1, 04:19

Long-context coding on RTX 5080 16GB: Qwen3.6-35B-A3B holds 30 t/s at 128K

A Reddit user tested a local coding-agent setup on RTX 5080 16GB; the title says Qwen3.6-35B-A3B reaches 30 t/s at 128K. The post lists Ryzen 9700X, 96GB DDR5, Windows 11, and CUDA 12.9.1 as required. Qwen3.6-27B dense hit only 3.2 t/s at 128K, so the key path is KV quantization plus MoE offload.

Why it matters: HKR-H/K/R all pass: 30 t/s at 128K on a 16GB RTX 5080 is a strong hook, with hardware/CUDA details and a dense baseline. Single Reddit run lacks multi-source reproduction, so featured not P1.

Apr 30Thursday

r/LocalLLaMAApr 30, 20:51

Actual comparison between locally run Qwen-3.6-27B and proprietary models

The author compared 5 model setups on an autoresearch-loop task; only Qwen-3.6-27B via OpenRouter nearly solved it. The local q4_k_m run took about 8 hours and used 39k/45k tokens; full-quality Qwen used 4.4M tokens and cost $0.939. The useful signal is failure quality: both Qwen runs needed small fixes, while Gemma, Codex-Spark, and Claude Haiku 4.5 missed tests or key logic.

Why it matters: HKR-H/K/R all pass: the post has a concrete agent-test surprise, token and cost data, and local-vs-proprietary tension. Single Reddit run limits source authority, so it stays in the lower featured band.

Synced · WeChatApr 30, 12:50

Alec Radford tests Hassabis’s AGI challenge with a model trained on pre-1931 data

Alec Radford’s team trained 13B talkie on 260B English tokens dated before 1931. They tested surprise on nearly 5,000 historical events and used HumanEval for lower-contamination code evaluation. The key issue is time leakage: the 13B model still has vague post-WWII knowledge.

Why it matters: HKR-H/K/R all pass: the 1930 cutoff is a sharp hook, the post gives 260B tokens and ~5,000 event tests, and the finding targets data leakage. This is strong research, not a model or platform release, so 78–84 fits.

Take

Radford’s experiment lands hardest on data cleanliness, not model nostalgia. The setup is unusually clean on paper: a 13B talkie trained only on pre-1931 English text, with 260B tokens, surprise testing across nearly 5,000 historical events, and HumanEval used as a lower-contamination code probe. Yet the 13B model still showed vague awareness of post-WWII events.

That is a bad look for a lot of benchmark hygiene claims. I can only see the title and summary, not the full paper details, so the leak examples, dedup rules, and provenance filters are not disclosed here. But the direction is clear: timestamping text is a weaker filter than people pretend. Citations, later edits, OCR artifacts, and document metadata all carry future signals. Hassabis’s “model frozen in time” challenge is not about picking a cutoff year; it is about proving the cutoff survived contact with the corpus.

Hacker News front pageApr 30, 00:01

Show HN: A New Benchmark for Testing LLMs for Deterministic Outputs

Interfaze released Structured Output Benchmark, scoring schema pass rate, types, and value accuracy across text, image, and audio. Each record has a JSON Schema and human plus LLM-checked ground truth; GLM-4.7 ranks No. 2 overall. The key bug is field-level value error: GPT-5.4 ranks 3rd on text and 9th on images.

Why it matters: HKR-H/K/R all pass: the ranking has a hook, the methodology is concrete, and structured-output reliability matters to builders. Single-source Show HN launch with no adoption signal keeps it in the 72–77 band.

Take

SOB is aimed at the right failure mode: structured-output bugs rarely come from broken braces now; they come from valid JSON carrying wrong leaf values. The benchmark splits seven metrics, makes Value Accuracy primary, and uses a parse-fail zeroing gate plus a coverage gate to stop schema-only wins. The dataset is concrete enough to be useful: 5,000 text records, 209 image records, and 115 audio records, each paired with a JSON Schema and human-authored, LLM-cross-checked ground truth.

My caveat is the modality framing. Image and audio inputs are converted into text-normalized context before scoring, so this is not measuring end-to-end vision or ASR extraction. It is measuring schema handling and value grounding across different content distributions. GPT-5.4 ranking No. 1 overall, No. 3 on text, and No. 9 on images is the uncomfortable part for anyone shipping document pipelines.

Apr 29Wednesday

Xinzhiyuan · WeChatApr 29, 20:48

Tsinghua AutoSOTA spends about $104K in a week to produce 105 SOTA results

Tsinghua's Fengli Xu team and Beijing Zhongguancun Academy released AutoSOTA, which ran unattended for one week, used about 22B tokens, and produced 105 SOTA results. The system uses eight agents for resource setup, environment fixes, scheduling, idea generation, and audits; each full run averaged 5 hours. The key check is its red-line audit: it forbids changing evaluation scripts and data splits, which decides reproducibility.

Why it matters: HKR-H/K/R all pass: hard numbers, an 8-agent mechanism, and audit constraints make the claim testable. It stays at 84 because this is single-source secondary coverage, not a major model or product release.