Why LLM self-reported confidence scores can't be trusted for calibration
What happened
On October 21, Hacker News' front page featured an article on the reliability of confidence scores generated by LLMs. It argues that a model's self-reported confidence is text generated token by token, and those numbers have no calibration basis against actual task accuracy. A reliable confidence score should draw on the information behind the prediction and be checked continuously against real outcomes on similar problems; matching a historical average accuracy is not enough. Using a customer-service agent as an example, the article suggests logging retrieval hits, consistency across repeated answers and human review results, building scoring rules from historical accuracy, and checking calibration regularly.
Written by AI from the coverage · updated 1 hour ago
Coverage
Follow the reports to see the story from different sides.
- Hacker News front pageUsing LLM for confidence scoring is a bad idea
LLM 自报的置信度是逐个 token 生成的文本，文章指出，这些数字缺乏针对实际任务正确率的校准依据。可靠的置信度需要利用预测所依据的信息，并通过同类问题的实际结果持续检验，单纯符合历史平均正确率还不够。文章以客服智能体为例，提出记录检索命中、重复回答一致性和人工复核结果，按历史正确率建立评分规则并定期检查校准情况。
Heat over time
Not enough continuous observations to draw a trend yet.