Skip to content
r/LocalLLaMA

Evaluated a RAG Chatbot: The Most Expensive Model Was the Worst Performer

Evaluated a RAG chatbot and the most expensive model was the worst performer. Notes on what actually moved the needle.

The author evaluated a customer-support RAG bot and raised the quality score from 6.62 to 7.88 while cutting per-session cost from $0.002420 to $0.000509, using retrieval logging, LLM-as-judge scoring, chunk deduplication, stricter grounding, and a five-model sweep.

Why it matters: A builder used a model as judge to tune a customer-service RAG bot, lifting quality from 6.62 to 7.88 and cutting cost per session by nearly 80%, while the priciest model performed worst.

Read the original ↗Export Markdown