kapa.ai taught a small LLM to prune 68% of RAG context while keeping 96% recall
Pruning RAG context down to what the answer actually needs
kapa.ai inserted a cheap small LLM between retriever and generator to prune context. The pruner reads the question and all retrieved chunks together, grades them on a five-level scale, and drops 68% of context before the expensive model sees it. Net query cost fell by a third while recall held at 96%. Cutting on rerank scores failed because scores aren't calibrated across queries and can't see cross-chunk relevance. Anchor documents fixed calibration but not the underlying scores. The working design judges the set, not individual chunks.
Why it matters: kapa.ai uses a small model to read the question and retrieved passages together and prune, handling reranking's trouble with one cutoff across queries and with ignoring how passages relate.