Skip to content
Hacker News front page

Grep beats LSP? Why coding agents ignore your fancier tools

An AgentConnect engineer tested three Claude models on code retrieval and editing tasks. When both grep and LSP tools were available, models chose the semantic tool only 0–6% of the time for localization and rename tasks; forcing LSP-first dropped success from 100% to 89%. On reference-completeness tasks, models routed to LSP 45–57% of the time, lifting precision from 0.76 to 1.00, but recall stayed at 0.66 for both—the limit was agent thoroughness, not retrieval accuracy. The LSP tool initially returned only file locations, forcing extra file reads; switching to inline source context raised rename Pass@1 from 0.67 to 0.83 and cut follow-up reads from 15.2 to 3.2 per episode, below grep's 4.3. Codebase noise was the decisive factor: on a clean repo where grep precision was 1.00, LSP added zero F1 gain and cost 16% more tokens; on a noisy repo where grep precision was 0.51, LSP improved F1 by 0.246 while saving 12% tokens. LLM-friendliness depends on output shape and interface design, not just result precision.

Why it matters: Pengcheng Xu's test across three Claude models and several Python and TypeScript repos shows agents pick grep for simple lookups, and forcing the semantic path drops success from 100% to 89%.

Read the original ↗Export Markdown