Grep beats LSP? Why coding agents ignore your fancier tools

An AgentConnect engineer tested three Claude models on code retrieval and editing tasks. When both grep and LSP tools were available, models chose the semantic tool only 0–6% of the time for localization and rename tasks; forcing LSP-first dropped success from 100% to 89%. On reference-completeness tasks, models routed to LSP 45–57% of the time, lifting precision from 0.76 to 1.00, but recall stayed at 0.66 for both—the limit was agent thoroughness, not retrieval accuracy. The LSP tool initially returned only file locations, forcing extra file reads; switching to inline source context raised rename Pass@1 from 0.67 to 0.83 and cut follow-up reads from 15.2 to 3.2 per episode, below grep's 4.3. Codebase noise was the decisive factor: on a clean repo where grep precision was 1.00, LSP added zero F1 gain and cost 16% more tokens; on a noisy repo where grep precision was 0.51, LSP improved F1 by 0.246 while saving 12% tokens. LLM-friendliness depends on output shape and interface design, not just result precision.

Why it matters: AgentConnect ran a clean, small-scale experiment across three Claude models comparing grep vs. LSP for code retrieval. The numbers are concrete (0–6% voluntary LSP usage, success drop when forced). Directly useful for coding agent builders. Points off for small sample size, un...