MemStitch: Zero-copy KV cache stitching for vLLM cuts multi-agent TTFT by up to 25x
Show HN: MemStitch – Zero-copy context bridging for vLLM (25x TTFT speedup)
DaqulaLin open-sourced MemStitch, a gateway that sits in front of vLLM and stitches KV caches across requests at the memory level using PagedAttention. It skips redundant prefill in multi-agent workflows, cutting TTFT by up to 25x and saving over 40% VRAM. The post doesn't disclose the test model or GPU setup, so I'd discount the 25x claim until those details surface.
Why it matters: MemStitch stitches KV caches from different requests directly in vLLM's memory layer, skipping repeated prefill in multi-agent collaboration, cutting time to first token by up to 25x and memory use by over 40%.