// blog·

We benchmarked our own tool against a strong grep agent. We lost half the time.

Grafema’s pitch used to lean on a benchmark: agents with graph tools answered 77% of hard codebase questions correctly vs 67% for grep-and-read. Honest numbers — but measured in March, against March-era agents. Agent tooling has improved fast since then, mostly in one direction: modern agents orchestrate text search very well — parallel greps, subagent fan-out, naming-convention sweeps.

So before launching anything, we re-ran the question ourselves, adversarially: if a strong agent with plain grep can match an agent with our graph, we’d rather find out than have a Hacker News commenter find out for us.

This post is the result: 22 probes, two codebases, four rounds. We lost about half of them. The half we won reshaped how we describe the product.

Setup

Same model for both arms of every probe (Claude Sonnet, full agentic loop with subagents allowed). Same question, verbatim. One arm gets Grafema’s MCP tools and a routing skill; the other gets the standard toolkit — grep, file reads, subagents. We measure answer completeness, session cost (bytes of session transcript as a token proxy), and tool choices. Two codebases:

  • express — 142 files, 67K graph nodes. Small enough to fit in one head.
  • grafema itself — a multi-language monorepo (TypeScript, Rust, Haskell), 372K graph nodes. Does not fit in one head.

Raw probe transcripts are in our ops repo; every number below has a session log behind it.

Round 1: if you can grep it, grep it

Question: “Who calls defineGetter from lib/utils.js? List every call site.”

The grep agent answered with one grep call: all 12 call sites, with line numbers — and it correctly noticed the premise was wrong (the function lives in lib/request.js, not utils.js). Session cost: ~1/7th of the graph-guided run, identical answer quality.

That’s the pattern for any question with a textual anchor — a symbol name you can type into a search box. Caller enumeration, value tracing, even cross-language impact: if the name is in the question, text search finds it at any repo size. Our graph-guided agent got the same answers; it just paid more for them.

We expected scale to change this. It didn’t, by itself.

Round 2: scale alone doesn’t save the graph

Question, on the 372K-node monorepo: “I want to change the signature of commitBatch — what breaks? Find all callers across the monorepo, including cross-language ones.”

We assumed the Rust↔TypeScript boundary would defeat grep. It didn’t: commitBatch and commit_batch are two grep patterns, and the grep agent ran 13 of them, found the wire-protocol coupling, the v2 handler, nine call sites in the Rust orchestrator and 23 affected unit tests. Complete answer, lower cost.

The lesson we wrote down: the discriminator is not repo size — it’s whether the question carries its own anchor. A symbol-named question is greppable at a million lines. What repo size actually does is amplify the cases where there is no anchor.

Round 3: no anchor — and the first real win

Question: “Find every place where a resolution-phase error is swallowed without logging.”

No function name to search for. Both agents had to characterize code rather than locate it. The result surprised us in a different way than rounds 1–2:

  • The graph agent (one Datalog query + targeted reads, 33% cheaper) found the TypeScript swallows: catch { return null; } in the manifest resolver, a chain of enrichers that log only at debug level.
  • The grep agent found ~40 swallow points in Rust (unwrap_or_default(), filter_map(.ok())) and Haskell (Nothing -> [] cascades).

Different subsets of the truth. The graph sees semantic structure — what a catch block does with the error. Grep sees textual patterns — every idiom you know to search for. Neither list was complete; the union was the real answer.

That’s not the result we were hoping for. It’s better: it’s reproducible, defensible, and it tells you exactly how to use both tools.

Round 4: nobody’s answers are stable — but the graph’s facts are

We ran one aggregate question three times in each arm and compared the answers’ identifier sets. The grep agent’s three answers shared zero common core. The graph agent’s shared a small one (4 of 64 identifiers) — about twice as consistent, and still nowhere near deterministic.

LLM narratives drift, whatever tools sit underneath. What doesn’t drift: the raw graph queries. grafema query, grafema impact, grafema who return byte-identical output every run, with file:line provenance from real edges. If you need an answer you can act on twice — an architecture check in CI, a guarantee that a layering rule holds — the deterministic layer is the product, and no amount of prompt engineering gives that to a text-search narrative.

What we now claim (and what we don’t)

We don’t claim the graph makes agents better at everything. On anchored questions, a strong agent with grep matches it at lower cost, on any repo we tested. Our own routing skill now tells agents exactly that: if the symbol is in the question, grep first.

We do claim three things, each backed by the rounds above:

  1. The graph makes the implicit explicit. Structure, flow, hidden side effects — the relationships that have no textual anchor. On those questions it’s a different instrument, not a faster grep (round 3).
  2. It’s the missing half of a review. For questions that characterize code rather than locate it, graph and text search surface different findings. Critical reviews should run both and take the union (round 3, measured).
  3. Deterministic, citable facts. Raw graph queries are reproducible and carry provenance — which is what makes architectural CI checks (guarantees) possible at all (round 4).

And one qualifier we attach to all three: this matters at scale. On a nine-file demo, or any project a developer holds comfortably in their head, memory and grep win — and they should. The graph earns its keep where the codebase has outgrown every individual head on the team: multi-service monorepos, multi-language boundaries, the 4-million-line class of repository where “ask the person who knows” stopped working years ago.

Reproduce it

Everything here runs on public code. Install (npm install -g grafema), analyze any repo (grafema analyze --quickstart), and ask both your agent and your graph the hardest no-anchor question you know about your own codebase. If grep wins, tell us — that data point is as useful to us as the other kind. The beta Telegram and GitHub issues are open.