LLMGRAPH benchmarks
Every number on this page comes from a run we can hand you: the method, the questions fixed before the run, and the raw answers. These are our own tests, run by us or by a tester we briefed, and each section says which. Rerun them and tell us where they don't hold.
No confident answer was wrong: blind test, 50 questions
Our own test, LLMGRAPH 0.4.2, 22 October 2026. A tester with no access to the engine's code, using only the public docs, asked 50 graded questions about callers and change impact on three open-source repos and checked every answer against the code.
| Repo | Graded | Correct | Wrong | Silently incomplete | Misleading | Unclear |
|---|---|---|---|---|---|---|
| requests v2.32.3 | 20 | 18 | 0 | 0 | 0 | 2 |
| rich v13.9.4 | 16 | 8 | 0 | 0 | 5 | 3 |
| zod v3.23.8 | 14 | 4 | 0 | 1 | 6 | 3 |
A confident answer is a line labelled certain, exactly these or within these. None of them was wrong. Limits seen in the run: long open lists padded with unrelated candidates (labelled open, never confident; work in progress), and one alias call in zod dropped instead of flagged (flagged as a blind spot since 0.4.3).
Method and raw answers: https://github.com/brainfoods-studio/llmgraph-agent-plugin/tree/main/benchmarks/blind-test
Find callers on noisy names: LLMGRAPH against a plain text search
Our own test, LLMGRAPH 0.4.1, 20 October 2026. 19 function and method names on requests, rich, werkzeug and hono, with the true callers decided by reading every site, independent of both tools. The names were fixed before the runs.
| Text search (ripgrep), one search | LLMGRAPH | |
|---|---|---|
| True callers found | 125 of 126 | 113 of 126 listed: 85 claimed, 28 as open items |
| Wrong-call lines shown | 413 | 0 claimed wrong; 79 open items that were not callers, all labelled maybe more or can't see, never confident |
| Lines that are not calls | 234 | 0 |
On the 9 names that also appear in unrelated code, the coding agent read 12,234 tokens with one text search and 5,013 with LLMGRAPH: less than half as many, and no wrong caller claimed.
Limits: on clean, unique names a text search finds the same callers for fewer tokens; the first answer from a fresh store takes seconds; 13 true callers were not listed (two of them answered about the class instead, fixed since).
Method, the fixed list of names and raw data: https://github.com/brainfoods-studio/llmgraph-agent-plugin/tree/main/benchmarks/noisy-names
Your code stays on your machine: privacy run on the released program
Our own test, npm 0.4.3, 24 October 2026. We ran the released package twice with a DNS trace and a process-tree sampler. Start-up and the licence check contacted only beta.llmgraph.in; reading, checking and answering contacted nothing. A separate run on the test build read the contents of what is sent: no code, paths, symbol or repo names, queries or answers.
Method and logs: https://github.com/brainfoods-studio/llmgraph-agent-plugin/tree/main/benchmarks/privacy
Check the rulebook examples yourself
Each chimera in the Agent Rules rulebook is a small project we wrote to show one hidden call path breaking after an ordinary edit. All ten are in Code Chimeras (MIT): https://github.com/brainfoods-studio/Code_Chimeras. One command runs the tests on the broken code, then on the fix:
python reproduce.py CH-06How we run these
- Questions, names and ground truth are fixed before a run, and decided by reading the code, not by either tool.
- Every answer is captured by script and published unedited; where a block on our pages is trimmed, it says so.
- A result goes on this page only with its raw data, and it stays here when a later version changes it: we add the new run beside it.
Real answers from these runs, as an agent sees them: Real answers.