From 2ce4a20764a61482fe3825984e64541cf1af2de5 Mon Sep 17 00:00:00 2001
From: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
Date: Mon, 5 Oct 2026 11:51:47 +0000
Subject: [PATCH] Fix #3690: benchmarks: code-intelligence result inaccessible;
 BENCHMARK

Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
(cherry picked from commit c7bbf6c834cef8875126c5d6e763a70035ede488)
---
 BENCHMARKS.md | 16 ++++++++++++++++
 README.md     |  1 +
 2 files changed, 17 insertions(+)

diff --git a/BENCHMARKS.md b/BENCHMARKS.md
index 6c1a6d331..cdab7867a 100644
--- a/BENCHMARKS.md
+++ b/BENCHMARKS.md
@@ -30,6 +30,7 @@ Highlights:
 | Memory | LOCOMO (300) | QA accuracy | 45.3% | supermemory 49.7% (11x ingest cost), bm25 31.3%, mem0 27.3% |
 | Memory | LOCOMO (300) | recall@10 | 0.497 | bm25 0.362, mem0 0.048 |
 | Memory | LongMemEval-S (50) | QA accuracy | 76% | dense RAG 76%, hybrid 74%, mem0 70% |
+| Code | ERPNext cross-tool (6) | key-fact coverage | 82.0% | grep/read baseline 70.8% |
 | Cost | LOCOMO ingest | USD | ~$1.40 | supermemory $15.67, mem0 $3.48 |
 | Cost | graph build | LLM credits | $0 | n/a |
 
@@ -148,6 +149,21 @@ graphify pays for itself in accuracy against searching raw files, and avoids the
 context-stuffing anti-pattern of packing the whole repo into every turn (which
 costs roughly 20x the tokens for lower coverage).
 
+### INFERRED cross-file edge precision (FastAPI demo corpus)
+
+The README hero graph is the upstream [FastAPI](https://github.com/fastapi/fastapi)
+library extracted AST-only (no semantic pass). To check the `INFERRED` confidence
+tag on cross-file resolver edges, we manually audited every such edge
+(`calls`, `indirect_call`, `imports`, `imports_from`, `inherits`) whose source and
+target both live under `fastapi/` (production package sources, excluding the test
+tree). This is a dated manual spot-check, not a continuously-measured metric: on a
+fresh AST-only extract (2026-10-05) all 12 such edges were correct. Reproduce by
+running `graphify .` against a FastAPI checkout and inspecting the INFERRED
+cross-file edges whose endpoints both sit under `fastapi/`. Homonymous helpers in
+`tests/` produce noisier INFERRED `calls` edges; the tag is most meaningful on
+production cross-file resolution, where the resolver prefers no edge over a wrong
+one.
+
 ## Results: temporal (15 years of ERPNext)
 
 689 weekly AST checkpoints, 2011 to 2026, built deterministically with no LLM.
diff --git a/README.md b/README.md
index fd72e793a..fb8eaf372 100644
--- a/README.md
+++ b/README.md
@@ -126,6 +126,7 @@ What you get out of the box:
 | LOCOMO (n=300) | recall@10 | **0.497** | mem0 0.048, supermemory 0.149 |
 | LOCOMO (n=300) | QA accuracy | 45.3% | supermemory 49.7%, mem0 27.3% |
 | LongMemEval-S (n=50) | QA accuracy | **76%** | tied with dense RAG |
+| ERPNext cross-tool (n=6) | key-fact coverage | **82.0%** | grep/read baseline 70.8% |
 | Graph build | LLM credits | **0** | per-token for most systems |
 
 Every system ran on the same harness with the same model and budgets, scored by a judge blind-validated against a second judge (90.6% agreement, Cohen's kappa 0.81). Full per-system tables, the code-intelligence result, and reproduction commands: **[BENCHMARKS.md](./BENCHMARKS.md)**.