Files
rea/tests/boundary/providers/kuna/kunaReferenceFiles.test.ts
T
Kaoru0822andClaude Opus 5.5 8ab5024397 fix(kuna): stream string scans instead of capping them at 16 MiB (#4318)
* fix(kuna): stream string scans instead of capping them at 16 MiB

`list_strings` and `search_strings` buffered Kuna's whole string JSON and
refused anything above 16 MiB. Kuna writes about 290 bytes per row, so
every app above roughly 10 MB failed with a non-retryable memory error:
a 20 MB Ghostty binary produces 31 MB, Raycast 30 MB and an arm64
Firefox XUL 179 MB.

Parse the scan as Kuna writes it. `runOwnedCommand` accepts a stdout
sink, and `KunaStringInventoryReader` owns the interpretation: it
tokenizes synchronously with stream-json, keeps exact u64 address
tokens, validates each row before retaining it, skips members it does
not interpret, and admits retained rows against live heap headroom
rather than a fixed scan size. The first contradiction stops Kuna
instead of waiting for the rest; an incomplete document is judged only
after Kuna's own exit status. Diagnostics keep the 8 MiB Kuna budget,
now shared with the project export.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): admit string scan text as it streams and drop sinks with their command

Review found that packed string tokens were assembled whole before the
reader could discard them or check heap headroom, and that a stdout sink
could outlive a cancelled or timed-out command.

Move the token mechanics into `KunaJsonDocumentReader`: keys and numbers
(Kuna's own fields) stay packed, while strings, which carry target-derived
text, arrive in chunks. Ignored strings are dropped unassembled and kept
ones are admitted against live heap headroom as they grow. Kept text is
copied out of the decoded chunk it was sliced from, so retained rows no
longer pin whole chunks (XUL: 278 MB retained before, 156 MB after).
`KunaStringInventoryReader` keeps only the string-scan semantics.

`runOwnedCommand` now ties the stdout pipeline to the command's deadline
signal and aborts it when collection fails, destroying the sink.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): type the stream-json tokenizer where it is imported

The Reverse tools quality lane type-checks a partial project that reaches
KunaJsonStream without the separate streamJsonParser.d.ts augmentation,
so jsonParser had no declaration there. Declare the tokenizer's shape at
its only import site and remove the augmentation file, so every
compilation of the reader is self-contained. Split two helpers out of
the token dispatcher and simplify the stdout sink wait in
runOwnedCommand, which keeps the added complexity minimal.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): bound key and number assembly in streamed Kuna JSON

Review found that packed keys (and packed numbers) were assembled whole
by the tokenizer before the reader could discard or constrain them, so a
malformed scan with one enormous key could still exhaust the heap.

Stream every scalar. Keys are assembled only up to 64 code units, past
every field name the reader keeps, so a longer key is skipped unmatched;
kept numbers longer than that are rejected, and ignored numbers are
dropped unassembled. XUL's 611k-row scan parses in the same time.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): stream the call graph instead of capping it at 32 MiB (#4329)

* fix(kuna): stream the call graph instead of capping it at 32 MiB

Call relationships buffered `kuna decompile-graph` and refused output
above 32 MiB. A 53 MB Raycast binary produces a 157 MB graph (180k
functions, 773k edges), so every call query failed with a non-retryable
memory error after Kuna's minute-long whole-program walk.

`KunaCallGraphReader` validates the graph through the shared streaming
document reader. It retains only the facts queries use (126 MB for
Raycast) and checks edges against the complete node set once Kuna
exits. Unrequested bodies are refused before any of their text is
assembled, through new must-be-null row fields; address fields may be
null, as `forwardsTo` is. Malformed graphs are now `AnalysisOutputError`
like the string scan.

`KunaQueryWork.scan` owns the run-and-validate sequence both producers
shared: diagnostics budget, early rejection and retention of
deterministic overflow.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): index call-graph edges as they stream

Review found that the reader kept every parsed edge and built the edge
indexes only after streaming, outside any heap admission, so a large
graph could exhaust the heap instead of returning a memory constraint.

Index each edge when it arrives: duplicate pairs and callee orders are
checked per caller, call edges go straight into the caller/callee
indexes, and no edge array is kept, so the whole working set grows under
the per-row admission. Endpoints and order completeness are checked once
all nodes are known, and the validation-only indexes are released then.
Raycast's graph now peaks at 346 MB instead of 503 MB and retains 93 MB
instead of 126 MB, in the same time.

Also drop imports the shared scan made unused.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(kuna): state the graph's retained size after incremental indexing

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 21:53:59 -07:00

328 lines
8.9 KiB
TypeScript

import { Readable } from "node:stream";
import { pipeline } from "node:stream/promises";
import * as v8 from "node:v8";
import { expect, it, vi } from "vitest";
import { normalizeKunaAddress } from "../../../../src/kuna/KunaAddress.js";
import {
KunaCallGraphReader,
parseKunaIncomingReferences,
} from "../../../../src/kuna/KunaReferenceFiles.js";
vi.mock("node:v8", { spy: true });
const input = "/snapshot/program";
// Fixture-only numeric placeholders preserve decimal u64 tokens in JSON text.
const wire = (value: unknown) =>
JSON.stringify(value).replace(/"@(\d+)"/gu, "$1");
const node = (address: string, name: string) => ({
address: `@${address}`,
name,
kind: "normal",
hasIndirectCalls: false,
forwardsTo: null,
codeC: null,
assembly: null,
error: null,
});
const graph = () => ({
schemaVersion: 4,
binary: { sourcePath: input, functionCount: 3, edgeCount: 2 },
functions: [
node("9007199254740993", "caller"),
node("18446744073709551615", "callee"),
node("1", "jump-target"),
],
edges: [
{
callerAddress: "@9007199254740993",
calleeAddress: "@18446744073709551615",
kind: "call",
calleeOrder: 0,
},
{
callerAddress: "@9007199254740993",
calleeAddress: "@1",
kind: "jump",
calleeOrder: 1,
},
],
});
/** Stream a graph through the reader in small pieces, as Kuna's stdout arrives. */
const parseKunaCallGraph = async (text: string, selected: string) => {
const bytes = Buffer.from(text);
const chunks: Buffer[] = [];
for (let offset = 0; offset < bytes.byteLength; offset += 7)
chunks.push(bytes.subarray(offset, offset + 7));
const reader = new KunaCallGraphReader();
await pipeline(Readable.from(chunks), reader);
return reader.finish(selected);
};
it("preserves u64 numeric tokens and excludes non-call edges", async () => {
const result = await parseKunaCallGraph(wire(graph()), input);
expect(result.ok).toBe(true);
if (!result.ok) return;
expect([...result.value.nodes.keys()]).toEqual([
"0x20000000000001",
"0xffffffffffffffff",
"0x1",
]);
expect(result.value.callees.get("0x20000000000001")).toEqual([
"0xffffffffffffffff",
]);
expect(result.value.callers.get("0xffffffffffffffff")).toEqual([
"0x20000000000001",
]);
expect(result.value.callers.has("0x1")).toBe(false);
});
it("keeps a forwarding instruction address separate from function identity", async () => {
const value = graph();
const result = await parseKunaCallGraph(
wire({
...value,
functions: value.functions.map((fn, index) =>
index === 0 ? { ...fn, forwardsTo: "@9007199254741000" } : fn,
),
}),
input,
);
expect(result.ok).toBe(true);
if (!result.ok) return;
expect(result.value.nodes.get("0x20000000000001")?.forwardsTo).toBe(
"0x20000000000008",
);
expect(result.value.nodes.has("0x20000000000008")).toBe(false);
expect(result.value.callees.get("0x20000000000001")).toEqual([
"0xffffffffffffffff",
]);
});
it.each([
[
"version",
(value: ReturnType<typeof graph>) => {
value.schemaVersion = 5;
},
],
[
"count",
(value: ReturnType<typeof graph>) => {
value.binary.edgeCount = 1;
},
],
[
"duplicate node",
(value: ReturnType<typeof graph>) => {
value.functions[1]!.address = value.functions[0]!.address;
},
],
[
"unknown endpoint",
(value: ReturnType<typeof graph>) => {
value.edges[0]!.calleeAddress = "@2";
},
],
[
"duplicate edge",
(value: ReturnType<typeof graph>) => {
value.edges[1]!.calleeAddress = value.edges[0]!.calleeAddress;
},
],
[
"duplicate order",
(value: ReturnType<typeof graph>) => {
value.edges[1]!.calleeOrder = 0;
},
],
[
"missing order",
(value: ReturnType<typeof graph>) => {
value.edges[1]!.calleeOrder = 2;
},
],
[
"overflow",
(value: ReturnType<typeof graph>) => {
value.functions[0]!.address = "@18446744073709551616";
},
],
] as const)(
"rejects graph %s rather than returning an empty relationship set",
async (_name, mutate) => {
const value = graph();
mutate(value);
expect(await parseKunaCallGraph(wire(value), input)).toMatchObject({
ok: false,
error: { _tag: "AnalysisOutputError", operation: "kuna:call-graph" },
});
},
);
it("rejects unrequested bodies and a document for a different input", async () => {
const value = graph();
expect((await parseKunaCallGraph(wire(value), "/other/program")).ok).toBe(
false,
);
const body = {
...value,
functions: value.functions.map((fn) => ({
...fn,
codeC: "must not be materialized as reference data",
})),
};
expect(await parseKunaCallGraph(wire(body), input)).toMatchObject({
ok: false,
error: { reason: "functions row codeC must be null" },
});
});
it("admits the edge index while edges stream and refuses it under low headroom", async () => {
const nodes = Array.from({ length: 300 }, (_, index) =>
node(String(index + 1), `f${String(index)}`),
);
const edges = Array.from({ length: 3000 }, (_, index) => ({
callerAddress: `@${String((index % 10) + 1)}`,
calleeAddress: `@${String(Math.floor(index / 10) + 1)}`,
kind: "call",
calleeOrder: Math.floor(index / 10),
}));
const text = wire({
schemaVersion: 4,
binary: { sourcePath: input, functionCount: 300, edgeCount: 3000 },
functions: nodes,
edges,
});
expect(await parseKunaCallGraph(text, input)).toMatchObject({
ok: true,
value: { edgeCount: 3000 },
});
const heap = v8.getHeapStatistics();
let samples = 0;
// Headroom runs out after the node rows have been admitted.
vi.mocked(v8.getHeapStatistics).mockImplementation(() => ({
...heap,
total_available_size:
(samples += 1) <= 2 ? heap.total_available_size : 1024,
}));
try {
const refused = await parseKunaCallGraph(text, input);
expect(refused).toMatchObject({
ok: false,
error: {
_tag: "AnalysisResourceConstraintError",
resource: "memory",
},
});
if (refused.ok) return;
const retained = (
refused.error as unknown as {
reportedLimits: { retained_rows: number };
}
).reportedLimits.retained_rows;
expect(retained).toBeGreaterThan(nodes.length);
expect(retained).toBeLessThan(nodes.length + edges.length);
} finally {
vi.mocked(v8.getHeapStatistics).mockReset();
}
});
it("skips fields it does not interpret, including nested parameters", async () => {
const value = graph();
const result = await parseKunaCallGraph(
wire({
...value,
binary: { ...value.binary, name: "program", analysisImageBase: 0 },
functions: value.functions.map((fn) => ({
...fn,
size: 16,
parameters: [{ name: "a0", type: "int" }],
signature: "int f(int a0)",
isEntryPoint: false,
})),
}),
input,
);
expect(result).toMatchObject({ ok: true, value: { edgeCount: 2 } });
});
const identity = (name: string, value: number) => ({
name,
address: `@${value}`,
address_hex: `0x${value.toString(16)}`,
});
const row = (from: number, to: number) => ({
address: `@${from}`,
address_hex: `0x${from.toString(16)}`,
from_address: `@${from}`,
from_address_hex: `0x${from.toString(16)}`,
to_address: `@${to}`,
to_address_hex: `0x${to.toString(16)}`,
kind: "call",
from_function: identity("caller", 4096),
to_function: null,
instruction: "CALL",
});
const xrefs = () => ({
binary: input,
direction: "to",
count: 3,
target: { ...identity("callee", 8192), aliases: [identity("slot", 12288)] },
xrefs: [row(4100, 8192), row(4100, 8192), row(4200, 12288)],
});
it("keeps exact endpoints, deduplicates sites and counts excluded forwarding aliases", () => {
expect(
parseKunaIncomingReferences(wire(xrefs()), input, "0x2000"),
).toMatchObject({
ok: true,
value: { addresses: ["0x1004"], rows: 3, excludedAliasRows: 1 },
});
});
it.each([
[
"count",
(value: ReturnType<typeof xrefs>) => {
value.count = 2;
},
],
[
"target",
(value: ReturnType<typeof xrefs>) => {
value.target.address = "@8193";
},
],
[
"unadvertised alias",
(value: ReturnType<typeof xrefs>) => {
value.target.aliases = [];
},
],
[
"numeric/hex mismatch",
(value: ReturnType<typeof xrefs>) => {
value.xrefs[0]!.from_address_hex = "0x1005";
},
],
] as const)(
"rejects xref %s without inventing absent references",
(_name, mutate) => {
const value = xrefs();
mutate(value);
expect(parseKunaIncomingReferences(wire(value), input, "0x2000").ok).toBe(
false,
);
},
);
it("bounds u64 input parsing before BigInt allocation", () => {
expect(normalizeKunaAddress(`0X${"0".repeat(1000)}ABC`)).toBe("0xabc");
expect(normalizeKunaAddress("0xffffffffffffffff")).toBe("0xffffffffffffffff");
expect(normalizeKunaAddress("0x10000000000000000")).toBeUndefined();
expect(normalizeKunaAddress("caller")).toBeUndefined();
});