Files
291059adb4 feat(analysis-view): project an environment manifest from web network captures (#4954)
* feat(analysis-view): project an environment manifest from web network captures

Add the environment-manifest view kind to inspect_analysis_view for retained
HAR Evidence: the primary origin's recorded request lines become a
deterministic tool surface (names, path templates, parameters with their
rule and basis, observed query names, value-free body shapes, record
pointers), with excluded groups, three fixed unknowns and the shared
projection budget. The four format rules always apply; after-uid-literal,
opaque-token and sibling-variance are opt-in and labelled, and literal
fold-in never touches a format-rule parameter. The api-surface accumulator
moves onto shared route, media-type and budget helpers without behaviour
change.

Refs morluto/rea#4933

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(web-network): check the manifest refusal text like the api-surface refusal

The real-capture verifier stringified the whole MCP result for the
environment-manifest refusal, so a declared "[" literal matched the result
envelope itself; compare the parsed refusal as the api-surface branch does.

Refs morluto/rea#4933

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(specs): record the hosted web-network lane run for the manifest view

Refs morluto/rea#4933

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(specs): reconcile the manifest view verification scope note with the hosted run

Refs morluto/rea#4933

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(analysis-view): bound body, route and name work in the environment manifest

Addresses the automated review findings on the PR: read request bodies at the
top level only instead of walking nested values; admit every literal route
part before materializing it; compare at most 32 segments in sibling-variance
and say so; keep stop words literal after a uid literal; keep the next
collision suffix per tool-name base; preserve a templated exclusion when two
groups print the same row.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(analysis-view): keep manifest folding, naming and media-type reads linear

Addresses the second automated review round: fold literal siblings through
set membership, key collision suffixes by the truncated name they share, and
charge a request media type before it is scanned.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(analysis-view): charge the manifest group key before building it

Addresses the third automated review round: the composite key copies the
method, origin, template and rules, so that copy is reserved before the key
is stringified.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(analysis-view): accept a +json suffix only on application media types

Addresses the fourth automated review round: `image/example+json` or
`text/example+json` no longer counts as JSON for origin selection, exclusion
or body shapes, matching the documented `application/json` and
`application/*+json` contract.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(analysis-view): require the capture parent's exact subject for a manifest

Addresses the fifth automated review round: the manifest cites the report
path and digest, so a parent whose subject is absent or names another path or
SHA-256 is refused at admission, as the capture view already does; the
api-surface view keeps its existing admission.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(analysis-view): share truncated suffix allocation work

Track suffix progress by the actual truncated prefix and suffix width. Distinct full names can collide only after suffix truncation; sharing that progress prevents quadratic rescanning while preserving shorter free names and existing allocation order.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* fix(analysis-view): merge only identifier-like siblings under sibling-variance

Addresses the sixth automated review finding: a bucket merged whenever one
literal carried a digit, folding plain words such as `mine` into the
parameter. The rule now partitions the bucket, merges the identifier-like
siblings and leaves the other literals as their own tools.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(analysis-view): combine qualifying sibling cohorts

Keep coded identifiers and a qualifying plural hyphenated cohort in one eligibility Set before checking the minimum merge size. Read each literal once and reuse the same membership at the consumer, preserving non-identifier routes and the three-hyphen minimum.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

* docs(analysis-view): clarify opt-in sibling folding

State the additional sibling-variance selection in the guide, acceptance specification and MCP parameter description. Preserve the existing single-rule behavior and caller-selected inference rules.

Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
2026-10-19 07:07:37 +08:00

306 lines
10 KiB
TypeScript

import { apiSurfaceHar, environmentManifestHar } from "./historicalHar.js";
import { webNetworkCaptureSchema } from "../../src/domain/webNetworkCapture.js";
import { BINARY_LAYOUT_TEST_PROVIDER } from "./binaryDiagnostics/layout.js";
import { binaryLayoutFixture } from "./binaryDiagnostics/layout.js";
import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../../src/contracts/javascript/javascriptRuntimeReconciliationExample.js";
import { createEvidence, type Evidence } from "../../src/domain/evidence.js";
import type { JavaScriptApplicationAnalysisResult } from "../../src/domain/javascript/javascriptApplicationAnalysis.js";
import { javascriptApplicationAnalysisResultSchema } from "../../src/domain/javascript/javascriptApplicationAnalysis.js";
import {
createJavaScriptApplicationGraph,
createJavaScriptApplicationNode,
} from "../../src/domain/javascript/javascriptApplicationGraph.js";
import { createJavaScriptSemanticGraph } from "../../src/domain/javascript/javascriptSemanticGraph.js";
import type { BinaryLayout } from "../../src/domain/native/binaryLayout.js";
import { binaryLayoutSchema } from "../../src/domain/native/binaryLayout.js";
import { ghidraFunctionDossier } from "../../src/domain/ghidraValues.fixture.js";
import { functionDossierSchema } from "../../src/domain/hopperValues.js";
import type { RelationshipCoverage } from "../../src/domain/relationshipCoverage.js";
/** Canonical portable Evidence; no native provider is started by this fixture. */
export const analysisViewNativeEvidence = (
relationshipCoverage?: RelationshipCoverage,
): Evidence =>
createEvidence(
{ path: "/fixtures/native.exe", format: "pe", sha256: "a".repeat(64) },
{ id: "fixture", name: "Fixture", version: "1" },
{
operation: "analyze_function",
parameters: { address: "0x401000" },
result: {
...functionDossierSchema.parse(ghidraFunctionDossier()),
limitations: ["Fixture dossier caveat"],
},
...(relationshipCoverage === undefined ? {} : { relationshipCoverage }),
},
);
const hex = (value: number): string =>
value === 0 ? "0x0" : `0x${value.toString(16)}`;
const layoutName = (display: string, offset: number) => ({
display,
bytes_base64: Buffer.from(display, "utf8").toString("base64"),
location: { offset: hex(offset), bytes: hex(display.length + 1) },
unknown_reason: null,
});
const layoutSection = (
index: number,
display: string,
nameOffset: number,
): BinaryLayout["sections"][number] => ({
index,
name: layoutName(display, nameOffset),
name_offset: hex(nameOffset),
type: 1,
header_location: { offset: hex(index * 16), bytes: "0x10" },
address: "0x0",
offset: hex(64 + index * 16),
size: "0x10",
alignment: "0x1",
flags: "0x6",
link: 0,
info: 0,
entry_size: "0x0",
file_backing: "file",
});
const layoutSymbol = (
entryIndex: number,
display: string,
nameOffset: number,
): BinaryLayout["symbols"][number] => ({
table_index: 0,
entry_index: entryIndex,
name: layoutName(display, nameOffset),
name_offset: hex(nameOffset),
location: { offset: hex(nameOffset), bytes: hex(display.length + 1) },
value: "0x0",
value_meaning: "no-address",
size: "0x0",
binding: 1,
type: 2,
visibility: 0,
section_index: 1,
});
/** Layout fixture with duplicate section/symbol names for selected-view tests. */
export const analysisViewLayoutFixture = (): BinaryLayout =>
binaryLayoutSchema.parse({
...binaryLayoutFixture("/artifacts/view-source.elf"),
artifact: {
path: "/artifacts/view-source.elf",
sha256: "b".repeat(64),
bytes: 512,
},
sections: [
layoutSection(0, ".text", 256),
layoutSection(1, ".text", 262),
layoutSection(2, ".data", 268),
],
symbols: [
layoutSymbol(0, "main", 280),
layoutSymbol(1, "main", 285),
layoutSymbol(2, "start", 290),
],
});
/** Wrap a layout observation as inspect_binary_layout Evidence. */
export const analysisViewLayoutEvidence = (
layout: BinaryLayout = analysisViewLayoutFixture(),
): Evidence =>
createEvidence(
{
path: layout.artifact.path,
format: "elf",
architecture: "x86_64",
sha256: layout.artifact.sha256,
},
BINARY_LAYOUT_TEST_PROVIDER,
{
operation: "inspect_binary_layout",
parameters: { path: layout.artifact.path },
result: layout,
rawResult: null,
confidence: "observed",
limitations: layout.limitations,
locations: [{ kind: "artifact-path", path: layout.artifact.path }],
},
);
/** Minimal authenticated JavaScript application analysis used by view tests. */
export const analysisViewJavaScriptAnalysis =
(): JavaScriptApplicationAnalysisResult =>
javascriptApplicationAnalysisResultSchema.parse(
JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE.normalized_result,
);
/** Application analysis whose module observations include source text. */
export const analysisViewJavaScriptAnalysisWithSource =
(): JavaScriptApplicationAnalysisResult => {
const analysis = analysisViewJavaScriptAnalysis();
const original = analysis.graph.nodes[0];
if (original === undefined)
throw new Error("JavaScript application example is missing its module");
const withSource = createJavaScriptApplicationNode({
kind: original.kind,
identity: original.identity,
observations: original.observations.map((observation) => ({
label: observation.label,
properties: {
...observation.properties,
source: "export const secret = 1;\n",
text: "should not leak",
},
evidence: observation.evidence,
})),
});
const graph = createJavaScriptApplicationGraph({
schema: "JavaScriptApplicationGraph",
root_node_ids: [withSource.node_id],
nodes: [withSource],
edges: [],
coverage: analysis.graph.coverage,
limitations: analysis.graph.limitations,
});
const { graph_id: _graphId, ...semantic } = analysis.semantic_graph;
return javascriptApplicationAnalysisResultSchema.parse({
...analysis,
graph,
semantic_graph: createJavaScriptSemanticGraph({
...semantic,
application_graph_id: graph.graph_id,
}),
});
};
/** Rebind a companion semantic graph after replacing the application graph. */
export const analysisViewBindJavaScriptGraphs = (
analysis: JavaScriptApplicationAnalysisResult,
graph: ReturnType<typeof createJavaScriptApplicationGraph>,
): JavaScriptApplicationAnalysisResult => {
const { graph_id: _graphId, ...semantic } = analysis.semantic_graph;
return javascriptApplicationAnalysisResultSchema.parse({
...analysis,
graph,
semantic_graph: createJavaScriptSemanticGraph({
...semantic,
application_graph_id: graph.graph_id,
}),
});
};
/** Wrap JavaScript application analysis as producer Evidence. */
export const analysisViewJavaScriptEvidence = (
analysis: JavaScriptApplicationAnalysisResult = analysisViewJavaScriptAnalysis(),
): Evidence =>
createEvidence(
{
path: analysis.input_path,
format: analysis.format,
sha256: analysis.root_artifact_sha256,
},
{
id: "rea-javascript-application",
name: "REA JavaScript application analyzer",
version: "1",
},
{
predicateType: "rea.javascript-application-analysis",
operation: "analyze_javascript_application",
parameters: { input_path: analysis.input_path, format: analysis.format },
result: analysis,
confidence: "derived",
authority: "shipped-artifact",
limitations: analysis.limitations,
},
);
/** Capture parent with explicit producer ordinals, including excluded URL values. */
export const analysisViewCaptureFixture = () =>
webNetworkCaptureSchema.parse({
artifact: {
path: "/fixtures/surface.har",
sha256: "c".repeat(64),
bytes: 1024,
},
format: "har",
decoder: { id: "fixture", name: "fixture", version: "1" },
container: {
reported: {},
numeric_literals: [],
redactions: [],
records_pointer: "/log/entries",
},
total_records: 15,
records: [
...apiSurfaceHar().log.entries,
{ request: { method: "GET", url: null } },
{ request: { method: "GET", url: "invalid URL" } },
].map((reported, ordinal) => ({
ordinal,
location: { kind: "json-pointer", pointer: `/log/entries/${ordinal}` },
reported,
binary_fields: [],
numeric_literals: [],
redactions: [],
limitations: [],
})),
runtime_attribution: "unknown",
limitations: [],
});
export const analysisViewCaptureEvidence = (
capture = analysisViewCaptureFixture(),
): Evidence =>
createEvidence(
capture.artifact.sha256 === null
? undefined
: {
path: capture.artifact.path,
sha256: capture.artifact.sha256,
format: "file",
},
{ id: "fixture", name: "fixture", version: "1" },
{
operation: "inspect_web_network_capture",
parameters: { sensitive_value_count: 0 },
result: capture,
confidence: "observed",
},
);
/** A small encoded capture whose many short objects expand during validation. */
export const analysisViewObjectDenseCapture = (headerCount = 100_000) => {
const capture = analysisViewCaptureFixture();
capture.records = capture.records.slice(0, 1);
capture.total_records = 1;
capture.records[0]!.reported = {
request: {
method: "GET",
url: "https://api.test/items/1",
headers: Array.from({ length: headerCount }, () => ({
name: "x",
value: "y",
})),
},
};
return capture;
};
/** Capture parent decoded from the Grafana-shaped HAR with producer ordinals intact. */
export const environmentManifestCaptureFixture = () => {
const capture = analysisViewCaptureFixture();
const entries = environmentManifestHar().log.entries;
capture.artifact.path = "/fixtures/environment-manifest.har";
capture.total_records = entries.length;
capture.records = entries.map((reported, ordinal) => ({
...capture.records[0]!,
ordinal,
location: { kind: "json-pointer", pointer: `/log/entries/${ordinal}` },
reported,
}));
return capture;
};