* feat(environment): render more read tools from the state and describe responses
Keep main's environment serving and fold in coverage for non-entity reads,
response summaries, and MCP tool annotations.
Squashed rebase of #5352 onto main.
* fix(environment): keep rendered documents and quoted samples inside their budgets
A recorded body of exponential numbers can render past 256 KiB and turn a
working read into a state size error, so that document stays a recorded replay.
Description samples are measured after JSON quoting, and the replay-fallback
catalog check uses a tool count the longer response descriptions still separate.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* feat(environment): apply evidence-gated write effects to the served state
Serve write tools whose later reads confirm the effect, and keep main's
state-file ownership and part-wise reply measurement.
Squashed rebase of #5600 onto the rebased #5352 tip.
* fix(environment): bound and validate write state history
* feat(environment): render more read tools from the state and describe responses
Keep main's environment serving and fold in coverage for non-entity reads,
response summaries, and MCP tool annotations.
Squashed rebase of #5352 onto main.
* fix(environment): keep rendered documents and quoted samples inside their budgets
A recorded body of exponential numbers can render past 256 KiB and turn a
working read into a state size error, so that document stays a recorded replay.
Description samples are measured after JSON quoting, and the replay-fallback
catalog check uses a tool count the longer response descriptions still separate.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* feat(environment): apply evidence-gated write effects to the served state
Serve write tools whose later reads confirm the effect, and keep main's
state-file ownership and part-wise reply measurement.
Squashed rebase of #5600 onto the rebased #5352 tip.
* fix(environment): bound and validate write state history
* fix(environment): bound additional collection history
* fix(environment): credit sent fields, negative sequences, and encoded targets
A later overwrite copied surviving derived fields into the sent-member
claim, so a members-only check rejected an update whose sent field still
matched. Serving floored sequences at zero and stored missing history as
zero, so a collection sequence learned from negative ranks skipped the
next value. An absence after a delete forgot the resolved collection, and
an encoded read could not observe an update spelled with another
collection encoding.
Regression tests cover each case.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* test(environment): cover resolved encoded update identities
* fix: validate timed update observations
* fix(environment): share write overlap and check clocks on earlier calls
Observation search skips a read begun before the write ended. That
comparison now uses the same overlap as the check path.
A clock learned on a later call was not checked against earlier calls
that kept the field. Serving would stamp those calls too, so the plan
stays unserved. A clock learned earlier is still refused when a later
call does not advance it.
Regression tests cover an overlapping empty update and a published_at
clock that one recorded call leaves unchanged.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix: preserve partial update evidence
* fix(environment): let an unmodelled write explain only the list members that differ
A failed whole-list read now reports which members alone account for the
difference. An unmodelled write excuses the read's writes only when it came
after each such member's last check and pending writes, so an older change
elsewhere no longer hides a newer write the list shows undone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 7b6aab9d7191e600051693f2c192afcfc743d083)
* fix(environment): explain added fields, tie creates to serving's collection, and bound write descriptions
An update explains a field the entity lacked before when the observing
view had shown the entity, so a server-filled field is not dropped. A
create is tied to the held collection serving picks for its path. Write
descriptions and body refusals quote member names and sample them within
a byte bound. The request body is indexed once per explanation, so copied
and slugged fields no longer rescan it.
Replays 07aca54e onto the restacked branch. Its rule that a read begun
before a write ended teaches nothing is already here (fb6785dc), so only
its test is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(environment): state how list reads weigh unmodelled writes, added update fields and quoted members
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(environment): keep the observing view helpers with the observations
shownBefore and the item-or-list fields check move to the observe module,
keeping the write effects within the line limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(environment): resolve unmodelled writes onto the held collection
An unmodelled path kept the request's collection spelling. An archive under
the plain spelling of a collection held encoded marked only that plain
collection unknown, so a later update was blamed for the archive's fields
and dropped. Name that entity the way a modelled write does.
Co-authored-by: morluto <morluto@users.noreply.github.com>
* fix(environment): confirm no write through volatile fields or overwritten replacements, and bound the served clock
A members-only check no longer compares volatile fields, which render
from the read itself. A document replacement credits the members it
wrote or removed at its object, so a later merge that overwrites them
takes its confirmation away. A recorded time past the range of a date
gives no served clock instead of throwing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(environment): leave unknown entities unchecked by failed reads, volatile fields and unchecked calls
A failed item read that is no absence leaves an unknown entity unknown, so
a later usable read still checks its writes. A read showing an unknown
entity again checks no write on a volatile field. An update of an unknown
entity takes its recorded outcome and adds nothing to a built plan. A later
update that does not answer with its entity is observed only by a read
whose view shows every learned clock; without one it is unmodelled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(environment): fill a create's path fields from the collection serving selects
The gate built a later create's path fields from the recorded request path,
while serving fills them from the held collection it selects for that path,
so a read showing the other spelling's parent could confirm a create serving
cannot reproduce. Rules still learn from the recorded path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(environment): name unmodelled writes as modelled ones, count fields updates write, and keep later members' types
An unmodelled write below an item path names its entity through itemTarget,
and one at or above a collection resolves the held collection, as modelled
writes do, so an encoded spelling no longer hides the entity it changed.
An update's answer counts the fields its rules write, other than increments,
as available, so a field only the update writes no longer makes it
incomplete-entities. A member a later call first stores keeps that call's
JSON type in the body's kinds, which the body check falls back to.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(environment): let a document replacement overwrite everything under its object
A replacement claims only the members it wrote or removed, but it now
overwrites every earlier effect under its object, so a merge that wrote
the object it replaces loses its confirmation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(environment): start the served clock from the records whose times a date can hold
A record timed past the range of a date no longer leaves the served clock
without a start, which the store read as zero and stamped writes in 1970;
the clock starts after the latest record a date can hold.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: morluto <morluto@users.noreply.github.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
362 lines
12 KiB
TypeScript
362 lines
12 KiB
TypeScript
import type { JsonValue } from "../../src/domain/jsonValue.js";
|
|
import {
|
|
apiSurfaceHar,
|
|
environmentManifestHar,
|
|
environmentStateHar,
|
|
} from "./historicalHar.js";
|
|
import { webNetworkCaptureSchema } from "../../src/domain/webNetworkCapture.js";
|
|
import { BINARY_LAYOUT_TEST_PROVIDER } from "./binaryDiagnostics/layout.js";
|
|
import { binaryLayoutFixture } from "./binaryDiagnostics/layout.js";
|
|
import { JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE } from "../../src/contracts/javascript/javascriptRuntimeReconciliationExample.js";
|
|
import { createEvidence, type Evidence } from "../../src/domain/evidence.js";
|
|
import type { JavaScriptApplicationAnalysisResult } from "../../src/domain/javascript/javascriptApplicationAnalysis.js";
|
|
import { javascriptApplicationAnalysisResultSchema } from "../../src/domain/javascript/javascriptApplicationAnalysis.js";
|
|
import {
|
|
createJavaScriptApplicationGraph,
|
|
createJavaScriptApplicationNode,
|
|
} from "../../src/domain/javascript/javascriptApplicationGraph.js";
|
|
import { createJavaScriptSemanticGraph } from "../../src/domain/javascript/javascriptSemanticGraph.js";
|
|
import type { BinaryLayout } from "../../src/domain/native/binaryLayout.js";
|
|
import { binaryLayoutSchema } from "../../src/domain/native/binaryLayout.js";
|
|
import { ghidraFunctionDossier } from "../../src/domain/ghidraValues.fixture.js";
|
|
import { functionDossierSchema } from "../../src/domain/hopperValues.js";
|
|
import type { RelationshipCoverage } from "../../src/domain/relationshipCoverage.js";
|
|
|
|
/** Canonical portable Evidence; no native provider is started by this fixture. */
|
|
export const analysisViewNativeEvidence = (
|
|
relationshipCoverage?: RelationshipCoverage,
|
|
): Evidence =>
|
|
createEvidence(
|
|
{ path: "/fixtures/native.exe", format: "pe", sha256: "a".repeat(64) },
|
|
{ id: "fixture", name: "Fixture", version: "1" },
|
|
{
|
|
operation: "analyze_function",
|
|
parameters: { address: "0x401000" },
|
|
result: {
|
|
...functionDossierSchema.parse(ghidraFunctionDossier()),
|
|
limitations: ["Fixture dossier caveat"],
|
|
},
|
|
...(relationshipCoverage === undefined ? {} : { relationshipCoverage }),
|
|
},
|
|
);
|
|
|
|
const hex = (value: number): string =>
|
|
value === 0 ? "0x0" : `0x${value.toString(16)}`;
|
|
|
|
const layoutName = (display: string, offset: number) => ({
|
|
display,
|
|
bytes_base64: Buffer.from(display, "utf8").toString("base64"),
|
|
location: { offset: hex(offset), bytes: hex(display.length + 1) },
|
|
unknown_reason: null,
|
|
});
|
|
|
|
const layoutSection = (
|
|
index: number,
|
|
display: string,
|
|
nameOffset: number,
|
|
): BinaryLayout["sections"][number] => ({
|
|
index,
|
|
name: layoutName(display, nameOffset),
|
|
name_offset: hex(nameOffset),
|
|
type: 1,
|
|
header_location: { offset: hex(index * 16), bytes: "0x10" },
|
|
address: "0x0",
|
|
offset: hex(64 + index * 16),
|
|
size: "0x10",
|
|
alignment: "0x1",
|
|
flags: "0x6",
|
|
link: 0,
|
|
info: 0,
|
|
entry_size: "0x0",
|
|
file_backing: "file",
|
|
});
|
|
|
|
const layoutSymbol = (
|
|
entryIndex: number,
|
|
display: string,
|
|
nameOffset: number,
|
|
): BinaryLayout["symbols"][number] => ({
|
|
table_index: 0,
|
|
entry_index: entryIndex,
|
|
name: layoutName(display, nameOffset),
|
|
name_offset: hex(nameOffset),
|
|
location: { offset: hex(nameOffset), bytes: hex(display.length + 1) },
|
|
value: "0x0",
|
|
value_meaning: "no-address",
|
|
size: "0x0",
|
|
binding: 1,
|
|
type: 2,
|
|
visibility: 0,
|
|
section_index: 1,
|
|
});
|
|
|
|
/** Layout fixture with duplicate section/symbol names for selected-view tests. */
|
|
export const analysisViewLayoutFixture = (): BinaryLayout =>
|
|
binaryLayoutSchema.parse({
|
|
...binaryLayoutFixture("/artifacts/view-source.elf"),
|
|
artifact: {
|
|
path: "/artifacts/view-source.elf",
|
|
sha256: "b".repeat(64),
|
|
bytes: 512,
|
|
},
|
|
sections: [
|
|
layoutSection(0, ".text", 256),
|
|
layoutSection(1, ".text", 262),
|
|
layoutSection(2, ".data", 268),
|
|
],
|
|
symbols: [
|
|
layoutSymbol(0, "main", 280),
|
|
layoutSymbol(1, "main", 285),
|
|
layoutSymbol(2, "start", 290),
|
|
],
|
|
});
|
|
|
|
/** Wrap a layout observation as inspect_binary_layout Evidence. */
|
|
export const analysisViewLayoutEvidence = (
|
|
layout: BinaryLayout = analysisViewLayoutFixture(),
|
|
): Evidence =>
|
|
createEvidence(
|
|
{
|
|
path: layout.artifact.path,
|
|
format: "elf",
|
|
architecture: "x86_64",
|
|
sha256: layout.artifact.sha256,
|
|
},
|
|
BINARY_LAYOUT_TEST_PROVIDER,
|
|
{
|
|
operation: "inspect_binary_layout",
|
|
parameters: { path: layout.artifact.path },
|
|
result: layout,
|
|
rawResult: null,
|
|
confidence: "observed",
|
|
limitations: layout.limitations,
|
|
locations: [{ kind: "artifact-path", path: layout.artifact.path }],
|
|
},
|
|
);
|
|
|
|
/** Minimal authenticated JavaScript application analysis used by view tests. */
|
|
export const analysisViewJavaScriptAnalysis =
|
|
(): JavaScriptApplicationAnalysisResult =>
|
|
javascriptApplicationAnalysisResultSchema.parse(
|
|
JAVASCRIPT_APPLICATION_EVIDENCE_EXAMPLE.normalized_result,
|
|
);
|
|
|
|
/** Application analysis whose module observations include source text. */
|
|
export const analysisViewJavaScriptAnalysisWithSource =
|
|
(): JavaScriptApplicationAnalysisResult => {
|
|
const analysis = analysisViewJavaScriptAnalysis();
|
|
const original = analysis.graph.nodes[0];
|
|
if (original === undefined)
|
|
throw new Error("JavaScript application example is missing its module");
|
|
const withSource = createJavaScriptApplicationNode({
|
|
kind: original.kind,
|
|
identity: original.identity,
|
|
observations: original.observations.map((observation) => ({
|
|
label: observation.label,
|
|
properties: {
|
|
...observation.properties,
|
|
source: "export const secret = 1;\n",
|
|
text: "should not leak",
|
|
},
|
|
evidence: observation.evidence,
|
|
})),
|
|
});
|
|
const graph = createJavaScriptApplicationGraph({
|
|
schema: "JavaScriptApplicationGraph",
|
|
root_node_ids: [withSource.node_id],
|
|
nodes: [withSource],
|
|
edges: [],
|
|
coverage: analysis.graph.coverage,
|
|
limitations: analysis.graph.limitations,
|
|
});
|
|
const { graph_id: _graphId, ...semantic } = analysis.semantic_graph;
|
|
return javascriptApplicationAnalysisResultSchema.parse({
|
|
...analysis,
|
|
graph,
|
|
semantic_graph: createJavaScriptSemanticGraph({
|
|
...semantic,
|
|
application_graph_id: graph.graph_id,
|
|
}),
|
|
});
|
|
};
|
|
|
|
/** Rebind a companion semantic graph after replacing the application graph. */
|
|
export const analysisViewBindJavaScriptGraphs = (
|
|
analysis: JavaScriptApplicationAnalysisResult,
|
|
graph: ReturnType<typeof createJavaScriptApplicationGraph>,
|
|
): JavaScriptApplicationAnalysisResult => {
|
|
const { graph_id: _graphId, ...semantic } = analysis.semantic_graph;
|
|
return javascriptApplicationAnalysisResultSchema.parse({
|
|
...analysis,
|
|
graph,
|
|
semantic_graph: createJavaScriptSemanticGraph({
|
|
...semantic,
|
|
application_graph_id: graph.graph_id,
|
|
}),
|
|
});
|
|
};
|
|
|
|
/** Wrap JavaScript application analysis as producer Evidence. */
|
|
export const analysisViewJavaScriptEvidence = (
|
|
analysis: JavaScriptApplicationAnalysisResult = analysisViewJavaScriptAnalysis(),
|
|
): Evidence =>
|
|
createEvidence(
|
|
{
|
|
path: analysis.input_path,
|
|
format: analysis.format,
|
|
sha256: analysis.root_artifact_sha256,
|
|
},
|
|
{
|
|
id: "rea-javascript-application",
|
|
name: "REA JavaScript application analyzer",
|
|
version: "1",
|
|
},
|
|
{
|
|
predicateType: "rea.javascript-application-analysis",
|
|
operation: "analyze_javascript_application",
|
|
parameters: { input_path: analysis.input_path, format: analysis.format },
|
|
result: analysis,
|
|
confidence: "derived",
|
|
authority: "shipped-artifact",
|
|
limitations: analysis.limitations,
|
|
},
|
|
);
|
|
|
|
/** Capture parent with explicit producer ordinals, including excluded URL values. */
|
|
export const analysisViewCaptureFixture = () =>
|
|
webNetworkCaptureSchema.parse({
|
|
artifact: {
|
|
path: "/fixtures/surface.har",
|
|
sha256: "c".repeat(64),
|
|
bytes: 1024,
|
|
},
|
|
format: "har",
|
|
decoder: { id: "fixture", name: "fixture", version: "1" },
|
|
container: {
|
|
reported: {},
|
|
numeric_literals: [],
|
|
redactions: [],
|
|
records_pointer: "/log/entries",
|
|
},
|
|
total_records: 15,
|
|
records: [
|
|
...apiSurfaceHar().log.entries,
|
|
{ request: { method: "GET", url: null } },
|
|
{ request: { method: "GET", url: "invalid URL" } },
|
|
].map((reported, ordinal) => ({
|
|
ordinal,
|
|
location: { kind: "json-pointer", pointer: `/log/entries/${ordinal}` },
|
|
reported,
|
|
binary_fields: [],
|
|
numeric_literals: [],
|
|
redactions: [],
|
|
limitations: [],
|
|
})),
|
|
runtime_attribution: "unknown",
|
|
limitations: [],
|
|
});
|
|
|
|
export const analysisViewCaptureEvidence = (
|
|
capture = analysisViewCaptureFixture(),
|
|
): Evidence =>
|
|
createEvidence(
|
|
capture.artifact.sha256 === null
|
|
? undefined
|
|
: {
|
|
path: capture.artifact.path,
|
|
sha256: capture.artifact.sha256,
|
|
format: "file",
|
|
},
|
|
{ id: "fixture", name: "fixture", version: "1" },
|
|
{
|
|
operation: "inspect_web_network_capture",
|
|
parameters: { sensitive_value_count: 0 },
|
|
result: capture,
|
|
confidence: "observed",
|
|
},
|
|
);
|
|
|
|
/** A small encoded capture whose many short objects expand during validation. */
|
|
export const analysisViewObjectDenseCapture = (headerCount = 100_000) => {
|
|
const capture = analysisViewCaptureFixture();
|
|
capture.records = capture.records.slice(0, 1);
|
|
capture.total_records = 1;
|
|
capture.records[0]!.reported = {
|
|
request: {
|
|
method: "GET",
|
|
url: "https://api.test/items/1",
|
|
headers: Array.from({ length: headerCount }, () => ({
|
|
name: "x",
|
|
value: "y",
|
|
})),
|
|
},
|
|
};
|
|
return capture;
|
|
};
|
|
|
|
/** Capture parent decoded from the Grafana-shaped HAR with producer ordinals intact. */
|
|
export const environmentManifestCaptureFixture = (
|
|
entries: readonly JsonValue[] = environmentManifestHar().log.entries,
|
|
) => {
|
|
const capture = analysisViewCaptureFixture();
|
|
capture.artifact.path = "/fixtures/environment-manifest.har";
|
|
capture.total_records = entries.length;
|
|
capture.records = entries.map((reported, ordinal) => ({
|
|
...capture.records[0]!,
|
|
ordinal,
|
|
location: { kind: "json-pointer", pointer: `/log/entries/${ordinal}` },
|
|
reported,
|
|
}));
|
|
return capture;
|
|
};
|
|
|
|
/** The same capture with distinct start times, as the environment-state view orders it. */
|
|
export const environmentStateCaptureFixture = () =>
|
|
environmentManifestCaptureFixture(environmentStateHar().log.entries);
|
|
|
|
/** One synthetic exchange: method, path, JSON response body (a string is recorded as written), status, JSON request body and duration in milliseconds. */
|
|
export type TimedCaptureRow = readonly [
|
|
method: string,
|
|
path: string,
|
|
body?: JsonValue,
|
|
status?: number,
|
|
request?: JsonValue,
|
|
time?: number,
|
|
];
|
|
|
|
/** Capture Evidence of HAR entries one second apart on one origin, as the environment-state view orders them. */
|
|
export const timedCaptureEvidence = (
|
|
rows: readonly TimedCaptureRow[],
|
|
): Evidence =>
|
|
analysisViewCaptureEvidence(
|
|
environmentManifestCaptureFixture(
|
|
rows.map(
|
|
([method, path, body = null, status = 200, sent, time = 5], index) => ({
|
|
startedDateTime: new Date(
|
|
Date.parse("2026-10-17T18:00:00.000Z") + index * 1000,
|
|
).toISOString(),
|
|
time,
|
|
request: {
|
|
method,
|
|
url: `http://app.example${path}`,
|
|
...(sent === undefined
|
|
? {}
|
|
: {
|
|
postData: {
|
|
mimeType: "application/json",
|
|
text: JSON.stringify(sent),
|
|
},
|
|
}),
|
|
},
|
|
response: {
|
|
status,
|
|
content: {
|
|
mimeType: "application/json",
|
|
text: typeof body === "string" ? body : JSON.stringify(body),
|
|
},
|
|
},
|
|
}),
|
|
),
|
|
),
|
|
);
|