The --update runbook's Step 9 stamped the entire detected corpus into the
manifest, so a semantic file (doc/paper/image) whose chunk failed or was omitted
was recorded as done and never re-queued on the next update — losing its content
permanently. The runbook now builds the manifest the way the library extract
path does: cli._stamped_manifest_files stamps only files that actually produced
nodes/edges/hyperedges, dispatched-but-empty files have their stale hash cleared,
and scan_corpus drops newly-excluded in-root rows.
Applied to the Claude, Aider, and Devin skill bodies and the shared update
reference; all 134 artifacts regenerated. gen.py gains a sanctioned monolith-
diff predicate for the new stamping lines.
build_merge now prunes a deleted file's nodes, edges, and hyperedges regardless
of whether their stored source_file is absolute or relative. When the caller
passed no root (the --update runbook), a node that kept an absolute path slipped
past the relative prune set and the deleted file's graph survived silently.
Matching is now form-insensitive (raw, normalized-relative, then an absolute-
identity fallback); a re-extracted file is still never pruned (#1796 preserved).
`graphify extract` also writes the .graphify_root marker after every graph write
so a later build_merge relativizes deleted-file paths correctly even under a
custom --out (its grandparent-of-graph.json fallback pointed at the wrong dir).
Regression tests in tests/test_build_merge_hyperedges_and_prune.py.
Three silent-data-loss fixes in the update/reconcile path:
- #2051: a full `graphify update` now evicts semantic nodes whose non-AST
source (a .txt/.pdf/.png with no code extractor) was deleted from disk.
The corpus sweep only checked re-extractable files, so deleted docs'/images'
LLM nodes survived as authoritative forever. Disk absence is now the deletion
signal; remote/virtual sources (`://`) are left untouched.
- #2056: an incremental rebuild whose change set names a present-but-
unextractable file no longer treats it as a deletion (which evicted its
semantic nodes and disabled the shrink guard). The guard now falls through to
per-source accounting instead of a wholesale bypass on any deletion.
- #2014: code-typed nodes the semantic pass surfaces from within a document now
count as that doc's semantic layer, so a rebuild doesn't re-scan and drop them.
Regression tests for each in tests/test_watch.py.
The barrel-chain resolver keyed (source, symbol) -> target as last-write-wins, so
a barrel re-exporting the same local name from two modules collapsed an importer's
edge onto whichever was learned last — a fabricated wrong edge. Learn into a set
per key and refuse to resolve when a name maps to more than one target; the edge
falls to the dangling-canonical fallback (dropped at build) instead. Adds a
regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PR resolved OLLAMA_HOST for the client base_url but not in detect_backend(),
so the headline #1940 case (OLLAMA_HOST set, no --backend) still errored with
"no LLM API key found". detect_backend() now uses _resolve_ollama_base_url (empty
default stays falsy, so ollama remains opt-in and never shadows a paid key). Also
default the port to 11434 when OLLAMA_HOST omits it (a bare host would otherwise
resolve to port 80), and handle bare-port / :port forms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ollama's own server has no OLLAMA_BASE_URL concept — it reads
OLLAMA_HOST (https://docs.ollama.com/faq#how-do-i-configure-ollama-server).
graphify only ever read OLLAMA_BASE_URL, so anyone who configured Ollama
the way its own docs describe (OLLAMA_HOST) had graphify silently ignore
it and fall back to the localhost:11434 default.
Add _resolve_ollama_base_url(default): OLLAMA_BASE_URL still wins when
set (unchanged behavior, graphify's existing convention for OpenAI-
compatible base_url overrides across backends). Otherwise falls back to
OLLAMA_HOST, normalized into an OpenAI-compatible URL (adds a scheme if
missing, ensures a trailing /v1). Wired into the BACKENDS dict's ollama
entry and the two ollama_url call sites that read the env var directly.
Fixes#1940.
Publishes graphifyy to PyPI via GitHub OIDC on a published Release — no API
token. Builds sdist+wheel with uv, guards that the package version matches the
release tag, twine-checks, then uploads via pypa/gh-action-pypi-publish with
id-token: write and the `pypi` environment. Requires a matching GitHub trusted
publisher configured on PyPI (Owner Graphify-Labs, repo graphify, workflow
publish.yml, environment pypi).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Homepage/Repository/Issues were stale from the org move (safishamsi/graphify).
Aligning them to Graphify-Labs/graphify fixes the PyPI "Issues" link and is a
prerequisite for PyPI to mark the Repository link Verified once releases publish
via a GitHub trusted publisher.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With `graphify extract --out <dir>`, the semantic cache write and read
sides disagreed on both location and key anchoring, breaking the cache
round-trip in two ways:
- Checkpoints (#1990): `_checkpoint_chunk` called `save_semantic_cache`
with only `root=target`, so per-chunk recovery checkpoints were written
under `<corpus>/graphify-out/` while the reader consulted
`<out>/graphify-out/` — creating an unwanted graphify-out/ inside the
analyzed source tree and making every interrupted run re-extract (and
re-bill) completed chunks.
- Final save (#1991): cli.py passed `root=out_root`, so corpus-relative
`source_file` paths resolved against the --out directory, failed
`p.is_file()`, and every result group was silently skipped — the cache
the reader would consult was never populated at all, with no warning.
Fix, following the split the AST cache already uses (#1774):
- `save_semantic_cache` and `check_semantic_cache` gain a `cache_root`
parameter mirroring `load_cached`/`save_cached`: `root` stays the
source-key anchor (content-hash keys, source_file resolution and
relativization), `cache_root` selects where cache files live. Omitting
it keeps `root` for both, so existing callers are unchanged.
- `extract_corpus_parallel` plumbs `cache_root` into `_checkpoint_chunk`.
- cli.py extract passes `root=target, cache_root=out_root` at the cache
read, the checkpoint path, and the final save, and re-anchors the
prune sweep's live hashes to `target` (keys anchored to out_root would
mismatch every entry and sweep the fresh cache as orphaned).
- `save_semantic_cache` now warns loudly when every result group is
dropped because its source_file does not resolve to a real file — the
silent-0-writes failure mode #1991 asked to surface.
Regression tests cover: checkpoint written under cache_root (not the
corpus, no corpus graphify-out/ created), recovery read finds the
checkpoint via the same root/cache_root split, the final-save call shape
writes entries where the reader looks, the all-groups-dropped warning,
and backward compatibility when cache_root is omitted.
Fixes#1990Fixes#1991
ee1df22 narrowed the Claude Code search-guard matcher from "Glob|Grep" to
"Bash" on the premise that dedicated search tools were removed and searches
go through Bash. Current Claude Code routes content search through its
first-class Grep tool (its Bash tool description actively steers away from
shell grep), so the graphify-first nudge never fired on the agent's primary
exploration path and the graph was silently bypassed.
Three-part fix, per the issue's analysis:
- Matcher: "Bash" -> "Bash|Grep" in _claude_pretooluse_hooks. Glob already
fires the read nudge via "Read|Glob", so Grep was the only orphaned tool.
- Guard body: the hook-guard search branch only inspected tool_input.command,
which a Grep call doesn't carry (it has pattern/path/glob). A Grep-shaped
input (pattern present, no command) is now treated as a search — it IS one
by definition — and nudges whenever a fresh graph exists. The Bash
token-matching path is unchanged, and a command-carrying input never
triggers the Grep shape, so non-search Bash calls stay silent.
- Idempotency: "Bash|Grep" added to the four install/uninstall dedup filters
(claude + codebuddy), so upgrading replaces the stale "Bash" hook in place
instead of appending a duplicate — verified against a pre-fix settings.json.
Tests: new regression tests feed Grep-shaped tool_input through
hook-guard search and assert the nudge (with graph), silence (without),
valid PreToolUse JSON, and no blocking; plus a guard that a non-search Bash
command with a stray pattern key does not nudge. Existing matcher assertions
updated across test_search_hook/test_install/test_claude_md/test_codebuddy/
test_hook_strict. Hook+install suites: 397 passed. Full suite: 3224 passed;
the 13 failures are pre-existing on clean v8 in this environment.
Fixes#1986
Claude Code runs command-type PreToolUse hooks through Git Bash by default on
Windows. The resolved exe was emitted as a raw backslash path and quoted only
when it contained a space, so a space-free path like C:\Users\me\graphify.EXE
reached settings.json unquoted. Git Bash treats the unquoted backslashes as
escapes and strips them, producing 'C:Usersmegraphify.EXE: command not found',
so every graph guard silently fails. Normalize the path to forward slashes in
_resolve_graphify_exe (a no-op on POSIX), fixing the Claude, Gemini, and Codex
hook emitters at once.
Rename the "Penpax" section to "graphify Enterprise" (linking graphify.com) and
add a Website badge to the Community and links footer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stamp the 0.9.19 release date, add a README note for the strict PreToolUse hook
(graphify install --project --strict), and credit both the #1814 reporter
(@Greg-Moskalenko) and the fix author (@alphanury).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PR always wrote gitignore=not no_gitignore into .graphify_build.json, so a
flag-less `graphify extract` after `--no-gitignore` reset it to True and the
git-ignored code silently disappeared again — the exact #1971 complaint. Write
False only when the flag is set (None = leave as-is, mirroring #1886 excludes),
and honor the persisted value for the run when the flag is absent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify prs` reads gh, git, and claude output via subprocess.run(text=True)
with no explicit encoding=, so on Windows stdout is decoded with the cp1252
locale codec. gh emits UTF-8 JSON whose PR titles and logins routinely carry
non-Latin1 bytes (emoji, or the Persian title in PR #1281), and cp1252 has no
mapping for bytes such as 0x81. The capture reader thread raises
UnicodeDecodeError, subprocess.run then returns stdout=None, and the caller dies
one line later on json.loads(None) (TypeError) or None.splitlines()
(AttributeError). Neither is caught by _gh's except clause, so the whole `prs`
command aborts, with a spurious reader-thread traceback on stderr, against any
repo that has non-ASCII PR metadata.
Pass encoding="utf-8", errors="replace" to all five subprocess reads in prs.py
so decoding matches Linux and macOS. This is the decode-side sibling of #1505,
which applied the same change to the llm.py claude-cli subprocess. CI runs Linux
only (a UTF-8 locale), so this path is never exercised there.
Reproduced on Windows 11 / CPython 3.13 with a cp1252 locale:
prs._gh("pr", "view", "1281", ...) crashed with TypeError before and returns the
decoded title after. The added regression tests assert encoding="utf-8" on each
call, mirroring tests/test_charmap_encoding.py, and are red without the fix.
`extract_dm` normalized include paths with `raw.lstrip("./")`. `str.lstrip`
treats its argument as a *set of characters*, not a prefix, so it strips
every leading `.` and `/` — destroying parent-relative includes:
"../shared/base.dm" -> "shared/base.dm"
"../../a.dm" -> "a.dm"
".hidden/x.dm" -> "hidden/x.dm"
`(path.parent / norm).resolve()` then points at the wrong directory,
`.exists()` returns False, and a correct internal `imports_from` edge to the
real included file becomes a bogus external `imports` edge to a phantom node,
silently losing the cross-file link.
The sibling local-include resolver in `extractors/bash.py` resolves
`(path.parent / raw)` from the raw path without stripping `..`, confirming
the intent. Strip only a literal leading `./` (re is already imported) so
`../` includes resolve correctly; the `./`-prefix and no-prefix cases are
unchanged.
The digest salts content with the path relative to root (portability, #1774),
but the stat-index memo was keyed by absolute path only — so the same file
hashed under two roots (which happens within one `--out` run) served whichever
digest was computed first, making file_hash order-dependent and poisoning the
persisted stat-index across runs. Store one digest per salt under a "hashes"
map; legacy un-salted "hash" entries are never trusted (recompute once). Digest
computation is byte-identical, so existing cache entries still hit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents routinely ignore the advisory "run graphify query first" nudge and read
raw files anyway. `graphify install --project --strict` (or `graphify claude
install --strict`) now installs a hook that BLOCKS the first raw source read of a
session via permissionDecision:"deny" with a redirect to graphify query, then
downgrades to the soft nudge — it fires at most once per session (atomic
per-session marker) so it can never strand the agent, and a recent
query/explain/path refreshes a stamp that suppresses it. Claude Code only;
Bash-grep and Glob stay nudge-only; Gemini/Codex/OpenCode are unchanged.
GRAPHIFY_HOOK_STRICT=1/0 toggles at runtime without a reinstall; default installs
are byte-identical (soft nudge).
Also fixes#1840 for the default soft nudge: the guard no longer fires for reads
of out-of-project files, and softens to a non-mandatory nudge when the graph is
stale for the target file. Gating is ~3 stat calls (no corpus walk) and fails
open. Begins the 0.9.19 cycle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stamp the 0.9.18 release date and add a README troubleshooting entry for the
new #1951 behavior: an incomplete extraction refuses to overwrite a larger
existing graph, with --allow-partial as the escape hatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`graphify extract <root> --out <dir>` reduced every node's `source_file` to a
bare filename, so a graph.json could no longer be resolved back to files on
disk by joining source_file onto the scan root.
--out passes the output dir as cache_root to relocate the cache, but that value
also anchored relativization. Every scanned file then failed relative_to(root),
fell through to the #1899 out-of-root fallback, tripped its `updepth > 3`
walk-up guard -- written for a stray ProjectReference, not a whole corpus -- and
collapsed to a basename. On Windows an --out on another drive hit the
cross-drive branch and basenamed unconditionally, which is what the reporter
saw: 0 of ~120k source_files kept a separator. The directory survived only in
the node id slug, lossily (`.`, `/`, `\`, `-`, spaces all map to `_`), and no
other field carried it -- origin_file is stripped (#1516) and the export has no
file table -- so 0% of nodes resolved.
extract() now takes an explicit `root` anchor for source_file/ids/symbol
resolution, which the CLI pins to the scan root independent of where the cache
lives. This completes the cache/anchor decoupling #1774 started and matches
build(root=target), which already anchored on the scan root -- extract was the
lone component keying off --out.
cache_root keeps its fallback-anchor role, so callers that pass the scan root as
cache_root (watch, the no---out CLI path, tests) are unchanged, as are cache
location and out-of-root portability (#1899).
Follow-up to #1956: a doc stamped complete on a prior run that truncates
(partial) this run must have its stale semantic_hash cleared so it is re-queued.
Partial files are dropped by _stamped_manifest_files and therefore land in
clear_semantic — pin that composed behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #1943 carve-out rescues genuine source under secrets/ / credentials/, but
.tfvars sits in CODE_EXTENSIONS while being Terraform's canonical values store
(routinely real secrets), so it would now be indexed. Add .tfvars to
_SECRET_PRONE_DATA_EXTS so the shared graphable-source predicate drops it in
both Stage 1 and Stage 3; .tf/.hcl stay graphable as genuine infra source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An explicit "weight": null in the extraction JSON survived .get("weight", 1.0)
(the key is present, so the default never applied) and reached Louvain/Leiden as
None, crashing modularity with a TypeError (graspologic's Leiden even panics on
NaN). build_from_json now coerces weight and confidence_score to float at the
ingest choke point, falling back to 1.0 for null / non-numeric / NaN / inf /
negative values while preserving valid ones. Repairs (not drops) the key so
graph.json round-trips clean and a cluster-only/--update reload never re-ingests
the null.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PR flagged unverifiable code nodes as confidence="UNVERIFIED", but node
confidence is read by nothing and UNVERIFIED isn't in the validated (edge-only)
confidence vocabulary {EXTRACTED,INFERRED,AMBIGUOUS} — a dead, colliding field.
- Move the flag to a dedicated verification="unverified" node field, off the
confidence key. A node the model already hedged (INFERRED/AMBIGUOUS) is left
untouched, as before.
- Also check the node id's identifiers (not just the label): ids carry the
verbatim symbol, cutting false flags on prettified labels.
- Skip the (potentially PDF-re-extracting) source read entirely when a result
has no code-typed node with a source_file.
- Wire a consumer: diagnose_extraction now counts and reports unverified nodes,
so the persisted flag is actually surfaced.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The semantic (LLM) extraction runs on documents/papers/images; code files are
handled by the deterministic AST engine and never reach the model. A node the
model tags file_type="code" is therefore a symbol it surfaced from within a
document (a name in a fenced code block, an API referenced in a paper), and it
enters graph.json today with no check that the symbol actually appears in the
source the model read. `_out_of_scope` (#1895) only rejects nodes attributed to a
file that was NOT dispatched, so a fabricated symbol on a dispatched file passes.
`extract_files_direct` now verifies every file_type=="code" node whose
source_file was dispatched in the call: if no identifier from its label occurs
(case-insensitive substring) in that file's source bytes, the node's confidence
is downgraded to "UNVERIFIED" (never dropped) and the count is reported to
stderr. Concept/document nodes, nodes without a source_file, nodes attributed to
undispatched files, and labels with no checkable identifier are left untouched.
Best-effort; never aborts extraction; one unreadable file is skipped, not fatal.
The _partial item-marker approach couldn't fire on the most common truncation
shape: a mid-JSON cut parses to zero items, so a sliced document whose second
slice truncated empty was still stamped complete (the PR's own headline case).
- The adaptive-retry give-up sites now record the chunk's own source files in a
result-level _partial_files list, independent of parsed items; it propagates
through _merge_two / the recursion merges / _merge_into so it reaches both the
per-chunk checkpoint and the run-level manifest stamp.
- _partial_source_files unions _partial_files with the item markers.
- save_semantic_cache seeds an empty group for a named partial file with no
items so its entry is stamped partial, and carries a partial prev entry's flag
forward so a later clean slice merging over it can't re-promote it to complete.
- the CLI final save now passes partial_source_files (computed before the save)
so an empty-parse file isn't written back as a complete entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A chunk whose LLM response is truncated (`finish_reason="length"`) and can't be
recovered by splitting, or that hits the adaptive-retry depth cap, returns a
partial node set. Today that set is checkpointed and written to the content-hash
semantic cache + manifest-stamped as complete, so the incomplete nodes are
served forever until the file content changes or `--force`.
Truncated give-up results are now tagged with an internal `_partial` marker.
`save_semantic_cache` stamps the affected file's entry `partial: True` (detected
from the marker or an explicit `partial_source_files` arg), and `load_cached`
treats a partial entry as a cache MISS, so the file re-dispatches and retries.
The file is also left unstamped in the manifest (like a failed chunk, #933) so
detect_incremental re-queues it on the next incremental run — not only on a full
/ `--force` / content-change run. A file sliced across chunks accumulates via a
partial-aware `merge_existing` peek (`load_cached(allow_partial=True)`) so a
truncated slice is never dropped or silently promoted to complete. Self-heals: a
later complete extraction overwrites the same key with a non-partial entry. The
marker is stripped after the final save so it never leaks into graph.json.
The atomic-write PR left several writers on the old truncate-then-write path.
Route them through write_json_atomic so a crash mid-write can't corrupt them:
- the --no-cluster raw graph.json dump (a core graph.json writer)
- merge-graphs / merge-chunks / merge-semantic output
- .graphify_analysis.json and .graphify_labels.json sidecars
- global_graph.py's global-graph.json and global-manifest.json
write_json_atomic gains an ensure_ascii flag so the raw-UTF-8 writers
(labels, merge outputs) keep byte-for-byte output. Adds tests for the
Windows PermissionError copy fallback and ensure_ascii=False.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
graph.json (the clustered `to_json` write and the `--no-cluster`/merge raw
dumps) and manifest.json were written with a direct `open()`/`write_text`, so a
crash, kill, or disk-full mid-write left a truncated, unparseable file that the
next load or `detect_incremental` then failed on.
Add `write_text_atomic`/`write_json_atomic` in graphify.paths (temp file in the
same directory + `os.replace`; JSON is streamed into the temp, not materialized
as one string) and route the graph.json writers (export.to_json,
cli._prune_graph_json_sources, the merge driver) plus detect.save_manifest
through them. The helper preserves the destination's mode (an atomic replace
never tightens 0644 to mkstemp's 0600), writes through a symlinked destination
(shared-output setups), and falls back to copy-then-delete on a Windows
os.replace lock — matching graphify.cache's existing atomic writer. On failure
the previous file is left intact and the temp removed. Not a power-loss
durability guarantee (no fsync, consistent with the rest of the codebase).
Two gaps the review found in the incomplete-build shrink guard:
- existing_graph_node_count() returned None ("proceed") on a present-but-
unparseable graph.json, so the --no-cluster path could clobber a complete
graph whose file was corrupt/mid-write. It now returns a MALFORMED_GRAPH
sentinel and the caller fails closed, matching to_json's #479 handling.
- A walk that couldn't fully enumerate the corpus (permission-denied subtree,
I/O error) is now treated as an incomplete extraction: detect()/
detect_incremental() already record walk_errors; the extract path consumes
them so a walk-truncated graph can't force-overwrite a complete one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
safishamsi