mirror of
https://github.com/safishamsi/graphify.git
synced 2026-09-24 22:45:43 +00:00
With `graphify extract --out <dir>`, the semantic cache write and read sides disagreed on both location and key anchoring, breaking the cache round-trip in two ways: - Checkpoints (#1990): `_checkpoint_chunk` called `save_semantic_cache` with only `root=target`, so per-chunk recovery checkpoints were written under `<corpus>/graphify-out/` while the reader consulted `<out>/graphify-out/` — creating an unwanted graphify-out/ inside the analyzed source tree and making every interrupted run re-extract (and re-bill) completed chunks. - Final save (#1991): cli.py passed `root=out_root`, so corpus-relative `source_file` paths resolved against the --out directory, failed `p.is_file()`, and every result group was silently skipped — the cache the reader would consult was never populated at all, with no warning. Fix, following the split the AST cache already uses (#1774): - `save_semantic_cache` and `check_semantic_cache` gain a `cache_root` parameter mirroring `load_cached`/`save_cached`: `root` stays the source-key anchor (content-hash keys, source_file resolution and relativization), `cache_root` selects where cache files live. Omitting it keeps `root` for both, so existing callers are unchanged. - `extract_corpus_parallel` plumbs `cache_root` into `_checkpoint_chunk`. - cli.py extract passes `root=target, cache_root=out_root` at the cache read, the checkpoint path, and the final save, and re-anchors the prune sweep's live hashes to `target` (keys anchored to out_root would mismatch every entry and sweep the fresh cache as orphaned). - `save_semantic_cache` now warns loudly when every result group is dropped because its source_file does not resolve to a real file — the silent-0-writes failure mode #1991 asked to surface. Regression tests cover: checkpoint written under cache_root (not the corpus, no corpus graphify-out/ created), recovery read finds the checkpoint via the same root/cache_root split, the final-save call shape writes entries where the reader looks, the all-groups-dropped warning, and backward compatibility when cache_root is omitted. Fixes #1990 Fixes #1991
This commit is contained in:
@@ -2044,6 +2044,7 @@ def extract_corpus_parallel(
|
||||
max_concurrency: int = 4,
|
||||
max_retry_depth: int = 3,
|
||||
deep_mode: bool = False,
|
||||
cache_root: "Path | None" = None,
|
||||
) -> dict:
|
||||
"""Extract a corpus in chunks, merging results.
|
||||
|
||||
@@ -2079,6 +2080,14 @@ def extract_corpus_parallel(
|
||||
output_tokens. Failed chunks are logged to stderr and skipped — one bad
|
||||
chunk does not abort the run.
|
||||
|
||||
``cache_root`` (when given) is where per-chunk checkpoint cache entries are
|
||||
written, decoupled from ``root`` which anchors content-hash keys and
|
||||
``source_file`` resolution — the same split the AST cache uses (#1774).
|
||||
With ``--out``, cli.py passes the corpus as ``root`` and the output
|
||||
directory as ``cache_root`` so checkpoints land where the recovery read
|
||||
looks, instead of creating an unwanted ``graphify-out/`` inside the
|
||||
analyzed source tree (#1990).
|
||||
|
||||
Accepts ``str`` paths as well as ``Path``; string entries are coerced up
|
||||
front so packing/slicing helpers can rely on ``Path`` semantics (#1386).
|
||||
"""
|
||||
@@ -2154,6 +2163,7 @@ def extract_corpus_parallel(
|
||||
result.get("edges", []),
|
||||
result.get("hyperedges", []),
|
||||
root=root,
|
||||
cache_root=cache_root,
|
||||
merge_existing=True,
|
||||
allowed_source_files=allowed,
|
||||
mode="deep" if deep_mode else None,
|
||||
|
||||
Reference in New Issue
Block a user