Commit Graph

7 Commits

Author SHA1 Message Date
Safi 7467c1b6a4 feat/fix: land PRs #1118 #1110 #1159 #1107 #1103 (graph quality + new features)
#1118 — prune stale AST nodes on full re-extraction (#1116)
Stamps every AST-extracted node with _origin="ast" in extract(). On a
full rebuild _rebuild_code drops any AST-marked node absent from the
fresh output even when its source file survives, fixing stale symbols.
Backward-compat: marker-less nodes from pre-1118 graphs survive one
cycle then self-heal.

#1110 — stop reading images and PDFs as garbage in headless extract
Images route through per-backend vision payloads (base64/data-URI/bytes
for claude/openai/bedrock); non-vision backends get _strip_pixels for
graceful degradation. PDFs reuse pypdf. 5MB cap, 20-image chunk limit.

#1159 — Salesforce Apex extractor (.cls, .trigger)
Regex-based extractor: classes, interfaces, enums, methods, triggers,
SOQL/DML edges. No new dependency. Dispatched as .cls and .trigger.

#1107 — Azure OpenAI Service backend (--backend azure)
Uses AzureOpenAI SDK client (from existing openai package). Auto-detects
when AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT both set. Uses
max_completion_tokens (not deprecated max_tokens).

#1103 — live PostgreSQL introspection (--postgres DSN)
graphify extract --postgres "postgresql://..." introspects tables, views,
routines, and FK relations via information_schema (SERIALIZABLE READ ONLY).
Credentials sanitized on error. New graphify[postgres] extra (psycopg3).

Union-resolved llm.py conflict: Azure functions + bedrock images= param.
Fixed test_image_vision.py mock to accept timeout= kwarg (our #1112).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-07 01:20:38 +01:00
Safi 8c4c67f3ad fix Ollama num_ctx: derive from actual chunk size instead of hardcoding 131072 (#798) 2026-05-09 23:42:44 +01:00
Safi 8c00287e84 fix Ollama context saturation: set num_ctx=131072, keep_alive=30m, serial by default (#798) 2026-05-09 22:51:36 +01:00
Safi 23f598f3a0 fix MultiGraph crash, hollow LLM response retry, and skill --help (#796, #795, #792)
#796: add edge_data()/edge_datas() helpers in build.py that tolerate
MultiGraph/MultiDiGraph; replace all G.edges[u,v] 2-tuple call sites in
__main__.py, serve.py, wiki.py, export.py, analyze.py, benchmark.py;
fix same pattern in 10 skill file inline heredocs

#795: all 12 skill files now short-circuit on /graphify --help or -h
and print the Usage block without running any pipeline steps

#792 (hollow response): add _response_is_hollow() predicate in llm.py;
when Ollama (or any backend) returns empty/null/whitespace content or a
parsed result with no nodes/edges, rewrite finish_reason="length" so
_extract_with_adaptive_retry bisects the chunk instead of silently
dropping it; applied to _call_openai_compat, _call_claude, _call_bedrock

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 21:17:25 +01:00
tmaeder f88567b114 Recover from context-window-exceeded API errors in adaptive retry (#789)
Adaptive retry only recovered from `finish_reason="length"` (output
truncation). It did not handle the other shape of overflow: the API
rejecting the prompt outright with a 400 because the input plus
`max_completion_tokens` doesn't fit in the model's context window.

This shows up immediately on local OpenAI-compatible servers (LM
Studio, llama.cpp, vLLM) where the default context is small (4K-32K)
and a 60K-token chunk packed for cloud Kimi/Claude blows past it.
Without retry the whole chunk fails with no output, even though the
two halves would each fit cleanly.

Catch a heuristic set of context-overflow exception messages,
classify them as the same kind of recoverable failure as
`finish_reason="length"`, and split-recurse on the same path. Single-
file overflow returns an empty fragment so the rest of the corpus
keeps running. Unrelated errors (rate limit, auth, etc.) still
propagate.

Tested with qwen3.5-9b on LM Studio (32K ctx) against a 215-file
corpus where chunks 4-12 of 12 previously failed; with this change
the overflowing chunks self-heal by splitting in half.
2026-05-09 12:59:35 +01:00
Daniel Graham cc63a1711b Make Gemini extraction model configurable
The initial Gemini backend defaulted to 2.5 Flash, but large semantic extraction chunks can benefit from newer models and more output headroom. Move the default to Gemini 3 Flash Preview, add CLI and environment model overrides, and increase the Gemini completion budget while keeping low reasoning effort for cost control.

Constraint: Google exposes Gemini through an OpenAI-compatible chat-completions endpoint

Rejected: Hardcode Gemini 3.1 Pro as the default | higher cost for routine repository indexing

Confidence: medium

Scope-risk: narrow

Directive: Keep --model and GRAPHIFY_GEMINI_MODEL working before changing Gemini defaults again

Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q

Not-tested: Live Gemini 3 extraction on the full cloud-edge repo before this commit
2026-05-05 10:11:12 -04:00
Daniel Graham a9cb692961 Prefer accessible semantic extraction backends
Gemini is often the cheaper available quota for low-stakes semantic graph extraction, while OpenAI is a useful fallback. Extend the direct extraction backend registry, CLI validation, docs, and tests so headless extraction can use GEMINI_API_KEY, GOOGLE_API_KEY, or OPENAI_API_KEY without changing the existing Claude and Kimi paths.

Constraint: Gemini supports OpenAI-compatible chat completions at the Google generative-language endpoint

Rejected: Native google-genai integration | higher dependency and response-shape churn for the same chat-completions path

Confidence: medium

Scope-risk: moderate

Directive: Keep backend detection explicit and test every accepted API-key environment variable before adding new providers

Tested: uv run --directory vendor/graphify pytest tests/test_llm_backends.py tests/test_chunking.py -q

Not-tested: Live Gemini/OpenAI API calls; no GEMINI_API_KEY or OPENAI_API_KEY present in this environment
2026-05-05 08:59:37 -04:00