76 Commits
Author SHA1 Message Date
Sergey KozyrenkoandClaude Opus 5.5 843dfeb4bf docs: the README says what PentAGI is, who it is for, how it differs and how to install it
The overview was generic; it now answers those four questions in the wording of pentagi.com.
The README, config.md and the installation guide no longer list MCP among the server-side
settings: the open-source edition has no MCP code, and Enterprise adds MCP servers. SearXNG is
spelled as its project spells it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 03:53:25 +07:00
Dmitry Ng b611fbebcf fix: installer stack detection, tests that can fail, proxy and port docs
- installer: decide whether the stack is extracted by the provider examples
  beside the env file, not in the working directory
- tests: assert table cells in the markdown editor pipe tests; pin the
  prompter in provider failover, the permission error on a password change
  and the built-in Bedrock config against its shipped example
- docs: state what PROXY_URL covers and which ports flows publish; name the
  cgr.dev registry of the Langfuse stack
- comments: drop history and restated code, correct those that had drifted
  from the code
2026-10-04 01:18:50 +03:00
Dmitry Ng 55a063ecb3 feat(provider): add Bedrock provider configuration and update related documentation
- Introduced a new example configuration file for the Bedrock provider, ensuring compatibility with the existing stack.
- Updated Docker and docker-compose configurations to include the new Bedrock provider, enhancing the overall functionality of the application.
- Revised README and other documentation to reflect the addition of the Bedrock provider, including environment variable descriptions and usage instructions.
- Improved installer logic to handle the new provider configuration seamlessly, ensuring a smooth user experience during setup.
2026-10-03 00:15:29 +03:00
Dmitry Ng 6f3d29c3f1 feat: port major fixes 2026-09-30 01:48:35 +03:00
Dmitry Ng 6f34a010e1 feat(cloud): enhance VXControl Cloud API usage and add PentAGI version checks 2026-09-02 02:31:59 +03:00
Dmitry Ng ebb228b19a feat(graphiti): enhance configuration and integration settings
- Added new environment variables to `.env.example` and `docker-compose-graphiti.yml` for Graphiti, including `GRAPHITI_CPUS`, `GRAPHITI_MEMORY`, and various ingestion and logging settings.
- Updated the README to clarify the optional nature of the Graphiti integration and its configuration requirements.
- Enhanced the backend installer to support new Graphiti configuration options, ensuring proper handling of the Graphiti stack.
- Improved the user interface for Graphiti settings in the installer wizard, allowing for better configuration management.
- Revised documentation to reflect changes in Graphiti's deployment modes and integration capabilities.
2026-07-30 16:26:16 +03:00
Dmitry Ng ca482686d7 feat(database): enhance PostgreSQL schema handling for multi-tenant support
- Added new environment variables `DATABASE_EXTENSIONS_SCHEMA` and `DATABASE_SEARCH_PATH_VIA_OPTIONS` to `.env.example` and `docker-compose.yml` for better management of PostgreSQL schemas in multi-tenant deployments.
- Updated the README and configuration documentation to explain the purpose and usage of the new variables, particularly for setups using Supabase.
- Implemented schema verification and initialization logic in the backend to ensure proper handling of tenant-specific schemas during database connections.
- Enhanced the installer and server settings forms to include the new configuration options, improving user experience and clarity.
2026-07-30 00:14:30 +03:00
Dmitry Ng 0da3892b0d fix(docker): clarify container capability management in Docker setup
- Updated the documentation in `docker.md` to provide a detailed explanation of the explicit capability allow-list used for primary containers, emphasizing the rationale behind the selected capabilities and the deliberate omission of `MKNOD`.
- Revised comments in `client.go` and `tools.go` to reflect the decision against using `no-new-privileges`, clarifying its impact on privilege escalation testing and container security.
- Enhanced the overall clarity and completeness of the capability management section to aid understanding of security measures in the Docker-in-Docker setup.
2026-07-29 22:33:14 +03:00
Dmitry Ng 131694b8aa feat(docker): enhance worker container Docker access configuration
- Added new environment variables `DOCKER_INSIDE_HOST`, `DOCKER_INSIDE_TLS_VERIFY`, and `DOCKER_INSIDE_CERT_PATH` to facilitate secure Docker access for worker containers.
- Updated `.env.example` and `docker-compose.yml` to reflect these new configurations.
- Enhanced backend logic to support the new Docker settings, ensuring proper isolation and security for worker containers.
- Expanded documentation to clarify the usage and implications of the new Docker access configurations.
2026-07-29 16:01:30 +03:00
Dmitry Ng 34a9cd2ee8 feat(config): enhance multi-instance support and Docker configurations
- Introduced `TENANT_ID` to namespace resources for multi-instance deployments, preventing collisions in shared services.
- Added new environment variables in `.env.example` and `docker-compose.yml` for Docker daemon configurations (`DOCKER_INSIDE_HOST`, `DOCKER_INSIDE_TLS_VERIFY`, `DOCKER_INSIDE_CERT_PATH`).
- Updated backend logic to handle tenant-specific configurations, ensuring proper isolation of resources across instances.
- Enhanced documentation to clarify the usage and implications of `TENANT_ID` and related Docker settings.
2026-07-29 16:01:24 +03:00
Dmitry Ng 5a4f1f54dc feat(search): introduce internal analytics engine for web search
- Added configuration options for an optional internal analytics engine in `.env.example`, `docker-compose.yml`, and related files.
- Updated documentation to include details about the new internal analytics engine, which scrapes and summarizes pages as a fallback for the web_search tool.
- Enhanced the `ftester` to support testing the new internal engine alongside existing search engines.
- Adjusted various components to integrate the new engine, including configuration forms and validation logic.

This feature allows users to enable a browser-based fallback for analytic queries, enhancing the flexibility of the web search capabilities.
2026-07-25 01:16:18 +03:00
Dmitry Ng 74c2663114 Merge branch 'integrate/open-prs-v2' into feature/next-release 2026-07-22 01:25:25 +03:00
Dmitry Ng e772a5cd82 Merge pull request #373 from rakshith48/feat/firecrawl-search
feat(tools): add Firecrawl web search provider
2026-07-18 19:33:57 +04:00
Dmitry Ng 6ca92ea40c Merge pull request #347 from mason5052/codex/issue-344-qwen-long-flow-troubleshooting
docs: add Qwen long-flow troubleshooting
2026-07-18 10:11:50 +04:00
Dmitry Ng 1ec1aef37f Merge pull request #274 from mason5052/codex/issue-249-macos-installer-warning
docs: add macOS installer warning guidance
2026-07-17 15:11:07 +04:00
rramprakash 7ffa30ddc9 address review: bound & align firecrawl summarizer prompt, fix docs
- Summarizer prompt now uses one-based source IDs (aligned with the numbered
  links) and the metadata-fallback title/URL, and truncates each document's
  markdown to the same per-result limit as the non-summarized path, so an
  oversized or hostile page can't blow up the prompt.
- Reword tool description, searcher prompt and docs to say 'main-content
  markdown' since the scrape uses onlyMainContent=true.
- config.md example now uses the real NewFirecrawlTool(...) constructor,
  matching the searxng/sploitus entries.
2026-07-16 12:53:11 +05:30
rramprakash 3f68960d98 feat(tools): add Firecrawl web search provider
Add Firecrawl as a network search provider alongside Tavily, Perplexity,
Traversaal, SearXNG and the others. Firecrawl's /v2/search endpoint combines
web search with page scraping, returning full-page markdown content for each
result, which the searcher agent summarizes like the other content-rich
engines.

- New firecrawl tool mirrors the Tavily provider layer-for-layer: registry
  definition, tool-type mapping, factory wiring in both tool builders, and
  ftester executor/args/mocks support.
- Configurable endpoint: FIRECRAWL_API_KEY plus an optional FIRECRAWL_API_URL
  (default https://api.firecrawl.dev) so self-hosted Firecrawl instances work.
- Adds 'firecrawl' to the searchengine_type enum (goose migration + models)
  and the search-log server model so searches are recorded per engine.
- Installer wizard, .env.example, docker-compose, prompts and docs updated.

The tool is only registered when FIRECRAWL_API_KEY is set, so existing
deployments are unaffected.
2026-07-16 12:53:11 +05:30
Sergey KozyrenkoandClaude Opus 4.8 18234f05a3 docs(docker): refresh the container-listing doc, drop a stale telemetry test comment
The container-listing section still described the removed `ls -1` parse,
`pkg/queue`, and the old `[]container.PathStat` return. Update it to the current
`find -print0` + errgroup fan-out returning `ContainerDirListing{Files, Failures,
Truncated}`. Also remove the stale `WithBlock`/`DefaultDialTimeout` paragraph left
above the non-blocking `grpc.NewClient` test — its own next paragraph already
contradicted it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 22:45:34 +07:00
Sergey KozyrenkoandClaude Opus 4.8 b8b90f8349 refactor(providers): use native adaptive thinking from the langchaingo fork
Adaptive thinking (Claude Opus 4.7+) is now implemented natively in the
vxcontrol/langchaingo fork (v0.1.14-update.6): the anthropic and bedrock clients
emit thinking.type=adaptive + output_config.effort and omit the sampling params
adaptive models reject. PentAGI no longer patches the serialized request body.

- pconfig.PrepareAdaptiveCallOptions appends llms.WithAdaptiveReasoning; the
  ctx-effort plumbing (WithAdaptiveEffort/AdaptiveEffortFromContext) is removed.
- Delete the bedrock smithy Build middleware and anthropic http RoundTripper
  (adaptive_thinking.go + tests in both packages) that rewrote the request body.
- Bump the langchaingo require to v0.1.14-update.6; refresh llms_how_to.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 00:47:32 +07:00
Sergey KozyrenkoandClaude Opus 4.8 d765345051 docs: propagate MiniMax + Claude 4.7/4.8 across deploy files and guides (refs #288, #328)
Code review found the providers were fully wired in code but missing from
several user-facing surfaces:

- MiniMax (#328): add to .env.example, docker-compose env passthrough,
  backend/docs/config.md env table, README provider list + a MiniMax
  configuration section, and backend/docs/database.md PROVIDER_TYPE enum.
- Claude Opus 4.7/4.8 (#288): add to the README Claude and Bedrock model
  catalogs and correct the model counts (10→11 Claude, 21→24 Bedrock); note
  4.7/4.8 are adaptive-thinking-only in the Adaptive Thinking section.
- llms_how_to.md: scope the "Temperature=1.0" rule to budget thinking and add
  an adaptive-thinking caveat (4.7/4.8 reject sampling params), so the guide no
  longer contradicts the shipped adaptive code path.
- docker-compose: point the Bedrock config/models default mount at the tracked
  examples/configs/bedrock-glm-flash.* files (the README-documented path) so a
  plain `docker compose up` no longer creates empty placeholder directories.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 18:04:43 +07:00
Sergey KozyrenkoandClaude Opus 4.8 06e37d8026 feat(bedrock): support external config and model catalog paths (refs #233)
Reworks PR #233 to follow the existing config-path pattern and make an
externally-added Bedrock model fully usable from the web UI.

- BEDROCK_CONFIG_PATH: replace the embedded per-agent config (model
  assignments + prices), mirroring OLLAMA_SERVER_CONFIG_PATH /
  LLM_SERVER_CONFIG_PATH. DefaultProviderConfig now takes *config.Config
  and reads the external file when set, else the embedded config.yml.
- BEDROCK_MODELS_PATH: merge an external model catalog onto the embedded
  models.yml so new ids appear and are selectable under Settings ->
  Providers. DefaultModels now takes *config.Config and the resolver +
  provider construction pass it through; new ids are added, a matching
  name overrides the embedded entry.

Wires both through docker-compose, .env.example, config.md and the README,
and ships examples/configs/bedrock-glm-flash.{provider,models}.yml using the
real AWS Bedrock id zai.glm-4.7-flash (In-Region, no inference-profile
prefix, 4K max output).

Dropped from #233: the ProviderConfig.Name field and the SeedDefaultProviders
mechanism (a per-user DB write fired on startup and inside the read-only
SettingsProviders resolver) -- the config already reaches the UI through
DefaultProvidersConfig without any DB rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 08:30:10 +07:00
mason5052 5be3a409f2 docs: tighten macOS installer warning guidance 2026-06-19 04:45:28 +07:00
mason5052 f37d7c97ed docs: refine macOS installer warning guidance 2026-06-19 04:45:27 +07:00
mason5052 305e071975 docs: add macOS installer warning guidance 2026-06-19 04:45:27 +07:00
mason5052 6f93856716 docs: add qwen long-flow troubleshooting 2026-06-19 04:45:05 +07:00
mason5052andClaude Opus 4.8 d5a86ff27b feat: add evidence receipt hash chain prototype (#279)
Disabled-by-default audit feature (EVIDENCE_RECEIPTS_ENABLED): appends a
hash-chained JSONL receipt per finished/failed toolcall under
<DATA_DIR>/flow-<id>/evidence/receipts.jsonl, recording toolcall
provenance plus SHA-256 hashes of args/result (no raw content).

Integration adapted to current main: the original PR built the receipt
from a database.Toolcall returned by ce.db.UpdateToolcall*Result, which
the executor no longer uses after the ToolCallLogProvider (tclp) refactor.
Receipts are now built from the in-scope toolcall data at the tclp log
sites and recorded non-fatally, so a receipt failure is logged and never
fails an otherwise-successful toolcall.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-19 04:41:57 +07:00
mason5052 1e19b70988 docs: add qwen long-flow troubleshooting 2026-06-09 20:09:55 -04:00
Dmitry Ng 9d68f570a8 Merge pull request #318 from mason5052/codex/issue-310-vertex-ai-config-docs
docs(llm): clarify Vertex AI configuration options
2026-05-28 00:15:15 +04:00
Dmitry Ng e674113745 Merge pull request #317 from mason5052/codex/issue-314-deepseek-v4-models
fix(deepseek): update default model names to DeepSeek V4
2026-05-28 00:09:25 +04:00
mason5052 5ff63cc4a8 docs(llm): clarify Vertex AI configuration options
Issue #310 asks how to provide a Google Vertex AI API key in .env for
Anthropic Claude. PentAGI currently has no dedicated Vertex AI provider
path in code: backend/pkg/config and backend/cmd/installer do not read
VERTEX_API_KEY, GOOGLE_APPLICATION_CREDENTIALS, or any vertex_ai
variable. The supported routes for Claude today are direct Anthropic
(ANTHROPIC_API_KEY / ANTHROPIC_SERVER_URL) and AWS Bedrock (BEDROCK_*).

Document this explicitly so users do not assume a hidden Vertex AI
configuration path exists:

- README.md: add a NOTE callout inside the Anthropic Provider
  Configuration section listing the supported routes and pointing
  users who need Vertex AI today at the OpenAI-compatible custom LLM
  provider path (LLM_SERVER_URL / LLM_SERVER_KEY / LLM_SERVER_MODEL)
  fronted by a translating gateway, with a caveat that reliability
  depends on the gateway.
- backend/docs/config.md: add a matching Note paragraph under the
  Anthropic section that points at the AWS Bedrock and custom LLM
  provider sections, and states that no VERTEX_API_KEY or
  GOOGLE_APPLICATION_CREDENTIALS variable is wired into provider
  initialization today.

Docs-only change. No runtime Go code, no installer behavior, no
generated files, no new environment variables. All env var names cited
in the new text already exist in the current PentAGI .env.example,
backend/pkg/config, and backend/cmd/installer.
2026-05-22 12:50:19 -04:00
Dmitry Ng db88ef5eaa Merge pull request #305 from mason5052/codex/issue-187-graphiti-limitations-docs
docs(graphiti): note beta status and OpenAI-only provider limitation
2026-05-22 01:12:43 +04:00
mason5052 3113ff3aa3 fix(deepseek): update default model names to DeepSeek V4
The DeepSeek provider config still defaulted to the legacy
`deepseek-chat` and `deepseek-reasoner` model names, which the
upstream DeepSeek API has announced for deprecation on 2026-07-24.
A first-run install therefore breaks once the legacy names are
removed.

Swap the defaults to the current DeepSeek V4 family:

- non-thinking roles use `deepseek-v4-flash`
- reasoning-heavy roles use `deepseek-v4-pro`

The change is limited to the embedded `config.yml` / `models.yml`
inside `backend/pkg/providers/deepseek`, the matching example at
`examples/configs/deepseek.provider.yml`, the `DeepSeekAgentModel`
fallback constant in `deepseek.go`, and three doc references
(README.md, backend/docs/config.md, backend/docs/llms_how_to.md)
plus one installer help string in
`backend/cmd/installer/wizard/locale/locale.go`. LiteLLM prefix
behavior is untouched.
2026-05-21 15:44:30 -04:00
Dmitry Ng 39f122467d feat(config): add new embedding and rename database connection pool settings
- Introduced `EMBEDDING_MAX_TEXT_BYTES` to limit the maximum byte size of text sent to the embedding model.
- Renamed database connection pool settings: `DATABASE_MAX_OPEN_CONNS`, `DATABASE_MAX_IDLE_CONNS`, and `DATABASE_VECTOR_MAX_CONNS` for improved PostgreSQL connection management.
- Updated relevant documentation to reflect these new configuration options and their usage.
- Adjusted various components to utilize the new settings for enhanced performance and resource management.
2026-05-18 18:26:52 +03:00
Dmitry Ng 2ce863ec1a feat(toolcall): implement ToolCall logging functionality
- Added ToolCallLogProvider interface with methods for logging tool calls, updating success and failure statuses.
- Introduced proxyToolCallLogProvider to handle ToolCall logging operations.
- Updated flow execution components to integrate ToolCall logging, including flow workers and controllers.
- Enhanced GraphQL schema to support ToolCall logs, including queries and subscriptions for real-time updates.
- Updated documentation to reflect the new ToolCall logging features and their usage.
2026-05-18 11:21:56 +03:00
Dmitry Ng 077ddce476 feat(database): enhance PostgreSQL connection pooling and configuration
- Introduced shared connection pooling for PostgreSQL using `*sql.DB` for sqlc and GORM, optimizing resource usage.
- Added new environment variables: `DB_MAX_OPEN_CONNS`, `DB_MAX_IDLE_CONNS`, and `DB_VECTOR_MAX_CONNS` for configurable connection limits.
- Updated documentation to reflect new connection pooling strategy and provide operational commands for monitoring.
- Implemented shared `pgxpool` for pgvector stores to reduce connection overhead and improve performance.
- Adjusted various components to utilize the new connection pooling setup, ensuring efficient database interactions.
2026-05-18 11:13:43 +03:00
mason5052 c0e9ed74fd docs(graphiti): use user-facing env vars for OpenAI endpoint
Address Copilot review feedback on PR #305: the 'Current Limitations' bullet for the Graphiti integration mixed PentAGI's user-facing .env variables with the container env vars defined in docker-compose-graphiti.yml.

Reword both README.md and backend/docs/config.md to lead with the user-facing OPEN_AI_KEY and OPEN_AI_SERVER_URL .env variables and explicitly note that docker-compose-graphiti.yml maps them into the bundled vxcontrol/graphiti container as OPENAI_API_KEY and OPENAI_BASE_URL. Operators set the .env variables; the container variables are an implementation detail.

Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-05-07 13:16:01 -04:00
mason5052 fa53a4e1d2 docs(graphiti): note beta status and OpenAI-only provider limitation
The Graphiti container shipped with docker-compose-graphiti.yml only
takes OPENAI_API_KEY and OPEN_AI_SERVER_URL for entity extraction.
PentAGI configures many other LLM providers (Anthropic, Google AI,
AWS Bedrock, DeepSeek, GLM, Kimi, Qwen) for the main flow, but those
credentials are not consumed by Graphiti today. Until that changes,
operators need to plan around an OpenAI-compatible endpoint just for
the knowledge graph.

This commit makes the limitation visible in two surfaces without
changing runtime behavior:

- README.md: Adds a beta callout at the top of the Knowledge Graph
  Integration section and a new 'Current Limitations' subsection
  covering provider scope, fixed model, independent billing, and the
  lack of an in-app graph explorer.
- backend/docs/config.md: Mirrors the beta callout under Graphiti
  Knowledge Graph Settings and adds a 'Current Limitations (Beta)'
  subsection with the same constraints, so config-focused readers
  see the same message.

Both notes explicitly point at the simple fallback: leave
GRAPHITI_ENABLED=false if the deployment cannot reach an
OpenAI-compatible endpoint.

Refs #187

Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-05-07 12:22:38 -04:00
mason5052 1cd643f72c docs: tighten macOS installer warning guidance 2026-05-05 20:14:58 +00:00
Dmitry Ng c0035ac130 Merge pull request #293 from mason5052/codex/issue-291-fix-processor-doc-links
docs: fix processor wizard integration links
2026-04-30 12:03:07 +04:00
Dmitry Ng 47de4e44e6 docs: update links in configuration and flow execution documents, add directory for proposals 2026-04-30 10:09:03 +03:00
MasonandMason Kim 6257612689 docs: add OSINT integration scenarios (#278)
* docs: add OSINT integration scenarios

* docs: clarify OSINT scenario payloads

* docs: clarify OSINT provider identifiers

* docs: distinguish OSINT provider ids

---------

Co-authored-by: Mason Kim(ZINUS US_SALES) <mkim@zinus.com>
2026-04-30 10:47:08 +04:00
mason5052 29d5723620 docs: fix processor wizard integration links 2026-04-29 10:58:03 -04:00
mason5052 ffdeec280f docs: refine macOS installer warning guidance 2026-04-28 23:03:09 +00:00
Dmitry Ng 402dfbe0ee Merge pull request #277 from mason5052/codex/issue-235-evidence-chain-rfc
docs: add evidence chain RFC
2026-04-29 01:35:59 +04:00
Dmitry Ng 425f075a96 Merge pull request #276 from mason5052/codex/issue-71-scope-of-work-template
docs: add scope-of-work pentest template
2026-04-29 01:08:29 +04:00
Dmitry NgandCopilot de6f527e6f Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dmitry Ng <19asdek91@gmail.com>
2026-04-29 01:07:47 +04:00
Dmitry Ng aa4b70eaba feat: add user resources system and flow integration
- UserResource model with MD5-deduplicated blob storage and virtual path filesystem
- REST API for resource CRUD (upload, mkdir, move, copy, delete, download)
- GraphQL query/mutations with resourceIds support on createFlow, putUserInput, createAssistant, callAssistant
- Resource → flow copy with hierarchy restore; incremental container sync (find missing, copy once)
- FlowWorker.PutResources delegates copy, docker push and flowFileAdded events
- Agent prompts updated with {{.UserFiles}} XML listing of /work/uploads and /work/resources
- Resource subscriptions: resourceAdded/Updated/Deleted
2026-04-28 17:00:17 +03:00
Dmitry Ng 7c4ebda2c2 feat: enhance Docker client with container file operations and API integration
- Added methods for non-recursive directory listing and file stat operations in the Docker client.
- Implemented a new API endpoint to retrieve files from a running container's directory.
- Updated documentation to reflect new file operations and API changes.
- Introduced data structures for container file metadata and integrated them into the flow file service.
- Enhanced flow file management capabilities with improved synchronization between local and container file systems.
2026-04-27 13:19:39 +03:00
Dmitry Ng 5e902ca255 docs: update API tokens section to reflect new naming convention in settings 2026-04-24 03:00:48 +03:00
Dmitry Ng 378db10732 Merge pull request #270 from mason5052/codex/issue-191-settings-coverage
docs: clarify current web settings coverage
2026-04-24 03:52:41 +04:00