Commit Graph

39 Commits

Author SHA1 Message Date
Sergey Kozyrenko 06e37d8026 feat(bedrock): support external config and model catalog paths (refs #233)
Reworks PR #233 to follow the existing config-path pattern and make an
externally-added Bedrock model fully usable from the web UI.

- BEDROCK_CONFIG_PATH: replace the embedded per-agent config (model
  assignments + prices), mirroring OLLAMA_SERVER_CONFIG_PATH /
  LLM_SERVER_CONFIG_PATH. DefaultProviderConfig now takes *config.Config
  and reads the external file when set, else the embedded config.yml.
- BEDROCK_MODELS_PATH: merge an external model catalog onto the embedded
  models.yml so new ids appear and are selectable under Settings ->
  Providers. DefaultModels now takes *config.Config and the resolver +
  provider construction pass it through; new ids are added, a matching
  name overrides the embedded entry.

Wires both through docker-compose, .env.example, config.md and the README,
and ships examples/configs/bedrock-glm-flash.{provider,models}.yml using the
real AWS Bedrock id zai.glm-4.7-flash (In-Region, no inference-profile
prefix, 4K max output).

Dropped from #233: the ProviderConfig.Name field and the SeedDefaultProviders
mechanism (a per-user DB write fired on startup and inside the read-only
SettingsProviders resolver) -- the config already reaches the UI through
DefaultProvidersConfig without any DB rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 08:30:10 +07:00
Sergey Kozyrenko ebe0618dee docs: add Atlas Cloud as a custom OpenAI-compatible provider (refs #332)
Reworks PR #332 to follow the repo's provider-doc convention instead of
its README-marketing form:
- add examples/configs/atlas.provider.yml, modeled on the existing
  aggregator example configs (deepinfra/openrouter/novita)
- list Atlas Cloud in the README aggregators line
- LLM_SERVER_PROVIDER left EMPTY for direct access — it is a LiteLLM
  model-name prefix, and #332 incorrectly recommended `openai`, which
  would break /models discovery for Atlas's vendor-prefixed model ids

Dropped from #332 (not matching any existing aggregator's docs): vendor
logo/banner, UTM-tracked links, and the static 59-model table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 07:43:20 +07:00
mason5052 3ef8b51f36 docs: address review feedback on install/config guide
- Use the package-path form for the tester utilities
  (go run ./cmd/ctester, go run ./cmd/etester test) instead of shell
  glob / single-file invocations, which are more portable and avoid the
  multi-file compile pitfall.
- Add -verbose to the container etester example so it matches the local
  example.
- Wrap the new README Quick Start pointer across multiple lines; rendered
  Markdown is unchanged.
2026-06-19 04:45:20 +07:00
mason5052 ebfd2bd9e9 docs: add installing and configuring guide for first deployment
Issue #61 asked for a proper guide on how to start using PentAGI. A
first-use guide (How to Use PentAGI After Login) already covers what to
do after login, but the maintainer kept the issue open until an article
about installing and configuring the product is published.

Add examples/guides/installation_configuration.md, a concise ordered
walkthrough that connects the existing detailed README sections into a
single path: choose installer vs manual Docker Compose, set core server
variables (PUBLIC_URL, CORS_ORIGINS, security vars), configure and test
an LLM provider with ctester, configure and test the embedding provider
with etester, optionally add search providers and the Graphiti/Langfuse/
observability stacks, then start and verify the stack and change the
default admin password. It ends with a quick checklist and links forward
to the existing post-login usage guide.

The guide links to the relevant README reference sections rather than
duplicating them. A single pointer link is added at the top of the
README Quick Start section for discoverability.

Docs only. No installer behavior, runtime code, env vars, or schema
changes. Every environment variable named in the guide already exists in
.env.example and backend/pkg/config/config.go.
2026-06-19 04:45:20 +07:00
mason5052 6f93856716 docs: add qwen long-flow troubleshooting 2026-06-19 04:45:05 +07:00
mason5052 e8a26c6d2c docs: clarify crawler RFC config sketch and scope_decision
Address Copilot review on PR #350: note that default_backend selects a crawl backend and passive extractors such as jsfinder are not selectable there; model scope_decision as a decision plus reason in both the field list and the JSON example; add an illustrative scope_entries example and clarify when allowed_hosts applies.

Refs #336
2026-06-19 04:12:40 +07:00
mason5052 7af46322ca docs: add headless crawler integration RFC
Add examples/proposals/headless_crawler_integration.md, a docs-only RFC proposing an optional, tool-agnostic crawler / URL discovery capability for PentAGI agents. Candidate backends are katana, crawlergo, rad, and jsfinder, framed as candidates with no mandatory default. The RFC keeps dictionary fuzzing (ffuf/dirsearch) unchanged, defines structured discovery artifacts (URLs, forms, parameters, JavaScript endpoints, status codes, source page, depth, scope decision), and keeps crawler URL discovery separate from the BrowserOS MCP interactive browser backend.

Refs #336
2026-06-19 04:12:39 +07:00
mason5052 5980c02854 docs: add recommendation and requested decision to Kubernetes RFC 2026-06-19 04:12:39 +07:00
mason5052 5bf41e0ea1 docs: fix grammar in Kubernetes RFC open question (#324)
Address review feedback on PR #326: rephrase the executor-model open
question so the final clause reads clearly.

Refs #324
2026-06-19 04:12:39 +07:00
mason5052 8dbf8889e0 docs: add Kubernetes deployment compatibility RFC (#324)
Add examples/proposals/kubernetes_deployment.md, an RFC-style design
surface responding to the Kubernetes deployment request in #324. The
document is docs-only: no Helm charts, manifests, operator, compose,
installer, or environment-variable changes, and it does not claim
Kubernetes is supported today.

It records the current Compose/installer deployment assumptions, maps
each one (secrets/config, volumes, service discovery, ingress/TLS,
health checks, network policies, the Docker-socket flow executor,
observability, image overrides, migrations) to candidate Kubernetes
equivalents, and proposes an incremental, docs-first path with open
questions, security/operational considerations, and a test/validation
strategy. The hardest item -- the Docker-socket worker executor -- is
laid out as candidate options without choosing one, and keeps flow
lifecycle explicit and inspectable per the #268 review lesson.

Refs #324
2026-06-19 04:12:39 +07:00
mason5052 760574677b docs: address review feedback on Vertex AI RFC
- Drop the hard-coded provider count to avoid doc drift as providers are
  added; list the current provider types instead.
- Use the ADC-specific command (gcloud auth application-default login)
  rather than the ambiguous 'gcloud login'.
- Reference the enum-swap migration pattern generically under
  backend/migrations/sql/ instead of a single timestamped filename that
  may be renamed or squashed.
2026-06-19 04:12:39 +07:00
mason5052 16669696b6 docs: add native Google Vertex AI provider RFC
Issue #321 requests a native Vertex AI provider with service-account /
ADC auth, opened after #310 clarified Vertex is not supported today. The
issue carries a full implementation outline and four open questions, and
the author asks for direction on adapter strategy and Gemini-vs-Claude
scope before implementing.

Add examples/proposals/vertex_ai_provider.md, a planning RFC that frames
the work before any code is written:

- Distinguishes the current options (AI Studio gemini, direct Anthropic,
  AWS Bedrock, and the custom OpenAI-compatible LiteLLM proxy workaround)
  from the proposed native vertex provider.
- Recommends a staged approach: v1 Gemini-on-Vertex with ADC /
  service-account auth; Claude-on-Vertex deferred to a maintainer
  decision, noting it likely belongs on the Anthropic adapter (Vertex
  auth/endpoint mode) rather than the Gemini-shaped path because the
  message schema differs.
- Models auth on the existing Bedrock multi-auth precedent (ADC default
  chain plus service-account file), and flags that service-account JSON
  is sensitive and must be file-mounted/secret-managed, never pasted into
  UI or logs.
- Captures config/migration touch points (provider checklist, REST
  Valid() whitelist, PROVIDER_TYPE enum-swap migration) so the eventual
  implementation size is clear, and restates the issue's open questions.

Docs/RFC only. No code, schema, migration, generated, frontend, or
provider runtime files are touched, and no new env vars or types are
added; every VERTEX_* key and type named is labeled as a candidate. No
overlap with the provider files in open PR #328.
2026-06-19 04:12:38 +07:00
mason5052 040c69ff28 docs: add recommendation and requested decision to BrowserOS MCP RFC 2026-06-19 04:12:38 +07:00
mason5052 b6f1e61dae docs: add BrowserOS MCP browser backend RFC (#342) 2026-06-19 04:12:38 +07:00
mason5052 92426b9b98 docs: add v1 recommendation and requested decision to fallback RFC 2026-06-19 04:12:38 +07:00
mason5052 44393b0f06 docs: add tool and model fallback RFC 2026-06-19 04:12:37 +07:00
Dmitry Ng 83c263e98e feat(qwen): enhance agent configurations with thinking control parameters
Updated the Qwen agent configuration to include `extra_body` parameters for thinking control across various models. Added `enable_thinking` and `preserve_thinking` options for reasoning agents, while utility agents have `enable_thinking` set to false. Adjusted the Qwen client initialization to support these new configurations. Updated test report to reflect changes in success rates and latencies.
2026-05-29 00:06:47 +03:00
Dmitry Ng 643ff7d218 feat(providers): Update model configurations for various LLM providers: qwen, kimi, glm, deepseek, gemini 2026-05-28 15:41:17 +03:00
Dmitry Ng 2f827a54ef Merge pull request #307 from mason5052/codex/issue-296-mcp-client-rfc
docs(rfc): propose MCP client integration design
2026-05-28 00:11:02 +04:00
Dmitry Ng e674113745 Merge pull request #317 from mason5052/codex/issue-314-deepseek-v4-models
fix(deepseek): update default model names to DeepSeek V4
2026-05-28 00:09:25 +04:00
Dmitry Ng 084670dd97 Merge pull request #306 from mason5052/codex/issue-298-flow-concurrency-rfc
docs(rfc): propose persistent flow queue and completion webhooks
2026-05-23 16:06:23 +04:00
mason5052 24176c2805 fix(deepseek): explicitly disable thinking mode for non-thinking Flash roles
DeepSeek V4 thinking mode defaults to enabled on both deepseek-v4-flash
and deepseek-v4-pro; non-thinking behavior requires an explicit toggle.
Per official docs, in thinking mode temperature/top_p/presence_penalty/
frequency_penalty are ignored, so the existing Flash role sampling knobs
would have been silently no-ops without the toggle.

Add extra_body.thinking.type=disabled to the five non-thinking Flash
roles so deepseek-v4-flash actually runs in non-thinking mode and
honors the role's temperature/top_p settings:

- simple, simple_json, adviser, searcher, enricher

Pro roles (primary_agent, assistant, generator, refiner, reflector,
coder, installer, pentester) intentionally keep thinking enabled (the
V4 default) for reasoning, tool-use, and security analysis.

PentAGI provider config already supports extra_body as a first-class
yaml field on AgentConfig and forwards it through openai.WithExtraBody,
which the vxcontrol langchaingo fork serializes at the top level of the
Chat Completions request - the same pattern Kimi uses for tool_choice.
No code, schema, or LiteLLM prefix changes required.

Touches:
- backend/pkg/providers/deepseek/config.yml (embedded production config)
- examples/configs/deepseek.provider.yml (user-facing example)

No change to role-to-model mapping, model metadata, pricing, README
wording, LiteLLM prefix, unrelated providers, lifecycle, queues, or
installer flow.
2026-05-22 12:26:28 -04:00
mason5052 67bf76514b docs(deepseek): align V4 model metadata with official pricing and context
- Update model descriptions to reflect V4 1M context window (up to 384K output)
  instead of legacy 128K wording in models.yml and README.
- Split Flash and Pro pricing per official DeepSeek API docs:
  - deepseek-v4-flash: input 0.14 / output 0.28 / cache_hit 0.0028 per 1M tokens
  - deepseek-v4-pro:   input 0.435 / output 0.87 / cache_hit 0.003625 per 1M tokens
- Apply per-role price split across all 13 role configs in both the embedded
  config.yml and the user-facing examples/configs/deepseek.provider.yml.
- Replace stale "cache pricing is 10% of input cost" claim in the README,
  which no longer holds for either V4 model.
- No change to LiteLLM prefix behavior, role-to-model mapping, lifecycle,
  queues, GraphQL schema, migrations, frontend, or installer flow.
2026-05-22 12:04:14 -04:00
mason5052 3113ff3aa3 fix(deepseek): update default model names to DeepSeek V4
The DeepSeek provider config still defaulted to the legacy
`deepseek-chat` and `deepseek-reasoner` model names, which the
upstream DeepSeek API has announced for deprecation on 2026-07-24.
A first-run install therefore breaks once the legacy names are
removed.

Swap the defaults to the current DeepSeek V4 family:

- non-thinking roles use `deepseek-v4-flash`
- reasoning-heavy roles use `deepseek-v4-pro`

The change is limited to the embedded `config.yml` / `models.yml`
inside `backend/pkg/providers/deepseek`, the matching example at
`examples/configs/deepseek.provider.yml`, the `DeepSeekAgentModel`
fallback constant in `deepseek.go`, and three doc references
(README.md, backend/docs/config.md, backend/docs/llms_how_to.md)
plus one installer help string in
`backend/cmd/installer/wizard/locale/locale.go`. LiteLLM prefix
behavior is untouched.
2026-05-21 15:44:30 -04:00
Dmitry Ng bff332b647 feat(docker): add new provider configurations for Qwen 3.6 35B models
- Included two new provider YAML files for Qwen 3.6 35B models: `vllm-qwen3.6-35b-a3b-fp8-no-think.provider.yml` and `vllm-qwen3.6-35b-a3b-fp8.provider.yml`.
- Updated Dockerfile to copy the new configuration files into the appropriate directory.
2026-05-14 14:11:27 +03:00
mason5052 5af902872a docs(rfc): address Copilot review feedback on MCP client RFC
- Reword PR #268 references to talk about review feedback rather
  than the PR being rejected.
- Clarify host.docker.internal availability: not universally provided
  by the core compose stack; Docker Desktop typically resolves it,
  while Linux/operator-managed compose stacks may need an explicit
  extra_hosts: host.docker.internal:host-gateway entry or another
  controlled endpoint.
- Make the safe initial Burp example unambiguously read-only: drop
  start_active_scan from the illustrative allowlist and call out
  that active capabilities belong to a later, explicitly gated
  milestone with scope and approval controls.
- Rephrase the awkward 'PentAGI must not be inferred...' sentence
  for clarity.

Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-05-08 16:09:01 -04:00
mason5052 ba8a687cb6 docs(rfc): propose MCP client integration design
Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-05-08 15:55:23 -04:00
mason5052 2d407b928e docs(rfc): clarify lifecycle, terminal semantics, and webhook event names
Address Copilot review feedback on PR #306:

- Lifecycle diagram: show running <-> waiting (waiting is a paused state that resumes to running when user input arrives) and document that both running and waiting can reach the terminal statuses finished or failed.

- Replace overloaded 'finished' wording with explicit 'terminal' semantics throughout. finished and failed remain distinct terminal statuses; the queue and webhook layers treat both as terminal.

- Align webhook event names with status terminology: flow.finished for success, flow.failed for failure. Update payload example accordingly and note the failed-flow shape.

Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-05-07 13:18:34 -04:00
mason5052 295e7d3b01 docs(rfc): propose persistent flow queue and completion webhooks
Proposes a design direction for native flow concurrency control and
completion notifications. The RFC follows the maintainer's relocated
proposal pattern at examples/proposals/<topic>.md and explicitly
builds on the lessons from PR #268 (rejected because the in-memory
queue was hidden lifecycle state).

The RFC covers:

- Goals limited to capping concurrent flows, persisting queued flows
  as first-class lifecycle, replacing external polling with at-least-
  once webhooks, and preserving the existing createFlow contract.
- Non-Goals that explicitly forbid hidden in-memory queues, multi-
  tenant scheduling, generic event bus features, and changing the
  meaning of 'finished' for tasks/subtasks/toolcalls.
- Design Principles for persistence, visibility, manageability,
  explicit promotion, clear finished semantics, and at-least-once
  delivery.
- A proposed concurrency model with a new persisted 'queued' status,
  a single MAX_CONCURRENT_FLOWS knob, an explicit promoter, and
  full UI/API visibility plus user cancellation.
- A proposed completion webhook model with per-flow and global URLs,
  HMAC-SHA256 signatures, persisted deliveries, bounded retries,
  and SSRF mitigations.
- Storage and API surface sketches that do not commit to a final
  schema.
- Open Questions covering per-user limits, blocking semantics on
  createFlow, signature alignment with the issue #235 receipt
  direction, and behavior of resources/uploads against queued flows.
- A Suggested First Milestone that lands the queue end-to-end before
  webhooks, to keep PR sizes reviewable.

This is documentation only. No runtime code, schema, GraphQL, REST,
or UI behavior changes here.

Refs #298

Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-05-07 12:28:06 -04:00
Dmitry Ng c068d86bf0 feat(docker): update Dockerfile and add new vLLM configurations
- Added new provider configurations for vLLM Qwen 3.6 in both thinking and non-thinking modes.
- Updated the Dockerfile to include the new configuration files for vLLM Qwen 3.6 and ensure proper setup for deployment.
2026-05-02 18:57:42 +03:00
Dmitry Ng 47de4e44e6 docs: update links in configuration and flow execution documents, add directory for proposals 2026-04-30 10:09:03 +03:00
Mason 6257612689 docs: add OSINT integration scenarios (#278)
* docs: add OSINT integration scenarios

* docs: clarify OSINT scenario payloads

* docs: clarify OSINT provider identifiers

* docs: distinguish OSINT provider ids

---------

Co-authored-by: Mason Kim(ZINUS US_SALES) <mkim@zinus.com>
2026-04-30 10:47:08 +04:00
Dmitry Ng 425f075a96 Merge pull request #276 from mason5052/codex/issue-71-scope-of-work-template
docs: add scope-of-work pentest template
2026-04-29 01:08:29 +04:00
Dmitry Ng de6f527e6f Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dmitry Ng <19asdek91@gmail.com>
2026-04-29 01:07:47 +04:00
Dmitry Ng fb607a8bc7 Merge branch 'feature/next-release' into codex/issue-69-openvas-custom-image-guide
Signed-off-by: Dmitry Ng <19asdek91@gmail.com>
2026-04-29 01:05:07 +04:00
mason5052 5bc534c590 docs: add scope-of-work pentest template 2026-04-22 13:05:40 -04:00
mason5052 3e4e800c44 docs: add OpenVAS custom image guide 2026-04-22 13:04:01 -04:00
Dmitry Ng 211931839e feat: add Azure OpenAI provider configuration file 2026-04-22 20:00:17 +03:00
Dmitry Ng b90ea4711e repo final state 2026-03-26 06:16:07 +03:00