Commit Graph
26 Commits
Author SHA1 Message Date
Sergey KozyrenkoandClaude Opus 4.8 2c524faea9 fix(observability,flows): drain telemetry sinks in parallel, stop leaking docker errors, dedup listing failures
Three defects on the shutdown and container-listing paths:

- Telemetry drain ran the langfuse and otel sinks serially under one deadline.
  langfuse's ForceFlush/Shutdown ignore the caller ctx and block on their own
  timeout, so a langfuse outage burned the whole budget and the healthy otel
  sink's final batch was never flushed. Add observer.Drain, which flushes then
  shuts down each sink in its own goroutine bounded by the deadline; main uses
  it, retiring the inline flush-then-shutdown dance.

- The partial-success listing put the raw stat/list error into the 200 body's
  Failures[].Message, exposing docker-layer detail (container ids, the daemon
  address) that response.Error only reveals under develop mode. Send a generic
  message to the client and keep the raw text dev-gated, logged server-side.

- A path read successfully by one query could also appear in Failures when
  another query failed on it, order-dependent. Reconcile at assembly: drop any
  failure whose path was read successfully, so no path is in both arrays.

Also bound the exec stdout buffered before parsing so a compromised sandbox
can't stream unbounded output into memory.

Tests: Drain delivers a healthy otel sink while a langfuse sink blocks and
returns within its deadline; the listing failure message is dev-gated; a path
never lands in both Files and Failures. Live: unreachable collector -> shutdown
stays bounded (~5s) and exits cleanly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 22:24:02 +07:00
Sergey KozyrenkoandClaude Opus 4.8 bc8714db6e fix(observability): flush telemetry on graceful shutdown, drain every sink
The graceful-shutdown branch only logged "cleaning up resources" and exited
without flushing telemetry, so the last buffered batch was dropped on every
SIGTERM. Drain it, with three corrections:

- main: flush then shut down the observer on shutdown, bounded by a 5s
  deadline in a goroutine so an unreachable collector can't stall exit.
  Flush must precede Shutdown because the langfuse observer's Shutdown
  cancels its context without flushing; its final batch drains only via
  ForceFlush while the context is still live.
- observer Flush/Shutdown: drain both the langfuse and otel sinks and join
  their errors instead of returning after the first, which skipped the
  second sink entirely when both were configured.
- telemetryClient Flush/Shutdown: attempt all three providers and always
  close the grpc connection, joining errors, so a stalled provider flush
  no longer skips the rest or leaks the connection.

Tests: the langfuse observer test proves Shutdown drops the buffered batch
while ForceFlush drains it to the sink; observer-level tests prove both
sinks drain and errors aggregate. Live: with an unreachable collector,
shutdown stays bounded (~5s) and exits cleanly instead of stalling ~10s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 17:30:31 +07:00
Sergey KozyrenkoandClaude Opus 4.8 d6fa58e5d4 fix(observability): don't let an unreachable telemetry collector down the app
A set-but-unreachable OTEL collector hung startup: NewTelemetryClient dialed
with grpc.WithBlock() and no timeout on the deadline-free signal context, so
the process blocked forever before it ever served. And a non-ErrNotConfigured
init error from either observability client was log.Fatalf, killing the
process. Both let an OPTIONAL integration take the whole app down.

Bound the dial with a 10s timeout, and on init failure degrade to a no-op
observer with a logged warning instead of exiting. Verified live: a bad
OTEL_HOST that hung startup indefinitely now boots in ~14s with a warning and
runs a full flow; a reachable collector still boots in ~4s with no warning.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 23:20:43 +07:00
Sergey KozyrenkoandClaude Opus 4.8 55335feb17 feat(minimax): add MiniMax provider with full app/UI/installer parity (refs #328)
Reworks external PR #328 (octo-patch). The PR added only the provider core;
this brings MiniMax to full parity with the other providers (qwen) so it is
selectable and configurable in the UI and installer.

Applied from the PR (verified against MiniMax's official API docs — M3/M2.7/
M2.7-highspeed are real current models; corrected the M3 description from the
PR's "512K" to the documented ~1M context):
- minimax provider package (OpenAI-compatible https://api.minimax.io/v1),
  config.yml, models.yml, tests; MINIMAX_API_KEY/SERVER_URL/PROVIDER env vars;
  ProviderMiniMax type + DefaultProviderNameMiniMax; providers.go wiring;
  Valid() whitelist.

Added for completeness:
- goose migration adding 'minimax' to the PROVIDER_TYPE enum + database
  ProviderTypeMinimax const.
- GraphQL: minimax in ProviderType enum, ProvidersModelsList,
  ProvidersReadinessStatus, DefaultProvidersConfig; resolvers wire default
  config/models + enabled status; gqlgen regenerated.
- Frontend: MiniMax icon (lobehub), provider-icon + settings-providers
  registration + provider type list; regenerated GraphQL types.
- Installer wizard: provider form, screen, list, registry, env-var mappings,
  locale strings + help text.
- ctester/ftester: -type/-provider minimax support.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 15:24:04 +07:00
Sergey KozyrenkoandClaude Opus 4.8 06e37d8026 feat(bedrock): support external config and model catalog paths (refs #233)
Reworks PR #233 to follow the existing config-path pattern and make an
externally-added Bedrock model fully usable from the web UI.

- BEDROCK_CONFIG_PATH: replace the embedded per-agent config (model
  assignments + prices), mirroring OLLAMA_SERVER_CONFIG_PATH /
  LLM_SERVER_CONFIG_PATH. DefaultProviderConfig now takes *config.Config
  and reads the external file when set, else the embedded config.yml.
- BEDROCK_MODELS_PATH: merge an external model catalog onto the embedded
  models.yml so new ids appear and are selectable under Settings ->
  Providers. DefaultModels now takes *config.Config and the resolver +
  provider construction pass it through; new ids are added, a matching
  name overrides the embedded entry.

Wires both through docker-compose, .env.example, config.md and the README,
and ships examples/configs/bedrock-glm-flash.{provider,models}.yml using the
real AWS Bedrock id zai.glm-4.7-flash (In-Region, no inference-profile
prefix, 4K max output).

Dropped from #233: the ProviderConfig.Name field and the SeedDefaultProviders
mechanism (a per-user DB write fired on startup and inside the read-only
SettingsProviders resolver) -- the config already reaches the UI through
DefaultProvidersConfig without any DB rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 08:30:10 +07:00
Dmitry Ng 879e87c2c2 fix(installer): skip destructive compose ops when compose file is missing
Remove/Purge/FactoryReset failed with "file does not exist" for stacks
that were never extracted (no docker-compose-*.yml). Now such stacks are
skipped instead of aborting the whole operation.

- Add composeFileExists check before destructive compose commands
- Add isDestructiveComposeOperation helper
2026-05-31 15:41:54 +03:00
Mriganka 6bea1581f6 fix(installer): propagate swallowed GatherUpdatesInfo errors
Problem:
In the installer processor, post-operation state refresh failures from
checker.GatherUpdatesInfo were silently discarded, so apply/install/
update/remove/purge could report success even when the refresh failed.

Root cause:
Two distinct shadowing/fall-through patterns wrapped the error into a
variable that was never returned:
- In the deferred closures of applyChanges/install/update/purge, the
  inner `if err := GatherUpdatesInfo(ctx); err != nil` shadowed the
  named return `err`, so `err = fmt.Errorf(...)` assigned the shadow and
  the wrap never reached the `sendCompletion(stack, err)` defer.
- In the ProductStackWorker cases of remove/purge, the block-local err
  was wrapped but the case fell through without returning, while every
  sibling check in the same switch returns on error.

Solution:
- Deferred closures: wrap into a `gatherErr` local and assign the named
  return `err`, so the failure propagates to sendCompletion.
- Switch cases: `return` the wrapped error, matching the sibling checks.

Testing:
go build ./cmd/installer/...        -> OK
go vet ./cmd/installer/processor/...  -> OK
go test ./cmd/installer/processor/... -> ok
golangci-lint (ineffassign) on logic.go -> clean
2026-05-31 11:53:15 +05:30
Dmitry Ng e674113745 Merge pull request #317 from mason5052/codex/issue-314-deepseek-v4-models
fix(deepseek): update default model names to DeepSeek V4
2026-05-28 00:09:25 +04:00
mason5052 20b0633521 docs(deepseek): address Copilot review feedback
- README: align supported-models intro with the local convention used
  by every other provider section ("Models marked with `*` are used
  in default configuration"), so the asterisk on each model ID has a
  near-by explanation.
- Installer help (`LLMFormDeepSeekHelp`): swap the legacy
  "DeepSeek-Chat" / "DeepSeek-Reasoner" bullets in "Default PentAGI
  Models" for the current `deepseek-v4-flash` / `deepseek-v4-pro`
  defaults so the wizard guidance matches the bundled config.

No code, schema, or LiteLLM prefix behavior changes.
2026-05-22 11:41:09 -04:00
mason5052 3113ff3aa3 fix(deepseek): update default model names to DeepSeek V4
The DeepSeek provider config still defaulted to the legacy
`deepseek-chat` and `deepseek-reasoner` model names, which the
upstream DeepSeek API has announced for deprecation on 2026-07-24.
A first-run install therefore breaks once the legacy names are
removed.

Swap the defaults to the current DeepSeek V4 family:

- non-thinking roles use `deepseek-v4-flash`
- reasoning-heavy roles use `deepseek-v4-pro`

The change is limited to the embedded `config.yml` / `models.yml`
inside `backend/pkg/providers/deepseek`, the matching example at
`examples/configs/deepseek.provider.yml`, the `DeepSeekAgentModel`
fallback constant in `deepseek.go`, and three doc references
(README.md, backend/docs/config.md, backend/docs/llms_how_to.md)
plus one installer help string in
`backend/cmd/installer/wizard/locale/locale.go`. LiteLLM prefix
behavior is untouched.
2026-05-21 15:44:30 -04:00
Dmitry Ng 39f122467d feat(config): add new embedding and rename database connection pool settings
- Introduced `EMBEDDING_MAX_TEXT_BYTES` to limit the maximum byte size of text sent to the embedding model.
- Renamed database connection pool settings: `DATABASE_MAX_OPEN_CONNS`, `DATABASE_MAX_IDLE_CONNS`, and `DATABASE_VECTOR_MAX_CONNS` for improved PostgreSQL connection management.
- Updated relevant documentation to reflect these new configuration options and their usage.
- Adjusted various components to utilize the new settings for enhanced performance and resource management.
2026-05-18 18:26:52 +03:00
Dmitry Ng 2ce863ec1a feat(toolcall): implement ToolCall logging functionality
- Added ToolCallLogProvider interface with methods for logging tool calls, updating success and failure statuses.
- Introduced proxyToolCallLogProvider to handle ToolCall logging operations.
- Updated flow execution components to integrate ToolCall logging, including flow workers and controllers.
- Enhanced GraphQL schema to support ToolCall logs, including queries and subscriptions for real-time updates.
- Updated documentation to reflect the new ToolCall logging features and their usage.
2026-05-18 11:21:56 +03:00
Dmitry Ng 077ddce476 feat(database): enhance PostgreSQL connection pooling and configuration
- Introduced shared connection pooling for PostgreSQL using `*sql.DB` for sqlc and GORM, optimizing resource usage.
- Added new environment variables: `DB_MAX_OPEN_CONNS`, `DB_MAX_IDLE_CONNS`, and `DB_VECTOR_MAX_CONNS` for configurable connection limits.
- Updated documentation to reflect new connection pooling strategy and provide operational commands for monitoring.
- Implemented shared `pgxpool` for pgvector stores to reduce connection overhead and improve performance.
- Adjusted various components to utilize the new connection pooling setup, ensuring efficient database interactions.
2026-05-18 11:13:43 +03:00
Dmitry Ng 3a52079278 feat(knowledge): add pgvector knowledge base management
- GraphQL/REST CRUD + semantic search for knowledge documents
- KnowledgeStore with admin/user-scoped filtering, re-embedding on update
- Real-time subscriptions (created/updated/deleted) per user and admin
- user_id tracking in all agent-stored documents (guide/answer/code/memory)
- sqlc queries, goose migrations, privilege grants, user_id backfill
- Memory cleanup on flow deletion; stale orphan purge via migration
- Unit tests for all KnowledgeStore operations including security cases
- Frontend GraphQL schema and TypeScript types regenerated
2026-05-05 01:09:20 +03:00
Dmitry Ng 46c7807af9 fix(main): router initialization
- Updated router initialization to include the Docker client
2026-05-03 00:47:04 +03:00
Dmitry Ng 72b1c8489e feat: implement flow file management with upload and retrieval capabilities
- Added new endpoints for managing flow files, including listing, uploading, and deleting files within flow workspaces.
- Introduced FlowFile model to represent file metadata.
- Enhanced GraphQL schema to support flow file operations and subscriptions for real-time updates.
- Updated API documentation to reflect new flow file functionalities.
2026-04-27 10:03:42 +03:00
Dmitry Ng ec449dd414 fix: update troubleshooting instructions for Docker Compose installation 2026-04-22 20:47:53 +03:00
Dmitry Ng 329c9ae1d0 Merge pull request #265 from mason5052/codex/issue-254-compose-check-alignment
fix: align compose checks with docker compose
2026-04-22 21:35:45 +04:00
Dmitry Ng 78e46bd77a refactor: update TERMINAL_TOOL_TIMEOUT to 1200 seconds with detailed documentation
- Changed default terminal tool timeout from 600 to 1200 seconds.
- Updated related documentation across .env.example, README.md, and config files to reflect the new timeout settings and their constraints.
- Enhanced descriptions in code comments and documentation to clarify timeout behavior, including clamping rules for values outside the accepted range.
2026-04-22 13:22:32 +03:00
Mason Kim(ZINUS US_SALES) 92232f5755 fix: accept compose version output on error
Signed-off-by: Mason Kim(ZINUS US_SALES) <mkim@zinus.com>
2026-04-15 17:51:28 -04:00
Mason Kim(ZINUS US_SALES) eea0a86c85 fix: align compose checks with docker compose
Signed-off-by: Mason Kim(ZINUS US_SALES) <mkim@zinus.com>
2026-04-15 16:18:36 -04:00
Mason Kim(ZINUS US_SALES) 1d6da349d4 fix: clamp oversized terminal timeout 2026-04-15 13:14:16 -04:00
Mason Kim(ZINUS US_SALES) 86d7666c86 feat: add configurable terminal tool timeout
Signed-off-by: Mason Kim(ZINUS US_SALES) <mkim@zinus.com>
2026-04-15 12:34:03 -04:00
Dmitry Ng 7c25f356ed feat: enable runtime provider switching
Adds conditional chain normalization in processChain to preserve reasoning cache when provider unchanged while fixing incomplete tool_calls and converting IDs when switching providers. Extends GraphQL API with modelProvider parameter for seamless provider changes without restart.
2026-04-08 23:58:56 +03:00
Dmitry Ng c8cd0e68f9 feat: add Docker host network mode support and improve agent terminal execution
- Add host network mode support in Docker client (DOCKER_NETWORK=host)
- Update documentation for network modes (bridge vs host)
- Enhance OOB port allocation guidance with mandatory directives
- Improve terminal command execution descriptions (detach, timeout)
- Fix MSF workflow issues: add process isolation rules and RPC daemon patterns
- Add terminal execution mechanics to adviser prompts for better monitoring
- Update installer locale with host network mode explanation

Fixes agent issues with msfconsole hanging, port conflicts, and process isolation.
2026-03-29 15:53:30 +03:00
Dmitry Ng b90ea4711e repo final state 2026-03-26 06:16:07 +03:00