Commit Graph
86 Commits
Author SHA1 Message Date
Dmitry Ng a348e41a31 feat(settings): add version and isDevelopMode fields to Settings model and GraphQL schema
- Introduced new fields `version` and `isDevelopMode` in the Settings model to provide application versioning and development mode status.
- Updated GraphQL schema and resolvers to support the new fields, ensuring they are accessible via the Settings query.
- Enhanced Swagger documentation to reflect the changes in the Settings API endpoint.
- Added necessary validation and response handling for the new fields in the Settings service.
2026-05-08 13:06:07 +03:00
Dmitry Ng 6890584c64 feat(resources): add multi-source move, copy and flow-file promotion
- Add Sources []string to MoveResourceRequest, CopyResourceRequest and AddResourceFromFlowRequest; merged with Source, deduplicated; multi-source uses destination as base dir and runs in a single atomic DB transaction
- Fix MoveResource response to return Added + Updated (not Updated only) so Apollo cache receives new parent directory entries alongside moved items
- Add missing errResourceNotFound case in CopyResource (was 500 instead of 404)
- Cover all new behaviour with table-driven tests (basename conflict, force overwrite, missing source, empty input, dir-into-itself guard, etc.)
2026-05-06 17:03:13 +03:00
Dmitry Ng 3a52079278 feat(knowledge): add pgvector knowledge base management
- GraphQL/REST CRUD + semantic search for knowledge documents
- KnowledgeStore with admin/user-scoped filtering, re-embedding on update
- Real-time subscriptions (created/updated/deleted) per user and admin
- user_id tracking in all agent-stored documents (guide/answer/code/memory)
- sqlc queries, goose migrations, privilege grants, user_id backfill
- Memory cleanup on flow deletion; stale orphan purge via migration
- Unit tests for all KnowledgeStore operations including security cases
- Frontend GraphQL schema and TypeScript types regenerated
2026-05-05 01:09:20 +03:00
Dmitry Ng ca9f4a0211 feat: add multi-path support and ZIP improvements across file APIs
- Added `paths[]` query/body parameter to DeleteFlowFile, DownloadFlowFile,
  GetFlowContainerFiles, PullFlowFiles, ListResources, DeleteResource, and
  DownloadResource; single `path` parameter retained for backward compatibility.
- Introduced `DeduplicatePaths` in flowfiles package with coverage-based
  deduplication (parent covers children), path normalization, and traversal safety.
- Added `ZipRelativePaths` to create ZIP archives from cache-relative paths,
  sharing `zipWriteFile` helper with refactored `ZipDirectory`.
- Switched all ZIP and single-file responses to buffered `DataFromReader` with
  explicit `Content-Length`, fixing Swagger UI download rendering.
- Expanded response payloads: delete and pull operations now enumerate all
  affected nested files; list responses include ancestor directories for tree
  completeness.
- Extended test coverage across flowfiles, flow_files, and resources packages
  with batch, deduplication, atomicity, Docker exec, and security scenarios.
2026-05-04 14:32:39 +03:00
Dmitry Ng f57e988586 feat(cast): add JSON control character sanitization for tool call arguments
- Implemented `SanitizeJSONControlChars` to escape literal control characters in JSON string values, ensuring compliance with the JSON specification.
- Enhanced `SanitizeToolCallArguments` method to apply sanitization across tool call arguments in the chain.
- Added comprehensive tests for both sanitization functions to validate behavior with various input scenarios.
2026-05-03 00:48:47 +03:00
Dmitry Ng 9cd52b102a fix(csum): prevent out-of-range errors in recent section determination
- Updated the logic in `determineRecentSectionsToKeep` to clamp the lower bound of the recent sections to keep, ensuring it does not exceed the available sections. This prevents potential out-of-range errors when `keepQASections` is greater than the total number of sections.
2026-05-03 00:47:47 +03:00
Dmitry Ng 46c7807af9 fix(main): router initialization
- Updated router initialization to include the Docker client
2026-05-03 00:47:04 +03:00
Dmitry Ng e370450dab feat(browser): enhance HTML and MD content handling with warnings for small content and errors for binary URLs
- Updated `getHTML` and `getMD` methods to return warnings for small content instead of errors.
- Implemented checks for binary URLs, returning descriptive errors when such URLs are encountered.
- Added new tests to validate the updated behavior for small and empty content handling.
2026-05-03 00:45:27 +03:00
Dmitry Ng b254ff6f90 refactor(flow_manager): improve error handling for running tasks 2026-05-03 00:43:51 +03:00
Dmitry Ng 75eb8e0f1e fix(aslog): implement TryLock to prevent deadlock in workerMsgUpdater
- Added TryLock mechanism to avoid deadlock situations when reading from the channel while the mutex is held.
- Enhanced timer handling to reset when the mutex is not available, ensuring continuous operation of the stream.
2026-05-03 00:42:17 +03:00
Dmitry Ng 8830ae7010 fix: update resource handling and API documentation
- Adjusted descriptions in API documentation to clarify paths for uploads, resources, and containers.
- Updated data models to use consistent naming conventions (e.g., `isDir` to `is_dir`, `modifiedAt` to `modified_at`).
- Changed resource ID types from string to uint64 for better consistency across the application.
- Enhanced flow file upload functionality to support batch processing of resource files.
- Removed deprecated code related to resource entry responses in tests.
2026-05-01 16:11:09 +03:00
Dmitry Ng 629c018a68 fix: bug with retrieving resources recursive via graphql 2026-05-01 16:09:06 +03:00
Dmitry Ng b6f7fd7ef8 feat: enhance task and subtask handling with interruption management 2026-05-01 16:05:12 +03:00
Dmitry NgandOctopus 956c11eadc feat: introduce engagement-log/technical-channel language policy across agent prompts
- Replaced ambiguous "user's language" guidance in tools/args.go with explicit engagement-log vs technical-channel markers per field, with strong English-only requirement for vector-store and search-engine queries.
- Added a unified LANGUAGE POLICY block to every agent prompt (primary_agent, assistant, pentester, coder, installer, searcher, memorist, generator, refiner, reporter, enricher), tailored per agent based on its actual tool set.
- Extended template variables and tool access (TerminalToolName, FileToolName) for coder, pentester, installer, memorist, generator, refiner, and enricher to match their runtime tool registrations.
- Fixed inverted UseAgents condition and removed misleading vector-store write references in assistant prompt; corrected MEMORY SYSTEM INTEGRATION for mode-specific tool references.
- Compressed COMPLETION REQUIREMENTS across templates and aligned closing-tool guidance with the channel mapping (engagement-log message vs technical-channel result).

Fixes #285.

Co-Authored-By: Octopus <liyuan851277048@icloud.com>
2026-04-30 15:29:56 +03:00
Dmitry Ng c0035ac130 Merge pull request #293 from mason5052/codex/issue-291-fix-processor-doc-links
docs: fix processor wizard integration links
2026-04-30 12:03:07 +04:00
Dmitry Ng 47de4e44e6 docs: update links in configuration and flow execution documents, add directory for proposals 2026-04-30 10:09:03 +03:00
MasonandMason Kim 6257612689 docs: add OSINT integration scenarios (#278)
* docs: add OSINT integration scenarios

* docs: clarify OSINT scenario payloads

* docs: clarify OSINT provider identifiers

* docs: distinguish OSINT provider ids

---------

Co-authored-by: Mason Kim(ZINUS US_SALES) <mkim@zinus.com>
2026-04-30 10:47:08 +04:00
mason5052 29d5723620 docs: fix processor wizard integration links 2026-04-29 10:58:03 -04:00
Dmitry Ng 402dfbe0ee Merge pull request #277 from mason5052/codex/issue-235-evidence-chain-rfc
docs: add evidence chain RFC
2026-04-29 01:35:59 +04:00
Dmitry Ng 425f075a96 Merge pull request #276 from mason5052/codex/issue-71-scope-of-work-template
docs: add scope-of-work pentest template
2026-04-29 01:08:29 +04:00
Dmitry NgandCopilot de6f527e6f Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dmitry Ng <19asdek91@gmail.com>
2026-04-29 01:07:47 +04:00
Dmitry Ng 1d6f842bb9 test: add comprehensive coverage for FlowFileService HTTP handlers
- Added table-driven scenarios for all 8 endpoints (Get/Upload/Delete/Download flow files, Pull from container, GetContainerFiles, AddResourcesToFlow, AddResourceFromFlow) covering success paths, all privilege combinations (view/upload/admin/cross-user), error responses (forbidden/not-found/conflict/invalid request), and security checks (path traversal, symlink rejection).
- Introduced reusable test infrastructure: sqlite-backed flows/user_resources schema, fakeDockerClient implementing the full docker.DockerClient interface, flowFileCaptureSubscriptions recording both FlowPublisher and ResourcePublisher events, and helpers for multipart upload bodies and container TAR fixtures.
- Added direct unit tests for shellQuote, parseFlowIDParam, cleanupPendingUploads, and flowScopeForFiles privilege matrix.
- Lifts handler coverage from 0% to 60-90% across the file and total services package coverage from 10.2% to 28.6%.
2026-04-28 22:48:23 +03:00
Dmitry Ng 452a2d51e2 fix: expand ResourceService coverage with comprehensive scenarios
- Restructured single-case tests for List, Mkdir, Download, and CleanupOrphanBlobs into table-driven scenarios.
- Extended Upload, Delete, Copy, and Move scenarios with admin, forbidden, malformed JSON, invalid path, missing source, and trailing-slash hint cases.
- Added dedicated tests for blob deduplication on upload/copy, non-multipart bodies, orphan blob preservation on rollback, and direct deleteOrphanBlob coverage.
- Introduced helpers for parameterized multipart field names, custom UID context, and on-disk blob counting.
2026-04-28 22:02:49 +03:00
Dmitry Ng aa4b70eaba feat: add user resources system and flow integration
- UserResource model with MD5-deduplicated blob storage and virtual path filesystem
- REST API for resource CRUD (upload, mkdir, move, copy, delete, download)
- GraphQL query/mutations with resourceIds support on createFlow, putUserInput, createAssistant, callAssistant
- Resource → flow copy with hierarchy restore; incremental container sync (find missing, copy once)
- FlowWorker.PutResources delegates copy, docker push and flowFileAdded events
- Agent prompts updated with {{.UserFiles}} XML listing of /work/uploads and /work/resources
- Resource subscriptions: resourceAdded/Updated/Deleted
2026-04-28 17:00:17 +03:00
Dmitry Ng 7c4ebda2c2 feat: enhance Docker client with container file operations and API integration
- Added methods for non-recursive directory listing and file stat operations in the Docker client.
- Implemented a new API endpoint to retrieve files from a running container's directory.
- Updated documentation to reflect new file operations and API changes.
- Introduced data structures for container file metadata and integrated them into the flow file service.
- Enhanced flow file management capabilities with improved synchronization between local and container file systems.
2026-04-27 13:19:39 +03:00
Dmitry Ng 72b1c8489e feat: implement flow file management with upload and retrieval capabilities
- Added new endpoints for managing flow files, including listing, uploading, and deleting files within flow workspaces.
- Introduced FlowFile model to represent file metadata.
- Enhanced GraphQL schema to support flow file operations and subscriptions for real-time updates.
- Updated API documentation to reflect new flow file functionalities.
2026-04-27 10:03:42 +03:00
Dmitry Ng 7cdb3a61cb Merge pull request #272 from mason5052/codex/issue-193-flow-file-uploads-v1
feat: add flow-scoped file uploads
2026-04-24 04:03:39 +04:00
Dmitry Ng 5e902ca255 docs: update API tokens section to reflect new naming convention in settings 2026-04-24 03:00:48 +03:00
Dmitry Ng 378db10732 Merge pull request #270 from mason5052/codex/issue-191-settings-coverage
docs: clarify current web settings coverage
2026-04-24 03:52:41 +04:00
Dmitry Ng 5067e8f5a4 feat: add assistant flow management tools and summarizer cache
- new get_flow_status / stop_flow / submit_flow_input / patch_flow_subtasks tools in assistant executor, backed by FlowWorker callbacks
- flowStatusTool supports 5 detail levels with verbose mode, polling for task readiness, and per-size summarization
- summarizer LRU cache (1000 entries, 4 h TTL, SHA-256 key) on flowProvider to skip redundant LLM calls
- updated assistant.tmpl with full flow management protocol: state reference, decision guide, constraints
- updated flow_execution.md with terminal timeout config, new tools, and cache
2026-04-24 02:37:09 +03:00
Dmitry Ng 172fb8754f Merge pull request #267 from mason5052/codex/issue-74-memory-lifecycle-docs
docs: clarify memory lifecycle across flows
2026-04-22 22:15:23 +04:00
Dmitry Ng 6919511022 Merge pull request #266 from mason5052/codex/issue-79-pentest-methodology-guide
docs: add pentesting prompt methodology guidance
2026-04-22 21:49:33 +04:00
Dmitry Ng ec449dd414 fix: update troubleshooting instructions for Docker Compose installation 2026-04-22 20:47:53 +03:00
Dmitry Ng 329c9ae1d0 Merge pull request #265 from mason5052/codex/issue-254-compose-check-alignment
fix: align compose checks with docker compose
2026-04-22 21:35:45 +04:00
mason5052 31a502ff73 docs: add evidence chain RFC 2026-04-22 13:07:27 -04:00
mason5052 5bc534c590 docs: add scope-of-work pentest template 2026-04-22 13:05:40 -04:00
Dmitry Ng 239dd31c91 feat: add GetUsageStatsByModelAgentsForFlow query and related GraphQL types
- Implemented a new SQL query to retrieve usage statistics by model and agents for a specific flow.
- Added corresponding Go types and methods to handle the new query in the database layer.
- Updated GraphQL schema to include the new ModelAgentsUsageStats type and the usageStatsByModelAgentsForFlow query.
- Enhanced frontend components to display the new usage statistics in the flow dashboard.
2026-04-22 19:51:20 +03:00
mason5052 c893bb817e docs: fix pentesting methodology links
Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-04-22 12:36:18 -04:00
mason5052 aa718e2330 docs: clarify settings config paths
Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-04-22 12:36:12 -04:00
mason5052 a285a99e28 docs: tighten memory lifecycle guidance
Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-04-22 12:36:08 -04:00
mason5052 263eaf56ab fix: harden flow file uploads
Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-04-22 12:35:55 -04:00
Dmitry Ng fa78acc04c Merge remote-tracking branch 'origin/feature/frontend' into feature/next-release 2026-04-22 16:00:32 +03:00
Dmitry Ng 897bb7ec53 chore: update pgx dependency to version 5.8.0 in go.mod and go.sum 2026-04-22 15:04:51 +03:00
Dmitry Ng 8fed9a8cd8 docs: clarify proxy configuration and Docker pull test in network validation documentation 2026-04-22 14:46:45 +03:00
Dmitry Ng 246abf7367 Merge pull request #262 from mason5052/codex/issue-258-docker-mirror-docs
docs: add Docker mirror guidance for restricted networks
2026-04-22 15:30:51 +04:00
Dmitry Ng 78e46bd77a refactor: update TERMINAL_TOOL_TIMEOUT to 1200 seconds with detailed documentation
- Changed default terminal tool timeout from 600 to 1200 seconds.
- Updated related documentation across .env.example, README.md, and config files to reflect the new timeout settings and their constraints.
- Enhanced descriptions in code comments and documentation to clarify timeout behavior, including clamping rules for values outside the accepted range.
2026-04-22 13:22:32 +03:00
Dmitry Ng 0e7c238cad Merge pull request #261 from mason5052/codex/issue-256-terminal-timeout
feat: add TERMINAL_TOOL_TIMEOUT configuration
2026-04-21 15:33:33 +04:00
Dmitry Ng 4294eac935 docs: update authentication settings in configuration documentation to reflect changes in OAuth client initialization 2026-04-21 13:19:27 +03:00
Dmitry Ng 25524cb271 Merge pull request #260 from mason5052/codex/issue-219-oauth-docs
docs: clarify OAuth callback setup for GitHub and Google
2026-04-21 13:29:42 +04:00
mason5052 4d7c3678a9 feat: add flow-scoped file uploads 2026-04-19 14:27:06 -04:00