Commit Graph
55 Commits
Author SHA1 Message Date
Dmitry Ng 2f4da118e0 feat(anonymization): implement anonymizeText mutation and associated service
- Added a new SQL migration to insert the 'anonymize.call' privilege into the privileges table.
- Introduced the `anonymizeText` mutation in the GraphQL schema, allowing users to anonymize sensitive text.
- Implemented the `AnonymizerService` to handle text anonymization requests via a REST API endpoint.
- Updated the GraphQL resolver to integrate the new mutation and ensure proper permission checks.
- Enhanced documentation with Swagger and OpenAPI specifications for the new endpoint.
- Added error handling for invalid requests and unavailable anonymizer configurations.
2026-05-12 21:47:40 +03:00
Dmitry Ng fbf917a18a fix(aslog, msglog): remove message length truncation logic
Eliminate the message length truncation from both `putMsg` methods in `aslog.go` and `msglog.go`. This change simplifies the message handling process by allowing messages to be processed without arbitrary length restrictions.
2026-05-12 21:37:16 +03:00
Dmitry Ng 0f0a7bd2d0 fix(database/knowledge): replace SimilaritySearch with direct SQLC vector queries
- Fix empty ID bug: langchaingo SimilaritySearch discarded document UUIDs; new SearchKnowledgeDocuments/SearchUserKnowledgeDocuments return them directly
- Remove unsafe fmt.Sprintf SQL filter interpolation, use parameterised queries
- Exclude memory documents from search results at SQL level
- Add FlowID support to passesSearchFilter
- Convert all $N positional params to sqlc.arg(name) across knowledge queries
- Update tests: replace TestBuildSearchFilters with TestPassesSearchFilter, add TestSearchDocuments and TestSearchUserDocuments
2026-05-12 20:47:46 +03:00
Dmitry Ng ca1dfa138c feat(knowledge): enhance document update functionality with docType handling
- Introduced a new `docType` field in the `UpdateKnowledgeDocumentInput` to manage document type changes.
- Updated the `doUpdate` method to clear subtype fields (GuideType, AnswerType, CodeLang) when the document type changes.
- Added comprehensive tests for various document type transitions to ensure correct behavior and state preservation.
- Updated GraphQL schema and generated models to accommodate the new `docType` field.
2026-05-11 10:29:13 +03:00
Dmitry Ng ba0cee43c8 fix(router): add new frontend routes for resources and knowledges to preserve route in the browser
- Introduced "/resources" and "/knowledges" routes to the frontendRoutes array in the router configuration, expanding the application's routing capabilities.
2026-05-08 20:46:47 +03:00
Dmitry Ng a9aeb81e4e refactor(resources): update MoveResourceRequest to allow empty destination
- Removed the requirement for the "destination" field in MoveResourceRequest, allowing it to be an empty string, which signifies moving to the root directory.
- Updated related documentation in swagger.json, swagger.yaml, and docs.go to reflect the new behavior.
- Adjusted the MoveResource function to handle cases where the destination is empty, ensuring proper path sanitization and resource movement semantics.
- Enhanced test cases to cover scenarios involving moving resources to the root directory and handling conflicts appropriately.
2026-05-08 20:24:39 +03:00
Dmitry Ng dc2fd43928 feat(assistants): support flowID=0 in REST API to create assistant with new flow
- Allow POST /flows/0/assistants/ to create a new flow together with the assistant, mirroring the existing GraphQL createAssistant(flowID: 0) behavior
- Require both assistants.create and flows.create permissions when flowID=0
- Add explicit flow ownership check for non-zero flowID using flows.admin scope
- Load flow data in the response by fetching it via assistant.FlowID after creation, ensuring AssistantFlow is fully populated in all cases
2026-05-08 18:54:14 +03:00
Dmitry Ng a348e41a31 feat(settings): add version and isDevelopMode fields to Settings model and GraphQL schema
- Introduced new fields `version` and `isDevelopMode` in the Settings model to provide application versioning and development mode status.
- Updated GraphQL schema and resolvers to support the new fields, ensuring they are accessible via the Settings query.
- Enhanced Swagger documentation to reflect the changes in the Settings API endpoint.
- Added necessary validation and response handling for the new fields in the Settings service.
2026-05-08 13:06:07 +03:00
Dmitry Ng 6890584c64 feat(resources): add multi-source move, copy and flow-file promotion
- Add Sources []string to MoveResourceRequest, CopyResourceRequest and AddResourceFromFlowRequest; merged with Source, deduplicated; multi-source uses destination as base dir and runs in a single atomic DB transaction
- Fix MoveResource response to return Added + Updated (not Updated only) so Apollo cache receives new parent directory entries alongside moved items
- Add missing errResourceNotFound case in CopyResource (was 500 instead of 404)
- Cover all new behaviour with table-driven tests (basename conflict, force overwrite, missing source, empty input, dir-into-itself guard, etc.)
2026-05-06 17:03:13 +03:00
Dmitry Ng 3a52079278 feat(knowledge): add pgvector knowledge base management
- GraphQL/REST CRUD + semantic search for knowledge documents
- KnowledgeStore with admin/user-scoped filtering, re-embedding on update
- Real-time subscriptions (created/updated/deleted) per user and admin
- user_id tracking in all agent-stored documents (guide/answer/code/memory)
- sqlc queries, goose migrations, privilege grants, user_id backfill
- Memory cleanup on flow deletion; stale orphan purge via migration
- Unit tests for all KnowledgeStore operations including security cases
- Frontend GraphQL schema and TypeScript types regenerated
2026-05-05 01:09:20 +03:00
Dmitry Ng ca9f4a0211 feat: add multi-path support and ZIP improvements across file APIs
- Added `paths[]` query/body parameter to DeleteFlowFile, DownloadFlowFile,
  GetFlowContainerFiles, PullFlowFiles, ListResources, DeleteResource, and
  DownloadResource; single `path` parameter retained for backward compatibility.
- Introduced `DeduplicatePaths` in flowfiles package with coverage-based
  deduplication (parent covers children), path normalization, and traversal safety.
- Added `ZipRelativePaths` to create ZIP archives from cache-relative paths,
  sharing `zipWriteFile` helper with refactored `ZipDirectory`.
- Switched all ZIP and single-file responses to buffered `DataFromReader` with
  explicit `Content-Length`, fixing Swagger UI download rendering.
- Expanded response payloads: delete and pull operations now enumerate all
  affected nested files; list responses include ancestor directories for tree
  completeness.
- Extended test coverage across flowfiles, flow_files, and resources packages
  with batch, deduplication, atomicity, Docker exec, and security scenarios.
2026-05-04 14:32:39 +03:00
Dmitry Ng f57e988586 feat(cast): add JSON control character sanitization for tool call arguments
- Implemented `SanitizeJSONControlChars` to escape literal control characters in JSON string values, ensuring compliance with the JSON specification.
- Enhanced `SanitizeToolCallArguments` method to apply sanitization across tool call arguments in the chain.
- Added comprehensive tests for both sanitization functions to validate behavior with various input scenarios.
2026-05-03 00:48:47 +03:00
Dmitry Ng 9cd52b102a fix(csum): prevent out-of-range errors in recent section determination
- Updated the logic in `determineRecentSectionsToKeep` to clamp the lower bound of the recent sections to keep, ensuring it does not exceed the available sections. This prevents potential out-of-range errors when `keepQASections` is greater than the total number of sections.
2026-05-03 00:47:47 +03:00
Dmitry Ng e370450dab feat(browser): enhance HTML and MD content handling with warnings for small content and errors for binary URLs
- Updated `getHTML` and `getMD` methods to return warnings for small content instead of errors.
- Implemented checks for binary URLs, returning descriptive errors when such URLs are encountered.
- Added new tests to validate the updated behavior for small and empty content handling.
2026-05-03 00:45:27 +03:00
Dmitry Ng b254ff6f90 refactor(flow_manager): improve error handling for running tasks 2026-05-03 00:43:51 +03:00
Dmitry Ng 75eb8e0f1e fix(aslog): implement TryLock to prevent deadlock in workerMsgUpdater
- Added TryLock mechanism to avoid deadlock situations when reading from the channel while the mutex is held.
- Enhanced timer handling to reset when the mutex is not available, ensuring continuous operation of the stream.
2026-05-03 00:42:17 +03:00
Dmitry Ng 8830ae7010 fix: update resource handling and API documentation
- Adjusted descriptions in API documentation to clarify paths for uploads, resources, and containers.
- Updated data models to use consistent naming conventions (e.g., `isDir` to `is_dir`, `modifiedAt` to `modified_at`).
- Changed resource ID types from string to uint64 for better consistency across the application.
- Enhanced flow file upload functionality to support batch processing of resource files.
- Removed deprecated code related to resource entry responses in tests.
2026-05-01 16:11:09 +03:00
Dmitry Ng 629c018a68 fix: bug with retrieving resources recursive via graphql 2026-05-01 16:09:06 +03:00
Dmitry Ng b6f7fd7ef8 feat: enhance task and subtask handling with interruption management 2026-05-01 16:05:12 +03:00
Dmitry NgandOctopus 956c11eadc feat: introduce engagement-log/technical-channel language policy across agent prompts
- Replaced ambiguous "user's language" guidance in tools/args.go with explicit engagement-log vs technical-channel markers per field, with strong English-only requirement for vector-store and search-engine queries.
- Added a unified LANGUAGE POLICY block to every agent prompt (primary_agent, assistant, pentester, coder, installer, searcher, memorist, generator, refiner, reporter, enricher), tailored per agent based on its actual tool set.
- Extended template variables and tool access (TerminalToolName, FileToolName) for coder, pentester, installer, memorist, generator, refiner, and enricher to match their runtime tool registrations.
- Fixed inverted UseAgents condition and removed misleading vector-store write references in assistant prompt; corrected MEMORY SYSTEM INTEGRATION for mode-specific tool references.
- Compressed COMPLETION REQUIREMENTS across templates and aligned closing-tool guidance with the channel mapping (engagement-log message vs technical-channel result).

Fixes #285.

Co-Authored-By: Octopus <liyuan851277048@icloud.com>
2026-04-30 15:29:56 +03:00
Dmitry Ng 1d6f842bb9 test: add comprehensive coverage for FlowFileService HTTP handlers
- Added table-driven scenarios for all 8 endpoints (Get/Upload/Delete/Download flow files, Pull from container, GetContainerFiles, AddResourcesToFlow, AddResourceFromFlow) covering success paths, all privilege combinations (view/upload/admin/cross-user), error responses (forbidden/not-found/conflict/invalid request), and security checks (path traversal, symlink rejection).
- Introduced reusable test infrastructure: sqlite-backed flows/user_resources schema, fakeDockerClient implementing the full docker.DockerClient interface, flowFileCaptureSubscriptions recording both FlowPublisher and ResourcePublisher events, and helpers for multipart upload bodies and container TAR fixtures.
- Added direct unit tests for shellQuote, parseFlowIDParam, cleanupPendingUploads, and flowScopeForFiles privilege matrix.
- Lifts handler coverage from 0% to 60-90% across the file and total services package coverage from 10.2% to 28.6%.
2026-04-28 22:48:23 +03:00
Dmitry Ng 452a2d51e2 fix: expand ResourceService coverage with comprehensive scenarios
- Restructured single-case tests for List, Mkdir, Download, and CleanupOrphanBlobs into table-driven scenarios.
- Extended Upload, Delete, Copy, and Move scenarios with admin, forbidden, malformed JSON, invalid path, missing source, and trailing-slash hint cases.
- Added dedicated tests for blob deduplication on upload/copy, non-multipart bodies, orphan blob preservation on rollback, and direct deleteOrphanBlob coverage.
- Introduced helpers for parameterized multipart field names, custom UID context, and on-disk blob counting.
2026-04-28 22:02:49 +03:00
Dmitry Ng aa4b70eaba feat: add user resources system and flow integration
- UserResource model with MD5-deduplicated blob storage and virtual path filesystem
- REST API for resource CRUD (upload, mkdir, move, copy, delete, download)
- GraphQL query/mutations with resourceIds support on createFlow, putUserInput, createAssistant, callAssistant
- Resource → flow copy with hierarchy restore; incremental container sync (find missing, copy once)
- FlowWorker.PutResources delegates copy, docker push and flowFileAdded events
- Agent prompts updated with {{.UserFiles}} XML listing of /work/uploads and /work/resources
- Resource subscriptions: resourceAdded/Updated/Deleted
2026-04-28 17:00:17 +03:00
Dmitry Ng 7c4ebda2c2 feat: enhance Docker client with container file operations and API integration
- Added methods for non-recursive directory listing and file stat operations in the Docker client.
- Implemented a new API endpoint to retrieve files from a running container's directory.
- Updated documentation to reflect new file operations and API changes.
- Introduced data structures for container file metadata and integrated them into the flow file service.
- Enhanced flow file management capabilities with improved synchronization between local and container file systems.
2026-04-27 13:19:39 +03:00
Dmitry Ng 72b1c8489e feat: implement flow file management with upload and retrieval capabilities
- Added new endpoints for managing flow files, including listing, uploading, and deleting files within flow workspaces.
- Introduced FlowFile model to represent file metadata.
- Enhanced GraphQL schema to support flow file operations and subscriptions for real-time updates.
- Updated API documentation to reflect new flow file functionalities.
2026-04-27 10:03:42 +03:00
Dmitry Ng 7cdb3a61cb Merge pull request #272 from mason5052/codex/issue-193-flow-file-uploads-v1
feat: add flow-scoped file uploads
2026-04-24 04:03:39 +04:00
Dmitry Ng 5067e8f5a4 feat: add assistant flow management tools and summarizer cache
- new get_flow_status / stop_flow / submit_flow_input / patch_flow_subtasks tools in assistant executor, backed by FlowWorker callbacks
- flowStatusTool supports 5 detail levels with verbose mode, polling for task readiness, and per-size summarization
- summarizer LRU cache (1000 entries, 4 h TTL, SHA-256 key) on flowProvider to skip redundant LLM calls
- updated assistant.tmpl with full flow management protocol: state reference, decision guide, constraints
- updated flow_execution.md with terminal timeout config, new tools, and cache
2026-04-24 02:37:09 +03:00
Dmitry Ng 239dd31c91 feat: add GetUsageStatsByModelAgentsForFlow query and related GraphQL types
- Implemented a new SQL query to retrieve usage statistics by model and agents for a specific flow.
- Added corresponding Go types and methods to handle the new query in the database layer.
- Updated GraphQL schema to include the new ModelAgentsUsageStats type and the usageStatsByModelAgentsForFlow query.
- Enhanced frontend components to display the new usage statistics in the flow dashboard.
2026-04-22 19:51:20 +03:00
mason5052 263eaf56ab fix: harden flow file uploads
Signed-off-by: mason5052 <ehehwnwjs5052@gmail.com>
2026-04-22 12:35:55 -04:00
Dmitry Ng 78e46bd77a refactor: update TERMINAL_TOOL_TIMEOUT to 1200 seconds with detailed documentation
- Changed default terminal tool timeout from 600 to 1200 seconds.
- Updated related documentation across .env.example, README.md, and config files to reflect the new timeout settings and their constraints.
- Enhanced descriptions in code comments and documentation to clarify timeout behavior, including clamping rules for values outside the accepted range.
2026-04-22 13:22:32 +03:00
mason5052 4d7c3678a9 feat: add flow-scoped file uploads 2026-04-19 14:27:06 -04:00
Mason Kim(ZINUS US_SALES) 1d6da349d4 fix: clamp oversized terminal timeout 2026-04-15 13:14:16 -04:00
Mason Kim(ZINUS US_SALES) 86d7666c86 feat: add configurable terminal tool timeout
Signed-off-by: Mason Kim(ZINUS US_SALES) <mkim@zinus.com>
2026-04-15 12:34:03 -04:00
Dmitry Ng 87ac00f195 test(graphiti): strengthen search method tests with proper types 2026-04-11 15:20:52 +03:00
Dmitry Ng 51385079ed Merge pull request #244 from mason5052/test/graphiti-client-coverage
test: add unit tests for graphiti client disabled mode
2026-04-11 16:13:25 +04:00
Dmitry Ng 6753e2b3d6 test(langfuse): strengthen noop tests with safer assertions 2026-04-11 14:02:21 +03:00
Mason b323a8ace0 test: add unit tests for langfuse context and noop observer (#242)
* test: add unit tests for langfuse context and noop observer

Add tests for observation context put/get round-trip, nested context
override behavior, and noop observer: trace ID generation, parent
inheritance, explicit override, Shutdown, ForceFlush, and enqueue.

* test: add edge case tests for noop observer branch coverage

Add two missing branch tests: explicit ObservationID with no parent
context (verifies new trace generation), and explicit TraceID with
parent (verifies observation ID is NOT inherited when trace is explicit).
2026-04-11 14:43:36 +04:00
Dmitry Ng 10e77fb546 test(langfuse): fix float assertions and strengthen type checks 2026-04-11 13:40:15 +03:00
Dmitry Ng 930830e742 test(schema): improve test clarity with null byte comments 2026-04-11 13:17:55 +03:00
Dmitry Ng 7c25f356ed feat: enable runtime provider switching
Adds conditional chain normalization in processChain to preserve reasoning cache when provider unchanged while fixing incomplete tool_calls and converting IDs when switching providers. Extends GraphQL API with modelProvider parameter for seamless provider changes without restart.
2026-04-08 23:58:56 +03:00
mason5052 166035b3de test: add unit tests for graphiti client disabled mode
Add tests for NewClient with enabled=false, IsEnabled nil/disabled/enabled
receivers, GetTimeout nil/configured, AddMessages no-op on disabled client,
and all 7 search methods returning "graphiti is not enabled" error when
disabled. All tests are hermetic with no network dependency.
2026-04-08 12:06:32 -04:00
mason5052 161b122d91 test: add getCurrentTimeRef test and strengthen ModelParameters assertions
Add missing TestGetCurrentTimeRef coverage. Replace shallow key-presence
checks in ModelParameters.ToLangfuse tests with full value assertions:
verify MapValue string/integer/boolean/list payloads for inf default,
explicit max_tokens, formatted floats, json mode, and stop words.
2026-04-08 11:29:00 -04:00
mason5052 cad6b01ebb test: add unit tests for langfuse helpers package
Add comprehensive unit tests for all utility functions in the langfuse
helpers package: mergeMaps, ObservationLevel, GenerationUsageUnit,
GenerationUsage, ModelParameters, GetLangchainModelParameters, trace/span
ID generation, time helpers, and pointer helpers.
2026-04-08 09:44:01 -04:00
Dmitry Ng 36ac0c876b Merge pull request #234 from octo-patch/fix/issue-220-user-defined-provider-precedence
fix: user-defined providers take precedence over built-in providers
2026-04-08 04:58:07 +04:00
Dmitry Ng fc91758b9c Merge pull request #238 from vxcontrol/feature/frontend
Add Analytics Dashboard and Template Management System
2026-04-08 04:36:05 +04:00
Dmitry Ng 089dcb36b4 feat: implement flow templates management 2026-04-08 03:34:32 +03:00
Dmitry Ng 51cb312b57 Merge pull request #230 from mason5052/test/server-models-validation
test: add unit tests for key server/models validation types
2026-04-08 01:20:43 +04:00
mason5052 1b2681cd99 test: shadow loop variable to prevent parallel subtest capture
Add tt := tt inside each for-range loop before t.Run to ensure
parallel subtests capture their own immutable copy of the test case.
2026-04-06 19:15:29 -04:00
Octopus a12ffe9c01 fix: user-defined providers take precedence over built-in providers (fixes #220) 2026-04-05 09:44:53 +08:00
mason5052 8ddc25f411 test: add failure-path coverage for schema validation
Add missing negative-path tests:
- Schema.Valid() with invalid $ref that fails compilation
- GetValidator() with same invalid schema returns error
- ValidateBytes() with malformed JSON input returns error
2026-03-31 21:54:51 -04:00