lee/pentagi
1
0
Fork 0
mirror of https://github.com/vxcontrol/pentagi.git synced 2026-08-25 12:36:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add Kubernetes deployment compatibility RFC (#324)

Browse Source 
Add examples/proposals/kubernetes_deployment.md, an RFC-style design
surface responding to the Kubernetes deployment request in #324. The
document is docs-only: no Helm charts, manifests, operator, compose,
installer, or environment-variable changes, and it does not claim
Kubernetes is supported today.

It records the current Compose/installer deployment assumptions, maps
each one (secrets/config, volumes, service discovery, ingress/TLS,
health checks, network policies, the Docker-socket flow executor,
observability, image overrides, migrations) to candidate Kubernetes
equivalents, and proposes an incremental, docs-first path with open
questions, security/operational considerations, and a test/validation
strategy. The hardest item -- the Docker-socket worker executor -- is
laid out as candidate options without choosing one, and keeps flow
lifecycle explicit and inspectable per the #268 review lesson.

Refs #324
This commit is contained in:
mason5052
2026-06-01 17:16:17 -04:00
parent 879e87c2c2
commit 6141975338
1 changed files with 360 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/proposals/kubernetes_deployment.md
+360
View File
@@ -0,0 +1,360 @@
# Kubernetes Deployment RFC
## Summary
Issue [#324](https://github.com/vxcontrol/pentagi/issues/324) asks
whether PentAGI can run on Kubernetes. Today PentAGI is built and
documented around Docker Compose and the installer, and there is no
supported Kubernetes path. This RFC sketches what a future,
incremental Kubernetes-compatibility effort could look like and names
the parts of the current design that make it non-trivial.
This document does not implement runtime behavior. It does not add
Helm charts, Kubernetes manifests, Kustomize bases, an operator, or
CRDs. It does not change `docker-compose.yml`, the installer, the
backend, the database schema, or any environment variable. It does
not claim that PentAGI runs on Kubernetes today, because it does not.
It is a design surface for maintainers to push back on before any
deployment code lands.
The RFC is intentionally staged and docs-first. PentAGI's flow
executor currently talks to a Docker daemon over a bind-mounted
socket, and that single fact drives most of the difficulty below. A
naive "wrap the containers in a Deployment" approach would either
break flow execution or smuggle in implicit, hard-to-inspect
lifecycle behavior -- close to the patterns pushed back on during PR
[#268](https://github.com/vxcontrol/pentagi/pull/268) review. The
proposed path below stays explicit and reviewable: every Kubernetes
resource a future implementation would create should be something an
operator can see with `kubectl`, not a hidden background mechanism.
## Goals
- Capture, in one place, the concrete reasons PentAGI does not run on
Kubernetes today, grounded in the current Compose and installer
design rather than in guesswork.
- Map each Compose-era assumption (secrets, volumes, service
discovery, TLS, health, networking, the container executor,
observability, image selection, migrations) to its candidate
Kubernetes equivalent.
- Identify the one genuinely hard problem -- the Docker-socket flow
executor -- and lay out candidate approaches with their trade-offs,
without choosing one.
- Propose an incremental, docs-first path so that any later
implementation can be reviewed in small, self-contained slices.
- Keep operators in control of secrets, persistence, network reach,
and the privilege level of flow execution at every step.
- Give maintainers a single artifact to accept, reject, or reshape
before any chart, manifest, or operator code is written.
## Non-Goals
- This RFC does not add Helm charts, raw manifests, Kustomize
overlays, an operator, or CRDs. No deployment artifact ships with
this document.
- This RFC does not modify `docker-compose.yml`, the installer, or the
current supported deployment path. Compose remains the only
supported deployment model until maintainers decide otherwise.
- This RFC does not add, rename, or change any environment variable,
and does not change any default in `.env.example` or the backend
configuration.
- This RFC does not change the backend, the database schema, the
generated code, the GraphQL or REST surface, or the flow executor.
- This RFC does not propose hidden background orchestration, an
implicit queue, or out-of-band lifecycle state to make Kubernetes
work. Carrying forward the explicit lesson from PR
[#268](https://github.com/vxcontrol/pentagi/pull/268) review: any
future Kubernetes resource (Pod, Job, PVC, Secret) must be visible
and manageable through the standard Kubernetes API, not buried in
process memory.
- This RFC does not claim parity with the Compose deployment. Some
capabilities (notably the privileged Docker-socket executor) may
never map cleanly, and this document does not promise that they
will.
- This RFC does not pick a single executor strategy, a single
ingress controller, a single storage class, or a single secret
backend. Those are deferred to a later implementation RFC.
## Current Deployment Assumptions
This section describes how PentAGI is deployed today, because the
Kubernetes considerations only make sense against the current shape.
Everything here is drawn from `docker-compose.yml`, the installer
docs, and the backend, not from a hypothetical setup.
- **Compose-oriented topology.** The supported deployment is Docker
Compose (directly or via the installer). The core stack is the
`pentagi` backend, a `pgvector` PostgreSQL instance, a `pgexporter`
metrics sidecar, and a `scraper` service. Optional stacks add
Graphiti / Neo4j, Langfuse, and an observability bundle
(OpenTelemetry collector, Grafana, VictoriaMetrics, and friends).
- **The flow executor uses the Docker socket.** This is the central
fact for Kubernetes. The `pentagi` service bind-mounts the host
Docker socket (`${PENTAGI_DOCKER_SOCKET:-/var/run/docker.sock}` to
`/var/run/docker.sock`) and the backend's Docker client connects
via `client.FromEnv`, honoring
`DOCKER_HOST` (default `unix:///var/run/docker.sock`). During a
flow, PentAGI creates and destroys terminal/worker containers
against that daemon. The executor is effectively "talk to a Docker
daemon and spawn sibling containers," not "run one long-lived
process."
- **Elevated privileges by design.** Because the backend drives the
Docker socket, the `pentagi` service runs `user: root:root`
(commented in-file as "while using docker.sock") and carries
Docker-related toggles (`DOCKER_INSIDE`, `DOCKER_NET_ADMIN`,
`DOCKER_GID=998`, `DOCKER_WORK_DIR`). This privilege level is
intrinsic to the current executor, not incidental.
- **Local-named volumes for state.** Persistent state uses Docker
local volumes: `pentagi-data` mounted at `/opt/pentagi/data`,
Postgres data in `pentagi-postgres-data`, plus `pentagi-ssl`,
`scraper-ssl`, and `pentagi-ollama`. These assume a single host
with local volume drivers.
- **Configuration and secrets via `.env`.** Provider keys, the
database DSN, embedding settings, TLS material, and feature toggles
are passed as environment variables sourced from `.env`. There is no
externalized secret store in the default path; the env file is the
source of truth.
- **Service discovery by Compose DNS.** Services find each other by
Compose service name on user-defined bridge networks
(`pentagi-network`, and the optional `observability-network` and
`langfuse-network`). The backend reaches Postgres, the scraper, and
optional services by name.
- **TLS terminates at the backend.** The backend listens on `8443`
and is published to `${PENTAGI_LISTEN_IP:-127.0.0.1}:8443`,
defaulting to loopback. There is no separate ingress or reverse
proxy in the core stack; TLS is handled inside the container.
- **Health via Compose healthchecks.** Ordering uses Compose
`depends_on` with `condition: service_healthy` (for example the
backend waits on `pgvector`). Health is expressed as container
healthchecks, not as orchestrator probes.
- **Database migrations on startup.** The backend embeds its SQL
migrations and runs them with goose at process start
(`goose.Up`). There is no separate migration step; the backend
migrates itself when it boots.
- **Image selection via env override.** The backend image is
`${PENTAGI_IMAGE:-vxcontrol/pentagi:latest}`, and worker/tool images
are similarly overridable. Air-gapped and mirror setups already rely
on these overrides (see the README's note on restricted networks,
Docker mirrors, and proxies).
## Kubernetes Compatibility Considerations
For each Compose-era assumption above, this section names the
candidate Kubernetes equivalent and the friction. Nothing here is a
committed design; it is a map of the problem space.
- **Secrets and configuration.** The `.env` model maps to Kubernetes
`Secret` objects (provider keys, DB credentials, TLS material) and
`ConfigMap` objects (non-secret toggles). This is mostly mechanical.
The open part is whether to keep a flat env-injection model
(`envFrom` a Secret/ConfigMap) or move toward referenced secrets,
and whether to integrate external secret managers. No change to the
variable names themselves is needed.
- **Persistent volumes.** The local-named volumes map to
`PersistentVolumeClaim`s backed by a cluster `StorageClass`. Postgres
state in particular wants a `StatefulSet` with a stable claim, or an
external managed Postgres. The friction is that several volumes today
assume single-host locality and `ReadWriteOnce` semantics; a future
design has to be explicit about access modes and about whether
Postgres is in-cluster or external.
- **Service discovery.** Compose service-name DNS maps cleanly to
Kubernetes `Service` objects and in-cluster DNS. This is among the
lowest-friction items; the backend would address Postgres and the
scraper by Service name instead of Compose name.
- **Ingress and TLS.** Today TLS terminates in the backend on `8443`
bound to loopback. On Kubernetes the candidate is an `Ingress` (or
Gateway API) with TLS via cert-manager, or preserving in-pod TLS and
exposing it through a passthrough Service. The open question is
whether to keep TLS in the backend or move termination to the edge;
both are viable and have different operational profiles.
- **Health checks.** Compose healthchecks and `depends_on` map to
`readinessProbe` and `livenessProbe`. Startup ordering that Compose
expresses with `service_healthy` becomes readiness-gated rollout
plus application-level retry, since Kubernetes does not block one
workload's start on another's health the way Compose does.
- **Network policies.** The implicit isolation of Compose user-defined
networks maps to Kubernetes `NetworkPolicy`. This is an opportunity
to make the currently-implicit segmentation explicit, but it is also
net-new surface that has to be designed rather than translated.
- **Flow / container execution model (the hard problem).** This is the
item that does not translate mechanically. The backend expects a
Docker daemon and spawns sibling containers over the socket.
Kubernetes does not hand workloads a Docker socket, and modern
clusters do not run Docker as the node runtime. Candidate
approaches, each with real trade-offs and none free of cost:
- **Kubernetes-native execution.** Teach the executor to create
ephemeral `Pod`s or `Job`s through the Kubernetes API instead of
Docker containers. Most idiomatic and the most inspectable
(`kubectl get pods/jobs` shows exactly what a flow is running),
but the largest backend change, and it requires an in-cluster
`ServiceAccount` with pod-create RBAC, which is its own risk.
- **Docker-in-Docker sidecar.** Run a DinD daemon next to the
backend and keep the existing socket-based executor. Smallest
backend change, but DinD typically needs a privileged container,
has known stability and storage caveats, and concentrates risk in
one privileged pod.
- **Sandboxed runtimes.** Pair Kubernetes-native execution with a
stronger isolation runtime (gVisor, Kata, sysbox, or similar) for
the worker pods, since flow workers run untrusted, agent-driven
commands. This is a hardening layer on top of native execution,
not an alternative to it.
Whatever is chosen, the PR
[#268](https://github.com/vxcontrol/pentagi/pull/268) lesson
applies: the running work must be visible and manageable through
standard Kubernetes objects, not tracked only inside the backend
process.
- **Observability.** The optional OpenTelemetry / Grafana /
VictoriaMetrics stack maps to in-cluster deployments or, more
likely, to whatever the operator's cluster already runs. The
candidate direction is to make PentAGI emit to existing cluster
observability rather than bundling its own, with the bundled stack
as an opt-in for clusters that have none.
- **Image overrides.** The existing `PENTAGI_IMAGE` and related
per-image overrides map directly to image fields in pod specs, which
is helpful for air-gapped and mirror deployments. This is
low-friction and reuses an existing mechanism rather than inventing
one.
- **Upgrade and migration path.** Because the backend runs goose
migrations on startup, a rolling update could run migrations from
whichever replica starts first. On Compose with a single backend
this is fine; on Kubernetes with multiple replicas it is not. A
future design needs an explicit decision: a one-shot migration
`Job` (or init container) gated ahead of the rollout, or an
enforced single-writer constraint. This must be settled before any
multi-replica backend deployment is suggested.
## Proposed Incremental Path
The path is deliberately docs-first so each step is small enough to
review and reject in isolation. No step below is started by this RFC;
this is the proposed sequence, not a commitment.
1. **This RFC.** Land the design surface, confirm the boundaries
(docs-only, no charts, no executor change yet), and let maintainers
accept, reshape, or decline the direction.
2. **Executor strategy decision.** Before any manifest exists, settle
the single hardest question in a follow-up RFC: how flow workers
run on Kubernetes (native Pods/Jobs vs DinD vs sandboxed runtime),
and what privilege and RBAC that implies. Everything else depends
on this.
3. **Stateless-core reference manifests.** Once the executor decision
exists, a minimal, clearly-labeled reference for the parts that do
translate cleanly -- backend Deployment/Service, Postgres via
StatefulSet or external, Secrets/ConfigMaps, probes, a migration
Job -- explicitly marked experimental and excluding flow execution.
4. **Flow execution on the chosen model.** Implement the executor
decision from step 2 behind the existing Docker path, so Compose
keeps working unchanged and Kubernetes execution is additive and
opt-in.
5. **Packaging and operator guide.** Only after the above is proven,
consider a Helm chart or operator and a Kubernetes operator guide
(in the spirit of the existing `examples/` material), so packaging
lands on top of a working deployment rather than ahead of it.
Each step is self-contained: maintainers can stop after any step
without leaving PentAGI in a half-migrated state, and Compose remains
the supported path throughout.
## Open Questions
- Which executor model should PentAGI target first -- Kubernetes-native
Pods/Jobs, a DinD sidecar, or a sandboxed runtime -- and is more than
one worth supporting?
- Should Postgres (and pgvector) run in-cluster as a StatefulSet, or
should the Kubernetes path assume an external managed database?
- Should TLS continue to terminate in the backend, or move to an
Ingress / Gateway with cert-manager?
- How should the startup goose migration be handled under multiple
backend replicas -- a gating migration Job, an init container, or an
enforced single-writer?
- What RBAC is acceptable for the backend's ServiceAccount if it
creates worker Pods/Jobs, and how is that least-privileged?
- Should the observability stack be bundled, or should PentAGI default
to emitting into the operator's existing cluster observability?
- Is Helm, an operator, or plain manifests the right packaging once a
working deployment exists, and which should ship first?
- How should air-gapped and mirror deployments be expressed on
Kubernetes, reusing the existing image-override mechanism?
- What is the minimum Kubernetes version and feature set
(StorageClass, Ingress/Gateway, NetworkPolicy support) a future
reference deployment should assume?
## Security and Operational Considerations
Moving PentAGI onto Kubernetes changes its security posture, and the
changes should be designed in rather than discovered later.
- **Privilege of the executor.** The current model effectively grants
the backend host-level container control via the Docker socket. Any
Kubernetes equivalent (pod-create RBAC, a privileged DinD sidecar,
or a sandboxed runtime) carries comparable or different risk. The
privilege level must be explicit, least-privilege, and visible to
operators -- not an unstated side effect of "making it work."
- **RBAC and namespacing.** If the backend creates worker Pods/Jobs,
its ServiceAccount needs scoped permissions in a dedicated
namespace, never cluster-admin. Flow workers should be confined to
that namespace with their own constrained ServiceAccount.
- **Untrusted workloads.** Flow workers run agent-driven, untrusted
commands. On Kubernetes that argues for pod security standards,
seccomp/AppArmor profiles, dropped capabilities, and a sandboxed
runtime for worker pods, rather than running them as ordinary
privileged pods.
- **Secret handling.** Kubernetes `Secret`s are base64, not encrypted,
at rest by default. A future design should call out
encryption-at-rest, optional external secret managers, and the fact
that provider keys and the DB DSN are sensitive. No secret should be
baked into an image or committed to a manifest.
- **Network segmentation.** The implicit Compose-network isolation
should be reproduced with explicit `NetworkPolicy`, defaulting to
deny and opening only the required backend-to-Postgres,
backend-to-scraper, and worker egress paths.
- **No unsafe defaults.** Any future reference deployment must not
default to a privileged or host-network pod, must not expose the
backend publicly without TLS, and must not widen RBAC for
convenience. The Compose default already binds the backend to
loopback; the Kubernetes default should be equally conservative.
- **Inspectable lifecycle.** Per the PR
[#268](https://github.com/vxcontrol/pentagi/pull/268) lesson, flow
execution state on Kubernetes should be representable as real
objects an operator can list and delete, so a stuck or runaway flow
is visible and stoppable through the cluster API rather than only
through backend internals.
## Test and Validation Strategy
A future implementation should be validated against the points below
before being described as anything more than experimental. This RFC
itself is validated only as documentation.
- **Local clusters.** Bring-up and teardown on kind and minikube as
the baseline developer-facing validation, since they need no cloud
account.
- **Manifest and chart linting.** If/when manifests or a chart exist,
`kubectl apply --dry-run=server`, `kubeconform` (or equivalent), and
`helm lint` / `helm template` in CI before anything is published.
- **Migration validation.** Verify the chosen migration approach is
safe under a rolling update with more than one backend replica, so
goose does not run concurrently from multiple pods.
- **End-to-end flow test.** Run at least one real flow on the chosen
executor model and confirm worker Pods/Jobs are created, complete,
are cleaned up, and are visible via `kubectl` for their lifetime.
- **Security review.** Run pod security and RBAC checks (for example
with a policy linter) to confirm least-privilege, deny-by-default
network policies, and no privileged or host-network defaults.
- **Compose parity guard.** Confirm the existing Docker Compose path
is unchanged and still the supported default, so the Kubernetes work
remains additive and opt-in throughout.
## References
- Issue [#324](https://github.com/vxcontrol/pentagi/issues/324):
Kubernetes deployment request.
- PR [#268](https://github.com/vxcontrol/pentagi/pull/268): source of
the explicit-lifecycle / no-hidden-state lesson carried forward
here.
- `docker-compose.yml`: current service topology, the Docker-socket
mount, the `root:root` executor, named volumes, and networks
described in "Current Deployment Assumptions."
- The README sections on Docker image configuration and on restricted
networks, Docker mirrors, and proxies: the existing image-override
mechanism reused under "Image overrides."