lee/pentagi
1
0
Fork 0
mirror of https://github.com/vxcontrol/pentagi.git synced 2026-08-25 04:26:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(worker_node): enhance monitoring setup and documentation for worker node

Browse Source 
- Updated `METRICS_IP` documentation to reflect new metrics ports (8080, 9100) for cAdvisor and node-exporter.
- Added detailed instructions for deploying cAdvisor and node-exporter on the worker node's host Docker, including scripts for setup.
- Enhanced the security and firewall configuration section to include new metrics endpoints.
- Improved overall documentation clarity for metrics integration with PentAGI's observability stack.
This commit is contained in:
Dmitry Ng
2026-08-01 20:16:15 +03:00
parent 339aa958f0
commit f6fe867a99
1 changed files with 132 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/guides/worker_node.md
+132 -5
View File
@@ -62,11 +62,11 @@ Set the private IP address that will be used throughout this setup, the metrics
```bash
export PRIVATE_IP=192.168.10.10 # Replace with your worker node IP
export METRICS_IP=127.0.0.1 # Bind address for Docker metrics ports 9323 / 9324
export METRICS_IP=127.0.0.1 # Bind address for metrics ports 8080 / 9100 / 9323 / 9324
export GUIDE_URL=https://raw.githubusercontent.com/vxcontrol/pentagi/main/examples/guides/worker_node
```
`METRICS_IP` defaults to `127.0.0.1` so Prometheus metrics stay local to the worker node. Set it to `${PRIVATE_IP}` only if a remote scraper must reach ports `9323` / `9324`.
`METRICS_IP` defaults to `127.0.0.1` so Prometheus metrics (Docker daemon, dind, cAdvisor, node-exporter) stay local to the worker node. Set it to `${PRIVATE_IP}` only if a remote scraper must reach ports `8080`, `9100`, `9323` / `9324`.
## Install Docker on Both Nodes
@@ -554,6 +554,131 @@ SCRAPER_PRIVATE_URL=https://${SCRAPER_USERNAME}:${SCRAPER_PASSWORD}@${PRIVATE_IP
> Nested containers may use `--network host` and can therefore reach `${PRIVATE_IP}:9443`. Basic Auth is the gate — keep the password strong and do not expose 9443 beyond the main PentAGI node.
## Deploy Monitoring Exporters on Worker Node (Host Docker)
Run cAdvisor (per-container metrics) and node-exporter (host-level metrics) on the **worker node's host Docker** (outside dind), bound to `${METRICS_IP}` — the same bind address already used for the Docker daemon and dind metrics endpoints (`9323`/`9324`). PentAGI's bundled [observability stack](mdc:observability/otel/config.yml) scrapes both under the `docker-container-collector` and `otel-collector` job names; on a distributed setup point those Prometheus targets at `${METRICS_IP}:8080` and `${METRICS_IP}:9100` instead of the in-network container names.
Both exporters are read-only introspection tools with no PentAGI-specific credentials, so — unlike the scraper — they need no secrets step.
### Create and Install `run-cadvisor`
```bash
sudo tee /usr/local/bin/run-cadvisor > /dev/null << EOF
#!/bin/bash
# Launch the cAdvisor container-metrics exporter on the worker host Docker daemon.
# Usage: run-cadvisor
set -e
CONTAINER_NAME=cadvisor
IMAGE=ghcr.io/google/cadvisor:0.56.2
LISTEN_IP=${METRICS_IP}
LISTEN_PORT=8080
RESTART_POLICY=unless-stopped
LOG_MAX_SIZE=50m
LOG_MAX_FILE=7
if docker ps --format '{{.Names}}' | grep -q "^\${CONTAINER_NAME}\$"; then
echo "container \${CONTAINER_NAME} is already running"
echo "to recreate, first remove: docker rm -f \${CONTAINER_NAME}"
exit 0
fi
docker rm "\${CONTAINER_NAME}" >/dev/null 2>&1 || true
docker run -d \\
--name "\${CONTAINER_NAME}" \\
--hostname cadvisor \\
--restart "\${RESTART_POLICY}" \\
-p "\${LISTEN_IP}:\${LISTEN_PORT}:8080" \\
-v /:/rootfs:ro \\
-v /var/run:/var/run:rw \\
-v /sys:/sys:ro \\
-v /var/lib/docker/:/var/lib/docker:ro \\
--log-opt "max-size=\${LOG_MAX_SIZE}" \\
--log-opt "max-file=\${LOG_MAX_FILE}" \\
"\${IMAGE}" \\
--store_container_labels=false \\
--docker_only=true \\
--disable_root_cgroup_stats=true
echo "cadvisor metrics listening on http://\${LISTEN_IP}:\${LISTEN_PORT}/metrics"
EOF
sudo chmod +x /usr/local/bin/run-cadvisor
```
> cAdvisor needs read-write access to `/var/run` to reach `docker.sock` for container metadata — the same trust level the host Docker daemon itself already extends to anything on this node. Keep `${METRICS_IP}` at the default `127.0.0.1` unless a remote scraper needs it.
### Create and Install `run-node-exporter`
```bash
sudo tee /usr/local/bin/run-node-exporter > /dev/null << EOF
#!/bin/bash
# Launch the Prometheus node-exporter host-metrics exporter on the worker host Docker daemon.
# Usage: run-node-exporter
set -e
CONTAINER_NAME=node_exporter
IMAGE=prom/node-exporter:v1.5.0
LISTEN_IP=${METRICS_IP}
LISTEN_PORT=9100
TEXTFILE_DIR=/var/lib/node_exporter/textfile_collector
RESTART_POLICY=unless-stopped
LOG_MAX_SIZE=50m
LOG_MAX_FILE=7
if docker ps --format '{{.Names}}' | grep -q "^\${CONTAINER_NAME}\$"; then
echo "container \${CONTAINER_NAME} is already running"
echo "to recreate, first remove: docker rm -f \${CONTAINER_NAME}"
exit 0
fi
sudo mkdir -p "\${TEXTFILE_DIR}"
docker rm "\${CONTAINER_NAME}" >/dev/null 2>&1 || true
docker run -d \\
--name "\${CONTAINER_NAME}" \\
--hostname node-exporter \\
--restart "\${RESTART_POLICY}" \\
-p "\${LISTEN_IP}:\${LISTEN_PORT}:9100" \\
-v /proc:/host/proc:ro \\
-v /sys:/host/sys:ro \\
-v /:/rootfs:ro \\
-v "\${TEXTFILE_DIR}:/textfile_collector:ro" \\
--log-opt "max-size=\${LOG_MAX_SIZE}" \\
--log-opt "max-file=\${LOG_MAX_FILE}" \\
"\${IMAGE}" \\
--path.procfs=/host/proc \\
--path.sysfs=/host/sys \\
--path.rootfs=/rootfs \\
--collector.filesystem.mount-points-exclude \\
'^/(dev|proc|sys|run|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)' \\
--collector.filesystem.fs-types-exclude \\
'^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|efivarfs|fusectl|hugetlbfs|mqueue|nsfs|overlay|proc|pstore|ramfs|rpc_pipefs|securityfs|squashfs|sysfs|tmpfs|tracefs)$' \\
--collector.textfile.directory=/textfile_collector
echo "node-exporter metrics listening on http://\${LISTEN_IP}:\${LISTEN_PORT}/metrics"
EOF
sudo chmod +x /usr/local/bin/run-node-exporter
```
> The compose reference this is derived from sets `deploy: mode: global`, a Swarm-only directive with no equivalent for a plain `docker run` container — one instance per worker node is already what this script produces.
### Start and Verify
```bash
sudo run-cadvisor
sudo run-node-exporter
curl -s "http://${METRICS_IP}:8080/metrics" | head -5
curl -s "http://${METRICS_IP}:9100/metrics" | head -5
```
Both endpoints should return Prometheus-formatted `# HELP` / `# TYPE` lines immediately — neither exporter has the cold-start delay the scraper's browser backend has.
## Security & Firewall Configuration
### Required Port Access
@@ -567,10 +692,12 @@ The worker node exposes the following services:
| 9443 | `${PRIVATE_IP}` | Scraper (Browserless) | HTTPS browser scraper for PentAGI browser tool |
| 9323 | `${METRICS_IP}` | Host Docker Metrics | Prometheus metrics endpoint for host Docker |
| 9324 | `${METRICS_IP}` | dind Metrics | Prometheus metrics endpoint for dind |
| 8080 | `${METRICS_IP}` | cAdvisor Metrics | Prometheus per-container metrics (`run-cadvisor`) |
| 9100 | `${METRICS_IP}` | node-exporter Metrics | Prometheus host-level metrics (`run-node-exporter`) |
With the default `METRICS_IP=127.0.0.1`, metrics are reachable only from the worker node itself. Set `METRICS_IP=${PRIVATE_IP}` if a remote metrics scraper must reach them.
**Metrics Integration:** The metrics ports (9323, 9324) can be configured in PentAGI's `observability/otel/config.yml` under the `docker-engine-collector` job name for monitoring integration.
**Metrics Integration:** The metrics ports (9323, 9324, 8080, 9100) can be configured in PentAGI's `observability/otel/config.yml` under the `docker-engine-collector`, `docker-container-collector` and `otel-collector` job names for monitoring integration — point their `targets` at `${METRICS_IP}:<port>` instead of the in-network container names when the worker node is remote.
### OOB Attack Port Range
@@ -578,11 +705,11 @@ Each worker container (`pentagi-terminal-N`) dynamically allocates **2 ports** f
**Firewall Requirements:**
- **Inbound**: Allow access to ports 2376, 3376, 9443 on `${PRIVATE_IP}` from the main PentAGI node
- **Inbound**: If `METRICS_IP=${PRIVATE_IP}`, also allow 9323, 9324 on `${PRIVATE_IP}` from the metrics scraper host; with the default `127.0.0.1` no remote metrics access is needed
- **Inbound**: If `METRICS_IP=${PRIVATE_IP}`, also allow 9323, 9324, 8080, 9100 on `${PRIVATE_IP}` from the metrics scraper host; with the default `127.0.0.1` no remote metrics access is needed
- **Inbound**: Allow access to port range 28000-30000 from target networks being tested
- Configure perimeter firewall to permit OOB traffic from target networks to worker node
Both Docker APIs and the scraper must be reachable **only** from the main node — nested containers may use `--network host` and can therefore probe every port on this node.
Both Docker APIs and the scraper must be reachable **only** from the main node — nested containers may use `--network host` and can therefore probe every port on this node. The same applies to cAdvisor and node-exporter: cAdvisor mounts `docker.sock`, so exposing `8080` beyond the metrics scraper host would hand any reachable client the same access level as the host Docker daemon.
## Transfer Certificates to Main Node