diff --git a/notes/observability-guide.md b/notes/observability-guide.md index 6e8fe16..133d011 100644 --- a/notes/observability-guide.md +++ b/notes/observability-guide.md @@ -6,8 +6,10 @@ and log aggregation (Loki) for Incus nodes, HAProxy load balancers, and host-level resources. The stack runs entirely on OVN and is managed by the `deploy-observability` -script. All containers use the `monitoring-allow` ACL to permit scrape -traffic on the shared network. +script. Cluster nodes are **auto-discovered** — the script queries +`incus cluster list` to find nodes, eliminating hardcoded IPs. All +containers use the `monitoring-allow` ACL to permit scrape traffic on the +shared network. ## Architecture @@ -27,10 +29,15 @@ flowchart TD promtail["Promtail"] end - subgraph targets["Scrape Targets"] - incus["Incus nodes :8443
.140 · .141 · .142"] + subgraph targets["Scrape Targets (auto-discovered)"] + incus["Incus nodes :8443
auto-discovered from cluster"] haproxy["HAProxy :8404
.50 · .51"] - ne["node-exporters :9100
.71 · .72 · .73"] + ne["node-exporters :9100
one per cluster node (OVN)"] + end + + subgraph ocserver["oc-server (optional, --oc-server)"] + oc_incus["Incus API :8443
192.168.102.120"] + oc_ne["node-exporter :9100
proxy device on host IP"] end lan --> fwd @@ -38,6 +45,8 @@ flowchart TD prometheus -->|"client cert"| incus prometheus --> haproxy prometheus --> ne + prometheus -->|"client cert"| oc_incus + prometheus --> oc_ne classDef external fill:#f5f5f5,color:#333,stroke:#999 classDef network fill:#0072B2,color:#fff,stroke:#005a8e @@ -49,10 +58,12 @@ flowchart TD class fwd_grafana,fwd_prom network class prometheus,grafana,loki,promtail mgmt class incus,haproxy,ne instance + class oc_incus,oc_ne node style fwd fill:#e0eef8,stroke:#0072B2 style mon fill:#f5e6f0,stroke:#CC79A7 style targets fill:#e0f2fe,stroke:#56B4E9 + style ocserver fill:#e0f5ed,stroke:#009E73 ``` ## Components @@ -97,13 +108,18 @@ Log shipping agent, co-located with Loki in the monitoring container. Host-level metrics via privileged Alpine containers. -- One container per cluster node, pinned with `--target` placement +- One container per cluster node, auto-discovered and pinned with `--target` - Privileged containers with host filesystem bind-mounts: - `/proc` (host) -> `/host/proc` (read-only) - `/sys` (host) -> `/host/sys` (read-only) - - `/` (host) -> `/host/rootfs` (read-only) + - `/` (host) -> `/host/root` (read-only) +- Configured via `/etc/conf.d/node-exporter` with `ARGS="--path.procfs=/host/proc + --path.sysfs=/host/sys --path.rootfs=/host/root"` (Alpine's init script reads + the `ARGS` variable, not `NODE_EXPORTER_OPTS`) - Exposes standard node_exporter metrics on `:9100` - Uses `monitoring-allow` ACL for network access +- Optional: oc-server gets its own node-exporter with a proxy device + exposing `:9100` on the host IP (since oc-server uses incusbr0, not OVN) ## Access @@ -147,7 +163,8 @@ Overview of all instances with instance/node template variables. ![Cluster Overview](../screenshots/cluster-overview.png) *Cluster-wide view showing 8 instances across 3 nodes with 28 total CPUs, -active memory utilization, and the instance status table.* +20.4 GiB total memory, cluster memory utilization gauge, and the instance +status table with per-instance drill-down links.* ### Incus Cluster — Instance Deep Dive @@ -190,8 +207,9 @@ Physical host monitoring with node template variable. - Network traffic by device, network errors and drops ![Host Resources](../screenshots/host-resources.png) -*Host-level metrics from node-exporter showing system info, CPU/memory -breakdown, filesystem usage, and network traffic for all 3 IncusOS nodes.* +*Host-level metrics from node-exporter showing system info (19.5 GiB RAM per +node, kernel version, boot time), CPU usage/mode breakdown, load average, and +memory utilization for all 3 auto-discovered IncusOS nodes.* ### Incus Infrastructure — Storage & Filesystem @@ -280,8 +298,9 @@ Prometheus self-monitoring (no template variables). - TSDB size trend, samples ingested trend, compactions ![Prometheus Health](../screenshots/prometheus-health.png) -*Prometheus self-monitoring showing 7 targets UP (3 Incus, 3 node-exporter, -1 self), 2 DOWN (HAProxy), scrape performance, and TSDB stats.* +*Prometheus self-monitoring showing 7 targets UP (3 Incus nodes, 3 +node-exporters with friendly names, 1 self), 2 DOWN (HAProxy not deployed), +scrape duration by job, and samples scraped.* ### Incus Operations — Capacity Planning @@ -293,8 +312,8 @@ Resource planning with 7-day default time range (no template variables). - Instance count trend, memory allocation vs active memory per instance ![Capacity Planning](../screenshots/capacity-planning.png) -*Capacity planning with CPU/memory headroom bars, 7-day usage trends, 30-day -disk space forecast using predict_linear, and instance count over time.* +*Capacity planning with CPU/memory headroom bars per node, 7-day CPU and +memory usage trends, and 30-day disk space forecast using predict_linear.* ## ACL Configuration @@ -340,6 +359,8 @@ manually or restart the container. ## Resource Budget +Node-exporter containers are auto-discovered from the cluster — one per node. + | Container | Image | RAM | Disk | Placement | |-----------|-------|-----|------|-----------| | monitoring | Debian/12 | 2 GiB | 20 GiB | oc-node-02 | @@ -347,6 +368,12 @@ manually or restart the container. | node-exp-02 | Alpine | 128 MiB | — | oc-node-02 | | node-exp-03 | Alpine | 128 MiB | — | oc-node-03 | +**OC server monitoring** (optional, deployed via `--oc-server`): + +| Container | Image | RAM | Placement | Notes | +|-----------|-------|-----|-----------|-------| +| node-exp-oc-server | Alpine | 128 MiB | oc-server | Proxy device exposes :9100 on host IP | + **Dummy workloads** (optional, deployed via `--workloads`): | Container | Image | RAM | Placement | Purpose | @@ -355,14 +382,15 @@ manually or restart the container. | workload-api | Alpine | 128 MiB | oc-node-03 | Python HTTP + memory sawtooth | **Totals:** -- RAM: ~2.4 GiB core (2 GiB + 3 x 128 MiB), +256 MiB with workloads +- RAM: ~2.4 GiB core (2 GiB + 3 x 128 MiB), +128 MiB with oc-server, +256 MiB with workloads - Disk: 20 GiB (only the monitoring container needs significant storage) - OVN forward IPs: 1 (192.168.103.201) - OVN network IPs: 4 core (10.10.10.70-73), +2 with workloads (.80, .81) ## Management -The `deploy-observability` script handles the full lifecycle. +The `deploy-observability` script handles the full lifecycle. Cluster nodes +are **auto-discovered** from `incus cluster list` — no hardcoded IPs needed. ### Deploy the stack @@ -370,10 +398,27 @@ The `deploy-observability` script handles the full lifecycle. incusos/deploy-observability --deploy ``` -Creates the monitoring container and all node-exporter containers, -installs and configures Prometheus, Grafana, Loki, and Promtail, -provisions dashboards, sets up the OVN network forward, and configures -ACLs. +Auto-discovers cluster nodes, creates the monitoring container and one +node-exporter per node, installs and configures Prometheus, Grafana, +Loki, and Promtail, provisions dashboards, sets up the OVN network +forward, and configures ACLs. + +To also monitor the standalone OC management server: + +```bash +incusos/deploy-observability --deploy --oc-server 192.168.102.120 +``` + +This deploys an additional node-exporter on oc-server with a proxy device +to expose `:9100` on the host IP (since oc-server uses incusbr0, not OVN). +Requires an `oc-server` Incus remote — the script provides `incus remote add` +instructions if it's missing. + +To override auto-discovery with a manual node list: + +```bash +incusos/deploy-observability --deploy --nodes oc-node-01,oc-node-02,oc-node-03 +``` ### Deploy dummy workloads @@ -447,7 +492,7 @@ If Prometheus is not running, check its service status: incus exec monitoring -- systemctl status prometheus ``` -### No node metrics +### No node metrics (or RAM shows 128 MiB) 1. Verify the node-exporter containers are running and privileged: ```bash @@ -457,7 +502,19 @@ incus exec monitoring -- systemctl status prometheus ```bash incus exec node-exp-01 -- ls /host/proc/stat ``` -3. Verify node_exporter is listening: +3. Verify node_exporter is running with host path flags: + ```bash + incus exec node-exp-01 -- ps aux | grep node_exporter + # Should show: --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root + ``` +4. If flags are missing, check `/etc/conf.d/node-exporter` uses `ARGS=` + (not `NODE_EXPORTER_OPTS=`). Alpine's init script reads the `ARGS` + variable: + ```bash + incus exec node-exp-01 -- cat /etc/conf.d/node-exporter + # Should show: ARGS="--path.procfs=/host/proc ..." + ``` +5. Verify node_exporter is listening: ```bash incus exec node-exp-01 -- wget -qO- http://localhost:9100/metrics | head ``` diff --git a/screenshots/capacity-planning.png b/screenshots/capacity-planning.png index 1f808b5..5f3f5f2 100644 Binary files a/screenshots/capacity-planning.png and b/screenshots/capacity-planning.png differ diff --git a/screenshots/cluster-overview.png b/screenshots/cluster-overview.png index 87ae773..f18a657 100644 Binary files a/screenshots/cluster-overview.png and b/screenshots/cluster-overview.png differ diff --git a/screenshots/haproxy-traffic.png b/screenshots/haproxy-traffic.png index fe03fb8..5fd2a10 100644 Binary files a/screenshots/haproxy-traffic.png and b/screenshots/haproxy-traffic.png differ diff --git a/screenshots/host-resources.png b/screenshots/host-resources.png index 8304bbc..afed397 100644 Binary files a/screenshots/host-resources.png and b/screenshots/host-resources.png differ diff --git a/screenshots/instance-deep-dive.png b/screenshots/instance-deep-dive.png index d53bbf1..4d409ed 100644 Binary files a/screenshots/instance-deep-dive.png and b/screenshots/instance-deep-dive.png differ diff --git a/screenshots/logs-explorer.png b/screenshots/logs-explorer.png index d6c429f..f361364 100644 Binary files a/screenshots/logs-explorer.png and b/screenshots/logs-explorer.png differ diff --git a/screenshots/network-deep-dive.png b/screenshots/network-deep-dive.png index 325ddb9..45c91e9 100644 Binary files a/screenshots/network-deep-dive.png and b/screenshots/network-deep-dive.png differ diff --git a/screenshots/prometheus-health.png b/screenshots/prometheus-health.png index 37f679a..761cf6c 100644 Binary files a/screenshots/prometheus-health.png and b/screenshots/prometheus-health.png differ diff --git a/screenshots/storage-filesystem.png b/screenshots/storage-filesystem.png index 724d4a7..93310a3 100644 Binary files a/screenshots/storage-filesystem.png and b/screenshots/storage-filesystem.png differ