Update observability guide and refresh all dashboard screenshots

Guide changes:
- Architecture diagram now shows auto-discovered targets and optional
  oc-server monitoring
- Fix /host/rootfs → /host/root mount path in node_exporter section
- Document ARGS variable name (not NODE_EXPORTER_OPTS) for Alpine init
- Add --oc-server and --nodes flag documentation to Management section
- Update resource budget with oc-server row
- Expand "No node metrics" troubleshooting with ARGS diagnostic steps
- Update screenshot captions with correct RAM values (19.5 GiB)

Screenshots refreshed after fixing node-exporter RAM bug — all 9
dashboards now show correct host metrics (20 GiB RAM, not 128 MiB).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

notes/observability-guide.md

@@ -6,8 +6,10 @@ and log aggregation (Loki) for Incus nodes, HAProxy load balancers, and
 host-level resources.
 
 The stack runs entirely on OVN and is managed by the `deploy-observability`
-script. All containers use the `monitoring-allow` ACL to permit scrape
+script. Cluster nodes are **auto-discovered** — the script queries
-traffic on the shared network.
+`incus cluster list` to find nodes, eliminating hardcoded IPs. All
+containers use the `monitoring-allow` ACL to permit scrape traffic on the
+shared network.
 
 ## Architecture
 
@@ -27,10 +29,15 @@ flowchart TD
         promtail["Promtail"]
     end
 
-    subgraph targets["Scrape Targets"]
+    subgraph targets["Scrape Targets (auto-discovered)"]
-        incus["Incus nodes :8443<br/>.140 · .141 · .142"]
+        incus["Incus nodes :8443<br/>auto-discovered from cluster"]
         haproxy["HAProxy :8404<br/>.50 · .51"]
-        ne["node-exporters :9100<br/>.71 · .72 · .73"]
+        ne["node-exporters :9100<br/>one per cluster node (OVN)"]
+    end
+
+    subgraph ocserver["oc-server (optional, --oc-server)"]
+        oc_incus["Incus API :8443<br/>192.168.102.120"]
+        oc_ne["node-exporter :9100<br/>proxy device on host IP"]
     end
 
     lan --> fwd
@@ -38,6 +45,8 @@ flowchart TD
     prometheus -->|"client cert"| incus
     prometheus --> haproxy
     prometheus --> ne
+    prometheus -->|"client cert"| oc_incus
+    prometheus --> oc_ne
 
     classDef external fill:#f5f5f5,color:#333,stroke:#999
     classDef network fill:#0072B2,color:#fff,stroke:#005a8e
@@ -49,10 +58,12 @@ flowchart TD
     class fwd_grafana,fwd_prom network
     class prometheus,grafana,loki,promtail mgmt
     class incus,haproxy,ne instance
+    class oc_incus,oc_ne node
 
     style fwd fill:#e0eef8,stroke:#0072B2
     style mon fill:#f5e6f0,stroke:#CC79A7
     style targets fill:#e0f2fe,stroke:#56B4E9
+    style ocserver fill:#e0f5ed,stroke:#009E73
 ```
 
 ## Components
@@ -97,13 +108,18 @@ Log shipping agent, co-located with Loki in the monitoring container.
 
 Host-level metrics via privileged Alpine containers.
 
-- One container per cluster node, pinned with `--target` placement
+- One container per cluster node, auto-discovered and pinned with `--target`
 - Privileged containers with host filesystem bind-mounts:
   - `/proc` (host) -> `/host/proc` (read-only)
   - `/sys` (host) -> `/host/sys` (read-only)
-  - `/` (host) -> `/host/rootfs` (read-only)
+  - `/` (host) -> `/host/root` (read-only)
+- Configured via `/etc/conf.d/node-exporter` with `ARGS="--path.procfs=/host/proc
+  --path.sysfs=/host/sys --path.rootfs=/host/root"` (Alpine's init script reads
+  the `ARGS` variable, not `NODE_EXPORTER_OPTS`)
 - Exposes standard node_exporter metrics on `:9100`
 - Uses `monitoring-allow` ACL for network access
+- Optional: oc-server gets its own node-exporter with a proxy device
+  exposing `:9100` on the host IP (since oc-server uses incusbr0, not OVN)
 
 ## Access
 
@@ -147,7 +163,8 @@ Overview of all instances with instance/node template variables.
 
 ![Cluster Overview](../screenshots/cluster-overview.png)
 *Cluster-wide view showing 8 instances across 3 nodes with 28 total CPUs,
-active memory utilization, and the instance status table.*
+20.4 GiB total memory, cluster memory utilization gauge, and the instance
+status table with per-instance drill-down links.*
 
 ### Incus Cluster — Instance Deep Dive
 
@@ -190,8 +207,9 @@ Physical host monitoring with node template variable.
 - Network traffic by device, network errors and drops
 
 ![Host Resources](../screenshots/host-resources.png)
-*Host-level metrics from node-exporter showing system info, CPU/memory
+*Host-level metrics from node-exporter showing system info (19.5 GiB RAM per
-breakdown, filesystem usage, and network traffic for all 3 IncusOS nodes.*
+node, kernel version, boot time), CPU usage/mode breakdown, load average, and
+memory utilization for all 3 auto-discovered IncusOS nodes.*
 
 ### Incus Infrastructure — Storage & Filesystem
 
@@ -280,8 +298,9 @@ Prometheus self-monitoring (no template variables).
 - TSDB size trend, samples ingested trend, compactions
 
 ![Prometheus Health](../screenshots/prometheus-health.png)
-*Prometheus self-monitoring showing 7 targets UP (3 Incus, 3 node-exporter,
+*Prometheus self-monitoring showing 7 targets UP (3 Incus nodes, 3
-1 self), 2 DOWN (HAProxy), scrape performance, and TSDB stats.*
+node-exporters with friendly names, 1 self), 2 DOWN (HAProxy not deployed),
+scrape duration by job, and samples scraped.*
 
 ### Incus Operations — Capacity Planning
 
@@ -293,8 +312,8 @@ Resource planning with 7-day default time range (no template variables).
 - Instance count trend, memory allocation vs active memory per instance
 
 ![Capacity Planning](../screenshots/capacity-planning.png)
-*Capacity planning with CPU/memory headroom bars, 7-day usage trends, 30-day
+*Capacity planning with CPU/memory headroom bars per node, 7-day CPU and
-disk space forecast using predict_linear, and instance count over time.*
+memory usage trends, and 30-day disk space forecast using predict_linear.*
 
 ## ACL Configuration
 
@@ -340,6 +359,8 @@ manually or restart the container.
 
 ## Resource Budget
 
+Node-exporter containers are auto-discovered from the cluster — one per node.
+
 | Container | Image | RAM | Disk | Placement |
 |-----------|-------|-----|------|-----------|
 | monitoring | Debian/12 | 2 GiB | 20 GiB | oc-node-02 |
@@ -347,6 +368,12 @@ manually or restart the container.
 | node-exp-02 | Alpine | 128 MiB | — | oc-node-02 |
 | node-exp-03 | Alpine | 128 MiB | — | oc-node-03 |
 
+**OC server monitoring** (optional, deployed via `--oc-server`):
+
+| Container | Image | RAM | Placement | Notes |
+|-----------|-------|-----|-----------|-------|
+| node-exp-oc-server | Alpine | 128 MiB | oc-server | Proxy device exposes :9100 on host IP |
+
 **Dummy workloads** (optional, deployed via `--workloads`):
 
 | Container | Image | RAM | Placement | Purpose |
@@ -355,14 +382,15 @@ manually or restart the container.
 | workload-api | Alpine | 128 MiB | oc-node-03 | Python HTTP + memory sawtooth |
 
 **Totals:**
-- RAM: ~2.4 GiB core (2 GiB + 3 x 128 MiB), +256 MiB with workloads
+- RAM: ~2.4 GiB core (2 GiB + 3 x 128 MiB), +128 MiB with oc-server, +256 MiB with workloads
 - Disk: 20 GiB (only the monitoring container needs significant storage)
 - OVN forward IPs: 1 (192.168.103.201)
 - OVN network IPs: 4 core (10.10.10.70-73), +2 with workloads (.80, .81)
 
 ## Management
 
-The `deploy-observability` script handles the full lifecycle.
+The `deploy-observability` script handles the full lifecycle. Cluster nodes
+are **auto-discovered** from `incus cluster list` — no hardcoded IPs needed.
 
 ### Deploy the stack
 
@@ -370,10 +398,27 @@ The `deploy-observability` script handles the full lifecycle.
 incusos/deploy-observability --deploy
 ```
 
-Creates the monitoring container and all node-exporter containers,
+Auto-discovers cluster nodes, creates the monitoring container and one
-installs and configures Prometheus, Grafana, Loki, and Promtail,
+node-exporter per node, installs and configures Prometheus, Grafana,
-provisions dashboards, sets up the OVN network forward, and configures
+Loki, and Promtail, provisions dashboards, sets up the OVN network
-ACLs.
+forward, and configures ACLs.
+
+To also monitor the standalone OC management server:
+
+```bash
+incusos/deploy-observability --deploy --oc-server 192.168.102.120
+```
+
+This deploys an additional node-exporter on oc-server with a proxy device
+to expose `:9100` on the host IP (since oc-server uses incusbr0, not OVN).
+Requires an `oc-server` Incus remote — the script provides `incus remote add`
+instructions if it's missing.
+
+To override auto-discovery with a manual node list:
+
+```bash
+incusos/deploy-observability --deploy --nodes oc-node-01,oc-node-02,oc-node-03
+```
 
 ### Deploy dummy workloads
 
@@ -447,7 +492,7 @@ If Prometheus is not running, check its service status:
 incus exec monitoring -- systemctl status prometheus
 ```
 
-### No node metrics
+### No node metrics (or RAM shows 128 MiB)
 
 1. Verify the node-exporter containers are running and privileged:
    ```bash
@@ -457,7 +502,19 @@ incus exec monitoring -- systemctl status prometheus
    ```bash
    incus exec node-exp-01 -- ls /host/proc/stat
    ```
-3. Verify node_exporter is listening:
+3. Verify node_exporter is running with host path flags:
+   ```bash
+   incus exec node-exp-01 -- ps aux | grep node_exporter
+   # Should show: --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root
+   ```
+4. If flags are missing, check `/etc/conf.d/node-exporter` uses `ARGS=`
+   (not `NODE_EXPORTER_OPTS=`). Alpine's init script reads the `ARGS`
+   variable:
+   ```bash
+   incus exec node-exp-01 -- cat /etc/conf.d/node-exporter
+   # Should show: ARGS="--path.procfs=/host/proc ..."
+   ```
+5. Verify node_exporter is listening:
    ```bash
    incus exec node-exp-01 -- wget -qO- http://localhost:9100/metrics | head
    ```