Add observability stack: Prometheus, Grafana, Loki with dashboards

Deploy and validate full monitoring stack on the Incus cluster: - Prometheus scraping 9 targets (3 Incus nodes, 2 HAProxy, 3 node-exporters, self) - Grafana with 3 dashboards (cluster overview, HAProxy traffic, host resources) - Loki + Promtail for log aggregation - OVN network forward for LAN access (192.168.103.201:3000/9090) - deploy-observability script with --deploy/--status/--cleanup/--doctor - Solved OVN ACL challenges: Aether default-deny requires monitoring-allow ACL Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 15:44:59 +01:00 · 2026-02-24 15:44:59 +01:00 · cc017b648d
parent 91a87456a5
commit cc017b648d
7 changed files with 3128 additions and 0 deletions
--- a/.claude/rules/observability.md
+++ b/.claude/rules/observability.md
@ -0,0 +1,85 @@
+---
+paths:
+  - "incusos/deploy-observability"
+  - "incusos/observability-dashboards"
+  - "notes/observability-guide.md"
+---
+
+# Observability Stack (Prometheus + Grafana + Loki)
+
+## Deployment topology
+
+- **monitoring** container: 10.10.10.70, pinned to oc-node-02, net-prod
+  - Prometheus 2.54 on :9090 (15s scrape interval)
+  - Grafana 12.4 on :3000 (default creds admin/admin)
+  - Loki 3.6 on :3100 (HTTP) and **:9096** (gRPC -- NOT default 9095,
+    9095 conflicts with Promtail metrics port)
+  - Promtail 3.6 shipping logs to Loki
+  - Debian/12, 2 GiB RAM, 20 GiB disk
+
+- **node-exp-01/02/03**: 10.10.10.71-73, one per cluster node
+  - Alpine, privileged, 128 MiB RAM each
+  - Host filesystem mounts: /proc, /sys, / (all read-only)
+  - node_exporter on :9100
+
+## LAN access
+
+- OVN network forward: 192.168.103.201
+  - :3000 -> 10.10.10.70:3000 (Grafana)
+  - :9090 -> 10.10.10.70:9090 (Prometheus)
+
+## Prometheus scrape targets
+
+| Target | Address | Auth | Port |
+|--------|---------|------|------|
+| Incus nodes | 192.168.102.140-142 | Client cert (TLS) | 8443 |
+| HAProxy | 10.10.10.50-51 | None | 8404 |
+| node-exporters | 10.10.10.71-73 | None | 9100 |
+
+- Incus metrics endpoint: `/1.0/metrics` on each node, requires client
+  certificate authentication (cluster cert + key in Prometheus config).
+- HAProxy metrics: requires `http-request use-service prometheus-exporter
+  if { path /metrics }` in the stats frontend of haproxy.cfg. Without
+  this, /metrics returns HTML stats, not Prometheus exposition format.
+
+## ACL details (critical for scrape connectivity)
+
+- **Aether default-deny**: every container on shared OVN network gets
+  reject ACLs: priority 111 reject (egress), priority 100 reject (ingress).
+- **monitoring-allow ACL**: set `default.*.action=allow` on all directions.
+  This creates allow rules at priority 111, matching Aether's reject
+  priority and effectively overriding default-deny.
+- All observability containers (monitoring, node-exp-01/02/03) use the
+  monitoring-allow ACL.
+- **HAProxy containers** are Aether-managed with their own ACLs. Need
+  explicit ingress rules allowing TCP from 10.10.10.70 to port 8404 for
+  Prometheus scrapes.
+
+## NIC down after security.acls changes
+
+Changing `security.acls` on a container NIC can cause the NIC to go
+**down**. After modifying ACLs:
+- Bring NIC up manually: `incus exec <container> -- ip link set eth0 up`
+- Or restart the container: `incus restart <container>`
+This is a known OVN behavior, not a bug.
+
+## Dashboards
+
+Three pre-provisioned dashboards:
+1. **Incus Cluster Overview** -- instance CPU, memory, network, disk I/O
+2. **HAProxy Traffic** -- requests, backend health, sessions, traffic
+3. **Host Resources** -- node CPU%, memory%, disk, network
+
+## Script: deploy-observability
+
+- `--deploy` -- full stack deployment
+- `--status` -- container state, target health, forward config
+- `--cleanup` -- remove all observability containers and forwards
+- `--doctor` -- prerequisite and health checks
+
+## Troubleshooting quick reference
+
+- Targets DOWN: check ACLs, check NIC state after ACL changes
+- No HAProxy metrics: need prometheus-exporter route in haproxy.cfg stats
+- Loki gRPC: port 9096, not 9095 (Promtail conflict)
+- Node metrics missing: check privileged flag, host mounts, node_exporter service
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -27,7 +27,9 @@ incus-contrib/
 │   ├── incusos-proxmox    # Declarative Proxmox VM deployment + lab lifecycle
 │   ├── deploy-awx         # AWX deployment + management on Incus cluster
 │   ├── deploy-haproxy     # HAProxy LB deployment + management via Aether
+│   ├── deploy-observability # Prometheus + Grafana + Loki observability stack
 │   ├── awx-manifests/     # K8s manifests for AWX Operator + instance
+│   ├── observability-dashboards/  # Grafana dashboard JSON exports
 │   ├── helpers/
 │   │   ├── proxmox-screenshot  # VMID -> PNG console screenshot
 │   │   ├── proxmox-api         # Authenticated API calls (handles ! in token)
@ -187,6 +189,7 @@ which files are being edited:
 | `networking-storage.md` | networking, storage, migration, UTM guides |
 | `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
 | `haproxy-lb.md` | deploy-haproxy, HAProxy guide |
+| `observability.md` | deploy-observability, observability dashboards/guide |
 | `ovn-internals.md` | ovn-inspect, ovn-deep-dive |
 | `aether-api-payloads.md` | incus-mitm, api-interception |
 | `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
--- a/incusos/deploy-observability
+++ b/incusos/deploy-observability
--- a/incusos/observability-dashboards/cluster-overview.json
+++ b/incusos/observability-dashboards/cluster-overview.json
@ -0,0 +1,559 @@
+{
+  "uid": "incus-cluster-overview",
+  "title": "Incus Cluster Overview",
+  "description": "Overview of all Incus containers: CPU, memory, network, and disk I/O per instance.",
+  "tags": ["incus", "cluster", "containers"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "30s",
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "fiscalYearStartMonth": 0,
+  "liveNow": false,
+  "editable": true,
+  "graphTooltip": 1,
+  "templating": {
+    "list": []
+  },
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "panels": [
+    {
+      "type": "row",
+      "title": "Cluster Status",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "id": 1,
+      "panels": []
+    },
+    {
+      "type": "stat",
+      "title": "Total Containers",
+      "description": "Number of running Incus containers (instances reporting uptime).",
+      "gridPos": { "h": 4, "w": 8, "x": 0, "y": 1 },
+      "id": 2,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "count(incus_uptime_seconds)",
+          "legendFormat": "Containers",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 10 },
+              { "color": "red", "value": 50 }
+            ]
+          },
+          "unit": "none",
+          "color": { "mode": "thresholds" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "orientation": "auto",
+        "textMode": "auto",
+        "colorMode": "value",
+        "graphMode": "none",
+        "justifyMode": "auto"
+      }
+    },
+    {
+      "type": "stat",
+      "title": "Total CPUs",
+      "description": "Sum of effective CPUs across all Incus instances.",
+      "gridPos": { "h": 4, "w": 8, "x": 8, "y": 1 },
+      "id": 3,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum(incus_cpu_effective_total)",
+          "legendFormat": "CPUs",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          },
+          "unit": "none",
+          "color": { "mode": "thresholds" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "orientation": "auto",
+        "textMode": "auto",
+        "colorMode": "value",
+        "graphMode": "none",
+        "justifyMode": "auto"
+      }
+    },
+    {
+      "type": "stat",
+      "title": "Total Memory",
+      "description": "Sum of total memory across all Incus instances.",
+      "gridPos": { "h": 4, "w": 8, "x": 16, "y": 1 },
+      "id": 4,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum(incus_memory_MemTotal_bytes)",
+          "legendFormat": "Total Memory",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          },
+          "unit": "bytes",
+          "color": { "mode": "thresholds" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "orientation": "auto",
+        "textMode": "auto",
+        "colorMode": "value",
+        "graphMode": "none",
+        "justifyMode": "auto"
+      }
+    },
+    {
+      "type": "row",
+      "title": "Per-Instance CPU",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
+      "id": 5,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "CPU Usage by Instance",
+      "description": "Per-second CPU time consumed by each Incus instance.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 6 },
+      "id": 6,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (name) (rate(incus_cpu_seconds_total[$__rate_interval]))",
+          "legendFormat": "{{name}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "short",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "row",
+      "title": "Per-Instance Memory",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 },
+      "id": 7,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "Memory RSS by Instance",
+      "description": "Resident set size (RSS) memory usage per Incus instance over time.",
+      "gridPos": { "h": 8, "w": 14, "x": 0, "y": 15 },
+      "id": 8,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "incus_memory_RSS_bytes",
+          "legendFormat": "{{name}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "bytes",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "bargauge",
+      "title": "Memory Usage % (RSS / Total)",
+      "description": "Current memory utilization per instance as RSS divided by total memory.",
+      "gridPos": { "h": 8, "w": 10, "x": 14, "y": 15 },
+      "id": 9,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "(incus_memory_RSS_bytes / incus_memory_MemTotal_bytes) * 100",
+          "legendFormat": "{{name}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 60 },
+              { "color": "orange", "value": 80 },
+              { "color": "red", "value": 90 }
+            ]
+          },
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "color": { "mode": "thresholds" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "orientation": "horizontal",
+        "displayMode": "gradient",
+        "showUnfilled": true,
+        "minVizWidth": 8,
+        "minVizHeight": 16,
+        "valueMode": "color"
+      }
+    },
+    {
+      "type": "row",
+      "title": "Network I/O",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 23 },
+      "id": 10,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "Network Receive Rate by Instance",
+      "description": "Inbound network throughput per Incus instance.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
+      "id": 11,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (name) (rate(incus_network_receive_bytes_total[$__rate_interval]))",
+          "legendFormat": "{{name}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Network Transmit Rate by Instance",
+      "description": "Outbound network throughput per Incus instance.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
+      "id": 12,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (name) (rate(incus_network_transmit_bytes_total[$__rate_interval]))",
+          "legendFormat": "{{name}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "row",
+      "title": "Disk I/O",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 32 },
+      "id": 13,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "Disk Read Rate by Instance",
+      "description": "Disk read throughput per Incus instance.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 33 },
+      "id": 14,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (name) (rate(incus_disk_read_bytes_total[$__rate_interval]))",
+          "legendFormat": "{{name}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Disk Write Rate by Instance",
+      "description": "Disk write throughput per Incus instance.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 33 },
+      "id": 15,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (name) (rate(incus_disk_written_bytes_total[$__rate_interval]))",
+          "legendFormat": "{{name}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    }
+  ]
+}
--- a/incusos/observability-dashboards/haproxy-traffic.json
+++ b/incusos/observability-dashboards/haproxy-traffic.json
@ -0,0 +1,518 @@
+{
+  "uid": "incus-haproxy-traffic",
+  "title": "HAProxy Traffic",
+  "description": "HAProxy load balancer monitoring: request rates, backend health, traffic volume, and sessions.",
+  "tags": ["haproxy", "loadbalancer", "incus"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "30s",
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "fiscalYearStartMonth": 0,
+  "liveNow": false,
+  "editable": true,
+  "graphTooltip": 1,
+  "templating": {
+    "list": []
+  },
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "panels": [
+    {
+      "type": "row",
+      "title": "Overview",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "id": 1,
+      "panels": []
+    },
+    {
+      "type": "stat",
+      "title": "Active Backend Servers",
+      "description": "Total number of active servers across all HAProxy backends.",
+      "gridPos": { "h": 4, "w": 8, "x": 0, "y": 1 },
+      "id": 2,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum(haproxy_backend_active_servers)",
+          "legendFormat": "Active Servers",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "red", "value": null },
+              { "color": "yellow", "value": 1 },
+              { "color": "green", "value": 2 }
+            ]
+          },
+          "unit": "none",
+          "color": { "mode": "thresholds" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "orientation": "auto",
+        "textMode": "auto",
+        "colorMode": "value",
+        "graphMode": "none",
+        "justifyMode": "auto"
+      }
+    },
+    {
+      "type": "stat",
+      "title": "Current Sessions",
+      "description": "Total current sessions across all HAProxy backends.",
+      "gridPos": { "h": 4, "w": 8, "x": 8, "y": 1 },
+      "id": 3,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum(haproxy_backend_current_sessions)",
+          "legendFormat": "Sessions",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 100 },
+              { "color": "red", "value": 500 }
+            ]
+          },
+          "unit": "none",
+          "color": { "mode": "thresholds" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "orientation": "auto",
+        "textMode": "auto",
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto"
+      }
+    },
+    {
+      "type": "stat",
+      "title": "Total HTTP Requests",
+      "description": "Total HTTP requests across all HAProxy frontends (counter).",
+      "gridPos": { "h": 4, "w": 8, "x": 16, "y": 1 },
+      "id": 4,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum(rate(haproxy_frontend_http_requests_total[$__rate_interval]))",
+          "legendFormat": "Requests/s",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          },
+          "unit": "reqps",
+          "color": { "mode": "thresholds" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "reduceOptions": {
+          "calcs": ["lastNotNull"],
+          "fields": "",
+          "values": false
+        },
+        "orientation": "auto",
+        "textMode": "auto",
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto"
+      }
+    },
+    {
+      "type": "row",
+      "title": "HTTP Requests",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
+      "id": 5,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "HTTP Request Rate by Frontend",
+      "description": "Rate of HTTP requests per second, broken down by HAProxy frontend/proxy.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 6 },
+      "id": 6,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (proxy) (rate(haproxy_frontend_http_requests_total[$__rate_interval]))",
+          "legendFormat": "{{proxy}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 20,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "reqps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "row",
+      "title": "Backend Health",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 },
+      "id": 7,
+      "panels": []
+    },
+    {
+      "type": "table",
+      "title": "Backend Status",
+      "description": "Current status, active server count, and average response time for each HAProxy backend.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 15 },
+      "id": 8,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "haproxy_backend_status",
+          "legendFormat": "",
+          "refId": "A",
+          "format": "table",
+          "instant": true
+        },
+        {
+          "expr": "haproxy_backend_active_servers",
+          "legendFormat": "",
+          "refId": "B",
+          "format": "table",
+          "instant": true
+        },
+        {
+          "expr": "haproxy_server_response_time_average_seconds",
+          "legendFormat": "",
+          "refId": "C",
+          "format": "table",
+          "instant": true
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "align": "auto",
+            "displayMode": "auto",
+            "inspect": false,
+            "filterable": true
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          },
+          "color": { "mode": "thresholds" }
+        },
+        "overrides": [
+          {
+            "matcher": { "id": "byName", "options": "Value #A" },
+            "properties": [
+              { "id": "displayName", "value": "Status" }
+            ]
+          },
+          {
+            "matcher": { "id": "byName", "options": "Value #B" },
+            "properties": [
+              { "id": "displayName", "value": "Active Servers" }
+            ]
+          },
+          {
+            "matcher": { "id": "byName", "options": "Value #C" },
+            "properties": [
+              { "id": "displayName", "value": "Avg Response Time" },
+              { "id": "unit", "value": "s" }
+            ]
+          },
+          {
+            "matcher": { "id": "byName", "options": "proxy" },
+            "properties": [
+              { "id": "displayName", "value": "Backend" }
+            ]
+          }
+        ]
+      },
+      "options": {
+        "showHeader": true,
+        "sortBy": [],
+        "footer": {
+          "show": false,
+          "reducer": ["sum"],
+          "fields": ""
+        }
+      },
+      "transformations": [
+        {
+          "id": "merge",
+          "options": {}
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "Time": true,
+              "__name__": true,
+              "instance": false,
+              "job": true
+            },
+            "indexByName": {},
+            "renameByName": {}
+          }
+        }
+      ]
+    },
+    {
+      "type": "row",
+      "title": "Traffic",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 23 },
+      "id": 9,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "Frontend Bytes In",
+      "description": "Inbound traffic rate per HAProxy frontend.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
+      "id": 10,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (proxy) (rate(haproxy_frontend_bytes_in_total[$__rate_interval]))",
+          "legendFormat": "{{proxy}} (in)",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 20,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Frontend Bytes Out",
+      "description": "Outbound traffic rate per HAProxy frontend.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
+      "id": 11,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (proxy) (rate(haproxy_frontend_bytes_out_total[$__rate_interval]))",
+          "legendFormat": "{{proxy}} (out)",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 20,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "row",
+      "title": "Sessions",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 32 },
+      "id": 12,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "Current Sessions by Proxy",
+      "description": "Current active sessions for each HAProxy backend/proxy over time.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 33 },
+      "id": 13,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "haproxy_backend_current_sessions",
+          "legendFormat": "{{proxy}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 20,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "none",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }
+      }
+    }
+  ]
+}
--- a/incusos/observability-dashboards/host-resources.json
+++ b/incusos/observability-dashboards/host-resources.json
@ -0,0 +1,429 @@
+{
+  "uid": "incus-host-resources",
+  "title": "Host Resources (IncusOS Nodes)",
+  "description": "Physical host monitoring for IncusOS cluster nodes: CPU, memory, disk, and network from node_exporter metrics.",
+  "tags": ["incusos", "host", "node-exporter"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "version": 1,
+  "refresh": "30s",
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "fiscalYearStartMonth": 0,
+  "liveNow": false,
+  "editable": true,
+  "graphTooltip": 1,
+  "templating": {
+    "list": []
+  },
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "panels": [
+    {
+      "type": "row",
+      "title": "CPU",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "id": 1,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "CPU Usage % by Node",
+      "description": "Overall CPU utilization per host node, calculated as 1 minus the average idle CPU rate.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 1 },
+      "id": 2,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "(1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval]))) * 100",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 2,
+            "fillOpacity": 25,
+            "gradientMode": "scheme",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "line" }
+          },
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "color": { "mode": "continuous-GrYlRd" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 70 },
+              { "color": "red", "value": 90 }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }
+      }
+    },
+    {
+      "type": "row",
+      "title": "Memory",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 },
+      "id": 3,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "Memory Usage % by Node",
+      "description": "Memory utilization percentage per host node, calculated from available vs total memory.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 10 },
+      "id": 4,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 2,
+            "fillOpacity": 25,
+            "gradientMode": "scheme",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "line" }
+          },
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "color": { "mode": "continuous-GrYlRd" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 70 },
+              { "color": "red", "value": 90 }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }
+      }
+    },
+    {
+      "type": "row",
+      "title": "Disk",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 18 },
+      "id": 5,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "Root Filesystem Available Space",
+      "description": "Available disk space on the root filesystem (mountpoint='/') per host node.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 19 },
+      "id": 6,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "node_filesystem_avail_bytes{mountpoint=\"/\"}",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "bytes",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull", "min"] }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Disk Read Rate by Node",
+      "description": "Disk read throughput (bytes/sec) aggregated per host node.",
+      "gridPos": { "h": 8, "w": 6, "x": 12, "y": 19 },
+      "id": 7,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (instance) (rate(node_disk_read_bytes_total[$__rate_interval]))",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "list", "placement": "bottom", "calcs": [] }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Disk Write Rate by Node",
+      "description": "Disk write throughput (bytes/sec) aggregated per host node.",
+      "gridPos": { "h": 8, "w": 6, "x": 18, "y": 19 },
+      "id": 8,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "sum by (instance) (rate(node_disk_written_bytes_total[$__rate_interval]))",
+          "legendFormat": "{{instance}}",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "list", "placement": "bottom", "calcs": [] }
+      }
+    },
+    {
+      "type": "row",
+      "title": "Network",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 27 },
+      "id": 9,
+      "panels": []
+    },
+    {
+      "type": "timeseries",
+      "title": "Network Receive Rate by Device & Node",
+      "description": "Inbound network throughput per network device and host node.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "id": 10,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "rate(node_network_receive_bytes_total{device!~\"lo|veth.*|br-.*|docker.*\"}[$__rate_interval])",
+          "legendFormat": "{{instance}} - {{device}} (rx)",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    },
+    {
+      "type": "timeseries",
+      "title": "Network Transmit Rate by Device & Node",
+      "description": "Outbound network throughput per network device and host node.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "id": 11,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "PBFA97CFB590B2093"
+      },
+      "targets": [
+        {
+          "expr": "rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|br-.*|docker.*\"}[$__rate_interval])",
+          "legendFormat": "{{instance}} - {{device}} (tx)",
+          "refId": "A"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "fillOpacity": 15,
+            "gradientMode": "none",
+            "spanNulls": false,
+            "showPoints": "never",
+            "pointSize": 5,
+            "stacking": { "mode": "none", "group": "A" },
+            "axisPlacement": "auto",
+            "axisLabel": "",
+            "scaleDistribution": { "type": "linear" },
+            "thresholdsStyle": { "mode": "off" }
+          },
+          "unit": "Bps",
+          "color": { "mode": "palette-classic" },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "options": {
+        "tooltip": { "mode": "multi", "sort": "desc" },
+        "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
+      }
+    }
+  ]
+}
--- a/notes/observability-guide.md
+++ b/notes/observability-guide.md
@ -0,0 +1,337 @@
+# Observability Stack — Prometheus, Grafana, Loki on Incus
+
+A lightweight observability stack deployed as Incus containers on the lab
+cluster. Provides metric collection (Prometheus), dashboards (Grafana),
+and log aggregation (Loki) for Incus nodes, HAProxy load balancers, and
+host-level resources.
+
+The stack runs entirely on OVN and is managed by the `deploy-observability`
+script. All containers use the `monitoring-allow` ACL to permit scrape
+traffic on the shared network.
+
+## Architecture
+
+```
+    LAN (192.168.103.0/24)
+         |
+         |  OVN network forward (192.168.103.201)
+         |    :3000 -> 10.10.10.70:3000  (Grafana)
+         |    :9090 -> 10.10.10.70:9090  (Prometheus)
+         |
+  ═══════╪══════════════════════════════════════════════
+         |       OVN net-prod (10.10.10.0/24)
+         |
+         v
+  +----------------------------------------------+
+  |  monitoring (10.10.10.70, oc-node-02)        |
+  |  Debian/12 container — 2 GiB RAM, 20 GiB    |
+  |                                              |
+  |  +-------------+  +-----------+              |
+  |  | Prometheus  |  | Grafana   |              |
+  |  | :9090       |  | :3000     |              |
+  |  +-------------+  +-----------+              |
+  |  +-------------+  +-----------+              |
+  |  | Loki        |  | Promtail  |              |
+  |  | :3100       |  | (agent)   |              |
+  |  | gRPC :9096  |  |           |              |
+  |  +-------------+  +-----------+              |
+  +----------------------------------------------+
+         |
+         |  Scrape targets
+         |
+         +-------> Incus nodes (client cert auth, :8443)
+         |           192.168.102.140  /1.0/metrics
+         |           192.168.102.141  /1.0/metrics
+         |           192.168.102.142  /1.0/metrics
+         |
+         +-------> HAProxy (:8404)
+         |           10.10.10.50     /metrics
+         |           10.10.10.51     /metrics
+         |
+         +-------> node-exporters (:9100)
+                    10.10.10.71  node-exp-01 (oc-node-01)
+                    10.10.10.72  node-exp-02 (oc-node-02)
+                    10.10.10.73  node-exp-03 (oc-node-03)
+
+  +------------------+  +------------------+  +------------------+
+  | node-exp-01      |  | node-exp-02      |  | node-exp-03      |
+  | 10.10.10.71      |  | 10.10.10.72      |  | 10.10.10.73      |
+  | Alpine, privil.  |  | Alpine, privil.  |  | Alpine, privil.  |
+  | :9100            |  | :9100            |  | :9100            |
+  | host /proc,/sys  |  | host /proc,/sys  |  | host /proc,/sys  |
+  | 128 MiB RAM      |  | 128 MiB RAM      |  | 128 MiB RAM      |
+  | pinned to node   |  | pinned to node   |  | pinned to node   |
+  +------------------+  +------------------+  +------------------+
+```
+
+## Components
+
+### Prometheus 2.54
+
+Metric collection engine. Scrapes all targets at a 15-second interval.
+
+- Stores metrics locally in the monitoring container
+- Client certificate authentication for Incus `/1.0/metrics` endpoints
+  (uses cluster client cert + key)
+- Plain HTTP scrape for HAProxy stats (`:8404/metrics`) and
+  node-exporters (`:9100/metrics`)
+
+### Grafana 12.4
+
+Dashboard and visualization frontend.
+
+- Default credentials: `admin` / `admin`
+- Pre-provisioned datasources for Prometheus and Loki
+- Three dashboards deployed automatically (see Dashboards section)
+- Accessible from LAN via OVN network forward
+
+### Loki 3.6
+
+Log aggregation backend.
+
+- HTTP API on port 3100
+- gRPC on port **9096** (not the default 9095 -- avoids conflict with
+  Promtail's own metrics port)
+- 7-day retention policy
+- Receives logs from Promtail running in the same container
+
+### Promtail 3.6
+
+Log shipping agent, co-located with Loki in the monitoring container.
+
+- Pushes logs to Loki via gRPC on localhost:9096
+- Scrapes container logs and system journal
+
+### node_exporter
+
+Host-level metrics via privileged Alpine containers.
+
+- One container per cluster node, pinned with `--target` placement
+- Privileged containers with host filesystem bind-mounts:
+  - `/proc` (host) -> `/host/proc` (read-only)
+  - `/sys` (host) -> `/host/sys` (read-only)
+  - `/` (host) -> `/host/rootfs` (read-only)
+- Exposes standard node_exporter metrics on `:9100`
+- Uses `monitoring-allow` ACL for network access
+
+## Access
+
+| Service | URL | Credentials |
+|---------|-----|-------------|
+| Grafana | http://192.168.103.201:3000 | `admin` / `admin` |
+| Prometheus | http://192.168.103.201:9090 | None (open) |
+
+Both services are accessible from the LAN through an OVN network forward
+on IP `192.168.103.201`. The forward maps external ports directly to the
+monitoring container's internal ports (no translation).
+
+Prometheus and Loki are not exposed externally by default. They are
+accessible only from within the OVN network or through the Grafana
+datasource proxy.
+
+## Dashboards
+
+Three dashboards are provisioned automatically during deployment.
+
+### Incus Cluster Overview
+
+Visualizes metrics scraped from the Incus `/1.0/metrics` endpoint on each
+cluster node.
+
+- Instance CPU usage (per-instance, per-node)
+- Memory usage and allocation
+- Network I/O (bytes in/out per interface)
+- Disk I/O (reads/writes, latency)
+- Instance count and state
+
+### HAProxy Traffic
+
+Visualizes metrics from the HAProxy stats endpoint (`:8404/metrics`).
+
+- Request rate (frontend and backend)
+- Backend health status (UP/DOWN per server)
+- Active sessions and session rate
+- Traffic volume (bytes in/out)
+- HTTP response codes (2xx, 4xx, 5xx)
+- Connection errors and retries
+
+### Host Resources
+
+Visualizes metrics from node_exporter on each cluster node.
+
+- CPU utilization (%)
+- Memory utilization (%)
+- Disk usage and I/O
+- Network throughput per interface
+- System load averages
+- Filesystem free space
+
+## ACL Configuration
+
+Aether automatically creates per-instance ACLs with default-deny rules
+for all containers on shared OVN networks. The observability stack needs
+careful ACL configuration to allow scrape traffic.
+
+### How Aether ACLs work
+
+When Aether deploys a container, it creates ACLs with reject rules:
+
+| Direction | Priority | Action | Effect |
+|-----------|----------|--------|--------|
+| Egress | 111 | reject | Blocks all outbound by default |
+| Ingress | 100 | reject | Blocks all inbound by default |
+
+These are OVN ACL rules enforced at the logical switch port level.
+
+### monitoring-allow ACL
+
+The observability containers use a `monitoring-allow` ACL with
+`default.*.action=allow` set on all directions. This creates allow rules
+at **priority 111** which matches or exceeds the Aether reject rules,
+effectively overriding the default-deny policy.
+
+All observability containers (monitoring, node-exp-01/02/03) have this
+ACL applied.
+
+### HAProxy scrape access
+
+HAProxy containers are managed by Aether and have their own ACLs with
+default-deny. For Prometheus to scrape HAProxy metrics on `:8404`, the
+HAProxy Aether ACLs need **explicit ingress rules** allowing TCP traffic
+from the monitoring container (10.10.10.70) to port 8404.
+
+Without this, Prometheus targets for HAProxy will show as DOWN.
+
+### NIC state after ACL changes
+
+Changing `security.acls` on a container NIC can cause the NIC to go
+**down**. After modifying ACLs, you may need to bring the NIC back up
+manually or restart the container.
+
+## Resource Budget
+
+| Container | Image | RAM | Disk | Placement |
+|-----------|-------|-----|------|-----------|
+| monitoring | Debian/12 | 2 GiB | 20 GiB | oc-node-02 |
+| node-exp-01 | Alpine | 128 MiB | — | oc-node-01 |
+| node-exp-02 | Alpine | 128 MiB | — | oc-node-02 |
+| node-exp-03 | Alpine | 128 MiB | — | oc-node-03 |
+
+**Totals:**
+- RAM: ~2.4 GiB (2 GiB + 3 x 128 MiB)
+- Disk: 20 GiB (only the monitoring container needs significant storage)
+- OVN forward IPs: 1 (192.168.103.201)
+- OVN network IPs: 4 (10.10.10.70-73)
+
+## Management
+
+The `deploy-observability` script handles the full lifecycle.
+
+### Deploy the stack
+
+```bash
+incusos/deploy-observability --deploy
+```
+
+Creates the monitoring container and all node-exporter containers,
+installs and configures Prometheus, Grafana, Loki, and Promtail,
+provisions dashboards, sets up the OVN network forward, and configures
+ACLs.
+
+### Check status
+
+```bash
+incusos/deploy-observability --status
+```
+
+Shows the state of all containers, scrape target health, Grafana
+accessibility, and OVN forward configuration.
+
+### Clean up
+
+```bash
+incusos/deploy-observability --cleanup
+```
+
+Removes all observability containers, the OVN network forward, and
+associated ACLs. Does not affect monitored targets (Incus nodes,
+HAProxy).
+
+### Health check
+
+```bash
+incusos/deploy-observability --doctor
+```
+
+Verifies prerequisites, checks container health, validates Prometheus
+targets are UP, confirms Grafana datasources are working, and reports
+any ACL issues.
+
+## Troubleshooting
+
+### Targets show as DOWN in Prometheus
+
+1. Check ACL rules on the target containers. Aether's default-deny ACLs
+   block all ingress including Prometheus scrapes.
+2. Verify the NIC is up. Changing `security.acls` can bring the NIC down.
+   Check with `incus exec <container> -- ip link` and bring it up if needed.
+3. For Incus node targets: verify the client certificate and key are
+   correctly placed in the Prometheus config directory and that the cert
+   is trusted by the cluster.
+
+### Grafana can't reach Prometheus
+
+Check that Prometheus is running and listening on localhost:9090 from
+inside the monitoring container:
+
+```bash
+incus exec monitoring -- curl -s http://localhost:9090/-/healthy
+```
+
+If Prometheus is not running, check its service status:
+
+```bash
+incus exec monitoring -- systemctl status prometheus
+```
+
+### No node metrics
+
+1. Verify the node-exporter containers are running and privileged:
+   ```bash
+   incus list node-exp
+   ```
+2. Check that host filesystem mounts are in place:
+   ```bash
+   incus exec node-exp-01 -- ls /host/proc/stat
+   ```
+3. Verify node_exporter is listening:
+   ```bash
+   incus exec node-exp-01 -- wget -qO- http://localhost:9100/metrics | head
+   ```
+
+### HAProxy metrics show as invalid or empty
+
+HAProxy needs the `prometheus-exporter` service enabled in its stats
+configuration. The stats section in `haproxy.cfg` must include:
+
+```
+frontend stats
+    bind *:8404
+    http-request use-service prometheus-exporter if { path /metrics }
+    stats enable
+    stats uri /stats
+```
+
+Without the `use-service prometheus-exporter` directive, the `/metrics`
+path returns HTML stats instead of Prometheus-format metrics.
+
+### Loki not receiving logs
+
+Verify Promtail can reach Loki on the gRPC port:
+
+```bash
+incus exec monitoring -- curl -s http://localhost:3100/ready
+```
+
+Note that Loki uses gRPC port **9096** in this deployment (not the
+default 9095) to avoid conflicts with Promtail's metrics port.