--- paths: - "incusos/deploy-observability" - "incusos/observability-dashboards" - "notes/observability-guide.md" --- # Observability Stack (Prometheus + Grafana + Loki) ## Deployment topology - **monitoring** container: 10.10.10.70, pinned to oc-node-02, net-prod - Prometheus 2.54 on :9090 (15s scrape interval) - Grafana 12.4 on :3000 (default creds admin/admin) - Loki 3.6 on :3100 (HTTP) and **:9096** (gRPC -- NOT default 9095, 9095 conflicts with Promtail metrics port) - Promtail 3.6 shipping logs to Loki - Debian/12, 2 GiB RAM, 20 GiB disk - **node-exp-01/02/03**: 10.10.10.71-73, one per cluster node - Alpine, privileged, 128 MiB RAM each - Host filesystem mounts: /proc, /sys, / (all read-only) - node_exporter on :9100 ## LAN access - OVN network forward: 192.168.103.201 - :3000 -> 10.10.10.70:3000 (Grafana) - :9090 -> 10.10.10.70:9090 (Prometheus) ## Prometheus scrape targets | Target | Address | Auth | Port | |--------|---------|------|------| | Incus nodes | 192.168.102.140-142 | Client cert (TLS) | 8443 | | HAProxy | 10.10.10.50-51 | None | 8404 | | node-exporters | 10.10.10.71-73 | None | 9100 | - Incus metrics endpoint: `/1.0/metrics` on each node, requires client certificate authentication (cluster cert + key in Prometheus config). - HAProxy metrics: requires `http-request use-service prometheus-exporter if { path /metrics }` in the stats frontend of haproxy.cfg. Without this, /metrics returns HTML stats, not Prometheus exposition format. ## ACL details (critical for scrape connectivity) - **Aether default-deny**: every container on shared OVN network gets reject ACLs: priority 111 reject (egress), priority 100 reject (ingress). - **monitoring-allow ACL**: set `default.*.action=allow` on all directions. This creates allow rules at priority 111, matching Aether's reject priority and effectively overriding default-deny. - All observability containers (monitoring, node-exp-01/02/03) use the monitoring-allow ACL. - **HAProxy containers** are Aether-managed with their own ACLs. Need explicit ingress rules allowing TCP from 10.10.10.70 to port 8404 for Prometheus scrapes. ## NIC down after security.acls changes Changing `security.acls` on a container NIC can cause the NIC to go **down**. After modifying ACLs: - Bring NIC up manually: `incus exec -- ip link set eth0 up` - Or restart the container: `incus restart ` This is a known OVN behavior, not a bug. ## Dashboards Three pre-provisioned dashboards: 1. **Incus Cluster Overview** -- instance CPU, memory, network, disk I/O 2. **HAProxy Traffic** -- requests, backend health, sessions, traffic 3. **Host Resources** -- node CPU%, memory%, disk, network ## Script: deploy-observability - `--deploy` -- full stack deployment - `--status` -- container state, target health, forward config - `--cleanup` -- remove all observability containers and forwards - `--doctor` -- prerequisite and health checks ## Troubleshooting quick reference - Targets DOWN: check ACLs, check NIC state after ACL changes - No HAProxy metrics: need prometheus-exporter route in haproxy.cfg stats - Loki gRPC: port 9096, not 9095 (Promtail conflict) - Node metrics missing: check privileged flag, host mounts, node_exporter service