incus-contrib/.claude/rules/observability.md

paths
incusos/deploy-observability incusos/observability-dashboards notes/observability-guide.md

Observability Stack (Prometheus + Grafana + Loki)

Deployment topology

• monitoring container: 10.10.10.70, pinned to oc-node-02, net-prod

  • Prometheus 2.54 on :9090 (15s scrape interval)
  • Grafana 12.4 on :3000 (default creds admin/admin)
  • Loki 3.6 on :3100 (HTTP) and :9096 (gRPC -- NOT default 9095, 9095 conflicts with Promtail metrics port)
  • Promtail 3.6 shipping logs to Loki
  • Debian/12, 2 GiB RAM, 20 GiB disk

• node-exp-01/02/03: 10.10.10.71-73, one per cluster node

  • Alpine, privileged, 128 MiB RAM each
  • Host filesystem mounts: /proc, /sys, / (all read-only)
  • node_exporter on :9100

LAN access

• OVN network forward: 192.168.103.201
  • :3000 -> 10.10.10.70:3000 (Grafana)
  • :9090 -> 10.10.10.70:9090 (Prometheus)

Prometheus scrape targets

Target	Address	Auth	Port
Incus nodes	192.168.102.140-142	Client cert (TLS)	8443
HAProxy	10.10.10.50-51	None	8404
node-exporters	10.10.10.71-73	None	9100

• Incus metrics endpoint: /1.0/metrics on each node, requires client certificate authentication (cluster cert + key in Prometheus config).
• HAProxy metrics: requires http-request use-service prometheus-exporter if { path /metrics } in the stats frontend of haproxy.cfg. Without this, /metrics returns HTML stats, not Prometheus exposition format.

ACL details (critical for scrape connectivity)

• Aether default-deny: every container on shared OVN network gets reject ACLs: priority 111 reject (egress), priority 100 reject (ingress).
• monitoring-allow ACL: set default.*.action=allow on all directions. This creates allow rules at priority 111, matching Aether's reject priority and effectively overriding default-deny.
• All observability containers (monitoring, node-exp-01/02/03) use the monitoring-allow ACL.
• HAProxy containers are Aether-managed with their own ACLs. Need explicit ingress rules allowing TCP from 10.10.10.70 to port 8404 for Prometheus scrapes.

NIC down after security.acls changes

Changing security.acls on a container NIC can cause the NIC to go down. After modifying ACLs:

• Bring NIC up manually: incus exec <container> -- ip link set eth0 up
• Or restart the container: incus restart <container> This is a known OVN behavior, not a bug.

Dashboards

Three pre-provisioned dashboards:

1. Incus Cluster Overview -- instance CPU, memory, network, disk I/O
2. HAProxy Traffic -- requests, backend health, sessions, traffic
3. Host Resources -- node CPU%, memory%, disk, network

Script: deploy-observability

• --deploy -- full stack deployment
• --status -- container state, target health, forward config
• --cleanup -- remove all observability containers and forwards
• --doctor -- prerequisite and health checks

Troubleshooting quick reference

• Targets DOWN: check ACLs, check NIC state after ACL changes
• No HAProxy metrics: need prometheus-exporter route in haproxy.cfg stats
• Loki gRPC: port 9096, not 9095 (Promtail conflict)
• Node metrics missing: check privileged flag, host mounts, node_exporter service