---
paths:
  - "incusos/deploy-observability"
  - "incusos/observability-dashboards"
  - "notes/observability-guide.md"
---

# Observability Stack (Prometheus + Grafana + Loki)

## Deployment topology

- **monitoring** container: 10.10.10.70, pinned to oc-node-02, net-prod
  - Prometheus 2.54 on :9090 (15s scrape interval)
  - Grafana 12.4 on :3000 (default creds admin/admin)
  - Loki 3.6 on :3100 (HTTP) and **:9096** (gRPC -- NOT default 9095,
    9095 conflicts with Promtail metrics port)
  - Promtail 3.6 shipping logs to Loki
  - Debian/12, 2 GiB RAM, 20 GiB disk

- **node-exp-01/02/03**: 10.10.10.71-73, one per cluster node
  - Alpine, privileged, 128 MiB RAM each
  - Host filesystem mounts: /proc, /sys, / (all read-only)
  - node_exporter on :9100

## LAN access

- OVN network forward: 192.168.103.201
  - :3000 -> 10.10.10.70:3000 (Grafana)
  - :9090 -> 10.10.10.70:9090 (Prometheus)

## Prometheus scrape targets

| Target | Address | Auth | Port |
|--------|---------|------|------|
| Incus nodes | 192.168.102.140-142 | Client cert (TLS) | 8443 |
| HAProxy | 10.10.10.50-51 | None | 8404 |
| node-exporters | 10.10.10.71-73 | None | 9100 |

- Incus metrics endpoint: `/1.0/metrics` on each node, requires client
  certificate authentication (cluster cert + key in Prometheus config).
- HAProxy metrics: requires `http-request use-service prometheus-exporter
  if { path /metrics }` in the stats frontend of haproxy.cfg. Without
  this, /metrics returns HTML stats, not Prometheus exposition format.

## ACL details (critical for scrape connectivity)

- **Aether default-deny**: every container on shared OVN network gets
  reject ACLs: priority 111 reject (egress), priority 100 reject (ingress).
- **monitoring-allow ACL**: set `default.*.action=allow` on all directions.
  This creates allow rules at priority 111, matching Aether's reject
  priority and effectively overriding default-deny.
- All observability containers (monitoring, node-exp-01/02/03) use the
  monitoring-allow ACL.
- **HAProxy containers** are Aether-managed with their own ACLs. Need
  explicit ingress rules allowing TCP from 10.10.10.70 to port 8404 for
  Prometheus scrapes.

## NIC down after security.acls changes

Changing `security.acls` on a container NIC can cause the NIC to go
**down**. After modifying ACLs:
- Bring NIC up manually: `incus exec <container> -- ip link set eth0 up`
- Or restart the container: `incus restart <container>`
This is a known OVN behavior, not a bug.

## Dashboards

Three pre-provisioned dashboards:
1. **Incus Cluster Overview** -- instance CPU, memory, network, disk I/O
2. **HAProxy Traffic** -- requests, backend health, sessions, traffic
3. **Host Resources** -- node CPU%, memory%, disk, network

## Script: deploy-observability

- `--deploy` -- full stack deployment
- `--status` -- container state, target health, forward config
- `--cleanup` -- remove all observability containers and forwards
- `--doctor` -- prerequisite and health checks

## Troubleshooting quick reference

- Targets DOWN: check ACLs, check NIC state after ACL changes
- No HAProxy metrics: need prometheus-exporter route in haproxy.cfg stats
- Loki gRPC: port 9096, not 9095 (Promtail conflict)
- Node metrics missing: check privileged flag, host mounts, node_exporter service
