86 lines
3.2 KiB
Markdown
86 lines
3.2 KiB
Markdown
---
|
|
paths:
|
|
- "incusos/deploy-observability"
|
|
- "incusos/observability-dashboards"
|
|
- "notes/observability-guide.md"
|
|
---
|
|
|
|
# Observability Stack (Prometheus + Grafana + Loki)
|
|
|
|
## Deployment topology
|
|
|
|
- **monitoring** container: 10.10.10.70, pinned to oc-node-02, net-prod
|
|
- Prometheus 2.54 on :9090 (15s scrape interval)
|
|
- Grafana 12.4 on :3000 (default creds admin/admin)
|
|
- Loki 3.6 on :3100 (HTTP) and **:9096** (gRPC -- NOT default 9095,
|
|
9095 conflicts with Promtail metrics port)
|
|
- Promtail 3.6 shipping logs to Loki
|
|
- Debian/12, 2 GiB RAM, 20 GiB disk
|
|
|
|
- **node-exp-01/02/03**: 10.10.10.71-73, one per cluster node
|
|
- Alpine, privileged, 128 MiB RAM each
|
|
- Host filesystem mounts: /proc, /sys, / (all read-only)
|
|
- node_exporter on :9100
|
|
|
|
## LAN access
|
|
|
|
- OVN network forward: 192.168.103.201
|
|
- :3000 -> 10.10.10.70:3000 (Grafana)
|
|
- :9090 -> 10.10.10.70:9090 (Prometheus)
|
|
|
|
## Prometheus scrape targets
|
|
|
|
| Target | Address | Auth | Port |
|
|
|--------|---------|------|------|
|
|
| Incus nodes | 192.168.102.140-142 | Client cert (TLS) | 8443 |
|
|
| HAProxy | 10.10.10.50-51 | None | 8404 |
|
|
| node-exporters | 10.10.10.71-73 | None | 9100 |
|
|
|
|
- Incus metrics endpoint: `/1.0/metrics` on each node, requires client
|
|
certificate authentication (cluster cert + key in Prometheus config).
|
|
- HAProxy metrics: requires `http-request use-service prometheus-exporter
|
|
if { path /metrics }` in the stats frontend of haproxy.cfg. Without
|
|
this, /metrics returns HTML stats, not Prometheus exposition format.
|
|
|
|
## ACL details (critical for scrape connectivity)
|
|
|
|
- **Aether default-deny**: every container on shared OVN network gets
|
|
reject ACLs: priority 111 reject (egress), priority 100 reject (ingress).
|
|
- **monitoring-allow ACL**: set `default.*.action=allow` on all directions.
|
|
This creates allow rules at priority 111, matching Aether's reject
|
|
priority and effectively overriding default-deny.
|
|
- All observability containers (monitoring, node-exp-01/02/03) use the
|
|
monitoring-allow ACL.
|
|
- **HAProxy containers** are Aether-managed with their own ACLs. Need
|
|
explicit ingress rules allowing TCP from 10.10.10.70 to port 8404 for
|
|
Prometheus scrapes.
|
|
|
|
## NIC down after security.acls changes
|
|
|
|
Changing `security.acls` on a container NIC can cause the NIC to go
|
|
**down**. After modifying ACLs:
|
|
- Bring NIC up manually: `incus exec <container> -- ip link set eth0 up`
|
|
- Or restart the container: `incus restart <container>`
|
|
This is a known OVN behavior, not a bug.
|
|
|
|
## Dashboards
|
|
|
|
Three pre-provisioned dashboards:
|
|
1. **Incus Cluster Overview** -- instance CPU, memory, network, disk I/O
|
|
2. **HAProxy Traffic** -- requests, backend health, sessions, traffic
|
|
3. **Host Resources** -- node CPU%, memory%, disk, network
|
|
|
|
## Script: deploy-observability
|
|
|
|
- `--deploy` -- full stack deployment
|
|
- `--status` -- container state, target health, forward config
|
|
- `--cleanup` -- remove all observability containers and forwards
|
|
- `--doctor` -- prerequisite and health checks
|
|
|
|
## Troubleshooting quick reference
|
|
|
|
- Targets DOWN: check ACLs, check NIC state after ACL changes
|
|
- No HAProxy metrics: need prometheus-exporter route in haproxy.cfg stats
|
|
- Loki gRPC: port 9096, not 9095 (Promtail conflict)
|
|
- Node metrics missing: check privileged flag, host mounts, node_exporter service
|