338 lines
11 KiB
Markdown
338 lines
11 KiB
Markdown
# Observability Stack — Prometheus, Grafana, Loki on Incus
|
|
|
|
A lightweight observability stack deployed as Incus containers on the lab
|
|
cluster. Provides metric collection (Prometheus), dashboards (Grafana),
|
|
and log aggregation (Loki) for Incus nodes, HAProxy load balancers, and
|
|
host-level resources.
|
|
|
|
The stack runs entirely on OVN and is managed by the `deploy-observability`
|
|
script. All containers use the `monitoring-allow` ACL to permit scrape
|
|
traffic on the shared network.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
LAN (192.168.103.0/24)
|
|
|
|
|
| OVN network forward (192.168.103.201)
|
|
| :3000 -> 10.10.10.70:3000 (Grafana)
|
|
| :9090 -> 10.10.10.70:9090 (Prometheus)
|
|
|
|
|
═══════╪══════════════════════════════════════════════
|
|
| OVN net-prod (10.10.10.0/24)
|
|
|
|
|
v
|
|
+----------------------------------------------+
|
|
| monitoring (10.10.10.70, oc-node-02) |
|
|
| Debian/12 container — 2 GiB RAM, 20 GiB |
|
|
| |
|
|
| +-------------+ +-----------+ |
|
|
| | Prometheus | | Grafana | |
|
|
| | :9090 | | :3000 | |
|
|
| +-------------+ +-----------+ |
|
|
| +-------------+ +-----------+ |
|
|
| | Loki | | Promtail | |
|
|
| | :3100 | | (agent) | |
|
|
| | gRPC :9096 | | | |
|
|
| +-------------+ +-----------+ |
|
|
+----------------------------------------------+
|
|
|
|
|
| Scrape targets
|
|
|
|
|
+-------> Incus nodes (client cert auth, :8443)
|
|
| 192.168.102.140 /1.0/metrics
|
|
| 192.168.102.141 /1.0/metrics
|
|
| 192.168.102.142 /1.0/metrics
|
|
|
|
|
+-------> HAProxy (:8404)
|
|
| 10.10.10.50 /metrics
|
|
| 10.10.10.51 /metrics
|
|
|
|
|
+-------> node-exporters (:9100)
|
|
10.10.10.71 node-exp-01 (oc-node-01)
|
|
10.10.10.72 node-exp-02 (oc-node-02)
|
|
10.10.10.73 node-exp-03 (oc-node-03)
|
|
|
|
+------------------+ +------------------+ +------------------+
|
|
| node-exp-01 | | node-exp-02 | | node-exp-03 |
|
|
| 10.10.10.71 | | 10.10.10.72 | | 10.10.10.73 |
|
|
| Alpine, privil. | | Alpine, privil. | | Alpine, privil. |
|
|
| :9100 | | :9100 | | :9100 |
|
|
| host /proc,/sys | | host /proc,/sys | | host /proc,/sys |
|
|
| 128 MiB RAM | | 128 MiB RAM | | 128 MiB RAM |
|
|
| pinned to node | | pinned to node | | pinned to node |
|
|
+------------------+ +------------------+ +------------------+
|
|
```
|
|
|
|
## Components
|
|
|
|
### Prometheus 2.54
|
|
|
|
Metric collection engine. Scrapes all targets at a 15-second interval.
|
|
|
|
- Stores metrics locally in the monitoring container
|
|
- Client certificate authentication for Incus `/1.0/metrics` endpoints
|
|
(uses cluster client cert + key)
|
|
- Plain HTTP scrape for HAProxy stats (`:8404/metrics`) and
|
|
node-exporters (`:9100/metrics`)
|
|
|
|
### Grafana 12.4
|
|
|
|
Dashboard and visualization frontend.
|
|
|
|
- Default credentials: `admin` / `admin`
|
|
- Pre-provisioned datasources for Prometheus and Loki
|
|
- Three dashboards deployed automatically (see Dashboards section)
|
|
- Accessible from LAN via OVN network forward
|
|
|
|
### Loki 3.6
|
|
|
|
Log aggregation backend.
|
|
|
|
- HTTP API on port 3100
|
|
- gRPC on port **9096** (not the default 9095 -- avoids conflict with
|
|
Promtail's own metrics port)
|
|
- 7-day retention policy
|
|
- Receives logs from Promtail running in the same container
|
|
|
|
### Promtail 3.6
|
|
|
|
Log shipping agent, co-located with Loki in the monitoring container.
|
|
|
|
- Pushes logs to Loki via gRPC on localhost:9096
|
|
- Scrapes container logs and system journal
|
|
|
|
### node_exporter
|
|
|
|
Host-level metrics via privileged Alpine containers.
|
|
|
|
- One container per cluster node, pinned with `--target` placement
|
|
- Privileged containers with host filesystem bind-mounts:
|
|
- `/proc` (host) -> `/host/proc` (read-only)
|
|
- `/sys` (host) -> `/host/sys` (read-only)
|
|
- `/` (host) -> `/host/rootfs` (read-only)
|
|
- Exposes standard node_exporter metrics on `:9100`
|
|
- Uses `monitoring-allow` ACL for network access
|
|
|
|
## Access
|
|
|
|
| Service | URL | Credentials |
|
|
|---------|-----|-------------|
|
|
| Grafana | http://192.168.103.201:3000 | `admin` / `admin` |
|
|
| Prometheus | http://192.168.103.201:9090 | None (open) |
|
|
|
|
Both services are accessible from the LAN through an OVN network forward
|
|
on IP `192.168.103.201`. The forward maps external ports directly to the
|
|
monitoring container's internal ports (no translation).
|
|
|
|
Prometheus and Loki are not exposed externally by default. They are
|
|
accessible only from within the OVN network or through the Grafana
|
|
datasource proxy.
|
|
|
|
## Dashboards
|
|
|
|
Three dashboards are provisioned automatically during deployment.
|
|
|
|
### Incus Cluster Overview
|
|
|
|
Visualizes metrics scraped from the Incus `/1.0/metrics` endpoint on each
|
|
cluster node.
|
|
|
|
- Instance CPU usage (per-instance, per-node)
|
|
- Memory usage and allocation
|
|
- Network I/O (bytes in/out per interface)
|
|
- Disk I/O (reads/writes, latency)
|
|
- Instance count and state
|
|
|
|
### HAProxy Traffic
|
|
|
|
Visualizes metrics from the HAProxy stats endpoint (`:8404/metrics`).
|
|
|
|
- Request rate (frontend and backend)
|
|
- Backend health status (UP/DOWN per server)
|
|
- Active sessions and session rate
|
|
- Traffic volume (bytes in/out)
|
|
- HTTP response codes (2xx, 4xx, 5xx)
|
|
- Connection errors and retries
|
|
|
|
### Host Resources
|
|
|
|
Visualizes metrics from node_exporter on each cluster node.
|
|
|
|
- CPU utilization (%)
|
|
- Memory utilization (%)
|
|
- Disk usage and I/O
|
|
- Network throughput per interface
|
|
- System load averages
|
|
- Filesystem free space
|
|
|
|
## ACL Configuration
|
|
|
|
Aether automatically creates per-instance ACLs with default-deny rules
|
|
for all containers on shared OVN networks. The observability stack needs
|
|
careful ACL configuration to allow scrape traffic.
|
|
|
|
### How Aether ACLs work
|
|
|
|
When Aether deploys a container, it creates ACLs with reject rules:
|
|
|
|
| Direction | Priority | Action | Effect |
|
|
|-----------|----------|--------|--------|
|
|
| Egress | 111 | reject | Blocks all outbound by default |
|
|
| Ingress | 100 | reject | Blocks all inbound by default |
|
|
|
|
These are OVN ACL rules enforced at the logical switch port level.
|
|
|
|
### monitoring-allow ACL
|
|
|
|
The observability containers use a `monitoring-allow` ACL with
|
|
`default.*.action=allow` set on all directions. This creates allow rules
|
|
at **priority 111** which matches or exceeds the Aether reject rules,
|
|
effectively overriding the default-deny policy.
|
|
|
|
All observability containers (monitoring, node-exp-01/02/03) have this
|
|
ACL applied.
|
|
|
|
### HAProxy scrape access
|
|
|
|
HAProxy containers are managed by Aether and have their own ACLs with
|
|
default-deny. For Prometheus to scrape HAProxy metrics on `:8404`, the
|
|
HAProxy Aether ACLs need **explicit ingress rules** allowing TCP traffic
|
|
from the monitoring container (10.10.10.70) to port 8404.
|
|
|
|
Without this, Prometheus targets for HAProxy will show as DOWN.
|
|
|
|
### NIC state after ACL changes
|
|
|
|
Changing `security.acls` on a container NIC can cause the NIC to go
|
|
**down**. After modifying ACLs, you may need to bring the NIC back up
|
|
manually or restart the container.
|
|
|
|
## Resource Budget
|
|
|
|
| Container | Image | RAM | Disk | Placement |
|
|
|-----------|-------|-----|------|-----------|
|
|
| monitoring | Debian/12 | 2 GiB | 20 GiB | oc-node-02 |
|
|
| node-exp-01 | Alpine | 128 MiB | — | oc-node-01 |
|
|
| node-exp-02 | Alpine | 128 MiB | — | oc-node-02 |
|
|
| node-exp-03 | Alpine | 128 MiB | — | oc-node-03 |
|
|
|
|
**Totals:**
|
|
- RAM: ~2.4 GiB (2 GiB + 3 x 128 MiB)
|
|
- Disk: 20 GiB (only the monitoring container needs significant storage)
|
|
- OVN forward IPs: 1 (192.168.103.201)
|
|
- OVN network IPs: 4 (10.10.10.70-73)
|
|
|
|
## Management
|
|
|
|
The `deploy-observability` script handles the full lifecycle.
|
|
|
|
### Deploy the stack
|
|
|
|
```bash
|
|
incusos/deploy-observability --deploy
|
|
```
|
|
|
|
Creates the monitoring container and all node-exporter containers,
|
|
installs and configures Prometheus, Grafana, Loki, and Promtail,
|
|
provisions dashboards, sets up the OVN network forward, and configures
|
|
ACLs.
|
|
|
|
### Check status
|
|
|
|
```bash
|
|
incusos/deploy-observability --status
|
|
```
|
|
|
|
Shows the state of all containers, scrape target health, Grafana
|
|
accessibility, and OVN forward configuration.
|
|
|
|
### Clean up
|
|
|
|
```bash
|
|
incusos/deploy-observability --cleanup
|
|
```
|
|
|
|
Removes all observability containers, the OVN network forward, and
|
|
associated ACLs. Does not affect monitored targets (Incus nodes,
|
|
HAProxy).
|
|
|
|
### Health check
|
|
|
|
```bash
|
|
incusos/deploy-observability --doctor
|
|
```
|
|
|
|
Verifies prerequisites, checks container health, validates Prometheus
|
|
targets are UP, confirms Grafana datasources are working, and reports
|
|
any ACL issues.
|
|
|
|
## Troubleshooting
|
|
|
|
### Targets show as DOWN in Prometheus
|
|
|
|
1. Check ACL rules on the target containers. Aether's default-deny ACLs
|
|
block all ingress including Prometheus scrapes.
|
|
2. Verify the NIC is up. Changing `security.acls` can bring the NIC down.
|
|
Check with `incus exec <container> -- ip link` and bring it up if needed.
|
|
3. For Incus node targets: verify the client certificate and key are
|
|
correctly placed in the Prometheus config directory and that the cert
|
|
is trusted by the cluster.
|
|
|
|
### Grafana can't reach Prometheus
|
|
|
|
Check that Prometheus is running and listening on localhost:9090 from
|
|
inside the monitoring container:
|
|
|
|
```bash
|
|
incus exec monitoring -- curl -s http://localhost:9090/-/healthy
|
|
```
|
|
|
|
If Prometheus is not running, check its service status:
|
|
|
|
```bash
|
|
incus exec monitoring -- systemctl status prometheus
|
|
```
|
|
|
|
### No node metrics
|
|
|
|
1. Verify the node-exporter containers are running and privileged:
|
|
```bash
|
|
incus list node-exp
|
|
```
|
|
2. Check that host filesystem mounts are in place:
|
|
```bash
|
|
incus exec node-exp-01 -- ls /host/proc/stat
|
|
```
|
|
3. Verify node_exporter is listening:
|
|
```bash
|
|
incus exec node-exp-01 -- wget -qO- http://localhost:9100/metrics | head
|
|
```
|
|
|
|
### HAProxy metrics show as invalid or empty
|
|
|
|
HAProxy needs the `prometheus-exporter` service enabled in its stats
|
|
configuration. The stats section in `haproxy.cfg` must include:
|
|
|
|
```
|
|
frontend stats
|
|
bind *:8404
|
|
http-request use-service prometheus-exporter if { path /metrics }
|
|
stats enable
|
|
stats uri /stats
|
|
```
|
|
|
|
Without the `use-service prometheus-exporter` directive, the `/metrics`
|
|
path returns HTML stats instead of Prometheus-format metrics.
|
|
|
|
### Loki not receiving logs
|
|
|
|
Verify Promtail can reach Loki on the gRPC port:
|
|
|
|
```bash
|
|
incus exec monitoring -- curl -s http://localhost:3100/ready
|
|
```
|
|
|
|
Note that Loki uses gRPC port **9096** in this deployment (not the
|
|
default 9095) to avoid conflicts with Promtail's metrics port.
|