incus-contrib/notes/observability-guide.md

# Observability Stack — Prometheus, Grafana, Loki on Incus

A lightweight observability stack deployed as Incus containers on the lab
cluster. Provides metric collection (Prometheus), dashboards (Grafana),
and log aggregation (Loki) for Incus nodes, HAProxy load balancers, and
host-level resources.

The stack runs entirely on OVN and is managed by the `deploy-observability`
script. All containers use the `monitoring-allow` ACL to permit scrape
traffic on the shared network.

## Architecture

```mermaid
flowchart TD
    lan(["LAN 192.168.103.0/24"])

    subgraph fwd["OVN Forward · 192.168.103.201"]
        fwd_grafana[":3000 → Grafana"]
        fwd_prom[":9090 → Prometheus"]
    end

    subgraph mon["monitoring · 10.10.10.70 · oc-node-02"]
        prometheus["Prometheus :9090"]
        grafana["Grafana :3000"]
        loki["Loki :3100<br/>gRPC :9096"]
        promtail["Promtail"]
    end

    subgraph targets["Scrape Targets"]
        incus["Incus nodes :8443<br/>.140 · .141 · .142"]
        haproxy["HAProxy :8404<br/>.50 · .51"]
        ne["node-exporters :9100<br/>.71 · .72 · .73"]
    end

    lan --> fwd
    fwd --> mon
    prometheus -->|"client cert"| incus
    prometheus --> haproxy
    prometheus --> ne

    classDef external fill:#f5f5f5,color:#333,stroke:#999
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
    classDef instance fill:#56B4E9,color:#fff,stroke:#3a8fbf
    classDef node fill:#009E73,color:#fff,stroke:#007a5e

    class lan external
    class fwd_grafana,fwd_prom network
    class prometheus,grafana,loki,promtail mgmt
    class incus,haproxy,ne instance

    style fwd fill:#e0eef8,stroke:#0072B2
    style mon fill:#f5e6f0,stroke:#CC79A7
    style targets fill:#e0f2fe,stroke:#56B4E9
```

## Components

### Prometheus 2.54

Metric collection engine. Scrapes all targets at a 15-second interval.

- Stores metrics locally in the monitoring container
- Client certificate authentication for Incus `/1.0/metrics` endpoints
  (uses cluster client cert + key)
- Plain HTTP scrape for HAProxy stats (`:8404/metrics`) and
  node-exporters (`:9100/metrics`)

### Grafana 12.4

Dashboard and visualization frontend.

- Default credentials: `admin` / `admin`
- Pre-provisioned datasources for Prometheus and Loki
- Three dashboards deployed automatically (see Dashboards section)
- Accessible from LAN via OVN network forward

### Loki 3.6

Log aggregation backend.

- HTTP API on port 3100
- gRPC on port **9096** (not the default 9095 -- avoids conflict with
  Promtail's own metrics port)
- 7-day retention policy
- Receives logs from Promtail running in the same container

### Promtail 3.6

Log shipping agent, co-located with Loki in the monitoring container.

- Pushes logs to Loki via gRPC on localhost:9096
- Scrapes container logs and system journal

### node_exporter

Host-level metrics via privileged Alpine containers.

- One container per cluster node, pinned with `--target` placement
- Privileged containers with host filesystem bind-mounts:
  - `/proc` (host) -> `/host/proc` (read-only)
  - `/sys` (host) -> `/host/sys` (read-only)
  - `/` (host) -> `/host/rootfs` (read-only)
- Exposes standard node_exporter metrics on `:9100`
- Uses `monitoring-allow` ACL for network access

## Access

| Service | URL | Credentials |
|---------|-----|-------------|
| Grafana | http://192.168.103.201:3000 | `admin` / `admin` |
| Prometheus | http://192.168.103.201:9090 | None (open) |

Both services are accessible from the LAN through an OVN network forward
on IP `192.168.103.201`. The forward maps external ports directly to the
monitoring container's internal ports (no translation).

Prometheus and Loki are not exposed externally by default. They are
accessible only from within the OVN network or through the Grafana
datasource proxy.

## Dashboards

Three dashboards are provisioned automatically during deployment.

### Incus Cluster Overview

Visualizes metrics scraped from the Incus `/1.0/metrics` endpoint on each
cluster node.

- Instance CPU usage (per-instance, per-node)
- Memory usage and allocation
- Network I/O (bytes in/out per interface)
- Disk I/O (reads/writes, latency)
- Instance count and state

### HAProxy Traffic

Visualizes metrics from the HAProxy stats endpoint (`:8404/metrics`).

- Request rate (frontend and backend)
- Backend health status (UP/DOWN per server)
- Active sessions and session rate
- Traffic volume (bytes in/out)
- HTTP response codes (2xx, 4xx, 5xx)
- Connection errors and retries

### Host Resources

Visualizes metrics from node_exporter on each cluster node.

- CPU utilization (%)
- Memory utilization (%)
- Disk usage and I/O
- Network throughput per interface
- System load averages
- Filesystem free space

## ACL Configuration

Aether automatically creates per-instance ACLs with default-deny rules
for all containers on shared OVN networks. The observability stack needs
careful ACL configuration to allow scrape traffic.

### How Aether ACLs work

When Aether deploys a container, it creates ACLs with reject rules:

| Direction | Priority | Action | Effect |
|-----------|----------|--------|--------|
| Egress | 111 | reject | Blocks all outbound by default |
| Ingress | 100 | reject | Blocks all inbound by default |

These are OVN ACL rules enforced at the logical switch port level.

### monitoring-allow ACL

The observability containers use a `monitoring-allow` ACL with
`default.*.action=allow` set on all directions. This creates allow rules
at **priority 111** which matches or exceeds the Aether reject rules,
effectively overriding the default-deny policy.

All observability containers (monitoring, node-exp-01/02/03) have this
ACL applied.

### HAProxy scrape access

HAProxy containers are managed by Aether and have their own ACLs with
default-deny. For Prometheus to scrape HAProxy metrics on `:8404`, the
HAProxy Aether ACLs need **explicit ingress rules** allowing TCP traffic
from the monitoring container (10.10.10.70) to port 8404.

Without this, Prometheus targets for HAProxy will show as DOWN.

### NIC state after ACL changes

Changing `security.acls` on a container NIC can cause the NIC to go
**down**. After modifying ACLs, you may need to bring the NIC back up
manually or restart the container.

## Resource Budget

| Container | Image | RAM | Disk | Placement |
|-----------|-------|-----|------|-----------|
| monitoring | Debian/12 | 2 GiB | 20 GiB | oc-node-02 |
| node-exp-01 | Alpine | 128 MiB | — | oc-node-01 |
| node-exp-02 | Alpine | 128 MiB | — | oc-node-02 |
| node-exp-03 | Alpine | 128 MiB | — | oc-node-03 |

**Totals:**
- RAM: ~2.4 GiB (2 GiB + 3 x 128 MiB)
- Disk: 20 GiB (only the monitoring container needs significant storage)
- OVN forward IPs: 1 (192.168.103.201)
- OVN network IPs: 4 (10.10.10.70-73)

## Management

The `deploy-observability` script handles the full lifecycle.

### Deploy the stack

```bash
incusos/deploy-observability --deploy
```

Creates the monitoring container and all node-exporter containers,
installs and configures Prometheus, Grafana, Loki, and Promtail,
provisions dashboards, sets up the OVN network forward, and configures
ACLs.

### Check status

```bash
incusos/deploy-observability --status
```

Shows the state of all containers, scrape target health, Grafana
accessibility, and OVN forward configuration.

### Clean up

```bash
incusos/deploy-observability --cleanup
```

Removes all observability containers, the OVN network forward, and
associated ACLs. Does not affect monitored targets (Incus nodes,
HAProxy).

### Health check

```bash
incusos/deploy-observability --doctor
```

Verifies prerequisites, checks container health, validates Prometheus
targets are UP, confirms Grafana datasources are working, and reports
any ACL issues.

## Troubleshooting

### Targets show as DOWN in Prometheus

1. Check ACL rules on the target containers. Aether's default-deny ACLs
   block all ingress including Prometheus scrapes.
2. Verify the NIC is up. Changing `security.acls` can bring the NIC down.
   Check with `incus exec <container> -- ip link` and bring it up if needed.
3. For Incus node targets: verify the client certificate and key are
   correctly placed in the Prometheus config directory and that the cert
   is trusted by the cluster.

### Grafana can't reach Prometheus

Check that Prometheus is running and listening on localhost:9090 from
inside the monitoring container:

```bash
incus exec monitoring -- curl -s http://localhost:9090/-/healthy
```

If Prometheus is not running, check its service status:

```bash
incus exec monitoring -- systemctl status prometheus
```

### No node metrics

1. Verify the node-exporter containers are running and privileged:
   ```bash
   incus list node-exp
   ```
2. Check that host filesystem mounts are in place:
   ```bash
   incus exec node-exp-01 -- ls /host/proc/stat
   ```
3. Verify node_exporter is listening:
   ```bash
   incus exec node-exp-01 -- wget -qO- http://localhost:9100/metrics | head
   ```

### HAProxy metrics show as invalid or empty

HAProxy needs the `prometheus-exporter` service enabled in its stats
configuration. The stats section in `haproxy.cfg` must include:

```
frontend stats
    bind *:8404
    http-request use-service prometheus-exporter if { path /metrics }
    stats enable
    stats uri /stats
```

Without the `use-service prometheus-exporter` directive, the `/metrics`
path returns HTML stats instead of Prometheus-format metrics.

### Loki not receiving logs

Verify Promtail can reach Loki on the gRPC port:

```bash
incus exec monitoring -- curl -s http://localhost:3100/ready
```

Note that Loki uses gRPC port **9096** in this deployment (not the
default 9095) to avoid conflicts with Promtail's metrics port.