From cc017b648d43b4d47f104cde2f6fe80cbb4a2515 Mon Sep 17 00:00:00 2001 From: Maarten Date: Tue, 24 Feb 2026 15:44:59 +0100 Subject: [PATCH] Add observability stack: Prometheus, Grafana, Loki with dashboards Deploy and validate full monitoring stack on the Incus cluster: - Prometheus scraping 9 targets (3 Incus nodes, 2 HAProxy, 3 node-exporters, self) - Grafana with 3 dashboards (cluster overview, HAProxy traffic, host resources) - Loki + Promtail for log aggregation - OVN network forward for LAN access (192.168.103.201:3000/9090) - deploy-observability script with --deploy/--status/--cleanup/--doctor - Solved OVN ACL challenges: Aether default-deny requires monitoring-allow ACL Co-Authored-By: Claude Opus 4.6 --- .claude/rules/observability.md | 85 ++ CLAUDE.md | 3 + incusos/deploy-observability | 1197 +++++++++++++++++ .../cluster-overview.json | 559 ++++++++ .../haproxy-traffic.json | 518 +++++++ .../host-resources.json | 429 ++++++ notes/observability-guide.md | 337 +++++ 7 files changed, 3128 insertions(+) create mode 100644 .claude/rules/observability.md create mode 100755 incusos/deploy-observability create mode 100644 incusos/observability-dashboards/cluster-overview.json create mode 100644 incusos/observability-dashboards/haproxy-traffic.json create mode 100644 incusos/observability-dashboards/host-resources.json create mode 100644 notes/observability-guide.md diff --git a/.claude/rules/observability.md b/.claude/rules/observability.md new file mode 100644 index 0000000..df45ac2 --- /dev/null +++ b/.claude/rules/observability.md @@ -0,0 +1,85 @@ +--- +paths: + - "incusos/deploy-observability" + - "incusos/observability-dashboards" + - "notes/observability-guide.md" +--- + +# Observability Stack (Prometheus + Grafana + Loki) + +## Deployment topology + +- **monitoring** container: 10.10.10.70, pinned to oc-node-02, net-prod + - Prometheus 2.54 on :9090 (15s scrape interval) + - Grafana 12.4 on :3000 (default creds admin/admin) + - Loki 3.6 on :3100 (HTTP) and **:9096** (gRPC -- NOT default 9095, + 9095 conflicts with Promtail metrics port) + - Promtail 3.6 shipping logs to Loki + - Debian/12, 2 GiB RAM, 20 GiB disk + +- **node-exp-01/02/03**: 10.10.10.71-73, one per cluster node + - Alpine, privileged, 128 MiB RAM each + - Host filesystem mounts: /proc, /sys, / (all read-only) + - node_exporter on :9100 + +## LAN access + +- OVN network forward: 192.168.103.201 + - :3000 -> 10.10.10.70:3000 (Grafana) + - :9090 -> 10.10.10.70:9090 (Prometheus) + +## Prometheus scrape targets + +| Target | Address | Auth | Port | +|--------|---------|------|------| +| Incus nodes | 192.168.102.140-142 | Client cert (TLS) | 8443 | +| HAProxy | 10.10.10.50-51 | None | 8404 | +| node-exporters | 10.10.10.71-73 | None | 9100 | + +- Incus metrics endpoint: `/1.0/metrics` on each node, requires client + certificate authentication (cluster cert + key in Prometheus config). +- HAProxy metrics: requires `http-request use-service prometheus-exporter + if { path /metrics }` in the stats frontend of haproxy.cfg. Without + this, /metrics returns HTML stats, not Prometheus exposition format. + +## ACL details (critical for scrape connectivity) + +- **Aether default-deny**: every container on shared OVN network gets + reject ACLs: priority 111 reject (egress), priority 100 reject (ingress). +- **monitoring-allow ACL**: set `default.*.action=allow` on all directions. + This creates allow rules at priority 111, matching Aether's reject + priority and effectively overriding default-deny. +- All observability containers (monitoring, node-exp-01/02/03) use the + monitoring-allow ACL. +- **HAProxy containers** are Aether-managed with their own ACLs. Need + explicit ingress rules allowing TCP from 10.10.10.70 to port 8404 for + Prometheus scrapes. + +## NIC down after security.acls changes + +Changing `security.acls` on a container NIC can cause the NIC to go +**down**. After modifying ACLs: +- Bring NIC up manually: `incus exec -- ip link set eth0 up` +- Or restart the container: `incus restart ` +This is a known OVN behavior, not a bug. + +## Dashboards + +Three pre-provisioned dashboards: +1. **Incus Cluster Overview** -- instance CPU, memory, network, disk I/O +2. **HAProxy Traffic** -- requests, backend health, sessions, traffic +3. **Host Resources** -- node CPU%, memory%, disk, network + +## Script: deploy-observability + +- `--deploy` -- full stack deployment +- `--status` -- container state, target health, forward config +- `--cleanup` -- remove all observability containers and forwards +- `--doctor` -- prerequisite and health checks + +## Troubleshooting quick reference + +- Targets DOWN: check ACLs, check NIC state after ACL changes +- No HAProxy metrics: need prometheus-exporter route in haproxy.cfg stats +- Loki gRPC: port 9096, not 9095 (Promtail conflict) +- Node metrics missing: check privileged flag, host mounts, node_exporter service diff --git a/CLAUDE.md b/CLAUDE.md index 4aca880..29d2537 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,9 @@ incus-contrib/ │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle │ ├── deploy-awx # AWX deployment + management on Incus cluster │ ├── deploy-haproxy # HAProxy LB deployment + management via Aether +│ ├── deploy-observability # Prometheus + Grafana + Loki observability stack │ ├── awx-manifests/ # K8s manifests for AWX Operator + instance +│ ├── observability-dashboards/ # Grafana dashboard JSON exports │ ├── helpers/ │ │ ├── proxmox-screenshot # VMID -> PNG console screenshot │ │ ├── proxmox-api # Authenticated API calls (handles ! in token) @@ -187,6 +189,7 @@ which files are being edited: | `networking-storage.md` | networking, storage, migration, UTM guides | | `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide | | `haproxy-lb.md` | deploy-haproxy, HAProxy guide | +| `observability.md` | deploy-observability, observability dashboards/guide | | `ovn-internals.md` | ovn-inspect, ovn-deep-dive | | `aether-api-payloads.md` | incus-mitm, api-interception | | `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) | diff --git a/incusos/deploy-observability b/incusos/deploy-observability new file mode 100755 index 0000000..88533a5 --- /dev/null +++ b/incusos/deploy-observability @@ -0,0 +1,1197 @@ +#!/usr/bin/env bash +# deploy-observability - Deploy and manage the Incus observability stack +# +# Deploys Prometheus, Grafana, Loki, and Promtail in a monitoring container, +# plus node-exporter containers on each cluster node. Configures Prometheus +# to scrape Incus metrics (with client certs), HAProxy stats, and node +# exporters. Sets up Grafana with datasources and dashboards. +# +# Usage: deploy-observability [OPTIONS] +# Run 'deploy-observability --help' for full usage information. + +set -euo pipefail + +readonly VERSION="0.1.0" +readonly SCRIPT_NAME="deploy-observability" +readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly DASHBOARDS_DIR="${SCRIPT_DIR}/observability-dashboards" + +# --------------------------------------------------------------------------- +# Defaults (overridden by CLI flags) +# --------------------------------------------------------------------------- + +CLUSTER_REMOTE="oc-node-01" +OVN_NETWORK="net-prod" +MONITORING_IP="10.10.10.70" +MONITORING_TARGET="oc-node-02" +FORWARD_VIP="192.168.103.201" + +INCUS_NODES=("192.168.102.140" "192.168.102.141" "192.168.102.142") +HAPROXY_IPS=("10.10.10.50" "10.10.10.51") +NODE_EXP_IPS=("10.10.10.71" "10.10.10.72" "10.10.10.73") +NODE_EXP_TARGETS=("oc-node-01" "oc-node-02" "oc-node-03") + +HAPROXY_CLUSTER_ID="52" + +# Runtime flags +DRY_RUN=false +VERBOSE=false +QUIET=false +ACTION="" + +# --------------------------------------------------------------------------- +# Color and formatting +# --------------------------------------------------------------------------- + +setup_colors() { + if [[ -t 1 ]] && [[ "${NO_COLOR:-}" != "1" ]] && [[ "${TERM:-}" != "dumb" ]]; then + RED='\033[0;31m' + GREEN='\033[0;32m' + YELLOW='\033[0;33m' + BLUE='\033[0;34m' + CYAN='\033[0;36m' + BOLD='\033[1m' + DIM='\033[2m' + RESET='\033[0m' + else + RED='' GREEN='' YELLOW='' BLUE='' CYAN='' BOLD='' DIM='' RESET='' + fi +} + +info() { [[ "$QUIET" == true ]] && return; echo -e "${BLUE}[info]${RESET} $*"; } +success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} $*"; } +warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; } +error() { echo -e "${RED}[error]${RESET} $*" >&2; } +step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; } +detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; } + +dry_run_guard() { + if [[ "$DRY_RUN" == true ]]; then + info "(dry-run) $*" + return 1 + fi + return 0 +} + +# --------------------------------------------------------------------------- +# Help +# --------------------------------------------------------------------------- + +usage() { + cat < Grafana (:3000), Prometheus (:9090) + +${BOLD}EXAMPLES${RESET} + # Check prerequisites and connectivity + ${SCRIPT_NAME} --doctor + + # Preview what would be deployed + ${SCRIPT_NAME} --deploy --dry-run + + # Full deployment + ${SCRIPT_NAME} --deploy + + # Check health of all components + ${SCRIPT_NAME} --status + + # Tear down everything + ${SCRIPT_NAME} --cleanup +EOF +} + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +container_ref() { + echo "${CLUSTER_REMOTE}:$1" +} + +container_exists() { + incus info "$(container_ref "$1")" &>/dev/null 2>&1 +} + +container_running() { + local state + state=$(incus info "$(container_ref "$1")" 2>/dev/null | grep -i '^status:' | awk '{print $2}') + [[ "$state" == "RUNNING" || "$state" == "Running" ]] +} + +container_exec() { + local name="$1" + shift + incus exec "$(container_ref "$name")" -- "$@" +} + +wait_for_container_agent() { + local name="$1" + local timeout="${2:-60}" + local elapsed=0 + while ! container_exec "$name" true &>/dev/null 2>&1; do + sleep 2 + elapsed=$((elapsed + 2)) + if [[ $elapsed -ge $timeout ]]; then + error "Container agent for '${name}' not available after ${timeout}s" + return 1 + fi + done +} + +node_exp_name() { + local idx="$1" + printf "node-exp-%02d" "$idx" +} + +# --------------------------------------------------------------------------- +# Deploy action +# --------------------------------------------------------------------------- + +action_deploy() { + step "Deploying observability stack on ${CLUSTER_REMOTE}" + echo + info " Monitoring: ${MONITORING_IP} (on ${MONITORING_TARGET})" + info " Node-exporters: ${NODE_EXP_IPS[*]}" + info " Incus nodes: ${INCUS_NODES[*]}" + info " HAProxy stats: ${HAPROXY_IPS[*]}" + info " Network fwd: ${FORWARD_VIP} -> Grafana/Prometheus" + echo + + if container_exists "monitoring"; then + error "Monitoring container already exists" + error "Use --status to check health, or --cleanup first." + return 1 + fi + + phase_deploy_monitoring + phase_install_prometheus + phase_install_loki + phase_install_grafana + phase_deploy_node_exporters + phase_create_acl + phase_configure_haproxy_exporter + phase_create_network_forward + phase_upload_dashboards + + echo + success "Observability stack deployed!" + echo + info " Grafana: http://${FORWARD_VIP}:3000 (admin / admin)" + info " Prometheus: http://${FORWARD_VIP}:9090" + info " Loki: http://${MONITORING_IP}:3100 (internal)" + echo + info "Next steps:" + info " ${SCRIPT_NAME} --status # verify all targets are up" + info " Open Grafana and explore the pre-loaded dashboards" +} + +# --------------------------------------------------------------------------- +# Phase 1: Create monitoring container +# --------------------------------------------------------------------------- + +phase_deploy_monitoring() { + step "Phase 1: Create monitoring container" + + if ! dry_run_guard "Launch monitoring on ${OVN_NETWORK} at ${MONITORING_IP}"; then + return 0 + fi + + incus launch images:debian/12 "$(container_ref monitoring)" \ + -n "$OVN_NETWORK" \ + --target "$MONITORING_TARGET" \ + -c limits.memory=2GiB \ + -d root,size=20GiB + + info "Waiting for container agent..." + wait_for_container_agent monitoring 60 + + # Configure static IP + container_exec monitoring bash -c " + cat > /etc/systemd/network/10-static.network << 'NETCFG' +[Match] +Name=eth0 + +[Network] +Address=${MONITORING_IP}/24 +Gateway=10.10.10.1 +DNS=10.10.10.1 +NETCFG + systemctl restart systemd-networkd + " + + sleep 3 + success "Monitoring container created at ${MONITORING_IP}" +} + +# --------------------------------------------------------------------------- +# Phase 2: Install and configure Prometheus +# --------------------------------------------------------------------------- + +phase_install_prometheus() { + step "Phase 2: Install and configure Prometheus" + + if ! dry_run_guard "Install Prometheus in monitoring container"; then + return 0 + fi + + # Copy Incus client certs into the container for metrics scraping + local incus_cert="${HOME}/.config/incus/client.crt" + local incus_key="${HOME}/.config/incus/client.key" + + if [[ ! -f "$incus_cert" || ! -f "$incus_key" ]]; then + error "Incus client certificates not found at ~/.config/incus/" + error "Prometheus needs these to scrape Incus metrics endpoints." + error "Install the Incus client and run 'incus remote add' first." + return 1 + fi + + incus file push "$incus_cert" "$(container_ref monitoring)/tmp/incus-client.crt" + incus file push "$incus_key" "$(container_ref monitoring)/tmp/incus-client.key" + + container_exec monitoring bash -c " + export DEBIAN_FRONTEND=noninteractive + apt-get update -qq + apt-get install -y -qq prometheus 2>&1 | tail -1 + + # Place Incus client certs where Prometheus can read them + mkdir -p /etc/prometheus/certs + cp /tmp/incus-client.crt /etc/prometheus/certs/ + cp /tmp/incus-client.key /etc/prometheus/certs/ + chown -R prometheus:prometheus /etc/prometheus/certs + chmod 600 /etc/prometheus/certs/incus-client.key + " + + # Build scrape targets + local incus_targets="" + local idx=0 + for node in "${INCUS_NODES[@]}"; do + if [[ $idx -gt 0 ]]; then incus_targets+=", "; fi + incus_targets+="'${node}:8443'" + idx=$((idx + 1)) + done + + local haproxy_targets="" + idx=0 + for ip in "${HAPROXY_IPS[@]}"; do + if [[ $idx -gt 0 ]]; then haproxy_targets+=", "; fi + haproxy_targets+="'${ip}:8404'" + idx=$((idx + 1)) + done + + local node_exp_targets="" + idx=0 + for ip in "${NODE_EXP_IPS[@]}"; do + if [[ $idx -gt 0 ]]; then node_exp_targets+=", "; fi + node_exp_targets+="'${ip}:9100'" + idx=$((idx + 1)) + done + + container_exec monitoring bash -c "cat > /etc/prometheus/prometheus.yml << 'PROMCFG' +global: + scrape_interval: 15s + evaluation_interval: 15s + +scrape_configs: + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] + + - job_name: 'incus' + scheme: https + tls_config: + cert_file: /etc/prometheus/certs/incus-client.crt + key_file: /etc/prometheus/certs/incus-client.key + insecure_skip_verify: true + metrics_path: /1.0/metrics + static_configs: + - targets: [${incus_targets}] + + - job_name: 'haproxy' + static_configs: + - targets: [${haproxy_targets}] + + - job_name: 'node-exporter' + static_configs: + - targets: [${node_exp_targets}] +PROMCFG" + + container_exec monitoring systemctl enable prometheus + container_exec monitoring systemctl restart prometheus + sleep 2 + + success "Prometheus installed and configured" +} + +# --------------------------------------------------------------------------- +# Phase 3: Install and configure Loki + Promtail +# --------------------------------------------------------------------------- + +phase_install_loki() { + step "Phase 3: Install Loki and Promtail" + + if ! dry_run_guard "Install Loki + Promtail in monitoring container"; then + return 0 + fi + + container_exec monitoring bash -c " + export DEBIAN_FRONTEND=noninteractive + apt-get install -y -qq loki promtail 2>&1 | tail -1 + " + + # Configure Loki + container_exec monitoring bash -c "cat > /etc/loki/config.yml << 'LOKICFG' +auth_enabled: false + +server: + http_listen_port: 3100 + grpc_listen_port: 9096 + +common: + path_prefix: /var/lib/loki + storage: + filesystem: + chunks_directory: /var/lib/loki/chunks + rules_directory: /var/lib/loki/rules + replication_factor: 1 + ring: + kvstore: + store: inmemory + +schema_config: + configs: + - from: 2020-10-24 + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: index_ + period: 24h + +limits_config: + reject_old_samples: true + reject_old_samples_max_age: 168h + allow_structured_metadata: false +LOKICFG" + + # Configure Promtail to ship local logs + container_exec monitoring bash -c "cat > /etc/promtail/config.yml << 'PTCFG' +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /var/lib/promtail/positions.yaml + +clients: + - url: http://localhost:3100/loki/api/v1/push + +scrape_configs: + - job_name: system + static_configs: + - targets: + - localhost + labels: + job: varlogs + host: monitoring + __path__: /var/log/**/*.log + + - job_name: journal + journal: + max_age: 12h + labels: + job: systemd-journal + host: monitoring + relabel_configs: + - source_labels: ['__journal__systemd_unit'] + target_label: unit +PTCFG" + + container_exec monitoring systemctl enable loki + container_exec monitoring systemctl restart loki + sleep 2 + + container_exec monitoring systemctl enable promtail + container_exec monitoring systemctl restart promtail + + success "Loki and Promtail installed and configured" +} + +# --------------------------------------------------------------------------- +# Phase 4: Install and configure Grafana +# --------------------------------------------------------------------------- + +phase_install_grafana() { + step "Phase 4: Install and configure Grafana" + + if ! dry_run_guard "Install Grafana in monitoring container"; then + return 0 + fi + + container_exec monitoring bash -c " + export DEBIAN_FRONTEND=noninteractive + apt-get install -y -qq grafana 2>&1 | tail -1 + " + + # Provision datasources + container_exec monitoring bash -c "mkdir -p /etc/grafana/provisioning/datasources +cat > /etc/grafana/provisioning/datasources/observability.yaml << 'DSCFG' +apiVersion: 1 +datasources: + - name: Prometheus + type: prometheus + access: proxy + url: http://localhost:9090 + isDefault: true + editable: true + + - name: Loki + type: loki + access: proxy + url: http://localhost:3100 + editable: true +DSCFG" + + # Provision dashboard directory + container_exec monitoring bash -c "mkdir -p /etc/grafana/provisioning/dashboards +cat > /etc/grafana/provisioning/dashboards/observability.yaml << 'DBCFG' +apiVersion: 1 +providers: + - name: Observability + orgId: 1 + folder: '' + type: file + disableDeletion: false + editable: true + options: + path: /var/lib/grafana/dashboards + foldersFromFilesStructure: false +DBCFG +mkdir -p /var/lib/grafana/dashboards" + + container_exec monitoring systemctl enable grafana-server + container_exec monitoring systemctl restart grafana-server + sleep 2 + + success "Grafana installed with Prometheus + Loki datasources" +} + +# --------------------------------------------------------------------------- +# Phase 5: Deploy node-exporter containers +# --------------------------------------------------------------------------- + +phase_deploy_node_exporters() { + step "Phase 5: Deploy node-exporter containers" + + local idx=1 + for ip in "${NODE_EXP_IPS[@]}"; do + local name + name=$(node_exp_name "$idx") + local target="${NODE_EXP_TARGETS[$((idx - 1))]}" + + if container_exists "$name"; then + if container_running "$name"; then + success " ${name} already running" + idx=$((idx + 1)) + continue + fi + fi + + if ! dry_run_guard "Launch ${name} on ${target} at ${ip}"; then + idx=$((idx + 1)) + continue + fi + + info "Launching ${name} (${ip}) on ${target}..." + incus launch images:alpine/3.20 "$(container_ref "$name")" \ + -n "$OVN_NETWORK" \ + --target "$target" \ + -c limits.memory=128MiB \ + -c security.privileged=true + + wait_for_container_agent "$name" 30 + + # Mount host filesystems for full node metrics + incus config device add "$(container_ref "$name")" host-proc disk \ + source=/proc path=/host/proc readonly=true + incus config device add "$(container_ref "$name")" host-sys disk \ + source=/sys path=/host/sys readonly=true + incus config device add "$(container_ref "$name")" host-root disk \ + source=/ path=/host/root readonly=true + + # Install and configure node_exporter + container_exec "$name" sh -c " + apk add --no-cache prometheus-node-exporter > /dev/null 2>&1 + + # Configure to use host mount paths + mkdir -p /etc/conf.d + cat > /etc/conf.d/node-exporter << 'NEXPCFG' +NODE_EXPORTER_OPTS=\"--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root\" +NEXPCFG + + rc-update add node-exporter default 2>/dev/null || true + rc-service node-exporter start 2>/dev/null || true + " + + # Configure static IP + container_exec "$name" sh -c " + cat > /etc/network/interfaces << NETCFG +auto lo +iface lo inet loopback + +auto eth0 +iface eth0 inet static + address ${ip}/24 + gateway 10.10.10.1 +NETCFG + rc-service networking restart 2>/dev/null || true + " + + success " ${name} deployed at ${ip} on ${target}" + idx=$((idx + 1)) + done + + success "Node-exporter containers deployed" +} + +# --------------------------------------------------------------------------- +# Phase 6: Create ACL and attach to containers +# --------------------------------------------------------------------------- + +phase_create_acl() { + step "Phase 6: Create monitoring ACL" + + if ! dry_run_guard "Create monitoring-allow ACL"; then + return 0 + fi + + # Create the ACL if it does not exist + if ! incus network acl show "$(container_ref monitoring-allow)" &>/dev/null 2>&1; then + incus network acl create "$(container_ref monitoring-allow)" \ + description="Allow all traffic for observability stack" + + # Add allow-all egress and ingress rules + incus network acl rule add "$(container_ref monitoring-allow)" egress \ + action=allow description="Allow all egress" + incus network acl rule add "$(container_ref monitoring-allow)" ingress \ + action=allow description="Allow all ingress" + success "ACL 'monitoring-allow' created with allow-all rules" + else + success "ACL 'monitoring-allow' already exists" + fi + + # Attach ACL to monitoring container + incus config device set "$(container_ref monitoring)" eth0 \ + security.acls=monitoring-allow security.acls.default.ingress.action=allow \ + security.acls.default.egress.action=allow 2>/dev/null || true + detail "ACL attached to monitoring" + + # Attach ACL to node-exporter containers + local idx=1 + for ip in "${NODE_EXP_IPS[@]}"; do + local name + name=$(node_exp_name "$idx") + if container_exists "$name"; then + incus config device set "$(container_ref "$name")" eth0 \ + security.acls=monitoring-allow security.acls.default.ingress.action=allow \ + security.acls.default.egress.action=allow 2>/dev/null || true + detail "ACL attached to ${name}" + fi + idx=$((idx + 1)) + done + + success "ACL attached to all observability containers" +} + +# --------------------------------------------------------------------------- +# Phase 7: Configure HAProxy prometheus exporter +# --------------------------------------------------------------------------- + +phase_configure_haproxy_exporter() { + step "Phase 7: Configure HAProxy prometheus exporter + ACL rules" + + if ! dry_run_guard "Add prometheus exporter to HAProxy containers"; then + return 0 + fi + + local haproxy_remote="${CLUSTER_REMOTE}" + + for n in 01 02; do + local cname="ffsdn-haproxy-${HAPROXY_CLUSTER_ID}-${n}" + + if ! container_exists "$cname"; then + warn "HAProxy container ${cname} not found -- skipping" + continue + fi + + # Add prometheus exporter line to the stats frontend in haproxy.cfg + # Only add if not already present + local has_exporter + has_exporter=$(container_exec "$cname" grep -c 'prometheus-exporter' /etc/haproxy/haproxy.cfg 2>/dev/null || echo "0") + if [[ "$has_exporter" == "0" ]]; then + container_exec "$cname" bash -c " + # Insert the prometheus exporter line into the stats section + sed -i '/^[[:space:]]*stats enable/a\\ http-request use-service prometheus-exporter if { path /metrics }' /etc/haproxy/haproxy.cfg + haproxy -c -f /etc/haproxy/haproxy.cfg 2>/dev/null && systemctl reload haproxy || true + " + success " ${cname}: prometheus exporter route added" + else + success " ${cname}: prometheus exporter already configured" + fi + + # Add monitoring ingress rules to Aether ACL + local acl_name="${HAPROXY_CLUSTER_ID}-${cname}-aether-acl" + local has_monitoring_rule + has_monitoring_rule=$(incus network acl show "${haproxy_remote}:${acl_name}" 2>/dev/null \ + | grep -c "monitoring-prometheus" || echo "0") + + if [[ "$has_monitoring_rule" == "0" ]]; then + incus network acl rule add "${haproxy_remote}:${acl_name}" ingress \ + action=allow protocol=tcp source="${MONITORING_IP}/32" destination_port=8404 \ + description="monitoring-prometheus" 2>/dev/null || true + incus network acl rule add "${haproxy_remote}:${acl_name}" ingress \ + action=allow protocol=icmp source="${MONITORING_IP}/32" \ + description="monitoring-icmp" 2>/dev/null || true + success " ${cname}: monitoring ACL rules added" + else + success " ${cname}: monitoring ACL rules already present" + fi + done + + success "HAProxy prometheus exporter configured" +} + +# --------------------------------------------------------------------------- +# Phase 8: Create OVN network forward +# --------------------------------------------------------------------------- + +phase_create_network_forward() { + step "Phase 8: Create network forward ${FORWARD_VIP}" + + if ! dry_run_guard "Create network forward ${FORWARD_VIP}"; then + return 0 + fi + + # Check if forward already exists + if incus network forward show "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" &>/dev/null 2>&1; then + success "Network forward ${FORWARD_VIP} already exists" + return 0 + fi + + incus network forward create "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" + + # Grafana port + incus network forward port add "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" \ + tcp 3000 "${MONITORING_IP}" 3000 + + # Prometheus port + incus network forward port add "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" \ + tcp 9090 "${MONITORING_IP}" 9090 + + success "Network forward created: ${FORWARD_VIP} -> Grafana(:3000), Prometheus(:9090)" +} + +# --------------------------------------------------------------------------- +# Phase 9: Upload Grafana dashboards +# --------------------------------------------------------------------------- + +phase_upload_dashboards() { + step "Phase 9: Upload Grafana dashboards" + + if ! dry_run_guard "Upload dashboards from ${DASHBOARDS_DIR}"; then + return 0 + fi + + if [[ ! -d "$DASHBOARDS_DIR" ]]; then + warn "Dashboard directory not found: ${DASHBOARDS_DIR}" + warn "Skipping dashboard upload -- create dashboards manually in Grafana." + return 0 + fi + + local count=0 + for dashboard in "${DASHBOARDS_DIR}"/*.json; do + [[ -f "$dashboard" ]] || continue + local basename + basename=$(basename "$dashboard") + incus file push "$dashboard" "$(container_ref monitoring)/var/lib/grafana/dashboards/${basename}" + detail "Uploaded ${basename}" + count=$((count + 1)) + done + + if [[ $count -gt 0 ]]; then + # Restart Grafana to pick up new dashboards + container_exec monitoring systemctl restart grafana-server 2>/dev/null || true + success "${count} dashboard(s) uploaded" + else + info "No dashboard files found in ${DASHBOARDS_DIR}" + fi +} + +# --------------------------------------------------------------------------- +# Status action +# --------------------------------------------------------------------------- + +action_status() { + step "Observability Stack Status" + echo + + # --- Container status --- + info "Containers:" + local all_ok=true + + for cname in monitoring; do + if container_exists "$cname"; then + if container_running "$cname"; then + success " ${cname}: running" + else + error " ${cname}: stopped" + all_ok=false + fi + else + error " ${cname}: not found" + all_ok=false + fi + done + + local idx=1 + for ip in "${NODE_EXP_IPS[@]}"; do + local name + name=$(node_exp_name "$idx") + if container_exists "$name"; then + if container_running "$name"; then + success " ${name} (${ip}): running" + else + warn " ${name} (${ip}): stopped" + fi + else + warn " ${name} (${ip}): not found" + fi + idx=$((idx + 1)) + done + + if [[ "$all_ok" == false ]]; then + echo + error "Some containers are missing or stopped." + echo + fi + + # --- Service status --- + if container_exists monitoring && container_running monitoring; then + echo + info "Services:" + for svc in prometheus grafana-server loki promtail; do + local status + status=$(container_exec monitoring systemctl is-active "$svc" 2>/dev/null || echo "inactive") + if [[ "$status" == "active" ]]; then + success " ${svc}: active" + else + error " ${svc}: ${status}" + fi + done + + # --- Prometheus targets --- + echo + info "Prometheus targets:" + local targets_json + targets_json=$(curl -s --connect-timeout 5 "http://${MONITORING_IP}:9090/api/v1/targets" 2>/dev/null || echo "") + if [[ -n "$targets_json" ]]; then + echo "$targets_json" | python3 -c " +import sys, json +try: + data = json.load(sys.stdin) + targets = data.get('data', {}).get('activeTargets', []) + for t in targets: + job = t.get('labels', {}).get('job', 'unknown') + instance = t.get('labels', {}).get('instance', 'unknown') + health = t.get('health', 'unknown') + icon = 'UP' if health == 'up' else 'DOWN' + print(f' {job:20s} {instance:30s} {icon}') +except Exception as e: + print(f' (error parsing targets: {e})') +" 2>/dev/null || warn " Could not parse Prometheus targets" + else + warn " Prometheus not reachable at http://${MONITORING_IP}:9090" + fi + + # --- Grafana health --- + echo + info "Grafana health:" + local grafana_code + grafana_code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 5 \ + "http://${MONITORING_IP}:3000/api/health" 2>/dev/null || echo "000") + if [[ "$grafana_code" == "200" ]]; then + success " Grafana API: healthy (HTTP ${grafana_code})" + else + error " Grafana API: HTTP ${grafana_code}" + fi + + # --- Loki labels --- + echo + info "Loki status:" + local loki_labels + loki_labels=$(curl -s --connect-timeout 5 "http://${MONITORING_IP}:3100/loki/api/v1/labels" 2>/dev/null || echo "") + if [[ -n "$loki_labels" ]]; then + local label_count + label_count=$(echo "$loki_labels" | python3 -c " +import sys, json +try: + data = json.load(sys.stdin) + labels = data.get('data', []) + print(len(labels)) +except: + print(0) +" 2>/dev/null || echo "0") + if [[ "$label_count" -gt 0 ]]; then + success " Loki: responding (${label_count} label(s))" + else + info " Loki: responding (no labels yet -- logs may not have been ingested)" + fi + else + warn " Loki not reachable at http://${MONITORING_IP}:3100" + fi + fi + + # --- Network forward --- + echo + info "Network forward:" + if incus network forward show "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" &>/dev/null 2>&1; then + success " ${FORWARD_VIP}: configured" + local grafana_fwd + grafana_fwd=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 5 \ + "http://${FORWARD_VIP}:3000/api/health" 2>/dev/null || echo "000") + if [[ "$grafana_fwd" == "200" ]]; then + success " Grafana via forward: reachable (HTTP ${grafana_fwd})" + else + warn " Grafana via forward: HTTP ${grafana_fwd}" + fi + else + warn " ${FORWARD_VIP}: not found" + fi +} + +# --------------------------------------------------------------------------- +# Cleanup action +# --------------------------------------------------------------------------- + +action_cleanup() { + step "Cleaning up observability stack" + echo + + local has_anything=false + + if container_exists monitoring; then has_anything=true; fi + local idx=1 + for ip in "${NODE_EXP_IPS[@]}"; do + if container_exists "$(node_exp_name "$idx")"; then has_anything=true; fi + idx=$((idx + 1)) + done + if incus network forward show "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" &>/dev/null 2>&1; then + has_anything=true + fi + + if [[ "$has_anything" == false ]]; then + info "Nothing to clean up" + return 0 + fi + + warn "This will delete:" + if container_exists monitoring; then + warn " - monitoring container (Prometheus, Grafana, Loki data)" + fi + idx=1 + for ip in "${NODE_EXP_IPS[@]}"; do + local name + name=$(node_exp_name "$idx") + if container_exists "$name"; then + warn " - ${name}" + fi + idx=$((idx + 1)) + done + if incus network forward show "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" &>/dev/null 2>&1; then + warn " - network forward ${FORWARD_VIP}" + fi + if incus network acl show "$(container_ref monitoring-allow)" &>/dev/null 2>&1; then + warn " - ACL monitoring-allow" + fi + echo + + if [[ "$DRY_RUN" == true ]]; then + info "(dry-run) Would delete all observability infrastructure" + return 0 + fi + + read -r -p "Type 'yes' to confirm: " confirm + if [[ "$confirm" != "yes" ]]; then + info "Aborted" + return 0 + fi + + # Remove monitoring ACL rules from HAProxy ACLs + step "Removing monitoring rules from HAProxy ACLs" + for n in 01 02; do + local cname="ffsdn-haproxy-${HAPROXY_CLUSTER_ID}-${n}" + local acl_name="${HAPROXY_CLUSTER_ID}-${cname}-aether-acl" + # Remove rules with monitoring descriptions + local rule_ids + rule_ids=$(incus network acl show "${CLUSTER_REMOTE}:${acl_name}" 2>/dev/null \ + | python3 -c " +import sys, yaml +try: + data = yaml.safe_load(sys.stdin) + for rule in data.get('ingress', []): + desc = rule.get('description', '') + if desc.startswith('monitoring-'): + # ACL rules are removed by matching all fields, not by ID + print(desc) +except: pass +" 2>/dev/null || true) + if [[ -n "$rule_ids" ]]; then + for desc in $rule_ids; do + incus network acl rule remove "${CLUSTER_REMOTE}:${acl_name}" ingress \ + description="$desc" 2>/dev/null || true + done + info " Removed monitoring rules from ${acl_name}" + fi + done + + # Delete containers + step "Deleting containers" + if container_exists monitoring; then + info "Deleting monitoring..." + incus delete "$(container_ref monitoring)" --force 2>/dev/null || true + success "Deleted monitoring" + fi + + idx=1 + for ip in "${NODE_EXP_IPS[@]}"; do + local name + name=$(node_exp_name "$idx") + if container_exists "$name"; then + info "Deleting ${name}..." + incus delete "$(container_ref "$name")" --force 2>/dev/null || true + success "Deleted ${name}" + fi + idx=$((idx + 1)) + done + + # Delete network forward + step "Removing network forward" + if incus network forward show "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" &>/dev/null 2>&1; then + incus network forward delete "${CLUSTER_REMOTE}:${OVN_NETWORK}" "$FORWARD_VIP" + success "Deleted network forward ${FORWARD_VIP}" + fi + + # Delete ACL + step "Removing ACL" + if incus network acl show "$(container_ref monitoring-allow)" &>/dev/null 2>&1; then + incus network acl delete "$(container_ref monitoring-allow)" 2>/dev/null || true + success "Deleted ACL monitoring-allow" + fi + + echo + success "Cleanup complete" +} + +# --------------------------------------------------------------------------- +# Doctor action +# --------------------------------------------------------------------------- + +action_doctor() { + step "Diagnosing observability stack" + echo + + if ! container_exists monitoring || ! container_running monitoring; then + error "Monitoring container is not running" + error "Deploy with: ${SCRIPT_NAME} --deploy" + return 1 + fi + + local issues=0 + + # Check container networking + info "Container networking:" + if container_exec monitoring ping -c 1 -W 3 10.10.10.1 &>/dev/null; then + success " Gateway (10.10.10.1): reachable" + else + error " Gateway (10.10.10.1): unreachable" + issues=$((issues + 1)) + fi + + local idx=1 + for ip in "${NODE_EXP_IPS[@]}"; do + local name + name=$(node_exp_name "$idx") + if container_exec monitoring ping -c 1 -W 3 "$ip" &>/dev/null; then + success " ${name} (${ip}): reachable" + else + warn " ${name} (${ip}): unreachable" + issues=$((issues + 1)) + fi + idx=$((idx + 1)) + done + + for ip in "${HAPROXY_IPS[@]}"; do + if container_exec monitoring ping -c 1 -W 3 "$ip" &>/dev/null; then + success " HAProxy (${ip}): reachable" + else + warn " HAProxy (${ip}): unreachable" + issues=$((issues + 1)) + fi + done + + # Check Prometheus scrape errors + echo + info "Prometheus scrape health:" + local targets_json + targets_json=$(curl -s --connect-timeout 5 "http://${MONITORING_IP}:9090/api/v1/targets" 2>/dev/null || echo "") + if [[ -n "$targets_json" ]]; then + echo "$targets_json" | python3 -c " +import sys, json +try: + data = json.load(sys.stdin) + targets = data.get('data', {}).get('activeTargets', []) + down = [t for t in targets if t.get('health') != 'up'] + if down: + for t in down: + job = t.get('labels', {}).get('job', '?') + inst = t.get('labels', {}).get('instance', '?') + err = t.get('lastError', 'unknown error') + print(f' DOWN: {job} {inst}') + print(f' {err}') + else: + print(f' All {len(targets)} target(s) are UP') +except Exception as e: + print(f' (parse error: {e})') +" 2>/dev/null || warn " Could not query Prometheus" + else + error " Prometheus not reachable" + issues=$((issues + 1)) + fi + + # Check systemd service failures + echo + info "Service health:" + for svc in prometheus grafana-server loki promtail; do + local status + status=$(container_exec monitoring systemctl is-active "$svc" 2>/dev/null || echo "inactive") + if [[ "$status" == "active" ]]; then + success " ${svc}: active" + else + error " ${svc}: ${status}" + # Show recent logs + local logs + logs=$(container_exec monitoring journalctl -u "$svc" --no-pager -n 5 --no-hostname 2>/dev/null || true) + if [[ -n "$logs" ]]; then + echo "$logs" | while IFS= read -r line; do + detail " ${line}" + done + fi + issues=$((issues + 1)) + fi + done + + # Check disk space + echo + info "Disk space (monitoring container):" + local disk_usage + disk_usage=$(container_exec monitoring df -h / 2>/dev/null | tail -1 || echo "") + if [[ -n "$disk_usage" ]]; then + local use_pct + use_pct=$(echo "$disk_usage" | awk '{print $5}' | tr -d '%') + if [[ -n "$use_pct" ]] && [[ "$use_pct" -gt 85 ]]; then + warn " Disk usage: ${use_pct}% -- consider increasing disk size" + issues=$((issues + 1)) + else + success " Disk usage: ${use_pct}%" + fi + else + warn " Could not check disk usage" + fi + + echo + if [[ $issues -eq 0 ]]; then + success "No issues found" + else + warn "${issues} issue(s) found -- review the output above" + fi +} + +# --------------------------------------------------------------------------- +# Argument parsing +# --------------------------------------------------------------------------- + +parse_args() { + while [[ $# -gt 0 ]]; do + case "$1" in + --deploy) ACTION="deploy";; + --status) ACTION="status";; + --cleanup) ACTION="cleanup";; + --doctor) ACTION="doctor";; + -r|--remote) CLUSTER_REMOTE="$2"; shift;; + -n|--dry-run) DRY_RUN=true;; + -v|--verbose) VERBOSE=true;; + -q|--quiet) QUIET=true;; + -h|--help) usage; exit 0;; + *) + error "Unknown option: $1" + echo "Run '${SCRIPT_NAME} --help' for usage." + exit 1 + ;; + esac + shift + done + + if [[ -z "$ACTION" ]]; then + error "No action specified" + echo "Run '${SCRIPT_NAME} --help' for usage." + exit 1 + fi +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +main() { + setup_colors + parse_args "$@" + + echo + case "$ACTION" in + deploy) action_deploy;; + status) action_status;; + cleanup) action_cleanup;; + doctor) action_doctor;; + esac +} + +main "$@" diff --git a/incusos/observability-dashboards/cluster-overview.json b/incusos/observability-dashboards/cluster-overview.json new file mode 100644 index 0000000..059d736 --- /dev/null +++ b/incusos/observability-dashboards/cluster-overview.json @@ -0,0 +1,559 @@ +{ + "uid": "incus-cluster-overview", + "title": "Incus Cluster Overview", + "description": "Overview of all Incus containers: CPU, memory, network, and disk I/O per instance.", + "tags": ["incus", "cluster", "containers"], + "timezone": "browser", + "schemaVersion": 39, + "version": 1, + "refresh": "30s", + "time": { + "from": "now-1h", + "to": "now" + }, + "fiscalYearStartMonth": 0, + "liveNow": false, + "editable": true, + "graphTooltip": 1, + "templating": { + "list": [] + }, + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "panels": [ + { + "type": "row", + "title": "Cluster Status", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, + "id": 1, + "panels": [] + }, + { + "type": "stat", + "title": "Total Containers", + "description": "Number of running Incus containers (instances reporting uptime).", + "gridPos": { "h": 4, "w": 8, "x": 0, "y": 1 }, + "id": 2, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "count(incus_uptime_seconds)", + "legendFormat": "Containers", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "yellow", "value": 10 }, + { "color": "red", "value": 50 } + ] + }, + "unit": "none", + "color": { "mode": "thresholds" } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto" + } + }, + { + "type": "stat", + "title": "Total CPUs", + "description": "Sum of effective CPUs across all Incus instances.", + "gridPos": { "h": 4, "w": 8, "x": 8, "y": 1 }, + "id": 3, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum(incus_cpu_effective_total)", + "legendFormat": "CPUs", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + }, + "unit": "none", + "color": { "mode": "thresholds" } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto" + } + }, + { + "type": "stat", + "title": "Total Memory", + "description": "Sum of total memory across all Incus instances.", + "gridPos": { "h": 4, "w": 8, "x": 16, "y": 1 }, + "id": 4, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum(incus_memory_MemTotal_bytes)", + "legendFormat": "Total Memory", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + }, + "unit": "bytes", + "color": { "mode": "thresholds" } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto" + } + }, + { + "type": "row", + "title": "Per-Instance CPU", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 }, + "id": 5, + "panels": [] + }, + { + "type": "timeseries", + "title": "CPU Usage by Instance", + "description": "Per-second CPU time consumed by each Incus instance.", + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 6 }, + "id": 6, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (name) (rate(incus_cpu_seconds_total[$__rate_interval]))", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "short", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "row", + "title": "Per-Instance Memory", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 }, + "id": 7, + "panels": [] + }, + { + "type": "timeseries", + "title": "Memory RSS by Instance", + "description": "Resident set size (RSS) memory usage per Incus instance over time.", + "gridPos": { "h": 8, "w": 14, "x": 0, "y": 15 }, + "id": 8, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "incus_memory_RSS_bytes", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "bytes", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "bargauge", + "title": "Memory Usage % (RSS / Total)", + "description": "Current memory utilization per instance as RSS divided by total memory.", + "gridPos": { "h": 8, "w": 10, "x": 14, "y": 15 }, + "id": 9, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "(incus_memory_RSS_bytes / incus_memory_MemTotal_bytes) * 100", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "yellow", "value": 60 }, + { "color": "orange", "value": 80 }, + { "color": "red", "value": 90 } + ] + }, + "unit": "percent", + "min": 0, + "max": 100, + "color": { "mode": "thresholds" } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "orientation": "horizontal", + "displayMode": "gradient", + "showUnfilled": true, + "minVizWidth": 8, + "minVizHeight": 16, + "valueMode": "color" + } + }, + { + "type": "row", + "title": "Network I/O", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 23 }, + "id": 10, + "panels": [] + }, + { + "type": "timeseries", + "title": "Network Receive Rate by Instance", + "description": "Inbound network throughput per Incus instance.", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 }, + "id": 11, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (name) (rate(incus_network_receive_bytes_total[$__rate_interval]))", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "timeseries", + "title": "Network Transmit Rate by Instance", + "description": "Outbound network throughput per Incus instance.", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 }, + "id": 12, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (name) (rate(incus_network_transmit_bytes_total[$__rate_interval]))", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "row", + "title": "Disk I/O", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 32 }, + "id": 13, + "panels": [] + }, + { + "type": "timeseries", + "title": "Disk Read Rate by Instance", + "description": "Disk read throughput per Incus instance.", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 33 }, + "id": 14, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (name) (rate(incus_disk_read_bytes_total[$__rate_interval]))", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "timeseries", + "title": "Disk Write Rate by Instance", + "description": "Disk write throughput per Incus instance.", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 33 }, + "id": 15, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (name) (rate(incus_disk_written_bytes_total[$__rate_interval]))", + "legendFormat": "{{name}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + } + ] +} diff --git a/incusos/observability-dashboards/haproxy-traffic.json b/incusos/observability-dashboards/haproxy-traffic.json new file mode 100644 index 0000000..8f89cbe --- /dev/null +++ b/incusos/observability-dashboards/haproxy-traffic.json @@ -0,0 +1,518 @@ +{ + "uid": "incus-haproxy-traffic", + "title": "HAProxy Traffic", + "description": "HAProxy load balancer monitoring: request rates, backend health, traffic volume, and sessions.", + "tags": ["haproxy", "loadbalancer", "incus"], + "timezone": "browser", + "schemaVersion": 39, + "version": 1, + "refresh": "30s", + "time": { + "from": "now-1h", + "to": "now" + }, + "fiscalYearStartMonth": 0, + "liveNow": false, + "editable": true, + "graphTooltip": 1, + "templating": { + "list": [] + }, + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "panels": [ + { + "type": "row", + "title": "Overview", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, + "id": 1, + "panels": [] + }, + { + "type": "stat", + "title": "Active Backend Servers", + "description": "Total number of active servers across all HAProxy backends.", + "gridPos": { "h": 4, "w": 8, "x": 0, "y": 1 }, + "id": 2, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum(haproxy_backend_active_servers)", + "legendFormat": "Active Servers", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "red", "value": null }, + { "color": "yellow", "value": 1 }, + { "color": "green", "value": 2 } + ] + }, + "unit": "none", + "color": { "mode": "thresholds" } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto" + } + }, + { + "type": "stat", + "title": "Current Sessions", + "description": "Total current sessions across all HAProxy backends.", + "gridPos": { "h": 4, "w": 8, "x": 8, "y": 1 }, + "id": 3, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum(haproxy_backend_current_sessions)", + "legendFormat": "Sessions", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "yellow", "value": 100 }, + { "color": "red", "value": 500 } + ] + }, + "unit": "none", + "color": { "mode": "thresholds" } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "type": "stat", + "title": "Total HTTP Requests", + "description": "Total HTTP requests across all HAProxy frontends (counter).", + "gridPos": { "h": 4, "w": 8, "x": 16, "y": 1 }, + "id": 4, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum(rate(haproxy_frontend_http_requests_total[$__rate_interval]))", + "legendFormat": "Requests/s", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + }, + "unit": "reqps", + "color": { "mode": "thresholds" } + }, + "overrides": [] + }, + "options": { + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "orientation": "auto", + "textMode": "auto", + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto" + } + }, + { + "type": "row", + "title": "HTTP Requests", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 }, + "id": 5, + "panels": [] + }, + { + "type": "timeseries", + "title": "HTTP Request Rate by Frontend", + "description": "Rate of HTTP requests per second, broken down by HAProxy frontend/proxy.", + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 6 }, + "id": 6, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (proxy) (rate(haproxy_frontend_http_requests_total[$__rate_interval]))", + "legendFormat": "{{proxy}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 20, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "reqps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "row", + "title": "Backend Health", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 }, + "id": 7, + "panels": [] + }, + { + "type": "table", + "title": "Backend Status", + "description": "Current status, active server count, and average response time for each HAProxy backend.", + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 15 }, + "id": 8, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "haproxy_backend_status", + "legendFormat": "", + "refId": "A", + "format": "table", + "instant": true + }, + { + "expr": "haproxy_backend_active_servers", + "legendFormat": "", + "refId": "B", + "format": "table", + "instant": true + }, + { + "expr": "haproxy_server_response_time_average_seconds", + "legendFormat": "", + "refId": "C", + "format": "table", + "instant": true + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "align": "auto", + "displayMode": "auto", + "inspect": false, + "filterable": true + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + }, + "color": { "mode": "thresholds" } + }, + "overrides": [ + { + "matcher": { "id": "byName", "options": "Value #A" }, + "properties": [ + { "id": "displayName", "value": "Status" } + ] + }, + { + "matcher": { "id": "byName", "options": "Value #B" }, + "properties": [ + { "id": "displayName", "value": "Active Servers" } + ] + }, + { + "matcher": { "id": "byName", "options": "Value #C" }, + "properties": [ + { "id": "displayName", "value": "Avg Response Time" }, + { "id": "unit", "value": "s" } + ] + }, + { + "matcher": { "id": "byName", "options": "proxy" }, + "properties": [ + { "id": "displayName", "value": "Backend" } + ] + } + ] + }, + "options": { + "showHeader": true, + "sortBy": [], + "footer": { + "show": false, + "reducer": ["sum"], + "fields": "" + } + }, + "transformations": [ + { + "id": "merge", + "options": {} + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "__name__": true, + "instance": false, + "job": true + }, + "indexByName": {}, + "renameByName": {} + } + } + ] + }, + { + "type": "row", + "title": "Traffic", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 23 }, + "id": 9, + "panels": [] + }, + { + "type": "timeseries", + "title": "Frontend Bytes In", + "description": "Inbound traffic rate per HAProxy frontend.", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 }, + "id": 10, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (proxy) (rate(haproxy_frontend_bytes_in_total[$__rate_interval]))", + "legendFormat": "{{proxy}} (in)", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 20, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "timeseries", + "title": "Frontend Bytes Out", + "description": "Outbound traffic rate per HAProxy frontend.", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 }, + "id": 11, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (proxy) (rate(haproxy_frontend_bytes_out_total[$__rate_interval]))", + "legendFormat": "{{proxy}} (out)", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 20, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "row", + "title": "Sessions", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 32 }, + "id": 12, + "panels": [] + }, + { + "type": "timeseries", + "title": "Current Sessions by Proxy", + "description": "Current active sessions for each HAProxy backend/proxy over time.", + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 33 }, + "id": 13, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "haproxy_backend_current_sessions", + "legendFormat": "{{proxy}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 20, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "none", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] } + } + } + ] +} diff --git a/incusos/observability-dashboards/host-resources.json b/incusos/observability-dashboards/host-resources.json new file mode 100644 index 0000000..282f7fa --- /dev/null +++ b/incusos/observability-dashboards/host-resources.json @@ -0,0 +1,429 @@ +{ + "uid": "incus-host-resources", + "title": "Host Resources (IncusOS Nodes)", + "description": "Physical host monitoring for IncusOS cluster nodes: CPU, memory, disk, and network from node_exporter metrics.", + "tags": ["incusos", "host", "node-exporter"], + "timezone": "browser", + "schemaVersion": 39, + "version": 1, + "refresh": "30s", + "time": { + "from": "now-1h", + "to": "now" + }, + "fiscalYearStartMonth": 0, + "liveNow": false, + "editable": true, + "graphTooltip": 1, + "templating": { + "list": [] + }, + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "panels": [ + { + "type": "row", + "title": "CPU", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, + "id": 1, + "panels": [] + }, + { + "type": "timeseries", + "title": "CPU Usage % by Node", + "description": "Overall CPU utilization per host node, calculated as 1 minus the average idle CPU rate.", + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 1 }, + "id": 2, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "(1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval]))) * 100", + "legendFormat": "{{instance}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 2, + "fillOpacity": 25, + "gradientMode": "scheme", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "line" } + }, + "unit": "percent", + "min": 0, + "max": 100, + "color": { "mode": "continuous-GrYlRd" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "yellow", "value": 70 }, + { "color": "red", "value": 90 } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] } + } + }, + { + "type": "row", + "title": "Memory", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 }, + "id": 3, + "panels": [] + }, + { + "type": "timeseries", + "title": "Memory Usage % by Node", + "description": "Memory utilization percentage per host node, calculated from available vs total memory.", + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 10 }, + "id": 4, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100", + "legendFormat": "{{instance}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 2, + "fillOpacity": 25, + "gradientMode": "scheme", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "line" } + }, + "unit": "percent", + "min": 0, + "max": 100, + "color": { "mode": "continuous-GrYlRd" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "yellow", "value": 70 }, + { "color": "red", "value": 90 } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] } + } + }, + { + "type": "row", + "title": "Disk", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 18 }, + "id": 5, + "panels": [] + }, + { + "type": "timeseries", + "title": "Root Filesystem Available Space", + "description": "Available disk space on the root filesystem (mountpoint='/') per host node.", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 19 }, + "id": 6, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "node_filesystem_avail_bytes{mountpoint=\"/\"}", + "legendFormat": "{{instance}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "bytes", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull", "min"] } + } + }, + { + "type": "timeseries", + "title": "Disk Read Rate by Node", + "description": "Disk read throughput (bytes/sec) aggregated per host node.", + "gridPos": { "h": 8, "w": 6, "x": 12, "y": 19 }, + "id": 7, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (instance) (rate(node_disk_read_bytes_total[$__rate_interval]))", + "legendFormat": "{{instance}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "list", "placement": "bottom", "calcs": [] } + } + }, + { + "type": "timeseries", + "title": "Disk Write Rate by Node", + "description": "Disk write throughput (bytes/sec) aggregated per host node.", + "gridPos": { "h": 8, "w": 6, "x": 18, "y": 19 }, + "id": 8, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "sum by (instance) (rate(node_disk_written_bytes_total[$__rate_interval]))", + "legendFormat": "{{instance}}", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "list", "placement": "bottom", "calcs": [] } + } + }, + { + "type": "row", + "title": "Network", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 27 }, + "id": 9, + "panels": [] + }, + { + "type": "timeseries", + "title": "Network Receive Rate by Device & Node", + "description": "Inbound network throughput per network device and host node.", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 }, + "id": 10, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "rate(node_network_receive_bytes_total{device!~\"lo|veth.*|br-.*|docker.*\"}[$__rate_interval])", + "legendFormat": "{{instance}} - {{device}} (rx)", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + }, + { + "type": "timeseries", + "title": "Network Transmit Rate by Device & Node", + "description": "Outbound network throughput per network device and host node.", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 }, + "id": 11, + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "targets": [ + { + "expr": "rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|br-.*|docker.*\"}[$__rate_interval])", + "legendFormat": "{{instance}} - {{device}} (tx)", + "refId": "A" + } + ], + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "lineInterpolation": "smooth", + "lineWidth": 1, + "fillOpacity": 15, + "gradientMode": "none", + "spanNulls": false, + "showPoints": "never", + "pointSize": 5, + "stacking": { "mode": "none", "group": "A" }, + "axisPlacement": "auto", + "axisLabel": "", + "scaleDistribution": { "type": "linear" }, + "thresholdsStyle": { "mode": "off" } + }, + "unit": "Bps", + "color": { "mode": "palette-classic" }, + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null } + ] + } + }, + "overrides": [] + }, + "options": { + "tooltip": { "mode": "multi", "sort": "desc" }, + "legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] } + } + } + ] +} diff --git a/notes/observability-guide.md b/notes/observability-guide.md new file mode 100644 index 0000000..15de7c8 --- /dev/null +++ b/notes/observability-guide.md @@ -0,0 +1,337 @@ +# Observability Stack — Prometheus, Grafana, Loki on Incus + +A lightweight observability stack deployed as Incus containers on the lab +cluster. Provides metric collection (Prometheus), dashboards (Grafana), +and log aggregation (Loki) for Incus nodes, HAProxy load balancers, and +host-level resources. + +The stack runs entirely on OVN and is managed by the `deploy-observability` +script. All containers use the `monitoring-allow` ACL to permit scrape +traffic on the shared network. + +## Architecture + +``` + LAN (192.168.103.0/24) + | + | OVN network forward (192.168.103.201) + | :3000 -> 10.10.10.70:3000 (Grafana) + | :9090 -> 10.10.10.70:9090 (Prometheus) + | + ═══════╪══════════════════════════════════════════════ + | OVN net-prod (10.10.10.0/24) + | + v + +----------------------------------------------+ + | monitoring (10.10.10.70, oc-node-02) | + | Debian/12 container — 2 GiB RAM, 20 GiB | + | | + | +-------------+ +-----------+ | + | | Prometheus | | Grafana | | + | | :9090 | | :3000 | | + | +-------------+ +-----------+ | + | +-------------+ +-----------+ | + | | Loki | | Promtail | | + | | :3100 | | (agent) | | + | | gRPC :9096 | | | | + | +-------------+ +-----------+ | + +----------------------------------------------+ + | + | Scrape targets + | + +-------> Incus nodes (client cert auth, :8443) + | 192.168.102.140 /1.0/metrics + | 192.168.102.141 /1.0/metrics + | 192.168.102.142 /1.0/metrics + | + +-------> HAProxy (:8404) + | 10.10.10.50 /metrics + | 10.10.10.51 /metrics + | + +-------> node-exporters (:9100) + 10.10.10.71 node-exp-01 (oc-node-01) + 10.10.10.72 node-exp-02 (oc-node-02) + 10.10.10.73 node-exp-03 (oc-node-03) + + +------------------+ +------------------+ +------------------+ + | node-exp-01 | | node-exp-02 | | node-exp-03 | + | 10.10.10.71 | | 10.10.10.72 | | 10.10.10.73 | + | Alpine, privil. | | Alpine, privil. | | Alpine, privil. | + | :9100 | | :9100 | | :9100 | + | host /proc,/sys | | host /proc,/sys | | host /proc,/sys | + | 128 MiB RAM | | 128 MiB RAM | | 128 MiB RAM | + | pinned to node | | pinned to node | | pinned to node | + +------------------+ +------------------+ +------------------+ +``` + +## Components + +### Prometheus 2.54 + +Metric collection engine. Scrapes all targets at a 15-second interval. + +- Stores metrics locally in the monitoring container +- Client certificate authentication for Incus `/1.0/metrics` endpoints + (uses cluster client cert + key) +- Plain HTTP scrape for HAProxy stats (`:8404/metrics`) and + node-exporters (`:9100/metrics`) + +### Grafana 12.4 + +Dashboard and visualization frontend. + +- Default credentials: `admin` / `admin` +- Pre-provisioned datasources for Prometheus and Loki +- Three dashboards deployed automatically (see Dashboards section) +- Accessible from LAN via OVN network forward + +### Loki 3.6 + +Log aggregation backend. + +- HTTP API on port 3100 +- gRPC on port **9096** (not the default 9095 -- avoids conflict with + Promtail's own metrics port) +- 7-day retention policy +- Receives logs from Promtail running in the same container + +### Promtail 3.6 + +Log shipping agent, co-located with Loki in the monitoring container. + +- Pushes logs to Loki via gRPC on localhost:9096 +- Scrapes container logs and system journal + +### node_exporter + +Host-level metrics via privileged Alpine containers. + +- One container per cluster node, pinned with `--target` placement +- Privileged containers with host filesystem bind-mounts: + - `/proc` (host) -> `/host/proc` (read-only) + - `/sys` (host) -> `/host/sys` (read-only) + - `/` (host) -> `/host/rootfs` (read-only) +- Exposes standard node_exporter metrics on `:9100` +- Uses `monitoring-allow` ACL for network access + +## Access + +| Service | URL | Credentials | +|---------|-----|-------------| +| Grafana | http://192.168.103.201:3000 | `admin` / `admin` | +| Prometheus | http://192.168.103.201:9090 | None (open) | + +Both services are accessible from the LAN through an OVN network forward +on IP `192.168.103.201`. The forward maps external ports directly to the +monitoring container's internal ports (no translation). + +Prometheus and Loki are not exposed externally by default. They are +accessible only from within the OVN network or through the Grafana +datasource proxy. + +## Dashboards + +Three dashboards are provisioned automatically during deployment. + +### Incus Cluster Overview + +Visualizes metrics scraped from the Incus `/1.0/metrics` endpoint on each +cluster node. + +- Instance CPU usage (per-instance, per-node) +- Memory usage and allocation +- Network I/O (bytes in/out per interface) +- Disk I/O (reads/writes, latency) +- Instance count and state + +### HAProxy Traffic + +Visualizes metrics from the HAProxy stats endpoint (`:8404/metrics`). + +- Request rate (frontend and backend) +- Backend health status (UP/DOWN per server) +- Active sessions and session rate +- Traffic volume (bytes in/out) +- HTTP response codes (2xx, 4xx, 5xx) +- Connection errors and retries + +### Host Resources + +Visualizes metrics from node_exporter on each cluster node. + +- CPU utilization (%) +- Memory utilization (%) +- Disk usage and I/O +- Network throughput per interface +- System load averages +- Filesystem free space + +## ACL Configuration + +Aether automatically creates per-instance ACLs with default-deny rules +for all containers on shared OVN networks. The observability stack needs +careful ACL configuration to allow scrape traffic. + +### How Aether ACLs work + +When Aether deploys a container, it creates ACLs with reject rules: + +| Direction | Priority | Action | Effect | +|-----------|----------|--------|--------| +| Egress | 111 | reject | Blocks all outbound by default | +| Ingress | 100 | reject | Blocks all inbound by default | + +These are OVN ACL rules enforced at the logical switch port level. + +### monitoring-allow ACL + +The observability containers use a `monitoring-allow` ACL with +`default.*.action=allow` set on all directions. This creates allow rules +at **priority 111** which matches or exceeds the Aether reject rules, +effectively overriding the default-deny policy. + +All observability containers (monitoring, node-exp-01/02/03) have this +ACL applied. + +### HAProxy scrape access + +HAProxy containers are managed by Aether and have their own ACLs with +default-deny. For Prometheus to scrape HAProxy metrics on `:8404`, the +HAProxy Aether ACLs need **explicit ingress rules** allowing TCP traffic +from the monitoring container (10.10.10.70) to port 8404. + +Without this, Prometheus targets for HAProxy will show as DOWN. + +### NIC state after ACL changes + +Changing `security.acls` on a container NIC can cause the NIC to go +**down**. After modifying ACLs, you may need to bring the NIC back up +manually or restart the container. + +## Resource Budget + +| Container | Image | RAM | Disk | Placement | +|-----------|-------|-----|------|-----------| +| monitoring | Debian/12 | 2 GiB | 20 GiB | oc-node-02 | +| node-exp-01 | Alpine | 128 MiB | — | oc-node-01 | +| node-exp-02 | Alpine | 128 MiB | — | oc-node-02 | +| node-exp-03 | Alpine | 128 MiB | — | oc-node-03 | + +**Totals:** +- RAM: ~2.4 GiB (2 GiB + 3 x 128 MiB) +- Disk: 20 GiB (only the monitoring container needs significant storage) +- OVN forward IPs: 1 (192.168.103.201) +- OVN network IPs: 4 (10.10.10.70-73) + +## Management + +The `deploy-observability` script handles the full lifecycle. + +### Deploy the stack + +```bash +incusos/deploy-observability --deploy +``` + +Creates the monitoring container and all node-exporter containers, +installs and configures Prometheus, Grafana, Loki, and Promtail, +provisions dashboards, sets up the OVN network forward, and configures +ACLs. + +### Check status + +```bash +incusos/deploy-observability --status +``` + +Shows the state of all containers, scrape target health, Grafana +accessibility, and OVN forward configuration. + +### Clean up + +```bash +incusos/deploy-observability --cleanup +``` + +Removes all observability containers, the OVN network forward, and +associated ACLs. Does not affect monitored targets (Incus nodes, +HAProxy). + +### Health check + +```bash +incusos/deploy-observability --doctor +``` + +Verifies prerequisites, checks container health, validates Prometheus +targets are UP, confirms Grafana datasources are working, and reports +any ACL issues. + +## Troubleshooting + +### Targets show as DOWN in Prometheus + +1. Check ACL rules on the target containers. Aether's default-deny ACLs + block all ingress including Prometheus scrapes. +2. Verify the NIC is up. Changing `security.acls` can bring the NIC down. + Check with `incus exec -- ip link` and bring it up if needed. +3. For Incus node targets: verify the client certificate and key are + correctly placed in the Prometheus config directory and that the cert + is trusted by the cluster. + +### Grafana can't reach Prometheus + +Check that Prometheus is running and listening on localhost:9090 from +inside the monitoring container: + +```bash +incus exec monitoring -- curl -s http://localhost:9090/-/healthy +``` + +If Prometheus is not running, check its service status: + +```bash +incus exec monitoring -- systemctl status prometheus +``` + +### No node metrics + +1. Verify the node-exporter containers are running and privileged: + ```bash + incus list node-exp + ``` +2. Check that host filesystem mounts are in place: + ```bash + incus exec node-exp-01 -- ls /host/proc/stat + ``` +3. Verify node_exporter is listening: + ```bash + incus exec node-exp-01 -- wget -qO- http://localhost:9100/metrics | head + ``` + +### HAProxy metrics show as invalid or empty + +HAProxy needs the `prometheus-exporter` service enabled in its stats +configuration. The stats section in `haproxy.cfg` must include: + +``` +frontend stats + bind *:8404 + http-request use-service prometheus-exporter if { path /metrics } + stats enable + stats uri /stats +``` + +Without the `use-service prometheus-exporter` directive, the `/metrics` +path returns HTML stats instead of Prometheus-format metrics. + +### Loki not receiving logs + +Verify Promtail can reach Loki on the gRPC port: + +```bash +incus exec monitoring -- curl -s http://localhost:3100/ready +``` + +Note that Loki uses gRPC port **9096** in this deployment (not the +default 9095) to avoid conflicts with Promtail's metrics port.