Add observability stack: Prometheus, Grafana, Loki with dashboards
Deploy and validate full monitoring stack on the Incus cluster: - Prometheus scraping 9 targets (3 Incus nodes, 2 HAProxy, 3 node-exporters, self) - Grafana with 3 dashboards (cluster overview, HAProxy traffic, host resources) - Loki + Promtail for log aggregation - OVN network forward for LAN access (192.168.103.201:3000/9090) - deploy-observability script with --deploy/--status/--cleanup/--doctor - Solved OVN ACL challenges: Aether default-deny requires monitoring-allow ACL Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
91a87456a5
commit
cc017b648d
|
|
@ -0,0 +1,85 @@
|
|||
---
|
||||
paths:
|
||||
- "incusos/deploy-observability"
|
||||
- "incusos/observability-dashboards"
|
||||
- "notes/observability-guide.md"
|
||||
---
|
||||
|
||||
# Observability Stack (Prometheus + Grafana + Loki)
|
||||
|
||||
## Deployment topology
|
||||
|
||||
- **monitoring** container: 10.10.10.70, pinned to oc-node-02, net-prod
|
||||
- Prometheus 2.54 on :9090 (15s scrape interval)
|
||||
- Grafana 12.4 on :3000 (default creds admin/admin)
|
||||
- Loki 3.6 on :3100 (HTTP) and **:9096** (gRPC -- NOT default 9095,
|
||||
9095 conflicts with Promtail metrics port)
|
||||
- Promtail 3.6 shipping logs to Loki
|
||||
- Debian/12, 2 GiB RAM, 20 GiB disk
|
||||
|
||||
- **node-exp-01/02/03**: 10.10.10.71-73, one per cluster node
|
||||
- Alpine, privileged, 128 MiB RAM each
|
||||
- Host filesystem mounts: /proc, /sys, / (all read-only)
|
||||
- node_exporter on :9100
|
||||
|
||||
## LAN access
|
||||
|
||||
- OVN network forward: 192.168.103.201
|
||||
- :3000 -> 10.10.10.70:3000 (Grafana)
|
||||
- :9090 -> 10.10.10.70:9090 (Prometheus)
|
||||
|
||||
## Prometheus scrape targets
|
||||
|
||||
| Target | Address | Auth | Port |
|
||||
|--------|---------|------|------|
|
||||
| Incus nodes | 192.168.102.140-142 | Client cert (TLS) | 8443 |
|
||||
| HAProxy | 10.10.10.50-51 | None | 8404 |
|
||||
| node-exporters | 10.10.10.71-73 | None | 9100 |
|
||||
|
||||
- Incus metrics endpoint: `/1.0/metrics` on each node, requires client
|
||||
certificate authentication (cluster cert + key in Prometheus config).
|
||||
- HAProxy metrics: requires `http-request use-service prometheus-exporter
|
||||
if { path /metrics }` in the stats frontend of haproxy.cfg. Without
|
||||
this, /metrics returns HTML stats, not Prometheus exposition format.
|
||||
|
||||
## ACL details (critical for scrape connectivity)
|
||||
|
||||
- **Aether default-deny**: every container on shared OVN network gets
|
||||
reject ACLs: priority 111 reject (egress), priority 100 reject (ingress).
|
||||
- **monitoring-allow ACL**: set `default.*.action=allow` on all directions.
|
||||
This creates allow rules at priority 111, matching Aether's reject
|
||||
priority and effectively overriding default-deny.
|
||||
- All observability containers (monitoring, node-exp-01/02/03) use the
|
||||
monitoring-allow ACL.
|
||||
- **HAProxy containers** are Aether-managed with their own ACLs. Need
|
||||
explicit ingress rules allowing TCP from 10.10.10.70 to port 8404 for
|
||||
Prometheus scrapes.
|
||||
|
||||
## NIC down after security.acls changes
|
||||
|
||||
Changing `security.acls` on a container NIC can cause the NIC to go
|
||||
**down**. After modifying ACLs:
|
||||
- Bring NIC up manually: `incus exec <container> -- ip link set eth0 up`
|
||||
- Or restart the container: `incus restart <container>`
|
||||
This is a known OVN behavior, not a bug.
|
||||
|
||||
## Dashboards
|
||||
|
||||
Three pre-provisioned dashboards:
|
||||
1. **Incus Cluster Overview** -- instance CPU, memory, network, disk I/O
|
||||
2. **HAProxy Traffic** -- requests, backend health, sessions, traffic
|
||||
3. **Host Resources** -- node CPU%, memory%, disk, network
|
||||
|
||||
## Script: deploy-observability
|
||||
|
||||
- `--deploy` -- full stack deployment
|
||||
- `--status` -- container state, target health, forward config
|
||||
- `--cleanup` -- remove all observability containers and forwards
|
||||
- `--doctor` -- prerequisite and health checks
|
||||
|
||||
## Troubleshooting quick reference
|
||||
|
||||
- Targets DOWN: check ACLs, check NIC state after ACL changes
|
||||
- No HAProxy metrics: need prometheus-exporter route in haproxy.cfg stats
|
||||
- Loki gRPC: port 9096, not 9095 (Promtail conflict)
|
||||
- Node metrics missing: check privileged flag, host mounts, node_exporter service
|
||||
|
|
@ -27,7 +27,9 @@ incus-contrib/
|
|||
│ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle
|
||||
│ ├── deploy-awx # AWX deployment + management on Incus cluster
|
||||
│ ├── deploy-haproxy # HAProxy LB deployment + management via Aether
|
||||
│ ├── deploy-observability # Prometheus + Grafana + Loki observability stack
|
||||
│ ├── awx-manifests/ # K8s manifests for AWX Operator + instance
|
||||
│ ├── observability-dashboards/ # Grafana dashboard JSON exports
|
||||
│ ├── helpers/
|
||||
│ │ ├── proxmox-screenshot # VMID -> PNG console screenshot
|
||||
│ │ ├── proxmox-api # Authenticated API calls (handles ! in token)
|
||||
|
|
@ -187,6 +189,7 @@ which files are being edited:
|
|||
| `networking-storage.md` | networking, storage, migration, UTM guides |
|
||||
| `awx-integration.md` | ansible/, deploy-awx, awx-manifests, AWX guide |
|
||||
| `haproxy-lb.md` | deploy-haproxy, HAProxy guide |
|
||||
| `observability.md` | deploy-observability, observability dashboards/guide |
|
||||
| `ovn-internals.md` | ovn-inspect, ovn-deep-dive |
|
||||
| `aether-api-payloads.md` | incus-mitm, api-interception |
|
||||
| `proxmox-ssh-rules.md` | **Always loaded** (no paths filter) |
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,559 @@
|
|||
{
|
||||
"uid": "incus-cluster-overview",
|
||||
"title": "Incus Cluster Overview",
|
||||
"description": "Overview of all Incus containers: CPU, memory, network, and disk I/O per instance.",
|
||||
"tags": ["incus", "cluster", "containers"],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "30s",
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"fiscalYearStartMonth": 0,
|
||||
"liveNow": false,
|
||||
"editable": true,
|
||||
"graphTooltip": 1,
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Cluster Status",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
|
||||
"id": 1,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Total Containers",
|
||||
"description": "Number of running Incus containers (instances reporting uptime).",
|
||||
"gridPos": { "h": 4, "w": 8, "x": 0, "y": 1 },
|
||||
"id": 2,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "count(incus_uptime_seconds)",
|
||||
"legendFormat": "Containers",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 10 },
|
||||
{ "color": "red", "value": 50 }
|
||||
]
|
||||
},
|
||||
"unit": "none",
|
||||
"color": { "mode": "thresholds" }
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "auto",
|
||||
"textMode": "auto",
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Total CPUs",
|
||||
"description": "Sum of effective CPUs across all Incus instances.",
|
||||
"gridPos": { "h": 4, "w": 8, "x": 8, "y": 1 },
|
||||
"id": 3,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(incus_cpu_effective_total)",
|
||||
"legendFormat": "CPUs",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
},
|
||||
"unit": "none",
|
||||
"color": { "mode": "thresholds" }
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "auto",
|
||||
"textMode": "auto",
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Total Memory",
|
||||
"description": "Sum of total memory across all Incus instances.",
|
||||
"gridPos": { "h": 4, "w": 8, "x": 16, "y": 1 },
|
||||
"id": 4,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(incus_memory_MemTotal_bytes)",
|
||||
"legendFormat": "Total Memory",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
},
|
||||
"unit": "bytes",
|
||||
"color": { "mode": "thresholds" }
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "auto",
|
||||
"textMode": "auto",
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Per-Instance CPU",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
|
||||
"id": 5,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "CPU Usage by Instance",
|
||||
"description": "Per-second CPU time consumed by each Incus instance.",
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 6 },
|
||||
"id": 6,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (name) (rate(incus_cpu_seconds_total[$__rate_interval]))",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "short",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Per-Instance Memory",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 },
|
||||
"id": 7,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Memory RSS by Instance",
|
||||
"description": "Resident set size (RSS) memory usage per Incus instance over time.",
|
||||
"gridPos": { "h": 8, "w": 14, "x": 0, "y": 15 },
|
||||
"id": 8,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "incus_memory_RSS_bytes",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "bytes",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "bargauge",
|
||||
"title": "Memory Usage % (RSS / Total)",
|
||||
"description": "Current memory utilization per instance as RSS divided by total memory.",
|
||||
"gridPos": { "h": 8, "w": 10, "x": 14, "y": 15 },
|
||||
"id": 9,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(incus_memory_RSS_bytes / incus_memory_MemTotal_bytes) * 100",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 60 },
|
||||
{ "color": "orange", "value": 80 },
|
||||
{ "color": "red", "value": 90 }
|
||||
]
|
||||
},
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100,
|
||||
"color": { "mode": "thresholds" }
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "horizontal",
|
||||
"displayMode": "gradient",
|
||||
"showUnfilled": true,
|
||||
"minVizWidth": 8,
|
||||
"minVizHeight": 16,
|
||||
"valueMode": "color"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Network I/O",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 23 },
|
||||
"id": 10,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Network Receive Rate by Instance",
|
||||
"description": "Inbound network throughput per Incus instance.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
|
||||
"id": 11,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (name) (rate(incus_network_receive_bytes_total[$__rate_interval]))",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Network Transmit Rate by Instance",
|
||||
"description": "Outbound network throughput per Incus instance.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
|
||||
"id": 12,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (name) (rate(incus_network_transmit_bytes_total[$__rate_interval]))",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Disk I/O",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 32 },
|
||||
"id": 13,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Disk Read Rate by Instance",
|
||||
"description": "Disk read throughput per Incus instance.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 33 },
|
||||
"id": 14,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (name) (rate(incus_disk_read_bytes_total[$__rate_interval]))",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Disk Write Rate by Instance",
|
||||
"description": "Disk write throughput per Incus instance.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 33 },
|
||||
"id": 15,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (name) (rate(incus_disk_written_bytes_total[$__rate_interval]))",
|
||||
"legendFormat": "{{name}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,518 @@
|
|||
{
|
||||
"uid": "incus-haproxy-traffic",
|
||||
"title": "HAProxy Traffic",
|
||||
"description": "HAProxy load balancer monitoring: request rates, backend health, traffic volume, and sessions.",
|
||||
"tags": ["haproxy", "loadbalancer", "incus"],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "30s",
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"fiscalYearStartMonth": 0,
|
||||
"liveNow": false,
|
||||
"editable": true,
|
||||
"graphTooltip": 1,
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Overview",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
|
||||
"id": 1,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Active Backend Servers",
|
||||
"description": "Total number of active servers across all HAProxy backends.",
|
||||
"gridPos": { "h": 4, "w": 8, "x": 0, "y": 1 },
|
||||
"id": 2,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(haproxy_backend_active_servers)",
|
||||
"legendFormat": "Active Servers",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "red", "value": null },
|
||||
{ "color": "yellow", "value": 1 },
|
||||
{ "color": "green", "value": 2 }
|
||||
]
|
||||
},
|
||||
"unit": "none",
|
||||
"color": { "mode": "thresholds" }
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "auto",
|
||||
"textMode": "auto",
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Current Sessions",
|
||||
"description": "Total current sessions across all HAProxy backends.",
|
||||
"gridPos": { "h": 4, "w": 8, "x": 8, "y": 1 },
|
||||
"id": 3,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(haproxy_backend_current_sessions)",
|
||||
"legendFormat": "Sessions",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 100 },
|
||||
{ "color": "red", "value": 500 }
|
||||
]
|
||||
},
|
||||
"unit": "none",
|
||||
"color": { "mode": "thresholds" }
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "auto",
|
||||
"textMode": "auto",
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "stat",
|
||||
"title": "Total HTTP Requests",
|
||||
"description": "Total HTTP requests across all HAProxy frontends (counter).",
|
||||
"gridPos": { "h": 4, "w": 8, "x": 16, "y": 1 },
|
||||
"id": 4,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(haproxy_frontend_http_requests_total[$__rate_interval]))",
|
||||
"legendFormat": "Requests/s",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
},
|
||||
"unit": "reqps",
|
||||
"color": { "mode": "thresholds" }
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"orientation": "auto",
|
||||
"textMode": "auto",
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "HTTP Requests",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
|
||||
"id": 5,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "HTTP Request Rate by Frontend",
|
||||
"description": "Rate of HTTP requests per second, broken down by HAProxy frontend/proxy.",
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 6 },
|
||||
"id": 6,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (proxy) (rate(haproxy_frontend_http_requests_total[$__rate_interval]))",
|
||||
"legendFormat": "{{proxy}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 20,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "reqps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Backend Health",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 14 },
|
||||
"id": 7,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "table",
|
||||
"title": "Backend Status",
|
||||
"description": "Current status, active server count, and average response time for each HAProxy backend.",
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 15 },
|
||||
"id": 8,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "haproxy_backend_status",
|
||||
"legendFormat": "",
|
||||
"refId": "A",
|
||||
"format": "table",
|
||||
"instant": true
|
||||
},
|
||||
{
|
||||
"expr": "haproxy_backend_active_servers",
|
||||
"legendFormat": "",
|
||||
"refId": "B",
|
||||
"format": "table",
|
||||
"instant": true
|
||||
},
|
||||
{
|
||||
"expr": "haproxy_server_response_time_average_seconds",
|
||||
"legendFormat": "",
|
||||
"refId": "C",
|
||||
"format": "table",
|
||||
"instant": true
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"displayMode": "auto",
|
||||
"inspect": false,
|
||||
"filterable": true
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
},
|
||||
"color": { "mode": "thresholds" }
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Value #A" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Status" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Value #B" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Active Servers" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Value #C" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Avg Response Time" },
|
||||
{ "id": "unit", "value": "s" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "proxy" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Backend" }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"sortBy": [],
|
||||
"footer": {
|
||||
"show": false,
|
||||
"reducer": ["sum"],
|
||||
"fields": ""
|
||||
}
|
||||
},
|
||||
"transformations": [
|
||||
{
|
||||
"id": "merge",
|
||||
"options": {}
|
||||
},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {
|
||||
"Time": true,
|
||||
"__name__": true,
|
||||
"instance": false,
|
||||
"job": true
|
||||
},
|
||||
"indexByName": {},
|
||||
"renameByName": {}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Traffic",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 23 },
|
||||
"id": 9,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Frontend Bytes In",
|
||||
"description": "Inbound traffic rate per HAProxy frontend.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 24 },
|
||||
"id": 10,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (proxy) (rate(haproxy_frontend_bytes_in_total[$__rate_interval]))",
|
||||
"legendFormat": "{{proxy}} (in)",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 20,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Frontend Bytes Out",
|
||||
"description": "Outbound traffic rate per HAProxy frontend.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 24 },
|
||||
"id": 11,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (proxy) (rate(haproxy_frontend_bytes_out_total[$__rate_interval]))",
|
||||
"legendFormat": "{{proxy}} (out)",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 20,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Sessions",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 32 },
|
||||
"id": 12,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Current Sessions by Proxy",
|
||||
"description": "Current active sessions for each HAProxy backend/proxy over time.",
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 33 },
|
||||
"id": 13,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "haproxy_backend_current_sessions",
|
||||
"legendFormat": "{{proxy}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 20,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "none",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,429 @@
|
|||
{
|
||||
"uid": "incus-host-resources",
|
||||
"title": "Host Resources (IncusOS Nodes)",
|
||||
"description": "Physical host monitoring for IncusOS cluster nodes: CPU, memory, disk, and network from node_exporter metrics.",
|
||||
"tags": ["incusos", "host", "node-exporter"],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "30s",
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"fiscalYearStartMonth": 0,
|
||||
"liveNow": false,
|
||||
"editable": true,
|
||||
"graphTooltip": 1,
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"type": "row",
|
||||
"title": "CPU",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
|
||||
"id": 1,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "CPU Usage % by Node",
|
||||
"description": "Overall CPU utilization per host node, calculated as 1 minus the average idle CPU rate.",
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 1 },
|
||||
"id": 2,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval]))) * 100",
|
||||
"legendFormat": "{{instance}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"fillOpacity": 25,
|
||||
"gradientMode": "scheme",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "line" }
|
||||
},
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100,
|
||||
"color": { "mode": "continuous-GrYlRd" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 70 },
|
||||
{ "color": "red", "value": 90 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Memory",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 },
|
||||
"id": 3,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Memory Usage % by Node",
|
||||
"description": "Memory utilization percentage per host node, calculated from available vs total memory.",
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 10 },
|
||||
"id": 4,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100",
|
||||
"legendFormat": "{{instance}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 2,
|
||||
"fillOpacity": 25,
|
||||
"gradientMode": "scheme",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "line" }
|
||||
},
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100,
|
||||
"color": { "mode": "continuous-GrYlRd" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 70 },
|
||||
{ "color": "red", "value": 90 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max", "lastNotNull"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Disk",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 18 },
|
||||
"id": 5,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Root Filesystem Available Space",
|
||||
"description": "Available disk space on the root filesystem (mountpoint='/') per host node.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 19 },
|
||||
"id": 6,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "node_filesystem_avail_bytes{mountpoint=\"/\"}",
|
||||
"legendFormat": "{{instance}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "bytes",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull", "min"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Disk Read Rate by Node",
|
||||
"description": "Disk read throughput (bytes/sec) aggregated per host node.",
|
||||
"gridPos": { "h": 8, "w": 6, "x": 12, "y": 19 },
|
||||
"id": 7,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (instance) (rate(node_disk_read_bytes_total[$__rate_interval]))",
|
||||
"legendFormat": "{{instance}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "list", "placement": "bottom", "calcs": [] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Disk Write Rate by Node",
|
||||
"description": "Disk write throughput (bytes/sec) aggregated per host node.",
|
||||
"gridPos": { "h": 8, "w": 6, "x": 18, "y": 19 },
|
||||
"id": 8,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (instance) (rate(node_disk_written_bytes_total[$__rate_interval]))",
|
||||
"legendFormat": "{{instance}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "list", "placement": "bottom", "calcs": [] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Network",
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 27 },
|
||||
"id": 9,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Network Receive Rate by Device & Node",
|
||||
"description": "Inbound network throughput per network device and host node.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
|
||||
"id": 10,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(node_network_receive_bytes_total{device!~\"lo|veth.*|br-.*|docker.*\"}[$__rate_interval])",
|
||||
"legendFormat": "{{instance}} - {{device}} (rx)",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Network Transmit Rate by Device & Node",
|
||||
"description": "Outbound network throughput per network device and host node.",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
|
||||
"id": 11,
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "PBFA97CFB590B2093"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|br-.*|docker.*\"}[$__rate_interval])",
|
||||
"legendFormat": "{{instance}} - {{device}} (tx)",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 15,
|
||||
"gradientMode": "none",
|
||||
"spanNulls": false,
|
||||
"showPoints": "never",
|
||||
"pointSize": 5,
|
||||
"stacking": { "mode": "none", "group": "A" },
|
||||
"axisPlacement": "auto",
|
||||
"axisLabel": "",
|
||||
"scaleDistribution": { "type": "linear" },
|
||||
"thresholdsStyle": { "mode": "off" }
|
||||
},
|
||||
"unit": "Bps",
|
||||
"color": { "mode": "palette-classic" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"tooltip": { "mode": "multi", "sort": "desc" },
|
||||
"legend": { "displayMode": "table", "placement": "right", "calcs": ["mean", "max"] }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,337 @@
|
|||
# Observability Stack — Prometheus, Grafana, Loki on Incus
|
||||
|
||||
A lightweight observability stack deployed as Incus containers on the lab
|
||||
cluster. Provides metric collection (Prometheus), dashboards (Grafana),
|
||||
and log aggregation (Loki) for Incus nodes, HAProxy load balancers, and
|
||||
host-level resources.
|
||||
|
||||
The stack runs entirely on OVN and is managed by the `deploy-observability`
|
||||
script. All containers use the `monitoring-allow` ACL to permit scrape
|
||||
traffic on the shared network.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
LAN (192.168.103.0/24)
|
||||
|
|
||||
| OVN network forward (192.168.103.201)
|
||||
| :3000 -> 10.10.10.70:3000 (Grafana)
|
||||
| :9090 -> 10.10.10.70:9090 (Prometheus)
|
||||
|
|
||||
═══════╪══════════════════════════════════════════════
|
||||
| OVN net-prod (10.10.10.0/24)
|
||||
|
|
||||
v
|
||||
+----------------------------------------------+
|
||||
| monitoring (10.10.10.70, oc-node-02) |
|
||||
| Debian/12 container — 2 GiB RAM, 20 GiB |
|
||||
| |
|
||||
| +-------------+ +-----------+ |
|
||||
| | Prometheus | | Grafana | |
|
||||
| | :9090 | | :3000 | |
|
||||
| +-------------+ +-----------+ |
|
||||
| +-------------+ +-----------+ |
|
||||
| | Loki | | Promtail | |
|
||||
| | :3100 | | (agent) | |
|
||||
| | gRPC :9096 | | | |
|
||||
| +-------------+ +-----------+ |
|
||||
+----------------------------------------------+
|
||||
|
|
||||
| Scrape targets
|
||||
|
|
||||
+-------> Incus nodes (client cert auth, :8443)
|
||||
| 192.168.102.140 /1.0/metrics
|
||||
| 192.168.102.141 /1.0/metrics
|
||||
| 192.168.102.142 /1.0/metrics
|
||||
|
|
||||
+-------> HAProxy (:8404)
|
||||
| 10.10.10.50 /metrics
|
||||
| 10.10.10.51 /metrics
|
||||
|
|
||||
+-------> node-exporters (:9100)
|
||||
10.10.10.71 node-exp-01 (oc-node-01)
|
||||
10.10.10.72 node-exp-02 (oc-node-02)
|
||||
10.10.10.73 node-exp-03 (oc-node-03)
|
||||
|
||||
+------------------+ +------------------+ +------------------+
|
||||
| node-exp-01 | | node-exp-02 | | node-exp-03 |
|
||||
| 10.10.10.71 | | 10.10.10.72 | | 10.10.10.73 |
|
||||
| Alpine, privil. | | Alpine, privil. | | Alpine, privil. |
|
||||
| :9100 | | :9100 | | :9100 |
|
||||
| host /proc,/sys | | host /proc,/sys | | host /proc,/sys |
|
||||
| 128 MiB RAM | | 128 MiB RAM | | 128 MiB RAM |
|
||||
| pinned to node | | pinned to node | | pinned to node |
|
||||
+------------------+ +------------------+ +------------------+
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
### Prometheus 2.54
|
||||
|
||||
Metric collection engine. Scrapes all targets at a 15-second interval.
|
||||
|
||||
- Stores metrics locally in the monitoring container
|
||||
- Client certificate authentication for Incus `/1.0/metrics` endpoints
|
||||
(uses cluster client cert + key)
|
||||
- Plain HTTP scrape for HAProxy stats (`:8404/metrics`) and
|
||||
node-exporters (`:9100/metrics`)
|
||||
|
||||
### Grafana 12.4
|
||||
|
||||
Dashboard and visualization frontend.
|
||||
|
||||
- Default credentials: `admin` / `admin`
|
||||
- Pre-provisioned datasources for Prometheus and Loki
|
||||
- Three dashboards deployed automatically (see Dashboards section)
|
||||
- Accessible from LAN via OVN network forward
|
||||
|
||||
### Loki 3.6
|
||||
|
||||
Log aggregation backend.
|
||||
|
||||
- HTTP API on port 3100
|
||||
- gRPC on port **9096** (not the default 9095 -- avoids conflict with
|
||||
Promtail's own metrics port)
|
||||
- 7-day retention policy
|
||||
- Receives logs from Promtail running in the same container
|
||||
|
||||
### Promtail 3.6
|
||||
|
||||
Log shipping agent, co-located with Loki in the monitoring container.
|
||||
|
||||
- Pushes logs to Loki via gRPC on localhost:9096
|
||||
- Scrapes container logs and system journal
|
||||
|
||||
### node_exporter
|
||||
|
||||
Host-level metrics via privileged Alpine containers.
|
||||
|
||||
- One container per cluster node, pinned with `--target` placement
|
||||
- Privileged containers with host filesystem bind-mounts:
|
||||
- `/proc` (host) -> `/host/proc` (read-only)
|
||||
- `/sys` (host) -> `/host/sys` (read-only)
|
||||
- `/` (host) -> `/host/rootfs` (read-only)
|
||||
- Exposes standard node_exporter metrics on `:9100`
|
||||
- Uses `monitoring-allow` ACL for network access
|
||||
|
||||
## Access
|
||||
|
||||
| Service | URL | Credentials |
|
||||
|---------|-----|-------------|
|
||||
| Grafana | http://192.168.103.201:3000 | `admin` / `admin` |
|
||||
| Prometheus | http://192.168.103.201:9090 | None (open) |
|
||||
|
||||
Both services are accessible from the LAN through an OVN network forward
|
||||
on IP `192.168.103.201`. The forward maps external ports directly to the
|
||||
monitoring container's internal ports (no translation).
|
||||
|
||||
Prometheus and Loki are not exposed externally by default. They are
|
||||
accessible only from within the OVN network or through the Grafana
|
||||
datasource proxy.
|
||||
|
||||
## Dashboards
|
||||
|
||||
Three dashboards are provisioned automatically during deployment.
|
||||
|
||||
### Incus Cluster Overview
|
||||
|
||||
Visualizes metrics scraped from the Incus `/1.0/metrics` endpoint on each
|
||||
cluster node.
|
||||
|
||||
- Instance CPU usage (per-instance, per-node)
|
||||
- Memory usage and allocation
|
||||
- Network I/O (bytes in/out per interface)
|
||||
- Disk I/O (reads/writes, latency)
|
||||
- Instance count and state
|
||||
|
||||
### HAProxy Traffic
|
||||
|
||||
Visualizes metrics from the HAProxy stats endpoint (`:8404/metrics`).
|
||||
|
||||
- Request rate (frontend and backend)
|
||||
- Backend health status (UP/DOWN per server)
|
||||
- Active sessions and session rate
|
||||
- Traffic volume (bytes in/out)
|
||||
- HTTP response codes (2xx, 4xx, 5xx)
|
||||
- Connection errors and retries
|
||||
|
||||
### Host Resources
|
||||
|
||||
Visualizes metrics from node_exporter on each cluster node.
|
||||
|
||||
- CPU utilization (%)
|
||||
- Memory utilization (%)
|
||||
- Disk usage and I/O
|
||||
- Network throughput per interface
|
||||
- System load averages
|
||||
- Filesystem free space
|
||||
|
||||
## ACL Configuration
|
||||
|
||||
Aether automatically creates per-instance ACLs with default-deny rules
|
||||
for all containers on shared OVN networks. The observability stack needs
|
||||
careful ACL configuration to allow scrape traffic.
|
||||
|
||||
### How Aether ACLs work
|
||||
|
||||
When Aether deploys a container, it creates ACLs with reject rules:
|
||||
|
||||
| Direction | Priority | Action | Effect |
|
||||
|-----------|----------|--------|--------|
|
||||
| Egress | 111 | reject | Blocks all outbound by default |
|
||||
| Ingress | 100 | reject | Blocks all inbound by default |
|
||||
|
||||
These are OVN ACL rules enforced at the logical switch port level.
|
||||
|
||||
### monitoring-allow ACL
|
||||
|
||||
The observability containers use a `monitoring-allow` ACL with
|
||||
`default.*.action=allow` set on all directions. This creates allow rules
|
||||
at **priority 111** which matches or exceeds the Aether reject rules,
|
||||
effectively overriding the default-deny policy.
|
||||
|
||||
All observability containers (monitoring, node-exp-01/02/03) have this
|
||||
ACL applied.
|
||||
|
||||
### HAProxy scrape access
|
||||
|
||||
HAProxy containers are managed by Aether and have their own ACLs with
|
||||
default-deny. For Prometheus to scrape HAProxy metrics on `:8404`, the
|
||||
HAProxy Aether ACLs need **explicit ingress rules** allowing TCP traffic
|
||||
from the monitoring container (10.10.10.70) to port 8404.
|
||||
|
||||
Without this, Prometheus targets for HAProxy will show as DOWN.
|
||||
|
||||
### NIC state after ACL changes
|
||||
|
||||
Changing `security.acls` on a container NIC can cause the NIC to go
|
||||
**down**. After modifying ACLs, you may need to bring the NIC back up
|
||||
manually or restart the container.
|
||||
|
||||
## Resource Budget
|
||||
|
||||
| Container | Image | RAM | Disk | Placement |
|
||||
|-----------|-------|-----|------|-----------|
|
||||
| monitoring | Debian/12 | 2 GiB | 20 GiB | oc-node-02 |
|
||||
| node-exp-01 | Alpine | 128 MiB | — | oc-node-01 |
|
||||
| node-exp-02 | Alpine | 128 MiB | — | oc-node-02 |
|
||||
| node-exp-03 | Alpine | 128 MiB | — | oc-node-03 |
|
||||
|
||||
**Totals:**
|
||||
- RAM: ~2.4 GiB (2 GiB + 3 x 128 MiB)
|
||||
- Disk: 20 GiB (only the monitoring container needs significant storage)
|
||||
- OVN forward IPs: 1 (192.168.103.201)
|
||||
- OVN network IPs: 4 (10.10.10.70-73)
|
||||
|
||||
## Management
|
||||
|
||||
The `deploy-observability` script handles the full lifecycle.
|
||||
|
||||
### Deploy the stack
|
||||
|
||||
```bash
|
||||
incusos/deploy-observability --deploy
|
||||
```
|
||||
|
||||
Creates the monitoring container and all node-exporter containers,
|
||||
installs and configures Prometheus, Grafana, Loki, and Promtail,
|
||||
provisions dashboards, sets up the OVN network forward, and configures
|
||||
ACLs.
|
||||
|
||||
### Check status
|
||||
|
||||
```bash
|
||||
incusos/deploy-observability --status
|
||||
```
|
||||
|
||||
Shows the state of all containers, scrape target health, Grafana
|
||||
accessibility, and OVN forward configuration.
|
||||
|
||||
### Clean up
|
||||
|
||||
```bash
|
||||
incusos/deploy-observability --cleanup
|
||||
```
|
||||
|
||||
Removes all observability containers, the OVN network forward, and
|
||||
associated ACLs. Does not affect monitored targets (Incus nodes,
|
||||
HAProxy).
|
||||
|
||||
### Health check
|
||||
|
||||
```bash
|
||||
incusos/deploy-observability --doctor
|
||||
```
|
||||
|
||||
Verifies prerequisites, checks container health, validates Prometheus
|
||||
targets are UP, confirms Grafana datasources are working, and reports
|
||||
any ACL issues.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Targets show as DOWN in Prometheus
|
||||
|
||||
1. Check ACL rules on the target containers. Aether's default-deny ACLs
|
||||
block all ingress including Prometheus scrapes.
|
||||
2. Verify the NIC is up. Changing `security.acls` can bring the NIC down.
|
||||
Check with `incus exec <container> -- ip link` and bring it up if needed.
|
||||
3. For Incus node targets: verify the client certificate and key are
|
||||
correctly placed in the Prometheus config directory and that the cert
|
||||
is trusted by the cluster.
|
||||
|
||||
### Grafana can't reach Prometheus
|
||||
|
||||
Check that Prometheus is running and listening on localhost:9090 from
|
||||
inside the monitoring container:
|
||||
|
||||
```bash
|
||||
incus exec monitoring -- curl -s http://localhost:9090/-/healthy
|
||||
```
|
||||
|
||||
If Prometheus is not running, check its service status:
|
||||
|
||||
```bash
|
||||
incus exec monitoring -- systemctl status prometheus
|
||||
```
|
||||
|
||||
### No node metrics
|
||||
|
||||
1. Verify the node-exporter containers are running and privileged:
|
||||
```bash
|
||||
incus list node-exp
|
||||
```
|
||||
2. Check that host filesystem mounts are in place:
|
||||
```bash
|
||||
incus exec node-exp-01 -- ls /host/proc/stat
|
||||
```
|
||||
3. Verify node_exporter is listening:
|
||||
```bash
|
||||
incus exec node-exp-01 -- wget -qO- http://localhost:9100/metrics | head
|
||||
```
|
||||
|
||||
### HAProxy metrics show as invalid or empty
|
||||
|
||||
HAProxy needs the `prometheus-exporter` service enabled in its stats
|
||||
configuration. The stats section in `haproxy.cfg` must include:
|
||||
|
||||
```
|
||||
frontend stats
|
||||
bind *:8404
|
||||
http-request use-service prometheus-exporter if { path /metrics }
|
||||
stats enable
|
||||
stats uri /stats
|
||||
```
|
||||
|
||||
Without the `use-service prometheus-exporter` directive, the `/metrics`
|
||||
path returns HTML stats instead of Prometheus-format metrics.
|
||||
|
||||
### Loki not receiving logs
|
||||
|
||||
Verify Promtail can reach Loki on the gRPC port:
|
||||
|
||||
```bash
|
||||
incus exec monitoring -- curl -s http://localhost:3100/ready
|
||||
```
|
||||
|
||||
Note that Loki uses gRPC port **9096** in this deployment (not the
|
||||
default 9095) to avoid conflicts with Promtail's metrics port.
|
||||
Loading…
Reference in New Issue