Maarten
3520eb76ea
Fix Promtail log shipping and Logs Explorer dashboard queries
...
Promtail was not shipping logs due to missing /var/lib/promtail directory
and insufficient permissions (not in adm or systemd-journal groups).
Logs Explorer dashboard queries failed on Loki 3.x because allValue ".*"
is rejected as an empty matcher — changed to ".+" which requires at least
one character. Removed unit filter from main Logs query since varlog
streams lack the unit label.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 12:43:03 +01:00
Maarten
32d341fc99
Fix dashboard query bugs, add screenshots, update observability guide
...
Replace incorrect Prometheus queries across 4 dashboard JSON files:
- incus_uptime_seconds → incus_cpu_effective_total for template vars and counts
- incus_memory_RSS_bytes → incus_memory_Active_bytes (RSS only reports for VMs)
- Uptime panel → Processes panel using incus_procs_total
- Remove wrong instance filter from network deep dive instance panels
- Fix Loki data directory permissions in deploy-observability
Add viewport screenshots of all 9 Grafana dashboards and embed them in
the observability guide with captions and updated descriptions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 09:44:03 +01:00
Maarten
6c7714459b
Add professional Grafana dashboard suite with management tooling
...
- New manage-dashboards script: install/export/list/validate/status
actions for Grafana dashboard lifecycle via HTTP API, with datasource
UID portability (${DS_PROMETHEUS}/${DS_LOKI} variables) and folder
organization
- Reorganize dashboards into core/ and community/ subdirectories with
dashboard.yaml manifest
- Enhance 3 existing dashboards: template variables (instance/node/
proxy/server), cross-dashboard navigation links, new panels (instance
status table, CPU mode breakdown, load averages, memory breakdown,
filesystem bar gauge, response code distribution, server state
timeline, connection errors)
- Add 6 new dashboards: Instance Deep Dive, Storage & Filesystem,
Network Deep Dive, Logs Explorer (Loki), Prometheus Health, Capacity
Planning (with predict_linear forecasting)
- Update deploy-observability: explicit datasource UIDs in provisioning,
phase 9 uses manage-dashboards --install, new --workloads action for
dummy metric generation (workload-web + workload-api containers)
- Update observability guide with all 9 dashboard descriptions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 07:50:13 +01:00
Maarten
671d79b77f
Convert all ASCII diagrams to mermaid with consistent styling
...
Replace ASCII art diagrams in 9 guide files with mermaid flowcharts
using a unified style system:
- 5 semantic colors (Okabe-Ito derived, color-blind safe):
teal (nodes), sky blue (instances), blue (network),
amber (load balancers), mauve (management)
- classDef for reusable styles, tinted subgraph fills
- Max 2 lines per node, specs in companion tables
- <br/> for line breaks (not \n, which Gitea renders literally)
- Solid arrows for data flow, dashed for control plane
Files converted:
- haproxy-guide.md (1 diagram)
- observability-guide.md (1 diagram)
- api-interception-guide.md (1 diagram)
- incusos-break-fix.md (3 diagrams: partitions, boot chain, updates)
- shared-storage-guide.md (3 diagrams: iSCSI arch, topology, multi-host)
- networking-guide.md (2 diagrams: bridge, OVN topology)
- operations-center-guide.md (1 diagram)
- ovn-deep-dive.md (3 diagrams: lab, bridges, Geneve mesh)
- production-lab-guide.md (1 diagram)
- mermaid-test.md updated as style reference with <br/> fix
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 17:31:29 +01:00
Maarten
cc017b648d
Add observability stack: Prometheus, Grafana, Loki with dashboards
...
Deploy and validate full monitoring stack on the Incus cluster:
- Prometheus scraping 9 targets (3 Incus nodes, 2 HAProxy, 3 node-exporters, self)
- Grafana with 3 dashboards (cluster overview, HAProxy traffic, host resources)
- Loki + Promtail for log aggregation
- OVN network forward for LAN access (192.168.103.201:3000/9090)
- deploy-observability script with --deploy/--status/--cleanup/--doctor
- Solved OVN ACL challenges: Aether default-deny requires monitoring-allow ACL
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 15:44:59 +01:00
Maarten