The incus scrape job (/1.0/metrics) exports both incus_* and node_* metrics,
duplicating the dedicated node-exporter containers. This caused 6 "nodes"
instead of 3 in dashboards, with the incus-sourced entries showing container
artifacts (epoch-0 boot time, 128 MiB RAM).
Prometheus config changes:
- Drop node_* metrics from incus job via metric_relabel_configs
- Relabel node-exporter instances to friendly names (oc-node-01/02/03)
using per-target nodename labels + relabel_configs
Dashboard changes (belt-and-suspenders):
- Add job="node-exporter" to all template variables and panel queries
in host-resources, network-deep-dive, storage-filesystem, capacity-planning
- Add Logs Drilldown link to logs-explorer dashboard
- Add Promtail journal relabels for level and service_name
Also: install Logs Drilldown plugin, update observability guide with
Logs Drilldown documentation and screenshots.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Promtail was not shipping logs due to missing /var/lib/promtail directory
and insufficient permissions (not in adm or systemd-journal groups).
Logs Explorer dashboard queries failed on Loki 3.x because allValue ".*"
is rejected as an empty matcher — changed to ".+" which requires at least
one character. Removed unit filter from main Logs query since varlog
streams lack the unit label.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace incorrect Prometheus queries across 4 dashboard JSON files:
- incus_uptime_seconds → incus_cpu_effective_total for template vars and counts
- incus_memory_RSS_bytes → incus_memory_Active_bytes (RSS only reports for VMs)
- Uptime panel → Processes panel using incus_procs_total
- Remove wrong instance filter from network deep dive instance panels
- Fix Loki data directory permissions in deploy-observability
Add viewport screenshots of all 9 Grafana dashboards and embed them in
the observability guide with captions and updated descriptions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
deploy-observability:
- Discover DNS from UPLINK config instead of hardcoding 8.8.8.8
- Set ipv4.address on OVN NICs (port security drops unknown IPs)
- Install ca-certificates/curl/gnupg before adding Grafana apt repo
- Fix ACL create syntax (--description flag, not key=value)
- Add NIC recovery after ACL changes (NICs go DOWN)
manage-dashboards:
- Rewrite read_manifest() without PyYAML dependency
- Fix ensure_folder() mixing debug output with return value
- Rewrite install payload generation with heredoc + file I/O
- Separate incus file push from Grafana API call
- Fix while-read loops consuming stdin from incus exec (fd3)
Dashboard JSON:
- Add allValue:".*" to all template variables with includeAll
Screenshots:
- Add Playwright screenshots of all 9 Grafana dashboards
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Maarten