maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
56 Commits 1 Branch 0 Tags 5.5 MiB
Commit Graph

4 Commits

Author SHA1 Message Date
Maarten 45801dc349 Fix metric source mixing: drop node_* from incus job, relabel node-exporter to friendly names 
The incus scrape job (/1.0/metrics) exports both incus_* and node_* metrics,
duplicating the dedicated node-exporter containers. This caused 6 "nodes"
instead of 3 in dashboards, with the incus-sourced entries showing container
artifacts (epoch-0 boot time, 128 MiB RAM).

Prometheus config changes:
- Drop node_* metrics from incus job via metric_relabel_configs
- Relabel node-exporter instances to friendly names (oc-node-01/02/03)
  using per-target nodename labels + relabel_configs

Dashboard changes (belt-and-suspenders):
- Add job="node-exporter" to all template variables and panel queries
  in host-resources, network-deep-dive, storage-filesystem, capacity-planning
- Add Logs Drilldown link to logs-explorer dashboard
- Add Promtail journal relabels for level and service_name

Also: install Logs Drilldown plugin, update observability guide with
Logs Drilldown documentation and screenshots.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 14:05:55 +01:00
Maarten 3520eb76ea Fix Promtail log shipping and Logs Explorer dashboard queries 
Promtail was not shipping logs due to missing /var/lib/promtail directory
and insufficient permissions (not in adm or systemd-journal groups).
Logs Explorer dashboard queries failed on Loki 3.x because allValue ".*"
is rejected as an empty matcher — changed to ".+" which requires at least
one character. Removed unit filter from main Logs query since varlog
streams lack the unit label.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 12:43:03 +01:00
Maarten f1f85d21ea Fix deploy-observability and manage-dashboards issues found during live deployment 
deploy-observability:
- Discover DNS from UPLINK config instead of hardcoding 8.8.8.8
- Set ipv4.address on OVN NICs (port security drops unknown IPs)
- Install ca-certificates/curl/gnupg before adding Grafana apt repo
- Fix ACL create syntax (--description flag, not key=value)
- Add NIC recovery after ACL changes (NICs go DOWN)

manage-dashboards:
- Rewrite read_manifest() without PyYAML dependency
- Fix ensure_folder() mixing debug output with return value
- Rewrite install payload generation with heredoc + file I/O
- Separate incus file push from Grafana API call
- Fix while-read loops consuming stdin from incus exec (fd3)

Dashboard JSON:
- Add allValue:".*" to all template variables with includeAll

Screenshots:
- Add Playwright screenshots of all 9 Grafana dashboards

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 08:54:58 +01:00
Maarten 6c7714459b Add professional Grafana dashboard suite with management tooling 
- New manage-dashboards script: install/export/list/validate/status
  actions for Grafana dashboard lifecycle via HTTP API, with datasource
  UID portability (${DS_PROMETHEUS}/${DS_LOKI} variables) and folder
  organization
- Reorganize dashboards into core/ and community/ subdirectories with
  dashboard.yaml manifest
- Enhance 3 existing dashboards: template variables (instance/node/
  proxy/server), cross-dashboard navigation links, new panels (instance
  status table, CPU mode breakdown, load averages, memory breakdown,
  filesystem bar gauge, response code distribution, server state
  timeline, connection errors)
- Add 6 new dashboards: Instance Deep Dive, Storage & Filesystem,
  Network Deep Dive, Logs Explorer (Loki), Prometheus Health, Capacity
  Planning (with predict_linear forecasting)
- Update deploy-observability: explicit datasource UIDs in provisioning,
  phase 9 uses manage-dashboards --install, new --workloads action for
  dummy metric generation (workload-web + workload-api containers)
- Update observability guide with all 9 dashboard descriptions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 07:50:13 +01:00