Three end-to-end blueprint examples demonstrating AWX configuring real
applications via Aether lifecycle hooks:
- webapp: nginx+PHP web tier + MariaDB with cross-instance service
discovery, credential sharing, and OVN ACL management
- staticsite: horizontally scaled nginx with per-instance branding
- k3scluster: K3s control plane + workers with automatic token
discovery and cluster join (fixes Incus files API symlink issue
where node-token symlink returns target path instead of content)
Includes dispatcher playbooks, shared helpers (sibling discovery,
temporary internet NIC), decommission handlers, and deploy-awx
changes to push showcase playbooks and create AWX templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Guide changes:
- Architecture diagram now shows auto-discovered targets and optional
oc-server monitoring
- Fix /host/rootfs → /host/root mount path in node_exporter section
- Document ARGS variable name (not NODE_EXPORTER_OPTS) for Alpine init
- Add --oc-server and --nodes flag documentation to Management section
- Update resource budget with oc-server row
- Expand "No node metrics" troubleshooting with ARGS diagnostic steps
- Update screenshot captions with correct RAM values (19.5 GiB)
Screenshots refreshed after fixing node-exporter RAM bug — all 9
dashboards now show correct host metrics (20 GiB RAM, not 128 MiB).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three changes to deploy-observability:
1. Fix node-exporter showing 128 MiB RAM: the conf.d template wrote
NODE_EXPORTER_OPTS but Alpine's init script reads ARGS, so
node-exporter started with no flags and read container /proc
instead of host /proc. Changed to ARGS.
2. Auto-discover cluster nodes from `incus cluster list` instead of
hardcoded arrays. Old values become fallback defaults for --dry-run.
Added --nodes flag for manual override.
3. Add --oc-server flag to monitor the standalone OC management VM.
Deploys node-exporter with host mounts and a proxy device to expose
9100 on the host IP (oc-server uses incusbr0, not OVN). Adds to
both Incus API and node-exporter Prometheus targets. Updated status
and cleanup to handle the oc-server container.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace merge transformation with joinByField on instance — the merge
couldn't combine rows from queries with different label cardinality
(node_uname_info has many extra labels), producing 7 rows instead of 3.
Fix boot time showing "56 years ago" by multiplying node_boot_time_seconds
by 1000 — Grafana's dateTimeFromNow unit expects epoch milliseconds.
Also hide the exported_nodename column (container hostname, not useful).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The incus scrape job (/1.0/metrics) exports both incus_* and node_* metrics,
duplicating the dedicated node-exporter containers. This caused 6 "nodes"
instead of 3 in dashboards, with the incus-sourced entries showing container
artifacts (epoch-0 boot time, 128 MiB RAM).
Prometheus config changes:
- Drop node_* metrics from incus job via metric_relabel_configs
- Relabel node-exporter instances to friendly names (oc-node-01/02/03)
using per-target nodename labels + relabel_configs
Dashboard changes (belt-and-suspenders):
- Add job="node-exporter" to all template variables and panel queries
in host-resources, network-deep-dive, storage-filesystem, capacity-planning
- Add Logs Drilldown link to logs-explorer dashboard
- Add Promtail journal relabels for level and service_name
Also: install Logs Drilldown plugin, update observability guide with
Logs Drilldown documentation and screenshots.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Promtail was not shipping logs due to missing /var/lib/promtail directory
and insufficient permissions (not in adm or systemd-journal groups).
Logs Explorer dashboard queries failed on Loki 3.x because allValue ".*"
is rejected as an empty matcher — changed to ".+" which requires at least
one character. Removed unit filter from main Logs query since varlog
streams lack the unit label.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace incorrect Prometheus queries across 4 dashboard JSON files:
- incus_uptime_seconds → incus_cpu_effective_total for template vars and counts
- incus_memory_RSS_bytes → incus_memory_Active_bytes (RSS only reports for VMs)
- Uptime panel → Processes panel using incus_procs_total
- Remove wrong instance filter from network deep dive instance panels
- Fix Loki data directory permissions in deploy-observability
Add viewport screenshots of all 9 Grafana dashboards and embed them in
the observability guide with captions and updated descriptions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
deploy-observability:
- Discover DNS from UPLINK config instead of hardcoding 8.8.8.8
- Set ipv4.address on OVN NICs (port security drops unknown IPs)
- Install ca-certificates/curl/gnupg before adding Grafana apt repo
- Fix ACL create syntax (--description flag, not key=value)
- Add NIC recovery after ACL changes (NICs go DOWN)
manage-dashboards:
- Rewrite read_manifest() without PyYAML dependency
- Fix ensure_folder() mixing debug output with return value
- Rewrite install payload generation with heredoc + file I/O
- Separate incus file push from Grafana API call
- Fix while-read loops consuming stdin from incus exec (fd3)
Dashboard JSON:
- Add allValue:".*" to all template variables with includeAll
Screenshots:
- Add Playwright screenshots of all 9 Grafana dashboards
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Define 5-color semantic palette (Okabe-Ito derived, color-blind safe)
- Use classDef for reusable styles instead of per-node statements
- Limit node text to 2 lines max, move specs to tables
- Consistent node shapes by role (rounded for network, stadium for external)
- Dashed arrows for control plane, solid for data plane
- Tinted subgraph fills matching dominant role color
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Test file with 6 diagrams converted from ASCII to mermaid format
to verify Gitea renders them correctly before updating all docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 2: captured Aether↔Incus API traffic using the event stream.
Identified callers by TLS certificate fingerprint (CLI vs Aether vs
cluster nodes). Documented API patterns for stop/start/snapshot/exec.
Key finding: Aether subscribes to lifecycle events and reactively
queries affected resources.
Deliverables:
- notes/api-interception-guide.md: capture setup + all API patterns
- incusos/helpers/incus-mitm: --capture/--live/--analyze/--callers
- .claude/rules/aether-api-payloads.md: context for AI assistants
- .mcp.json: add --ignore-https-errors for Playwright
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 of observability plan: mapped the complete OVN topology at
logical (NB), physical (SB), and OVS layers. Traced packet path from
LAN through VIP LB to HAProxy backends. Documented naming conventions,
gateway failover, Geneve tunnels, and MTU implications.
Deliverables:
- notes/ovn-deep-dive.md: comprehensive OVN internals guide
- incusos/helpers/ovn-inspect: --nb/--sb/--ovs/--full/--trace actions
- .claude/rules/ovn-internals.md: context for AI assistants
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tear down and fully re-deploy HAProxy infrastructure from scratch to
validate the guide reads correctly. Key fixes from re-validation:
- Fix session API field names: lb_vip (not vip), target_ip/target_port
(not ip/port), health_check_enabled (not health_check)
- Document that Aether does NOT create the OVN load balancer — must be
created manually via incus network load-balancer
- Fix internal VIP test: OVN LB doesn't hairpin, use localhost instead
- Document OVN LB has no health checking (HAProxy failover not instant)
- Add cleanup step for manually-created OVN LB
- Add aether-browser Playwright helper and .mcp.json config
- Add deploy-haproxy script, haproxy-lb rules, full guide
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AWX guide: fix wrong project ID (6→9), add dynamic ID lookups,
add missing Step 5 push commands, add migration pod to expected
output, replace PostgreSQL workaround with Aether UI instructions,
reference deploy-awx as automated path
- Aether guide: document JWT REST API (auth, clusters, ACLs),
swagger UI at /api/docs, rate limits, coverage vs UI-only features
- CLAUDE.md: add Aether API capability, add interaction hierarchy
(API first → UI fallback → never root/DB in guides)
- deploy-awx: fix PLAYBOOK_DIR from "ansible/playbooks" to "playbooks"
- post-deploy.yml: clarify cert push path vs EE runtime mount path
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Work-in-progress commit to enable AWX project sync from git.
Files will be updated with validated output from real deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aether deployed on oc-node-01, connected to Incus cluster (trust token)
and Operations Center (mutual TLS). Extracted all 16 built-in help pages
(426 KB) covering cluster management, storage, profiles, instances,
HAProxy LB, firewall ACLs, blueprints, and health monitoring.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
README.md: add networking, shared-storage, production-lab guides;
reorder notes in learning sequence.
CLAUDE.md: add production-lab, version-compat, iso-download to structure.
incusos/README.md: add 5 missing example configs, organize by category.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validated end-to-end on OC-managed 3-node cluster: iSCSI target
container (tgt), IncusOS iSCSI/LVM services, lvmcluster pool,
container migration (0.12s), VM live migration (~6s, RAM only).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full teardown and rebuild of the OC-managed cluster lab to validate
the guide works end-to-end with zero manual interventions. Key fixes:
- Add missing `provisioning channel add old-stable` step (was causing
"Not found" error on `assign-channels`)
- Fix command paths: `examples/` → `incusos/examples/` (repo-relative)
- Fix version numbers: Incus 6.12 → 6.21, OC output format
- Fix update list table format (Severity not Sev, add Published At note)
- Fix cluster list: remove premature ovn-chassis roles at Section 4.4
- Add WARNING line to `admin os show` output
- Add TLS cert acceptance note for `remote add`
- Add cleanup --yes flag to teardown commands
- Add test container cleanup command in Section 5.7
- Note variable-per-deployment values (incusbr0 subnet, token UUIDs)
- Add `provisioning channel add` to Quick Reference and CLI table
- Add channel creation to troubleshooting table
- Use consistent <token-UUID> placeholder throughout
All 16 verification checklist items PASS. Lab left running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy a 3-node OC-managed Incus cluster with OVN overlay networking
and an HA nginx workload behind an OVN load balancer. Key discoveries:
- OC needs_update blocker: OC tracks update delivery through its pipeline,
not version comparison. Nodes deployed from the latest ISO are stuck at
needs_update=true. Fix: generate ISO from older channel (--channel
old-stable), let OC push the real update after registration.
- ISO upload skip bug: incusos-proxmox silently reused stale ISOs with the
same filename. Fixed: --iso now replaces existing same-named ISOs.
- Empty app-seed-config ({}): avoids "Certificate already in trust store"
Terraform error when cert is already injected via SEED_DATA.
Files changed:
- notes/operations-center-guide.md: full rewrite with real captured output
- CLAUDE.md: needs_update findings, ISO fix, OVN LB behavior
- incusos/examples/lab-oc-deploy.yaml: static IP for VLAN 69
- incusos/examples/lab-oc-nodes.yaml: static IPs + disk sizing
- incusos/incusos-proxmox: fix stale ISO reuse when --iso provided
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive 1775-line guide (notes/production-lab-guide.md) covering
full lab lifecycle: OC dashboard, 3-node manual clustering, OVN overlay
networking, mixed workloads, live migration, network security, load
balancers, and cluster evacuation/restore. Every command was executed
on real infrastructure (2026-02-22) and output captured.
Key findings incorporated:
- IncusOS NIC is named `mgmt`, not `ens18` (UPLINK parent)
- OC CLI requires `remote switch`, not `remote:` suffix syntax
- LB/forward backends need IP addresses, not instance names
- Cluster auto-schedules without --target flag
- Migration ~7s at ~140 MB/s with heartbeat continuity
Also: lab-production.yaml config, lab-oc.yaml apply_defaults fix,
CLAUDE.md and networking-guide.md updates for mgmt/OC/LB findings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VLAN tagging: incusos-proxmox now reads optional `vlan` key from
proxmox.yaml and lab configs, appending `tag=N` to the VM NIC spec
in both SSH and API creation paths. Tested manually on VMID 850 —
VM received IP in 192.168.100.0/22 (Homelab VLAN). VLAN tag is
Proxmox-level only; IncusOS and Incus instances are unaware of it.
Networking guide: comprehensive tutorial covering bridge networking,
OVN overlay setup on IncusOS (services disabled by default, must
enable via REST API), cross-node connectivity, isolation, ACLs,
peering, load balancers, forwards, and DNS. All scenarios tested
on a 4-node OC-managed cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parallel investigation of IncusOS issue #843 across 20 VM deploys confirms
the "invalid crontab expression" error hits 67% of first boots. The bug is
a race condition in the daemon startup — NOT related to ISO/CDN version
mismatch, update downloads, or seed config. Sysext downloads occur on every
first boot even with matching versions.
New tools: investigate-parallel (parallel VM testing), pve-api (Python
Proxmox API helper avoiding bash ! quoting), investigate-boot (sequential
testing), test-boot-reliability (batch reliability testing).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Observational single-VM deploy tool (observe-deploy) captures frame-by-frame
console screenshots via Proxmox SSH root screendump to diagnose IncusOS boot
failures. Four deploys revealed ~50% "invalid crontab expression" hit rate
with no correlation to update.yaml presence.
Deep analysis of IncusOS source code (main.go, state/, scheduling/, install.go)
traced the full crash path: registerJobs() validates ScrubSchedule via gocron
at step 19 of startup(), after applications start at step 16 and REST API
starts even earlier. Most likely a race condition or gocron edge case.
Key fixes:
- incusos-proxmox: automatic retry on first-boot failure (stop+start cycle)
- incusos-proxmox: pve_destroy_vm now waits for async UPID task completion
- observe-deploy: same UPID wait for destroy, full screenshot pipeline
- incusos-seed: always generate update.yaml with explicit check_frequency
- CLAUDE.md: complete boot sequence timeline, source-level root cause analysis,
proposed upstream fix for IncusOS registerJobs() defensive validation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scripts now search for an 'env' file up the directory tree from the
script location and source it automatically. No manual 'source env'
needed. Falls back gracefully -- if no env file exists and the secret
isn't exported, require_api_auth() catches it with a clear error.
Also fix unbound variable in --labs when pool has no VMs (declare -A
needs explicit =() initialization under set -u).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Safety fix: --cleanup-all, --resources, --labs silently failed when
PROXMOX_TOKEN_SECRET was not set, reporting "No VMs found" instead of
an auth error. Added require_api_auth() gate and error propagation in
pve_list_vms() to fail loudly on API auth failures.
Also:
- Standardize role name to IncusOSDeployer across all docs and scripts
- Add Sys.Audit privilege to all setup command examples
- Document /nodes/pve ACL requirement for scoped permissions
- Update lab-cluster.yaml: apply_defaults: false for joining nodes
(upstream IncusOS recommendation to avoid 8-command cleanup)
- Add full cluster automation to lab-test phase_cluster:
core.https_address setup, remote cert fixup, automated join prompts,
conditional defaults cleanup for apply_defaults: true nodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Volume testing with live Proxmox infrastructure proved that guest reboot
does NOT break standalone IncusOS nodes. Tested:
- 4 consecutive Proxmox stop/start cycles: all clean, data intact
- Single-node guest reboot: recovered in ~50s, containers auto-started
- Simultaneous 3-node guest reboot: all recovered, data integrity verified
The "reboot breaks IncusOS" issue is OC-specific. Console errors showed:
- Node 1: "failed to parse update frequency" (OC pushed bad crontab)
- Node 2: stuck at "starting application" (waiting for peers in loops)
- Node 3: "constraint violation" (OC re-registering existing server)
Updated all 8 references across CLAUDE.md, operations-center-guide.md,
and TESTING.md to reflect the corrected diagnosis.
Also fixed one more bare return bug in do_list_labs (empty pool path).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and
lab-replace.yaml (single replacement node) for advanced migration and
cluster lifecycle testing
- New notes/clustering-guide.md with full clustering walkthrough: cluster
formation, storage pool conflict fix, join workflow, VM live migration
with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs),
edge cases (I/O, memory, concurrent), and node replacement lifecycle
- Updated CLAUDE.md with clustering docs, rebalancing config keys,
cluster enable syntax fix (TWO args not one), cluster remove syntax,
agent reconnect timing (3-4s not 1-2s), and lifecycle procedure
- proxmox.yaml.example for Proxmox connection config reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the first wave of lab validation infrastructure:
incusos-proxmox enhancements:
- --doctor flag: standalone environment health check (tool versions,
IncusOS CDN availability, Proxmox connectivity). Works with or
without a config file.
- --cleanup --deep: extended cleanup that also removes seed ISOs,
IncusOS ISOs from Proxmox storage, matching incus remotes, and
the local ISO cache (~/.cache/incusos-proxmox/iso-cache/).
- --cleanup-all: pool-wide cleanup that destroys ALL VMs with the
[incusos-lab:managed] marker in the configured resource pool.
Requires a pool to be set in config for safety scoping.
New script -- lab-test:
- Guided lab validation with 5 phases: deploy & verify, single-node
workloads (container + VM lifecycle), cluster formation (enable +
join via incus remotes), cluster workloads (scheduler-placed +
targeted), and migration & evacuation (stop/move + evacuate/restore).
- Interactive by default with --yes for unattended runs.
- Reports PASS/FAIL/SKIP per test with a summary.
- Supports --dry-run, --skip-deploy, --cleanup, per-phase execution.
- Reads the same YAML config format as incusos-proxmox.
New example configs:
- lab-cluster.yaml: 3-node Incus cluster (VMID 900+) for testing
clustering, workloads, and migration.
- lab-oc.yaml: Operations Center + 3 Incus nodes (VMID 910+) for
testing OC-managed provisioning and cluster formation.
Documentation:
- CLAUDE.md: added clustering workflow (incus cluster enable/add/join
via remotes), lab-test overview, Operations Center CLI notes,
doctor and cleanup feature descriptions.
- README.md: added lab-test section with usage/phases/options, doctor
check docs, deep cleanup docs, new example files in listing.
- TESTING.md: added full lab validation guide (quick start, phase-by-
phase testing, manual cluster formation alternative, OC lab workflow
with CLI setup and provisioning steps), cleanup instructions.
Not yet tested against live infrastructure -- next step is to deploy
lab-cluster.yaml, add incus remotes, and run lab-test end-to-end.
Known areas to validate: cluster join syntax with remotes, VM nested
virt support, evacuation behavior, apply_defaults consistency across
cluster members.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Major changes to incusos-proxmox and incusos-seed to achieve fully automated
IncusOS deployment to Proxmox VE. Fixes multiple blockers discovered during
testing: FAT→ISO 9660 seed format (CD-ROM label detection), preseed YAML
structure (Go yaml:",inline"), disk target auto-detection, install completion
monitoring via blockstat disk I/O, install media ejection, and ARP-based IP
detection for the agent-less IncusOS.
Key changes:
- incusos-seed: add --format iso using genisoimage, fix preseed structure
- incusos-proxmox: ISO seeds, blockstat install detection, media ejection,
ARP-based IP lookup, force_reboot requirement
- Documentation: comprehensive updates to CLAUDE.md, README.md, TESTING.md
- Examples: fix preseed.certificates path in all YAML examples
- Notes: full session progress with all bugs found and fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Maarten