- AWX guide: fix wrong project ID (6→9), add dynamic ID lookups,
add missing Step 5 push commands, add migration pod to expected
output, replace PostgreSQL workaround with Aether UI instructions,
reference deploy-awx as automated path
- Aether guide: document JWT REST API (auth, clusters, ACLs),
swagger UI at /api/docs, rate limits, coverage vs UI-only features
- CLAUDE.md: add Aether API capability, add interaction hierarchy
(API first → UI fallback → never root/DB in guides)
- deploy-awx: fix PLAYBOOK_DIR from "ansible/playbooks" to "playbooks"
- post-deploy.yml: clarify cert push path vs EE runtime mount path
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Work-in-progress commit to enable AWX project sync from git.
Files will be updated with validated output from real deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aether deployed on oc-node-01, connected to Incus cluster (trust token)
and Operations Center (mutual TLS). Extracted all 16 built-in help pages
(426 KB) covering cluster management, storage, profiles, instances,
HAProxy LB, firewall ACLs, blueprints, and health monitoring.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validated end-to-end on OC-managed 3-node cluster: iSCSI target
container (tgt), IncusOS iSCSI/LVM services, lvmcluster pool,
container migration (0.12s), VM live migration (~6s, RAM only).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full teardown and rebuild of the OC-managed cluster lab to validate
the guide works end-to-end with zero manual interventions. Key fixes:
- Add missing `provisioning channel add old-stable` step (was causing
"Not found" error on `assign-channels`)
- Fix command paths: `examples/` → `incusos/examples/` (repo-relative)
- Fix version numbers: Incus 6.12 → 6.21, OC output format
- Fix update list table format (Severity not Sev, add Published At note)
- Fix cluster list: remove premature ovn-chassis roles at Section 4.4
- Add WARNING line to `admin os show` output
- Add TLS cert acceptance note for `remote add`
- Add cleanup --yes flag to teardown commands
- Add test container cleanup command in Section 5.7
- Note variable-per-deployment values (incusbr0 subnet, token UUIDs)
- Add `provisioning channel add` to Quick Reference and CLI table
- Add channel creation to troubleshooting table
- Use consistent <token-UUID> placeholder throughout
All 16 verification checklist items PASS. Lab left running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy a 3-node OC-managed Incus cluster with OVN overlay networking
and an HA nginx workload behind an OVN load balancer. Key discoveries:
- OC needs_update blocker: OC tracks update delivery through its pipeline,
not version comparison. Nodes deployed from the latest ISO are stuck at
needs_update=true. Fix: generate ISO from older channel (--channel
old-stable), let OC push the real update after registration.
- ISO upload skip bug: incusos-proxmox silently reused stale ISOs with the
same filename. Fixed: --iso now replaces existing same-named ISOs.
- Empty app-seed-config ({}): avoids "Certificate already in trust store"
Terraform error when cert is already injected via SEED_DATA.
Files changed:
- notes/operations-center-guide.md: full rewrite with real captured output
- CLAUDE.md: needs_update findings, ISO fix, OVN LB behavior
- incusos/examples/lab-oc-deploy.yaml: static IP for VLAN 69
- incusos/examples/lab-oc-nodes.yaml: static IPs + disk sizing
- incusos/incusos-proxmox: fix stale ISO reuse when --iso provided
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive 1775-line guide (notes/production-lab-guide.md) covering
full lab lifecycle: OC dashboard, 3-node manual clustering, OVN overlay
networking, mixed workloads, live migration, network security, load
balancers, and cluster evacuation/restore. Every command was executed
on real infrastructure (2026-02-22) and output captured.
Key findings incorporated:
- IncusOS NIC is named `mgmt`, not `ens18` (UPLINK parent)
- OC CLI requires `remote switch`, not `remote:` suffix syntax
- LB/forward backends need IP addresses, not instance names
- Cluster auto-schedules without --target flag
- Migration ~7s at ~140 MB/s with heartbeat continuity
Also: lab-production.yaml config, lab-oc.yaml apply_defaults fix,
CLAUDE.md and networking-guide.md updates for mgmt/OC/LB findings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VLAN tagging: incusos-proxmox now reads optional `vlan` key from
proxmox.yaml and lab configs, appending `tag=N` to the VM NIC spec
in both SSH and API creation paths. Tested manually on VMID 850 —
VM received IP in 192.168.100.0/22 (Homelab VLAN). VLAN tag is
Proxmox-level only; IncusOS and Incus instances are unaware of it.
Networking guide: comprehensive tutorial covering bridge networking,
OVN overlay setup on IncusOS (services disabled by default, must
enable via REST API), cross-node connectivity, isolation, ACLs,
peering, load balancers, forwards, and DNS. All scenarios tested
on a 4-node OC-managed cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parallel investigation of IncusOS issue #843 across 20 VM deploys confirms
the "invalid crontab expression" error hits 67% of first boots. The bug is
a race condition in the daemon startup — NOT related to ISO/CDN version
mismatch, update downloads, or seed config. Sysext downloads occur on every
first boot even with matching versions.
New tools: investigate-parallel (parallel VM testing), pve-api (Python
Proxmox API helper avoiding bash ! quoting), investigate-boot (sequential
testing), test-boot-reliability (batch reliability testing).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Safety fix: --cleanup-all, --resources, --labs silently failed when
PROXMOX_TOKEN_SECRET was not set, reporting "No VMs found" instead of
an auth error. Added require_api_auth() gate and error propagation in
pve_list_vms() to fail loudly on API auth failures.
Also:
- Standardize role name to IncusOSDeployer across all docs and scripts
- Add Sys.Audit privilege to all setup command examples
- Document /nodes/pve ACL requirement for scoped permissions
- Update lab-cluster.yaml: apply_defaults: false for joining nodes
(upstream IncusOS recommendation to avoid 8-command cleanup)
- Add full cluster automation to lab-test phase_cluster:
core.https_address setup, remote cert fixup, automated join prompts,
conditional defaults cleanup for apply_defaults: true nodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Volume testing with live Proxmox infrastructure proved that guest reboot
does NOT break standalone IncusOS nodes. Tested:
- 4 consecutive Proxmox stop/start cycles: all clean, data intact
- Single-node guest reboot: recovered in ~50s, containers auto-started
- Simultaneous 3-node guest reboot: all recovered, data integrity verified
The "reboot breaks IncusOS" issue is OC-specific. Console errors showed:
- Node 1: "failed to parse update frequency" (OC pushed bad crontab)
- Node 2: stuck at "starting application" (waiting for peers in loops)
- Node 3: "constraint violation" (OC re-registering existing server)
Updated all 8 references across CLAUDE.md, operations-center-guide.md,
and TESTING.md to reflect the corrected diagnosis.
Also fixed one more bare return bug in do_list_labs (empty pool path).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and
lab-replace.yaml (single replacement node) for advanced migration and
cluster lifecycle testing
- New notes/clustering-guide.md with full clustering walkthrough: cluster
formation, storage pool conflict fix, join workflow, VM live migration
with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs),
edge cases (I/O, memory, concurrent), and node replacement lifecycle
- Updated CLAUDE.md with clustering docs, rebalancing config keys,
cluster enable syntax fix (TWO args not one), cluster remove syntax,
agent reconnect timing (3-4s not 1-2s), and lifecycle procedure
- proxmox.yaml.example for Proxmox connection config reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Major changes to incusos-proxmox and incusos-seed to achieve fully automated
IncusOS deployment to Proxmox VE. Fixes multiple blockers discovered during
testing: FAT→ISO 9660 seed format (CD-ROM label detection), preseed YAML
structure (Go yaml:",inline"), disk target auto-detection, install completion
monitoring via blockstat disk I/O, install media ejection, and ARP-based IP
detection for the agent-less IncusOS.
Key changes:
- incusos-seed: add --format iso using genisoimage, fix preseed structure
- incusos-proxmox: ISO seeds, blockstat install detection, media ejection,
ARP-based IP lookup, force_reboot requirement
- Documentation: comprehensive updates to CLAUDE.md, README.md, TESTING.md
- Examples: fix preseed.certificates path in all YAML examples
- Notes: full session progress with all bugs found and fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Maarten