Phase 2: captured Aether↔Incus API traffic using the event stream.
Identified callers by TLS certificate fingerprint (CLI vs Aether vs
cluster nodes). Documented API patterns for stop/start/snapshot/exec.
Key finding: Aether subscribes to lifecycle events and reactively
queries affected resources.
Deliverables:
- notes/api-interception-guide.md: capture setup + all API patterns
- incusos/helpers/incus-mitm: --capture/--live/--analyze/--callers
- .claude/rules/aether-api-payloads.md: context for AI assistants
- .mcp.json: add --ignore-https-errors for Playwright
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 of observability plan: mapped the complete OVN topology at
logical (NB), physical (SB), and OVS layers. Traced packet path from
LAN through VIP LB to HAProxy backends. Documented naming conventions,
gateway failover, Geneve tunnels, and MTU implications.
Deliverables:
- notes/ovn-deep-dive.md: comprehensive OVN internals guide
- incusos/helpers/ovn-inspect: --nb/--sb/--ovs/--full/--trace actions
- .claude/rules/ovn-internals.md: context for AI assistants
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tear down and fully re-deploy HAProxy infrastructure from scratch to
validate the guide reads correctly. Key fixes from re-validation:
- Fix session API field names: lb_vip (not vip), target_ip/target_port
(not ip/port), health_check_enabled (not health_check)
- Document that Aether does NOT create the OVN load balancer — must be
created manually via incus network load-balancer
- Fix internal VIP test: OVN LB doesn't hairpin, use localhost instead
- Document OVN LB has no health checking (HAProxy failover not instant)
- Add cleanup step for manually-created OVN LB
- Add aether-browser Playwright helper and .mcp.json config
- Add deploy-haproxy script, haproxy-lb rules, full guide
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AWX guide: fix wrong project ID (6→9), add dynamic ID lookups,
add missing Step 5 push commands, add migration pod to expected
output, replace PostgreSQL workaround with Aether UI instructions,
reference deploy-awx as automated path
- Aether guide: document JWT REST API (auth, clusters, ACLs),
swagger UI at /api/docs, rate limits, coverage vs UI-only features
- CLAUDE.md: add Aether API capability, add interaction hierarchy
(API first → UI fallback → never root/DB in guides)
- deploy-awx: fix PLAYBOOK_DIR from "ansible/playbooks" to "playbooks"
- post-deploy.yml: clarify cert push path vs EE runtime mount path
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Work-in-progress commit to enable AWX project sync from git.
Files will be updated with validated output from real deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aether deployed on oc-node-01, connected to Incus cluster (trust token)
and Operations Center (mutual TLS). Extracted all 16 built-in help pages
(426 KB) covering cluster management, storage, profiles, instances,
HAProxy LB, firewall ACLs, blueprints, and health monitoring.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validated end-to-end on OC-managed 3-node cluster: iSCSI target
container (tgt), IncusOS iSCSI/LVM services, lvmcluster pool,
container migration (0.12s), VM live migration (~6s, RAM only).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full teardown and rebuild of the OC-managed cluster lab to validate
the guide works end-to-end with zero manual interventions. Key fixes:
- Add missing `provisioning channel add old-stable` step (was causing
"Not found" error on `assign-channels`)
- Fix command paths: `examples/` → `incusos/examples/` (repo-relative)
- Fix version numbers: Incus 6.12 → 6.21, OC output format
- Fix update list table format (Severity not Sev, add Published At note)
- Fix cluster list: remove premature ovn-chassis roles at Section 4.4
- Add WARNING line to `admin os show` output
- Add TLS cert acceptance note for `remote add`
- Add cleanup --yes flag to teardown commands
- Add test container cleanup command in Section 5.7
- Note variable-per-deployment values (incusbr0 subnet, token UUIDs)
- Add `provisioning channel add` to Quick Reference and CLI table
- Add channel creation to troubleshooting table
- Use consistent <token-UUID> placeholder throughout
All 16 verification checklist items PASS. Lab left running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy a 3-node OC-managed Incus cluster with OVN overlay networking
and an HA nginx workload behind an OVN load balancer. Key discoveries:
- OC needs_update blocker: OC tracks update delivery through its pipeline,
not version comparison. Nodes deployed from the latest ISO are stuck at
needs_update=true. Fix: generate ISO from older channel (--channel
old-stable), let OC push the real update after registration.
- ISO upload skip bug: incusos-proxmox silently reused stale ISOs with the
same filename. Fixed: --iso now replaces existing same-named ISOs.
- Empty app-seed-config ({}): avoids "Certificate already in trust store"
Terraform error when cert is already injected via SEED_DATA.
Files changed:
- notes/operations-center-guide.md: full rewrite with real captured output
- CLAUDE.md: needs_update findings, ISO fix, OVN LB behavior
- incusos/examples/lab-oc-deploy.yaml: static IP for VLAN 69
- incusos/examples/lab-oc-nodes.yaml: static IPs + disk sizing
- incusos/incusos-proxmox: fix stale ISO reuse when --iso provided
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive 1775-line guide (notes/production-lab-guide.md) covering
full lab lifecycle: OC dashboard, 3-node manual clustering, OVN overlay
networking, mixed workloads, live migration, network security, load
balancers, and cluster evacuation/restore. Every command was executed
on real infrastructure (2026-02-22) and output captured.
Key findings incorporated:
- IncusOS NIC is named `mgmt`, not `ens18` (UPLINK parent)
- OC CLI requires `remote switch`, not `remote:` suffix syntax
- LB/forward backends need IP addresses, not instance names
- Cluster auto-schedules without --target flag
- Migration ~7s at ~140 MB/s with heartbeat continuity
Also: lab-production.yaml config, lab-oc.yaml apply_defaults fix,
CLAUDE.md and networking-guide.md updates for mgmt/OC/LB findings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VLAN tagging: incusos-proxmox now reads optional `vlan` key from
proxmox.yaml and lab configs, appending `tag=N` to the VM NIC spec
in both SSH and API creation paths. Tested manually on VMID 850 —
VM received IP in 192.168.100.0/22 (Homelab VLAN). VLAN tag is
Proxmox-level only; IncusOS and Incus instances are unaware of it.
Networking guide: comprehensive tutorial covering bridge networking,
OVN overlay setup on IncusOS (services disabled by default, must
enable via REST API), cross-node connectivity, isolation, ACLs,
peering, load balancers, forwards, and DNS. All scenarios tested
on a 4-node OC-managed cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parallel investigation of IncusOS issue #843 across 20 VM deploys confirms
the "invalid crontab expression" error hits 67% of first boots. The bug is
a race condition in the daemon startup — NOT related to ISO/CDN version
mismatch, update downloads, or seed config. Sysext downloads occur on every
first boot even with matching versions.
New tools: investigate-parallel (parallel VM testing), pve-api (Python
Proxmox API helper avoiding bash ! quoting), investigate-boot (sequential
testing), test-boot-reliability (batch reliability testing).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Safety fix: --cleanup-all, --resources, --labs silently failed when
PROXMOX_TOKEN_SECRET was not set, reporting "No VMs found" instead of
an auth error. Added require_api_auth() gate and error propagation in
pve_list_vms() to fail loudly on API auth failures.
Also:
- Standardize role name to IncusOSDeployer across all docs and scripts
- Add Sys.Audit privilege to all setup command examples
- Document /nodes/pve ACL requirement for scoped permissions
- Update lab-cluster.yaml: apply_defaults: false for joining nodes
(upstream IncusOS recommendation to avoid 8-command cleanup)
- Add full cluster automation to lab-test phase_cluster:
core.https_address setup, remote cert fixup, automated join prompts,
conditional defaults cleanup for apply_defaults: true nodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Volume testing with live Proxmox infrastructure proved that guest reboot
does NOT break standalone IncusOS nodes. Tested:
- 4 consecutive Proxmox stop/start cycles: all clean, data intact
- Single-node guest reboot: recovered in ~50s, containers auto-started
- Simultaneous 3-node guest reboot: all recovered, data integrity verified
The "reboot breaks IncusOS" issue is OC-specific. Console errors showed:
- Node 1: "failed to parse update frequency" (OC pushed bad crontab)
- Node 2: stuck at "starting application" (waiting for peers in loops)
- Node 3: "constraint violation" (OC re-registering existing server)
Updated all 8 references across CLAUDE.md, operations-center-guide.md,
and TESTING.md to reflect the corrected diagnosis.
Also fixed one more bare return bug in do_list_labs (empty pool path).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and
lab-replace.yaml (single replacement node) for advanced migration and
cluster lifecycle testing
- New notes/clustering-guide.md with full clustering walkthrough: cluster
formation, storage pool conflict fix, join workflow, VM live migration
with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs),
edge cases (I/O, memory, concurrent), and node replacement lifecycle
- Updated CLAUDE.md with clustering docs, rebalancing config keys,
cluster enable syntax fix (TWO args not one), cluster remove syntax,
agent reconnect timing (3-4s not 1-2s), and lifecycle procedure
- proxmox.yaml.example for Proxmox connection config reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Major changes to incusos-proxmox and incusos-seed to achieve fully automated
IncusOS deployment to Proxmox VE. Fixes multiple blockers discovered during
testing: FAT→ISO 9660 seed format (CD-ROM label detection), preseed YAML
structure (Go yaml:",inline"), disk target auto-detection, install completion
monitoring via blockstat disk I/O, install media ejection, and ARP-based IP
detection for the agent-less IncusOS.
Key changes:
- incusos-seed: add --format iso using genisoimage, fix preseed structure
- incusos-proxmox: ISO seeds, blockstat install detection, media ejection,
ARP-based IP lookup, force_reboot requirement
- Documentation: comprehensive updates to CLAUDE.md, README.md, TESTING.md
- Examples: fix preseed.certificates path in all YAML examples
- Notes: full session progress with all bugs found and fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Maarten