Phase 1 of observability plan: mapped the complete OVN topology at
logical (NB), physical (SB), and OVS layers. Traced packet path from
LAN through VIP LB to HAProxy backends. Documented naming conventions,
gateway failover, Geneve tunnels, and MTU implications.
Deliverables:
- notes/ovn-deep-dive.md: comprehensive OVN internals guide
- incusos/helpers/ovn-inspect: --nb/--sb/--ovs/--full/--trace actions
- .claude/rules/ovn-internals.md: context for AI assistants
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tear down and fully re-deploy HAProxy infrastructure from scratch to
validate the guide reads correctly. Key fixes from re-validation:
- Fix session API field names: lb_vip (not vip), target_ip/target_port
(not ip/port), health_check_enabled (not health_check)
- Document that Aether does NOT create the OVN load balancer — must be
created manually via incus network load-balancer
- Fix internal VIP test: OVN LB doesn't hairpin, use localhost instead
- Document OVN LB has no health checking (HAProxy failover not instant)
- Add cleanup step for manually-created OVN LB
- Add aether-browser Playwright helper and .mcp.json config
- Add deploy-haproxy script, haproxy-lb rules, full guide
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AWX guide: fix wrong project ID (6→9), add dynamic ID lookups,
add missing Step 5 push commands, add migration pod to expected
output, replace PostgreSQL workaround with Aether UI instructions,
reference deploy-awx as automated path
- Aether guide: document JWT REST API (auth, clusters, ACLs),
swagger UI at /api/docs, rate limits, coverage vs UI-only features
- CLAUDE.md: add Aether API capability, add interaction hierarchy
(API first → UI fallback → never root/DB in guides)
- deploy-awx: fix PLAYBOOK_DIR from "ansible/playbooks" to "playbooks"
- post-deploy.yml: clarify cert push path vs EE runtime mount path
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Work-in-progress commit to enable AWX project sync from git.
Files will be updated with validated output from real deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
README.md: add networking, shared-storage, production-lab guides;
reorder notes in learning sequence.
CLAUDE.md: add production-lab, version-compat, iso-download to structure.
incusos/README.md: add 5 missing example configs, organize by category.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Validated end-to-end on OC-managed 3-node cluster: iSCSI target
container (tgt), IncusOS iSCSI/LVM services, lvmcluster pool,
container migration (0.12s), VM live migration (~6s, RAM only).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy a 3-node OC-managed Incus cluster with OVN overlay networking
and an HA nginx workload behind an OVN load balancer. Key discoveries:
- OC needs_update blocker: OC tracks update delivery through its pipeline,
not version comparison. Nodes deployed from the latest ISO are stuck at
needs_update=true. Fix: generate ISO from older channel (--channel
old-stable), let OC push the real update after registration.
- ISO upload skip bug: incusos-proxmox silently reused stale ISOs with the
same filename. Fixed: --iso now replaces existing same-named ISOs.
- Empty app-seed-config ({}): avoids "Certificate already in trust store"
Terraform error when cert is already injected via SEED_DATA.
Files changed:
- notes/operations-center-guide.md: full rewrite with real captured output
- CLAUDE.md: needs_update findings, ISO fix, OVN LB behavior
- incusos/examples/lab-oc-deploy.yaml: static IP for VLAN 69
- incusos/examples/lab-oc-nodes.yaml: static IPs + disk sizing
- incusos/incusos-proxmox: fix stale ISO reuse when --iso provided
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive 1775-line guide (notes/production-lab-guide.md) covering
full lab lifecycle: OC dashboard, 3-node manual clustering, OVN overlay
networking, mixed workloads, live migration, network security, load
balancers, and cluster evacuation/restore. Every command was executed
on real infrastructure (2026-02-22) and output captured.
Key findings incorporated:
- IncusOS NIC is named `mgmt`, not `ens18` (UPLINK parent)
- OC CLI requires `remote switch`, not `remote:` suffix syntax
- LB/forward backends need IP addresses, not instance names
- Cluster auto-schedules without --target flag
- Migration ~7s at ~140 MB/s with heartbeat continuity
Also: lab-production.yaml config, lab-oc.yaml apply_defaults fix,
CLAUDE.md and networking-guide.md updates for mgmt/OC/LB findings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VLAN tagging: incusos-proxmox now reads optional `vlan` key from
proxmox.yaml and lab configs, appending `tag=N` to the VM NIC spec
in both SSH and API creation paths. Tested manually on VMID 850 —
VM received IP in 192.168.100.0/22 (Homelab VLAN). VLAN tag is
Proxmox-level only; IncusOS and Incus instances are unaware of it.
Networking guide: comprehensive tutorial covering bridge networking,
OVN overlay setup on IncusOS (services disabled by default, must
enable via REST API), cross-node connectivity, isolation, ACLs,
peering, load balancers, forwards, and DNS. All scenarios tested
on a 4-node OC-managed cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parallel investigation of IncusOS issue #843 across 20 VM deploys confirms
the "invalid crontab expression" error hits 67% of first boots. The bug is
a race condition in the daemon startup — NOT related to ISO/CDN version
mismatch, update downloads, or seed config. Sysext downloads occur on every
first boot even with matching versions.
New tools: investigate-parallel (parallel VM testing), pve-api (Python
Proxmox API helper avoiding bash ! quoting), investigate-boot (sequential
testing), test-boot-reliability (batch reliability testing).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Observational single-VM deploy tool (observe-deploy) captures frame-by-frame
console screenshots via Proxmox SSH root screendump to diagnose IncusOS boot
failures. Four deploys revealed ~50% "invalid crontab expression" hit rate
with no correlation to update.yaml presence.
Deep analysis of IncusOS source code (main.go, state/, scheduling/, install.go)
traced the full crash path: registerJobs() validates ScrubSchedule via gocron
at step 19 of startup(), after applications start at step 16 and REST API
starts even earlier. Most likely a race condition or gocron edge case.
Key fixes:
- incusos-proxmox: automatic retry on first-boot failure (stop+start cycle)
- incusos-proxmox: pve_destroy_vm now waits for async UPID task completion
- observe-deploy: same UPID wait for destroy, full screenshot pipeline
- incusos-seed: always generate update.yaml with explicit check_frequency
- CLAUDE.md: complete boot sequence timeline, source-level root cause analysis,
proposed upstream fix for IncusOS registerJobs() defensive validation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scripts now search for an 'env' file up the directory tree from the
script location and source it automatically. No manual 'source env'
needed. Falls back gracefully -- if no env file exists and the secret
isn't exported, require_api_auth() catches it with a clear error.
Also fix unbound variable in --labs when pool has no VMs (declare -A
needs explicit =() initialization under set -u).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Safety fix: --cleanup-all, --resources, --labs silently failed when
PROXMOX_TOKEN_SECRET was not set, reporting "No VMs found" instead of
an auth error. Added require_api_auth() gate and error propagation in
pve_list_vms() to fail loudly on API auth failures.
Also:
- Standardize role name to IncusOSDeployer across all docs and scripts
- Add Sys.Audit privilege to all setup command examples
- Document /nodes/pve ACL requirement for scoped permissions
- Update lab-cluster.yaml: apply_defaults: false for joining nodes
(upstream IncusOS recommendation to avoid 8-command cleanup)
- Add full cluster automation to lab-test phase_cluster:
core.https_address setup, remote cert fixup, automated join prompts,
conditional defaults cleanup for apply_defaults: true nodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Volume testing with live Proxmox infrastructure proved that guest reboot
does NOT break standalone IncusOS nodes. Tested:
- 4 consecutive Proxmox stop/start cycles: all clean, data intact
- Single-node guest reboot: recovered in ~50s, containers auto-started
- Simultaneous 3-node guest reboot: all recovered, data integrity verified
The "reboot breaks IncusOS" issue is OC-specific. Console errors showed:
- Node 1: "failed to parse update frequency" (OC pushed bad crontab)
- Node 2: stuck at "starting application" (waiting for peers in loops)
- Node 3: "constraint violation" (OC re-registering existing server)
Updated all 8 references across CLAUDE.md, operations-center-guide.md,
and TESTING.md to reflect the corrected diagnosis.
Also fixed one more bare return bug in do_list_labs (empty pool path).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and
lab-replace.yaml (single replacement node) for advanced migration and
cluster lifecycle testing
- New notes/clustering-guide.md with full clustering walkthrough: cluster
formation, storage pool conflict fix, join workflow, VM live migration
with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs),
edge cases (I/O, memory, concurrent), and node replacement lifecycle
- Updated CLAUDE.md with clustering docs, rebalancing config keys,
cluster enable syntax fix (TWO args not one), cluster remove syntax,
agent reconnect timing (3-4s not 1-2s), and lifecycle procedure
- proxmox.yaml.example for Proxmox connection config reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the first wave of lab validation infrastructure:
incusos-proxmox enhancements:
- --doctor flag: standalone environment health check (tool versions,
IncusOS CDN availability, Proxmox connectivity). Works with or
without a config file.
- --cleanup --deep: extended cleanup that also removes seed ISOs,
IncusOS ISOs from Proxmox storage, matching incus remotes, and
the local ISO cache (~/.cache/incusos-proxmox/iso-cache/).
- --cleanup-all: pool-wide cleanup that destroys ALL VMs with the
[incusos-lab:managed] marker in the configured resource pool.
Requires a pool to be set in config for safety scoping.
New script -- lab-test:
- Guided lab validation with 5 phases: deploy & verify, single-node
workloads (container + VM lifecycle), cluster formation (enable +
join via incus remotes), cluster workloads (scheduler-placed +
targeted), and migration & evacuation (stop/move + evacuate/restore).
- Interactive by default with --yes for unattended runs.
- Reports PASS/FAIL/SKIP per test with a summary.
- Supports --dry-run, --skip-deploy, --cleanup, per-phase execution.
- Reads the same YAML config format as incusos-proxmox.
New example configs:
- lab-cluster.yaml: 3-node Incus cluster (VMID 900+) for testing
clustering, workloads, and migration.
- lab-oc.yaml: Operations Center + 3 Incus nodes (VMID 910+) for
testing OC-managed provisioning and cluster formation.
Documentation:
- CLAUDE.md: added clustering workflow (incus cluster enable/add/join
via remotes), lab-test overview, Operations Center CLI notes,
doctor and cleanup feature descriptions.
- README.md: added lab-test section with usage/phases/options, doctor
check docs, deep cleanup docs, new example files in listing.
- TESTING.md: added full lab validation guide (quick start, phase-by-
phase testing, manual cluster formation alternative, OC lab workflow
with CLI setup and provisioning steps), cleanup instructions.
Not yet tested against live infrastructure -- next step is to deploy
lab-cluster.yaml, add incus remotes, and run lab-test end-to-end.
Known areas to validate: cluster join syntax with remotes, VM nested
virt support, evacuation behavior, apply_defaults consistency across
cluster members.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Major changes to incusos-proxmox and incusos-seed to achieve fully automated
IncusOS deployment to Proxmox VE. Fixes multiple blockers discovered during
testing: FAT→ISO 9660 seed format (CD-ROM label detection), preseed YAML
structure (Go yaml:",inline"), disk target auto-detection, install completion
monitoring via blockstat disk I/O, install media ejection, and ARP-based IP
detection for the agent-less IncusOS.
Key changes:
- incusos-seed: add --format iso using genisoimage, fix preseed structure
- incusos-proxmox: ISO seeds, blockstat install detection, media ejection,
ARP-based IP lookup, force_reboot requirement
- Documentation: comprehensive updates to CLAUDE.md, README.md, TESTING.md
- Examples: fix preseed.certificates path in all YAML examples
- Notes: full session progress with all bugs found and fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Maarten