IncusOS Immutability & Break-Fix Context

Architecture

IncusOS is an immutable OS with A/B partition scheme for safe updates.
Version format: YYYYMMDDHHMI (build timestamp, e.g., 202602240349).
TPM-based full disk encryption: root + swap volumes unlocked via TPM measured boot. Secure Boot enforced with PK, KEK, and 2x db certs.
ZFS storage pool "local" on partition 11, contains volume "incus".
Single SCSI disk per node (64-100 GiB in lab, QEMU HARDDISK).
MAC-dependent boot: udev renames NIC by MAC. Changing MAC = boot hang.

Node	VMID	IP	Version
node-01	400	192.168.102.140	202602240349
node-02	401	192.168.102.141	202602240349
node-03	402	192.168.102.142	202602240349

All on port 8443, authenticated with Incus client certificate:

/os/1.0 -- version, hostname, basic info
/os/1.0/system/security -- TPM, Secure Boot, encryption/pool recovery keys
/os/1.0/system/storage -- disks, partitions, ZFS pools
/os/1.0/system/resources -- CPU, memory, hardware
/os/1.0/system/network -- interfaces, DNS, routes
/os/1.0/system/update -- GET: state; PUT: change config (frequency, auto_reboot)
/os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}

incusos/helpers/incusos-health queries the IncusOS API across all cluster nodes. Actions:

Flag	Purpose
`--status`	Version, hostname, TPM status, Secure Boot
`--partitions`	Disk/partition layout, A/B state
`--tpm`	TPM details, Secure Boot certs, encryption keys
`--services`	Enabled/disabled services
`--network`	Interface config, DNS, gateway
`--update`	Channel, frequency, pending updates
`--all`	All of the above

NEVER hard-stop during first boot -- corrupts TPM permanently, requires full reinstallation. Hard-stop is safe ONLY for already- provisioned nodes (TPM state already sealed).
One node at a time for break-fix exercises. 3-node cluster needs 2/3 quorum. Taking down 2 nodes = cluster failure.
Proxmox snapshots before every destructive test (VMID 400-402). Note: API token may lack VM.Snapshot permission.
Target node-03 (VMID 402) for destructive tests -- not the leader, fewest workloads.
Verify health with incusos-health --all before and after exercises.
Monitor via Grafana during all exercises.
Never change MAC addresses on IncusOS VMs. Use --data-urlencode when modifying NIC config via Proxmox API.

os_version = current running version.
os_version_next = version on standby partition (same as current if no update pending, different if update downloaded).
needs_reboot = true when update is downloaded and waiting for reboot.
auto_reboot: false in lab -- updates download but don't reboot.
Check frequency configurable via PUT. Default may be "never" on some nodes.
POST to trigger immediate check returns 501 (not implemented).
Boot-time update check: IncusOS checks for updates during boot sequence.
Nodes update independently (rolling, not coordinated).

Normal Update Observation -- Completed. Observed download, reboot, A/B switch on node-02. Database-leader transfer seamless during reboot.
Simulated Failed Update -- Blocked (no update available to interrupt).
Network Isolation -- Completed. OVN gateway is SPOF (no failover). East-west traffic works, north-south broken. Recovery: 26s cluster, 75s gateway.
Full Node Failure -- Completed. 2/3 quorum held. Detection: ~20s. Recovery: ~50s to cluster rejoin. All instances auto-restarted.

See notes/incusos-break-fix.md for full exercise details and results.