incus-contrib/.claude/rules/incusos-immutability.md

paths
incusos/helpers/incusos-health notes/incusos-break-fix.md

IncusOS Immutability & Break-Fix Context

Architecture

• IncusOS is an immutable OS with A/B partition scheme for safe updates.
• Version format: YYYYMMDDHHMI (build timestamp, e.g., 202602240349).
• TPM-based full disk encryption: root + swap volumes unlocked via TPM measured boot. Secure Boot enforced with PK, KEK, and 2x db certs.
• ZFS storage pool "local" on partition 11, contains volume "incus".
• Single SCSI disk per node (64 GiB in lab, QEMU HARDDISK).

Lab cluster nodes

Node	VMID	IP	Version
node-01	900	192.168.102.140	202602240349
node-02	901	192.168.102.141	202602230420
node-03	902	192.168.102.142	202602230420

• Management + cluster traffic on interface "mgmt" (static IPs, /22 subnet).
• DNS + gateway: 192.168.100.1.
• OVN enabled, all other services disabled.

IncusOS API endpoints

All on port 8443, authenticated with Incus client certificate:

• /os/1.0 -- version, hostname, basic info
• /os/1.0/system/security -- TPM, Secure Boot, encryption/pool recovery keys
• /os/1.0/system/storage -- disks, partitions, ZFS pools
• /os/1.0/system/resources -- CPU, memory, hardware
• /os/1.0/system/network -- interfaces, DNS, routes
• /os/1.0/system/update -- channel, frequency, pending updates
• /os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}

incusos-health helper

incusos/helpers/incusos-health queries the IncusOS API across all cluster nodes. Actions:

Flag	Purpose
--status	Version, hostname, TPM status, Secure Boot
--partitions	Disk/partition layout, A/B state
--tpm	TPM details, Secure Boot certs, encryption keys
--services	Enabled/disabled services
--network	Interface config, DNS, gateway
--update	Channel, frequency, pending updates
--all	All of the above

Safety rules (MANDATORY)

• NEVER hard-stop during first boot -- corrupts TPM permanently, requires full reinstallation. Hard-stop is safe ONLY for already- provisioned nodes (TPM state already sealed).
• One node at a time for break-fix exercises. 3-node cluster needs 2/3 quorum. Taking down 2 nodes = cluster failure.
• Proxmox snapshots before every destructive test (VMID range 900-939).
• Target node-03 (VMID 902) for destructive tests -- not the leader, fewest workloads.
• Verify health with incusos-health --all before and after exercises.
• Monitor via Grafana during all exercises.

Update mechanics

• os_version = current running version.
• os_version_next = version on standby partition (same as current if no update pending, different if update downloaded).
• needs_reboot = true when update is downloaded and waiting for reboot.
• auto_reboot: false in lab -- updates download but don't reboot.
• Check frequency: 6 hours, stable channel.
• Nodes update independently (rolling, not coordinated).

Break-fix exercises (defined, not yet executed)

1. Normal Update Observation -- watch download, reboot, A/B switch
2. Simulated Failed Update -- hard-stop node-03 mid-update, expect rollback
3. Network Isolation -- disconnect node-03 NIC, observe OVN tunnel loss
4. Full Node Failure -- hard-stop node-03, verify 2/3 quorum holds

See notes/incusos-break-fix.md for full exercise details, prerequisites, expected behavior, and recovery steps.