# IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing IncusOS is an immutable, purpose-built operating system for running Incus clusters. This guide documents what we discovered about its internals through the `/os/1.0` API, and defines a catalog of break-fix exercises for testing cluster resilience in a safe lab environment. All observations come from a 3-node Proxmox-hosted cluster (VMID 900-902) running IncusOS with virtual TPM, Secure Boot, and OVN networking. ## IncusOS Architecture IncusOS is an immutable OS designed for a single purpose: running Incus. Key architectural properties discovered via the `/os/1.0` API: - **Immutable root filesystem** with A/B partition scheme. The running system boots from one partition; updates install to the other. Rollback is automatic if the new partition fails validation. - **TPM-based full disk encryption** -- root and swap volumes are encrypted and unlocked automatically via TPM measured boot. No passphrase required during normal operation. - **Secure Boot enforced** with 4 certificates in the UEFI firmware: - PK (Platform Key) -- root of trust - KEK (Key Exchange Key) -- authorizes db updates - db (2025 signing cert) -- validates current signed binaries - db (2026 signing cert) -- validates next-year signed binaries - **Version format**: `YYYYMMDDHHMI` (build timestamp, not semver). Example: `202602240349` = 2026-02-24 at 03:49 UTC. - **ZFS storage**: pool "local" on a dedicated partition (raid0 on single disk, ~30 GiB usable), encrypted with its own pool recovery key. - **Update system**: stable channel, 6-hour check frequency, `auto_reboot: false` (updates download but do not reboot automatically). ### Lab cluster state | Node | VMID | IP | IncusOS Version | |------|------|----|-----------------| | node-01 | 900 | 192.168.102.140 | 202602240349 | | node-02 | 901 | 192.168.102.141 | 202602230420 | | node-03 | 902 | 192.168.102.142 | 202602230420 | node-01 received a newer build than nodes 02/03, proving that nodes update independently (rolling updates, not cluster-wide atomic upgrades). ## Partition & Disk Layout Each node has a single 64 GiB SCSI disk (QEMU HARDDISK) with this layout: ``` +------------------------------------------------------+ | SCSI Disk (64 GiB, QEMU HARDDISK) | |------------------------------------------------------| | Partition 1 EFI System Partition | | Partition A Root filesystem (active or standby) | | Partition B Root filesystem (standby or active) | | Partition X Swap (encrypted, TPM-unlocked) | | Partition 11 ZFS data pool ("local") | +------------------------------------------------------+ ``` Key observations from the storage API: - **Partition 11** is the ZFS data partition, hosting pool "local". - **Pool "local"**: raid0, single vdev, ~30 GiB total, contains volume "incus" (~10 GiB used for Incus database, images, instances). - **Root + swap**: both encrypted, both unlocked by TPM at boot. No manual key entry needed unless TPM state is corrupted. ## Security Chain The boot security chain, as validated through the API: ``` UEFI firmware --> Secure Boot certificate validation (PK -> KEK -> db certs) --> Signed kernel + initrd loaded --> TPM measured boot (PCR measurements recorded) --> TPM validates measurements match expected policy --> Root partition decrypted and mounted --> Swap partition decrypted and activated --> System boots into trusted state ``` API-reported security state: | Property | Value | |----------|-------| | TPM status | ok | | Secure Boot | enabled, enforced | | System state | trusted | | Encryption recovery keys | 1 key (retrievable via API) | | Pool recovery keys | 1 key for pool "local" | **Recovery keys** are a safety net. If TPM state is corrupted (e.g., by a hard-stop during first boot), the encryption recovery key allows manual unlock. The pool recovery key allows ZFS pool import on a different system. ## Services Configuration Discovered via `/os/1.0/services/*`: | Service | Status | Notes | |---------|--------|-------| | OVN | enabled | Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels | | iSCSI | disabled | | | LVM | disabled | | | Multipath | disabled | | | NVMe | disabled | | | Tailscale | disabled | | | USB/IP | disabled | | OVN is the only enabled service beyond Incus itself, providing the overlay network for cross-node container connectivity and network policies. ## Network Configuration Each node has a single management interface: | Property | Value | |----------|-------| | Interface name | mgmt | | Role | management + cluster traffic | | Addressing | Static | | Subnet | /22 (192.168.100.0/22) | | DNS | 192.168.100.1 | | Gateway | 192.168.100.1 | Node IP assignments: - node-01: `192.168.102.140/22` - node-02: `192.168.102.141/22` - node-03: `192.168.102.142/22` The management interface carries both management traffic (API, cluster heartbeats) and OVN Geneve tunnel traffic. In production, these should be separated. ## Update Mechanics (Observed) The IncusOS update system uses A/B partitions for safe, rollback-capable updates. Observations from querying the update API: 1. **No pending update**: `os_version` and `os_version_next` are identical. This means no update has been downloaded or is waiting for reboot. 2. **Pending update**: `os_version_next` would differ from `os_version`, indicating a new build has been downloaded to the standby partition. 3. **Auto-reboot disabled**: `auto_reboot: false` means updates download to the standby partition but the node continues running the current version until explicitly rebooted (or until an admin triggers reboot via API / UI). 4. **No reboot needed**: `needs_reboot: false` confirms no downloaded update is waiting for a reboot to activate. 5. **Independent node updates**: node-01 is on `202602240349` while nodes 02/03 are on `202602230420`. This proves each node checks for and applies updates independently. There is no cluster-wide coordinated update mechanism at the OS level. 6. **Check frequency**: 6 hours. The stable channel is checked automatically on this interval. ### Update lifecycle (theoretical) ``` Check timer fires (every 6h) --> Query stable channel for new version --> Download new rootfs to standby partition (A or B) --> os_version_next updated, needs_reboot = true --> (if auto_reboot) Reboot automatically --> (if !auto_reboot) Wait for manual reboot --> On reboot: boot from new partition --> TPM re-measures, validates new boot chain --> If valid: new partition becomes active --> If invalid: rollback to previous partition ``` ## Break-Fix Exercise Catalog These exercises are designed to test IncusOS cluster resilience in the Proxmox lab. All exercises follow strict safety rules (see Safety Rules section below). **Current status**: All exercises are defined but not yet executed. --- ### Exercise 1: Normal Update Observation **Goal**: Observe the full update lifecycle -- download, reboot, A/B partition switch, version verification. **Prerequisites**: - Proxmox snapshot of all cluster nodes (VMID 900-902) - Grafana monitoring active (observe cluster metrics during update) - Verify cluster health with `incusos-health --all` **Steps**: 1. Record current versions: `incusos-health --update` 2. Trigger update via Operations Center UI, or wait for 6h check interval 3. Monitor via Grafana for download activity and node state changes 4. When `needs_reboot: true`, reboot one node at a time 5. After reboot, verify new version: `incusos-health --status` 6. Confirm A/B partition switch: `incusos-health --partitions` **What to observe**: - Download phase duration - Reboot duration (typically 30-60s for IncusOS) - Cluster behavior while one node reboots (2/3 quorum maintained) - New version number in `os_version` - Previous version still available on standby partition **Status**: Defined, not yet executed. --- ### Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update) **Goal**: Verify that IncusOS rolls back to the previous partition after a failed update (simulated by hard-stopping the VM during update). **Prerequisites**: - Proxmox snapshot of node-03 (VMID 902) -- non-leader, fewest workloads - Verify node-03 has no critical workloads - Confirm 3/3 nodes healthy before starting **Target node**: node-03 (VMID 902) ONLY. Never the cluster leader. **Steps**: 1. Take Proxmox snapshot of node-03: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/snapshot -d '{"snapname":"pre-break-fix-2"}'` 2. Trigger an update on node-03 (if one is available) 3. During the update download or early reboot phase, hard-stop node-03 via Proxmox API: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop` 4. Wait 10 seconds, then start node-03: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start` 5. Monitor boot via console screenshots: `incusos/helpers/proxmox-screenshot 902` 6. After boot, check version: `incusos-health --status` on node-03 **Expected behavior**: - Node boots from the previous (known-good) partition - Failed partition is marked as bad / not bootable - `os_version` returns to the pre-update version - Node rejoins the cluster automatically **Safety**: - ONLY node-03 -- maintains 2/3 quorum (node-01 + node-02 continue) - Do NOT hard-stop during *first boot* (corrupts TPM permanently) - If recovery fails, restore from Proxmox snapshot **Status**: Defined, not yet executed. --- ### Exercise 3: Network Isolation **Goal**: Observe how the cluster handles a node losing network connectivity -- OVN tunnel loss, cluster membership changes, and automatic recovery on reconnection. **Prerequisites**: - Evacuate all workloads from node-03 before disconnecting - Grafana monitoring active (watch OVN tunnel metrics, cluster events) - Verify cluster health: `incusos-health --all` **Steps**: 1. Evacuate workloads from node-03: `incus cluster evacuate oc-node-03 --target oc-node-01` 2. Disconnect node-03 NIC via Proxmox API: `incusos/helpers/proxmox-api PUT /nodes/pve/qemu/902/config -d '{"net0":"virtio=...,link_down=1"}'` 3. Observe in Grafana: OVN Geneve tunnel loss, cluster detecting missing node 4. Wait 2-5 minutes for cluster to mark node-03 as offline 5. Reconnect NIC: set `link_down=0` via Proxmox API 6. Monitor recovery: node-03 should rejoin cluster, OVN tunnels re-establish 7. Measure total recovery time from reconnection to healthy state **Expected behavior**: - Cluster detects node-03 offline within heartbeat timeout - OVN tunnels from/to node-03 fail (Geneve encap packets lost) - Cluster continues operating with 2/3 quorum - On reconnection: node-03 rejoins, tunnels re-establish, workloads can be restored **Recovery**: - If node-03 does not rejoin: check OVN service status, restart if needed - If cluster state is inconsistent: restore from Proxmox snapshot **Status**: Defined, not yet executed. --- ### Exercise 4: Full Node Failure **Goal**: Verify that the cluster survives a complete node loss and maintains operations with 2/3 quorum. Measure cluster rejoin time after node recovery. **Prerequisites**: - Proxmox snapshot of node-03 (VMID 902) - Evacuate ALL workloads from node-03 - Verify cluster health: `incusos-health --all` **Steps**: 1. Evacuate workloads: `incus cluster evacuate oc-node-03` 2. Take Proxmox snapshot of node-03 3. Hard-stop node-03 via Proxmox: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop` 4. Verify cluster continues operating: - `incus cluster list` should show 2 online, 1 offline - Existing workloads on node-01/02 remain accessible - OVN gateway should failover if node-03 was a gateway 5. Wait 5 minutes, observing Grafana metrics 6. Restart node-03: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start` 7. Monitor boot: `incusos/helpers/proxmox-screenshot 902` 8. Measure time from start to cluster rejoin (node shows online) 9. Restore workloads: `incus cluster restore oc-node-03` **Expected behavior**: - 2/3 quorum maintained -- all cluster operations continue - OVN gateway failover occurs if node-03 was elected gateway - After restart: node-03 boots, TPM unlocks, Incus starts, rejoins cluster - Typical rejoin time: 1-3 minutes after boot **Safety**: - This is NOT a first-boot scenario -- hard-stop is safe for already- provisioned nodes (TPM state is already sealed) - One node at a time ONLY - If node-03 fails to rejoin after restart, restore snapshot **Status**: Defined, not yet executed. ## Safety Rules These rules are non-negotiable for all break-fix exercises: 1. **Never hard-stop during first boot.** First boot seals TPM measurements and writes encryption keys. Interrupting this corrupts TPM state permanently, requiring full reinstallation. 2. **One node at a time.** A 3-node cluster requires 2/3 quorum. Never take down more than one node simultaneously, or the cluster loses quorum and all operations halt. 3. **Proxmox snapshots before every destructive test.** Snapshot the target node's VM (VMID 900-939) before any exercise that involves stopping, disconnecting, or modifying the node. 4. **Verify cluster health before and after.** Run `incusos-health --all` (or equivalent API checks) before starting an exercise and after completing it. Do not proceed if the cluster is already degraded. 5. **Monitor via Grafana during all exercises.** Visual monitoring catches issues that API polling might miss (e.g., OVN tunnel flapping, storage I/O spikes). 6. **Target node-03 for destructive tests.** node-03 (VMID 902) is the preferred target because it is not the cluster leader and typically has the fewest workloads. Evacuate before testing. 7. **Keep recovery keys accessible.** The encryption recovery key and ZFS pool recovery key should be retrieved and stored securely before any exercise that might corrupt TPM state. ## Helper Script: incusos-health The `incusos/helpers/incusos-health` script queries the IncusOS API on cluster nodes to report system state. It is the primary tool for verifying cluster health before and after break-fix exercises. ### Usage ```bash incusos/helpers/incusos-health [ACTION] [OPTIONS] ``` ### Actions | Action | Description | |--------|-------------| | `--status` | Basic system info: version, hostname, TPM status, Secure Boot | | `--partitions` | Disk and partition layout, A/B partition state | | `--tpm` | TPM details, Secure Boot certificates, encryption keys | | `--services` | Enabled/disabled services (OVN, iSCSI, LVM, etc.) | | `--network` | Network interface configuration, DNS, gateway | | `--update` | Update channel, check frequency, pending update status | | `--all` | Run all of the above in sequence | ### Example output workflow ```bash # Before exercise: verify all nodes healthy incusos/helpers/incusos-health --all # After exercise: verify recovery incusos/helpers/incusos-health --status # Quick version check incusos/helpers/incusos-health --all # Full health report ``` ## API Reference All IncusOS system information is available via the REST API on each node. | Endpoint | Returns | |----------|---------| | `/os/1.0` | Version, hostname, basic system info | | `/os/1.0/system/security` | TPM status, Secure Boot, encryption keys | | `/os/1.0/system/storage` | Disks, partitions, ZFS pools | | `/os/1.0/system/resources` | CPU, memory, hardware info | | `/os/1.0/system/network` | Interfaces, DNS, routes | | `/os/1.0/system/update` | Update channel, version, pending updates | | `/os/1.0/services/ovn` | OVN configuration and status | | `/os/1.0/services/iscsi` | iSCSI configuration | | `/os/1.0/services/lvm` | LVM configuration | | `/os/1.0/services/multipath` | Multipath configuration | | `/os/1.0/services/nvme` | NVMe-oF configuration | | `/os/1.0/services/tailscale` | Tailscale VPN configuration | | `/os/1.0/services/usbip` | USB/IP configuration | The API listens on the management interface, HTTPS, port 8443 (same as Incus). Authentication uses the Incus client certificate.