# IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing IncusOS is an immutable, purpose-built operating system for running Incus clusters. This guide documents what we discovered about its internals through the `/os/1.0` API, and records the results of break-fix exercises testing cluster resilience in a safe lab environment. All observations come from a 3-node Proxmox-hosted cluster (VMID 400-402) running IncusOS with virtual TPM, Secure Boot, and OVN networking. ## IncusOS Architecture IncusOS is an immutable OS designed for a single purpose: running Incus. Key architectural properties discovered via the `/os/1.0` API: - **Immutable root filesystem** with A/B partition scheme. The running system boots from one partition; updates install to the other. Rollback is automatic if the new partition fails validation. - **TPM-based full disk encryption** -- root and swap volumes are encrypted and unlocked automatically via TPM measured boot. No passphrase required during normal operation. - **Secure Boot enforced** with 4 certificates in the UEFI firmware: - PK (Platform Key) -- root of trust - KEK (Key Exchange Key) -- authorizes db updates - db (2025 signing cert) -- validates current signed binaries - db (2026 signing cert) -- validates next-year signed binaries - **Version format**: `YYYYMMDDHHMI` (build timestamp, not semver). Example: `202602240349` = 2026-02-24 at 03:49 UTC. - **ZFS storage**: pool "local" on a dedicated partition (raid0 on single disk, ~30 GiB usable), encrypted with its own pool recovery key. - **Update system**: stable channel, configurable check frequency, `auto_reboot: false` (updates download but do not reboot automatically). - **MAC-dependent boot**: IncusOS uses udev rules to rename the network interface by MAC address at boot. Changing the VM's MAC address will cause `ERROR timed out waiting for udev to rename interface(s)` and a boot hang. ### Lab cluster state (post-exercises) | Node | VMID | IP | IncusOS Version | |------|------|----|-----------------| | node-01 | 400 | 192.168.102.140 | 202602240349 | | node-02 | 401 | 192.168.102.141 | 202602240349 | | node-03 | 402 | 192.168.102.142 | 202602240349 | All 3 nodes updated to `202602240349` during the break-fix exercises. node-01 was already on this version; node-02 and node-03 were on `202602230420` and updated independently (proving rolling updates work). ## Partition & Disk Layout Each node has a single SCSI disk (QEMU HARDDISK) with this layout: ``` +------------------------------------------------------+ | SCSI Disk (64-100 GiB, QEMU HARDDISK) | |------------------------------------------------------| | Partition 1 EFI System Partition | | Partition A Root filesystem (active or standby) | | Partition B Root filesystem (standby or active) | | Partition X Swap (encrypted, TPM-unlocked) | | Partition 11 ZFS data pool ("local") | +------------------------------------------------------+ ``` Key observations from the storage API: - **Partition 11** is the ZFS data partition, hosting pool "local". - **Pool "local"**: raid0, single vdev, 30-66 GiB depending on disk size, contains volume "incus" (Incus database, images, instances). - **Root + swap**: both encrypted, both unlocked by TPM at boot. No manual key entry needed unless TPM state is corrupted. ## Security Chain The boot security chain, as validated through the API: ``` UEFI firmware --> Secure Boot certificate validation (PK -> KEK -> db certs) --> Signed kernel + initrd loaded --> TPM measured boot (PCR measurements recorded) --> TPM validates measurements match expected policy --> Root partition decrypted and mounted --> Swap partition decrypted and activated --> System boots into trusted state ``` API-reported security state: | Property | Value | |----------|-------| | TPM status | ok | | Secure Boot | enabled, enforced | | System state | trusted | | Encryption recovery keys | 1 key (retrievable via API) | | Pool recovery keys | 1 key for pool "local" | **Recovery keys** are a safety net. If TPM state is corrupted (e.g., by a hard-stop during first boot), the encryption recovery key allows manual unlock. The pool recovery key allows ZFS pool import on a different system. ## Services Configuration Discovered via `/os/1.0/services/*`: | Service | Status | Notes | |---------|--------|-------| | OVN | enabled | Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels | | iSCSI | disabled | | | LVM | disabled | | | Multipath | disabled | | | NVMe | disabled | | | Tailscale | disabled | | | USB/IP | disabled | | OVN is the only enabled service beyond Incus itself, providing the overlay network for cross-node container connectivity and network policies. ## Network Configuration Each node has a single management interface: | Property | Value | |----------|-------| | Interface name | mgmt | | Role | management + cluster traffic | | Addressing | Static | | Subnet | /22 (192.168.100.0/22) | | DNS | 192.168.100.1 | | Gateway | 192.168.100.1 | Node IP assignments: - node-01: `192.168.102.140/22` - node-02: `192.168.102.141/22` - node-03: `192.168.102.142/22` The management interface carries both management traffic (API, cluster heartbeats) and OVN Geneve tunnel traffic. In production, these should be separated. ## Update Mechanics (Observed and Tested) The IncusOS update system uses A/B partitions for safe, rollback-capable updates. Observations from querying the update API and live testing: 1. **No pending update**: `os_version` and `os_version_next` are identical. This means no update has been downloaded or is waiting for reboot. 2. **Pending update**: `os_version_next` differs from `os_version`, indicating a new build has been downloaded to the standby partition. The update status shows: `"IncusOS has been updated to version YYYYMMDDHHMI"`. 3. **Auto-reboot disabled**: `auto_reboot: false` means updates download to the standby partition but the node continues running the current version until explicitly rebooted. 4. **`needs_reboot: true`** confirms a downloaded update is waiting for reboot to activate. 5. **Independent node updates**: node-01 was on `202602240349` while nodes 02/03 were on `202602230420`. Each node checks for and applies updates independently. There is no cluster-wide coordinated update. 6. **Check frequency is configurable via API**: Use PUT to `/os/1.0/system/update` with `{"config":{"check_frequency":"6h"}}`. Default may be "never" on some nodes -- verify and set explicitly. POST to trigger an immediate check returns 501 (not implemented). 7. **Boot-time update check**: When a node reboots, IncusOS checks for updates during the boot sequence. node-03 updated from `202602230420` to `202602240349` during a hard-stop/restart cycle, even though the update timer had not yet fired. ### Update lifecycle (observed) ``` Config: check_frequency set via PUT /os/1.0/system/update --> Timer fires (or boot-time check runs) --> Query stable channel for new version --> Download new rootfs to standby partition --> os_version_next updated, needs_reboot = true --> status = "IncusOS has been updated to version YYYYMMDDHHMI" --> (if auto_reboot) Reboot automatically --> (if !auto_reboot) Wait for manual reboot --> On reboot: boot from new partition --> TPM re-measures, validates new boot chain --> If valid: new partition becomes active --> If invalid: rollback to previous partition ``` --- ## Break-Fix Exercise Results These exercises were executed on 2026-02-24 against the 3-node lab cluster. All exercises followed the safety rules (see Safety Rules section below). --- ### Exercise 1: Normal Update Observation **Goal**: Observe the full update lifecycle -- download, reboot, A/B partition switch, version verification. **Executed**: 2026-02-24 #### Pre-exercise state | Node | Version | Check Frequency | Last Check | |------|---------|----------------|------------| | node-01 | 202602240349 | 6h | 2026-02-24T15:05 | | node-02 | 202602230420 | never | never | | node-03 | 202602230420 | never | never | Key finding: nodes 02/03 had `check_frequency: never` by default, which is why they never downloaded the available update. The check frequency must be explicitly configured via the API. #### Steps and observations 1. Changed check frequency on nodes 02/03 via API PUT to `6h`. 2. The update timer is not immediately responsive -- setting `10s` or `6h` does not trigger an immediate check. POST returns 501 (not implemented). 3. **node-03 updated during a hard-stop/restart** (Exercise 4 triggered the reboot). The boot-time check found and applied the update. 4. **node-02 download observed**: After setting frequency to `6h`, the node eventually checked and downloaded the update: - Status: `"IncusOS has been updated to version 202602240349"` - `os_version: 202602230420`, `os_version_next: 202602240349` - `needs_reboot: true` 5. Rebooted node-02 (graceful, via Proxmox API). The database-leader role seamlessly transferred during the reboot. 6. Post-reboot: node-02 running `202602240349`, `needs_reboot: false`. #### Timing | Event | Duration | |-------|----------| | node-02 reboot (offline → online) | ~50s | | Database-leader role transfer | seamless (no cluster disruption) | | All instances on node-02 auto-restart | within boot time | #### Result All 3 nodes successfully updated to `202602240349`. The A/B partition scheme works as documented -- updates download to the standby partition and activate on reboot. **Status**: Completed successfully. --- ### Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update) **Goal**: Verify that IncusOS rolls back to the previous partition after a failed update (simulated by hard-stopping the VM during update). **Status**: Not executable. All 3 nodes were already on the latest version (`202602240349`) after Exercises 1 and 4. No pending update was available to interrupt. **To execute in the future**: Wait for a new IncusOS build to be published to the stable channel, then: 1. Take Proxmox snapshot of node-03 (VMID 402) 2. Trigger update check (change frequency, wait for download) 3. During early reboot phase (after download, during boot), hard-stop the VM via `proxmox-api POST /nodes/pve/qemu/402/status/stop` 4. Start the VM and observe which partition boots (A or B) 5. Check `os_version` -- should be the pre-update version if rollback worked --- ### Exercise 3: Network Isolation **Goal**: Observe how the cluster handles a node losing network connectivity -- OVN tunnel loss, cluster membership changes, and automatic recovery on reconnection. **Executed**: 2026-02-24 #### Pre-exercise state All 3 nodes healthy, 20 instances running. node-03 hosted 6 instances including `ovn-central` and `node-exp-03`. #### Steps and observations 1. **Evacuated workloads** from node-03 via `incus cluster evacuate`. 3 instances migrated to nodes 01/02, 3 stopped in place. 2. **Disconnected NIC** via Proxmox API: ``` proxmox-api PUT /nodes/pve/qemu/402/config \ --data-urlencode 'net0=virtio=BC:24:11:11:6E:F9,bridge=vmbr0,tag=69,link_down=1' ``` 3. **Cluster detection**: node-03 detected as OFFLINE within ~20s of NIC disconnect. Heartbeat message shows exact last heartbeat timestamp. 4. **Critical finding -- OVN gateway is a SPOF**: The OVN router's external gateway was scheduled on node-03's chassis. When node-03 lost network: - **East-west traffic** (container ↔ container on OVN): **still worked** - **North-south traffic** (OVN ↔ physical LAN): **completely broken** - Grafana from LAN: unreachable (OVN forward goes through gateway) - Monitoring → management network (SNAT): broken - Monitoring → other OVN containers (10.10.10.x): still working - **No gateway failover occurred** even after 3+ minutes of waiting. This is a single point of failure in the current OVN configuration. 5. **NIC reconnection gotcha**: Using Proxmox `PUT /config` with `-d` flag regenerates the MAC address because curl interprets `:` in the MAC as URL parameters. **Must use `--data-urlencode`** to preserve the MAC address. Changing the MAC causes IncusOS boot failure: `ERROR timed out waiting for udev to rename interface(s)` 6. **Recovery after correct MAC restored**: - Cluster rejoin: ~26s after VM start - OVN gateway recovery: ~75s (north-south traffic restored) #### Key findings | Metric | Value | |--------|-------| | Detection time | ~20s (heartbeat timeout) | | OVN east-west during isolation | Working | | OVN north-south during isolation | Broken (no failover) | | Cluster quorum | 2/3 maintained | | Recovery (cluster rejoin) | ~26s | | Recovery (OVN gateway) | ~75s | #### Lessons learned - **OVN gateway HA**: The OVN logical router gateway chassis does not automatically failover when a node goes offline. In production, this would need to be addressed with gateway chassis groups or redundant uplinks. This is likely an Incus OVN configuration issue, not an IncusOS limitation. - **Proxmox NIC manipulation**: Use `--data-urlencode` for any Proxmox API PUT that includes MAC addresses. The `-d` flag with raw data corrupts the MAC, regenerating a new one. - **IncusOS MAC dependency**: The boot process uses udev rules tied to the NIC's MAC address. A MAC change = boot failure. This is important for VM migration, NIC replacement, or Proxmox config changes. **Status**: Completed with significant findings. --- ### Exercise 4: Full Node Failure **Goal**: Verify that the cluster survives a complete node loss and maintains operations with 2/3 quorum. Measure cluster rejoin time after node recovery. **Executed**: 2026-02-24 #### Pre-exercise state All 3 nodes healthy. node-03 hosted 6 instances. No evacuation was performed before the hard-stop (to test realistic failure behavior). #### Steps and observations 1. **Hard-stopped node-03** via Proxmox API at 16:12:19 CET: ``` proxmox-api POST /nodes/pve/qemu/402/status/stop ``` 2. **Detection**: Cluster marked node-03 OFFLINE within ~20s. The status message includes the exact last heartbeat timestamp: `No heartbeat for 26.79s (2026-02-24 15:12:10 UTC)` 3. **Cluster behavior with node-03 down**: - 2/3 quorum maintained -- node-01 and node-02 "Fully operational" - All 14 instances on nodes 01/02 remained RUNNING - 6 instances on node-03 went to ERROR state - `incus cluster list`, `incus list`, `incus exec` all continued working - Storage pool listing worked normally - **OVN operations continued** despite `network.ovn.northbound_connection` pointing to node-03 (192.168.102.142:6641). The Incus database layer handles OVN NB database failover transparently. 4. **Prometheus impact**: 7/9 targets UP (node-03 Incus metrics + node-exp-03 down) 5. **Grafana**: accessible from LAN (monitoring container on node-02) 6. **Restarted node-03** at 16:17:17 CET. Recovery timeline: - +50s: Cluster shows node-03 "Fully operational" - All 6 instances on node-03 auto-restarted (RUNNING within 1-2 min) - 9/9 Prometheus targets back to UP 7. **Bonus finding**: node-03 updated from `202602230420` to `202602240349` during the reboot! IncusOS performs an update check at boot time, found the newer version, and activated it. #### Timing | Event | Time | |-------|------| | Hard-stop → OFFLINE detection | ~20s | | Cluster operations during failure | Fully functional (2/3 quorum) | | VM start → cluster rejoin | ~50s | | VM start → all instances RUNNING | ~2 min | | VM start → 9/9 Prometheus targets | ~3 min | #### Key findings - Hard-stop of a non-leader node is safe (TPM state preserved for already-provisioned nodes, as expected). - Cluster quorum with 2/3 nodes provides full operational capability. - Instance auto-restart on node recovery is automatic -- no manual intervention needed. - OVN NB database failover is transparent (even when the configured connection target is the downed node). - Boot-time update checks can apply pending updates during recovery. **Status**: Completed successfully. --- ## Safety Rules These rules are non-negotiable for all break-fix exercises: 1. **Never hard-stop during first boot.** First boot seals TPM measurements and writes encryption keys. Interrupting this corrupts TPM state permanently, requiring full reinstallation. 2. **One node at a time.** A 3-node cluster requires 2/3 quorum. Never take down more than one node simultaneously, or the cluster loses quorum and all operations halt. 3. **Proxmox snapshots before every destructive test.** Snapshot the target node's VM (VMID 400-402) before any exercise that involves stopping, disconnecting, or modifying the node. Note: the API token may lack `VM.Snapshot` permissions -- verify before relying on this. 4. **Verify cluster health before and after.** Run `incusos-health --all` (or equivalent API checks) before starting an exercise and after completing it. Do not proceed if the cluster is already degraded. 5. **Monitor via Grafana during all exercises.** Visual monitoring catches issues that API polling might miss (e.g., OVN tunnel flapping, storage I/O spikes). 6. **Target node-03 for destructive tests.** node-03 (VMID 402) is the preferred target because it is not the cluster leader and typically has the fewest workloads. Evacuate before testing. 7. **Keep recovery keys accessible.** The encryption recovery key and ZFS pool recovery key should be retrieved and stored securely before any exercise that might corrupt TPM state. 8. **Never change MAC addresses on IncusOS VMs.** IncusOS uses udev rules tied to the NIC MAC. Changing the MAC causes a boot hang. When using Proxmox API for NIC operations, always use `--data-urlencode` to preserve the MAC address. ## Helper Script: incusos-health The `incusos/helpers/incusos-health` script queries the IncusOS API on cluster nodes to report system state. It is the primary tool for verifying cluster health before and after break-fix exercises. ### Usage ```bash incusos/helpers/incusos-health [ACTION] [OPTIONS] ``` ### Actions | Action | Description | |--------|-------------| | `--status` | Basic system info: version, hostname, TPM status, Secure Boot | | `--partitions` | Disk and partition layout, A/B partition state | | `--tpm` | TPM details, Secure Boot certificates, encryption keys | | `--services` | Enabled/disabled services (OVN, iSCSI, LVM, etc.) | | `--network` | Network interface configuration, DNS, gateway | | `--update` | Update channel, check frequency, pending update status | | `--all` | Run all of the above in sequence | ### Example output workflow ```bash # Before exercise: verify all nodes healthy incusos/helpers/incusos-health --all # After exercise: verify recovery incusos/helpers/incusos-health --status # Quick version check incusos/helpers/incusos-health --all # Full health report ``` ## API Reference All IncusOS system information is available via the REST API on each node. | Endpoint | Method | Returns | |----------|--------|---------| | `/os/1.0` | GET | Version, hostname, basic system info | | `/os/1.0/system/security` | GET | TPM status, Secure Boot, encryption keys | | `/os/1.0/system/storage` | GET | Disks, partitions, ZFS pools | | `/os/1.0/system/resources` | GET | CPU, memory, hardware info | | `/os/1.0/system/network` | GET | Interfaces, DNS, routes | | `/os/1.0/system/update` | GET | Update channel, version, pending updates | | `/os/1.0/system/update` | PUT | Change update config (check_frequency, auto_reboot, channel) | | `/os/1.0/services/ovn` | GET | OVN configuration and status | | `/os/1.0/services/iscsi` | GET | iSCSI configuration | | `/os/1.0/services/lvm` | GET | LVM configuration | | `/os/1.0/services/multipath` | GET | Multipath configuration | | `/os/1.0/services/nvme` | GET | NVMe-oF configuration | | `/os/1.0/services/tailscale` | GET | Tailscale VPN configuration | | `/os/1.0/services/usbip` | GET | USB/IP configuration | The API listens on the management interface, HTTPS, port 8443 (same as Incus). Authentication uses the Incus client certificate. ### Update config via API ```bash # Change check frequency (accepted values: "never", "1h", "6h", "12h", "24h") curl -sk --cert "$CERT" --key "$KEY" -X PUT \ https://NODE_IP:8443/os/1.0/system/update \ -H "Content-Type: application/json" \ -d '{"config":{"auto_reboot":false,"channel":"stable","check_frequency":"6h"}}' # Note: POST to trigger immediate check returns 501 (not implemented) ```