maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/incusos-break-fix.md

16 KiB
Raw Blame History

IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing

IncusOS is an immutable, purpose-built operating system for running Incus clusters. This guide documents what we discovered about its internals through the /os/1.0 API, and defines a catalog of break-fix exercises for testing cluster resilience in a safe lab environment.

All observations come from a 3-node Proxmox-hosted cluster (VMID 900-902) running IncusOS with virtual TPM, Secure Boot, and OVN networking.

IncusOS Architecture

IncusOS is an immutable OS designed for a single purpose: running Incus. Key architectural properties discovered via the /os/1.0 API:

  • Immutable root filesystem with A/B partition scheme. The running system boots from one partition; updates install to the other. Rollback is automatic if the new partition fails validation.
  • TPM-based full disk encryption -- root and swap volumes are encrypted and unlocked automatically via TPM measured boot. No passphrase required during normal operation.
  • Secure Boot enforced with 4 certificates in the UEFI firmware:
    • PK (Platform Key) -- root of trust
    • KEK (Key Exchange Key) -- authorizes db updates
    • db (2025 signing cert) -- validates current signed binaries
    • db (2026 signing cert) -- validates next-year signed binaries
  • Version format: YYYYMMDDHHMI (build timestamp, not semver). Example: 202602240349 = 2026-02-24 at 03:49 UTC.
  • ZFS storage: pool "local" on a dedicated partition (raid0 on single disk, ~30 GiB usable), encrypted with its own pool recovery key.
  • Update system: stable channel, 6-hour check frequency, auto_reboot: false (updates download but do not reboot automatically).

Lab cluster state

Node VMID IP IncusOS Version
node-01 900 192.168.102.140 202602240349
node-02 901 192.168.102.141 202602230420
node-03 902 192.168.102.142 202602230420

node-01 received a newer build than nodes 02/03, proving that nodes update independently (rolling updates, not cluster-wide atomic upgrades).

Partition & Disk Layout

Each node has a single 64 GiB SCSI disk (QEMU HARDDISK) with this layout:

+------------------------------------------------------+
| SCSI Disk  (64 GiB, QEMU HARDDISK)                  |
|------------------------------------------------------|
| Partition 1   EFI System Partition                   |
| Partition A   Root filesystem (active or standby)    |
| Partition B   Root filesystem (standby or active)    |
| Partition X   Swap (encrypted, TPM-unlocked)         |
| Partition 11  ZFS data pool ("local")                |
+------------------------------------------------------+

Key observations from the storage API:

  • Partition 11 is the ZFS data partition, hosting pool "local".
  • Pool "local": raid0, single vdev, ~30 GiB total, contains volume "incus" (~10 GiB used for Incus database, images, instances).
  • Root + swap: both encrypted, both unlocked by TPM at boot. No manual key entry needed unless TPM state is corrupted.

Security Chain

The boot security chain, as validated through the API:

UEFI firmware
  --> Secure Boot certificate validation (PK -> KEK -> db certs)
    --> Signed kernel + initrd loaded
      --> TPM measured boot (PCR measurements recorded)
        --> TPM validates measurements match expected policy
          --> Root partition decrypted and mounted
          --> Swap partition decrypted and activated
            --> System boots into trusted state

API-reported security state:

Property Value
TPM status ok
Secure Boot enabled, enforced
System state trusted
Encryption recovery keys 1 key (retrievable via API)
Pool recovery keys 1 key for pool "local"

Recovery keys are a safety net. If TPM state is corrupted (e.g., by a hard-stop during first boot), the encryption recovery key allows manual unlock. The pool recovery key allows ZFS pool import on a different system.

Services Configuration

Discovered via /os/1.0/services/*:

Service Status Notes
OVN enabled Connects to cluster DB at tcp:192.168.102.142:6642, Geneve tunnels
iSCSI disabled
LVM disabled
Multipath disabled
NVMe disabled
Tailscale disabled
USB/IP disabled

OVN is the only enabled service beyond Incus itself, providing the overlay network for cross-node container connectivity and network policies.

Network Configuration

Each node has a single management interface:

Property Value
Interface name mgmt
Role management + cluster traffic
Addressing Static
Subnet /22 (192.168.100.0/22)
DNS 192.168.100.1
Gateway 192.168.100.1

Node IP assignments:

  • node-01: 192.168.102.140/22
  • node-02: 192.168.102.141/22
  • node-03: 192.168.102.142/22

The management interface carries both management traffic (API, cluster heartbeats) and OVN Geneve tunnel traffic. In production, these should be separated.

Update Mechanics (Observed)

The IncusOS update system uses A/B partitions for safe, rollback-capable updates. Observations from querying the update API:

  1. No pending update: os_version and os_version_next are identical. This means no update has been downloaded or is waiting for reboot.

  2. Pending update: os_version_next would differ from os_version, indicating a new build has been downloaded to the standby partition.

  3. Auto-reboot disabled: auto_reboot: false means updates download to the standby partition but the node continues running the current version until explicitly rebooted (or until an admin triggers reboot via API / UI).

  4. No reboot needed: needs_reboot: false confirms no downloaded update is waiting for a reboot to activate.

  5. Independent node updates: node-01 is on 202602240349 while nodes 02/03 are on 202602230420. This proves each node checks for and applies updates independently. There is no cluster-wide coordinated update mechanism at the OS level.

  6. Check frequency: 6 hours. The stable channel is checked automatically on this interval.

Update lifecycle (theoretical)

Check timer fires (every 6h)
  --> Query stable channel for new version
    --> Download new rootfs to standby partition (A or B)
      --> os_version_next updated, needs_reboot = true
        --> (if auto_reboot) Reboot automatically
        --> (if !auto_reboot) Wait for manual reboot
          --> On reboot: boot from new partition
            --> TPM re-measures, validates new boot chain
              --> If valid: new partition becomes active
              --> If invalid: rollback to previous partition

Break-Fix Exercise Catalog

These exercises are designed to test IncusOS cluster resilience in the Proxmox lab. All exercises follow strict safety rules (see Safety Rules section below).

Current status: All exercises are defined but not yet executed.

Exercise 1: Normal Update Observation

Goal: Observe the full update lifecycle -- download, reboot, A/B partition switch, version verification.

Prerequisites:

  • Proxmox snapshot of all cluster nodes (VMID 900-902)
  • Grafana monitoring active (observe cluster metrics during update)
  • Verify cluster health with incusos-health --all

Steps:

  1. Record current versions: incusos-health --update
  2. Trigger update via Operations Center UI, or wait for 6h check interval
  3. Monitor via Grafana for download activity and node state changes
  4. When needs_reboot: true, reboot one node at a time
  5. After reboot, verify new version: incusos-health --status
  6. Confirm A/B partition switch: incusos-health --partitions

What to observe:

  • Download phase duration
  • Reboot duration (typically 30-60s for IncusOS)
  • Cluster behavior while one node reboots (2/3 quorum maintained)
  • New version number in os_version
  • Previous version still available on standby partition

Status: Defined, not yet executed.

Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update)

Goal: Verify that IncusOS rolls back to the previous partition after a failed update (simulated by hard-stopping the VM during update).

Prerequisites:

  • Proxmox snapshot of node-03 (VMID 902) -- non-leader, fewest workloads
  • Verify node-03 has no critical workloads
  • Confirm 3/3 nodes healthy before starting

Target node: node-03 (VMID 902) ONLY. Never the cluster leader.

Steps:

  1. Take Proxmox snapshot of node-03: incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/snapshot -d '{"snapname":"pre-break-fix-2"}'
  2. Trigger an update on node-03 (if one is available)
  3. During the update download or early reboot phase, hard-stop node-03 via Proxmox API: incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop
  4. Wait 10 seconds, then start node-03: incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start
  5. Monitor boot via console screenshots: incusos/helpers/proxmox-screenshot 902
  6. After boot, check version: incusos-health --status on node-03

Expected behavior:

  • Node boots from the previous (known-good) partition
  • Failed partition is marked as bad / not bootable
  • os_version returns to the pre-update version
  • Node rejoins the cluster automatically

Safety:

  • ONLY node-03 -- maintains 2/3 quorum (node-01 + node-02 continue)
  • Do NOT hard-stop during first boot (corrupts TPM permanently)
  • If recovery fails, restore from Proxmox snapshot

Status: Defined, not yet executed.

Exercise 3: Network Isolation

Goal: Observe how the cluster handles a node losing network connectivity -- OVN tunnel loss, cluster membership changes, and automatic recovery on reconnection.

Prerequisites:

  • Evacuate all workloads from node-03 before disconnecting
  • Grafana monitoring active (watch OVN tunnel metrics, cluster events)
  • Verify cluster health: incusos-health --all

Steps:

  1. Evacuate workloads from node-03: incus cluster evacuate oc-node-03 --target oc-node-01
  2. Disconnect node-03 NIC via Proxmox API: incusos/helpers/proxmox-api PUT /nodes/pve/qemu/902/config -d '{"net0":"virtio=...,link_down=1"}'
  3. Observe in Grafana: OVN Geneve tunnel loss, cluster detecting missing node
  4. Wait 2-5 minutes for cluster to mark node-03 as offline
  5. Reconnect NIC: set link_down=0 via Proxmox API
  6. Monitor recovery: node-03 should rejoin cluster, OVN tunnels re-establish
  7. Measure total recovery time from reconnection to healthy state

Expected behavior:

  • Cluster detects node-03 offline within heartbeat timeout
  • OVN tunnels from/to node-03 fail (Geneve encap packets lost)
  • Cluster continues operating with 2/3 quorum
  • On reconnection: node-03 rejoins, tunnels re-establish, workloads can be restored

Recovery:

  • If node-03 does not rejoin: check OVN service status, restart if needed
  • If cluster state is inconsistent: restore from Proxmox snapshot

Status: Defined, not yet executed.

Exercise 4: Full Node Failure

Goal: Verify that the cluster survives a complete node loss and maintains operations with 2/3 quorum. Measure cluster rejoin time after node recovery.

Prerequisites:

  • Proxmox snapshot of node-03 (VMID 902)
  • Evacuate ALL workloads from node-03
  • Verify cluster health: incusos-health --all

Steps:

  1. Evacuate workloads: incus cluster evacuate oc-node-03
  2. Take Proxmox snapshot of node-03
  3. Hard-stop node-03 via Proxmox: incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop
  4. Verify cluster continues operating:
    • incus cluster list should show 2 online, 1 offline
    • Existing workloads on node-01/02 remain accessible
    • OVN gateway should failover if node-03 was a gateway
  5. Wait 5 minutes, observing Grafana metrics
  6. Restart node-03: incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start
  7. Monitor boot: incusos/helpers/proxmox-screenshot 902
  8. Measure time from start to cluster rejoin (node shows online)
  9. Restore workloads: incus cluster restore oc-node-03

Expected behavior:

  • 2/3 quorum maintained -- all cluster operations continue
  • OVN gateway failover occurs if node-03 was elected gateway
  • After restart: node-03 boots, TPM unlocks, Incus starts, rejoins cluster
  • Typical rejoin time: 1-3 minutes after boot

Safety:

  • This is NOT a first-boot scenario -- hard-stop is safe for already- provisioned nodes (TPM state is already sealed)
  • One node at a time ONLY
  • If node-03 fails to rejoin after restart, restore snapshot

Status: Defined, not yet executed.

Safety Rules

These rules are non-negotiable for all break-fix exercises:

  1. Never hard-stop during first boot. First boot seals TPM measurements and writes encryption keys. Interrupting this corrupts TPM state permanently, requiring full reinstallation.

  2. One node at a time. A 3-node cluster requires 2/3 quorum. Never take down more than one node simultaneously, or the cluster loses quorum and all operations halt.

  3. Proxmox snapshots before every destructive test. Snapshot the target node's VM (VMID 900-939) before any exercise that involves stopping, disconnecting, or modifying the node.

  4. Verify cluster health before and after. Run incusos-health --all (or equivalent API checks) before starting an exercise and after completing it. Do not proceed if the cluster is already degraded.

  5. Monitor via Grafana during all exercises. Visual monitoring catches issues that API polling might miss (e.g., OVN tunnel flapping, storage I/O spikes).

  6. Target node-03 for destructive tests. node-03 (VMID 902) is the preferred target because it is not the cluster leader and typically has the fewest workloads. Evacuate before testing.

  7. Keep recovery keys accessible. The encryption recovery key and ZFS pool recovery key should be retrieved and stored securely before any exercise that might corrupt TPM state.

Helper Script: incusos-health

The incusos/helpers/incusos-health script queries the IncusOS API on cluster nodes to report system state. It is the primary tool for verifying cluster health before and after break-fix exercises.

Usage

incusos/helpers/incusos-health [ACTION] [OPTIONS]

Actions

Action Description
--status Basic system info: version, hostname, TPM status, Secure Boot
--partitions Disk and partition layout, A/B partition state
--tpm TPM details, Secure Boot certificates, encryption keys
--services Enabled/disabled services (OVN, iSCSI, LVM, etc.)
--network Network interface configuration, DNS, gateway
--update Update channel, check frequency, pending update status
--all Run all of the above in sequence

Example output workflow

# Before exercise: verify all nodes healthy
incusos/helpers/incusos-health --all

# After exercise: verify recovery
incusos/helpers/incusos-health --status   # Quick version check
incusos/helpers/incusos-health --all      # Full health report

API Reference

All IncusOS system information is available via the REST API on each node.

Endpoint Returns
/os/1.0 Version, hostname, basic system info
/os/1.0/system/security TPM status, Secure Boot, encryption keys
/os/1.0/system/storage Disks, partitions, ZFS pools
/os/1.0/system/resources CPU, memory, hardware info
/os/1.0/system/network Interfaces, DNS, routes
/os/1.0/system/update Update channel, version, pending updates
/os/1.0/services/ovn OVN configuration and status
/os/1.0/services/iscsi iSCSI configuration
/os/1.0/services/lvm LVM configuration
/os/1.0/services/multipath Multipath configuration
/os/1.0/services/nvme NVMe-oF configuration
/os/1.0/services/tailscale Tailscale VPN configuration
/os/1.0/services/usbip USB/IP configuration

The API listens on the management interface, HTTPS, port 8443 (same as Incus). Authentication uses the Incus client certificate.