21 KiB

Raw Blame History

IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing

IncusOS is an immutable, purpose-built operating system for running Incus clusters. This guide documents what we discovered about its internals through the /os/1.0 API, and records the results of break-fix exercises testing cluster resilience in a safe lab environment.

All observations come from a 3-node Proxmox-hosted cluster (VMID 400-402) running IncusOS with virtual TPM, Secure Boot, and OVN networking.

IncusOS Architecture

IncusOS is an immutable OS designed for a single purpose: running Incus. Key architectural properties discovered via the /os/1.0 API:

Immutable root filesystem with A/B partition scheme. The running system boots from one partition; updates install to the other. Rollback is automatic if the new partition fails validation.
TPM-based full disk encryption -- root and swap volumes are encrypted and unlocked automatically via TPM measured boot. No passphrase required during normal operation.
Secure Boot enforced with 4 certificates in the UEFI firmware:
- PK (Platform Key) -- root of trust
- KEK (Key Exchange Key) -- authorizes db updates
- db (2025 signing cert) -- validates current signed binaries
- db (2026 signing cert) -- validates next-year signed binaries
Version format: YYYYMMDDHHMI (build timestamp, not semver). Example: 202602240349 = 2026-02-24 at 03:49 UTC.
ZFS storage: pool "local" on a dedicated partition (raid0 on single disk, ~30 GiB usable), encrypted with its own pool recovery key.
Update system: stable channel, configurable check frequency, auto_reboot: false (updates download but do not reboot automatically).
MAC-dependent boot: IncusOS uses udev rules to rename the network interface by MAC address at boot. Changing the VM's MAC address will cause ERROR timed out waiting for udev to rename interface(s) and a boot hang.

Lab cluster state (post-exercises)

Node	VMID	IP	IncusOS Version
node-01	400	192.168.102.140	202602240349
node-02	401	192.168.102.141	202602240349
node-03	402	192.168.102.142	202602240349

All 3 nodes updated to 202602240349 during the break-fix exercises. node-01 was already on this version; node-02 and node-03 were on 202602230420 and updated independently (proving rolling updates work).

Partition & Disk Layout

Each node has a single SCSI disk (QEMU HARDDISK) with this layout:

block-beta
    columns 1
    disk["SCSI Disk · 64-100 GiB · QEMU HARDDISK"]
    p1["Partition 1 — EFI System Partition"]
    pA["Partition A — Root filesystem (active or standby)"]
    pB["Partition B — Root filesystem (standby or active)"]
    pX["Partition X — Swap (encrypted, TPM-unlocked)"]
    p11["Partition 11 — ZFS data pool 'local'"]

    style disk fill:#f5f5f5,color:#333,stroke:#999
    style p1 fill:#E69F00,color:#fff,stroke:#b87d00
    style pA fill:#009E73,color:#fff,stroke:#007a5e
    style pB fill:#56B4E9,color:#fff,stroke:#3a8fbf
    style pX fill:#CC79A7,color:#fff,stroke:#a36088
    style p11 fill:#0072B2,color:#fff,stroke:#005a8e

Key observations from the storage API:

Partition 11 is the ZFS data partition, hosting pool "local".
Pool "local": raid0, single vdev, 30-66 GiB depending on disk size, contains volume "incus" (Incus database, images, instances).
Root + swap: both encrypted, both unlocked by TPM at boot. No manual key entry needed unless TPM state is corrupted.

Security Chain

The boot security chain, as validated through the API:

flowchart TD
    uefi["UEFI Firmware"]
    sb["Secure Boot validation<br/>PK → KEK → db certs"]
    kernel["Signed kernel + initrd loaded"]
    tpm["TPM measured boot<br/>PCR measurements recorded"]
    validate["TPM validates measurements"]
    root["Root partition decrypted"]
    swap["Swap partition decrypted"]
    boot["System boots into trusted state"]

    uefi --> sb --> kernel --> tpm --> validate
    validate --> root & swap --> boot

    classDef security fill:#CC79A7,color:#fff,stroke:#a36088
    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e

    class uefi,sb,kernel security
    class tpm,validate security
    class root,swap network
    class boot node

API-reported security state:

Property	Value
TPM status	ok
Secure Boot	enabled, enforced
System state	trusted
Encryption recovery keys	1 key (retrievable via API)
Pool recovery keys	1 key for pool "local"

Recovery keys are a safety net. If TPM state is corrupted (e.g., by a hard-stop during first boot), the encryption recovery key allows manual unlock. The pool recovery key allows ZFS pool import on a different system.

Services Configuration

Discovered via /os/1.0/services/*:

Service	Status	Notes
OVN	enabled	Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels
iSCSI	disabled
LVM	disabled
Multipath	disabled
NVMe	disabled
Tailscale	disabled
USB/IP	disabled

OVN is the only enabled service beyond Incus itself, providing the overlay network for cross-node container connectivity and network policies.

Network Configuration

Each node has a single management interface:

Property	Value
Interface name	mgmt
Role	management + cluster traffic
Addressing	Static
Subnet	/22 (192.168.100.0/22)
DNS	192.168.100.1
Gateway	192.168.100.1

Node IP assignments:

node-01: 192.168.102.140/22
node-02: 192.168.102.141/22
node-03: 192.168.102.142/22

The management interface carries both management traffic (API, cluster heartbeats) and OVN Geneve tunnel traffic. In production, these should be separated.

Update Mechanics (Observed and Tested)

The IncusOS update system uses A/B partitions for safe, rollback-capable updates. Observations from querying the update API and live testing:

No pending update: os_version and os_version_next are identical. This means no update has been downloaded or is waiting for reboot.
Pending update: os_version_next differs from os_version, indicating a new build has been downloaded to the standby partition. The update status shows: "IncusOS has been updated to version YYYYMMDDHHMI".
Auto-reboot disabled: auto_reboot: false means updates download to the standby partition but the node continues running the current version until explicitly rebooted.
needs_reboot: true confirms a downloaded update is waiting for reboot to activate.
Independent node updates: node-01 was on 202602240349 while nodes 02/03 were on 202602230420. Each node checks for and applies updates independently. There is no cluster-wide coordinated update.
Check frequency is configurable via API: Use PUT to /os/1.0/system/update with {"config":{"check_frequency":"6h"}}. Default may be "never" on some nodes -- verify and set explicitly. POST to trigger an immediate check returns 501 (not implemented).
Boot-time update check: When a node reboots, IncusOS checks for updates during the boot sequence. node-03 updated from 202602230420 to 202602240349 during a hard-stop/restart cycle, even though the update timer had not yet fired.

Update lifecycle (observed)

flowchart TD
    config["check_frequency set via<br/>PUT /os/1.0/system/update"]
    timer["Timer fires<br/>(or boot-time check)"]
    query["Query stable channel"]
    download["Download rootfs to<br/>standby partition"]
    ready["os_version_next updated<br/>needs_reboot = true"]

    config --> timer --> query --> download --> ready

    ready -->|auto_reboot| reboot
    ready -->|manual| wait["Wait for<br/>manual reboot"]
    wait --> reboot["Reboot from<br/>new partition"]

    reboot --> tpm["TPM re-measures<br/>validates boot chain"]
    tpm -->|valid| active["New partition<br/>becomes active"]
    tpm -->|invalid| rollback["Rollback to<br/>previous partition"]

    classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef lb fill:#E69F00,color:#fff,stroke:#b87d00

    class config,timer,query mgmt
    class download,ready network
    class reboot,tpm,wait lb
    class active node
    class rollback fill:#D55E00,color:#fff,stroke:#a34a00

Break-Fix Exercise Results

These exercises were executed on 2026-02-24 against the 3-node lab cluster. All exercises followed the safety rules (see Safety Rules section below).

Exercise 1: Normal Update Observation

Goal: Observe the full update lifecycle -- download, reboot, A/B partition switch, version verification.

Executed: 2026-02-24

Pre-exercise state

Node	Version	Check Frequency	Last Check
node-01	202602240349	6h	2026-02-24T15:05
node-02	202602230420	never	never
node-03	202602230420	never	never

Key finding: nodes 02/03 had check_frequency: never by default, which is why they never downloaded the available update. The check frequency must be explicitly configured via the API.

Steps and observations

Changed check frequency on nodes 02/03 via API PUT to 6h.
The update timer is not immediately responsive -- setting 10s or 6h does not trigger an immediate check. POST returns 501 (not implemented).
node-03 updated during a hard-stop/restart (Exercise 4 triggered the reboot). The boot-time check found and applied the update.
node-02 download observed: After setting frequency to 6h, the node eventually checked and downloaded the update:
- Status: "IncusOS has been updated to version 202602240349"
- os_version: 202602230420, os_version_next: 202602240349
- needs_reboot: true
Rebooted node-02 (graceful, via Proxmox API). The database-leader role seamlessly transferred during the reboot.
Post-reboot: node-02 running 202602240349, needs_reboot: false.

Timing

Event	Duration
node-02 reboot (offline → online)	~50s
Database-leader role transfer	seamless (no cluster disruption)
All instances on node-02 auto-restart	within boot time

Result

All 3 nodes successfully updated to 202602240349. The A/B partition scheme works as documented -- updates download to the standby partition and activate on reboot.

Status: Completed successfully.

Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update)

Goal: Verify that IncusOS rolls back to the previous partition after a failed update (simulated by hard-stopping the VM during update).

Status: Not executable. All 3 nodes were already on the latest version (202602240349) after Exercises 1 and 4. No pending update was available to interrupt.

To execute in the future: Wait for a new IncusOS build to be published to the stable channel, then:

Take Proxmox snapshot of node-03 (VMID 402)
Trigger update check (change frequency, wait for download)
During early reboot phase (after download, during boot), hard-stop the VM via proxmox-api POST /nodes/pve/qemu/402/status/stop
Start the VM and observe which partition boots (A or B)
Check os_version -- should be the pre-update version if rollback worked

Exercise 3: Network Isolation

Goal: Observe how the cluster handles a node losing network connectivity -- OVN tunnel loss, cluster membership changes, and automatic recovery on reconnection.

Executed: 2026-02-24

Pre-exercise state

All 3 nodes healthy, 20 instances running. node-03 hosted 6 instances including ovn-central and node-exp-03.

Steps and observations

Evacuated workloads from node-03 via incus cluster evacuate. 3 instances migrated to nodes 01/02, 3 stopped in place.

Disconnected NIC via Proxmox API:

proxmox-api PUT /nodes/pve/qemu/402/config \
  --data-urlencode 'net0=virtio=BC:24:11:11:6E:F9,bridge=vmbr0,tag=69,link_down=1'

Cluster detection: node-03 detected as OFFLINE within ~20s of NIC disconnect. Heartbeat message shows exact last heartbeat timestamp.
Critical finding -- OVN gateway is a SPOF: The OVN router's external gateway was scheduled on node-03's chassis. When node-03 lost network:
- East-west traffic (container ↔ container on OVN): still worked
- North-south traffic (OVN ↔ physical LAN): completely broken
- Grafana from LAN: unreachable (OVN forward goes through gateway)
- Monitoring → management network (SNAT): broken
- Monitoring → other OVN containers (10.10.10.x): still working
- No gateway failover occurred even after 3+ minutes of waiting. This is a single point of failure in the current OVN configuration.
NIC reconnection gotcha: Using Proxmox PUT /config with -d flag regenerates the MAC address because curl interprets : in the MAC as URL parameters. Must use --data-urlencode to preserve the MAC address. Changing the MAC causes IncusOS boot failure: ERROR timed out waiting for udev to rename interface(s)
Recovery after correct MAC restored:
- Cluster rejoin: ~26s after VM start
- OVN gateway recovery: ~75s (north-south traffic restored)

Key findings

Metric	Value
Detection time	~20s (heartbeat timeout)
OVN east-west during isolation	Working
OVN north-south during isolation	Broken (no failover)
Cluster quorum	2/3 maintained
Recovery (cluster rejoin)	~26s
Recovery (OVN gateway)	~75s

Lessons learned

OVN gateway HA: The OVN logical router gateway chassis does not automatically failover when a node goes offline. In production, this would need to be addressed with gateway chassis groups or redundant uplinks. This is likely an Incus OVN configuration issue, not an IncusOS limitation.
Proxmox NIC manipulation: Use --data-urlencode for any Proxmox API PUT that includes MAC addresses. The -d flag with raw data corrupts the MAC, regenerating a new one.
IncusOS MAC dependency: The boot process uses udev rules tied to the NIC's MAC address. A MAC change = boot failure. This is important for VM migration, NIC replacement, or Proxmox config changes.

Status: Completed with significant findings.

Exercise 4: Full Node Failure

Goal: Verify that the cluster survives a complete node loss and maintains operations with 2/3 quorum. Measure cluster rejoin time after node recovery.

Executed: 2026-02-24

Pre-exercise state

All 3 nodes healthy. node-03 hosted 6 instances. No evacuation was performed before the hard-stop (to test realistic failure behavior).

Steps and observations

Hard-stopped node-03 via Proxmox API at 16:12:19 CET:
```
proxmox-api POST /nodes/pve/qemu/402/status/stop
```
Detection: Cluster marked node-03 OFFLINE within ~20s. The status message includes the exact last heartbeat timestamp: No heartbeat for 26.79s (2026-02-24 15:12:10 UTC)
Cluster behavior with node-03 down:
- 2/3 quorum maintained -- node-01 and node-02 "Fully operational"
- All 14 instances on nodes 01/02 remained RUNNING
- 6 instances on node-03 went to ERROR state
- incus cluster list, incus list, incus exec all continued working
- Storage pool listing worked normally
- OVN operations continued despite network.ovn.northbound_connection pointing to node-03 (192.168.102.142:6641). The Incus database layer handles OVN NB database failover transparently.
Prometheus impact: 7/9 targets UP (node-03 Incus metrics + node-exp-03 down)
Grafana: accessible from LAN (monitoring container on node-02)
Restarted node-03 at 16:17:17 CET. Recovery timeline:
- +50s: Cluster shows node-03 "Fully operational"
- All 6 instances on node-03 auto-restarted (RUNNING within 1-2 min)
- 9/9 Prometheus targets back to UP
Bonus finding: node-03 updated from 202602230420 to 202602240349 during the reboot! IncusOS performs an update check at boot time, found the newer version, and activated it.

Timing

Event	Time
Hard-stop → OFFLINE detection	~20s
Cluster operations during failure	Fully functional (2/3 quorum)
VM start → cluster rejoin	~50s
VM start → all instances RUNNING	~2 min
VM start → 9/9 Prometheus targets	~3 min

Key findings

Hard-stop of a non-leader node is safe (TPM state preserved for already-provisioned nodes, as expected).
Cluster quorum with 2/3 nodes provides full operational capability.
Instance auto-restart on node recovery is automatic -- no manual intervention needed.
OVN NB database failover is transparent (even when the configured connection target is the downed node).
Boot-time update checks can apply pending updates during recovery.

Status: Completed successfully.

Safety Rules

These rules are non-negotiable for all break-fix exercises:

Never hard-stop during first boot. First boot seals TPM measurements and writes encryption keys. Interrupting this corrupts TPM state permanently, requiring full reinstallation.
One node at a time. A 3-node cluster requires 2/3 quorum. Never take down more than one node simultaneously, or the cluster loses quorum and all operations halt.
Proxmox snapshots before every destructive test. Snapshot the target node's VM (VMID 400-402) before any exercise that involves stopping, disconnecting, or modifying the node. Note: the API token may lack VM.Snapshot permissions -- verify before relying on this.
Verify cluster health before and after. Run incusos-health --all (or equivalent API checks) before starting an exercise and after completing it. Do not proceed if the cluster is already degraded.
Monitor via Grafana during all exercises. Visual monitoring catches issues that API polling might miss (e.g., OVN tunnel flapping, storage I/O spikes).
Target node-03 for destructive tests. node-03 (VMID 402) is the preferred target because it is not the cluster leader and typically has the fewest workloads. Evacuate before testing.
Keep recovery keys accessible. The encryption recovery key and ZFS pool recovery key should be retrieved and stored securely before any exercise that might corrupt TPM state.
Never change MAC addresses on IncusOS VMs. IncusOS uses udev rules tied to the NIC MAC. Changing the MAC causes a boot hang. When using Proxmox API for NIC operations, always use --data-urlencode to preserve the MAC address.

Helper Script: incusos-health

The incusos/helpers/incusos-health script queries the IncusOS API on cluster nodes to report system state. It is the primary tool for verifying cluster health before and after break-fix exercises.

Usage

incusos/helpers/incusos-health [ACTION] [OPTIONS]

Actions

Action	Description
`--status`	Basic system info: version, hostname, TPM status, Secure Boot
`--partitions`	Disk and partition layout, A/B partition state
`--tpm`	TPM details, Secure Boot certificates, encryption keys
`--services`	Enabled/disabled services (OVN, iSCSI, LVM, etc.)
`--network`	Network interface configuration, DNS, gateway
`--update`	Update channel, check frequency, pending update status
`--all`	Run all of the above in sequence

Example output workflow

# Before exercise: verify all nodes healthy
incusos/helpers/incusos-health --all

# After exercise: verify recovery
incusos/helpers/incusos-health --status   # Quick version check
incusos/helpers/incusos-health --all      # Full health report

API Reference

All IncusOS system information is available via the REST API on each node.

Endpoint	Method	Returns
`/os/1.0`	GET	Version, hostname, basic system info
`/os/1.0/system/security`	GET	TPM status, Secure Boot, encryption keys
`/os/1.0/system/storage`	GET	Disks, partitions, ZFS pools
`/os/1.0/system/resources`	GET	CPU, memory, hardware info
`/os/1.0/system/network`	GET	Interfaces, DNS, routes
`/os/1.0/system/update`	GET	Update channel, version, pending updates
`/os/1.0/system/update`	PUT	Change update config (check_frequency, auto_reboot, channel)
`/os/1.0/services/ovn`	GET	OVN configuration and status
`/os/1.0/services/iscsi`	GET	iSCSI configuration
`/os/1.0/services/lvm`	GET	LVM configuration
`/os/1.0/services/multipath`	GET	Multipath configuration
`/os/1.0/services/nvme`	GET	NVMe-oF configuration
`/os/1.0/services/tailscale`	GET	Tailscale VPN configuration
`/os/1.0/services/usbip`	GET	USB/IP configuration

The API listens on the management interface, HTTPS, port 8443 (same as Incus). Authentication uses the Incus client certificate.

Update config via API

# Change check frequency (accepted values: "never", "1h", "6h", "12h", "24h")
curl -sk --cert "$CERT" --key "$KEY" -X PUT \
  https://NODE_IP:8443/os/1.0/system/update \
  -H "Content-Type: application/json" \
  -d '{"config":{"auto_reboot":false,"channel":"stable","check_frequency":"6h"}}'

# Note: POST to trigger immediate check returns 501 (not implemented)

21 KiB Raw Blame History

IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing

IncusOS Architecture

Lab cluster state (post-exercises)

Partition & Disk Layout

Security Chain

Services Configuration

Network Configuration

Update Mechanics (Observed and Tested)

Update lifecycle (observed)

Break-Fix Exercise Results

Exercise 1: Normal Update Observation

Pre-exercise state

Steps and observations

Timing

Result

Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update)

Exercise 3: Network Isolation

Pre-exercise state

Steps and observations

Key findings

Lessons learned

Exercise 4: Full Node Failure

Pre-exercise state

Steps and observations

Timing

Key findings

Safety Rules

Helper Script: incusos-health

Usage

Actions

Example output workflow

API Reference

Update config via API

21 KiB

Raw Blame History