maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/incusos-break-fix.md

564 lines
21 KiB
Markdown
Raw Blame History

# IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing
IncusOS is an immutable, purpose-built operating system for running Incus
clusters. This guide documents what we discovered about its internals through
the `/os/1.0` API, and records the results of break-fix exercises testing
cluster resilience in a safe lab environment.
All observations come from a 3-node Proxmox-hosted cluster (VMID 400-402)
running IncusOS with virtual TPM, Secure Boot, and OVN networking.
## IncusOS Architecture
IncusOS is an immutable OS designed for a single purpose: running Incus.
Key architectural properties discovered via the `/os/1.0` API:
- **Immutable root filesystem** with A/B partition scheme. The running system
boots from one partition; updates install to the other. Rollback is automatic
if the new partition fails validation.
- **TPM-based full disk encryption** -- root and swap volumes are encrypted
and unlocked automatically via TPM measured boot. No passphrase required
during normal operation.
- **Secure Boot enforced** with 4 certificates in the UEFI firmware:
- PK (Platform Key) -- root of trust
- KEK (Key Exchange Key) -- authorizes db updates
- db (2025 signing cert) -- validates current signed binaries
- db (2026 signing cert) -- validates next-year signed binaries
- **Version format**: `YYYYMMDDHHMI` (build timestamp, not semver).
Example: `202602240349` = 2026-02-24 at 03:49 UTC.
- **ZFS storage**: pool "local" on a dedicated partition (raid0 on single
disk, ~30 GiB usable), encrypted with its own pool recovery key.
- **Update system**: stable channel, configurable check frequency,
`auto_reboot: false` (updates download but do not reboot automatically).
- **MAC-dependent boot**: IncusOS uses udev rules to rename the network
interface by MAC address at boot. Changing the VM's MAC address will cause
`ERROR timed out waiting for udev to rename interface(s)` and a boot hang.
### Lab cluster state (post-exercises)
| Node | VMID | IP | IncusOS Version |
|------|------|----|-----------------|
| node-01 | 400 | 192.168.102.140 | 202602240349 |
| node-02 | 401 | 192.168.102.141 | 202602240349 |
| node-03 | 402 | 192.168.102.142 | 202602240349 |
All 3 nodes updated to `202602240349` during the break-fix exercises.
node-01 was already on this version; node-02 and node-03 were on
`202602230420` and updated independently (proving rolling updates work).
## Partition & Disk Layout
Each node has a single SCSI disk (QEMU HARDDISK) with this layout:
```mermaid
block-beta
columns 1
disk["SCSI Disk · 64-100 GiB · QEMU HARDDISK"]
p1["Partition 1 — EFI System Partition"]
pA["Partition A — Root filesystem (active or standby)"]
pB["Partition B — Root filesystem (standby or active)"]
pX["Partition X — Swap (encrypted, TPM-unlocked)"]
p11["Partition 11 — ZFS data pool 'local'"]
style disk fill:#f5f5f5,color:#333,stroke:#999
style p1 fill:#E69F00,color:#fff,stroke:#b87d00
style pA fill:#009E73,color:#fff,stroke:#007a5e
style pB fill:#56B4E9,color:#fff,stroke:#3a8fbf
style pX fill:#CC79A7,color:#fff,stroke:#a36088
style p11 fill:#0072B2,color:#fff,stroke:#005a8e
```
Key observations from the storage API:
- **Partition 11** is the ZFS data partition, hosting pool "local".
- **Pool "local"**: raid0, single vdev, 30-66 GiB depending on disk size,
contains volume "incus" (Incus database, images, instances).
- **Root + swap**: both encrypted, both unlocked by TPM at boot.
No manual key entry needed unless TPM state is corrupted.
## Security Chain
The boot security chain, as validated through the API:
```mermaid
flowchart TD
uefi["UEFI Firmware"]
sb["Secure Boot validation<br/>PK → KEK → db certs"]
kernel["Signed kernel + initrd loaded"]
tpm["TPM measured boot<br/>PCR measurements recorded"]
validate["TPM validates measurements"]
root["Root partition decrypted"]
swap["Swap partition decrypted"]
boot["System boots into trusted state"]
uefi --> sb --> kernel --> tpm --> validate
validate --> root & swap --> boot
classDef security fill:#CC79A7,color:#fff,stroke:#a36088
classDef node fill:#009E73,color:#fff,stroke:#007a5e
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
class uefi,sb,kernel security
class tpm,validate security
class root,swap network
class boot node
```
API-reported security state:
| Property | Value |
|----------|-------|
| TPM status | ok |
| Secure Boot | enabled, enforced |
| System state | trusted |
| Encryption recovery keys | 1 key (retrievable via API) |
| Pool recovery keys | 1 key for pool "local" |
**Recovery keys** are a safety net. If TPM state is corrupted (e.g., by a
hard-stop during first boot), the encryption recovery key allows manual
unlock. The pool recovery key allows ZFS pool import on a different system.
## Services Configuration
Discovered via `/os/1.0/services/*`:
| Service | Status | Notes |
|---------|--------|-------|
| OVN | enabled | Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels |
| iSCSI | disabled | |
| LVM | disabled | |
| Multipath | disabled | |
| NVMe | disabled | |
| Tailscale | disabled | |
| USB/IP | disabled | |
OVN is the only enabled service beyond Incus itself, providing the overlay
network for cross-node container connectivity and network policies.
## Network Configuration
Each node has a single management interface:
| Property | Value |
|----------|-------|
| Interface name | mgmt |
| Role | management + cluster traffic |
| Addressing | Static |
| Subnet | /22 (192.168.100.0/22) |
| DNS | 192.168.100.1 |
| Gateway | 192.168.100.1 |
Node IP assignments:
- node-01: `192.168.102.140/22`
- node-02: `192.168.102.141/22`
- node-03: `192.168.102.142/22`
The management interface carries both management traffic (API, cluster
heartbeats) and OVN Geneve tunnel traffic. In production, these should
be separated.
## Update Mechanics (Observed and Tested)
The IncusOS update system uses A/B partitions for safe, rollback-capable
updates. Observations from querying the update API and live testing:
1. **No pending update**: `os_version` and `os_version_next` are identical.
This means no update has been downloaded or is waiting for reboot.
2. **Pending update**: `os_version_next` differs from `os_version`,
indicating a new build has been downloaded to the standby partition.
The update status shows: `"IncusOS has been updated to version YYYYMMDDHHMI"`.
3. **Auto-reboot disabled**: `auto_reboot: false` means updates download
to the standby partition but the node continues running the current
version until explicitly rebooted.
4. **`needs_reboot: true`** confirms a downloaded update is waiting for
reboot to activate.
5. **Independent node updates**: node-01 was on `202602240349` while
nodes 02/03 were on `202602230420`. Each node checks for and applies
updates independently. There is no cluster-wide coordinated update.
6. **Check frequency is configurable via API**: Use PUT to
`/os/1.0/system/update` with `{"config":{"check_frequency":"6h"}}`.
Default may be "never" on some nodes -- verify and set explicitly.
POST to trigger an immediate check returns 501 (not implemented).
7. **Boot-time update check**: When a node reboots, IncusOS checks for
updates during the boot sequence. node-03 updated from `202602230420`
to `202602240349` during a hard-stop/restart cycle, even though the
update timer had not yet fired.
### Update lifecycle (observed)
```mermaid
flowchart TD
config["check_frequency set via<br/>PUT /os/1.0/system/update"]
timer["Timer fires<br/>(or boot-time check)"]
query["Query stable channel"]
download["Download rootfs to<br/>standby partition"]
ready["os_version_next updated<br/>needs_reboot = true"]
config --> timer --> query --> download --> ready
ready -->|auto_reboot| reboot
ready -->|manual| wait["Wait for<br/>manual reboot"]
wait --> reboot["Reboot from<br/>new partition"]
reboot --> tpm["TPM re-measures<br/>validates boot chain"]
tpm -->|valid| active["New partition<br/>becomes active"]
tpm -->|invalid| rollback["Rollback to<br/>previous partition"]
classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
classDef node fill:#009E73,color:#fff,stroke:#007a5e
classDef network fill:#0072B2,color:#fff,stroke:#005a8e
classDef lb fill:#E69F00,color:#fff,stroke:#b87d00
class config,timer,query mgmt
class download,ready network
class reboot,tpm,wait lb
class active node
class rollback fill:#D55E00,color:#fff,stroke:#a34a00
```
---
## Break-Fix Exercise Results
These exercises were executed on 2026-02-24 against the 3-node lab cluster.
All exercises followed the safety rules (see Safety Rules section below).
---
### Exercise 1: Normal Update Observation
**Goal**: Observe the full update lifecycle -- download, reboot, A/B
partition switch, version verification.
**Executed**: 2026-02-24
#### Pre-exercise state
| Node | Version | Check Frequency | Last Check |
|------|---------|----------------|------------|
| node-01 | 202602240349 | 6h | 2026-02-24T15:05 |
| node-02 | 202602230420 | never | never |
| node-03 | 202602230420 | never | never |
Key finding: nodes 02/03 had `check_frequency: never` by default, which
is why they never downloaded the available update. The check frequency
must be explicitly configured via the API.
#### Steps and observations
1. Changed check frequency on nodes 02/03 via API PUT to `6h`.
2. The update timer is not immediately responsive -- setting `10s` or `6h`
does not trigger an immediate check. POST returns 501 (not implemented).
3. **node-03 updated during a hard-stop/restart** (Exercise 4 triggered
the reboot). The boot-time check found and applied the update.
4. **node-02 download observed**: After setting frequency to `6h`, the
node eventually checked and downloaded the update:
- Status: `"IncusOS has been updated to version 202602240349"`
- `os_version: 202602230420`, `os_version_next: 202602240349`
- `needs_reboot: true`
5. Rebooted node-02 (graceful, via Proxmox API). The database-leader
role seamlessly transferred during the reboot.
6. Post-reboot: node-02 running `202602240349`, `needs_reboot: false`.
#### Timing
| Event | Duration |
|-------|----------|
| node-02 reboot (offline → online) | ~50s |
| Database-leader role transfer | seamless (no cluster disruption) |
| All instances on node-02 auto-restart | within boot time |
#### Result
All 3 nodes successfully updated to `202602240349`. The A/B partition
scheme works as documented -- updates download to the standby partition
and activate on reboot.
**Status**: Completed successfully.
---
### Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update)
**Goal**: Verify that IncusOS rolls back to the previous partition after
a failed update (simulated by hard-stopping the VM during update).
**Status**: Not executable. All 3 nodes were already on the latest
version (`202602240349`) after Exercises 1 and 4. No pending update
was available to interrupt.
**To execute in the future**: Wait for a new IncusOS build to be
published to the stable channel, then:
1. Take Proxmox snapshot of node-03 (VMID 402)
2. Trigger update check (change frequency, wait for download)
3. During early reboot phase (after download, during boot), hard-stop
the VM via `proxmox-api POST /nodes/pve/qemu/402/status/stop`
4. Start the VM and observe which partition boots (A or B)
5. Check `os_version` -- should be the pre-update version if rollback worked
---
### Exercise 3: Network Isolation
**Goal**: Observe how the cluster handles a node losing network
connectivity -- OVN tunnel loss, cluster membership changes, and
automatic recovery on reconnection.
**Executed**: 2026-02-24
#### Pre-exercise state
All 3 nodes healthy, 20 instances running. node-03 hosted 6 instances
including `ovn-central` and `node-exp-03`.
#### Steps and observations
1. **Evacuated workloads** from node-03 via `incus cluster evacuate`.
3 instances migrated to nodes 01/02, 3 stopped in place.
2. **Disconnected NIC** via Proxmox API:
```
proxmox-api PUT /nodes/pve/qemu/402/config \
--data-urlencode 'net0=virtio=BC:24:11:11:6E:F9,bridge=vmbr0,tag=69,link_down=1'
```
3. **Cluster detection**: node-03 detected as OFFLINE within ~20s of
NIC disconnect. Heartbeat message shows exact last heartbeat timestamp.
4. **Critical finding -- OVN gateway is a SPOF**: The OVN router's
external gateway was scheduled on node-03's chassis. When node-03
lost network:
- **East-west traffic** (container ↔ container on OVN): **still worked**
- **North-south traffic** (OVN ↔ physical LAN): **completely broken**
- Grafana from LAN: unreachable (OVN forward goes through gateway)
- Monitoring → management network (SNAT): broken
- Monitoring → other OVN containers (10.10.10.x): still working
- **No gateway failover occurred** even after 3+ minutes of waiting.
This is a single point of failure in the current OVN configuration.
5. **NIC reconnection gotcha**: Using Proxmox `PUT /config` with `-d`
flag regenerates the MAC address because curl interprets `:` in the
MAC as URL parameters. **Must use `--data-urlencode`** to preserve
the MAC address. Changing the MAC causes IncusOS boot failure:
`ERROR timed out waiting for udev to rename interface(s)`
6. **Recovery after correct MAC restored**:
- Cluster rejoin: ~26s after VM start
- OVN gateway recovery: ~75s (north-south traffic restored)
#### Key findings
| Metric | Value |
|--------|-------|
| Detection time | ~20s (heartbeat timeout) |
| OVN east-west during isolation | Working |
| OVN north-south during isolation | Broken (no failover) |
| Cluster quorum | 2/3 maintained |
| Recovery (cluster rejoin) | ~26s |
| Recovery (OVN gateway) | ~75s |
#### Lessons learned
- **OVN gateway HA**: The OVN logical router gateway chassis does not
automatically failover when a node goes offline. In production, this
would need to be addressed with gateway chassis groups or redundant
uplinks. This is likely an Incus OVN configuration issue, not an
IncusOS limitation.
- **Proxmox NIC manipulation**: Use `--data-urlencode` for any Proxmox
API PUT that includes MAC addresses. The `-d` flag with raw data
corrupts the MAC, regenerating a new one.
- **IncusOS MAC dependency**: The boot process uses udev rules tied to
the NIC's MAC address. A MAC change = boot failure. This is important
for VM migration, NIC replacement, or Proxmox config changes.
**Status**: Completed with significant findings.
---
### Exercise 4: Full Node Failure
**Goal**: Verify that the cluster survives a complete node loss and
maintains operations with 2/3 quorum. Measure cluster rejoin time
after node recovery.
**Executed**: 2026-02-24
#### Pre-exercise state
All 3 nodes healthy. node-03 hosted 6 instances. No evacuation was
performed before the hard-stop (to test realistic failure behavior).
#### Steps and observations
1. **Hard-stopped node-03** via Proxmox API at 16:12:19 CET:
```
proxmox-api POST /nodes/pve/qemu/402/status/stop
```
2. **Detection**: Cluster marked node-03 OFFLINE within ~20s. The status
message includes the exact last heartbeat timestamp:
`No heartbeat for 26.79s (2026-02-24 15:12:10 UTC)`
3. **Cluster behavior with node-03 down**:
- 2/3 quorum maintained -- node-01 and node-02 "Fully operational"
- All 14 instances on nodes 01/02 remained RUNNING
- 6 instances on node-03 went to ERROR state
- `incus cluster list`, `incus list`, `incus exec` all continued working
- Storage pool listing worked normally
- **OVN operations continued** despite `network.ovn.northbound_connection`
pointing to node-03 (192.168.102.142:6641). The Incus database layer
handles OVN NB database failover transparently.
4. **Prometheus impact**: 7/9 targets UP (node-03 Incus metrics + node-exp-03 down)
5. **Grafana**: accessible from LAN (monitoring container on node-02)
6. **Restarted node-03** at 16:17:17 CET. Recovery timeline:
- +50s: Cluster shows node-03 "Fully operational"
- All 6 instances on node-03 auto-restarted (RUNNING within 1-2 min)
- 9/9 Prometheus targets back to UP
7. **Bonus finding**: node-03 updated from `202602230420` to `202602240349`
during the reboot! IncusOS performs an update check at boot time,
found the newer version, and activated it.
#### Timing
| Event | Time |
|-------|------|
| Hard-stop → OFFLINE detection | ~20s |
| Cluster operations during failure | Fully functional (2/3 quorum) |
| VM start → cluster rejoin | ~50s |
| VM start → all instances RUNNING | ~2 min |
| VM start → 9/9 Prometheus targets | ~3 min |
#### Key findings
- Hard-stop of a non-leader node is safe (TPM state preserved for
already-provisioned nodes, as expected).
- Cluster quorum with 2/3 nodes provides full operational capability.
- Instance auto-restart on node recovery is automatic -- no manual
intervention needed.
- OVN NB database failover is transparent (even when the configured
connection target is the downed node).
- Boot-time update checks can apply pending updates during recovery.
**Status**: Completed successfully.
---
## Safety Rules
These rules are non-negotiable for all break-fix exercises:
1. **Never hard-stop during first boot.** First boot seals TPM measurements
and writes encryption keys. Interrupting this corrupts TPM state
permanently, requiring full reinstallation.
2. **One node at a time.** A 3-node cluster requires 2/3 quorum. Never
take down more than one node simultaneously, or the cluster loses
quorum and all operations halt.
3. **Proxmox snapshots before every destructive test.** Snapshot the
target node's VM (VMID 400-402) before any exercise that involves
stopping, disconnecting, or modifying the node. Note: the API token
may lack `VM.Snapshot` permissions -- verify before relying on this.
4. **Verify cluster health before and after.** Run `incusos-health --all`
(or equivalent API checks) before starting an exercise and after
completing it. Do not proceed if the cluster is already degraded.
5. **Monitor via Grafana during all exercises.** Visual monitoring catches
issues that API polling might miss (e.g., OVN tunnel flapping,
storage I/O spikes).
6. **Target node-03 for destructive tests.** node-03 (VMID 402) is the
preferred target because it is not the cluster leader and typically
has the fewest workloads. Evacuate before testing.
7. **Keep recovery keys accessible.** The encryption recovery key and
ZFS pool recovery key should be retrieved and stored securely before
any exercise that might corrupt TPM state.
8. **Never change MAC addresses on IncusOS VMs.** IncusOS uses udev
rules tied to the NIC MAC. Changing the MAC causes a boot hang.
When using Proxmox API for NIC operations, always use
`--data-urlencode` to preserve the MAC address.
## Helper Script: incusos-health
The `incusos/helpers/incusos-health` script queries the IncusOS API on
cluster nodes to report system state. It is the primary tool for verifying
cluster health before and after break-fix exercises.
### Usage
```bash
incusos/helpers/incusos-health [ACTION] [OPTIONS]
```
### Actions
| Action | Description |
|--------|-------------|
| `--status` | Basic system info: version, hostname, TPM status, Secure Boot |
| `--partitions` | Disk and partition layout, A/B partition state |
| `--tpm` | TPM details, Secure Boot certificates, encryption keys |
| `--services` | Enabled/disabled services (OVN, iSCSI, LVM, etc.) |
| `--network` | Network interface configuration, DNS, gateway |
| `--update` | Update channel, check frequency, pending update status |
| `--all` | Run all of the above in sequence |
### Example output workflow
```bash
# Before exercise: verify all nodes healthy
incusos/helpers/incusos-health --all
# After exercise: verify recovery
incusos/helpers/incusos-health --status # Quick version check
incusos/helpers/incusos-health --all # Full health report
```
## API Reference
All IncusOS system information is available via the REST API on each node.
| Endpoint | Method | Returns |
|----------|--------|---------|
| `/os/1.0` | GET | Version, hostname, basic system info |
| `/os/1.0/system/security` | GET | TPM status, Secure Boot, encryption keys |
| `/os/1.0/system/storage` | GET | Disks, partitions, ZFS pools |
| `/os/1.0/system/resources` | GET | CPU, memory, hardware info |
| `/os/1.0/system/network` | GET | Interfaces, DNS, routes |
| `/os/1.0/system/update` | GET | Update channel, version, pending updates |
| `/os/1.0/system/update` | PUT | Change update config (check_frequency, auto_reboot, channel) |
| `/os/1.0/services/ovn` | GET | OVN configuration and status |
| `/os/1.0/services/iscsi` | GET | iSCSI configuration |
| `/os/1.0/services/lvm` | GET | LVM configuration |
| `/os/1.0/services/multipath` | GET | Multipath configuration |
| `/os/1.0/services/nvme` | GET | NVMe-oF configuration |
| `/os/1.0/services/tailscale` | GET | Tailscale VPN configuration |
| `/os/1.0/services/usbip` | GET | USB/IP configuration |
The API listens on the management interface, HTTPS, port 8443 (same as
Incus). Authentication uses the Incus client certificate.
### Update config via API
```bash
# Change check frequency (accepted values: "never", "1h", "6h", "12h", "24h")
curl -sk --cert "$CERT" --key "$KEY" -X PUT \
https://NODE_IP:8443/os/1.0/system/update \
-H "Content-Type: application/json" \
-d '{"config":{"auto_reboot":false,"channel":"stable","check_frequency":"6h"}}'
# Note: POST to trigger immediate check returns 501 (not implemented)
```
Reference in New Issue View Git Blame Copy Permalink