maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
incus-contrib/notes/incusos-break-fix.md

422 lines
16 KiB
Markdown
Raw Blame History

# IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing
IncusOS is an immutable, purpose-built operating system for running Incus
clusters. This guide documents what we discovered about its internals through
the `/os/1.0` API, and defines a catalog of break-fix exercises for testing
cluster resilience in a safe lab environment.
All observations come from a 3-node Proxmox-hosted cluster (VMID 900-902)
running IncusOS with virtual TPM, Secure Boot, and OVN networking.
## IncusOS Architecture
IncusOS is an immutable OS designed for a single purpose: running Incus.
Key architectural properties discovered via the `/os/1.0` API:
- **Immutable root filesystem** with A/B partition scheme. The running system
boots from one partition; updates install to the other. Rollback is automatic
if the new partition fails validation.
- **TPM-based full disk encryption** -- root and swap volumes are encrypted
and unlocked automatically via TPM measured boot. No passphrase required
during normal operation.
- **Secure Boot enforced** with 4 certificates in the UEFI firmware:
- PK (Platform Key) -- root of trust
- KEK (Key Exchange Key) -- authorizes db updates
- db (2025 signing cert) -- validates current signed binaries
- db (2026 signing cert) -- validates next-year signed binaries
- **Version format**: `YYYYMMDDHHMI` (build timestamp, not semver).
Example: `202602240349` = 2026-02-24 at 03:49 UTC.
- **ZFS storage**: pool "local" on a dedicated partition (raid0 on single
disk, ~30 GiB usable), encrypted with its own pool recovery key.
- **Update system**: stable channel, 6-hour check frequency,
`auto_reboot: false` (updates download but do not reboot automatically).
### Lab cluster state
| Node | VMID | IP | IncusOS Version |
|------|------|----|-----------------|
| node-01 | 900 | 192.168.102.140 | 202602240349 |
| node-02 | 901 | 192.168.102.141 | 202602230420 |
| node-03 | 902 | 192.168.102.142 | 202602230420 |
node-01 received a newer build than nodes 02/03, proving that nodes update
independently (rolling updates, not cluster-wide atomic upgrades).
## Partition & Disk Layout
Each node has a single 64 GiB SCSI disk (QEMU HARDDISK) with this layout:
```
+------------------------------------------------------+
| SCSI Disk (64 GiB, QEMU HARDDISK) |
|------------------------------------------------------|
| Partition 1 EFI System Partition |
| Partition A Root filesystem (active or standby) |
| Partition B Root filesystem (standby or active) |
| Partition X Swap (encrypted, TPM-unlocked) |
| Partition 11 ZFS data pool ("local") |
+------------------------------------------------------+
```
Key observations from the storage API:
- **Partition 11** is the ZFS data partition, hosting pool "local".
- **Pool "local"**: raid0, single vdev, ~30 GiB total, contains volume
"incus" (~10 GiB used for Incus database, images, instances).
- **Root + swap**: both encrypted, both unlocked by TPM at boot.
No manual key entry needed unless TPM state is corrupted.
## Security Chain
The boot security chain, as validated through the API:
```
UEFI firmware
--> Secure Boot certificate validation (PK -> KEK -> db certs)
--> Signed kernel + initrd loaded
--> TPM measured boot (PCR measurements recorded)
--> TPM validates measurements match expected policy
--> Root partition decrypted and mounted
--> Swap partition decrypted and activated
--> System boots into trusted state
```
API-reported security state:
| Property | Value |
|----------|-------|
| TPM status | ok |
| Secure Boot | enabled, enforced |
| System state | trusted |
| Encryption recovery keys | 1 key (retrievable via API) |
| Pool recovery keys | 1 key for pool "local" |
**Recovery keys** are a safety net. If TPM state is corrupted (e.g., by a
hard-stop during first boot), the encryption recovery key allows manual
unlock. The pool recovery key allows ZFS pool import on a different system.
## Services Configuration
Discovered via `/os/1.0/services/*`:
| Service | Status | Notes |
|---------|--------|-------|
| OVN | enabled | Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels |
| iSCSI | disabled | |
| LVM | disabled | |
| Multipath | disabled | |
| NVMe | disabled | |
| Tailscale | disabled | |
| USB/IP | disabled | |
OVN is the only enabled service beyond Incus itself, providing the overlay
network for cross-node container connectivity and network policies.
## Network Configuration
Each node has a single management interface:
| Property | Value |
|----------|-------|
| Interface name | mgmt |
| Role | management + cluster traffic |
| Addressing | Static |
| Subnet | /22 (192.168.100.0/22) |
| DNS | 192.168.100.1 |
| Gateway | 192.168.100.1 |
Node IP assignments:
- node-01: `192.168.102.140/22`
- node-02: `192.168.102.141/22`
- node-03: `192.168.102.142/22`
The management interface carries both management traffic (API, cluster
heartbeats) and OVN Geneve tunnel traffic. In production, these should
be separated.
## Update Mechanics (Observed)
The IncusOS update system uses A/B partitions for safe, rollback-capable
updates. Observations from querying the update API:
1. **No pending update**: `os_version` and `os_version_next` are identical.
This means no update has been downloaded or is waiting for reboot.
2. **Pending update**: `os_version_next` would differ from `os_version`,
indicating a new build has been downloaded to the standby partition.
3. **Auto-reboot disabled**: `auto_reboot: false` means updates download
to the standby partition but the node continues running the current
version until explicitly rebooted (or until an admin triggers reboot
via API / UI).
4. **No reboot needed**: `needs_reboot: false` confirms no downloaded
update is waiting for a reboot to activate.
5. **Independent node updates**: node-01 is on `202602240349` while
nodes 02/03 are on `202602230420`. This proves each node checks for
and applies updates independently. There is no cluster-wide coordinated
update mechanism at the OS level.
6. **Check frequency**: 6 hours. The stable channel is checked
automatically on this interval.
### Update lifecycle (theoretical)
```
Check timer fires (every 6h)
--> Query stable channel for new version
--> Download new rootfs to standby partition (A or B)
--> os_version_next updated, needs_reboot = true
--> (if auto_reboot) Reboot automatically
--> (if !auto_reboot) Wait for manual reboot
--> On reboot: boot from new partition
--> TPM re-measures, validates new boot chain
--> If valid: new partition becomes active
--> If invalid: rollback to previous partition
```
## Break-Fix Exercise Catalog
These exercises are designed to test IncusOS cluster resilience in the
Proxmox lab. All exercises follow strict safety rules (see Safety Rules
section below).
**Current status**: All exercises are defined but not yet executed.
---
### Exercise 1: Normal Update Observation
**Goal**: Observe the full update lifecycle -- download, reboot, A/B
partition switch, version verification.
**Prerequisites**:
- Proxmox snapshot of all cluster nodes (VMID 900-902)
- Grafana monitoring active (observe cluster metrics during update)
- Verify cluster health with `incusos-health --all`
**Steps**:
1. Record current versions: `incusos-health --update`
2. Trigger update via Operations Center UI, or wait for 6h check interval
3. Monitor via Grafana for download activity and node state changes
4. When `needs_reboot: true`, reboot one node at a time
5. After reboot, verify new version: `incusos-health --status`
6. Confirm A/B partition switch: `incusos-health --partitions`
**What to observe**:
- Download phase duration
- Reboot duration (typically 30-60s for IncusOS)
- Cluster behavior while one node reboots (2/3 quorum maintained)
- New version number in `os_version`
- Previous version still available on standby partition
**Status**: Defined, not yet executed.
---
### Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update)
**Goal**: Verify that IncusOS rolls back to the previous partition after
a failed update (simulated by hard-stopping the VM during update).
**Prerequisites**:
- Proxmox snapshot of node-03 (VMID 902) -- non-leader, fewest workloads
- Verify node-03 has no critical workloads
- Confirm 3/3 nodes healthy before starting
**Target node**: node-03 (VMID 902) ONLY. Never the cluster leader.
**Steps**:
1. Take Proxmox snapshot of node-03: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/snapshot -d '{"snapname":"pre-break-fix-2"}'`
2. Trigger an update on node-03 (if one is available)
3. During the update download or early reboot phase, hard-stop node-03
via Proxmox API: `incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop`
4. Wait 10 seconds, then start node-03:
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start`
5. Monitor boot via console screenshots: `incusos/helpers/proxmox-screenshot 902`
6. After boot, check version: `incusos-health --status` on node-03
**Expected behavior**:
- Node boots from the previous (known-good) partition
- Failed partition is marked as bad / not bootable
- `os_version` returns to the pre-update version
- Node rejoins the cluster automatically
**Safety**:
- ONLY node-03 -- maintains 2/3 quorum (node-01 + node-02 continue)
- Do NOT hard-stop during *first boot* (corrupts TPM permanently)
- If recovery fails, restore from Proxmox snapshot
**Status**: Defined, not yet executed.
---
### Exercise 3: Network Isolation
**Goal**: Observe how the cluster handles a node losing network
connectivity -- OVN tunnel loss, cluster membership changes, and
automatic recovery on reconnection.
**Prerequisites**:
- Evacuate all workloads from node-03 before disconnecting
- Grafana monitoring active (watch OVN tunnel metrics, cluster events)
- Verify cluster health: `incusos-health --all`
**Steps**:
1. Evacuate workloads from node-03:
`incus cluster evacuate oc-node-03 --target oc-node-01`
2. Disconnect node-03 NIC via Proxmox API:
`incusos/helpers/proxmox-api PUT /nodes/pve/qemu/902/config -d '{"net0":"virtio=...,link_down=1"}'`
3. Observe in Grafana: OVN Geneve tunnel loss, cluster detecting missing node
4. Wait 2-5 minutes for cluster to mark node-03 as offline
5. Reconnect NIC: set `link_down=0` via Proxmox API
6. Monitor recovery: node-03 should rejoin cluster, OVN tunnels re-establish
7. Measure total recovery time from reconnection to healthy state
**Expected behavior**:
- Cluster detects node-03 offline within heartbeat timeout
- OVN tunnels from/to node-03 fail (Geneve encap packets lost)
- Cluster continues operating with 2/3 quorum
- On reconnection: node-03 rejoins, tunnels re-establish, workloads
can be restored
**Recovery**:
- If node-03 does not rejoin: check OVN service status, restart if needed
- If cluster state is inconsistent: restore from Proxmox snapshot
**Status**: Defined, not yet executed.
---
### Exercise 4: Full Node Failure
**Goal**: Verify that the cluster survives a complete node loss and
maintains operations with 2/3 quorum. Measure cluster rejoin time
after node recovery.
**Prerequisites**:
- Proxmox snapshot of node-03 (VMID 902)
- Evacuate ALL workloads from node-03
- Verify cluster health: `incusos-health --all`
**Steps**:
1. Evacuate workloads: `incus cluster evacuate oc-node-03`
2. Take Proxmox snapshot of node-03
3. Hard-stop node-03 via Proxmox:
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/stop`
4. Verify cluster continues operating:
- `incus cluster list` should show 2 online, 1 offline
- Existing workloads on node-01/02 remain accessible
- OVN gateway should failover if node-03 was a gateway
5. Wait 5 minutes, observing Grafana metrics
6. Restart node-03:
`incusos/helpers/proxmox-api POST /nodes/pve/qemu/902/status/start`
7. Monitor boot: `incusos/helpers/proxmox-screenshot 902`
8. Measure time from start to cluster rejoin (node shows online)
9. Restore workloads: `incus cluster restore oc-node-03`
**Expected behavior**:
- 2/3 quorum maintained -- all cluster operations continue
- OVN gateway failover occurs if node-03 was elected gateway
- After restart: node-03 boots, TPM unlocks, Incus starts, rejoins cluster
- Typical rejoin time: 1-3 minutes after boot
**Safety**:
- This is NOT a first-boot scenario -- hard-stop is safe for already-
provisioned nodes (TPM state is already sealed)
- One node at a time ONLY
- If node-03 fails to rejoin after restart, restore snapshot
**Status**: Defined, not yet executed.
## Safety Rules
These rules are non-negotiable for all break-fix exercises:
1. **Never hard-stop during first boot.** First boot seals TPM measurements
and writes encryption keys. Interrupting this corrupts TPM state
permanently, requiring full reinstallation.
2. **One node at a time.** A 3-node cluster requires 2/3 quorum. Never
take down more than one node simultaneously, or the cluster loses
quorum and all operations halt.
3. **Proxmox snapshots before every destructive test.** Snapshot the
target node's VM (VMID 900-939) before any exercise that involves
stopping, disconnecting, or modifying the node.
4. **Verify cluster health before and after.** Run `incusos-health --all`
(or equivalent API checks) before starting an exercise and after
completing it. Do not proceed if the cluster is already degraded.
5. **Monitor via Grafana during all exercises.** Visual monitoring catches
issues that API polling might miss (e.g., OVN tunnel flapping,
storage I/O spikes).
6. **Target node-03 for destructive tests.** node-03 (VMID 902) is the
preferred target because it is not the cluster leader and typically
has the fewest workloads. Evacuate before testing.
7. **Keep recovery keys accessible.** The encryption recovery key and
ZFS pool recovery key should be retrieved and stored securely before
any exercise that might corrupt TPM state.
## Helper Script: incusos-health
The `incusos/helpers/incusos-health` script queries the IncusOS API on
cluster nodes to report system state. It is the primary tool for verifying
cluster health before and after break-fix exercises.
### Usage
```bash
incusos/helpers/incusos-health [ACTION] [OPTIONS]
```
### Actions
| Action | Description |
|--------|-------------|
| `--status` | Basic system info: version, hostname, TPM status, Secure Boot |
| `--partitions` | Disk and partition layout, A/B partition state |
| `--tpm` | TPM details, Secure Boot certificates, encryption keys |
| `--services` | Enabled/disabled services (OVN, iSCSI, LVM, etc.) |
| `--network` | Network interface configuration, DNS, gateway |
| `--update` | Update channel, check frequency, pending update status |
| `--all` | Run all of the above in sequence |
### Example output workflow
```bash
# Before exercise: verify all nodes healthy
incusos/helpers/incusos-health --all
# After exercise: verify recovery
incusos/helpers/incusos-health --status # Quick version check
incusos/helpers/incusos-health --all # Full health report
```
## API Reference
All IncusOS system information is available via the REST API on each node.
| Endpoint | Returns |
|----------|---------|
| `/os/1.0` | Version, hostname, basic system info |
| `/os/1.0/system/security` | TPM status, Secure Boot, encryption keys |
| `/os/1.0/system/storage` | Disks, partitions, ZFS pools |
| `/os/1.0/system/resources` | CPU, memory, hardware info |
| `/os/1.0/system/network` | Interfaces, DNS, routes |
| `/os/1.0/system/update` | Update channel, version, pending updates |
| `/os/1.0/services/ovn` | OVN configuration and status |
| `/os/1.0/services/iscsi` | iSCSI configuration |
| `/os/1.0/services/lvm` | LVM configuration |
| `/os/1.0/services/multipath` | Multipath configuration |
| `/os/1.0/services/nvme` | NVMe-oF configuration |
| `/os/1.0/services/tailscale` | Tailscale VPN configuration |
| `/os/1.0/services/usbip` | USB/IP configuration |
The API listens on the management interface, HTTPS, port 8443 (same as
Incus). Authentication uses the Incus client certificate.
Reference in New Issue View Git Blame Copy Permalink