# IncusOS Break-Fix Lab -- Immutability Exploration & Resilience Testing

IncusOS is an immutable, purpose-built operating system for running Incus
clusters. This guide documents what we discovered about its internals through
the `/os/1.0` API, and records the results of break-fix exercises testing
cluster resilience in a safe lab environment.

All observations come from a 3-node Proxmox-hosted cluster (VMID 400-402)
running IncusOS with virtual TPM, Secure Boot, and OVN networking.

## IncusOS Architecture

IncusOS is an immutable OS designed for a single purpose: running Incus.
Key architectural properties discovered via the `/os/1.0` API:

- **Immutable root filesystem** with A/B partition scheme. The running system
  boots from one partition; updates install to the other. Rollback is automatic
  if the new partition fails validation.
- **TPM-based full disk encryption** -- root and swap volumes are encrypted
  and unlocked automatically via TPM measured boot. No passphrase required
  during normal operation.
- **Secure Boot enforced** with 4 certificates in the UEFI firmware:
  - PK (Platform Key) -- root of trust
  - KEK (Key Exchange Key) -- authorizes db updates
  - db (2025 signing cert) -- validates current signed binaries
  - db (2026 signing cert) -- validates next-year signed binaries
- **Version format**: `YYYYMMDDHHMI` (build timestamp, not semver).
  Example: `202602240349` = 2026-02-24 at 03:49 UTC.
- **ZFS storage**: pool "local" on a dedicated partition (raid0 on single
  disk, ~30 GiB usable), encrypted with its own pool recovery key.
- **Update system**: stable channel, configurable check frequency,
  `auto_reboot: false` (updates download but do not reboot automatically).
- **MAC-dependent boot**: IncusOS uses udev rules to rename the network
  interface by MAC address at boot. Changing the VM's MAC address will cause
  `ERROR timed out waiting for udev to rename interface(s)` and a boot hang.

### Lab cluster state (post-exercises)

| Node | VMID | IP | IncusOS Version |
|------|------|----|-----------------|
| node-01 | 400 | 192.168.102.140 | 202602240349 |
| node-02 | 401 | 192.168.102.141 | 202602240349 |
| node-03 | 402 | 192.168.102.142 | 202602240349 |

All 3 nodes updated to `202602240349` during the break-fix exercises.
node-01 was already on this version; node-02 and node-03 were on
`202602230420` and updated independently (proving rolling updates work).

## Partition & Disk Layout

Each node has a single SCSI disk (QEMU HARDDISK) with this layout:

```mermaid
block-beta
    columns 1
    disk["SCSI Disk · 64-100 GiB · QEMU HARDDISK"]
    p1["Partition 1 — EFI System Partition"]
    pA["Partition A — Root filesystem (active or standby)"]
    pB["Partition B — Root filesystem (standby or active)"]
    pX["Partition X — Swap (encrypted, TPM-unlocked)"]
    p11["Partition 11 — ZFS data pool 'local'"]

    style disk fill:#f5f5f5,color:#333,stroke:#999
    style p1 fill:#E69F00,color:#fff,stroke:#b87d00
    style pA fill:#009E73,color:#fff,stroke:#007a5e
    style pB fill:#56B4E9,color:#fff,stroke:#3a8fbf
    style pX fill:#CC79A7,color:#fff,stroke:#a36088
    style p11 fill:#0072B2,color:#fff,stroke:#005a8e
```

Key observations from the storage API:

- **Partition 11** is the ZFS data partition, hosting pool "local".
- **Pool "local"**: raid0, single vdev, 30-66 GiB depending on disk size,
  contains volume "incus" (Incus database, images, instances).
- **Root + swap**: both encrypted, both unlocked by TPM at boot.
  No manual key entry needed unless TPM state is corrupted.

## Security Chain

The boot security chain, as validated through the API:

```mermaid
flowchart TD
    uefi["UEFI Firmware"]
    sb["Secure Boot validation<br/>PK → KEK → db certs"]
    kernel["Signed kernel + initrd loaded"]
    tpm["TPM measured boot<br/>PCR measurements recorded"]
    validate["TPM validates measurements"]
    root["Root partition decrypted"]
    swap["Swap partition decrypted"]
    boot["System boots into trusted state"]

    uefi --> sb --> kernel --> tpm --> validate
    validate --> root & swap --> boot

    classDef security fill:#CC79A7,color:#fff,stroke:#a36088
    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e

    class uefi,sb,kernel security
    class tpm,validate security
    class root,swap network
    class boot node
```

API-reported security state:

| Property | Value |
|----------|-------|
| TPM status | ok |
| Secure Boot | enabled, enforced |
| System state | trusted |
| Encryption recovery keys | 1 key (retrievable via API) |
| Pool recovery keys | 1 key for pool "local" |

**Recovery keys** are a safety net. If TPM state is corrupted (e.g., by a
hard-stop during first boot), the encryption recovery key allows manual
unlock. The pool recovery key allows ZFS pool import on a different system.

## Services Configuration

Discovered via `/os/1.0/services/*`:

| Service | Status | Notes |
|---------|--------|-------|
| OVN | enabled | Connects to cluster DB at `tcp:192.168.102.142:6642`, Geneve tunnels |
| iSCSI | disabled | |
| LVM | disabled | |
| Multipath | disabled | |
| NVMe | disabled | |
| Tailscale | disabled | |
| USB/IP | disabled | |

OVN is the only enabled service beyond Incus itself, providing the overlay
network for cross-node container connectivity and network policies.

## Network Configuration

Each node has a single management interface:

| Property | Value |
|----------|-------|
| Interface name | mgmt |
| Role | management + cluster traffic |
| Addressing | Static |
| Subnet | /22 (192.168.100.0/22) |
| DNS | 192.168.100.1 |
| Gateway | 192.168.100.1 |

Node IP assignments:
- node-01: `192.168.102.140/22`
- node-02: `192.168.102.141/22`
- node-03: `192.168.102.142/22`

The management interface carries both management traffic (API, cluster
heartbeats) and OVN Geneve tunnel traffic. In production, these should
be separated.

## Update Mechanics (Observed and Tested)

The IncusOS update system uses A/B partitions for safe, rollback-capable
updates. Observations from querying the update API and live testing:

1. **No pending update**: `os_version` and `os_version_next` are identical.
   This means no update has been downloaded or is waiting for reboot.

2. **Pending update**: `os_version_next` differs from `os_version`,
   indicating a new build has been downloaded to the standby partition.
   The update status shows: `"IncusOS has been updated to version YYYYMMDDHHMI"`.

3. **Auto-reboot disabled**: `auto_reboot: false` means updates download
   to the standby partition but the node continues running the current
   version until explicitly rebooted.

4. **`needs_reboot: true`** confirms a downloaded update is waiting for
   reboot to activate.

5. **Independent node updates**: node-01 was on `202602240349` while
   nodes 02/03 were on `202602230420`. Each node checks for and applies
   updates independently. There is no cluster-wide coordinated update.

6. **Check frequency is configurable via API**: Use PUT to
   `/os/1.0/system/update` with `{"config":{"check_frequency":"6h"}}`.
   Default may be "never" on some nodes -- verify and set explicitly.
   POST to trigger an immediate check returns 501 (not implemented).

7. **Boot-time update check**: When a node reboots, IncusOS checks for
   updates during the boot sequence. node-03 updated from `202602230420`
   to `202602240349` during a hard-stop/restart cycle, even though the
   update timer had not yet fired.

### Update lifecycle (observed)

```mermaid
flowchart TD
    config["check_frequency set via<br/>PUT /os/1.0/system/update"]
    timer["Timer fires<br/>(or boot-time check)"]
    query["Query stable channel"]
    download["Download rootfs to<br/>standby partition"]
    ready["os_version_next updated<br/>needs_reboot = true"]

    config --> timer --> query --> download --> ready

    ready -->|auto_reboot| reboot
    ready -->|manual| wait["Wait for<br/>manual reboot"]
    wait --> reboot["Reboot from<br/>new partition"]

    reboot --> tpm["TPM re-measures<br/>validates boot chain"]
    tpm -->|valid| active["New partition<br/>becomes active"]
    tpm -->|invalid| rollback["Rollback to<br/>previous partition"]

    classDef mgmt fill:#CC79A7,color:#fff,stroke:#a36088
    classDef node fill:#009E73,color:#fff,stroke:#007a5e
    classDef network fill:#0072B2,color:#fff,stroke:#005a8e
    classDef lb fill:#E69F00,color:#fff,stroke:#b87d00

    class config,timer,query mgmt
    class download,ready network
    class reboot,tpm,wait lb
    class active node
    class rollback fill:#D55E00,color:#fff,stroke:#a34a00
```

---

## Break-Fix Exercise Results

These exercises were executed on 2026-02-24 against the 3-node lab cluster.
All exercises followed the safety rules (see Safety Rules section below).

---

### Exercise 1: Normal Update Observation

**Goal**: Observe the full update lifecycle -- download, reboot, A/B
partition switch, version verification.

**Executed**: 2026-02-24

#### Pre-exercise state

| Node | Version | Check Frequency | Last Check |
|------|---------|----------------|------------|
| node-01 | 202602240349 | 6h | 2026-02-24T15:05 |
| node-02 | 202602230420 | never | never |
| node-03 | 202602230420 | never | never |

Key finding: nodes 02/03 had `check_frequency: never` by default, which
is why they never downloaded the available update. The check frequency
must be explicitly configured via the API.

#### Steps and observations

1. Changed check frequency on nodes 02/03 via API PUT to `6h`.
2. The update timer is not immediately responsive -- setting `10s` or `6h`
   does not trigger an immediate check. POST returns 501 (not implemented).
3. **node-03 updated during a hard-stop/restart** (Exercise 4 triggered
   the reboot). The boot-time check found and applied the update.
4. **node-02 download observed**: After setting frequency to `6h`, the
   node eventually checked and downloaded the update:
   - Status: `"IncusOS has been updated to version 202602240349"`
   - `os_version: 202602230420`, `os_version_next: 202602240349`
   - `needs_reboot: true`
5. Rebooted node-02 (graceful, via Proxmox API). The database-leader
   role seamlessly transferred during the reboot.
6. Post-reboot: node-02 running `202602240349`, `needs_reboot: false`.

#### Timing

| Event | Duration |
|-------|----------|
| node-02 reboot (offline → online) | ~50s |
| Database-leader role transfer | seamless (no cluster disruption) |
| All instances on node-02 auto-restart | within boot time |

#### Result

All 3 nodes successfully updated to `202602240349`. The A/B partition
scheme works as documented -- updates download to the standby partition
and activate on reboot.

**Status**: Completed successfully.

---

### Exercise 2: Simulated Failed Update (Hard-Stop Mid-Update)

**Goal**: Verify that IncusOS rolls back to the previous partition after
a failed update (simulated by hard-stopping the VM during update).

**Status**: Not executable. All 3 nodes were already on the latest
version (`202602240349`) after Exercises 1 and 4. No pending update
was available to interrupt.

**To execute in the future**: Wait for a new IncusOS build to be
published to the stable channel, then:
1. Take Proxmox snapshot of node-03 (VMID 402)
2. Trigger update check (change frequency, wait for download)
3. During early reboot phase (after download, during boot), hard-stop
   the VM via `proxmox-api POST /nodes/pve/qemu/402/status/stop`
4. Start the VM and observe which partition boots (A or B)
5. Check `os_version` -- should be the pre-update version if rollback worked

---

### Exercise 3: Network Isolation

**Goal**: Observe how the cluster handles a node losing network
connectivity -- OVN tunnel loss, cluster membership changes, and
automatic recovery on reconnection.

**Executed**: 2026-02-24

#### Pre-exercise state

All 3 nodes healthy, 20 instances running. node-03 hosted 6 instances
including `ovn-central` and `node-exp-03`.

#### Steps and observations

1. **Evacuated workloads** from node-03 via `incus cluster evacuate`.
   3 instances migrated to nodes 01/02, 3 stopped in place.

2. **Disconnected NIC** via Proxmox API:
   ```
   proxmox-api PUT /nodes/pve/qemu/402/config \
     --data-urlencode 'net0=virtio=BC:24:11:11:6E:F9,bridge=vmbr0,tag=69,link_down=1'
   ```

3. **Cluster detection**: node-03 detected as OFFLINE within ~20s of
   NIC disconnect. Heartbeat message shows exact last heartbeat timestamp.

4. **Critical finding -- OVN gateway is a SPOF**: The OVN router's
   external gateway was scheduled on node-03's chassis. When node-03
   lost network:
   - **East-west traffic** (container ↔ container on OVN): **still worked**
   - **North-south traffic** (OVN ↔ physical LAN): **completely broken**
   - Grafana from LAN: unreachable (OVN forward goes through gateway)
   - Monitoring → management network (SNAT): broken
   - Monitoring → other OVN containers (10.10.10.x): still working
   - **No gateway failover occurred** even after 3+ minutes of waiting.
     This is a single point of failure in the current OVN configuration.

5. **NIC reconnection gotcha**: Using Proxmox `PUT /config` with `-d`
   flag regenerates the MAC address because curl interprets `:` in the
   MAC as URL parameters. **Must use `--data-urlencode`** to preserve
   the MAC address. Changing the MAC causes IncusOS boot failure:
   `ERROR timed out waiting for udev to rename interface(s)`

6. **Recovery after correct MAC restored**:
   - Cluster rejoin: ~26s after VM start
   - OVN gateway recovery: ~75s (north-south traffic restored)

#### Key findings

| Metric | Value |
|--------|-------|
| Detection time | ~20s (heartbeat timeout) |
| OVN east-west during isolation | Working |
| OVN north-south during isolation | Broken (no failover) |
| Cluster quorum | 2/3 maintained |
| Recovery (cluster rejoin) | ~26s |
| Recovery (OVN gateway) | ~75s |

#### Lessons learned

- **OVN gateway HA**: The OVN logical router gateway chassis does not
  automatically failover when a node goes offline. In production, this
  would need to be addressed with gateway chassis groups or redundant
  uplinks. This is likely an Incus OVN configuration issue, not an
  IncusOS limitation.
- **Proxmox NIC manipulation**: Use `--data-urlencode` for any Proxmox
  API PUT that includes MAC addresses. The `-d` flag with raw data
  corrupts the MAC, regenerating a new one.
- **IncusOS MAC dependency**: The boot process uses udev rules tied to
  the NIC's MAC address. A MAC change = boot failure. This is important
  for VM migration, NIC replacement, or Proxmox config changes.

**Status**: Completed with significant findings.

---

### Exercise 4: Full Node Failure

**Goal**: Verify that the cluster survives a complete node loss and
maintains operations with 2/3 quorum. Measure cluster rejoin time
after node recovery.

**Executed**: 2026-02-24

#### Pre-exercise state

All 3 nodes healthy. node-03 hosted 6 instances. No evacuation was
performed before the hard-stop (to test realistic failure behavior).

#### Steps and observations

1. **Hard-stopped node-03** via Proxmox API at 16:12:19 CET:
   ```
   proxmox-api POST /nodes/pve/qemu/402/status/stop
   ```

2. **Detection**: Cluster marked node-03 OFFLINE within ~20s. The status
   message includes the exact last heartbeat timestamp:
   `No heartbeat for 26.79s (2026-02-24 15:12:10 UTC)`

3. **Cluster behavior with node-03 down**:
   - 2/3 quorum maintained -- node-01 and node-02 "Fully operational"
   - All 14 instances on nodes 01/02 remained RUNNING
   - 6 instances on node-03 went to ERROR state
   - `incus cluster list`, `incus list`, `incus exec` all continued working
   - Storage pool listing worked normally
   - **OVN operations continued** despite `network.ovn.northbound_connection`
     pointing to node-03 (192.168.102.142:6641). The Incus database layer
     handles OVN NB database failover transparently.

4. **Prometheus impact**: 7/9 targets UP (node-03 Incus metrics + node-exp-03 down)

5. **Grafana**: accessible from LAN (monitoring container on node-02)

6. **Restarted node-03** at 16:17:17 CET. Recovery timeline:
   - +50s: Cluster shows node-03 "Fully operational"
   - All 6 instances on node-03 auto-restarted (RUNNING within 1-2 min)
   - 9/9 Prometheus targets back to UP

7. **Bonus finding**: node-03 updated from `202602230420` to `202602240349`
   during the reboot! IncusOS performs an update check at boot time,
   found the newer version, and activated it.

#### Timing

| Event | Time |
|-------|------|
| Hard-stop → OFFLINE detection | ~20s |
| Cluster operations during failure | Fully functional (2/3 quorum) |
| VM start → cluster rejoin | ~50s |
| VM start → all instances RUNNING | ~2 min |
| VM start → 9/9 Prometheus targets | ~3 min |

#### Key findings

- Hard-stop of a non-leader node is safe (TPM state preserved for
  already-provisioned nodes, as expected).
- Cluster quorum with 2/3 nodes provides full operational capability.
- Instance auto-restart on node recovery is automatic -- no manual
  intervention needed.
- OVN NB database failover is transparent (even when the configured
  connection target is the downed node).
- Boot-time update checks can apply pending updates during recovery.

**Status**: Completed successfully.

---

## Safety Rules

These rules are non-negotiable for all break-fix exercises:

1. **Never hard-stop during first boot.** First boot seals TPM measurements
   and writes encryption keys. Interrupting this corrupts TPM state
   permanently, requiring full reinstallation.

2. **One node at a time.** A 3-node cluster requires 2/3 quorum. Never
   take down more than one node simultaneously, or the cluster loses
   quorum and all operations halt.

3. **Proxmox snapshots before every destructive test.** Snapshot the
   target node's VM (VMID 400-402) before any exercise that involves
   stopping, disconnecting, or modifying the node. Note: the API token
   may lack `VM.Snapshot` permissions -- verify before relying on this.

4. **Verify cluster health before and after.** Run `incusos-health --all`
   (or equivalent API checks) before starting an exercise and after
   completing it. Do not proceed if the cluster is already degraded.

5. **Monitor via Grafana during all exercises.** Visual monitoring catches
   issues that API polling might miss (e.g., OVN tunnel flapping,
   storage I/O spikes).

6. **Target node-03 for destructive tests.** node-03 (VMID 402) is the
   preferred target because it is not the cluster leader and typically
   has the fewest workloads. Evacuate before testing.

7. **Keep recovery keys accessible.** The encryption recovery key and
   ZFS pool recovery key should be retrieved and stored securely before
   any exercise that might corrupt TPM state.

8. **Never change MAC addresses on IncusOS VMs.** IncusOS uses udev
   rules tied to the NIC MAC. Changing the MAC causes a boot hang.
   When using Proxmox API for NIC operations, always use
   `--data-urlencode` to preserve the MAC address.

## Helper Script: incusos-health

The `incusos/helpers/incusos-health` script queries the IncusOS API on
cluster nodes to report system state. It is the primary tool for verifying
cluster health before and after break-fix exercises.

### Usage

```bash
incusos/helpers/incusos-health [ACTION] [OPTIONS]
```

### Actions

| Action | Description |
|--------|-------------|
| `--status` | Basic system info: version, hostname, TPM status, Secure Boot |
| `--partitions` | Disk and partition layout, A/B partition state |
| `--tpm` | TPM details, Secure Boot certificates, encryption keys |
| `--services` | Enabled/disabled services (OVN, iSCSI, LVM, etc.) |
| `--network` | Network interface configuration, DNS, gateway |
| `--update` | Update channel, check frequency, pending update status |
| `--all` | Run all of the above in sequence |

### Example output workflow

```bash
# Before exercise: verify all nodes healthy
incusos/helpers/incusos-health --all

# After exercise: verify recovery
incusos/helpers/incusos-health --status   # Quick version check
incusos/helpers/incusos-health --all      # Full health report
```

## API Reference

All IncusOS system information is available via the REST API on each node.

| Endpoint | Method | Returns |
|----------|--------|---------|
| `/os/1.0` | GET | Version, hostname, basic system info |
| `/os/1.0/system/security` | GET | TPM status, Secure Boot, encryption keys |
| `/os/1.0/system/storage` | GET | Disks, partitions, ZFS pools |
| `/os/1.0/system/resources` | GET | CPU, memory, hardware info |
| `/os/1.0/system/network` | GET | Interfaces, DNS, routes |
| `/os/1.0/system/update` | GET | Update channel, version, pending updates |
| `/os/1.0/system/update` | PUT | Change update config (check_frequency, auto_reboot, channel) |
| `/os/1.0/services/ovn` | GET | OVN configuration and status |
| `/os/1.0/services/iscsi` | GET | iSCSI configuration |
| `/os/1.0/services/lvm` | GET | LVM configuration |
| `/os/1.0/services/multipath` | GET | Multipath configuration |
| `/os/1.0/services/nvme` | GET | NVMe-oF configuration |
| `/os/1.0/services/tailscale` | GET | Tailscale VPN configuration |
| `/os/1.0/services/usbip` | GET | USB/IP configuration |

The API listens on the management interface, HTTPS, port 8443 (same as
Incus). Authentication uses the Incus client certificate.

### Update config via API

```bash
# Change check frequency (accepted values: "never", "1h", "6h", "12h", "24h")
curl -sk --cert "$CERT" --key "$KEY" -X PUT \
  https://NODE_IP:8443/os/1.0/system/update \
  -H "Content-Type: application/json" \
  -d '{"config":{"auto_reboot":false,"channel":"stable","check_frequency":"6h"}}'

# Note: POST to trigger immediate check returns 501 (not implemented)
```
