---
paths:
  - "incusos/helpers/incusos-health"
  - "notes/incusos-break-fix.md"
---

# IncusOS Immutability & Break-Fix Context

## Architecture

- IncusOS is an immutable OS with A/B partition scheme for safe updates.
- Version format: `YYYYMMDDHHMI` (build timestamp, e.g., `202602240349`).
- TPM-based full disk encryption: root + swap volumes unlocked via TPM
  measured boot. Secure Boot enforced with PK, KEK, and 2x db certs.
- ZFS storage pool "local" on partition 11, contains volume "incus".
- Single SCSI disk per node (64-100 GiB in lab, QEMU HARDDISK).
- MAC-dependent boot: udev renames NIC by MAC. Changing MAC = boot hang.

## Lab cluster nodes

| Node | VMID | IP | Version |
|------|------|----|---------|
| node-01 | 400 | 192.168.102.140 | 202602240349 |
| node-02 | 401 | 192.168.102.141 | 202602240349 |
| node-03 | 402 | 192.168.102.142 | 202602240349 |

- Management + cluster traffic on interface "mgmt" (static IPs, /22 subnet).
- DNS + gateway: 192.168.100.1.
- OVN enabled, all other services disabled.

## IncusOS API endpoints

All on port 8443, authenticated with Incus client certificate:

- `/os/1.0` -- version, hostname, basic info
- `/os/1.0/system/security` -- TPM, Secure Boot, encryption/pool recovery keys
- `/os/1.0/system/storage` -- disks, partitions, ZFS pools
- `/os/1.0/system/resources` -- CPU, memory, hardware
- `/os/1.0/system/network` -- interfaces, DNS, routes
- `/os/1.0/system/update` -- GET: state; PUT: change config (frequency, auto_reboot)
- `/os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}`

## incusos-health helper

`incusos/helpers/incusos-health` queries the IncusOS API across all
cluster nodes. Actions:

| Flag | Purpose |
|------|---------|
| `--status` | Version, hostname, TPM status, Secure Boot |
| `--partitions` | Disk/partition layout, A/B state |
| `--tpm` | TPM details, Secure Boot certs, encryption keys |
| `--services` | Enabled/disabled services |
| `--network` | Interface config, DNS, gateway |
| `--update` | Channel, frequency, pending updates |
| `--all` | All of the above |

## Safety rules (MANDATORY)

- **NEVER hard-stop during first boot** -- corrupts TPM permanently,
  requires full reinstallation. Hard-stop is safe ONLY for already-
  provisioned nodes (TPM state already sealed).
- **One node at a time** for break-fix exercises. 3-node cluster needs
  2/3 quorum. Taking down 2 nodes = cluster failure.
- **Proxmox snapshots** before every destructive test (VMID 400-402).
  Note: API token may lack `VM.Snapshot` permission.
- **Target node-03** (VMID 402) for destructive tests -- not the leader,
  fewest workloads.
- **Verify health** with `incusos-health --all` before and after exercises.
- **Monitor via Grafana** during all exercises.
- **Never change MAC addresses** on IncusOS VMs. Use `--data-urlencode`
  when modifying NIC config via Proxmox API.

## Update mechanics (tested)

- `os_version` = current running version.
- `os_version_next` = version on standby partition (same as current if
  no update pending, different if update downloaded).
- `needs_reboot` = true when update is downloaded and waiting for reboot.
- `auto_reboot: false` in lab -- updates download but don't reboot.
- Check frequency configurable via PUT. Default may be "never" on some nodes.
- POST to trigger immediate check returns 501 (not implemented).
- Boot-time update check: IncusOS checks for updates during boot sequence.
- Nodes update independently (rolling, not coordinated).

## Break-fix exercise results (2026-02-24)

1. **Normal Update Observation** -- Completed. Observed download, reboot,
   A/B switch on node-02. Database-leader transfer seamless during reboot.
2. **Simulated Failed Update** -- Blocked (no update available to interrupt).
3. **Network Isolation** -- Completed. OVN gateway is SPOF (no failover).
   East-west traffic works, north-south broken. Recovery: 26s cluster, 75s gateway.
4. **Full Node Failure** -- Completed. 2/3 quorum held. Detection: ~20s.
   Recovery: ~50s to cluster rejoin. All instances auto-restarted.

See `notes/incusos-break-fix.md` for full exercise details and results.