---
paths:
  - "incusos/helpers/incusos-health"
  - "notes/incusos-break-fix.md"
---

# IncusOS Immutability & Break-Fix Context

## Architecture

- IncusOS is an immutable OS with A/B partition scheme for safe updates.
- Version format: `YYYYMMDDHHMI` (build timestamp, e.g., `202602240349`).
- TPM-based full disk encryption: root + swap volumes unlocked via TPM
  measured boot. Secure Boot enforced with PK, KEK, and 2x db certs.
- ZFS storage pool "local" on partition 11, contains volume "incus".
- Single SCSI disk per node (64 GiB in lab, QEMU HARDDISK).

## Lab cluster nodes

| Node | VMID | IP | Version |
|------|------|----|---------|
| node-01 | 900 | 192.168.102.140 | 202602240349 |
| node-02 | 901 | 192.168.102.141 | 202602230420 |
| node-03 | 902 | 192.168.102.142 | 202602230420 |

- Management + cluster traffic on interface "mgmt" (static IPs, /22 subnet).
- DNS + gateway: 192.168.100.1.
- OVN enabled, all other services disabled.

## IncusOS API endpoints

All on port 8443, authenticated with Incus client certificate:

- `/os/1.0` -- version, hostname, basic info
- `/os/1.0/system/security` -- TPM, Secure Boot, encryption/pool recovery keys
- `/os/1.0/system/storage` -- disks, partitions, ZFS pools
- `/os/1.0/system/resources` -- CPU, memory, hardware
- `/os/1.0/system/network` -- interfaces, DNS, routes
- `/os/1.0/system/update` -- channel, frequency, pending updates
- `/os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}`

## incusos-health helper

`incusos/helpers/incusos-health` queries the IncusOS API across all
cluster nodes. Actions:

| Flag | Purpose |
|------|---------|
| `--status` | Version, hostname, TPM status, Secure Boot |
| `--partitions` | Disk/partition layout, A/B state |
| `--tpm` | TPM details, Secure Boot certs, encryption keys |
| `--services` | Enabled/disabled services |
| `--network` | Interface config, DNS, gateway |
| `--update` | Channel, frequency, pending updates |
| `--all` | All of the above |

## Safety rules (MANDATORY)

- **NEVER hard-stop during first boot** -- corrupts TPM permanently,
  requires full reinstallation. Hard-stop is safe ONLY for already-
  provisioned nodes (TPM state already sealed).
- **One node at a time** for break-fix exercises. 3-node cluster needs
  2/3 quorum. Taking down 2 nodes = cluster failure.
- **Proxmox snapshots** before every destructive test (VMID range 900-939).
- **Target node-03** (VMID 902) for destructive tests -- not the leader,
  fewest workloads.
- **Verify health** with `incusos-health --all` before and after exercises.
- **Monitor via Grafana** during all exercises.

## Update mechanics

- `os_version` = current running version.
- `os_version_next` = version on standby partition (same as current if
  no update pending, different if update downloaded).
- `needs_reboot` = true when update is downloaded and waiting for reboot.
- `auto_reboot: false` in lab -- updates download but don't reboot.
- Check frequency: 6 hours, stable channel.
- Nodes update independently (rolling, not coordinated).

## Break-fix exercises (defined, not yet executed)

1. **Normal Update Observation** -- watch download, reboot, A/B switch
2. **Simulated Failed Update** -- hard-stop node-03 mid-update, expect rollback
3. **Network Isolation** -- disconnect node-03 NIC, observe OVN tunnel loss
4. **Full Node Failure** -- hard-stop node-03, verify 2/3 quorum holds

See `notes/incusos-break-fix.md` for full exercise details, prerequisites,
expected behavior, and recovery steps.