89 lines
3.4 KiB
Markdown
89 lines
3.4 KiB
Markdown
---
|
|
paths:
|
|
- "incusos/helpers/incusos-health"
|
|
- "notes/incusos-break-fix.md"
|
|
---
|
|
|
|
# IncusOS Immutability & Break-Fix Context
|
|
|
|
## Architecture
|
|
|
|
- IncusOS is an immutable OS with A/B partition scheme for safe updates.
|
|
- Version format: `YYYYMMDDHHMI` (build timestamp, e.g., `202602240349`).
|
|
- TPM-based full disk encryption: root + swap volumes unlocked via TPM
|
|
measured boot. Secure Boot enforced with PK, KEK, and 2x db certs.
|
|
- ZFS storage pool "local" on partition 11, contains volume "incus".
|
|
- Single SCSI disk per node (64 GiB in lab, QEMU HARDDISK).
|
|
|
|
## Lab cluster nodes
|
|
|
|
| Node | VMID | IP | Version |
|
|
|------|------|----|---------|
|
|
| node-01 | 900 | 192.168.102.140 | 202602240349 |
|
|
| node-02 | 901 | 192.168.102.141 | 202602230420 |
|
|
| node-03 | 902 | 192.168.102.142 | 202602230420 |
|
|
|
|
- Management + cluster traffic on interface "mgmt" (static IPs, /22 subnet).
|
|
- DNS + gateway: 192.168.100.1.
|
|
- OVN enabled, all other services disabled.
|
|
|
|
## IncusOS API endpoints
|
|
|
|
All on port 8443, authenticated with Incus client certificate:
|
|
|
|
- `/os/1.0` -- version, hostname, basic info
|
|
- `/os/1.0/system/security` -- TPM, Secure Boot, encryption/pool recovery keys
|
|
- `/os/1.0/system/storage` -- disks, partitions, ZFS pools
|
|
- `/os/1.0/system/resources` -- CPU, memory, hardware
|
|
- `/os/1.0/system/network` -- interfaces, DNS, routes
|
|
- `/os/1.0/system/update` -- channel, frequency, pending updates
|
|
- `/os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}`
|
|
|
|
## incusos-health helper
|
|
|
|
`incusos/helpers/incusos-health` queries the IncusOS API across all
|
|
cluster nodes. Actions:
|
|
|
|
| Flag | Purpose |
|
|
|------|---------|
|
|
| `--status` | Version, hostname, TPM status, Secure Boot |
|
|
| `--partitions` | Disk/partition layout, A/B state |
|
|
| `--tpm` | TPM details, Secure Boot certs, encryption keys |
|
|
| `--services` | Enabled/disabled services |
|
|
| `--network` | Interface config, DNS, gateway |
|
|
| `--update` | Channel, frequency, pending updates |
|
|
| `--all` | All of the above |
|
|
|
|
## Safety rules (MANDATORY)
|
|
|
|
- **NEVER hard-stop during first boot** -- corrupts TPM permanently,
|
|
requires full reinstallation. Hard-stop is safe ONLY for already-
|
|
provisioned nodes (TPM state already sealed).
|
|
- **One node at a time** for break-fix exercises. 3-node cluster needs
|
|
2/3 quorum. Taking down 2 nodes = cluster failure.
|
|
- **Proxmox snapshots** before every destructive test (VMID range 900-939).
|
|
- **Target node-03** (VMID 902) for destructive tests -- not the leader,
|
|
fewest workloads.
|
|
- **Verify health** with `incusos-health --all` before and after exercises.
|
|
- **Monitor via Grafana** during all exercises.
|
|
|
|
## Update mechanics
|
|
|
|
- `os_version` = current running version.
|
|
- `os_version_next` = version on standby partition (same as current if
|
|
no update pending, different if update downloaded).
|
|
- `needs_reboot` = true when update is downloaded and waiting for reboot.
|
|
- `auto_reboot: false` in lab -- updates download but don't reboot.
|
|
- Check frequency: 6 hours, stable channel.
|
|
- Nodes update independently (rolling, not coordinated).
|
|
|
|
## Break-fix exercises (defined, not yet executed)
|
|
|
|
1. **Normal Update Observation** -- watch download, reboot, A/B switch
|
|
2. **Simulated Failed Update** -- hard-stop node-03 mid-update, expect rollback
|
|
3. **Network Isolation** -- disconnect node-03 NIC, observe OVN tunnel loss
|
|
4. **Full Node Failure** -- hard-stop node-03, verify 2/3 quorum holds
|
|
|
|
See `notes/incusos-break-fix.md` for full exercise details, prerequisites,
|
|
expected behavior, and recovery steps.
|