97 lines
4.0 KiB
Markdown
97 lines
4.0 KiB
Markdown
---
|
|
paths:
|
|
- "incusos/helpers/incusos-health"
|
|
- "notes/incusos-break-fix.md"
|
|
---
|
|
|
|
# IncusOS Immutability & Break-Fix Context
|
|
|
|
## Architecture
|
|
|
|
- IncusOS is an immutable OS with A/B partition scheme for safe updates.
|
|
- Version format: `YYYYMMDDHHMI` (build timestamp, e.g., `202602240349`).
|
|
- TPM-based full disk encryption: root + swap volumes unlocked via TPM
|
|
measured boot. Secure Boot enforced with PK, KEK, and 2x db certs.
|
|
- ZFS storage pool "local" on partition 11, contains volume "incus".
|
|
- Single SCSI disk per node (64-100 GiB in lab, QEMU HARDDISK).
|
|
- MAC-dependent boot: udev renames NIC by MAC. Changing MAC = boot hang.
|
|
|
|
## Lab cluster nodes
|
|
|
|
| Node | VMID | IP | Version |
|
|
|------|------|----|---------|
|
|
| node-01 | 400 | 192.168.102.140 | 202602240349 |
|
|
| node-02 | 401 | 192.168.102.141 | 202602240349 |
|
|
| node-03 | 402 | 192.168.102.142 | 202602240349 |
|
|
|
|
- Management + cluster traffic on interface "mgmt" (static IPs, /22 subnet).
|
|
- DNS + gateway: 192.168.100.1.
|
|
- OVN enabled, all other services disabled.
|
|
|
|
## IncusOS API endpoints
|
|
|
|
All on port 8443, authenticated with Incus client certificate:
|
|
|
|
- `/os/1.0` -- version, hostname, basic info
|
|
- `/os/1.0/system/security` -- TPM, Secure Boot, encryption/pool recovery keys
|
|
- `/os/1.0/system/storage` -- disks, partitions, ZFS pools
|
|
- `/os/1.0/system/resources` -- CPU, memory, hardware
|
|
- `/os/1.0/system/network` -- interfaces, DNS, routes
|
|
- `/os/1.0/system/update` -- GET: state; PUT: change config (frequency, auto_reboot)
|
|
- `/os/1.0/services/{ovn,iscsi,lvm,multipath,nvme,tailscale,usbip}`
|
|
|
|
## incusos-health helper
|
|
|
|
`incusos/helpers/incusos-health` queries the IncusOS API across all
|
|
cluster nodes. Actions:
|
|
|
|
| Flag | Purpose |
|
|
|------|---------|
|
|
| `--status` | Version, hostname, TPM status, Secure Boot |
|
|
| `--partitions` | Disk/partition layout, A/B state |
|
|
| `--tpm` | TPM details, Secure Boot certs, encryption keys |
|
|
| `--services` | Enabled/disabled services |
|
|
| `--network` | Interface config, DNS, gateway |
|
|
| `--update` | Channel, frequency, pending updates |
|
|
| `--all` | All of the above |
|
|
|
|
## Safety rules (MANDATORY)
|
|
|
|
- **NEVER hard-stop during first boot** -- corrupts TPM permanently,
|
|
requires full reinstallation. Hard-stop is safe ONLY for already-
|
|
provisioned nodes (TPM state already sealed).
|
|
- **One node at a time** for break-fix exercises. 3-node cluster needs
|
|
2/3 quorum. Taking down 2 nodes = cluster failure.
|
|
- **Proxmox snapshots** before every destructive test (VMID 400-402).
|
|
Note: API token may lack `VM.Snapshot` permission.
|
|
- **Target node-03** (VMID 402) for destructive tests -- not the leader,
|
|
fewest workloads.
|
|
- **Verify health** with `incusos-health --all` before and after exercises.
|
|
- **Monitor via Grafana** during all exercises.
|
|
- **Never change MAC addresses** on IncusOS VMs. Use `--data-urlencode`
|
|
when modifying NIC config via Proxmox API.
|
|
|
|
## Update mechanics (tested)
|
|
|
|
- `os_version` = current running version.
|
|
- `os_version_next` = version on standby partition (same as current if
|
|
no update pending, different if update downloaded).
|
|
- `needs_reboot` = true when update is downloaded and waiting for reboot.
|
|
- `auto_reboot: false` in lab -- updates download but don't reboot.
|
|
- Check frequency configurable via PUT. Default may be "never" on some nodes.
|
|
- POST to trigger immediate check returns 501 (not implemented).
|
|
- Boot-time update check: IncusOS checks for updates during boot sequence.
|
|
- Nodes update independently (rolling, not coordinated).
|
|
|
|
## Break-fix exercise results (2026-02-24)
|
|
|
|
1. **Normal Update Observation** -- Completed. Observed download, reboot,
|
|
A/B switch on node-02. Database-leader transfer seamless during reboot.
|
|
2. **Simulated Failed Update** -- Blocked (no update available to interrupt).
|
|
3. **Network Isolation** -- Completed. OVN gateway is SPOF (no failover).
|
|
East-west traffic works, north-south broken. Recovery: 26s cluster, 75s gateway.
|
|
4. **Full Node Failure** -- Completed. 2/3 quorum held. Detection: ~20s.
|
|
Recovery: ~50s to cluster rejoin. All instances auto-restarted.
|
|
|
|
See `notes/incusos-break-fix.md` for full exercise details and results.
|